2023-09-10T00:33:06.687 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-10T00:33:06.688 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534 --owner scheduled_pdonnell@teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} -- /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/orig.config.yaml 2023-09-10T00:33:06.746 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-10T00:33:06.834 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534 branch: wip-batrick-testing-20230909.145638 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7392534' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-batrick-testing-20230909.145638 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 9670cca5bc8e17916865dc84939edfc10b1f2c00 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 9670cca5bc8e17916865dc84939edfc10b1f2c00 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20230909.145638 sha1: 9670cca5bc8e17916865dc84939edfc10b1f2c00 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5964 sha1: 9670cca5bc8e17916865dc84939edfc10b1f2c00 sleep_before_teardown: 0 subset: 225/256 suite: fs suite_branch: wip-batrick-testing-20230909.145638 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9670cca5bc8e17916865dc84939edfc10b1f2c00 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-09_19:00:20 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1394 2023-09-10T00:33:06.834 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa; will attempt to use it 2023-09-10T00:33:06.835 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa/tasks 2023-09-10T00:33:06.836 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-10T00:33:06.838 INFO:teuthology.task.internal:Checking packages... 2023-09-10T00:33:06.861 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '9670cca5bc8e17916865dc84939edfc10b1f2c00' 2023-09-10T00:33:06.861 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-10T00:33:06.862 INFO:teuthology.packaging:ref: None 2023-09-10T00:33:06.862 INFO:teuthology.packaging:tag: None 2023-09-10T00:33:06.862 INFO:teuthology.packaging:branch: wip-batrick-testing-20230909.145638 2023-09-10T00:33:06.863 INFO:teuthology.packaging:sha1: 9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:33:06.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-batrick-testing-20230909.145638 2023-09-10T00:33:07.079 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6025.g9670cca5 2023-09-10T00:33:07.080 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-10T00:33:07.087 INFO:teuthology.task.internal:no buildpackages task found 2023-09-10T00:33:07.087 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-10T00:33:07.137 INFO:teuthology.task.internal:Saving configuration 2023-09-10T00:33:07.158 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-10T00:33:07.170 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-10T00:33:07.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-10 00:28:12.072759', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-10T00:33:07.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-10 00:28:12.075313', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-10T00:33:07.240 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-10 00:28:12.074198', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-10T00:33:07.240 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-10T00:33:07.252 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-09-10T00:33:07.252 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-09-10T00:33:07.252 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-09-10T00:33:07.252 INFO:teuthology.run_tasks:Running task console_log... 2023-09-10T00:33:07.389 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f967602eca0>, signals=[15]) 2023-09-10T00:33:07.390 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-10T00:33:07.401 INFO:teuthology.task.internal:Opening connections... 2023-09-10T00:33:07.402 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T00:33:07.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:33:07.474 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2023-09-10T00:33:07.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:33:07.542 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2023-09-10T00:33:07.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:33:07.651 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-10T00:33:07.661 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2023-09-10T00:33:07.684 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-09-10T00:33:07.685 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2023-09-10T00:33:07.745 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Red Hat Enterprise Linux" 2023-09-10T00:33:07.745 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8.6 (Ootpa)" 2023-09-10T00:33:07.745 INFO:teuthology.orchestra.run.smithi003.stdout:ID="rhel" 2023-09-10T00:33:07.745 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="fedora" 2023-09-10T00:33:07.746 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8.6" 2023-09-10T00:33:07.746 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2023-09-10T00:33:07.746 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-09-10T00:33:07.746 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-09-10T00:33:07.747 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-09-10T00:33:07.747 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.redhat.com/" 2023-09-10T00:33:07.747 INFO:teuthology.orchestra.run.smithi003.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-10T00:33:07.747 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-10T00:33:07.747 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:33:07.747 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-10T00:33:07.748 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-09-10T00:33:07.748 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-10T00:33:07.748 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-09-10T00:33:07.749 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-09-10T00:33:07.770 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2023-09-10T00:33:07.791 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2023-09-10T00:33:07.791 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2023-09-10T00:33:07.851 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Red Hat Enterprise Linux" 2023-09-10T00:33:07.851 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="8.6 (Ootpa)" 2023-09-10T00:33:07.852 INFO:teuthology.orchestra.run.smithi092.stdout:ID="rhel" 2023-09-10T00:33:07.852 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="fedora" 2023-09-10T00:33:07.852 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="8.6" 2023-09-10T00:33:07.853 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el8" 2023-09-10T00:33:07.853 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-09-10T00:33:07.853 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2023-09-10T00:33:07.854 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-09-10T00:33:07.854 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.redhat.com/" 2023-09-10T00:33:07.854 INFO:teuthology.orchestra.run.smithi092.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-10T00:33:07.854 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-10T00:33:07.855 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:33:07.855 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-10T00:33:07.855 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-09-10T00:33:07.855 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-10T00:33:07.855 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-09-10T00:33:07.856 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2023-09-10T00:33:07.878 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2023-09-10T00:33:07.898 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2023-09-10T00:33:07.899 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2023-09-10T00:33:07.958 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Red Hat Enterprise Linux" 2023-09-10T00:33:07.959 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8.6 (Ootpa)" 2023-09-10T00:33:07.959 INFO:teuthology.orchestra.run.smithi136.stdout:ID="rhel" 2023-09-10T00:33:07.959 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="fedora" 2023-09-10T00:33:07.959 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8.6" 2023-09-10T00:33:07.960 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2023-09-10T00:33:07.960 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-09-10T00:33:07.960 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2023-09-10T00:33:07.960 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-09-10T00:33:07.961 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.redhat.com/" 2023-09-10T00:33:07.961 INFO:teuthology.orchestra.run.smithi136.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-10T00:33:07.961 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-10T00:33:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:33:07.961 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-10T00:33:07.962 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-09-10T00:33:07.962 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-10T00:33:07.962 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-09-10T00:33:07.963 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2023-09-10T00:33:07.986 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-10T00:33:07.999 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-10T00:33:08.020 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-10T00:33:08.021 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2023-09-10T00:33:08.024 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2023-09-10T00:33:08.026 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2023-09-10T00:33:08.044 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-10T00:33:08.057 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-10T00:33:08.057 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2023-09-10T00:33:08.083 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2023-09-10T00:33:08.086 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2023-09-10T00:33:08.119 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-10T00:33:08.205 INFO:teuthology.run_tasks:Running task kernel... 2023-09-10T00:33:08.225 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-09-10T00:33:08.226 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-09-10T00:33:08.226 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-09-10T00:33:08.226 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-10T00:33:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T00:33:08.247 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-09-10T00:33:08.560 DEBUG:teuthology.task.kernel:sha1 for client.0 is 533e5fc8ca061dbf84e56917eecb852edf10a398 2023-09-10T00:33:08.561 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "533e5fc8ca061dbf84e56917eecb852edf10a398"... 2023-09-10T00:33:08.561 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-09-10T00:33:08.581 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-372.9.1.el8.x86_64 2023-09-10T00:33:08.582 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 533e5fc8ca061dbf84e56917eecb852edf10a398 2023-09-10T00:33:08.582 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-09-10T00:33:08.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum upgrade -y linux-firmware 2023-09-10T00:33:09.769 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-09-10T00:33:09.769 INFO:teuthology.orchestra.run.smithi003.stdout:Unable to read consumer identity 2023-09-10T00:33:11.163 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-09-10T00:33:11.880 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-09-10T00:33:15.730 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 131 kB/s | 24 kB 00:00 2023-09-10T00:33:16.517 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-09-10T00:33:16.518 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-09-10T00:33:16.518 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-09-10T00:33:16.648 INFO:teuthology.task.kernel:Downloading kernel 533e5fc8ca061dbf84e56917eecb852edf10a398 on client.0... 2023-09-10T00:33:16.649 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=533e5fc8ca061dbf84e56917eecb852edf10a398 2023-09-10T00:33:16.810 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/533e5fc8ca061dbf84e56917eecb852edf10a398/centos/8/flavors/default/x86_64/ 2023-09-10T00:33:16.811 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc7_g533e5fc8ca06-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/533e5fc8ca061dbf84e56917eecb852edf10a398/centos/8/flavors/default/x86_64/ --input-file=- 2023-09-10T00:33:16.813 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi003.front.sepia.ceph.com to complete... 2023-09-10T00:33:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10 00:33:29 URL:https://4.chacra.ceph.com/r/kernel/testing/533e5fc8ca061dbf84e56917eecb852edf10a398/centos/8/flavors/default/x86_64/kernel-6.5.0_rc7_g533e5fc8ca06-1.x86_64.rpm [380759500/380759500] -> "/tmp/kernel.x86_64.rpm" [1] 2023-09-10T00:33:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:FINISHED --2023-09-10 00:33:29-- 2023-09-10T00:33:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:Total wall clock time: 13s 2023-09-10T00:33:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:Downloaded: 1 files, 363M in 13s (28.9 MB/s) 2023-09-10T00:33:29.660 INFO:teuthology.task.kernel:Installing kernel 533e5fc8ca061dbf84e56917eecb852edf10a398 on client.0... 2023-09-10T00:33:29.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-09-10T00:33:31.310 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying... ######################################## 2023-09-10T00:33:31.364 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing... ######################################## 2023-09-10T00:33:31.452 INFO:teuthology.orchestra.run.smithi003.stdout:Updating / installing... 2023-09-10T00:34:12.049 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-6.5.0_rc7_g533e5fc8ca06-1 ######################################## 2023-09-10T00:34:35.585 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-09-10T00:34:35.585 DEBUG:teuthology.orchestra.run.smithi003:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-09-10T00:34:35.638 INFO:teuthology.orchestra.run.smithi003.stdout:/boot/System.map-6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:35.639 INFO:teuthology.orchestra.run.smithi003.stdout:/boot/config-6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:35.639 INFO:teuthology.orchestra.run.smithi003.stdout:/boot/vmlinuz-6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:35.639 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:35.639 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel 2023-09-10T00:34:35.639 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch 2023-09-10T00:34:35.640 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86 2023-09-10T00:34:35.640 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto 2023-09-10T00:34:35.640 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-09-10T00:34:35.640 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-09-10T00:34:35.640 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-09-10T00:34:35.641 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-09-10T00:34:35.641 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-09-10T00:34:35.641 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-09-10T00:34:35.641 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-09-10T00:34:35.641 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/crc32c-intel.ko 2023-09-10T00:34:35.641 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-09-10T00:34:35.642 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-09-10T00:34:35.642 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-09-10T00:34:35.642 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-09-10T00:34:35.642 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/serpent-avx2.ko 2023-09-10T00:34:35.642 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-09-10T00:34:35.643 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-09-10T00:34:35.643 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-09-10T00:34:35.643 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-09-10T00:34:35.644 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-09-10T00:34:35.644 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-09-10T00:34:35.644 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-09-10T00:34:35.645 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kernel 2023-09-10T00:34:35.645 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kernel/cpu 2023-09-10T00:34:35.645 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kernel/cpu/mce 2023-09-10T00:34:35.645 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-09-10T00:34:35.645 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kvm 2023-09-10T00:34:35.646 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kvm/kvm-amd.ko 2023-09-10T00:34:35.646 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kvm/kvm-intel.ko 2023-09-10T00:34:35.646 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/arch/x86/kvm/kvm.ko 2023-09-10T00:34:35.646 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto 2023-09-10T00:34:35.647 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/ansi_cprng.ko 2023-09-10T00:34:35.647 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/anubis.ko 2023-09-10T00:34:35.647 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/arc4.ko 2023-09-10T00:34:35.647 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/async_tx 2023-09-10T00:34:35.647 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/async_tx/async_memcpy.ko 2023-09-10T00:34:35.648 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/async_tx/async_pq.ko 2023-09-10T00:34:35.648 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/async_tx/async_raid6_recov.ko 2023-09-10T00:34:35.648 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/async_tx/async_tx.ko 2023-09-10T00:34:35.648 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/async_tx/async_xor.ko 2023-09-10T00:34:35.648 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/async_tx/raid6test.ko 2023-09-10T00:34:35.648 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/authenc.ko 2023-09-10T00:34:35.649 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/authencesn.ko 2023-09-10T00:34:35.649 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/blake2b_generic.ko 2023-09-10T00:34:35.649 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/blowfish_common.ko 2023-09-10T00:34:35.649 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/blowfish_generic.ko 2023-09-10T00:34:35.649 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/camellia_generic.ko 2023-09-10T00:34:35.650 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/cast5_generic.ko 2023-09-10T00:34:35.650 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/cast6_generic.ko 2023-09-10T00:34:35.650 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/cast_common.ko 2023-09-10T00:34:35.650 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/ccm.ko 2023-09-10T00:34:35.650 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/cmac.ko 2023-09-10T00:34:35.651 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/crc32_generic.ko 2023-09-10T00:34:35.651 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/crypto_engine.ko 2023-09-10T00:34:35.651 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/crypto_user.ko 2023-09-10T00:34:35.651 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/cts.ko 2023-09-10T00:34:35.651 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/des_generic.ko 2023-09-10T00:34:35.652 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/dh_generic.ko 2023-09-10T00:34:35.652 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/echainiv.ko 2023-09-10T00:34:35.652 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/essiv.ko 2023-09-10T00:34:35.652 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/fcrypt.ko 2023-09-10T00:34:35.652 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/khazad.ko 2023-09-10T00:34:35.652 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/lz4.ko 2023-09-10T00:34:35.653 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/lz4hc.ko 2023-09-10T00:34:35.653 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/md4.ko 2023-09-10T00:34:35.653 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/michael_mic.ko 2023-09-10T00:34:35.653 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/pcbc.ko 2023-09-10T00:34:35.654 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/pcrypt.ko 2023-09-10T00:34:35.654 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/rmd160.ko 2023-09-10T00:34:35.654 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/seed.ko 2023-09-10T00:34:35.654 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/serpent_generic.ko 2023-09-10T00:34:35.654 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/tcrypt.ko 2023-09-10T00:34:35.655 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/tea.ko 2023-09-10T00:34:35.655 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/twofish_common.ko 2023-09-10T00:34:35.655 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/twofish_generic.ko 2023-09-10T00:34:35.655 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/vmac.ko 2023-09-10T00:34:35.655 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/wp512.ko 2023-09-10T00:34:35.655 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/xcbc.ko 2023-09-10T00:34:35.656 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/xor.ko 2023-09-10T00:34:35.656 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/crypto/xxhash_generic.ko 2023-09-10T00:34:35.656 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers 2023-09-10T00:34:35.656 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi 2023-09-10T00:34:35.656 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/acpi_ipmi.ko 2023-09-10T00:34:35.657 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/acpi_pad.ko 2023-09-10T00:34:35.657 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/custom_method.ko 2023-09-10T00:34:35.657 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/ec_sys.ko 2023-09-10T00:34:35.657 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/platform_profile.ko 2023-09-10T00:34:35.657 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/sbs.ko 2023-09-10T00:34:35.658 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/sbshc.ko 2023-09-10T00:34:35.658 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/acpi/video.ko 2023-09-10T00:34:35.658 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata 2023-09-10T00:34:35.658 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/acard-ahci.ko 2023-09-10T00:34:35.658 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/ahci_platform.ko 2023-09-10T00:34:35.659 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/ata_generic.ko 2023-09-10T00:34:35.659 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/libahci_platform.ko 2023-09-10T00:34:35.659 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_acpi.ko 2023-09-10T00:34:35.659 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_ali.ko 2023-09-10T00:34:35.659 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_amd.ko 2023-09-10T00:34:35.660 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_artop.ko 2023-09-10T00:34:35.660 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_atiixp.ko 2023-09-10T00:34:35.660 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_atp867x.ko 2023-09-10T00:34:35.660 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_cmd640.ko 2023-09-10T00:34:35.660 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_cmd64x.ko 2023-09-10T00:34:35.660 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_cypress.ko 2023-09-10T00:34:35.661 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_efar.ko 2023-09-10T00:34:35.661 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_hpt366.ko 2023-09-10T00:34:35.661 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_hpt37x.ko 2023-09-10T00:34:35.661 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_hpt3x2n.ko 2023-09-10T00:34:35.661 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_hpt3x3.ko 2023-09-10T00:34:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_it8213.ko 2023-09-10T00:34:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_it821x.ko 2023-09-10T00:34:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_jmicron.ko 2023-09-10T00:34:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_marvell.ko 2023-09-10T00:34:35.662 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_mpiix.ko 2023-09-10T00:34:35.663 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_netcell.ko 2023-09-10T00:34:35.663 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_ninja32.ko 2023-09-10T00:34:35.663 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_ns87410.ko 2023-09-10T00:34:35.663 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_ns87415.ko 2023-09-10T00:34:35.663 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_oldpiix.ko 2023-09-10T00:34:35.664 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_opti.ko 2023-09-10T00:34:35.664 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_optidma.ko 2023-09-10T00:34:35.664 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_pcmcia.ko 2023-09-10T00:34:35.664 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_pdc2027x.ko 2023-09-10T00:34:35.664 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-09-10T00:34:35.664 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_piccolo.ko 2023-09-10T00:34:35.665 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_rdc.ko 2023-09-10T00:34:35.665 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_sch.ko 2023-09-10T00:34:35.665 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_serverworks.ko 2023-09-10T00:34:35.665 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_sil680.ko 2023-09-10T00:34:35.665 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_sis.ko 2023-09-10T00:34:35.666 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_sl82c105.ko 2023-09-10T00:34:35.666 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_triflex.ko 2023-09-10T00:34:35.666 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pata_via.ko 2023-09-10T00:34:35.666 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/pdc_adma.ko 2023-09-10T00:34:35.666 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_inic162x.ko 2023-09-10T00:34:35.667 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_mv.ko 2023-09-10T00:34:35.667 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_nv.ko 2023-09-10T00:34:35.667 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_promise.ko 2023-09-10T00:34:35.667 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_qstor.ko 2023-09-10T00:34:35.667 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_sil.ko 2023-09-10T00:34:35.667 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_sil24.ko 2023-09-10T00:34:35.668 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_sis.ko 2023-09-10T00:34:35.668 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_svw.ko 2023-09-10T00:34:35.668 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_sx4.ko 2023-09-10T00:34:35.668 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_uli.ko 2023-09-10T00:34:35.668 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_via.ko 2023-09-10T00:34:35.669 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ata/sata_vsc.ko 2023-09-10T00:34:35.669 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/auxdisplay 2023-09-10T00:34:35.669 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/auxdisplay/ks0108.ko 2023-09-10T00:34:35.669 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/base 2023-09-10T00:34:35.669 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/base/regmap 2023-09-10T00:34:35.670 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/base/regmap/regmap-i2c.ko 2023-09-10T00:34:35.670 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/base/regmap/regmap-mmio.ko 2023-09-10T00:34:35.670 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/bcma 2023-09-10T00:34:35.670 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/bcma/bcma.ko 2023-09-10T00:34:35.670 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block 2023-09-10T00:34:35.671 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/aoe 2023-09-10T00:34:35.671 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/aoe/aoe.ko 2023-09-10T00:34:35.671 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/brd.ko 2023-09-10T00:34:35.671 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/drbd 2023-09-10T00:34:35.671 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/drbd/drbd.ko 2023-09-10T00:34:35.672 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/floppy.ko 2023-09-10T00:34:35.672 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/loop.ko 2023-09-10T00:34:35.672 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/mtip32xx 2023-09-10T00:34:35.672 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-09-10T00:34:35.672 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/nbd.ko 2023-09-10T00:34:35.673 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/null_blk 2023-09-10T00:34:35.673 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/null_blk/null_blk.ko 2023-09-10T00:34:35.673 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/pktcdvd.ko 2023-09-10T00:34:35.673 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/rbd.ko 2023-09-10T00:34:35.673 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/virtio_blk.ko 2023-09-10T00:34:35.674 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/xen-blkback 2023-09-10T00:34:35.674 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-09-10T00:34:35.674 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/xen-blkfront.ko 2023-09-10T00:34:35.674 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/zram 2023-09-10T00:34:35.674 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/block/zram/zram.ko 2023-09-10T00:34:35.674 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char 2023-09-10T00:34:35.675 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/hangcheck-timer.ko 2023-09-10T00:34:35.675 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/hw_random 2023-09-10T00:34:35.675 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/hw_random/amd-rng.ko 2023-09-10T00:34:35.675 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/hw_random/intel-rng.ko 2023-09-10T00:34:35.675 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-09-10T00:34:35.676 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/hw_random/via-rng.ko 2023-09-10T00:34:35.676 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/hw_random/virtio-rng.ko 2023-09-10T00:34:35.676 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ipmi 2023-09-10T00:34:35.676 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-09-10T00:34:35.676 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-09-10T00:34:35.677 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-09-10T00:34:35.677 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ipmi/ipmi_si.ko 2023-09-10T00:34:35.677 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-09-10T00:34:35.677 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-09-10T00:34:35.677 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/lp.ko 2023-09-10T00:34:35.678 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/mwave 2023-09-10T00:34:35.678 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/mwave/mwave.ko 2023-09-10T00:34:35.678 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/ppdev.ko 2023-09-10T00:34:35.678 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tlclk.ko 2023-09-10T00:34:35.678 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tpm 2023-09-10T00:34:35.679 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tpm/tpm.ko 2023-09-10T00:34:35.679 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tpm/tpm_atmel.ko 2023-09-10T00:34:35.679 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tpm/tpm_infineon.ko 2023-09-10T00:34:35.679 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tpm/tpm_nsc.ko 2023-09-10T00:34:35.679 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tpm/tpm_tis.ko 2023-09-10T00:34:35.680 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-09-10T00:34:35.680 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/uv_mmtimer.ko 2023-09-10T00:34:35.680 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/char/virtio_console.ko 2023-09-10T00:34:35.680 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/cpufreq 2023-09-10T00:34:35.680 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-09-10T00:34:35.680 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-09-10T00:34:35.681 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/cpufreq/p4-clockmod.ko 2023-09-10T00:34:35.681 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-09-10T00:34:35.681 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/cpufreq/powernow-k8.ko 2023-09-10T00:34:35.681 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/cpufreq/speedstep-lib.ko 2023-09-10T00:34:35.681 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto 2023-09-10T00:34:35.682 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/ccp 2023-09-10T00:34:35.682 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-09-10T00:34:35.682 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/ccp/ccp.ko 2023-09-10T00:34:35.682 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/intel 2023-09-10T00:34:35.682 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/intel/qat 2023-09-10T00:34:35.683 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/intel/qat/qat_common 2023-09-10T00:34:35.683 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-09-10T00:34:35.683 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-09-10T00:34:35.683 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-09-10T00:34:35.683 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/padlock-aes.ko 2023-09-10T00:34:35.684 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/padlock-sha.ko 2023-09-10T00:34:35.684 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/virtio 2023-09-10T00:34:35.684 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-09-10T00:34:35.684 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/dca 2023-09-10T00:34:35.684 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/dca/dca.ko 2023-09-10T00:34:35.685 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/devfreq 2023-09-10T00:34:35.685 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-09-10T00:34:35.685 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/dma 2023-09-10T00:34:35.685 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/dma/dw 2023-09-10T00:34:35.685 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/dma/dw/dw_dmac.ko 2023-09-10T00:34:35.686 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/dma/ioat 2023-09-10T00:34:35.686 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/dma/ioat/ioatdma.ko 2023-09-10T00:34:35.686 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac 2023-09-10T00:34:35.686 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/amd64_edac.ko 2023-09-10T00:34:35.686 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/e752x_edac.ko 2023-09-10T00:34:35.687 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/edac_mce_amd.ko 2023-09-10T00:34:35.687 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/i3000_edac.ko 2023-09-10T00:34:35.687 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/i3200_edac.ko 2023-09-10T00:34:35.687 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/i5100_edac.ko 2023-09-10T00:34:35.687 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/i5400_edac.ko 2023-09-10T00:34:35.687 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/i7300_edac.ko 2023-09-10T00:34:35.688 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/i7core_edac.ko 2023-09-10T00:34:35.688 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/i82975x_edac.ko 2023-09-10T00:34:35.688 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/ie31200_edac.ko 2023-09-10T00:34:35.688 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/sb_edac.ko 2023-09-10T00:34:35.688 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/edac/x38_edac.ko 2023-09-10T00:34:35.689 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/firmware 2023-09-10T00:34:35.689 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/firmware/edd.ko 2023-09-10T00:34:35.689 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/firmware/iscsi_ibft.ko 2023-09-10T00:34:35.689 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/gpio 2023-09-10T00:34:35.689 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/gpio/gpio-ich.ko 2023-09-10T00:34:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/gpio/gpio-viperboard.ko 2023-09-10T00:34:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid 2023-09-10T00:34:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-appleir.ko 2023-09-10T00:34:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-aureal.ko 2023-09-10T00:34:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-axff.ko 2023-09-10T00:34:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-dr.ko 2023-09-10T00:34:35.691 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-elecom.ko 2023-09-10T00:34:35.691 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-elo.ko 2023-09-10T00:34:35.691 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-emsff.ko 2023-09-10T00:34:35.691 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-gaff.ko 2023-09-10T00:34:35.693 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-gt683r.ko 2023-09-10T00:34:35.693 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-gyration.ko 2023-09-10T00:34:35.693 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-holtek-kbd.ko 2023-09-10T00:34:35.693 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-holtek-mouse.ko 2023-09-10T00:34:35.694 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-holtekff.ko 2023-09-10T00:34:35.694 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-hyperv.ko 2023-09-10T00:34:35.694 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-icade.ko 2023-09-10T00:34:35.694 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-keytouch.ko 2023-09-10T00:34:35.694 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-kye.ko 2023-09-10T00:34:35.695 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-lcpower.ko 2023-09-10T00:34:35.695 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-led.ko 2023-09-10T00:34:35.695 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-lenovo.ko 2023-09-10T00:34:35.695 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-logitech-dj.ko 2023-09-10T00:34:35.695 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-09-10T00:34:35.696 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-multitouch.ko 2023-09-10T00:34:35.696 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-ortek.ko 2023-09-10T00:34:35.696 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-penmount.ko 2023-09-10T00:34:35.696 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-petalynx.ko 2023-09-10T00:34:35.696 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-picolcd.ko 2023-09-10T00:34:35.696 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-pl.ko 2023-09-10T00:34:35.697 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-plantronics.ko 2023-09-10T00:34:35.697 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-primax.ko 2023-09-10T00:34:35.697 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-rmi.ko 2023-09-10T00:34:35.697 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-arvo.ko 2023-09-10T00:34:35.697 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-common.ko 2023-09-10T00:34:35.698 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-isku.ko 2023-09-10T00:34:35.698 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-kone.ko 2023-09-10T00:34:35.698 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-09-10T00:34:35.698 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-konepure.ko 2023-09-10T00:34:35.698 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-09-10T00:34:35.699 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-lua.ko 2023-09-10T00:34:35.699 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-pyra.ko 2023-09-10T00:34:35.699 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-ryos.ko 2023-09-10T00:34:35.699 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat-savu.ko 2023-09-10T00:34:35.699 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-roccat.ko 2023-09-10T00:34:35.700 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-saitek.ko 2023-09-10T00:34:35.700 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-samsung.ko 2023-09-10T00:34:35.700 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-sensor-hub.ko 2023-09-10T00:34:35.700 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-sjoy.ko 2023-09-10T00:34:35.700 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-sony.ko 2023-09-10T00:34:35.700 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-speedlink.ko 2023-09-10T00:34:35.701 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-steelseries.ko 2023-09-10T00:34:35.701 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-sunplus.ko 2023-09-10T00:34:35.701 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-thrustmaster.ko 2023-09-10T00:34:35.701 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-tivo.ko 2023-09-10T00:34:35.701 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-tmff.ko 2023-09-10T00:34:35.702 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-topseed.ko 2023-09-10T00:34:35.702 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-twinhan.ko 2023-09-10T00:34:35.702 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-uclogic.ko 2023-09-10T00:34:35.702 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-waltop.ko 2023-09-10T00:34:35.702 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-wiimote.ko 2023-09-10T00:34:35.703 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-xinmo.ko 2023-09-10T00:34:35.703 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-zpff.ko 2023-09-10T00:34:35.703 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/hid-zydacron.ko 2023-09-10T00:34:35.703 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/uhid.ko 2023-09-10T00:34:35.703 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hid/wacom.ko 2023-09-10T00:34:35.704 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hv 2023-09-10T00:34:35.704 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hv/hv_balloon.ko 2023-09-10T00:34:35.704 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hv/hv_utils.ko 2023-09-10T00:34:35.704 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hv/hv_vmbus.ko 2023-09-10T00:34:35.704 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon 2023-09-10T00:34:35.705 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/abituguru.ko 2023-09-10T00:34:35.705 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/abituguru3.ko 2023-09-10T00:34:35.705 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/acpi_power_meter.ko 2023-09-10T00:34:35.705 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ad7414.ko 2023-09-10T00:34:35.705 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ad7418.ko 2023-09-10T00:34:35.706 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adc128d818.ko 2023-09-10T00:34:35.706 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adm1025.ko 2023-09-10T00:34:35.706 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adm1026.ko 2023-09-10T00:34:35.706 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adm1029.ko 2023-09-10T00:34:35.706 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adm1031.ko 2023-09-10T00:34:35.707 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adm9240.ko 2023-09-10T00:34:35.707 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ads7828.ko 2023-09-10T00:34:35.707 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adt7410.ko 2023-09-10T00:34:35.707 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adt7411.ko 2023-09-10T00:34:35.707 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adt7462.ko 2023-09-10T00:34:35.708 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adt7470.ko 2023-09-10T00:34:35.708 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adt7475.ko 2023-09-10T00:34:35.708 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/adt7x10.ko 2023-09-10T00:34:35.708 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/amc6821.ko 2023-09-10T00:34:35.708 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/applesmc.ko 2023-09-10T00:34:35.709 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/asb100.ko 2023-09-10T00:34:35.709 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/asc7621.ko 2023-09-10T00:34:35.709 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/asus_atk0110.ko 2023-09-10T00:34:35.709 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/atxp1.ko 2023-09-10T00:34:35.709 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/coretemp.ko 2023-09-10T00:34:35.710 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-09-10T00:34:35.710 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/dme1737.ko 2023-09-10T00:34:35.710 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ds1621.ko 2023-09-10T00:34:35.710 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ds620.ko 2023-09-10T00:34:35.710 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/emc1403.ko 2023-09-10T00:34:35.711 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/emc6w201.ko 2023-09-10T00:34:35.711 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/f71805f.ko 2023-09-10T00:34:35.711 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/f71882fg.ko 2023-09-10T00:34:35.711 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/f75375s.ko 2023-09-10T00:34:35.711 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/fam15h_power.ko 2023-09-10T00:34:35.712 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/fschmd.ko 2023-09-10T00:34:35.712 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/g760a.ko 2023-09-10T00:34:35.712 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/g762.ko 2023-09-10T00:34:35.712 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/gl518sm.ko 2023-09-10T00:34:35.712 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/gl520sm.ko 2023-09-10T00:34:35.713 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/hwmon-vid.ko 2023-09-10T00:34:35.713 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/i5500_temp.ko 2023-09-10T00:34:35.713 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/i5k_amb.ko 2023-09-10T00:34:35.713 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ibmaem.ko 2023-09-10T00:34:35.713 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ibmpex.ko 2023-09-10T00:34:35.714 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ina209.ko 2023-09-10T00:34:35.714 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ina2xx.ko 2023-09-10T00:34:35.714 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/it87.ko 2023-09-10T00:34:35.714 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/k10temp.ko 2023-09-10T00:34:35.715 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/k8temp.ko 2023-09-10T00:34:35.715 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lineage-pem.ko 2023-09-10T00:34:35.715 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm63.ko 2023-09-10T00:34:35.715 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm73.ko 2023-09-10T00:34:35.715 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm75.ko 2023-09-10T00:34:35.715 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm77.ko 2023-09-10T00:34:35.716 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm78.ko 2023-09-10T00:34:35.716 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm80.ko 2023-09-10T00:34:35.716 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm83.ko 2023-09-10T00:34:35.716 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm85.ko 2023-09-10T00:34:35.716 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm87.ko 2023-09-10T00:34:35.717 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm90.ko 2023-09-10T00:34:35.717 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm92.ko 2023-09-10T00:34:35.717 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm93.ko 2023-09-10T00:34:35.717 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm95234.ko 2023-09-10T00:34:35.717 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm95241.ko 2023-09-10T00:34:35.717 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/lm95245.ko 2023-09-10T00:34:35.718 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ltc2945.ko 2023-09-10T00:34:35.718 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ltc4151.ko 2023-09-10T00:34:35.718 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ltc4215.ko 2023-09-10T00:34:35.718 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ltc4222.ko 2023-09-10T00:34:35.718 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ltc4245.ko 2023-09-10T00:34:35.719 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ltc4260.ko 2023-09-10T00:34:35.719 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/ltc4261.ko 2023-09-10T00:34:35.719 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/max16065.ko 2023-09-10T00:34:35.719 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/max1619.ko 2023-09-10T00:34:35.719 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/max1668.ko 2023-09-10T00:34:35.720 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/max197.ko 2023-09-10T00:34:35.720 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/max6639.ko 2023-09-10T00:34:35.720 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/max6650.ko 2023-09-10T00:34:35.720 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/max6697.ko 2023-09-10T00:34:35.720 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/mcp3021.ko 2023-09-10T00:34:35.720 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/nct6683.ko 2023-09-10T00:34:35.721 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/nct6775-core.ko 2023-09-10T00:34:35.721 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/nct6775.ko 2023-09-10T00:34:35.721 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/nct7802.ko 2023-09-10T00:34:35.721 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pc87360.ko 2023-09-10T00:34:35.721 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pc87427.ko 2023-09-10T00:34:35.722 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pcf8591.ko 2023-09-10T00:34:35.722 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus 2023-09-10T00:34:35.722 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-09-10T00:34:35.722 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-09-10T00:34:35.722 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-09-10T00:34:35.723 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/max16064.ko 2023-09-10T00:34:35.723 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/max34440.ko 2023-09-10T00:34:35.723 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/max8688.ko 2023-09-10T00:34:35.723 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-09-10T00:34:35.723 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-09-10T00:34:35.724 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-09-10T00:34:35.724 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-09-10T00:34:35.724 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-09-10T00:34:35.724 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-09-10T00:34:35.724 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/powr1220.ko 2023-09-10T00:34:35.724 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/sch5627.ko 2023-09-10T00:34:35.725 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/sch5636.ko 2023-09-10T00:34:35.725 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/sch56xx-common.ko 2023-09-10T00:34:35.725 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/sht15.ko 2023-09-10T00:34:35.725 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/sht21.ko 2023-09-10T00:34:35.725 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/shtc1.ko 2023-09-10T00:34:35.726 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/sis5595.ko 2023-09-10T00:34:35.726 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/smsc47b397.ko 2023-09-10T00:34:35.726 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/smsc47m1.ko 2023-09-10T00:34:35.726 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/smsc47m192.ko 2023-09-10T00:34:35.726 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/thmc50.ko 2023-09-10T00:34:35.727 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/tmp102.ko 2023-09-10T00:34:35.727 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/tmp103.ko 2023-09-10T00:34:35.727 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/tmp401.ko 2023-09-10T00:34:35.727 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/tmp421.ko 2023-09-10T00:34:35.727 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/via-cputemp.ko 2023-09-10T00:34:35.728 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/via686a.ko 2023-09-10T00:34:35.728 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/vt1211.ko 2023-09-10T00:34:35.728 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/vt8231.ko 2023-09-10T00:34:35.728 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83627ehf.ko 2023-09-10T00:34:35.728 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83627hf.ko 2023-09-10T00:34:35.728 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83781d.ko 2023-09-10T00:34:35.729 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83791d.ko 2023-09-10T00:34:35.729 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83792d.ko 2023-09-10T00:34:35.729 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83793.ko 2023-09-10T00:34:35.729 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83795.ko 2023-09-10T00:34:35.729 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83l785ts.ko 2023-09-10T00:34:35.730 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/hwmon/w83l786ng.ko 2023-09-10T00:34:35.730 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c 2023-09-10T00:34:35.730 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/algos 2023-09-10T00:34:35.730 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-09-10T00:34:35.730 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-09-10T00:34:35.730 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses 2023-09-10T00:34:35.731 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-09-10T00:34:35.731 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-09-10T00:34:35.731 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-09-10T00:34:35.731 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-09-10T00:34:35.731 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-09-10T00:34:35.732 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-09-10T00:34:35.732 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-09-10T00:34:35.732 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-09-10T00:34:35.732 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-i801.ko 2023-09-10T00:34:35.732 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-isch.ko 2023-09-10T00:34:35.733 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-09-10T00:34:35.733 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-09-10T00:34:35.733 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-09-10T00:34:35.733 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-parport.ko 2023-09-10T00:34:35.733 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-09-10T00:34:35.734 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-09-10T00:34:35.734 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-09-10T00:34:35.734 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-09-10T00:34:35.734 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-09-10T00:34:35.734 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-09-10T00:34:35.735 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-via.ko 2023-09-10T00:34:35.735 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-09-10T00:34:35.735 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-09-10T00:34:35.735 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/i2c-dev.ko 2023-09-10T00:34:35.735 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/i2c-mux.ko 2023-09-10T00:34:35.735 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-09-10T00:34:35.736 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/i2c-smbus.ko 2023-09-10T00:34:35.736 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/i2c/i2c-stub.ko 2023-09-10T00:34:35.736 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband 2023-09-10T00:34:35.736 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core 2023-09-10T00:34:35.736 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core/ib_cm.ko 2023-09-10T00:34:35.737 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core/ib_core.ko 2023-09-10T00:34:35.737 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core/ib_umad.ko 2023-09-10T00:34:35.737 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-09-10T00:34:35.737 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core/iw_cm.ko 2023-09-10T00:34:35.737 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core/rdma_cm.ko 2023-09-10T00:34:35.738 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-09-10T00:34:35.738 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw 2023-09-10T00:34:35.738 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/cxgb4 2023-09-10T00:34:35.738 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-09-10T00:34:35.738 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/mlx4 2023-09-10T00:34:35.739 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-09-10T00:34:35.739 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/mlx5 2023-09-10T00:34:35.739 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-09-10T00:34:35.739 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/mthca 2023-09-10T00:34:35.739 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-09-10T00:34:35.739 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp 2023-09-10T00:34:35.740 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/ipoib 2023-09-10T00:34:35.740 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-09-10T00:34:35.740 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/iser 2023-09-10T00:34:35.740 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-09-10T00:34:35.740 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/isert 2023-09-10T00:34:35.741 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-09-10T00:34:35.741 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/srp 2023-09-10T00:34:35.741 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-09-10T00:34:35.741 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/srpt 2023-09-10T00:34:35.741 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-09-10T00:34:35.742 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input 2023-09-10T00:34:35.742 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/gameport 2023-09-10T00:34:35.742 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-09-10T00:34:35.742 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/gameport/fm801-gp.ko 2023-09-10T00:34:35.742 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/gameport/gameport.ko 2023-09-10T00:34:35.743 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/gameport/lightning.ko 2023-09-10T00:34:35.743 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/gameport/ns558.ko 2023-09-10T00:34:35.743 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/joydev.ko 2023-09-10T00:34:35.743 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/keyboard 2023-09-10T00:34:35.743 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/keyboard/gpio_keys.ko 2023-09-10T00:34:35.743 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse 2023-09-10T00:34:35.744 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/appletouch.ko 2023-09-10T00:34:35.744 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/bcm5974.ko 2023-09-10T00:34:35.744 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/cyapatp.ko 2023-09-10T00:34:35.744 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/elan_i2c.ko 2023-09-10T00:34:35.744 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/sermouse.ko 2023-09-10T00:34:35.745 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-09-10T00:34:35.745 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/synaptics_usb.ko 2023-09-10T00:34:35.745 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/mouse/vsxxxaa.ko 2023-09-10T00:34:35.745 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/rmi4 2023-09-10T00:34:35.745 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/rmi4/rmi_core.ko 2023-09-10T00:34:35.746 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/serio 2023-09-10T00:34:35.746 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/serio/altera_ps2.ko 2023-09-10T00:34:35.746 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/serio/arc_ps2.ko 2023-09-10T00:34:35.746 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-09-10T00:34:35.746 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/serio/serio_raw.ko 2023-09-10T00:34:35.747 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/input/sparse-keymap.ko 2023-09-10T00:34:35.747 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/iommu 2023-09-10T00:34:35.747 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/iommu/amd 2023-09-10T00:34:35.747 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/iommu/amd/iommu_v2.ko 2023-09-10T00:34:35.747 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds 2023-09-10T00:34:35.748 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/leds-blinkm.ko 2023-09-10T00:34:35.748 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/leds-lm3530.ko 2023-09-10T00:34:35.748 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/leds-lp3944.ko 2023-09-10T00:34:35.748 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/leds-ss4200.ko 2023-09-10T00:34:35.748 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger 2023-09-10T00:34:35.748 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-09-10T00:34:35.749 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-09-10T00:34:35.749 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-09-10T00:34:35.749 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-09-10T00:34:35.749 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-09-10T00:34:35.749 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-09-10T00:34:35.750 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-09-10T00:34:35.750 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-09-10T00:34:35.750 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md 2023-09-10T00:34:35.750 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/bcache 2023-09-10T00:34:35.750 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/bcache/bcache.ko 2023-09-10T00:34:35.751 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-bio-prison.ko 2023-09-10T00:34:35.751 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-cache-smq.ko 2023-09-10T00:34:35.751 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-cache.ko 2023-09-10T00:34:35.751 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-crypt.ko 2023-09-10T00:34:35.751 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-delay.ko 2023-09-10T00:34:35.751 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-flakey.ko 2023-09-10T00:34:35.752 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-log-userspace.ko 2023-09-10T00:34:35.752 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-multipath.ko 2023-09-10T00:34:35.752 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-queue-length.ko 2023-09-10T00:34:35.752 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-raid.ko 2023-09-10T00:34:35.752 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-round-robin.ko 2023-09-10T00:34:35.753 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-service-time.ko 2023-09-10T00:34:35.753 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-switch.ko 2023-09-10T00:34:35.753 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-thin-pool.ko 2023-09-10T00:34:35.753 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/dm-verity.ko 2023-09-10T00:34:35.753 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/faulty.ko 2023-09-10T00:34:35.754 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/linear.ko 2023-09-10T00:34:35.754 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/multipath.ko 2023-09-10T00:34:35.754 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/persistent-data 2023-09-10T00:34:35.754 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-09-10T00:34:35.754 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/raid0.ko 2023-09-10T00:34:35.754 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/raid1.ko 2023-09-10T00:34:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/raid10.ko 2023-09-10T00:34:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/md/raid456.ko 2023-09-10T00:34:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message 2023-09-10T00:34:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion 2023-09-10T00:34:35.755 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion/mptbase.ko 2023-09-10T00:34:35.756 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion/mptctl.ko 2023-09-10T00:34:35.756 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion/mptfc.ko 2023-09-10T00:34:35.756 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion/mptlan.ko 2023-09-10T00:34:35.756 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion/mptsas.ko 2023-09-10T00:34:35.756 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion/mptscsih.ko 2023-09-10T00:34:35.757 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/message/fusion/mptspi.ko 2023-09-10T00:34:35.757 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd 2023-09-10T00:34:35.757 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd/lpc_ich.ko 2023-09-10T00:34:35.757 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd/lpc_sch.ko 2023-09-10T00:34:35.757 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd/mfd-core.ko 2023-09-10T00:34:35.758 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd/sm501.ko 2023-09-10T00:34:35.758 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd/viperboard.ko 2023-09-10T00:34:35.758 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd/vx855.ko 2023-09-10T00:34:35.758 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/mfd/wl1273-core.ko 2023-09-10T00:34:35.758 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc 2023-09-10T00:34:35.759 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/altera-stapl 2023-09-10T00:34:35.759 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-09-10T00:34:35.759 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/apds9802als.ko 2023-09-10T00:34:35.759 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/apds990x.ko 2023-09-10T00:34:35.759 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/bh1770glc.ko 2023-09-10T00:34:35.760 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/cb710 2023-09-10T00:34:35.760 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/cb710/cb710.ko 2023-09-10T00:34:35.760 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/eeprom 2023-09-10T00:34:35.760 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/eeprom/at24.ko 2023-09-10T00:34:35.760 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/eeprom/eeprom.ko 2023-09-10T00:34:35.761 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-09-10T00:34:35.761 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/eeprom/max6875.ko 2023-09-10T00:34:35.761 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/enclosure.ko 2023-09-10T00:34:35.761 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/hpilo.ko 2023-09-10T00:34:35.761 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/isl29003.ko 2023-09-10T00:34:35.762 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/isl29020.ko 2023-09-10T00:34:35.762 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/lis3lv02d 2023-09-10T00:34:35.762 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-09-10T00:34:35.762 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-09-10T00:34:35.762 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/mei 2023-09-10T00:34:35.763 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/mei/mei-me.ko 2023-09-10T00:34:35.763 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/mei/mei-txe.ko 2023-09-10T00:34:35.763 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/mei/mei.ko 2023-09-10T00:34:35.763 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/sgi-gru 2023-09-10T00:34:35.763 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/sgi-gru/gru.ko 2023-09-10T00:34:35.763 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/sgi-xp 2023-09-10T00:34:35.764 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/sgi-xp/xp.ko 2023-09-10T00:34:35.764 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/sgi-xp/xpc.ko 2023-09-10T00:34:35.764 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-09-10T00:34:35.764 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/tifm_7xx1.ko 2023-09-10T00:34:35.764 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/tifm_core.ko 2023-09-10T00:34:35.765 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/tsl2550.ko 2023-09-10T00:34:35.765 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/vmw_balloon.ko 2023-09-10T00:34:35.765 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/vmw_vmci 2023-09-10T00:34:35.765 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-09-10T00:34:35.765 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net 2023-09-10T00:34:35.766 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/bonding 2023-09-10T00:34:35.766 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/bonding/bonding.ko 2023-09-10T00:34:35.766 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dsa 2023-09-10T00:34:35.766 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dsa/b53 2023-09-10T00:34:35.766 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dsa/b53/b53_common.ko 2023-09-10T00:34:35.767 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dsa/bcm-sf2.ko 2023-09-10T00:34:35.767 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dsa/mv88e6060.ko 2023-09-10T00:34:35.767 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dsa/mv88e6xxx 2023-09-10T00:34:35.767 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-09-10T00:34:35.767 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/dummy.ko 2023-09-10T00:34:35.768 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/eql.ko 2023-09-10T00:34:35.768 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet 2023-09-10T00:34:35.768 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/3com 2023-09-10T00:34:35.768 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-09-10T00:34:35.768 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-09-10T00:34:35.768 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-09-10T00:34:35.769 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-09-10T00:34:35.769 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/8390 2023-09-10T00:34:35.769 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/8390/8390.ko 2023-09-10T00:34:35.769 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-09-10T00:34:35.769 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-09-10T00:34:35.770 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-09-10T00:34:35.770 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/adaptec 2023-09-10T00:34:35.770 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-09-10T00:34:35.770 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/agere 2023-09-10T00:34:35.770 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/agere/et131x.ko 2023-09-10T00:34:35.771 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/alteon 2023-09-10T00:34:35.771 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-09-10T00:34:35.771 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/altera 2023-09-10T00:34:35.771 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-09-10T00:34:35.771 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/amd 2023-09-10T00:34:35.771 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-09-10T00:34:35.772 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-09-10T00:34:35.772 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-09-10T00:34:35.772 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros 2023-09-10T00:34:35.772 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/alx 2023-09-10T00:34:35.772 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-09-10T00:34:35.773 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/atl1c 2023-09-10T00:34:35.773 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-09-10T00:34:35.773 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/atl1e 2023-09-10T00:34:35.773 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-09-10T00:34:35.773 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/atlx 2023-09-10T00:34:35.774 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-09-10T00:34:35.774 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-09-10T00:34:35.774 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom 2023-09-10T00:34:35.774 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-09-10T00:34:35.774 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-09-10T00:34:35.775 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-09-10T00:34:35.775 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-09-10T00:34:35.775 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-09-10T00:34:35.775 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/genet 2023-09-10T00:34:35.775 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-09-10T00:34:35.776 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-09-10T00:34:35.776 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/brocade 2023-09-10T00:34:35.776 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/brocade/bna 2023-09-10T00:34:35.776 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-09-10T00:34:35.776 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio 2023-09-10T00:34:35.777 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb 2023-09-10T00:34:35.777 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-09-10T00:34:35.777 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-09-10T00:34:35.777 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-09-10T00:34:35.777 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-09-10T00:34:35.777 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-09-10T00:34:35.778 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-09-10T00:34:35.778 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-09-10T00:34:35.778 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-09-10T00:34:35.778 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-09-10T00:34:35.778 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/cisco 2023-09-10T00:34:35.779 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/cisco/enic 2023-09-10T00:34:35.779 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-09-10T00:34:35.779 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec 2023-09-10T00:34:35.779 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec/tulip 2023-09-10T00:34:35.779 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-09-10T00:34:35.780 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-09-10T00:34:35.780 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-09-10T00:34:35.780 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-09-10T00:34:35.780 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-09-10T00:34:35.780 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-09-10T00:34:35.780 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dlink 2023-09-10T00:34:35.781 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-09-10T00:34:35.781 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-09-10T00:34:35.781 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/dnet.ko 2023-09-10T00:34:35.781 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/emulex 2023-09-10T00:34:35.781 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/emulex/benet 2023-09-10T00:34:35.782 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-09-10T00:34:35.782 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/ethoc.ko 2023-09-10T00:34:35.782 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/fealnx.ko 2023-09-10T00:34:35.782 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel 2023-09-10T00:34:35.782 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/e100.ko 2023-09-10T00:34:35.783 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/e1000 2023-09-10T00:34:35.783 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-09-10T00:34:35.783 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/e1000e 2023-09-10T00:34:35.783 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-09-10T00:34:35.783 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/fm10k 2023-09-10T00:34:35.783 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-09-10T00:34:35.784 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/i40e 2023-09-10T00:34:35.784 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-09-10T00:34:35.784 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/iavf 2023-09-10T00:34:35.784 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-09-10T00:34:35.784 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/igb 2023-09-10T00:34:35.785 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-09-10T00:34:35.785 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/igbvf 2023-09-10T00:34:35.785 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-09-10T00:34:35.785 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/ixgbe 2023-09-10T00:34:35.785 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-09-10T00:34:35.785 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/ixgbevf 2023-09-10T00:34:35.786 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-09-10T00:34:35.786 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/jme.ko 2023-09-10T00:34:35.786 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/marvell 2023-09-10T00:34:35.786 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-09-10T00:34:35.786 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/marvell/skge.ko 2023-09-10T00:34:35.787 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-09-10T00:34:35.787 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/mellanox 2023-09-10T00:34:35.787 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/mellanox/mlx4 2023-09-10T00:34:35.787 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-09-10T00:34:35.787 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-09-10T00:34:35.788 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/mellanox/mlx5 2023-09-10T00:34:35.788 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-09-10T00:34:35.788 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-09-10T00:34:35.788 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/micrel 2023-09-10T00:34:35.788 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-09-10T00:34:35.789 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/myricom 2023-09-10T00:34:35.789 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/myricom/myri10ge 2023-09-10T00:34:35.789 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-09-10T00:34:35.789 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/natsemi 2023-09-10T00:34:35.789 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-09-10T00:34:35.789 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-09-10T00:34:35.790 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/neterion 2023-09-10T00:34:35.790 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-09-10T00:34:35.790 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/nvidia 2023-09-10T00:34:35.790 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-09-10T00:34:35.790 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/packetengines 2023-09-10T00:34:35.791 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-09-10T00:34:35.791 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-09-10T00:34:35.791 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/qlogic 2023-09-10T00:34:35.791 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/qlogic/netxen 2023-09-10T00:34:35.791 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-09-10T00:34:35.792 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-09-10T00:34:35.792 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-09-10T00:34:35.792 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-09-10T00:34:35.792 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/rdc 2023-09-10T00:34:35.792 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-09-10T00:34:35.792 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/realtek 2023-09-10T00:34:35.793 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-09-10T00:34:35.793 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-09-10T00:34:35.793 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/realtek/atp.ko 2023-09-10T00:34:35.793 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-09-10T00:34:35.793 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/rocker 2023-09-10T00:34:35.794 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-09-10T00:34:35.794 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sfc 2023-09-10T00:34:35.794 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-09-10T00:34:35.794 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/silan 2023-09-10T00:34:35.794 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-09-10T00:34:35.795 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sis 2023-09-10T00:34:35.795 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sis/sis190.ko 2023-09-10T00:34:35.795 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sis/sis900.ko 2023-09-10T00:34:35.795 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/smsc 2023-09-10T00:34:35.795 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-09-10T00:34:35.796 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-09-10T00:34:35.796 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-09-10T00:34:35.796 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-09-10T00:34:35.796 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/stmicro 2023-09-10T00:34:35.796 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/stmicro/stmmac 2023-09-10T00:34:35.797 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-09-10T00:34:35.797 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-09-10T00:34:35.797 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sun 2023-09-10T00:34:35.797 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sun/cassini.ko 2023-09-10T00:34:35.797 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sun/niu.ko 2023-09-10T00:34:35.797 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sun/sungem.ko 2023-09-10T00:34:35.798 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-09-10T00:34:35.798 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/tehuti 2023-09-10T00:34:35.798 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-09-10T00:34:35.798 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/ti 2023-09-10T00:34:35.798 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/ti/tlan.ko 2023-09-10T00:34:35.799 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/via 2023-09-10T00:34:35.799 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-09-10T00:34:35.799 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-09-10T00:34:35.799 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/wiznet 2023-09-10T00:34:35.799 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-09-10T00:34:35.800 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-09-10T00:34:35.800 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/xircom 2023-09-10T00:34:35.800 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-09-10T00:34:35.800 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/geneve.ko 2023-09-10T00:34:35.800 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/hyperv 2023-09-10T00:34:35.800 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-09-10T00:34:35.801 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ieee802154 2023-09-10T00:34:35.801 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ieee802154/fakelb.ko 2023-09-10T00:34:35.801 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ifb.ko 2023-09-10T00:34:35.801 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ipvlan 2023-09-10T00:34:35.801 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ipvlan/ipvlan.ko 2023-09-10T00:34:35.802 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/macvlan.ko 2023-09-10T00:34:35.802 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/macvtap.ko 2023-09-10T00:34:35.802 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/mdio 2023-09-10T00:34:35.802 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/mdio.ko 2023-09-10T00:34:35.802 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-09-10T00:34:35.802 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-09-10T00:34:35.803 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/mdio/mdio-regmap.ko 2023-09-10T00:34:35.803 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/mii.ko 2023-09-10T00:34:35.803 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/net_failover.ko 2023-09-10T00:34:35.803 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/netconsole.ko 2023-09-10T00:34:35.803 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/nlmon.ko 2023-09-10T00:34:35.804 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/pcs 2023-09-10T00:34:35.804 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/pcs/pcs-lynx.ko 2023-09-10T00:34:35.804 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-09-10T00:34:35.804 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy 2023-09-10T00:34:35.804 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/amd.ko 2023-09-10T00:34:35.805 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-09-10T00:34:35.805 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-09-10T00:34:35.805 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/bcm7xxx.ko 2023-09-10T00:34:35.805 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/bcm87xx.ko 2023-09-10T00:34:35.805 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/broadcom.ko 2023-09-10T00:34:35.805 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/cicada.ko 2023-09-10T00:34:35.806 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/davicom.ko 2023-09-10T00:34:35.806 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/dp83640.ko 2023-09-10T00:34:35.806 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/et1011c.ko 2023-09-10T00:34:35.806 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/icplus.ko 2023-09-10T00:34:35.806 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/lxt.ko 2023-09-10T00:34:35.807 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/marvell.ko 2023-09-10T00:34:35.807 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/micrel.ko 2023-09-10T00:34:35.807 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/national.ko 2023-09-10T00:34:35.807 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/phylink.ko 2023-09-10T00:34:35.807 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/qsemi.ko 2023-09-10T00:34:35.808 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/realtek.ko 2023-09-10T00:34:35.808 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/smsc.ko 2023-09-10T00:34:35.808 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/ste10Xp.ko 2023-09-10T00:34:35.808 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/phy/vitesse.ko 2023-09-10T00:34:35.808 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp 2023-09-10T00:34:35.809 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/bsd_comp.ko 2023-09-10T00:34:35.809 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/ppp_async.ko 2023-09-10T00:34:35.809 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/ppp_deflate.ko 2023-09-10T00:34:35.809 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/ppp_generic.ko 2023-09-10T00:34:35.809 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/ppp_mppe.ko 2023-09-10T00:34:35.809 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/ppp_synctty.ko 2023-09-10T00:34:35.810 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/pppoe.ko 2023-09-10T00:34:35.810 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/pppox.ko 2023-09-10T00:34:35.810 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/ppp/pptp.ko 2023-09-10T00:34:35.810 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/slip 2023-09-10T00:34:35.810 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/slip/slhc.ko 2023-09-10T00:34:35.811 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/slip/slip.ko 2023-09-10T00:34:35.811 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/sungem_phy.ko 2023-09-10T00:34:35.811 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/tap.ko 2023-09-10T00:34:35.811 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/team 2023-09-10T00:34:35.811 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/team/team.ko 2023-09-10T00:34:35.812 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/team/team_mode_activebackup.ko 2023-09-10T00:34:35.812 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/team/team_mode_broadcast.ko 2023-09-10T00:34:35.812 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-09-10T00:34:35.812 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/team/team_mode_random.ko 2023-09-10T00:34:35.812 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-09-10T00:34:35.812 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/tun.ko 2023-09-10T00:34:35.813 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/veth.ko 2023-09-10T00:34:35.813 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/virtio_net.ko 2023-09-10T00:34:35.813 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/vmxnet3 2023-09-10T00:34:35.813 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-09-10T00:34:35.813 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/vxlan 2023-09-10T00:34:35.814 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/vxlan/vxlan.ko 2023-09-10T00:34:35.814 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/xen-netback 2023-09-10T00:34:35.814 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/xen-netback/xen-netback.ko 2023-09-10T00:34:35.814 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/net/xen-netfront.ko 2023-09-10T00:34:35.814 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ntb 2023-09-10T00:34:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ntb/ntb.ko 2023-09-10T00:34:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/nvme 2023-09-10T00:34:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/nvme/host 2023-09-10T00:34:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/nvme/host/nvme-core.ko 2023-09-10T00:34:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/nvme/host/nvme.ko 2023-09-10T00:34:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/parport 2023-09-10T00:34:35.816 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/parport/parport.ko 2023-09-10T00:34:35.816 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/parport/parport_cs.ko 2023-09-10T00:34:35.816 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/parport/parport_pc.ko 2023-09-10T00:34:35.816 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/parport/parport_serial.ko 2023-09-10T00:34:35.816 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pci 2023-09-10T00:34:35.817 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pci/hotplug 2023-09-10T00:34:35.817 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-09-10T00:34:35.817 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pci/pcie 2023-09-10T00:34:35.817 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pci/pcie/aer_inject.ko 2023-09-10T00:34:35.817 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pci/xen-pcifront.ko 2023-09-10T00:34:35.818 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pcmcia 2023-09-10T00:34:35.818 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pcmcia/i82092.ko 2023-09-10T00:34:35.818 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pcmcia/pd6729.ko 2023-09-10T00:34:35.818 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pcmcia/yenta_socket.ko 2023-09-10T00:34:35.818 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pinctrl 2023-09-10T00:34:35.819 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pinctrl/intel 2023-09-10T00:34:35.819 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-09-10T00:34:35.819 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform 2023-09-10T00:34:35.819 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86 2023-09-10T00:34:35.819 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/acer-wmi.ko 2023-09-10T00:34:35.819 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/acerhdf.ko 2023-09-10T00:34:35.820 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/apple-gmux.ko 2023-09-10T00:34:35.820 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/asus-laptop.ko 2023-09-10T00:34:35.820 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-09-10T00:34:35.820 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/asus-wmi.ko 2023-09-10T00:34:35.820 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/classmate-laptop.ko 2023-09-10T00:34:35.821 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-09-10T00:34:35.821 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-09-10T00:34:35.821 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-09-10T00:34:35.821 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-09-10T00:34:35.821 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/hdaps.ko 2023-09-10T00:34:35.821 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/intel 2023-09-10T00:34:35.822 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-09-10T00:34:35.822 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/intel_ips.ko 2023-09-10T00:34:35.822 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/msi-wmi.ko 2023-09-10T00:34:35.822 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/mxm-wmi.ko 2023-09-10T00:34:35.822 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-09-10T00:34:35.823 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/samsung-laptop.ko 2023-09-10T00:34:35.823 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/samsung-q10.ko 2023-09-10T00:34:35.823 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/topstar-laptop.ko 2023-09-10T00:34:35.823 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-09-10T00:34:35.823 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/toshiba_haps.ko 2023-09-10T00:34:35.823 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/wmi-bmof.ko 2023-09-10T00:34:35.824 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/platform/x86/wmi.ko 2023-09-10T00:34:35.824 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/powercap 2023-09-10T00:34:35.824 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/powercap/intel_rapl_common.ko 2023-09-10T00:34:35.824 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/powercap/intel_rapl_msr.ko 2023-09-10T00:34:35.824 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pps 2023-09-10T00:34:35.825 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pps/clients 2023-09-10T00:34:35.825 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pps/clients/pps-gpio.ko 2023-09-10T00:34:35.825 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pps/clients/pps-ldisc.ko 2023-09-10T00:34:35.825 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pps/clients/pps_parport.ko 2023-09-10T00:34:35.825 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/pps/pps_core.ko 2023-09-10T00:34:35.826 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ptp 2023-09-10T00:34:35.826 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ptp/ptp.ko 2023-09-10T00:34:35.826 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ptp/ptp_kvm.ko 2023-09-10T00:34:35.826 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc 2023-09-10T00:34:35.826 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-bq32k.ko 2023-09-10T00:34:35.826 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-bq4802.ko 2023-09-10T00:34:35.827 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds1286.ko 2023-09-10T00:34:35.827 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds1307.ko 2023-09-10T00:34:35.827 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds1374.ko 2023-09-10T00:34:35.827 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds1511.ko 2023-09-10T00:34:35.827 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds1553.ko 2023-09-10T00:34:35.828 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds1672.ko 2023-09-10T00:34:35.828 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds1742.ko 2023-09-10T00:34:35.828 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds2404.ko 2023-09-10T00:34:35.828 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-ds3232.ko 2023-09-10T00:34:35.828 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-em3027.ko 2023-09-10T00:34:35.829 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-fm3130.ko 2023-09-10T00:34:35.829 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-isl12022.ko 2023-09-10T00:34:35.829 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-isl1208.ko 2023-09-10T00:34:35.829 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-m41t80.ko 2023-09-10T00:34:35.829 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-m48t35.ko 2023-09-10T00:34:35.829 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-m48t59.ko 2023-09-10T00:34:35.830 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-max6900.ko 2023-09-10T00:34:35.830 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-msm6242.ko 2023-09-10T00:34:35.830 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-pcf2127.ko 2023-09-10T00:34:35.830 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-pcf85063.ko 2023-09-10T00:34:35.830 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-pcf8523.ko 2023-09-10T00:34:35.831 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-pcf8563.ko 2023-09-10T00:34:35.831 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-pcf8583.ko 2023-09-10T00:34:35.831 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-rp5c01.ko 2023-09-10T00:34:35.831 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-rs5c372.ko 2023-09-10T00:34:35.831 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-09-10T00:34:35.832 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-rx8025.ko 2023-09-10T00:34:35.832 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-rx8581.ko 2023-09-10T00:34:35.832 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-09-10T00:34:35.832 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/rtc/rtc-x1205.ko 2023-09-10T00:34:35.832 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi 2023-09-10T00:34:35.832 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/3w-9xxx.ko 2023-09-10T00:34:35.833 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/3w-sas.ko 2023-09-10T00:34:35.833 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/3w-xxxx.ko 2023-09-10T00:34:35.833 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/BusLogic.ko 2023-09-10T00:34:35.833 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/a100u2w.ko 2023-09-10T00:34:35.833 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/aacraid 2023-09-10T00:34:35.834 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/aacraid/aacraid.ko 2023-09-10T00:34:35.834 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/advansys.ko 2023-09-10T00:34:35.834 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/aic7xxx 2023-09-10T00:34:35.834 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-09-10T00:34:35.834 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-09-10T00:34:35.835 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/am53c974.ko 2023-09-10T00:34:35.835 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/arcmsr 2023-09-10T00:34:35.835 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-09-10T00:34:35.835 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/atp870u.ko 2023-09-10T00:34:35.835 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/be2iscsi 2023-09-10T00:34:35.835 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-09-10T00:34:35.836 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/bfa 2023-09-10T00:34:35.836 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/bfa/bfa.ko 2023-09-10T00:34:35.836 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/bnx2fc 2023-09-10T00:34:35.836 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-09-10T00:34:35.836 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/bnx2i 2023-09-10T00:34:35.837 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-09-10T00:34:35.837 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/ch.ko 2023-09-10T00:34:35.837 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/csiostor 2023-09-10T00:34:35.837 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/csiostor/csiostor.ko 2023-09-10T00:34:35.837 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/cxgbi 2023-09-10T00:34:35.837 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/cxgbi/cxgb3i 2023-09-10T00:34:35.838 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-09-10T00:34:35.838 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/cxgbi/cxgb4i 2023-09-10T00:34:35.838 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-09-10T00:34:35.838 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-09-10T00:34:35.838 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/dc395x.ko 2023-09-10T00:34:35.839 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/device_handler 2023-09-10T00:34:35.839 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-09-10T00:34:35.839 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-09-10T00:34:35.839 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-09-10T00:34:35.839 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-09-10T00:34:35.840 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/dmx3191d.ko 2023-09-10T00:34:35.840 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/esas2r 2023-09-10T00:34:35.840 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/esas2r/esas2r.ko 2023-09-10T00:34:35.840 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/esp_scsi.ko 2023-09-10T00:34:35.840 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/fcoe 2023-09-10T00:34:35.840 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/fcoe/fcoe.ko 2023-09-10T00:34:35.841 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-09-10T00:34:35.841 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/fnic 2023-09-10T00:34:35.841 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/fnic/fnic.ko 2023-09-10T00:34:35.841 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/hpsa.ko 2023-09-10T00:34:35.841 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/hptiop.ko 2023-09-10T00:34:35.841 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/hv_storvsc.ko 2023-09-10T00:34:35.842 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/initio.ko 2023-09-10T00:34:35.842 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/ipr.ko 2023-09-10T00:34:35.842 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/ips.ko 2023-09-10T00:34:35.842 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/isci 2023-09-10T00:34:35.842 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/isci/isci.ko 2023-09-10T00:34:35.843 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-09-10T00:34:35.843 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/iscsi_tcp.ko 2023-09-10T00:34:35.843 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/libfc 2023-09-10T00:34:35.843 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/libfc/libfc.ko 2023-09-10T00:34:35.843 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/libiscsi.ko 2023-09-10T00:34:35.843 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/libiscsi_tcp.ko 2023-09-10T00:34:35.844 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/libsas 2023-09-10T00:34:35.844 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/libsas/libsas.ko 2023-09-10T00:34:35.844 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/lpfc 2023-09-10T00:34:35.844 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/lpfc/lpfc.ko 2023-09-10T00:34:35.844 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/megaraid 2023-09-10T00:34:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/megaraid.ko 2023-09-10T00:34:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-09-10T00:34:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-09-10T00:34:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-09-10T00:34:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/mpt3sas 2023-09-10T00:34:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-09-10T00:34:35.846 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/mvsas 2023-09-10T00:34:35.846 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/mvsas/mvsas.ko 2023-09-10T00:34:35.846 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/mvumi.ko 2023-09-10T00:34:35.846 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/pm8001 2023-09-10T00:34:35.846 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-09-10T00:34:35.847 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/pmcraid.ko 2023-09-10T00:34:35.847 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/qla1280.ko 2023-09-10T00:34:35.847 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/qla2xxx 2023-09-10T00:34:35.847 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-09-10T00:34:35.847 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-09-10T00:34:35.847 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/qla4xxx 2023-09-10T00:34:35.848 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-09-10T00:34:35.848 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/raid_class.ko 2023-09-10T00:34:35.848 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/scsi_debug.ko 2023-09-10T00:34:35.848 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/scsi_transport_fc.ko 2023-09-10T00:34:35.848 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-09-10T00:34:35.849 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/scsi_transport_sas.ko 2023-09-10T00:34:35.849 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/scsi_transport_spi.ko 2023-09-10T00:34:35.849 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/scsi_transport_srp.ko 2023-09-10T00:34:35.849 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/ses.ko 2023-09-10T00:34:35.849 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/st.ko 2023-09-10T00:34:35.849 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/stex.ko 2023-09-10T00:34:35.850 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/sym53c8xx_2 2023-09-10T00:34:35.850 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-09-10T00:34:35.850 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/virtio_scsi.ko 2023-09-10T00:34:35.850 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/vmw_pvscsi.ko 2023-09-10T00:34:35.850 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/scsi/wd719x.ko 2023-09-10T00:34:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ssb 2023-09-10T00:34:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ssb/ssb.ko 2023-09-10T00:34:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target 2023-09-10T00:34:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/iscsi 2023-09-10T00:34:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-09-10T00:34:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/loopback 2023-09-10T00:34:35.852 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/loopback/tcm_loop.ko 2023-09-10T00:34:35.852 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/target_core_file.ko 2023-09-10T00:34:35.852 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/target_core_iblock.ko 2023-09-10T00:34:35.852 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/target_core_mod.ko 2023-09-10T00:34:35.852 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/target_core_pscsi.ko 2023-09-10T00:34:35.853 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/target_core_user.ko 2023-09-10T00:34:35.853 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/tcm_fc 2023-09-10T00:34:35.853 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-09-10T00:34:35.853 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal 2023-09-10T00:34:35.853 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel 2023-09-10T00:34:35.853 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal 2023-09-10T00:34:35.854 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-09-10T00:34:35.854 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-09-10T00:34:35.854 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-09-10T00:34:35.854 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-09-10T00:34:35.854 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-09-10T00:34:35.855 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-09-10T00:34:35.855 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-09-10T00:34:35.855 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-09-10T00:34:35.855 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-09-10T00:34:35.855 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-09-10T00:34:35.856 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-09-10T00:34:35.856 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-09-10T00:34:35.856 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-09-10T00:34:35.856 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-09-10T00:34:35.856 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-09-10T00:34:35.857 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty 2023-09-10T00:34:35.857 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/ipwireless 2023-09-10T00:34:35.857 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-09-10T00:34:35.857 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/n_gsm.ko 2023-09-10T00:34:35.857 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/n_hdlc.ko 2023-09-10T00:34:35.857 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/nozomi.ko 2023-09-10T00:34:35.858 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/serial 2023-09-10T00:34:35.858 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/serial/8250 2023-09-10T00:34:35.858 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-09-10T00:34:35.858 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/serial/arc_uart.ko 2023-09-10T00:34:35.858 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/serial/jsm 2023-09-10T00:34:35.859 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/serial/jsm/jsm.ko 2023-09-10T00:34:35.859 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/tty/synclink_gt.ko 2023-09-10T00:34:35.859 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ufs 2023-09-10T00:34:35.859 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ufs/core 2023-09-10T00:34:35.859 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ufs/core/ufshcd-core.ko 2023-09-10T00:34:35.859 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ufs/host 2023-09-10T00:34:35.860 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-09-10T00:34:35.860 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/uio 2023-09-10T00:34:35.860 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/uio/uio.ko 2023-09-10T00:34:35.860 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/uio/uio_aec.ko 2023-09-10T00:34:35.860 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/uio/uio_cif.ko 2023-09-10T00:34:35.861 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/uio/uio_pci_generic.ko 2023-09-10T00:34:35.861 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/uio/uio_sercos3.ko 2023-09-10T00:34:35.861 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb 2023-09-10T00:34:35.861 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/class 2023-09-10T00:34:35.861 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/class/cdc-acm.ko 2023-09-10T00:34:35.862 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/class/cdc-wdm.ko 2023-09-10T00:34:35.862 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/class/usblp.ko 2023-09-10T00:34:35.862 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/class/usbtmc.ko 2023-09-10T00:34:35.862 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/host 2023-09-10T00:34:35.862 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/host/sl811-hcd.ko 2023-09-10T00:34:35.862 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/image 2023-09-10T00:34:35.863 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/image/mdc800.ko 2023-09-10T00:34:35.863 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/image/microtek.ko 2023-09-10T00:34:35.863 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc 2023-09-10T00:34:35.863 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/adutux.ko 2023-09-10T00:34:35.863 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/appledisplay.ko 2023-09-10T00:34:35.864 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/emi26.ko 2023-09-10T00:34:35.864 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/emi62.ko 2023-09-10T00:34:35.864 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/ezusb.ko 2023-09-10T00:34:35.864 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/idmouse.ko 2023-09-10T00:34:35.864 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/iowarrior.ko 2023-09-10T00:34:35.864 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/isight_firmware.ko 2023-09-10T00:34:35.865 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/ldusb.ko 2023-09-10T00:34:35.865 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/legousbtower.ko 2023-09-10T00:34:35.865 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/sisusbvga 2023-09-10T00:34:35.865 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-09-10T00:34:35.865 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/trancevibrator.ko 2023-09-10T00:34:35.866 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/usb3503.ko 2023-09-10T00:34:35.866 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/usblcd.ko 2023-09-10T00:34:35.866 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/usbsevseg.ko 2023-09-10T00:34:35.866 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/uss720.ko 2023-09-10T00:34:35.866 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/misc/yurex.ko 2023-09-10T00:34:35.866 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/phy 2023-09-10T00:34:35.867 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/phy/phy-generic.ko 2023-09-10T00:34:35.867 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial 2023-09-10T00:34:35.867 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/aircable.ko 2023-09-10T00:34:35.867 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ark3116.ko 2023-09-10T00:34:35.867 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/belkin_sa.ko 2023-09-10T00:34:35.868 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ch341.ko 2023-09-10T00:34:35.868 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/cp210x.ko 2023-09-10T00:34:35.868 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/cyberjack.ko 2023-09-10T00:34:35.868 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/cypress_m8.ko 2023-09-10T00:34:35.868 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/digi_acceleport.ko 2023-09-10T00:34:35.868 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/empeg.ko 2023-09-10T00:34:35.869 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ftdi_sio.ko 2023-09-10T00:34:35.869 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/garmin_gps.ko 2023-09-10T00:34:35.869 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/io_edgeport.ko 2023-09-10T00:34:35.869 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/io_ti.ko 2023-09-10T00:34:35.869 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ipaq.ko 2023-09-10T00:34:35.870 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ipw.ko 2023-09-10T00:34:35.870 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ir-usb.ko 2023-09-10T00:34:35.870 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-09-10T00:34:35.870 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/keyspan.ko 2023-09-10T00:34:35.870 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/keyspan_pda.ko 2023-09-10T00:34:35.870 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/kl5kusb105.ko 2023-09-10T00:34:35.871 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/kobil_sct.ko 2023-09-10T00:34:35.871 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/mct_u232.ko 2023-09-10T00:34:35.871 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/mos7720.ko 2023-09-10T00:34:35.871 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/mos7840.ko 2023-09-10T00:34:35.871 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/navman.ko 2023-09-10T00:34:35.872 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/omninet.ko 2023-09-10T00:34:35.872 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/opticon.ko 2023-09-10T00:34:35.872 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/option.ko 2023-09-10T00:34:35.872 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/oti6858.ko 2023-09-10T00:34:35.872 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/pl2303.ko 2023-09-10T00:34:35.872 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/qcaux.ko 2023-09-10T00:34:35.873 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/qcserial.ko 2023-09-10T00:34:35.873 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/quatech2.ko 2023-09-10T00:34:35.873 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/safe_serial.ko 2023-09-10T00:34:35.873 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/sierra.ko 2023-09-10T00:34:35.873 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/spcp8x5.ko 2023-09-10T00:34:35.874 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ssu100.ko 2023-09-10T00:34:35.874 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/symbolserial.ko 2023-09-10T00:34:35.874 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-09-10T00:34:35.874 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-09-10T00:34:35.874 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/usb_debug.ko 2023-09-10T00:34:35.874 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/usb_wwan.ko 2023-09-10T00:34:35.875 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/visor.ko 2023-09-10T00:34:35.875 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/whiteheat.ko 2023-09-10T00:34:35.875 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/serial/xsens_mt.ko 2023-09-10T00:34:35.875 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage 2023-09-10T00:34:35.875 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/uas.ko 2023-09-10T00:34:35.876 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-alauda.ko 2023-09-10T00:34:35.876 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-cypress.ko 2023-09-10T00:34:35.876 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-datafab.ko 2023-09-10T00:34:35.876 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-09-10T00:34:35.876 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-freecom.ko 2023-09-10T00:34:35.876 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-isd200.ko 2023-09-10T00:34:35.877 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-09-10T00:34:35.877 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-karma.ko 2023-09-10T00:34:35.877 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-onetouch.ko 2023-09-10T00:34:35.877 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-realtek.ko 2023-09-10T00:34:35.878 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-sddr09.ko 2023-09-10T00:34:35.878 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-sddr55.ko 2023-09-10T00:34:35.878 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/ums-usbat.ko 2023-09-10T00:34:35.878 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/storage/usb-storage.ko 2023-09-10T00:34:35.878 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/usbip 2023-09-10T00:34:35.879 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/usbip/usbip-core.ko 2023-09-10T00:34:35.879 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/usbip/usbip-host.ko 2023-09-10T00:34:35.879 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-09-10T00:34:35.879 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vfio 2023-09-10T00:34:35.879 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vfio/pci 2023-09-10T00:34:35.879 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-09-10T00:34:35.880 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vfio/pci/vfio-pci.ko 2023-09-10T00:34:35.880 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vfio/vfio.ko 2023-09-10T00:34:35.880 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-09-10T00:34:35.880 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vhost 2023-09-10T00:34:35.880 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vhost/vhost.ko 2023-09-10T00:34:35.881 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vhost/vhost_iotlb.ko 2023-09-10T00:34:35.881 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vhost/vhost_net.ko 2023-09-10T00:34:35.881 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/vhost/vhost_scsi.ko 2023-09-10T00:34:35.881 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/video 2023-09-10T00:34:35.881 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/video/backlight 2023-09-10T00:34:35.882 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/video/backlight/apple_bl.ko 2023-09-10T00:34:35.882 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/video/backlight/lcd.ko 2023-09-10T00:34:35.882 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/video/backlight/platform_lcd.ko 2023-09-10T00:34:35.882 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio 2023-09-10T00:34:35.882 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio/virtio.ko 2023-09-10T00:34:35.883 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio/virtio_balloon.ko 2023-09-10T00:34:35.883 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio/virtio_mmio.ko 2023-09-10T00:34:35.883 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio/virtio_pci.ko 2023-09-10T00:34:35.883 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-09-10T00:34:35.883 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-09-10T00:34:35.884 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/virtio/virtio_ring.ko 2023-09-10T00:34:35.884 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1 2023-09-10T00:34:35.884 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/masters 2023-09-10T00:34:35.884 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/masters/ds2482.ko 2023-09-10T00:34:35.884 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/masters/ds2490.ko 2023-09-10T00:34:35.884 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves 2023-09-10T00:34:35.885 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-09-10T00:34:35.885 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-09-10T00:34:35.885 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-09-10T00:34:35.885 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-09-10T00:34:35.885 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-09-10T00:34:35.886 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-09-10T00:34:35.886 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-09-10T00:34:35.886 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-09-10T00:34:35.886 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-09-10T00:34:35.886 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_smem.ko 2023-09-10T00:34:35.887 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/slaves/w1_therm.ko 2023-09-10T00:34:35.887 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/w1/wire.ko 2023-09-10T00:34:35.887 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog 2023-09-10T00:34:35.887 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/alim1535_wdt.ko 2023-09-10T00:34:35.887 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/alim7101_wdt.ko 2023-09-10T00:34:35.888 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/f71808e_wdt.ko 2023-09-10T00:34:35.888 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/hpwdt.ko 2023-09-10T00:34:35.888 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/i6300esb.ko 2023-09-10T00:34:35.888 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-09-10T00:34:35.888 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/iTCO_wdt.ko 2023-09-10T00:34:35.889 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/ib700wdt.ko 2023-09-10T00:34:35.889 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/ibmasr.ko 2023-09-10T00:34:35.889 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-09-10T00:34:35.889 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/it8712f_wdt.ko 2023-09-10T00:34:35.889 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/it87_wdt.ko 2023-09-10T00:34:35.890 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/machzwd.ko 2023-09-10T00:34:35.890 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/nv_tco.ko 2023-09-10T00:34:35.890 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/pcwd_pci.ko 2023-09-10T00:34:35.890 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/pcwd_usb.ko 2023-09-10T00:34:35.890 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-09-10T00:34:35.891 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/sch311x_wdt.ko 2023-09-10T00:34:35.891 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/softdog.ko 2023-09-10T00:34:35.891 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/sp5100_tco.ko 2023-09-10T00:34:35.891 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/via_wdt.ko 2023-09-10T00:34:35.891 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-09-10T00:34:35.891 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/w83877f_wdt.ko 2023-09-10T00:34:35.892 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/w83977f_wdt.ko 2023-09-10T00:34:35.892 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/wdt_pci.ko 2023-09-10T00:34:35.892 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/watchdog/xen_wdt.ko 2023-09-10T00:34:35.892 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen 2023-09-10T00:34:35.892 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xen-acpi-processor.ko 2023-09-10T00:34:35.893 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xen-evtchn.ko 2023-09-10T00:34:35.893 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xen-gntalloc.ko 2023-09-10T00:34:35.893 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xen-gntdev.ko 2023-09-10T00:34:35.893 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xen-pciback 2023-09-10T00:34:35.893 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-09-10T00:34:35.893 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xen-privcmd.ko 2023-09-10T00:34:35.894 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xenfs 2023-09-10T00:34:35.894 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/drivers/xen/xenfs/xenfs.ko 2023-09-10T00:34:35.894 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs 2023-09-10T00:34:35.894 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/9p 2023-09-10T00:34:35.894 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/9p/9p.ko 2023-09-10T00:34:35.895 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/affs 2023-09-10T00:34:35.895 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/affs/affs.ko 2023-09-10T00:34:35.895 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/befs 2023-09-10T00:34:35.896 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/befs/befs.ko 2023-09-10T00:34:35.897 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/binfmt_misc.ko 2023-09-10T00:34:35.897 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/btrfs 2023-09-10T00:34:35.897 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/btrfs/btrfs.ko 2023-09-10T00:34:35.897 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/cachefiles 2023-09-10T00:34:35.897 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/cachefiles/cachefiles.ko 2023-09-10T00:34:35.898 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ceph 2023-09-10T00:34:35.898 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ceph/ceph.ko 2023-09-10T00:34:35.898 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/coda 2023-09-10T00:34:35.898 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/coda/coda.ko 2023-09-10T00:34:35.898 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/cramfs 2023-09-10T00:34:35.899 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/cramfs/cramfs.ko 2023-09-10T00:34:35.899 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/dlm 2023-09-10T00:34:35.899 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/dlm/dlm.ko 2023-09-10T00:34:35.899 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ecryptfs 2023-09-10T00:34:35.899 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ecryptfs/ecryptfs.ko 2023-09-10T00:34:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/f2fs 2023-09-10T00:34:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/f2fs/f2fs.ko 2023-09-10T00:34:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fat 2023-09-10T00:34:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fat/fat.ko 2023-09-10T00:34:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fat/msdos.ko 2023-09-10T00:34:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fat/vfat.ko 2023-09-10T00:34:35.901 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fscache 2023-09-10T00:34:35.901 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fscache/fscache.ko 2023-09-10T00:34:35.901 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fuse 2023-09-10T00:34:35.901 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fuse/cuse.ko 2023-09-10T00:34:35.901 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/fuse/fuse.ko 2023-09-10T00:34:35.902 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/gfs2 2023-09-10T00:34:35.902 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/gfs2/gfs2.ko 2023-09-10T00:34:35.902 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/hfs 2023-09-10T00:34:35.902 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/hfs/hfs.ko 2023-09-10T00:34:35.902 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/hfsplus 2023-09-10T00:34:35.903 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/hfsplus/hfsplus.ko 2023-09-10T00:34:35.903 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/isofs 2023-09-10T00:34:35.903 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/isofs/isofs.ko 2023-09-10T00:34:35.903 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/jfs 2023-09-10T00:34:35.903 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/jfs/jfs.ko 2023-09-10T00:34:35.903 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/lockd 2023-09-10T00:34:35.904 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/lockd/lockd.ko 2023-09-10T00:34:35.904 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/minix 2023-09-10T00:34:35.904 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/minix/minix.ko 2023-09-10T00:34:35.904 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/netfs 2023-09-10T00:34:35.904 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/netfs/netfs.ko 2023-09-10T00:34:35.905 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs 2023-09-10T00:34:35.905 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/blocklayout 2023-09-10T00:34:35.905 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-09-10T00:34:35.905 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/filelayout 2023-09-10T00:34:35.905 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-09-10T00:34:35.906 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/flexfilelayout 2023-09-10T00:34:35.906 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-09-10T00:34:35.906 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/nfs.ko 2023-09-10T00:34:35.906 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/nfsv3.ko 2023-09-10T00:34:35.906 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs/nfsv4.ko 2023-09-10T00:34:35.906 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs_common 2023-09-10T00:34:35.907 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs_common/grace.ko 2023-09-10T00:34:35.907 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfs_common/nfs_acl.ko 2023-09-10T00:34:35.907 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfsd 2023-09-10T00:34:35.907 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nfsd/nfsd.ko 2023-09-10T00:34:35.907 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nilfs2 2023-09-10T00:34:35.908 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nilfs2/nilfs2.ko 2023-09-10T00:34:35.908 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls 2023-09-10T00:34:35.908 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-celtic.ko 2023-09-10T00:34:35.908 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-centeuro.ko 2023-09-10T00:34:35.908 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-croatian.ko 2023-09-10T00:34:35.909 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-cyrillic.ko 2023-09-10T00:34:35.909 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-gaelic.ko 2023-09-10T00:34:35.909 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-greek.ko 2023-09-10T00:34:35.909 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-iceland.ko 2023-09-10T00:34:35.909 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-inuit.ko 2023-09-10T00:34:35.909 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-roman.ko 2023-09-10T00:34:35.910 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-romanian.ko 2023-09-10T00:34:35.910 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/mac-turkish.ko 2023-09-10T00:34:35.910 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp1250.ko 2023-09-10T00:34:35.910 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp1251.ko 2023-09-10T00:34:35.910 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp1255.ko 2023-09-10T00:34:35.911 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp737.ko 2023-09-10T00:34:35.911 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp775.ko 2023-09-10T00:34:35.911 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp850.ko 2023-09-10T00:34:35.911 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp852.ko 2023-09-10T00:34:35.911 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp855.ko 2023-09-10T00:34:35.912 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp857.ko 2023-09-10T00:34:35.912 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp860.ko 2023-09-10T00:34:35.912 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp861.ko 2023-09-10T00:34:35.912 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp862.ko 2023-09-10T00:34:35.912 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp863.ko 2023-09-10T00:34:35.913 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp864.ko 2023-09-10T00:34:35.913 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp865.ko 2023-09-10T00:34:35.913 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp866.ko 2023-09-10T00:34:35.913 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp869.ko 2023-09-10T00:34:35.913 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp874.ko 2023-09-10T00:34:35.914 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp932.ko 2023-09-10T00:34:35.914 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp936.ko 2023-09-10T00:34:35.914 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp949.ko 2023-09-10T00:34:35.914 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_cp950.ko 2023-09-10T00:34:35.914 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_euc-jp.ko 2023-09-10T00:34:35.914 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-1.ko 2023-09-10T00:34:35.915 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-13.ko 2023-09-10T00:34:35.915 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-14.ko 2023-09-10T00:34:35.915 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-15.ko 2023-09-10T00:34:35.915 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-2.ko 2023-09-10T00:34:35.915 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-3.ko 2023-09-10T00:34:35.916 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-4.ko 2023-09-10T00:34:35.916 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-5.ko 2023-09-10T00:34:35.916 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-6.ko 2023-09-10T00:34:35.916 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-7.ko 2023-09-10T00:34:35.916 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_iso8859-9.ko 2023-09-10T00:34:35.917 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_koi8-r.ko 2023-09-10T00:34:35.917 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_koi8-ru.ko 2023-09-10T00:34:35.917 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_koi8-u.ko 2023-09-10T00:34:35.917 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/nls/nls_utf8.ko 2023-09-10T00:34:35.917 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2 2023-09-10T00:34:35.918 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/cluster 2023-09-10T00:34:35.918 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-09-10T00:34:35.918 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/dlm 2023-09-10T00:34:35.918 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-09-10T00:34:35.918 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/dlmfs 2023-09-10T00:34:35.919 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-09-10T00:34:35.919 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/ocfs2.ko 2023-09-10T00:34:35.919 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-09-10T00:34:35.919 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-09-10T00:34:35.919 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-09-10T00:34:35.920 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/overlayfs 2023-09-10T00:34:35.920 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/overlayfs/overlay.ko 2023-09-10T00:34:35.920 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/pstore 2023-09-10T00:34:35.920 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/pstore/ramoops.ko 2023-09-10T00:34:35.920 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/reiserfs 2023-09-10T00:34:35.921 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/reiserfs/reiserfs.ko 2023-09-10T00:34:35.921 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/romfs 2023-09-10T00:34:35.921 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/romfs/romfs.ko 2023-09-10T00:34:35.921 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/smb 2023-09-10T00:34:35.921 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/smb/client 2023-09-10T00:34:35.922 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/smb/client/cifs.ko 2023-09-10T00:34:35.922 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/smb/common 2023-09-10T00:34:35.922 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/smb/common/cifs_arc4.ko 2023-09-10T00:34:35.922 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/smb/common/cifs_md4.ko 2023-09-10T00:34:35.922 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/squashfs 2023-09-10T00:34:35.923 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/squashfs/squashfs.ko 2023-09-10T00:34:35.923 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/sysv 2023-09-10T00:34:35.923 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/sysv/sysv.ko 2023-09-10T00:34:35.923 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/udf 2023-09-10T00:34:35.923 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/udf/udf.ko 2023-09-10T00:34:35.923 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ufs 2023-09-10T00:34:35.924 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/ufs/ufs.ko 2023-09-10T00:34:35.924 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/xfs 2023-09-10T00:34:35.924 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/fs/xfs/xfs.ko 2023-09-10T00:34:35.924 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/kernel 2023-09-10T00:34:35.925 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/kernel/rcu 2023-09-10T00:34:35.925 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/kernel/rcu/rcutorture.ko 2023-09-10T00:34:35.925 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/kernel/torture.ko 2023-09-10T00:34:35.925 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/kernel/trace 2023-09-10T00:34:35.925 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-09-10T00:34:35.926 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib 2023-09-10T00:34:35.926 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/asn1_encoder.ko 2023-09-10T00:34:35.926 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/crc-itu-t.ko 2023-09-10T00:34:35.926 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/crc8.ko 2023-09-10T00:34:35.926 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/crypto 2023-09-10T00:34:35.927 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/crypto/libarc4.ko 2023-09-10T00:34:35.927 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/crypto/libdes.ko 2023-09-10T00:34:35.927 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/libcrc32c.ko 2023-09-10T00:34:35.927 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/lru_cache.ko 2023-09-10T00:34:35.927 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/lz4 2023-09-10T00:34:35.928 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/lz4/lz4_compress.ko 2023-09-10T00:34:35.928 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/lz4/lz4hc_compress.ko 2023-09-10T00:34:35.928 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/math 2023-09-10T00:34:35.928 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/math/cordic.ko 2023-09-10T00:34:35.928 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/raid6 2023-09-10T00:34:35.928 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/raid6/raid6_pq.ko 2023-09-10T00:34:35.929 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/reed_solomon 2023-09-10T00:34:35.929 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/reed_solomon/reed_solomon.ko 2023-09-10T00:34:35.929 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/ts_bm.ko 2023-09-10T00:34:35.929 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/ts_fsm.ko 2023-09-10T00:34:35.929 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/ts_kmp.ko 2023-09-10T00:34:35.930 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/zstd 2023-09-10T00:34:35.930 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/lib/zstd/zstd_compress.ko 2023-09-10T00:34:35.930 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/mm 2023-09-10T00:34:35.930 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/mm/hwpoison-inject.ko 2023-09-10T00:34:35.930 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net 2023-09-10T00:34:35.931 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan 2023-09-10T00:34:35.931 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/6lowpan.ko 2023-09-10T00:34:35.931 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/nhc_dest.ko 2023-09-10T00:34:35.931 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/nhc_fragment.ko 2023-09-10T00:34:35.931 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/nhc_hop.ko 2023-09-10T00:34:35.932 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/nhc_ipv6.ko 2023-09-10T00:34:35.932 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/nhc_mobility.ko 2023-09-10T00:34:35.932 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/nhc_routing.ko 2023-09-10T00:34:35.932 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/6lowpan/nhc_udp.ko 2023-09-10T00:34:35.932 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/802 2023-09-10T00:34:35.933 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/802/garp.ko 2023-09-10T00:34:35.933 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/802/mrp.ko 2023-09-10T00:34:35.933 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/802/p8022.ko 2023-09-10T00:34:35.933 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/802/psnap.ko 2023-09-10T00:34:35.933 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/802/stp.ko 2023-09-10T00:34:35.933 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/8021q 2023-09-10T00:34:35.934 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/8021q/8021q.ko 2023-09-10T00:34:35.934 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/9p 2023-09-10T00:34:35.934 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/9p/9pnet.ko 2023-09-10T00:34:35.934 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/9p/9pnet_fd.ko 2023-09-10T00:34:35.934 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/9p/9pnet_rdma.ko 2023-09-10T00:34:35.935 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/9p/9pnet_virtio.ko 2023-09-10T00:34:35.935 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/batman-adv 2023-09-10T00:34:35.935 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/batman-adv/batman-adv.ko 2023-09-10T00:34:35.935 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge 2023-09-10T00:34:35.935 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/br_netfilter.ko 2023-09-10T00:34:35.936 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/bridge.ko 2023-09-10T00:34:35.936 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter 2023-09-10T00:34:35.936 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-09-10T00:34:35.936 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_among.ko 2023-09-10T00:34:35.936 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_arp.ko 2023-09-10T00:34:35.936 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-09-10T00:34:35.937 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-09-10T00:34:35.937 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_ip.ko 2023-09-10T00:34:35.937 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-09-10T00:34:35.937 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_limit.ko 2023-09-10T00:34:35.937 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_log.ko 2023-09-10T00:34:35.937 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_mark.ko 2023-09-10T00:34:35.938 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-09-10T00:34:35.938 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-09-10T00:34:35.938 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-09-10T00:34:35.938 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-09-10T00:34:35.938 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_snat.ko 2023-09-10T00:34:35.939 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_stp.ko 2023-09-10T00:34:35.939 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-09-10T00:34:35.939 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-09-10T00:34:35.939 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-09-10T00:34:35.939 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-09-10T00:34:35.940 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/ebtables.ko 2023-09-10T00:34:35.940 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-09-10T00:34:35.940 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-09-10T00:34:35.940 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-09-10T00:34:35.940 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ceph 2023-09-10T00:34:35.941 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ceph/libceph.ko 2023-09-10T00:34:35.941 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/core 2023-09-10T00:34:35.941 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/core/failover.ko 2023-09-10T00:34:35.941 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/core/pktgen.ko 2023-09-10T00:34:35.942 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dccp 2023-09-10T00:34:35.942 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dccp/dccp.ko 2023-09-10T00:34:35.942 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dccp/dccp_diag.ko 2023-09-10T00:34:35.942 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dccp/dccp_ipv4.ko 2023-09-10T00:34:35.943 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dccp/dccp_ipv6.ko 2023-09-10T00:34:35.943 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dns_resolver 2023-09-10T00:34:35.943 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dns_resolver/dns_resolver.ko 2023-09-10T00:34:35.943 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dsa 2023-09-10T00:34:35.943 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dsa/dsa_core.ko 2023-09-10T00:34:35.943 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dsa/tag_brcm.ko 2023-09-10T00:34:35.944 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dsa/tag_dsa.ko 2023-09-10T00:34:35.944 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dsa/tag_none.ko 2023-09-10T00:34:35.944 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/dsa/tag_trailer.ko 2023-09-10T00:34:35.944 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ieee802154 2023-09-10T00:34:35.944 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ieee802154/6lowpan 2023-09-10T00:34:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-09-10T00:34:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ieee802154/ieee802154.ko 2023-09-10T00:34:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ieee802154/ieee802154_socket.ko 2023-09-10T00:34:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4 2023-09-10T00:34:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/ah4.ko 2023-09-10T00:34:35.945 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/esp4.ko 2023-09-10T00:34:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/fou.ko 2023-09-10T00:34:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/gre.ko 2023-09-10T00:34:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/inet_diag.ko 2023-09-10T00:34:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/ip_gre.ko 2023-09-10T00:34:35.946 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/ip_tunnel.ko 2023-09-10T00:34:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/ip_vti.ko 2023-09-10T00:34:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/ipcomp.ko 2023-09-10T00:34:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/ipip.ko 2023-09-10T00:34:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter 2023-09-10T00:34:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/arp_tables.ko 2023-09-10T00:34:35.947 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-09-10T00:34:35.948 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-09-10T00:34:35.948 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-09-10T00:34:35.948 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-09-10T00:34:35.948 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-09-10T00:34:35.948 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-09-10T00:34:35.949 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-09-10T00:34:35.949 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-09-10T00:34:35.949 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-09-10T00:34:35.949 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/iptable_security.ko 2023-09-10T00:34:35.949 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-09-10T00:34:35.950 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-09-10T00:34:35.950 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-09-10T00:34:35.950 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-09-10T00:34:35.950 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-09-10T00:34:35.950 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-09-10T00:34:35.950 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-09-10T00:34:35.951 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-09-10T00:34:35.951 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-09-10T00:34:35.951 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-09-10T00:34:35.951 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_bic.ko 2023-09-10T00:34:35.951 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_dctcp.ko 2023-09-10T00:34:35.952 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_diag.ko 2023-09-10T00:34:35.952 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_highspeed.ko 2023-09-10T00:34:35.952 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_htcp.ko 2023-09-10T00:34:35.952 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_hybla.ko 2023-09-10T00:34:35.952 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_illinois.ko 2023-09-10T00:34:35.952 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_lp.ko 2023-09-10T00:34:35.953 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_scalable.ko 2023-09-10T00:34:35.953 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_vegas.ko 2023-09-10T00:34:35.953 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_veno.ko 2023-09-10T00:34:35.953 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_westwood.ko 2023-09-10T00:34:35.954 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tcp_yeah.ko 2023-09-10T00:34:35.954 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/tunnel4.ko 2023-09-10T00:34:35.954 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/udp_diag.ko 2023-09-10T00:34:35.954 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/udp_tunnel.ko 2023-09-10T00:34:35.954 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv4/xfrm4_tunnel.ko 2023-09-10T00:34:35.955 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6 2023-09-10T00:34:35.955 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/ah6.ko 2023-09-10T00:34:35.955 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/esp6.ko 2023-09-10T00:34:35.955 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/fou6.ko 2023-09-10T00:34:35.956 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/ip6_tunnel.ko 2023-09-10T00:34:35.956 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-09-10T00:34:35.956 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/ip6_vti.ko 2023-09-10T00:34:35.956 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/ipcomp6.ko 2023-09-10T00:34:35.956 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter 2023-09-10T00:34:35.956 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-09-10T00:34:35.957 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-09-10T00:34:35.957 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-09-10T00:34:35.957 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-09-10T00:34:35.957 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-09-10T00:34:35.957 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-09-10T00:34:35.958 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-09-10T00:34:35.958 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-09-10T00:34:35.958 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-09-10T00:34:35.958 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-09-10T00:34:35.958 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-09-10T00:34:35.958 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-09-10T00:34:35.959 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-09-10T00:34:35.959 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-09-10T00:34:35.959 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-09-10T00:34:35.959 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-09-10T00:34:35.959 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-09-10T00:34:35.960 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-09-10T00:34:35.960 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-09-10T00:34:35.960 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-09-10T00:34:35.960 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-09-10T00:34:35.960 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-09-10T00:34:35.961 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-09-10T00:34:35.961 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-09-10T00:34:35.961 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/sit.ko 2023-09-10T00:34:35.961 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/tunnel6.ko 2023-09-10T00:34:35.961 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/ipv6/xfrm6_tunnel.ko 2023-09-10T00:34:35.961 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/key 2023-09-10T00:34:35.962 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/key/af_key.ko 2023-09-10T00:34:35.962 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp 2023-09-10T00:34:35.962 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp/l2tp_core.ko 2023-09-10T00:34:35.962 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp/l2tp_debugfs.ko 2023-09-10T00:34:35.962 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp/l2tp_eth.ko 2023-09-10T00:34:35.963 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp/l2tp_ip.ko 2023-09-10T00:34:35.963 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp/l2tp_ip6.ko 2023-09-10T00:34:35.963 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp/l2tp_netlink.ko 2023-09-10T00:34:35.963 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/l2tp/l2tp_ppp.ko 2023-09-10T00:34:35.963 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/llc 2023-09-10T00:34:35.963 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/llc/llc.ko 2023-09-10T00:34:35.964 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/mac802154 2023-09-10T00:34:35.964 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/mac802154/mac802154.ko 2023-09-10T00:34:35.964 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/mpls 2023-09-10T00:34:35.964 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/mpls/mpls_gso.ko 2023-09-10T00:34:35.964 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter 2023-09-10T00:34:35.965 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset 2023-09-10T00:34:35.965 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set.ko 2023-09-10T00:34:35.965 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-09-10T00:34:35.965 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-09-10T00:34:35.965 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-09-10T00:34:35.965 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-09-10T00:34:35.966 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-09-10T00:34:35.966 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-09-10T00:34:35.966 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-09-10T00:34:35.966 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-09-10T00:34:35.966 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-09-10T00:34:35.967 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-09-10T00:34:35.967 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-09-10T00:34:35.967 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-09-10T00:34:35.967 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-09-10T00:34:35.967 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-09-10T00:34:35.967 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-09-10T00:34:35.968 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs 2023-09-10T00:34:35.968 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs.ko 2023-09-10T00:34:35.968 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-09-10T00:34:35.968 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-09-10T00:34:35.968 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-09-10T00:34:35.969 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-09-10T00:34:35.969 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-09-10T00:34:35.969 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-09-10T00:34:35.969 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-09-10T00:34:35.969 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-09-10T00:34:35.970 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-09-10T00:34:35.970 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-09-10T00:34:35.970 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-09-10T00:34:35.970 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-09-10T00:34:35.970 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-09-10T00:34:35.971 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conncount.ko 2023-09-10T00:34:35.971 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack.ko 2023-09-10T00:34:35.971 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-09-10T00:34:35.971 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-09-10T00:34:35.971 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-09-10T00:34:35.971 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_h323.ko 2023-09-10T00:34:35.972 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_irc.ko 2023-09-10T00:34:35.972 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-09-10T00:34:35.972 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-09-10T00:34:35.972 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-09-10T00:34:35.972 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_sane.ko 2023-09-10T00:34:35.973 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_sip.ko 2023-09-10T00:34:35.973 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-09-10T00:34:35.973 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-09-10T00:34:35.973 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_dup_netdev.ko 2023-09-10T00:34:35.973 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_flow_table.ko 2023-09-10T00:34:35.974 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_flow_table_inet.ko 2023-09-10T00:34:35.974 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_log_syslog.ko 2023-09-10T00:34:35.974 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_nat.ko 2023-09-10T00:34:35.974 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_nat_amanda.ko 2023-09-10T00:34:35.974 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_nat_ftp.ko 2023-09-10T00:34:35.974 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_nat_irc.ko 2023-09-10T00:34:35.975 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_nat_sip.ko 2023-09-10T00:34:35.975 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_nat_tftp.ko 2023-09-10T00:34:35.975 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_synproxy_core.ko 2023-09-10T00:34:35.975 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nf_tables.ko 2023-09-10T00:34:35.975 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nfnetlink.ko 2023-09-10T00:34:35.976 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nfnetlink_acct.ko 2023-09-10T00:34:35.976 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nfnetlink_log.ko 2023-09-10T00:34:35.976 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nfnetlink_osf.ko 2023-09-10T00:34:35.976 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nfnetlink_queue.ko 2023-09-10T00:34:35.976 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_chain_nat.ko 2023-09-10T00:34:35.976 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_compat.ko 2023-09-10T00:34:35.977 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_ct.ko 2023-09-10T00:34:35.977 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_dup_netdev.ko 2023-09-10T00:34:35.977 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_fib.ko 2023-09-10T00:34:35.977 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_fwd_netdev.ko 2023-09-10T00:34:35.978 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_hash.ko 2023-09-10T00:34:35.978 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_limit.ko 2023-09-10T00:34:35.978 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_log.ko 2023-09-10T00:34:35.978 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_masq.ko 2023-09-10T00:34:35.978 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_nat.ko 2023-09-10T00:34:35.978 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_numgen.ko 2023-09-10T00:34:35.979 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_queue.ko 2023-09-10T00:34:35.979 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_quota.ko 2023-09-10T00:34:35.979 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_redir.ko 2023-09-10T00:34:35.979 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_reject.ko 2023-09-10T00:34:35.979 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_reject_inet.ko 2023-09-10T00:34:35.980 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_socket.ko 2023-09-10T00:34:35.980 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/nft_tunnel.ko 2023-09-10T00:34:35.980 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_AUDIT.ko 2023-09-10T00:34:35.980 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_CHECKSUM.ko 2023-09-10T00:34:35.980 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_CLASSIFY.ko 2023-09-10T00:34:35.980 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-09-10T00:34:35.981 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_CT.ko 2023-09-10T00:34:35.981 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_DSCP.ko 2023-09-10T00:34:35.981 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_HL.ko 2023-09-10T00:34:35.981 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_HMARK.ko 2023-09-10T00:34:35.981 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_IDLETIMER.ko 2023-09-10T00:34:35.982 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_LED.ko 2023-09-10T00:34:35.982 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_LOG.ko 2023-09-10T00:34:35.982 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_MASQUERADE.ko 2023-09-10T00:34:35.982 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_NETMAP.ko 2023-09-10T00:34:35.982 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_NFLOG.ko 2023-09-10T00:34:35.982 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_NFQUEUE.ko 2023-09-10T00:34:35.983 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_RATEEST.ko 2023-09-10T00:34:35.983 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_REDIRECT.ko 2023-09-10T00:34:35.983 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_SECMARK.ko 2023-09-10T00:34:35.983 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_TCPMSS.ko 2023-09-10T00:34:35.983 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-09-10T00:34:35.984 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_TEE.ko 2023-09-10T00:34:35.984 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_TPROXY.ko 2023-09-10T00:34:35.984 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_TRACE.ko 2023-09-10T00:34:35.984 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_addrtype.ko 2023-09-10T00:34:35.984 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_bpf.ko 2023-09-10T00:34:35.985 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_cgroup.ko 2023-09-10T00:34:35.985 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_cluster.ko 2023-09-10T00:34:35.985 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_comment.ko 2023-09-10T00:34:35.985 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_connbytes.ko 2023-09-10T00:34:35.985 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_connlabel.ko 2023-09-10T00:34:35.985 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_connlimit.ko 2023-09-10T00:34:35.986 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_connmark.ko 2023-09-10T00:34:35.986 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_conntrack.ko 2023-09-10T00:34:35.986 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_cpu.ko 2023-09-10T00:34:35.986 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_dccp.ko 2023-09-10T00:34:35.986 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_devgroup.ko 2023-09-10T00:34:35.987 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_dscp.ko 2023-09-10T00:34:35.987 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_ecn.ko 2023-09-10T00:34:35.987 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_esp.ko 2023-09-10T00:34:35.987 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_hashlimit.ko 2023-09-10T00:34:35.987 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_helper.ko 2023-09-10T00:34:35.987 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_hl.ko 2023-09-10T00:34:35.988 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_ipcomp.ko 2023-09-10T00:34:35.988 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_iprange.ko 2023-09-10T00:34:35.988 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_ipvs.ko 2023-09-10T00:34:35.988 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_l2tp.ko 2023-09-10T00:34:35.988 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_length.ko 2023-09-10T00:34:35.989 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_limit.ko 2023-09-10T00:34:35.989 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_mac.ko 2023-09-10T00:34:35.989 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_mark.ko 2023-09-10T00:34:35.989 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_multiport.ko 2023-09-10T00:34:35.989 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_nat.ko 2023-09-10T00:34:35.989 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_nfacct.ko 2023-09-10T00:34:35.990 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_osf.ko 2023-09-10T00:34:35.990 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_owner.ko 2023-09-10T00:34:35.990 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_physdev.ko 2023-09-10T00:34:35.990 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_pkttype.ko 2023-09-10T00:34:35.990 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_policy.ko 2023-09-10T00:34:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_quota.ko 2023-09-10T00:34:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_rateest.ko 2023-09-10T00:34:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_realm.ko 2023-09-10T00:34:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_recent.ko 2023-09-10T00:34:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_sctp.ko 2023-09-10T00:34:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_set.ko 2023-09-10T00:34:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_socket.ko 2023-09-10T00:34:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_state.ko 2023-09-10T00:34:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_statistic.ko 2023-09-10T00:34:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_string.ko 2023-09-10T00:34:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_tcpmss.ko 2023-09-10T00:34:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_time.ko 2023-09-10T00:34:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netfilter/xt_u32.ko 2023-09-10T00:34:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netlink 2023-09-10T00:34:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/netlink/netlink_diag.ko 2023-09-10T00:34:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/nsh 2023-09-10T00:34:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/nsh/nsh.ko 2023-09-10T00:34:35.994 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/openvswitch 2023-09-10T00:34:35.994 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/openvswitch/openvswitch.ko 2023-09-10T00:34:35.994 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/openvswitch/vport-geneve.ko 2023-09-10T00:34:35.994 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/openvswitch/vport-gre.ko 2023-09-10T00:34:35.995 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/openvswitch/vport-vxlan.ko 2023-09-10T00:34:35.995 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/packet 2023-09-10T00:34:35.995 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/packet/af_packet_diag.ko 2023-09-10T00:34:35.995 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/rds 2023-09-10T00:34:35.995 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/rds/rds.ko 2023-09-10T00:34:35.996 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/rds/rds_rdma.ko 2023-09-10T00:34:35.996 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/rds/rds_tcp.ko 2023-09-10T00:34:35.996 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched 2023-09-10T00:34:35.996 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_csum.ko 2023-09-10T00:34:35.996 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_gact.ko 2023-09-10T00:34:35.996 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_ipt.ko 2023-09-10T00:34:35.997 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_mirred.ko 2023-09-10T00:34:35.997 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_nat.ko 2023-09-10T00:34:35.997 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_pedit.ko 2023-09-10T00:34:35.997 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_police.ko 2023-09-10T00:34:35.997 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_simple.ko 2023-09-10T00:34:35.998 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_skbedit.ko 2023-09-10T00:34:35.998 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/act_vlan.ko 2023-09-10T00:34:35.998 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/cls_basic.ko 2023-09-10T00:34:35.998 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/cls_bpf.ko 2023-09-10T00:34:35.998 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/cls_flow.ko 2023-09-10T00:34:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/cls_fw.ko 2023-09-10T00:34:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/cls_route.ko 2023-09-10T00:34:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/cls_u32.ko 2023-09-10T00:34:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/em_cmp.ko 2023-09-10T00:34:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/em_ipset.ko 2023-09-10T00:34:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/em_meta.ko 2023-09-10T00:34:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/em_nbyte.ko 2023-09-10T00:34:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/em_text.ko 2023-09-10T00:34:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/em_u32.ko 2023-09-10T00:34:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_choke.ko 2023-09-10T00:34:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_codel.ko 2023-09-10T00:34:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_drr.ko 2023-09-10T00:34:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_fq.ko 2023-09-10T00:34:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_gred.ko 2023-09-10T00:34:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_hfsc.ko 2023-09-10T00:34:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_hhf.ko 2023-09-10T00:34:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_htb.ko 2023-09-10T00:34:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_ingress.ko 2023-09-10T00:34:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_mqprio.ko 2023-09-10T00:34:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_mqprio_lib.ko 2023-09-10T00:34:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_multiq.ko 2023-09-10T00:34:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_netem.ko 2023-09-10T00:34:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_pie.ko 2023-09-10T00:34:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_plug.ko 2023-09-10T00:34:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_prio.ko 2023-09-10T00:34:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_qfq.ko 2023-09-10T00:34:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_red.ko 2023-09-10T00:34:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_sfb.ko 2023-09-10T00:34:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_sfq.ko 2023-09-10T00:34:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_tbf.ko 2023-09-10T00:34:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sched/sch_teql.ko 2023-09-10T00:34:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sctp 2023-09-10T00:34:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sctp/sctp.ko 2023-09-10T00:34:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sctp/sctp_diag.ko 2023-09-10T00:34:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sunrpc 2023-09-10T00:34:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sunrpc/auth_gss 2023-09-10T00:34:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-09-10T00:34:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-09-10T00:34:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sunrpc/sunrpc.ko 2023-09-10T00:34:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sunrpc/xprtrdma 2023-09-10T00:34:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-09-10T00:34:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/tipc 2023-09-10T00:34:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/tipc/diag.ko 2023-09-10T00:34:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/tipc/tipc.ko 2023-09-10T00:34:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/unix 2023-09-10T00:34:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/unix/unix_diag.ko 2023-09-10T00:34:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/vmw_vsock 2023-09-10T00:34:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-09-10T00:34:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-09-10T00:34:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/vmw_vsock/vsock.ko 2023-09-10T00:34:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/vmw_vsock/vsock_diag.ko 2023-09-10T00:34:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/vmw_vsock/vsock_loopback.ko 2023-09-10T00:34:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/xfrm 2023-09-10T00:34:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/net/xfrm/xfrm_ipcomp.ko 2023-09-10T00:34:36.009 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/security 2023-09-10T00:34:36.009 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/security/keys 2023-09-10T00:34:36.009 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/security/keys/encrypted-keys 2023-09-10T00:34:36.009 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-09-10T00:34:36.010 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/security/keys/trusted-keys 2023-09-10T00:34:36.010 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/security/keys/trusted-keys/trusted.ko 2023-09-10T00:34:36.010 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/virt 2023-09-10T00:34:36.010 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/virt/lib 2023-09-10T00:34:36.010 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/kernel/virt/lib/irqbypass.ko 2023-09-10T00:34:36.010 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.alias 2023-09-10T00:34:36.011 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.alias.bin 2023-09-10T00:34:36.011 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.builtin 2023-09-10T00:34:36.011 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.builtin.bin 2023-09-10T00:34:36.011 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.builtin.modinfo 2023-09-10T00:34:36.011 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.dep 2023-09-10T00:34:36.012 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.dep.bin 2023-09-10T00:34:36.012 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.devname 2023-09-10T00:34:36.012 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.order 2023-09-10T00:34:36.012 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.softdep 2023-09-10T00:34:36.012 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.symbols 2023-09-10T00:34:36.012 INFO:teuthology.orchestra.run.smithi003.stdout:/lib/modules/6.5.0-rc7-g533e5fc8ca06/modules.symbols.bin 2023-09-10T00:34:36.013 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:36.014 DEBUG:teuthology.orchestra.run.smithi003:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-09-10T00:34:36.042 INFO:teuthology.orchestra.run.smithi003.stdout:postinstall scriptlet (using /bin/sh): 2023-09-10T00:34:36.042 INFO:teuthology.orchestra.run.smithi003.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc7-g533e5fc8ca06 -a -r /boot/System.map-6.5.0-rc7-g533e5fc8ca06 ]; then 2023-09-10T00:34:36.042 INFO:teuthology.orchestra.run.smithi003.stdout:cp /boot/vmlinuz-6.5.0-rc7-g533e5fc8ca06 /boot/.vmlinuz-6.5.0-rc7-g533e5fc8ca06-rpm 2023-09-10T00:34:36.042 INFO:teuthology.orchestra.run.smithi003.stdout:cp /boot/System.map-6.5.0-rc7-g533e5fc8ca06 /boot/.System.map-6.5.0-rc7-g533e5fc8ca06-rpm 2023-09-10T00:34:36.043 INFO:teuthology.orchestra.run.smithi003.stdout:rm -f /boot/vmlinuz-6.5.0-rc7-g533e5fc8ca06 /boot/System.map-6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:36.043 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/installkernel 6.5.0-rc7-g533e5fc8ca06 /boot/.vmlinuz-6.5.0-rc7-g533e5fc8ca06-rpm /boot/.System.map-6.5.0-rc7-g533e5fc8ca06-rpm 2023-09-10T00:34:36.043 INFO:teuthology.orchestra.run.smithi003.stdout:rm -f /boot/.vmlinuz-6.5.0-rc7-g533e5fc8ca06-rpm /boot/.System.map-6.5.0-rc7-g533e5fc8ca06-rpm 2023-09-10T00:34:36.043 INFO:teuthology.orchestra.run.smithi003.stdout:fi 2023-09-10T00:34:36.043 INFO:teuthology.orchestra.run.smithi003.stdout:preuninstall scriptlet (using /bin/sh): 2023-09-10T00:34:36.044 INFO:teuthology.orchestra.run.smithi003.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-09-10T00:34:36.044 INFO:teuthology.orchestra.run.smithi003.stdout:new-kernel-pkg --remove 6.5.0-rc7-g533e5fc8ca06 --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc7-g533e5fc8ca06.img 2023-09-10T00:34:36.044 INFO:teuthology.orchestra.run.smithi003.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-09-10T00:34:36.044 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-install remove 6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:36.044 INFO:teuthology.orchestra.run.smithi003.stdout:fi 2023-09-10T00:34:36.044 INFO:teuthology.orchestra.run.smithi003.stdout:postuninstall scriptlet (using /bin/sh): 2023-09-10T00:34:36.045 INFO:teuthology.orchestra.run.smithi003.stdout:if [ -x /sbin/update-bootloader ]; then 2023-09-10T00:34:36.045 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/update-bootloader --remove 6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:36.045 INFO:teuthology.orchestra.run.smithi003.stdout:fi 2023-09-10T00:34:36.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2023-09-10T00:34:36.126 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2023-09-10T00:34:36.127 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2023-09-10T00:34:36.127 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.02 2023-09-10T00:34:36.127 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 123.el8 2023-09-10T00:34:36.127 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2023-09-10T00:34:36.128 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-09-10T00:34:36.128 INFO:teuthology.orchestra.run.smithi003.stdout:Group : System Environment/Base 2023-09-10T00:34:36.128 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 9265450 2023-09-10T00:34:36.128 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2023-09-10T00:34:36.128 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-09-10T00:34:36.129 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-09-10T00:34:36.129 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-09-10T00:34:36.129 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-09-10T00:34:36.129 INFO:teuthology.orchestra.run.smithi003.stdout:Relocations : (not relocatable) 2023-09-10T00:34:36.129 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : Red Hat, Inc. 2023-09-10T00:34:36.130 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : Red Hat, Inc. 2023-09-10T00:34:36.130 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-10T00:34:36.130 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2023-09-10T00:34:36.130 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2023-09-10T00:34:36.130 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:34:36.130 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-10T00:34:36.131 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-10T00:34:36.131 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-10T00:34:36.131 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2023-09-10T00:34:36.131 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:34:36.131 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2023-09-10T00:34:36.132 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-10T00:34:36.133 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:36.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-10T00:34:40.813 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2023-09-10T00:34:42.772 INFO:teuthology.orchestra.run.smithi003.stderr:done 2023-09-10T00:34:42.774 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-09-10T00:34:42.806 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.bUfetnjmhX 2023-09-10T00:34:42.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bUfetnjmhX 2023-09-10T00:34:42.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.bUfetnjmhX 2023-09-10T00:34:43.031 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.bUfetnjmhX is 6KB 2023-09-10T00:34:43.083 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.bUfetnjmhX 2023-09-10T00:34:43.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2023-09-10T00:34:43.167 INFO:teuthology.orchestra.run.smithi003.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-09-10T00:34:43.167 INFO:teuthology.orchestra.run.smithi003.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-09-10T00:34:43.167 INFO:teuthology.orchestra.run.smithi003.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc7-g533e5fc8ca06.conf 2023-09-10T00:34:43.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:34:43.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2023-09-10T00:35:13.459 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-09-10T00:35:13.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-09-10T00:35:13.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:36:13.464 DEBUG:teuthology.orchestra.remote:timed out 2023-09-10T00:36:22.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-09-10T00:36:22.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:36:25.857 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-09-10T00:36:26.284 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-09-10T00:36:26.284 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.5.0-rc7-g533e5fc8ca06"... 2023-09-10T00:36:26.284 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-09-10T00:36:26.327 INFO:teuthology.orchestra.run.smithi003.stdout:6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:36:26.327 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc7-g533e5fc8ca06 vs 6.5.0-rc7-g533e5fc8ca06 2023-09-10T00:36:26.327 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-10T00:36:26.328 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-10T00:36:27.329 DEBUG:teuthology.parallel:result is None 2023-09-10T00:36:27.329 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-10T00:36:27.337 INFO:teuthology.task.internal:Creating test directory... 2023-09-10T00:36:27.337 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-10T00:36:27.343 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-10T00:36:27.346 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-10T00:36:27.383 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-10T00:36:27.390 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-10T00:36:27.418 INFO:teuthology.task.internal:Creating archive directory... 2023-09-10T00:36:27.419 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-10T00:36:27.428 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-10T00:36:27.430 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-10T00:36:27.493 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-10T00:36:27.502 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-10T00:36:27.502 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-10T00:36:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T00:36:27.523 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-10T00:36:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T00:36:27.542 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-10T00:36:27.550 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-10T00:36:27.565 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-10T00:36:27.603 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-10T00:36:27.625 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-10T00:36:27.626 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-10T00:36:27.641 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-10T00:36:27.800 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-10T00:36:27.834 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-10T00:36:27.838 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-10T00:36:27.846 INFO:teuthology.task.internal:Configuring sudo... 2023-09-10T00:36:27.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-10T00:36:27.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-10T00:36:27.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-10T00:36:27.918 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-10T00:36:27.929 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-10T00:36:27.929 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-10T00:36:27.934 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-10T00:36:27.936 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-10T00:36:27.973 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-10T00:36:28.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-10T00:36:28.171 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-10T00:36:28.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-10T00:36:28.314 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:36:28.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-10T00:36:28.386 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-10T00:36:28.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-10T00:36:28.493 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-10T00:36:28.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-10T00:36:28.628 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:36:28.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-10T00:36:28.701 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-10T00:36:28.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-10T00:36:28.801 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-10T00:36:28.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-10T00:36:28.931 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:36:28.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-10T00:36:29.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2023-09-10T00:36:29.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2023-09-10T00:36:29.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2023-09-10T00:36:29.077 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-10T00:36:29.106 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-10T00:36:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-10T00:36:29.477 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-10T00:36:29.486 INFO:teuthology.task.internal:Starting timer... 2023-09-10T00:36:29.487 INFO:teuthology.run_tasks:Running task pcp... 2023-09-10T00:36:29.522 INFO:teuthology.run_tasks:Running task selinux... 2023-09-10T00:36:29.572 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-10T00:36:29.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2023-09-10T00:36:29.698 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2023-09-10T00:36:29.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2023-09-10T00:36:29.763 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2023-09-10T00:36:29.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2023-09-10T00:36:29.824 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2023-09-10T00:36:29.826 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-10T00:36:29.827 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2023-09-10T00:36:29.875 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2023-09-10T00:36:29.876 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2023-09-10T00:36:29.902 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2023-09-10T00:36:29.903 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2023-09-10T00:36:29.928 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2023-09-10T00:36:29.928 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2023-09-10T00:36:29.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-10T00:36:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T00:36:29.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-10T00:36:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T00:36:30.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-10T00:36:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T00:36:30.055 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-10T00:36:30.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2023-09-10T00:36:30.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2023-09-10T00:36:30.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2023-09-10T00:36:30.189 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-10T00:36:30.199 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-10T00:36:30.199 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-10T00:36:30.216 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-10T00:36:30.217 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi092.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-10T00:48:58.176 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2023-09-10T00:48:58.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-09-10T00:48:58.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:48:58.294 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-09-10T00:48:58.362 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-09-10T00:48:58.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-09-10T00:48:58.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:48:58.431 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-09-10T00:48:58.502 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-09-10T00:48:58.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-09-10T00:48:58.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-10T00:48:58.572 DEBUG:teuthology.orchestra.run.smithi136:> true 2023-09-10T00:48:58.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-09-10T00:48:58.642 INFO:teuthology.run_tasks:Running task clock... 2023-09-10T00:48:58.653 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-10T00:48:58.654 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-10T00:48:58.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-10T00:48:58.659 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-10T00:48:58.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-10T00:48:58.661 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-10T00:48:58.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-10T00:48:58.697 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-10T00:48:58.714 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-10T00:48:58.715 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-10T00:48:58.730 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-10T00:48:58.731 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-10T00:48:58.766 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-10T00:48:58.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2023-09-10T00:48:58.806 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2023-09-10T00:48:58.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2023-09-10T00:48:58.824 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-10T00:48:58.824 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2023-09-10T00:48:58.841 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-10T00:48:58.842 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-10T00:48:58.853 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-10T00:48:58.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2023-09-10T00:48:58.892 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2023-09-10T00:48:58.919 INFO:teuthology.orchestra.run.smithi092.stderr:bash: ntpq: command not found 2023-09-10T00:48:58.924 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-10T00:48:58.924 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-09-10T00:48:58.924 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.924 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.924 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.925 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.927 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-10T00:48:58.934 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2023-09-10T00:48:58.938 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-10T00:48:58.938 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2023-09-10T00:48:58.939 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.939 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.939 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.939 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:58.963 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-10T00:48:59.073 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-09-10T00:48:59.080 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-10T00:48:59.081 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-09-10T00:48:59.081 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:59.081 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:59.081 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:59.081 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T00:48:59.083 INFO:teuthology.run_tasks:Running task pexec... 2023-09-10T00:48:59.093 INFO:teuthology.task.pexec:Executing custom commands... 2023-09-10T00:48:59.093 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T00:48:59.094 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-09-10T00:48:59.094 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi092.front.sepia.ceph.com 2023-09-10T00:48:59.095 DEBUG:teuthology.orchestra.run.smithi092:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-09-10T00:48:59.095 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi136.front.sepia.ceph.com 2023-09-10T00:48:59.095 DEBUG:teuthology.orchestra.run.smithi136:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-09-10T00:48:59.469 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-09-10T00:48:59.483 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-09-10T00:48:59.607 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:00.985 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:21 ago on Sun 10 Sep 2023 12:44:39 AM UTC. 2023-09-10T00:49:00.990 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:04:24 ago on Sun 10 Sep 2023 12:44:36 AM UTC. 2023-09-10T00:49:01.298 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:52 ago on Sun 10 Sep 2023 12:47:08 AM UTC. 2023-09-10T00:49:02.744 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-09-10T00:49:02.745 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-09-10T00:49:02.745 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-09-10T00:49:02.745 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-09-10T00:49:02.745 INFO:teuthology.orchestra.run.smithi092.stdout:Resetting modules: 2023-09-10T00:49:02.745 INFO:teuthology.orchestra.run.smithi092.stdout: container-tools 2023-09-10T00:49:02.745 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:49:02.746 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-09-10T00:49:02.746 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-09-10T00:49:02.746 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:49:02.866 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2023-09-10T00:49:02.867 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2023-09-10T00:49:02.867 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2023-09-10T00:49:02.867 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2023-09-10T00:49:02.867 INFO:teuthology.orchestra.run.smithi136.stdout:Resetting modules: 2023-09-10T00:49:02.867 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools 2023-09-10T00:49:02.868 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:02.868 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2023-09-10T00:49:02.868 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2023-09-10T00:49:02.868 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:03.346 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-09-10T00:49:03.347 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-09-10T00:49:03.347 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-09-10T00:49:03.348 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-09-10T00:49:03.348 INFO:teuthology.orchestra.run.smithi003.stdout:Resetting modules: 2023-09-10T00:49:03.348 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools 2023-09-10T00:49:03.348 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:49:03.348 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-09-10T00:49:03.349 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-09-10T00:49:03.349 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:49:03.747 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:04:27 ago on Sun 10 Sep 2023 12:44:36 AM UTC. 2023-09-10T00:49:03.831 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:24 ago on Sun 10 Sep 2023 12:44:39 AM UTC. 2023-09-10T00:49:04.635 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:56 ago on Sun 10 Sep 2023 12:47:08 AM UTC. 2023-09-10T00:49:08.988 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-09-10T00:49:09.465 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:09.874 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2023-09-10T00:49:10.394 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:10.476 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-09-10T00:49:10.661 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:04:34 ago on Sun 10 Sep 2023 12:44:36 AM UTC. 2023-09-10T00:49:11.130 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:11.894 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:32 ago on Sun 10 Sep 2023 12:44:39 AM UTC. 2023-09-10T00:49:12.361 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-09-10T00:49:12.457 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================================ 2023-09-10T00:49:12.457 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-09-10T00:49:12.458 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================================ 2023-09-10T00:49:12.458 INFO:teuthology.orchestra.run.smithi092.stdout:Installing group/module packages: 2023-09-10T00:49:12.458 INFO:teuthology.orchestra.run.smithi092.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-09-10T00:49:12.458 INFO:teuthology.orchestra.run.smithi092.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-09-10T00:49:12.459 INFO:teuthology.orchestra.run.smithi092.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-09-10T00:49:12.459 INFO:teuthology.orchestra.run.smithi092.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-09-10T00:49:12.459 INFO:teuthology.orchestra.run.smithi092.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-09-10T00:49:12.459 INFO:teuthology.orchestra.run.smithi092.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-09-10T00:49:12.459 INFO:teuthology.orchestra.run.smithi092.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-09-10T00:49:12.460 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-09-10T00:49:12.460 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-09-10T00:49:12.460 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-09-10T00:49:12.460 INFO:teuthology.orchestra.run.smithi092.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-09-10T00:49:12.460 INFO:teuthology.orchestra.run.smithi092.stdout:Installing module profiles: 2023-09-10T00:49:12.461 INFO:teuthology.orchestra.run.smithi092.stdout: container-tools/common 2023-09-10T00:49:12.461 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:49:12.461 INFO:teuthology.orchestra.run.smithi092.stdout:Enabling module streams: 2023-09-10T00:49:12.461 INFO:teuthology.orchestra.run.smithi092.stdout: container-tools rhel8 2023-09-10T00:49:12.461 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:49:12.462 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-09-10T00:49:12.462 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================================ 2023-09-10T00:49:12.462 INFO:teuthology.orchestra.run.smithi092.stdout:Install 10 Packages 2023-09-10T00:49:12.462 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:49:12.463 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 18 M 2023-09-10T00:49:12.463 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 65 M 2023-09-10T00:49:12.463 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-09-10T00:49:12.694 INFO:teuthology.orchestra.run.smithi092.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 402 kB/s | 94 kB 00:00 2023-09-10T00:49:12.719 INFO:teuthology.orchestra.run.smithi092.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 157 kB/s | 41 kB 00:00 2023-09-10T00:49:12.744 INFO:teuthology.orchestra.run.smithi092.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 87 kB/s | 25 kB 00:00 2023-09-10T00:49:12.820 INFO:teuthology.orchestra.run.smithi092.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 393 kB/s | 49 kB 00:00 2023-09-10T00:49:12.853 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:04 ago on Sun 10 Sep 2023 12:47:08 AM UTC. 2023-09-10T00:49:12.854 INFO:teuthology.orchestra.run.smithi092.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-09-10T00:49:12.953 INFO:teuthology.orchestra.run.smithi092.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.1 MB/s | 149 kB 00:00 2023-09-10T00:49:13.029 INFO:teuthology.orchestra.run.smithi092.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.2 MB/s | 213 kB 00:00 2023-09-10T00:49:13.192 INFO:teuthology.orchestra.run.smithi092.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 18 MB/s | 8.0 MB 00:00 2023-09-10T00:49:13.262 INFO:teuthology.orchestra.run.smithi092.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 9.4 MB/s | 2.2 MB 00:00 2023-09-10T00:49:13.571 INFO:teuthology.orchestra.run.smithi092.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 11 MB/s | 7.0 MB 00:00 2023-09-10T00:49:13.572 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-09-10T00:49:13.572 INFO:teuthology.orchestra.run.smithi092.stdout:Total 16 MB/s | 18 MB 00:01 2023-09-10T00:49:13.599 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2023-09-10T00:49:13.692 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================================ 2023-09-10T00:49:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2023-09-10T00:49:13.692 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================================ 2023-09-10T00:49:13.693 INFO:teuthology.orchestra.run.smithi136.stdout:Installing group/module packages: 2023-09-10T00:49:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-09-10T00:49:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-09-10T00:49:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-09-10T00:49:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-09-10T00:49:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-09-10T00:49:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-09-10T00:49:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-09-10T00:49:13.695 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2023-09-10T00:49:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-09-10T00:49:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-09-10T00:49:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-09-10T00:49:13.695 INFO:teuthology.orchestra.run.smithi136.stdout:Installing module profiles: 2023-09-10T00:49:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools/common 2023-09-10T00:49:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:13.696 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling module streams: 2023-09-10T00:49:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: container-tools rhel8 2023-09-10T00:49:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:13.697 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2023-09-10T00:49:13.697 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================================ 2023-09-10T00:49:13.697 INFO:teuthology.orchestra.run.smithi136.stdout:Install 10 Packages 2023-09-10T00:49:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:13.698 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 18 M 2023-09-10T00:49:13.698 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 65 M 2023-09-10T00:49:13.698 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2023-09-10T00:49:13.771 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-09-10T00:49:13.790 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-09-10T00:49:13.790 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-09-10T00:49:13.849 INFO:teuthology.orchestra.run.smithi136.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 161 kB/s | 25 kB 00:00 2023-09-10T00:49:13.874 INFO:teuthology.orchestra.run.smithi136.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 227 kB/s | 41 kB 00:00 2023-09-10T00:49:13.900 INFO:teuthology.orchestra.run.smithi136.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 458 kB/s | 94 kB 00:00 2023-09-10T00:49:13.971 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-09-10T00:49:13.974 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-09-10T00:49:14.008 INFO:teuthology.orchestra.run.smithi136.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 311 kB/s | 49 kB 00:00 2023-09-10T00:49:14.050 INFO:teuthology.orchestra.run.smithi136.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.9 MB/s | 524 kB 00:00 2023-09-10T00:49:14.200 INFO:teuthology.orchestra.run.smithi136.stdout:(6/10): buildah-1.26.2-1.module+el8.6.0+16771+2 26 MB/s | 8.0 MB 00:00 2023-09-10T00:49:14.226 INFO:teuthology.orchestra.run.smithi136.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 687 kB/s | 149 kB 00:00 2023-09-10T00:49:14.251 INFO:teuthology.orchestra.run.smithi136.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.0 MB/s | 213 kB 00:00 2023-09-10T00:49:14.385 INFO:teuthology.orchestra.run.smithi136.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 14 MB/s | 2.2 MB 00:00 2023-09-10T00:49:14.497 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-09-10T00:49:14.585 INFO:teuthology.orchestra.run.smithi136.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 18 MB/s | 7.0 MB 00:00 2023-09-10T00:49:14.586 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2023-09-10T00:49:14.586 INFO:teuthology.orchestra.run.smithi136.stdout:Total 20 MB/s | 18 MB 00:00 2023-09-10T00:49:14.671 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-09-10T00:49:14.785 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2023-09-10T00:49:14.803 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2023-09-10T00:49:14.804 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2023-09-10T00:49:14.813 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-09-10T00:49:14.828 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-09-10T00:49:14.919 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================================ 2023-09-10T00:49:14.919 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-09-10T00:49:14.919 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================================ 2023-09-10T00:49:14.919 INFO:teuthology.orchestra.run.smithi003.stdout:Installing group/module packages: 2023-09-10T00:49:14.920 INFO:teuthology.orchestra.run.smithi003.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-09-10T00:49:14.920 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-09-10T00:49:14.920 INFO:teuthology.orchestra.run.smithi003.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-09-10T00:49:14.920 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-09-10T00:49:14.920 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-09-10T00:49:14.920 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-09-10T00:49:14.921 INFO:teuthology.orchestra.run.smithi003.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-09-10T00:49:14.921 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-09-10T00:49:14.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-09-10T00:49:14.921 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-09-10T00:49:14.921 INFO:teuthology.orchestra.run.smithi003.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-09-10T00:49:14.922 INFO:teuthology.orchestra.run.smithi003.stdout:Installing module profiles: 2023-09-10T00:49:14.922 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools/common 2023-09-10T00:49:14.922 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:49:14.922 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling module streams: 2023-09-10T00:49:14.922 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools rhel8 2023-09-10T00:49:14.923 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:49:14.923 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-09-10T00:49:14.923 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================================ 2023-09-10T00:49:14.923 INFO:teuthology.orchestra.run.smithi003.stdout:Install 10 Packages 2023-09-10T00:49:14.923 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:49:14.924 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 18 M 2023-09-10T00:49:14.924 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 65 M 2023-09-10T00:49:14.924 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-09-10T00:49:14.986 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2023-09-10T00:49:14.990 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2023-09-10T00:49:15.030 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-09-10T00:49:15.092 INFO:teuthology.orchestra.run.smithi003.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 553 kB/s | 94 kB 00:00 2023-09-10T00:49:15.117 INFO:teuthology.orchestra.run.smithi003.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 209 kB/s | 41 kB 00:00 2023-09-10T00:49:15.159 INFO:teuthology.orchestra.run.smithi003.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 105 kB/s | 25 kB 00:00 2023-09-10T00:49:15.218 INFO:teuthology.orchestra.run.smithi003.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 394 kB/s | 49 kB 00:00 2023-09-10T00:49:15.242 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-09-10T00:49:15.260 INFO:teuthology.orchestra.run.smithi003.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.6 MB/s | 524 kB 00:00 2023-09-10T00:49:15.377 INFO:teuthology.orchestra.run.smithi003.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 941 kB/s | 149 kB 00:00 2023-09-10T00:49:15.410 INFO:teuthology.orchestra.run.smithi003.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.4 MB/s | 213 kB 00:00 2023-09-10T00:49:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2023-09-10T00:49:15.552 INFO:teuthology.orchestra.run.smithi003.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 20 MB/s | 8.0 MB 00:00 2023-09-10T00:49:15.582 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-09-10T00:49:15.645 INFO:teuthology.orchestra.run.smithi003.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 9.4 MB/s | 2.2 MB 00:00 2023-09-10T00:49:15.664 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-09-10T00:49:15.836 INFO:teuthology.orchestra.run.smithi003.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 15 MB/s | 7.0 MB 00:00 2023-09-10T00:49:15.837 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-09-10T00:49:15.837 INFO:teuthology.orchestra.run.smithi003.stdout:Total 20 MB/s | 18 MB 00:00 2023-09-10T00:49:15.862 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-09-10T00:49:16.040 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-09-10T00:49:16.179 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-09-10T00:49:16.203 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-09-10T00:49:16.204 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-09-10T00:49:16.276 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-09-10T00:49:16.361 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-09-10T00:49:16.648 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-09-10T00:49:16.734 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-09-10T00:49:16.738 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-09-10T00:49:17.250 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-09-10T00:49:17.344 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-09-10T00:49:17.460 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-09-10T00:49:17.469 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-09-10T00:49:17.540 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-09-10T00:49:17.683 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-09-10T00:49:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-09-10T00:49:17.879 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-09-10T00:49:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-09-10T00:49:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-09-10T00:49:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-09-10T00:49:18.136 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-09-10T00:49:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-09-10T00:49:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-09-10T00:49:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-09-10T00:49:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-09-10T00:49:18.137 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-09-10T00:49:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-09-10T00:49:18.138 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-09-10T00:49:18.220 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-09-10T00:49:18.337 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-09-10T00:49:18.579 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-09-10T00:49:18.580 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-09-10T00:49:18.752 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-09-10T00:49:18.962 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-09-10T00:49:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-09-10T00:49:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-09-10T00:49:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-09-10T00:49:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-09-10T00:49:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-09-10T00:49:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-09-10T00:49:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-09-10T00:49:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-09-10T00:49:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-09-10T00:49:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-09-10T00:49:19.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-09-10T00:49:19.258 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:04:42 ago on Sun 10 Sep 2023 12:44:36 AM UTC. 2023-09-10T00:49:19.419 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-09-10T00:49:20.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-09-10T00:49:20.414 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:40 ago on Sun 10 Sep 2023 12:44:39 AM UTC. 2023-09-10T00:49:20.415 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-09-10T00:49:20.693 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-09-10T00:49:20.881 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-09-10T00:49:21.105 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-09-10T00:49:21.507 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-09-10T00:49:21.507 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-09-10T00:49:21.508 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-09-10T00:49:21.508 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-09-10T00:49:21.508 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-09-10T00:49:21.508 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-09-10T00:49:21.508 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-09-10T00:49:21.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-09-10T00:49:21.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-09-10T00:49:21.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-09-10T00:49:22.362 INFO:teuthology.orchestra.run.smithi092.stdout:Installed products updated. 2023-09-10T00:49:22.621 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:49:22.622 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-09-10T00:49:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-09-10T00:49:22.622 INFO:teuthology.orchestra.run.smithi092.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-09-10T00:49:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-09-10T00:49:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-09-10T00:49:22.623 INFO:teuthology.orchestra.run.smithi092.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-09-10T00:49:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: yajl-2.1.0-10.el8.x86_64 2023-09-10T00:49:22.624 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:49:22.624 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-09-10T00:49:22.819 DEBUG:teuthology.parallel:result is None 2023-09-10T00:49:22.974 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-09-10T00:49:22.975 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:14 ago on Sun 10 Sep 2023 12:47:08 AM UTC. 2023-09-10T00:49:23.261 INFO:teuthology.orchestra.run.smithi136.stdout:Installed products updated. 2023-09-10T00:49:23.485 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:23.485 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2023-09-10T00:49:23.486 INFO:teuthology.orchestra.run.smithi136.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:23.486 INFO:teuthology.orchestra.run.smithi136.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-09-10T00:49:23.487 INFO:teuthology.orchestra.run.smithi136.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:23.487 INFO:teuthology.orchestra.run.smithi136.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-09-10T00:49:23.487 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-09-10T00:49:23.488 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-09-10T00:49:23.488 INFO:teuthology.orchestra.run.smithi136.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:23.488 INFO:teuthology.orchestra.run.smithi136.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:23.489 INFO:teuthology.orchestra.run.smithi136.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-09-10T00:49:23.489 INFO:teuthology.orchestra.run.smithi136.stdout: yajl-2.1.0-10.el8.x86_64 2023-09-10T00:49:23.490 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:23.490 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2023-09-10T00:49:23.681 DEBUG:teuthology.parallel:result is None 2023-09-10T00:49:27.064 INFO:teuthology.orchestra.run.smithi003.stdout:Installed products updated. 2023-09-10T00:49:27.291 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:49:27.292 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-09-10T00:49:27.292 INFO:teuthology.orchestra.run.smithi003.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:27.292 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-09-10T00:49:27.292 INFO:teuthology.orchestra.run.smithi003.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:27.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-09-10T00:49:27.293 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-09-10T00:49:27.293 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-09-10T00:49:27.293 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:27.293 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-09-10T00:49:27.293 INFO:teuthology.orchestra.run.smithi003.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-09-10T00:49:27.294 INFO:teuthology.orchestra.run.smithi003.stdout: yajl-2.1.0-10.el8.x86_64 2023-09-10T00:49:27.294 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:49:27.294 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-09-10T00:49:27.515 DEBUG:teuthology.parallel:result is None 2023-09-10T00:49:27.515 INFO:teuthology.run_tasks:Running task install... 2023-09-10T00:49:27.524 DEBUG:teuthology.task.install:project ceph 2023-09-10T00:49:27.525 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9670cca5bc8e17916865dc84939edfc10b1f2c00'}} 2023-09-10T00:49:27.525 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9670cca5bc8e17916865dc84939edfc10b1f2c00'} 2023-09-10T00:49:27.525 INFO:teuthology.task.install:Using flavor: default 2023-09-10T00:49:27.531 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-10T00:49:27.532 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-09-10T00:49:27.532 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9670cca5bc8e17916865dc84939edfc10b1f2c00', 'tag': None, 'wait_for_package': False} 2023-09-10T00:49:27.533 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:49:27.535 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9670cca5bc8e17916865dc84939edfc10b1f2c00', 'tag': None, 'wait_for_package': False} 2023-09-10T00:49:27.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:49:27.536 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9670cca5bc8e17916865dc84939edfc10b1f2c00', 'tag': None, 'wait_for_package': False} 2023-09-10T00:49:27.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:49:27.678 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/ 2023-09-10T00:49:27.679 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6025.g9670cca5 2023-09-10T00:49:27.684 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/ 2023-09-10T00:49:27.684 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6025.g9670cca5 2023-09-10T00:49:27.747 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/ 2023-09-10T00:49:27.748 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6025.g9670cca5 2023-09-10T00:49:27.850 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-10T00:49:27.850 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:49:27.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-10T00:49:27.853 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-10T00:49:27.853 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:49:27.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-10T00:49:27.856 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-10T00:49:27.856 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:49:27.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-10T00:49:27.885 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-09-10T00:49:27.885 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9670cca5bc8e17916865dc84939edfc10b1f2c00/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-10T00:49:27.887 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-09-10T00:49:27.887 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9670cca5bc8e17916865dc84939edfc10b1f2c00/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-10T00:49:27.924 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-09-10T00:49:27.925 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9670cca5bc8e17916865dc84939edfc10b1f2c00/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-10T00:49:27.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-10T00:49:27.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-10T00:49:28.032 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-10T00:49:28.033 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-10T00:49:28.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-10T00:49:28.097 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2023-09-10T00:49:28.098 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2023-09-10T00:49:28.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2023-09-10T00:49:28.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2023-09-10T00:49:28.131 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-10T00:49:28.200 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2023-09-10T00:49:28.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2023-09-10T00:49:28.505 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:28.533 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:28.684 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:28.865 INFO:teuthology.orchestra.run.smithi092.stdout:44 files removed 2023-09-10T00:49:28.879 INFO:teuthology.orchestra.run.smithi136.stdout:44 files removed 2023-09-10T00:49:28.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-09-10T00:49:28.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-09-10T00:49:29.078 INFO:teuthology.orchestra.run.smithi003.stdout:44 files removed 2023-09-10T00:49:29.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-09-10T00:49:29.260 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:29.288 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:29.613 INFO:teuthology.orchestra.run.smithi003.stdout:Updating Subscription Management repositories. 2023-09-10T00:49:30.530 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 215 kB/s | 81 kB 00:00 2023-09-10T00:49:30.697 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 248 kB/s | 81 kB 00:00 2023-09-10T00:49:30.698 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 155 kB/s | 81 kB 00:00 2023-09-10T00:49:30.839 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 34 kB/s | 9.4 kB 00:00 2023-09-10T00:49:30.973 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 39 kB/s | 9.4 kB 00:00 2023-09-10T00:49:31.016 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 33 kB/s | 9.4 kB 00:00 2023-09-10T00:49:31.224 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2023-09-10T00:49:31.273 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 5.4 kB/s | 1.5 kB 00:00 2023-09-10T00:49:31.326 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 5.2 kB/s | 1.5 kB 00:00 2023-09-10T00:49:31.358 INFO:teuthology.orchestra.run.smithi136.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-09-10T00:49:31.416 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-09-10T00:49:31.469 INFO:teuthology.orchestra.run.smithi092.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-09-10T00:49:32.367 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-09-10T00:49:33.409 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 16 MB 00:02 2023-09-10T00:49:36.806 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 265 kB/s | 24 kB 00:00 2023-09-10T00:49:37.050 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 383 kB/s | 24 kB 00:00 2023-09-10T00:49:38.160 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 47 MB 00:01 2023-09-10T00:49:38.277 INFO:teuthology.orchestra.run.smithi003.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 MB/s | 47 MB 00:01 2023-09-10T00:49:46.185 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-09-10T00:49:47.644 INFO:teuthology.orchestra.run.smithi003.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 MB/s | 53 MB 00:01 2023-09-10T00:49:49.463 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 901 kB/s | 16 MB 00:17 2023-09-10T00:49:52.946 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 438 kB/s | 24 kB 00:00 2023-09-10T00:49:53.847 INFO:teuthology.orchestra.run.smithi092.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 53 MB 00:00 2023-09-10T00:49:57.274 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-09-10T00:49:57.275 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-09-10T00:49:57.290 INFO:teuthology.orchestra.run.smithi136.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-09-10T00:49:57.291 INFO:teuthology.orchestra.run.smithi136.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-09-10T00:49:57.292 INFO:teuthology.orchestra.run.smithi136.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-09-10T00:49:57.293 INFO:teuthology.orchestra.run.smithi136.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-09-10T00:49:57.295 INFO:teuthology.orchestra.run.smithi136.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-09-10T00:49:57.298 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-09-10T00:49:57.299 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-09-10T00:49:57.432 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2023-09-10T00:49:57.444 INFO:teuthology.orchestra.run.smithi136.stdout:=========================================================================================================================== 2023-09-10T00:49:57.445 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2023-09-10T00:49:57.445 INFO:teuthology.orchestra.run.smithi136.stdout:=========================================================================================================================== 2023-09-10T00:49:57.445 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2023-09-10T00:49:57.445 INFO:teuthology.orchestra.run.smithi136.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-09-10T00:49:57.446 INFO:teuthology.orchestra.run.smithi136.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-09-10T00:49:57.446 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 6.4 k 2023-09-10T00:49:57.446 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 5.4 M 2023-09-10T00:49:57.446 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 907 k 2023-09-10T00:49:57.447 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 167 k 2023-09-10T00:49:57.447 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 1.6 M 2023-09-10T00:49:57.447 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 132 k 2023-09-10T00:49:57.447 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 1.7 M 2023-09-10T00:49:57.448 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 7.4 M 2023-09-10T00:49:57.448 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 47 k 2023-09-10T00:49:57.448 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 14 M 2023-09-10T00:49:57.448 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 49 M 2023-09-10T00:49:57.448 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 261 k 2023-09-10T00:49:57.449 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 176 k 2023-09-10T00:49:57.449 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 260 k 2023-09-10T00:49:57.449 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 18 k 2023-09-10T00:49:57.449 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-09-10T00:49:57.449 INFO:teuthology.orchestra.run.smithi136.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-09-10T00:49:57.450 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-09-10T00:49:57.450 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 31 k 2023-09-10T00:49:57.450 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 827 k 2023-09-10T00:49:57.450 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 139 k 2023-09-10T00:49:57.451 INFO:teuthology.orchestra.run.smithi136.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-09-10T00:49:57.451 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-09-10T00:49:57.451 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-09-10T00:49:57.451 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-09-10T00:49:57.451 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-09-10T00:49:57.452 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-09-10T00:49:57.452 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 197 k 2023-09-10T00:49:57.452 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 372 k 2023-09-10T00:49:57.452 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 364 k 2023-09-10T00:49:57.452 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 107 k 2023-09-10T00:49:57.453 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 94 k 2023-09-10T00:49:57.453 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.6 M 2023-09-10T00:49:57.453 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 179 k 2023-09-10T00:49:57.453 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-09-10T00:49:57.453 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2023-09-10T00:49:57.454 INFO:teuthology.orchestra.run.smithi136.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-09-10T00:49:57.454 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.7 M 2023-09-10T00:49:57.454 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.7 M 2023-09-10T00:49:57.454 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2023-09-10T00:49:57.455 INFO:teuthology.orchestra.run.smithi136.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-09-10T00:49:57.455 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 22 M 2023-09-10T00:49:57.455 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 23 k 2023-09-10T00:49:57.455 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 2.3 M 2023-09-10T00:49:57.455 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 242 k 2023-09-10T00:49:57.456 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 4.6 M 2023-09-10T00:49:57.456 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 18 M 2023-09-10T00:49:57.456 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 15 k 2023-09-10T00:49:57.456 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 24 k 2023-09-10T00:49:57.457 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-10T00:49:57.457 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-10T00:49:57.457 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-09-10T00:49:57.457 INFO:teuthology.orchestra.run.smithi136.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-09-10T00:49:57.458 INFO:teuthology.orchestra.run.smithi136.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-09-10T00:49:57.458 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 184 k 2023-09-10T00:49:57.458 INFO:teuthology.orchestra.run.smithi136.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-09-10T00:49:57.458 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-09-10T00:49:57.458 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-10T00:49:57.459 INFO:teuthology.orchestra.run.smithi136.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-09-10T00:49:57.459 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-09-10T00:49:57.459 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-10T00:49:57.459 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 519 k 2023-09-10T00:49:57.460 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-09-10T00:49:57.460 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 6.8 M 2023-09-10T00:49:57.460 INFO:teuthology.orchestra.run.smithi136.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-09-10T00:49:57.460 INFO:teuthology.orchestra.run.smithi136.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-09-10T00:49:57.460 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-10T00:49:57.461 INFO:teuthology.orchestra.run.smithi136.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-09-10T00:49:57.461 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-10T00:49:57.461 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-10T00:49:57.461 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-10T00:49:57.461 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-09-10T00:49:57.462 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-09-10T00:49:57.462 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-09-10T00:49:57.462 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-09-10T00:49:57.462 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-10T00:49:57.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-10T00:49:57.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-10T00:49:57.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-10T00:49:57.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-10T00:49:57.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 45 k 2023-09-10T00:49:57.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 126 k 2023-09-10T00:49:57.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-10T00:49:57.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-09-10T00:49:57.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-10T00:49:57.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-10T00:49:57.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-09-10T00:49:57.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-10T00:49:57.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-10T00:49:57.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-10T00:49:57.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-10T00:49:57.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-10T00:49:57.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-10T00:49:57.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-10T00:49:57.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-10T00:49:57.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-10T00:49:57.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-10T00:49:57.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-10T00:49:57.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-10T00:49:57.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-10T00:49:57.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-10T00:49:57.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-10T00:49:57.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-10T00:49:57.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-10T00:49:57.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-10T00:49:57.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-10T00:49:57.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-09-10T00:49:57.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-10T00:49:57.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-10T00:49:57.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-09-10T00:49:57.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-10T00:49:57.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-10T00:49:57.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-10T00:49:57.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-10T00:49:57.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-10T00:49:57.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-09-10T00:49:57.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-10T00:49:57.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-10T00:49:57.472 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-10T00:49:57.472 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-10T00:49:57.472 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-10T00:49:57.472 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-10T00:49:57.472 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-10T00:49:57.473 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-10T00:49:57.473 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-10T00:49:57.473 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-10T00:49:57.473 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-09-10T00:49:57.474 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-10T00:49:57.474 INFO:teuthology.orchestra.run.smithi136.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-09-10T00:49:57.474 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-10T00:49:57.474 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-09-10T00:49:57.475 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling module streams: 2023-09-10T00:49:57.475 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql 10 2023-09-10T00:49:57.475 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:57.475 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2023-09-10T00:49:57.475 INFO:teuthology.orchestra.run.smithi136.stdout:=========================================================================================================================== 2023-09-10T00:49:57.476 INFO:teuthology.orchestra.run.smithi136.stdout:Install 128 Packages 2023-09-10T00:49:57.476 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 3 Packages 2023-09-10T00:49:57.476 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:49:57.477 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 208 M 2023-09-10T00:49:57.477 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2023-09-10T00:49:57.735 INFO:teuthology.orchestra.run.smithi136.stdout:(1/131): ceph-18.0.0-6025.g9670cca5.el8.x86_64. 53 kB/s | 6.4 kB 00:00 2023-09-10T00:49:57.961 INFO:teuthology.orchestra.run.smithi136.stdout:(2/131): ceph-fuse-18.0.0-6025.g9670cca5.el8.x8 3.9 MB/s | 907 kB 00:00 2023-09-10T00:49:58.019 INFO:teuthology.orchestra.run.smithi136.stdout:(3/131): ceph-immutable-object-cache-18.0.0-602 2.8 MB/s | 167 kB 00:00 2023-09-10T00:49:58.382 INFO:teuthology.orchestra.run.smithi136.stdout:(4/131): ceph-base-18.0.0-6025.g9670cca5.el8.x8 7.0 MB/s | 5.4 MB 00:00 2023-09-10T00:49:58.437 INFO:teuthology.orchestra.run.smithi136.stdout:(5/131): ceph-mds-18.0.0-6025.g9670cca5.el8.x86 5.6 MB/s | 2.3 MB 00:00 2023-09-10T00:49:58.571 INFO:teuthology.orchestra.run.smithi136.stdout:(6/131): ceph-mgr-18.0.0-6025.g9670cca5.el8.x86 8.6 MB/s | 1.6 MB 00:00 2023-09-10T00:49:59.229 INFO:teuthology.orchestra.run.smithi136.stdout:(7/131): ceph-mon-18.0.0-6025.g9670cca5.el8.x86 5.8 MB/s | 4.6 MB 00:00 2023-09-10T00:49:59.754 INFO:teuthology.orchestra.run.smithi136.stdout:(8/131): ceph-common-18.0.0-6025.g9670cca5.el8. 10 MB/s | 22 MB 00:02 2023-09-10T00:49:59.813 INFO:teuthology.orchestra.run.smithi136.stdout:(9/131): ceph-selinux-18.0.0-6025.g9670cca5.el8 413 kB/s | 24 kB 00:00 2023-09-10T00:50:00.293 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-09-10T00:50:00.294 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-09-10T00:50:00.310 INFO:teuthology.orchestra.run.smithi003.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-09-10T00:50:00.311 INFO:teuthology.orchestra.run.smithi003.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-09-10T00:50:00.312 INFO:teuthology.orchestra.run.smithi003.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-09-10T00:50:00.313 INFO:teuthology.orchestra.run.smithi003.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-09-10T00:50:00.317 INFO:teuthology.orchestra.run.smithi003.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-09-10T00:50:00.319 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-09-10T00:50:00.320 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-09-10T00:50:00.338 INFO:teuthology.orchestra.run.smithi136.stdout:(10/131): ceph-osd-18.0.0-6025.g9670cca5.el8.x8 10 MB/s | 18 MB 00:01 2023-09-10T00:50:00.413 INFO:teuthology.orchestra.run.smithi136.stdout:(11/131): cephfs-mirror-18.0.0-6025.g9670cca5.e 3.4 MB/s | 260 kB 00:00 2023-09-10T00:50:00.475 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-09-10T00:50:00.480 INFO:teuthology.orchestra.run.smithi136.stdout:(12/131): libcephfs-devel-18.0.0-6025.g9670cca5 466 kB/s | 31 kB 00:00 2023-09-10T00:50:00.488 INFO:teuthology.orchestra.run.smithi003.stdout:=========================================================================================================================== 2023-09-10T00:50:00.489 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-09-10T00:50:00.489 INFO:teuthology.orchestra.run.smithi003.stdout:=========================================================================================================================== 2023-09-10T00:50:00.489 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-09-10T00:50:00.489 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-09-10T00:50:00.489 INFO:teuthology.orchestra.run.smithi003.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-09-10T00:50:00.490 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 6.4 k 2023-09-10T00:50:00.490 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 5.4 M 2023-09-10T00:50:00.490 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 907 k 2023-09-10T00:50:00.490 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 167 k 2023-09-10T00:50:00.490 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 1.6 M 2023-09-10T00:50:00.491 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 132 k 2023-09-10T00:50:00.491 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 1.7 M 2023-09-10T00:50:00.491 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 7.4 M 2023-09-10T00:50:00.491 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 47 k 2023-09-10T00:50:00.491 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 14 M 2023-09-10T00:50:00.492 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 49 M 2023-09-10T00:50:00.492 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 261 k 2023-09-10T00:50:00.492 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 176 k 2023-09-10T00:50:00.492 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 260 k 2023-09-10T00:50:00.492 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 18 k 2023-09-10T00:50:00.493 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-09-10T00:50:00.493 INFO:teuthology.orchestra.run.smithi003.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-09-10T00:50:00.493 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-09-10T00:50:00.493 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 31 k 2023-09-10T00:50:00.493 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 827 k 2023-09-10T00:50:00.494 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 139 k 2023-09-10T00:50:00.494 INFO:teuthology.orchestra.run.smithi003.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-09-10T00:50:00.494 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-09-10T00:50:00.494 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-09-10T00:50:00.494 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-09-10T00:50:00.495 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-09-10T00:50:00.495 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-09-10T00:50:00.495 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 197 k 2023-09-10T00:50:00.495 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 372 k 2023-09-10T00:50:00.495 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 364 k 2023-09-10T00:50:00.496 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 107 k 2023-09-10T00:50:00.496 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 94 k 2023-09-10T00:50:00.496 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.6 M 2023-09-10T00:50:00.496 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 179 k 2023-09-10T00:50:00.496 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-09-10T00:50:00.496 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2023-09-10T00:50:00.497 INFO:teuthology.orchestra.run.smithi003.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-09-10T00:50:00.497 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.7 M 2023-09-10T00:50:00.497 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.7 M 2023-09-10T00:50:00.497 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-09-10T00:50:00.497 INFO:teuthology.orchestra.run.smithi003.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-09-10T00:50:00.498 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 22 M 2023-09-10T00:50:00.498 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 23 k 2023-09-10T00:50:00.498 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 2.3 M 2023-09-10T00:50:00.498 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 242 k 2023-09-10T00:50:00.499 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 4.6 M 2023-09-10T00:50:00.499 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 18 M 2023-09-10T00:50:00.499 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 15 k 2023-09-10T00:50:00.499 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 24 k 2023-09-10T00:50:00.499 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-10T00:50:00.499 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-10T00:50:00.500 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-09-10T00:50:00.500 INFO:teuthology.orchestra.run.smithi003.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-09-10T00:50:00.500 INFO:teuthology.orchestra.run.smithi003.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-09-10T00:50:00.500 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 184 k 2023-09-10T00:50:00.500 INFO:teuthology.orchestra.run.smithi003.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-09-10T00:50:00.501 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-09-10T00:50:00.501 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-10T00:50:00.501 INFO:teuthology.orchestra.run.smithi003.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-09-10T00:50:00.501 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-09-10T00:50:00.501 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-10T00:50:00.501 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 519 k 2023-09-10T00:50:00.502 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-09-10T00:50:00.502 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 6.8 M 2023-09-10T00:50:00.502 INFO:teuthology.orchestra.run.smithi003.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-09-10T00:50:00.502 INFO:teuthology.orchestra.run.smithi003.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-09-10T00:50:00.502 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-10T00:50:00.503 INFO:teuthology.orchestra.run.smithi003.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-09-10T00:50:00.503 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-10T00:50:00.503 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-10T00:50:00.503 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-10T00:50:00.503 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-09-10T00:50:00.504 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-09-10T00:50:00.504 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-09-10T00:50:00.504 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-09-10T00:50:00.504 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-10T00:50:00.504 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-10T00:50:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-10T00:50:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-10T00:50:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-10T00:50:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 45 k 2023-09-10T00:50:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 126 k 2023-09-10T00:50:00.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-10T00:50:00.506 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-09-10T00:50:00.506 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-10T00:50:00.506 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-10T00:50:00.506 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-09-10T00:50:00.506 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-10T00:50:00.507 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-10T00:50:00.507 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-10T00:50:00.507 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-10T00:50:00.507 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-10T00:50:00.507 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-10T00:50:00.508 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-10T00:50:00.508 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-10T00:50:00.508 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-10T00:50:00.508 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-10T00:50:00.508 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-10T00:50:00.509 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-10T00:50:00.509 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-10T00:50:00.509 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-10T00:50:00.509 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-10T00:50:00.509 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-10T00:50:00.510 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-10T00:50:00.510 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-10T00:50:00.510 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-10T00:50:00.510 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-09-10T00:50:00.510 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-10T00:50:00.510 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-10T00:50:00.511 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-09-10T00:50:00.511 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-10T00:50:00.511 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-10T00:50:00.511 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-10T00:50:00.511 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-10T00:50:00.512 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-10T00:50:00.512 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-09-10T00:50:00.512 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-10T00:50:00.512 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-10T00:50:00.512 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-10T00:50:00.513 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-10T00:50:00.513 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-10T00:50:00.513 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-10T00:50:00.513 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-10T00:50:00.513 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-10T00:50:00.514 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-10T00:50:00.514 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-10T00:50:00.514 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-09-10T00:50:00.514 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-10T00:50:00.514 INFO:teuthology.orchestra.run.smithi003.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-09-10T00:50:00.515 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-10T00:50:00.515 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-09-10T00:50:00.515 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling module streams: 2023-09-10T00:50:00.515 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql 10 2023-09-10T00:50:00.515 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:50:00.515 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-09-10T00:50:00.516 INFO:teuthology.orchestra.run.smithi003.stdout:=========================================================================================================================== 2023-09-10T00:50:00.516 INFO:teuthology.orchestra.run.smithi003.stdout:Install 128 Packages 2023-09-10T00:50:00.516 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 3 Packages 2023-09-10T00:50:00.516 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:50:00.517 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 208 M 2023-09-10T00:50:00.517 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-09-10T00:50:00.614 INFO:teuthology.orchestra.run.smithi136.stdout:(13/131): libcephfs2-18.0.0-6025.g9670cca5.el8. 6.1 MB/s | 827 kB 00:00 2023-09-10T00:50:00.689 INFO:teuthology.orchestra.run.smithi136.stdout:(14/131): libcephsqlite-18.0.0-6025.g9670cca5.e 2.4 MB/s | 184 kB 00:00 2023-09-10T00:50:00.756 INFO:teuthology.orchestra.run.smithi136.stdout:(15/131): librados-devel-18.0.0-6025.g9670cca5. 2.0 MB/s | 139 kB 00:00 2023-09-10T00:50:00.848 INFO:teuthology.orchestra.run.smithi136.stdout:(16/131): libradosstriper1-18.0.0-6025.g9670cca 5.5 MB/s | 519 kB 00:00 2023-09-10T00:50:01.178 INFO:teuthology.orchestra.run.smithi003.stdout:(1/131): ceph-18.0.0-6025.g9670cca5.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2023-09-10T00:50:01.632 INFO:teuthology.orchestra.run.smithi136.stdout:(17/131): librgw2-18.0.0-6025.g9670cca5.el8.x86 8.7 MB/s | 6.8 MB 00:00 2023-09-10T00:50:01.723 INFO:teuthology.orchestra.run.smithi136.stdout:(18/131): python3-ceph-argparse-18.0.0-6025.g96 488 kB/s | 45 kB 00:00 2023-09-10T00:50:01.807 INFO:teuthology.orchestra.run.smithi136.stdout:(19/131): python3-ceph-common-18.0.0-6025.g9670 1.5 MB/s | 126 kB 00:00 2023-09-10T00:50:01.818 INFO:teuthology.orchestra.run.smithi003.stdout:(2/131): ceph-fuse-18.0.0-6025.g9670cca5.el8.x8 1.4 MB/s | 907 kB 00:00 2023-09-10T00:50:01.882 INFO:teuthology.orchestra.run.smithi136.stdout:(20/131): python3-cephfs-18.0.0-6025.g9670cca5. 2.6 MB/s | 197 kB 00:00 2023-09-10T00:50:01.954 INFO:teuthology.orchestra.run.smithi003.stdout:(3/131): ceph-immutable-object-cache-18.0.0-602 1.2 MB/s | 167 kB 00:00 2023-09-10T00:50:01.974 INFO:teuthology.orchestra.run.smithi136.stdout:(21/131): python3-rados-18.0.0-6025.g9670cca5.e 4.0 MB/s | 372 kB 00:00 2023-09-10T00:50:02.031 INFO:teuthology.orchestra.run.smithi092.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 47 MB 00:00 2023-09-10T00:50:02.083 INFO:teuthology.orchestra.run.smithi136.stdout:(22/131): python3-rbd-18.0.0-6025.g9670cca5.el8 3.3 MB/s | 364 kB 00:00 2023-09-10T00:50:02.166 INFO:teuthology.orchestra.run.smithi136.stdout:(23/131): python3-rgw-18.0.0-6025.g9670cca5.el8 1.2 MB/s | 107 kB 00:00 2023-09-10T00:50:02.259 INFO:teuthology.orchestra.run.smithi136.stdout:(24/131): rbd-fuse-18.0.0-6025.g9670cca5.el8.x8 1.0 MB/s | 94 kB 00:00 2023-09-10T00:50:02.946 INFO:teuthology.orchestra.run.smithi003.stdout:(4/131): ceph-mds-18.0.0-6025.g9670cca5.el8.x86 2.4 MB/s | 2.3 MB 00:00 2023-09-10T00:50:02.950 INFO:teuthology.orchestra.run.smithi136.stdout:(25/131): ceph-radosgw-18.0.0-6025.g9670cca5.el 3.7 MB/s | 14 MB 00:03 2023-09-10T00:50:03.009 INFO:teuthology.orchestra.run.smithi136.stdout:(26/131): rbd-mirror-18.0.0-6025.g9670cca5.el8. 4.8 MB/s | 3.6 MB 00:00 2023-09-10T00:50:03.034 INFO:teuthology.orchestra.run.smithi136.stdout:(27/131): rbd-nbd-18.0.0-6025.g9670cca5.el8.x86 2.1 MB/s | 179 kB 00:00 2023-09-10T00:50:03.068 INFO:teuthology.orchestra.run.smithi136.stdout:(28/131): ceph-grafana-dashboards-18.0.0-6025.g 395 kB/s | 23 kB 00:00 2023-09-10T00:50:03.148 INFO:teuthology.orchestra.run.smithi136.stdout:(29/131): ceph-mgr-cephadm-18.0.0-6025.g9670cca 2.2 MB/s | 132 kB 00:00 2023-09-10T00:50:03.413 INFO:teuthology.orchestra.run.smithi003.stdout:(5/131): ceph-mgr-18.0.0-6025.g9670cca5.el8.x86 3.5 MB/s | 1.6 MB 00:00 2023-09-10T00:50:03.727 INFO:teuthology.orchestra.run.smithi136.stdout:(30/131): ceph-test-18.0.0-6025.g9670cca5.el8.x 12 MB/s | 49 MB 00:03 2023-09-10T00:50:03.769 INFO:teuthology.orchestra.run.smithi136.stdout:(31/131): ceph-mgr-dashboard-18.0.0-6025.g9670c 2.5 MB/s | 1.7 MB 00:00 2023-09-10T00:50:03.794 INFO:teuthology.orchestra.run.smithi136.stdout:(32/131): ceph-mgr-modules-core-18.0.0-6025.g96 3.6 MB/s | 242 kB 00:00 2023-09-10T00:50:03.820 INFO:teuthology.orchestra.run.smithi136.stdout:(33/131): ceph-mgr-rook-18.0.0-6025.g9670cca5.e 949 kB/s | 47 kB 00:00 2023-09-10T00:50:03.886 INFO:teuthology.orchestra.run.smithi136.stdout:(34/131): ceph-prometheus-alerts-18.0.0-6025.g9 158 kB/s | 15 kB 00:00 2023-09-10T00:50:03.928 INFO:teuthology.orchestra.run.smithi136.stdout:(35/131): ceph-volume-18.0.0-6025.g9670cca5.el8 2.4 MB/s | 261 kB 00:00 2023-09-10T00:50:03.953 INFO:teuthology.orchestra.run.smithi136.stdout:(36/131): cephadm-18.0.0-6025.g9670cca5.el8.noa 2.6 MB/s | 176 kB 00:00 2023-09-10T00:50:03.955 INFO:teuthology.orchestra.run.smithi003.stdout:(6/131): ceph-base-18.0.0-6025.g9670cca5.el8.x8 1.8 MB/s | 5.4 MB 00:03 2023-09-10T00:50:03.979 INFO:teuthology.orchestra.run.smithi136.stdout:(37/131): cephfs-top-18.0.0-6025.g9670cca5.el8. 357 kB/s | 18 kB 00:00 2023-09-10T00:50:04.071 INFO:teuthology.orchestra.run.smithi136.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 643 kB/s | 59 kB 00:00 2023-09-10T00:50:04.114 INFO:teuthology.orchestra.run.smithi136.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2023-09-10T00:50:04.163 INFO:teuthology.orchestra.run.smithi136.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 818 kB/s | 75 kB 00:00 2023-09-10T00:50:04.213 INFO:teuthology.orchestra.run.smithi136.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.6 MB/s | 476 kB 00:00 2023-09-10T00:50:04.239 INFO:teuthology.orchestra.run.smithi003.stdout:(7/131): ceph-mon-18.0.0-6025.g9670cca5.el8.x86 5.6 MB/s | 4.6 MB 00:00 2023-09-10T00:50:04.240 INFO:teuthology.orchestra.run.smithi136.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 579 kB/s | 44 kB 00:00 2023-09-10T00:50:04.364 INFO:teuthology.orchestra.run.smithi136.stdout:(43/131): ceph-mgr-diskprediction-local-18.0.0- 5.8 MB/s | 7.4 MB 00:01 2023-09-10T00:50:04.389 INFO:teuthology.orchestra.run.smithi136.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 219 kB/s | 33 kB 00:00 2023-09-10T00:50:04.415 INFO:teuthology.orchestra.run.smithi136.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 922 kB/s | 185 kB 00:00 2023-09-10T00:50:04.473 INFO:teuthology.orchestra.run.smithi136.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 6.5 MB/s | 384 kB 00:00 2023-09-10T00:50:04.499 INFO:teuthology.orchestra.run.smithi136.stdout:(47/131): python3-certifi-2018.10.15-7.el8.noar 122 kB/s | 16 kB 00:00 2023-09-10T00:50:04.524 INFO:teuthology.orchestra.run.smithi136.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-09-10T00:50:04.549 INFO:teuthology.orchestra.run.smithi136.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-09-10T00:50:04.574 INFO:teuthology.orchestra.run.smithi136.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2023-09-10T00:50:04.591 INFO:teuthology.orchestra.run.smithi136.stdout:(51/131): python3-google-auth-1.1.1-10.el8.noar 690 kB/s | 81 kB 00:00 2023-09-10T00:50:04.616 INFO:teuthology.orchestra.run.smithi136.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 1.1 MB/s | 49 kB 00:00 2023-09-10T00:50:04.642 INFO:teuthology.orchestra.run.smithi136.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-09-10T00:50:04.667 INFO:teuthology.orchestra.run.smithi136.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-09-10T00:50:04.692 INFO:teuthology.orchestra.run.smithi136.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-09-10T00:50:04.718 INFO:teuthology.orchestra.run.smithi136.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-09-10T00:50:04.771 INFO:teuthology.orchestra.run.smithi136.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 433 kB/s | 34 kB 00:00 2023-09-10T00:50:04.810 INFO:teuthology.orchestra.run.smithi136.stdout:(58/131): python3-kubernetes-11.0.0-6.el8.noarc 4.5 MB/s | 1.2 MB 00:00 2023-09-10T00:50:04.836 INFO:teuthology.orchestra.run.smithi136.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.6 MB/s | 196 kB 00:00 2023-09-10T00:50:04.861 INFO:teuthology.orchestra.run.smithi136.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 714 kB/s | 64 kB 00:00 2023-09-10T00:50:04.877 INFO:teuthology.orchestra.run.smithi136.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 278 kB/s | 19 kB 00:00 2023-09-10T00:50:04.903 INFO:teuthology.orchestra.run.smithi136.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 357 kB/s | 24 kB 00:00 2023-09-10T00:50:04.905 INFO:teuthology.orchestra.run.smithi003.stdout:(8/131): ceph-common-18.0.0-6025.g9670cca5.el8. 5.6 MB/s | 22 MB 00:04 2023-09-10T00:50:04.920 INFO:teuthology.orchestra.run.smithi136.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 701 kB/s | 29 kB 00:00 2023-09-10T00:50:04.937 INFO:teuthology.orchestra.run.smithi136.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 471 kB/s | 27 kB 00:00 2023-09-10T00:50:04.979 INFO:teuthology.orchestra.run.smithi136.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2023-09-10T00:50:04.984 INFO:teuthology.orchestra.run.smithi003.stdout:(9/131): ceph-selinux-18.0.0-6025.g9670cca5.el8 309 kB/s | 24 kB 00:00 2023-09-10T00:50:05.004 INFO:teuthology.orchestra.run.smithi136.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 2.5 MB/s | 255 kB 00:00 2023-09-10T00:50:05.029 INFO:teuthology.orchestra.run.smithi136.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-09-10T00:50:05.063 INFO:teuthology.orchestra.run.smithi136.stdout:(68/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2023-09-10T00:50:05.080 INFO:teuthology.orchestra.run.smithi136.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 304 kB/s | 23 kB 00:00 2023-09-10T00:50:05.105 INFO:teuthology.orchestra.run.smithi136.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2023-09-10T00:50:05.172 INFO:teuthology.orchestra.run.smithi136.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.8 MB/s | 110 kB 00:00 2023-09-10T00:50:05.239 INFO:teuthology.orchestra.run.smithi136.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 811 kB/s | 54 kB 00:00 2023-09-10T00:50:05.340 INFO:teuthology.orchestra.run.smithi136.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 5.6 MB/s | 1.7 MB 00:00 2023-09-10T00:50:05.373 INFO:teuthology.orchestra.run.smithi136.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2023-09-10T00:50:05.423 INFO:teuthology.orchestra.run.smithi136.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 871 kB/s | 39 kB 00:00 2023-09-10T00:50:05.457 INFO:teuthology.orchestra.run.smithi136.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-09-10T00:50:05.490 INFO:teuthology.orchestra.run.smithi136.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-09-10T00:50:05.524 INFO:teuthology.orchestra.run.smithi136.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.9 MB/s | 538 kB 00:00 2023-09-10T00:50:05.549 INFO:teuthology.orchestra.run.smithi136.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-09-10T00:50:05.574 INFO:teuthology.orchestra.run.smithi136.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-09-10T00:50:05.608 INFO:teuthology.orchestra.run.smithi136.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-09-10T00:50:05.677 INFO:teuthology.orchestra.run.smithi136.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 388 kB/s | 27 kB 00:00 2023-09-10T00:50:05.725 INFO:teuthology.orchestra.run.smithi136.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 6.5 MB/s | 320 kB 00:00 2023-09-10T00:50:05.767 INFO:teuthology.orchestra.run.smithi136.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-09-10T00:50:05.843 INFO:teuthology.orchestra.run.smithi136.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-09-10T00:50:05.985 INFO:teuthology.orchestra.run.smithi136.stdout:(86/131): python3-scipy-1.0.0-21.module+el8.5.0 15 MB/s | 14 MB 00:00 2023-09-10T00:50:06.018 INFO:teuthology.orchestra.run.smithi136.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.6 MB/s | 457 kB 00:00 2023-09-10T00:50:06.052 INFO:teuthology.orchestra.run.smithi136.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.3 MB/s | 225 kB 00:00 2023-09-10T00:50:06.077 INFO:teuthology.orchestra.run.smithi136.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2023-09-10T00:50:06.114 INFO:teuthology.orchestra.run.smithi003.stdout:(10/131): ceph-radosgw-18.0.0-6025.g9670cca5.el 7.3 MB/s | 14 MB 00:01 2023-09-10T00:50:06.122 INFO:teuthology.orchestra.run.smithi136.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.8 MB/s | 126 kB 00:00 2023-09-10T00:50:06.153 INFO:teuthology.orchestra.run.smithi136.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 266 kB/s | 20 kB 00:00 2023-09-10T00:50:06.170 INFO:teuthology.orchestra.run.smithi136.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.1 MB/s | 103 kB 00:00 2023-09-10T00:50:06.181 INFO:teuthology.orchestra.run.smithi003.stdout:(11/131): cephfs-mirror-18.0.0-6025.g9670cca5.e 3.8 MB/s | 260 kB 00:00 2023-09-10T00:50:06.203 INFO:teuthology.orchestra.run.smithi136.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2023-09-10T00:50:06.231 INFO:teuthology.orchestra.run.smithi003.stdout:(12/131): libcephfs-devel-18.0.0-6025.g9670cca5 621 kB/s | 31 kB 00:00 2023-09-10T00:50:06.270 INFO:teuthology.orchestra.run.smithi136.stdout:(94/131): python3-numpy-1.14.3-10.el8.x86_64.rp 7.4 MB/s | 3.7 MB 00:00 2023-09-10T00:50:06.287 INFO:teuthology.orchestra.run.smithi136.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.3 MB/s | 198 kB 00:00 2023-09-10T00:50:06.329 INFO:teuthology.orchestra.run.smithi136.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-09-10T00:50:06.373 INFO:teuthology.orchestra.run.smithi003.stdout:(13/131): libcephfs2-18.0.0-6025.g9670cca5.el8. 5.7 MB/s | 827 kB 00:00 2023-09-10T00:50:06.379 INFO:teuthology.orchestra.run.smithi136.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-09-10T00:50:06.432 INFO:teuthology.orchestra.run.smithi003.stdout:(14/131): libcephsqlite-18.0.0-6025.g9670cca5.e 3.1 MB/s | 184 kB 00:00 2023-09-10T00:50:06.455 INFO:teuthology.orchestra.run.smithi136.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-09-10T00:50:06.482 INFO:teuthology.orchestra.run.smithi003.stdout:(15/131): librados-devel-18.0.0-6025.g9670cca5. 2.7 MB/s | 139 kB 00:00 2023-09-10T00:50:06.530 INFO:teuthology.orchestra.run.smithi136.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-09-10T00:50:06.582 INFO:teuthology.orchestra.run.smithi003.stdout:(16/131): libradosstriper1-18.0.0-6025.g9670cca 5.1 MB/s | 519 kB 00:00 2023-09-10T00:50:06.605 INFO:teuthology.orchestra.run.smithi136.stdout:(100/131): postgresql-contrib-10.21-2.module+el 5.3 MB/s | 808 kB 00:00 2023-09-10T00:50:06.647 INFO:teuthology.orchestra.run.smithi136.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-09-10T00:50:06.681 INFO:teuthology.orchestra.run.smithi136.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 10 MB/s | 1.5 MB 00:00 2023-09-10T00:50:06.706 INFO:teuthology.orchestra.run.smithi136.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 3.3 MB/s | 200 kB 00:00 2023-09-10T00:50:06.732 INFO:teuthology.orchestra.run.smithi136.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 849 kB/s | 43 kB 00:00 2023-09-10T00:50:06.757 INFO:teuthology.orchestra.run.smithi136.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-09-10T00:50:06.782 INFO:teuthology.orchestra.run.smithi136.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 961 kB/s | 48 kB 00:00 2023-09-10T00:50:06.807 INFO:teuthology.orchestra.run.smithi136.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-09-10T00:50:06.833 INFO:teuthology.orchestra.run.smithi136.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 845 kB/s | 43 kB 00:00 2023-09-10T00:50:06.858 INFO:teuthology.orchestra.run.smithi136.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-09-10T00:50:06.883 INFO:teuthology.orchestra.run.smithi136.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-09-10T00:50:06.917 INFO:teuthology.orchestra.run.smithi136.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-09-10T00:50:06.942 INFO:teuthology.orchestra.run.smithi136.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 320 kB/s | 19 kB 00:00 2023-09-10T00:50:07.036 INFO:teuthology.orchestra.run.smithi136.stdout:(113/131): postgresql-server-10.21-2.module+el8 7.7 MB/s | 5.1 MB 00:00 2023-09-10T00:50:07.068 INFO:teuthology.orchestra.run.smithi136.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.0 MB/s | 155 kB 00:00 2023-09-10T00:50:07.101 INFO:teuthology.orchestra.run.smithi136.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.4 MB/s | 559 kB 00:00 2023-09-10T00:50:07.126 INFO:teuthology.orchestra.run.smithi136.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 307 kB/s | 18 kB 00:00 2023-09-10T00:50:07.152 INFO:teuthology.orchestra.run.smithi136.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.6 MB/s | 238 kB 00:00 2023-09-10T00:50:07.186 INFO:teuthology.orchestra.run.smithi136.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2023-09-10T00:50:07.211 INFO:teuthology.orchestra.run.smithi136.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.2 MB/s | 72 kB 00:00 2023-09-10T00:50:07.245 INFO:teuthology.orchestra.run.smithi136.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2023-09-10T00:50:07.270 INFO:teuthology.orchestra.run.smithi136.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.0 MB/s | 60 kB 00:00 2023-09-10T00:50:07.295 INFO:teuthology.orchestra.run.smithi136.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-09-10T00:50:07.299 INFO:teuthology.orchestra.run.smithi003.stdout:(17/131): librgw2-18.0.0-6025.g9670cca5.el8.x86 9.5 MB/s | 6.8 MB 00:00 2023-09-10T00:50:07.320 INFO:teuthology.orchestra.run.smithi136.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-09-10T00:50:07.346 INFO:teuthology.orchestra.run.smithi136.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 768 kB/s | 39 kB 00:00 2023-09-10T00:50:07.378 INFO:teuthology.orchestra.run.smithi003.stdout:(18/131): python3-ceph-argparse-18.0.0-6025.g96 571 kB/s | 45 kB 00:00 2023-09-10T00:50:07.379 INFO:teuthology.orchestra.run.smithi136.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 11 MB/s | 644 kB 00:00 2023-09-10T00:50:07.405 INFO:teuthology.orchestra.run.smithi136.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-09-10T00:50:07.430 INFO:teuthology.orchestra.run.smithi136.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-09-10T00:50:07.441 INFO:teuthology.orchestra.run.smithi003.stdout:(19/131): python3-ceph-common-18.0.0-6025.g9670 2.0 MB/s | 126 kB 00:00 2023-09-10T00:50:07.500 INFO:teuthology.orchestra.run.smithi003.stdout:(20/131): python3-cephfs-18.0.0-6025.g9670cca5. 3.3 MB/s | 197 kB 00:00 2023-09-10T00:50:07.522 INFO:teuthology.orchestra.run.smithi136.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 5.9 MB/s | 2.3 MB 00:00 2023-09-10T00:50:07.572 INFO:teuthology.orchestra.run.smithi136.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-09-10T00:50:07.583 INFO:teuthology.orchestra.run.smithi003.stdout:(21/131): python3-rados-18.0.0-6025.g9670cca5.e 4.4 MB/s | 372 kB 00:00 2023-09-10T00:50:07.692 INFO:teuthology.orchestra.run.smithi003.stdout:(22/131): python3-rbd-18.0.0-6025.g9670cca5.el8 3.3 MB/s | 364 kB 00:00 2023-09-10T00:50:07.768 INFO:teuthology.orchestra.run.smithi003.stdout:(23/131): python3-rgw-18.0.0-6025.g9670cca5.el8 1.4 MB/s | 107 kB 00:00 2023-09-10T00:50:07.835 INFO:teuthology.orchestra.run.smithi003.stdout:(24/131): rbd-fuse-18.0.0-6025.g9670cca5.el8.x8 1.4 MB/s | 94 kB 00:00 2023-09-10T00:50:07.981 INFO:teuthology.orchestra.run.smithi136.stdout:(130/131): librbd1-18.0.0-6025.g9670cca5.el8.x8 6.7 MB/s | 3.7 MB 00:00 2023-09-10T00:50:08.056 INFO:teuthology.orchestra.run.smithi136.stdout:(131/131): librados2-18.0.0-6025.g9670cca5.el8. 5.7 MB/s | 3.7 MB 00:00 2023-09-10T00:50:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2023-09-10T00:50:08.059 INFO:teuthology.orchestra.run.smithi136.stdout:Total 20 MB/s | 208 MB 00:10 2023-09-10T00:50:08.268 INFO:teuthology.orchestra.run.smithi003.stdout:(25/131): rbd-mirror-18.0.0-6025.g9670cca5.el8. 8.3 MB/s | 3.6 MB 00:00 2023-09-10T00:50:08.310 INFO:teuthology.orchestra.run.smithi003.stdout:(26/131): rbd-nbd-18.0.0-6025.g9670cca5.el8.x86 4.2 MB/s | 179 kB 00:00 2023-09-10T00:50:08.361 INFO:teuthology.orchestra.run.smithi003.stdout:(27/131): ceph-grafana-dashboards-18.0.0-6025.g 462 kB/s | 23 kB 00:00 2023-09-10T00:50:08.436 INFO:teuthology.orchestra.run.smithi003.stdout:(28/131): ceph-mgr-cephadm-18.0.0-6025.g9670cca 1.7 MB/s | 132 kB 00:00 2023-09-10T00:50:08.628 INFO:teuthology.orchestra.run.smithi003.stdout:(29/131): ceph-mgr-dashboard-18.0.0-6025.g9670c 9.1 MB/s | 1.7 MB 00:00 2023-09-10T00:50:08.895 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2023-09-10T00:50:09.014 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2023-09-10T00:50:09.015 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2023-09-10T00:50:09.378 INFO:teuthology.orchestra.run.smithi003.stdout:(30/131): ceph-test-18.0.0-6025.g9670cca5.el8.x 11 MB/s | 49 MB 00:04 2023-09-10T00:50:09.454 INFO:teuthology.orchestra.run.smithi003.stdout:(31/131): ceph-mgr-modules-core-18.0.0-6025.g96 3.2 MB/s | 242 kB 00:00 2023-09-10T00:50:09.495 INFO:teuthology.orchestra.run.smithi003.stdout:(32/131): ceph-mgr-rook-18.0.0-6025.g9670cca5.e 1.1 MB/s | 47 kB 00:00 2023-09-10T00:50:09.529 INFO:teuthology.orchestra.run.smithi003.stdout:(33/131): ceph-prometheus-alerts-18.0.0-6025.g9 435 kB/s | 15 kB 00:00 2023-09-10T00:50:09.579 INFO:teuthology.orchestra.run.smithi003.stdout:(34/131): ceph-volume-18.0.0-6025.g9670cca5.el8 5.1 MB/s | 261 kB 00:00 2023-09-10T00:50:09.723 INFO:teuthology.orchestra.run.smithi003.stdout:(35/131): ceph-mgr-diskprediction-local-18.0.0- 6.8 MB/s | 7.4 MB 00:01 2023-09-10T00:50:09.755 INFO:teuthology.orchestra.run.smithi003.stdout:(36/131): cephadm-18.0.0-6025.g9670cca5.el8.noa 1.0 MB/s | 176 kB 00:00 2023-09-10T00:50:09.772 INFO:teuthology.orchestra.run.smithi003.stdout:(37/131): cephfs-top-18.0.0-6025.g9670cca5.el8. 367 kB/s | 18 kB 00:00 2023-09-10T00:50:10.130 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2023-09-10T00:50:10.134 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2023-09-10T00:50:10.173 INFO:teuthology.orchestra.run.smithi003.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 147 kB/s | 59 kB 00:00 2023-09-10T00:50:10.273 INFO:teuthology.orchestra.run.smithi003.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 592 kB/s | 306 kB 00:00 2023-09-10T00:50:10.356 INFO:teuthology.orchestra.run.smithi003.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 410 kB/s | 75 kB 00:00 2023-09-10T00:50:10.465 INFO:teuthology.orchestra.run.smithi003.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.4 MB/s | 476 kB 00:00 2023-09-10T00:50:10.490 INFO:teuthology.orchestra.run.smithi003.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 326 kB/s | 44 kB 00:00 2023-09-10T00:50:10.566 INFO:teuthology.orchestra.run.smithi003.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-09-10T00:50:10.591 INFO:teuthology.orchestra.run.smithi003.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 329 kB/s | 33 kB 00:00 2023-09-10T00:50:10.666 INFO:teuthology.orchestra.run.smithi003.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 163 kB/s | 16 kB 00:00 2023-09-10T00:50:10.800 INFO:teuthology.orchestra.run.smithi003.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.8 MB/s | 384 kB 00:00 2023-09-10T00:50:10.825 INFO:teuthology.orchestra.run.smithi003.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 738 kB/s | 173 kB 00:00 2023-09-10T00:50:10.917 INFO:teuthology.orchestra.run.smithi003.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 695 kB/s | 81 kB 00:00 2023-09-10T00:50:10.942 INFO:teuthology.orchestra.run.smithi003.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-09-10T00:50:11.010 INFO:teuthology.orchestra.run.smithi003.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 191 kB/s | 18 kB 00:00 2023-09-10T00:50:11.118 INFO:teuthology.orchestra.run.smithi003.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 450 kB/s | 49 kB 00:00 2023-09-10T00:50:11.227 INFO:teuthology.orchestra.run.smithi003.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 543 kB/s | 59 kB 00:00 2023-09-10T00:50:11.319 INFO:teuthology.orchestra.run.smithi003.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 658 kB/s | 60 kB 00:00 2023-09-10T00:50:11.452 INFO:teuthology.orchestra.run.smithi003.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2023-09-10T00:50:11.552 INFO:teuthology.orchestra.run.smithi003.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-09-10T00:50:11.611 INFO:teuthology.orchestra.run.smithi003.stdout:(56/131): python3-kubernetes-11.0.0-6.el8.noarc 1.7 MB/s | 1.2 MB 00:00 2023-09-10T00:50:11.645 INFO:teuthology.orchestra.run.smithi003.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 364 kB/s | 34 kB 00:00 2023-09-10T00:50:11.712 INFO:teuthology.orchestra.run.smithi003.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-09-10T00:50:11.737 INFO:teuthology.orchestra.run.smithi003.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 700 kB/s | 64 kB 00:00 2023-09-10T00:50:11.821 INFO:teuthology.orchestra.run.smithi003.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2023-09-10T00:50:11.846 INFO:teuthology.orchestra.run.smithi003.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 220 kB/s | 24 kB 00:00 2023-09-10T00:50:11.930 INFO:teuthology.orchestra.run.smithi003.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 269 kB/s | 29 kB 00:00 2023-09-10T00:50:11.955 INFO:teuthology.orchestra.run.smithi003.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-09-10T00:50:12.056 INFO:teuthology.orchestra.run.smithi003.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.5 MB/s | 251 kB 00:00 2023-09-10T00:50:12.156 INFO:teuthology.orchestra.run.smithi003.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.1 MB/s | 255 kB 00:00 2023-09-10T00:50:12.181 INFO:teuthology.orchestra.run.smithi003.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 489 kB/s | 61 kB 00:00 2023-09-10T00:50:12.273 INFO:teuthology.orchestra.run.smithi003.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 731 kB/s | 85 kB 00:00 2023-09-10T00:50:12.298 INFO:teuthology.orchestra.run.smithi003.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 196 kB/s | 23 kB 00:00 2023-09-10T00:50:12.324 INFO:teuthology.orchestra.run.smithi003.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-09-10T00:50:12.724 INFO:teuthology.orchestra.run.smithi003.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 34 MB/s | 14 MB 00:00 2023-09-10T00:50:12.766 INFO:teuthology.orchestra.run.smithi003.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-09-10T00:50:12.833 INFO:teuthology.orchestra.run.smithi003.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 814 kB/s | 54 kB 00:00 2023-09-10T00:50:12.892 INFO:teuthology.orchestra.run.smithi003.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 2.8 MB/s | 1.7 MB 00:00 2023-09-10T00:50:12.925 INFO:teuthology.orchestra.run.smithi003.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.8 MB/s | 259 kB 00:00 2023-09-10T00:50:12.975 INFO:teuthology.orchestra.run.smithi003.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 936 kB/s | 39 kB 00:00 2023-09-10T00:50:12.976 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-09-10T00:50:12.977 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-09-10T00:50:12.992 INFO:teuthology.orchestra.run.smithi092.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-09-10T00:50:12.993 INFO:teuthology.orchestra.run.smithi092.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-09-10T00:50:12.994 INFO:teuthology.orchestra.run.smithi092.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-09-10T00:50:12.995 INFO:teuthology.orchestra.run.smithi092.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-09-10T00:50:12.997 INFO:teuthology.orchestra.run.smithi092.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-09-10T00:50:13.001 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-09-10T00:50:13.002 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-09-10T00:50:13.017 INFO:teuthology.orchestra.run.smithi003.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2023-09-10T00:50:13.059 INFO:teuthology.orchestra.run.smithi003.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 8.0 MB/s | 688 kB 00:00 2023-09-10T00:50:13.093 INFO:teuthology.orchestra.run.smithi003.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.0 MB/s | 538 kB 00:00 2023-09-10T00:50:13.136 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-09-10T00:50:13.138 INFO:teuthology.orchestra.run.smithi003.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.9 MB/s | 157 kB 00:00 2023-09-10T00:50:13.149 INFO:teuthology.orchestra.run.smithi092.stdout:=========================================================================================================================== 2023-09-10T00:50:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-09-10T00:50:13.149 INFO:teuthology.orchestra.run.smithi092.stdout:=========================================================================================================================== 2023-09-10T00:50:13.149 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-09-10T00:50:13.149 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-09-10T00:50:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-09-10T00:50:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 6.4 k 2023-09-10T00:50:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 5.4 M 2023-09-10T00:50:13.150 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 907 k 2023-09-10T00:50:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 167 k 2023-09-10T00:50:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 1.6 M 2023-09-10T00:50:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 132 k 2023-09-10T00:50:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 1.7 M 2023-09-10T00:50:13.151 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 7.4 M 2023-09-10T00:50:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 47 k 2023-09-10T00:50:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 14 M 2023-09-10T00:50:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 49 M 2023-09-10T00:50:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 261 k 2023-09-10T00:50:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 176 k 2023-09-10T00:50:13.152 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 260 k 2023-09-10T00:50:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 18 k 2023-09-10T00:50:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-09-10T00:50:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-09-10T00:50:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-09-10T00:50:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 31 k 2023-09-10T00:50:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 827 k 2023-09-10T00:50:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 139 k 2023-09-10T00:50:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-09-10T00:50:13.154 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-09-10T00:50:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-09-10T00:50:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-09-10T00:50:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-09-10T00:50:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-09-10T00:50:13.155 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 197 k 2023-09-10T00:50:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 372 k 2023-09-10T00:50:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 364 k 2023-09-10T00:50:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 107 k 2023-09-10T00:50:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 94 k 2023-09-10T00:50:13.156 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.6 M 2023-09-10T00:50:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 179 k 2023-09-10T00:50:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-09-10T00:50:13.157 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2023-09-10T00:50:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-09-10T00:50:13.157 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.7 M 2023-09-10T00:50:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 3.7 M 2023-09-10T00:50:13.158 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-09-10T00:50:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-09-10T00:50:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 22 M 2023-09-10T00:50:13.158 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 23 k 2023-09-10T00:50:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 2.3 M 2023-09-10T00:50:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 242 k 2023-09-10T00:50:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 4.6 M 2023-09-10T00:50:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 18 M 2023-09-10T00:50:13.159 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6025.g9670cca5.el8 ceph-noarch 15 k 2023-09-10T00:50:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 24 k 2023-09-10T00:50:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-10T00:50:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-10T00:50:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-09-10T00:50:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-09-10T00:50:13.160 INFO:teuthology.orchestra.run.smithi092.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-09-10T00:50:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 184 k 2023-09-10T00:50:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-09-10T00:50:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-09-10T00:50:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-10T00:50:13.161 INFO:teuthology.orchestra.run.smithi092.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-09-10T00:50:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-09-10T00:50:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-10T00:50:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 519 k 2023-09-10T00:50:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-09-10T00:50:13.162 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 6.8 M 2023-09-10T00:50:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-09-10T00:50:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-09-10T00:50:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-10T00:50:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-09-10T00:50:13.163 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-10T00:50:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-10T00:50:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-10T00:50:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-09-10T00:50:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-09-10T00:50:13.164 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-09-10T00:50:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-09-10T00:50:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-10T00:50:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-10T00:50:13.165 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-10T00:50:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-10T00:50:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-10T00:50:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 45 k 2023-09-10T00:50:13.166 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:18.0.0-6025.g9670cca5.el8 ceph 126 k 2023-09-10T00:50:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-10T00:50:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-09-10T00:50:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-10T00:50:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-10T00:50:13.167 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-09-10T00:50:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-10T00:50:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-10T00:50:13.168 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-10T00:50:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-10T00:50:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-10T00:50:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-10T00:50:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-10T00:50:13.169 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-10T00:50:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-10T00:50:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-10T00:50:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-10T00:50:13.170 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-10T00:50:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-10T00:50:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-10T00:50:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-10T00:50:13.171 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-10T00:50:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-10T00:50:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-10T00:50:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-10T00:50:13.172 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-09-10T00:50:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-10T00:50:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-10T00:50:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-09-10T00:50:13.173 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-10T00:50:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-10T00:50:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-10T00:50:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-10T00:50:13.174 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-10T00:50:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-09-10T00:50:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-10T00:50:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-10T00:50:13.175 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-10T00:50:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-10T00:50:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-10T00:50:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-10T00:50:13.176 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-10T00:50:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-10T00:50:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-10T00:50:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-10T00:50:13.177 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-09-10T00:50:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-10T00:50:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-09-10T00:50:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-10T00:50:13.178 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-09-10T00:50:13.179 INFO:teuthology.orchestra.run.smithi092.stdout:Enabling module streams: 2023-09-10T00:50:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql 10 2023-09-10T00:50:13.179 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:50:13.179 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-09-10T00:50:13.180 INFO:teuthology.orchestra.run.smithi092.stdout:=========================================================================================================================== 2023-09-10T00:50:13.180 INFO:teuthology.orchestra.run.smithi092.stdout:Install 128 Packages 2023-09-10T00:50:13.180 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 3 Packages 2023-09-10T00:50:13.180 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:50:13.181 INFO:teuthology.orchestra.run.smithi003.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2023-09-10T00:50:13.181 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 208 M 2023-09-10T00:50:13.181 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-09-10T00:50:13.219 INFO:teuthology.orchestra.run.smithi003.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 8.7 MB/s | 713 kB 00:00 2023-09-10T00:50:13.261 INFO:teuthology.orchestra.run.smithi003.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 639 kB/s | 27 kB 00:00 2023-09-10T00:50:13.320 INFO:teuthology.orchestra.run.smithi003.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 5.3 MB/s | 320 kB 00:00 2023-09-10T00:50:13.362 INFO:teuthology.orchestra.run.smithi003.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-09-10T00:50:13.437 INFO:teuthology.orchestra.run.smithi003.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-09-10T00:50:13.512 INFO:teuthology.orchestra.run.smithi003.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 6.0 MB/s | 457 kB 00:00 2023-09-10T00:50:13.688 INFO:teuthology.orchestra.run.smithi003.stdout:(87/131): ceph-osd-18.0.0-6025.g9670cca5.el8.x8 1.8 MB/s | 18 MB 00:09 2023-09-10T00:50:13.721 INFO:teuthology.orchestra.run.smithi003.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.1 MB/s | 225 kB 00:00 2023-09-10T00:50:13.747 INFO:teuthology.orchestra.run.smithi092.stdout:(1/131): ceph-18.0.0-6025.g9670cca5.el8.x86_64. 33 kB/s | 6.4 kB 00:00 2023-09-10T00:50:13.771 INFO:teuthology.orchestra.run.smithi003.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 769 kB/s | 64 kB 00:00 2023-09-10T00:50:13.797 INFO:teuthology.orchestra.run.smithi003.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.9 MB/s | 126 kB 00:00 2023-09-10T00:50:13.847 INFO:teuthology.orchestra.run.smithi003.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 266 kB/s | 20 kB 00:00 2023-09-10T00:50:13.872 INFO:teuthology.orchestra.run.smithi003.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.3 MB/s | 103 kB 00:00 2023-09-10T00:50:13.939 INFO:teuthology.orchestra.run.smithi003.stdout:(93/131): python3-numpy-1.14.3-10.el8.x86_64.rp 6.5 MB/s | 3.7 MB 00:00 2023-09-10T00:50:13.973 INFO:teuthology.orchestra.run.smithi003.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 5.5 MB/s | 709 kB 00:00 2023-09-10T00:50:14.007 INFO:teuthology.orchestra.run.smithi003.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.9 MB/s | 198 kB 00:00 2023-09-10T00:50:14.057 INFO:teuthology.orchestra.run.smithi003.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 3.9 MB/s | 202 kB 00:00 2023-09-10T00:50:14.107 INFO:teuthology.orchestra.run.smithi003.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-09-10T00:50:14.214 INFO:teuthology.orchestra.run.smithi092.stdout:(2/131): ceph-fuse-18.0.0-6025.g9670cca5.el8.x8 1.9 MB/s | 907 kB 00:00 2023-09-10T00:50:14.233 INFO:teuthology.orchestra.run.smithi003.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 13 MB/s | 4.9 MB 00:00 2023-09-10T00:50:14.300 INFO:teuthology.orchestra.run.smithi003.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-09-10T00:50:14.306 INFO:teuthology.orchestra.run.smithi092.stdout:(3/131): ceph-immutable-object-cache-18.0.0-602 1.8 MB/s | 167 kB 00:00 2023-09-10T00:50:14.350 INFO:teuthology.orchestra.run.smithi003.stdout:(100/131): postgresql-contrib-10.21-2.module+el 6.8 MB/s | 808 kB 00:00 2023-09-10T00:50:14.392 INFO:teuthology.orchestra.run.smithi003.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-09-10T00:50:14.442 INFO:teuthology.orchestra.run.smithi003.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-09-10T00:50:14.476 INFO:teuthology.orchestra.run.smithi003.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.3 MB/s | 200 kB 00:00 2023-09-10T00:50:14.523 INFO:teuthology.orchestra.run.smithi092.stdout:(4/131): ceph-base-18.0.0-6025.g9670cca5.el8.x8 5.5 MB/s | 5.4 MB 00:00 2023-09-10T00:50:14.526 INFO:teuthology.orchestra.run.smithi003.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 512 kB/s | 43 kB 00:00 2023-09-10T00:50:14.560 INFO:teuthology.orchestra.run.smithi003.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2023-09-10T00:50:14.576 INFO:teuthology.orchestra.run.smithi003.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 956 kB/s | 48 kB 00:00 2023-09-10T00:50:14.610 INFO:teuthology.orchestra.run.smithi003.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-09-10T00:50:14.627 INFO:teuthology.orchestra.run.smithi003.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 846 kB/s | 43 kB 00:00 2023-09-10T00:50:14.703 INFO:teuthology.orchestra.run.smithi003.stdout:(109/131): postgresql-server-10.21-2.module+el8 8.5 MB/s | 5.1 MB 00:00 2023-09-10T00:50:14.728 INFO:teuthology.orchestra.run.smithi003.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 931 kB/s | 109 kB 00:00 2023-09-10T00:50:14.753 INFO:teuthology.orchestra.run.smithi003.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2023-09-10T00:50:14.778 INFO:teuthology.orchestra.run.smithi003.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 371 kB/s | 19 kB 00:00 2023-09-10T00:50:14.804 INFO:teuthology.orchestra.run.smithi003.stdout:(113/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.4 MB/s | 560 kB 00:00 2023-09-10T00:50:14.823 INFO:teuthology.orchestra.run.smithi092.stdout:(5/131): ceph-mgr-18.0.0-6025.g9670cca5.el8.x86 5.4 MB/s | 1.6 MB 00:00 2023-09-10T00:50:14.865 INFO:teuthology.orchestra.run.smithi003.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.4 MB/s | 155 kB 00:00 2023-09-10T00:50:14.913 INFO:teuthology.orchestra.run.smithi003.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 4.1 MB/s | 559 kB 00:00 2023-09-10T00:50:15.090 INFO:teuthology.orchestra.run.smithi092.stdout:(6/131): ceph-common-18.0.0-6025.g9670cca5.el8. 15 MB/s | 22 MB 00:01 2023-09-10T00:50:15.140 INFO:teuthology.orchestra.run.smithi092.stdout:(7/131): ceph-mds-18.0.0-6025.g9670cca5.el8.x86 2.8 MB/s | 2.3 MB 00:00 2023-09-10T00:50:15.155 INFO:teuthology.orchestra.run.smithi003.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 51 kB/s | 18 kB 00:00 2023-09-10T00:50:15.214 INFO:teuthology.orchestra.run.smithi003.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 790 kB/s | 238 kB 00:00 2023-09-10T00:50:15.247 INFO:teuthology.orchestra.run.smithi003.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 1.5 MB/s | 605 kB 00:00 2023-09-10T00:50:15.272 INFO:teuthology.orchestra.run.smithi003.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.2 MB/s | 72 kB 00:00 2023-09-10T00:50:15.298 INFO:teuthology.orchestra.run.smithi003.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-09-10T00:50:15.323 INFO:teuthology.orchestra.run.smithi003.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-09-10T00:50:15.340 INFO:teuthology.orchestra.run.smithi092.stdout:(8/131): ceph-mon-18.0.0-6025.g9670cca5.el8.x86 8.9 MB/s | 4.6 MB 00:00 2023-09-10T00:50:15.348 INFO:teuthology.orchestra.run.smithi003.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-09-10T00:50:15.359 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2023-09-10T00:50:15.373 INFO:teuthology.orchestra.run.smithi003.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-09-10T00:50:15.399 INFO:teuthology.orchestra.run.smithi003.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 771 kB/s | 39 kB 00:00 2023-09-10T00:50:15.424 INFO:teuthology.orchestra.run.smithi092.stdout:(9/131): ceph-selinux-18.0.0-6025.g9670cca5.el8 292 kB/s | 24 kB 00:00 2023-09-10T00:50:15.441 INFO:teuthology.orchestra.run.smithi003.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.4 MB/s | 644 kB 00:00 2023-09-10T00:50:15.474 INFO:teuthology.orchestra.run.smithi003.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.2 MB/s | 171 kB 00:00 2023-09-10T00:50:15.500 INFO:teuthology.orchestra.run.smithi003.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.3 MB/s | 138 kB 00:00 2023-09-10T00:50:15.534 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-09-10T00:50:15.575 INFO:teuthology.orchestra.run.smithi003.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 5.5 MB/s | 2.3 MB 00:00 2023-09-10T00:50:15.642 INFO:teuthology.orchestra.run.smithi003.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-09-10T00:50:15.739 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-09-10T00:50:15.959 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-09-10T00:50:16.541 INFO:teuthology.orchestra.run.smithi092.stdout:(10/131): ceph-osd-18.0.0-6025.g9670cca5.el8.x8 12 MB/s | 18 MB 00:01 2023-09-10T00:50:16.599 INFO:teuthology.orchestra.run.smithi092.stdout:(11/131): cephfs-mirror-18.0.0-6025.g9670cca5.e 4.4 MB/s | 260 kB 00:00 2023-09-10T00:50:16.600 INFO:teuthology.orchestra.run.smithi003.stdout:(130/131): librados2-18.0.0-6025.g9670cca5.el8. 3.3 MB/s | 3.7 MB 00:01 2023-09-10T00:50:16.641 INFO:teuthology.orchestra.run.smithi092.stdout:(12/131): libcephfs-devel-18.0.0-6025.g9670cca5 748 kB/s | 31 kB 00:00 2023-09-10T00:50:16.678 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-09-10T00:50:16.758 INFO:teuthology.orchestra.run.smithi092.stdout:(13/131): libcephfs2-18.0.0-6025.g9670cca5.el8. 6.9 MB/s | 827 kB 00:00 2023-09-10T00:50:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-09-10T00:50:16.834 INFO:teuthology.orchestra.run.smithi092.stdout:(14/131): libcephsqlite-18.0.0-6025.g9670cca5.e 2.4 MB/s | 184 kB 00:00 2023-09-10T00:50:16.884 INFO:teuthology.orchestra.run.smithi092.stdout:(15/131): librados-devel-18.0.0-6025.g9670cca5. 2.7 MB/s | 139 kB 00:00 2023-09-10T00:50:16.959 INFO:teuthology.orchestra.run.smithi092.stdout:(16/131): libradosstriper1-18.0.0-6025.g9670cca 6.8 MB/s | 519 kB 00:00 2023-09-10T00:50:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-09-10T00:50:17.473 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-09-10T00:50:17.526 INFO:teuthology.orchestra.run.smithi092.stdout:(17/131): librgw2-18.0.0-6025.g9670cca5.el8.x86 12 MB/s | 6.8 MB 00:00 2023-09-10T00:50:17.568 INFO:teuthology.orchestra.run.smithi092.stdout:(18/131): python3-ceph-argparse-18.0.0-6025.g96 1.1 MB/s | 45 kB 00:00 2023-09-10T00:50:17.618 INFO:teuthology.orchestra.run.smithi092.stdout:(19/131): python3-ceph-common-18.0.0-6025.g9670 2.5 MB/s | 126 kB 00:00 2023-09-10T00:50:17.685 INFO:teuthology.orchestra.run.smithi092.stdout:(20/131): python3-cephfs-18.0.0-6025.g9670cca5. 2.9 MB/s | 197 kB 00:00 2023-09-10T00:50:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:50:17.785 INFO:teuthology.orchestra.run.smithi092.stdout:(21/131): python3-rados-18.0.0-6025.g9670cca5.e 3.6 MB/s | 372 kB 00:00 2023-09-10T00:50:17.800 INFO:teuthology.orchestra.run.smithi003.stdout:(131/131): librbd1-18.0.0-6025.g9670cca5.el8.x8 1.6 MB/s | 3.7 MB 00:02 2023-09-10T00:50:17.804 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-09-10T00:50:17.804 INFO:teuthology.orchestra.run.smithi003.stdout:Total 12 MB/s | 208 MB 00:17 2023-09-10T00:50:17.852 INFO:teuthology.orchestra.run.smithi092.stdout:(22/131): python3-rbd-18.0.0-6025.g9670cca5.el8 5.4 MB/s | 364 kB 00:00 2023-09-10T00:50:17.911 INFO:teuthology.orchestra.run.smithi092.stdout:(23/131): python3-rgw-18.0.0-6025.g9670cca5.el8 1.8 MB/s | 107 kB 00:00 2023-09-10T00:50:17.978 INFO:teuthology.orchestra.run.smithi092.stdout:(24/131): rbd-fuse-18.0.0-6025.g9670cca5.el8.x8 1.4 MB/s | 94 kB 00:00 2023-09-10T00:50:18.094 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:50:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:50:18.295 INFO:teuthology.orchestra.run.smithi092.stdout:(25/131): rbd-mirror-18.0.0-6025.g9670cca5.el8. 11 MB/s | 3.6 MB 00:00 2023-09-10T00:50:18.345 INFO:teuthology.orchestra.run.smithi092.stdout:(26/131): rbd-nbd-18.0.0-6025.g9670cca5.el8.x86 3.5 MB/s | 179 kB 00:00 2023-09-10T00:50:18.396 INFO:teuthology.orchestra.run.smithi092.stdout:(27/131): ceph-grafana-dashboards-18.0.0-6025.g 461 kB/s | 23 kB 00:00 2023-09-10T00:50:18.410 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:50:18.463 INFO:teuthology.orchestra.run.smithi092.stdout:(28/131): ceph-mgr-cephadm-18.0.0-6025.g9670cca 1.9 MB/s | 132 kB 00:00 2023-09-10T00:50:18.618 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:50:18.629 INFO:teuthology.orchestra.run.smithi092.stdout:(29/131): ceph-mgr-dashboard-18.0.0-6025.g9670c 11 MB/s | 1.7 MB 00:00 2023-09-10T00:50:18.702 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:50:18.889 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 9/134 2023-09-10T00:50:19.132 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-09-10T00:50:19.197 INFO:teuthology.orchestra.run.smithi092.stdout:(30/131): ceph-mgr-diskprediction-local-18.0.0- 13 MB/s | 7.4 MB 00:00 2023-09-10T00:50:19.264 INFO:teuthology.orchestra.run.smithi092.stdout:(31/131): ceph-mgr-modules-core-18.0.0-6025.g96 3.6 MB/s | 242 kB 00:00 2023-09-10T00:50:19.321 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-09-10T00:50:19.338 INFO:teuthology.orchestra.run.smithi092.stdout:(32/131): ceph-mgr-rook-18.0.0-6025.g9670cca5.e 632 kB/s | 47 kB 00:00 2023-09-10T00:50:19.407 INFO:teuthology.orchestra.run.smithi092.stdout:(33/131): ceph-prometheus-alerts-18.0.0-6025.g9 218 kB/s | 15 kB 00:00 2023-09-10T00:50:19.473 INFO:teuthology.orchestra.run.smithi092.stdout:(34/131): ceph-volume-18.0.0-6025.g9670cca5.el8 3.8 MB/s | 261 kB 00:00 2023-09-10T00:50:19.489 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-09-10T00:50:19.523 INFO:teuthology.orchestra.run.smithi092.stdout:(35/131): cephadm-18.0.0-6025.g9670cca5.el8.noa 3.5 MB/s | 176 kB 00:00 2023-09-10T00:50:19.565 INFO:teuthology.orchestra.run.smithi092.stdout:(36/131): cephfs-top-18.0.0-6025.g9670cca5.el8. 432 kB/s | 18 kB 00:00 2023-09-10T00:50:19.636 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-09-10T00:50:19.637 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-09-10T00:50:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-09-10T00:50:19.984 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-09-10T00:50:20.152 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 14/134 2023-09-10T00:50:20.262 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 14/134 2023-09-10T00:50:20.426 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 15/134 2023-09-10T00:50:20.473 INFO:teuthology.orchestra.run.smithi092.stdout:(37/131): ceph-test-18.0.0-6025.g9670cca5.el8.x 9.6 MB/s | 49 MB 00:05 2023-09-10T00:50:20.502 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 15/134 2023-09-10T00:50:20.635 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-09-10T00:50:20.640 INFO:teuthology.orchestra.run.smithi092.stdout:(38/131): ceph-radosgw-18.0.0-6025.g9670cca5.el 2.5 MB/s | 14 MB 00:05 2023-09-10T00:50:20.732 INFO:teuthology.orchestra.run.smithi092.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 262 kB/s | 306 kB 00:01 2023-09-10T00:50:20.800 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-09-10T00:50:20.965 INFO:teuthology.orchestra.run.smithi092.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 119 kB/s | 59 kB 00:00 2023-09-10T00:50:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-09-10T00:50:21.074 INFO:teuthology.orchestra.run.smithi092.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 402 kB/s | 44 kB 00:00 2023-09-10T00:50:21.133 INFO:teuthology.orchestra.run.smithi092.stdout:(42/131): libunwind-1.3.1-3.el8.x86_64.rpm 152 kB/s | 75 kB 00:00 2023-09-10T00:50:21.233 INFO:teuthology.orchestra.run.smithi092.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 950 kB/s | 476 kB 00:00 2023-09-10T00:50:21.258 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-09-10T00:50:21.263 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-09-10T00:50:21.299 INFO:teuthology.orchestra.run.smithi092.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 199 kB/s | 33 kB 00:00 2023-09-10T00:50:21.342 INFO:teuthology.orchestra.run.smithi092.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 150 kB/s | 16 kB 00:00 2023-09-10T00:50:21.417 INFO:teuthology.orchestra.run.smithi092.stdout:(46/131): python3-beautifulsoup4-4.6.3-2.el8.1. 540 kB/s | 185 kB 00:00 2023-09-10T00:50:21.551 INFO:teuthology.orchestra.run.smithi092.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 610 kB/s | 81 kB 00:00 2023-09-10T00:50:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-09-10T00:50:21.668 INFO:teuthology.orchestra.run.smithi092.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-09-10T00:50:21.768 INFO:teuthology.orchestra.run.smithi092.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 903 kB/s | 384 kB 00:00 2023-09-10T00:50:21.785 INFO:teuthology.orchestra.run.smithi092.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 150 kB/s | 18 kB 00:00 2023-09-10T00:50:21.835 INFO:teuthology.orchestra.run.smithi092.stdout:(51/131): python3-cheroot-8.5.2-1.el8.noarch.rp 322 kB/s | 173 kB 00:00 2023-09-10T00:50:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-09-10T00:50:21.944 INFO:teuthology.orchestra.run.smithi092.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 307 kB/s | 49 kB 00:00 2023-09-10T00:50:21.978 INFO:teuthology.orchestra.run.smithi092.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 413 kB/s | 59 kB 00:00 2023-09-10T00:50:22.061 INFO:teuthology.orchestra.run.smithi092.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 514 kB/s | 60 kB 00:00 2023-09-10T00:50:22.178 INFO:teuthology.orchestra.run.smithi092.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2023-09-10T00:50:22.295 INFO:teuthology.orchestra.run.smithi092.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 288 kB/s | 34 kB 00:00 2023-09-10T00:50:22.387 INFO:teuthology.orchestra.run.smithi092.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 692 kB/s | 283 kB 00:00 2023-09-10T00:50:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-09-10T00:50:22.504 INFO:teuthology.orchestra.run.smithi092.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 551 kB/s | 64 kB 00:00 2023-09-10T00:50:22.529 INFO:teuthology.orchestra.run.smithi092.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 840 kB/s | 196 kB 00:00 2023-09-10T00:50:22.646 INFO:teuthology.orchestra.run.smithi092.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 205 kB/s | 24 kB 00:00 2023-09-10T00:50:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-09-10T00:50:22.763 INFO:teuthology.orchestra.run.smithi092.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 250 kB/s | 29 kB 00:00 2023-09-10T00:50:22.813 INFO:teuthology.orchestra.run.smithi092.stdout:(62/131): python3-kubernetes-11.0.0-6.el8.noarc 1.1 MB/s | 1.2 MB 00:01 2023-09-10T00:50:22.880 INFO:teuthology.orchestra.run.smithi092.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 235 kB/s | 27 kB 00:00 2023-09-10T00:50:22.906 INFO:teuthology.orchestra.run.smithi092.stdout:(64/131): python3-simplegeneric-0.8.1-17.el8.no 47 kB/s | 19 kB 00:00 2023-09-10T00:50:23.114 INFO:teuthology.orchestra.run.smithi092.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 850 kB/s | 255 kB 00:00 2023-09-10T00:50:23.214 INFO:teuthology.orchestra.run.smithi092.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 753 kB/s | 251 kB 00:00 2023-09-10T00:50:23.240 INFO:teuthology.orchestra.run.smithi092.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 183 kB/s | 61 kB 00:00 2023-09-10T00:50:23.265 INFO:teuthology.orchestra.run.smithi092.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 567 kB/s | 85 kB 00:00 2023-09-10T00:50:23.290 INFO:teuthology.orchestra.run.smithi092.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-09-10T00:50:23.332 INFO:teuthology.orchestra.run.smithi092.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 195 kB/s | 23 kB 00:00 2023-09-10T00:50:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-09-10T00:50:23.416 INFO:teuthology.orchestra.run.smithi092.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 373 kB/s | 47 kB 00:00 2023-09-10T00:50:23.483 INFO:teuthology.orchestra.run.smithi092.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.3 MB/s | 200 kB 00:00 2023-09-10T00:50:23.533 INFO:teuthology.orchestra.run.smithi092.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-09-10T00:50:23.575 INFO:teuthology.orchestra.run.smithi092.stdout:(74/131): python3-jwt-1.6.1-2.el8.noarch.rpm 268 kB/s | 43 kB 00:00 2023-09-10T00:50:23.650 INFO:teuthology.orchestra.run.smithi092.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 412 kB/s | 48 kB 00:00 2023-09-10T00:50:23.666 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-09-10T00:50:23.692 INFO:teuthology.orchestra.run.smithi092.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 745 kB/s | 87 kB 00:00 2023-09-10T00:50:23.751 INFO:teuthology.orchestra.run.smithi092.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 426 kB/s | 43 kB 00:00 2023-09-10T00:50:23.876 INFO:teuthology.orchestra.run.smithi092.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2023-09-10T00:50:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-09-10T00:50:23.901 INFO:teuthology.orchestra.run.smithi092.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 522 kB/s | 109 kB 00:00 2023-09-10T00:50:24.001 INFO:teuthology.orchestra.run.smithi092.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.4 MB/s | 560 kB 00:00 2023-09-10T00:50:24.027 INFO:teuthology.orchestra.run.smithi092.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 149 kB/s | 19 kB 00:00 2023-09-10T00:50:24.104 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-09-10T00:50:24.161 INFO:teuthology.orchestra.run.smithi092.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 975 kB/s | 155 kB 00:00 2023-09-10T00:50:24.194 INFO:teuthology.orchestra.run.smithi092.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 3.3 MB/s | 559 kB 00:00 2023-09-10T00:50:24.320 INFO:teuthology.orchestra.run.smithi092.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 113 kB/s | 18 kB 00:00 2023-09-10T00:50:24.332 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-09-10T00:50:24.387 INFO:teuthology.orchestra.run.smithi092.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.1 MB/s | 605 kB 00:00 2023-09-10T00:50:24.512 INFO:teuthology.orchestra.run.smithi092.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.2 MB/s | 238 kB 00:00 2023-09-10T00:50:24.532 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-09-10T00:50:24.579 INFO:teuthology.orchestra.run.smithi092.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 12 MB/s | 2.3 MB 00:00 2023-09-10T00:50:24.646 INFO:teuthology.orchestra.run.smithi092.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 541 kB/s | 72 kB 00:00 2023-09-10T00:50:24.679 INFO:teuthology.orchestra.run.smithi092.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 826 kB/s | 83 kB 00:00 2023-09-10T00:50:24.738 INFO:teuthology.orchestra.run.smithi092.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 652 kB/s | 60 kB 00:00 2023-09-10T00:50:24.761 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-09-10T00:50:24.763 INFO:teuthology.orchestra.run.smithi092.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.1 MB/s | 98 kB 00:00 2023-09-10T00:50:24.880 INFO:teuthology.orchestra.run.smithi092.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 406 kB/s | 58 kB 00:00 2023-09-10T00:50:24.905 INFO:teuthology.orchestra.run.smithi092.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 272 kB/s | 39 kB 00:00 2023-09-10T00:50:24.949 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:18.0.0-6025.g9670cca5.el8. 26/134 2023-09-10T00:50:25.055 INFO:teuthology.orchestra.run.smithi092.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.1 MB/s | 171 kB 00:00 2023-09-10T00:50:25.089 INFO:teuthology.orchestra.run.smithi092.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.0 MB/s | 644 kB 00:00 2023-09-10T00:50:25.095 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el 27/134 2023-09-10T00:50:25.240 INFO:teuthology.orchestra.run.smithi092.stdout:(96/131): NetworkManager-initscripts-updown-1.3 750 kB/s | 138 kB 00:00 2023-09-10T00:50:25.473 INFO:teuthology.orchestra.run.smithi092.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 471 kB/s | 110 kB 00:00 2023-09-10T00:50:25.640 INFO:teuthology.orchestra.run.smithi092.stdout:(98/131): python3-scipy-1.0.0-21.module+el8.5.0 25 MB/s | 14 MB 00:00 2023-09-10T00:50:25.665 INFO:teuthology.orchestra.run.smithi092.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 282 kB/s | 54 kB 00:00 2023-09-10T00:50:25.783 INFO:teuthology.orchestra.run.smithi092.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-09-10T00:50:25.808 INFO:teuthology.orchestra.run.smithi092.stdout:(101/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-09-10T00:50:25.833 INFO:teuthology.orchestra.run.smithi092.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 772 kB/s | 39 kB 00:00 2023-09-10T00:50:25.867 INFO:teuthology.orchestra.run.smithi092.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-09-10T00:50:25.900 INFO:teuthology.orchestra.run.smithi092.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2023-09-10T00:50:25.925 INFO:teuthology.orchestra.run.smithi092.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-09-10T00:50:25.951 INFO:teuthology.orchestra.run.smithi092.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-09-10T00:50:25.984 INFO:teuthology.orchestra.run.smithi092.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-09-10T00:50:26.043 INFO:teuthology.orchestra.run.smithi092.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 455 kB/s | 27 kB 00:00 2023-09-10T00:50:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_6 28/134 2023-09-10T00:50:26.118 INFO:teuthology.orchestra.run.smithi092.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 29 MB/s | 4.8 MB 00:00 2023-09-10T00:50:26.144 INFO:teuthology.orchestra.run.smithi092.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 3.1 MB/s | 320 kB 00:00 2023-09-10T00:50:26.186 INFO:teuthology.orchestra.run.smithi092.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 653 kB/s | 44 kB 00:00 2023-09-10T00:50:26.294 INFO:teuthology.orchestra.run.smithi092.stdout:(112/131): python3-numpy-1.14.3-10.el8.x86_64.r 25 MB/s | 3.7 MB 00:00 2023-09-10T00:50:26.336 INFO:teuthology.orchestra.run.smithi092.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 3.0 MB/s | 457 kB 00:00 2023-09-10T00:50:26.361 INFO:teuthology.orchestra.run.smithi092.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 3.3 MB/s | 225 kB 00:00 2023-09-10T00:50:26.398 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-09-10T00:50:26.405 INFO:teuthology.orchestra.run.smithi092.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 930 kB/s | 64 kB 00:00 2023-09-10T00:50:26.437 INFO:teuthology.orchestra.run.smithi092.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.6 MB/s | 126 kB 00:00 2023-09-10T00:50:26.442 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-09-10T00:50:26.462 INFO:teuthology.orchestra.run.smithi092.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 352 kB/s | 20 kB 00:00 2023-09-10T00:50:26.488 INFO:teuthology.orchestra.run.smithi092.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-09-10T00:50:26.521 INFO:teuthology.orchestra.run.smithi092.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-09-10T00:50:26.571 INFO:teuthology.orchestra.run.smithi092.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2023-09-10T00:50:26.645 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-09-10T00:50:26.655 INFO:teuthology.orchestra.run.smithi092.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 29 MB/s | 4.9 MB 00:00 2023-09-10T00:50:26.705 INFO:teuthology.orchestra.run.smithi092.stdout:(122/131): thrift-0.13.0-2.el8.x86_64.rpm 512 kB/s | 1.7 MB 00:03 2023-09-10T00:50:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-09-10T00:50:26.731 INFO:teuthology.orchestra.run.smithi092.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 2.6 MB/s | 202 kB 00:00 2023-09-10T00:50:26.781 INFO:teuthology.orchestra.run.smithi092.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2023-09-10T00:50:26.796 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-09-10T00:50:26.848 INFO:teuthology.orchestra.run.smithi092.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-09-10T00:50:26.890 INFO:teuthology.orchestra.run.smithi092.stdout:(126/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-09-10T00:50:26.940 INFO:teuthology.orchestra.run.smithi092.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 16 MB/s | 1.5 MB 00:00 2023-09-10T00:50:27.033 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-09-10T00:50:27.050 INFO:teuthology.orchestra.run.smithi092.stdout:(128/131): postgresql-server-10.21-2.module+el8 16 MB/s | 5.1 MB 00:00 2023-09-10T00:50:27.132 INFO:teuthology.orchestra.run.smithi092.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 9.5 MB/s | 798 kB 00:00 2023-09-10T00:50:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-09-10T00:50:27.246 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-09-10T00:50:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-09-10T00:50:27.403 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-09-10T00:50:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-09-10T00:50:27.716 INFO:teuthology.orchestra.run.smithi092.stdout:(130/131): librbd1-18.0.0-6025.g9670cca5.el8.x8 4.8 MB/s | 3.7 MB 00:00 2023-09-10T00:50:28.099 INFO:teuthology.orchestra.run.smithi092.stdout:(131/131): librados2-18.0.0-6025.g9670cca5.el8. 3.1 MB/s | 3.7 MB 00:01 2023-09-10T00:50:28.102 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-09-10T00:50:28.102 INFO:teuthology.orchestra.run.smithi092.stdout:Total 14 MB/s | 208 MB 00:14 2023-09-10T00:50:28.152 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-09-10T00:50:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:50:28.868 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:50:28.955 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-09-10T00:50:29.037 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:50:29.074 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-09-10T00:50:29.075 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-09-10T00:50:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-09-10T00:50:29.201 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:50:29.387 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:50:29.497 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:50:29.501 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-09-10T00:50:29.687 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 9/134 2023-09-10T00:50:29.712 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-09-10T00:50:29.915 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-09-10T00:50:29.944 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-09-10T00:50:30.156 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 36/134 2023-09-10T00:50:30.160 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-09-10T00:50:30.191 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-09-10T00:50:30.195 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-09-10T00:50:30.379 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_6 37/134 2023-09-10T00:50:30.540 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-09-10T00:50:30.683 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-09-10T00:50:30.735 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-09-10T00:50:30.938 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-09-10T00:50:30.977 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-09-10T00:50:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 14/134 2023-09-10T00:50:31.161 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-09-10T00:50:31.249 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 14/134 2023-09-10T00:50:31.382 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-09-10T00:50:31.407 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 15/134 2023-09-10T00:50:31.511 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 15/134 2023-09-10T00:50:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-09-10T00:50:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-09-10T00:50:31.813 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-09-10T00:50:31.966 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-09-10T00:50:31.978 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-09-10T00:50:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-09-10T00:50:32.559 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-09-10T00:50:32.721 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-09-10T00:50:32.771 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-09-10T00:50:32.928 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-09-10T00:50:33.071 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-09-10T00:50:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-09-10T00:50:33.607 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-09-10T00:50:33.652 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-09-10T00:50:33.858 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-09-10T00:50:33.869 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-09-10T00:50:34.086 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-09-10T00:50:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-09-10T00:50:34.460 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-09-10T00:50:34.750 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-09-10T00:50:34.756 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-09-10T00:50:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-09-10T00:50:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-09-10T00:50:35.235 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-09-10T00:50:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-09-10T00:50:35.469 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-09-10T00:50:35.506 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-09-10T00:50:35.538 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-09-10T00:50:35.688 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-09-10T00:50:35.705 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-09-10T00:50:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-09-10T00:50:35.862 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-09-10T00:50:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-09-10T00:50:35.976 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-09-10T00:50:36.065 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-09-10T00:50:36.115 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-09-10T00:50:36.172 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-09-10T00:50:36.181 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-09-10T00:50:36.270 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-09-10T00:50:36.390 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-09-10T00:50:36.428 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:18.0.0-6025.g9670cca5.el8. 26/134 2023-09-10T00:50:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-09-10T00:50:36.592 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el 27/134 2023-09-10T00:50:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-09-10T00:50:36.780 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-09-10T00:50:36.840 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-09-10T00:50:36.918 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-09-10T00:50:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 5/134 2023-09-10T00:50:37.191 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-09-10T00:50:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-09-10T00:50:37.379 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 5/134 2023-09-10T00:50:37.539 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:50:37.582 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-09-10T00:50:37.656 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_6 28/134 2023-09-10T00:50:37.669 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:50:37.779 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-09-10T00:50:37.847 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:50:37.974 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:50:37.991 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-09-10T00:50:38.097 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-09-10T00:50:38.119 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:50:38.281 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-09-10T00:50:38.298 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-09-10T00:50:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-09-10T00:50:38.511 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-09-10T00:50:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-09-10T00:50:38.681 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-09-10T00:50:38.693 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-09-10T00:50:38.734 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-09-10T00:50:38.861 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-09-10T00:50:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-09-10T00:50:39.110 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-09-10T00:50:39.135 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-09-10T00:50:39.300 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-09-10T00:50:39.392 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-09-10T00:50:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-09-10T00:50:39.530 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 14/134 2023-09-10T00:50:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-09-10T00:50:39.637 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 14/134 2023-09-10T00:50:39.780 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 15/134 2023-09-10T00:50:39.919 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-09-10T00:50:39.963 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 15/134 2023-09-10T00:50:40.160 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-09-10T00:50:40.184 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-09-10T00:50:40.361 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-09-10T00:50:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-09-10T00:50:40.562 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-09-10T00:50:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-09-10T00:50:40.688 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-09-10T00:50:40.814 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-09-10T00:50:40.938 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-09-10T00:50:40.980 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-09-10T00:50:41.057 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-09-10T00:50:41.171 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-09-10T00:50:41.414 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-09-10T00:50:41.438 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-09-10T00:50:41.583 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-09-10T00:50:41.659 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-09-10T00:50:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-09-10T00:50:42.022 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-09-10T00:50:42.077 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 36/134 2023-09-10T00:50:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_6 37/134 2023-09-10T00:50:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-09-10T00:50:42.548 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-09-10T00:50:42.590 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-09-10T00:50:42.611 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-09-10T00:50:42.771 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-09-10T00:50:42.797 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-09-10T00:50:42.884 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-09-10T00:50:42.935 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-09-10T00:50:43.178 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-09-10T00:50:43.188 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-09-10T00:50:43.395 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-09-10T00:50:43.476 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-09-10T00:50:43.548 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-09-10T00:50:43.694 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-09-10T00:50:43.816 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-09-10T00:50:43.820 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-09-10T00:50:43.911 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-09-10T00:50:44.045 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-09-10T00:50:44.127 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-09-10T00:50:44.140 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-09-10T00:50:44.309 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-09-10T00:50:44.319 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:18.0.0-6025.g9670cca5.el8. 26/134 2023-09-10T00:50:44.432 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el 27/134 2023-09-10T00:50:44.560 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-09-10T00:50:44.659 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_6 28/134 2023-09-10T00:50:44.829 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-09-10T00:50:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-09-10T00:50:44.855 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-09-10T00:50:45.013 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 92/134 2023-09-10T00:50:45.087 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 92/134 2023-09-10T00:50:45.180 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-09-10T00:50:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 93/134 2023-09-10T00:50:45.378 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-09-10T00:50:45.572 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-09-10T00:50:45.789 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-09-10T00:50:45.976 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-09-10T00:50:46.187 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-09-10T00:50:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-09-10T00:50:46.588 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-09-10T00:50:46.812 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-09-10T00:50:47.094 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-09-10T00:50:47.343 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-09-10T00:50:47.552 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-09-10T00:50:47.616 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-09-10T00:50:47.752 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-09-10T00:50:47.942 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-09-10T00:50:47.977 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-09-10T00:50:48.032 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-09-10T00:50:48.239 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-09-10T00:50:48.270 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-09-10T00:50:48.475 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-09-10T00:50:48.483 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-09-10T00:50:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-09-10T00:50:48.690 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-09-10T00:50:48.768 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-09-10T00:50:48.864 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-09-10T00:50:49.019 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-09-10T00:50:49.068 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 36/134 2023-09-10T00:50:49.156 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:50:49.193 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_6 37/134 2023-09-10T00:50:49.196 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-09-10T00:50:49.335 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-09-10T00:50:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:50:49.387 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-09-10T00:50:49.479 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-09-10T00:50:49.579 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-09-10T00:50:49.669 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-09-10T00:50:49.741 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-09-10T00:50:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-09-10T00:50:49.958 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-09-10T00:50:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:50:50.110 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-09-10T00:50:50.182 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-09-10T00:50:50.257 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-09-10T00:50:50.258 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 95/134 2023-09-10T00:50:50.371 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-09-10T00:50:50.371 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 95/134 2023-09-10T00:50:50.437 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-09-10T00:50:50.444 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-09-10T00:50:50.495 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 96/134 2023-09-10T00:50:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-09-10T00:50:50.636 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-09-10T00:50:50.782 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-09-10T00:50:50.846 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-09-10T00:50:50.977 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-09-10T00:50:51.070 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-09-10T00:50:51.146 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-09-10T00:50:51.255 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-09-10T00:50:51.314 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-09-10T00:50:51.434 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-09-10T00:50:51.465 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-09-10T00:50:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-09-10T00:50:51.676 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-09-10T00:50:51.865 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-09-10T00:50:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-09-10T00:50:52.117 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-09-10T00:50:52.237 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-09-10T00:50:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-09-10T00:50:52.513 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-09-10T00:50:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-09-10T00:50:52.713 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-09-10T00:50:52.789 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-09-10T00:50:52.956 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-09-10T00:50:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-09-10T00:50:53.165 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-09-10T00:50:53.477 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-09-10T00:50:53.702 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-09-10T00:50:53.869 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-09-10T00:50:54.091 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-09-10T00:50:54.290 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-09-10T00:50:54.396 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-09-10T00:50:54.456 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-09-10T00:50:54.656 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-09-10T00:50:54.739 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-09-10T00:50:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-09-10T00:50:54.981 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-09-10T00:50:55.241 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-09-10T00:50:55.299 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-09-10T00:50:55.486 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-09-10T00:50:55.602 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-09-10T00:50:55.635 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-09-10T00:50:55.790 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-09-10T00:50:55.822 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-09-10T00:50:56.115 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-09-10T00:50:56.317 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-09-10T00:50:56.447 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-09-10T00:50:56.596 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-09-10T00:50:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-09-10T00:50:56.803 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-09-10T00:50:56.891 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 92/134 2023-09-10T00:50:56.986 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 92/134 2023-09-10T00:50:56.997 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-09-10T00:50:57.174 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-09-10T00:50:57.192 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 93/134 2023-09-10T00:50:57.371 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-09-10T00:50:57.507 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-09-10T00:50:57.721 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-09-10T00:50:58.014 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-09-10T00:50:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-09-10T00:50:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-09-10T00:50:58.676 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-09-10T00:50:58.825 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-09-10T00:50:59.025 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-09-10T00:51:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-09-10T00:51:00.388 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-09-10T00:51:00.650 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-09-10T00:51:00.933 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-09-10T00:51:01.242 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-09-10T00:51:01.436 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-09-10T00:51:02.222 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-09-10T00:51:02.384 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 92/134 2023-09-10T00:51:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:51:02.476 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 92/134 2023-09-10T00:51:02.599 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:51:02.642 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 93/134 2023-09-10T00:51:03.332 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:51:03.620 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 95/134 2023-09-10T00:51:03.796 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 95/134 2023-09-10T00:51:03.957 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 96/134 2023-09-10T00:51:06.387 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:51:06.593 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:51:07.182 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 94/134 2023-09-10T00:51:07.461 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 95/134 2023-09-10T00:51:07.547 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 95/134 2023-09-10T00:51:07.666 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 96/134 2023-09-10T00:51:09.292 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 96/134 2023-09-10T00:51:09.293 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2023-09-10T00:51:09.293 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2023-09-10T00:51:09.293 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2023-09-10T00:51:09.293 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /run 2023-09-10T00:51:09.293 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2023-09-10T00:51:09.294 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2023-09-10T00:51:09.294 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2023-09-10T00:51:09.294 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2023-09-10T00:51:09.294 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2023-09-10T00:51:09.294 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:51:09.666 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 97/134 2023-09-10T00:51:11.244 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 97/134 2023-09-10T00:51:11.437 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 98/134 2023-09-10T00:51:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 98/134 2023-09-10T00:51:11.882 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 99/134 2023-09-10T00:51:12.337 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 99/134 2023-09-10T00:51:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 100/134 2023-09-10T00:51:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 100/134 2023-09-10T00:51:14.257 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 101/134 2023-09-10T00:51:14.455 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 101/134 2023-09-10T00:51:14.621 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.e 102/134 2023-09-10T00:51:14.833 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5. 103/134 2023-09-10T00:51:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 104/134 2023-09-10T00:51:15.266 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 104/134 2023-09-10T00:51:15.544 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 105/134 2023-09-10T00:51:16.505 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 105/134 2023-09-10T00:51:16.723 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 106/134 2023-09-10T00:51:16.813 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 106/134 2023-09-10T00:51:17.144 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el 107/134 2023-09-10T00:51:17.390 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 108/134 2023-09-10T00:51:17.426 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 108/134 2023-09-10T00:51:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 109/134 2023-09-10T00:51:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 109/134 2023-09-10T00:51:18.953 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 110/134 2023-09-10T00:51:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 111/134 2023-09-10T00:51:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 111/134 2023-09-10T00:51:19.389 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6025.g9670c 112/134 2023-09-10T00:51:19.431 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6025.g9670c 112/134 2023-09-10T00:51:19.631 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 113/134 2023-09-10T00:51:19.990 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 113/134 2023-09-10T00:51:20.164 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 114/134 2023-09-10T00:51:24.796 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 114/134 2023-09-10T00:51:25.097 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 115/134 2023-09-10T00:51:25.262 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-09-10T00:51:25.452 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-09-10T00:51:25.817 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 96/134 2023-09-10T00:51:25.817 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2023-09-10T00:51:25.818 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2023-09-10T00:51:25.818 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-09-10T00:51:25.818 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /run 2023-09-10T00:51:25.818 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2023-09-10T00:51:25.819 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2023-09-10T00:51:25.819 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2023-09-10T00:51:25.819 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2023-09-10T00:51:25.819 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-09-10T00:51:25.819 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:51:26.221 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 97/134 2023-09-10T00:51:26.315 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-09-10T00:51:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 96/134 2023-09-10T00:51:26.438 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2023-09-10T00:51:26.439 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2023-09-10T00:51:26.439 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /dev 2023-09-10T00:51:26.439 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /run 2023-09-10T00:51:26.439 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2023-09-10T00:51:26.439 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2023-09-10T00:51:26.440 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2023-09-10T00:51:26.440 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2023-09-10T00:51:26.440 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /dev 2023-09-10T00:51:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:51:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-09-10T00:51:26.804 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 97/134 2023-09-10T00:51:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 119/134 2023-09-10T00:51:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-09-10T00:51:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_ 121/134 2023-09-10T00:51:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-09-10T00:51:28.089 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 97/134 2023-09-10T00:51:28.275 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 98/134 2023-09-10T00:51:28.389 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 97/134 2023-09-10T00:51:28.540 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 98/134 2023-09-10T00:51:28.578 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 98/134 2023-09-10T00:51:28.737 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 99/134 2023-09-10T00:51:28.763 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 98/134 2023-09-10T00:51:28.937 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 99/134 2023-09-10T00:51:29.308 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 99/134 2023-09-10T00:51:29.392 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 99/134 2023-09-10T00:51:29.478 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 100/134 2023-09-10T00:51:29.521 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 100/134 2023-09-10T00:51:29.536 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 100/134 2023-09-10T00:51:29.553 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 100/134 2023-09-10T00:51:29.582 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:29.820 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:29.839 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 124/134 2023-09-10T00:51:30.361 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 125/134 2023-09-10T00:51:30.681 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 126/134 2023-09-10T00:51:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-09-10T00:51:31.144 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-09-10T00:51:31.153 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-09-10T00:51:31.211 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 101/134 2023-09-10T00:51:31.361 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-09-10T00:51:31.409 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 101/134 2023-09-10T00:51:31.533 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.e 102/134 2023-09-10T00:51:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-09-10T00:51:31.625 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-09-10T00:51:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5. 103/134 2023-09-10T00:51:31.861 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-09-10T00:51:31.873 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-09-10T00:51:31.874 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:31.935 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 104/134 2023-09-10T00:51:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-09-10T00:51:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 104/134 2023-09-10T00:51:32.358 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-09-10T00:51:32.481 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 105/134 2023-09-10T00:51:32.493 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 101/134 2023-09-10T00:51:32.568 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:32.694 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 101/134 2023-09-10T00:51:32.802 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.e 102/134 2023-09-10T00:51:33.027 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5. 103/134 2023-09-10T00:51:33.115 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:33.116 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 1/134 2023-09-10T00:51:33.116 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 2/134 2023-09-10T00:51:33.116 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 3/134 2023-09-10T00:51:33.116 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 4/134 2023-09-10T00:51:33.117 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6025.g9670c 5/134 2023-09-10T00:51:33.117 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:51:33.117 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:51:33.117 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:51:33.117 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 9/134 2023-09-10T00:51:33.118 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 10/134 2023-09-10T00:51:33.118 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 11/134 2023-09-10T00:51:33.118 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 12/134 2023-09-10T00:51:33.118 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 13/134 2023-09-10T00:51:33.118 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_ 14/134 2023-09-10T00:51:33.119 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 15/134 2023-09-10T00:51:33.119 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 16/134 2023-09-10T00:51:33.119 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_6 17/134 2023-09-10T00:51:33.119 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 18/134 2023-09-10T00:51:33.120 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 19/134 2023-09-10T00:51:33.120 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el 20/134 2023-09-10T00:51:33.120 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:18.0.0-6025.g9670cca5.el8. 21/134 2023-09-10T00:51:33.120 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_6 22/134 2023-09-10T00:51:33.120 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 23/134 2023-09-10T00:51:33.120 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 24/134 2023-09-10T00:51:33.121 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 25/134 2023-09-10T00:51:33.121 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 26/134 2023-09-10T00:51:33.121 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 27/134 2023-09-10T00:51:33.121 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 28/134 2023-09-10T00:51:33.121 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5. 29/134 2023-09-10T00:51:33.122 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 30/134 2023-09-10T00:51:33.122 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 31/134 2023-09-10T00:51:33.122 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 32/134 2023-09-10T00:51:33.122 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el 33/134 2023-09-10T00:51:33.122 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 34/134 2023-09-10T00:51:33.123 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.e 35/134 2023-09-10T00:51:33.123 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 36/134 2023-09-10T00:51:33.123 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 37/134 2023-09-10T00:51:33.123 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 38/134 2023-09-10T00:51:33.123 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-09-10T00:51:33.124 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-09-10T00:51:33.124 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-09-10T00:51:33.124 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-09-10T00:51:33.124 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-09-10T00:51:33.124 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-09-10T00:51:33.125 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-09-10T00:51:33.125 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-09-10T00:51:33.125 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-09-10T00:51:33.125 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-09-10T00:51:33.125 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-09-10T00:51:33.126 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-09-10T00:51:33.126 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-09-10T00:51:33.126 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-09-10T00:51:33.126 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-09-10T00:51:33.126 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-09-10T00:51:33.127 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-09-10T00:51:33.127 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-09-10T00:51:33.127 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-09-10T00:51:33.127 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-09-10T00:51:33.127 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-09-10T00:51:33.128 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-09-10T00:51:33.128 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-09-10T00:51:33.128 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-09-10T00:51:33.128 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-09-10T00:51:33.128 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-09-10T00:51:33.129 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-09-10T00:51:33.129 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-09-10T00:51:33.129 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-09-10T00:51:33.129 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-09-10T00:51:33.129 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-09-10T00:51:33.130 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-09-10T00:51:33.130 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-09-10T00:51:33.130 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-09-10T00:51:33.130 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-09-10T00:51:33.131 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-09-10T00:51:33.131 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-09-10T00:51:33.131 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-09-10T00:51:33.131 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-09-10T00:51:33.131 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-09-10T00:51:33.131 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-09-10T00:51:33.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-09-10T00:51:33.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-09-10T00:51:33.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-09-10T00:51:33.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-09-10T00:51:33.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-09-10T00:51:33.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-09-10T00:51:33.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-09-10T00:51:33.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-09-10T00:51:33.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-09-10T00:51:33.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-09-10T00:51:33.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-09-10T00:51:33.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-09-10T00:51:33.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-09-10T00:51:33.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-09-10T00:51:33.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-09-10T00:51:33.135 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-09-10T00:51:33.135 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-09-10T00:51:33.135 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-09-10T00:51:33.135 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-09-10T00:51:33.135 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-09-10T00:51:33.136 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-09-10T00:51:33.136 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-09-10T00:51:33.136 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-09-10T00:51:33.137 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-09-10T00:51:33.137 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-09-10T00:51:33.137 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-09-10T00:51:33.137 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-09-10T00:51:33.137 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-09-10T00:51:33.138 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-09-10T00:51:33.138 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-09-10T00:51:33.138 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-09-10T00:51:33.139 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-09-10T00:51:33.139 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-09-10T00:51:33.139 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-09-10T00:51:33.140 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-09-10T00:51:33.140 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-09-10T00:51:33.140 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-09-10T00:51:33.140 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-09-10T00:51:33.141 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-09-10T00:51:33.141 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-09-10T00:51:33.141 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-09-10T00:51:33.142 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-09-10T00:51:33.142 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-09-10T00:51:33.142 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-09-10T00:51:33.143 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-09-10T00:51:33.143 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-09-10T00:51:33.143 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-09-10T00:51:33.144 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-09-10T00:51:33.144 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-09-10T00:51:33.144 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 129/134 2023-09-10T00:51:33.144 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-09-10T00:51:33.145 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 131/134 2023-09-10T00:51:33.145 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:33.145 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-09-10T00:51:33.194 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 104/134 2023-09-10T00:51:33.439 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 105/134 2023-09-10T00:51:33.472 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 104/134 2023-09-10T00:51:33.635 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 106/134 2023-09-10T00:51:33.690 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 105/134 2023-09-10T00:51:33.725 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 106/134 2023-09-10T00:51:34.071 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el 107/134 2023-09-10T00:51:34.277 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 108/134 2023-09-10T00:51:34.312 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 108/134 2023-09-10T00:51:34.510 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 109/134 2023-09-10T00:51:34.521 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 109/134 2023-09-10T00:51:34.873 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 105/134 2023-09-10T00:51:35.226 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 106/134 2023-09-10T00:51:35.362 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 106/134 2023-09-10T00:51:35.648 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:35.649 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:50 ago on Sun 10 Sep 2023 12:49:45 AM UTC. 2023-09-10T00:51:35.747 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el 107/134 2023-09-10T00:51:35.847 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 110/134 2023-09-10T00:51:36.016 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 108/134 2023-09-10T00:51:36.021 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 111/134 2023-09-10T00:51:36.055 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 111/134 2023-09-10T00:51:36.084 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 108/134 2023-09-10T00:51:36.201 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6025.g9670c 112/134 2023-09-10T00:51:36.244 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6025.g9670c 112/134 2023-09-10T00:51:36.310 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 109/134 2023-09-10T00:51:36.339 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 109/134 2023-09-10T00:51:36.418 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 113/134 2023-09-10T00:51:36.777 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 113/134 2023-09-10T00:51:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 114/134 2023-09-10T00:51:37.964 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 110/134 2023-09-10T00:51:38.151 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 111/134 2023-09-10T00:51:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 111/134 2023-09-10T00:51:38.234 INFO:teuthology.orchestra.run.smithi136.stdout:Installed products updated. 2023-09-10T00:51:38.385 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6025.g9670c 112/134 2023-09-10T00:51:38.447 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:51:38.447 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2023-09-10T00:51:38.447 INFO:teuthology.orchestra.run.smithi136.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-09-10T00:51:38.448 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.448 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.448 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2023-09-10T00:51:38.448 INFO:teuthology.orchestra.run.smithi136.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-09-10T00:51:38.448 INFO:teuthology.orchestra.run.smithi136.stdout: automake-1.16.1-7.el8.noarch 2023-09-10T00:51:38.449 INFO:teuthology.orchestra.run.smithi136.stdout: bison-3.0.4-10.el8.x86_64 2023-09-10T00:51:38.449 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.449 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.449 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.449 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.450 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.450 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.450 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.450 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.450 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.452 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.452 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.452 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.452 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.452 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.453 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.453 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.453 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.453 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:38.453 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-09-10T00:51:38.454 INFO:teuthology.orchestra.run.smithi136.stdout: flex-2.6.1-9.el8.x86_64 2023-09-10T00:51:38.454 INFO:teuthology.orchestra.run.smithi136.stdout: fuse-2.9.7-15.el8.x86_64 2023-09-10T00:51:38.454 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-10T00:51:38.454 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-3.el8.x86_64 2023-09-10T00:51:38.454 INFO:teuthology.orchestra.run.smithi136.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-09-10T00:51:38.455 INFO:teuthology.orchestra.run.smithi136.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-09-10T00:51:38.455 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-09-10T00:51:38.455 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.455 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.455 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.456 INFO:teuthology.orchestra.run.smithi136.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-09-10T00:51:38.456 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-09-10T00:51:38.456 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-10T00:51:38.456 INFO:teuthology.orchestra.run.smithi136.stdout: libpq-13.5-1.el8.x86_64 2023-09-10T00:51:38.456 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-09-10T00:51:38.456 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-10T00:51:38.457 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.457 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.457 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-10T00:51:38.457 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.457 INFO:teuthology.orchestra.run.smithi136.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-09-10T00:51:38.458 INFO:teuthology.orchestra.run.smithi136.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-09-10T00:51:38.458 INFO:teuthology.orchestra.run.smithi136.stdout: libtool-2.4.6-25.el8.x86_64 2023-09-10T00:51:38.458 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-10T00:51:38.458 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-09-10T00:51:38.458 INFO:teuthology.orchestra.run.smithi136.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-09-10T00:51:38.459 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-10T00:51:38.459 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-10T00:51:38.459 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.15-3.el8.x86_64 2023-09-10T00:51:38.459 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-09-10T00:51:38.459 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-09-10T00:51:38.460 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-09-10T00:51:38.460 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-09-10T00:51:38.460 INFO:teuthology.orchestra.run.smithi136.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-09-10T00:51:38.460 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:51:38.460 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:51:38.461 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:51:38.461 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-10T00:51:38.461 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-10T00:51:38.461 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-10T00:51:38.461 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-10T00:51:38.462 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-10T00:51:38.462 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.462 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.462 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.462 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-10T00:51:38.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-09-10T00:51:38.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-10T00:51:38.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-10T00:51:38.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-09-10T00:51:38.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-10T00:51:38.463 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-10T00:51:38.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-10T00:51:38.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-10T00:51:38.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-10T00:51:38.464 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-10T00:51:38.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-10T00:51:38.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-10T00:51:38.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-10T00:51:38.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-10T00:51:38.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-10T00:51:38.465 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-10T00:51:38.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-10T00:51:38.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-10T00:51:38.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-10T00:51:38.466 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-10T00:51:38.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-10T00:51:38.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-2.6-1.el8.noarch 2023-09-10T00:51:38.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-10T00:51:38.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-10T00:51:38.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-10T00:51:38.467 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-10T00:51:38.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-10T00:51:38.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-10T00:51:38.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.468 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-10T00:51:38.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-10T00:51:38.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-10T00:51:38.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-10T00:51:38.469 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-09-10T00:51:38.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-10T00:51:38.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-10T00:51:38.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-10T00:51:38.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-10T00:51:38.470 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-10T00:51:38.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-10T00:51:38.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-10T00:51:38.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-10T00:51:38.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-10T00:51:38.471 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-10T00:51:38.472 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.472 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.472 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:38.472 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-10T00:51:38.472 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-10T00:51:38.473 INFO:teuthology.orchestra.run.smithi136.stdout: uuid-1.6.2-43.el8.x86_64 2023-09-10T00:51:38.473 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-09-10T00:51:38.473 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-10T00:51:38.474 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6025.g9670c 112/134 2023-09-10T00:51:38.474 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-09-10T00:51:38.474 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:51:38.475 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2023-09-10T00:51:38.602 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 113/134 2023-09-10T00:51:38.650 DEBUG:teuthology.parallel:result is None 2023-09-10T00:51:39.050 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 113/134 2023-09-10T00:51:39.360 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 114/134 2023-09-10T00:51:41.592 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 114/134 2023-09-10T00:51:42.450 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 115/134 2023-09-10T00:51:42.782 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-09-10T00:51:42.944 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 117/134 2023-09-10T00:51:43.196 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-09-10T00:51:43.391 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-09-10T00:51:43.582 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-09-10T00:51:43.829 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-09-10T00:51:44.012 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_ 121/134 2023-09-10T00:51:44.175 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-09-10T00:51:44.798 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 114/134 2023-09-10T00:51:45.091 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 115/134 2023-09-10T00:51:45.267 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-09-10T00:51:45.534 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-09-10T00:51:46.339 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:46.615 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:46.634 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:46.799 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 124/134 2023-09-10T00:51:46.839 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-09-10T00:51:47.065 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 125/134 2023-09-10T00:51:47.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-09-10T00:51:47.395 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 126/134 2023-09-10T00:51:47.482 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 119/134 2023-09-10T00:51:47.697 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-09-10T00:51:47.781 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-09-10T00:51:47.889 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_ 121/134 2023-09-10T00:51:47.990 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-09-10T00:51:48.026 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-09-10T00:51:48.076 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-09-10T00:51:48.290 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-09-10T00:51:48.303 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-09-10T00:51:48.473 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-09-10T00:51:48.482 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-09-10T00:51:48.609 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-09-10T00:51:48.609 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:48.785 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:48.785 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-09-10T00:51:49.012 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-09-10T00:51:49.231 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:49.771 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:49.772 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 1/134 2023-09-10T00:51:49.772 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 2/134 2023-09-10T00:51:49.772 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 3/134 2023-09-10T00:51:49.772 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 4/134 2023-09-10T00:51:49.773 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6025.g9670c 5/134 2023-09-10T00:51:49.773 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:51:49.773 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:51:49.773 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:51:49.773 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 9/134 2023-09-10T00:51:49.774 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 10/134 2023-09-10T00:51:49.774 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 11/134 2023-09-10T00:51:49.774 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 12/134 2023-09-10T00:51:49.774 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 13/134 2023-09-10T00:51:49.774 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_ 14/134 2023-09-10T00:51:49.775 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 15/134 2023-09-10T00:51:49.775 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 16/134 2023-09-10T00:51:49.775 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_6 17/134 2023-09-10T00:51:49.775 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 18/134 2023-09-10T00:51:49.776 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 19/134 2023-09-10T00:51:49.776 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el 20/134 2023-09-10T00:51:49.776 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:18.0.0-6025.g9670cca5.el8. 21/134 2023-09-10T00:51:49.776 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_6 22/134 2023-09-10T00:51:49.776 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 23/134 2023-09-10T00:51:49.777 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 24/134 2023-09-10T00:51:49.777 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 25/134 2023-09-10T00:51:49.777 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 26/134 2023-09-10T00:51:49.777 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 27/134 2023-09-10T00:51:49.777 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 28/134 2023-09-10T00:51:49.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5. 29/134 2023-09-10T00:51:49.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 30/134 2023-09-10T00:51:49.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 31/134 2023-09-10T00:51:49.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 32/134 2023-09-10T00:51:49.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el 33/134 2023-09-10T00:51:49.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 34/134 2023-09-10T00:51:49.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.e 35/134 2023-09-10T00:51:49.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 36/134 2023-09-10T00:51:49.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 37/134 2023-09-10T00:51:49.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 38/134 2023-09-10T00:51:49.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-09-10T00:51:49.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-09-10T00:51:49.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-09-10T00:51:49.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-09-10T00:51:49.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-09-10T00:51:49.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-09-10T00:51:49.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-09-10T00:51:49.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-09-10T00:51:49.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-09-10T00:51:49.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-09-10T00:51:49.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-09-10T00:51:49.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-09-10T00:51:49.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-09-10T00:51:49.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-09-10T00:51:49.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-09-10T00:51:49.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-09-10T00:51:49.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-09-10T00:51:49.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-09-10T00:51:49.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-09-10T00:51:49.793 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-09-10T00:51:49.793 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-09-10T00:51:49.793 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-09-10T00:51:49.793 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-09-10T00:51:49.793 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-09-10T00:51:49.793 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-09-10T00:51:49.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-09-10T00:51:49.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-09-10T00:51:49.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-09-10T00:51:49.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-09-10T00:51:49.795 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-09-10T00:51:49.795 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-09-10T00:51:49.795 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-09-10T00:51:49.795 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-09-10T00:51:49.795 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-09-10T00:51:49.796 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-09-10T00:51:49.796 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-09-10T00:51:49.796 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-09-10T00:51:49.796 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-09-10T00:51:49.796 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-09-10T00:51:49.796 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-09-10T00:51:49.797 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-09-10T00:51:49.797 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-09-10T00:51:49.797 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-09-10T00:51:49.797 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-09-10T00:51:49.797 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-09-10T00:51:49.798 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-09-10T00:51:49.798 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-09-10T00:51:49.798 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-09-10T00:51:49.798 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-09-10T00:51:49.798 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-09-10T00:51:49.799 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-09-10T00:51:49.799 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-09-10T00:51:49.799 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-09-10T00:51:49.799 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-09-10T00:51:49.799 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-09-10T00:51:49.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-09-10T00:51:49.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-09-10T00:51:49.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-09-10T00:51:49.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-09-10T00:51:49.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-09-10T00:51:49.800 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-09-10T00:51:49.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-09-10T00:51:49.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-09-10T00:51:49.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-09-10T00:51:49.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-09-10T00:51:49.801 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-09-10T00:51:49.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-09-10T00:51:49.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-09-10T00:51:49.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-09-10T00:51:49.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-09-10T00:51:49.802 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-09-10T00:51:49.803 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-09-10T00:51:49.803 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-09-10T00:51:49.803 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-09-10T00:51:49.803 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-09-10T00:51:49.803 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-09-10T00:51:49.804 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-09-10T00:51:49.804 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-09-10T00:51:49.804 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-09-10T00:51:49.804 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-09-10T00:51:49.804 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-09-10T00:51:49.805 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-09-10T00:51:49.805 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-09-10T00:51:49.805 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-09-10T00:51:49.805 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-09-10T00:51:49.805 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-09-10T00:51:49.806 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-09-10T00:51:49.806 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-09-10T00:51:49.806 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-09-10T00:51:49.806 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-09-10T00:51:49.806 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 129/134 2023-09-10T00:51:49.807 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-09-10T00:51:49.807 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 131/134 2023-09-10T00:51:49.807 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:49.807 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-09-10T00:51:51.526 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-09-10T00:51:52.109 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 124/134 2023-09-10T00:51:52.248 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:52.248 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:51 ago on Sun 10 Sep 2023 12:50:01 AM UTC. 2023-09-10T00:51:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 125/134 2023-09-10T00:51:52.851 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 126/134 2023-09-10T00:51:53.144 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-09-10T00:51:53.308 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-09-10T00:51:53.331 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-09-10T00:51:53.565 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-09-10T00:51:53.782 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-09-10T00:51:53.897 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-09-10T00:51:54.124 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-09-10T00:51:54.153 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-09-10T00:51:54.154 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:54.364 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:54.365 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-09-10T00:51:54.598 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-09-10T00:51:54.816 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:55.079 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:55.079 INFO:teuthology.orchestra.run.smithi003.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-09-10T00:51:55.079 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:51:55.105 INFO:teuthology.orchestra.run.smithi092.stdout:Installed products updated. 2023-09-10T00:51:55.318 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:51:55.318 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2023-09-10T00:51:55.319 INFO:teuthology.orchestra.run.smithi092.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-09-10T00:51:55.319 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.319 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.319 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-09-10T00:51:55.320 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-09-10T00:51:55.320 INFO:teuthology.orchestra.run.smithi092.stdout: automake-1.16.1-7.el8.noarch 2023-09-10T00:51:55.320 INFO:teuthology.orchestra.run.smithi092.stdout: bison-3.0.4-10.el8.x86_64 2023-09-10T00:51:55.320 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.321 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.321 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.321 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.322 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.322 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.322 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.322 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.323 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.323 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.323 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.323 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.324 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.324 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.324 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.325 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.325 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.326 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.326 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.326 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.327 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.327 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.327 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:51:55.327 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-09-10T00:51:55.328 INFO:teuthology.orchestra.run.smithi092.stdout: flex-2.6.1-9.el8.x86_64 2023-09-10T00:51:55.328 INFO:teuthology.orchestra.run.smithi092.stdout: fuse-2.9.7-15.el8.x86_64 2023-09-10T00:51:55.328 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-10T00:51:55.329 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-3.el8.x86_64 2023-09-10T00:51:55.329 INFO:teuthology.orchestra.run.smithi092.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-09-10T00:51:55.329 INFO:teuthology.orchestra.run.smithi092.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-09-10T00:51:55.330 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-09-10T00:51:55.330 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.330 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.330 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.331 INFO:teuthology.orchestra.run.smithi092.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-09-10T00:51:55.331 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-09-10T00:51:55.331 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-10T00:51:55.331 INFO:teuthology.orchestra.run.smithi092.stdout: libpq-13.5-1.el8.x86_64 2023-09-10T00:51:55.332 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-09-10T00:51:55.332 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-10T00:51:55.332 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.332 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.333 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-10T00:51:55.333 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.333 INFO:teuthology.orchestra.run.smithi092.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-09-10T00:51:55.333 INFO:teuthology.orchestra.run.smithi092.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-09-10T00:51:55.335 INFO:teuthology.orchestra.run.smithi092.stdout: libtool-2.4.6-25.el8.x86_64 2023-09-10T00:51:55.335 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-10T00:51:55.335 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-09-10T00:51:55.336 INFO:teuthology.orchestra.run.smithi092.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-09-10T00:51:55.336 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-10T00:51:55.337 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-10T00:51:55.337 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.15-3.el8.x86_64 2023-09-10T00:51:55.337 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-09-10T00:51:55.337 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-09-10T00:51:55.338 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-09-10T00:51:55.338 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-09-10T00:51:55.338 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-09-10T00:51:55.338 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:51:55.339 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:51:55.339 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:51:55.339 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-10T00:51:55.339 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-10T00:51:55.340 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-10T00:51:55.340 INFO:teuthology.orchestra.run.smithi092.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-10T00:51:55.340 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-10T00:51:55.340 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.340 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.341 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.341 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-10T00:51:55.341 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-09-10T00:51:55.341 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-10T00:51:55.342 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-10T00:51:55.342 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-09-10T00:51:55.342 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-10T00:51:55.343 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-10T00:51:55.343 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-10T00:51:55.343 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-10T00:51:55.343 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-10T00:51:55.344 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-10T00:51:55.344 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-10T00:51:55.344 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-10T00:51:55.344 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-10T00:51:55.345 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-10T00:51:55.345 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-10T00:51:55.345 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-10T00:51:55.345 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-10T00:51:55.346 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-10T00:51:55.346 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-10T00:51:55.346 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-10T00:51:55.346 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-10T00:51:55.347 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-2.6-1.el8.noarch 2023-09-10T00:51:55.347 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-10T00:51:55.347 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-10T00:51:55.347 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-10T00:51:55.348 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-10T00:51:55.348 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-10T00:51:55.348 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-10T00:51:55.348 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.348 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.349 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-10T00:51:55.349 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-10T00:51:55.349 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.349 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-10T00:51:55.349 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-10T00:51:55.350 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-09-10T00:51:55.350 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-10T00:51:55.350 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-10T00:51:55.350 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-10T00:51:55.350 INFO:teuthology.orchestra.run.smithi092.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-10T00:51:55.351 INFO:teuthology.orchestra.run.smithi092.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-10T00:51:55.351 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-10T00:51:55.351 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-10T00:51:55.352 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-10T00:51:55.352 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-10T00:51:55.352 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-10T00:51:55.352 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.353 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.353 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:51:55.353 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-10T00:51:55.354 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-10T00:51:55.354 INFO:teuthology.orchestra.run.smithi092.stdout: uuid-1.6.2-43.el8.x86_64 2023-09-10T00:51:55.354 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-09-10T00:51:55.355 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-10T00:51:55.355 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-09-10T00:51:55.356 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-09-10T00:51:55.356 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-09-10T00:51:55.511 DEBUG:teuthology.parallel:result is None 2023-09-10T00:51:55.700 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 1/134 2023-09-10T00:51:55.700 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 2/134 2023-09-10T00:51:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 3/134 2023-09-10T00:51:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 4/134 2023-09-10T00:51:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6025.g9670c 5/134 2023-09-10T00:51:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 6/134 2023-09-10T00:51:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 7/134 2023-09-10T00:51:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 8/134 2023-09-10T00:51:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 9/134 2023-09-10T00:51:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 10/134 2023-09-10T00:51:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 11/134 2023-09-10T00:51:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 12/134 2023-09-10T00:51:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 13/134 2023-09-10T00:51:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_ 14/134 2023-09-10T00:51:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 15/134 2023-09-10T00:51:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 16/134 2023-09-10T00:51:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_6 17/134 2023-09-10T00:51:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86 18/134 2023-09-10T00:51:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 19/134 2023-09-10T00:51:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el 20/134 2023-09-10T00:51:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:18.0.0-6025.g9670cca5.el8. 21/134 2023-09-10T00:51:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_6 22/134 2023-09-10T00:51:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 23/134 2023-09-10T00:51:55.705 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 24/134 2023-09-10T00:51:55.705 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 25/134 2023-09-10T00:51:55.705 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 26/134 2023-09-10T00:51:55.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 27/134 2023-09-10T00:51:55.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 28/134 2023-09-10T00:51:55.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5. 29/134 2023-09-10T00:51:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noa 30/134 2023-09-10T00:51:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.n 31/134 2023-09-10T00:51:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6025.g967 32/134 2023-09-10T00:51:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el 33/134 2023-09-10T00:51:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 34/134 2023-09-10T00:51:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.e 35/134 2023-09-10T00:51:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 36/134 2023-09-10T00:51:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 37/134 2023-09-10T00:51:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 38/134 2023-09-10T00:51:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-09-10T00:51:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-09-10T00:51:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-09-10T00:51:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-09-10T00:51:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-09-10T00:51:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-09-10T00:51:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-09-10T00:51:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-09-10T00:51:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-09-10T00:51:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-09-10T00:51:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-09-10T00:51:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-09-10T00:51:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-09-10T00:51:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-09-10T00:51:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-09-10T00:51:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-09-10T00:51:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-09-10T00:51:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-09-10T00:51:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-09-10T00:51:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-09-10T00:51:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-09-10T00:51:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-09-10T00:51:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-09-10T00:51:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-09-10T00:51:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-09-10T00:51:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-09-10T00:51:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-09-10T00:51:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-09-10T00:51:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-09-10T00:51:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-09-10T00:51:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-09-10T00:51:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-09-10T00:51:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-09-10T00:51:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-09-10T00:51:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-09-10T00:51:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-09-10T00:51:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-09-10T00:51:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-09-10T00:51:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-09-10T00:51:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-09-10T00:51:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-09-10T00:51:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-09-10T00:51:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-09-10T00:51:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-09-10T00:51:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-09-10T00:51:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-09-10T00:51:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-09-10T00:51:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-09-10T00:51:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-09-10T00:51:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-09-10T00:51:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-09-10T00:51:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-09-10T00:51:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-09-10T00:51:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-09-10T00:51:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-09-10T00:51:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-09-10T00:51:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-09-10T00:51:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-09-10T00:51:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-09-10T00:51:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-09-10T00:51:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-09-10T00:51:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-09-10T00:51:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-09-10T00:51:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-09-10T00:51:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-09-10T00:51:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-09-10T00:51:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-09-10T00:51:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-09-10T00:51:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-09-10T00:51:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-09-10T00:51:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-09-10T00:51:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-09-10T00:51:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-09-10T00:51:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-09-10T00:51:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-09-10T00:51:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-09-10T00:51:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-09-10T00:51:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-09-10T00:51:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-09-10T00:51:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-09-10T00:51:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-09-10T00:51:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-09-10T00:51:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-09-10T00:51:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-09-10T00:51:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-09-10T00:51:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-09-10T00:51:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-09-10T00:51:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-09-10T00:51:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-09-10T00:51:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-09-10T00:51:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 129/134 2023-09-10T00:51:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-09-10T00:51:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 131/134 2023-09-10T00:51:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-09-10T00:51:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-09-10T00:51:58.609 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-09-10T00:51:58.609 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:11 ago on Sun 10 Sep 2023 12:49:47 AM UTC. 2023-09-10T00:52:01.489 INFO:teuthology.orchestra.run.smithi003.stdout:Installed products updated. 2023-09-10T00:52:01.732 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:52:01.733 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2023-09-10T00:52:01.733 INFO:teuthology.orchestra.run.smithi003.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-09-10T00:52:01.733 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.733 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.734 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-09-10T00:52:01.734 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-09-10T00:52:01.734 INFO:teuthology.orchestra.run.smithi003.stdout: automake-1.16.1-7.el8.noarch 2023-09-10T00:52:01.734 INFO:teuthology.orchestra.run.smithi003.stdout: bison-3.0.4-10.el8.x86_64 2023-09-10T00:52:01.734 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.735 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.735 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.735 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.735 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.735 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.736 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.736 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.736 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.736 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.736 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.737 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.737 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.737 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.737 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.737 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.738 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.738 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.738 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.738 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.738 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.739 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.739 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top-2:18.0.0-6025.g9670cca5.el8.noarch 2023-09-10T00:52:01.739 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-09-10T00:52:01.739 INFO:teuthology.orchestra.run.smithi003.stdout: flex-2.6.1-9.el8.x86_64 2023-09-10T00:52:01.739 INFO:teuthology.orchestra.run.smithi003.stdout: fuse-2.9.7-15.el8.x86_64 2023-09-10T00:52:01.740 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-10T00:52:01.740 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-3.el8.x86_64 2023-09-10T00:52:01.740 INFO:teuthology.orchestra.run.smithi003.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-09-10T00:52:01.740 INFO:teuthology.orchestra.run.smithi003.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-09-10T00:52:01.740 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-09-10T00:52:01.741 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.741 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.741 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.741 INFO:teuthology.orchestra.run.smithi003.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-09-10T00:52:01.741 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-09-10T00:52:01.741 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-10T00:52:01.742 INFO:teuthology.orchestra.run.smithi003.stdout: libpq-13.5-1.el8.x86_64 2023-09-10T00:52:01.742 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-09-10T00:52:01.742 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-10T00:52:01.742 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.743 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.743 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-10T00:52:01.743 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.743 INFO:teuthology.orchestra.run.smithi003.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-09-10T00:52:01.743 INFO:teuthology.orchestra.run.smithi003.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-09-10T00:52:01.743 INFO:teuthology.orchestra.run.smithi003.stdout: libtool-2.4.6-25.el8.x86_64 2023-09-10T00:52:01.744 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-10T00:52:01.744 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-09-10T00:52:01.744 INFO:teuthology.orchestra.run.smithi003.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-09-10T00:52:01.744 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-10T00:52:01.744 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-10T00:52:01.745 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.15-3.el8.x86_64 2023-09-10T00:52:01.745 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-09-10T00:52:01.745 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-09-10T00:52:01.745 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-09-10T00:52:01.745 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-09-10T00:52:01.746 INFO:teuthology.orchestra.run.smithi003.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-09-10T00:52:01.746 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:52:01.746 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:52:01.746 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-09-10T00:52:01.746 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-10T00:52:01.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-10T00:52:01.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-10T00:52:01.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-10T00:52:01.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-10T00:52:01.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.748 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.748 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-10T00:52:01.748 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-09-10T00:52:01.748 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-10T00:52:01.748 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-10T00:52:01.749 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-09-10T00:52:01.749 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-10T00:52:01.749 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-10T00:52:01.749 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-10T00:52:01.749 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-10T00:52:01.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-10T00:52:01.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-10T00:52:01.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-10T00:52:01.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-10T00:52:01.750 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-10T00:52:01.751 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-10T00:52:01.751 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-10T00:52:01.751 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-10T00:52:01.751 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-10T00:52:01.751 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-10T00:52:01.752 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-10T00:52:01.752 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-10T00:52:01.752 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-10T00:52:01.752 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-2.6-1.el8.noarch 2023-09-10T00:52:01.752 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-10T00:52:01.753 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-10T00:52:01.753 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-10T00:52:01.753 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-10T00:52:01.754 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-10T00:52:01.754 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-10T00:52:01.754 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.754 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.755 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-10T00:52:01.755 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-10T00:52:01.755 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.755 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-10T00:52:01.755 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-10T00:52:01.756 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-09-10T00:52:01.756 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-10T00:52:01.756 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-10T00:52:01.756 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-10T00:52:01.756 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-10T00:52:01.757 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-10T00:52:01.757 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-10T00:52:01.757 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-10T00:52:01.757 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-10T00:52:01.757 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-10T00:52:01.757 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-10T00:52:01.758 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.758 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.758 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:18.0.0-6025.g9670cca5.el8.x86_64 2023-09-10T00:52:01.758 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-10T00:52:01.759 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-10T00:52:01.759 INFO:teuthology.orchestra.run.smithi003.stdout: uuid-1.6.2-43.el8.x86_64 2023-09-10T00:52:01.759 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-09-10T00:52:01.759 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-10T00:52:01.760 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-09-10T00:52:01.760 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:52:01.760 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-09-10T00:52:02.047 DEBUG:teuthology.parallel:result is None 2023-09-10T00:52:02.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:52:02.385 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-10T00:52:02.440 INFO:teuthology.orchestra.run.smithi003.stdout:18.0.0-6025.g9670cca5.el8 2023-09-10T00:52:02.441 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6025.g9670cca5.el8 2023-09-10T00:52:02.441 INFO:teuthology.task.install:The correct ceph version 18.0.0-6025.g9670cca5 is installed. 2023-09-10T00:52:02.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:52:02.585 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-10T00:52:02.613 INFO:teuthology.orchestra.run.smithi092.stdout:18.0.0-6025.g9670cca5.el8 2023-09-10T00:52:02.614 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6025.g9670cca5.el8 2023-09-10T00:52:02.614 INFO:teuthology.task.install:The correct ceph version 18.0.0-6025.g9670cca5 is installed. 2023-09-10T00:52:02.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:52:02.875 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-10T00:52:02.904 INFO:teuthology.orchestra.run.smithi136.stdout:18.0.0-6025.g9670cca5.el8 2023-09-10T00:52:02.904 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6025.g9670cca5.el8 2023-09-10T00:52:02.905 INFO:teuthology.task.install:The correct ceph version 18.0.0-6025.g9670cca5 is installed. 2023-09-10T00:52:02.905 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-10T00:52:02.906 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:52:02.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-10T00:52:03.000 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:52:03.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-10T00:52:03.047 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:52:03.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-10T00:52:03.094 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-10T00:52:03.095 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:52:03.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2023-09-10T00:52:03.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-10T00:52:03.278 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:52:03.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2023-09-10T00:52:03.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-10T00:52:03.378 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:52:03.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2023-09-10T00:52:03.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-10T00:52:03.481 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-10T00:52:03.481 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:52:03.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-10T00:52:03.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-10T00:52:03.659 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:52:03.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-10T00:52:03.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-10T00:52:03.764 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:52:03.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-10T00:52:03.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-10T00:52:03.867 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-10T00:52:03.868 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:52:03.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2023-09-10T00:52:03.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-10T00:52:04.044 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:52:04.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2023-09-10T00:52:04.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-10T00:52:04.149 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:52:04.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2023-09-10T00:52:04.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-10T00:52:04.251 INFO:teuthology.run_tasks:Running task cephadm... 2023-09-10T00:52:04.349 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '9670cca5bc8e17916865dc84939edfc10b1f2c00', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-09-10T00:52:04.349 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:52:04.350 INFO:tasks.cephadm:Cluster fsid is 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:52:04.350 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-09-10T00:52:04.351 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.3', 'mon.b': '172.21.15.92', 'mon.c': '172.21.15.136'} 2023-09-10T00:52:04.352 INFO:tasks.cephadm:First mon is mon.a on smithi003 2023-09-10T00:52:04.352 INFO:tasks.cephadm:First mgr is x 2023-09-10T00:52:04.353 INFO:tasks.cephadm:Normalizing hostnames... 2023-09-10T00:52:04.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2023-09-10T00:52:04.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2023-09-10T00:52:04.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2023-09-10T00:52:04.496 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-09-10T00:52:04.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:52:04.714 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9670cca5bc8e17916865dc84939edfc10b1f2c00', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/72502/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-6025-g9670cca5', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-6025.g9670cca5'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-09-09 16:32:07.237982', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20230909.145638', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2023-09-10T00:52:04.851 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-batrick-testing-20230909.145638, sha1 9670cca5bc8e17916865dc84939edfc10b1f2c00 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T00:52:04.851 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/x86_64/flavors/default/cephadm 2023-09-10T00:52:04.852 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:05.461 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-rw-r--. 1 ubuntu ubuntu 724179 Sep 10 00:52 /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:05.461 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:05.810 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-rw-r--. 1 ubuntu ubuntu 724179 Sep 10 00:52 /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:05.810 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20230909.145638/9670cca5bc8e17916865dc84939edfc10b1f2c00/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:06.135 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-rw-r--. 1 ubuntu ubuntu 724179 Sep 10 00:52 /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:06.136 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:06.187 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:06.208 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-09-10T00:52:06.267 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 on all hosts... 2023-09-10T00:52:06.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 pull 2023-09-10T00:52:06.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 pull 2023-09-10T00:52:06.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 pull 2023-09-10T00:52:07.594 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00... 2023-09-10T00:52:10.840 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00... 2023-09-10T00:52:11.357 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00... 2023-09-10T00:52:36.787 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2023-09-10T00:52:36.787 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev)", 2023-09-10T00:52:36.788 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "20c3861ef6c981f72050dd1dd3edbdc62747039bda543319e240a6e02f850eea", 2023-09-10T00:52:36.788 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2023-09-10T00:52:36.788 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8bb87e43f4b5fd23ed47b24bb48a71d3a37a4d237bf8cf5ca0615963ecee90da" 2023-09-10T00:52:36.788 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2023-09-10T00:52:36.788 INFO:teuthology.orchestra.run.smithi136.stdout:} 2023-09-10T00:52:40.742 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2023-09-10T00:52:40.742 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev)", 2023-09-10T00:52:40.742 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "20c3861ef6c981f72050dd1dd3edbdc62747039bda543319e240a6e02f850eea", 2023-09-10T00:52:40.743 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2023-09-10T00:52:40.743 INFO:teuthology.orchestra.run.smithi092.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8bb87e43f4b5fd23ed47b24bb48a71d3a37a4d237bf8cf5ca0615963ecee90da" 2023-09-10T00:52:40.743 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2023-09-10T00:52:40.743 INFO:teuthology.orchestra.run.smithi092.stdout:} 2023-09-10T00:52:42.613 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T00:52:42.613 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev)", 2023-09-10T00:52:42.614 INFO:teuthology.orchestra.run.smithi003.stdout: "image_id": "20c3861ef6c981f72050dd1dd3edbdc62747039bda543319e240a6e02f850eea", 2023-09-10T00:52:42.614 INFO:teuthology.orchestra.run.smithi003.stdout: "repo_digests": [ 2023-09-10T00:52:42.614 INFO:teuthology.orchestra.run.smithi003.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8bb87e43f4b5fd23ed47b24bb48a71d3a37a4d237bf8cf5ca0615963ecee90da" 2023-09-10T00:52:42.614 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T00:52:42.614 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T00:52:42.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2023-09-10T00:52:42.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2023-09-10T00:52:42.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2023-09-10T00:52:42.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2023-09-10T00:52:42.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2023-09-10T00:52:42.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2023-09-10T00:52:42.916 INFO:tasks.cephadm:Writing seed config... 2023-09-10T00:52:42.917 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-09-10T00:52:42.917 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-09-10T00:52:42.917 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-09-10T00:52:42.918 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-09-10T00:52:42.918 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-09-10T00:52:42.918 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-09-10T00:52:42.919 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-09-10T00:52:42.919 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-09-10T00:52:42.919 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-09-10T00:52:42.919 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-09-10T00:52:42.920 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-09-10T00:52:42.920 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-09-10T00:52:42.920 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-09-10T00:52:42.920 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-09-10T00:52:42.921 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-09-10T00:52:42.921 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-09-10T00:52:42.921 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-09-10T00:52:42.922 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-09-10T00:52:42.922 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-09-10T00:52:42.922 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-09-10T00:52:42.922 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-09-10T00:52:42.923 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-09-10T00:52:42.923 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-09-10T00:52:42.924 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:52:42.924 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-09-10T00:52:42.967 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4260c16a-4f74-11ee-9ab7-7b867c8bd7da ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-09-10T00:52:42.967 DEBUG:teuthology.orchestra.run.smithi003:mon.a> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a.service 2023-09-10T00:52:43.012 DEBUG:teuthology.orchestra.run.smithi003:mgr.x> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.x.service 2023-09-10T00:52:43.059 INFO:tasks.cephadm:Bootstrapping... 2023-09-10T00:52:43.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 -v bootstrap --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.3 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-09-10T00:52:43.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T00:52:43.129 INFO:journalctl@ceph.mgr.x.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T00:52:43.292 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-09-10T00:52:43.292 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00', '-v', 'bootstrap', '--fsid', '4260c16a-4f74-11ee-9ab7-7b867c8bd7da', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.3', '--skip-admin-label'] 2023-09-10T00:52:43.467 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.1.1 2023-09-10T00:52:43.467 INFO:teuthology.orchestra.run.smithi003.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-09-10T00:52:43.468 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying podman|docker is present... 2023-09-10T00:52:43.638 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.1.1 2023-09-10T00:52:43.639 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying lvm2 is present... 2023-09-10T00:52:43.639 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying time synchronization is in place... 2023-09-10T00:52:43.653 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-09-10T00:52:43.653 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-09-10T00:52:43.666 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-09-10T00:52:43.667 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-09-10T00:52:43.680 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2023-09-10T00:52:43.693 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2023-09-10T00:52:43.694 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2023-09-10T00:52:43.694 INFO:teuthology.orchestra.run.smithi003.stdout:Repeating the final host check... 2023-09-10T00:52:43.867 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.1.1 2023-09-10T00:52:43.868 INFO:teuthology.orchestra.run.smithi003.stdout:podman (/bin/podman) version 4.1.1 is present 2023-09-10T00:52:43.868 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl is present 2023-09-10T00:52:43.868 INFO:teuthology.orchestra.run.smithi003.stdout:lvcreate is present 2023-09-10T00:52:43.881 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-09-10T00:52:43.881 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-09-10T00:52:43.893 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-09-10T00:52:43.893 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-09-10T00:52:43.907 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2023-09-10T00:52:43.920 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2023-09-10T00:52:43.920 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2023-09-10T00:52:43.920 INFO:teuthology.orchestra.run.smithi003.stdout:Host looks OK 2023-09-10T00:52:43.921 INFO:teuthology.orchestra.run.smithi003.stdout:Cluster fsid: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:52:43.921 INFO:teuthology.orchestra.run.smithi003.stdout:Acquiring lock 140579095325160 on /run/cephadm/4260c16a-4f74-11ee-9ab7-7b867c8bd7da.lock 2023-09-10T00:52:43.922 INFO:teuthology.orchestra.run.smithi003.stdout:Lock 140579095325160 acquired on /run/cephadm/4260c16a-4f74-11ee-9ab7-7b867c8bd7da.lock 2023-09-10T00:52:43.922 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 3300 ... 2023-09-10T00:52:43.922 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 6789 ... 2023-09-10T00:52:43.923 INFO:teuthology.orchestra.run.smithi003.stdout:Base mon IP(s) is [172.21.15.3:3300, 172.21.15.3:6789], mon addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-09-10T00:52:43.929 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-09-10T00:52:43.930 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2023-09-10T00:52:43.937 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-09-10T00:52:43.937 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-09-10T00:52:43.937 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1783sec hoplimit 64 pref medium 2023-09-10T00:52:43.943 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-09-10T00:52:43.943 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-09-10T00:52:43.943 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-09-10T00:52:43.944 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-09-10T00:52:43.944 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2023-09-10T00:52:43.944 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-09-10T00:52:43.945 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2023-09-10T00:52:43.945 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2023-09-10T00:52:43.945 INFO:teuthology.orchestra.run.smithi003.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-09-10T00:52:43.946 INFO:teuthology.orchestra.run.smithi003.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-09-10T00:52:43.946 INFO:teuthology.orchestra.run.smithi003.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00... 2023-09-10T00:52:44.619 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 20c3861ef6c981f72050dd1dd3edbdc62747039bda543319e240a6e02f850eea 2023-09-10T00:52:44.620 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00... 2023-09-10T00:52:44.620 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Getting image source signatures 2023-09-10T00:52:44.620 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:2d7a35eed4a09aaaa91d235ae74efd6175638ed8fde15ffafe0c3c53db478b03 2023-09-10T00:52:44.620 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:d3e1674b1d277d1d98aa3561ad0e3209c8e2f26183d8e8b51d0d9fbac358121d 2023-09-10T00:52:44.620 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying config sha256:20c3861ef6c981f72050dd1dd3edbdc62747039bda543319e240a6e02f850eea 2023-09-10T00:52:44.621 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Writing manifest to image destination 2023-09-10T00:52:44.621 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Storing signatures 2023-09-10T00:52:45.599 INFO:teuthology.orchestra.run.smithi003.stdout:ceph: stdout ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev) 2023-09-10T00:52:45.600 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph version: ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev) 2023-09-10T00:52:45.601 INFO:teuthology.orchestra.run.smithi003.stdout:Extracting ceph user uid/gid from container image... 2023-09-10T00:52:46.478 INFO:teuthology.orchestra.run.smithi003.stdout:stat: stdout 167 167 2023-09-10T00:52:46.478 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial keys... 2023-09-10T00:52:47.453 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBeE/1k86lKMxAAGYPqxqNXTaNQD0oSUjhT/A== 2023-09-10T00:52:48.329 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBfE/1kn/2vNRAAfbbygNP6AiMtv3ii1tRbqg== 2023-09-10T00:52:49.102 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBgE/1kinlKKxAAT1rDzVtJVlKTZsRD2A2R8w== 2023-09-10T00:52:49.104 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial monmap... 2023-09-10T00:52:49.980 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-09-10T00:52:49.981 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-09-10T00:52:49.981 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:52:49.981 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-09-10T00:52:49.981 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool for a [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-09-10T00:52:49.981 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = pacific 2023-09-10T00:52:49.982 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: set fsid to 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:52:49.982 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-09-10T00:52:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:52:49.982 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mon... 2023-09-10T00:52:50.956 INFO:teuthology.orchestra.run.smithi003.stdout:create mon.a on 2023-09-10T00:52:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-09-10T00:52:51.405 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-09-10T00:52:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da.target → /etc/systemd/system/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da.target. 2023-09-10T00:52:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da.target → /etc/systemd/system/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da.target. 2023-09-10T00:52:52.068 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a 2023-09-10T00:52:52.068 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a.service: Unit ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a.service not loaded. 2023-09-10T00:52:52.280 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da.target.wants/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a.service → /etc/systemd/system/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@.service. 2023-09-10T00:52:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:52 smithi003 systemd[1]: Starting Ceph mon.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T00:52:53.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:52 smithi003 podman[97104]: 2023-09-10T00:52:53.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:53 smithi003 bash[97104]: c06b123e3abe7137317e5971792d1a8470a219e51698e15d10be98731a7db550 2023-09-10T00:52:53.094 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-09-10T00:52:53.094 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-09-10T00:52:53.110 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-09-10T00:52:53.110 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-09-10T00:52:53.111 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-09-10T00:52:53.111 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2023-09-10T00:52:53.111 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon to start... 2023-09-10T00:52:53.112 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon... 2023-09-10T00:52:53.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:53 smithi003 ceph-mon[97144]: mkfs 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:52:53.401 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:53 smithi003 systemd[1]: Started Ceph mon.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:52:53.401 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:53 smithi003 ceph-mon[97144]: mon.a is new leader, mons a in quorum (ranks 0) 2023-09-10T00:52:54.195 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout cluster: 2023-09-10T00:52:54.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout id: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:52:54.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-09-10T00:52:54.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:52:54.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout services: 2023-09-10T00:52:54.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.689682s) 2023-09-10T00:52:54.198 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-09-10T00:52:54.198 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-09-10T00:52:54.198 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:52:54.198 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout data: 2023-09-10T00:52:54.198 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-09-10T00:52:54.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-09-10T00:52:54.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-09-10T00:52:54.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pgs: 2023-09-10T00:52:54.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:52:54.199 INFO:teuthology.orchestra.run.smithi003.stdout:mon is available 2023-09-10T00:52:54.200 INFO:teuthology.orchestra.run.smithi003.stdout:Assimilating anything we can from ceph.conf... 2023-09-10T00:52:54.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:54 smithi003 ceph-mon[97144]: mon.a is new leader, mons a in quorum (ranks 0) 2023-09-10T00:52:54.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:54 smithi003 ceph-mon[97144]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} 2023-09-10T00:52:54.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:54 smithi003 ceph-mon[97144]: fsmap 2023-09-10T00:52:54.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:54 smithi003 ceph-mon[97144]: osdmap e1: 0 total, 0 up, 0 in 2023-09-10T00:52:54.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:54 smithi003 ceph-mon[97144]: mgrmap e1: no daemons active 2023-09-10T00:52:54.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:54 smithi003 ceph-mon[97144]: from='client.? 172.21.15.3:0/967976310' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-09-10T00:52:55.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:55 smithi003 ceph-mon[97144]: from='client.? 172.21.15.3:0/2999638909' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-09-10T00:52:55.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:55 smithi003 ceph-mon[97144]: from='client.? 172.21.15.3:0/2999638909' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-09-10T00:52:55.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:52:55.278 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2023-09-10T00:52:55.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:52:55.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-09-10T00:52:55.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-09-10T00:52:55.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-09-10T00:52:55.279 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-09-10T00:52:55.280 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-09-10T00:52:55.280 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:52:55.280 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2023-09-10T00:52:55.280 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-09-10T00:52:55.280 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:52:55.281 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2023-09-10T00:52:55.281 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-09-10T00:52:55.281 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-09-10T00:52:55.281 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-09-10T00:52:55.281 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-09-10T00:52:55.282 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-09-10T00:52:55.282 INFO:teuthology.orchestra.run.smithi003.stdout:Generating new minimal ceph.conf... 2023-09-10T00:52:57.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:57 smithi003 ceph-mon[97144]: from='client.? 172.21.15.3:0/1892226888' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:52:57.659 INFO:teuthology.orchestra.run.smithi003.stdout:Restarting the monitor... 2023-09-10T00:52:58.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:57 smithi003 systemd[1]: Stopping Ceph mon.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T00:52:58.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-a[97120]: 2023-09-10T00:52:58.116+0000 7f6c411e4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T00:52:58.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-a[97120]: 2023-09-10T00:52:58.116+0000 7f6c411e4700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-09-10T00:52:58.764 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:58 smithi003 bash[97482]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-a 2023-09-10T00:52:59.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:58 smithi003 systemd[1]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a.service: Succeeded. 2023-09-10T00:52:59.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:58 smithi003 systemd[1]: Stopped Ceph mon.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:52:59.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:58 smithi003 systemd[1]: Starting Ceph mon.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T00:52:59.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 podman[97614]: 2023-09-10T00:52:59.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 bash[97614]: 939386b04bd110d64ea546dc9f8920659aeab412360f441af3048e3e4be81d18 2023-09-10T00:52:59.788 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: set uid:gid to 167:167 (ceph:ceph) 2023-09-10T00:52:59.798 INFO:teuthology.orchestra.run.smithi003.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-09-10T00:53:00.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev), process ceph-mon, pid 7 2023-09-10T00:53:00.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: pidfile_write: ignore empty --pid-file 2023-09-10T00:53:00.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: load: jerasure load: lrc 2023-09-10T00:53:00.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: RocksDB version: 7.9.2 2023-09-10T00:53:00.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Git sha 0 2023-09-10T00:53:00.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Compile date 2023-09-09 15:16:58 2023-09-10T00:53:00.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: DB SUMMARY 2023-09-10T00:53:00.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: DB Session ID: DW9AVI6Q4EQEPS0CZKJG 2023-09-10T00:53:00.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: CURRENT file: CURRENT 2023-09-10T00:53:00.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: IDENTITY file: IDENTITY 2023-09-10T00:53:00.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-09-10T00:53:00.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-09-10T00:53:00.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 89589 ; 2023-09-10T00:53:00.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.error_if_exists: 0 2023-09-10T00:53:00.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.create_if_missing: 0 2023-09-10T00:53:00.065 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.paranoid_checks: 1 2023-09-10T00:53:00.065 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.flush_verify_memtable_count: 1 2023-09-10T00:53:00.065 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-09-10T00:53:00.065 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-09-10T00:53:00.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.env: 0x56431acdbda0 2023-09-10T00:53:00.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.fs: PosixFileSystem 2023-09-10T00:53:00.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.info_log: 0x56431bd6b4e0 2023-09-10T00:53:00.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_file_opening_threads: 16 2023-09-10T00:53:00.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.statistics: (nil) 2023-09-10T00:53:00.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.use_fsync: 0 2023-09-10T00:53:00.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_log_file_size: 0 2023-09-10T00:53:00.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-09-10T00:53:00.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.log_file_time_to_roll: 0 2023-09-10T00:53:00.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.keep_log_file_num: 1000 2023-09-10T00:53:00.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.recycle_log_file_num: 0 2023-09-10T00:53:00.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.allow_fallocate: 1 2023-09-10T00:53:00.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.allow_mmap_reads: 0 2023-09-10T00:53:00.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.allow_mmap_writes: 0 2023-09-10T00:53:00.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.use_direct_reads: 0 2023-09-10T00:53:00.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-09-10T00:53:00.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.create_missing_column_families: 0 2023-09-10T00:53:00.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.db_log_dir: 2023-09-10T00:53:00.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.wal_dir: 2023-09-10T00:53:00.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.table_cache_numshardbits: 6 2023-09-10T00:53:00.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.WAL_ttl_seconds: 0 2023-09-10T00:53:00.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.WAL_size_limit_MB: 0 2023-09-10T00:53:00.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-09-10T00:53:00.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-09-10T00:53:00.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.is_fd_close_on_exec: 1 2023-09-10T00:53:00.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.advise_random_on_open: 1 2023-09-10T00:53:00.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.db_write_buffer_size: 0 2023-09-10T00:53:00.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.write_buffer_manager: 0x56431bd254a0 2023-09-10T00:53:00.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-09-10T00:53:00.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-09-10T00:53:00.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.use_adaptive_mutex: 0 2023-09-10T00:53:00.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.rate_limiter: (nil) 2023-09-10T00:53:00.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-09-10T00:53:00.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.wal_recovery_mode: 2 2023-09-10T00:53:00.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.enable_thread_tracking: 0 2023-09-10T00:53:00.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.enable_pipelined_write: 0 2023-09-10T00:53:00.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.unordered_write: 0 2023-09-10T00:53:00.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-09-10T00:53:00.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-09-10T00:53:00.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-09-10T00:53:00.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-09-10T00:53:00.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.row_cache: None 2023-09-10T00:53:00.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.wal_filter: None 2023-09-10T00:53:00.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-09-10T00:53:00.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.allow_ingest_behind: 0 2023-09-10T00:53:00.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.two_write_queues: 0 2023-09-10T00:53:00.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.manual_wal_flush: 0 2023-09-10T00:53:00.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.wal_compression: 0 2023-09-10T00:53:00.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.atomic_flush: 0 2023-09-10T00:53:00.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 systemd[1]: Started Ceph mon.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:53:00.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-09-10T00:53:00.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.persist_stats_to_disk: 0 2023-09-10T00:53:00.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.write_dbid_to_manifest: 0 2023-09-10T00:53:00.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.log_readahead_size: 0 2023-09-10T00:53:00.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-09-10T00:53:00.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.best_efforts_recovery: 0 2023-09-10T00:53:00.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-09-10T00:53:00.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-09-10T00:53:00.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.allow_data_in_errors: 0 2023-09-10T00:53:00.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.db_host_id: __hostname__ 2023-09-10T00:53:00.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.enforce_single_del_contracts: true 2023-09-10T00:53:00.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_background_jobs: 2 2023-09-10T00:53:00.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_background_compactions: -1 2023-09-10T00:53:00.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_subcompactions: 1 2023-09-10T00:53:00.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-09-10T00:53:00.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-09-10T00:53:00.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.delayed_write_rate : 16777216 2023-09-10T00:53:00.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_total_wal_size: 0 2023-09-10T00:53:00.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-09-10T00:53:00.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.stats_dump_period_sec: 600 2023-09-10T00:53:00.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.stats_persist_period_sec: 600 2023-09-10T00:53:00.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-09-10T00:53:00.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_open_files: -1 2023-09-10T00:53:00.080 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bytes_per_sync: 0 2023-09-10T00:53:00.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.wal_bytes_per_sync: 0 2023-09-10T00:53:00.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.strict_bytes_per_sync: 0 2023-09-10T00:53:00.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_readahead_size: 0 2023-09-10T00:53:00.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_background_flushes: -1 2023-09-10T00:53:00.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Compression algorithms supported: 2023-09-10T00:53:00.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-09-10T00:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kZSTD supported: 0 2023-09-10T00:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kXpressCompression supported: 0 2023-09-10T00:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kLZ4HCCompression supported: 1 2023-09-10T00:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kZlibCompression supported: 1 2023-09-10T00:53:00.082 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kSnappyCompression supported: 1 2023-09-10T00:53:00.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kLZ4Compression supported: 1 2023-09-10T00:53:00.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: kBZip2Compression supported: 0 2023-09-10T00:53:00.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Fast CRC32 supported: Supported on x86 2023-09-10T00:53:00.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: DMutex implementation: pthread_mutex_t 2023-09-10T00:53:00.083 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-09-10T00:53:00.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-09-10T00:53:00.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-09-10T00:53:00.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.merge_operator: 2023-09-10T00:53:00.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_filter: None 2023-09-10T00:53:00.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_filter_factory: None 2023-09-10T00:53:00.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.sst_partitioner_factory: None 2023-09-10T00:53:00.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.memtable_factory: SkipListFactory 2023-09-10T00:53:00.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.table_factory: BlockBasedTable 2023-09-10T00:53:00.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56431bd6b5e0) 2023-09-10T00:53:00.085 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2023-09-10T00:53:00.086 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-09-10T00:53:00.086 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-09-10T00:53:00.086 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2023-09-10T00:53:00.086 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2023-09-10T00:53:00.086 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2023-09-10T00:53:00.087 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2023-09-10T00:53:00.087 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2023-09-10T00:53:00.087 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 4 2023-09-10T00:53:00.087 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2023-09-10T00:53:00.087 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x56431bdc31f0 2023-09-10T00:53:00.088 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2023-09-10T00:53:00.088 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2023-09-10T00:53:00.088 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2023-09-10T00:53:00.088 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2023-09-10T00:53:00.088 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2023-09-10T00:53:00.089 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2023-09-10T00:53:00.089 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2023-09-10T00:53:00.089 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2023-09-10T00:53:00.089 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2023-09-10T00:53:00.089 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2023-09-10T00:53:00.090 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2023-09-10T00:53:00.090 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2023-09-10T00:53:00.090 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2023-09-10T00:53:00.090 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2023-09-10T00:53:00.090 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2023-09-10T00:53:00.091 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: bloomfilter 2023-09-10T00:53:00.091 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2023-09-10T00:53:00.091 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2023-09-10T00:53:00.091 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2023-09-10T00:53:00.091 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 5 2023-09-10T00:53:00.092 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2023-09-10T00:53:00.092 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2023-09-10T00:53:00.092 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_auto_readahead_size: 262144 2023-09-10T00:53:00.092 INFO:journalctl@ceph.mon.a.smithi003.stdout: prepopulate_block_cache: 0 2023-09-10T00:53:00.092 INFO:journalctl@ceph.mon.a.smithi003.stdout: initial_auto_readahead_size: 8192 2023-09-10T00:53:00.093 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2023-09-10T00:53:00.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.write_buffer_size: 33554432 2023-09-10T00:53:00.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_write_buffer_number: 2 2023-09-10T00:53:00.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression: NoCompression 2023-09-10T00:53:00.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression: Disabled 2023-09-10T00:53:00.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.prefix_extractor: nullptr 2023-09-10T00:53:00.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-09-10T00:53:00.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.num_levels: 7 2023-09-10T00:53:00.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-09-10T00:53:00.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-09-10T00:53:00.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-09-10T00:53:00.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-09-10T00:53:00.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-09-10T00:53:00.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-09-10T00:53:00.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-09-10T00:53:00.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-09-10T00:53:00.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-09-10T00:53:00.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-09-10T00:53:00.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-09-10T00:53:00.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-09-10T00:53:00.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.window_bits: -14 2023-09-10T00:53:00.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.level: 32767 2023-09-10T00:53:00.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.strategy: 0 2023-09-10T00:53:00.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-09-10T00:53:00.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-09-10T00:53:00.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-09-10T00:53:00.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-09-10T00:53:00.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.enabled: false 2023-09-10T00:53:00.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-09-10T00:53:00.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-09-10T00:53:00.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-09-10T00:53:00.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-09-10T00:53:00.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.target_file_size_base: 67108864 2023-09-10T00:53:00.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.target_file_size_multiplier: 1 2023-09-10T00:53:00.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-09-10T00:53:00.100 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-09-10T00:53:00.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-09-10T00:53:00.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-09-10T00:53:00.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-09-10T00:53:00.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-09-10T00:53:00.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-09-10T00:53:00.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-09-10T00:53:00.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-09-10T00:53:00.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-09-10T00:53:00.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-09-10T00:53:00.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-09-10T00:53:00.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-09-10T00:53:00.103 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.arena_block_size: 1048576 2023-09-10T00:53:00.103 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-09-10T00:53:00.103 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-09-10T00:53:00.103 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.disable_auto_compactions: 0 2023-09-10T00:53:00.103 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-09-10T00:53:00.104 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-09-10T00:53:00.104 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-09-10T00:53:00.104 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-09-10T00:53:00.104 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-09-10T00:53:00.104 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-09-10T00:53:00.104 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-09-10T00:53:00.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-09-10T00:53:00.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-09-10T00:53:00.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-09-10T00:53:00.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.table_properties_collectors: 2023-09-10T00:53:00.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.inplace_update_support: 0 2023-09-10T00:53:00.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.inplace_update_num_locks: 10000 2023-09-10T00:53:00.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-09-10T00:53:00.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-09-10T00:53:00.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.memtable_huge_page_size: 0 2023-09-10T00:53:00.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.bloom_locality: 0 2023-09-10T00:53:00.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.max_successive_merges: 0 2023-09-10T00:53:00.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.optimize_filters_for_hits: 0 2023-09-10T00:53:00.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.paranoid_file_checks: 0 2023-09-10T00:53:00.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.force_consistency_checks: 1 2023-09-10T00:53:00.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.report_bg_io_stats: 0 2023-09-10T00:53:00.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.ttl: 2592000 2023-09-10T00:53:00.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.periodic_compaction_seconds: 0 2023-09-10T00:53:00.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-09-10T00:53:00.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-09-10T00:53:00.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.enable_blob_files: false 2023-09-10T00:53:00.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.min_blob_size: 0 2023-09-10T00:53:00.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.blob_file_size: 268435456 2023-09-10T00:53:00.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.blob_compression_type: NoCompression 2023-09-10T00:53:00.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.enable_blob_garbage_collection: false 2023-09-10T00:53:00.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-09-10T00:53:00.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-09-10T00:53:00.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-09-10T00:53:00.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.blob_file_starting_level: 0 2023-09-10T00:53:00.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-09-10T00:53:00.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-09-10T00:53:00.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-09-10T00:53:00.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a55904c8-6a0a-4f3b-92d3-9fdc44444c29 2023-09-10T00:53:00.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307179796519, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-09-10T00:53:00.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-09-10T00:53:00.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307179798128, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85555, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 281, "table_properties": {"data_size": 83658, "index_size": 243, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11225, "raw_average_key_size": 48, "raw_value_size": 77371, "raw_average_value_size": 333, "num_data_blocks": 10, "num_entries": 232, "num_filter_entries": 232, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1694307179, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a55904c8-6a0a-4f3b-92d3-9fdc44444c29", "db_session_id": "DW9AVI6Q4EQEPS0CZKJG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-09-10T00:53:00.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307179798272, "job": 1, "event": "recovery_finished"} 2023-09-10T00:53:00.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-09-10T00:53:00.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-09-10T00:53:00.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56431be5e000 2023-09-10T00:53:00.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: DB pointer 0x56431be4a000 2023-09-10T00:53:00.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-09-10T00:53:00.113 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-09-10T00:53:00.113 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2023-09-10T00:53:00.113 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-09-10T00:53:00.113 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-09-10T00:53:00.113 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-09-10T00:53:00.113 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-09-10T00:53:00.114 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-09-10T00:53:00.114 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-09-10T00:53:00.114 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-09-10T00:53:00.115 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-09-10T00:53:00.115 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2023-09-10T00:53:00.115 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-09-10T00:53:00.115 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-09-10T00:53:00.115 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 85.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:53:00.115 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 85.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:53:00.116 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:53:00.116 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-09-10T00:53:00.116 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2023-09-10T00:53:00.116 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-09-10T00:53:00.116 INFO:journalctl@ceph.mon.a.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-09-10T00:53:00.117 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:53:00.117 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-09-10T00:53:00.117 INFO:journalctl@ceph.mon.a.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-09-10T00:53:00.117 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-09-10T00:53:00.117 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-09-10T00:53:00.117 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-09-10T00:53:00.118 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-09-10T00:53:00.118 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-09-10T00:53:00.118 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-09-10T00:53:00.118 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2023-09-10T00:53:00.118 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 10.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-09-10T00:53:00.119 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 10.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-09-10T00:53:00.119 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-09-10T00:53:00.119 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache BinnedLRUCache@0x56431bdc31f0#7 capacity: 512.00 MB usage: 1.27 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2023-09-10T00:53:00.119 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2023-09-10T00:53:00.119 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-09-10T00:53:00.119 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2023-09-10T00:53:00.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:53:00.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???) e1 preinit fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:53:00.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???).mds e1 new map 2023-09-10T00:53:00.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???).mds e1 print_map 2023-09-10T00:53:00.120 INFO:journalctl@ceph.mon.a.smithi003.stdout: e1 2023-09-10T00:53:00.121 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-09-10T00:53:00.121 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-09-10T00:53:00.121 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: -1 2023-09-10T00:53:00.121 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2023-09-10T00:53:00.121 INFO:journalctl@ceph.mon.a.smithi003.stdout: No filesystems configured 2023-09-10T00:53:00.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-09-10T00:53:00.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:53:00.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:53:00.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:53:00.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-09-10T00:53:00.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expand map: {default=false} 2023-09-10T00:53:00.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta from 'false' to 'false' 2023-09-10T00:53:00.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expanded map: {default=false} 2023-09-10T00:53:00.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expand map: {default=info} 2023-09-10T00:53:00.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta from 'info' to 'info' 2023-09-10T00:53:00.124 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expanded map: {default=info} 2023-09-10T00:53:00.124 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expand map: {default=daemon} 2023-09-10T00:53:00.124 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta from 'daemon' to 'daemon' 2023-09-10T00:53:00.124 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expanded map: {default=daemon} 2023-09-10T00:53:00.124 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expand map: {default=debug} 2023-09-10T00:53:00.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta from 'debug' to 'debug' 2023-09-10T00:53:00.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: expand_channel_meta expanded map: {default=debug} 2023-09-10T00:53:00.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mon.a is new leader, mons a in quorum (ranks 0) 2023-09-10T00:53:00.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} 2023-09-10T00:53:00.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: fsmap 2023-09-10T00:53:00.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: osdmap e1: 0 total, 0 up, 0 in 2023-09-10T00:53:00.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:52:59 smithi003 ceph-mon[97654]: mgrmap e1: no daemons active 2023-09-10T00:53:01.084 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote config to /etc/ceph/ceph.conf 2023-09-10T00:53:01.125 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-09-10T00:53:01.126 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mgr... 2023-09-10T00:53:01.126 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:9283 ... 2023-09-10T00:53:01.127 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:8765 ... 2023-09-10T00:53:01.515 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.x 2023-09-10T00:53:01.515 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.x.service: Unit ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.x.service not loaded. 2023-09-10T00:53:01.723 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da.target.wants/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.x.service → /etc/systemd/system/ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@.service. 2023-09-10T00:53:01.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1445123672' entity='client.admin' 2023-09-10T00:53:01.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:01 smithi003 systemd[1]: Starting Ceph mgr.x for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T00:53:02.471 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:02 smithi003 podman[97886]: 2023-09-10T00:53:02.533 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-09-10T00:53:02.534 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-09-10T00:53:02.550 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-09-10T00:53:02.550 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-09-10T00:53:02.550 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-09-10T00:53:02.551 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2023-09-10T00:53:02.570 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-09-10T00:53:02.571 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-09-10T00:53:02.590 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-09-10T00:53:02.590 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-09-10T00:53:02.591 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-09-10T00:53:02.591 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-09-10T00:53:02.592 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr to start... 2023-09-10T00:53:02.592 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr... 2023-09-10T00:53:02.770 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:02 smithi003 bash[97886]: 50ba7b7298296730e895690617b9d449bf3550ab5c7c36bd617076230d306c5d 2023-09-10T00:53:02.770 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:02 smithi003 systemd[1]: Started Ceph mgr.x for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:53:02.771 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:02.649+0000 7fc807e7d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T00:53:02.771 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:02.769+0000 7fc807e7d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T00:53:03.022 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:02.889+0000 7fc807e7d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T00:53:03.023 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:03.021+0000 7fc807e7d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T00:53:03.308 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:03.270+0000 7fc807e7d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T00:53:03.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3618031353' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-09-10T00:53:03.701 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:53:03.709 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-09-10T00:53:03.709 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T00:53:03.709 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2023-09-10T00:53:03.710 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-09-10T00:53:03.710 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-09-10T00:53:03.710 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-09-10T00:53:03.710 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:03.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-09-10T00:53:03.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-09-10T00:53:03.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2023-09-10T00:53:03.711 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-09-10T00:53:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2023-09-10T00:53:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-09-10T00:53:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2023-09-10T00:53:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-09-10T00:53:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-09-10T00:53:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:03.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-09-10T00:53:03.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:03.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-09-10T00:53:03.714 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-09-10T00:53:03.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-09-10T00:53:03.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-09-10T00:53:03.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-09-10T00:53:03.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-09-10T00:53:03.715 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:03.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-09-10T00:53:03.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-09-10T00:53:03.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-09-10T00:53:03.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-09-10T00:53:03.716 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-09-10T00:53:03.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-09-10T00:53:03.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-09-10T00:53:03.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-09-10T00:53:03.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-09-10T00:53:03.717 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:03.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-09-10T00:53:03.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:03.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-09-10T00:53:03.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-09-10T00:53:03.718 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:03.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-09-10T00:53:03.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2023-09-10T00:53:03.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-09-10T00:53:03.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2023-09-10T00:53:03.719 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2023-09-10T00:53:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2023-09-10T00:53:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2023-09-10T00:53:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-09-10T00:53:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:03.720 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-09-10T00:53:03.721 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:03.721 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2023-09-10T00:52:53.067628+0000", 2023-09-10T00:53:03.721 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-09-10T00:53:03.721 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:03.721 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-09-10T00:53:03.722 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-09-10T00:53:03.722 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (1/15)... 2023-09-10T00:53:04.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:03.772+0000 7fc807e7d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T00:53:04.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:03.944+0000 7fc807e7d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T00:53:04.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:04 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:04.064+0000 7fc807e7d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T00:53:04.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:04 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:04.192+0000 7fc807e7d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T00:53:05.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:04 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:04.764+0000 7fc807e7d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T00:53:05.965 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:05 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:05.692+0000 7fc807e7d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T00:53:05.965 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:05 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:05.810+0000 7fc807e7d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T00:53:06.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:06 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1850090059' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-09-10T00:53:07.193 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:06 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:06.836+0000 7fc807e7d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:53:07.194 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:06 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:06.961+0000 7fc807e7d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T00:53:07.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:07 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:07.192+0000 7fc807e7d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:53:07.885 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:07 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:07.583+0000 7fc807e7d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T00:53:08.308 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:07 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:07.884+0000 7fc807e7d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T00:53:08.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:08 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:08.093+0000 7fc807e7d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T00:53:08.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:08 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:08.444+0000 7fc807e7d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T00:53:09.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:09 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:09.406+0000 7fc807e7d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T00:53:09.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:09 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:09.586+0000 7fc807e7d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T00:53:10.276 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:09 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:09.841+0000 7fc807e7d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T00:53:10.276 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:09 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:09.992+0000 7fc807e7d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T00:53:10.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:10.275+0000 7fc807e7d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T00:53:11.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:11.384+0000 7fc807e7d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T00:53:11.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: Activating manager daemon x 2023-09-10T00:53:11.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: mgrmap e2: x(active, starting, since 0.00420078s) 2023-09-10T00:53:11.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-09-10T00:53:11.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-09-10T00:53:11.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-09-10T00:53:11.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:53:11.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-09-10T00:53:11.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:11 smithi003 ceph-mon[97654]: Manager daemon x is now available 2023-09-10T00:53:12.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:12 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T00:53:12.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:12 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T00:53:12.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:12 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' 2023-09-10T00:53:12.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:12 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' 2023-09-10T00:53:12.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:12 smithi003 ceph-mon[97654]: from='mgr.14100 172.21.15.3:0/2257261836' entity='mgr.x' 2023-09-10T00:53:12.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:12 smithi003 ceph-mon[97654]: mgrmap e3: x(active, since 1.01001s) 2023-09-10T00:53:14.009 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:53:14.009 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-09-10T00:53:14.009 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T00:53:14.009 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2023-09-10T00:53:14.010 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-09-10T00:53:14.010 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-09-10T00:53:14.010 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-09-10T00:53:14.010 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:14.010 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-09-10T00:53:14.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-09-10T00:53:14.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2023-09-10T00:53:14.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:14.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-09-10T00:53:14.011 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2023-09-10T00:53:14.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:14.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2023-09-10T00:53:14.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2023-09-10T00:53:14.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:14.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-09-10T00:53:14.012 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-09-10T00:53:14.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:14.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-09-10T00:53:14.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:14.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-09-10T00:53:14.013 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-09-10T00:53:14.014 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-09-10T00:53:14.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-09-10T00:53:14.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-09-10T00:53:14.017 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-09-10T00:53:14.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:14.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-09-10T00:53:14.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-09-10T00:53:14.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-09-10T00:53:14.018 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-09-10T00:53:14.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-09-10T00:53:14.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-09-10T00:53:14.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-09-10T00:53:14.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-09-10T00:53:14.019 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-09-10T00:53:14.020 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:14.020 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-09-10T00:53:14.020 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:14.020 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-09-10T00:53:14.020 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-09-10T00:53:14.021 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:14.021 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-09-10T00:53:14.021 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2023-09-10T00:53:14.021 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-09-10T00:53:14.021 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2023-09-10T00:53:14.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2023-09-10T00:53:14.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2023-09-10T00:53:14.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2023-09-10T00:53:14.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:14.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-09-10T00:53:14.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:14.023 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-09-10T00:53:14.023 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:14.023 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2023-09-10T00:52:53.067628+0000", 2023-09-10T00:53:14.023 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-09-10T00:53:14.023 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:14.024 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-09-10T00:53:14.024 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-09-10T00:53:14.024 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (2/15)... 2023-09-10T00:53:14.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:14 smithi003 ceph-mon[97654]: mgrmap e4: x(active, since 2s) 2023-09-10T00:53:17.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:16 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2344007546' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-09-10T00:53:17.303 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:53:17.304 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-09-10T00:53:17.304 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T00:53:17.304 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2023-09-10T00:53:17.304 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-09-10T00:53:17.304 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-09-10T00:53:17.305 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-09-10T00:53:17.305 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:17.305 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-09-10T00:53:17.305 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-09-10T00:53:17.305 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2023-09-10T00:53:17.306 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:17.306 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-09-10T00:53:17.306 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2023-09-10T00:53:17.306 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:17.306 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2023-09-10T00:53:17.307 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2023-09-10T00:53:17.307 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:17.307 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-09-10T00:53:17.307 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-09-10T00:53:17.307 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:17.307 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-09-10T00:53:17.308 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:17.308 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-09-10T00:53:17.308 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-09-10T00:53:17.308 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-09-10T00:53:17.308 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-09-10T00:53:17.312 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-09-10T00:53:17.312 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-09-10T00:53:17.312 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:17.312 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-09-10T00:53:17.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-09-10T00:53:17.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-09-10T00:53:17.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-09-10T00:53:17.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-09-10T00:53:17.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-09-10T00:53:17.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-09-10T00:53:17.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-09-10T00:53:17.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-09-10T00:53:17.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:17.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-09-10T00:53:17.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:17.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-09-10T00:53:17.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-09-10T00:53:17.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:17.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-09-10T00:53:17.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2023-09-10T00:53:17.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-09-10T00:53:17.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2023-09-10T00:53:17.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2023-09-10T00:53:17.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2023-09-10T00:53:17.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2023-09-10T00:53:17.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2023-09-10T00:53:17.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-09-10T00:53:17.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:17.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-09-10T00:53:17.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-09-10T00:53:17.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2023-09-10T00:52:53.067628+0000", 2023-09-10T00:53:17.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2023-09-10T00:53:17.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2023-09-10T00:53:17.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-09-10T00:53:17.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-09-10T00:53:17.318 INFO:teuthology.orchestra.run.smithi003.stdout:mgr is available 2023-09-10T00:53:18.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:18 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/682719580' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-09-10T00:53:18.585 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:53:18.585 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2023-09-10T00:53:18.585 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:53:18.585 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-09-10T00:53:18.586 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-09-10T00:53:18.586 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-09-10T00:53:18.586 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-09-10T00:53:18.586 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:53:18.586 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2023-09-10T00:53:18.587 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-09-10T00:53:18.587 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2023-09-10T00:53:18.587 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2023-09-10T00:53:18.587 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-09-10T00:53:18.587 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-09-10T00:53:18.588 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-09-10T00:53:18.588 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-09-10T00:53:18.588 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-09-10T00:53:18.588 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling cephadm module... 2023-09-10T00:53:19.726 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:19 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/119786896' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-09-10T00:53:19.727 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:19 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: ignoring --setuser ceph since I am not root 2023-09-10T00:53:19.727 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:19 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: ignoring --setgroup ceph since I am not root 2023-09-10T00:53:20.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:19 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:19.725+0000 7ff5e2758200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T00:53:20.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:19 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:19.834+0000 7ff5e2758200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T00:53:20.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:19 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:19.965+0000 7ff5e2758200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T00:53:20.463 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:20 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:20.094+0000 7ff5e2758200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T00:53:20.463 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:20 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:20.338+0000 7ff5e2758200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T00:53:20.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:20 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/119786896' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-09-10T00:53:20.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:20 smithi003 ceph-mon[97654]: mgrmap e5: x(active, since 8s) 2023-09-10T00:53:21.041 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-09-10T00:53:21.042 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-09-10T00:53:21.042 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2023-09-10T00:53:21.042 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-09-10T00:53:21.042 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-09-10T00:53:21.043 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-09-10T00:53:21.043 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2023-09-10T00:53:21.043 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 5... 2023-09-10T00:53:21.093 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:20 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:20.827+0000 7ff5e2758200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T00:53:21.093 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:20 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:20.996+0000 7ff5e2758200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T00:53:21.359 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:21 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:21.132+0000 7ff5e2758200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T00:53:21.360 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:21 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:21.264+0000 7ff5e2758200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T00:53:21.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:21 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3019817299' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-09-10T00:53:22.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:21 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:21.836+0000 7ff5e2758200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T00:53:23.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:22 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:22.714+0000 7ff5e2758200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T00:53:23.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:22 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:22.824+0000 7ff5e2758200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T00:53:24.173 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:23 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:23.816+0000 7ff5e2758200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:53:24.173 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:23 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:23.941+0000 7ff5e2758200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T00:53:24.550 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:24 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:24.172+0000 7ff5e2758200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:53:24.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:24 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:24.549+0000 7ff5e2758200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T00:53:25.308 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:24 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:24.832+0000 7ff5e2758200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T00:53:25.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:25 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:25.033+0000 7ff5e2758200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T00:53:25.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:25 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:25.387+0000 7ff5e2758200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T00:53:26.789 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:26.351+0000 7ff5e2758200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T00:53:26.790 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:26.532+0000 7ff5e2758200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T00:53:27.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:26.788+0000 7ff5e2758200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T00:53:27.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:26.940+0000 7ff5e2758200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T00:53:27.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:27 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:27.225+0000 7ff5e2758200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T00:53:28.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:28.330+0000 7ff5e2758200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T00:53:28.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: Active manager daemon x restarted 2023-09-10T00:53:28.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: Activating manager daemon x 2023-09-10T00:53:28.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: osdmap e2: 0 total, 0 up, 0 in 2023-09-10T00:53:28.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: mgrmap e6: x(active, starting, since 0.00378034s) 2023-09-10T00:53:28.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:53:28.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-09-10T00:53:28.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-09-10T00:53:28.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-09-10T00:53:28.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-09-10T00:53:28.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: Manager daemon x is now available 2023-09-10T00:53:28.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:28.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:28.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:28.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:28 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:29.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-09-10T00:53:29.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-09-10T00:53:29.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2023-09-10T00:53:29.742 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-09-10T00:53:29.743 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 5 is available 2023-09-10T00:53:29.743 INFO:teuthology.orchestra.run.smithi003.stdout:Setting orchestrator backend to cephadm... 2023-09-10T00:53:29.765 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:29 smithi003 ceph-mon[97654]: Found migration_current of "None". Setting to last migration. 2023-09-10T00:53:29.765 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:29 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:29.765 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:29 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T00:53:29.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:29 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T00:53:29.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:29 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:29.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:29 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:29.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:29 smithi003 ceph-mon[97654]: mgrmap e7: x(active, since 1.00637s) 2023-09-10T00:53:30.648 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:30 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:29] ENGINE Bus STARTING 2023-09-10T00:53:30.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:30 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:29] ENGINE Serving on https://172.21.15.3:7150 2023-09-10T00:53:30.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:30 smithi003 ceph-mon[97654]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-09-10T00:53:30.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:30 smithi003 ceph-mon[97654]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-09-10T00:53:30.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:30 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:29] ENGINE Serving on http://172.21.15.3:8765 2023-09-10T00:53:30.649 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:30 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:29] ENGINE Bus STARTED 2023-09-10T00:53:30.650 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:30 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:31.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:31 smithi003 ceph-mon[97654]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:31.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:31 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:31.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:31 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:31.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:31 smithi003 ceph-mon[97654]: mgrmap e8: x(active, since 2s) 2023-09-10T00:53:32.098 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout value unchanged 2023-09-10T00:53:32.098 INFO:teuthology.orchestra.run.smithi003.stdout:Generating ssh key... 2023-09-10T00:53:32.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:32 smithi003 ceph-mon[97654]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:33.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: Generating public/private rsa key pair. 2023-09-10T00:53:33.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: Your identification has been saved in /tmp/tmpqsd7vepm/key. 2023-09-10T00:53:33.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: Your public key has been saved in /tmp/tmpqsd7vepm/key.pub. 2023-09-10T00:53:33.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: The key fingerprint is: 2023-09-10T00:53:33.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: SHA256:+MRBIKLsAaDXM3J21E2rl3qUqjoAPk+qtQYK+QfNWx0 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:53:33.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: The key's randomart image is: 2023-09-10T00:53:33.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: +---[RSA 3072]----+ 2023-09-10T00:53:33.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |+ . . oo.o. | 2023-09-10T00:53:33.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |+. o o .. .. | 2023-09-10T00:53:33.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |ooo * . . . | 2023-09-10T00:53:33.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |o..+ + oEo o | 2023-09-10T00:53:33.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |.+ o ..S.= | 2023-09-10T00:53:33.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |+oo.o .o.= | 2023-09-10T00:53:33.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |oo*o o + . | 2023-09-10T00:53:33.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |.ooo+ . . | 2023-09-10T00:53:33.562 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: |o....o. | 2023-09-10T00:53:33.562 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: +----[SHA256]-----+ 2023-09-10T00:53:34.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:34 smithi003 ceph-mon[97654]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:34.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:34 smithi003 ceph-mon[97654]: Generating ssh key... 2023-09-10T00:53:34.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:34 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:34.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:34 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:34.749 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78XTDjJO/eM150Gjc6JMF6S0KDWAGPJoaiSgBBCoxY28V3BBUoFMk9VR9QqvFLjy43ahSiCFLTCba2Ptas8rCqfmOcSaJtrmlQWsWvTYy3d6SdPlUwBtkpC9qCSkeleOpEJunvrGGH1ix8jFiIGE2dzi9W35cBHDIqWtqgr2wi6Uo9TKLkzWQJw1SgkhDtrm7/zUgzlOGGqIIwJKYDzYR7a+T03iqX9NzT/ppcJduCKthGJQD/tv+Ml9S5SynChmcCrSTrc2rFtX8M6Xh76xXJTycJ4DWQ1YsZVG5c/HTl6h5BkX7OfuQZcbFEwmBe8jSfymyxbPDS0kQfHDoeUKztktDNUZNWUwmVckntvepTdlPA+FYfyETT55HcrSm/rT9AAyv+Mtppoq+GAtSr0h0wnyGxm/ajbw/+sTkI+Dln3N+p0fwE+wfKJ9oSryyWWtwWaaGpPnjlnQHP5hZR1YY6wttYtcPbEmhppuNncEq9gtHgUe5Chk5+nDaoHY0EkM= ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:53:34.749 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-09-10T00:53:34.750 INFO:teuthology.orchestra.run.smithi003.stdout:Adding key to root@localhost authorized_keys... 2023-09-10T00:53:34.751 INFO:teuthology.orchestra.run.smithi003.stdout:Adding host smithi003... 2023-09-10T00:53:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:35 smithi003 ceph-mon[97654]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:36.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:36 smithi003 ceph-mon[97654]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:37.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:37 smithi003 ceph-mon[97654]: Deploying cephadm binary to smithi003 2023-09-10T00:53:39.340 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Added host 'smithi003' with addr '172.21.15.3' 2023-09-10T00:53:39.341 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mon service... 2023-09-10T00:53:40.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:39 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:40.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:39 smithi003 ceph-mon[97654]: Added host smithi003 2023-09-10T00:53:40.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:39 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:40.821 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-09-10T00:53:40.822 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mgr service... 2023-09-10T00:53:41.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:41 smithi003 ceph-mon[97654]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:41.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:41 smithi003 ceph-mon[97654]: Saving service mon spec with placement count:5 2023-09-10T00:53:41.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:41 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:42.450 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-09-10T00:53:42.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:42 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:42.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:42 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:42.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:42 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:43.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:43 smithi003 ceph-mon[97654]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:43.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:43 smithi003 ceph-mon[97654]: Saving service mgr spec with placement count:2 2023-09-10T00:53:43.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:43 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1756419103' entity='client.admin' 2023-09-10T00:53:45.161 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling the dashboard module... 2023-09-10T00:53:45.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:45 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:45.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:45 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2850631721' entity='client.admin' 2023-09-10T00:53:47.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:46 smithi003 ceph-mon[97654]: from='mgr.14118 172.21.15.3:0/3675141756' entity='mgr.x' 2023-09-10T00:53:47.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:46 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2251693014' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-09-10T00:53:47.081 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:46 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: ignoring --setuser ceph since I am not root 2023-09-10T00:53:47.082 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:46 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: ignoring --setgroup ceph since I am not root 2023-09-10T00:53:47.364 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:47.128+0000 7f62007ff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T00:53:47.365 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:47.236+0000 7f62007ff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T00:53:47.687 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:47.363+0000 7f62007ff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T00:53:47.687 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:47.521+0000 7f62007ff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T00:53:47.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:47 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2251693014' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-09-10T00:53:47.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:47 smithi003 ceph-mon[97654]: mgrmap e9: x(active, since 18s) 2023-09-10T00:53:47.941 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:47.752+0000 7f62007ff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T00:53:48.538 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:48.242+0000 7f62007ff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T00:53:48.538 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:48.409+0000 7f62007ff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T00:53:48.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:48.537+0000 7f62007ff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T00:53:48.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:48.667+0000 7f62007ff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T00:53:48.861 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-09-10T00:53:48.861 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-09-10T00:53:48.862 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2023-09-10T00:53:48.862 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-09-10T00:53:48.862 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-09-10T00:53:48.863 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-09-10T00:53:48.863 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2023-09-10T00:53:48.863 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 9... 2023-09-10T00:53:49.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:48 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/326082937' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-09-10T00:53:49.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:49 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:49.263+0000 7f62007ff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T00:53:50.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:50.148+0000 7f62007ff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T00:53:50.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:50.259+0000 7f62007ff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T00:53:51.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:51.245+0000 7f62007ff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:53:51.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:51.368+0000 7f62007ff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T00:53:51.975 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:51.597+0000 7f62007ff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:53:52.257 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:51.973+0000 7f62007ff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T00:53:52.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:52.255+0000 7f62007ff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T00:53:52.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:52.456+0000 7f62007ff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T00:53:53.308 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:52.808+0000 7f62007ff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T00:53:54.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:53.766+0000 7f62007ff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T00:53:54.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:53.947+0000 7f62007ff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T00:53:54.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:54.202+0000 7f62007ff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T00:53:54.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:54.355+0000 7f62007ff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T00:53:55.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:54.639+0000 7f62007ff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T00:53:56.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:53:55.750+0000 7f62007ff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T00:53:56.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: Active manager daemon x restarted 2023-09-10T00:53:56.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: Activating manager daemon x 2023-09-10T00:53:56.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: osdmap e3: 0 total, 0 up, 0 in 2023-09-10T00:53:56.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: mgrmap e10: x(active, starting, since 0.00636773s) 2023-09-10T00:53:56.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:53:56.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-09-10T00:53:56.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-09-10T00:53:56.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-09-10T00:53:56.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-09-10T00:53:56.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: Manager daemon x is now available 2023-09-10T00:53:56.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:56.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:53:57.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T00:53:57.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T00:53:57.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:56 smithi003 ceph-mon[97654]: mgrmap e11: x(active, since 1.00715s) 2023-09-10T00:53:57.169 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2023-09-10T00:53:57.169 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-09-10T00:53:57.169 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2023-09-10T00:53:57.169 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2023-09-10T00:53:57.170 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 9 is available 2023-09-10T00:53:57.170 INFO:teuthology.orchestra.run.smithi003.stdout:Generating a dashboard self-signed certificate... 2023-09-10T00:53:58.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:57 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:56] ENGINE Bus STARTING 2023-09-10T00:53:58.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:57 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:56] ENGINE Serving on http://172.21.15.3:8765 2023-09-10T00:53:58.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:57 smithi003 ceph-mon[97654]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-09-10T00:53:58.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:57 smithi003 ceph-mon[97654]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-09-10T00:53:58.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:57 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:56] ENGINE Serving on https://172.21.15.3:7150 2023-09-10T00:53:58.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:57 smithi003 ceph-mon[97654]: [10/Sep/2023:00:53:56] ENGINE Bus STARTED 2023-09-10T00:53:58.450 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-09-10T00:53:58.451 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial admin user... 2023-09-10T00:53:59.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:58 smithi003 ceph-mon[97654]: mgrmap e12: x(active, since 2s) 2023-09-10T00:53:59.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:58 smithi003 ceph-mon[97654]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:53:59.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:53:59.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:53:59.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:53:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:00.025 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$mmcMUahAh8bx/5OV4dCybekvuRnvVLeRdEw54W57y2BF94GzdzpB.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1694307239, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-09-10T00:54:00.025 INFO:teuthology.orchestra.run.smithi003.stdout:Fetching dashboard port number... 2023-09-10T00:54:00.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:00 smithi003 ceph-mon[97654]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:01.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 8443 2023-09-10T00:54:01.425 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-09-10T00:54:01.425 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2023-09-10T00:54:01.445 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-09-10T00:54:01.445 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2023-09-10T00:54:01.445 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2023-09-10T00:54:01.445 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-09-10T00:54:01.447 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph Dashboard is now available at: 2023-09-10T00:54:01.447 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: URL: https://smithi003.front.sepia.ceph.com:8443/ 2023-09-10T00:54:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: User: admin 2023-09-10T00:54:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: Password: dc6je80jez 2023-09-10T00:54:01.448 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:01.449 INFO:teuthology.orchestra.run.smithi003.stdout:Saving cluster configuration to /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config directory 2023-09-10T00:54:01.449 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling autotune for osd_memory_target 2023-09-10T00:54:01.673 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2164791553' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-09-10T00:54:03.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:03 smithi003 ceph-mon[97654]: mgrmap e13: x(active, since 6s) 2023-09-10T00:54:04.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-09-10T00:54:04.033 INFO:teuthology.orchestra.run.smithi003.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-09-10T00:54:04.033 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.034 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:04.034 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.034 INFO:teuthology.orchestra.run.smithi003.stdout:Or, if you are only running a single cluster on this host: 2023-09-10T00:54:04.034 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.035 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-09-10T00:54:04.035 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.035 INFO:teuthology.orchestra.run.smithi003.stdout:Please consider enabling telemetry to help improve Ceph: 2023-09-10T00:54:04.035 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.035 INFO:teuthology.orchestra.run.smithi003.stdout: ceph telemetry on 2023-09-10T00:54:04.036 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.036 INFO:teuthology.orchestra.run.smithi003.stdout:For more information see: 2023-09-10T00:54:04.036 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.036 INFO:teuthology.orchestra.run.smithi003.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-09-10T00:54:04.036 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:04.037 INFO:teuthology.orchestra.run.smithi003.stdout:Bootstrap complete. 2023-09-10T00:54:04.092 INFO:tasks.cephadm:Fetching config... 2023-09-10T00:54:04.093 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:54:04.093 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-09-10T00:54:04.141 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-09-10T00:54:04.141 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:54:04.142 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-09-10T00:54:04.227 INFO:tasks.cephadm:Fetching mon keyring... 2023-09-10T00:54:04.228 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:54:04.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/keyring of=/dev/stdout 2023-09-10T00:54:04.337 INFO:tasks.cephadm:Fetching pub ssh key... 2023-09-10T00:54:04.338 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:54:04.338 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-09-10T00:54:04.422 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-09-10T00:54:04.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78XTDjJO/eM150Gjc6JMF6S0KDWAGPJoaiSgBBCoxY28V3BBUoFMk9VR9QqvFLjy43ahSiCFLTCba2Ptas8rCqfmOcSaJtrmlQWsWvTYy3d6SdPlUwBtkpC9qCSkeleOpEJunvrGGH1ix8jFiIGE2dzi9W35cBHDIqWtqgr2wi6Uo9TKLkzWQJw1SgkhDtrm7/zUgzlOGGqIIwJKYDzYR7a+T03iqX9NzT/ppcJduCKthGJQD/tv+Ml9S5SynChmcCrSTrc2rFtX8M6Xh76xXJTycJ4DWQ1YsZVG5c/HTl6h5BkX7OfuQZcbFEwmBe8jSfymyxbPDS0kQfHDoeUKztktDNUZNWUwmVckntvepTdlPA+FYfyETT55HcrSm/rT9AAyv+Mtppoq+GAtSr0h0wnyGxm/ajbw/+sTkI+Dln3N+p0fwE+wfKJ9oSryyWWtwWaaGpPnjlnQHP5hZR1YY6wttYtcPbEmhppuNncEq9gtHgUe5Chk5+nDaoHY0EkM= ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-09-10T00:54:04.569 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78XTDjJO/eM150Gjc6JMF6S0KDWAGPJoaiSgBBCoxY28V3BBUoFMk9VR9QqvFLjy43ahSiCFLTCba2Ptas8rCqfmOcSaJtrmlQWsWvTYy3d6SdPlUwBtkpC9qCSkeleOpEJunvrGGH1ix8jFiIGE2dzi9W35cBHDIqWtqgr2wi6Uo9TKLkzWQJw1SgkhDtrm7/zUgzlOGGqIIwJKYDzYR7a+T03iqX9NzT/ppcJduCKthGJQD/tv+Ml9S5SynChmcCrSTrc2rFtX8M6Xh76xXJTycJ4DWQ1YsZVG5c/HTl6h5BkX7OfuQZcbFEwmBe8jSfymyxbPDS0kQfHDoeUKztktDNUZNWUwmVckntvepTdlPA+FYfyETT55HcrSm/rT9AAyv+Mtppoq+GAtSr0h0wnyGxm/ajbw/+sTkI+Dln3N+p0fwE+wfKJ9oSryyWWtwWaaGpPnjlnQHP5hZR1YY6wttYtcPbEmhppuNncEq9gtHgUe5Chk5+nDaoHY0EkM= ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:04.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78XTDjJO/eM150Gjc6JMF6S0KDWAGPJoaiSgBBCoxY28V3BBUoFMk9VR9QqvFLjy43ahSiCFLTCba2Ptas8rCqfmOcSaJtrmlQWsWvTYy3d6SdPlUwBtkpC9qCSkeleOpEJunvrGGH1ix8jFiIGE2dzi9W35cBHDIqWtqgr2wi6Uo9TKLkzWQJw1SgkhDtrm7/zUgzlOGGqIIwJKYDzYR7a+T03iqX9NzT/ppcJduCKthGJQD/tv+Ml9S5SynChmcCrSTrc2rFtX8M6Xh76xXJTycJ4DWQ1YsZVG5c/HTl6h5BkX7OfuQZcbFEwmBe8jSfymyxbPDS0kQfHDoeUKztktDNUZNWUwmVckntvepTdlPA+FYfyETT55HcrSm/rT9AAyv+Mtppoq+GAtSr0h0wnyGxm/ajbw/+sTkI+Dln3N+p0fwE+wfKJ9oSryyWWtwWaaGpPnjlnQHP5hZR1YY6wttYtcPbEmhppuNncEq9gtHgUe5Chk5+nDaoHY0EkM= ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-09-10T00:54:04.643 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78XTDjJO/eM150Gjc6JMF6S0KDWAGPJoaiSgBBCoxY28V3BBUoFMk9VR9QqvFLjy43ahSiCFLTCba2Ptas8rCqfmOcSaJtrmlQWsWvTYy3d6SdPlUwBtkpC9qCSkeleOpEJunvrGGH1ix8jFiIGE2dzi9W35cBHDIqWtqgr2wi6Uo9TKLkzWQJw1SgkhDtrm7/zUgzlOGGqIIwJKYDzYR7a+T03iqX9NzT/ppcJduCKthGJQD/tv+Ml9S5SynChmcCrSTrc2rFtX8M6Xh76xXJTycJ4DWQ1YsZVG5c/HTl6h5BkX7OfuQZcbFEwmBe8jSfymyxbPDS0kQfHDoeUKztktDNUZNWUwmVckntvepTdlPA+FYfyETT55HcrSm/rT9AAyv+Mtppoq+GAtSr0h0wnyGxm/ajbw/+sTkI+Dln3N+p0fwE+wfKJ9oSryyWWtwWaaGpPnjlnQHP5hZR1YY6wttYtcPbEmhppuNncEq9gtHgUe5Chk5+nDaoHY0EkM= ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:04.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78XTDjJO/eM150Gjc6JMF6S0KDWAGPJoaiSgBBCoxY28V3BBUoFMk9VR9QqvFLjy43ahSiCFLTCba2Ptas8rCqfmOcSaJtrmlQWsWvTYy3d6SdPlUwBtkpC9qCSkeleOpEJunvrGGH1ix8jFiIGE2dzi9W35cBHDIqWtqgr2wi6Uo9TKLkzWQJw1SgkhDtrm7/zUgzlOGGqIIwJKYDzYR7a+T03iqX9NzT/ppcJduCKthGJQD/tv+Ml9S5SynChmcCrSTrc2rFtX8M6Xh76xXJTycJ4DWQ1YsZVG5c/HTl6h5BkX7OfuQZcbFEwmBe8jSfymyxbPDS0kQfHDoeUKztktDNUZNWUwmVckntvepTdlPA+FYfyETT55HcrSm/rT9AAyv+Mtppoq+GAtSr0h0wnyGxm/ajbw/+sTkI+Dln3N+p0fwE+wfKJ9oSryyWWtwWaaGpPnjlnQHP5hZR1YY6wttYtcPbEmhppuNncEq9gtHgUe5Chk5+nDaoHY0EkM= ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-09-10T00:54:04.701 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78XTDjJO/eM150Gjc6JMF6S0KDWAGPJoaiSgBBCoxY28V3BBUoFMk9VR9QqvFLjy43ahSiCFLTCba2Ptas8rCqfmOcSaJtrmlQWsWvTYy3d6SdPlUwBtkpC9qCSkeleOpEJunvrGGH1ix8jFiIGE2dzi9W35cBHDIqWtqgr2wi6Uo9TKLkzWQJw1SgkhDtrm7/zUgzlOGGqIIwJKYDzYR7a+T03iqX9NzT/ppcJduCKthGJQD/tv+Ml9S5SynChmcCrSTrc2rFtX8M6Xh76xXJTycJ4DWQ1YsZVG5c/HTl6h5BkX7OfuQZcbFEwmBe8jSfymyxbPDS0kQfHDoeUKztktDNUZNWUwmVckntvepTdlPA+FYfyETT55HcrSm/rT9AAyv+Mtppoq+GAtSr0h0wnyGxm/ajbw/+sTkI+Dln3N+p0fwE+wfKJ9oSryyWWtwWaaGpPnjlnQHP5hZR1YY6wttYtcPbEmhppuNncEq9gtHgUe5Chk5+nDaoHY0EkM= ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:04.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-09-10T00:54:05.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:04 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2260418808' entity='client.admin' 2023-09-10T00:54:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:07.955 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-09-10T00:54:07.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-09-10T00:54:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:08.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:08 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2254196231' entity='client.admin' 2023-09-10T00:54:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi003", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:54:10.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:10.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:10.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:11.102 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi092 2023-09-10T00:54:11.102 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:54:11.103 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.conf 2023-09-10T00:54:11.126 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:54:11.126 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:11.185 INFO:tasks.cephadm:Adding host smithi092 to orchestrator... 2023-09-10T00:54:11.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch host add smithi092 2023-09-10T00:54:11.940 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:12.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:11 smithi003 ceph-mon[97654]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:12.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:12.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:12.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:12.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:12.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:11 smithi003 ceph-mon[97654]: Updating smithi003:/etc/ceph/ceph.conf 2023-09-10T00:54:12.719 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:12 smithi003 ceph-mon[97654]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:12.719 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:12 smithi003 ceph-mon[97654]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:14.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:13 smithi003 ceph-mon[97654]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T00:54:14.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:14.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:14.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:15.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:14 smithi003 ceph-mon[97654]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi092", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:16.006 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi092' with addr '172.21.15.92' 2023-09-10T00:54:16.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:15 smithi003 ceph-mon[97654]: Deploying cephadm binary to smithi092 2023-09-10T00:54:16.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch host ls --format=json 2023-09-10T00:54:17.272 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:17.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:16 smithi003 ceph-mon[97654]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:18.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:17 smithi003 ceph-mon[97654]: Added host smithi092 2023-09-10T00:54:18.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:19.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:18 smithi003 ceph-mon[97654]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:19.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:18 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:19.126 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:19.126 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}] 2023-09-10T00:54:19.548 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2023-09-10T00:54:19.548 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:54:19.549 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2023-09-10T00:54:19.570 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:54:19.570 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:19.629 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2023-09-10T00:54:19.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch host add smithi136 2023-09-10T00:54:20.365 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:20.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:20 smithi003 ceph-mon[97654]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T00:54:20.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:20 smithi003 ceph-mon[97654]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:22.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:22.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:22.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:22.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:22.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:54:22.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:22.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:22.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T00:54:22.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:22.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:22.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:22.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:22.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:22.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:23.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:23 smithi003 ceph-mon[97654]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T00:54:23.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:23 smithi003 ceph-mon[97654]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:23.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:23 smithi003 ceph-mon[97654]: Deploying cephadm binary to smithi136 2023-09-10T00:54:24.654 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi136' with addr '172.21.15.136' 2023-09-10T00:54:24.768 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:24 smithi003 ceph-mon[97654]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:25.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch host ls --format=json 2023-09-10T00:54:25.775 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:26.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:25 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:26.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:25 smithi003 ceph-mon[97654]: Added host smithi136 2023-09-10T00:54:26.025 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:25 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:26.025 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:25 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:26 smithi003 ceph-mon[97654]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:27.350 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:54:27.351 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2023-09-10T00:54:27.759 INFO:tasks.cephadm:Setting crush tunables to default 2023-09-10T00:54:27.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd crush tunables default 2023-09-10T00:54:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:28 smithi003 ceph-mon[97654]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T00:54:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:28 smithi003 ceph-mon[97654]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:30.862 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2023-09-10T00:54:31.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:31.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:31.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:31.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:31.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:31.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:54:31.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:31.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:31.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T00:54:31.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1014505104' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-09-10T00:54:31.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:31.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:31.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:31.272 INFO:tasks.cephadm:Adding mon.a on smithi003 2023-09-10T00:54:31.272 INFO:tasks.cephadm:Adding mon.b on smithi092 2023-09-10T00:54:31.273 INFO:tasks.cephadm:Adding mon.c on smithi136 2023-09-10T00:54:31.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch apply mon '3;smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c' 2023-09-10T00:54:31.848 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:31 smithi003 ceph-mon[97654]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:31 smithi003 ceph-mon[97654]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:31 smithi003 ceph-mon[97654]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T00:54:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:31 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1014505104' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-09-10T00:54:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:31 smithi003 ceph-mon[97654]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:33.123 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mon update... 2023-09-10T00:54:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:32 smithi003 ceph-mon[97654]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:33.513 DEBUG:teuthology.orchestra.run.smithi092:mon.b> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.b.service 2023-09-10T00:54:33.516 DEBUG:teuthology.orchestra.run.smithi136:mon.c> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.c.service 2023-09-10T00:54:33.518 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-09-10T00:54:33.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph mon dump -f json 2023-09-10T00:54:33.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T00:54:33.552 INFO:journalctl@ceph.mon.b.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T00:54:34.269 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:34.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c;count:3 2023-09-10T00:54:34.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:34.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:34.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:34.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:34.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:34.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:34.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:34.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: Deploying daemon mon.c on smithi136 2023-09-10T00:54:34.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:34 smithi003 ceph-mon[97654]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:36.171 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:54:36.171 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","modified":"2023-09-10T00:52:49.540158Z","created":"2023-09-10T00:52:49.540158Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-09-10T00:54:36.173 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2023-09-10T00:54:36.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 podman[133668]: 2023-09-10T00:54:37.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: set uid:gid to 167:167 (ceph:ceph) 2023-09-10T00:54:37.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev), process ceph-mon, pid 7 2023-09-10T00:54:37.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 bash[133668]: e313c55c3f67a73815c8b58a125ba7582c114918aec7abb635a390163fd0ba86 2023-09-10T00:54:37.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pidfile_write: ignore empty --pid-file 2023-09-10T00:54:37.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: load: jerasure load: lrc 2023-09-10T00:54:37.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: RocksDB version: 7.9.2 2023-09-10T00:54:37.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Git sha 0 2023-09-10T00:54:37.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Compile date 2023-09-09 15:16:58 2023-09-10T00:54:37.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: DB SUMMARY 2023-09-10T00:54:37.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: DB Session ID: 2WLWFHBT75NZS4K7TUQM 2023-09-10T00:54:37.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: CURRENT file: CURRENT 2023-09-10T00:54:37.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: IDENTITY file: IDENTITY 2023-09-10T00:54:37.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-09-10T00:54:37.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-09-10T00:54:37.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-09-10T00:54:37.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.error_if_exists: 0 2023-09-10T00:54:37.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.create_if_missing: 0 2023-09-10T00:54:37.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.paranoid_checks: 1 2023-09-10T00:54:37.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.flush_verify_memtable_count: 1 2023-09-10T00:54:37.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-09-10T00:54:37.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-09-10T00:54:37.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.env: 0x5644e6c76da0 2023-09-10T00:54:37.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.fs: PosixFileSystem 2023-09-10T00:54:37.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.info_log: 0x5644e84354e0 2023-09-10T00:54:37.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_file_opening_threads: 16 2023-09-10T00:54:37.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.statistics: (nil) 2023-09-10T00:54:37.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.use_fsync: 0 2023-09-10T00:54:37.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_log_file_size: 0 2023-09-10T00:54:37.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-09-10T00:54:37.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.log_file_time_to_roll: 0 2023-09-10T00:54:37.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.keep_log_file_num: 1000 2023-09-10T00:54:37.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.recycle_log_file_num: 0 2023-09-10T00:54:37.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.allow_fallocate: 1 2023-09-10T00:54:37.174 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.allow_mmap_reads: 0 2023-09-10T00:54:37.174 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.allow_mmap_writes: 0 2023-09-10T00:54:37.174 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.use_direct_reads: 0 2023-09-10T00:54:37.174 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-09-10T00:54:37.174 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.create_missing_column_families: 0 2023-09-10T00:54:37.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.db_log_dir: 2023-09-10T00:54:37.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.wal_dir: 2023-09-10T00:54:37.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.table_cache_numshardbits: 6 2023-09-10T00:54:37.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.WAL_ttl_seconds: 0 2023-09-10T00:54:37.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.WAL_size_limit_MB: 0 2023-09-10T00:54:37.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-09-10T00:54:37.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-09-10T00:54:37.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.is_fd_close_on_exec: 1 2023-09-10T00:54:37.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.advise_random_on_open: 1 2023-09-10T00:54:37.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.db_write_buffer_size: 0 2023-09-10T00:54:37.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.write_buffer_manager: 0x5644e83ef4a0 2023-09-10T00:54:37.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-09-10T00:54:37.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-09-10T00:54:37.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.use_adaptive_mutex: 0 2023-09-10T00:54:37.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.rate_limiter: (nil) 2023-09-10T00:54:37.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-09-10T00:54:37.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.wal_recovery_mode: 2 2023-09-10T00:54:37.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.enable_thread_tracking: 0 2023-09-10T00:54:37.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.enable_pipelined_write: 0 2023-09-10T00:54:37.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.unordered_write: 0 2023-09-10T00:54:37.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-09-10T00:54:37.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-09-10T00:54:37.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-09-10T00:54:37.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-09-10T00:54:37.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.row_cache: None 2023-09-10T00:54:37.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.wal_filter: None 2023-09-10T00:54:37.180 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-09-10T00:54:37.180 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.allow_ingest_behind: 0 2023-09-10T00:54:37.180 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.two_write_queues: 0 2023-09-10T00:54:37.180 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.manual_wal_flush: 0 2023-09-10T00:54:37.180 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.wal_compression: 0 2023-09-10T00:54:37.181 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.atomic_flush: 0 2023-09-10T00:54:37.181 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-09-10T00:54:37.181 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.persist_stats_to_disk: 0 2023-09-10T00:54:37.181 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.write_dbid_to_manifest: 0 2023-09-10T00:54:37.181 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.log_readahead_size: 0 2023-09-10T00:54:37.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-09-10T00:54:37.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.best_efforts_recovery: 0 2023-09-10T00:54:37.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-09-10T00:54:37.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-09-10T00:54:37.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.allow_data_in_errors: 0 2023-09-10T00:54:37.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.db_host_id: __hostname__ 2023-09-10T00:54:37.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.enforce_single_del_contracts: true 2023-09-10T00:54:37.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_background_jobs: 2 2023-09-10T00:54:37.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_background_compactions: -1 2023-09-10T00:54:37.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_subcompactions: 1 2023-09-10T00:54:37.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-09-10T00:54:37.184 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-09-10T00:54:37.184 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.delayed_write_rate : 16777216 2023-09-10T00:54:37.184 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_total_wal_size: 0 2023-09-10T00:54:37.184 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-09-10T00:54:37.184 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.stats_dump_period_sec: 600 2023-09-10T00:54:37.185 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.stats_persist_period_sec: 600 2023-09-10T00:54:37.185 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-09-10T00:54:37.185 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_open_files: -1 2023-09-10T00:54:37.185 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bytes_per_sync: 0 2023-09-10T00:54:37.185 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.wal_bytes_per_sync: 0 2023-09-10T00:54:37.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.strict_bytes_per_sync: 0 2023-09-10T00:54:37.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_readahead_size: 0 2023-09-10T00:54:37.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_background_flushes: -1 2023-09-10T00:54:37.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Compression algorithms supported: 2023-09-10T00:54:37.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-09-10T00:54:37.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kZSTD supported: 0 2023-09-10T00:54:37.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kXpressCompression supported: 0 2023-09-10T00:54:37.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kLZ4HCCompression supported: 1 2023-09-10T00:54:37.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kZlibCompression supported: 1 2023-09-10T00:54:37.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kSnappyCompression supported: 1 2023-09-10T00:54:37.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kLZ4Compression supported: 1 2023-09-10T00:54:37.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: kBZip2Compression supported: 0 2023-09-10T00:54:37.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Fast CRC32 supported: Supported on x86 2023-09-10T00:54:37.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: DMutex implementation: pthread_mutex_t 2023-09-10T00:54:37.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-09-10T00:54:37.188 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-09-10T00:54:37.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-09-10T00:54:37.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.merge_operator: 2023-09-10T00:54:37.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_filter: None 2023-09-10T00:54:37.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_filter_factory: None 2023-09-10T00:54:37.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.sst_partitioner_factory: None 2023-09-10T00:54:37.190 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.memtable_factory: SkipListFactory 2023-09-10T00:54:37.190 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.table_factory: BlockBasedTable 2023-09-10T00:54:37.190 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5644e84355e0) 2023-09-10T00:54:37.190 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks: 1 2023-09-10T00:54:37.190 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-09-10T00:54:37.191 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-09-10T00:54:37.191 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_top_level_index_and_filter: 1 2023-09-10T00:54:37.191 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_type: 0 2023-09-10T00:54:37.191 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_index_type: 0 2023-09-10T00:54:37.191 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_shortening: 1 2023-09-10T00:54:37.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2023-09-10T00:54:37.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: checksum: 4 2023-09-10T00:54:37.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: no_block_cache: 0 2023-09-10T00:54:37.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache: 0x5644e848d1f0 2023-09-10T00:54:37.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_name: BinnedLRUCache 2023-09-10T00:54:37.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_options: 2023-09-10T00:54:37.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: capacity : 536870912 2023-09-10T00:54:37.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_shard_bits : 4 2023-09-10T00:54:37.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: strict_capacity_limit : 0 2023-09-10T00:54:37.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: high_pri_pool_ratio: 0.000 2023-09-10T00:54:37.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_compressed: (nil) 2023-09-10T00:54:37.194 INFO:journalctl@ceph.mon.c.smithi136.stdout: persistent_cache: (nil) 2023-09-10T00:54:37.194 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size: 4096 2023-09-10T00:54:37.194 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size_deviation: 10 2023-09-10T00:54:37.194 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_restart_interval: 16 2023-09-10T00:54:37.194 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_block_restart_interval: 1 2023-09-10T00:54:37.195 INFO:journalctl@ceph.mon.c.smithi136.stdout: metadata_block_size: 4096 2023-09-10T00:54:37.195 INFO:journalctl@ceph.mon.c.smithi136.stdout: partition_filters: 0 2023-09-10T00:54:37.195 INFO:journalctl@ceph.mon.c.smithi136.stdout: use_delta_encoding: 1 2023-09-10T00:54:37.195 INFO:journalctl@ceph.mon.c.smithi136.stdout: filter_policy: bloomfilter 2023-09-10T00:54:37.195 INFO:journalctl@ceph.mon.c.smithi136.stdout: whole_key_filtering: 1 2023-09-10T00:54:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout: verify_compression: 0 2023-09-10T00:54:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout: read_amp_bytes_per_bit: 0 2023-09-10T00:54:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout: format_version: 5 2023-09-10T00:54:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_index_compression: 1 2023-09-10T00:54:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_align: 0 2023-09-10T00:54:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_auto_readahead_size: 262144 2023-09-10T00:54:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout: prepopulate_block_cache: 0 2023-09-10T00:54:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout: initial_auto_readahead_size: 8192 2023-09-10T00:54:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2023-09-10T00:54:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.write_buffer_size: 33554432 2023-09-10T00:54:37.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_write_buffer_number: 2 2023-09-10T00:54:37.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression: NoCompression 2023-09-10T00:54:37.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression: Disabled 2023-09-10T00:54:37.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.prefix_extractor: nullptr 2023-09-10T00:54:37.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-09-10T00:54:37.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.num_levels: 7 2023-09-10T00:54:37.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-09-10T00:54:37.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-09-10T00:54:37.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-09-10T00:54:37.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-09-10T00:54:37.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-09-10T00:54:37.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-09-10T00:54:37.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-09-10T00:54:37.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-09-10T00:54:37.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-09-10T00:54:37.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-09-10T00:54:37.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-09-10T00:54:37.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-09-10T00:54:37.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.window_bits: -14 2023-09-10T00:54:37.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.level: 32767 2023-09-10T00:54:37.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.strategy: 0 2023-09-10T00:54:37.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-09-10T00:54:37.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-09-10T00:54:37.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-09-10T00:54:37.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-09-10T00:54:37.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.enabled: false 2023-09-10T00:54:37.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-09-10T00:54:37.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-09-10T00:54:37.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-09-10T00:54:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-09-10T00:54:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.target_file_size_base: 67108864 2023-09-10T00:54:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.target_file_size_multiplier: 1 2023-09-10T00:54:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-09-10T00:54:37.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-09-10T00:54:37.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-09-10T00:54:37.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-09-10T00:54:37.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-09-10T00:54:37.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-09-10T00:54:37.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-09-10T00:54:37.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-09-10T00:54:37.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-09-10T00:54:37.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-09-10T00:54:37.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-09-10T00:54:37.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-09-10T00:54:37.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-09-10T00:54:37.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.arena_block_size: 1048576 2023-09-10T00:54:37.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-09-10T00:54:37.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-09-10T00:54:37.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.disable_auto_compactions: 0 2023-09-10T00:54:37.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-09-10T00:54:37.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-09-10T00:54:37.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-09-10T00:54:37.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-09-10T00:54:37.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-09-10T00:54:37.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-09-10T00:54:37.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-09-10T00:54:37.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-09-10T00:54:37.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-09-10T00:54:37.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-09-10T00:54:37.210 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.table_properties_collectors: 2023-09-10T00:54:37.210 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.inplace_update_support: 0 2023-09-10T00:54:37.210 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.inplace_update_num_locks: 10000 2023-09-10T00:54:37.210 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-09-10T00:54:37.210 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-09-10T00:54:37.211 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.memtable_huge_page_size: 0 2023-09-10T00:54:37.211 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.bloom_locality: 0 2023-09-10T00:54:37.211 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.max_successive_merges: 0 2023-09-10T00:54:37.211 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.optimize_filters_for_hits: 0 2023-09-10T00:54:37.211 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.paranoid_file_checks: 0 2023-09-10T00:54:37.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.force_consistency_checks: 1 2023-09-10T00:54:37.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.report_bg_io_stats: 0 2023-09-10T00:54:37.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.ttl: 2592000 2023-09-10T00:54:37.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.periodic_compaction_seconds: 0 2023-09-10T00:54:37.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-09-10T00:54:37.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-09-10T00:54:37.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.enable_blob_files: false 2023-09-10T00:54:37.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.min_blob_size: 0 2023-09-10T00:54:37.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.blob_file_size: 268435456 2023-09-10T00:54:37.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.blob_compression_type: NoCompression 2023-09-10T00:54:37.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.enable_blob_garbage_collection: false 2023-09-10T00:54:37.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-09-10T00:54:37.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-09-10T00:54:37.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-09-10T00:54:37.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.blob_file_starting_level: 0 2023-09-10T00:54:37.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-09-10T00:54:37.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-09-10T00:54:37.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-09-10T00:54:37.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4b674059-b9c0-4fa8-ac36-ae4141d13f29 2023-09-10T00:54:37.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307276842846, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-09-10T00:54:37.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-09-10T00:54:37.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307276843434, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1694307276, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4b674059-b9c0-4fa8-ac36-ae4141d13f29", "db_session_id": "2WLWFHBT75NZS4K7TUQM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-09-10T00:54:37.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307276843525, "job": 1, "event": "recovery_finished"} 2023-09-10T00:54:37.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-09-10T00:54:37.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-09-10T00:54:37.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5644e8528000 2023-09-10T00:54:37.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: DB pointer 0x5644e8514000 2023-09-10T00:54:37.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-09-10T00:54:37.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-09-10T00:54:37.217 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** DB Stats ** 2023-09-10T00:54:37.217 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-09-10T00:54:37.218 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-09-10T00:54:37.218 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-09-10T00:54:37.218 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-09-10T00:54:37.218 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-09-10T00:54:37.218 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-09-10T00:54:37.219 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-09-10T00:54:37.219 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2023-09-10T00:54:37.219 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2023-09-10T00:54:37.219 INFO:journalctl@ceph.mon.c.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-09-10T00:54:37.219 INFO:journalctl@ceph.mon.c.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-09-10T00:54:37.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:54:37.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:54:37.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:54:37.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2023-09-10T00:54:37.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2023-09-10T00:54:37.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-09-10T00:54:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-09-10T00:54:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-09-10T00:54:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2023-09-10T00:54:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-09-10T00:54:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2023-09-10T00:54:37.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-09-10T00:54:37.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-09-10T00:54:37.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-09-10T00:54:37.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-09-10T00:54:37.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-09-10T00:54:37.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2023-09-10T00:54:37.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-09-10T00:54:37.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-09-10T00:54:37.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-09-10T00:54:37.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache BinnedLRUCache@0x5644e848d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-09-10T00:54:37.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-09-10T00:54:37.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2023-09-10T00:54:37.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2023-09-10T00:54:37.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-09-10T00:54:37.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: using public_addr v2:172.21.15.136:0/0 -> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2023-09-10T00:54:37.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: starting mon.c rank -1 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:37.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(???) e0 preinit fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:37.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).mds e1 new map 2023-09-10T00:54:37.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).mds e1 print_map 2023-09-10T00:54:37.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: e1 2023-09-10T00:54:37.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-09-10T00:54:37.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-09-10T00:54:37.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: legacy client fscid: -1 2023-09-10T00:54:37.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2023-09-10T00:54:37.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: No filesystems configured 2023-09-10T00:54:37.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-09-10T00:54:37.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-09-10T00:54:37.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-09-10T00:54:37.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-09-10T00:54:37.227 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-09-10T00:54:37.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-09-10T00:54:37.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-09-10T00:54:37.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:54:37.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:54:37.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:54:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T00:54:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:54:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:37.230 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:37.231 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T00:54:37.231 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:37.231 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.231 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:37.231 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.232 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.232 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.232 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T00:54:37.232 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:37.232 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Deploying cephadm binary to smithi136 2023-09-10T00:54:37.233 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.233 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.233 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Added host smithi136 2023-09-10T00:54:37.233 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:37.233 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.234 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.234 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.234 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.234 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T00:54:37.234 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.235 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.235 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.235 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.235 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.235 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.235 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:54:37.236 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:37.236 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:37.236 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T00:54:37.236 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1014505104' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-09-10T00:54:37.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:37.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:37.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T00:54:37.238 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1014505104' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-09-10T00:54:37.238 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:37.238 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.238 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:37.238 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c;count:3 2023-09-10T00:54:37.239 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.239 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:37.239 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:37.239 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:37.239 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:37.240 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:37.240 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:37.240 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: Deploying daemon mon.c on smithi136 2023-09-10T00:54:37.240 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.240 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.241 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/3079757541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:37.241 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-09-10T00:54:37.241 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expand map: {default=false} 2023-09-10T00:54:37.241 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta from 'false' to 'false' 2023-09-10T00:54:37.241 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expanded map: {default=false} 2023-09-10T00:54:37.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expand map: {default=info} 2023-09-10T00:54:37.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta from 'info' to 'info' 2023-09-10T00:54:37.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expanded map: {default=info} 2023-09-10T00:54:37.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expand map: {default=daemon} 2023-09-10T00:54:37.242 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta from 'daemon' to 'daemon' 2023-09-10T00:54:37.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expanded map: {default=daemon} 2023-09-10T00:54:37.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expand map: {default=debug} 2023-09-10T00:54:37.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta from 'debug' to 'debug' 2023-09-10T00:54:37.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: expand_channel_meta expanded map: {default=debug} 2023-09-10T00:54:37.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 ceph-mon[133708]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-09-10T00:54:37.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:36 smithi136 systemd[1]: Started Ceph mon.c for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:54:37.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:36 smithi003 ceph-mon[97654]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:37.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:36 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/3079757541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:37.550 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-09-10T00:54:37.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph mon dump -f json 2023-09-10T00:54:38.110 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:54:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:54:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: mon.a calling monitor election 2023-09-10T00:54:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: mon.c calling monitor election 2023-09-10T00:54:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-09-10T00:54:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: fsmap 2023-09-10T00:54:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: mgrmap e13: x(active, since 46s) 2023-09-10T00:54:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: overall HEALTH_OK 2023-09-10T00:54:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:42.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:42.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:42.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:42.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:54:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: mon.a calling monitor election 2023-09-10T00:54:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: mon.c calling monitor election 2023-09-10T00:54:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-09-10T00:54:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: fsmap 2023-09-10T00:54:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: mgrmap e13: x(active, since 46s) 2023-09-10T00:54:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: overall HEALTH_OK 2023-09-10T00:54:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:42.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:42.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:42.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:42.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:42.462 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:54:42.462 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","modified":"2023-09-10T00:54:36.871784Z","created":"2023-09-10T00:52:49.540158Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-09-10T00:54:42.465 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 2 2023-09-10T00:54:43.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:42 smithi136 ceph-mon[133708]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:43.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:42 smithi136 ceph-mon[133708]: Deploying daemon mon.b on smithi092 2023-09-10T00:54:43.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:42 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/1468951415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:43.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:42 smithi003 ceph-mon[97654]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:42 smithi003 ceph-mon[97654]: Deploying daemon mon.b on smithi092 2023-09-10T00:54:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:42 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/1468951415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:43.870 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-09-10T00:54:43.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph mon dump -f json 2023-09-10T00:54:44.436 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:54:44.663 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 systemd[1]: Starting Ceph mon.b for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T00:54:44.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 podman[134027]: 2023-09-10T00:54:44.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: set uid:gid to 167:167 (ceph:ceph) 2023-09-10T00:54:44.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev), process ceph-mon, pid 7 2023-09-10T00:54:44.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pidfile_write: ignore empty --pid-file 2023-09-10T00:54:44.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 bash[134027]: f9e8feb9fa289434e8bc295b1ecd612bd6af1546c67ce9707248de3989fd1def 2023-09-10T00:54:44.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: load: jerasure load: lrc 2023-09-10T00:54:44.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: RocksDB version: 7.9.2 2023-09-10T00:54:44.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Git sha 0 2023-09-10T00:54:44.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Compile date 2023-09-09 15:16:58 2023-09-10T00:54:44.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: DB SUMMARY 2023-09-10T00:54:44.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: DB Session ID: 0GZU0EK7AO9XLBLQEO7P 2023-09-10T00:54:44.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: CURRENT file: CURRENT 2023-09-10T00:54:44.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: IDENTITY file: IDENTITY 2023-09-10T00:54:44.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-09-10T00:54:44.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-09-10T00:54:44.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-09-10T00:54:44.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.error_if_exists: 0 2023-09-10T00:54:44.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.create_if_missing: 0 2023-09-10T00:54:44.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.paranoid_checks: 1 2023-09-10T00:54:44.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.flush_verify_memtable_count: 1 2023-09-10T00:54:44.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-09-10T00:54:44.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-09-10T00:54:44.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.env: 0x55f46e337da0 2023-09-10T00:54:44.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.fs: PosixFileSystem 2023-09-10T00:54:44.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.info_log: 0x55f46f2694e0 2023-09-10T00:54:44.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_file_opening_threads: 16 2023-09-10T00:54:44.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.statistics: (nil) 2023-09-10T00:54:44.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.use_fsync: 0 2023-09-10T00:54:44.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_log_file_size: 0 2023-09-10T00:54:44.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-09-10T00:54:44.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.log_file_time_to_roll: 0 2023-09-10T00:54:44.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.keep_log_file_num: 1000 2023-09-10T00:54:44.973 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.recycle_log_file_num: 0 2023-09-10T00:54:44.973 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.allow_fallocate: 1 2023-09-10T00:54:44.973 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.allow_mmap_reads: 0 2023-09-10T00:54:44.973 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.allow_mmap_writes: 0 2023-09-10T00:54:44.973 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.use_direct_reads: 0 2023-09-10T00:54:44.974 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-09-10T00:54:44.974 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.create_missing_column_families: 0 2023-09-10T00:54:44.974 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.db_log_dir: 2023-09-10T00:54:44.974 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.wal_dir: 2023-09-10T00:54:44.974 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.table_cache_numshardbits: 6 2023-09-10T00:54:44.974 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.WAL_ttl_seconds: 0 2023-09-10T00:54:44.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.WAL_size_limit_MB: 0 2023-09-10T00:54:44.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-09-10T00:54:44.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-09-10T00:54:44.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.is_fd_close_on_exec: 1 2023-09-10T00:54:44.975 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.advise_random_on_open: 1 2023-09-10T00:54:44.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.db_write_buffer_size: 0 2023-09-10T00:54:44.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.write_buffer_manager: 0x55f46f2234a0 2023-09-10T00:54:44.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-09-10T00:54:44.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-09-10T00:54:44.976 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.use_adaptive_mutex: 0 2023-09-10T00:54:44.977 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.rate_limiter: (nil) 2023-09-10T00:54:44.977 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-09-10T00:54:44.977 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.wal_recovery_mode: 2 2023-09-10T00:54:44.977 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.enable_thread_tracking: 0 2023-09-10T00:54:44.977 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.enable_pipelined_write: 0 2023-09-10T00:54:44.978 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.unordered_write: 0 2023-09-10T00:54:44.978 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-09-10T00:54:44.978 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-09-10T00:54:44.978 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-09-10T00:54:44.978 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-09-10T00:54:44.979 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.row_cache: None 2023-09-10T00:54:44.979 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.wal_filter: None 2023-09-10T00:54:44.979 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-09-10T00:54:44.979 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.allow_ingest_behind: 0 2023-09-10T00:54:44.979 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.two_write_queues: 0 2023-09-10T00:54:44.979 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.manual_wal_flush: 0 2023-09-10T00:54:44.980 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.wal_compression: 0 2023-09-10T00:54:44.980 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.atomic_flush: 0 2023-09-10T00:54:44.980 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-09-10T00:54:44.980 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.persist_stats_to_disk: 0 2023-09-10T00:54:44.980 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.write_dbid_to_manifest: 0 2023-09-10T00:54:44.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.log_readahead_size: 0 2023-09-10T00:54:44.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-09-10T00:54:44.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.best_efforts_recovery: 0 2023-09-10T00:54:44.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-09-10T00:54:44.981 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-09-10T00:54:44.983 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.allow_data_in_errors: 0 2023-09-10T00:54:44.983 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.db_host_id: __hostname__ 2023-09-10T00:54:44.983 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.enforce_single_del_contracts: true 2023-09-10T00:54:44.984 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_background_jobs: 2 2023-09-10T00:54:44.984 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_background_compactions: -1 2023-09-10T00:54:44.984 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_subcompactions: 1 2023-09-10T00:54:44.984 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-09-10T00:54:44.984 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-09-10T00:54:44.985 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.delayed_write_rate : 16777216 2023-09-10T00:54:44.985 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_total_wal_size: 0 2023-09-10T00:54:44.985 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-09-10T00:54:44.985 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.stats_dump_period_sec: 600 2023-09-10T00:54:44.985 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.stats_persist_period_sec: 600 2023-09-10T00:54:44.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-09-10T00:54:44.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_open_files: -1 2023-09-10T00:54:44.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bytes_per_sync: 0 2023-09-10T00:54:44.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.wal_bytes_per_sync: 0 2023-09-10T00:54:44.986 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.strict_bytes_per_sync: 0 2023-09-10T00:54:44.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_readahead_size: 0 2023-09-10T00:54:44.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_background_flushes: -1 2023-09-10T00:54:44.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Compression algorithms supported: 2023-09-10T00:54:44.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-09-10T00:54:44.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kZSTD supported: 0 2023-09-10T00:54:44.987 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kXpressCompression supported: 0 2023-09-10T00:54:44.988 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kLZ4HCCompression supported: 1 2023-09-10T00:54:44.988 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kZlibCompression supported: 1 2023-09-10T00:54:44.988 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kSnappyCompression supported: 1 2023-09-10T00:54:44.988 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kLZ4Compression supported: 1 2023-09-10T00:54:44.988 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: kBZip2Compression supported: 0 2023-09-10T00:54:44.989 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Fast CRC32 supported: Supported on x86 2023-09-10T00:54:44.989 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: DMutex implementation: pthread_mutex_t 2023-09-10T00:54:44.989 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-09-10T00:54:44.989 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-09-10T00:54:44.989 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-09-10T00:54:44.989 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.merge_operator: 2023-09-10T00:54:44.990 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_filter: None 2023-09-10T00:54:44.990 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_filter_factory: None 2023-09-10T00:54:44.990 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.sst_partitioner_factory: None 2023-09-10T00:54:44.990 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.memtable_factory: SkipListFactory 2023-09-10T00:54:44.990 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.table_factory: BlockBasedTable 2023-09-10T00:54:44.991 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f46f2695e0) 2023-09-10T00:54:44.991 INFO:journalctl@ceph.mon.b.smithi092.stdout: cache_index_and_filter_blocks: 1 2023-09-10T00:54:44.991 INFO:journalctl@ceph.mon.b.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-09-10T00:54:44.991 INFO:journalctl@ceph.mon.b.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-09-10T00:54:44.991 INFO:journalctl@ceph.mon.b.smithi092.stdout: pin_top_level_index_and_filter: 1 2023-09-10T00:54:44.992 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_type: 0 2023-09-10T00:54:44.992 INFO:journalctl@ceph.mon.b.smithi092.stdout: data_block_index_type: 0 2023-09-10T00:54:44.992 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_shortening: 1 2023-09-10T00:54:44.992 INFO:journalctl@ceph.mon.b.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2023-09-10T00:54:44.992 INFO:journalctl@ceph.mon.b.smithi092.stdout: checksum: 4 2023-09-10T00:54:44.993 INFO:journalctl@ceph.mon.b.smithi092.stdout: no_block_cache: 0 2023-09-10T00:54:44.993 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache: 0x55f46f2c11f0 2023-09-10T00:54:44.993 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_name: BinnedLRUCache 2023-09-10T00:54:44.993 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_options: 2023-09-10T00:54:44.993 INFO:journalctl@ceph.mon.b.smithi092.stdout: capacity : 536870912 2023-09-10T00:54:44.994 INFO:journalctl@ceph.mon.b.smithi092.stdout: num_shard_bits : 4 2023-09-10T00:54:44.994 INFO:journalctl@ceph.mon.b.smithi092.stdout: strict_capacity_limit : 0 2023-09-10T00:54:44.994 INFO:journalctl@ceph.mon.b.smithi092.stdout: high_pri_pool_ratio: 0.000 2023-09-10T00:54:44.994 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_compressed: (nil) 2023-09-10T00:54:44.994 INFO:journalctl@ceph.mon.b.smithi092.stdout: persistent_cache: (nil) 2023-09-10T00:54:44.994 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_size: 4096 2023-09-10T00:54:44.995 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_size_deviation: 10 2023-09-10T00:54:44.995 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_restart_interval: 16 2023-09-10T00:54:44.995 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_block_restart_interval: 1 2023-09-10T00:54:44.995 INFO:journalctl@ceph.mon.b.smithi092.stdout: metadata_block_size: 4096 2023-09-10T00:54:44.995 INFO:journalctl@ceph.mon.b.smithi092.stdout: partition_filters: 0 2023-09-10T00:54:44.996 INFO:journalctl@ceph.mon.b.smithi092.stdout: use_delta_encoding: 1 2023-09-10T00:54:44.996 INFO:journalctl@ceph.mon.b.smithi092.stdout: filter_policy: bloomfilter 2023-09-10T00:54:44.996 INFO:journalctl@ceph.mon.b.smithi092.stdout: whole_key_filtering: 1 2023-09-10T00:54:44.996 INFO:journalctl@ceph.mon.b.smithi092.stdout: verify_compression: 0 2023-09-10T00:54:44.996 INFO:journalctl@ceph.mon.b.smithi092.stdout: read_amp_bytes_per_bit: 0 2023-09-10T00:54:44.997 INFO:journalctl@ceph.mon.b.smithi092.stdout: format_version: 5 2023-09-10T00:54:44.997 INFO:journalctl@ceph.mon.b.smithi092.stdout: enable_index_compression: 1 2023-09-10T00:54:44.997 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_align: 0 2023-09-10T00:54:44.997 INFO:journalctl@ceph.mon.b.smithi092.stdout: max_auto_readahead_size: 262144 2023-09-10T00:54:44.997 INFO:journalctl@ceph.mon.b.smithi092.stdout: prepopulate_block_cache: 0 2023-09-10T00:54:44.997 INFO:journalctl@ceph.mon.b.smithi092.stdout: initial_auto_readahead_size: 8192 2023-09-10T00:54:44.998 INFO:journalctl@ceph.mon.b.smithi092.stdout: num_file_reads_for_auto_readahead: 2 2023-09-10T00:54:44.998 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.write_buffer_size: 33554432 2023-09-10T00:54:44.998 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_write_buffer_number: 2 2023-09-10T00:54:44.998 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression: NoCompression 2023-09-10T00:54:44.998 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression: Disabled 2023-09-10T00:54:44.999 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.prefix_extractor: nullptr 2023-09-10T00:54:44.999 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-09-10T00:54:44.999 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.num_levels: 7 2023-09-10T00:54:44.999 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-09-10T00:54:44.999 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-09-10T00:54:45.000 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-09-10T00:54:45.000 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-09-10T00:54:45.000 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-09-10T00:54:45.000 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-09-10T00:54:45.000 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-09-10T00:54:45.001 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-09-10T00:54:45.001 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-09-10T00:54:45.001 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-09-10T00:54:45.001 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-09-10T00:54:45.001 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-09-10T00:54:45.001 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.window_bits: -14 2023-09-10T00:54:45.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.level: 32767 2023-09-10T00:54:45.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.strategy: 0 2023-09-10T00:54:45.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-09-10T00:54:45.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-09-10T00:54:45.002 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-09-10T00:54:45.003 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-09-10T00:54:45.003 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.enabled: false 2023-09-10T00:54:45.003 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-09-10T00:54:45.003 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-09-10T00:54:45.003 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-09-10T00:54:45.004 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-09-10T00:54:45.004 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.target_file_size_base: 67108864 2023-09-10T00:54:45.004 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.target_file_size_multiplier: 1 2023-09-10T00:54:45.004 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-09-10T00:54:45.004 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-09-10T00:54:45.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-09-10T00:54:45.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-09-10T00:54:45.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-09-10T00:54:45.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-09-10T00:54:45.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-09-10T00:54:45.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-09-10T00:54:45.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-09-10T00:54:45.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-09-10T00:54:45.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-09-10T00:54:45.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-09-10T00:54:45.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-09-10T00:54:45.007 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.arena_block_size: 1048576 2023-09-10T00:54:45.007 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-09-10T00:54:45.007 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-09-10T00:54:45.007 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.disable_auto_compactions: 0 2023-09-10T00:54:45.007 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-09-10T00:54:45.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-09-10T00:54:45.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-09-10T00:54:45.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-09-10T00:54:45.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-09-10T00:54:45.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-09-10T00:54:45.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-09-10T00:54:45.009 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-09-10T00:54:45.009 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-09-10T00:54:45.009 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-09-10T00:54:45.009 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.table_properties_collectors: 2023-09-10T00:54:45.009 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.inplace_update_support: 0 2023-09-10T00:54:45.010 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.inplace_update_num_locks: 10000 2023-09-10T00:54:45.010 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-09-10T00:54:45.010 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-09-10T00:54:45.010 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.memtable_huge_page_size: 0 2023-09-10T00:54:45.010 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.bloom_locality: 0 2023-09-10T00:54:45.011 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.max_successive_merges: 0 2023-09-10T00:54:45.011 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.optimize_filters_for_hits: 0 2023-09-10T00:54:45.011 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.paranoid_file_checks: 0 2023-09-10T00:54:45.011 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.force_consistency_checks: 1 2023-09-10T00:54:45.011 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.report_bg_io_stats: 0 2023-09-10T00:54:45.012 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.ttl: 2592000 2023-09-10T00:54:45.012 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.periodic_compaction_seconds: 0 2023-09-10T00:54:45.012 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-09-10T00:54:45.012 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-09-10T00:54:45.012 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.enable_blob_files: false 2023-09-10T00:54:45.012 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.min_blob_size: 0 2023-09-10T00:54:45.013 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.blob_file_size: 268435456 2023-09-10T00:54:45.013 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.blob_compression_type: NoCompression 2023-09-10T00:54:45.013 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.enable_blob_garbage_collection: false 2023-09-10T00:54:45.013 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-09-10T00:54:45.013 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-09-10T00:54:45.014 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-09-10T00:54:45.014 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.blob_file_starting_level: 0 2023-09-10T00:54:45.014 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-09-10T00:54:45.014 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-09-10T00:54:45.014 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-09-10T00:54:45.015 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a52df050-3816-4dcb-985f-671cf3b1b3aa 2023-09-10T00:54:45.015 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307284845958, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-09-10T00:54:45.015 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-09-10T00:54:45.015 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307284846374, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1694307284, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a52df050-3816-4dcb-985f-671cf3b1b3aa", "db_session_id": "0GZU0EK7AO9XLBLQEO7P", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-09-10T00:54:45.015 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: EVENT_LOG_v1 {"time_micros": 1694307284846442, "job": 1, "event": "recovery_finished"} 2023-09-10T00:54:45.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-09-10T00:54:45.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-09-10T00:54:45.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f46f35c000 2023-09-10T00:54:45.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: DB pointer 0x55f46f348000 2023-09-10T00:54:45.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-09-10T00:54:45.017 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-09-10T00:54:45.017 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** DB Stats ** 2023-09-10T00:54:45.017 INFO:journalctl@ceph.mon.b.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-09-10T00:54:45.017 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-09-10T00:54:45.017 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-09-10T00:54:45.017 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-09-10T00:54:45.018 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-09-10T00:54:45.018 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-09-10T00:54:45.018 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-09-10T00:54:45.018 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-09-10T00:54:45.018 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2023-09-10T00:54:45.019 INFO:journalctl@ceph.mon.b.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-09-10T00:54:45.019 INFO:journalctl@ceph.mon.b.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-09-10T00:54:45.019 INFO:journalctl@ceph.mon.b.smithi092.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-09-10T00:54:45.019 INFO:journalctl@ceph.mon.b.smithi092.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-09-10T00:54:45.019 INFO:journalctl@ceph.mon.b.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-09-10T00:54:45.020 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-09-10T00:54:45.020 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2023-09-10T00:54:45.020 INFO:journalctl@ceph.mon.b.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-09-10T00:54:45.020 INFO:journalctl@ceph.mon.b.smithi092.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-09-10T00:54:45.020 INFO:journalctl@ceph.mon.b.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-09-10T00:54:45.021 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-09-10T00:54:45.021 INFO:journalctl@ceph.mon.b.smithi092.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-09-10T00:54:45.021 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-09-10T00:54:45.021 INFO:journalctl@ceph.mon.b.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-09-10T00:54:45.021 INFO:journalctl@ceph.mon.b.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-09-10T00:54:45.022 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-09-10T00:54:45.022 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-09-10T00:54:45.022 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-09-10T00:54:45.022 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2023-09-10T00:54:45.022 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-09-10T00:54:45.023 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-09-10T00:54:45.023 INFO:journalctl@ceph.mon.b.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-09-10T00:54:45.023 INFO:journalctl@ceph.mon.b.smithi092.stdout: Block cache BinnedLRUCache@0x55f46f2c11f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-09-10T00:54:45.023 INFO:journalctl@ceph.mon.b.smithi092.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-09-10T00:54:45.023 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-09-10T00:54:45.023 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2023-09-10T00:54:45.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-09-10T00:54:45.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: using public_addr v2:172.21.15.92:0/0 -> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-09-10T00:54:45.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: starting mon.b rank -1 at public addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] at bind addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:45.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(???) e0 preinit fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:45.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 systemd[1]: Started Ceph mon.b for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:54:45.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).mds e1 new map 2023-09-10T00:54:45.025 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).mds e1 print_map 2023-09-10T00:54:45.025 INFO:journalctl@ceph.mon.b.smithi092.stdout: e1 2023-09-10T00:54:45.025 INFO:journalctl@ceph.mon.b.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-09-10T00:54:45.025 INFO:journalctl@ceph.mon.b.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-09-10T00:54:45.025 INFO:journalctl@ceph.mon.b.smithi092.stdout: legacy client fscid: -1 2023-09-10T00:54:45.026 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-09-10T00:54:45.026 INFO:journalctl@ceph.mon.b.smithi092.stdout: No filesystems configured 2023-09-10T00:54:45.026 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-09-10T00:54:45.026 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-09-10T00:54:45.027 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-09-10T00:54:45.027 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-09-10T00:54:45.027 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-09-10T00:54:45.027 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-09-10T00:54:45.027 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-09-10T00:54:45.027 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:54:45.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:54:45.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-09-10T00:54:45.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T00:54:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:54:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T00:54:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:45.031 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.031 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:45.031 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.031 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.031 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.031 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T00:54:45.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:45.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Deploying cephadm binary to smithi136 2023-09-10T00:54:45.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.032 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Added host smithi136 2023-09-10T00:54:45.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:45.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.033 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T00:54:45.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.034 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:54:45.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:45.035 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:45.036 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T00:54:45.036 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1014505104' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-09-10T00:54:45.036 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.036 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.036 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.036 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:45.037 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:54:45.037 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T00:54:45.037 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1014505104' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-09-10T00:54:45.037 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:45.037 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.038 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:54:45.038 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c;count:3 2023-09-10T00:54:45.038 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.038 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:45.038 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:45.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:45.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:45.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:45.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Deploying daemon mon.c on smithi136 2023-09-10T00:54:45.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.040 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.040 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/3079757541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:45.040 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:54:45.040 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:45.040 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.a calling monitor election 2023-09-10T00:54:45.041 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.041 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:45.041 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:45.041 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.c calling monitor election 2023-09-10T00:54:45.041 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:45.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:45.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:45.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-09-10T00:54:45.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:45.043 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: fsmap 2023-09-10T00:54:45.043 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:45.043 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mgrmap e13: x(active, since 46s) 2023-09-10T00:54:45.043 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: overall HEALTH_OK 2023-09-10T00:54:45.043 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.043 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.044 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:45.044 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:45.044 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:45.044 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:45.044 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: Deploying daemon mon.b on smithi092 2023-09-10T00:54:45.044 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/1468951415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:45.045 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:45.045 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-09-10T00:54:45.045 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-09-10T00:54:45.045 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expand map: {default=false} 2023-09-10T00:54:45.045 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta from 'false' to 'false' 2023-09-10T00:54:45.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expanded map: {default=false} 2023-09-10T00:54:45.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expand map: {default=info} 2023-09-10T00:54:45.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta from 'info' to 'info' 2023-09-10T00:54:45.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expanded map: {default=info} 2023-09-10T00:54:45.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expand map: {default=daemon} 2023-09-10T00:54:45.047 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta from 'daemon' to 'daemon' 2023-09-10T00:54:45.047 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expanded map: {default=daemon} 2023-09-10T00:54:45.047 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expand map: {default=debug} 2023-09-10T00:54:45.047 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta from 'debug' to 'debug' 2023-09-10T00:54:45.047 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:44 smithi092 ceph-mon[134067]: expand_channel_meta expanded map: {default=debug} 2023-09-10T00:54:50.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:54:50.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: mon.a calling monitor election 2023-09-10T00:54:50.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: mon.c calling monitor election 2023-09-10T00:54:50.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:50.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:50.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:50.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-09-10T00:54:50.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:50.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: fsmap 2023-09-10T00:54:50.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:50.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: mgrmap e13: x(active, since 54s) 2023-09-10T00:54:50.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-09-10T00:54:50.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-09-10T00:54:50.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-09-10T00:54:50.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: mon.b (rank 2) addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] is down (out of quorum) 2023-09-10T00:54:50.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:50.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:50.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:50.317 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-09-10T00:54:50.317 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":3,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","modified":"2023-09-10T00:54:44.878523Z","created":"2023-09-10T00:52:49.540158Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-09-10T00:54:50.317 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 3 2023-09-10T00:54:50.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:54:50.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: mon.a calling monitor election 2023-09-10T00:54:50.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: mon.c calling monitor election 2023-09-10T00:54:50.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:50.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:50.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:50.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:50.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-09-10T00:54:50.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:50.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: fsmap 2023-09-10T00:54:50.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:50.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: mgrmap e13: x(active, since 54s) 2023-09-10T00:54:50.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-09-10T00:54:50.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-09-10T00:54:50.351 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-09-10T00:54:50.351 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: mon.b (rank 2) addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] is down (out of quorum) 2023-09-10T00:54:50.351 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:50.351 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:50.351 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:50.792 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-09-10T00:54:50.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config generate-minimal-conf 2023-09-10T00:54:51.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:50 smithi136 ceph-mon[133708]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:51.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:51.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:51.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/2708294024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:51.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:51.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:50 smithi003 ceph-mon[97654]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/2708294024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:51.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:51.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:51.536 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:52.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:54:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: mon.a calling monitor election 2023-09-10T00:54:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: mon.c calling monitor election 2023-09-10T00:54:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:52.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:54:52.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:52.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:52.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:52.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:52.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:52.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:52.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:52.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-09-10T00:54:52.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:52.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: fsmap 2023-09-10T00:54:52.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:52.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: mgrmap e13: x(active, since 54s) 2023-09-10T00:54:52.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-09-10T00:54:52.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-09-10T00:54:52.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-09-10T00:54:52.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: mon.b (rank 2) addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] is down (out of quorum) 2023-09-10T00:54:52.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:52.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:52.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:52.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:52.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:52.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:52.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/2708294024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-09-10T00:54:52.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:52.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:53.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: mon.b calling monitor election 2023-09-10T00:54:53.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: Updating smithi003:/etc/ceph/ceph.conf 2023-09-10T00:54:53.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T00:54:53.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T00:54:53.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:53.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:53.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:53.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: mon.b calling monitor election 2023-09-10T00:54:53.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: mon.a calling monitor election 2023-09-10T00:54:53.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: mon.c calling monitor election 2023-09-10T00:54:53.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-09-10T00:54:53.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:53.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: fsmap 2023-09-10T00:54:53.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:53.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: mgrmap e13: x(active, since 56s) 2023-09-10T00:54:53.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-09-10T00:54:53.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: Cluster is now healthy 2023-09-10T00:54:53.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: overall HEALTH_OK 2023-09-10T00:54:53.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: mon.b calling monitor election 2023-09-10T00:54:53.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: Updating smithi003:/etc/ceph/ceph.conf 2023-09-10T00:54:53.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T00:54:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T00:54:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: mon.b calling monitor election 2023-09-10T00:54:53.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: mon.a calling monitor election 2023-09-10T00:54:53.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: mon.c calling monitor election 2023-09-10T00:54:53.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-09-10T00:54:53.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:53.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: fsmap 2023-09-10T00:54:53.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:53.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: mgrmap e13: x(active, since 56s) 2023-09-10T00:54:53.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-09-10T00:54:53.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: Cluster is now healthy 2023-09-10T00:54:53.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: overall HEALTH_OK 2023-09-10T00:54:53.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: mon.b calling monitor election 2023-09-10T00:54:53.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: Updating smithi003:/etc/ceph/ceph.conf 2023-09-10T00:54:53.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T00:54:53.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T00:54:53.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:53.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:53.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:53.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: mon.b calling monitor election 2023-09-10T00:54:53.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: mon.a calling monitor election 2023-09-10T00:54:53.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: mon.c calling monitor election 2023-09-10T00:54:53.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-09-10T00:54:53.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} 2023-09-10T00:54:53.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: fsmap 2023-09-10T00:54:53.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: osdmap e4: 0 total, 0 up, 0 in 2023-09-10T00:54:53.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: mgrmap e13: x(active, since 56s) 2023-09-10T00:54:53.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-09-10T00:54:53.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: Cluster is now healthy 2023-09-10T00:54:53.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: overall HEALTH_OK 2023-09-10T00:54:53.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:53.222 INFO:teuthology.orchestra.run.smithi003.stdout:# minimal ceph.conf for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:53.222 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2023-09-10T00:54:53.223 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T00:54:53.223 INFO:teuthology.orchestra.run.smithi003.stdout: mon_host = [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2023-09-10T00:54:53.624 INFO:tasks.cephadm:['ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'] 2023-09-10T00:54:53.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set client client_mount_timeout 600 2023-09-10T00:54:53.629 INFO:tasks.cephadm:['ceph', 'config', 'set', 'client', 'debug_client', '20'] 2023-09-10T00:54:53.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set client debug_client 20 2023-09-10T00:54:53.676 INFO:tasks.cephadm:['ceph', 'config', 'set', 'client', 'debug_ms', '1'] 2023-09-10T00:54:53.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set client debug_ms 1 2023-09-10T00:54:53.722 INFO:tasks.cephadm:['ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'] 2023-09-10T00:54:53.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set client rados_mon_op_timeout 900 2023-09-10T00:54:53.767 INFO:tasks.cephadm:['ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'] 2023-09-10T00:54:53.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set client rados_osd_op_timeout 900 2023-09-10T00:54:53.817 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'debug_mds', '20'] 2023-09-10T00:54:53.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds debug_mds 20 2023-09-10T00:54:53.867 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'] 2023-09-10T00:54:53.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds debug_mds_balancer 20 2023-09-10T00:54:53.916 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'debug_ms', '1'] 2023-09-10T00:54:53.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds debug_ms 1 2023-09-10T00:54:53.972 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'] 2023-09-10T00:54:53.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_bal_fragment_size_max 10000 2023-09-10T00:54:54.037 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'] 2023-09-10T00:54:54.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_bal_merge_size 5 2023-09-10T00:54:54.098 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'] 2023-09-10T00:54:54.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_bal_split_bits 3 2023-09-10T00:54:54.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:54.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:54.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: Reconfiguring mon.a (unknown last config time)... 2023-09-10T00:54:54.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:54.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:54.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:54.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: Reconfiguring daemon mon.a on smithi003 2023-09-10T00:54:54.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:53 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/651225791' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:54.166 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'] 2023-09-10T00:54:54.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_bal_split_size 100 2023-09-10T00:54:54.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:54.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:54.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.168 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: Reconfiguring mon.a (unknown last config time)... 2023-09-10T00:54:54.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:54.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:54.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:54.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: Reconfiguring daemon mon.a on smithi003 2023-09-10T00:54:54.169 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/651225791' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:54.170 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 00:54:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T00:54:53.880+0000 7f61b9edc700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-10T00:54:54.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T00:54:54.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:54:54.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:54.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: Reconfiguring mon.a (unknown last config time)... 2023-09-10T00:54:54.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:54.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:54.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:54.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: Reconfiguring daemon mon.a on smithi003 2023-09-10T00:54:54.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:53 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/651225791' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:54.226 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'] 2023-09-10T00:54:54.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_debug_frag True 2023-09-10T00:54:54.291 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'] 2023-09-10T00:54:54.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_debug_scatterstat True 2023-09-10T00:54:54.350 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'] 2023-09-10T00:54:54.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_op_complaint_time 180 2023-09-10T00:54:54.405 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'] 2023-09-10T00:54:54.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds mds_verify_scatter True 2023-09-10T00:54:54.469 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'] 2023-09-10T00:54:54.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds osd_op_complaint_time 180 2023-09-10T00:54:54.524 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'] 2023-09-10T00:54:54.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds rados_mon_op_timeout 900 2023-09-10T00:54:54.604 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'] 2023-09-10T00:54:54.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mds rados_osd_op_timeout 900 2023-09-10T00:54:54.685 INFO:tasks.cephadm:['ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'] 2023-09-10T00:54:54.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set mon mon_op_complaint_time 120 2023-09-10T00:54:54.816 INFO:tasks.cephadm:['ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'] 2023-09-10T00:54:54.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config set osd osd_op_complaint_time 180 2023-09-10T00:54:54.871 DEBUG:tasks.cephadm:set 21 configs 2023-09-10T00:54:55.041 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:55.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:54 smithi003 ceph-mon[97654]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:55.247 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:55.277 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:55.302 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:55.349 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:55.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:54 smithi136 ceph-mon[133708]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:55.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:54 smithi092 ceph-mon[134067]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:55.545 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:55.799 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:56.109 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:56.112 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:56.212 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:54:56.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:56.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:56.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: Reconfiguring mon.b (monmap changed)... 2023-09-10T00:54:56.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:56.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:56.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:56.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: Reconfiguring daemon mon.b on smithi092 2023-09-10T00:54:56.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:56 smithi003 ceph-mon[97654]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: Reconfiguring mon.b (monmap changed)... 2023-09-10T00:54:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: Reconfiguring daemon mon.b on smithi092 2023-09-10T00:54:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:56 smithi136 ceph-mon[133708]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: Reconfiguring mon.b (monmap changed)... 2023-09-10T00:54:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:56.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: Reconfiguring daemon mon.b on smithi092 2023-09-10T00:54:56.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:56 smithi092 ceph-mon[134067]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:57.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:57.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:57.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:57 smithi003 ceph-mon[97654]: Reconfiguring mon.c (monmap changed)... 2023-09-10T00:54:57.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:57.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:57.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:57.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:57 smithi003 ceph-mon[97654]: Reconfiguring daemon mon.c on smithi136 2023-09-10T00:54:57.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:57.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:57.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:57 smithi136 ceph-mon[133708]: Reconfiguring mon.c (monmap changed)... 2023-09-10T00:54:57.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:57.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:57.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:57.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:57 smithi136 ceph-mon[133708]: Reconfiguring daemon mon.c on smithi136 2023-09-10T00:54:57.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:57 smithi092 ceph-mon[134067]: Reconfiguring mon.c (monmap changed)... 2023-09-10T00:54:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-09-10T00:54:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-09-10T00:54:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:57.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:57 smithi092 ceph-mon[134067]: Reconfiguring daemon mon.c on smithi136 2023-09-10T00:54:58.789 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.789 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.790 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:58.790 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:58.790 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:58.790 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.790 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:54:58 smithi003 ceph-mon[97654]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:58.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:58.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:58.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:58.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:54:58 smithi136 ceph-mon[133708]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:54:58.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:54:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:54:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:54:58.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:54:58.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:54:58 smithi092 ceph-mon[134067]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:01.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:00 smithi003 ceph-mon[97654]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:00 smithi136 ceph-mon[133708]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:01.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:00 smithi092 ceph-mon[134067]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:01.400 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:01.449 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:01.475 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:01.497 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:02.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:02 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:02.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:02 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/891607353' entity='client.admin' 2023-09-10T00:55:02.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:02 smithi003 ceph-mon[97654]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:02 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:02 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/891607353' entity='client.admin' 2023-09-10T00:55:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:02 smithi136 ceph-mon[133708]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:02.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:02 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:02.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:02 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/891607353' entity='client.admin' 2023-09-10T00:55:02.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:02 smithi092 ceph-mon[134067]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:03.027 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:03.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:03 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:03.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:03 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:03.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:03 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:03.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:03 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:03 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:03 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:03 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:03 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:03 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:03 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:03 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:03 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:04.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:04 smithi003 ceph-mon[97654]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:04.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:04 smithi092 ceph-mon[134067]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:05.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:04 smithi136 ceph-mon[133708]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:05.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:06.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 -- 172.21.15.3:0/4610845 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9614038630 msgr2=0x7f961403aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:06.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 --2- 172.21.15.3:0/4610845 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9614038630 0x7f961403aaf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f962401a8f0 tx=0x7f962402a040 comp rx=0 tx=0).stop 2023-09-10T00:55:06.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 -- 172.21.15.3:0/4610845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9628111630 msgr2=0x7f96281a0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:06.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 --2- 172.21.15.3:0/4610845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9628111630 0x7f96281a0340 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f962000bb90 tx=0x7f962000bea0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 -- 172.21.15.3:0/4610845 shutdown_connections 2023-09-10T00:55:06.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 --2- 172.21.15.3:0/4610845 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9614038630 0x7f961403aaf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 --2- 172.21.15.3:0/4610845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96280729a0 0x7f962810c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 --2- 172.21.15.3:0/4610845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9628111630 0x7f96281a0340 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 --2- 172.21.15.3:0/4610845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96280732c0 0x7f962810ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 -- 172.21.15.3:0/4610845 >> 172.21.15.3:0/4610845 conn(0x7f96280fc0f0 msgr2=0x7f96280fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:06.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 -- 172.21.15.3:0/4610845 shutdown_connections 2023-09-10T00:55:06.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.154+0000 7f96304fe700 1 -- 172.21.15.3:0/4610845 wait complete. 2023-09-10T00:55:06.357 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:06.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 -- 172.21.15.3:0/376567318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 msgr2=0x7f2c6c10c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:06.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 --2- 172.21.15.3:0/376567318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c10c920 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c009990 tx=0x7f2c5c009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 -- 172.21.15.3:0/376567318 shutdown_connections 2023-09-10T00:55:06.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 --2- 172.21.15.3:0/376567318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c6c10ce60 0x7f2c6c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 --2- 172.21.15.3:0/376567318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c6c072a30 0x7f2c6c072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 --2- 172.21.15.3:0/376567318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c10c920 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 -- 172.21.15.3:0/376567318 >> 172.21.15.3:0/376567318 conn(0x7f2c6c0fc0f0 msgr2=0x7f2c6c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:06.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 -- 172.21.15.3:0/376567318 shutdown_connections 2023-09-10T00:55:06.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.580+0000 7f2c73369700 1 -- 172.21.15.3:0/376567318 wait complete. 2023-09-10T00:55:06.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.581+0000 7f2c73369700 1 Processor -- start 2023-09-10T00:55:06.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.581+0000 7f2c73369700 1 -- start start 2023-09-10T00:55:06.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.581+0000 7f2c73369700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c6c072a30 0x7f2c6c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.581+0000 7f2c73369700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.581+0000 7f2c73369700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c6c10ce60 0x7f2c6c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.581+0000 7f2c73369700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c198d40 con 0x7f2c6c073350 2023-09-10T00:55:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c73369700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c198eb0 con 0x7f2c6c10ce60 2023-09-10T00:55:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c73369700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c199070 con 0x7f2c6c072a30 2023-09-10T00:55:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41826/0 (socket says 172.21.15.3:41826) 2023-09-10T00:55:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 -- 172.21.15.3:0/143956621 learned_addr learned my addr 172.21.15.3:0/143956621 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c71906700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c6c10ce60 0x7f2c6c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c71105700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c6c072a30 0x7f2c6c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 -- 172.21.15.3:0/143956621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c6c072a30 msgr2=0x7f2c6c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c6c072a30 0x7f2c6c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 -- 172.21.15.3:0/143956621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c6c10ce60 msgr2=0x7f2c6c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c6c10ce60 0x7f2c6c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 -- 172.21.15.3:0/143956621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c0095f0 con 0x7f2c6c073350 2023-09-10T00:55:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.582+0000 7f2c70904700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c19eb90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c005910 tx=0x7f2c5c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.583+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c01d070 con 0x7f2c6c073350 2023-09-10T00:55:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.583+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2c5c004500 con 0x7f2c6c073350 2023-09-10T00:55:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.583+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c004b30 con 0x7f2c6c073350 2023-09-10T00:55:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.583+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6c1992f0 con 0x7f2c6c073350 2023-09-10T00:55:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.583+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c1a5640 con 0x7f2c6c073350 2023-09-10T00:55:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.584+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7f2c5c004c90 con 0x7f2c6c073350 2023-09-10T00:55:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.584+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6c04e300 con 0x7f2c6c073350 2023-09-10T00:55:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.584+0000 7f2c627fc700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2c58038630 0x7f2c5803aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.584+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2c5c04bf10 con 0x7f2c6c073350 2023-09-10T00:55:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.585+0000 7f2c71105700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2c58038630 0x7f2c5803aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.585+0000 7f2c71105700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2c58038630 0x7f2c5803aaf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005950 tx=0x7f2c6800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:06.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.589+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f2c5c029540 con 0x7f2c6c073350 2023-09-10T00:55:06.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.828+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) v1 -- 0x7f2c6c112830 con 0x7f2c6c073350 2023-09-10T00:55:06.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.831+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v18)=0 v18) v1 ==== 114+0+0 (secure 0 0 0) 0x7f2c5c026070 con 0x7f2c6c073350 2023-09-10T00:55:06.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.832+0000 7f2c627fc700 1 -- 172.21.15.3:0/143956621 <== mon.0 v2:172.21.15.3:3300/0 8 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c5c029540 con 0x7f2c6c073350 2023-09-10T00:55:06.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2c58038630 msgr2=0x7f2c5803aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:06.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2c58038630 0x7f2c5803aaf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005950 tx=0x7f2c6800b410 comp rx=0 tx=0).stop 2023-09-10T00:55:06.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 msgr2=0x7f2c6c19eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:06.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c19eb90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c005910 tx=0x7f2c5c00f690 comp rx=0 tx=0).stop 2023-09-10T00:55:06.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 shutdown_connections 2023-09-10T00:55:06.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2c58038630 0x7f2c5803aaf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c6c072a30 0x7f2c6c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2c6c10ce60 0x7f2c6c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 --2- 172.21.15.3:0/143956621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c6c073350 0x7f2c6c19eb90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:06.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 >> 172.21.15.3:0/143956621 conn(0x7f2c6c0fc0f0 msgr2=0x7f2c6c10f050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:06.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 shutdown_connections 2023-09-10T00:55:06.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:06.834+0000 7f2c73369700 1 -- 172.21.15.3:0/143956621 wait complete. 2023-09-10T00:55:07.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:06 smithi003 ceph-mon[97654]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:07.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:06 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3115423865' entity='client.admin' 2023-09-10T00:55:07.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:06 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:07.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:06 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/143956621' entity='client.admin' 2023-09-10T00:55:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:06 smithi136 ceph-mon[133708]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:06 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3115423865' entity='client.admin' 2023-09-10T00:55:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:06 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:07.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:06 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/143956621' entity='client.admin' 2023-09-10T00:55:07.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:06 smithi092 ceph-mon[134067]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:06 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3115423865' entity='client.admin' 2023-09-10T00:55:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:06 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:06 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/143956621' entity='client.admin' 2023-09-10T00:55:08.000 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:08.048 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:08.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.426+0000 7fbcbc77e700 1 -- 172.21.15.3:0/1568475892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4106620 msgr2=0x7fbcb4106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:08.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.426+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/1568475892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4106620 0x7fbcb4106a00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009990 tx=0x7fbcb0009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.427+0000 7fbcbc77e700 1 -- 172.21.15.3:0/1568475892 shutdown_connections 2023-09-10T00:55:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.427+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/1568475892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcb4100fe0 0x7fbcb410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.427+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/1568475892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4106620 0x7fbcb4106a00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.427+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/1568475892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4100620 0x7fbcb4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.427+0000 7fbcbc77e700 1 -- 172.21.15.3:0/1568475892 >> 172.21.15.3:0/1568475892 conn(0x7fbcb40fc0f0 msgr2=0x7fbcb40fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:08.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.427+0000 7fbcbc77e700 1 -- 172.21.15.3:0/1568475892 shutdown_connections 2023-09-10T00:55:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.427+0000 7fbcbc77e700 1 -- 172.21.15.3:0/1568475892 wait complete. 2023-09-10T00:55:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 Processor -- start 2023-09-10T00:55:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 -- start start 2023-09-10T00:55:08.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcb4100620 0x7fbcb419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4100fe0 0x7fbcb419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4106620 0x7fbcb4196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4196bf0 con 0x7fbcb4106620 2023-09-10T00:55:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4196d60 con 0x7fbcb4100fe0 2023-09-10T00:55:08.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.428+0000 7fbcbc77e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4196f20 con 0x7fbcb4100620 2023-09-10T00:55:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4106620 0x7fbcb4196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4106620 0x7fbcb4196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41840/0 (socket says 172.21.15.3:41840) 2023-09-10T00:55:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 -- 172.21.15.3:0/4159502592 learned_addr learned my addr 172.21.15.3:0/4159502592 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcb9d19700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4100fe0 0x7fbcb419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcba51a700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcb4100620 0x7fbcb419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:08.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 -- 172.21.15.3:0/4159502592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcb4100620 msgr2=0x7fbcb419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:08.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcb4100620 0x7fbcb419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 -- 172.21.15.3:0/4159502592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4100fe0 msgr2=0x7fbcb419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:08.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4100fe0 0x7fbcb419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 -- 172.21.15.3:0/4159502592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb00095f0 con 0x7fbcb4106620 2023-09-10T00:55:08.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcb9d19700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4100fe0 0x7fbcb419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T00:55:08.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcbad1b700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4106620 0x7fbcb4196580 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbcac007ee0 tx=0x7fbcac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:08.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.429+0000 7fbcab7fe700 1 -- 172.21.15.3:0/4159502592 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac017070 con 0x7fbcb4106620 2023-09-10T00:55:08.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.430+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb4197200 con 0x7fbcb4106620 2023-09-10T00:55:08.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.430+0000 7fbcab7fe700 1 -- 172.21.15.3:0/4159502592 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcac00f040 con 0x7fbcb4106620 2023-09-10T00:55:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.430+0000 7fbcab7fe700 1 -- 172.21.15.3:0/4159502592 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac005550 con 0x7fbcb4106620 2023-09-10T00:55:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.430+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb41a34f0 con 0x7fbcb4106620 2023-09-10T00:55:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.431+0000 7fbcab7fe700 1 -- 172.21.15.3:0/4159502592 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7fbcac00db20 con 0x7fbcb4106620 2023-09-10T00:55:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.431+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb404e300 con 0x7fbcb4106620 2023-09-10T00:55:08.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.431+0000 7fbcab7fe700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbca0038630 0x7fbca003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:08.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.431+0000 7fbcab7fe700 1 -- 172.21.15.3:0/4159502592 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbcac04a580 con 0x7fbcb4106620 2023-09-10T00:55:08.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.432+0000 7fbcba51a700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbca0038630 0x7fbca003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:08.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.432+0000 7fbcba51a700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbca0038630 0x7fbca003aaf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000b5c0 tx=0x7fbcb0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:08.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.438+0000 7fbcab7fe700 1 -- 172.21.15.3:0/4159502592 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbcac0194a0 con 0x7fbcb4106620 2023-09-10T00:55:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.664+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) v1 -- 0x7fbcb4110780 con 0x7fbcb4106620 2023-09-10T00:55:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.667+0000 7fbcab7fe700 1 -- 172.21.15.3:0/4159502592 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v19)=0 v19) v1 ==== 106+0+0 (secure 0 0 0) 0x7fbcac0192c0 con 0x7fbcb4106620 2023-09-10T00:55:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbca0038630 msgr2=0x7fbca003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbca0038630 0x7fbca003aaf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000b5c0 tx=0x7fbcb0005fd0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4106620 msgr2=0x7fbcb4196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4106620 0x7fbcb4196580 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbcac007ee0 tx=0x7fbcac00d3b0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 shutdown_connections 2023-09-10T00:55:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbca0038630 0x7fbca003aaf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcb4100620 0x7fbcb419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcb4106620 0x7fbcb4196580 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 --2- 172.21.15.3:0/4159502592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcb4100fe0 0x7fbcb419ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 >> 172.21.15.3:0/4159502592 conn(0x7fbcb40fc0f0 msgr2=0x7fbcb40fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 shutdown_connections 2023-09-10T00:55:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:08.670+0000 7fbcbc77e700 1 -- 172.21.15.3:0/4159502592 wait complete. 2023-09-10T00:55:09.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:08 smithi136 ceph-mon[133708]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:08 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4159502592' entity='client.admin' 2023-09-10T00:55:09.201 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:08 smithi003 ceph-mon[97654]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:09.201 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:08 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4159502592' entity='client.admin' 2023-09-10T00:55:09.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:08 smithi092 ceph-mon[134067]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:08 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4159502592' entity='client.admin' 2023-09-10T00:55:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 -- 172.21.15.3:0/1310558134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100620 msgr2=0x7fb530100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 --2- 172.21.15.3:0/1310558134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100620 0x7fb530100aa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb524009990 tx=0x7fb524009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 -- 172.21.15.3:0/1310558134 shutdown_connections 2023-09-10T00:55:10.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 --2- 172.21.15.3:0/1310558134 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb530106620 0x7fb530106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 --2- 172.21.15.3:0/1310558134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb530100fe0 0x7fb53010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 --2- 172.21.15.3:0/1310558134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100620 0x7fb530100aa0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 -- 172.21.15.3:0/1310558134 >> 172.21.15.3:0/1310558134 conn(0x7fb5300fc0f0 msgr2=0x7fb5300fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:10.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 -- 172.21.15.3:0/1310558134 shutdown_connections 2023-09-10T00:55:10.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.657+0000 7fb5349e8700 1 -- 172.21.15.3:0/1310558134 wait complete. 2023-09-10T00:55:10.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.659+0000 7fb5349e8700 1 Processor -- start 2023-09-10T00:55:10.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.659+0000 7fb5349e8700 1 -- start start 2023-09-10T00:55:10.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.659+0000 7fb5349e8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb530100620 0x7fb53019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:10.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.659+0000 7fb5349e8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100fe0 0x7fb53019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:10.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb5349e8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb530106620 0x7fb530196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:10.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb5349e8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530196bf0 con 0x7fb530100fe0 2023-09-10T00:55:10.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb5349e8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530196d60 con 0x7fb530106620 2023-09-10T00:55:10.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb5349e8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb530196f20 con 0x7fb530100620 2023-09-10T00:55:10.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100fe0 0x7fb53019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:10.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100fe0 0x7fb53019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41862/0 (socket says 172.21.15.3:41862) 2023-09-10T00:55:10.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 -- 172.21.15.3:0/3404150045 learned_addr learned my addr 172.21.15.3:0/3404150045 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:10.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52e59c700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb530100620 0x7fb53019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:10.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52ed9d700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb530106620 0x7fb530196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:10.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 -- 172.21.15.3:0/3404150045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb530100620 msgr2=0x7fb53019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:10.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb530100620 0x7fb53019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 -- 172.21.15.3:0/3404150045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb530106620 msgr2=0x7fb530196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:10.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb530106620 0x7fb530196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 -- 172.21.15.3:0/3404150045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5240095f0 con 0x7fb530100fe0 2023-09-10T00:55:10.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.660+0000 7fb52dd9b700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100fe0 0x7fb53019ca40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb524005910 tx=0x7fb524004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:10.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.661+0000 7fb51f7fe700 1 -- 172.21.15.3:0/3404150045 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52401d070 con 0x7fb530100fe0 2023-09-10T00:55:10.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.661+0000 7fb51f7fe700 1 -- 172.21.15.3:0/3404150045 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb524022470 con 0x7fb530100fe0 2023-09-10T00:55:10.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.661+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5301971a0 con 0x7fb530100fe0 2023-09-10T00:55:10.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.661+0000 7fb51f7fe700 1 -- 172.21.15.3:0/3404150045 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb524017650 con 0x7fb530100fe0 2023-09-10T00:55:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.661+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5301a34f0 con 0x7fb530100fe0 2023-09-10T00:55:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.662+0000 7fb51f7fe700 1 -- 172.21.15.3:0/3404150045 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7fb5240225e0 con 0x7fb530100fe0 2023-09-10T00:55:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.662+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53004e300 con 0x7fb530100fe0 2023-09-10T00:55:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.662+0000 7fb51f7fe700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb518038630 0x7fb51803aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.662+0000 7fb51f7fe700 1 -- 172.21.15.3:0/3404150045 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb52404c2d0 con 0x7fb530100fe0 2023-09-10T00:55:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.663+0000 7fb52e59c700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb518038630 0x7fb51803aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.663+0000 7fb52e59c700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb518038630 0x7fb51803aaf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb520009be0 tx=0x7fb520009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.668+0000 7fb51f7fe700 1 -- 172.21.15.3:0/3404150045 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb52400fb10 con 0x7fb530100fe0 2023-09-10T00:55:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.898+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) v1 -- 0x7fb530110660 con 0x7fb530100fe0 2023-09-10T00:55:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.901+0000 7fb51f7fe700 1 -- 172.21.15.3:0/3404150045 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb524027020 con 0x7fb530100fe0 2023-09-10T00:55:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb518038630 msgr2=0x7fb51803aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb518038630 0x7fb51803aaf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb520009be0 tx=0x7fb520009260 comp rx=0 tx=0).stop 2023-09-10T00:55:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100fe0 msgr2=0x7fb53019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100fe0 0x7fb53019ca40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb524005910 tx=0x7fb524004710 comp rx=0 tx=0).stop 2023-09-10T00:55:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 shutdown_connections 2023-09-10T00:55:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb518038630 0x7fb51803aaf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb530100620 0x7fb53019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb530106620 0x7fb530196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 --2- 172.21.15.3:0/3404150045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb530100fe0 0x7fb53019ca40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 >> 172.21.15.3:0/3404150045 conn(0x7fb5300fc0f0 msgr2=0x7fb5300fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 shutdown_connections 2023-09-10T00:55:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:09.903+0000 7fb5349e8700 1 -- 172.21.15.3:0/3404150045 wait complete. 2023-09-10T00:55:10.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:10 smithi003 ceph-mon[97654]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:10.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:10 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3404150045' entity='client.admin' 2023-09-10T00:55:11.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:10 smithi136 ceph-mon[133708]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:10 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3404150045' entity='client.admin' 2023-09-10T00:55:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 -- 172.21.15.3:0/528791353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 msgr2=0x7f1bdc0685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 --2- 172.21.15.3:0/528791353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc0685a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009990 tx=0x7f1bc8009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 -- 172.21.15.3:0/528791353 shutdown_connections 2023-09-10T00:55:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 --2- 172.21.15.3:0/528791353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bdc068ae0 0x7f1bdc10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 --2- 172.21.15.3:0/528791353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bdc106770 0x7f1bdc106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 --2- 172.21.15.3:0/528791353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc0685a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 -- 172.21.15.3:0/528791353 >> 172.21.15.3:0/528791353 conn(0x7f1bdc076600 msgr2=0x7f1bdc076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:11.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 -- 172.21.15.3:0/528791353 shutdown_connections 2023-09-10T00:55:11.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.192+0000 7f1be2e85700 1 -- 172.21.15.3:0/528791353 wait complete. 2023-09-10T00:55:11.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 Processor -- start 2023-09-10T00:55:11.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 -- start start 2023-09-10T00:55:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bdc068ae0 0x7f1bdc19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bdc106770 0x7f1bdc1965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc196c10 con 0x7f1bdc068120 2023-09-10T00:55:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc196d80 con 0x7f1bdc068ae0 2023-09-10T00:55:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be2e85700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc196f40 con 0x7f1bdc106770 2023-09-10T00:55:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be0c21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be0c21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc19c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41884/0 (socket says 172.21.15.3:41884) 2023-09-10T00:55:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.194+0000 7f1be0c21700 1 -- 172.21.15.3:0/1936831034 learned_addr learned my addr 172.21.15.3:0/1936831034 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1bdbfff700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bdc068ae0 0x7f1bdc19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1be0c21700 1 -- 172.21.15.3:0/1936831034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bdc106770 msgr2=0x7f1bdc1965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1be1422700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bdc106770 0x7f1bdc1965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1be0c21700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bdc106770 0x7f1bdc1965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1be0c21700 1 -- 172.21.15.3:0/1936831034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bdc068ae0 msgr2=0x7f1bdc19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1be0c21700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bdc068ae0 0x7f1bdc19ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1be0c21700 1 -- 172.21.15.3:0/1936831034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd0009520 con 0x7f1bdc068120 2023-09-10T00:55:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1bdbfff700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bdc068ae0 0x7f1bdc19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T00:55:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1be0c21700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc19c520 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000eb30 tx=0x7f1bd000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.195+0000 7f1bd9ffb700 1 -- 172.21.15.3:0/1936831034 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0009880 con 0x7f1bdc068120 2023-09-10T00:55:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.196+0000 7f1bd9ffb700 1 -- 172.21.15.3:0/1936831034 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bd00044e0 con 0x7f1bdc068120 2023-09-10T00:55:11.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.196+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc80095f0 con 0x7f1bdc068120 2023-09-10T00:55:11.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.196+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bdc197410 con 0x7f1bdc068120 2023-09-10T00:55:11.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.196+0000 7f1bd9ffb700 1 -- 172.21.15.3:0/1936831034 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0010430 con 0x7f1bdc068120 2023-09-10T00:55:11.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.197+0000 7f1bd9ffb700 1 -- 172.21.15.3:0/1936831034 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7f1bd0010590 con 0x7f1bdc068120 2023-09-10T00:55:11.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.197+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc0005320 con 0x7f1bdc068120 2023-09-10T00:55:11.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.198+0000 7f1bd9ffb700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bcc038630 0x7f1bcc03aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.198+0000 7f1bd9ffb700 1 -- 172.21.15.3:0/1936831034 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1bd0014070 con 0x7f1bdc068120 2023-09-10T00:55:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.198+0000 7f1bdbfff700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bcc038630 0x7f1bcc03aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.199+0000 7f1bdbfff700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bcc038630 0x7f1bcc03aaf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800b2e0 tx=0x7f1bc8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:11.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.204+0000 7f1bd9ffb700 1 -- 172.21.15.3:0/1936831034 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1bd000fe20 con 0x7f1bdc068120 2023-09-10T00:55:11.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:10 smithi092 ceph-mon[134067]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:11.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:10 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3404150045' entity='client.admin' 2023-09-10T00:55:11.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.458+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) v1 -- 0x7f1bc0005b50 con 0x7f1bdc068120 2023-09-10T00:55:11.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.462+0000 7f1bd9ffb700 1 -- 172.21.15.3:0/1936831034 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v21)=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1bd000f3c0 con 0x7f1bdc068120 2023-09-10T00:55:11.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bcc038630 msgr2=0x7f1bcc03aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:11.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bcc038630 0x7f1bcc03aaf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800b2e0 tx=0x7f1bc8005fd0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 msgr2=0x7f1bdc19c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:11.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc19c520 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000eb30 tx=0x7f1bd000eef0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 shutdown_connections 2023-09-10T00:55:11.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bcc038630 0x7f1bcc03aaf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bdc106770 0x7f1bdc1965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bdc068120 0x7f1bdc19c520 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 --2- 172.21.15.3:0/1936831034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bdc068ae0 0x7f1bdc19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:11.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 >> 172.21.15.3:0/1936831034 conn(0x7f1bdc076600 msgr2=0x7f1bdc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:11.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 shutdown_connections 2023-09-10T00:55:11.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:11.466+0000 7f1be2e85700 1 -- 172.21.15.3:0/1936831034 wait complete. 2023-09-10T00:55:12.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:12 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1936831034' entity='client.admin' 2023-09-10T00:55:12.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:12 smithi003 ceph-mon[97654]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:12.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:12 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1936831034' entity='client.admin' 2023-09-10T00:55:12.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:12 smithi136 ceph-mon[133708]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:12.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:12 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1936831034' entity='client.admin' 2023-09-10T00:55:12.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:12 smithi092 ceph-mon[134067]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:13.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 -- 172.21.15.3:0/2688597879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140732c0 msgr2=0x7fde141110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 --2- 172.21.15.3:0/2688597879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140732c0 0x7fde141110f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde08009990 tx=0x7fde08009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 -- 172.21.15.3:0/2688597879 shutdown_connections 2023-09-10T00:55:13.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 --2- 172.21.15.3:0/2688597879 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde140729a0 0x7fde14072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 --2- 172.21.15.3:0/2688597879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde14111630 0x7fde14113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 --2- 172.21.15.3:0/2688597879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140732c0 0x7fde141110f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 -- 172.21.15.3:0/2688597879 >> 172.21.15.3:0/2688597879 conn(0x7fde140fc0f0 msgr2=0x7fde140fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 -- 172.21.15.3:0/2688597879 shutdown_connections 2023-09-10T00:55:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.215+0000 7fde18db4700 1 -- 172.21.15.3:0/2688597879 wait complete. 2023-09-10T00:55:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.216+0000 7fde18db4700 1 Processor -- start 2023-09-10T00:55:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.216+0000 7fde18db4700 1 -- start start 2023-09-10T00:55:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde18db4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140729a0 0x7fde1419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde18db4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde140732c0 0x7fde1419d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde18db4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde14111630 0x7fde141a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde18db4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde141a1d50 con 0x7fde140729a0 2023-09-10T00:55:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde18db4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde141a1ec0 con 0x7fde140732c0 2023-09-10T00:55:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde18db4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde141a2080 con 0x7fde14111630 2023-09-10T00:55:13.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde1259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140729a0 0x7fde1419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde1259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140729a0 0x7fde1419cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41910/0 (socket says 172.21.15.3:41910) 2023-09-10T00:55:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde1259c700 1 -- 172.21.15.3:0/698878596 learned_addr learned my addr 172.21.15.3:0/698878596 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde11d9b700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde140732c0 0x7fde1419d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.217+0000 7fde12d9d700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde14111630 0x7fde141a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde11d9b700 1 -- 172.21.15.3:0/698878596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde14111630 msgr2=0x7fde141a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde11d9b700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde14111630 0x7fde141a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde11d9b700 1 -- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140729a0 msgr2=0x7fde1419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde11d9b700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140729a0 0x7fde1419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde11d9b700 1 -- 172.21.15.3:0/698878596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080095f0 con 0x7fde140732c0 2023-09-10T00:55:13.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde1259c700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140729a0 0x7fde1419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T00:55:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde11d9b700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde140732c0 0x7fde1419d2d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fde08000c00 tx=0x7fde08005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde037fe700 1 -- 172.21.15.3:0/698878596 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0801d070 con 0x7fde140732c0 2023-09-10T00:55:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde037fe700 1 -- 172.21.15.3:0/698878596 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde08021410 con 0x7fde140732c0 2023-09-10T00:55:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde141a2300 con 0x7fde140732c0 2023-09-10T00:55:13.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde037fe700 1 -- 172.21.15.3:0/698878596 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08017650 con 0x7fde140732c0 2023-09-10T00:55:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.218+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde141a9bc0 con 0x7fde140732c0 2023-09-10T00:55:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.219+0000 7fde037fe700 1 -- 172.21.15.3:0/698878596 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7fde0800fad0 con 0x7fde140732c0 2023-09-10T00:55:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.220+0000 7fde037fe700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fddfc038630 0x7fddfc03aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.220+0000 7fde037fe700 1 -- 172.21.15.3:0/698878596 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fde0804c190 con 0x7fde140732c0 2023-09-10T00:55:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.220+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde14103b10 con 0x7fde140732c0 2023-09-10T00:55:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.220+0000 7fde1259c700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fddfc038630 0x7fddfc03aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.221+0000 7fde1259c700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fddfc038630 0x7fddfc03aaf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde04011550 tx=0x7fde04009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:13.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.225+0000 7fde037fe700 1 -- 172.21.15.3:0/698878596 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fde08026070 con 0x7fde140732c0 2023-09-10T00:55:13.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 -- 172.21.15.3:0/2247478182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 msgr2=0x7ff3ac072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/2247478182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac072e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009990 tx=0x7ff3a0009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 -- 172.21.15.3:0/2247478182 shutdown_connections 2023-09-10T00:55:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/2247478182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac10cca0 0x7ff3ac10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/2247478182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac072e10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/2247478182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3ac073350 0x7ff3ac10c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 -- 172.21.15.3:0/2247478182 >> 172.21.15.3:0/2247478182 conn(0x7ff3ac0fbfb0 msgr2=0x7ff3ac0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 -- 172.21.15.3:0/2247478182 shutdown_connections 2023-09-10T00:55:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.267+0000 7ff3b1fba700 1 -- 172.21.15.3:0/2247478182 wait complete. 2023-09-10T00:55:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.268+0000 7ff3b1fba700 1 Processor -- start 2023-09-10T00:55:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.268+0000 7ff3b1fba700 1 -- start start 2023-09-10T00:55:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b1fba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b1fba700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3ac073350 0x7ff3ac19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b1fba700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac10cca0 0x7ff3ac198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b1fba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac198c90 con 0x7ff3ac072a30 2023-09-10T00:55:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b1fba700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac198e00 con 0x7ff3ac10cca0 2023-09-10T00:55:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b1fba700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac198fc0 con 0x7ff3ac073350 2023-09-10T00:55:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41926/0 (socket says 172.21.15.3:41926) 2023-09-10T00:55:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 -- 172.21.15.3:0/553928410 learned_addr learned my addr 172.21.15.3:0/553928410 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:13.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b17b9700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac10cca0 0x7ff3ac198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3abfff700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3ac073350 0x7ff3ac19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 -- 172.21.15.3:0/553928410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3ac073350 msgr2=0x7ff3ac19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3ac073350 0x7ff3ac19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 -- 172.21.15.3:0/553928410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac10cca0 msgr2=0x7ff3ac198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac10cca0 0x7ff3ac198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.269+0000 7ff3b0fb8700 1 -- 172.21.15.3:0/553928410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff398009520 con 0x7ff3ac072a30 2023-09-10T00:55:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.270+0000 7ff3b0fb8700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac19e5a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.270+0000 7ff3a9ffb700 1 -- 172.21.15.3:0/553928410 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a001d070 con 0x7ff3ac072a30 2023-09-10T00:55:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.270+0000 7ff3a9ffb700 1 -- 172.21.15.3:0/553928410 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3a0022470 con 0x7ff3ac072a30 2023-09-10T00:55:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.270+0000 7ff3a9ffb700 1 -- 172.21.15.3:0/553928410 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0017650 con 0x7ff3ac072a30 2023-09-10T00:55:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.270+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a00095f0 con 0x7ff3ac072a30 2023-09-10T00:55:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.270+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac1a53c0 con 0x7ff3ac072a30 2023-09-10T00:55:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.271+0000 7ff3a9ffb700 1 -- 172.21.15.3:0/553928410 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7ff3a000fa50 con 0x7ff3ac072a30 2023-09-10T00:55:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.271+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac04eb50 con 0x7ff3ac072a30 2023-09-10T00:55:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.272+0000 7ff3a9ffb700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff39c038630 0x7ff39c03aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.272+0000 7ff3a9ffb700 1 -- 172.21.15.3:0/553928410 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff3a004c6c0 con 0x7ff3ac072a30 2023-09-10T00:55:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.272+0000 7ff3abfff700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff39c038630 0x7ff39c03aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.272+0000 7ff3abfff700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff39c038630 0x7ff39c03aaf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac10e300 tx=0x7ff398009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.277+0000 7ff3a9ffb700 1 -- 172.21.15.3:0/553928410 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff3a0016b90 con 0x7ff3ac072a30 2023-09-10T00:55:13.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.445+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) v1 -- 0x7fde1419e1c0 con 0x7fde140732c0 2023-09-10T00:55:13.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.450+0000 7fde037fe700 1 -- 172.21.15.3:0/698878596 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v22)=0 v22) v1 ==== 116+0+0 (secure 0 0 0) 0x7fde08029360 con 0x7fde140732c0 2023-09-10T00:55:13.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.452+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fddfc038630 msgr2=0x7fddfc03aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.452+0000 7fde18db4700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fddfc038630 0x7fddfc03aaf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde04011550 tx=0x7fde04009260 comp rx=0 tx=0).stop 2023-09-10T00:55:13.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde140732c0 msgr2=0x7fde1419d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde140732c0 0x7fde1419d2d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fde08000c00 tx=0x7fde08005740 comp rx=0 tx=0).stop 2023-09-10T00:55:13.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 shutdown_connections 2023-09-10T00:55:13.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fddfc038630 0x7fddfc03aaf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde14111630 0x7fde141a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde140729a0 0x7fde1419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 --2- 172.21.15.3:0/698878596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde140732c0 0x7fde1419d2d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 >> 172.21.15.3:0/698878596 conn(0x7fde140fc0f0 msgr2=0x7fde140fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:13.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 shutdown_connections 2023-09-10T00:55:13.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.453+0000 7fde18db4700 1 -- 172.21.15.3:0/698878596 wait complete. 2023-09-10T00:55:13.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.505+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) v1 -- 0x7ff3ac199b30 con 0x7ff3ac072a30 2023-09-10T00:55:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.509+0000 7ff3a9ffb700 1 -- 172.21.15.3:0/553928410 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v23)=0 v23) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff3ac199b30 con 0x7ff3ac072a30 2023-09-10T00:55:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff39c038630 msgr2=0x7ff39c03aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff39c038630 0x7ff39c03aaf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac10e300 tx=0x7ff398009260 comp rx=0 tx=0).stop 2023-09-10T00:55:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 msgr2=0x7ff3ac19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac19e5a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a0005740 comp rx=0 tx=0).stop 2023-09-10T00:55:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 shutdown_connections 2023-09-10T00:55:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3ac10cca0 0x7ff3ac198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3ac072a30 0x7ff3ac19e5a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff39c038630 0x7ff39c03aaf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 --2- 172.21.15.3:0/553928410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3ac073350 0x7ff3ac19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:13.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 >> 172.21.15.3:0/553928410 conn(0x7ff3ac0fbfb0 msgr2=0x7ff3ac10ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:13.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.511+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 shutdown_connections 2023-09-10T00:55:13.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:13.512+0000 7ff3b1fba700 1 -- 172.21.15.3:0/553928410 wait complete. 2023-09-10T00:55:14.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 -- 172.21.15.3:0/2815606904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 msgr2=0x7f092c106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:14.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 --2- 172.21.15.3:0/2815606904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 0x7f092c106a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0928009990 tx=0x7f0928009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 -- 172.21.15.3:0/2815606904 shutdown_connections 2023-09-10T00:55:14.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 --2- 172.21.15.3:0/2815606904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f092c100fe0 0x7f092c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 --2- 172.21.15.3:0/2815606904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 0x7f092c106a00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 --2- 172.21.15.3:0/2815606904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f092c100620 0x7f092c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 -- 172.21.15.3:0/2815606904 >> 172.21.15.3:0/2815606904 conn(0x7f092c0fc0f0 msgr2=0x7f092c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:14.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.031+0000 7f093349d700 1 -- 172.21.15.3:0/2815606904 shutdown_connections 2023-09-10T00:55:14.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.032+0000 7f093349d700 1 -- 172.21.15.3:0/2815606904 wait complete. 2023-09-10T00:55:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 Processor -- start 2023-09-10T00:55:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 -- start start 2023-09-10T00:55:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f092c100620 0x7f092c072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f092c100fe0 0x7f092c072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 0x7f092c0733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c1a4c00 con 0x7f092c106620 2023-09-10T00:55:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c1a4d70 con 0x7f092c100fe0 2023-09-10T00:55:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.033+0000 7f093349d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c1a4f30 con 0x7f092c100620 2023-09-10T00:55:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931a3a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 0x7f092c0733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:14.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931239700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f092c100620 0x7f092c072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931a3a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 0x7f092c0733c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41940/0 (socket says 172.21.15.3:41940) 2023-09-10T00:55:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0930a38700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f092c100fe0 0x7f092c072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931a3a700 1 -- 172.21.15.3:0/3349691060 learned_addr learned my addr 172.21.15.3:0/3349691060 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931239700 1 -- 172.21.15.3:0/3349691060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f092c100fe0 msgr2=0x7f092c072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:14.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931239700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f092c100fe0 0x7f092c072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931239700 1 -- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 msgr2=0x7f092c0733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:14.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931239700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 0x7f092c0733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931239700 1 -- 172.21.15.3:0/3349691060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c009520 con 0x7f092c100620 2023-09-10T00:55:14.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.034+0000 7f0931239700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f092c100620 0x7f092c072900 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0928000c00 tx=0x7f0928005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:14.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.035+0000 7f09227fc700 1 -- 172.21.15.3:0/3349691060 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092801d070 con 0x7f092c100620 2023-09-10T00:55:14.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.035+0000 7f09227fc700 1 -- 172.21.15.3:0/3349691060 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0928021410 con 0x7f092c100620 2023-09-10T00:55:14.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.035+0000 7f09227fc700 1 -- 172.21.15.3:0/3349691060 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928017650 con 0x7f092c100620 2023-09-10T00:55:14.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.035+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09280095f0 con 0x7f092c100620 2023-09-10T00:55:14.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.035+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c1a53e0 con 0x7f092c100620 2023-09-10T00:55:14.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.036+0000 7f09227fc700 1 -- 172.21.15.3:0/3349691060 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7f0928021580 con 0x7f092c100620 2023-09-10T00:55:14.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.036+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c04e300 con 0x7f092c100620 2023-09-10T00:55:14.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.036+0000 7f09227fc700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0918038630 0x7f091803aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:14.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.036+0000 7f09227fc700 1 -- 172.21.15.3:0/3349691060 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f092804c230 con 0x7f092c100620 2023-09-10T00:55:14.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.037+0000 7f0930a38700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0918038630 0x7f091803aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:14.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.037+0000 7f0930a38700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0918038630 0x7f091803aaf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f092c0740f0 tx=0x7f091c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:14.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.042+0000 7f09227fc700 1 -- 172.21.15.3:0/3349691060 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0928026070 con 0x7f092c100620 2023-09-10T00:55:14.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.265+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) v1 -- 0x7f092c073e30 con 0x7f092c100620 2023-09-10T00:55:14.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.270+0000 7f09227fc700 1 -- 172.21.15.3:0/3349691060 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v24)=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0928029540 con 0x7f092c100620 2023-09-10T00:55:14.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.272+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0918038630 msgr2=0x7f091803aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:14.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.272+0000 7f093349d700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0918038630 0x7f091803aaf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f092c0740f0 tx=0x7f091c009260 comp rx=0 tx=0).stop 2023-09-10T00:55:14.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.272+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f092c100620 msgr2=0x7f092c072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:14.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.272+0000 7f093349d700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f092c100620 0x7f092c072900 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0928000c00 tx=0x7f0928005740 comp rx=0 tx=0).stop 2023-09-10T00:55:14.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 shutdown_connections 2023-09-10T00:55:14.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0918038630 0x7f091803aaf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f092c100620 0x7f092c072900 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f092c106620 0x7f092c0733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 --2- 172.21.15.3:0/3349691060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f092c100fe0 0x7f092c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:14.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 >> 172.21.15.3:0/3349691060 conn(0x7f092c0fc0f0 msgr2=0x7f092c0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:14.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 shutdown_connections 2023-09-10T00:55:14.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:14.273+0000 7f093349d700 1 -- 172.21.15.3:0/3349691060 wait complete. 2023-09-10T00:55:14.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:14 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:14.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:14 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/553928410' entity='client.admin' 2023-09-10T00:55:14.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:14 smithi003 ceph-mon[97654]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:14.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:14 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:14.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:14 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:14.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:14 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/553928410' entity='client.admin' 2023-09-10T00:55:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:14 smithi092 ceph-mon[134067]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:14 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:14.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:14 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:14.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:14 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/553928410' entity='client.admin' 2023-09-10T00:55:14.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:14 smithi136 ceph-mon[133708]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:14.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:14 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.538+0000 7f2a65cb2700 1 -- 172.21.15.3:0/3900200565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600732c0 msgr2=0x7f2a601110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.538+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/3900200565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600732c0 0x7f2a601110f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009990 tx=0x7f2a54009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.538+0000 7f2a65cb2700 1 -- 172.21.15.3:0/3900200565 shutdown_connections 2023-09-10T00:55:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.538+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/3900200565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a60111630 0x7f2a60113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.538+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/3900200565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a600729a0 0x7f2a60072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.538+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/3900200565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600732c0 0x7f2a601110f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.538+0000 7f2a65cb2700 1 -- 172.21.15.3:0/3900200565 >> 172.21.15.3:0/3900200565 conn(0x7f2a600fc0f0 msgr2=0x7f2a600fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.539+0000 7f2a65cb2700 1 -- 172.21.15.3:0/3900200565 shutdown_connections 2023-09-10T00:55:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.539+0000 7f2a65cb2700 1 -- 172.21.15.3:0/3900200565 wait complete. 2023-09-10T00:55:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 Processor -- start 2023-09-10T00:55:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 -- start start 2023-09-10T00:55:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600729a0 0x7f2a6019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a600732c0 0x7f2a6019d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a60111630 0x7f2a601a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a601a1d90 con 0x7f2a60111630 2023-09-10T00:55:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a601a1f00 con 0x7f2a600729a0 2023-09-10T00:55:15.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a65cb2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a601a20c0 con 0x7f2a600732c0 2023-09-10T00:55:15.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.540+0000 7f2a5f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600729a0 0x7f2a6019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:15.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600729a0 0x7f2a6019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39738/0 (socket says 172.21.15.3:39738) 2023-09-10T00:55:15.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 -- 172.21.15.3:0/591115646 learned_addr learned my addr 172.21.15.3:0/591115646 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:15.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5ffff700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a60111630 0x7f2a601a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:15.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5effd700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a600732c0 0x7f2a6019d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:15.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 -- 172.21.15.3:0/591115646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a600732c0 msgr2=0x7f2a6019d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:15.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a600732c0 0x7f2a6019d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 -- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a60111630 msgr2=0x7f2a601a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:15.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a60111630 0x7f2a601a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 -- 172.21.15.3:0/591115646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a540095f0 con 0x7f2a600729a0 2023-09-10T00:55:15.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5f7fe700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600729a0 0x7f2a6019cdd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000e9a0 tx=0x7f2a5000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:15.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.541+0000 7f2a5cff9700 1 -- 172.21.15.3:0/591115646 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a50009790 con 0x7f2a600729a0 2023-09-10T00:55:15.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.542+0000 7f2a5cff9700 1 -- 172.21.15.3:0/591115646 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a50010430 con 0x7f2a600729a0 2023-09-10T00:55:15.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.542+0000 7f2a5cff9700 1 -- 172.21.15.3:0/591115646 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a50018680 con 0x7f2a600729a0 2023-09-10T00:55:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.542+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a601a23a0 con 0x7f2a600729a0 2023-09-10T00:55:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.542+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a601a9c00 con 0x7f2a600729a0 2023-09-10T00:55:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.543+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a60103b10 con 0x7f2a600729a0 2023-09-10T00:55:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.543+0000 7f2a5cff9700 1 -- 172.21.15.3:0/591115646 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7f2a500105a0 con 0x7f2a600729a0 2023-09-10T00:55:15.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.544+0000 7f2a5cff9700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2a48038630 0x7f2a4803aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.544+0000 7f2a5cff9700 1 -- 172.21.15.3:0/591115646 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2a50014070 con 0x7f2a600729a0 2023-09-10T00:55:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.544+0000 7f2a5effd700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2a48038630 0x7f2a4803aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.544+0000 7f2a5effd700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2a48038630 0x7f2a4803aaf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a6019e3f0 tx=0x7f2a54005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.548+0000 7f2a5cff9700 1 -- 172.21.15.3:0/591115646 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f2a5001f780 con 0x7f2a600729a0 2023-09-10T00:55:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.762+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) v1 -- 0x7f2a6004e300 con 0x7f2a600729a0 2023-09-10T00:55:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.767+0000 7f2a5cff9700 1 -- 172.21.15.3:0/591115646 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v25)=0 v25) v1 ==== 119+0+0 (secure 0 0 0) 0x7f2a5001f780 con 0x7f2a600729a0 2023-09-10T00:55:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2a48038630 msgr2=0x7f2a4803aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2a48038630 0x7f2a4803aaf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a6019e3f0 tx=0x7f2a54005fb0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600729a0 msgr2=0x7f2a6019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600729a0 0x7f2a6019cdd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000e9a0 tx=0x7f2a5000ecb0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 shutdown_connections 2023-09-10T00:55:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a60111630 0x7f2a601a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a600729a0 0x7f2a6019cdd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2a48038630 0x7f2a4803aaf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 --2- 172.21.15.3:0/591115646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a600732c0 0x7f2a6019d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 >> 172.21.15.3:0/591115646 conn(0x7f2a600fc0f0 msgr2=0x7f2a600fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 shutdown_connections 2023-09-10T00:55:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:15.770+0000 7f2a65cb2700 1 -- 172.21.15.3:0/591115646 wait complete. 2023-09-10T00:55:16.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.269+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/970576760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c072a30 msgr2=0x7ff79c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.269+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/970576760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c072a30 0x7ff79c072e10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff798009990 tx=0x7ff798009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.269+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/970576760 shutdown_connections 2023-09-10T00:55:16.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.269+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/970576760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c072a30 0x7ff79c072e10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.269+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/970576760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c10ce60 0x7ff79c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.269+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/970576760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c073350 0x7ff79c10c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.269+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/970576760 >> 172.21.15.3:0/970576760 conn(0x7ff79c0fc0f0 msgr2=0x7ff79c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:16.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.270+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/970576760 shutdown_connections 2023-09-10T00:55:16.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.270+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/970576760 wait complete. 2023-09-10T00:55:16.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 Processor -- start 2023-09-10T00:55:16.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 -- start start 2023-09-10T00:55:16.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c072a30 0x7ff79c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c073350 0x7ff79c198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c10ce60 0x7ff79c19d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c19d950 con 0x7ff79c10ce60 2023-09-10T00:55:16.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c19dac0 con 0x7ff79c072a30 2023-09-10T00:55:16.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a3a4d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c19dc80 con 0x7ff79c073350 2023-09-10T00:55:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a1fea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c10ce60 0x7ff79c19d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.271+0000 7ff7a1fea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c10ce60 0x7ff79c19d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42118/0 (socket says 172.21.15.3:42118) 2023-09-10T00:55:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a1fea700 1 -- 172.21.15.3:0/4029402874 learned_addr learned my addr 172.21.15.3:0/4029402874 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:16.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a17e9700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c072a30 0x7ff79c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a0fe8700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c073350 0x7ff79c198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a1fea700 1 -- 172.21.15.3:0/4029402874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c073350 msgr2=0x7ff79c198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a1fea700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c073350 0x7ff79c198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a1fea700 1 -- 172.21.15.3:0/4029402874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c072a30 msgr2=0x7ff79c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a1fea700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c072a30 0x7ff79c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a1fea700 1 -- 172.21.15.3:0/4029402874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7980095f0 con 0x7ff79c10ce60 2023-09-10T00:55:16.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a1fea700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c10ce60 0x7ff79c19d380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff79400ef30 tx=0x7ff79400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7927fc700 1 -- 172.21.15.3:0/4029402874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794018070 con 0x7ff79c10ce60 2023-09-10T00:55:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7927fc700 1 -- 172.21.15.3:0/4029402874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff79400f040 con 0x7ff79c10ce60 2023-09-10T00:55:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c19df00 con 0x7ff79c10ce60 2023-09-10T00:55:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.272+0000 7ff7927fc700 1 -- 172.21.15.3:0/4029402874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79400cd00 con 0x7ff79c10ce60 2023-09-10T00:55:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.273+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c1a5800 con 0x7ff79c10ce60 2023-09-10T00:55:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.274+0000 7ff7927fc700 1 -- 172.21.15.3:0/4029402874 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7ff794003a00 con 0x7ff79c10ce60 2023-09-10T00:55:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.274+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c04e300 con 0x7ff79c10ce60 2023-09-10T00:55:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.274+0000 7ff7927fc700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff788038630 0x7ff78803aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.274+0000 7ff7927fc700 1 -- 172.21.15.3:0/4029402874 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff79404a8e0 con 0x7ff79c10ce60 2023-09-10T00:55:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.274+0000 7ff7a17e9700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff788038630 0x7ff78803aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.275+0000 7ff7a17e9700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff788038630 0x7ff78803aaf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff79800b2e0 tx=0x7ff798005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:16.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.280+0000 7ff7927fc700 1 -- 172.21.15.3:0/4029402874 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff794005810 con 0x7ff79c10ce60 2023-09-10T00:55:16.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.530+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) v1 -- 0x7ff79c199d00 con 0x7ff79c10ce60 2023-09-10T00:55:16.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.533+0000 7ff7927fc700 1 -- 172.21.15.3:0/4029402874 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v26)=0 v26) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff794005810 con 0x7ff79c10ce60 2023-09-10T00:55:16.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff788038630 msgr2=0x7ff78803aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff788038630 0x7ff78803aaf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff79800b2e0 tx=0x7ff798005fd0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c10ce60 msgr2=0x7ff79c19d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c10ce60 0x7ff79c19d380 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff79400ef30 tx=0x7ff79400c3a0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 shutdown_connections 2023-09-10T00:55:16.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff79c10ce60 0x7ff79c19d380 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c072a30 0x7ff79c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff788038630 0x7ff78803aaf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 --2- 172.21.15.3:0/4029402874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff79c073350 0x7ff79c198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 >> 172.21.15.3:0/4029402874 conn(0x7ff79c0fc0f0 msgr2=0x7ff79c10f090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:16.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 shutdown_connections 2023-09-10T00:55:16.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.537+0000 7ff7a3a4d700 1 -- 172.21.15.3:0/4029402874 wait complete. 2023-09-10T00:55:16.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:16 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:16.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 -- 172.21.15.3:0/614881998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 msgr2=0x7ff5a410d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 --2- 172.21.15.3:0/614881998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a410d500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00b0a0 tx=0x7ff59c00b3b0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 -- 172.21.15.3:0/614881998 shutdown_connections 2023-09-10T00:55:16.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 --2- 172.21.15.3:0/614881998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a410d500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 --2- 172.21.15.3:0/614881998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5a4106630 0x7ff5a4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 --2- 172.21.15.3:0/614881998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5a4100630 0x7ff5a4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 -- 172.21.15.3:0/614881998 >> 172.21.15.3:0/614881998 conn(0x7ff5a40fc120 msgr2=0x7ff5a40fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:16.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 -- 172.21.15.3:0/614881998 shutdown_connections 2023-09-10T00:55:16.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.927+0000 7ff5ab793700 1 -- 172.21.15.3:0/614881998 wait complete. 2023-09-10T00:55:16.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.928+0000 7ff5ab793700 1 Processor -- start 2023-09-10T00:55:16.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.929+0000 7ff5ab793700 1 -- start start 2023-09-10T00:55:16.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.929+0000 7ff5ab793700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5a4100630 0x7ff5a419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.929+0000 7ff5ab793700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.929+0000 7ff5ab793700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5a4106630 0x7ff5a4196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.929+0000 7ff5ab793700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4196cf0 con 0x7ff5a4100ff0 2023-09-10T00:55:16.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.929+0000 7ff5ab793700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4196e60 con 0x7ff5a4100630 2023-09-10T00:55:16.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.929+0000 7ff5ab793700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4197020 con 0x7ff5a4106630 2023-09-10T00:55:16.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a419cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42138/0 (socket says 172.21.15.3:42138) 2023-09-10T00:55:16.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 -- 172.21.15.3:0/4030045016 learned_addr learned my addr 172.21.15.3:0/4030045016 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:16.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a952f700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5a4100630 0x7ff5a419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a9d30700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5a4106630 0x7ff5a4196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 -- 172.21.15.3:0/4030045016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5a4106630 msgr2=0x7ff5a4196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5a4106630 0x7ff5a4196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 -- 172.21.15.3:0/4030045016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5a4100630 msgr2=0x7ff5a419c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:16.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5a4100630 0x7ff5a419c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:16.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 -- 172.21.15.3:0/4030045016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c009d00 con 0x7ff5a4100ff0 2023-09-10T00:55:16.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a9d30700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5a4106630 0x7ff5a4196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:55:16.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.930+0000 7ff5a8d2e700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a419cb40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff59400e9a0 tx=0x7ff59400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:16.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.931+0000 7ff59a7fc700 1 -- 172.21.15.3:0/4030045016 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594009790 con 0x7ff5a4100ff0 2023-09-10T00:55:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.931+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a4197300 con 0x7ff5a4100ff0 2023-09-10T00:55:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.931+0000 7ff59a7fc700 1 -- 172.21.15.3:0/4030045016 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff59400cbd0 con 0x7ff5a4100ff0 2023-09-10T00:55:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.931+0000 7ff59a7fc700 1 -- 172.21.15.3:0/4030045016 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594018680 con 0x7ff5a4100ff0 2023-09-10T00:55:16.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.931+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a41a35f0 con 0x7ff5a4100ff0 2023-09-10T00:55:16.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.932+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a404e300 con 0x7ff5a4100ff0 2023-09-10T00:55:16.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.932+0000 7ff59a7fc700 1 -- 172.21.15.3:0/4030045016 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7ff594010430 con 0x7ff5a4100ff0 2023-09-10T00:55:16.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.932+0000 7ff59a7fc700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff590038630 0x7ff59003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:16.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.932+0000 7ff59a7fc700 1 -- 172.21.15.3:0/4030045016 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff594014070 con 0x7ff5a4100ff0 2023-09-10T00:55:16.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.933+0000 7ff5a952f700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff590038630 0x7ff59003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:16.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.933+0000 7ff5a952f700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff590038630 0x7ff59003aaf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0005950 tx=0x7ff5a000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:16.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:16.937+0000 7ff59a7fc700 1 -- 172.21.15.3:0/4030045016 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff59400cd40 con 0x7ff5a4100ff0 2023-09-10T00:55:17.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:16 smithi003 ceph-mon[97654]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:17.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:16 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4029402874' entity='client.admin' 2023-09-10T00:55:17.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:16 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:17.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:16 smithi136 ceph-mon[133708]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:17.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:16 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4029402874' entity='client.admin' 2023-09-10T00:55:17.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.172+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) v1 -- 0x7ff5a41106c0 con 0x7ff5a4100ff0 2023-09-10T00:55:17.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.177+0000 7ff59a7fc700 1 -- 172.21.15.3:0/4030045016 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v27)=0 v27) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff59401fb00 con 0x7ff5a4100ff0 2023-09-10T00:55:17.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff590038630 msgr2=0x7ff59003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff590038630 0x7ff59003aaf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0005950 tx=0x7ff5a000b410 comp rx=0 tx=0).stop 2023-09-10T00:55:17.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 msgr2=0x7ff5a419cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a419cb40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff59400e9a0 tx=0x7ff59400ecb0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 shutdown_connections 2023-09-10T00:55:17.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff590038630 0x7ff59003aaf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5a4106630 0x7ff5a4196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5a4100630 0x7ff5a419c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 --2- 172.21.15.3:0/4030045016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5a4100ff0 0x7ff5a419cb40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 >> 172.21.15.3:0/4030045016 conn(0x7ff5a40fc120 msgr2=0x7ff5a40fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:17.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 shutdown_connections 2023-09-10T00:55:17.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7ff5ab793700 1 -- 172.21.15.3:0/4030045016 wait complete. 2023-09-10T00:55:17.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.175+0000 7f533963d700 1 -- 172.21.15.3:0/3512786499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 msgr2=0x7f53341041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.175+0000 7f533963d700 1 --2- 172.21.15.3:0/3512786499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 0x7f53341041f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5328009990 tx=0x7f5328009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.175+0000 7f533963d700 1 -- 172.21.15.3:0/3512786499 shutdown_connections 2023-09-10T00:55:17.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.175+0000 7f533963d700 1 --2- 172.21.15.3:0/3512786499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5334104810 0x7f5334104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.175+0000 7f533963d700 1 --2- 172.21.15.3:0/3512786499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 0x7f53340ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.175+0000 7f533963d700 1 --2- 172.21.15.3:0/3512786499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 0x7f53341041f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.175+0000 7f533963d700 1 -- 172.21.15.3:0/3512786499 >> 172.21.15.3:0/3512786499 conn(0x7f53340faea0 msgr2=0x7f53340fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:17.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.177+0000 7f533963d700 1 -- 172.21.15.3:0/3512786499 shutdown_connections 2023-09-10T00:55:17.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.177+0000 7f533963d700 1 -- 172.21.15.3:0/3512786499 wait complete. 2023-09-10T00:55:17.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.179+0000 7f533963d700 1 Processor -- start 2023-09-10T00:55:17.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.179+0000 7f533963d700 1 -- start start 2023-09-10T00:55:17.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.179+0000 7f533963d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 0x7f5334071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f533963d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 0x7f5334072020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f533963d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5334104810 0x7f533406aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f533963d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533406b1a0 con 0x7f53340ffd20 2023-09-10T00:55:17.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f533963d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533406b310 con 0x7f53340ff400 2023-09-10T00:55:17.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f533963d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533406b4d0 con 0x7f5334104810 2023-09-10T00:55:17.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 0x7f5334071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f53337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 0x7f5334072020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 0x7f5334071ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39812/0 (socket says 172.21.15.3:39812) 2023-09-10T00:55:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 -- 172.21.15.3:0/2292651883 learned_addr learned my addr 172.21.15.3:0/2292651883 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5338e3c700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5334104810 0x7f533406aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 -- 172.21.15.3:0/2292651883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5334104810 msgr2=0x7f533406aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5334104810 0x7f533406aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 -- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 msgr2=0x7f5334072020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 0x7f5334072020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.180+0000 7f5333fff700 1 -- 172.21.15.3:0/2292651883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53280095f0 con 0x7f53340ff400 2023-09-10T00:55:17.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7f5333fff700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 0x7f5334071ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f532400e9a0 tx=0x7f532400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7f53317fa700 1 -- 172.21.15.3:0/2292651883 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324009790 con 0x7f53340ff400 2023-09-10T00:55:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7f53317fa700 1 -- 172.21.15.3:0/2292651883 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f532400cbd0 con 0x7f53340ff400 2023-09-10T00:55:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7f53317fa700 1 -- 172.21.15.3:0/2292651883 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324018680 con 0x7f53340ff400 2023-09-10T00:55:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533406b7b0 con 0x7f53340ff400 2023-09-10T00:55:17.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.181+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533406f7e0 con 0x7f53340ff400 2023-09-10T00:55:17.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.183+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533404eb50 con 0x7f53340ff400 2023-09-10T00:55:17.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.183+0000 7f53317fa700 1 -- 172.21.15.3:0/2292651883 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7f532400cd40 con 0x7f53340ff400 2023-09-10T00:55:17.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.184+0000 7f53317fa700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f531c038680 0x7f531c03ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.184+0000 7f53317fa700 1 -- 172.21.15.3:0/2292651883 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5324014070 con 0x7f53340ff400 2023-09-10T00:55:17.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.185+0000 7f53337fe700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f531c038680 0x7f531c03ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.185+0000 7f53337fe700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f531c038680 0x7f531c03ab40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f532800b2e0 tx=0x7f5328005e00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:17.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.188+0000 7f53317fa700 1 -- 172.21.15.3:0/2292651883 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5324016740 con 0x7f53340ff400 2023-09-10T00:55:17.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:16 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:17.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:16 smithi092 ceph-mon[134067]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:16 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4029402874' entity='client.admin' 2023-09-10T00:55:17.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 -- 172.21.15.3:0/1958567798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 msgr2=0x7feac80685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 --2- 172.21.15.3:0/1958567798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac80685a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feabc009990 tx=0x7feabc009ca0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 -- 172.21.15.3:0/1958567798 shutdown_connections 2023-09-10T00:55:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 --2- 172.21.15.3:0/1958567798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feac8106770 0x7feac8106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 --2- 172.21.15.3:0/1958567798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac8068ae0 0x7feac810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 --2- 172.21.15.3:0/1958567798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac80685a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 -- 172.21.15.3:0/1958567798 >> 172.21.15.3:0/1958567798 conn(0x7feac8076600 msgr2=0x7feac8076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:17.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 -- 172.21.15.3:0/1958567798 shutdown_connections 2023-09-10T00:55:17.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.120+0000 7feacd15c700 1 -- 172.21.15.3:0/1958567798 wait complete. 2023-09-10T00:55:17.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 Processor -- start 2023-09-10T00:55:17.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 -- start start 2023-09-10T00:55:17.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feac8068ae0 0x7feac819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac8106770 0x7feac8198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac8198fb0 con 0x7feac8106770 2023-09-10T00:55:17.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac8199120 con 0x7feac8068120 2023-09-10T00:55:17.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.122+0000 7feacd15c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac81992e0 con 0x7feac8068ae0 2023-09-10T00:55:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac819e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39806/0 (socket says 172.21.15.3:39806) 2023-09-10T00:55:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 -- 172.21.15.3:0/1009601314 learned_addr learned my addr 172.21.15.3:0/1009601314 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:17.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac659c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feac8068ae0 0x7feac819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac759e700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac8106770 0x7feac8198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 -- 172.21.15.3:0/1009601314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feac8068ae0 msgr2=0x7feac819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feac8068ae0 0x7feac819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 -- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac8106770 msgr2=0x7feac8198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac8106770 0x7feac8198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac6d9d700 1 -- 172.21.15.3:0/1009601314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac0009c40 con 0x7feac8068120 2023-09-10T00:55:17.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.123+0000 7feac759e700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac8106770 0x7feac8198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T00:55:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.124+0000 7feac6d9d700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac819e810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feab800eb30 tx=0x7feab800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.124+0000 7feaaffff700 1 -- 172.21.15.3:0/1009601314 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8009880 con 0x7feac8068120 2023-09-10T00:55:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.124+0000 7feaaffff700 1 -- 172.21.15.3:0/1009601314 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feab80044e0 con 0x7feac8068120 2023-09-10T00:55:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.124+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab8009520 con 0x7feac8068120 2023-09-10T00:55:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.124+0000 7feaaffff700 1 -- 172.21.15.3:0/1009601314 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8003e90 con 0x7feac8068120 2023-09-10T00:55:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.124+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc0095f0 con 0x7feac8068120 2023-09-10T00:55:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.125+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac804e300 con 0x7feac8068120 2023-09-10T00:55:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.125+0000 7feaaffff700 1 -- 172.21.15.3:0/1009601314 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7feab8018430 con 0x7feac8068120 2023-09-10T00:55:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.126+0000 7feaaffff700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feab0038630 0x7feab003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.126+0000 7feaaffff700 1 -- 172.21.15.3:0/1009601314 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feab8014070 con 0x7feac8068120 2023-09-10T00:55:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.126+0000 7feac659c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feab0038630 0x7feab003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.127+0000 7feac659c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feab0038630 0x7feab003aaf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feac819a030 tx=0x7feabc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:17.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.131+0000 7feaaffff700 1 -- 172.21.15.3:0/1009601314 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7feab807de10 con 0x7feac8068120 2023-09-10T00:55:17.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.375+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) v1 -- 0x7feac81106c0 con 0x7feac8068120 2023-09-10T00:55:17.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.380+0000 7feaaffff700 1 -- 172.21.15.3:0/1009601314 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v28)=0 v28) v1 ==== 121+0+0 (secure 0 0 0) 0x7feab800fe20 con 0x7feac8068120 2023-09-10T00:55:17.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feab0038630 msgr2=0x7feab003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feab0038630 0x7feab003aaf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feac819a030 tx=0x7feabc005fb0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 msgr2=0x7feac819e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac819e810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feab800eb30 tx=0x7feab800ee40 comp rx=0 tx=0).stop 2023-09-10T00:55:17.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 shutdown_connections 2023-09-10T00:55:17.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feac8106770 0x7feac8198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feac8068120 0x7feac819e810 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feab0038630 0x7feab003aaf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 --2- 172.21.15.3:0/1009601314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feac8068ae0 0x7feac819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 >> 172.21.15.3:0/1009601314 conn(0x7feac8076600 msgr2=0x7feac810b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:17.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.382+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 shutdown_connections 2023-09-10T00:55:17.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.383+0000 7feacd15c700 1 -- 172.21.15.3:0/1009601314 wait complete. 2023-09-10T00:55:17.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.422+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) v1 -- 0x7f5334116a00 con 0x7f53340ff400 2023-09-10T00:55:17.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.433+0000 7f53317fa700 1 -- 172.21.15.3:0/2292651883 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v29)=0 v29) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5324016330 con 0x7f53340ff400 2023-09-10T00:55:17.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f531c038680 msgr2=0x7f531c03ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f531c038680 0x7f531c03ab40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f532800b2e0 tx=0x7f5328005e00 comp rx=0 tx=0).stop 2023-09-10T00:55:17.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 msgr2=0x7f5334071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:17.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 0x7f5334071ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f532400e9a0 tx=0x7f532400ecb0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 shutdown_connections 2023-09-10T00:55:17.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f531c038680 0x7f531c03ab40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5334104810 0x7f533406aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53340ff400 0x7f5334071ae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 --2- 172.21.15.3:0/2292651883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53340ffd20 0x7f5334072020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:17.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 >> 172.21.15.3:0/2292651883 conn(0x7f53340faea0 msgr2=0x7f5334102310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:17.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 shutdown_connections 2023-09-10T00:55:17.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:17.436+0000 7f533963d700 1 -- 172.21.15.3:0/2292651883 wait complete. 2023-09-10T00:55:18.245 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:18 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4030045016' entity='client.admin' 2023-09-10T00:55:18.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:18 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:18.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:18 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:18.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:18 smithi003 ceph-mon[97654]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:18 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4030045016' entity='client.admin' 2023-09-10T00:55:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:18 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:18 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:18 smithi092 ceph-mon[134067]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:18.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.638+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/2660271968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4106630 msgr2=0x7fd2c4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:18.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.638+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/2660271968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4106630 0x7fd2c4106a10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40098f0 tx=0x7fd2b4009c00 comp rx=0 tx=0).stop 2023-09-10T00:55:18.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.638+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/2660271968 shutdown_connections 2023-09-10T00:55:18.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.638+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/2660271968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4106630 0x7fd2c4106a10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.638+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/2660271968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2c4100ff0 0x7fd2c410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.638+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/2660271968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 0x7fd2c4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.638+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/2660271968 >> 172.21.15.3:0/2660271968 conn(0x7fd2c40fc120 msgr2=0x7fd2c40fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:18.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.639+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/2660271968 shutdown_connections 2023-09-10T00:55:18.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.639+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/2660271968 wait complete. 2023-09-10T00:55:18.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.639+0000 7fd2c9f8c700 1 Processor -- start 2023-09-10T00:55:18.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c9f8c700 1 -- start start 2023-09-10T00:55:18.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c9f8c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 0x7fd2c419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:18.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c9f8c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4100ff0 0x7fd2c419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:18.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c9f8c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2c4106630 0x7fd2c41965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:18.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c9f8c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4196c10 con 0x7fd2c4100630 2023-09-10T00:55:18.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c9f8c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4196d80 con 0x7fd2c4106630 2023-09-10T00:55:18.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c9f8c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4196f40 con 0x7fd2c4100ff0 2023-09-10T00:55:18.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 0x7fd2c419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:18.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4100ff0 0x7fd2c419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:18.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4100ff0 0x7fd2c419ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40144/0 (socket says 172.21.15.3:40144) 2023-09-10T00:55:18.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 0x7fd2c419c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42192/0 (socket says 172.21.15.3:42192) 2023-09-10T00:55:18.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.640+0000 7fd2c2ffd700 1 -- 172.21.15.3:0/515909667 learned_addr learned my addr 172.21.15.3:0/515909667 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:18.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c3fff700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2c4106630 0x7fd2c41965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:18.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c2ffd700 1 -- 172.21.15.3:0/515909667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2c4106630 msgr2=0x7fd2c41965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:18.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c2ffd700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2c4106630 0x7fd2c41965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c2ffd700 1 -- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 msgr2=0x7fd2c419c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:18.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c2ffd700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 0x7fd2c419c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c2ffd700 1 -- 172.21.15.3:0/515909667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8009520 con 0x7fd2c4100ff0 2023-09-10T00:55:18.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c3fff700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2c4106630 0x7fd2c41965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T00:55:18.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c37fe700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 0x7fd2c419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T00:55:18.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c2ffd700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4100ff0 0x7fd2c419ca60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800eb30 tx=0x7fd2b800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:18.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c0ff9700 1 -- 172.21.15.3:0/515909667 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8009880 con 0x7fd2c4100ff0 2023-09-10T00:55:18.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c0ff9700 1 -- 172.21.15.3:0/515909667 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2b80044e0 con 0x7fd2c4100ff0 2023-09-10T00:55:18.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b4009520 con 0x7fd2c4100ff0 2023-09-10T00:55:18.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.641+0000 7fd2c0ff9700 1 -- 172.21.15.3:0/515909667 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8003e90 con 0x7fd2c4100ff0 2023-09-10T00:55:18.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.642+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c4197410 con 0x7fd2c4100ff0 2023-09-10T00:55:18.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.643+0000 7fd2c0ff9700 1 -- 172.21.15.3:0/515909667 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7fd2b8018430 con 0x7fd2c4100ff0 2023-09-10T00:55:18.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.643+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c404e300 con 0x7fd2c4100ff0 2023-09-10T00:55:18.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.643+0000 7fd2c0ff9700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2ac038630 0x7fd2ac03aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:18.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.643+0000 7fd2c0ff9700 1 -- 172.21.15.3:0/515909667 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd2b8014070 con 0x7fd2c4100ff0 2023-09-10T00:55:18.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.643+0000 7fd2c37fe700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2ac038630 0x7fd2ac03aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:18.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.644+0000 7fd2c37fe700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2ac038630 0x7fd2ac03aaf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4005e90 tx=0x7fd2b4019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:18.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.648+0000 7fd2c0ff9700 1 -- 172.21.15.3:0/515909667 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd2b800f5a0 con 0x7fd2c4100ff0 2023-09-10T00:55:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:18 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4030045016' entity='client.admin' 2023-09-10T00:55:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:18 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:18 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:18 smithi136 ceph-mon[133708]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.859+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) v1 -- 0x7fd2c4197410 con 0x7fd2c4100ff0 2023-09-10T00:55:18.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.865+0000 7fd2c0ff9700 1 -- 172.21.15.3:0/515909667 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v30)=0 v30) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd2c4197410 con 0x7fd2c4100ff0 2023-09-10T00:55:18.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2ac038630 msgr2=0x7fd2ac03aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:18.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2ac038630 0x7fd2ac03aaf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4005e90 tx=0x7fd2b4019040 comp rx=0 tx=0).stop 2023-09-10T00:55:18.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4100ff0 msgr2=0x7fd2c419ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:18.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4100ff0 0x7fd2c419ca60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800eb30 tx=0x7fd2b800ee40 comp rx=0 tx=0).stop 2023-09-10T00:55:18.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 shutdown_connections 2023-09-10T00:55:18.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2c4106630 0x7fd2c41965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2c4100630 0x7fd2c419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2ac038630 0x7fd2ac03aaf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 --2- 172.21.15.3:0/515909667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2c4100ff0 0x7fd2c419ca60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:18.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 >> 172.21.15.3:0/515909667 conn(0x7fd2c40fc120 msgr2=0x7fd2c4104140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:18.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 shutdown_connections 2023-09-10T00:55:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:18.868+0000 7fd2c9f8c700 1 -- 172.21.15.3:0/515909667 wait complete. 2023-09-10T00:55:19.315 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-09-10T00:55:19.316 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:55:19.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.conf 2023-09-10T00:55:19.429 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:55:19.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:55:19.538 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:55:19.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.conf 2023-09-10T00:55:19.578 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:55:19.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:55:19.651 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:55:19.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2023-09-10T00:55:19.687 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:55:19.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-09-10T00:55:19.758 INFO:tasks.cephadm:Adding mgr.x on smithi003 2023-09-10T00:55:19.759 INFO:tasks.cephadm:Adding mgr.y on smithi092 2023-09-10T00:55:19.759 INFO:tasks.cephadm:Adding mgr.z on smithi136 2023-09-10T00:55:19.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch apply mgr '3;smithi003=x;smithi092=y;smithi136=z' 2023-09-10T00:55:20.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:19 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T00:55:20.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:19 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T00:55:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:19 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T00:55:20.355 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:55:21.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:20 smithi136 ceph-mon[133708]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:21.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:20 smithi092 ceph-mon[134067]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:20 smithi003 ceph-mon[97654]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:21.530 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 -- 172.21.15.136:0/535604750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c102230 msgr2=0x7f574c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:21.531 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 --2- 172.21.15.136:0/535604750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c102230 0x7f574c108ae0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f574400b0a0 tx=0x7f574400b3b0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.531 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 -- 172.21.15.136:0/535604750 shutdown_connections 2023-09-10T00:55:21.531 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 --2- 172.21.15.136:0/535604750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c102230 0x7f574c108ae0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.531 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 --2- 172.21.15.136:0/535604750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f574c100690 0x7f574c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.531 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 --2- 172.21.15.136:0/535604750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f574c101890 0x7f574c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.532 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 -- 172.21.15.136:0/535604750 >> 172.21.15.136:0/535604750 conn(0x7f574c0fbbf0 msgr2=0x7f574c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:21.532 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 -- 172.21.15.136:0/535604750 shutdown_connections 2023-09-10T00:55:21.532 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 -- 172.21.15.136:0/535604750 wait complete. 2023-09-10T00:55:21.535 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 Processor -- start 2023-09-10T00:55:21.535 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.530+0000 7f5752ae1700 1 -- start start 2023-09-10T00:55:21.535 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5752ae1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f574c100690 0x7f574c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:21.535 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5752ae1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c101890 0x7f574c196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:21.535 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5752ae1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f574c102230 0x7f574c19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:21.536 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5752ae1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c19e050 con 0x7f574c102230 2023-09-10T00:55:21.536 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5752ae1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c19e190 con 0x7f574c100690 2023-09-10T00:55:21.536 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5752ae1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c19e320 con 0x7f574c101890 2023-09-10T00:55:21.536 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c101890 0x7f574c196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:21.536 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c101890 0x7f574c196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39136/0 (socket says 172.21.15.136:39136) 2023-09-10T00:55:21.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 -- 172.21.15.136:0/4064096328 learned_addr learned my addr 172.21.15.136:0/4064096328 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-09-10T00:55:21.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f575087d700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f574c100690 0x7f574c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:21.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f575107e700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f574c102230 0x7f574c19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:21.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 -- 172.21.15.136:0/4064096328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f574c100690 msgr2=0x7f574c1964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:21.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f574c100690 0x7f574c1964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.538 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 -- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f574c102230 msgr2=0x7f574c19da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:21.538 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f574c102230 0x7f574c19da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.538 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 -- 172.21.15.136:0/4064096328 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5744009d00 con 0x7f574c101890 2023-09-10T00:55:21.538 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f574bfff700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c101890 0x7f574c196a00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f573800d980 tx=0x7f573800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:21.539 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5749ffb700 1 -- 172.21.15.136:0/4064096328 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738009b40 con 0x7f574c101890 2023-09-10T00:55:21.539 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5749ffb700 1 -- 172.21.15.136:0/4064096328 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f573800b6e0 con 0x7f574c101890 2023-09-10T00:55:21.539 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c19e600 con 0x7f574c101890 2023-09-10T00:55:21.539 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.531+0000 7f5749ffb700 1 -- 172.21.15.136:0/4064096328 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573800f620 con 0x7f574c101890 2023-09-10T00:55:21.539 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.532+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c19ebd0 con 0x7f574c101890 2023-09-10T00:55:21.540 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.532+0000 7f5749ffb700 1 -- 172.21.15.136:0/4064096328 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7f573800f780 con 0x7f574c101890 2023-09-10T00:55:21.540 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.532+0000 7f5749ffb700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f573c038630 0x7f573c03aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:21.540 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.533+0000 7f5749ffb700 1 -- 172.21.15.136:0/4064096328 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f573804b0d0 con 0x7f574c101890 2023-09-10T00:55:21.540 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.533+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c107bd0 con 0x7f574c101890 2023-09-10T00:55:21.540 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.533+0000 7f575087d700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f573c038630 0x7f573c03aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:21.541 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.534+0000 7f575087d700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f573c038630 0x7f573c03aaf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f574c1016f0 tx=0x7f5740009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:21.541 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.537+0000 7f5749ffb700 1 -- 172.21.15.136:0/4064096328 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5738016070 con 0x7f574c101890 2023-09-10T00:55:21.739 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.739+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi092=y;smithi136=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f574c10c020 con 0x7f573c038630 2023-09-10T00:55:21.746 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.746+0000 7f5749ffb700 1 -- 172.21.15.136:0/4064096328 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f574c10c020 con 0x7f573c038630 2023-09-10T00:55:21.746 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mgr update... 2023-09-10T00:55:21.748 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f573c038630 msgr2=0x7f573c03aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:21.748 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f573c038630 0x7f573c03aaf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f574c1016f0 tx=0x7f5740009190 comp rx=0 tx=0).stop 2023-09-10T00:55:21.749 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c101890 msgr2=0x7f574c196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:21.749 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c101890 0x7f574c196a00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f573800d980 tx=0x7f573800dc90 comp rx=0 tx=0).stop 2023-09-10T00:55:21.749 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 shutdown_connections 2023-09-10T00:55:21.749 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f574c102230 0x7f574c19da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.749 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f574c100690 0x7f574c1964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.750 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f573c038630 0x7f573c03aaf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.750 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 --2- 172.21.15.136:0/4064096328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f574c101890 0x7f574c196a00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:21.750 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 >> 172.21.15.136:0/4064096328 conn(0x7f574c0fbbf0 msgr2=0x7f574c104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:21.750 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 shutdown_connections 2023-09-10T00:55:21.750 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:55:21.748+0000 7f5752ae1700 1 -- 172.21.15.136:0/4064096328 wait complete. 2023-09-10T00:55:22.197 DEBUG:teuthology.orchestra.run.smithi092:mgr.y> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.y.service 2023-09-10T00:55:22.200 DEBUG:teuthology.orchestra.run.smithi136:mgr.z> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.z.service 2023-09-10T00:55:22.202 INFO:tasks.cephadm:Deploying OSDs... 2023-09-10T00:55:22.203 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T00:55:22.203 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2023-09-10T00:55:22.227 INFO:journalctl@ceph.mgr.y.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T00:55:22.231 INFO:journalctl@ceph.mgr.z.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T00:55:22.248 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-10T00:55:22.248 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2023-09-10T00:55:22.333 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-10T00:55:22.334 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:22.334 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 448 Links: 1 2023-09-10T00:55:22.334 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:22.334 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:22.334 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-09-10 00:54:08.147763375 +0000 2023-09-10T00:55:22.335 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-09-10 00:48:16.946009436 +0000 2023-09-10T00:55:22.335 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-09-10 00:48:16.946009436 +0000 2023-09-10T00:55:22.335 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-09-10 00:48:16.916009819 +0000 2023-09-10T00:55:22.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-10T00:55:22.440 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-09-10T00:55:22.441 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-09-10T00:55:22.441 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000294382 s, 1.7 MB/s 2023-09-10T00:55:22.444 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-10T00:55:22.493 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2023-09-10T00:55:22.542 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-10T00:55:22.543 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:22.543 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 454 Links: 1 2023-09-10T00:55:22.543 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:22.543 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:22.544 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-09-10 00:54:08.215762507 +0000 2023-09-10T00:55:22.544 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-09-10 00:48:17.775998856 +0000 2023-09-10T00:55:22.544 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-09-10 00:48:17.775998856 +0000 2023-09-10T00:55:22.544 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-09-10 00:48:17.747999213 +0000 2023-09-10T00:55:22.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-10T00:55:22.612 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-09-10T00:55:22.612 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-09-10T00:55:22.612 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000297203 s, 1.7 MB/s 2023-09-10T00:55:22.614 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-10T00:55:22.660 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2023-09-10T00:55:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-10T00:55:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:22.745 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 460 Links: 1 2023-09-10T00:55:22.745 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:22.745 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:22.746 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-09-10 00:54:08.287761588 +0000 2023-09-10T00:55:22.746 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-09-10 00:48:18.632987933 +0000 2023-09-10T00:55:22.746 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-09-10 00:48:18.632987933 +0000 2023-09-10T00:55:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-09-10 00:48:18.606988264 +0000 2023-09-10T00:55:22.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-10T00:55:22.850 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-09-10T00:55:22.851 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-09-10T00:55:22.851 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000300868 s, 1.7 MB/s 2023-09-10T00:55:22.853 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-10T00:55:22.899 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2023-09-10T00:55:22.984 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-10T00:55:22.984 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:22.984 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 466 Links: 1 2023-09-10T00:55:22.984 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:22.985 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:22.985 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-09-10 00:54:08.349760796 +0000 2023-09-10T00:55:22.985 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-09-10 00:48:19.582975824 +0000 2023-09-10T00:55:22.986 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-09-10 00:48:19.582975824 +0000 2023-09-10T00:55:22.987 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-09-10 00:48:19.556976155 +0000 2023-09-10T00:55:22.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-10T00:55:23.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi092=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: Saving service mgr spec with placement smithi003=x;smithi092=y;smithi136=z;count:3 2023-09-10T00:55:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:23.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-09-10T00:55:23.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:23.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-09-10T00:55:23.090 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-09-10T00:55:23.091 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000311171 s, 1.6 MB/s 2023-09-10T00:55:23.092 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-10T00:55:23.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi092=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:23.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: Saving service mgr spec with placement smithi003=x;smithi092=y;smithi136=z;count:3 2023-09-10T00:55:23.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:23.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:23.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:23.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:23.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:23.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:23.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:23.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-09-10T00:55:23.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:23.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:23.178 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-09-10T00:55:23.179 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2023-09-10T00:55:23.200 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-10T00:55:23.201 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2023-09-10T00:55:23.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi092=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: Saving service mgr spec with placement smithi003=x;smithi092=y;smithi136=z;count:3 2023-09-10T00:55:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-09-10T00:55:23.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:23.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-10T00:55:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:23.239 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 379817 Links: 1 2023-09-10T00:55:23.239 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:23.239 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:23.239 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-09-10 00:54:44.328560575 +0000 2023-09-10T00:55:23.239 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-09-10 00:45:39.500662364 +0000 2023-09-10T00:55:23.240 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-09-10 00:45:39.500662364 +0000 2023-09-10T00:55:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-09-10T00:55:23.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-10T00:55:23.309 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-09-10T00:55:23.310 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-09-10T00:55:23.310 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000246256 s, 2.1 MB/s 2023-09-10T00:55:23.311 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-10T00:55:23.373 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2023-09-10T00:55:23.434 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-10T00:55:23.435 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:23.435 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 385892 Links: 1 2023-09-10T00:55:23.435 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:23.435 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:23.435 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-09-10 00:54:44.328560575 +0000 2023-09-10T00:55:23.436 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-09-10 00:45:40.023647408 +0000 2023-09-10T00:55:23.436 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-09-10 00:45:40.023647408 +0000 2023-09-10T00:55:23.436 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-09-10T00:55:23.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-10T00:55:23.505 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-09-10T00:55:23.506 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-09-10T00:55:23.506 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000242331 s, 2.1 MB/s 2023-09-10T00:55:23.507 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-10T00:55:23.569 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2023-09-10T00:55:23.630 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-10T00:55:23.630 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:23.631 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 386000 Links: 1 2023-09-10T00:55:23.631 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:23.631 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:23.631 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-09-10 00:54:44.328560575 +0000 2023-09-10T00:55:23.631 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-09-10 00:45:40.539632652 +0000 2023-09-10T00:55:23.632 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-09-10 00:45:40.539632652 +0000 2023-09-10T00:55:23.632 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-09-10T00:55:23.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-10T00:55:23.702 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-09-10T00:55:23.702 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-09-10T00:55:23.703 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000270408 s, 1.9 MB/s 2023-09-10T00:55:23.704 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-10T00:55:23.766 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2023-09-10T00:55:23.828 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-10T00:55:23.828 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:23.829 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 392242 Links: 1 2023-09-10T00:55:23.829 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:23.829 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:23.829 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-09-10 00:54:44.328560575 +0000 2023-09-10T00:55:23.829 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-09-10 00:45:41.037618410 +0000 2023-09-10T00:55:23.830 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-09-10 00:45:41.037618410 +0000 2023-09-10T00:55:23.830 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-09-10T00:55:23.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-10T00:55:23.899 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-09-10T00:55:23.900 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-09-10T00:55:23.900 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000220789 s, 2.3 MB/s 2023-09-10T00:55:23.901 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-10T00:55:23.963 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-09-10T00:55:23.964 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2023-09-10T00:55:23.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:23 smithi092 ceph-mon[134067]: Deploying daemon mgr.z on smithi136 2023-09-10T00:55:23.982 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-10T00:55:23.982 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2023-09-10T00:55:24.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:23 smithi136 ceph-mon[133708]: Deploying daemon mgr.z on smithi136 2023-09-10T00:55:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-10T00:55:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 380622 Links: 1 2023-09-10T00:55:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-09-10 00:55:23.986950018 +0000 2023-09-10T00:55:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-09-10 00:45:41.074807976 +0000 2023-09-10T00:55:24.045 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-09-10 00:45:41.074807976 +0000 2023-09-10T00:55:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-09-10T00:55:24.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-10T00:55:24.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:23 smithi003 ceph-mon[97654]: Deploying daemon mgr.z on smithi136 2023-09-10T00:55:24.114 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-09-10T00:55:24.114 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-09-10T00:55:24.115 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00025598 s, 2.0 MB/s 2023-09-10T00:55:24.116 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-10T00:55:24.174 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2023-09-10T00:55:24.232 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-10T00:55:24.232 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:24.233 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 391198 Links: 1 2023-09-10T00:55:24.233 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:24.233 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:24.233 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-09-10 00:55:23.987949996 +0000 2023-09-10T00:55:24.234 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-09-10 00:45:41.577796621 +0000 2023-09-10T00:55:24.234 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-09-10 00:45:41.577796621 +0000 2023-09-10T00:55:24.234 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-09-10T00:55:24.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-10T00:55:24.296 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 systemd[1]: Starting Ceph mgr.z for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T00:55:24.298 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-09-10T00:55:24.299 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-09-10T00:55:24.299 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000222735 s, 2.3 MB/s 2023-09-10T00:55:24.300 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-10T00:55:24.360 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2023-09-10T00:55:24.418 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-10T00:55:24.418 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:24.418 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 381672 Links: 1 2023-09-10T00:55:24.419 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:24.419 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:24.419 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-09-10 00:55:23.987949996 +0000 2023-09-10T00:55:24.419 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-09-10 00:45:42.119784384 +0000 2023-09-10T00:55:24.419 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-09-10 00:45:42.119784384 +0000 2023-09-10T00:55:24.420 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-09-10T00:55:24.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-10T00:55:24.487 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-09-10T00:55:24.487 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-09-10T00:55:24.487 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000277832 s, 1.8 MB/s 2023-09-10T00:55:24.488 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-10T00:55:24.545 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2023-09-10T00:55:24.590 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 podman[135647]: 2023-09-10T00:55:24.603 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-10T00:55:24.603 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-10T00:55:24.603 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 391391 Links: 1 2023-09-10T00:55:24.604 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T00:55:24.604 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2023-09-10T00:55:24.604 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-09-10 00:55:23.987949996 +0000 2023-09-10T00:55:24.604 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-09-10 00:45:42.625772961 +0000 2023-09-10T00:55:24.604 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-09-10 00:45:42.625772961 +0000 2023-09-10T00:55:24.605 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-09-10T00:55:24.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-10T00:55:24.671 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-09-10T00:55:24.672 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-09-10T00:55:24.672 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00020459 s, 2.5 MB/s 2023-09-10T00:55:24.673 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-10T00:55:24.732 INFO:tasks.cephadm:Deploying osd.0 on smithi003 with /dev/vg_nvme/lv_4... 2023-09-10T00:55:24.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-09-10T00:55:24.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:24.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:24.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:24.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:24.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:24.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-09-10T00:55:24.892 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:24.892 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:24.892 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-mgr[135811]: -- 172.21.15.136:0/240977686 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f9acb051e0 con 0x55f9acadb000 2023-09-10T00:55:24.892 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 bash[135647]: 27520df7ae584d27746bb269b07efb1186dd9f40ca3ab0045c854375a5744fdc 2023-09-10T00:55:24.893 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 systemd[1]: Started Ceph mgr.z for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:55:24.893 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:24.717+0000 7f894253f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T00:55:24.893 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:24.800+0000 7f894253f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T00:55:24.893 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:24.889+0000 7f894253f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T00:55:25.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:25.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:25.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:25.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:25.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:25.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-09-10T00:55:25.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:25.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:25.151 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:24 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:24.982+0000 7f894253f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T00:55:25.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-09-10T00:55:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:25.411 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:25.149+0000 7f894253f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T00:55:25.466 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:25.824 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:25.511+0000 7f894253f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T00:55:25.825 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:25.641+0000 7f894253f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T00:55:25.825 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:25.729+0000 7f894253f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T00:55:25.957 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:25 smithi092 ceph-mon[134067]: Deploying daemon mgr.y on smithi092 2023-09-10T00:55:26.018 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:25 smithi003 ceph-mon[97654]: Deploying daemon mgr.y on smithi092 2023-09-10T00:55:26.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:25 smithi136 ceph-mon[133708]: Deploying daemon mgr.y on smithi092 2023-09-10T00:55:26.163 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:25.823+0000 7f894253f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T00:55:26.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:26 smithi092 systemd[1]: Starting Ceph mgr.y for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T00:55:26.662 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:26 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:26.244+0000 7f894253f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T00:55:26.768 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:26 smithi092 podman[135624]: 2023-09-10T00:55:26.768 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:26 smithi092 bash[135624]: 8b62e4f31d91b9ff236010d7b5dcb6b02ae81db6101639f6b936bfc60031bf8f 2023-09-10T00:55:26.768 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:26 smithi092 ceph-mgr[135664]: -- 172.21.15.92:0/1939162464 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fe12b971e0 con 0x55fe12b6cc00 2023-09-10T00:55:27.021 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:26 smithi092 ceph-mon[134067]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:27.021 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.022 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:26 smithi092 systemd[1]: Started Ceph mgr.y for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T00:55:27.022 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:26 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:26.846+0000 7fa267fc4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T00:55:27.022 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:26 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:26.928+0000 7fa267fc4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T00:55:27.023 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:27.020+0000 7fa267fc4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T00:55:27.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:26 smithi003 ceph-mon[97654]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.161 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:26 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:26.909+0000 7f894253f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T00:55:27.162 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:26 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:26.991+0000 7f894253f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T00:55:27.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:26 smithi136 ceph-mon[133708]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:27.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.307 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:27.125+0000 7fa267fc4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T00:55:27.394 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-09-10T00:55:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-09-10T00:55:27.395 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-09-10T00:55:27.395 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-09-10T00:55:27.395 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0358761 s, 292 MB/s 2023-09-10T00:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-09-10T00:55:27.684 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:27.306+0000 7fa267fc4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T00:55:27.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi003:vg_nvme/lv_4 2023-09-10T00:55:27.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:27.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:27.966 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:27.683+0000 7fa267fc4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T00:55:27.966 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:27.822+0000 7fa267fc4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T00:55:27.967 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:27.916+0000 7fa267fc4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T00:55:28.110 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:28.110 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:28.110 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:28.111 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:27.819+0000 7f894253f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:55:28.111 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:27.924+0000 7f894253f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T00:55:28.111 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:28.107+0000 7f894253f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:55:28.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:28.461 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:28 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:28.016+0000 7fa267fc4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T00:55:28.576 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:55:28.653 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:28.397+0000 7f894253f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T00:55:28.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:28 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:28.460+0000 7fa267fc4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T00:55:28.912 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:28.652+0000 7f894253f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T00:55:28.912 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:28.813+0000 7f894253f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T00:55:28.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-mon[133708]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:28.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:28.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:28.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:28.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:28 smithi003 ceph-mon[97654]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:29.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.150 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.208 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:29.125+0000 7fa267fc4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T00:55:29.208 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:28 smithi092 ceph-mon[134067]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:29.208 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.209 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.209 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.209 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:29.412 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:29.095+0000 7f894253f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T00:55:29.465 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:29.207+0000 7fa267fc4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T00:55:30.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 -- 172.21.15.3:0/4198473646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 msgr2=0x7f6328100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:30.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 --2- 172.21.15.3:0/4198473646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328100ab0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f63180099c0 tx=0x7f6318009cd0 comp rx=0 tx=0).stop 2023-09-10T00:55:30.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 -- 172.21.15.3:0/4198473646 shutdown_connections 2023-09-10T00:55:30.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 --2- 172.21.15.3:0/4198473646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328102230 0x7f6328108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:30.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 --2- 172.21.15.3:0/4198473646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328100ab0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 --2- 172.21.15.3:0/4198473646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6328101890 0x7f6328101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 -- 172.21.15.3:0/4198473646 >> 172.21.15.3:0/4198473646 conn(0x7f63280fbbf0 msgr2=0x7f63280fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 -- 172.21.15.3:0/4198473646 shutdown_connections 2023-09-10T00:55:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.101+0000 7f632d3c9700 1 -- 172.21.15.3:0/4198473646 wait complete. 2023-09-10T00:55:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.102+0000 7f632d3c9700 1 Processor -- start 2023-09-10T00:55:30.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.102+0000 7f632d3c9700 1 -- start start 2023-09-10T00:55:30.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.102+0000 7f632d3c9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:30.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.102+0000 7f632d3c9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6328101890 0x7f6328198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:30.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f632d3c9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328102230 0x7f632819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:30.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f632d3c9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63281a0110 con 0x7f6328100690 2023-09-10T00:55:30.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f632d3c9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63281a0250 con 0x7f6328102230 2023-09-10T00:55:30.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f632d3c9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63281a03e0 con 0x7f6328101890 2023-09-10T00:55:30.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:30.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f63267fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6328101890 0x7f6328198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:30.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43196/0 (socket says 172.21.15.3:43196) 2023-09-10T00:55:30.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 -- 172.21.15.3:0/2079501401 learned_addr learned my addr 172.21.15.3:0/2079501401 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:55:30.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f63277fe700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328102230 0x7f632819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:30.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 -- 172.21.15.3:0/2079501401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6328101890 msgr2=0x7f6328198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:30.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6328101890 0x7f6328198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:30.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 -- 172.21.15.3:0/2079501401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328102230 msgr2=0x7f632819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:30.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328102230 0x7f632819fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f6326ffd700 1 -- 172.21.15.3:0/2079501401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63180095f0 con 0x7f6328100690 2023-09-10T00:55:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.103+0000 7f63277fe700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328102230 0x7f632819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T00:55:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.104+0000 7f6326ffd700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328198580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f63180098e0 tx=0x7f631800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.104+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631801d070 con 0x7f6328100690 2023-09-10T00:55:30.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.104+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6318005050 con 0x7f6328100690 2023-09-10T00:55:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.104+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318004a70 con 0x7f6328100690 2023-09-10T00:55:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.104+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63281a0660 con 0x7f6328100690 2023-09-10T00:55:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.104+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63281a0a70 con 0x7f6328100690 2023-09-10T00:55:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.105+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45386+0+0 (secure 0 0 0) 0x7f63180051c0 con 0x7f6328100690 2023-09-10T00:55:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.105+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328192cd0 con 0x7f6328100690 2023-09-10T00:55:30.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.105+0000 7f630ffff700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6310038630 0x7f631003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:30.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.105+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f631804bcb0 con 0x7f6328100690 2023-09-10T00:55:30.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.106+0000 7f63267fc700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6310038630 0x7f631003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:30.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.106+0000 7f63267fc700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6310038630 0x7f631003aaf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f631c009c60 tx=0x7f631c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:30.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.111+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6318016b90 con 0x7f6328100690 2023-09-10T00:55:30.127 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:29.800+0000 7f894253f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T00:55:30.128 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:29.933+0000 7f894253f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T00:55:30.128 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:30.128 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:30.128 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:30.129 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:30.129 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:30.129 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:30.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:29.976+0000 7fa267fc4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:55:30.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:30.066+0000 7fa267fc4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T00:55:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:30.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:30.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:30.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-09-10T00:55:30.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-09-10T00:55:30.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:30.343+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f632802cf10 con 0x7f6310038630 2023-09-10T00:55:30.411 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:30.126+0000 7f894253f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T00:55:30.412 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:30.238+0000 7f894253f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T00:55:30.512 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:30.237+0000 7fa267fc4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T00:55:30.884 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:30.511+0000 7fa267fc4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T00:55:30.885 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:30.736+0000 7fa267fc4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T00:55:30.891 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:30.453+0000 7f894253f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T00:55:31.145 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:30.884+0000 7fa267fc4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T00:55:31.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-mon[134067]: Reconfiguring mgr.x (unknown last config time)... 2023-09-10T00:55:31.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-mon[134067]: Reconfiguring daemon mgr.x on smithi003 2023-09-10T00:55:31.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-mon[134067]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:31.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:55:31.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:55:31.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:31.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:31.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-mon[133708]: Reconfiguring mgr.x (unknown last config time)... 2023-09-10T00:55:31.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-mon[133708]: Reconfiguring daemon mgr.x on smithi003 2023-09-10T00:55:31.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-mon[133708]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:31.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:55:31.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:55:31.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:31.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:31.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:30 smithi003 ceph-mon[97654]: Reconfiguring mgr.x (unknown last config time)... 2023-09-10T00:55:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:30 smithi003 ceph-mon[97654]: Reconfiguring daemon mgr.x on smithi003 2023-09-10T00:55:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:30 smithi003 ceph-mon[97654]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:55:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:55:31.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:31.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:31.465 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:31.144+0000 7fa267fc4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T00:55:31.661 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 00:55:31 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T00:55:31.320+0000 7f894253f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T00:55:32.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:32.031+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 14) v1 ==== 90294+0+0 (secure 0 0 0) 0x7f63180163a0 con 0x7f6328100690 2023-09-10T00:55:32.169 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:31.840+0000 7fa267fc4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T00:55:32.169 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:31.972+0000 7fa267fc4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T00:55:32.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:32.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:32.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:32.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:32.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: Standby manager daemon z started 2023-09-10T00:55:32.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-09-10T00:55:32.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T00:55:32.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-09-10T00:55:32.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T00:55:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: Standby manager daemon z started 2023-09-10T00:55:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-09-10T00:55:32.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T00:55:32.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-09-10T00:55:32.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:32 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T00:55:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: Standby manager daemon z started 2023-09-10T00:55:32.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-09-10T00:55:32.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T00:55:32.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-09-10T00:55:32.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:32 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/2682152022' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T00:55:32.465 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:32.168+0000 7fa267fc4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T00:55:32.465 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:32.281+0000 7fa267fc4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T00:55:32.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:32.497+0000 7fa267fc4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T00:55:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:33 smithi003 ceph-mon[97654]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:33 smithi003 ceph-mon[97654]: mgrmap e14: x(active, since 96s), standbys: z 2023-09-10T00:55:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:33 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-09-10T00:55:33.352 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:33 smithi092 ceph-mon[134067]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:33.352 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:33 smithi092 ceph-mon[134067]: mgrmap e14: x(active, since 96s), standbys: z 2023-09-10T00:55:33.353 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:33 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-09-10T00:55:33.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:33 smithi136 ceph-mon[133708]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:33.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:33 smithi136 ceph-mon[133708]: mgrmap e14: x(active, since 96s), standbys: z 2023-09-10T00:55:33.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:33 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-09-10T00:55:33.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 00:55:33 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T00:55:33.351+0000 7fa267fc4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T00:55:34.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:34.014+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 15) v1 ==== 135202+0+0 (secure 0 0 0) 0x7f631800fae0 con 0x7f6328100690 2023-09-10T00:55:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: Standby manager daemon y started 2023-09-10T00:55:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-09-10T00:55:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T00:55:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-09-10T00:55:34.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T00:55:34.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3265960011' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaca1361-1d53-4fff-b909-d973a1ce185a"}]: dispatch 2023-09-10T00:55:34.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3265960011' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaca1361-1d53-4fff-b909-d973a1ce185a"}]': finished 2023-09-10T00:55:34.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: osdmap e5: 1 total, 0 up, 1 in 2023-09-10T00:55:34.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:34.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:34 smithi003 ceph-mon[97654]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: Standby manager daemon y started 2023-09-10T00:55:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-09-10T00:55:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T00:55:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-09-10T00:55:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T00:55:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3265960011' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaca1361-1d53-4fff-b909-d973a1ce185a"}]: dispatch 2023-09-10T00:55:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3265960011' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaca1361-1d53-4fff-b909-d973a1ce185a"}]': finished 2023-09-10T00:55:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: osdmap e5: 1 total, 0 up, 1 in 2023-09-10T00:55:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:34.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:34 smithi136 ceph-mon[133708]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:34.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: Standby manager daemon y started 2023-09-10T00:55:34.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-09-10T00:55:34.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T00:55:34.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-09-10T00:55:34.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/2335551633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T00:55:34.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3265960011' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaca1361-1d53-4fff-b909-d973a1ce185a"}]: dispatch 2023-09-10T00:55:34.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3265960011' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaca1361-1d53-4fff-b909-d973a1ce185a"}]': finished 2023-09-10T00:55:34.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: osdmap e5: 1 total, 0 up, 1 in 2023-09-10T00:55:34.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:34.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:34 smithi092 ceph-mon[134067]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:35.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:35 smithi003 ceph-mon[97654]: mgrmap e15: x(active, since 98s), standbys: z, y 2023-09-10T00:55:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:35 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-09-10T00:55:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:35 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2894436892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:55:35.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:35 smithi136 ceph-mon[133708]: mgrmap e15: x(active, since 98s), standbys: z, y 2023-09-10T00:55:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:35 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-09-10T00:55:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:35 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2894436892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:55:35.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:35 smithi092 ceph-mon[134067]: mgrmap e15: x(active, since 98s), standbys: z, y 2023-09-10T00:55:35.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:35 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-09-10T00:55:35.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:35 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2894436892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:55:36.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:36 smithi003 ceph-mon[97654]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:36.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:36 smithi136 ceph-mon[133708]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:36 smithi092 ceph-mon[134067]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:39.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:38 smithi003 ceph-mon[97654]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:39.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:38 smithi136 ceph-mon[133708]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:39.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:38 smithi092 ceph-mon[134067]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:41.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:40 smithi136 ceph-mon[133708]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:41.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:40 smithi092 ceph-mon[134067]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:41.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:40 smithi003 ceph-mon[97654]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:42.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-09-10T00:55:42.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:42.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:41 smithi136 ceph-mon[133708]: Deploying daemon osd.0 on smithi003 2023-09-10T00:55:42.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-09-10T00:55:42.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:42.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:41 smithi003 ceph-mon[97654]: Deploying daemon osd.0 on smithi003 2023-09-10T00:55:42.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-09-10T00:55:42.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:42.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:41 smithi092 ceph-mon[134067]: Deploying daemon osd.0 on smithi003 2023-09-10T00:55:43.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:42 smithi136 ceph-mon[133708]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:43.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:42 smithi092 ceph-mon[134067]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:43.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:42 smithi003 ceph-mon[97654]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:45.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:44 smithi136 ceph-mon[133708]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:45.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:44 smithi092 ceph-mon[134067]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:44 smithi003 ceph-mon[97654]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:47.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:46 smithi136 ceph-mon[133708]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:47.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:46 smithi092 ceph-mon[134067]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:47.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:46 smithi003 ceph-mon[97654]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:48.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:48.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:48.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:48.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:48.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:48.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:49.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:48 smithi003 ceph-mon[97654]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:49.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:48 smithi092 ceph-mon[134067]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:49.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:48 smithi136 ceph-mon[133708]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:50.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.307+0000 7f630ffff700 1 -- 172.21.15.3:0/2079501401 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f632802cf10 con 0x7f6310038630 2023-09-10T00:55:50.309 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 0 on host 'smithi003' 2023-09-10T00:55:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.309+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6310038630 msgr2=0x7f631003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.309+0000 7f632d3c9700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6310038630 0x7f631003aaf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f631c009c60 tx=0x7f631c009260 comp rx=0 tx=0).stop 2023-09-10T00:55:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.309+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 msgr2=0x7f6328198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:50.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.309+0000 7f632d3c9700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328198580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f63180098e0 tx=0x7f631800f690 comp rx=0 tx=0).stop 2023-09-10T00:55:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 shutdown_connections 2023-09-10T00:55:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6328102230 0x7f632819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6328100690 0x7f6328198580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6310038630 0x7f631003aaf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:50.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 --2- 172.21.15.3:0/2079501401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6328101890 0x7f6328198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:50.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 >> 172.21.15.3:0/2079501401 conn(0x7f63280fbbf0 msgr2=0x7f63281071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:50.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 shutdown_connections 2023-09-10T00:55:50.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:55:50.310+0000 7f632d3c9700 1 -- 172.21.15.3:0/2079501401 wait complete. 2023-09-10T00:55:50.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:50.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:50.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:50.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:50.711 DEBUG:teuthology.orchestra.run.smithi003:osd.0> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.0.service 2023-09-10T00:55:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:50.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:50.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:50.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:50.720 INFO:tasks.cephadm:Deploying osd.1 on smithi092 with /dev/vg_nvme/lv_4... 2023-09-10T00:55:50.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-09-10T00:55:50.788 INFO:journalctl@ceph.osd.0.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T00:55:50.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:50.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:50.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:50.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:50.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:55:51.059 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:55:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-0[119797]: 2023-09-10T00:55:50.912+0000 7f1e5eb3d640 -1 osd.0 0 log_to_monitors true 2023-09-10T00:55:51.294 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:55:51.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:51 smithi003 ceph-mon[97654]: from='osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-09-10T00:55:51.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:51 smithi003 ceph-mon[97654]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-09-10T00:55:51.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:51 smithi092 ceph-mon[134067]: from='osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-09-10T00:55:51.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:51 smithi092 ceph-mon[134067]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-09-10T00:55:51.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:51 smithi136 ceph-mon[133708]: from='osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-09-10T00:55:51.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:51 smithi136 ceph-mon[133708]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-09-10T00:55:52.559 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-0[119797]: 2023-09-10T00:55:52.462+0000 7f1e541b4700 -1 osd.0 0 waiting for initial osdmap 2023-09-10T00:55:52.559 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-0[119797]: 2023-09-10T00:55:52.466+0000 7f1e4f7a8700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:55:52.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-mon[97654]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-09-10T00:55:52.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-mon[97654]: from='osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:55:52.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-mon[97654]: osdmap e6: 1 total, 0 up, 1 in 2023-09-10T00:55:52.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:52.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-mon[97654]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:55:52.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:52 smithi003 ceph-mon[97654]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:52.676 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-09-10T00:55:52.676 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-09-10T00:55:52.676 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-09-10T00:55:52.677 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-09-10T00:55:52.677 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109785 s, 955 MB/s 2023-09-10T00:55:52.677 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-09-10T00:55:52.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:52 smithi092 ceph-mon[134067]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-09-10T00:55:52.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:52 smithi092 ceph-mon[134067]: from='osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:55:52.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:52 smithi092 ceph-mon[134067]: osdmap e6: 1 total, 0 up, 1 in 2023-09-10T00:55:52.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:52 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:52.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:52 smithi092 ceph-mon[134067]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:55:52.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:52 smithi092 ceph-mon[134067]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:52.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:52 smithi136 ceph-mon[133708]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-09-10T00:55:52.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:52 smithi136 ceph-mon[133708]: from='osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:55:52.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:52 smithi136 ceph-mon[133708]: osdmap e6: 1 total, 0 up, 1 in 2023-09-10T00:55:52.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:52 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:52.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:52 smithi136 ceph-mon[133708]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:55:52.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:52 smithi136 ceph-mon[133708]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:53.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi092:vg_nvme/lv_4 2023-09-10T00:55:53.600 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:55:53.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:53 smithi003 ceph-mon[97654]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:55:53.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:53 smithi003 ceph-mon[97654]: osdmap e7: 1 total, 0 up, 1 in 2023-09-10T00:55:53.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:53.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:53 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:53.776 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:53 smithi092 ceph-mon[134067]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:55:53.776 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:53 smithi092 ceph-mon[134067]: osdmap e7: 1 total, 0 up, 1 in 2023-09-10T00:55:53.777 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:53.777 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:53 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:53.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:53 smithi136 ceph-mon[133708]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:55:53.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:53 smithi136 ceph-mon[133708]: osdmap e7: 1 total, 0 up, 1 in 2023-09-10T00:55:53.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:53.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:53 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:54.593 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:55:54.594 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:55:54.594 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] boot 2023-09-10T00:55:54.594 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: osdmap e8: 1 total, 1 up, 1 in 2023-09-10T00:55:54.594 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:54.595 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:54.595 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi003 2023-09-10T00:55:54.595 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.595 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.595 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:55:54.596 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi003 to 17272M 2023-09-10T00:55:54.596 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.596 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:54.596 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:54.596 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.784 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 -- 172.21.15.92:0/291719855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 msgr2=0x7f7d20107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:54.785 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/291719855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20107a60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800b0a0 tx=0x7f7d1800b3b0 comp rx=0 tx=0).stop 2023-09-10T00:55:54.785 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 -- 172.21.15.92:0/291719855 shutdown_connections 2023-09-10T00:55:54.785 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/291719855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d20100ba0 0x7f7d20100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:54.785 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/291719855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20107a60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:54.786 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/291719855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d200ff800 0x7f7d200ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:54.786 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 -- 172.21.15.92:0/291719855 >> 172.21.15.92:0/291719855 conn(0x7f7d200fa2b0 msgr2=0x7f7d200fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:55:54.786 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 -- 172.21.15.92:0/291719855 shutdown_connections 2023-09-10T00:55:54.786 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.781+0000 7f7d27e7f700 1 -- 172.21.15.92:0/291719855 wait complete. 2023-09-10T00:55:54.786 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 Processor -- start 2023-09-10T00:55:54.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 -- start start 2023-09-10T00:55:54.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d200ff800 0x7f7d20196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:54.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:54.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d20100ba0 0x7f7d2019da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:54.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2019dfe0 con 0x7f7d20100ba0 2023-09-10T00:55:54.788 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2019e120 con 0x7f7d201001c0 2023-09-10T00:55:54.788 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d27e7f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2019e2b0 con 0x7f7d200ff800 2023-09-10T00:55:54.788 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d2541a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:54.788 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d2541a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:51058/0 (socket says 172.21.15.92:51058) 2023-09-10T00:55:54.788 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d2541a700 1 -- 172.21.15.92:0/1490103564 learned_addr learned my addr 172.21.15.92:0/1490103564 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-09-10T00:55:54.789 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d2641c700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d20100ba0 0x7f7d2019da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:54.789 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d25c1b700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d200ff800 0x7f7d20196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:54.789 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d2541a700 1 -- 172.21.15.92:0/1490103564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d200ff800 msgr2=0x7f7d20196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:54.789 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d2541a700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d200ff800 0x7f7d20196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:54.789 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.782+0000 7f7d2541a700 1 -- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d20100ba0 msgr2=0x7f7d2019da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:55:54.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d2541a700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d20100ba0 0x7f7d2019da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:55:54.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d2541a700 1 -- 172.21.15.92:0/1490103564 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d18009d00 con 0x7f7d201001c0 2023-09-10T00:55:54.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d25c1b700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d200ff800 0x7f7d20196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T00:55:54.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d2541a700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20196990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00c920 tx=0x7f7d1c00cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:54.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d16ffd700 1 -- 172.21.15.92:0/1490103564 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c007b00 con 0x7f7d201001c0 2023-09-10T00:55:54.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d16ffd700 1 -- 172.21.15.92:0/1490103564 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d1c004cf0 con 0x7f7d201001c0 2023-09-10T00:55:54.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2019e590 con 0x7f7d201001c0 2023-09-10T00:55:54.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d16ffd700 1 -- 172.21.15.92:0/1490103564 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0055e0 con 0x7f7d201001c0 2023-09-10T00:55:54.792 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.783+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2019ea90 con 0x7f7d201001c0 2023-09-10T00:55:54.792 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.784+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2010ae20 con 0x7f7d201001c0 2023-09-10T00:55:54.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.785+0000 7f7d16ffd700 1 -- 172.21.15.92:0/1490103564 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 15) v1 ==== 135202+0+0 (secure 0 0 0) 0x7f7d1c007c60 con 0x7f7d201001c0 2023-09-10T00:55:54.794 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.785+0000 7f7d16ffd700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7d0c0a0700 0x7f7d0c0a2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:55:54.794 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.785+0000 7f7d16ffd700 1 -- 172.21.15.92:0/1490103564 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f7d1c0c9c30 con 0x7f7d201001c0 2023-09-10T00:55:54.794 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.786+0000 7f7d25c1b700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7d0c0a0700 0x7f7d0c0a2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:55:54.795 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.786+0000 7f7d25c1b700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7d0c0a0700 0x7f7d0c0a2bc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d20067ea0 tx=0x7f7d10005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:55:54.795 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.790+0000 7f7d16ffd700 1 -- 172.21.15.92:0/1490103564 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7d1c0951d0 con 0x7f7d201001c0 2023-09-10T00:55:54.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:55:54.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:55:54.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] boot 2023-09-10T00:55:54.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: osdmap e8: 1 total, 1 up, 1 in 2023-09-10T00:55:54.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:54.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:54.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi003 2023-09-10T00:55:54.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:55:54.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi003 to 17272M 2023-09-10T00:55:54.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:54.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:54.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:55:54.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:55:54.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: osd.0 [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] boot 2023-09-10T00:55:54.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: osdmap e8: 1 total, 1 up, 1 in 2023-09-10T00:55:54.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T00:55:54.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-09-10T00:55:54.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi003 2023-09-10T00:55:54.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:55:54.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi003 to 17272M 2023-09-10T00:55:54.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:54.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:55:54.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:55:54.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:55:54.988+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d200fef00 con 0x7f7d0c0a0700 2023-09-10T00:55:55.758 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:55 smithi092 ceph-mon[134067]: osdmap e9: 1 total, 1 up, 1 in 2023-09-10T00:55:55.758 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:55 smithi092 ceph-mon[134067]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:55.759 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:55:55.759 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:55:55.759 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:55.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:55 smithi003 ceph-mon[97654]: osdmap e9: 1 total, 1 up, 1 in 2023-09-10T00:55:55.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:55 smithi003 ceph-mon[97654]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:55.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:55:55.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:55:55.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:55.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:55 smithi136 ceph-mon[133708]: osdmap e9: 1 total, 1 up, 1 in 2023-09-10T00:55:55.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:55 smithi136 ceph-mon[133708]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:55:55.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:55:55.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:55:55.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:55:56.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:56 smithi003 ceph-mon[97654]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:55:56.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:56 smithi136 ceph-mon[133708]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:55:56.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:56 smithi092 ceph-mon[134067]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:55:57.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:57 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/210440592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]: dispatch 2023-09-10T00:55:57.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:57 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]: dispatch 2023-09-10T00:55:57.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:57 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]': finished 2023-09-10T00:55:57.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:57 smithi003 ceph-mon[97654]: osdmap e10: 2 total, 1 up, 2 in 2023-09-10T00:55:57.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:55:57.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:57 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/210440592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]: dispatch 2023-09-10T00:55:57.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:57 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]: dispatch 2023-09-10T00:55:57.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:57 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]': finished 2023-09-10T00:55:57.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:57 smithi136 ceph-mon[133708]: osdmap e10: 2 total, 1 up, 2 in 2023-09-10T00:55:57.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:55:57.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:57 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/210440592' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]: dispatch 2023-09-10T00:55:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:57 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]: dispatch 2023-09-10T00:55:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:57 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1f77daa-c19a-4eba-aebf-27ca13859ee5"}]': finished 2023-09-10T00:55:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:57 smithi092 ceph-mon[134067]: osdmap e10: 2 total, 1 up, 2 in 2023-09-10T00:55:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:55:58.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:58 smithi003 ceph-mon[97654]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:55:58.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:55:58 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/2793384290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:55:58.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:58 smithi136 ceph-mon[133708]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:55:58.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:55:58 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/2793384290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:55:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:58 smithi092 ceph-mon[134067]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:55:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:55:58 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/2793384290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:56:01.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:00 smithi092 ceph-mon[134067]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:01.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:00 smithi136 ceph-mon[133708]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:00 smithi003 ceph-mon[97654]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:03.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:02 smithi136 ceph-mon[133708]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:03.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:02 smithi092 ceph-mon[134067]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:02 smithi003 ceph-mon[97654]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:03 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-09-10T00:56:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:03 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:04.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:03 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-09-10T00:56:04.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:03 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:03 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-09-10T00:56:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:03 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:04.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:04 smithi092 ceph-mon[134067]: Deploying daemon osd.1 on smithi092 2023-09-10T00:56:04.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:04 smithi092 ceph-mon[134067]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:05.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:04 smithi136 ceph-mon[133708]: Deploying daemon osd.1 on smithi092 2023-09-10T00:56:05.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:04 smithi136 ceph-mon[133708]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:05.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:04 smithi003 ceph-mon[97654]: Deploying daemon osd.1 on smithi092 2023-09-10T00:56:05.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:04 smithi003 ceph-mon[97654]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:06.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:06 smithi092 ceph-mon[134067]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:06 smithi136 ceph-mon[133708]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:07.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:06 smithi003 ceph-mon[97654]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:09.117 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:08 smithi092 ceph-mon[134067]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:09.117 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:09.117 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:09.117 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:08 smithi136 ceph-mon[133708]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:09.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:08 smithi003 ceph-mon[97654]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:10.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.551+0000 7f7d16ffd700 1 -- 172.21.15.92:0/1490103564 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7d200fef00 con 0x7f7d0c0a0700 2023-09-10T00:56:10.552 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 1 on host 'smithi092' 2023-09-10T00:56:10.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7d0c0a0700 msgr2=0x7f7d0c0a2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:10.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7d0c0a0700 0x7f7d0c0a2bc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d20067ea0 tx=0x7f7d10005f90 comp rx=0 tx=0).stop 2023-09-10T00:56:10.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 msgr2=0x7f7d20196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:10.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20196990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c00c920 tx=0x7f7d1c00cce0 comp rx=0 tx=0).stop 2023-09-10T00:56:10.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 shutdown_connections 2023-09-10T00:56:10.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7d0c0a0700 0x7f7d0c0a2bc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:10.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d200ff800 0x7f7d20196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:10.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d20100ba0 0x7f7d2019da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:10.555 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 --2- 172.21.15.92:0/1490103564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d201001c0 0x7f7d20196990 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:10.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 >> 172.21.15.92:0/1490103564 conn(0x7f7d200fa2b0 msgr2=0x7f7d200fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:56:10.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 shutdown_connections 2023-09-10T00:56:10.556 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:56:10.553+0000 7f7d27e7f700 1 -- 172.21.15.92:0/1490103564 wait complete. 2023-09-10T00:56:10.984 DEBUG:teuthology.orchestra.run.smithi092:osd.1> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.1.service 2023-09-10T00:56:10.987 INFO:tasks.cephadm:Deploying osd.2 on smithi136 with /dev/vg_nvme/lv_4... 2023-09-10T00:56:10.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-09-10T00:56:11.018 INFO:journalctl@ceph.osd.1.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T00:56:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:11.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:11.551 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:56:12.216 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-1[139135]: 2023-09-10T00:56:11.984+0000 7fe6a7f32640 -1 osd.1 0 log_to_monitors true 2023-09-10T00:56:13.019 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-09-10T00:56:13.019 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-09-10T00:56:13.020 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-09-10T00:56:13.020 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-09-10T00:56:13.020 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00767413 s, 1.4 GB/s 2023-09-10T00:56:13.020 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-09-10T00:56:13.149 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:12 smithi136 ceph-mon[133708]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:13.150 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:12 smithi136 ceph-mon[133708]: from='osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-09-10T00:56:13.150 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:12 smithi136 ceph-mon[133708]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-09-10T00:56:13.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:12 smithi092 ceph-mon[134067]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:12 smithi092 ceph-mon[134067]: from='osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-09-10T00:56:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:12 smithi092 ceph-mon[134067]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-09-10T00:56:13.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:12 smithi003 ceph-mon[97654]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:12 smithi003 ceph-mon[97654]: from='osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-09-10T00:56:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:12 smithi003 ceph-mon[97654]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-09-10T00:56:13.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2023-09-10T00:56:13.943 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:56:14.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-09-10T00:56:14.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: osdmap e11: 2 total, 1 up, 2 in 2023-09-10T00:56:14.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:14.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:56:14.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:56:14.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:56:14.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:14.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:14.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.215 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-1[139135]: 2023-09-10T00:56:13.903+0000 7fe69d5a9700 -1 osd.1 0 waiting for initial osdmap 2023-09-10T00:56:14.216 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-1[139135]: 2023-09-10T00:56:13.906+0000 7fe697b9b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:56:14.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-09-10T00:56:14.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: osdmap e11: 2 total, 1 up, 2 in 2023-09-10T00:56:14.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:14.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:56:14.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:56:14.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:56:14.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:14.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:14.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-09-10T00:56:14.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: osdmap e11: 2 total, 1 up, 2 in 2023-09-10T00:56:14.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:14.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:56:14.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:56:14.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:56:14.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:14.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:14.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:14.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:15.153 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 -- 172.21.15.136:0/518596097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb500ff590 msgr2=0x7ffb500ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:15.153 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 --2- 172.21.15.136:0/518596097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb500ff590 0x7ffb500ff9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffb44009990 tx=0x7ffb44009ca0 comp rx=0 tx=0).stop 2023-09-10T00:56:15.154 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 -- 172.21.15.136:0/518596097 shutdown_connections 2023-09-10T00:56:15.154 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 --2- 172.21.15.136:0/518596097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb500ff590 0x7ffb500ff9b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:15.154 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 --2- 172.21.15.136:0/518596097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb50101130 0x7ffb501079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:15.154 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 --2- 172.21.15.136:0/518596097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb50100790 0x7ffb50100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:15.154 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 -- 172.21.15.136:0/518596097 >> 172.21.15.136:0/518596097 conn(0x7ffb500fab10 msgr2=0x7ffb500fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:56:15.155 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 -- 172.21.15.136:0/518596097 shutdown_connections 2023-09-10T00:56:15.155 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.151+0000 7ffb54e99700 1 -- 172.21.15.136:0/518596097 wait complete. 2023-09-10T00:56:15.158 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.152+0000 7ffb54e99700 1 Processor -- start 2023-09-10T00:56:15.158 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.152+0000 7ffb54e99700 1 -- start start 2023-09-10T00:56:15.159 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb500ff590 0x7ffb501952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:15.159 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb50100790 0x7ffb50195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:15.159 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb50101130 0x7ffb5019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:15.159 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5019ce70 con 0x7ffb500ff590 2023-09-10T00:56:15.159 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5019cfb0 con 0x7ffb50101130 2023-09-10T00:56:15.160 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5019d140 con 0x7ffb50100790 2023-09-10T00:56:15.160 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb50100790 0x7ffb50195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:15.160 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb50100790 0x7ffb50195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39168/0 (socket says 172.21.15.136:39168) 2023-09-10T00:56:15.160 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 -- 172.21.15.136:0/3020904940 learned_addr learned my addr 172.21.15.136:0/3020904940 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-09-10T00:56:15.160 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 -- 172.21.15.136:0/3020904940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb50101130 msgr2=0x7ffb5019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:15.161 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4ed9d700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb50101130 0x7ffb5019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:15.161 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4e59c700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb500ff590 0x7ffb501952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:15.161 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb50101130 0x7ffb5019c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:15.161 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 -- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb500ff590 msgr2=0x7ffb501952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:15.161 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb500ff590 0x7ffb501952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:15.162 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 -- 172.21.15.136:0/3020904940 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb440095f0 con 0x7ffb50100790 2023-09-10T00:56:15.162 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4e59c700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb500ff590 0x7ffb501952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:56:15.162 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb4dd9b700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb50100790 0x7ffb50195820 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb400099a0 tx=0x7ffb40009d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:56:15.162 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb3f7fe700 1 -- 172.21.15.136:0/3020904940 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40012070 con 0x7ffb50100790 2023-09-10T00:56:15.162 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb3f7fe700 1 -- 172.21.15.136:0/3020904940 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb4000d260 con 0x7ffb50100790 2023-09-10T00:56:15.162 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb50068120 con 0x7ffb50100790 2023-09-10T00:56:15.163 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb3f7fe700 1 -- 172.21.15.136:0/3020904940 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4000caa0 con 0x7ffb50100790 2023-09-10T00:56:15.163 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.153+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb500686f0 con 0x7ffb50100790 2023-09-10T00:56:15.163 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.154+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5010ae20 con 0x7ffb50100790 2023-09-10T00:56:15.163 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.156+0000 7ffb3f7fe700 1 -- 172.21.15.136:0/3020904940 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) v1 ==== 135202+0+0 (secure 0 0 0) 0x7ffb40007ce0 con 0x7ffb50100790 2023-09-10T00:56:15.163 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.157+0000 7ffb3f7fe700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ffb380a0630 0x7ffb380a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:15.164 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.157+0000 7ffb3f7fe700 1 -- 172.21.15.136:0/3020904940 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7ffb400cb370 con 0x7ffb50100790 2023-09-10T00:56:15.164 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.157+0000 7ffb4e59c700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ffb380a0630 0x7ffb380a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:15.164 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.158+0000 7ffb4e59c700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ffb380a0630 0x7ffb380a2af0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400b5c0 tx=0x7ffb44005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:56:15.165 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.160+0000 7ffb3f7fe700 1 -- 172.21.15.136:0/3020904940 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ffb400966f0 con 0x7ffb50100790 2023-09-10T00:56:15.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:14 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi092 2023-09-10T00:56:15.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:14 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi092 to 17168M 2023-09-10T00:56:15.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:14 smithi136 ceph-mon[133708]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:15.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:14 smithi136 ceph-mon[133708]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:56:15.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:14 smithi136 ceph-mon[133708]: osdmap e12: 2 total, 1 up, 2 in 2023-09-10T00:56:15.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:15.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:15.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:14 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi092 2023-09-10T00:56:15.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:14 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi092 to 17168M 2023-09-10T00:56:15.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:14 smithi092 ceph-mon[134067]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:15.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:14 smithi092 ceph-mon[134067]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:56:15.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:14 smithi092 ceph-mon[134067]: osdmap e12: 2 total, 1 up, 2 in 2023-09-10T00:56:15.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:15.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:14 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi092 2023-09-10T00:56:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:14 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi092 to 17168M 2023-09-10T00:56:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:14 smithi003 ceph-mon[97654]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:14 smithi003 ceph-mon[97654]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:56:15.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:14 smithi003 ceph-mon[97654]: osdmap e12: 2 total, 1 up, 2 in 2023-09-10T00:56:15.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:15.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:15.354+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb50068ea0 con 0x7ffb380a0630 2023-09-10T00:56:16.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:56:16.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:56:16.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] boot 2023-09-10T00:56:16.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: osdmap e13: 2 total, 2 up, 2 in 2023-09-10T00:56:16.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:16.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:56:16.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:56:16.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:15 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:16.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:56:16.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:56:16.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] boot 2023-09-10T00:56:16.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: osdmap e13: 2 total, 2 up, 2 in 2023-09-10T00:56:16.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:16.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:56:16.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:56:16.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:15 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:16.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:56:16.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:56:16.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: osd.1 [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] boot 2023-09-10T00:56:16.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: osdmap e13: 2 total, 2 up, 2 in 2023-09-10T00:56:16.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T00:56:16.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:56:16.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:56:16.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:15 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:17.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:16 smithi092 ceph-mon[134067]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:56:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:16 smithi092 ceph-mon[134067]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:16 smithi092 ceph-mon[134067]: osdmap e14: 2 total, 2 up, 2 in 2023-09-10T00:56:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:16 smithi003 ceph-mon[97654]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:56:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:16 smithi003 ceph-mon[97654]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:16 smithi003 ceph-mon[97654]: osdmap e14: 2 total, 2 up, 2 in 2023-09-10T00:56:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:16 smithi136 ceph-mon[133708]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:56:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:16 smithi136 ceph-mon[133708]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-09-10T00:56:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:16 smithi136 ceph-mon[133708]: osdmap e14: 2 total, 2 up, 2 in 2023-09-10T00:56:18.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:17 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/1225504188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]: dispatch 2023-09-10T00:56:18.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:17 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]: dispatch 2023-09-10T00:56:18.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:17 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]': finished 2023-09-10T00:56:18.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:17 smithi092 ceph-mon[134067]: osdmap e15: 3 total, 2 up, 3 in 2023-09-10T00:56:18.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:18.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:17 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/1225504188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]: dispatch 2023-09-10T00:56:18.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:17 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]: dispatch 2023-09-10T00:56:18.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:17 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]': finished 2023-09-10T00:56:18.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:17 smithi003 ceph-mon[97654]: osdmap e15: 3 total, 2 up, 3 in 2023-09-10T00:56:18.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:17 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/1225504188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]: dispatch 2023-09-10T00:56:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:17 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]: dispatch 2023-09-10T00:56:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:17 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c7848b8-ced5-42a8-b5e1-59a572c3fca3"}]': finished 2023-09-10T00:56:18.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:17 smithi136 ceph-mon[133708]: osdmap e15: 3 total, 2 up, 3 in 2023-09-10T00:56:18.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:19.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:18 smithi092 ceph-mon[134067]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:19.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:18 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/1822636908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:56:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:18 smithi003 ceph-mon[97654]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:18 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/1822636908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:56:19.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:18 smithi136 ceph-mon[133708]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:19.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:18 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/1822636908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:56:21.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:20 smithi092 ceph-mon[134067]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:20 smithi003 ceph-mon[97654]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:21.342 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:20 smithi136 ceph-mon[133708]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:23.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:22 smithi136 ceph-mon[133708]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:23.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:22 smithi092 ceph-mon[134067]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:23.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:22 smithi003 ceph-mon[97654]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:25.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:24 smithi092 ceph-mon[134067]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-09-10T00:56:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:25.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:24 smithi003 ceph-mon[97654]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-09-10T00:56:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:25.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:24 smithi136 ceph-mon[133708]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:25.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-09-10T00:56:25.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:26.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:25 smithi136 ceph-mon[133708]: Deploying daemon osd.2 on smithi136 2023-09-10T00:56:26.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:25 smithi092 ceph-mon[134067]: Deploying daemon osd.2 on smithi136 2023-09-10T00:56:26.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:25 smithi003 ceph-mon[97654]: Deploying daemon osd.2 on smithi136 2023-09-10T00:56:27.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:26 smithi092 ceph-mon[134067]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:27.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:26 smithi003 ceph-mon[97654]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:26 smithi136 ceph-mon[133708]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:29.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:28 smithi136 ceph-mon[133708]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:29.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:29.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:29.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:29.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:28 smithi092 ceph-mon[134067]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:29.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:28 smithi003 ceph-mon[97654]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:29.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:29.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:29.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:30.976 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.975+0000 7ffb3f7fe700 1 -- 172.21.15.136:0/3020904940 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffb50068ea0 con 0x7ffb380a0630 2023-09-10T00:56:30.976 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 2 on host 'smithi136' 2023-09-10T00:56:30.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ffb380a0630 msgr2=0x7ffb380a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:30.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ffb380a0630 0x7ffb380a2af0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400b5c0 tx=0x7ffb44005fb0 comp rx=0 tx=0).stop 2023-09-10T00:56:30.978 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb50100790 msgr2=0x7ffb50195820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:30.979 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb50100790 0x7ffb50195820 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffb400099a0 tx=0x7ffb40009d60 comp rx=0 tx=0).stop 2023-09-10T00:56:30.979 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 shutdown_connections 2023-09-10T00:56:30.979 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb50101130 0x7ffb5019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:30.979 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb500ff590 0x7ffb501952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:30.979 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ffb380a0630 0x7ffb380a2af0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:30.980 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 --2- 172.21.15.136:0/3020904940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb50100790 0x7ffb50195820 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:30.980 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 >> 172.21.15.136:0/3020904940 conn(0x7ffb500fab10 msgr2=0x7ffb500fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:56:30.980 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 shutdown_connections 2023-09-10T00:56:30.980 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:56:30.978+0000 7ffb54e99700 1 -- 172.21.15.136:0/3020904940 wait complete. 2023-09-10T00:56:31.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:30 smithi136 ceph-mon[133708]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:31.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:30 smithi092 ceph-mon[134067]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:30 smithi003 ceph-mon[97654]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:31.431 DEBUG:teuthology.orchestra.run.smithi136:osd.2> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.2.service 2023-09-10T00:56:31.434 INFO:tasks.cephadm:Deploying osd.3 on smithi003 with /dev/vg_nvme/lv_3... 2023-09-10T00:56:31.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-09-10T00:56:31.466 INFO:journalctl@ceph.osd.2.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T00:56:32.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:31 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:32.177 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:56:32.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:31 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:32.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:31 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:32.458 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:32 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-2[139502]: 2023-09-10T00:56:32.315+0000 7ff5435b0640 -1 osd.2 0 log_to_monitors true 2023-09-10T00:56:33.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:32 smithi003 ceph-mon[97654]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:33.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:32 smithi003 ceph-mon[97654]: from='osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-09-10T00:56:33.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:32 smithi003 ceph-mon[97654]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-09-10T00:56:33.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:32 smithi136 ceph-mon[133708]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:33.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:32 smithi136 ceph-mon[133708]: from='osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-09-10T00:56:33.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:32 smithi136 ceph-mon[133708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-09-10T00:56:33.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:32 smithi092 ceph-mon[134067]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:33.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:32 smithi092 ceph-mon[134067]: from='osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-09-10T00:56:33.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:32 smithi092 ceph-mon[134067]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-09-10T00:56:34.061 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-09-10T00:56:34.061 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-09-10T00:56:34.061 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-09-10T00:56:34.062 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-09-10T00:56:34.062 INFO:teuthology.orchestra.run.smithi003.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0313303 s, 335 MB/s 2023-09-10T00:56:34.062 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-09-10T00:56:34.214 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:33 smithi003 ceph-mon[97654]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-09-10T00:56:34.214 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:33 smithi003 ceph-mon[97654]: from='osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:56:34.214 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:33 smithi003 ceph-mon[97654]: osdmap e16: 3 total, 2 up, 3 in 2023-09-10T00:56:34.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:33 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:34.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:33 smithi003 ceph-mon[97654]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:56:34.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:33 smithi092 ceph-mon[134067]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-09-10T00:56:34.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:33 smithi092 ceph-mon[134067]: from='osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:56:34.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:33 smithi092 ceph-mon[134067]: osdmap e16: 3 total, 2 up, 3 in 2023-09-10T00:56:34.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:33 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:34.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:33 smithi092 ceph-mon[134067]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:56:34.412 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:33 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-2[139502]: 2023-09-10T00:56:33.960+0000 7ff538c27700 -1 osd.2 0 waiting for initial osdmap 2023-09-10T00:56:34.412 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:33 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-2[139502]: 2023-09-10T00:56:33.964+0000 7ff530a14700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:56:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:33 smithi136 ceph-mon[133708]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-09-10T00:56:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:33 smithi136 ceph-mon[133708]: from='osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:56:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:33 smithi136 ceph-mon[133708]: osdmap e16: 3 total, 2 up, 3 in 2023-09-10T00:56:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:33 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:34.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:33 smithi136 ceph-mon[133708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:56:34.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi003:vg_nvme/lv_3 2023-09-10T00:56:35.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:56:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: osdmap e17: 3 total, 2 up, 3 in 2023-09-10T00:56:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:35.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi136 2023-09-10T00:56:35.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:56:35.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:35.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:35.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:35.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:35.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:35.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.299 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:56:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:56:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: osdmap e17: 3 total, 2 up, 3 in 2023-09-10T00:56:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:35.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:35.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi136 2023-09-10T00:56:35.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:56:35.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:35.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:35.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:35.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:35.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:35.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:56:35.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: osdmap e17: 3 total, 2 up, 3 in 2023-09-10T00:56:35.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:35.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:35.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi136 2023-09-10T00:56:35.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:56:35.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:35.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:35.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:35.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:35.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:35.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:35.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:36.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 -- 172.21.15.3:0/236372874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 msgr2=0x7f357410ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:36.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 --2- 172.21.15.3:0/236372874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357410ad30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f356c00b0d0 tx=0x7f356c00b3e0 comp rx=0 tx=0).stop 2023-09-10T00:56:36.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 -- 172.21.15.3:0/236372874 shutdown_connections 2023-09-10T00:56:36.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 --2- 172.21.15.3:0/236372874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357410ad30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:36.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 --2- 172.21.15.3:0/236372874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35741028e0 0x7f3574102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:36.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 --2- 172.21.15.3:0/236372874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3574103ae0 0x7f3574103f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:36.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 -- 172.21.15.3:0/236372874 >> 172.21.15.3:0/236372874 conn(0x7f35740fde60 msgr2=0x7f35741002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:56:36.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 -- 172.21.15.3:0/236372874 shutdown_connections 2023-09-10T00:56:36.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.581+0000 7f357a651700 1 -- 172.21.15.3:0/236372874 wait complete. 2023-09-10T00:56:36.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.582+0000 7f357a651700 1 Processor -- start 2023-09-10T00:56:36.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.582+0000 7f357a651700 1 -- start start 2023-09-10T00:56:36.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f357a651700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35741028e0 0x7f3574198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:36.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f357a651700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3574103ae0 0x7f3574198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:36.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f357a651700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:36.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f357a651700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35741a0220 con 0x7f3574104480 2023-09-10T00:56:36.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f357a651700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35741a0360 con 0x7f3574103ae0 2023-09-10T00:56:36.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f357a651700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35741a04f0 con 0x7f35741028e0 2023-09-10T00:56:36.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f3578bee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:36.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f3578bee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357419fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35188/0 (socket says 172.21.15.3:35188) 2023-09-10T00:56:36.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f3578bee700 1 -- 172.21.15.3:0/2230079053 learned_addr learned my addr 172.21.15.3:0/2230079053 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:56:36.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f3573fff700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35741028e0 0x7f3574198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:36.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.583+0000 7f35737fe700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3574103ae0 0x7f3574198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:36.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f3578bee700 1 -- 172.21.15.3:0/2230079053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35741028e0 msgr2=0x7f3574198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:36.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f3578bee700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35741028e0 0x7f3574198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:36.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f3578bee700 1 -- 172.21.15.3:0/2230079053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3574103ae0 msgr2=0x7f3574198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:36.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f3578bee700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3574103ae0 0x7f3574198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:36.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f3578bee700 1 -- 172.21.15.3:0/2230079053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c009d00 con 0x7f3574104480 2023-09-10T00:56:36.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f3578bee700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357419fc50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f356c008eb0 tx=0x7f356c008ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:56:36.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c00e040 con 0x7f3574104480 2023-09-10T00:56:36.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35741a0770 con 0x7f3574104480 2023-09-10T00:56:36.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f356c00bdb0 con 0x7f3574104480 2023-09-10T00:56:36.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c01d470 con 0x7f3574104480 2023-09-10T00:56:36.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.584+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35741a0b90 con 0x7f3574104480 2023-09-10T00:56:36.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.585+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3574109e10 con 0x7f3574104480 2023-09-10T00:56:36.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.587+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 15) v1 ==== 135202+0+0 (secure 0 0 0) 0x7f356c005020 con 0x7f3574104480 2023-09-10T00:56:36.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.589+0000 7f35717fa700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f355c0a0630 0x7f355c0a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:56:36.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.590+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f356c0cba50 con 0x7f3574104480 2023-09-10T00:56:36.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.590+0000 7f3573fff700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f355c0a0630 0x7f355c0a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:56:36.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.590+0000 7f3573fff700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f355c0a0630 0x7f355c0a2af0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3574103940 tx=0x7f3564009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:56:36.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.594+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f356c096a30 con 0x7f3574104480 2023-09-10T00:56:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:36 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:56:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:36 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:56:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:36 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi136 to 17168M 2023-09-10T00:56:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:36 smithi136 ceph-mon[133708]: osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] boot 2023-09-10T00:56:36.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:36 smithi136 ceph-mon[133708]: osdmap e18: 3 total, 3 up, 3 in 2023-09-10T00:56:36.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:36.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:36 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:56:36.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:36 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:56:36.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:36 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi136 to 17168M 2023-09-10T00:56:36.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:36 smithi092 ceph-mon[134067]: osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] boot 2023-09-10T00:56:36.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:36 smithi092 ceph-mon[134067]: osdmap e18: 3 total, 3 up, 3 in 2023-09-10T00:56:36.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:36.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:36 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:56:36.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:36 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:56:36.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:36 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi136 to 17168M 2023-09-10T00:56:36.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:36 smithi003 ceph-mon[97654]: osd.2 [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] boot 2023-09-10T00:56:36.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:36 smithi003 ceph-mon[97654]: osdmap e18: 3 total, 3 up, 3 in 2023-09-10T00:56:36.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T00:56:36.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:36.814+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f357402cf80 con 0x7f355c0a0630 2023-09-10T00:56:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-09-10T00:56:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-09-10T00:56:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: osdmap e19: 3 total, 3 up, 3 in 2023-09-10T00:56:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T00:56:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:56:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:56:37.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:56:37.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:37.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-09-10T00:56:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-09-10T00:56:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: osdmap e19: 3 total, 3 up, 3 in 2023-09-10T00:56:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T00:56:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:56:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:56:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:56:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:37.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-09-10T00:56:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-09-10T00:56:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-09-10T00:56:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: osdmap e19: 3 total, 3 up, 3 in 2023-09-10T00:56:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T00:56:37.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:56:37.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:56:37.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:56:37.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:38.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:38.403+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f356c096e00 con 0x7f3574104480 2023-09-10T00:56:38.444 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125441]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-09-10T00:56:38.445 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125441]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:38.445 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125441]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:38.445 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125441]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:38.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-09-10T00:56:38.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:38 smithi003 ceph-mon[97654]: osdmap e20: 3 total, 3 up, 3 in 2023-09-10T00:56:38.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:38 smithi003 ceph-mon[97654]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-09-10T00:56:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:38 smithi136 ceph-mon[133708]: osdmap e20: 3 total, 3 up, 3 in 2023-09-10T00:56:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:38 smithi136 ceph-mon[133708]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:38.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-09-10T00:56:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:38 smithi092 ceph-mon[134067]: osdmap e20: 3 total, 3 up, 3 in 2023-09-10T00:56:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:38 smithi092 ceph-mon[134067]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:39.059 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125489]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-09-10T00:56:39.059 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125489]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:39.059 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125489]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:39.059 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 00:56:38 smithi003 sudo[125489]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:39.387 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144085]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-09-10T00:56:39.387 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144085]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:39.388 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144085]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:39.388 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144085]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:39.655 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:39 smithi136 ceph-mon[133708]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-09-10T00:56:39.655 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:39 smithi136 ceph-mon[133708]: osdmap e21: 3 total, 3 up, 3 in 2023-09-10T00:56:39.655 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:39 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:39.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:39 smithi092 ceph-mon[134067]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-09-10T00:56:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:39 smithi092 ceph-mon[134067]: osdmap e21: 3 total, 3 up, 3 in 2023-09-10T00:56:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:39 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:39.716 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144088]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-09-10T00:56:39.717 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144088]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:39.717 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144088]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:39.717 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 00:56:39 smithi092 sudo[144088]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:39 smithi003 ceph-mon[97654]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-09-10T00:56:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:39 smithi003 ceph-mon[97654]: osdmap e21: 3 total, 3 up, 3 in 2023-09-10T00:56:39.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:39 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:39.908 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145026]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-09-10T00:56:39.908 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:39.909 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145026]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:39.909 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145026]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:40.162 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145029]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-09-10T00:56:40.162 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:40.162 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145029]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:40.162 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 00:56:39 smithi136 sudo[145029]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:40.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:40 smithi003 sudo[125569]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-09-10T00:56:40.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:40 smithi003 sudo[125569]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:40.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:40 smithi003 sudo[125569]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:40.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:40 smithi003 sudo[125569]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:41.278 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:41.279 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1724324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]: dispatch 2023-09-10T00:56:41.279 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]: dispatch 2023-09-10T00:56:41.279 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]': finished 2023-09-10T00:56:41.279 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: osdmap e22: 4 total, 3 up, 4 in 2023-09-10T00:56:41.279 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:56:41.280 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:41.280 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 ceph-mon[133708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:41.280 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145032]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-09-10T00:56:41.280 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:41.280 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145032]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:41.281 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145032]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 sudo[125622]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-09-10T00:56:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 sudo[125622]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 sudo[125622]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 sudo[125622]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1724324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]: dispatch 2023-09-10T00:56:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]: dispatch 2023-09-10T00:56:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]': finished 2023-09-10T00:56:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: osdmap e22: 4 total, 3 up, 4 in 2023-09-10T00:56:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:56:41.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:41.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:41 smithi003 ceph-mon[97654]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:41.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:41.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1724324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]: dispatch 2023-09-10T00:56:41.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]: dispatch 2023-09-10T00:56:41.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08fe6de-b501-43d2-bcf3-2ce57eb464f9"}]': finished 2023-09-10T00:56:41.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: osdmap e22: 4 total, 3 up, 4 in 2023-09-10T00:56:41.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:56:41.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:41.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 ceph-mon[134067]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145035]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-09-10T00:56:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145035]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:41 smithi136 sudo[145035]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:42.007 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 sudo[144091]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-09-10T00:56:42.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 sudo[144091]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:42.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 sudo[144091]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:42.008 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:41 smithi092 sudo[144091]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/525391192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:56:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:42 smithi003 ceph-mon[97654]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/525391192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:56:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:42 smithi136 ceph-mon[133708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 sudo[144094]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-09-10T00:56:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 sudo[144094]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-09-10T00:56:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 sudo[144094]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-09-10T00:56:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 sudo[144094]: pam_unix(sudo:session): session closed for user root 2023-09-10T00:56:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:42.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/525391192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:56:42.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:42.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:42.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-09-10T00:56:42.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T00:56:42.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T00:56:42.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T00:56:42.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:42.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:42 smithi092 ceph-mon[134067]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-09-10T00:56:45.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:44 smithi136 ceph-mon[133708]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:45.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:44 smithi003 ceph-mon[97654]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:45.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:44 smithi092 ceph-mon[134067]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:47.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:46 smithi003 ceph-mon[97654]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:47.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:46 smithi136 ceph-mon[133708]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:47.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:46 smithi092 ceph-mon[134067]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:56:48.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-09-10T00:56:48.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:48.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-09-10T00:56:48.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:48.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-09-10T00:56:48.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:49.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:48 smithi136 ceph-mon[133708]: Deploying daemon osd.3 on smithi003 2023-09-10T00:56:49.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:48 smithi136 ceph-mon[133708]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-09-10T00:56:49.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:48 smithi092 ceph-mon[134067]: Deploying daemon osd.3 on smithi003 2023-09-10T00:56:49.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:48 smithi092 ceph-mon[134067]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-09-10T00:56:49.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:48 smithi003 ceph-mon[97654]: Deploying daemon osd.3 on smithi003 2023-09-10T00:56:49.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:48 smithi003 ceph-mon[97654]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-09-10T00:56:51.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:50 smithi136 ceph-mon[133708]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-09-10T00:56:51.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:50 smithi092 ceph-mon[134067]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-09-10T00:56:51.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:50 smithi003 ceph-mon[97654]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-09-10T00:56:53.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:52 smithi003 ceph-mon[97654]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-09-10T00:56:53.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:52 smithi136 ceph-mon[133708]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-09-10T00:56:53.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:52 smithi092 ceph-mon[134067]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-09-10T00:56:54.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:54.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:54.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:54.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:54 smithi003 ceph-mon[97654]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:54.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:54 smithi136 ceph-mon[133708]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:54.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:54.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:54.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:54 smithi092 ceph-mon[134067]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:56.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.419+0000 7f35717fa700 1 -- 172.21.15.3:0/2230079053 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f357402cf80 con 0x7f355c0a0630 2023-09-10T00:56:56.421 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 3 on host 'smithi003' 2023-09-10T00:56:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f355c0a0630 msgr2=0x7f355c0a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f355c0a0630 0x7f355c0a2af0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3574103940 tx=0x7f3564009260 comp rx=0 tx=0).stop 2023-09-10T00:56:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 msgr2=0x7f357419fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:56:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357419fc50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f356c008eb0 tx=0x7f356c008ee0 comp rx=0 tx=0).stop 2023-09-10T00:56:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 shutdown_connections 2023-09-10T00:56:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f355c0a0630 0x7f355c0a2af0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35741028e0 0x7f3574198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3574104480 0x7f357419fc50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 --2- 172.21.15.3:0/2230079053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3574103ae0 0x7f3574198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:56:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 >> 172.21.15.3:0/2230079053 conn(0x7f35740fde60 msgr2=0x7f3574106f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:56:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.422+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 shutdown_connections 2023-09-10T00:56:56.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:56:56.423+0000 7f357a651700 1 -- 172.21.15.3:0/2230079053 wait complete. 2023-09-10T00:56:56.921 DEBUG:teuthology.orchestra.run.smithi003:osd.3> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.3.service 2023-09-10T00:56:56.926 INFO:tasks.cephadm:Deploying osd.4 on smithi092 with /dev/vg_nvme/lv_3... 2023-09-10T00:56:56.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-09-10T00:56:57.008 INFO:journalctl@ceph.osd.3.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T00:56:57.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:57.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:57.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:57.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:57.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:57.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:57.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:57.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:57.308 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 00:56:57 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T00:56:57.096+0000 7fc8b1ee1640 -1 osd.3 0 log_to_monitors true 2023-09-10T00:56:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:56:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:56:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:56:57.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:56:57.497 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:56:58.022 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:57 smithi092 ceph-mon[134067]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-09-10T00:56:58.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:57 smithi136 ceph-mon[133708]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-09-10T00:56:58.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:57 smithi003 ceph-mon[97654]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-09-10T00:56:58.947 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-09-10T00:56:58.948 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-09-10T00:56:58.948 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-09-10T00:56:58.948 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-09-10T00:56:58.948 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148718 s, 705 MB/s 2023-09-10T00:56:58.948 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-09-10T00:56:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:58 smithi136 ceph-mon[133708]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:58 smithi136 ceph-mon[133708]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-09-10T00:56:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:58 smithi136 ceph-mon[133708]: osdmap e23: 4 total, 3 up, 4 in 2023-09-10T00:56:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:58 smithi136 ceph-mon[133708]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:56:59.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:56:59.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:58 smithi092 ceph-mon[134067]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:59.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:58 smithi092 ceph-mon[134067]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-09-10T00:56:59.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:58 smithi092 ceph-mon[134067]: osdmap e23: 4 total, 3 up, 4 in 2023-09-10T00:56:59.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:58 smithi092 ceph-mon[134067]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:56:59.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:56:59.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:58 smithi003 ceph-mon[97654]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-09-10T00:56:59.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:58 smithi003 ceph-mon[97654]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-09-10T00:56:59.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:58 smithi003 ceph-mon[97654]: osdmap e23: 4 total, 3 up, 4 in 2023-09-10T00:56:59.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:58 smithi003 ceph-mon[97654]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:56:59.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:56:59.310 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 00:56:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T00:56:58.891+0000 7fc8a7558700 -1 osd.3 0 waiting for initial osdmap 2023-09-10T00:56:59.311 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 00:56:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T00:56:58.900+0000 7fc8a334d700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:56:59.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi092:vg_nvme/lv_3 2023-09-10T00:56:59.890 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:57:00.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:59 smithi136 ceph-mon[133708]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:57:00.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:59 smithi136 ceph-mon[133708]: osdmap e24: 4 total, 3 up, 4 in 2023-09-10T00:57:00.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:59 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:59 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:59 smithi136 ceph-mon[133708]: osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] boot 2023-09-10T00:57:00.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:59 smithi136 ceph-mon[133708]: osdmap e25: 4 total, 4 up, 4 in 2023-09-10T00:57:00.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:56:59 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:59 smithi092 ceph-mon[134067]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:57:00.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:59 smithi092 ceph-mon[134067]: osdmap e24: 4 total, 3 up, 4 in 2023-09-10T00:57:00.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:59 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:59 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:59 smithi092 ceph-mon[134067]: osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] boot 2023-09-10T00:57:00.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:59 smithi092 ceph-mon[134067]: osdmap e25: 4 total, 4 up, 4 in 2023-09-10T00:57:00.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:56:59 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:59 smithi003 ceph-mon[97654]: from='osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:57:00.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:59 smithi003 ceph-mon[97654]: osdmap e24: 4 total, 3 up, 4 in 2023-09-10T00:57:00.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:59 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:59 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:00.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:59 smithi003 ceph-mon[97654]: osd.3 [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] boot 2023-09-10T00:57:00.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:59 smithi003 ceph-mon[97654]: osdmap e25: 4 total, 4 up, 4 in 2023-09-10T00:57:00.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:56:59 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T00:57:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:57:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:57:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:57:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:01.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:01.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:01.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:01.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:00 smithi136 ceph-mon[133708]: osdmap e26: 4 total, 4 up, 4 in 2023-09-10T00:57:01.202 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/1623098305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf481007d0 msgr2=0x7fbf48100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/1623098305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf481007d0 0x7fbf48100c50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009990 tx=0x7fbf3c009ca0 comp rx=0 tx=0).stop 2023-09-10T00:57:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/1623098305 shutdown_connections 2023-09-10T00:57:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/1623098305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf48101190 0x7fbf48107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/1623098305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf480ffab0 0x7fbf480ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:01.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/1623098305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf481007d0 0x7fbf48100c50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:01.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/1623098305 >> 172.21.15.92:0/1623098305 conn(0x7fbf480fa2b0 msgr2=0x7fbf480fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:57:01.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/1623098305 shutdown_connections 2023-09-10T00:57:01.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.201+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/1623098305 wait complete. 2023-09-10T00:57:01.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 Processor -- start 2023-09-10T00:57:01.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 -- start start 2023-09-10T00:57:01.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf480ffab0 0x7fbf480ff190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:01.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf481007d0 0x7fbf4819b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:01.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf48101190 0x7fbf4819d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:01.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819de50 con 0x7fbf481007d0 2023-09-10T00:57:01.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819dfc0 con 0x7fbf480ffab0 2023-09-10T00:57:01.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf4e7d2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819e180 con 0x7fbf48101190 2023-09-10T00:57:01.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf47fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf480ffab0 0x7fbf480ff190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:01.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf47fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf480ffab0 0x7fbf480ff190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:51134/0 (socket says 172.21.15.92:51134) 2023-09-10T00:57:01.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.202+0000 7fbf47fff700 1 -- 172.21.15.92:0/413447877 learned_addr learned my addr 172.21.15.92:0/413447877 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-09-10T00:57:01.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf477fe700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf481007d0 0x7fbf4819b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:01.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf47fff700 1 -- 172.21.15.92:0/413447877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf48101190 msgr2=0x7fbf4819d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:01.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf47fff700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf48101190 0x7fbf4819d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:01.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf47fff700 1 -- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf481007d0 msgr2=0x7fbf4819b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:01.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf47fff700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf481007d0 0x7fbf4819b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:01.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf47fff700 1 -- 172.21.15.92:0/413447877 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0095f0 con 0x7fbf480ffab0 2023-09-10T00:57:01.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf47fff700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf480ffab0 0x7fbf480ff190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800d980 tx=0x7fbf3800dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:57:01.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf457fa700 1 -- 172.21.15.92:0/413447877 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3800de60 con 0x7fbf480ffab0 2023-09-10T00:57:01.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf457fa700 1 -- 172.21.15.92:0/413447877 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf380044e0 con 0x7fbf480ffab0 2023-09-10T00:57:01.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4819e460 con 0x7fbf480ffab0 2023-09-10T00:57:01.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf457fa700 1 -- 172.21.15.92:0/413447877 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38003be0 con 0x7fbf480ffab0 2023-09-10T00:57:01.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.203+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4819e9b0 con 0x7fbf480ffab0 2023-09-10T00:57:01.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.204+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4810adc0 con 0x7fbf480ffab0 2023-09-10T00:57:01.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.205+0000 7fbf457fa700 1 -- 172.21.15.92:0/413447877 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fbf38009bb0 con 0x7fbf480ffab0 2023-09-10T00:57:01.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.206+0000 7fbf457fa700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf300a06c0 0x7fbf300a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:01.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.206+0000 7fbf457fa700 1 -- 172.21.15.92:0/413447877 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fbf380caa00 con 0x7fbf480ffab0 2023-09-10T00:57:01.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.206+0000 7fbf477fe700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf300a06c0 0x7fbf300a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:01.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.207+0000 7fbf477fe700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf300a06c0 0x7fbf300a2b80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006010 tx=0x7fbf3c00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:57:01.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.209+0000 7fbf457fa700 1 -- 172.21.15.92:0/413447877 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbf380957b0 con 0x7fbf480ffab0 2023-09-10T00:57:01.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:57:01.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:57:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:57:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:01.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:01.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:01.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:00 smithi092 ceph-mon[134067]: osdmap e26: 4 total, 4 up, 4 in 2023-09-10T00:57:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:57:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:57:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-09-10T00:57:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:01.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:01.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:01.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:01.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:01.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:00 smithi003 ceph-mon[97654]: osdmap e26: 4 total, 4 up, 4 in 2023-09-10T00:57:01.407 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:01.406+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf48060a40 con 0x7fbf300a06c0 2023-09-10T00:57:02.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:01 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi003 2023-09-10T00:57:02.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:01 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi003 to 8636M 2023-09-10T00:57:02.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:01 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:02.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:01 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:02.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:01 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:02.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:01 smithi136 ceph-mon[133708]: osdmap e27: 4 total, 4 up, 4 in 2023-09-10T00:57:02.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:01 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi003 2023-09-10T00:57:02.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:01 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi003 to 8636M 2023-09-10T00:57:02.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:01 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:02.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:01 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:02.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:01 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:02.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:01 smithi092 ceph-mon[134067]: osdmap e27: 4 total, 4 up, 4 in 2023-09-10T00:57:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:01 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi003 2023-09-10T00:57:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:01 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi003 to 8636M 2023-09-10T00:57:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:01 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:01 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:02.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:01 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:02.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:01 smithi003 ceph-mon[97654]: osdmap e27: 4 total, 4 up, 4 in 2023-09-10T00:57:03.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:02 smithi136 ceph-mon[133708]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:03.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:02 smithi136 ceph-mon[133708]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:03.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:02 smithi092 ceph-mon[134067]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:03.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:02 smithi092 ceph-mon[134067]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:02 smithi003 ceph-mon[97654]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:02 smithi003 ceph-mon[97654]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:04.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:03 smithi136 ceph-mon[133708]: osdmap e28: 4 total, 4 up, 4 in 2023-09-10T00:57:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:03 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/3426722765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]: dispatch 2023-09-10T00:57:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:03 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]: dispatch 2023-09-10T00:57:04.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:03 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]': finished 2023-09-10T00:57:04.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:03 smithi136 ceph-mon[133708]: osdmap e29: 5 total, 4 up, 5 in 2023-09-10T00:57:04.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:03 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:04.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:03 smithi092 ceph-mon[134067]: osdmap e28: 4 total, 4 up, 4 in 2023-09-10T00:57:04.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:03 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/3426722765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]: dispatch 2023-09-10T00:57:04.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:03 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]: dispatch 2023-09-10T00:57:04.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:03 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]': finished 2023-09-10T00:57:04.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:03 smithi092 ceph-mon[134067]: osdmap e29: 5 total, 4 up, 5 in 2023-09-10T00:57:04.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:03 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:03 smithi003 ceph-mon[97654]: osdmap e28: 4 total, 4 up, 4 in 2023-09-10T00:57:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/3426722765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]: dispatch 2023-09-10T00:57:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:03 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]: dispatch 2023-09-10T00:57:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:03 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b1cf565e-0d0a-49ec-b39b-6dbee04e586f"}]': finished 2023-09-10T00:57:04.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:03 smithi003 ceph-mon[97654]: osdmap e29: 5 total, 4 up, 5 in 2023-09-10T00:57:04.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:03 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:05.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:04 smithi136 ceph-mon[133708]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:05.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:04 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/2163954311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:05.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:04 smithi092 ceph-mon[134067]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:05.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:04 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/2163954311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:05.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:04 smithi003 ceph-mon[97654]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:05.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:04 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/2163954311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:06 smithi136 ceph-mon[133708]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:07.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:06 smithi092 ceph-mon[134067]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:07.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:06 smithi003 ceph-mon[97654]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:09.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:08 smithi092 ceph-mon[134067]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:57:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:08 smithi003 ceph-mon[97654]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:57:09.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:08 smithi136 ceph-mon[133708]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:57:10.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:10 smithi092 ceph-mon[134067]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-09-10T00:57:10.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-09-10T00:57:10.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:11.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:10 smithi003 ceph-mon[97654]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-09-10T00:57:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-09-10T00:57:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:11.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:10 smithi136 ceph-mon[133708]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-09-10T00:57:11.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-09-10T00:57:11.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:12.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:11 smithi092 ceph-mon[134067]: Deploying daemon osd.4 on smithi092 2023-09-10T00:57:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:11 smithi003 ceph-mon[97654]: Deploying daemon osd.4 on smithi092 2023-09-10T00:57:12.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:11 smithi136 ceph-mon[133708]: Deploying daemon osd.4 on smithi092 2023-09-10T00:57:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:12 smithi092 ceph-mon[134067]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-09-10T00:57:13.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:12 smithi003 ceph-mon[97654]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-09-10T00:57:13.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:12 smithi136 ceph-mon[133708]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-09-10T00:57:15.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:14 smithi092 ceph-mon[134067]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-09-10T00:57:15.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:15.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:15.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:14 smithi003 ceph-mon[97654]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-09-10T00:57:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:15.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:14 smithi136 ceph-mon[133708]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-09-10T00:57:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:17.021 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.020+0000 7fbf457fa700 1 -- 172.21.15.92:0/413447877 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbf48060a40 con 0x7fbf300a06c0 2023-09-10T00:57:17.021 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 4 on host 'smithi092' 2023-09-10T00:57:17.023 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf300a06c0 msgr2=0x7fbf300a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:17.023 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf300a06c0 0x7fbf300a2b80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c006010 tx=0x7fbf3c00b540 comp rx=0 tx=0).stop 2023-09-10T00:57:17.024 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf480ffab0 msgr2=0x7fbf480ff190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:17.024 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf480ffab0 0x7fbf480ff190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800d980 tx=0x7fbf3800dd40 comp rx=0 tx=0).stop 2023-09-10T00:57:17.024 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 shutdown_connections 2023-09-10T00:57:17.024 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf300a06c0 0x7fbf300a2b80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:17.024 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf48101190 0x7fbf4819d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:17.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf480ffab0 0x7fbf480ff190 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:17.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 --2- 172.21.15.92:0/413447877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf481007d0 0x7fbf4819b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:17.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.022+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 >> 172.21.15.92:0/413447877 conn(0x7fbf480fa2b0 msgr2=0x7fbf480fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:57:17.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.023+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 shutdown_connections 2023-09-10T00:57:17.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:57:17.023+0000 7fbf4e7d2700 1 -- 172.21.15.92:0/413447877 wait complete. 2023-09-10T00:57:17.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:16 smithi092 ceph-mon[134067]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:57:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:16 smithi003 ceph-mon[97654]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:57:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:17.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:17.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.363 DEBUG:teuthology.orchestra.run.smithi092:osd.4> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.4.service 2023-09-10T00:57:17.366 INFO:tasks.cephadm:Deploying osd.5 on smithi136 with /dev/vg_nvme/lv_3... 2023-09-10T00:57:17.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-09-10T00:57:17.406 INFO:journalctl@ceph.osd.4.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T00:57:17.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:16 smithi136 ceph-mon[133708]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:57:17.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:17.943 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:57:18.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:18.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:18.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:18.281 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:18.282 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:18.282 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:18.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:18.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:18.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:18.465 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 00:57:18 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4[147180]: 2023-09-10T00:57:18.284+0000 7f5ec15a0640 -1 osd.4 0 log_to_monitors true 2023-09-10T00:57:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:18 smithi136 ceph-mon[133708]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:57:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:18 smithi136 ceph-mon[133708]: from='osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-09-10T00:57:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:18 smithi136 ceph-mon[133708]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-09-10T00:57:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:18 smithi003 ceph-mon[97654]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:57:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:18 smithi003 ceph-mon[97654]: from='osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-09-10T00:57:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:18 smithi003 ceph-mon[97654]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-09-10T00:57:19.355 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-09-10T00:57:19.355 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-09-10T00:57:19.355 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-09-10T00:57:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-09-10T00:57:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00908466 s, 1.2 GB/s 2023-09-10T00:57:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-09-10T00:57:19.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:18 smithi092 ceph-mon[134067]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:57:19.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:18 smithi092 ceph-mon[134067]: from='osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-09-10T00:57:19.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:18 smithi092 ceph-mon[134067]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-09-10T00:57:19.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2023-09-10T00:57:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:57:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:20 smithi003 ceph-mon[97654]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-09-10T00:57:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:20 smithi003 ceph-mon[97654]: from='osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:57:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:20 smithi003 ceph-mon[97654]: osdmap e30: 5 total, 4 up, 5 in 2023-09-10T00:57:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:20 smithi003 ceph-mon[97654]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:57:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:20 smithi136 ceph-mon[133708]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-09-10T00:57:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:20 smithi136 ceph-mon[133708]: from='osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:57:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:20 smithi136 ceph-mon[133708]: osdmap e30: 5 total, 4 up, 5 in 2023-09-10T00:57:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:20 smithi136 ceph-mon[133708]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:57:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:20.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-mon[134067]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-09-10T00:57:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-mon[134067]: from='osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:57:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-mon[134067]: osdmap e30: 5 total, 4 up, 5 in 2023-09-10T00:57:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-mon[134067]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:57:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:20.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:20.467 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4[147180]: 2023-09-10T00:57:20.019+0000 7f5eb6c17700 -1 osd.4 0 waiting for initial osdmap 2023-09-10T00:57:20.467 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 00:57:20 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4[147180]: 2023-09-10T00:57:20.027+0000 7f5eb0207700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:57:21.263 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:21.263 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi092 2023-09-10T00:57:21.264 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:21.264 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:21.264 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:57:21.264 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi092 to 8584M 2023-09-10T00:57:21.264 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: osdmap e31: 5 total, 4 up, 5 in 2023-09-10T00:57:21.265 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:21.265 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:21.265 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:21.265 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:21.265 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:21.266 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:21 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:21.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi092 2023-09-10T00:57:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:57:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi092 to 8584M 2023-09-10T00:57:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: osdmap e31: 5 total, 4 up, 5 in 2023-09-10T00:57:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:21.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:21.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:21 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:21.449 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.446+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1654888145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 msgr2=0x7ff428107160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:21.449 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.446+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1654888145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff428107160 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff42000b0a0 tx=0x7ff42000b3b0 comp rx=0 tx=0).stop 2023-09-10T00:57:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.446+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1654888145 shutdown_connections 2023-09-10T00:57:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.446+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1654888145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff428107160 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.446+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1654888145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4280fed10 0x7ff4280ff130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.446+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1654888145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4280fff10 0x7ff428100370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.446+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1654888145 >> 172.21.15.136:0/1654888145 conn(0x7ff4280fa2b0 msgr2=0x7ff4280fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:57:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1654888145 shutdown_connections 2023-09-10T00:57:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1654888145 wait complete. 2023-09-10T00:57:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 Processor -- start 2023-09-10T00:57:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 -- start start 2023-09-10T00:57:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4280fed10 0x7ff428196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4280fff10 0x7ff4281969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff42819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42819e020 con 0x7ff4280fff10 2023-09-10T00:57:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42819e160 con 0x7ff4280fed10 2023-09-10T00:57:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.447+0000 7ff42e0cb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42819e2f0 con 0x7ff4281008b0 2023-09-10T00:57:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff42819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff42819da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39246/0 (socket says 172.21.15.136:39246) 2023-09-10T00:57:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 -- 172.21.15.136:0/1048520114 learned_addr learned my addr 172.21.15.136:0/1048520114 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-09-10T00:57:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff4277fe700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4280fed10 0x7ff428196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff426ffd700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4280fff10 0x7ff4281969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 -- 172.21.15.136:0/1048520114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4280fed10 msgr2=0x7ff428196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4280fed10 0x7ff428196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 -- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4280fff10 msgr2=0x7ff4281969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4280fff10 0x7ff4281969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 -- 172.21.15.136:0/1048520114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420009d00 con 0x7ff4281008b0 2023-09-10T00:57:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff4277fe700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4280fed10 0x7ff428196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:57:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff426ffd700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4280fff10 0x7ff4281969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:57:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff427fff700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff42819da50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff420007ae0 tx=0x7ff420004db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:57:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff424ff9700 1 -- 172.21.15.136:0/1048520114 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42000e040 con 0x7ff4281008b0 2023-09-10T00:57:21.457 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff424ff9700 1 -- 172.21.15.136:0/1048520114 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff42001db50 con 0x7ff4281008b0 2023-09-10T00:57:21.457 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42819e570 con 0x7ff4281008b0 2023-09-10T00:57:21.457 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff424ff9700 1 -- 172.21.15.136:0/1048520114 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42001bc40 con 0x7ff4281008b0 2023-09-10T00:57:21.458 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.448+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42819ea30 con 0x7ff4281008b0 2023-09-10T00:57:21.458 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.449+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42810adb0 con 0x7ff4281008b0 2023-09-10T00:57:21.459 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.450+0000 7ff424ff9700 1 -- 172.21.15.136:0/1048520114 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7ff42001dcc0 con 0x7ff4281008b0 2023-09-10T00:57:21.459 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.451+0000 7ff424ff9700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff4140a06c0 0x7ff4140a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:21.460 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.451+0000 7ff424ff9700 1 -- 172.21.15.136:0/1048520114 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7ff4200cc630 con 0x7ff4281008b0 2023-09-10T00:57:21.460 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.451+0000 7ff4277fe700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff4140a06c0 0x7ff4140a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:21.460 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.451+0000 7ff4277fe700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff4140a06c0 0x7ff4140a2b80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff418005fd0 tx=0x7ff418005f00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:57:21.460 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.455+0000 7ff424ff9700 1 -- 172.21.15.136:0/1048520114 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff420097260 con 0x7ff4281008b0 2023-09-10T00:57:21.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:21.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi092 2023-09-10T00:57:21.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:21.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:21.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:57:21.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi092 to 8584M 2023-09-10T00:57:21.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: osdmap e31: 5 total, 4 up, 5 in 2023-09-10T00:57:21.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:21.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:21.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:21.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:21.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:21.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:21 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:21.651+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4281081e0 con 0x7ff4140a06c0 2023-09-10T00:57:22.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:57:22.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:57:22.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:22.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] boot 2023-09-10T00:57:22.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: osdmap e32: 5 total, 5 up, 5 in 2023-09-10T00:57:22.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:22.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:22.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:22.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:22.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:22.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:22 smithi003 ceph-mon[97654]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:22.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:57:22.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:57:22.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:22.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] boot 2023-09-10T00:57:22.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: osdmap e32: 5 total, 5 up, 5 in 2023-09-10T00:57:22.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:22.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:22.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:22.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:22.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:22.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:22 smithi136 ceph-mon[133708]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:57:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:57:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: osd.4 [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] boot 2023-09-10T00:57:22.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: osdmap e32: 5 total, 5 up, 5 in 2023-09-10T00:57:22.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T00:57:22.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:22.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:22.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:22.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:22.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:22 smithi092 ceph-mon[134067]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-09-10T00:57:23.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:23 smithi003 ceph-mon[97654]: osdmap e33: 5 total, 5 up, 5 in 2023-09-10T00:57:23.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:23 smithi136 ceph-mon[133708]: osdmap e33: 5 total, 5 up, 5 in 2023-09-10T00:57:23.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:23 smithi092 ceph-mon[134067]: osdmap e33: 5 total, 5 up, 5 in 2023-09-10T00:57:24.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:24 smithi003 ceph-mon[97654]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:24.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:24 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/2039009865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]: dispatch 2023-09-10T00:57:24.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:24 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]: dispatch 2023-09-10T00:57:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:24 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]': finished 2023-09-10T00:57:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:24 smithi003 ceph-mon[97654]: osdmap e34: 6 total, 5 up, 6 in 2023-09-10T00:57:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:24.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:24 smithi136 ceph-mon[133708]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:24.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:24 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/2039009865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]: dispatch 2023-09-10T00:57:24.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:24 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]: dispatch 2023-09-10T00:57:24.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:24 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]': finished 2023-09-10T00:57:24.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:24 smithi136 ceph-mon[133708]: osdmap e34: 6 total, 5 up, 6 in 2023-09-10T00:57:24.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:24.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:24 smithi092 ceph-mon[134067]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:24 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/2039009865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]: dispatch 2023-09-10T00:57:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:24 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]: dispatch 2023-09-10T00:57:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:24 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb4bd187-8a27-479f-a3c3-018993c8d986"}]': finished 2023-09-10T00:57:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:24 smithi092 ceph-mon[134067]: osdmap e34: 6 total, 5 up, 6 in 2023-09-10T00:57:24.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:25.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:25 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/1528737751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:25.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:25 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/1528737751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:25.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:25 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/1528737751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:26.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:26 smithi136 ceph-mon[133708]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:26 smithi092 ceph-mon[134067]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:26.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:26 smithi003 ceph-mon[97654]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:29.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:28 smithi136 ceph-mon[133708]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:29.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:28 smithi092 ceph-mon[134067]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:29.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:28 smithi003 ceph-mon[97654]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:31.146 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:30 smithi136 ceph-mon[133708]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:31.146 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-09-10T00:57:31.146 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:31.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:30 smithi092 ceph-mon[134067]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-09-10T00:57:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:30 smithi003 ceph-mon[97654]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-09-10T00:57:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:32.108 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:31 smithi136 ceph-mon[133708]: Deploying daemon osd.5 on smithi136 2023-09-10T00:57:32.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:31 smithi092 ceph-mon[134067]: Deploying daemon osd.5 on smithi136 2023-09-10T00:57:32.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:31 smithi003 ceph-mon[97654]: Deploying daemon osd.5 on smithi136 2023-09-10T00:57:33.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:32 smithi136 ceph-mon[133708]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:33.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:32 smithi092 ceph-mon[134067]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:33.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:32 smithi003 ceph-mon[97654]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:35.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:34 smithi136 ceph-mon[133708]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:35.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:35.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:35.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:35.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:34 smithi092 ceph-mon[134067]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:35.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:34 smithi003 ceph-mon[97654]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:37.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:36 smithi136 ceph-mon[133708]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:37.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:37.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:37.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:36 smithi092 ceph-mon[134067]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:37.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:37.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:37.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:36 smithi003 ceph-mon[97654]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:37.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:37.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:37.416 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.414+0000 7ff424ff9700 1 -- 172.21.15.136:0/1048520114 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff4281081e0 con 0x7ff4140a06c0 2023-09-10T00:57:37.417 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 5 on host 'smithi136' 2023-09-10T00:57:37.418 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff4140a06c0 msgr2=0x7ff4140a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:37.418 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff4140a06c0 0x7ff4140a2b80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff418005fd0 tx=0x7ff418005f00 comp rx=0 tx=0).stop 2023-09-10T00:57:37.418 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 msgr2=0x7ff42819da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:37.418 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff42819da50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff420007ae0 tx=0x7ff420004db0 comp rx=0 tx=0).stop 2023-09-10T00:57:37.419 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 shutdown_connections 2023-09-10T00:57:37.419 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff4140a06c0 0x7ff4140a2b80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:37.419 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4281008b0 0x7ff42819da50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:37.419 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4280fed10 0x7ff428196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:37.419 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 --2- 172.21.15.136:0/1048520114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4280fff10 0x7ff4281969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:37.420 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 >> 172.21.15.136:0/1048520114 conn(0x7ff4280fa2b0 msgr2=0x7ff428107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:57:37.420 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.416+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 shutdown_connections 2023-09-10T00:57:37.420 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:57:37.417+0000 7ff42e0cb700 1 -- 172.21.15.136:0/1048520114 wait complete. 2023-09-10T00:57:37.848 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.5.service 2023-09-10T00:57:37.851 INFO:tasks.cephadm:Deploying osd.6 on smithi003 with /dev/vg_nvme/lv_2... 2023-09-10T00:57:37.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-09-10T00:57:37.888 INFO:journalctl@ceph.osd.5.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T00:57:38.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:38.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:38.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:38.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:38.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:57:38.615 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:57:38.902 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 00:57:38 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T00:57:38.579+0000 7fbd7b483640 -1 osd.5 0 log_to_monitors true 2023-09-10T00:57:39.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:38 smithi136 ceph-mon[133708]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:39.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:38 smithi136 ceph-mon[133708]: from='osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-09-10T00:57:39.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:38 smithi136 ceph-mon[133708]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-09-10T00:57:39.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:38 smithi092 ceph-mon[134067]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:39.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:38 smithi092 ceph-mon[134067]: from='osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-09-10T00:57:39.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:38 smithi092 ceph-mon[134067]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-09-10T00:57:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:38 smithi003 ceph-mon[97654]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:38 smithi003 ceph-mon[97654]: from='osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-09-10T00:57:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:38 smithi003 ceph-mon[97654]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-09-10T00:57:40.662 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T00:57:40.395+0000 7fbd70afa700 -1 osd.5 0 waiting for initial osdmap 2023-09-10T00:57:40.662 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T00:57:40.402+0000 7fbd6b8ed700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:57:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-09-10T00:57:40.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:57:40.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: osdmap e35: 6 total, 5 up, 6 in 2023-09-10T00:57:40.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:40.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:57:40.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:40.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:40.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:40.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:40.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:40 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-09-10T00:57:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-09-10T00:57:40.710 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-09-10T00:57:40.710 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-09-10T00:57:40.710 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0570102 s, 184 MB/s 2023-09-10T00:57:40.710 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-09-10T00:57:40.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-09-10T00:57:40.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:57:40.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: osdmap e35: 6 total, 5 up, 6 in 2023-09-10T00:57:40.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:40.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:57:40.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:40.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:40.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:40.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:40.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:40 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:40.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-09-10T00:57:40.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:57:40.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: osdmap e35: 6 total, 5 up, 6 in 2023-09-10T00:57:40.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:40.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:57:40.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:40.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:40.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:40.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:40.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:40 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:57:41.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi003:vg_nvme/lv_2 2023-09-10T00:57:41.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi136 2023-09-10T00:57:41.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi136 to 8584M 2023-09-10T00:57:41.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi136 2023-09-10T00:57:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi136 to 8584M 2023-09-10T00:57:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:41.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:41.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:41.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:57:41.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: osdmap e36: 6 total, 5 up, 6 in 2023-09-10T00:57:41.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:41.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:41.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:41.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi136 2023-09-10T00:57:41.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi136 to 8584M 2023-09-10T00:57:41.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:41.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:41.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:41.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:57:41.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: osdmap e36: 6 total, 5 up, 6 in 2023-09-10T00:57:41.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:41.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:41.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:41.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:41.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:57:41.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:57:41.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: osdmap e36: 6 total, 5 up, 6 in 2023-09-10T00:57:41.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:41.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:41.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:57:41.932 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:57:42.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:42 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:57:42.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:42 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:57:42.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:42.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:42 smithi003 ceph-mon[97654]: osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] boot 2023-09-10T00:57:42.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:42 smithi003 ceph-mon[97654]: osdmap e37: 6 total, 6 up, 6 in 2023-09-10T00:57:42.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:42.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:42 smithi003 ceph-mon[97654]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:42 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:57:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:42 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:57:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:42 smithi136 ceph-mon[133708]: osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] boot 2023-09-10T00:57:42.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:42 smithi136 ceph-mon[133708]: osdmap e37: 6 total, 6 up, 6 in 2023-09-10T00:57:42.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:42.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:42 smithi136 ceph-mon[133708]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:42.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:42 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:57:42.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:42 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:57:42.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:42.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:42 smithi092 ceph-mon[134067]: osd.5 [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] boot 2023-09-10T00:57:42.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:42 smithi092 ceph-mon[134067]: osdmap e37: 6 total, 6 up, 6 in 2023-09-10T00:57:42.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T00:57:42.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:42 smithi092 ceph-mon[134067]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-09-10T00:57:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 -- 172.21.15.3:0/371881495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398100690 msgr2=0x7f1398100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 --2- 172.21.15.3:0/371881495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398100690 0x7f1398100ab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13940099c0 tx=0x7f1394009cd0 comp rx=0 tx=0).stop 2023-09-10T00:57:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 -- 172.21.15.3:0/371881495 shutdown_connections 2023-09-10T00:57:43.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 --2- 172.21.15.3:0/371881495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398102230 0x7f1398108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 --2- 172.21.15.3:0/371881495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398100690 0x7f1398100ab0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 --2- 172.21.15.3:0/371881495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1398101890 0x7f1398101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 -- 172.21.15.3:0/371881495 >> 172.21.15.3:0/371881495 conn(0x7f13980fbbf0 msgr2=0x7f13980fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:57:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 -- 172.21.15.3:0/371881495 shutdown_connections 2023-09-10T00:57:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.286+0000 7f13a0391700 1 -- 172.21.15.3:0/371881495 wait complete. 2023-09-10T00:57:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 Processor -- start 2023-09-10T00:57:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 -- start start 2023-09-10T00:57:43.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1398100690 0x7f1398198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101890 0x7f1398198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398102230 0x7f139819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13981a0120 con 0x7f1398102230 2023-09-10T00:57:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13981a0260 con 0x7f1398101890 2023-09-10T00:57:43.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.287+0000 7f13a0391700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13981a03f0 con 0x7f1398100690 2023-09-10T00:57:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101890 0x7f1398198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139e12d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1398100690 0x7f1398198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101890 0x7f1398198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:58332/0 (socket says 172.21.15.3:58332) 2023-09-10T00:57:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 -- 172.21.15.3:0/2725953749 learned_addr learned my addr 172.21.15.3:0/2725953749 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:57:43.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139e92e700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398102230 0x7f139819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 -- 172.21.15.3:0/2725953749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1398100690 msgr2=0x7f1398198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1398100690 0x7f1398198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 -- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398102230 msgr2=0x7f139819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:57:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398102230 0x7f139819fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:57:43.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 -- 172.21.15.3:0/2725953749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940095f0 con 0x7f1398101890 2023-09-10T00:57:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.288+0000 7f139d92c700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101890 0x7f1398198ad0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f138800d950 tx=0x7f138800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:57:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.289+0000 7f138f7fe700 1 -- 172.21.15.3:0/2725953749 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138800f890 con 0x7f1398101890 2023-09-10T00:57:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.289+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13981a06d0 con 0x7f1398101890 2023-09-10T00:57:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.289+0000 7f138f7fe700 1 -- 172.21.15.3:0/2725953749 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1388010460 con 0x7f1398101890 2023-09-10T00:57:43.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.289+0000 7f138f7fe700 1 -- 172.21.15.3:0/2725953749 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138800e610 con 0x7f1398101890 2023-09-10T00:57:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.289+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13981a0c20 con 0x7f1398101890 2023-09-10T00:57:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.290+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1398192cd0 con 0x7f1398101890 2023-09-10T00:57:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.292+0000 7f138f7fe700 1 -- 172.21.15.3:0/2725953749 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f13880105d0 con 0x7f1398101890 2023-09-10T00:57:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.293+0000 7f138f7fe700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f13840a0640 0x7f13840a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:57:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.293+0000 7f138f7fe700 1 -- 172.21.15.3:0/2725953749 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f13880cab10 con 0x7f1398101890 2023-09-10T00:57:43.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.294+0000 7f139e12d700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f13840a0640 0x7f13840a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:57:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.294+0000 7f139e12d700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f13840a0640 0x7f13840a2b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1394009990 tx=0x7f1394005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:57:43.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.295+0000 7f138f7fe700 1 -- 172.21.15.3:0/2725953749 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1388095440 con 0x7f1398101890 2023-09-10T00:57:43.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:57:43.516+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f139802cf50 con 0x7f13840a0640 2023-09-10T00:57:43.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:43 smithi136 ceph-mon[133708]: osdmap e38: 6 total, 6 up, 6 in 2023-09-10T00:57:43.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:43 smithi092 ceph-mon[134067]: osdmap e38: 6 total, 6 up, 6 in 2023-09-10T00:57:43.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:43 smithi003 ceph-mon[97654]: osdmap e38: 6 total, 6 up, 6 in 2023-09-10T00:57:44.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:44 smithi092 ceph-mon[134067]: osdmap e39: 6 total, 6 up, 6 in 2023-09-10T00:57:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:44 smithi092 ceph-mon[134067]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:44.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:44 smithi092 ceph-mon[134067]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:44.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:44 smithi003 ceph-mon[97654]: osdmap e39: 6 total, 6 up, 6 in 2023-09-10T00:57:44.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:44 smithi003 ceph-mon[97654]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:44.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:44.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:44.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:44.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:44 smithi003 ceph-mon[97654]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:44.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:44 smithi136 ceph-mon[133708]: osdmap e39: 6 total, 6 up, 6 in 2023-09-10T00:57:44.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:44 smithi136 ceph-mon[133708]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:57:44.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:57:44.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:57:44.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:44.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:44 smithi136 ceph-mon[133708]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:45.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:45 smithi092 ceph-mon[134067]: osdmap e40: 6 total, 6 up, 6 in 2023-09-10T00:57:45.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:45 smithi003 ceph-mon[97654]: osdmap e40: 6 total, 6 up, 6 in 2023-09-10T00:57:45.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:45 smithi136 ceph-mon[133708]: osdmap e40: 6 total, 6 up, 6 in 2023-09-10T00:57:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:46 smithi092 ceph-mon[134067]: osdmap e41: 6 total, 6 up, 6 in 2023-09-10T00:57:46.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:46 smithi092 ceph-mon[134067]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:46.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:46 smithi003 ceph-mon[97654]: osdmap e41: 6 total, 6 up, 6 in 2023-09-10T00:57:46.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:46 smithi003 ceph-mon[97654]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:46.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:46 smithi136 ceph-mon[133708]: osdmap e41: 6 total, 6 up, 6 in 2023-09-10T00:57:46.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:46 smithi136 ceph-mon[133708]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:47.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:47 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3947761129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]: dispatch 2023-09-10T00:57:47.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:47 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]: dispatch 2023-09-10T00:57:47.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:47 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]': finished 2023-09-10T00:57:47.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:47 smithi092 ceph-mon[134067]: osdmap e42: 7 total, 6 up, 7 in 2023-09-10T00:57:47.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:57:47.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:47 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1227449425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:47.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:47 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3947761129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]: dispatch 2023-09-10T00:57:47.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:47 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]: dispatch 2023-09-10T00:57:47.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:47 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]': finished 2023-09-10T00:57:47.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:47 smithi003 ceph-mon[97654]: osdmap e42: 7 total, 6 up, 7 in 2023-09-10T00:57:47.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:57:47.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:47 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1227449425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:47.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:47 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3947761129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]: dispatch 2023-09-10T00:57:47.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:47 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]: dispatch 2023-09-10T00:57:47.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:47 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1"}]': finished 2023-09-10T00:57:47.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:47 smithi136 ceph-mon[133708]: osdmap e42: 7 total, 6 up, 7 in 2023-09-10T00:57:47.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:57:47.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:47 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1227449425' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:57:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:48 smithi092 ceph-mon[134067]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-09-10T00:57:48.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:48 smithi003 ceph-mon[97654]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-09-10T00:57:48.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:48 smithi136 ceph-mon[133708]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-09-10T00:57:51.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:50 smithi136 ceph-mon[133708]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:57:51.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:50 smithi092 ceph-mon[134067]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:57:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:50 smithi003 ceph-mon[97654]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:57:53.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:52 smithi136 ceph-mon[133708]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-09-10T00:57:53.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:52 smithi092 ceph-mon[134067]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-09-10T00:57:53.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:52 smithi003 ceph-mon[97654]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-09-10T00:57:55.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:54 smithi003 ceph-mon[97654]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-09-10T00:57:55.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-09-10T00:57:55.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:55.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:54 smithi003 ceph-mon[97654]: Deploying daemon osd.6 on smithi003 2023-09-10T00:57:55.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:54 smithi136 ceph-mon[133708]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-09-10T00:57:55.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-09-10T00:57:55.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:55.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:54 smithi136 ceph-mon[133708]: Deploying daemon osd.6 on smithi003 2023-09-10T00:57:55.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:54 smithi092 ceph-mon[134067]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-09-10T00:57:55.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-09-10T00:57:55.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:57:55.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:54 smithi092 ceph-mon[134067]: Deploying daemon osd.6 on smithi003 2023-09-10T00:57:57.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:56 smithi136 ceph-mon[133708]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-09-10T00:57:57.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:56 smithi092 ceph-mon[134067]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-09-10T00:57:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:56 smithi003 ceph-mon[97654]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-09-10T00:57:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:57:58 smithi136 ceph-mon[133708]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:59.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:57:58 smithi092 ceph-mon[134067]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:57:59.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:57:58 smithi003 ceph-mon[97654]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:00 smithi136 ceph-mon[133708]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:01.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:00 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:01.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:00 smithi092 ceph-mon[134067]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:00 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:01.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:00 smithi003 ceph-mon[97654]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:00 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:03.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:02 smithi136 ceph-mon[133708]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:03.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:02 smithi092 ceph-mon[134067]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:03.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:02 smithi003 ceph-mon[97654]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:03.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.879+0000 7f138f7fe700 1 -- 172.21.15.3:0/2725953749 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f139802cf50 con 0x7f13840a0640 2023-09-10T00:58:03.881 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 6 on host 'smithi003' 2023-09-10T00:58:03.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.881+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f13840a0640 msgr2=0x7f13840a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:03.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.881+0000 7f13a0391700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f13840a0640 0x7f13840a2b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1394009990 tx=0x7f1394005fb0 comp rx=0 tx=0).stop 2023-09-10T00:58:03.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.881+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101890 msgr2=0x7f1398198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:03.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.881+0000 7f13a0391700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101890 0x7f1398198ad0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f138800d950 tx=0x7f138800dc60 comp rx=0 tx=0).stop 2023-09-10T00:58:03.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 shutdown_connections 2023-09-10T00:58:03.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f13840a0640 0x7f13840a2b00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:03.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1398100690 0x7f1398198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:03.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1398102230 0x7f139819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:03.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 --2- 172.21.15.3:0/2725953749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101890 0x7f1398198ad0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f138800d950 tx=0x7f138800dc60 comp rx=0 tx=0).stop 2023-09-10T00:58:03.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 >> 172.21.15.3:0/2725953749 conn(0x7f13980fbbf0 msgr2=0x7f13981071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:58:03.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 shutdown_connections 2023-09-10T00:58:03.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:03.882+0000 7f13a0391700 1 -- 172.21.15.3:0/2725953749 wait complete. 2023-09-10T00:58:04.356 DEBUG:teuthology.orchestra.run.smithi003:osd.6> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.6.service 2023-09-10T00:58:04.361 INFO:tasks.cephadm:Deploying osd.7 on smithi092 with /dev/vg_nvme/lv_2... 2023-09-10T00:58:04.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-09-10T00:58:04.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:04.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:04.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:04.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.423 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:04.423 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:04 smithi003 ceph-mon[97654]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-09-10T00:58:04.435 INFO:journalctl@ceph.osd.6.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T00:58:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:04.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:04 smithi136 ceph-mon[133708]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-09-10T00:58:04.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:04.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:04.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:04.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:04 smithi092 ceph-mon[134067]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-09-10T00:58:04.935 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:58:05.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:05 smithi092 ceph-mon[134067]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-09-10T00:58:05.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:05 smithi092 ceph-mon[134067]: osdmap e43: 7 total, 6 up, 7 in 2023-09-10T00:58:05.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:05 smithi092 ceph-mon[134067]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:58:05.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:05 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:06.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:05 smithi003 ceph-mon[97654]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-09-10T00:58:06.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:05 smithi003 ceph-mon[97654]: osdmap e43: 7 total, 6 up, 7 in 2023-09-10T00:58:06.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:05 smithi003 ceph-mon[97654]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:58:06.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:05 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:06.060 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 00:58:05 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T00:58:05.853+0000 7fa75e78e700 -1 osd.6 0 waiting for initial osdmap 2023-09-10T00:58:06.060 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 00:58:05 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T00:58:05.870+0000 7fa759d82700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:58:06.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:05 smithi136 ceph-mon[133708]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-09-10T00:58:06.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:05 smithi136 ceph-mon[133708]: osdmap e43: 7 total, 6 up, 7 in 2023-09-10T00:58:06.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:05 smithi136 ceph-mon[133708]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:58:06.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:05 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:06.414 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-09-10T00:58:06.414 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-09-10T00:58:06.414 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-09-10T00:58:06.415 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-09-10T00:58:06.415 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014917 s, 703 MB/s 2023-09-10T00:58:06.415 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-09-10T00:58:06.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi092:vg_nvme/lv_2 2023-09-10T00:58:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:06 smithi136 ceph-mon[133708]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:06 smithi136 ceph-mon[133708]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:58:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:06 smithi136 ceph-mon[133708]: osdmap e44: 7 total, 6 up, 7 in 2023-09-10T00:58:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:06 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:07.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:06 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:07.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:06 smithi092 ceph-mon[134067]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:06 smithi092 ceph-mon[134067]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:58:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:06 smithi092 ceph-mon[134067]: osdmap e44: 7 total, 6 up, 7 in 2023-09-10T00:58:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:06 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:06 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:07.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:06 smithi003 ceph-mon[97654]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-09-10T00:58:07.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:06 smithi003 ceph-mon[97654]: from='osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:58:07.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:06 smithi003 ceph-mon[97654]: osdmap e44: 7 total, 6 up, 7 in 2023-09-10T00:58:07.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:06 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:07.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:06 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:07.343 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:58:08.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:07 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:58:08.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:07 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:58:08.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:07 smithi003 ceph-mon[97654]: osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] boot 2023-09-10T00:58:08.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:07 smithi003 ceph-mon[97654]: osdmap e45: 7 total, 7 up, 7 in 2023-09-10T00:58:08.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:07 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:08.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:07 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:58:08.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:07 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:58:08.125 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:07 smithi092 ceph-mon[134067]: osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] boot 2023-09-10T00:58:08.126 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:07 smithi092 ceph-mon[134067]: osdmap e45: 7 total, 7 up, 7 in 2023-09-10T00:58:08.126 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:07 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:08.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:07 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:58:08.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:07 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:58:08.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:07 smithi136 ceph-mon[133708]: osd.6 [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] boot 2023-09-10T00:58:08.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:07 smithi136 ceph-mon[133708]: osdmap e45: 7 total, 7 up, 7 in 2023-09-10T00:58:08.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:07 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T00:58:08.632 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 -- 172.21.15.92:0/201315012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 msgr2=0x7f3ae0100bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:08.632 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/201315012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0100bf0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ad00099c0 tx=0x7f3ad0009cd0 comp rx=0 tx=0).stop 2023-09-10T00:58:08.633 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 -- 172.21.15.92:0/201315012 shutdown_connections 2023-09-10T00:58:08.633 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/201315012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae00ff590 0x7f3ae00ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:08.633 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/201315012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae0101130 0x7f3ae01079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:08.633 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/201315012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0100bf0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:08.633 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 -- 172.21.15.92:0/201315012 >> 172.21.15.92:0/201315012 conn(0x7f3ae00fab10 msgr2=0x7f3ae00fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:58:08.634 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 -- 172.21.15.92:0/201315012 shutdown_connections 2023-09-10T00:58:08.634 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.629+0000 7f3ae66ed700 1 -- 172.21.15.92:0/201315012 wait complete. 2023-09-10T00:58:08.634 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 Processor -- start 2023-09-10T00:58:08.634 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 -- start start 2023-09-10T00:58:08.635 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae00ff590 0x7f3ae01952d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:08.635 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:08.635 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae0101130 0x7f3ae019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:08.635 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae019ce60 con 0x7f3ae0101130 2023-09-10T00:58:08.636 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae019cfa0 con 0x7f3ae0100790 2023-09-10T00:58:08.636 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3ae66ed700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae019d130 con 0x7f3ae00ff590 2023-09-10T00:58:08.636 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3adf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:08.636 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3adf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:51246/0 (socket says 172.21.15.92:51246) 2023-09-10T00:58:08.637 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.630+0000 7f3adf7fe700 1 -- 172.21.15.92:0/15694900 learned_addr learned my addr 172.21.15.92:0/15694900 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-09-10T00:58:08.637 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adffff700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae00ff590 0x7f3ae01952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:08.637 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3ae4c8a700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae0101130 0x7f3ae019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:08.637 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adf7fe700 1 -- 172.21.15.92:0/15694900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae00ff590 msgr2=0x7f3ae01952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:08.637 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adf7fe700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae00ff590 0x7f3ae01952d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:08.638 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adf7fe700 1 -- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae0101130 msgr2=0x7f3ae019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:08.638 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adf7fe700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae0101130 0x7f3ae019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:08.640 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adf7fe700 1 -- 172.21.15.92:0/15694900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad00095f0 con 0x7f3ae0100790 2023-09-10T00:58:08.640 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3ae4c8a700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae0101130 0x7f3ae019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:58:08.641 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adffff700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae00ff590 0x7f3ae01952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:58:08.641 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3adf7fe700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0195810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009990 tx=0x7f3ad0005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:58:08.641 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3add7fa700 1 -- 172.21.15.92:0/15694900 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad001d070 con 0x7f3ae0100790 2023-09-10T00:58:08.641 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3add7fa700 1 -- 172.21.15.92:0/15694900 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ad0022470 con 0x7f3ae0100790 2023-09-10T00:58:08.642 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae019d3b0 con 0x7f3ae0100790 2023-09-10T00:58:08.642 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3add7fa700 1 -- 172.21.15.92:0/15694900 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0004230 con 0x7f3ae0100790 2023-09-10T00:58:08.642 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.631+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae019d850 con 0x7f3ae0100790 2023-09-10T00:58:08.643 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.632+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae0106b70 con 0x7f3ae0100790 2023-09-10T00:58:08.643 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.633+0000 7f3add7fa700 1 -- 172.21.15.92:0/15694900 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f3ad00225e0 con 0x7f3ae0100790 2023-09-10T00:58:08.643 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.634+0000 7f3add7fa700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3ac80a0640 0x7f3ac80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:08.643 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.634+0000 7f3add7fa700 1 -- 172.21.15.92:0/15694900 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f3ad00cc700 con 0x7f3ae0100790 2023-09-10T00:58:08.644 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.634+0000 7f3adffff700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3ac80a0640 0x7f3ac80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:08.644 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.634+0000 7f3adffff700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3ac80a0640 0x7f3ac80a2b00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ae01005f0 tx=0x7f3ad4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:58:08.644 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.637+0000 7f3add7fa700 1 -- 172.21.15.92:0/15694900 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3ad0096fd0 con 0x7f3ae0100790 2023-09-10T00:58:08.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:08.837+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ae002d010 con 0x7f3ac80a0640 2023-09-10T00:58:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: osdmap e46: 7 total, 7 up, 7 in 2023-09-10T00:58:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:09.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: osdmap e46: 7 total, 7 up, 7 in 2023-09-10T00:58:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:09.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:09.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:09.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:09.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: osdmap e46: 7 total, 7 up, 7 in 2023-09-10T00:58:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:09.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:09.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:09.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:09.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:09.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:09.922 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:09 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi003 2023-09-10T00:58:09.923 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:09 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi003 to 5757M 2023-09-10T00:58:09.923 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:09 smithi092 ceph-mon[134067]: from='client.24410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:10.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:09 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi003 2023-09-10T00:58:10.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:09 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi003 to 5757M 2023-09-10T00:58:10.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:09 smithi136 ceph-mon[133708]: from='client.24410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:09 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi003 2023-09-10T00:58:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:09 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi003 to 5757M 2023-09-10T00:58:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:09 smithi003 ceph-mon[97654]: from='client.24410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:10 smithi136 ceph-mon[133708]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:10 smithi092 ceph-mon[134067]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:11.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:10 smithi003 ceph-mon[97654]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:12.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:11 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/894820625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]: dispatch 2023-09-10T00:58:12.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:11 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]: dispatch 2023-09-10T00:58:12.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:11 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]': finished 2023-09-10T00:58:12.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:11 smithi136 ceph-mon[133708]: osdmap e47: 8 total, 7 up, 8 in 2023-09-10T00:58:12.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:12.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:11 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/515339667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:12.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:11 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/894820625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]: dispatch 2023-09-10T00:58:12.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:11 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]: dispatch 2023-09-10T00:58:12.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:11 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]': finished 2023-09-10T00:58:12.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:11 smithi092 ceph-mon[134067]: osdmap e47: 8 total, 7 up, 8 in 2023-09-10T00:58:12.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:12.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:11 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/515339667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:11 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/894820625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]: dispatch 2023-09-10T00:58:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:11 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]: dispatch 2023-09-10T00:58:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:11 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23003010-ab55-4574-8464-0f56aec3ddbd"}]': finished 2023-09-10T00:58:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:11 smithi003 ceph-mon[97654]: osdmap e47: 8 total, 7 up, 8 in 2023-09-10T00:58:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:12.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:11 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/515339667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:13.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:12 smithi136 ceph-mon[133708]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:13.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:12 smithi092 ceph-mon[134067]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:12 smithi003 ceph-mon[97654]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:15.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:14 smithi136 ceph-mon[133708]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:15.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:14 smithi092 ceph-mon[134067]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:14 smithi003 ceph-mon[97654]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:17.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:16 smithi136 ceph-mon[133708]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:16 smithi092 ceph-mon[134067]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:16 smithi003 ceph-mon[97654]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:18.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-09-10T00:58:18.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:18.209 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-09-10T00:58:18.210 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:18.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-09-10T00:58:18.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:18 smithi136 ceph-mon[133708]: Deploying daemon osd.7 on smithi092 2023-09-10T00:58:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:18 smithi136 ceph-mon[133708]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:19.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:18 smithi092 ceph-mon[134067]: Deploying daemon osd.7 on smithi092 2023-09-10T00:58:19.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:18 smithi092 ceph-mon[134067]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:19.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:18 smithi003 ceph-mon[97654]: Deploying daemon osd.7 on smithi092 2023-09-10T00:58:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:18 smithi003 ceph-mon[97654]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:21.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:20 smithi092 ceph-mon[134067]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:21.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:20 smithi003 ceph-mon[97654]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:21.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:20 smithi136 ceph-mon[133708]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:23.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:22 smithi092 ceph-mon[134067]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:23.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:23.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:23.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:23.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:22 smithi136 ceph-mon[133708]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:23.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:23.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:23.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:23.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:22 smithi003 ceph-mon[97654]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:23.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:23.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:23.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:24.550 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 7 on host 'smithi092' 2023-09-10T00:58:24.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.548+0000 7f3add7fa700 1 -- 172.21.15.92:0/15694900 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3ae002d010 con 0x7f3ac80a0640 2023-09-10T00:58:24.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3ac80a0640 msgr2=0x7f3ac80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:24.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3ac80a0640 0x7f3ac80a2b00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ae01005f0 tx=0x7f3ad4008040 comp rx=0 tx=0).stop 2023-09-10T00:58:24.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 msgr2=0x7f3ae0195810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:24.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0195810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009990 tx=0x7f3ad0005e80 comp rx=0 tx=0).stop 2023-09-10T00:58:24.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 shutdown_connections 2023-09-10T00:58:24.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3ac80a0640 0x7f3ac80a2b00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:24.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae00ff590 0x7f3ae01952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:24.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae0101130 0x7f3ae019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:24.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 --2- 172.21.15.92:0/15694900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ae0100790 0x7f3ae0195810 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:24.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 >> 172.21.15.92:0/15694900 conn(0x7f3ae00fab10 msgr2=0x7f3ae0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:58:24.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 shutdown_connections 2023-09-10T00:58:24.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:58:24.551+0000 7f3ae66ed700 1 -- 172.21.15.92:0/15694900 wait complete. 2023-09-10T00:58:25.027 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:25.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:25.028 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:25.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:25.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.030 DEBUG:teuthology.orchestra.run.smithi092:osd.7> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.7.service 2023-09-10T00:58:25.043 INFO:tasks.cephadm:Deploying osd.8 on smithi136 with /dev/vg_nvme/lv_2... 2023-09-10T00:58:25.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-09-10T00:58:25.077 INFO:journalctl@ceph.osd.7.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T00:58:25.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:25.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:25.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:25.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:25.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:25.617 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:58:26.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:25 smithi136 ceph-mon[133708]: from='osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-09-10T00:58:26.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:25 smithi136 ceph-mon[133708]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-09-10T00:58:26.185 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:25 smithi092 ceph-mon[134067]: from='osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-09-10T00:58:26.185 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:25 smithi092 ceph-mon[134067]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-09-10T00:58:26.185 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 00:58:25 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T00:58:25.857+0000 7fba78f17640 -1 osd.7 0 log_to_monitors true 2023-09-10T00:58:26.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:25 smithi003 ceph-mon[97654]: from='osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-09-10T00:58:26.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:25 smithi003 ceph-mon[97654]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-09-10T00:58:27.149 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-09-10T00:58:27.149 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-09-10T00:58:27.149 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-09-10T00:58:27.150 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-09-10T00:58:27.150 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00800803 s, 1.3 GB/s 2023-09-10T00:58:27.150 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-09-10T00:58:27.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:26 smithi136 ceph-mon[133708]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:27.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:26 smithi136 ceph-mon[133708]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-09-10T00:58:27.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:26 smithi136 ceph-mon[133708]: from='osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:58:27.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:26 smithi136 ceph-mon[133708]: osdmap e48: 8 total, 7 up, 8 in 2023-09-10T00:58:27.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:27.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:26 smithi136 ceph-mon[133708]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:58:27.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T00:58:26.912+0000 7fba6e58e700 -1 osd.7 0 waiting for initial osdmap 2023-09-10T00:58:27.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T00:58:26.924+0000 7fba66b7c700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:58:27.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-mon[134067]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:27.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-mon[134067]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-09-10T00:58:27.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-mon[134067]: from='osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:58:27.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-mon[134067]: osdmap e48: 8 total, 7 up, 8 in 2023-09-10T00:58:27.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:27.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:26 smithi092 ceph-mon[134067]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:58:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:26 smithi003 ceph-mon[97654]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:26 smithi003 ceph-mon[97654]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-09-10T00:58:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:26 smithi003 ceph-mon[97654]: from='osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:58:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:26 smithi003 ceph-mon[97654]: osdmap e48: 8 total, 7 up, 8 in 2023-09-10T00:58:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:26 smithi003 ceph-mon[97654]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:58:27.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2023-09-10T00:58:28.128 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:58:28.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:58:28.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: osdmap e49: 8 total, 7 up, 8 in 2023-09-10T00:58:28.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:28.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:28.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:28.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:28.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:27 smithi092 ceph-mon[134067]: osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] boot 2023-09-10T00:58:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:58:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: osdmap e49: 8 total, 7 up, 8 in 2023-09-10T00:58:28.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:28.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:28.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:28.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:28.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:27 smithi003 ceph-mon[97654]: osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] boot 2023-09-10T00:58:28.368 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:58:28.368 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: osdmap e49: 8 total, 7 up, 8 in 2023-09-10T00:58:28.368 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:28.368 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:28.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:28.370 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.370 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:28.370 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:28.370 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:28.370 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:27 smithi136 ceph-mon[133708]: osd.7 [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] boot 2023-09-10T00:58:29.190 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:28 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:58:29.191 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:28 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:58:29.191 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:28 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi092 2023-09-10T00:58:29.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:28 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi092 to 5722M 2023-09-10T00:58:29.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:28 smithi136 ceph-mon[133708]: pgmap v175: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:29.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:28 smithi136 ceph-mon[133708]: osdmap e50: 8 total, 8 up, 8 in 2023-09-10T00:58:29.193 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:29.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:28 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:58:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:28 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:58:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:28 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi092 2023-09-10T00:58:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:28 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi092 to 5722M 2023-09-10T00:58:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:28 smithi092 ceph-mon[134067]: pgmap v175: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:29.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:28 smithi092 ceph-mon[134067]: osdmap e50: 8 total, 8 up, 8 in 2023-09-10T00:58:29.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:29.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:28 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:58:29.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:28 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:58:29.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:28 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi092 2023-09-10T00:58:29.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:28 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi092 to 5722M 2023-09-10T00:58:29.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:28 smithi003 ceph-mon[97654]: pgmap v175: 1 pgs: 1 unknown; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-09-10T00:58:29.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:28 smithi003 ceph-mon[97654]: osdmap e50: 8 total, 8 up, 8 in 2023-09-10T00:58:29.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T00:58:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 -- 172.21.15.136:0/3801880958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 msgr2=0x7f97800ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 --2- 172.21.15.136:0/3801880958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f97800ff9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9770009990 tx=0x7f9770009ca0 comp rx=0 tx=0).stop 2023-09-10T00:58:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 -- 172.21.15.136:0/3801880958 shutdown_connections 2023-09-10T00:58:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 --2- 172.21.15.136:0/3801880958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f97800ff9b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 --2- 172.21.15.136:0/3801880958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9780101130 0x7f97801079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 --2- 172.21.15.136:0/3801880958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9780100790 0x7f9780100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 -- 172.21.15.136:0/3801880958 >> 172.21.15.136:0/3801880958 conn(0x7f97800fab10 msgr2=0x7f97800fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:58:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 -- 172.21.15.136:0/3801880958 shutdown_connections 2023-09-10T00:58:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.347+0000 7f9788173700 1 -- 172.21.15.136:0/3801880958 wait complete. 2023-09-10T00:58:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 Processor -- start 2023-09-10T00:58:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 -- start start 2023-09-10T00:58:29.353 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f9780068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:29.353 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9780100790 0x7f978019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:29.353 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9780101130 0x7f978019d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:29.353 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978019ded0 con 0x7f9780100790 2023-09-10T00:58:29.353 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978019e040 con 0x7f9780101130 2023-09-10T00:58:29.354 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9788173700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978019e200 con 0x7f97800ff590 2023-09-10T00:58:29.354 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f9780068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:29.354 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f9780068c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39362/0 (socket says 172.21.15.136:39362) 2023-09-10T00:58:29.354 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 -- 172.21.15.136:0/2039971225 learned_addr learned my addr 172.21.15.136:0/2039971225 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-09-10T00:58:29.354 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9786710700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9780101130 0x7f978019d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:29.355 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f978570e700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9780100790 0x7f978019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:29.355 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 -- 172.21.15.136:0/2039971225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9780101130 msgr2=0x7f978019d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:29.355 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9780101130 0x7f978019d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:29.355 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 -- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9780100790 msgr2=0x7f978019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:29.355 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9780100790 0x7f978019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:29.356 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 -- 172.21.15.136:0/2039971225 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700095f0 con 0x7f97800ff590 2023-09-10T00:58:29.356 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f978570e700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9780100790 0x7f978019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:58:29.356 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.348+0000 7f9785f0f700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f9780068c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f977000b5c0 tx=0x7f977000f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:58:29.356 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.349+0000 7f9776ffd700 1 -- 172.21.15.136:0/2039971225 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977001d070 con 0x7f97800ff590 2023-09-10T00:58:29.356 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.349+0000 7f9776ffd700 1 -- 172.21.15.136:0/2039971225 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9770003e20 con 0x7f97800ff590 2023-09-10T00:58:29.357 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.349+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978019e480 con 0x7f97800ff590 2023-09-10T00:58:29.357 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.349+0000 7f9776ffd700 1 -- 172.21.15.136:0/2039971225 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9770017650 con 0x7f97800ff590 2023-09-10T00:58:29.357 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.349+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978019e8f0 con 0x7f97800ff590 2023-09-10T00:58:29.358 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.350+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9780106520 con 0x7f97800ff590 2023-09-10T00:58:29.358 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.351+0000 7f9776ffd700 1 -- 172.21.15.136:0/2039971225 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f9770003960 con 0x7f97800ff590 2023-09-10T00:58:29.358 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.351+0000 7f9776ffd700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f976c0a06c0 0x7f976c0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:29.359 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.352+0000 7f9776ffd700 1 -- 172.21.15.136:0/2039971225 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f97700cc840 con 0x7f97800ff590 2023-09-10T00:58:29.359 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.352+0000 7f978570e700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f976c0a06c0 0x7f976c0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:29.359 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.352+0000 7f978570e700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f976c0a06c0 0x7f976c0a2b80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f977c005fd0 tx=0x7f977c005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:58:29.359 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.355+0000 7f9776ffd700 1 -- 172.21.15.136:0/2039971225 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9770096fc0 con 0x7f97800ff590 2023-09-10T00:58:29.552 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:29.550+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f978002cff0 con 0x7f976c0a06c0 2023-09-10T00:58:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:29 smithi092 ceph-mon[134067]: osdmap e51: 8 total, 8 up, 8 in 2023-09-10T00:58:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:30.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:30.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:29 smithi003 ceph-mon[97654]: osdmap e51: 8 total, 8 up, 8 in 2023-09-10T00:58:30.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:30.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:30.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:30.341 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:29 smithi136 ceph-mon[133708]: osdmap e51: 8 total, 8 up, 8 in 2023-09-10T00:58:30.342 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:30.342 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:30.342 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:31.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:30 smithi092 ceph-mon[134067]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:30 smithi092 ceph-mon[134067]: pgmap v178: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:30 smithi092 ceph-mon[134067]: osdmap e52: 8 total, 8 up, 8 in 2023-09-10T00:58:31.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:30 smithi136 ceph-mon[133708]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:31.221 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:30 smithi136 ceph-mon[133708]: pgmap v178: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:31.221 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:30 smithi136 ceph-mon[133708]: osdmap e52: 8 total, 8 up, 8 in 2023-09-10T00:58:31.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:30 smithi003 ceph-mon[97654]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:30 smithi003 ceph-mon[97654]: pgmap v178: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:30 smithi003 ceph-mon[97654]: osdmap e52: 8 total, 8 up, 8 in 2023-09-10T00:58:32.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:31 smithi092 ceph-mon[134067]: osdmap e53: 8 total, 8 up, 8 in 2023-09-10T00:58:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:31 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/1341381385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]: dispatch 2023-09-10T00:58:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:31 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]: dispatch 2023-09-10T00:58:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:31 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]': finished 2023-09-10T00:58:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:31 smithi092 ceph-mon[134067]: osdmap e54: 9 total, 8 up, 9 in 2023-09-10T00:58:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:31 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:31 smithi003 ceph-mon[97654]: osdmap e53: 8 total, 8 up, 8 in 2023-09-10T00:58:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:31 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/1341381385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]: dispatch 2023-09-10T00:58:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:31 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]: dispatch 2023-09-10T00:58:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:31 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]': finished 2023-09-10T00:58:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:31 smithi003 ceph-mon[97654]: osdmap e54: 9 total, 8 up, 9 in 2023-09-10T00:58:32.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:31 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:31 smithi136 ceph-mon[133708]: osdmap e53: 8 total, 8 up, 8 in 2023-09-10T00:58:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:31 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/1341381385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]: dispatch 2023-09-10T00:58:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:31 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]: dispatch 2023-09-10T00:58:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:31 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77f3d09f-2450-4861-9c06-172488429b52"}]': finished 2023-09-10T00:58:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:31 smithi136 ceph-mon[133708]: osdmap e54: 9 total, 8 up, 9 in 2023-09-10T00:58:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:31 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:33.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:32 smithi092 ceph-mon[134067]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:33.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:32 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/3406811277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:32 smithi003 ceph-mon[97654]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:32 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/3406811277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:33.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:32 smithi136 ceph-mon[133708]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:33.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:32 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/3406811277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:35.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:34 smithi092 ceph-mon[134067]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:35.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:34 smithi003 ceph-mon[97654]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:34 smithi136 ceph-mon[133708]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:37.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:36 smithi136 ceph-mon[133708]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:37.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:36 smithi092 ceph-mon[134067]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:37.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:36 smithi003 ceph-mon[97654]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:39.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:38 smithi092 ceph-mon[134067]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:39.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-09-10T00:58:39.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:39.267 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:38 smithi136 ceph-mon[133708]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:39.268 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-09-10T00:58:39.268 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:39.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:38 smithi003 ceph-mon[97654]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-09-10T00:58:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:40.124 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:39 smithi136 ceph-mon[133708]: Deploying daemon osd.8 on smithi136 2023-09-10T00:58:40.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:39 smithi092 ceph-mon[134067]: Deploying daemon osd.8 on smithi136 2023-09-10T00:58:40.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:39 smithi003 ceph-mon[97654]: Deploying daemon osd.8 on smithi136 2023-09-10T00:58:41.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:40 smithi136 ceph-mon[133708]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:41.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:40 smithi092 ceph-mon[134067]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:40 smithi003 ceph-mon[97654]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:43.153 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:42 smithi136 ceph-mon[133708]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:43.153 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:43.153 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:43.154 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:42 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:43.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:42 smithi092 ceph-mon[134067]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:43.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:43.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:43.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:42 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:43.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:42 smithi003 ceph-mon[97654]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:42 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:45.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:44 smithi136 ceph-mon[133708]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:44 smithi003 ceph-mon[97654]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:45.377 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.377+0000 7f9776ffd700 1 -- 172.21.15.136:0/2039971225 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f978002cff0 con 0x7f976c0a06c0 2023-09-10T00:58:45.378 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 8 on host 'smithi136' 2023-09-10T00:58:45.379 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f976c0a06c0 msgr2=0x7f976c0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:45.379 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f976c0a06c0 0x7f976c0a2b80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f977c005fd0 tx=0x7f977c005dc0 comp rx=0 tx=0).stop 2023-09-10T00:58:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 msgr2=0x7f9780068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f9780068c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f977000b5c0 tx=0x7f977000f7b0 comp rx=0 tx=0).stop 2023-09-10T00:58:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 shutdown_connections 2023-09-10T00:58:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f976c0a06c0 0x7f976c0a2b80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97800ff590 0x7f9780068c40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:45.381 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9780101130 0x7f978019d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:45.381 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 --2- 172.21.15.136:0/2039971225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9780100790 0x7f978019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:45.381 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 >> 172.21.15.136:0/2039971225 conn(0x7f97800fab10 msgr2=0x7f97800fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:58:45.381 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 shutdown_connections 2023-09-10T00:58:45.381 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:58:45.379+0000 7f9788173700 1 -- 172.21.15.136:0/2039971225 wait complete. 2023-09-10T00:58:45.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:44 smithi092 ceph-mon[134067]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:45.820 DEBUG:teuthology.orchestra.run.smithi136:osd.8> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.8.service 2023-09-10T00:58:45.822 INFO:tasks.cephadm:Deploying osd.9 on smithi003 with /dev/vg_nvme/lv_1... 2023-09-10T00:58:45.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-09-10T00:58:45.852 INFO:journalctl@ceph.osd.8.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T00:58:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:46.567 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:58:46.943 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 00:58:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T00:58:46.693+0000 7fde6f7b1640 -1 osd.8 0 log_to_monitors true 2023-09-10T00:58:47.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:47 smithi003 ceph-mon[97654]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:47.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:47 smithi003 ceph-mon[97654]: from='osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-09-10T00:58:47.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:47 smithi003 ceph-mon[97654]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-09-10T00:58:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:47 smithi136 ceph-mon[133708]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:47 smithi136 ceph-mon[133708]: from='osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-09-10T00:58:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:47 smithi136 ceph-mon[133708]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-09-10T00:58:47.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:47 smithi092 ceph-mon[134067]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:47.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:47 smithi092 ceph-mon[134067]: from='osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-09-10T00:58:47.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:47 smithi092 ceph-mon[134067]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-09-10T00:58:48.509 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-09-10T00:58:48.510 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-09-10T00:58:48.510 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2023-09-10T00:58:48.510 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2023-09-10T00:58:48.510 INFO:teuthology.orchestra.run.smithi003.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0578533 s, 181 MB/s 2023-09-10T00:58:48.511 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2023-09-10T00:58:48.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:48 smithi003 ceph-mon[97654]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-09-10T00:58:48.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:48 smithi003 ceph-mon[97654]: from='osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:58:48.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:48 smithi003 ceph-mon[97654]: osdmap e55: 9 total, 8 up, 9 in 2023-09-10T00:58:48.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:48 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:48.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:48 smithi003 ceph-mon[97654]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:58:48.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:48 smithi003 ceph-mon[97654]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:48.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T00:58:48.354+0000 7fde64e28700 -1 osd.8 0 waiting for initial osdmap 2023-09-10T00:58:48.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T00:58:48.363+0000 7fde5cc15700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:58:48.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-mon[133708]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-09-10T00:58:48.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-mon[133708]: from='osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:58:48.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-mon[133708]: osdmap e55: 9 total, 8 up, 9 in 2023-09-10T00:58:48.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:48.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-mon[133708]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:58:48.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:48 smithi136 ceph-mon[133708]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:48 smithi092 ceph-mon[134067]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-09-10T00:58:48.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:48 smithi092 ceph-mon[134067]: from='osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:58:48.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:48 smithi092 ceph-mon[134067]: osdmap e55: 9 total, 8 up, 9 in 2023-09-10T00:58:48.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:48 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:48.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:48 smithi092 ceph-mon[134067]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:58:48.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:48 smithi092 ceph-mon[134067]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-09-10T00:58:49.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi003:vg_nvme/lv_1 2023-09-10T00:58:49.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:58:49.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: osdmap e56: 9 total, 8 up, 9 in 2023-09-10T00:58:49.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi136 2023-09-10T00:58:49.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi136 to 5722M 2023-09-10T00:58:49.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:49.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:49.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:49.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:49.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:49.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] boot 2023-09-10T00:58:49.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: osdmap e57: 9 total, 9 up, 9 in 2023-09-10T00:58:49.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:49 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:58:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: osdmap e56: 9 total, 8 up, 9 in 2023-09-10T00:58:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi136 2023-09-10T00:58:49.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi136 to 5722M 2023-09-10T00:58:49.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:49.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:49.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:49.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:49.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:49.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] boot 2023-09-10T00:58:49.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: osdmap e57: 9 total, 9 up, 9 in 2023-09-10T00:58:49.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:49 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:58:49.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: osdmap e56: 9 total, 8 up, 9 in 2023-09-10T00:58:49.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi136 2023-09-10T00:58:49.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:58:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi136 to 5722M 2023-09-10T00:58:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:58:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:58:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:58:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: osd.8 [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] boot 2023-09-10T00:58:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: osdmap e57: 9 total, 9 up, 9 in 2023-09-10T00:58:49.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:49 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T00:58:49.884 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:58:50.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:50 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:58:50.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:50 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:58:50.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:50 smithi003 ceph-mon[97654]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:58:50.460 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:50 smithi003 ceph-mon[97654]: osdmap e58: 9 total, 9 up, 9 in 2023-09-10T00:58:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:50 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:58:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:50 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:58:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:50 smithi136 ceph-mon[133708]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:58:50.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:50 smithi136 ceph-mon[133708]: osdmap e58: 9 total, 9 up, 9 in 2023-09-10T00:58:50.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:50 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:58:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:50 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:58:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:50 smithi092 ceph-mon[134067]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:58:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:50 smithi092 ceph-mon[134067]: osdmap e58: 9 total, 9 up, 9 in 2023-09-10T00:58:51.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 -- 172.21.15.3:0/1066866322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494101890 msgr2=0x7fd494101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 --2- 172.21.15.3:0/1066866322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494101890 0x7fd494101cf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd4840099c0 tx=0x7fd484009cd0 comp rx=0 tx=0).stop 2023-09-10T00:58:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 -- 172.21.15.3:0/1066866322 shutdown_connections 2023-09-10T00:58:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 --2- 172.21.15.3:0/1066866322 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 0x7fd494100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 --2- 172.21.15.3:0/1066866322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd494102230 0x7fd494108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 --2- 172.21.15.3:0/1066866322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494101890 0x7fd494101cf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 -- 172.21.15.3:0/1066866322 >> 172.21.15.3:0/1066866322 conn(0x7fd4940fbbf0 msgr2=0x7fd4940fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:58:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 -- 172.21.15.3:0/1066866322 shutdown_connections 2023-09-10T00:58:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.475+0000 7fd49c11c700 1 -- 172.21.15.3:0/1066866322 wait complete. 2023-09-10T00:58:51.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.476+0000 7fd49c11c700 1 Processor -- start 2023-09-10T00:58:51.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49c11c700 1 -- start start 2023-09-10T00:58:51.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49c11c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 0x7fd494198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:51.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49c11c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd494101890 0x7fd494198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:51.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49c11c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494102230 0x7fd49419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:51.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49c11c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4941a00f0 con 0x7fd494102230 2023-09-10T00:58:51.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49c11c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4941a0230 con 0x7fd494101890 2023-09-10T00:58:51.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49c11c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4941a03c0 con 0x7fd494100690 2023-09-10T00:58:51.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd499eb8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 0x7fd494198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:51.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd499eb8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 0x7fd494198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44096/0 (socket says 172.21.15.3:44096) 2023-09-10T00:58:51.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd499eb8700 1 -- 172.21.15.3:0/372356847 learned_addr learned my addr 172.21.15.3:0/372356847 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:58:51.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd4996b7700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd494101890 0x7fd494198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:51.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.477+0000 7fd49a6b9700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494102230 0x7fd49419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:51.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd499eb8700 1 -- 172.21.15.3:0/372356847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd494101890 msgr2=0x7fd494198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:51.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd499eb8700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd494101890 0x7fd494198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:51.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd499eb8700 1 -- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494102230 msgr2=0x7fd49419fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:58:51.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd499eb8700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494102230 0x7fd49419fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:58:51.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd499eb8700 1 -- 172.21.15.3:0/372356847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4840095f0 con 0x7fd494100690 2023-09-10T00:58:51.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd4996b7700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd494101890 0x7fd494198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T00:58:51.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd49a6b9700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494102230 0x7fd49419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T00:58:51.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd499eb8700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 0x7fd494198560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd49000e9a0 tx=0x7fd49000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:58:51.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd48affd700 1 -- 172.21.15.3:0/372356847 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490009790 con 0x7fd494100690 2023-09-10T00:58:51.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4941a06a0 con 0x7fd494100690 2023-09-10T00:58:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd48affd700 1 -- 172.21.15.3:0/372356847 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd49000c810 con 0x7fd494100690 2023-09-10T00:58:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd48affd700 1 -- 172.21.15.3:0/372356847 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490003c70 con 0x7fd494100690 2023-09-10T00:58:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.478+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4941a0bf0 con 0x7fd494100690 2023-09-10T00:58:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.479+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd494192cd0 con 0x7fd494100690 2023-09-10T00:58:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.481+0000 7fd48affd700 1 -- 172.21.15.3:0/372356847 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fd49000cc80 con 0x7fd494100690 2023-09-10T00:58:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.483+0000 7fd48affd700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd4800a0640 0x7fd4800a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:58:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.483+0000 7fd48affd700 1 -- 172.21.15.3:0/372356847 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fd490014070 con 0x7fd494100690 2023-09-10T00:58:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.483+0000 7fd4996b7700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd4800a0640 0x7fd4800a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:58:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.483+0000 7fd4996b7700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd4800a0640 0x7fd4800a2b00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd48400b580 tx=0x7fd48401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:58:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.485+0000 7fd48affd700 1 -- 172.21.15.3:0/372356847 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd490095950 con 0x7fd494100690 2023-09-10T00:58:51.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:58:51.706+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd49402cf40 con 0x7fd4800a0640 2023-09-10T00:58:52.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:51 smithi092 ceph-mon[134067]: osdmap e59: 9 total, 9 up, 9 in 2023-09-10T00:58:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:52.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:51 smithi003 ceph-mon[97654]: osdmap e59: 9 total, 9 up, 9 in 2023-09-10T00:58:52.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:51 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:52.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:51 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:52.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:51 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:51 smithi136 ceph-mon[133708]: osdmap e59: 9 total, 9 up, 9 in 2023-09-10T00:58:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:51 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:58:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:51 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:58:52.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:51 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:58:53.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:52 smithi003 ceph-mon[97654]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:53.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:52 smithi003 ceph-mon[97654]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:58:53.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:52 smithi003 ceph-mon[97654]: osdmap e60: 9 total, 9 up, 9 in 2023-09-10T00:58:53.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:52 smithi092 ceph-mon[134067]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:53.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:52 smithi092 ceph-mon[134067]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:58:53.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:52 smithi092 ceph-mon[134067]: osdmap e60: 9 total, 9 up, 9 in 2023-09-10T00:58:53.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:52 smithi136 ceph-mon[133708]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:58:53.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:52 smithi136 ceph-mon[133708]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:58:53.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:52 smithi136 ceph-mon[133708]: osdmap e60: 9 total, 9 up, 9 in 2023-09-10T00:58:54.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:53 smithi092 ceph-mon[134067]: osdmap e61: 9 total, 9 up, 9 in 2023-09-10T00:58:54.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:53 smithi003 ceph-mon[97654]: osdmap e61: 9 total, 9 up, 9 in 2023-09-10T00:58:54.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:53 smithi136 ceph-mon[133708]: osdmap e61: 9 total, 9 up, 9 in 2023-09-10T00:58:55.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:54 smithi092 ceph-mon[134067]: pgmap v200: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-09-10T00:58:55.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:54 smithi003 ceph-mon[97654]: pgmap v200: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-09-10T00:58:55.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:54 smithi136 ceph-mon[133708]: pgmap v200: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-09-10T00:58:56.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2916779945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3bd058-2c37-4910-b1b9-661e888ef9aa"}]: dispatch 2023-09-10T00:58:56.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2916779945' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b3bd058-2c37-4910-b1b9-661e888ef9aa"}]': finished 2023-09-10T00:58:56.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:55 smithi092 ceph-mon[134067]: osdmap e62: 10 total, 9 up, 10 in 2023-09-10T00:58:56.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:58:56.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2869049606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2916779945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3bd058-2c37-4910-b1b9-661e888ef9aa"}]: dispatch 2023-09-10T00:58:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2916779945' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b3bd058-2c37-4910-b1b9-661e888ef9aa"}]': finished 2023-09-10T00:58:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:55 smithi003 ceph-mon[97654]: osdmap e62: 10 total, 9 up, 10 in 2023-09-10T00:58:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:58:56.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2869049606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:56.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2916779945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b3bd058-2c37-4910-b1b9-661e888ef9aa"}]: dispatch 2023-09-10T00:58:56.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2916779945' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b3bd058-2c37-4910-b1b9-661e888ef9aa"}]': finished 2023-09-10T00:58:56.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:55 smithi136 ceph-mon[133708]: osdmap e62: 10 total, 9 up, 10 in 2023-09-10T00:58:56.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:58:56.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2869049606' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:58:57.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:57 smithi003 ceph-mon[97654]: pgmap v202: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2023-09-10T00:58:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:57 smithi136 ceph-mon[133708]: pgmap v202: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2023-09-10T00:58:57.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:57 smithi092 ceph-mon[134067]: pgmap v202: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2023-09-10T00:58:59.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:58:59 smithi003 ceph-mon[97654]: pgmap v203: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:58:59.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:58:59 smithi136 ceph-mon[133708]: pgmap v203: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:58:59.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:58:59 smithi092 ceph-mon[134067]: pgmap v203: 1 pgs: 1 activating; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-09-10T00:59:01.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:01 smithi003 ceph-mon[97654]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-09-10T00:59:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:01 smithi136 ceph-mon[133708]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-09-10T00:59:01.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:01 smithi092 ceph-mon[134067]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-09-10T00:59:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:03 smithi003 ceph-mon[97654]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-09-10T00:59:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:03 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-09-10T00:59:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:03 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:03.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:03 smithi136 ceph-mon[133708]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-09-10T00:59:03.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:03 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-09-10T00:59:03.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:03 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:03.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:03 smithi092 ceph-mon[134067]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-09-10T00:59:03.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:03 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-09-10T00:59:03.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:03 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:04 smithi003 ceph-mon[97654]: Deploying daemon osd.9 on smithi003 2023-09-10T00:59:04.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:04 smithi136 ceph-mon[133708]: Deploying daemon osd.9 on smithi003 2023-09-10T00:59:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:04 smithi092 ceph-mon[134067]: Deploying daemon osd.9 on smithi003 2023-09-10T00:59:05.243 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:05 smithi003 ceph-mon[97654]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:05 smithi136 ceph-mon[133708]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:05.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:05 smithi092 ceph-mon[134067]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:07.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:07 smithi003 ceph-mon[97654]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:07.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:07 smithi136 ceph-mon[133708]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:07.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:07 smithi092 ceph-mon[134067]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:08.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:08 smithi136 ceph-mon[133708]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:08 smithi092 ceph-mon[134067]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:08.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:08 smithi003 ceph-mon[97654]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:10 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:10 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:10 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:11 smithi003 ceph-mon[97654]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:11.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:11 smithi136 ceph-mon[133708]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:11.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:11 smithi092 ceph-mon[134067]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:12.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.419+0000 7fd48affd700 1 -- 172.21.15.3:0/372356847 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd49402cf40 con 0x7fd4800a0640 2023-09-10T00:59:12.421 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 9 on host 'smithi003' 2023-09-10T00:59:12.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd4800a0640 msgr2=0x7fd4800a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:12.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd4800a0640 0x7fd4800a2b00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd48400b580 tx=0x7fd48401a040 comp rx=0 tx=0).stop 2023-09-10T00:59:12.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 msgr2=0x7fd494198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:12.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 0x7fd494198560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd49000e9a0 tx=0x7fd49000ecb0 comp rx=0 tx=0).stop 2023-09-10T00:59:12.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 shutdown_connections 2023-09-10T00:59:12.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd4800a0640 0x7fd4800a2b00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:12.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd494100690 0x7fd494198560 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:12.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd494102230 0x7fd49419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:12.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 --2- 172.21.15.3:0/372356847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd494101890 0x7fd494198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:12.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 >> 172.21.15.3:0/372356847 conn(0x7fd4940fbbf0 msgr2=0x7fd494107220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:59:12.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 shutdown_connections 2023-09-10T00:59:12.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:12.422+0000 7fd49c11c700 1 -- 172.21.15.3:0/372356847 wait complete. 2023-09-10T00:59:12.921 DEBUG:teuthology.orchestra.run.smithi003:osd.9> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.9.service 2023-09-10T00:59:12.925 INFO:tasks.cephadm:Deploying osd.10 on smithi092 with /dev/vg_nvme/lv_1... 2023-09-10T00:59:12.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-09-10T00:59:13.011 INFO:journalctl@ceph.osd.9.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T00:59:13.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-09-10T00:59:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:13 smithi092 ceph-mon[134067]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-09-10T00:59:13.309 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T00:59:13.019+0000 7fbcac189640 -1 osd.9 0 log_to_monitors true 2023-09-10T00:59:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-09-10T00:59:13.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:13 smithi003 ceph-mon[97654]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-09-10T00:59:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:13.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:13.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:13.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:13.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-09-10T00:59:13.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:13 smithi136 ceph-mon[133708]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-09-10T00:59:13.497 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:59:14.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:14 smithi092 ceph-mon[134067]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-09-10T00:59:14.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:14 smithi092 ceph-mon[134067]: osdmap e63: 10 total, 9 up, 10 in 2023-09-10T00:59:14.546 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:14.547 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:14 smithi092 ceph-mon[134067]: from='osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:59:14.547 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:14 smithi092 ceph-mon[134067]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:59:14.547 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:14 smithi092 ceph-mon[134067]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:14 smithi136 ceph-mon[133708]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-09-10T00:59:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:14 smithi136 ceph-mon[133708]: osdmap e63: 10 total, 9 up, 10 in 2023-09-10T00:59:14.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:14.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:14 smithi136 ceph-mon[133708]: from='osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:59:14.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:14 smithi136 ceph-mon[133708]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:59:14.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:14 smithi136 ceph-mon[133708]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:14.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-mon[97654]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-09-10T00:59:14.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-mon[97654]: osdmap e63: 10 total, 9 up, 10 in 2023-09-10T00:59:14.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:14.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-mon[97654]: from='osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:59:14.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-mon[97654]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2023-09-10T00:59:14.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-mon[97654]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-09-10T00:59:14.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T00:59:14.411+0000 7fbca1800700 -1 osd.9 0 waiting for initial osdmap 2023-09-10T00:59:14.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 00:59:14 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T00:59:14.433+0000 7fbc99dee700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:59:15.001 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-09-10T00:59:15.001 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-09-10T00:59:15.001 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-09-10T00:59:15.001 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-09-10T00:59:15.002 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140559 s, 746 MB/s 2023-09-10T00:59:15.002 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-09-10T00:59:15.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi092:vg_nvme/lv_1 2023-09-10T00:59:15.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:15 smithi092 ceph-mon[134067]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:59:15.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:15 smithi092 ceph-mon[134067]: osdmap e64: 10 total, 9 up, 10 in 2023-09-10T00:59:15.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:15 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:15.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:15 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:15.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:15 smithi003 ceph-mon[97654]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:59:15.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:15 smithi003 ceph-mon[97654]: osdmap e64: 10 total, 9 up, 10 in 2023-09-10T00:59:15.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:15 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:15.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:15 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:15.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:15 smithi136 ceph-mon[133708]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2023-09-10T00:59:15.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:15 smithi136 ceph-mon[133708]: osdmap e64: 10 total, 9 up, 10 in 2023-09-10T00:59:15.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:15 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:15.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:15 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:16.007 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T00:59:16.458 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:59:16.458 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:59:16.459 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] boot 2023-09-10T00:59:16.459 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: osdmap e65: 10 total, 10 up, 10 in 2023-09-10T00:59:16.459 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:16.459 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:16.459 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.460 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.460 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.460 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.460 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.460 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.461 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.461 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:16.461 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:16.461 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:59:16.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:59:16.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] boot 2023-09-10T00:59:16.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: osdmap e65: 10 total, 10 up, 10 in 2023-09-10T00:59:16.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:16.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:16.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:16.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:16.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:59:16.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:59:16.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: osd.9 [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] boot 2023-09-10T00:59:16.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: osdmap e65: 10 total, 10 up, 10 in 2023-09-10T00:59:16.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T00:59:16.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:16.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:16.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:16.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:16.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:16.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:17.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 -- 172.21.15.92:0/2159178790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13080ff590 msgr2=0x7f13080ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:17.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 --2- 172.21.15.92:0/2159178790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13080ff590 0x7f13080ff9b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0099c0 tx=0x7f12fc009cd0 comp rx=0 tx=0).stop 2023-09-10T00:59:17.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 -- 172.21.15.92:0/2159178790 shutdown_connections 2023-09-10T00:59:17.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 --2- 172.21.15.92:0/2159178790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1308101130 0x7f13081079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:17.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 --2- 172.21.15.92:0/2159178790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13080ff590 0x7f13080ff9b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:17.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 --2- 172.21.15.92:0/2159178790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1308100790 0x7f1308100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:17.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 -- 172.21.15.92:0/2159178790 >> 172.21.15.92:0/2159178790 conn(0x7f13080fab10 msgr2=0x7f13080fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:59:17.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 -- 172.21.15.92:0/2159178790 shutdown_connections 2023-09-10T00:59:17.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.187+0000 7f130ee77700 1 -- 172.21.15.92:0/2159178790 wait complete. 2023-09-10T00:59:17.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 Processor -- start 2023-09-10T00:59:17.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 -- start start 2023-09-10T00:59:17.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13080ff590 0x7f1308196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:17.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1308100790 0x7f1308196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:17.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1308101130 0x7f130819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:17.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130819dfa0 con 0x7f13080ff590 2023-09-10T00:59:17.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130819e0e0 con 0x7f1308101130 2023-09-10T00:59:17.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130ee77700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130819e270 con 0x7f1308100790 2023-09-10T00:59:17.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1308101130 0x7f130819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:17.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1308101130 0x7f130819d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:51402/0 (socket says 172.21.15.92:51402) 2023-09-10T00:59:17.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 -- 172.21.15.92:0/3984406607 learned_addr learned my addr 172.21.15.92:0/3984406607 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-09-10T00:59:17.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130cc13700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13080ff590 0x7f1308196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:17.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f1307fff700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1308100790 0x7f1308196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:17.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 -- 172.21.15.92:0/3984406607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1308100790 msgr2=0x7f1308196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:17.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1308100790 0x7f1308196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:17.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 -- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13080ff590 msgr2=0x7f1308196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:17.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13080ff590 0x7f1308196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:17.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 -- 172.21.15.92:0/3984406607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc0095f0 con 0x7f1308101130 2023-09-10T00:59:17.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f1307fff700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1308100790 0x7f1308196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T00:59:17.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.188+0000 7f130d414700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1308101130 0x7f130819d9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1300007ee0 tx=0x7f130000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:59:17.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.189+0000 7f1305ffb700 1 -- 172.21.15.92:0/3984406607 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300017070 con 0x7f1308101130 2023-09-10T00:59:17.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.189+0000 7f1305ffb700 1 -- 172.21.15.92:0/3984406607 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f130000f040 con 0x7f1308101130 2023-09-10T00:59:17.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.189+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130819e550 con 0x7f1308101130 2023-09-10T00:59:17.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.189+0000 7f1305ffb700 1 -- 172.21.15.92:0/3984406607 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300005740 con 0x7f1308101130 2023-09-10T00:59:17.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.189+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130819ea00 con 0x7f1308101130 2023-09-10T00:59:17.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.190+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130810ae10 con 0x7f1308101130 2023-09-10T00:59:17.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.191+0000 7f1305ffb700 1 -- 172.21.15.92:0/3984406607 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f130000dbd0 con 0x7f1308101130 2023-09-10T00:59:17.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.191+0000 7f1305ffb700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f12f80a0640 0x7f12f80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:17.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.192+0000 7f1305ffb700 1 -- 172.21.15.92:0/3984406607 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f13000cb280 con 0x7f1308101130 2023-09-10T00:59:17.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.192+0000 7f130cc13700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f12f80a0640 0x7f12f80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:17.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.192+0000 7f130cc13700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f12f80a0640 0x7f12f80a2b00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00b580 tx=0x7f12fc005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:59:17.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.195+0000 7f1305ffb700 1 -- 172.21.15.92:0/3984406607 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f13000955b0 con 0x7f1308101130 2023-09-10T00:59:17.392 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:17.391+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1308069080 con 0x7f12f80a0640 2023-09-10T00:59:17.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:17 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi003 2023-09-10T00:59:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:17 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi003 to 4318M 2023-09-10T00:59:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:17 smithi092 ceph-mon[134067]: osdmap e66: 10 total, 10 up, 10 in 2023-09-10T00:59:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:59:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:59:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:17.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:17 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi003 2023-09-10T00:59:17.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:17 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi003 to 4318M 2023-09-10T00:59:17.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:17 smithi003 ceph-mon[97654]: osdmap e66: 10 total, 10 up, 10 in 2023-09-10T00:59:17.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:59:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:59:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:17.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:17 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi003 2023-09-10T00:59:17.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:17 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi003 to 4318M 2023-09-10T00:59:17.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:17 smithi136 ceph-mon[133708]: osdmap e66: 10 total, 10 up, 10 in 2023-09-10T00:59:17.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:59:17.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:59:17.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:18.709 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:18 smithi092 ceph-mon[134067]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:59:18.709 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:18 smithi092 ceph-mon[134067]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:18.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:18 smithi003 ceph-mon[97654]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:59:18.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:18 smithi003 ceph-mon[97654]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:18.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:18 smithi136 ceph-mon[133708]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:59:18.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:18 smithi136 ceph-mon[133708]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:20.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:19 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/2321961097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]: dispatch 2023-09-10T00:59:20.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:19 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]: dispatch 2023-09-10T00:59:20.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:19 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]': finished 2023-09-10T00:59:20.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:19 smithi003 ceph-mon[97654]: osdmap e67: 11 total, 10 up, 11 in 2023-09-10T00:59:20.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:19 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:20.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:19 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/2321961097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]: dispatch 2023-09-10T00:59:20.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:19 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]: dispatch 2023-09-10T00:59:20.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:19 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]': finished 2023-09-10T00:59:20.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:19 smithi136 ceph-mon[133708]: osdmap e67: 11 total, 10 up, 11 in 2023-09-10T00:59:20.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:19 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:20.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:19 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/2321961097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]: dispatch 2023-09-10T00:59:20.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:19 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]: dispatch 2023-09-10T00:59:20.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:19 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b312bb63-21fa-4f9e-b676-c0cfca04749a"}]': finished 2023-09-10T00:59:20.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:19 smithi092 ceph-mon[134067]: osdmap e67: 11 total, 10 up, 11 in 2023-09-10T00:59:20.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:19 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:21.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:20 smithi136 ceph-mon[133708]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:21.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:20 smithi136 ceph-mon[133708]: from='client.? 172.21.15.92:0/3568188800' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:59:21.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:20 smithi092 ceph-mon[134067]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:21.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:20 smithi092 ceph-mon[134067]: from='client.? 172.21.15.92:0/3568188800' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:59:21.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:20 smithi003 ceph-mon[97654]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:20 smithi003 ceph-mon[97654]: from='client.? 172.21.15.92:0/3568188800' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:59:23.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:22 smithi092 ceph-mon[134067]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:23.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:22 smithi003 ceph-mon[97654]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:23.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:22 smithi136 ceph-mon[133708]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:25.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:24 smithi092 ceph-mon[134067]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:24 smithi003 ceph-mon[97654]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:25.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:24 smithi136 ceph-mon[133708]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:27.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:26 smithi092 ceph-mon[134067]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:27.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-09-10T00:59:27.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:26 smithi003 ceph-mon[97654]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-09-10T00:59:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:27.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:26 smithi136 ceph-mon[133708]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-09-10T00:59:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:28.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:27 smithi092 ceph-mon[134067]: Deploying daemon osd.10 on smithi092 2023-09-10T00:59:28.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:27 smithi003 ceph-mon[97654]: Deploying daemon osd.10 on smithi092 2023-09-10T00:59:28.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:27 smithi136 ceph-mon[133708]: Deploying daemon osd.10 on smithi092 2023-09-10T00:59:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:28 smithi092 ceph-mon[134067]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:29.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:28 smithi003 ceph-mon[97654]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:29.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:28 smithi136 ceph-mon[133708]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:31.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:30 smithi092 ceph-mon[134067]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:31.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:31.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:31.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:30 smithi003 ceph-mon[97654]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:31.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:30 smithi136 ceph-mon[133708]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:33.005 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 10 on host 'smithi092' 2023-09-10T00:59:33.005 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.003+0000 7f1305ffb700 1 -- 172.21.15.92:0/3984406607 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f1308069080 con 0x7f12f80a0640 2023-09-10T00:59:33.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f12f80a0640 msgr2=0x7f12f80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:33.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f12f80a0640 0x7f12f80a2b00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00b580 tx=0x7f12fc005e80 comp rx=0 tx=0).stop 2023-09-10T00:59:33.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1308101130 msgr2=0x7f130819d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:33.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1308101130 0x7f130819d9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1300007ee0 tx=0x7f130000d3b0 comp rx=0 tx=0).stop 2023-09-10T00:59:33.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 shutdown_connections 2023-09-10T00:59:33.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1308101130 0x7f130819d9d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:33.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13080ff590 0x7f1308196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:33.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f12f80a0640 0x7f12f80a2b00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:33.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 --2- 172.21.15.92:0/3984406607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1308100790 0x7f1308196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:33.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 >> 172.21.15.92:0/3984406607 conn(0x7f13080fab10 msgr2=0x7f1308068970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:59:33.009 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 shutdown_connections 2023-09-10T00:59:33.009 INFO:teuthology.orchestra.run.smithi092.stderr:2023-09-10T00:59:33.006+0000 7f130ee77700 1 -- 172.21.15.92:0/3984406607 wait complete. 2023-09-10T00:59:33.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:32 smithi092 ceph-mon[134067]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:32 smithi003 ceph-mon[97654]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:33.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:32 smithi136 ceph-mon[133708]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:33.484 DEBUG:teuthology.orchestra.run.smithi092:osd.10> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.10.service 2023-09-10T00:59:33.487 INFO:tasks.cephadm:Deploying osd.11 on smithi136 with /dev/vg_nvme/lv_1... 2023-09-10T00:59:33.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-09-10T00:59:33.515 INFO:journalctl@ceph.osd.10.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T00:59:34.056 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:59:34.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:33 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:33 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:33 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:33 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.261 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:33 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.262 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:33 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.262 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:33 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.262 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:33 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:33 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:33 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:33 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:33 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:34.566 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 00:59:34 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T00:59:34.340+0000 7f0a5289f640 -1 osd.10 0 log_to_monitors true 2023-09-10T00:59:35.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:34 smithi092 ceph-mon[134067]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:34 smithi092 ceph-mon[134067]: from='osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-09-10T00:59:35.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:34 smithi092 ceph-mon[134067]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-09-10T00:59:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:34 smithi003 ceph-mon[97654]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:34 smithi003 ceph-mon[97654]: from='osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-09-10T00:59:35.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:34 smithi003 ceph-mon[97654]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-09-10T00:59:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:34 smithi136 ceph-mon[133708]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:34 smithi136 ceph-mon[133708]: from='osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-09-10T00:59:35.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:34 smithi136 ceph-mon[133708]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-09-10T00:59:35.553 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-09-10T00:59:35.553 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-09-10T00:59:35.553 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2023-09-10T00:59:35.553 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2023-09-10T00:59:35.554 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0082322 s, 1.3 GB/s 2023-09-10T00:59:35.554 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2023-09-10T00:59:36.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2023-09-10T00:59:36.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T00:59:36.210+0000 7f0a47f16700 -1 osd.10 0 waiting for initial osdmap 2023-09-10T00:59:36.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T00:59:36.221+0000 7f0a40504700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:59:36.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-09-10T00:59:36.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: osdmap e68: 11 total, 10 up, 11 in 2023-09-10T00:59:36.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:36.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:59:36.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:59:36.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:36.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:36.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:36.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-09-10T00:59:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: osdmap e68: 11 total, 10 up, 11 in 2023-09-10T00:59:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:59:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:59:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:36.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:36.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.568 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T00:59:36.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-09-10T00:59:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: osdmap e68: 11 total, 10 up, 11 in 2023-09-10T00:59:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:59:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-09-10T00:59:36.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-09-10T00:59:36.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:36.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:36.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:36.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:36.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:37.344 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi092 2023-09-10T00:59:37.344 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi092 to 4292M 2023-09-10T00:59:37.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:59:37.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: osdmap e69: 11 total, 10 up, 11 in 2023-09-10T00:59:37.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:37.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:37.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:37.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:37.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:37.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:37.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi092 2023-09-10T00:59:37.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi092 to 4292M 2023-09-10T00:59:37.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:59:37.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: osdmap e69: 11 total, 10 up, 11 in 2023-09-10T00:59:37.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:37.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:37.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:37.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:37.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:37.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi092 2023-09-10T00:59:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi092 to 4292M 2023-09-10T00:59:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-09-10T00:59:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: osdmap e69: 11 total, 10 up, 11 in 2023-09-10T00:59:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:37.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:37.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:37.753 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 -- 172.21.15.136:0/817315537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5000100820 msgr2=0x7f5000100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:37.753 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 --2- 172.21.15.136:0/817315537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5000100820 0x7f5000100ca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009990 tx=0x7f4ff0009ca0 comp rx=0 tx=0).stop 2023-09-10T00:59:37.754 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 -- 172.21.15.136:0/817315537 shutdown_connections 2023-09-10T00:59:37.754 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 --2- 172.21.15.136:0/817315537 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50001011e0 0x7f5000107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:37.754 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 --2- 172.21.15.136:0/817315537 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50000ffb00 0x7f50000fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:37.754 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 --2- 172.21.15.136:0/817315537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5000100820 0x7f5000100ca0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:37.754 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 -- 172.21.15.136:0/817315537 >> 172.21.15.136:0/817315537 conn(0x7f50000fa2e0 msgr2=0x7f50000fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:59:37.755 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 -- 172.21.15.136:0/817315537 shutdown_connections 2023-09-10T00:59:37.755 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 -- 172.21.15.136:0/817315537 wait complete. 2023-09-10T00:59:37.755 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.751+0000 7f5005833700 1 Processor -- start 2023-09-10T00:59:37.755 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5005833700 1 -- start start 2023-09-10T00:59:37.755 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5005833700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50000ffb00 0x7f50000ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:37.756 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5005833700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5000100820 0x7f500019b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:37.756 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5005833700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50001011e0 0x7f500019d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:37.756 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5005833700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500019dfd0 con 0x7f5000100820 2023-09-10T00:59:37.756 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5005833700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500019e140 con 0x7f50000ffb00 2023-09-10T00:59:37.756 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5005833700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500019e300 con 0x7f50001011e0 2023-09-10T00:59:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50001011e0 0x7f500019d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50001011e0 0x7f500019d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39514/0 (socket says 172.21.15.136:39514) 2023-09-10T00:59:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 -- 172.21.15.136:0/691572605 learned_addr learned my addr 172.21.15.136:0/691572605 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-09-10T00:59:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4ffe7fc700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5000100820 0x7f500019b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4ffeffd700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50000ffb00 0x7f50000ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:37.758 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 -- 172.21.15.136:0/691572605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50000ffb00 msgr2=0x7f50000ff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:37.758 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50000ffb00 0x7f50000ff1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:37.758 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 -- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5000100820 msgr2=0x7f500019b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:37.758 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5000100820 0x7f500019b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:37.758 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 -- 172.21.15.136:0/691572605 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00095f0 con 0x7f50001011e0 2023-09-10T00:59:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4ffe7fc700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5000100820 0x7f500019b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T00:59:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f4fff7fe700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50001011e0 0x7f500019d830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800ada0 tx=0x7f4ff800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:59:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.752+0000 7f5004831700 1 -- 172.21.15.136:0/691572605 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff800f070 con 0x7f50001011e0 2023-09-10T00:59:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.753+0000 7f5004831700 1 -- 172.21.15.136:0/691572605 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ff8007b30 con 0x7f50001011e0 2023-09-10T00:59:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.753+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500019e5e0 con 0x7f50001011e0 2023-09-10T00:59:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.753+0000 7f5004831700 1 -- 172.21.15.136:0/691572605 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8005440 con 0x7f50001011e0 2023-09-10T00:59:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.753+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500019eb30 con 0x7f50001011e0 2023-09-10T00:59:37.761 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.753+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500010ae70 con 0x7f50001011e0 2023-09-10T00:59:37.761 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.755+0000 7f5004831700 1 -- 172.21.15.136:0/691572605 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f4ff8004ad0 con 0x7f50001011e0 2023-09-10T00:59:37.761 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.755+0000 7f5004831700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4fe80a06c0 0x7f4fe80a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:37.762 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.755+0000 7f5004831700 1 -- 172.21.15.136:0/691572605 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f4ff80cab30 con 0x7f50001011e0 2023-09-10T00:59:37.762 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.756+0000 7f4ffeffd700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4fe80a06c0 0x7f4fe80a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:37.762 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.756+0000 7f4ffeffd700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4fe80a06c0 0x7f4fe80a2b80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4006fd0 tx=0x7f4ff4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:59:37.762 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.759+0000 7f5004831700 1 -- 172.21.15.136:0/691572605 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f4ff8094d20 con 0x7f50001011e0 2023-09-10T00:59:37.957 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:37.957+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5000060a40 con 0x7f4fe80a06c0 2023-09-10T00:59:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:59:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:59:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] boot 2023-09-10T00:59:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: osdmap e70: 11 total, 11 up, 11 in 2023-09-10T00:59:38.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:38.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:38.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:59:38.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:59:38.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:59:38.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:59:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:59:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] boot 2023-09-10T00:59:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: osdmap e70: 11 total, 11 up, 11 in 2023-09-10T00:59:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:59:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:59:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:59:38.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:38.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:59:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:59:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: osd.10 [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] boot 2023-09-10T00:59:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: osdmap e70: 11 total, 11 up, 11 in 2023-09-10T00:59:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T00:59:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T00:59:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-09-10T00:59:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-09-10T00:59:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:39.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:39 smithi136 ceph-mon[133708]: osdmap e71: 11 total, 11 up, 11 in 2023-09-10T00:59:39.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:39 smithi136 ceph-mon[133708]: osdmap e72: 11 total, 11 up, 11 in 2023-09-10T00:59:39.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:39 smithi003 ceph-mon[97654]: osdmap e71: 11 total, 11 up, 11 in 2023-09-10T00:59:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:39 smithi003 ceph-mon[97654]: osdmap e72: 11 total, 11 up, 11 in 2023-09-10T00:59:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:39 smithi092 ceph-mon[134067]: osdmap e71: 11 total, 11 up, 11 in 2023-09-10T00:59:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:39 smithi092 ceph-mon[134067]: osdmap e72: 11 total, 11 up, 11 in 2023-09-10T00:59:41.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:41.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: osdmap e73: 11 total, 11 up, 11 in 2023-09-10T00:59:41.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/2589834379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]: dispatch 2023-09-10T00:59:41.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]: dispatch 2023-09-10T00:59:41.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]': finished 2023-09-10T00:59:41.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: osdmap e74: 12 total, 11 up, 12 in 2023-09-10T00:59:41.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:41.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.136:0/4035298027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:59:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: osdmap e73: 11 total, 11 up, 11 in 2023-09-10T00:59:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/2589834379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]: dispatch 2023-09-10T00:59:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]: dispatch 2023-09-10T00:59:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]': finished 2023-09-10T00:59:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: osdmap e74: 12 total, 11 up, 12 in 2023-09-10T00:59:41.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:41.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.136:0/4035298027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:59:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: osdmap e73: 11 total, 11 up, 11 in 2023-09-10T00:59:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/2589834379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]: dispatch 2023-09-10T00:59:41.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]: dispatch 2023-09-10T00:59:41.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d479a7b-430c-4707-89c8-d20e7b47d68e"}]': finished 2023-09-10T00:59:41.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: osdmap e74: 12 total, 11 up, 12 in 2023-09-10T00:59:41.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:41.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.136:0/4035298027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-09-10T00:59:43.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:42 smithi092 ceph-mon[134067]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:42 smithi003 ceph-mon[97654]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:42 smithi136 ceph-mon[133708]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:45.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:44 smithi092 ceph-mon[134067]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2023-09-10T00:59:45.263 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:44 smithi136 ceph-mon[133708]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2023-09-10T00:59:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:44 smithi003 ceph-mon[97654]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2023-09-10T00:59:47.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:46 smithi092 ceph-mon[134067]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-09-10T00:59:47.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-09-10T00:59:47.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:47.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:46 smithi003 ceph-mon[97654]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-09-10T00:59:47.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-09-10T00:59:47.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:47.317 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:46 smithi136 ceph-mon[133708]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-09-10T00:59:47.317 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-09-10T00:59:47.317 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:48.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:47 smithi092 ceph-mon[134067]: Deploying daemon osd.11 on smithi136 2023-09-10T00:59:48.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:47 smithi003 ceph-mon[97654]: Deploying daemon osd.11 on smithi136 2023-09-10T00:59:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:47 smithi136 ceph-mon[133708]: Deploying daemon osd.11 on smithi136 2023-09-10T00:59:49.032 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:48 smithi136 ceph-mon[133708]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-09-10T00:59:49.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:48 smithi092 ceph-mon[134067]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-09-10T00:59:49.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:48 smithi003 ceph-mon[97654]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-09-10T00:59:51.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:50 smithi136 ceph-mon[133708]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-09-10T00:59:51.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:50 smithi092 ceph-mon[134067]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-09-10T00:59:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:50 smithi003 ceph-mon[97654]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-09-10T00:59:52.101 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:51 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:52.101 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:51 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:52.101 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:51 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:52.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:52.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:51 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:52.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:51 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:52.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:51 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:52.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:51 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:53.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:52 smithi092 ceph-mon[134067]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-09-10T00:59:53.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:52 smithi136 ceph-mon[133708]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-09-10T00:59:53.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:52 smithi003 ceph-mon[97654]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-09-10T00:59:53.571 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 11 on host 'smithi136' 2023-09-10T00:59:53.571 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.569+0000 7f5004831700 1 -- 172.21.15.136:0/691572605 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f5000060a40 con 0x7f4fe80a06c0 2023-09-10T00:59:53.574 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4fe80a06c0 msgr2=0x7f4fe80a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:53.574 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4fe80a06c0 0x7f4fe80a2b80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4006fd0 tx=0x7f4ff4008040 comp rx=0 tx=0).stop 2023-09-10T00:59:53.574 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50001011e0 msgr2=0x7f500019d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:53.574 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50001011e0 0x7f500019d830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800ada0 tx=0x7f4ff800c330 comp rx=0 tx=0).stop 2023-09-10T00:59:53.575 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 shutdown_connections 2023-09-10T00:59:53.575 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4fe80a06c0 0x7f4fe80a2b80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:53.575 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50001011e0 0x7f500019d830 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:53.575 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50000ffb00 0x7f50000ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:53.575 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 --2- 172.21.15.136:0/691572605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5000100820 0x7f500019b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:53.576 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 >> 172.21.15.136:0/691572605 conn(0x7f50000fa2e0 msgr2=0x7f50000fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:59:53.576 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 shutdown_connections 2023-09-10T00:59:53.576 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T00:59:53.572+0000 7f5005833700 1 -- 172.21.15.136:0/691572605 wait complete. 2023-09-10T00:59:54.054 DEBUG:teuthology.orchestra.run.smithi136:osd.11> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.11.service 2023-09-10T00:59:54.056 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-09-10T00:59:54.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd stat -f json 2023-09-10T00:59:54.089 INFO:journalctl@ceph.osd.11.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T00:59:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:59:54.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:54 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:54 smithi003 ceph-mon[97654]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:59:54.864 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.864 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.865 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.865 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:54 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.865 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:54 smithi136 ceph-mon[133708]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:59:54.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:54 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:54.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:54 smithi092 ceph-mon[134067]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-09-10T00:59:55.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 00:59:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T00:59:54.862+0000 7f68ab4fc640 -1 osd.11 0 log_to_monitors true 2023-09-10T00:59:55.768 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:55 smithi136 ceph-mon[133708]: from='osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-09-10T00:59:55.768 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:55 smithi136 ceph-mon[133708]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-09-10T00:59:55.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:55 smithi092 ceph-mon[134067]: from='osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-09-10T00:59:55.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:55 smithi092 ceph-mon[134067]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-09-10T00:59:56.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:55 smithi003 ceph-mon[97654]: from='osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-09-10T00:59:56.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:55 smithi003 ceph-mon[97654]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-09-10T00:59:56.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/2110897362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 msgr2=0x7f1a140734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:56.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/2110897362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a140734e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1a040098f0 tx=0x7f1a04009c00 comp rx=0 tx=0).stop 2023-09-10T00:59:56.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/2110897362 shutdown_connections 2023-09-10T00:59:56.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/2110897362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a14108560 0x7f1a1410a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/2110897362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a14074630 0x7f1a14072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/2110897362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a140734e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/2110897362 >> 172.21.15.3:0/2110897362 conn(0x7f1a140fbbf0 msgr2=0x7f1a140fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:59:56.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/2110897362 shutdown_connections 2023-09-10T00:59:56.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.337+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/2110897362 wait complete. 2023-09-10T00:59:56.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.338+0000 7f1a1ba5f700 1 Processor -- start 2023-09-10T00:59:56.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a1ba5f700 1 -- start start 2023-09-10T00:59:56.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a1ba5f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a141985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:56.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a1ba5f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a14074630 0x7f1a14198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a1ba5f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a14108560 0x7f1a1419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a1ba5f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a141a0160 con 0x7f1a14073060 2023-09-10T00:59:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a1ba5f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a141a02a0 con 0x7f1a14108560 2023-09-10T00:59:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a1ba5f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a141a0430 con 0x7f1a14074630 2023-09-10T00:59:56.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.339+0000 7f1a197fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a141985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a18ffa700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a14074630 0x7f1a14198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a197fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a141985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36416/0 (socket says 172.21.15.3:36416) 2023-09-10T00:59:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a197fb700 1 -- 172.21.15.3:0/891364634 learned_addr learned my addr 172.21.15.3:0/891364634 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T00:59:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a19ffc700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a14108560 0x7f1a1419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a18ffa700 1 -- 172.21.15.3:0/891364634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a14108560 msgr2=0x7f1a1419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a18ffa700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a14108560 0x7f1a1419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a18ffa700 1 -- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 msgr2=0x7f1a141985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a18ffa700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a141985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a18ffa700 1 -- 172.21.15.3:0/891364634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04009520 con 0x7f1a14074630 2023-09-10T00:59:56.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a197fb700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a141985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T00:59:56.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a18ffa700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a14074630 0x7f1a14198b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009810 tx=0x7f1a0400b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:59:56.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.340+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/891364634 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0401c070 con 0x7f1a14074630 2023-09-10T00:59:56.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.341+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a141a06b0 con 0x7f1a14074630 2023-09-10T00:59:56.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.341+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/891364634 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a0400f7d0 con 0x7f1a14074630 2023-09-10T00:59:56.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.341+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/891364634 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04003bc0 con 0x7f1a14074630 2023-09-10T00:59:56.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.341+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a141a0b20 con 0x7f1a14074630 2023-09-10T00:59:56.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.342+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a14106c10 con 0x7f1a14074630 2023-09-10T00:59:56.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.344+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/891364634 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f1a04003d20 con 0x7f1a14074630 2023-09-10T00:59:56.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.345+0000 7f1a0a7fc700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1a000a0640 0x7f1a000a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T00:59:56.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.345+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/891364634 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f1a040cc7f0 con 0x7f1a14074630 2023-09-10T00:59:56.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.345+0000 7f1a197fb700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1a000a0640 0x7f1a000a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T00:59:56.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.346+0000 7f1a197fb700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1a000a0640 0x7f1a000a2b00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a10005950 tx=0x7f1a10009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T00:59:56.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.347+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/891364634 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1a04096a00 con 0x7f1a14074630 2023-09-10T00:59:56.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.555+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f1a1410e280 con 0x7f1a14074630 2023-09-10T00:59:56.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.556+0000 7f1a0a7fc700 1 -- 172.21.15.3:0/891364634 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7f1a0409a020 con 0x7f1a14074630 2023-09-10T00:59:56.557 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T00:59:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.558+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1a000a0640 msgr2=0x7f1a000a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.558+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1a000a0640 0x7f1a000a2b00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a10005950 tx=0x7f1a10009260 comp rx=0 tx=0).stop 2023-09-10T00:59:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.558+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a14074630 msgr2=0x7f1a14198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T00:59:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.558+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a14074630 0x7f1a14198b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009810 tx=0x7f1a0400b790 comp rx=0 tx=0).stop 2023-09-10T00:59:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 shutdown_connections 2023-09-10T00:59:56.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a14108560 0x7f1a1419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a14073060 0x7f1a141985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1a000a0640 0x7f1a000a2b00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 --2- 172.21.15.3:0/891364634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a14074630 0x7f1a14198b10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T00:59:56.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 >> 172.21.15.3:0/891364634 conn(0x7f1a140fbbf0 msgr2=0x7f1a140fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T00:59:56.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 shutdown_connections 2023-09-10T00:59:56.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T00:59:56.559+0000 7f1a1ba5f700 1 -- 172.21.15.3:0/891364634 wait complete. 2023-09-10T00:59:56.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T00:59:56.703+0000 7f68a0b73700 -1 osd.11 0 waiting for initial osdmap 2023-09-10T00:59:56.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T00:59:56.718+0000 7f6898960700 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-10T00:59:56.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-mon[133708]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-09-10T00:59:56.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-mon[133708]: osdmap e75: 12 total, 11 up, 12 in 2023-09-10T00:59:56.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:56.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-mon[133708]: from='osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:59:56.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-mon[133708]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:59:56.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-mon[133708]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:56.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:56 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/891364634' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-09-10T00:59:56.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:56 smithi003 ceph-mon[97654]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-09-10T00:59:56.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:56 smithi003 ceph-mon[97654]: osdmap e75: 12 total, 11 up, 12 in 2023-09-10T00:59:56.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:56.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:56 smithi003 ceph-mon[97654]: from='osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:59:56.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:56 smithi003 ceph-mon[97654]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:59:56.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:56 smithi003 ceph-mon[97654]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:56.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:56 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/891364634' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-09-10T00:59:56.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:56 smithi092 ceph-mon[134067]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-09-10T00:59:56.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:56 smithi092 ceph-mon[134067]: osdmap e75: 12 total, 11 up, 12 in 2023-09-10T00:59:56.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:56.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:56 smithi092 ceph-mon[134067]: from='osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:59:56.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:56 smithi092 ceph-mon[134067]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2023-09-10T00:59:56.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:56 smithi092 ceph-mon[134067]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:56.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:56 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/891364634' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-09-10T00:59:56.984 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1694307577,"num_in_osds":12,"osd_in_since":1694307580,"num_remapped_pgs":0} 2023-09-10T00:59:57.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:59:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: osdmap e76: 12 total, 11 up, 12 in 2023-09-10T00:59:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:57.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:57.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:57.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:57.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:57.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:57.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:57.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:57.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:57.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:57.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:57.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:57.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:57.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:57.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:57.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:57.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd stat -f json 2023-09-10T00:59:58.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:59:58.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: osdmap e76: 12 total, 11 up, 12 in 2023-09-10T00:59:58.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.053 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:58.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:58.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:58.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:58.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:58.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2023-09-10T00:59:58.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: osdmap e76: 12 total, 11 up, 12 in 2023-09-10T00:59:58.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T00:59:58.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:58.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:58.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T00:59:58.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T00:59:58.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T00:59:58.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T00:59:58.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.725 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T00:59:58.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: purged_snaps scrub starts 2023-09-10T00:59:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: purged_snaps scrub ok 2023-09-10T00:59:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: Detected new or changed devices on smithi136 2023-09-10T00:59:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi136 to 4292M 2023-09-10T00:59:58.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] boot 2023-09-10T00:59:58.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: osdmap e77: 12 total, 12 up, 12 in 2023-09-10T00:59:58.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:58 smithi092 ceph-mon[134067]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:58.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: purged_snaps scrub starts 2023-09-10T00:59:58.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: purged_snaps scrub ok 2023-09-10T00:59:58.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: Detected new or changed devices on smithi136 2023-09-10T00:59:58.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi136 to 4292M 2023-09-10T00:59:58.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] boot 2023-09-10T00:59:58.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: osdmap e77: 12 total, 12 up, 12 in 2023-09-10T00:59:58.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:58.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:58 smithi003 ceph-mon[97654]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: purged_snaps scrub starts 2023-09-10T00:59:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: purged_snaps scrub ok 2023-09-10T00:59:59.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: Detected new or changed devices on smithi136 2023-09-10T00:59:59.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi136 to 4292M 2023-09-10T00:59:59.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: osd.11 [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] boot 2023-09-10T00:59:59.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: osdmap e77: 12 total, 12 up, 12 in 2023-09-10T00:59:59.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T00:59:59.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:58 smithi136 ceph-mon[133708]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-09-10T00:59:59.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 00:59:59 smithi003 ceph-mon[97654]: osdmap e78: 12 total, 12 up, 12 in 2023-09-10T00:59:59.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 00:59:59 smithi092 ceph-mon[134067]: osdmap e78: 12 total, 12 up, 12 in 2023-09-10T01:00:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.153+0000 7f4781cb0700 1 -- 172.21.15.3:0/1775307196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c100690 msgr2=0x7f477c100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.153+0000 7f4781cb0700 1 --2- 172.21.15.3:0/1775307196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c100690 0x7f477c100ab0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f476c0099c0 tx=0x7f476c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.153+0000 7f4781cb0700 1 -- 172.21.15.3:0/1775307196 shutdown_connections 2023-09-10T01:00:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.153+0000 7f4781cb0700 1 --2- 172.21.15.3:0/1775307196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f477c102230 0x7f477c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.153+0000 7f4781cb0700 1 --2- 172.21.15.3:0/1775307196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c100690 0x7f477c100ab0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.153+0000 7f4781cb0700 1 --2- 172.21.15.3:0/1775307196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f477c101890 0x7f477c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.153+0000 7f4781cb0700 1 -- 172.21.15.3:0/1775307196 >> 172.21.15.3:0/1775307196 conn(0x7f477c0fbbf0 msgr2=0x7f477c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.154+0000 7f4781cb0700 1 -- 172.21.15.3:0/1775307196 shutdown_connections 2023-09-10T01:00:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.154+0000 7f4781cb0700 1 -- 172.21.15.3:0/1775307196 wait complete. 2023-09-10T01:00:00.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 Processor -- start 2023-09-10T01:00:00.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 -- start start 2023-09-10T01:00:00.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f477c100690 0x7f477c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:00.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f477c101890 0x7f477c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:00.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c102230 0x7f477c19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:00.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c1a0100 con 0x7f477c102230 2023-09-10T01:00:00.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c1a0240 con 0x7f477c100690 2023-09-10T01:00:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.155+0000 7f4781cb0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c1a03d0 con 0x7f477c101890 2023-09-10T01:00:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c102230 0x7f477c19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f477c100690 0x7f477c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c102230 0x7f477c19fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36430/0 (socket says 172.21.15.3:36430) 2023-09-10T01:00:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f477c101890 0x7f477c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 -- 172.21.15.3:0/3257909779 learned_addr learned my addr 172.21.15.3:0/3257909779 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 -- 172.21.15.3:0/3257909779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f477c101890 msgr2=0x7f477c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f477c101890 0x7f477c198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 -- 172.21.15.3:0/3257909779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f477c100690 msgr2=0x7f477c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f477c100690 0x7f477c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 -- 172.21.15.3:0/3257909779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476c0095f0 con 0x7f477c102230 2023-09-10T01:00:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f477bfff700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c102230 0x7f477c19fb30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f477400ef60 tx=0x7f477400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f4778ff9700 1 -- 172.21.15.3:0/3257909779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774018070 con 0x7f477c102230 2023-09-10T01:00:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477c1a0650 con 0x7f477c102230 2023-09-10T01:00:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f4778ff9700 1 -- 172.21.15.3:0/3257909779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f477400f040 con 0x7f477c102230 2023-09-10T01:00:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.156+0000 7f4778ff9700 1 -- 172.21.15.3:0/3257909779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477400cc70 con 0x7f477c102230 2023-09-10T01:00:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.157+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c1a0b50 con 0x7f477c102230 2023-09-10T01:00:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.158+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477c192c80 con 0x7f477c102230 2023-09-10T01:00:00.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 00:59:59 smithi136 ceph-mon[133708]: osdmap e78: 12 total, 12 up, 12 in 2023-09-10T01:00:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.160+0000 7f4778ff9700 1 -- 172.21.15.3:0/3257909779 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f4774003a00 con 0x7f477c102230 2023-09-10T01:00:00.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.161+0000 7f4778ff9700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f47640a0640 0x7f47640a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:00.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.161+0000 7f4778ff9700 1 -- 172.21.15.3:0/3257909779 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f47740cbc00 con 0x7f477c102230 2023-09-10T01:00:00.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.161+0000 7f477b7fe700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f47640a0640 0x7f47640a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:00.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.162+0000 7f477b7fe700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f47640a0640 0x7f47640a2b00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f476c009990 tx=0x7f476c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:00.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.163+0000 7f4778ff9700 1 -- 172.21.15.3:0/3257909779 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f4774095bc0 con 0x7f477c102230 2023-09-10T01:00:00.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.373+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f477c10c060 con 0x7f477c102230 2023-09-10T01:00:00.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.374+0000 7f4778ff9700 1 -- 172.21.15.3:0/3257909779 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7f47740991e0 con 0x7f477c102230 2023-09-10T01:00:00.375 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f47640a0640 msgr2=0x7f47640a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f47640a0640 0x7f47640a2b00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f476c009990 tx=0x7f476c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c102230 msgr2=0x7f477c19fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:00.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c102230 0x7f477c19fb30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f477400ef60 tx=0x7f477400ef90 comp rx=0 tx=0).stop 2023-09-10T01:00:00.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 shutdown_connections 2023-09-10T01:00:00.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f477c102230 0x7f477c19fb30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f477c100690 0x7f477c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f47640a0640 0x7f47640a2b00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 --2- 172.21.15.3:0/3257909779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f477c101890 0x7f477c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:00.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 >> 172.21.15.3:0/3257909779 conn(0x7f477c0fbbf0 msgr2=0x7f477c1071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:00.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 shutdown_connections 2023-09-10T01:00:00.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:00.376+0000 7f4781cb0700 1 -- 172.21.15.3:0/3257909779 wait complete. 2023-09-10T01:00:00.901 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1694307597,"num_in_osds":12,"osd_in_since":1694307580,"num_remapped_pgs":0} 2023-09-10T01:00:00.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd dump --format=json 2023-09-10T01:00:01.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:00 smithi092 ceph-mon[134067]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:00 smithi092 ceph-mon[134067]: overall HEALTH_OK 2023-09-10T01:00:01.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:00 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3257909779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-09-10T01:00:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:00 smithi003 ceph-mon[97654]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:00 smithi003 ceph-mon[97654]: overall HEALTH_OK 2023-09-10T01:00:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:00 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3257909779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-09-10T01:00:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:00 smithi136 ceph-mon[133708]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:00 smithi136 ceph-mon[133708]: overall HEALTH_OK 2023-09-10T01:00:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:00 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3257909779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-09-10T01:00:01.642 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:00:03.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.144+0000 7f76f8d0f700 1 -- 172.21.15.3:0/2964063407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100fe0 msgr2=0x7f76f410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:03.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.144+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/2964063407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100fe0 0x7f76f410d4f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009f90 tx=0x7f76ec00b330 comp rx=0 tx=0).stop 2023-09-10T01:00:03.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.144+0000 7f76f8d0f700 1 -- 172.21.15.3:0/2964063407 shutdown_connections 2023-09-10T01:00:03.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.144+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/2964063407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76f4106620 0x7f76f4106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.144+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/2964063407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100fe0 0x7f76f410d4f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.144+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/2964063407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f4100620 0x7f76f4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.144+0000 7f76f8d0f700 1 -- 172.21.15.3:0/2964063407 >> 172.21.15.3:0/2964063407 conn(0x7f76f40fc0f0 msgr2=0x7f76f40fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:03.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.145+0000 7f76f8d0f700 1 -- 172.21.15.3:0/2964063407 shutdown_connections 2023-09-10T01:00:03.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.145+0000 7f76f8d0f700 1 -- 172.21.15.3:0/2964063407 wait complete. 2023-09-10T01:00:03.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 Processor -- start 2023-09-10T01:00:03.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 -- start start 2023-09-10T01:00:03.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100620 0x7f76f419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:03.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76f4100fe0 0x7f76f419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:03.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f4106620 0x7f76f4198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:03.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4198f00 con 0x7f76f4100620 2023-09-10T01:00:03.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4199070 con 0x7f76f4106620 2023-09-10T01:00:03.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f8d0f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f4199230 con 0x7f76f4100fe0 2023-09-10T01:00:03.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.146+0000 7f76f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100620 0x7f76f419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:03.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f1d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76f4100fe0 0x7f76f419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:03.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100620 0x7f76f419e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36444/0 (socket says 172.21.15.3:36444) 2023-09-10T01:00:03.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f259c700 1 -- 172.21.15.3:0/255788871 learned_addr learned my addr 172.21.15.3:0/255788871 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:03.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f2d9d700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f4106620 0x7f76f4198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:03.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f1d9b700 1 -- 172.21.15.3:0/255788871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f4106620 msgr2=0x7f76f4198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:03.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f1d9b700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f4106620 0x7f76f4198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f1d9b700 1 -- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100620 msgr2=0x7f76f419e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:03.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f1d9b700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100620 0x7f76f419e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f1d9b700 1 -- 172.21.15.3:0/255788871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec009c40 con 0x7f76f4100fe0 2023-09-10T01:00:03.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f259c700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100620 0x7f76f419e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:03.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.147+0000 7f76f1d9b700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76f4100fe0 0x7f76f419ed50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f76e800e9f0 tx=0x7f76e800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:03.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.148+0000 7f76e37fe700 1 -- 172.21.15.3:0/255788871 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e800c9c0 con 0x7f76f4100fe0 2023-09-10T01:00:03.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.148+0000 7f76e37fe700 1 -- 172.21.15.3:0/255788871 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76e8010430 con 0x7f76f4100fe0 2023-09-10T01:00:03.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.148+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f4199510 con 0x7f76f4100fe0 2023-09-10T01:00:03.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.148+0000 7f76e37fe700 1 -- 172.21.15.3:0/255788871 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8018680 con 0x7f76f4100fe0 2023-09-10T01:00:03.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.148+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f41a5800 con 0x7f76f4100fe0 2023-09-10T01:00:03.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.149+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f404e300 con 0x7f76f4100fe0 2023-09-10T01:00:03.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.151+0000 7f76e37fe700 1 -- 172.21.15.3:0/255788871 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f76e8010860 con 0x7f76f4100fe0 2023-09-10T01:00:03.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.152+0000 7f76e37fe700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76dc0a0640 0x7f76dc0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:03.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.152+0000 7f76e37fe700 1 -- 172.21.15.3:0/255788871 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f76e8014070 con 0x7f76f4100fe0 2023-09-10T01:00:03.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.152+0000 7f76f259c700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76dc0a0640 0x7f76dc0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:03.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.153+0000 7f76f259c700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76dc0a0640 0x7f76dc0a2b00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f76e4005950 tx=0x7f76e4009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:03.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.154+0000 7f76e37fe700 1 -- 172.21.15.3:0/255788871 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f76e8095a30 con 0x7f76f4100fe0 2023-09-10T01:00:03.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:02 smithi092 ceph-mon[134067]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:03.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:02 smithi003 ceph-mon[97654]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:03.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.362+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76f41106b0 con 0x7f76f4100fe0 2023-09-10T01:00:03.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.363+0000 7f76e37fe700 1 -- 172.21.15.3:0/255788871 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18311 (secure 0 0 0) 0x7f76e8099050 con 0x7f76f4100fe0 2023-09-10T01:00:03.365 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:03.365 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T00:59:58.707893+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:00:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76dc0a0640 msgr2=0x7f76dc0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76dc0a0640 0x7f76dc0a2b00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f76e4005950 tx=0x7f76e4009310 comp rx=0 tx=0).stop 2023-09-10T01:00:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76f4100fe0 msgr2=0x7f76f419ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76f4100fe0 0x7f76f419ed50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f76e800e9f0 tx=0x7f76e800ed00 comp rx=0 tx=0).stop 2023-09-10T01:00:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 shutdown_connections 2023-09-10T01:00:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f4106620 0x7f76f4198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76f4100620 0x7f76f419e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76dc0a0640 0x7f76dc0a2b00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 --2- 172.21.15.3:0/255788871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76f4100fe0 0x7f76f419ed50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 >> 172.21.15.3:0/255788871 conn(0x7f76f40fc0f0 msgr2=0x7f76f410b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 shutdown_connections 2023-09-10T01:00:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:03.366+0000 7f76f8d0f700 1 -- 172.21.15.3:0/255788871 wait complete. 2023-09-10T01:00:03.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:02 smithi136 ceph-mon[133708]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:03.825 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-09-10T01:00:03.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd pool get .mgr pg_num 2023-09-10T01:00:04.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/255788871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:04.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:03 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/255788871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:04.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:03 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/255788871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:04.563 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:00:05.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:04 smithi003 ceph-mon[97654]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:05.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:04 smithi092 ceph-mon[134067]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:04 smithi136 ceph-mon[133708]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 -- 172.21.15.3:0/691603748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 msgr2=0x7f6c7c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 --2- 172.21.15.3:0/691603748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 0x7f6c7c100aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009990 tx=0x7f6c70009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 -- 172.21.15.3:0/691603748 shutdown_connections 2023-09-10T01:00:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 --2- 172.21.15.3:0/691603748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100fe0 0x7f6c7c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 --2- 172.21.15.3:0/691603748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c106620 0x7f6c7c106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 --2- 172.21.15.3:0/691603748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 0x7f6c7c100aa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 -- 172.21.15.3:0/691603748 >> 172.21.15.3:0/691603748 conn(0x7f6c7c0fc0f0 msgr2=0x7f6c7c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:06.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.098+0000 7f6c8133a700 1 -- 172.21.15.3:0/691603748 shutdown_connections 2023-09-10T01:00:06.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.099+0000 7f6c8133a700 1 -- 172.21.15.3:0/691603748 wait complete. 2023-09-10T01:00:06.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.099+0000 7f6c8133a700 1 Processor -- start 2023-09-10T01:00:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c8133a700 1 -- start start 2023-09-10T01:00:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c8133a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 0x7f6c7c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c8133a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100fe0 0x7f6c7c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c8133a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c106620 0x7f6c7c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c8133a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c198f90 con 0x7f6c7c100620 2023-09-10T01:00:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c8133a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c199100 con 0x7f6c7c106620 2023-09-10T01:00:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c8133a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c1992c0 con 0x7f6c7c100fe0 2023-09-10T01:00:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 0x7f6c7c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:06.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100fe0 0x7f6c7c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 0x7f6c7c19e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56050/0 (socket says 172.21.15.3:56050) 2023-09-10T01:00:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.100+0000 7f6c7affd700 1 -- 172.21.15.3:0/130579806 learned_addr learned my addr 172.21.15.3:0/130579806 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:06.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c106620 0x7f6c7c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/130579806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c106620 msgr2=0x7f6c7c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c106620 0x7f6c7c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 msgr2=0x7f6c7c19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 0x7f6c7c19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/130579806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c009520 con 0x7f6c7c100fe0 2023-09-10T01:00:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100fe0 0x7f6c7c19ed30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6c70000c00 tx=0x7f6c70005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c63fff700 1 -- 172.21.15.3:0/130579806 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7001d070 con 0x7f6c7c100fe0 2023-09-10T01:00:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c700095f0 con 0x7f6c7c100fe0 2023-09-10T01:00:06.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c63fff700 1 -- 172.21.15.3:0/130579806 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c70021410 con 0x7f6c7c100fe0 2023-09-10T01:00:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.101+0000 7f6c63fff700 1 -- 172.21.15.3:0/130579806 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70017650 con 0x7f6c7c100fe0 2023-09-10T01:00:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.102+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c1a55a0 con 0x7f6c7c100fe0 2023-09-10T01:00:06.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.103+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c04e300 con 0x7f6c7c100fe0 2023-09-10T01:00:06.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.105+0000 7f6c63fff700 1 -- 172.21.15.3:0/130579806 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f6c700217f0 con 0x7f6c7c100fe0 2023-09-10T01:00:06.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.106+0000 7f6c63fff700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6c640a0640 0x7f6c640a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:06.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.106+0000 7f6c63fff700 1 -- 172.21.15.3:0/130579806 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6c700cd390 con 0x7f6c7c100fe0 2023-09-10T01:00:06.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.106+0000 7f6c7affd700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6c640a0640 0x7f6c640a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:06.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.107+0000 7f6c7affd700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6c640a0640 0x7f6c640a2b00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009e70 tx=0x7f6c6c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:06.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.108+0000 7f6c63fff700 1 -- 172.21.15.3:0/130579806 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6c70097350 con 0x7f6c7c100fe0 2023-09-10T01:00:06.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.317+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c7c1105b0 con 0x7f6c7c100fe0 2023-09-10T01:00:06.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.318+0000 7f6c63fff700 1 -- 172.21.15.3:0/130579806 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c7009a970 con 0x7f6c7c100fe0 2023-09-10T01:00:06.319 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:00:06.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6c640a0640 msgr2=0x7f6c640a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:06.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6c640a0640 0x7f6c640a2b00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009e70 tx=0x7f6c6c009260 comp rx=0 tx=0).stop 2023-09-10T01:00:06.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100fe0 msgr2=0x7f6c7c19ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:06.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100fe0 0x7f6c7c19ed30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6c70000c00 tx=0x7f6c70005740 comp rx=0 tx=0).stop 2023-09-10T01:00:06.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 shutdown_connections 2023-09-10T01:00:06.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c106620 0x7f6c7c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100620 0x7f6c7c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6c640a0640 0x7f6c640a2b00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 --2- 172.21.15.3:0/130579806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100fe0 0x7f6c7c19ed30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:06.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 >> 172.21.15.3:0/130579806 conn(0x7f6c7c0fc0f0 msgr2=0x7f6c7c10ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:06.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 shutdown_connections 2023-09-10T01:00:06.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:06.321+0000 7f6c8133a700 1 -- 172.21.15.3:0/130579806 wait complete. 2023-09-10T01:00:06.842 INFO:tasks.cephadm:Adding mds.a on smithi003 2023-09-10T01:00:06.842 INFO:tasks.cephadm:Adding mds.d on smithi003 2023-09-10T01:00:06.843 INFO:tasks.cephadm:Adding mds.g on smithi003 2023-09-10T01:00:06.843 INFO:tasks.cephadm:Adding mds.j on smithi003 2023-09-10T01:00:06.843 INFO:tasks.cephadm:Adding mds.b on smithi092 2023-09-10T01:00:06.843 INFO:tasks.cephadm:Adding mds.e on smithi092 2023-09-10T01:00:06.844 INFO:tasks.cephadm:Adding mds.h on smithi092 2023-09-10T01:00:06.844 INFO:tasks.cephadm:Adding mds.k on smithi092 2023-09-10T01:00:06.844 INFO:tasks.cephadm:Adding mds.c on smithi136 2023-09-10T01:00:06.845 INFO:tasks.cephadm:Adding mds.f on smithi136 2023-09-10T01:00:06.845 INFO:tasks.cephadm:Adding mds.i on smithi136 2023-09-10T01:00:06.845 INFO:tasks.cephadm:Adding mds.l on smithi136 2023-09-10T01:00:06.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph orch apply mds all '12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l' 2023-09-10T01:00:07.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:06 smithi003 ceph-mon[97654]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:07.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:06 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/130579806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:07.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:06 smithi136 ceph-mon[133708]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:07.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:06 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/130579806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:07.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:06 smithi092 ceph-mon[134067]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:07.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:06 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/130579806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:07.419 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T01:00:08.599 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/1123506474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 msgr2=0x7fb0f4103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:08.600 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/1123506474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4103570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009990 tx=0x7fb0e4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.600 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/1123506474 shutdown_connections 2023-09-10T01:00:08.600 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/1123506474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4103570 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.600 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/1123506474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0f41063e0 0x7fb0f41087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.601 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/1123506474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f4103ab0 0x7fb0f4105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.601 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/1123506474 >> 172.21.15.136:0/1123506474 conn(0x7fb0f40fab10 msgr2=0x7fb0f40fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:08.601 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/1123506474 shutdown_connections 2023-09-10T01:00:08.601 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.597+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/1123506474 wait complete. 2023-09-10T01:00:08.601 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 Processor -- start 2023-09-10T01:00:08.602 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 -- start start 2023-09-10T01:00:08.602 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4100ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:08.602 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0f4103ab0 0x7fb0f40ff130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:08.602 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f41063e0 0x7fb0f40ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:08.602 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40ffdf0 con 0x7fb0f41063e0 2023-09-10T01:00:08.603 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40fff60 con 0x7fb0f4103ab0 2023-09-10T01:00:08.603 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fc4a6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f41a0350 con 0x7fb0f4101180 2023-09-10T01:00:08.603 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4100ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:08.603 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4100ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39634/0 (socket says 172.21.15.136:39634) 2023-09-10T01:00:08.603 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 -- 172.21.15.136:0/4168042196 learned_addr learned my addr 172.21.15.136:0/4168042196 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-09-10T01:00:08.604 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0f9a41700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0f4103ab0 0x7fb0f40ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:08.604 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0faa43700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f41063e0 0x7fb0f40ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:08.604 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 -- 172.21.15.136:0/4168042196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0f4103ab0 msgr2=0x7fb0f40ff130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:08.604 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0f4103ab0 0x7fb0f40ff130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.604 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 -- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f41063e0 msgr2=0x7fb0f40ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:08.605 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f41063e0 0x7fb0f40ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.605 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.598+0000 7fb0fa242700 1 -- 172.21.15.136:0/4168042196 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e40095f0 con 0x7fb0f4101180 2023-09-10T01:00:08.605 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.599+0000 7fb0fa242700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4100ae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4005910 tx=0x7fb0e400f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:08.605 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.599+0000 7fb0eb7fe700 1 -- 172.21.15.136:0/4168042196 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e401d070 con 0x7fb0f4101180 2023-09-10T01:00:08.605 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.599+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f41a05d0 con 0x7fb0f4101180 2023-09-10T01:00:08.605 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.599+0000 7fb0eb7fe700 1 -- 172.21.15.136:0/4168042196 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0e400fc30 con 0x7fb0f4101180 2023-09-10T01:00:08.606 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.599+0000 7fb0eb7fe700 1 -- 172.21.15.136:0/4168042196 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4017650 con 0x7fb0f4101180 2023-09-10T01:00:08.606 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.599+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f41a0a40 con 0x7fb0f4101180 2023-09-10T01:00:08.607 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.600+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f40fc690 con 0x7fb0f4101180 2023-09-10T01:00:08.607 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.601+0000 7fb0eb7fe700 1 -- 172.21.15.136:0/4168042196 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb0e4022470 con 0x7fb0f4101180 2023-09-10T01:00:08.607 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.601+0000 7fb0eb7fe700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0e00a0640 0x7fb0e00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:08.608 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.601+0000 7fb0eb7fe700 1 -- 172.21.15.136:0/4168042196 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb0e40cd050 con 0x7fb0f4101180 2023-09-10T01:00:08.608 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.601+0000 7fb0f9a41700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0e00a0640 0x7fb0e00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:08.608 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.602+0000 7fb0f9a41700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0e00a0640 0x7fb0e00a2b00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0005ea0 tx=0x7fb0f0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:08.608 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.605+0000 7fb0eb7fe700 1 -- 172.21.15.136:0/4168042196 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb0e4097140 con 0x7fb0f4101180 2023-09-10T01:00:08.804 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.804+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0f402d000 con 0x7fb0e00a0640 2023-09-10T01:00:08.813 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.813+0000 7fb0eb7fe700 1 -- 172.21.15.136:0/4168042196 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb0f402d000 con 0x7fb0e00a0640 2023-09-10T01:00:08.813 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mds.all update... 2023-09-10T01:00:08.815 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0e00a0640 msgr2=0x7fb0e00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:08.816 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0e00a0640 0x7fb0e00a2b00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0005ea0 tx=0x7fb0f0005e30 comp rx=0 tx=0).stop 2023-09-10T01:00:08.816 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 msgr2=0x7fb0f4100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:08.816 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4100ae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4005910 tx=0x7fb0e400f740 comp rx=0 tx=0).stop 2023-09-10T01:00:08.816 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 shutdown_connections 2023-09-10T01:00:08.816 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0e00a0640 0x7fb0e00a2b00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.817 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0f4101180 0x7fb0f4100ae0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.817 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0f41063e0 0x7fb0f40ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.817 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 --2- 172.21.15.136:0/4168042196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0f4103ab0 0x7fb0f40ff130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:08.817 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 >> 172.21.15.136:0/4168042196 conn(0x7fb0f40fab10 msgr2=0x7fb0f4101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:08.817 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 shutdown_connections 2023-09-10T01:00:08.818 INFO:teuthology.orchestra.run.smithi136.stderr:2023-09-10T01:00:08.815+0000 7fb0fc4a6700 1 -- 172.21.15.136:0/4168042196 wait complete. 2023-09-10T01:00:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:09.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:09.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:09.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:09.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:09.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:09.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:08 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:09.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:09.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:09.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:08 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:09.267 DEBUG:teuthology.orchestra.run.smithi003:mds.a> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.a.service 2023-09-10T01:00:09.272 DEBUG:teuthology.orchestra.run.smithi003:mds.d> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.d.service 2023-09-10T01:00:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:08 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:09.312 DEBUG:teuthology.orchestra.run.smithi003:mds.g> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.g.service 2023-09-10T01:00:09.358 DEBUG:teuthology.orchestra.run.smithi003:mds.j> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.j.service 2023-09-10T01:00:09.359 INFO:journalctl@ceph.mds.a.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T01:00:09.387 INFO:journalctl@ceph.mds.d.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T01:00:09.391 DEBUG:teuthology.orchestra.run.smithi092:mds.b> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.b.service 2023-09-10T01:00:09.395 DEBUG:teuthology.orchestra.run.smithi092:mds.e> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.e.service 2023-09-10T01:00:09.431 INFO:journalctl@ceph.mds.g.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T01:00:09.434 INFO:journalctl@ceph.mds.b.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T01:00:09.435 DEBUG:teuthology.orchestra.run.smithi092:mds.h> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.h.service 2023-09-10T01:00:09.460 INFO:journalctl@ceph.mds.j.smithi003.stdout:-- Logs begin at Sun 2023-09-10 00:36:06 UTC. -- 2023-09-10T01:00:09.467 INFO:journalctl@ceph.mds.e.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T01:00:09.468 DEBUG:teuthology.orchestra.run.smithi092:mds.k> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.k.service 2023-09-10T01:00:09.500 INFO:journalctl@ceph.mds.h.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T01:00:09.501 DEBUG:teuthology.orchestra.run.smithi136:mds.c> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.c.service 2023-09-10T01:00:09.504 DEBUG:teuthology.orchestra.run.smithi136:mds.f> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.f.service 2023-09-10T01:00:09.534 INFO:journalctl@ceph.mds.k.smithi092.stdout:-- Logs begin at Sun 2023-09-10 00:32:32 UTC. -- 2023-09-10T01:00:09.535 INFO:journalctl@ceph.mds.c.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T01:00:09.535 DEBUG:teuthology.orchestra.run.smithi136:mds.i> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.i.service 2023-09-10T01:00:09.561 INFO:journalctl@ceph.mds.f.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T01:00:09.562 DEBUG:teuthology.orchestra.run.smithi136:mds.l> sudo journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.l.service 2023-09-10T01:00:09.587 INFO:journalctl@ceph.mds.i.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T01:00:09.588 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-09-10T01:00:09.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:00:09.615 INFO:journalctl@ceph.mds.l.smithi136.stdout:-- Logs begin at Sun 2023-09-10 00:32:20 UTC. -- 2023-09-10T01:00:09.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 -- 172.21.15.3:0/3244821952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901018c0 msgr2=0x7f7190101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:09.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 --2- 172.21.15.3:0/3244821952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901018c0 0x7f7190101d20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f717c009990 tx=0x7f717c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 -- 172.21.15.3:0/3244821952 shutdown_connections 2023-09-10T01:00:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 --2- 172.21.15.3:0/3244821952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7190102260 0x7f7190108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 --2- 172.21.15.3:0/3244821952 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71901006d0 0x7f7190100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 --2- 172.21.15.3:0/3244821952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901018c0 0x7f7190101d20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:09.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 -- 172.21.15.3:0/3244821952 >> 172.21.15.3:0/3244821952 conn(0x7f71900fbc60 msgr2=0x7f71900fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 -- 172.21.15.3:0/3244821952 shutdown_connections 2023-09-10T01:00:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.818+0000 7f7196dff700 1 -- 172.21.15.3:0/3244821952 wait complete. 2023-09-10T01:00:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 Processor -- start 2023-09-10T01:00:09.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 -- start start 2023-09-10T01:00:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901006d0 0x7f71900743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71901018c0 0x7f71900729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7190102260 0x7f7190072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019a240 con 0x7f7190102260 2023-09-10T01:00:09.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7190073470 con 0x7f71901018c0 2023-09-10T01:00:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7196dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7190073630 con 0x7f71901006d0 2023-09-10T01:00:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7194b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901006d0 0x7f71900743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7194b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901006d0 0x7f71900743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42798/0 (socket says 172.21.15.3:42798) 2023-09-10T01:00:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f7194b9b700 1 -- 172.21.15.3:0/943202265 learned_addr learned my addr 172.21.15.3:0/943202265 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f719539c700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7190102260 0x7f7190072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.820+0000 7f718ffff700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71901018c0 0x7f71900729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f719539c700 1 -- 172.21.15.3:0/943202265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901006d0 msgr2=0x7f71900743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f719539c700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901006d0 0x7f71900743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f719539c700 1 -- 172.21.15.3:0/943202265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71901018c0 msgr2=0x7f71900729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f719539c700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71901018c0 0x7f71900729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f719539c700 1 -- 172.21.15.3:0/943202265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c0095f0 con 0x7f7190102260 2023-09-10T01:00:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f7194b9b700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901006d0 0x7f71900743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f719539c700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7190102260 0x7f7190072f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f718800efc0 tx=0x7f718800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f718dffb700 1 -- 172.21.15.3:0/943202265 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188018070 con 0x7f7190102260 2023-09-10T01:00:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71901a4bf0 con 0x7f7190102260 2023-09-10T01:00:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f718dffb700 1 -- 172.21.15.3:0/943202265 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f718800f040 con 0x7f7190102260 2023-09-10T01:00:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.821+0000 7f718dffb700 1 -- 172.21.15.3:0/943202265 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718800cd40 con 0x7f7190102260 2023-09-10T01:00:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.822+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71901a5010 con 0x7f7190102260 2023-09-10T01:00:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.823+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7190107600 con 0x7f7190102260 2023-09-10T01:00:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.825+0000 7f718dffb700 1 -- 172.21.15.3:0/943202265 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f7188003a00 con 0x7f7190102260 2023-09-10T01:00:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.826+0000 7f718dffb700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f71800a0640 0x7f71800a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.826+0000 7f718dffb700 1 -- 172.21.15.3:0/943202265 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f71880cbc80 con 0x7f7190102260 2023-09-10T01:00:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.826+0000 7f7194b9b700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f71800a0640 0x7f71800a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.827+0000 7f7194b9b700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f71800a0640 0x7f71800a2b00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7184005950 tx=0x7f7184009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:09.829+0000 7f718dffb700 1 -- 172.21.15.3:0/943202265 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7188095c40 con 0x7f7190102260 2023-09-10T01:00:10.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.044+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f719010c0c0 con 0x7f7190102260 2023-09-10T01:00:10.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.045+0000 7f718dffb700 1 -- 172.21.15.3:0/943202265 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18311 (secure 0 0 0) 0x7f7188099260 con 0x7f7190102260 2023-09-10T01:00:10.047 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:10.047 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T00:59:58.707893+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:00:10.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f71800a0640 msgr2=0x7f71800a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f71800a0640 0x7f71800a2b00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7184005950 tx=0x7f7184009260 comp rx=0 tx=0).stop 2023-09-10T01:00:10.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7190102260 msgr2=0x7f7190072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7190102260 0x7f7190072f30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f718800efc0 tx=0x7f718800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 shutdown_connections 2023-09-10T01:00:10.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f71800a0640 0x7f71800a2b00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71901006d0 0x7f71900743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7190102260 0x7f7190072f30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 --2- 172.21.15.3:0/943202265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71901018c0 0x7f71900729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 >> 172.21.15.3:0/943202265 conn(0x7f71900fbc60 msgr2=0x7f71900fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:10.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 shutdown_connections 2023-09-10T01:00:10.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.048+0000 7f7196dff700 1 -- 172.21.15.3:0/943202265 wait complete. 2023-09-10T01:00:10.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-09-10T01:00:10.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:00:10.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:09 smithi136 ceph-mon[133708]: from='client.24538 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:00:10.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:09 smithi136 ceph-mon[133708]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2023-09-10T01:00:10.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:09 smithi136 ceph-mon[133708]: Deploying daemon mds.l on smithi136 2023-09-10T01:00:10.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:09 smithi092 ceph-mon[134067]: from='client.24538 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:00:10.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:09 smithi092 ceph-mon[134067]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2023-09-10T01:00:10.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:09 smithi092 ceph-mon[134067]: Deploying daemon mds.l on smithi136 2023-09-10T01:00:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 -- 172.21.15.3:0/2049830613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81881006e0 msgr2=0x7f8188100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 --2- 172.21.15.3:0/2049830613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81881006e0 0x7f8188100b00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f81780099c0 tx=0x7f8178009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 -- 172.21.15.3:0/2049830613 shutdown_connections 2023-09-10T01:00:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 --2- 172.21.15.3:0/2049830613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8188102270 0x7f8188108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 --2- 172.21.15.3:0/2049830613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81881006e0 0x7f8188100b00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 --2- 172.21.15.3:0/2049830613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81881018d0 0x7f8188101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 -- 172.21.15.3:0/2049830613 >> 172.21.15.3:0/2049830613 conn(0x7f81880fbc50 msgr2=0x7f81880fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:10.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 -- 172.21.15.3:0/2049830613 shutdown_connections 2023-09-10T01:00:10.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.277+0000 7f818d1ff700 1 -- 172.21.15.3:0/2049830613 wait complete. 2023-09-10T01:00:10.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.278+0000 7f818d1ff700 1 Processor -- start 2023-09-10T01:00:10.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.278+0000 7f818d1ff700 1 -- start start 2023-09-10T01:00:10.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818d1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81881006e0 0x7f8188196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818d1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81881018d0 0x7f81881968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818d1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8188102270 0x7f818819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818d1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818819df00 con 0x7f8188102270 2023-09-10T01:00:10.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818d1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818819e040 con 0x7f81881018d0 2023-09-10T01:00:10.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818d1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818819e1d0 con 0x7f81881006e0 2023-09-10T01:00:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81881006e0 0x7f8188196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81881006e0 0x7f8188196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42826/0 (socket says 172.21.15.3:42826) 2023-09-10T01:00:10.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 -- 172.21.15.3:0/531449666 learned_addr learned my addr 172.21.15.3:0/531449666 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:10.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818659c700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81881018d0 0x7f81881968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f818759e700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8188102270 0x7f818819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 -- 172.21.15.3:0/531449666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81881018d0 msgr2=0x7f81881968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81881018d0 0x7f81881968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 -- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8188102270 msgr2=0x7f818819d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8188102270 0x7f818819d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.279+0000 7f8186d9d700 1 -- 172.21.15.3:0/531449666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81780095f0 con 0x7f81881006e0 2023-09-10T01:00:10.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.280+0000 7f8186d9d700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81881006e0 0x7f8188196380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8178009990 tx=0x7f8178004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:10.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.280+0000 7f816ffff700 1 -- 172.21.15.3:0/531449666 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817801d070 con 0x7f81881006e0 2023-09-10T01:00:10.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.280+0000 7f816ffff700 1 -- 172.21.15.3:0/531449666 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8178004010 con 0x7f81881006e0 2023-09-10T01:00:10.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.280+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818819e450 con 0x7f81881006e0 2023-09-10T01:00:10.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.280+0000 7f816ffff700 1 -- 172.21.15.3:0/531449666 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178022cd0 con 0x7f81881006e0 2023-09-10T01:00:10.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.280+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818819e950 con 0x7f81881006e0 2023-09-10T01:00:10.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.281+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8188107cc0 con 0x7f81881006e0 2023-09-10T01:00:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.283+0000 7f816ffff700 1 -- 172.21.15.3:0/531449666 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f8178022470 con 0x7f81881006e0 2023-09-10T01:00:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.284+0000 7f816ffff700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81700a06c0 0x7f81700a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.285+0000 7f816ffff700 1 -- 172.21.15.3:0/531449666 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f81780cce60 con 0x7f81881006e0 2023-09-10T01:00:10.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.285+0000 7f818659c700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81700a06c0 0x7f81700a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.285+0000 7f818659c700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81700a06c0 0x7f81700a2b80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f817c007c60 tx=0x7f817c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:10.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.286+0000 7f816ffff700 1 -- 172.21.15.3:0/531449666 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f8178096e20 con 0x7f81881006e0 2023-09-10T01:00:10.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:09 smithi003 ceph-mon[97654]: from='client.24538 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:00:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:09 smithi003 ceph-mon[97654]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2023-09-10T01:00:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:09 smithi003 ceph-mon[97654]: Deploying daemon mds.l on smithi136 2023-09-10T01:00:10.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.498+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f818810c080 con 0x7f81881006e0 2023-09-10T01:00:10.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.498+0000 7f816ffff700 1 -- 172.21.15.3:0/531449666 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f817809a440 con 0x7f81881006e0 2023-09-10T01:00:10.500 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:00:10.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81700a06c0 msgr2=0x7f81700a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81700a06c0 0x7f81700a2b80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f817c007c60 tx=0x7f817c0073d0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81881006e0 msgr2=0x7f8188196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81881006e0 0x7f8188196380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8178009990 tx=0x7f8178004730 comp rx=0 tx=0).stop 2023-09-10T01:00:10.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 shutdown_connections 2023-09-10T01:00:10.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81700a06c0 0x7f81700a2b80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81881006e0 0x7f8188196380 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8188102270 0x7f818819d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 --2- 172.21.15.3:0/531449666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81881018d0 0x7f81881968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 >> 172.21.15.3:0/531449666 conn(0x7f81880fbc50 msgr2=0x7f8188104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:10.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 shutdown_connections 2023-09-10T01:00:10.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.501+0000 7f818d1ff700 1 -- 172.21.15.3:0/531449666 wait complete. 2023-09-10T01:00:10.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:10.662 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:00:10 smithi136 systemd[1]: Starting Ceph mds.l for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:10.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.724+0000 7f0e85dff700 1 -- 172.21.15.3:0/2452348576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80074640 msgr2=0x7f0e80072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.724+0000 7f0e85dff700 1 --2- 172.21.15.3:0/2452348576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80074640 0x7f0e80072a90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0e700098f0 tx=0x7f0e70009c00 comp rx=0 tx=0).stop 2023-09-10T01:00:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.725+0000 7f0e85dff700 1 -- 172.21.15.3:0/2452348576 shutdown_connections 2023-09-10T01:00:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.725+0000 7f0e85dff700 1 --2- 172.21.15.3:0/2452348576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80108580 0x7f0e8010aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.725+0000 7f0e85dff700 1 --2- 172.21.15.3:0/2452348576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80074640 0x7f0e80072a90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.725+0000 7f0e85dff700 1 --2- 172.21.15.3:0/2452348576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e80073060 0x7f0e800734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.725+0000 7f0e85dff700 1 -- 172.21.15.3:0/2452348576 >> 172.21.15.3:0/2452348576 conn(0x7f0e800fbc30 msgr2=0x7f0e800fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.725+0000 7f0e85dff700 1 -- 172.21.15.3:0/2452348576 shutdown_connections 2023-09-10T01:00:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.725+0000 7f0e85dff700 1 -- 172.21.15.3:0/2452348576 wait complete. 2023-09-10T01:00:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 Processor -- start 2023-09-10T01:00:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 -- start start 2023-09-10T01:00:10.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80073060 0x7f0e801985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e80074640 0x7f0e80198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80108580 0x7f0e8019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e801a0150 con 0x7f0e80108580 2023-09-10T01:00:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e801a0290 con 0x7f0e80073060 2023-09-10T01:00:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e85dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e801a0420 con 0x7f0e80074640 2023-09-10T01:00:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e7f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80073060 0x7f0e801985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e7f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80073060 0x7f0e801985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48280/0 (socket says 172.21.15.3:48280) 2023-09-10T01:00:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.726+0000 7f0e7f7fe700 1 -- 172.21.15.3:0/3417403230 learned_addr learned my addr 172.21.15.3:0/3417403230 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7effd700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e80074640 0x7f0e80198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7ffff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80108580 0x7f0e8019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7f7fe700 1 -- 172.21.15.3:0/3417403230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e80074640 msgr2=0x7f0e80198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7f7fe700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e80074640 0x7f0e80198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7f7fe700 1 -- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80108580 msgr2=0x7f0e8019fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7f7fe700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80108580 0x7f0e8019fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7f7fe700 1 -- 172.21.15.3:0/3417403230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e740095f0 con 0x7f0e80073060 2023-09-10T01:00:10.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7ffff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80108580 0x7f0e8019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:10.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7f7fe700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80073060 0x7f0e801985d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0e70011040 tx=0x7f0e7000b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:10.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7cff9700 1 -- 172.21.15.3:0/3417403230 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7001c070 con 0x7f0e80073060 2023-09-10T01:00:10.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7cff9700 1 -- 172.21.15.3:0/3417403230 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e7000f7d0 con 0x7f0e80073060 2023-09-10T01:00:10.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e7cff9700 1 -- 172.21.15.3:0/3417403230 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70004a90 con 0x7f0e80073060 2023-09-10T01:00:10.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.727+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e70009520 con 0x7f0e80073060 2023-09-10T01:00:10.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.728+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e801a08e0 con 0x7f0e80073060 2023-09-10T01:00:10.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.728+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e80106c30 con 0x7f0e80073060 2023-09-10T01:00:10.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.731+0000 7f0e7cff9700 1 -- 172.21.15.3:0/3417403230 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f0e70020070 con 0x7f0e80073060 2023-09-10T01:00:10.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.732+0000 7f0e7cff9700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0e680a0640 0x7f0e680a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:10.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.732+0000 7f0e7cff9700 1 -- 172.21.15.3:0/3417403230 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0e700ccc70 con 0x7f0e80073060 2023-09-10T01:00:10.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.732+0000 7f0e7effd700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0e680a0640 0x7f0e680a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:10.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.733+0000 7f0e7effd700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0e680a0640 0x7f0e680a2b00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e80074360 tx=0x7f0e74009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:10.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.734+0000 7f0e7cff9700 1 -- 172.21.15.3:0/3417403230 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0e70096c30 con 0x7f0e80073060 2023-09-10T01:00:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:10 smithi136 ceph-mon[133708]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:10.946 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:00:10 smithi136 podman[170472]: 2023-09-10T01:00:10.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.992+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e8010e0b0 con 0x7f0e80073060 2023-09-10T01:00:10.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.993+0000 7f0e7cff9700 1 -- 172.21.15.3:0/3417403230 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f0e7009a250 con 0x7f0e80073060 2023-09-10T01:00:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:10.994 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-09-10T01:00:10.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0e680a0640 msgr2=0x7f0e680a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0e680a0640 0x7f0e680a2b00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e80074360 tx=0x7f0e74009310 comp rx=0 tx=0).stop 2023-09-10T01:00:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80073060 msgr2=0x7f0e801985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80073060 0x7f0e801985d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0e70011040 tx=0x7f0e7000b790 comp rx=0 tx=0).stop 2023-09-10T01:00:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 shutdown_connections 2023-09-10T01:00:10.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80108580 0x7f0e8019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e80073060 0x7f0e801985d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0e680a0640 0x7f0e680a2b00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 --2- 172.21.15.3:0/3417403230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e80074640 0x7f0e80198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.995+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 >> 172.21.15.3:0/3417403230 conn(0x7f0e800fbc30 msgr2=0x7f0e800fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:10.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.996+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 shutdown_connections 2023-09-10T01:00:10.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:10.996+0000 7f0e85dff700 1 -- 172.21.15.3:0/3417403230 wait complete. 2023-09-10T01:00:10.999 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 2 2023-09-10T01:00:11.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:10 smithi092 ceph-mon[134067]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:10 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/943202265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:10 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/531449666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:10 smithi003 ceph-mon[97654]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:10 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/943202265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:10 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/531449666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:11.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:10 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/943202265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:11.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:10 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/531449666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:11.412 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:00:10 smithi136 ceph-mds[170512]: -- 172.21.15.136:0/586197744 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a25c4331e0 con 0x55a25c408800 2023-09-10T01:00:11.412 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:00:10 smithi136 bash[170472]: 3e6969447a8f2f175d40da61001306a84ea05ade5be1a2039711efe782691cd4 2023-09-10T01:00:11.413 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:00:10 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-l[170488]: starting mds.l at 2023-09-10T01:00:11.413 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:00:10 smithi136 systemd[1]: Started Ceph mds.l for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:12.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6832/3536147716,v1:172.21.15.136:6833/3536147716] up:boot 2023-09-10T01:00:12.302 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: fsmap 1 up:standby 2023-09-10T01:00:12.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-09-10T01:00:12.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3417403230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:12.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.303 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.304 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:12.304 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:12.304 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:11 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6832/3536147716,v1:172.21.15.136:6833/3536147716] up:boot 2023-09-10T01:00:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: fsmap 1 up:standby 2023-09-10T01:00:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-09-10T01:00:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3417403230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:12.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:12.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:11 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6832/3536147716,v1:172.21.15.136:6833/3536147716] up:boot 2023-09-10T01:00:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: fsmap 1 up:standby 2023-09-10T01:00:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-09-10T01:00:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3417403230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:12.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:12.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:12.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:11 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:12.932 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:00:12 smithi092 systemd[1]: Starting Ceph mds.k for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:13.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:13.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:12 smithi092 ceph-mon[134067]: Deploying daemon mds.k on smithi092 2023-09-10T01:00:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:12 smithi092 ceph-mon[134067]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:13.216 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:00:12 smithi092 podman[170416]: 2023-09-10T01:00:13.216 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:00:13 smithi092 ceph-mds[170456]: -- 172.21.15.92:0/695300724 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558be97011e0 con 0x558be96d6800 2023-09-10T01:00:13.216 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:00:13 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-k[170432]: starting mds.k at 2023-09-10T01:00:13.217 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:00:13 smithi092 bash[170416]: f3c15383c9c447be84eea593fdc70bd4ff29209921db224b962d014ebe0220ec 2023-09-10T01:00:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 -- 172.21.15.3:0/2383726287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c108580 msgr2=0x7fda3c10aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:13.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 --2- 172.21.15.3:0/2383726287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c108580 0x7fda3c10aa10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fda3400b0a0 tx=0x7fda3400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 -- 172.21.15.3:0/2383726287 shutdown_connections 2023-09-10T01:00:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 --2- 172.21.15.3:0/2383726287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda3c074640 0x7fda3c072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 --2- 172.21.15.3:0/2383726287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c108580 0x7fda3c10aa10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 --2- 172.21.15.3:0/2383726287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c073060 0x7fda3c0734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 -- 172.21.15.3:0/2383726287 >> 172.21.15.3:0/2383726287 conn(0x7fda3c0fbc30 msgr2=0x7fda3c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 -- 172.21.15.3:0/2383726287 shutdown_connections 2023-09-10T01:00:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.229+0000 7fda41dff700 1 -- 172.21.15.3:0/2383726287 wait complete. 2023-09-10T01:00:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.230+0000 7fda41dff700 1 Processor -- start 2023-09-10T01:00:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.230+0000 7fda41dff700 1 -- start start 2023-09-10T01:00:13.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda41dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c073060 0x7fda3c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda41dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda3c074640 0x7fda3c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda41dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c108580 0x7fda3c19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda41dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a02b0 con 0x7fda3c073060 2023-09-10T01:00:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda41dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a03f0 con 0x7fda3c108580 2023-09-10T01:00:13.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda41dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a0580 con 0x7fda3c074640 2023-09-10T01:00:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c073060 0x7fda3c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c108580 0x7fda3c19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda3c074640 0x7fda3c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c073060 0x7fda3c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56152/0 (socket says 172.21.15.3:56152) 2023-09-10T01:00:13.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 -- 172.21.15.3:0/1673588587 learned_addr learned my addr 172.21.15.3:0/1673588587 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c108580 0x7fda3c19fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48298/0 (socket says 172.21.15.3:48298) 2023-09-10T01:00:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda3c074640 0x7fda3c198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42844/0 (socket says 172.21.15.3:42844) 2023-09-10T01:00:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 -- 172.21.15.3:0/1673588587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda3c074640 msgr2=0x7fda3c198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda3c074640 0x7fda3c198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 -- 172.21.15.3:0/1673588587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c108580 msgr2=0x7fda3c19fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c108580 0x7fda3c19fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3b7fe700 1 -- 172.21.15.3:0/1673588587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30009520 con 0x7fda3c073060 2023-09-10T01:00:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.231+0000 7fda3bfff700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c108580 0x7fda3c19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.232+0000 7fda3b7fe700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c073060 0x7fda3c198730 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00e9a0 tx=0x7fda2c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:13.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.232+0000 7fda38ff9700 1 -- 172.21.15.3:0/1673588587 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c009790 con 0x7fda3c073060 2023-09-10T01:00:13.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.232+0000 7fda38ff9700 1 -- 172.21.15.3:0/1673588587 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda2c010430 con 0x7fda3c073060 2023-09-10T01:00:13.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.232+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda34009d00 con 0x7fda3c073060 2023-09-10T01:00:13.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.232+0000 7fda38ff9700 1 -- 172.21.15.3:0/1673588587 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c018680 con 0x7fda3c073060 2023-09-10T01:00:13.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.232+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c1a0be0 con 0x7fda3c073060 2023-09-10T01:00:13.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.233+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c106c30 con 0x7fda3c073060 2023-09-10T01:00:13.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.235+0000 7fda38ff9700 1 -- 172.21.15.3:0/1673588587 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fda2c0105a0 con 0x7fda3c073060 2023-09-10T01:00:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.236+0000 7fda38ff9700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fda240a0640 0x7fda240a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.237+0000 7fda38ff9700 1 -- 172.21.15.3:0/1673588587 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda2c014070 con 0x7fda3c073060 2023-09-10T01:00:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.237+0000 7fda3affd700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fda240a0640 0x7fda240a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:13.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.237+0000 7fda3affd700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fda240a0640 0x7fda240a2b00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda3c074360 tx=0x7fda30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:13.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.240+0000 7fda38ff9700 1 -- 172.21.15.3:0/1673588587 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fda2c051e10 con 0x7fda3c073060 2023-09-10T01:00:13.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:12 smithi003 ceph-mon[97654]: Deploying daemon mds.k on smithi092 2023-09-10T01:00:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:12 smithi003 ceph-mon[97654]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:12 smithi136 ceph-mon[133708]: Deploying daemon mds.k on smithi092 2023-09-10T01:00:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:12 smithi136 ceph-mon[133708]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:13.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.502+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda3c10e0b0 con 0x7fda3c073060 2023-09-10T01:00:13.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.502+0000 7fda38ff9700 1 -- 172.21.15.3:0/1673588587 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fda2c0957d0 con 0x7fda3c073060 2023-09-10T01:00:13.503 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:13.504 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-09-10T01:00:13.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fda240a0640 msgr2=0x7fda240a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:13.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fda240a0640 0x7fda240a2b00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda3c074360 tx=0x7fda30009260 comp rx=0 tx=0).stop 2023-09-10T01:00:13.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c073060 msgr2=0x7fda3c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:13.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c073060 0x7fda3c198730 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00e9a0 tx=0x7fda2c00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 shutdown_connections 2023-09-10T01:00:13.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda3c108580 0x7fda3c19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda3c073060 0x7fda3c198730 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fda240a0640 0x7fda240a2b00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 --2- 172.21.15.3:0/1673588587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda3c074640 0x7fda3c198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.505+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 >> 172.21.15.3:0/1673588587 conn(0x7fda3c0fbc30 msgr2=0x7fda3c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.506+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 shutdown_connections 2023-09-10T01:00:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:13.506+0000 7fda41dff700 1 -- 172.21.15.3:0/1673588587 wait complete. 2023-09-10T01:00:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 2 2023-09-10T01:00:13.716 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:00:13 smithi092 systemd[1]: Started Ceph mds.k for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:14.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:14.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:14.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:14.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: Deploying daemon mds.j on smithi003 2023-09-10T01:00:14.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1673588587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:14.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:14.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6832/1626913605,v1:172.21.15.92:6833/1626913605] up:boot 2023-09-10T01:00:14.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: fsmap 2 up:standby 2023-09-10T01:00:14.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:14 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-09-10T01:00:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: Deploying daemon mds.j on smithi003 2023-09-10T01:00:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1673588587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:14.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:14.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6832/1626913605,v1:172.21.15.92:6833/1626913605] up:boot 2023-09-10T01:00:14.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: fsmap 2 up:standby 2023-09-10T01:00:14.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:14 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-09-10T01:00:14.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: Deploying daemon mds.j on smithi003 2023-09-10T01:00:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1673588587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6832/1626913605,v1:172.21.15.92:6833/1626913605] up:boot 2023-09-10T01:00:14.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: fsmap 2 up:standby 2023-09-10T01:00:14.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:14 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-09-10T01:00:15.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.826+0000 7f647abff700 1 -- 172.21.15.3:0/2680490807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 msgr2=0x7f6474101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:15.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.826+0000 7f647abff700 1 --2- 172.21.15.3:0/2680490807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f6474101d20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f64600099c0 tx=0x7f6460009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:15.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.826+0000 7f647abff700 1 -- 172.21.15.3:0/2680490807 shutdown_connections 2023-09-10T01:00:15.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.827+0000 7f647abff700 1 --2- 172.21.15.3:0/2680490807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64741006d0 0x7f6474100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:15.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.827+0000 7f647abff700 1 --2- 172.21.15.3:0/2680490807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6474102260 0x7f6474108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:15.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.827+0000 7f647abff700 1 --2- 172.21.15.3:0/2680490807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f6474101d20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:15.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.827+0000 7f647abff700 1 -- 172.21.15.3:0/2680490807 >> 172.21.15.3:0/2680490807 conn(0x7f64740fbc60 msgr2=0x7f64740fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:15.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.827+0000 7f647abff700 1 -- 172.21.15.3:0/2680490807 shutdown_connections 2023-09-10T01:00:15.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.827+0000 7f647abff700 1 -- 172.21.15.3:0/2680490807 wait complete. 2023-09-10T01:00:15.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 Processor -- start 2023-09-10T01:00:15.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 -- start start 2023-09-10T01:00:15.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64741006d0 0x7f64740743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:15.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f64740729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:15.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474102260 0x7f6474072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:15.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647419a2a0 con 0x7f64741018c0 2023-09-10T01:00:15.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474073470 con 0x7f64741006d0 2023-09-10T01:00:15.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.828+0000 7f647abff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474073630 con 0x7f6474102260 2023-09-10T01:00:15.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.829+0000 7f6473fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f64740729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:15.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.829+0000 7f6473fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f64740729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60664/0 (socket says 172.21.15.3:60664) 2023-09-10T01:00:15.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.829+0000 7f6473fff700 1 -- 172.21.15.3:0/349175496 learned_addr learned my addr 172.21.15.3:0/349175496 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:15.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.829+0000 7f647899b700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64741006d0 0x7f64740743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:15.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.829+0000 7f647919c700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474102260 0x7f6474072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:15.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6473fff700 1 -- 172.21.15.3:0/349175496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474102260 msgr2=0x7f6474072f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:15.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6473fff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474102260 0x7f6474072f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:15.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6473fff700 1 -- 172.21.15.3:0/349175496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64741006d0 msgr2=0x7f64740743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:15.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6473fff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64741006d0 0x7f64740743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:15.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6473fff700 1 -- 172.21.15.3:0/349175496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c009c40 con 0x7f64741018c0 2023-09-10T01:00:15.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6473fff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f64740729f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6460009fd0 tx=0x7f646000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:15.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6471ffb700 1 -- 172.21.15.3:0/349175496 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646001d070 con 0x7f64741018c0 2023-09-10T01:00:15.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64600095f0 con 0x7f64741018c0 2023-09-10T01:00:15.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6471ffb700 1 -- 172.21.15.3:0/349175496 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6460005510 con 0x7f64741018c0 2023-09-10T01:00:15.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.830+0000 7f6471ffb700 1 -- 172.21.15.3:0/349175496 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460004ae0 con 0x7f64741018c0 2023-09-10T01:00:15.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.831+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64741a4e30 con 0x7f64741018c0 2023-09-10T01:00:15.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.832+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647410bf40 con 0x7f64741018c0 2023-09-10T01:00:15.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.836+0000 7f6471ffb700 1 -- 172.21.15.3:0/349175496 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f6460004cc0 con 0x7f64741018c0 2023-09-10T01:00:15.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.838+0000 7f6471ffb700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f64640a0640 0x7f64640a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:15.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.838+0000 7f6471ffb700 1 -- 172.21.15.3:0/349175496 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64600cc610 con 0x7f64741018c0 2023-09-10T01:00:15.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.839+0000 7f647899b700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f64640a0640 0x7f64640a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:15.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.839+0000 7f647899b700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f64640a0640 0x7f64640a2b00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6474101720 tx=0x7f6468009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:15.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:15.843+0000 7f6471ffb700 1 -- 172.21.15.3:0/349175496 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f64600965d0 con 0x7f64741018c0 2023-09-10T01:00:16.059 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:00:15 smithi003 systemd[1]: Starting Ceph mds.j for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:16.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.152+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6474107620 con 0x7f64741018c0 2023-09-10T01:00:16.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.152+0000 7f6471ffb700 1 -- 172.21.15.3:0/349175496 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7f6460099bf0 con 0x7f64741018c0 2023-09-10T01:00:16.153 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:16.154 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-09-10T01:00:16.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f64640a0640 msgr2=0x7f64640a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:16.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f64640a0640 0x7f64640a2b00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6474101720 tx=0x7f6468009260 comp rx=0 tx=0).stop 2023-09-10T01:00:16.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 msgr2=0x7f64740729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:16.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f64740729f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6460009fd0 tx=0x7f646000f690 comp rx=0 tx=0).stop 2023-09-10T01:00:16.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 shutdown_connections 2023-09-10T01:00:16.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f64640a0640 0x7f64640a2b00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:16.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474102260 0x7f6474072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:16.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64741006d0 0x7f64740743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:16.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 --2- 172.21.15.3:0/349175496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64741018c0 0x7f64740729f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:16.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 >> 172.21.15.3:0/349175496 conn(0x7f64740fbc60 msgr2=0x7f64740fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:16.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 shutdown_connections 2023-09-10T01:00:16.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:16.155+0000 7f647abff700 1 -- 172.21.15.3:0/349175496 wait complete. 2023-09-10T01:00:16.158 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 3 2023-09-10T01:00:16.469 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:00:16 smithi003 podman[151908]: 2023-09-10T01:00:16.809 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:00:16 smithi003 bash[151908]: 83347b1b1a0ca5b34935d59ad3c909c244bf5469d0d45e91289967a974475837 2023-09-10T01:00:16.809 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mds[151950]: -- 172.21.15.3:0/1792394193 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5567c42fb1e0 con 0x5567c42d0c00 2023-09-10T01:00:16.809 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-j[151926]: starting mds.j at 2023-09-10T01:00:16.809 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:00:16 smithi003 systemd[1]: Started Ceph mds.j for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:17.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/349175496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:17.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:17.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:16 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/349175496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:17.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:17.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:16 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/349175496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:17.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:17.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:17.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:16 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:18.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:17 smithi003 ceph-mon[97654]: Deploying daemon mds.d on smithi003 2023-09-10T01:00:18.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:17 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.3:6834/2642590493,v1:172.21.15.3:6835/2642590493] up:boot 2023-09-10T01:00:18.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:17 smithi003 ceph-mon[97654]: fsmap 3 up:standby 2023-09-10T01:00:18.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:17 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-09-10T01:00:18.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:18.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:17 smithi092 ceph-mon[134067]: Deploying daemon mds.d on smithi003 2023-09-10T01:00:18.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:17 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.3:6834/2642590493,v1:172.21.15.3:6835/2642590493] up:boot 2023-09-10T01:00:18.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:17 smithi092 ceph-mon[134067]: fsmap 3 up:standby 2023-09-10T01:00:18.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:17 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-09-10T01:00:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:17 smithi136 ceph-mon[133708]: Deploying daemon mds.d on smithi003 2023-09-10T01:00:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:17 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.3:6834/2642590493,v1:172.21.15.3:6835/2642590493] up:boot 2023-09-10T01:00:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:17 smithi136 ceph-mon[133708]: fsmap 3 up:standby 2023-09-10T01:00:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:17 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-09-10T01:00:18.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.571+0000 7fad85dff700 1 -- 172.21.15.3:0/2197725456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 msgr2=0x7fad80108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:18.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.571+0000 7fad85dff700 1 --2- 172.21.15.3:0/2197725456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad80108b20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fad700099c0 tx=0x7fad70009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.571+0000 7fad85dff700 1 -- 172.21.15.3:0/2197725456 shutdown_connections 2023-09-10T01:00:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.571+0000 7fad85dff700 1 --2- 172.21.15.3:0/2197725456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801006e0 0x7fad80100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.572+0000 7fad85dff700 1 --2- 172.21.15.3:0/2197725456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad80108b20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.572+0000 7fad85dff700 1 --2- 172.21.15.3:0/2197725456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad801018d0 0x7fad80101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.572+0000 7fad85dff700 1 -- 172.21.15.3:0/2197725456 >> 172.21.15.3:0/2197725456 conn(0x7fad800fbc50 msgr2=0x7fad800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.573+0000 7fad85dff700 1 -- 172.21.15.3:0/2197725456 shutdown_connections 2023-09-10T01:00:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.573+0000 7fad85dff700 1 -- 172.21.15.3:0/2197725456 wait complete. 2023-09-10T01:00:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.574+0000 7fad85dff700 1 Processor -- start 2023-09-10T01:00:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.575+0000 7fad85dff700 1 -- start start 2023-09-10T01:00:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.575+0000 7fad85dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad801006e0 0x7fad8010e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.575+0000 7fad85dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801018d0 0x7fad8010e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.575+0000 7fad85dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad801acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.575+0000 7fad85dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad80111a70 con 0x7fad80102270 2023-09-10T01:00:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.575+0000 7fad85dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad80111bb0 con 0x7fad801006e0 2023-09-10T01:00:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.575+0000 7fad85dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad80111d40 con 0x7fad801018d0 2023-09-10T01:00:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.577+0000 7fad7ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad801acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.577+0000 7fad7ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad801acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60708/0 (socket says 172.21.15.3:60708) 2023-09-10T01:00:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.577+0000 7fad7ffff700 1 -- 172.21.15.3:0/1548449739 learned_addr learned my addr 172.21.15.3:0/1548449739 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.577+0000 7fad7f7fe700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad801006e0 0x7fad8010e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7ffff700 1 -- 172.21.15.3:0/1548449739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801018d0 msgr2=0x7fad8010e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7effd700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801018d0 0x7fad8010e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7ffff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801018d0 0x7fad8010e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7ffff700 1 -- 172.21.15.3:0/1548449739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad801006e0 msgr2=0x7fad8010e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7ffff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad801006e0 0x7fad8010e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7ffff700 1 -- 172.21.15.3:0/1548449739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700095f0 con 0x7fad80102270 2023-09-10T01:00:18.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7f7fe700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad801006e0 0x7fad8010e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:18.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7effd700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801018d0 0x7fad8010e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:00:18.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7ffff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad801acc60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fad70010d80 tx=0x7fad70005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:18.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7cff9700 1 -- 172.21.15.3:0/1548449739 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7000b990 con 0x7fad80102270 2023-09-10T01:00:18.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad7cff9700 1 -- 172.21.15.3:0/1548449739 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad70004500 con 0x7fad80102270 2023-09-10T01:00:18.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.578+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8006a110 con 0x7fad80102270 2023-09-10T01:00:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.579+0000 7fad7cff9700 1 -- 172.21.15.3:0/1548449739 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70018690 con 0x7fad80102270 2023-09-10T01:00:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.579+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8006a610 con 0x7fad80102270 2023-09-10T01:00:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.580+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad801125e0 con 0x7fad80102270 2023-09-10T01:00:18.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.584+0000 7fad7cff9700 1 -- 172.21.15.3:0/1548449739 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fad70004670 con 0x7fad80102270 2023-09-10T01:00:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.586+0000 7fad7cff9700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fad680a0710 0x7fad680a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.586+0000 7fad7cff9700 1 -- 172.21.15.3:0/1548449739 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fad700cdcc0 con 0x7fad80102270 2023-09-10T01:00:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.586+0000 7fad7f7fe700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fad680a0710 0x7fad680a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:18.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.587+0000 7fad7f7fe700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fad680a0710 0x7fad680a2bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fad80101730 tx=0x7fad74008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:18.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.588+0000 7fad7cff9700 1 -- 172.21.15.3:0/1548449739 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fad70097d20 con 0x7fad80102270 2023-09-10T01:00:18.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.871+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad8006a8f0 con 0x7fad80102270 2023-09-10T01:00:18.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.872+0000 7fad7cff9700 1 -- 172.21.15.3:0/1548449739 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7fad7009b340 con 0x7fad80102270 2023-09-10T01:00:18.873 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:18.873 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-09-10T01:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fad680a0710 msgr2=0x7fad680a2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fad680a0710 0x7fad680a2bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fad80101730 tx=0x7fad74008040 comp rx=0 tx=0).stop 2023-09-10T01:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 msgr2=0x7fad801acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad801acc60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fad70010d80 tx=0x7fad70005690 comp rx=0 tx=0).stop 2023-09-10T01:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 shutdown_connections 2023-09-10T01:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad80102270 0x7fad801acc60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad801006e0 0x7fad8010e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fad680a0710 0x7fad680a2bd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 --2- 172.21.15.3:0/1548449739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801018d0 0x7fad8010e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 >> 172.21.15.3:0/1548449739 conn(0x7fad800fbc50 msgr2=0x7fad800fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 shutdown_connections 2023-09-10T01:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:18.876+0000 7fad85dff700 1 -- 172.21.15.3:0/1548449739 wait complete. 2023-09-10T01:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 4 2023-09-10T01:00:19.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:18 smithi003 ceph-mon[97654]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:19.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:18 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1548449739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:19.060 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:00:18 smithi003 systemd[1]: Starting Ceph mds.d for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:19.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:18 smithi092 ceph-mon[134067]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:19.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:18 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1548449739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:19.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:18 smithi136 ceph-mon[133708]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:19.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:18 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1548449739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:19.552 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:00:19 smithi003 podman[152297]: 2023-09-10T01:00:19.552 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:00:19 smithi003 bash[152297]: 7fc775c1b52e75575ec27b9e854653506a2c03a2bde3d73e59d65f93cf53f5e7 2023-09-10T01:00:19.553 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:00:19 smithi003 ceph-mds[152337]: -- 172.21.15.3:0/354775331 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56430b2db1e0 con 0x56430b2b0800 2023-09-10T01:00:19.553 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:00:19 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-d[152313]: starting mds.d at 2023-09-10T01:00:19.809 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:00:19 smithi003 systemd[1]: Started Ceph mds.d for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:20.883 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.884 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.884 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.884 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:20.884 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:20.885 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:20.885 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: Deploying daemon mds.e on smithi092 2023-09-10T01:00:20.885 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:20.885 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.3:6836/3588843640,v1:172.21.15.3:6837/3588843640] up:boot 2023-09-10T01:00:20.885 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: fsmap 4 up:standby 2023-09-10T01:00:20.886 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:20 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-09-10T01:00:20.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:20.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:20.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:20.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:20.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: Deploying daemon mds.e on smithi092 2023-09-10T01:00:20.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:20.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.3:6836/3588843640,v1:172.21.15.3:6837/3588843640] up:boot 2023-09-10T01:00:20.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: fsmap 4 up:standby 2023-09-10T01:00:20.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:20 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-09-10T01:00:20.974 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.974 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.974 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:20.974 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:20.975 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:20.975 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:20.975 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: Deploying daemon mds.e on smithi092 2023-09-10T01:00:20.975 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:20.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.3:6836/3588843640,v1:172.21.15.3:6837/3588843640] up:boot 2023-09-10T01:00:20.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: fsmap 4 up:standby 2023-09-10T01:00:20.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:20 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-09-10T01:00:21.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.121+0000 7ff25e5ff700 1 -- 172.21.15.3:0/2405233217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 msgr2=0x7ff258101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:21.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.121+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/2405233217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 0x7ff258101cf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0099c0 tx=0x7ff24c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.122+0000 7ff25e5ff700 1 -- 172.21.15.3:0/2405233217 shutdown_connections 2023-09-10T01:00:21.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.122+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/2405233217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff258102230 0x7ff258108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.122+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/2405233217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2581006a0 0x7ff258100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.122+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/2405233217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 0x7ff258101cf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.122+0000 7ff25e5ff700 1 -- 172.21.15.3:0/2405233217 >> 172.21.15.3:0/2405233217 conn(0x7ff2580fbbf0 msgr2=0x7ff2580fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:21.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.122+0000 7ff25e5ff700 1 -- 172.21.15.3:0/2405233217 shutdown_connections 2023-09-10T01:00:21.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.122+0000 7ff25e5ff700 1 -- 172.21.15.3:0/2405233217 wait complete. 2023-09-10T01:00:21.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 Processor -- start 2023-09-10T01:00:21.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 -- start start 2023-09-10T01:00:21.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2581006a0 0x7ff258198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:21.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 0x7ff258198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:21.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff258102230 0x7ff25819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:21.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2581a0180 con 0x7ff258101890 2023-09-10T01:00:21.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2581a02c0 con 0x7ff258102230 2023-09-10T01:00:21.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25e5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2581a0450 con 0x7ff2581006a0 2023-09-10T01:00:21.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25cb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff258102230 0x7ff25819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:21.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff2577fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 0x7ff258198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:21.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25cb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff258102230 0x7ff25819fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59904/0 (socket says 172.21.15.3:59904) 2023-09-10T01:00:21.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff25cb9c700 1 -- 172.21.15.3:0/1328573637 learned_addr learned my addr 172.21.15.3:0/1328573637 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:21.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.123+0000 7ff257fff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2581006a0 0x7ff258198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:21.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25cb9c700 1 -- 172.21.15.3:0/1328573637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2581006a0 msgr2=0x7ff258198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:21.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25cb9c700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2581006a0 0x7ff258198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25cb9c700 1 -- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 msgr2=0x7ff258198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:21.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25cb9c700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 0x7ff258198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25cb9c700 1 -- 172.21.15.3:0/1328573637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0095f0 con 0x7ff258102230 2023-09-10T01:00:21.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff2577fe700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 0x7ff258198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:21.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25cb9c700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff258102230 0x7ff25819fbb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff25000ef30 tx=0x7ff25000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:21.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff2557fa700 1 -- 172.21.15.3:0/1328573637 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250018070 con 0x7ff258102230 2023-09-10T01:00:21.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff2557fa700 1 -- 172.21.15.3:0/1328573637 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff25000f040 con 0x7ff258102230 2023-09-10T01:00:21.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2581a06d0 con 0x7ff258102230 2023-09-10T01:00:21.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff2557fa700 1 -- 172.21.15.3:0/1328573637 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25000cd00 con 0x7ff258102230 2023-09-10T01:00:21.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.124+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2581a0c30 con 0x7ff258102230 2023-09-10T01:00:21.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.125+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff258192ca0 con 0x7ff258102230 2023-09-10T01:00:21.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.127+0000 7ff2557fa700 1 -- 172.21.15.3:0/1328573637 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7ff250003a00 con 0x7ff258102230 2023-09-10T01:00:21.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.129+0000 7ff2557fa700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff2400a06c0 0x7ff2400a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:21.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.129+0000 7ff2557fa700 1 -- 172.21.15.3:0/1328573637 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff2500cb870 con 0x7ff258102230 2023-09-10T01:00:21.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.129+0000 7ff257fff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff2400a06c0 0x7ff2400a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:21.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.129+0000 7ff257fff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff2400a06c0 0x7ff2400a2b80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2581016f0 tx=0x7ff248009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:21.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.132+0000 7ff2557fa700 1 -- 172.21.15.3:0/1328573637 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff250095830 con 0x7ff258102230 2023-09-10T01:00:21.216 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:00:21 smithi092 systemd[1]: Starting Ceph mds.e for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.390+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25810beb0 con 0x7ff258102230 2023-09-10T01:00:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.391+0000 7ff2557fa700 1 -- 172.21.15.3:0/1328573637 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3695 (secure 0 0 0) 0x7ff250098e50 con 0x7ff258102230 2023-09-10T01:00:21.392 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:21.392 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-09-10T01:00:21.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff2400a06c0 msgr2=0x7ff2400a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:21.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff2400a06c0 0x7ff2400a2b80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2581016f0 tx=0x7ff248009260 comp rx=0 tx=0).stop 2023-09-10T01:00:21.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff258102230 msgr2=0x7ff25819fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff258102230 0x7ff25819fbb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff25000ef30 tx=0x7ff25000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 shutdown_connections 2023-09-10T01:00:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff2400a06c0 0x7ff2400a2b80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2581006a0 0x7ff258198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff258102230 0x7ff25819fbb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 --2- 172.21.15.3:0/1328573637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff258101890 0x7ff258198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:21.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 >> 172.21.15.3:0/1328573637 conn(0x7ff2580fbbf0 msgr2=0x7ff2581071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:21.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 shutdown_connections 2023-09-10T01:00:21.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:21.393+0000 7ff25e5ff700 1 -- 172.21.15.3:0/1328573637 wait complete. 2023-09-10T01:00:21.397 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 5 2023-09-10T01:00:21.641 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:00:21 smithi092 podman[170755]: 2023-09-10T01:00:21.642 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:00:21 smithi092 ceph-mds[170795]: -- 172.21.15.92:0/3413546673 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b7e4c071e0 con 0x55b7e4bdd000 2023-09-10T01:00:21.642 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:00:21 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-e[170771]: starting mds.e at 2023-09-10T01:00:21.642 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:00:21 smithi092 bash[170755]: 131439bf27622ae4172b23c6ae690f7232207a54490eaffe396ca7b6bfd6bba0 2023-09-10T01:00:21.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:21 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1328573637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:21.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:21 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1328573637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:21.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:21 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1328573637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:21.966 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:00:21 smithi092 systemd[1]: Started Ceph mds.e for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:23.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:23.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:23.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:23.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: Deploying daemon mds.f on smithi136 2023-09-10T01:00:23.016 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:23.016 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6834/790897161,v1:172.21.15.92:6835/790897161] up:boot 2023-09-10T01:00:23.016 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: fsmap 5 up:standby 2023-09-10T01:00:23.016 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:22 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:00:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: Deploying daemon mds.f on smithi136 2023-09-10T01:00:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:23.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6834/790897161,v1:172.21.15.92:6835/790897161] up:boot 2023-09-10T01:00:23.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: fsmap 5 up:standby 2023-09-10T01:00:23.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:22 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:00:23.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:23.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: Deploying daemon mds.f on smithi136 2023-09-10T01:00:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6834/790897161,v1:172.21.15.92:6835/790897161] up:boot 2023-09-10T01:00:23.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: fsmap 5 up:standby 2023-09-10T01:00:23.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:22 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:00:23.412 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:00:23 smithi136 systemd[1]: Starting Ceph mds.f for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:23.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:23.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 -- 172.21.15.3:0/3940645140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1006d0 msgr2=0x7f067c100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:23.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 --2- 172.21.15.3:0/3940645140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1006d0 0x7f067c100af0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0678009990 tx=0x7f0678009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 -- 172.21.15.3:0/3940645140 shutdown_connections 2023-09-10T01:00:23.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 --2- 172.21.15.3:0/3940645140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1006d0 0x7f067c100af0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 --2- 172.21.15.3:0/3940645140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c102260 0x7f067c108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 --2- 172.21.15.3:0/3940645140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1018c0 0x7f067c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 -- 172.21.15.3:0/3940645140 >> 172.21.15.3:0/3940645140 conn(0x7f067c0fbc60 msgr2=0x7f067c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:23.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.626+0000 7f0683dff700 1 -- 172.21.15.3:0/3940645140 shutdown_connections 2023-09-10T01:00:23.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.627+0000 7f0683dff700 1 -- 172.21.15.3:0/3940645140 wait complete. 2023-09-10T01:00:23.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.627+0000 7f0683dff700 1 Processor -- start 2023-09-10T01:00:23.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0683dff700 1 -- start start 2023-09-10T01:00:23.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0683dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1006d0 0x7f067c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:23.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0683dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1018c0 0x7f067c198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:23.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0683dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c102260 0x7f067c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:23.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0683dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c1a0150 con 0x7f067c1006d0 2023-09-10T01:00:23.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0683dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c1a0290 con 0x7f067c102260 2023-09-10T01:00:23.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0683dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c1a0420 con 0x7f067c1018c0 2023-09-10T01:00:23.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0681b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1006d0 0x7f067c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:23.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f068139a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1018c0 0x7f067c198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:23.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0681b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1006d0 0x7f067c1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60754/0 (socket says 172.21.15.3:60754) 2023-09-10T01:00:23.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f0681b9b700 1 -- 172.21.15.3:0/2860259220 learned_addr learned my addr 172.21.15.3:0/2860259220 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:23.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.628+0000 7f068239c700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c102260 0x7f067c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:23.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f068139a700 1 -- 172.21.15.3:0/2860259220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c102260 msgr2=0x7f067c19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:23.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f068139a700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c102260 0x7f067c19fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f068139a700 1 -- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1006d0 msgr2=0x7f067c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:23.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f068139a700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1006d0 0x7f067c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f068139a700 1 -- 172.21.15.3:0/2860259220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c009520 con 0x7f067c1018c0 2023-09-10T01:00:23.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f0681b9b700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1006d0 0x7f067c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:23.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f068139a700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1018c0 0x7f067c198b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f066c00e9f0 tx=0x7f066c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:23.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f0672ffd700 1 -- 172.21.15.3:0/2860259220 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c00cb70 con 0x7f067c1018c0 2023-09-10T01:00:23.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06780095f0 con 0x7f067c1018c0 2023-09-10T01:00:23.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f0672ffd700 1 -- 172.21.15.3:0/2860259220 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f066c00ee90 con 0x7f067c1018c0 2023-09-10T01:00:23.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f0672ffd700 1 -- 172.21.15.3:0/2860259220 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c003cf0 con 0x7f067c1018c0 2023-09-10T01:00:23.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.629+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c1a0ad0 con 0x7f067c1018c0 2023-09-10T01:00:23.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.630+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067c192d20 con 0x7f067c1018c0 2023-09-10T01:00:23.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.632+0000 7f0672ffd700 1 -- 172.21.15.3:0/2860259220 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f066c00ccd0 con 0x7f067c1018c0 2023-09-10T01:00:23.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.633+0000 7f0672ffd700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f06680a0640 0x7f06680a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:23.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.634+0000 7f0672ffd700 1 -- 172.21.15.3:0/2860259220 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f066c014070 con 0x7f067c1018c0 2023-09-10T01:00:23.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.634+0000 7f0681b9b700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f06680a0640 0x7f06680a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:23.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.634+0000 7f0681b9b700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f06680a0640 0x7f06680a2b00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f067800b2e0 tx=0x7f0678005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:23.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.635+0000 7f0672ffd700 1 -- 172.21.15.3:0/2860259220 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f066c095bb0 con 0x7f067c1018c0 2023-09-10T01:00:23.836 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:00:23 smithi136 podman[170814]: 2023-09-10T01:00:23.837 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:00:23 smithi136 ceph-mds[170854]: -- 172.21.15.136:0/1528433932 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de64b071e0 con 0x55de64add000 2023-09-10T01:00:23.837 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:00:23 smithi136 bash[170814]: 9826d3dc44bad27becd8d02686575c387b7f3135be571bd4dd74e8cd49f7980c 2023-09-10T01:00:23.837 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:00:23 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-f[170830]: starting mds.f at 2023-09-10T01:00:23.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.894+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f067c10bf30 con 0x7f067c1018c0 2023-09-10T01:00:23.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.895+0000 7f0672ffd700 1 -- 172.21.15.3:0/2860259220 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4483 (secure 0 0 0) 0x7f066c00fe60 con 0x7f067c1018c0 2023-09-10T01:00:23.897 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:23.898 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2023-09-10T01:00:23.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f06680a0640 msgr2=0x7f06680a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:23.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f06680a0640 0x7f06680a2b00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f067800b2e0 tx=0x7f0678005fb0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1018c0 msgr2=0x7f067c198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:23.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1018c0 0x7f067c198b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f066c00e9f0 tx=0x7f066c00ed00 comp rx=0 tx=0).stop 2023-09-10T01:00:23.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 shutdown_connections 2023-09-10T01:00:23.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c102260 0x7f067c19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f067c1006d0 0x7f067c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f06680a0640 0x7f06680a2b00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 --2- 172.21.15.3:0/2860259220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f067c1018c0 0x7f067c198b10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:23.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 >> 172.21.15.3:0/2860259220 conn(0x7f067c0fbc60 msgr2=0x7f067c107190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:23.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 shutdown_connections 2023-09-10T01:00:23.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:23.899+0000 7f0683dff700 1 -- 172.21.15.3:0/2860259220 wait complete. 2023-09-10T01:00:23.904 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 6 2023-09-10T01:00:24.162 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:00:23 smithi136 systemd[1]: Started Ceph mds.f for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:25.139 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:25.139 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.140 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.140 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.141 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:25.141 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:25.141 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2860259220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:25.142 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:25.142 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: Deploying daemon mds.c on smithi136 2023-09-10T01:00:25.142 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6834/662717057,v1:172.21.15.136:6835/662717057] up:boot 2023-09-10T01:00:25.143 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: fsmap 6 up:standby 2023-09-10T01:00:25.143 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:24 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:00:25.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2860259220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: Deploying daemon mds.c on smithi136 2023-09-10T01:00:25.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6834/662717057,v1:172.21.15.136:6835/662717057] up:boot 2023-09-10T01:00:25.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: fsmap 6 up:standby 2023-09-10T01:00:25.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:24 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:00:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2860259220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: Deploying daemon mds.c on smithi136 2023-09-10T01:00:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6834/662717057,v1:172.21.15.136:6835/662717057] up:boot 2023-09-10T01:00:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: fsmap 6 up:standby 2023-09-10T01:00:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:24 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:00:25.412 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:00:25 smithi136 systemd[1]: Starting Ceph mds.c for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:25.912 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:00:25 smithi136 podman[171154]: 2023-09-10T01:00:25.912 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:00:25 smithi136 bash[171154]: 278e9befba9b93c3bc5ba5e921063815efd228df8c68b9146314b00b2bf0acfd 2023-09-10T01:00:25.913 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:00:25 smithi136 ceph-mds[171194]: -- 172.21.15.136:0/3604164738 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5625a6a091e0 con 0x5625a69de800 2023-09-10T01:00:25.913 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:00:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-c[171170]: starting mds.c at 2023-09-10T01:00:25.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:26.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.139+0000 7f761adff700 1 -- 172.21.15.3:0/4154955406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141006c0 msgr2=0x7f7614100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:26.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.139+0000 7f761adff700 1 --2- 172.21.15.3:0/4154955406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141006c0 0x7f7614100ae0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f76080099c0 tx=0x7f7608009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.140+0000 7f761adff700 1 -- 172.21.15.3:0/4154955406 shutdown_connections 2023-09-10T01:00:26.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.140+0000 7f761adff700 1 --2- 172.21.15.3:0/4154955406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7614102250 0x7f7614108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.140+0000 7f761adff700 1 --2- 172.21.15.3:0/4154955406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141006c0 0x7f7614100ae0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.140+0000 7f761adff700 1 --2- 172.21.15.3:0/4154955406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76141018b0 0x7f7614101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.140+0000 7f761adff700 1 -- 172.21.15.3:0/4154955406 >> 172.21.15.3:0/4154955406 conn(0x7f76140fbc30 msgr2=0x7f76140fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:26.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.140+0000 7f761adff700 1 -- 172.21.15.3:0/4154955406 shutdown_connections 2023-09-10T01:00:26.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.140+0000 7f761adff700 1 -- 172.21.15.3:0/4154955406 wait complete. 2023-09-10T01:00:26.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 Processor -- start 2023-09-10T01:00:26.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 -- start start 2023-09-10T01:00:26.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76141006c0 0x7f7614198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:26.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141018b0 0x7f7614198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:26.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7614102250 0x7f761419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:26.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141a01b0 con 0x7f76141018b0 2023-09-10T01:00:26.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141a02f0 con 0x7f76141006c0 2023-09-10T01:00:26.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.141+0000 7f761adff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141a0480 con 0x7f7614102250 2023-09-10T01:00:26.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141018b0 0x7f7614198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:26.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141018b0 0x7f7614198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48304/0 (socket says 172.21.15.3:48304) 2023-09-10T01:00:26.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 -- 172.21.15.3:0/2068542270 learned_addr learned my addr 172.21.15.3:0/2068542270 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:26.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7618b9b700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76141006c0 0x7f7614198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:26.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f761939c700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7614102250 0x7f761419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:26.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 -- 172.21.15.3:0/2068542270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7614102250 msgr2=0x7f761419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:26.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7614102250 0x7f761419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 -- 172.21.15.3:0/2068542270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76141006c0 msgr2=0x7f7614198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:26.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76141006c0 0x7f7614198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 -- 172.21.15.3:0/2068542270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080095f0 con 0x7f76141018b0 2023-09-10T01:00:26.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7613fff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141018b0 0x7f7614198b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f760000e9a0 tx=0x7f760000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:26.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7611ffb700 1 -- 172.21.15.3:0/2068542270 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600009790 con 0x7f76141018b0 2023-09-10T01:00:26.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f7611ffb700 1 -- 172.21.15.3:0/2068542270 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7600010430 con 0x7f76141018b0 2023-09-10T01:00:26.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.142+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76141a0760 con 0x7f76141018b0 2023-09-10T01:00:26.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.143+0000 7f7611ffb700 1 -- 172.21.15.3:0/2068542270 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600018680 con 0x7f76141018b0 2023-09-10T01:00:26.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.143+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76141a0cc0 con 0x7f76141018b0 2023-09-10T01:00:26.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.144+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7614192d10 con 0x7f76141018b0 2023-09-10T01:00:26.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.145+0000 7f7611ffb700 1 -- 172.21.15.3:0/2068542270 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f76000105a0 con 0x7f76141018b0 2023-09-10T01:00:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.147+0000 7f7611ffb700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76040a06c0 0x7f76040a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.147+0000 7f7611ffb700 1 -- 172.21.15.3:0/2068542270 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7600014070 con 0x7f76141018b0 2023-09-10T01:00:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.147+0000 7f7618b9b700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76040a06c0 0x7f76040a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.147+0000 7f7618b9b700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76040a06c0 0x7f76040a2b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7608009990 tx=0x7f7608005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:26.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.151+0000 7f7611ffb700 1 -- 172.21.15.3:0/2068542270 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7600051e10 con 0x7f76141018b0 2023-09-10T01:00:26.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.407+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f761410bf20 con 0x7f76141018b0 2023-09-10T01:00:26.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.408+0000 7f7611ffb700 1 -- 172.21.15.3:0/2068542270 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6068 (secure 0 0 0) 0x7f7600095820 con 0x7f76141018b0 2023-09-10T01:00:26.410 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:26.410 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2023-09-10T01:00:26.411 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:00:25 smithi136 systemd[1]: Started Ceph mds.c for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:26.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76040a06c0 msgr2=0x7f76040a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:26.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76040a06c0 0x7f76040a2b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7608009990 tx=0x7f7608005fb0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141018b0 msgr2=0x7f7614198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:26.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141018b0 0x7f7614198b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f760000e9a0 tx=0x7f760000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 shutdown_connections 2023-09-10T01:00:26.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76040a06c0 0x7f76040a2b80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7614102250 0x7f761419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76141006c0 0x7f7614198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 --2- 172.21.15.3:0/2068542270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76141018b0 0x7f7614198b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:26.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 >> 172.21.15.3:0/2068542270 conn(0x7f76140fbc30 msgr2=0x7f76141071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:26.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 shutdown_connections 2023-09-10T01:00:26.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:26.410+0000 7f761adff700 1 -- 172.21.15.3:0/2068542270 wait complete. 2023-09-10T01:00:26.415 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2023-09-10T01:00:27.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:27.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6836/3841438021,v1:172.21.15.136:6837/3841438021] up:boot 2023-09-10T01:00:27.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: fsmap 7 up:standby 2023-09-10T01:00:27.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:00:27.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:27.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:27.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:27.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: Deploying daemon mds.h on smithi092 2023-09-10T01:00:27.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:26 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2068542270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6836/3841438021,v1:172.21.15.136:6837/3841438021] up:boot 2023-09-10T01:00:27.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: fsmap 7 up:standby 2023-09-10T01:00:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:00:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:27.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:27.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:27.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: Deploying daemon mds.h on smithi092 2023-09-10T01:00:27.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:26 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2068542270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6836/3841438021,v1:172.21.15.136:6837/3841438021] up:boot 2023-09-10T01:00:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: fsmap 7 up:standby 2023-09-10T01:00:27.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:00:27.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:27.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:27.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:27.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:27.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: Deploying daemon mds.h on smithi092 2023-09-10T01:00:27.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:26 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2068542270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:27.802 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:00:27 smithi092 systemd[1]: Starting Ceph mds.h for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:28.057 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:00:27 smithi092 podman[171096]: 2023-09-10T01:00:28.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:28.466 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mds[171136]: -- 172.21.15.92:0/2876366129 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b9bdb7b1e0 con 0x55b9bdb50800 2023-09-10T01:00:28.466 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:00:28 smithi092 bash[171096]: 34e948f895e707ead3586ddee92d0958fe722646d4ae3286cfd70dfa313f7233 2023-09-10T01:00:28.466 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-h[171112]: starting mds.h at 2023-09-10T01:00:28.467 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:00:28 smithi092 systemd[1]: Started Ceph mds.h for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:28.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 -- 172.21.15.3:0/3916774641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac101170 msgr2=0x7fb1ac103560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:28.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/3916774641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac101170 0x7fb1ac103560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009990 tx=0x7fb19c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 -- 172.21.15.3:0/3916774641 shutdown_connections 2023-09-10T01:00:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/3916774641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1ac1063d0 0x7fb1ac108800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/3916774641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac101170 0x7fb1ac103560 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/3916774641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1ac103aa0 0x7fb1ac105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 -- 172.21.15.3:0/3916774641 >> 172.21.15.3:0/3916774641 conn(0x7fb1ac0fab40 msgr2=0x7fb1ac0fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:28.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 -- 172.21.15.3:0/3916774641 shutdown_connections 2023-09-10T01:00:28.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.646+0000 7fb1b21ff700 1 -- 172.21.15.3:0/3916774641 wait complete. 2023-09-10T01:00:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.647+0000 7fb1b21ff700 1 Processor -- start 2023-09-10T01:00:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.647+0000 7fb1b21ff700 1 -- start start 2023-09-10T01:00:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.647+0000 7fb1b21ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1ac101170 0x7fb1ac1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:28.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1b21ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac103aa0 0x7fb1ac196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:28.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1b21ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1ac1063d0 0x7fb1ac19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:28.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1b21ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac19df70 con 0x7fb1ac103aa0 2023-09-10T01:00:28.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1b21ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac19e0b0 con 0x7fb1ac101170 2023-09-10T01:00:28.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1b21ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac19e240 con 0x7fb1ac1063d0 2023-09-10T01:00:28.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac103aa0 0x7fb1ac196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:28.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac103aa0 0x7fb1ac196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48318/0 (socket says 172.21.15.3:48318) 2023-09-10T01:00:28.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 -- 172.21.15.3:0/668564565 learned_addr learned my addr 172.21.15.3:0/668564565 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:28.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1abfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1ac1063d0 0x7fb1ac19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:28.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1ab7fe700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1ac101170 0x7fb1ac1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:28.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 -- 172.21.15.3:0/668564565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1ac1063d0 msgr2=0x7fb1ac19d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:28.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1ac1063d0 0x7fb1ac19d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 -- 172.21.15.3:0/668564565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1ac101170 msgr2=0x7fb1ac1963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:28.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1ac101170 0x7fb1ac1963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 -- 172.21.15.3:0/668564565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0095f0 con 0x7fb1ac103aa0 2023-09-10T01:00:28.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1ab7fe700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1ac101170 0x7fb1ac1963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:28.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.648+0000 7fb1a3fff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac103aa0 0x7fb1ac196930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb19400e9a0 tx=0x7fb19400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:28.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.649+0000 7fb1a97fa700 1 -- 172.21.15.3:0/668564565 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb194009790 con 0x7fb1ac103aa0 2023-09-10T01:00:28.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.649+0000 7fb1a97fa700 1 -- 172.21.15.3:0/668564565 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb19400ce50 con 0x7fb1ac103aa0 2023-09-10T01:00:28.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.649+0000 7fb1a97fa700 1 -- 172.21.15.3:0/668564565 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb194018680 con 0x7fb1ac103aa0 2023-09-10T01:00:28.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.649+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac0fefc0 con 0x7fb1ac103aa0 2023-09-10T01:00:28.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.649+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac0ff4d0 con 0x7fb1ac103aa0 2023-09-10T01:00:28.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.650+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac0fc6c0 con 0x7fb1ac103aa0 2023-09-10T01:00:28.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.652+0000 7fb1a97fa700 1 -- 172.21.15.3:0/668564565 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb194010430 con 0x7fb1ac103aa0 2023-09-10T01:00:28.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.653+0000 7fb1a97fa700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1980a06c0 0x7fb1980a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:28.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.653+0000 7fb1a97fa700 1 -- 172.21.15.3:0/668564565 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb194014070 con 0x7fb1ac103aa0 2023-09-10T01:00:28.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.654+0000 7fb1ab7fe700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1980a06c0 0x7fb1980a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:28.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.654+0000 7fb1ab7fe700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1980a06c0 0x7fb1980a2b80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb19c000c00 tx=0x7fb19c00b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:28.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.655+0000 7fb1a97fa700 1 -- 172.21.15.3:0/668564565 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb194095b90 con 0x7fb1ac103aa0 2023-09-10T01:00:28.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.914+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1ac10c150 con 0x7fb1ac103aa0 2023-09-10T01:00:28.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.915+0000 7fb1a97fa700 1 -- 172.21.15.3:0/668564565 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6068 (secure 0 0 0) 0x7fb1940991b0 con 0x7fb1ac103aa0 2023-09-10T01:00:28.916 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:28.916 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2023-09-10T01:00:28.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1980a06c0 msgr2=0x7fb1980a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:28.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1980a06c0 0x7fb1980a2b80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb19c000c00 tx=0x7fb19c00b260 comp rx=0 tx=0).stop 2023-09-10T01:00:28.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac103aa0 msgr2=0x7fb1ac196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:28.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac103aa0 0x7fb1ac196930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb19400e9a0 tx=0x7fb19400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 shutdown_connections 2023-09-10T01:00:28.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1980a06c0 0x7fb1980a2b80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1ac1063d0 0x7fb1ac19d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1ac101170 0x7fb1ac1963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 --2- 172.21.15.3:0/668564565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1ac103aa0 0x7fb1ac196930 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:28.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 >> 172.21.15.3:0/668564565 conn(0x7fb1ac0fab40 msgr2=0x7fb1ac101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:28.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 shutdown_connections 2023-09-10T01:00:28.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:28.918+0000 7fb1b21ff700 1 -- 172.21.15.3:0/668564565 wait complete. 2023-09-10T01:00:28.922 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2023-09-10T01:00:29.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:29.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:29.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:29.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:29.104 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:28 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/668564565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:29.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:29.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:29.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:28 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/668564565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:29.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:29.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:29.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:29.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:29.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:29.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:28 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/668564565' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:29.912 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:00:29 smithi136 systemd[1]: Starting Ceph mds.i for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:30.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:29 smithi092 ceph-mon[134067]: Deploying daemon mds.i on smithi136 2023-09-10T01:00:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:29 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] up:boot 2023-09-10T01:00:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:29 smithi092 ceph-mon[134067]: fsmap 8 up:standby 2023-09-10T01:00:30.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:29 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:00:30.250 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:29 smithi136 ceph-mon[133708]: Deploying daemon mds.i on smithi136 2023-09-10T01:00:30.250 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:29 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] up:boot 2023-09-10T01:00:30.250 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:29 smithi136 ceph-mon[133708]: fsmap 8 up:standby 2023-09-10T01:00:30.250 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:29 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:00:30.251 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:00:29 smithi136 podman[171500]: 2023-09-10T01:00:30.251 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mds[171540]: -- 172.21.15.136:0/3671774913 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5596ac41b1e0 con 0x5596ac3f1000 2023-09-10T01:00:30.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:29 smithi003 ceph-mon[97654]: Deploying daemon mds.i on smithi136 2023-09-10T01:00:30.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:29 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] up:boot 2023-09-10T01:00:30.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:29 smithi003 ceph-mon[97654]: fsmap 8 up:standby 2023-09-10T01:00:30.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:29 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:00:30.662 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-i[171516]: starting mds.i at 2023-09-10T01:00:30.662 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:00:30 smithi136 bash[171500]: d359797a393059e48dddd737288bc4a5d2af08f48ac55297db254fec0f70c962 2023-09-10T01:00:30.662 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:00:30 smithi136 systemd[1]: Started Ceph mds.i for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:30.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:31.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 -- 172.21.15.3:0/3355417833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 msgr2=0x7f6f5c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:31.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3355417833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c101d10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6f480099c0 tx=0x7f6f48009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 -- 172.21.15.3:0/3355417833 shutdown_connections 2023-09-10T01:00:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3355417833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f5c1006c0 0x7f6f5c100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3355417833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f5c102250 0x7f6f5c108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3355417833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c101d10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 -- 172.21.15.3:0/3355417833 >> 172.21.15.3:0/3355417833 conn(0x7f6f5c0fbc30 msgr2=0x7f6f5c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 -- 172.21.15.3:0/3355417833 shutdown_connections 2023-09-10T01:00:31.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.156+0000 7f6f62dff700 1 -- 172.21.15.3:0/3355417833 wait complete. 2023-09-10T01:00:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.157+0000 7f6f62dff700 1 Processor -- start 2023-09-10T01:00:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.157+0000 7f6f62dff700 1 -- start start 2023-09-10T01:00:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.157+0000 7f6f62dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f5c1006c0 0x7f6f5c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.157+0000 7f6f62dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.157+0000 7f6f62dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f5c102250 0x7f6f5c19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.157+0000 7f6f62dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c1a0160 con 0x7f6f5c1018b0 2023-09-10T01:00:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.157+0000 7f6f62dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c1a02a0 con 0x7f6f5c102250 2023-09-10T01:00:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f62dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c1a0430 con 0x7f6f5c1006c0 2023-09-10T01:00:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48332/0 (socket says 172.21.15.3:48332) 2023-09-10T01:00:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 -- 172.21.15.3:0/3298769453 learned_addr learned my addr 172.21.15.3:0/3298769453 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f60b9b700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f5c1006c0 0x7f6f5c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f6139c700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f5c102250 0x7f6f5c19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 -- 172.21.15.3:0/3298769453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f5c1006c0 msgr2=0x7f6f5c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f5c1006c0 0x7f6f5c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 -- 172.21.15.3:0/3298769453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f5c102250 msgr2=0x7f6f5c19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f5c102250 0x7f6f5c19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 -- 172.21.15.3:0/3298769453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f480095f0 con 0x7f6f5c1018b0 2023-09-10T01:00:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.158+0000 7f6f5bfff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c198b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009fd0 tx=0x7f6f4800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.159+0000 7f6f59ffb700 1 -- 172.21.15.3:0/3298769453 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4801d070 con 0x7f6f5c1018b0 2023-09-10T01:00:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.159+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c1a06b0 con 0x7f6f5c1018b0 2023-09-10T01:00:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.159+0000 7f6f59ffb700 1 -- 172.21.15.3:0/3298769453 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f48005050 con 0x7f6f5c1018b0 2023-09-10T01:00:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.159+0000 7f6f59ffb700 1 -- 172.21.15.3:0/3298769453 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48004b30 con 0x7f6f5c1018b0 2023-09-10T01:00:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.159+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c1a0b30 con 0x7f6f5c1018b0 2023-09-10T01:00:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.160+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c10bf30 con 0x7f6f5c1018b0 2023-09-10T01:00:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.162+0000 7f6f59ffb700 1 -- 172.21.15.3:0/3298769453 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f6f48004c90 con 0x7f6f5c1018b0 2023-09-10T01:00:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.163+0000 7f6f59ffb700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6f4c0a0640 0x7f6f4c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.164+0000 7f6f59ffb700 1 -- 172.21.15.3:0/3298769453 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6f480cc940 con 0x7f6f5c1018b0 2023-09-10T01:00:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.164+0000 7f6f60b9b700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6f4c0a0640 0x7f6f4c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:31.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.164+0000 7f6f60b9b700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6f4c0a0640 0x7f6f4c0a2b00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c101710 tx=0x7f6f50009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:31.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.166+0000 7f6f59ffb700 1 -- 172.21.15.3:0/3298769453 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6f48096900 con 0x7f6f5c1018b0 2023-09-10T01:00:31.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:30 smithi092 ceph-mon[134067]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:31.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:31.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:30 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:30 smithi003 ceph-mon[97654]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:31.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:31.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:30 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mon[133708]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:31.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:31.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:31.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:30 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:31.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.442+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f5c10c120 con 0x7f6f5c1018b0 2023-09-10T01:00:31.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.443+0000 7f6f59ffb700 1 -- 172.21.15.3:0/3298769453 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7649 (secure 0 0 0) 0x7f6f48099f20 con 0x7f6f5c1018b0 2023-09-10T01:00:31.444 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:31.444 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24634,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24647,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2023-09-10T01:00:31.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6f4c0a0640 msgr2=0x7f6f4c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:31.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6f4c0a0640 0x7f6f4c0a2b00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c101710 tx=0x7f6f50009260 comp rx=0 tx=0).stop 2023-09-10T01:00:31.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 msgr2=0x7f6f5c198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:31.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c198b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009fd0 tx=0x7f6f4800f690 comp rx=0 tx=0).stop 2023-09-10T01:00:31.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 shutdown_connections 2023-09-10T01:00:31.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6f4c0a0640 0x7f6f4c0a2b00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f5c1006c0 0x7f6f5c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f5c102250 0x7f6f5c19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 --2- 172.21.15.3:0/3298769453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f5c1018b0 0x7f6f5c198b20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:31.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.445+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 >> 172.21.15.3:0/3298769453 conn(0x7f6f5c0fbc30 msgr2=0x7f6f5c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:31.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.446+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 shutdown_connections 2023-09-10T01:00:31.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:31.446+0000 7f6f62dff700 1 -- 172.21.15.3:0/3298769453 wait complete. 2023-09-10T01:00:31.449 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 10 2023-09-10T01:00:32.051 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:31 smithi003 ceph-mon[97654]: Deploying daemon mds.g on smithi003 2023-09-10T01:00:32.051 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:31 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] up:boot 2023-09-10T01:00:32.051 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:31 smithi003 ceph-mon[97654]: fsmap 9 up:standby 2023-09-10T01:00:32.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:31 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:00:32.052 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:31 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3298769453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:32.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:31 smithi092 ceph-mon[134067]: Deploying daemon mds.g on smithi003 2023-09-10T01:00:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:31 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] up:boot 2023-09-10T01:00:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:31 smithi092 ceph-mon[134067]: fsmap 9 up:standby 2023-09-10T01:00:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:31 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:00:32.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:31 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3298769453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:31 smithi136 ceph-mon[133708]: Deploying daemon mds.g on smithi003 2023-09-10T01:00:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:31 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] up:boot 2023-09-10T01:00:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:31 smithi136 ceph-mon[133708]: fsmap 9 up:standby 2023-09-10T01:00:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:31 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:00:32.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:31 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3298769453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:32.621 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:00:32 smithi003 systemd[1]: Starting Ceph mds.g for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:33.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:32 smithi092 ceph-mon[134067]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:33.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:32 smithi003 ceph-mon[97654]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:33.248 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:00:33 smithi003 podman[152867]: 2023-09-10T01:00:33.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:32 smithi136 ceph-mon[133708]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:33.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:33.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 -- 172.21.15.3:0/3986458298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81006c0 msgr2=0x7f15f8100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:33.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 --2- 172.21.15.3:0/3986458298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81006c0 0x7f15f8100ae0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f15e80098f0 tx=0x7f15e8009c00 comp rx=0 tx=0).stop 2023-09-10T01:00:33.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 -- 172.21.15.3:0/3986458298 shutdown_connections 2023-09-10T01:00:33.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 --2- 172.21.15.3:0/3986458298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81006c0 0x7f15f8100ae0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 --2- 172.21.15.3:0/3986458298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f8102250 0x7f15f8108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 --2- 172.21.15.3:0/3986458298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81018b0 0x7f15f8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 -- 172.21.15.3:0/3986458298 >> 172.21.15.3:0/3986458298 conn(0x7f15f80fbc30 msgr2=0x7f15f80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 -- 172.21.15.3:0/3986458298 shutdown_connections 2023-09-10T01:00:33.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.682+0000 7f15fdfff700 1 -- 172.21.15.3:0/3986458298 wait complete. 2023-09-10T01:00:33.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 Processor -- start 2023-09-10T01:00:33.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 -- start start 2023-09-10T01:00:33.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f81006c0 0x7f15f81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:33.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81018b0 0x7f15f8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:33.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f8102250 0x7f15f819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:33.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f81a0220 con 0x7f15f81006c0 2023-09-10T01:00:33.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f81a0360 con 0x7f15f8102250 2023-09-10T01:00:33.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.683+0000 7f15fdfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f81a04f0 con 0x7f15f81018b0 2023-09-10T01:00:33.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f81006c0 0x7f15f81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:33.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f81006c0 0x7f15f81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48374/0 (socket says 172.21.15.3:48374) 2023-09-10T01:00:33.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 -- 172.21.15.3:0/737175200 learned_addr learned my addr 172.21.15.3:0/737175200 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:33.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f7fff700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f8102250 0x7f15f819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:33.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f6ffd700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81018b0 0x7f15f8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:33.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 -- 172.21.15.3:0/737175200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81018b0 msgr2=0x7f15f8198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:33.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81018b0 0x7f15f8198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 -- 172.21.15.3:0/737175200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f8102250 msgr2=0x7f15f819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:33.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f8102250 0x7f15f819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 -- 172.21.15.3:0/737175200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0009c40 con 0x7f15f81006c0 2023-09-10T01:00:33.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f6ffd700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81018b0 0x7f15f8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:33.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.684+0000 7f15f77fe700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f81006c0 0x7f15f81986a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009fd0 tx=0x7f15e800b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:33.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.685+0000 7f15f4ff9700 1 -- 172.21.15.3:0/737175200 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e801c070 con 0x7f15f81006c0 2023-09-10T01:00:33.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.685+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e8009520 con 0x7f15f81006c0 2023-09-10T01:00:33.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.685+0000 7f15f4ff9700 1 -- 172.21.15.3:0/737175200 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15e8005040 con 0x7f15f81006c0 2023-09-10T01:00:33.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.685+0000 7f15f4ff9700 1 -- 172.21.15.3:0/737175200 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8003a20 con 0x7f15f81006c0 2023-09-10T01:00:33.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.685+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f81a0a60 con 0x7f15f81006c0 2023-09-10T01:00:33.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.686+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f8192d10 con 0x7f15f81006c0 2023-09-10T01:00:33.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.688+0000 7f15f4ff9700 1 -- 172.21.15.3:0/737175200 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f15e8020030 con 0x7f15f81006c0 2023-09-10T01:00:33.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.690+0000 7f15f4ff9700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15e00a0640 0x7f15e00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:33.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.690+0000 7f15f4ff9700 1 -- 172.21.15.3:0/737175200 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f15e80cc850 con 0x7f15f81006c0 2023-09-10T01:00:33.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.691+0000 7f15f6ffd700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15e00a0640 0x7f15e00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:33.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.691+0000 7f15f6ffd700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15e00a0640 0x7f15e00a2b00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f15ec005fd0 tx=0x7f15ec009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:33.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.693+0000 7f15f4ff9700 1 -- 172.21.15.3:0/737175200 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f15e8096810 con 0x7f15f81006c0 2023-09-10T01:00:33.809 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:00:33 smithi003 ceph-mds[152907]: -- 172.21.15.3:0/499296287 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5586c54e91e0 con 0x5586c54bec00 2023-09-10T01:00:33.809 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:00:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-g[152883]: starting mds.g at 2023-09-10T01:00:33.810 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:00:33 smithi003 bash[152867]: e8f6624d6bb3c871f00afec16bb048a5c872eaf02efeb99fd5b6ac7f38056a3a 2023-09-10T01:00:33.810 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:00:33 smithi003 systemd[1]: Started Ceph mds.g for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.955+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15f810c140 con 0x7f15f81006c0 2023-09-10T01:00:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.956+0000 7f15f4ff9700 1 -- 172.21.15.3:0/737175200 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7649 (secure 0 0 0) 0x7f15e80054d0 con 0x7f15f81006c0 2023-09-10T01:00:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:33.957 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24634,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24647,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2023-09-10T01:00:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15e00a0640 msgr2=0x7f15e00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15e00a0640 0x7f15e00a2b00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f15ec005fd0 tx=0x7f15ec009310 comp rx=0 tx=0).stop 2023-09-10T01:00:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f81006c0 msgr2=0x7f15f81986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f81006c0 0x7f15f81986a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009fd0 tx=0x7f15e800b610 comp rx=0 tx=0).stop 2023-09-10T01:00:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 shutdown_connections 2023-09-10T01:00:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f8102250 0x7f15f819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f81006c0 0x7f15f81986a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15e00a0640 0x7f15e00a2b00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 --2- 172.21.15.3:0/737175200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f81018b0 0x7f15f8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.958+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 >> 172.21.15.3:0/737175200 conn(0x7f15f80fbc30 msgr2=0x7f15f8107290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.959+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 shutdown_connections 2023-09-10T01:00:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:33.959+0000 7f15fdfff700 1 -- 172.21.15.3:0/737175200 wait complete. 2023-09-10T01:00:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 10 2023-09-10T01:00:34.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:34.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:34.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: Deploying daemon mds.a on smithi003 2023-09-10T01:00:34.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:34.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/737175200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:34.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.3:6838/640397108,v1:172.21.15.3:6839/640397108] up:boot 2023-09-10T01:00:34.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: fsmap 10 up:standby 2023-09-10T01:00:34.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:34 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-09-10T01:00:34.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:34.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:34.786 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:34.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: Deploying daemon mds.a on smithi003 2023-09-10T01:00:34.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:34.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/737175200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:34.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.3:6838/640397108,v1:172.21.15.3:6839/640397108] up:boot 2023-09-10T01:00:34.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: fsmap 10 up:standby 2023-09-10T01:00:34.788 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:34 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-09-10T01:00:34.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:34.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:34.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:34.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:34.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: Deploying daemon mds.a on smithi003 2023-09-10T01:00:34.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:34.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/737175200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:34.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.3:6838/640397108,v1:172.21.15.3:6839/640397108] up:boot 2023-09-10T01:00:34.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: fsmap 10 up:standby 2023-09-10T01:00:34.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:34 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-09-10T01:00:35.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:36.055 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:00:35 smithi003 systemd[1]: Starting Ceph mds.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:36.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1528996870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 msgr2=0x7ff9b41035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:36.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1528996870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41035c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40098f0 tx=0x7ff9a4009c00 comp rx=0 tx=0).stop 2023-09-10T01:00:36.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1528996870 shutdown_connections 2023-09-10T01:00:36.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1528996870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b4106430 0x7ff9b4108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1528996870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41035c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1528996870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b4103b00 0x7ff9b4105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1528996870 >> 172.21.15.3:0/1528996870 conn(0x7ff9b40fab40 msgr2=0x7ff9b40fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1528996870 shutdown_connections 2023-09-10T01:00:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.213+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1528996870 wait complete. 2023-09-10T01:00:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.214+0000 7ff9ba1ff700 1 Processor -- start 2023-09-10T01:00:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.214+0000 7ff9ba1ff700 1 -- start start 2023-09-10T01:00:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9ba1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9ba1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b4103b00 0x7ff9b4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9ba1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b4106430 0x7ff9b419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9ba1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b41a0160 con 0x7ff9b41011d0 2023-09-10T01:00:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9ba1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b41a02a0 con 0x7ff9b4106430 2023-09-10T01:00:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9ba1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b41a0430 con 0x7ff9b4103b00 2023-09-10T01:00:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60564/0 (socket says 172.21.15.3:60564) 2023-09-10T01:00:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 -- 172.21.15.3:0/1613755564 learned_addr learned my addr 172.21.15.3:0/1613755564 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b3fff700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b4106430 0x7ff9b419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b2ffd700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b4103b00 0x7ff9b4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 -- 172.21.15.3:0/1613755564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b4103b00 msgr2=0x7ff9b4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:36.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b4103b00 0x7ff9b4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 -- 172.21.15.3:0/1613755564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b4106430 msgr2=0x7ff9b419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:36.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b4106430 0x7ff9b419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.215+0000 7ff9b37fe700 1 -- 172.21.15.3:0/1613755564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a80095f0 con 0x7ff9b41011d0 2023-09-10T01:00:36.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.216+0000 7ff9b37fe700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41985e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009fd0 tx=0x7ff9a400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:36.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.216+0000 7ff9b0ff9700 1 -- 172.21.15.3:0/1613755564 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a401c070 con 0x7ff9b41011d0 2023-09-10T01:00:36.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.216+0000 7ff9b0ff9700 1 -- 172.21.15.3:0/1613755564 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9a400f7d0 con 0x7ff9b41011d0 2023-09-10T01:00:36.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.216+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a4009520 con 0x7ff9b41011d0 2023-09-10T01:00:36.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.216+0000 7ff9b0ff9700 1 -- 172.21.15.3:0/1613755564 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4004ad0 con 0x7ff9b41011d0 2023-09-10T01:00:36.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.216+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b41a0940 con 0x7ff9b41011d0 2023-09-10T01:00:36.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.217+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b4107fb0 con 0x7ff9b41011d0 2023-09-10T01:00:36.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.219+0000 7ff9b0ff9700 1 -- 172.21.15.3:0/1613755564 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7ff9a4020070 con 0x7ff9b41011d0 2023-09-10T01:00:36.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.220+0000 7ff9b0ff9700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff99c0a0640 0x7ff99c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:36.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.220+0000 7ff9b0ff9700 1 -- 172.21.15.3:0/1613755564 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9a40cc370 con 0x7ff9b41011d0 2023-09-10T01:00:36.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.221+0000 7ff9b2ffd700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff99c0a0640 0x7ff99c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:36.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.221+0000 7ff9b2ffd700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff99c0a0640 0x7ff99c0a2b00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8005fd0 tx=0x7ff9a8009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:36.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.225+0000 7ff9b0ff9700 1 -- 172.21.15.3:0/1613755564 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff9a405ce10 con 0x7ff9b41011d0 2023-09-10T01:00:36.309 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:00:36 smithi003 podman[153280]: 2023-09-10T01:00:36.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.493+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b410bf00 con 0x7ff9b41011d0 2023-09-10T01:00:36.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.494+0000 7ff9b0ff9700 1 -- 172.21.15.3:0/1613755564 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8435 (secure 0 0 0) 0x7ff9a4096330 con 0x7ff9b41011d0 2023-09-10T01:00:36.496 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:36.496 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24634,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24647,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2023-09-10T01:00:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.496+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff99c0a0640 msgr2=0x7ff99c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.496+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff99c0a0640 0x7ff99c0a2b00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8005fd0 tx=0x7ff9a8009310 comp rx=0 tx=0).stop 2023-09-10T01:00:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 msgr2=0x7ff9b41985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41985e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009fd0 tx=0x7ff9a400b790 comp rx=0 tx=0).stop 2023-09-10T01:00:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 shutdown_connections 2023-09-10T01:00:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9b4106430 0x7ff9b419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9b41011d0 0x7ff9b41985e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff99c0a0640 0x7ff99c0a2b00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 --2- 172.21.15.3:0/1613755564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b4103b00 0x7ff9b4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 >> 172.21.15.3:0/1613755564 conn(0x7ff9b40fab40 msgr2=0x7ff9b40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 shutdown_connections 2023-09-10T01:00:36.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:36.497+0000 7ff9ba1ff700 1 -- 172.21.15.3:0/1613755564 wait complete. 2023-09-10T01:00:36.503 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 11 2023-09-10T01:00:36.809 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:00:36 smithi003 bash[153280]: ea54adaa57cdaef890e34822c14c44ac9beb021d8b002303785e4236dcc2e067 2023-09-10T01:00:36.809 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mds[153342]: -- 172.21.15.3:0/670715264 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e165171e0 con 0x562e164ecc00 2023-09-10T01:00:36.809 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-a[153316]: starting mds.a at 2023-09-10T01:00:36.810 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:00:36 smithi003 systemd[1]: Started Ceph mds.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:37.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:37.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1613755564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:37.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:37.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:37.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:36 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:37.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:37.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1613755564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:37.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:37.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:37.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:36 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:37.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:37.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1613755564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:37.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:37.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-09-10T01:00:37.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-09-10T01:00:37.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:36 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:37 smithi092 ceph-mon[134067]: Deploying daemon mds.b on smithi092 2023-09-10T01:00:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:37 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.3:6840/1239897501,v1:172.21.15.3:6841/1239897501] up:boot 2023-09-10T01:00:38.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:37 smithi092 ceph-mon[134067]: fsmap 11 up:standby 2023-09-10T01:00:38.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:37 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-09-10T01:00:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:37 smithi003 ceph-mon[97654]: Deploying daemon mds.b on smithi092 2023-09-10T01:00:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:37 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.3:6840/1239897501,v1:172.21.15.3:6841/1239897501] up:boot 2023-09-10T01:00:38.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:37 smithi003 ceph-mon[97654]: fsmap 11 up:standby 2023-09-10T01:00:38.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:37 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-09-10T01:00:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:37 smithi136 ceph-mon[133708]: Deploying daemon mds.b on smithi092 2023-09-10T01:00:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:37 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.3:6840/1239897501,v1:172.21.15.3:6841/1239897501] up:boot 2023-09-10T01:00:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:37 smithi136 ceph-mon[133708]: fsmap 11 up:standby 2023-09-10T01:00:38.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:37 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-09-10T01:00:38.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:38.524 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:00:38 smithi092 systemd[1]: Starting Ceph mds.b for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:00:38.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.735+0000 7f7063bff700 1 -- 172.21.15.3:0/4012318777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c102250 msgr2=0x7f705c108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:38.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.735+0000 7f7063bff700 1 --2- 172.21.15.3:0/4012318777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c102250 0x7f705c108b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f705400b0d0 tx=0x7f705400b3e0 comp rx=0 tx=0).stop 2023-09-10T01:00:38.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.735+0000 7f7063bff700 1 -- 172.21.15.3:0/4012318777 shutdown_connections 2023-09-10T01:00:38.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.735+0000 7f7063bff700 1 --2- 172.21.15.3:0/4012318777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c102250 0x7f705c108b00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:38.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.735+0000 7f7063bff700 1 --2- 172.21.15.3:0/4012318777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f705c1006c0 0x7f705c100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:38.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.735+0000 7f7063bff700 1 --2- 172.21.15.3:0/4012318777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c1018b0 0x7f705c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:38.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.735+0000 7f7063bff700 1 -- 172.21.15.3:0/4012318777 >> 172.21.15.3:0/4012318777 conn(0x7f705c0fbc30 msgr2=0x7f705c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.736+0000 7f7063bff700 1 -- 172.21.15.3:0/4012318777 shutdown_connections 2023-09-10T01:00:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.736+0000 7f7063bff700 1 -- 172.21.15.3:0/4012318777 wait complete. 2023-09-10T01:00:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.736+0000 7f7063bff700 1 Processor -- start 2023-09-10T01:00:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f7063bff700 1 -- start start 2023-09-10T01:00:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f7063bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c1006c0 0x7f705c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f7063bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c1018b0 0x7f705c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f7063bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f705c102250 0x7f705c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f7063bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c1a0250 con 0x7f705c1018b0 2023-09-10T01:00:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f7063bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c1a0390 con 0x7f705c102250 2023-09-10T01:00:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f7063bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c1a0520 con 0x7f705c1006c0 2023-09-10T01:00:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f706119a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c1018b0 0x7f705c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f706219c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f705c102250 0x7f705c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:38.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f706119a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c1018b0 0x7f705c198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60586/0 (socket says 172.21.15.3:60586) 2023-09-10T01:00:38.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f706119a700 1 -- 172.21.15.3:0/380110639 learned_addr learned my addr 172.21.15.3:0/380110639 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:38.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.737+0000 7f706199b700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c1006c0 0x7f705c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:38.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f706219c700 1 -- 172.21.15.3:0/380110639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c1006c0 msgr2=0x7f705c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:38.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f706219c700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c1006c0 0x7f705c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f706219c700 1 -- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c1018b0 msgr2=0x7f705c198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f706219c700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c1018b0 0x7f705c198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f706219c700 1 -- 172.21.15.3:0/380110639 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7058009520 con 0x7f705c102250 2023-09-10T01:00:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f706119a700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c1018b0 0x7f705c198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f706219c700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f705c102250 0x7f705c19fc80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7054007a30 tx=0x7f7054008eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f7052ffd700 1 -- 172.21.15.3:0/380110639 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705400e040 con 0x7f705c102250 2023-09-10T01:00:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f7052ffd700 1 -- 172.21.15.3:0/380110639 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f705400bd30 con 0x7f705c102250 2023-09-10T01:00:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7054009d00 con 0x7f705c102250 2023-09-10T01:00:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.738+0000 7f7052ffd700 1 -- 172.21.15.3:0/380110639 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705401d470 con 0x7f705c102250 2023-09-10T01:00:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.739+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705c1a0a40 con 0x7f705c102250 2023-09-10T01:00:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.739+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705c192d20 con 0x7f705c102250 2023-09-10T01:00:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.742+0000 7f7052ffd700 1 -- 172.21.15.3:0/380110639 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f7054005000 con 0x7f705c102250 2023-09-10T01:00:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.743+0000 7f7052ffd700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f70480a0640 0x7f70480a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.743+0000 7f7052ffd700 1 -- 172.21.15.3:0/380110639 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f70540cd1b0 con 0x7f705c102250 2023-09-10T01:00:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.743+0000 7f706199b700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f70480a0640 0x7f70480a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.744+0000 7f706199b700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f70480a0640 0x7f70480a2b00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f705c101710 tx=0x7f7058009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:38.745+0000 7f7052ffd700 1 -- 172.21.15.3:0/380110639 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7054097170 con 0x7f705c102250 2023-09-10T01:00:38.804 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:00:38 smithi092 podman[171438]: 2023-09-10T01:00:39.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.026+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f705c10bf30 con 0x7f705c102250 2023-09-10T01:00:39.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.027+0000 7f7052ffd700 1 -- 172.21.15.3:0/380110639 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10013 (secure 0 0 0) 0x7f705409a790 con 0x7f705c102250 2023-09-10T01:00:39.028 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:39.029 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24634,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24647,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24698,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2023-09-10T01:00:39.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f70480a0640 msgr2=0x7f70480a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f70480a0640 0x7f70480a2b00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f705c101710 tx=0x7f7058009260 comp rx=0 tx=0).stop 2023-09-10T01:00:39.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f705c102250 msgr2=0x7f705c19fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f705c102250 0x7f705c19fc80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7054007a30 tx=0x7f7054008eb0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 shutdown_connections 2023-09-10T01:00:39.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f70480a0640 0x7f70480a2b00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f705c1006c0 0x7f705c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f705c102250 0x7f705c19fc80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 --2- 172.21.15.3:0/380110639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f705c1018b0 0x7f705c198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 >> 172.21.15.3:0/380110639 conn(0x7f705c0fbc30 msgr2=0x7f705c107160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:39.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 shutdown_connections 2023-09-10T01:00:39.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.030+0000 7f7063bff700 1 -- 172.21.15.3:0/380110639 wait complete. 2023-09-10T01:00:39.034 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 13 2023-09-10T01:00:39.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:00:39.215 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-mds[171478]: -- 172.21.15.92:0/1993152302 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d208f671e0 con 0x55d208f3d000 2023-09-10T01:00:39.216 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:00:38 smithi092 bash[171438]: f8c49417c46d1399e23419b9b255548cbf795bfc6c4b9781ee86e35f6af6f692 2023-09-10T01:00:39.216 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: starting mds.b at 2023-09-10T01:00:39.217 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:00:38 smithi092 systemd[1]: Started Ceph mds.b for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:00:39.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-mon[134067]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:39.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:38 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 -- 172.21.15.3:0/374425205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 msgr2=0x7f90d0102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 --2- 172.21.15.3:0/374425205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d0102d00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f90c00099c0 tx=0x7f90c0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 -- 172.21.15.3:0/374425205 shutdown_connections 2023-09-10T01:00:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 --2- 172.21.15.3:0/374425205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90d0104470 0x7f90d010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 --2- 172.21.15.3:0/374425205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d0102d00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 --2- 172.21.15.3:0/374425205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d0103ad0 0x7f90d0103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 -- 172.21.15.3:0/374425205 >> 172.21.15.3:0/374425205 conn(0x7f90d00fde50 msgr2=0x7f90d0100290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:39.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 -- 172.21.15.3:0/374425205 shutdown_connections 2023-09-10T01:00:39.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.272+0000 7f90d67ff700 1 -- 172.21.15.3:0/374425205 wait complete. 2023-09-10T01:00:39.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 Processor -- start 2023-09-10T01:00:39.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 -- start start 2023-09-10T01:00:39.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90d0103ad0 0x7f90d0198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d0104470 0x7f90d019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d01a0130 con 0x7f90d0103ad0 2023-09-10T01:00:39.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d01a0270 con 0x7f90d01028e0 2023-09-10T01:00:39.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.273+0000 7f90d67ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d01a0400 con 0x7f90d0104470 2023-09-10T01:00:39.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42428/0 (socket says 172.21.15.3:42428) 2023-09-10T01:00:39.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 -- 172.21.15.3:0/3997580476 learned_addr learned my addr 172.21.15.3:0/3997580476 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:39.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90d4d9c700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d0104470 0x7f90d019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cf7fe700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90d0103ad0 0x7f90d0198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 -- 172.21.15.3:0/3997580476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d0104470 msgr2=0x7f90d019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d0104470 0x7f90d019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 -- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90d0103ad0 msgr2=0x7f90d0198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90d0103ad0 0x7f90d0198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 -- 172.21.15.3:0/3997580476 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c00095f0 con 0x7f90d01028e0 2023-09-10T01:00:39.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90d4d9c700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d0104470 0x7f90d019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cffff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d01985b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009990 tx=0x7f90c000f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.274+0000 7f90cd7fa700 1 -- 172.21.15.3:0/3997580476 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c001d070 con 0x7f90d01028e0 2023-09-10T01:00:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.275+0000 7f90cd7fa700 1 -- 172.21.15.3:0/3997580476 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90c00054f0 con 0x7f90d01028e0 2023-09-10T01:00:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.275+0000 7f90cd7fa700 1 -- 172.21.15.3:0/3997580476 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0004bb0 con 0x7f90d01028e0 2023-09-10T01:00:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.275+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d01a0680 con 0x7f90d01028e0 2023-09-10T01:00:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.275+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d01a0a50 con 0x7f90d01028e0 2023-09-10T01:00:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.276+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d0109e10 con 0x7f90d01028e0 2023-09-10T01:00:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.278+0000 7f90cd7fa700 1 -- 172.21.15.3:0/3997580476 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f90c000fc30 con 0x7f90d01028e0 2023-09-10T01:00:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.279+0000 7f90cd7fa700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90b80a0640 0x7f90b80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.280+0000 7f90cd7fa700 1 -- 172.21.15.3:0/3997580476 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f90c00ccb10 con 0x7f90d01028e0 2023-09-10T01:00:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.280+0000 7f90cf7fe700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90b80a0640 0x7f90b80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.280+0000 7f90cf7fe700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90b80a0640 0x7f90b80a2b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009bf0 tx=0x7f90c4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.285+0000 7f90cd7fa700 1 -- 172.21.15.3:0/3997580476 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f90c0096ad0 con 0x7f90d01028e0 2023-09-10T01:00:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:38 smithi003 ceph-mon[97654]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:38 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:38 smithi136 ceph-mon[133708]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:39.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:38 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:39.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.503+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90d010e140 con 0x7f90d01028e0 2023-09-10T01:00:39.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.505+0000 7f90cd7fa700 1 -- 172.21.15.3:0/3997580476 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18311 (secure 0 0 0) 0x7f90c009a0f0 con 0x7f90d01028e0 2023-09-10T01:00:39.506 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:39.507 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":78,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T00:59:58.707893+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:00:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.507+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90b80a0640 msgr2=0x7f90b80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.507+0000 7f90d67ff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90b80a0640 0x7f90b80a2b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f90c4009bf0 tx=0x7f90c4009260 comp rx=0 tx=0).stop 2023-09-10T01:00:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.507+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 msgr2=0x7f90d01985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.507+0000 7f90d67ff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d01985b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009990 tx=0x7f90c000f740 comp rx=0 tx=0).stop 2023-09-10T01:00:39.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 shutdown_connections 2023-09-10T01:00:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90b80a0640 0x7f90b80a2b00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d0104470 0x7f90d019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90d01028e0 0x7f90d01985b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 --2- 172.21.15.3:0/3997580476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90d0103ad0 0x7f90d0198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 >> 172.21.15.3:0/3997580476 conn(0x7f90d00fde50 msgr2=0x7f90d0106f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:39.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 shutdown_connections 2023-09-10T01:00:39.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.508+0000 7f90d67ff700 1 -- 172.21.15.3:0/3997580476 wait complete. 2023-09-10T01:00:39.529 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-09-10T01:00:39.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:00:39.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.750+0000 7f4c6099d700 1 -- 172.21.15.3:0/3548734151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c100420 msgr2=0x7f4c5c100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.750+0000 7f4c6099d700 1 --2- 172.21.15.3:0/3548734151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c100420 0x7f4c5c100840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009920 tx=0x7f4c4c009c30 comp rx=0 tx=0).stop 2023-09-10T01:00:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.750+0000 7f4c6099d700 1 -- 172.21.15.3:0/3548734151 shutdown_connections 2023-09-10T01:00:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.751+0000 7f4c6099d700 1 --2- 172.21.15.3:0/3548734151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c101fb0 0x7f4c5c108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.751+0000 7f4c6099d700 1 --2- 172.21.15.3:0/3548734151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c100420 0x7f4c5c100840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.751+0000 7f4c6099d700 1 --2- 172.21.15.3:0/3548734151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c5c101610 0x7f4c5c101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.751+0000 7f4c6099d700 1 -- 172.21.15.3:0/3548734151 >> 172.21.15.3:0/3548734151 conn(0x7f4c5c0fb950 msgr2=0x7f4c5c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.751+0000 7f4c6099d700 1 -- 172.21.15.3:0/3548734151 shutdown_connections 2023-09-10T01:00:39.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.751+0000 7f4c6099d700 1 -- 172.21.15.3:0/3548734151 wait complete. 2023-09-10T01:00:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.752+0000 7f4c6099d700 1 Processor -- start 2023-09-10T01:00:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.752+0000 7f4c6099d700 1 -- start start 2023-09-10T01:00:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c6099d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c100420 0x7f4c5c196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c6099d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c5c101610 0x7f4c5c196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c6099d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c101fb0 0x7f4c5c19d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c6099d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c19ddc0 con 0x7f4c5c100420 2023-09-10T01:00:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c6099d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c19df00 con 0x7f4c5c101fb0 2023-09-10T01:00:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c6099d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c19e090 con 0x7f4c5c101610 2023-09-10T01:00:39.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c5b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c100420 0x7f4c5c196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c5b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c100420 0x7f4c5c196240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60622/0 (socket says 172.21.15.3:60622) 2023-09-10T01:00:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c5bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c101fb0 0x7f4c5c19d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c5b7fe700 1 -- 172.21.15.3:0/2613042785 learned_addr learned my addr 172.21.15.3:0/2613042785 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.753+0000 7f4c5affd700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c5c101610 0x7f4c5c196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5bfff700 1 -- 172.21.15.3:0/2613042785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c5c101610 msgr2=0x7f4c5c196780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5bfff700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c5c101610 0x7f4c5c196780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5bfff700 1 -- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c100420 msgr2=0x7f4c5c196240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5bfff700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c100420 0x7f4c5c196240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5bfff700 1 -- 172.21.15.3:0/2613042785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c500095f0 con 0x7f4c5c101fb0 2023-09-10T01:00:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5affd700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c5c101610 0x7f4c5c196780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5b7fe700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c100420 0x7f4c5c196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c5bfff700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c101fb0 0x7f4c5c19d7f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400efb0 tx=0x7f4c5400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c58ff9700 1 -- 172.21.15.3:0/2613042785 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54003a00 con 0x7f4c5c101fb0 2023-09-10T01:00:39.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c58ff9700 1 -- 172.21.15.3:0/2613042785 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c5400f040 con 0x7f4c5c101fb0 2023-09-10T01:00:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c58ff9700 1 -- 172.21.15.3:0/2613042785 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54004540 con 0x7f4c5c101fb0 2023-09-10T01:00:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c54009c40 con 0x7f4c5c101fb0 2023-09-10T01:00:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.754+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c009520 con 0x7f4c5c101fb0 2023-09-10T01:00:39.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.755+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5c10bc90 con 0x7f4c5c101fb0 2023-09-10T01:00:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.757+0000 7f4c58ff9700 1 -- 172.21.15.3:0/2613042785 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f4c54003bf0 con 0x7f4c5c101fb0 2023-09-10T01:00:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.758+0000 7f4c58ff9700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c440a0640 0x7f4c440a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.759+0000 7f4c58ff9700 1 -- 172.21.15.3:0/2613042785 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4c540cb620 con 0x7f4c5c101fb0 2023-09-10T01:00:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.759+0000 7f4c5b7fe700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c440a0640 0x7f4c440a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.759+0000 7f4c5b7fe700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c440a0640 0x7f4c440a2b00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00c010 tx=0x7f4c4c019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:39.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:39.761+0000 7f4c58ff9700 1 -- 172.21.15.3:0/2613042785 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f4c540955e0 con 0x7f4c5c101fb0 2023-09-10T01:00:40.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.007+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4c5c107470 con 0x7f4c5c101fb0 2023-09-10T01:00:40.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.008+0000 7f4c58ff9700 1 -- 172.21.15.3:0/2613042785 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4c54098c00 con 0x7f4c5c101fb0 2023-09-10T01:00:40.009 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:00:40.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c440a0640 msgr2=0x7f4c440a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c440a0640 0x7f4c440a2b00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00c010 tx=0x7f4c4c019040 comp rx=0 tx=0).stop 2023-09-10T01:00:40.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c101fb0 msgr2=0x7f4c5c19d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c101fb0 0x7f4c5c19d7f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400efb0 tx=0x7f4c5400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 shutdown_connections 2023-09-10T01:00:40.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c5c101fb0 0x7f4c5c19d7f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c5c100420 0x7f4c5c196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c440a0640 0x7f4c440a2b00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 --2- 172.21.15.3:0/2613042785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c5c101610 0x7f4c5c196780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 >> 172.21.15.3:0/2613042785 conn(0x7f4c5c0fb950 msgr2=0x7f4c5c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:40.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 shutdown_connections 2023-09-10T01:00:40.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.010+0000 7f4c6099d700 1 -- 172.21.15.3:0/2613042785 wait complete. 2023-09-10T01:00:40.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-09-10T01:00:40.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 -- 172.21.15.3:0/1106993813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1018b0 msgr2=0x7f37bc101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 --2- 172.21.15.3:0/1106993813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1018b0 0x7f37bc101d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37b00099c0 tx=0x7f37b0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 -- 172.21.15.3:0/1106993813 shutdown_connections 2023-09-10T01:00:40.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 --2- 172.21.15.3:0/1106993813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37bc102250 0x7f37bc108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 --2- 172.21.15.3:0/1106993813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37bc1006c0 0x7f37bc100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 --2- 172.21.15.3:0/1106993813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1018b0 0x7f37bc101d10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 -- 172.21.15.3:0/1106993813 >> 172.21.15.3:0/1106993813 conn(0x7f37bc0fbc30 msgr2=0x7f37bc0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:40.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 -- 172.21.15.3:0/1106993813 shutdown_connections 2023-09-10T01:00:40.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.240+0000 7f37c29ff700 1 -- 172.21.15.3:0/1106993813 wait complete. 2023-09-10T01:00:40.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.241+0000 7f37c29ff700 1 Processor -- start 2023-09-10T01:00:40.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37c29ff700 1 -- start start 2023-09-10T01:00:40.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37c29ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1006c0 0x7f37bc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37c29ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37bc1018b0 0x7f37bc198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37c29ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37bc102250 0x7f37bc19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37c29ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc1a0180 con 0x7f37bc1018b0 2023-09-10T01:00:40.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37c29ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc1a02c0 con 0x7f37bc1006c0 2023-09-10T01:00:40.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37c29ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc1a0450 con 0x7f37bc102250 2023-09-10T01:00:40.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1006c0 0x7f37bc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37bb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37bc1018b0 0x7f37bc198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1006c0 0x7f37bc198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42464/0 (socket says 172.21.15.3:42464) 2023-09-10T01:00:40.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.242+0000 7f37bbfff700 1 -- 172.21.15.3:0/394536003 learned_addr learned my addr 172.21.15.3:0/394536003 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:40.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37c0f9c700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37bc102250 0x7f37bc19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37bbfff700 1 -- 172.21.15.3:0/394536003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37bc102250 msgr2=0x7f37bc19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37bbfff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37bc102250 0x7f37bc19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37bbfff700 1 -- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37bc1018b0 msgr2=0x7f37bc198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37bbfff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37bc1018b0 0x7f37bc198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37bbfff700 1 -- 172.21.15.3:0/394536003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b00095f0 con 0x7f37bc1006c0 2023-09-10T01:00:40.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37c0f9c700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37bc102250 0x7f37bc19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:40.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37bb7fe700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37bc1018b0 0x7f37bc198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:40.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37bbfff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1006c0 0x7f37bc198600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00e970 tx=0x7f37ac00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:40.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37b97fa700 1 -- 172.21.15.3:0/394536003 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac009790 con 0x7f37bc1006c0 2023-09-10T01:00:40.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37b97fa700 1 -- 172.21.15.3:0/394536003 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37ac00eec0 con 0x7f37bc1006c0 2023-09-10T01:00:40.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37bc1a0730 con 0x7f37bc1006c0 2023-09-10T01:00:40.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.243+0000 7f37b97fa700 1 -- 172.21.15.3:0/394536003 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac003cf0 con 0x7f37bc1006c0 2023-09-10T01:00:40.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.244+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37bc1a0c90 con 0x7f37bc1006c0 2023-09-10T01:00:40.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.245+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37bc192d10 con 0x7f37bc1006c0 2023-09-10T01:00:40.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.248+0000 7f37b97fa700 1 -- 172.21.15.3:0/394536003 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f37ac00ccd0 con 0x7f37bc1006c0 2023-09-10T01:00:40.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.249+0000 7f37b97fa700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f37a40a06c0 0x7f37a40a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.250+0000 7f37b97fa700 1 -- 172.21.15.3:0/394536003 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f37ac014070 con 0x7f37bc1006c0 2023-09-10T01:00:40.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.250+0000 7f37bb7fe700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f37a40a06c0 0x7f37a40a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.250+0000 7f37bb7fe700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f37a40a06c0 0x7f37a40a2b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37b000b580 tx=0x7f37b001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:40.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.254+0000 7f37b97fa700 1 -- 172.21.15.3:0/394536003 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f37ac0958b0 con 0x7f37bc1006c0 2023-09-10T01:00:40.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:39 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] up:boot 2023-09-10T01:00:40.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:39 smithi003 ceph-mon[97654]: fsmap 12 up:standby 2023-09-10T01:00:40.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:39 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:00:40.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:39 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/380110639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:39 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:39 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3997580476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:39 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] up:boot 2023-09-10T01:00:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:39 smithi136 ceph-mon[133708]: fsmap 12 up:standby 2023-09-10T01:00:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:39 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:00:40.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:39 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/380110639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:39 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:39 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3997580476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:39 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] up:boot 2023-09-10T01:00:40.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:39 smithi092 ceph-mon[134067]: fsmap 12 up:standby 2023-09-10T01:00:40.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:39 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:00:40.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:39 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/380110639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:39 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:39 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3997580476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:40.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.495+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f37bc10bf20 con 0x7f37bc1006c0 2023-09-10T01:00:40.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.495+0000 7f37b97fa700 1 -- 172.21.15.3:0/394536003 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f37ac098ed0 con 0x7f37bc1006c0 2023-09-10T01:00:40.496 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:40.497 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2023-09-10T01:00:40.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:00:40.497 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2023-09-10T01:00:40.497 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": ".mgr" 2023-09-10T01:00:40.498 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:00:40.498 INFO:teuthology.orchestra.run.smithi003.stdout:] 2023-09-10T01:00:40.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f37a40a06c0 msgr2=0x7f37a40a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f37a40a06c0 0x7f37a40a2b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37b000b580 tx=0x7f37b001a040 comp rx=0 tx=0).stop 2023-09-10T01:00:40.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1006c0 msgr2=0x7f37bc198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1006c0 0x7f37bc198600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00e970 tx=0x7f37ac00ec80 comp rx=0 tx=0).stop 2023-09-10T01:00:40.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 shutdown_connections 2023-09-10T01:00:40.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f37a40a06c0 0x7f37a40a2b80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37bc102250 0x7f37bc19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc1006c0 0x7f37bc198600 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 --2- 172.21.15.3:0/394536003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f37bc1018b0 0x7f37bc198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 >> 172.21.15.3:0/394536003 conn(0x7f37bc0fbc30 msgr2=0x7f37bc107190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:40.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 shutdown_connections 2023-09-10T01:00:40.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.498+0000 7f37c29ff700 1 -- 172.21.15.3:0/394536003 wait complete. 2023-09-10T01:00:40.517 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-09-10T01:00:40.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-09-10T01:00:40.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 -- 172.21.15.3:0/828030193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01006f0 msgr2=0x7f56f0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 --2- 172.21.15.3:0/828030193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01006f0 0x7f56f0100b10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f56e40099c0 tx=0x7f56e4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 -- 172.21.15.3:0/828030193 shutdown_connections 2023-09-10T01:00:40.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 --2- 172.21.15.3:0/828030193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56f0102280 0x7f56f0108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 --2- 172.21.15.3:0/828030193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01006f0 0x7f56f0100b10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 --2- 172.21.15.3:0/828030193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f01018e0 0x7f56f0101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 -- 172.21.15.3:0/828030193 >> 172.21.15.3:0/828030193 conn(0x7f56f00fbc60 msgr2=0x7f56f00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:40.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 -- 172.21.15.3:0/828030193 shutdown_connections 2023-09-10T01:00:40.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.730+0000 7f56f6dff700 1 -- 172.21.15.3:0/828030193 wait complete. 2023-09-10T01:00:40.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.731+0000 7f56f6dff700 1 Processor -- start 2023-09-10T01:00:40.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f6dff700 1 -- start start 2023-09-10T01:00:40.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f6dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56f01006f0 0x7f56f0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f6dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01018e0 0x7f56f0198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f6dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f0102280 0x7f56f019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f6dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f01a01b0 con 0x7f56f01006f0 2023-09-10T01:00:40.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f6dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f01a02f0 con 0x7f56f01018e0 2023-09-10T01:00:40.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f6dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f01a0480 con 0x7f56f0102280 2023-09-10T01:00:40.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f539c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f0102280 0x7f56f019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f4b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56f01006f0 0x7f56f0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f539c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f0102280 0x7f56f019fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49362/0 (socket says 172.21.15.3:49362) 2023-09-10T01:00:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56f539c700 1 -- 172.21.15.3:0/2604741611 learned_addr learned my addr 172.21.15.3:0/2604741611 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.732+0000 7f56effff700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01018e0 0x7f56f0198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f539c700 1 -- 172.21.15.3:0/2604741611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01018e0 msgr2=0x7f56f0198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f539c700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01018e0 0x7f56f0198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f539c700 1 -- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56f01006f0 msgr2=0x7f56f0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f539c700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56f01006f0 0x7f56f0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f539c700 1 -- 172.21.15.3:0/2604741611 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e40095f0 con 0x7f56f0102280 2023-09-10T01:00:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f539c700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f0102280 0x7f56f019fbe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f56e800ef30 tx=0x7f56e800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:40.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56edffb700 1 -- 172.21.15.3:0/2604741611 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8018070 con 0x7f56f0102280 2023-09-10T01:00:40.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f01a0700 con 0x7f56f0102280 2023-09-10T01:00:40.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56edffb700 1 -- 172.21.15.3:0/2604741611 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56e800f040 con 0x7f56f0102280 2023-09-10T01:00:40.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56edffb700 1 -- 172.21.15.3:0/2604741611 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e800cd00 con 0x7f56f0102280 2023-09-10T01:00:40.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.733+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f01a0c60 con 0x7f56f0102280 2023-09-10T01:00:40.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.734+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f0192dc0 con 0x7f56f0102280 2023-09-10T01:00:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.737+0000 7f56edffb700 1 -- 172.21.15.3:0/2604741611 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f56e8003a00 con 0x7f56f0102280 2023-09-10T01:00:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.738+0000 7f56edffb700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56e00a0640 0x7f56e00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.738+0000 7f56edffb700 1 -- 172.21.15.3:0/2604741611 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f56e80cb8f0 con 0x7f56f0102280 2023-09-10T01:00:40.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.738+0000 7f56f4b9b700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56e00a0640 0x7f56e00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:40.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.739+0000 7f56f4b9b700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56e00a0640 0x7f56e00a2b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009990 tx=0x7f56e4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:40.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.740+0000 7f56edffb700 1 -- 172.21.15.3:0/2604741611 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f56e8095930 con 0x7f56f0102280 2023-09-10T01:00:40.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:40.970+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f56f010c220 con 0x7f56f0102280 2023-09-10T01:00:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:40 smithi003 ceph-mon[97654]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2613042785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/394536003' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-09-10T01:00:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:40 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:41.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:40 smithi136 ceph-mon[133708]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2613042785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/394536003' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-09-10T01:00:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:40 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:41.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:40 smithi092 ceph-mon[134067]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:41.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2613042785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:00:41.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/394536003' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-09-10T01:00:41.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:40 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:41.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.984+0000 7f56edffb700 1 -- 172.21.15.3:0/2604741611 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7f56e8098f50 con 0x7f56f0102280 2023-09-10T01:00:41.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56e00a0640 msgr2=0x7f56e00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56e00a0640 0x7f56e00a2b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009990 tx=0x7f56e4005fb0 comp rx=0 tx=0).stop 2023-09-10T01:00:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f0102280 msgr2=0x7f56f019fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f0102280 0x7f56f019fbe0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f56e800ef30 tx=0x7f56e800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:00:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 shutdown_connections 2023-09-10T01:00:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56e00a0640 0x7f56e00a2b00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f0102280 0x7f56f019fbe0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56f01006f0 0x7f56f0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:41.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 --2- 172.21.15.3:0/2604741611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56f01018e0 0x7f56f0198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:41.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 >> 172.21.15.3:0/2604741611 conn(0x7f56f00fbc60 msgr2=0x7f56f01071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:41.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 shutdown_connections 2023-09-10T01:00:41.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:41.987+0000 7f56f6dff700 1 -- 172.21.15.3:0/2604741611 wait complete. 2023-09-10T01:00:41.991 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2023-09-10T01:00:42.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-09-10T01:00:42.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.232+0000 7f6e385ff700 1 -- 172.21.15.3:0/600460423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30101160 msgr2=0x7f6e30103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:42.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.232+0000 7f6e385ff700 1 --2- 172.21.15.3:0/600460423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30101160 0x7f6e30103550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c009990 tx=0x7f6e2c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:42.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.232+0000 7f6e385ff700 1 -- 172.21.15.3:0/600460423 shutdown_connections 2023-09-10T01:00:42.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.232+0000 7f6e385ff700 1 --2- 172.21.15.3:0/600460423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e301063c0 0x7f6e30108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:42.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.232+0000 7f6e385ff700 1 --2- 172.21.15.3:0/600460423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30101160 0x7f6e30103550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:42.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.232+0000 7f6e385ff700 1 --2- 172.21.15.3:0/600460423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e30103a90 0x7f6e30105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:42.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.232+0000 7f6e385ff700 1 -- 172.21.15.3:0/600460423 >> 172.21.15.3:0/600460423 conn(0x7f6e300fab30 msgr2=0x7f6e300fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:42.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.233+0000 7f6e385ff700 1 -- 172.21.15.3:0/600460423 shutdown_connections 2023-09-10T01:00:42.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.233+0000 7f6e385ff700 1 -- 172.21.15.3:0/600460423 wait complete. 2023-09-10T01:00:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 Processor -- start 2023-09-10T01:00:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 -- start start 2023-09-10T01:00:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e30101160 0x7f6e30198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30103a90 0x7f6e30198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e301063c0 0x7f6e3019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e301a0110 con 0x7f6e30101160 2023-09-10T01:00:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e301a0250 con 0x7f6e30103a90 2023-09-10T01:00:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e385ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e301a03e0 con 0x7f6e301063c0 2023-09-10T01:00:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e3639b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e30101160 0x7f6e30198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:42.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e35b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30103a90 0x7f6e30198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:42.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e35b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30103a90 0x7f6e30198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42488/0 (socket says 172.21.15.3:42488) 2023-09-10T01:00:42.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e35b9a700 1 -- 172.21.15.3:0/3978533457 learned_addr learned my addr 172.21.15.3:0/3978533457 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:42.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.234+0000 7f6e36b9c700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e301063c0 0x7f6e3019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:42.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e35b9a700 1 -- 172.21.15.3:0/3978533457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e301063c0 msgr2=0x7f6e3019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:42.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e35b9a700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e301063c0 0x7f6e3019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:42.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e35b9a700 1 -- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e30101160 msgr2=0x7f6e30198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:42.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e35b9a700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e30101160 0x7f6e30198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:42.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e35b9a700 1 -- 172.21.15.3:0/3978533457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c0095f0 con 0x7f6e30103a90 2023-09-10T01:00:42.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e3639b700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e30101160 0x7f6e30198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:42.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e35b9a700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30103a90 0x7f6e30198ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000e9a0 tx=0x7f6e2000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:42.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e277fe700 1 -- 172.21.15.3:0/3978533457 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e20009790 con 0x7f6e30103a90 2023-09-10T01:00:42.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e277fe700 1 -- 172.21.15.3:0/3978533457 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e20010430 con 0x7f6e30103a90 2023-09-10T01:00:42.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e277fe700 1 -- 172.21.15.3:0/3978533457 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e20018680 con 0x7f6e30103a90 2023-09-10T01:00:42.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.235+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e301a06c0 con 0x7f6e30103a90 2023-09-10T01:00:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.236+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e301a0bd0 con 0x7f6e30103a90 2023-09-10T01:00:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.237+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e30107fa0 con 0x7f6e30103a90 2023-09-10T01:00:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.238+0000 7f6e277fe700 1 -- 172.21.15.3:0/3978533457 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f6e200105a0 con 0x7f6e30103a90 2023-09-10T01:00:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.240+0000 7f6e277fe700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e1c0a06c0 0x7f6e1c0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.240+0000 7f6e277fe700 1 -- 172.21.15.3:0/3978533457 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f6e20014070 con 0x7f6e30103a90 2023-09-10T01:00:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.240+0000 7f6e3639b700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e1c0a06c0 0x7f6e1c0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.241+0000 7f6e3639b700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e1c0a06c0 0x7f6e1c0a2b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00b2e0 tx=0x7f6e2c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.244+0000 7f6e277fe700 1 -- 172.21.15.3:0/3978533457 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6e20051e10 con 0x7f6e30103a90 2023-09-10T01:00:42.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:41 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2604741611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-09-10T01:00:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:41 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-09-10T01:00:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:41 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:42.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:41 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2604741611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-09-10T01:00:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:41 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-09-10T01:00:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:41 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:41 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2604741611' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-09-10T01:00:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:41 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-09-10T01:00:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:41 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:42.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.467+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f6e3010bf10 con 0x7f6e30103a90 2023-09-10T01:00:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:42.993+0000 7f6e277fe700 1 -- 172.21.15.3:0/3978533457 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v80) v1 ==== 171+0+0 (secure 0 0 0) 0x7f6e200992e0 con 0x7f6e30103a90 2023-09-10T01:00:43.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.006+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e1c0a06c0 msgr2=0x7f6e1c0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:43.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.006+0000 7f6e385ff700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e1c0a06c0 0x7f6e1c0a2b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00b2e0 tx=0x7f6e2c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.006+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30103a90 msgr2=0x7f6e30198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:43.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.006+0000 7f6e385ff700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30103a90 0x7f6e30198ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000e9a0 tx=0x7f6e2000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.008+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 shutdown_connections 2023-09-10T01:00:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.008+0000 7f6e385ff700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e1c0a06c0 0x7f6e1c0a2b80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.008+0000 7f6e385ff700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e301063c0 0x7f6e3019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.008+0000 7f6e385ff700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e30101160 0x7f6e30198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.008+0000 7f6e385ff700 1 --2- 172.21.15.3:0/3978533457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e30103a90 0x7f6e30198ad0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.008+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 >> 172.21.15.3:0/3978533457 conn(0x7f6e300fab30 msgr2=0x7f6e300ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.011+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 shutdown_connections 2023-09-10T01:00:43.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.011+0000 7f6e385ff700 1 -- 172.21.15.3:0/3978533457 wait complete. 2023-09-10T01:00:43.013 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2023-09-10T01:00:43.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-09-10T01:00:43.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.244+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1854328944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 msgr2=0x7ff1cc101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:43.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.244+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1854328944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc101d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009990 tx=0x7ff1bc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.244+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1854328944 shutdown_connections 2023-09-10T01:00:43.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.244+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1854328944 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1cc102270 0x7ff1cc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.244+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1854328944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1cc1006e0 0x7ff1cc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.244+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1854328944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc101d30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.244+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1854328944 >> 172.21.15.3:0/1854328944 conn(0x7ff1cc0fbc50 msgr2=0x7ff1cc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:43.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.245+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1854328944 shutdown_connections 2023-09-10T01:00:43.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.245+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1854328944 wait complete. 2023-09-10T01:00:43.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 Processor -- start 2023-09-10T01:00:43.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 -- start start 2023-09-10T01:00:43.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1cc1006e0 0x7ff1cc1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:43.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:43.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1cc102270 0x7ff1cc19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:43.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1a0130 con 0x7ff1cc102270 2023-09-10T01:00:43.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1a0270 con 0x7ff1cc1006e0 2023-09-10T01:00:43.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d3bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1a0400 con 0x7ff1cc1018d0 2023-09-10T01:00:43.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d119a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:43.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d219c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1cc102270 0x7ff1cc19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:43.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d119a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49390/0 (socket says 172.21.15.3:49390) 2023-09-10T01:00:43.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d199b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1cc1006e0 0x7ff1cc1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:43.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.246+0000 7ff1d119a700 1 -- 172.21.15.3:0/1975426756 learned_addr learned my addr 172.21.15.3:0/1975426756 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:43.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d119a700 1 -- 172.21.15.3:0/1975426756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1cc1006e0 msgr2=0x7ff1cc1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:43.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d119a700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1cc1006e0 0x7ff1cc1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d119a700 1 -- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1cc102270 msgr2=0x7ff1cc19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:43.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d119a700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1cc102270 0x7ff1cc19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:43.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d119a700 1 -- 172.21.15.3:0/1975426756 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8009520 con 0x7ff1cc1018d0 2023-09-10T01:00:43.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d199b700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1cc1006e0 0x7ff1cc1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:43.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d219c700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1cc102270 0x7ff1cc19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:43.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d119a700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc198af0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc004740 tx=0x7ff1bc009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:43.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1c2ffd700 1 -- 172.21.15.3:0/1975426756 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc01d070 con 0x7ff1cc1018d0 2023-09-10T01:00:43.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1c2ffd700 1 -- 172.21.15.3:0/1975426756 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1bc004830 con 0x7ff1cc1018d0 2023-09-10T01:00:43.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc0095f0 con 0x7ff1cc1018d0 2023-09-10T01:00:43.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.247+0000 7ff1c2ffd700 1 -- 172.21.15.3:0/1975426756 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0053c0 con 0x7ff1cc1018d0 2023-09-10T01:00:43.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.248+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1a08b0 con 0x7ff1cc1018d0 2023-09-10T01:00:43.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.248+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc10bf50 con 0x7ff1cc1018d0 2023-09-10T01:00:43.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.251+0000 7ff1c2ffd700 1 -- 172.21.15.3:0/1975426756 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7ff1bc005560 con 0x7ff1cc1018d0 2023-09-10T01:00:43.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.252+0000 7ff1c2ffd700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1b80a0640 0x7ff1b80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:43.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.252+0000 7ff1c2ffd700 1 -- 172.21.15.3:0/1975426756 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7879+0+0 (secure 0 0 0) 0x7ff1bc0cd0d0 con 0x7ff1cc1018d0 2023-09-10T01:00:43.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.252+0000 7ff1d199b700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1b80a0640 0x7ff1b80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:43.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.253+0000 7ff1d199b700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1b80a0640 0x7ff1b80a2b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009fd0 tx=0x7ff1c8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:43.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.254+0000 7ff1c2ffd700 1 -- 172.21.15.3:0/1975426756 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff1bc096da0 con 0x7ff1cc1018d0 2023-09-10T01:00:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:43 smithi003 ceph-mon[97654]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:43 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-09-10T01:00:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:43 smithi003 ceph-mon[97654]: osdmap e79: 12 total, 12 up, 12 in 2023-09-10T01:00:43.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:43 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:43.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:43 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:43.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:43 smithi136 ceph-mon[133708]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:43 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-09-10T01:00:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:43 smithi136 ceph-mon[133708]: osdmap e79: 12 total, 12 up, 12 in 2023-09-10T01:00:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:43 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:43 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:43.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:43 smithi092 ceph-mon[134067]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:43.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:43 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-09-10T01:00:43.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:43 smithi092 ceph-mon[134067]: osdmap e79: 12 total, 12 up, 12 in 2023-09-10T01:00:43.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:43 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:43.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:43 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:43.529+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff1cc10c110 con 0x7ff1cc1018d0 2023-09-10T01:00:44.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.044+0000 7ff1c2ffd700 1 -- 172.21.15.3:0/1975426756 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff1bc09a3c0 con 0x7ff1cc1018d0 2023-09-10T01:00:44.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1b80a0640 msgr2=0x7ff1b80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:44.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1b80a0640 0x7ff1b80a2b00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009fd0 tx=0x7ff1c8009260 comp rx=0 tx=0).stop 2023-09-10T01:00:44.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 msgr2=0x7ff1cc198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc198af0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc004740 tx=0x7ff1bc009580 comp rx=0 tx=0).stop 2023-09-10T01:00:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 shutdown_connections 2023-09-10T01:00:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1cc102270 0x7ff1cc19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1cc1006e0 0x7ff1cc1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1b80a0640 0x7ff1b80a2b00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 --2- 172.21.15.3:0/1975426756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1cc1018d0 0x7ff1cc198af0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.046+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 >> 172.21.15.3:0/1975426756 conn(0x7ff1cc0fbc50 msgr2=0x7ff1cc0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.047+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 shutdown_connections 2023-09-10T01:00:44.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.047+0000 7ff1d3bff700 1 -- 172.21.15.3:0/1975426756 wait complete. 2023-09-10T01:00:44.052 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 2 and data pool 3 2023-09-10T01:00:44.069 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-09-10T01:00:44.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-09-10T01:00:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/2961931070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 msgr2=0x7f4c34101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/2961931070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 0x7f4c34101d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c280099c0 tx=0x7f4c28009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/2961931070 shutdown_connections 2023-09-10T01:00:44.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/2961931070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c34102280 0x7f4c34108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/2961931070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 0x7f4c34100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/2961931070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 0x7f4c34101d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/2961931070 >> 172.21.15.3:0/2961931070 conn(0x7f4c340fbc60 msgr2=0x7f4c340fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:44.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/2961931070 shutdown_connections 2023-09-10T01:00:44.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.295+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/2961931070 wait complete. 2023-09-10T01:00:44.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 Processor -- start 2023-09-10T01:00:44.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 -- start start 2023-09-10T01:00:44.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 0x7f4c34196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:44.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 0x7f4c34196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:44.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c34102280 0x7f4c3419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:44.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3419ded0 con 0x7f4c341018e0 2023-09-10T01:00:44.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3419e010 con 0x7f4c341006f0 2023-09-10T01:00:44.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.296+0000 7f4c3a5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3419e1a0 con 0x7f4c34102280 2023-09-10T01:00:44.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 0x7f4c34196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:44.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 0x7f4c34196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42528/0 (socket says 172.21.15.3:42528) 2023-09-10T01:00:44.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 -- 172.21.15.3:0/4199659551 learned_addr learned my addr 172.21.15.3:0/4199659551 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:44.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c337fe700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 0x7f4c34196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:44.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c38b9c700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c34102280 0x7f4c3419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:44.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 -- 172.21.15.3:0/4199659551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c34102280 msgr2=0x7f4c3419d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:44.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c34102280 0x7f4c3419d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 -- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 msgr2=0x7f4c34196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:44.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 0x7f4c34196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:44.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 -- 172.21.15.3:0/4199659551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c280095f0 con 0x7f4c341006f0 2023-09-10T01:00:44.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c33fff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 0x7f4c34196350 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400e970 tx=0x7f4c2400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:44.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c317fa700 1 -- 172.21.15.3:0/4199659551 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24009790 con 0x7f4c341006f0 2023-09-10T01:00:44.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c317fa700 1 -- 172.21.15.3:0/4199659551 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c2400eec0 con 0x7f4c341006f0 2023-09-10T01:00:44.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3419e480 con 0x7f4c341006f0 2023-09-10T01:00:44.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.297+0000 7f4c317fa700 1 -- 172.21.15.3:0/4199659551 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24003cf0 con 0x7f4c341006f0 2023-09-10T01:00:44.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.298+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3419e990 con 0x7f4c341006f0 2023-09-10T01:00:44.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.299+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3410bf00 con 0x7f4c341006f0 2023-09-10T01:00:44.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.301+0000 7f4c317fa700 1 -- 172.21.15.3:0/4199659551 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f4c2400ccd0 con 0x7f4c341006f0 2023-09-10T01:00:44.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.302+0000 7f4c317fa700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c1c0a0640 0x7f4c1c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:44.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.302+0000 7f4c317fa700 1 -- 172.21.15.3:0/4199659551 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4c24014070 con 0x7f4c341006f0 2023-09-10T01:00:44.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.302+0000 7f4c337fe700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c1c0a0640 0x7f4c1c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:44.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.303+0000 7f4c337fe700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c1c0a0640 0x7f4c1c0a2b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:44.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.304+0000 7f4c317fa700 1 -- 172.21.15.3:0/4199659551 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f4c24095a70 con 0x7f4c341006f0 2023-09-10T01:00:44.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3978533457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-09-10T01:00:44.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-09-10T01:00:44.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-09-10T01:00:44.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: osdmap e80: 12 total, 12 up, 12 in 2023-09-10T01:00:44.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:44.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:44.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:44.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1975426756' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-09-10T01:00:44.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-09-10T01:00:44.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:44 smithi003 ceph-mon[97654]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-09-10T01:00:44.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3978533457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-09-10T01:00:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-09-10T01:00:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-09-10T01:00:44.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: osdmap e80: 12 total, 12 up, 12 in 2023-09-10T01:00:44.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:44.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:44.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:44.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1975426756' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-09-10T01:00:44.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-09-10T01:00:44.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:44 smithi136 ceph-mon[133708]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-09-10T01:00:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3978533457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-09-10T01:00:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-09-10T01:00:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-09-10T01:00:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: osdmap e80: 12 total, 12 up, 12 in 2023-09-10T01:00:44.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:44.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:44.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:44.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:00:44.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1975426756' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-09-10T01:00:44.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-09-10T01:00:44.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:44 smithi092 ceph-mon[134067]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-09-10T01:00:44.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:44.538+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f4c3419ec10 con 0x7f4c341006f0 2023-09-10T01:00:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.017+0000 7f4c317fa700 1 -- 172.21.15.3:0/4199659551 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v82) v1 ==== 157+0+0 (secure 0 0 0) 0x7f4c24099090 con 0x7f4c341006f0 2023-09-10T01:00:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c1c0a0640 msgr2=0x7f4c1c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c1c0a0640 0x7f4c1c0a2b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4c28000c00 tx=0x7f4c28019040 comp rx=0 tx=0).stop 2023-09-10T01:00:45.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 msgr2=0x7f4c34196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:45.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 0x7f4c34196350 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400e970 tx=0x7f4c2400ec80 comp rx=0 tx=0).stop 2023-09-10T01:00:45.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 shutdown_connections 2023-09-10T01:00:45.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c1c0a0640 0x7f4c1c0a2b00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c34102280 0x7f4c3419d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c341006f0 0x7f4c34196350 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 --2- 172.21.15.3:0/4199659551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c341018e0 0x7f4c34196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 >> 172.21.15.3:0/4199659551 conn(0x7f4c340fbc60 msgr2=0x7f4c340fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 shutdown_connections 2023-09-10T01:00:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.021+0000 7f4c3a5ff700 1 -- 172.21.15.3:0/4199659551 wait complete. 2023-09-10T01:00:45.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-09-10T01:00:45.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 -- 172.21.15.3:0/2753705155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 msgr2=0x7fafd8100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:45.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 --2- 172.21.15.3:0/2753705155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8100a50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0099c0 tx=0x7fafcc009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 -- 172.21.15.3:0/2753705155 shutdown_connections 2023-09-10T01:00:45.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 --2- 172.21.15.3:0/2753705155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafd81021c0 0x7fafd8108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 --2- 172.21.15.3:0/2753705155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8100a50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 --2- 172.21.15.3:0/2753705155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafd8101820 0x7fafd8101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 -- 172.21.15.3:0/2753705155 >> 172.21.15.3:0/2753705155 conn(0x7fafd80fbbc0 msgr2=0x7fafd80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:45.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 -- 172.21.15.3:0/2753705155 shutdown_connections 2023-09-10T01:00:45.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.268+0000 7fafdefff700 1 -- 172.21.15.3:0/2753705155 wait complete. 2023-09-10T01:00:45.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.270+0000 7fafdefff700 1 Processor -- start 2023-09-10T01:00:45.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.271+0000 7fafdefff700 1 -- start start 2023-09-10T01:00:45.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdefff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:45.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdefff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafd8101820 0x7fafd8196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:45.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdefff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafd81021c0 0x7fafd819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:45.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdefff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd819de80 con 0x7fafd81021c0 2023-09-10T01:00:45.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdefff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd819dfc0 con 0x7fafd8100630 2023-09-10T01:00:45.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdefff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd819e150 con 0x7fafd8101820 2023-09-10T01:00:45.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdcd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:45.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdd59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafd81021c0 0x7fafd819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:45.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdcd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39304/0 (socket says 172.21.15.3:39304) 2023-09-10T01:00:45.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.272+0000 7fafdcd9b700 1 -- 172.21.15.3:0/1303700711 learned_addr learned my addr 172.21.15.3:0/1303700711 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:45.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdcd9b700 1 -- 172.21.15.3:0/1303700711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafd8101820 msgr2=0x7fafd8196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdcd9b700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafd8101820 0x7fafd8196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdcd9b700 1 -- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafd81021c0 msgr2=0x7fafd819d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdcd9b700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafd81021c0 0x7fafd819d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdcd9b700 1 -- 172.21.15.3:0/1303700711 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc0095f0 con 0x7fafd8100630 2023-09-10T01:00:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdd59c700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafd81021c0 0x7fafd819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdcd9b700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8196300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0058e0 tx=0x7fafcc00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafd5ffb700 1 -- 172.21.15.3:0/1303700711 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc01d070 con 0x7fafd8100630 2023-09-10T01:00:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafd5ffb700 1 -- 172.21.15.3:0/1303700711 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafcc005510 con 0x7fafd8100630 2023-09-10T01:00:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafd5ffb700 1 -- 172.21.15.3:0/1303700711 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc004b10 con 0x7fafd8100630 2023-09-10T01:00:45.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.273+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd819e3d0 con 0x7fafd8100630 2023-09-10T01:00:45.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.274+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd819e7f0 con 0x7fafd8100630 2023-09-10T01:00:45.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.274+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd810be60 con 0x7fafd8100630 2023-09-10T01:00:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.278+0000 7fafd5ffb700 1 -- 172.21.15.3:0/1303700711 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fafcc00fbb0 con 0x7fafd8100630 2023-09-10T01:00:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.280+0000 7fafd5ffb700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fafc80a06c0 0x7fafc80a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.280+0000 7fafd5ffb700 1 -- 172.21.15.3:0/1303700711 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fafcc0cced0 con 0x7fafd8100630 2023-09-10T01:00:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.280+0000 7fafd7fff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fafc80a06c0 0x7fafc80a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.281+0000 7fafd7fff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fafc80a06c0 0x7fafc80a2b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005950 tx=0x7fafc40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.281+0000 7fafd5ffb700 1 -- 172.21.15.3:0/1303700711 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fafcc096a50 con 0x7fafd8100630 2023-09-10T01:00:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: pgmap v274: 97 pgs: 21 creating+peering, 75 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: osdmap e81: 12 total, 12 up, 12 in 2023-09-10T01:00:45.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-09-10T01:00:45.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-09-10T01:00:45.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-09-10T01:00:45.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-09-10T01:00:45.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4199659551' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-09-10T01:00:45.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-09-10T01:00:45.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] up:active 2023-09-10T01:00:45.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:45 smithi003 ceph-mon[97654]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:00:45.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: pgmap v274: 97 pgs: 21 creating+peering, 75 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:45.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: osdmap e81: 12 total, 12 up, 12 in 2023-09-10T01:00:45.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-09-10T01:00:45.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-09-10T01:00:45.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-09-10T01:00:45.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-09-10T01:00:45.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4199659551' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-09-10T01:00:45.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-09-10T01:00:45.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] up:active 2023-09-10T01:00:45.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:45 smithi136 ceph-mon[133708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:00:45.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: pgmap v274: 97 pgs: 21 creating+peering, 75 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:00:45.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: osdmap e81: 12 total, 12 up, 12 in 2023-09-10T01:00:45.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-09-10T01:00:45.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-09-10T01:00:45.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-09-10T01:00:45.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-09-10T01:00:45.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4199659551' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-09-10T01:00:45.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-09-10T01:00:45.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] up:active 2023-09-10T01:00:45.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:45 smithi092 ceph-mon[134067]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:00:45.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:45.499+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fafd8107510 con 0x7fafd8100630 2023-09-10T01:00:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-09-10T01:00:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: osdmap e82: 12 total, 12 up, 12 in 2023-09-10T01:00:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1303700711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-09-10T01:00:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-09-10T01:00:46.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:46 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-09-10T01:00:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: osdmap e82: 12 total, 12 up, 12 in 2023-09-10T01:00:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1303700711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-09-10T01:00:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-09-10T01:00:46.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:46 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-09-10T01:00:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: osdmap e82: 12 total, 12 up, 12 in 2023-09-10T01:00:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1303700711' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-09-10T01:00:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-09-10T01:00:46.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:46.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:46 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.036+0000 7fafd5ffb700 1 -- 172.21.15.3:0/1303700711 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v84) v1 ==== 228+0+0 (secure 0 0 0) 0x7fafcc09a070 con 0x7fafd8100630 2023-09-10T01:00:47.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fafc80a06c0 msgr2=0x7fafc80a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:47.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fafc80a06c0 0x7fafc80a2b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005950 tx=0x7fafc40058e0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 msgr2=0x7fafd8196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:47.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8196300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0058e0 tx=0x7fafcc00f690 comp rx=0 tx=0).stop 2023-09-10T01:00:47.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 shutdown_connections 2023-09-10T01:00:47.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafd81021c0 0x7fafd819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd8100630 0x7fafd8196300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.041+0000 7fafdefff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fafc80a06c0 0x7fafc80a2b80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.042+0000 7fafdefff700 1 --2- 172.21.15.3:0/1303700711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafd8101820 0x7fafd8196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.042+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 >> 172.21.15.3:0/1303700711 conn(0x7fafd80fbbc0 msgr2=0x7fafd80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:47.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.042+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 shutdown_connections 2023-09-10T01:00:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.042+0000 7fafdefff700 1 -- 172.21.15.3:0/1303700711 wait complete. 2023-09-10T01:00:47.045 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data_ec' created 2023-09-10T01:00:47.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-09-10T01:00:47.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 -- 172.21.15.3:0/4279293844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c1018e0 msgr2=0x7f9b6c101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:47.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 --2- 172.21.15.3:0/4279293844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c101d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009990 tx=0x7f9b60009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 -- 172.21.15.3:0/4279293844 shutdown_connections 2023-09-10T01:00:47.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 --2- 172.21.15.3:0/4279293844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c102280 0x7f9b6c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 --2- 172.21.15.3:0/4279293844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b6c1006f0 0x7f9b6c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 --2- 172.21.15.3:0/4279293844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c101d40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 -- 172.21.15.3:0/4279293844 >> 172.21.15.3:0/4279293844 conn(0x7f9b6c0fbc60 msgr2=0x7f9b6c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:47.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 -- 172.21.15.3:0/4279293844 shutdown_connections 2023-09-10T01:00:47.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.278+0000 7f9b71dff700 1 -- 172.21.15.3:0/4279293844 wait complete. 2023-09-10T01:00:47.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 Processor -- start 2023-09-10T01:00:47.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 -- start start 2023-09-10T01:00:47.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b6c1006f0 0x7f9b6c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:47.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:47.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c102280 0x7f9b6c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:47.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c1a0130 con 0x7f9b6c1018e0 2023-09-10T01:00:47.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c1a0270 con 0x7f9b6c1006f0 2023-09-10T01:00:47.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b71dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c1a0400 con 0x7f9b6c102280 2023-09-10T01:00:47.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.279+0000 7f9b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:47.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44624/0 (socket says 172.21.15.3:44624) 2023-09-10T01:00:47.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b6c1006f0 0x7f9b6c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:47.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 -- 172.21.15.3:0/1598759752 learned_addr learned my addr 172.21.15.3:0/1598759752 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:47.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6bfff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c102280 0x7f9b6c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:47.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 -- 172.21.15.3:0/1598759752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c102280 msgr2=0x7f9b6c19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:47.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c102280 0x7f9b6c19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 -- 172.21.15.3:0/1598759752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b6c1006f0 msgr2=0x7f9b6c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:47.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b6c1006f0 0x7f9b6c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:47.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 -- 172.21.15.3:0/1598759752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b600095f0 con 0x7f9b6c1018e0 2023-09-10T01:00:47.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6bfff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c102280 0x7f9b6c19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:47.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b6affd700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c198af0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9b60004740 tx=0x7f9b60009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:47.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b68ff9700 1 -- 172.21.15.3:0/1598759752 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6001d070 con 0x7f9b6c1018e0 2023-09-10T01:00:47.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c1a0680 con 0x7f9b6c1018e0 2023-09-10T01:00:47.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.280+0000 7f9b68ff9700 1 -- 172.21.15.3:0/1598759752 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b60004830 con 0x7f9b6c1018e0 2023-09-10T01:00:47.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.281+0000 7f9b68ff9700 1 -- 172.21.15.3:0/1598759752 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b60005450 con 0x7f9b6c1018e0 2023-09-10T01:00:47.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.281+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c1a0a50 con 0x7f9b6c1018e0 2023-09-10T01:00:47.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.282+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c10bf60 con 0x7f9b6c1018e0 2023-09-10T01:00:47.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.284+0000 7f9b68ff9700 1 -- 172.21.15.3:0/1598759752 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f9b600055d0 con 0x7f9b6c1018e0 2023-09-10T01:00:47.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.286+0000 7f9b68ff9700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9b540a0640 0x7f9b540a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:47.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.287+0000 7f9b68ff9700 1 -- 172.21.15.3:0/1598759752 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f9b600cd4a0 con 0x7f9b6c1018e0 2023-09-10T01:00:47.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.287+0000 7f9b6b7fe700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9b540a0640 0x7f9b540a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:47.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.287+0000 7f9b6b7fe700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9b540a0640 0x7f9b540a2b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c101740 tx=0x7f9b5c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:47.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.290+0000 7f9b68ff9700 1 -- 172.21.15.3:0/1598759752 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9b60096e30 con 0x7f9b6c1018e0 2023-09-10T01:00:47.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: pgmap v277: 97 pgs: 21 creating+peering, 42 unknown, 34 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 5 op/s 2023-09-10T01:00:47.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2023-09-10T01:00:47.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: Cluster is now healthy 2023-09-10T01:00:47.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: osdmap e83: 12 total, 12 up, 12 in 2023-09-10T01:00:47.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:47.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:47.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:47 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: pgmap v277: 97 pgs: 21 creating+peering, 42 unknown, 34 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 5 op/s 2023-09-10T01:00:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2023-09-10T01:00:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: Cluster is now healthy 2023-09-10T01:00:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: osdmap e83: 12 total, 12 up, 12 in 2023-09-10T01:00:47.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:47.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:47.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:47 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: pgmap v277: 97 pgs: 21 creating+peering, 42 unknown, 34 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 5 op/s 2023-09-10T01:00:47.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2023-09-10T01:00:47.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: Cluster is now healthy 2023-09-10T01:00:47.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: osdmap e83: 12 total, 12 up, 12 in 2023-09-10T01:00:47.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:00:47.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:00:47.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:47 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:00:47.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:47.502+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f9b6c10c140 con 0x7f9b6c1018e0 2023-09-10T01:00:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.036+0000 7f9b68ff9700 1 -- 172.21.15.3:0/1598759752 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9b6009a450 con 0x7f9b6c1018e0 2023-09-10T01:00:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.040+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9b540a0640 msgr2=0x7f9b540a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.040+0000 7f9b71dff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9b540a0640 0x7f9b540a2b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c101740 tx=0x7f9b5c009260 comp rx=0 tx=0).stop 2023-09-10T01:00:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.040+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c1018e0 msgr2=0x7f9b6c198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:48.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.040+0000 7f9b71dff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c198af0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9b60004740 tx=0x7f9b60009580 comp rx=0 tx=0).stop 2023-09-10T01:00:48.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.041+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 shutdown_connections 2023-09-10T01:00:48.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.041+0000 7f9b71dff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9b540a0640 0x7f9b540a2b00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.041+0000 7f9b71dff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b6c102280 0x7f9b6c19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.041+0000 7f9b71dff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b6c1006f0 0x7f9b6c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.041+0000 7f9b71dff700 1 --2- 172.21.15.3:0/1598759752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b6c1018e0 0x7f9b6c198af0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.041+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 >> 172.21.15.3:0/1598759752 conn(0x7f9b6c0fbc60 msgr2=0x7f9b6c0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:48.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.041+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 shutdown_connections 2023-09-10T01:00:48.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.042+0000 7f9b71dff700 1 -- 172.21.15.3:0/1598759752 wait complete. 2023-09-10T01:00:48.045 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 4 allow_ec_overwrites to true 2023-09-10T01:00:48.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-09-10T01:00:48.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 -- 172.21.15.3:0/66362384 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81006f0 msgr2=0x7f7fa8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:48.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/66362384 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81006f0 0x7f7fa8100b10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7f980099c0 tx=0x7f7f98009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 -- 172.21.15.3:0/66362384 shutdown_connections 2023-09-10T01:00:48.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/66362384 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fa8102280 0x7f7fa8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/66362384 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81006f0 0x7f7fa8100b10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/66362384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fa81018e0 0x7f7fa8101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 -- 172.21.15.3:0/66362384 >> 172.21.15.3:0/66362384 conn(0x7f7fa80fbc60 msgr2=0x7f7fa80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:48.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.283+0000 7f7fad3ff700 1 -- 172.21.15.3:0/66362384 shutdown_connections 2023-09-10T01:00:48.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.284+0000 7f7fad3ff700 1 -- 172.21.15.3:0/66362384 wait complete. 2023-09-10T01:00:48.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.284+0000 7f7fad3ff700 1 Processor -- start 2023-09-10T01:00:48.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fad3ff700 1 -- start start 2023-09-10T01:00:48.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fad3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fa81006f0 0x7f7fa81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:48.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fad3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81018e0 0x7f7fa8198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:48.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fad3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fa8102280 0x7f7fa819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:48.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fad3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa81a0150 con 0x7f7fa81018e0 2023-09-10T01:00:48.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fad3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa81a0290 con 0x7f7fa8102280 2023-09-10T01:00:48.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fad3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa81a0420 con 0x7f7fa81006f0 2023-09-10T01:00:48.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fa67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81018e0 0x7f7fa8198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:48.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fa67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81018e0 0x7f7fa8198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44636/0 (socket says 172.21.15.3:44636) 2023-09-10T01:00:48.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fa67fc700 1 -- 172.21.15.3:0/1596659725 learned_addr learned my addr 172.21.15.3:0/1596659725 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:48.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fa77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fa8102280 0x7f7fa819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:48.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.285+0000 7f7fa6ffd700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fa81006f0 0x7f7fa81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:48.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fa67fc700 1 -- 172.21.15.3:0/1596659725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fa81006f0 msgr2=0x7f7fa81985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:48.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fa67fc700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fa81006f0 0x7f7fa81985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fa67fc700 1 -- 172.21.15.3:0/1596659725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fa8102280 msgr2=0x7f7fa819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:48.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fa67fc700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fa8102280 0x7f7fa819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:48.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fa67fc700 1 -- 172.21.15.3:0/1596659725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f980095f0 con 0x7f7fa81018e0 2023-09-10T01:00:48.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fa67fc700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81018e0 0x7f7fa8198b10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00e9a0 tx=0x7f7f9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:48.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7f8ffff700 1 -- 172.21.15.3:0/1596659725 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c009790 con 0x7f7fa81018e0 2023-09-10T01:00:48.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa81a0700 con 0x7f7fa81018e0 2023-09-10T01:00:48.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7f8ffff700 1 -- 172.21.15.3:0/1596659725 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f9c010430 con 0x7f7fa81018e0 2023-09-10T01:00:48.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7f8ffff700 1 -- 172.21.15.3:0/1596659725 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c018680 con 0x7f7fa81018e0 2023-09-10T01:00:48.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.286+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa81a0c10 con 0x7f7fa81018e0 2023-09-10T01:00:48.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.287+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa8192d40 con 0x7f7fa81018e0 2023-09-10T01:00:48.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.289+0000 7f7f8ffff700 1 -- 172.21.15.3:0/1596659725 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f7f9c0105a0 con 0x7f7fa81018e0 2023-09-10T01:00:48.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.291+0000 7f7f8ffff700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7f900a0640 0x7f7f900a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:48.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.291+0000 7f7f8ffff700 1 -- 172.21.15.3:0/1596659725 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f7f9c014070 con 0x7f7fa81018e0 2023-09-10T01:00:48.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.291+0000 7f7fa6ffd700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7f900a0640 0x7f7f900a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:48.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.292+0000 7f7fa6ffd700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7f900a0640 0x7f7f900a2b00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7f98009990 tx=0x7f7f98005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:48.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.295+0000 7f7f8ffff700 1 -- 172.21.15.3:0/1596659725 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7f9c051e10 con 0x7f7fa81018e0 2023-09-10T01:00:48.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:48 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-09-10T01:00:48.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:48 smithi003 ceph-mon[97654]: osdmap e84: 12 total, 12 up, 12 in 2023-09-10T01:00:48.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:48 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1598759752' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-09-10T01:00:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:48 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-09-10T01:00:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:48 smithi136 ceph-mon[133708]: osdmap e84: 12 total, 12 up, 12 in 2023-09-10T01:00:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:48 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1598759752' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-09-10T01:00:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:48 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-09-10T01:00:48.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:48 smithi092 ceph-mon[134067]: osdmap e84: 12 total, 12 up, 12 in 2023-09-10T01:00:48.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:48 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1598759752' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-09-10T01:00:48.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:48.558+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f7fa810bf50 con 0x7f7fa81018e0 2023-09-10T01:00:49.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.039+0000 7f7f8ffff700 1 -- 172.21.15.3:0/1596659725 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7f7f9c0d0070 con 0x7f7fa81018e0 2023-09-10T01:00:49.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.042+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7f900a0640 msgr2=0x7f7f900a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.042+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7f900a0640 0x7f7f900a2b00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7f98009990 tx=0x7f7f98005fb0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.042+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81018e0 msgr2=0x7f7fa8198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.042+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81018e0 0x7f7fa8198b10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00e9a0 tx=0x7f7f9c00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 shutdown_connections 2023-09-10T01:00:49.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7f900a0640 0x7f7f900a2b00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fa81006f0 0x7f7fa81985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fa8102280 0x7f7fa819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 --2- 172.21.15.3:0/1596659725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fa81018e0 0x7f7fa8198b10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 >> 172.21.15.3:0/1596659725 conn(0x7f7fa80fbc60 msgr2=0x7f7fa81071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:49.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 shutdown_connections 2023-09-10T01:00:49.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.043+0000 7f7fad3ff700 1 -- 172.21.15.3:0/1596659725 wait complete. 2023-09-10T01:00:49.046 INFO:teuthology.orchestra.run.smithi003.stderr:added data pool 4 to fsmap 2023-09-10T01:00:49.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:49.216 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:00:48 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: 2023-09-10T01:00:48.955+0000 7f1e948f8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-09-10T01:00:49.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:49 smithi092 ceph-mon[134067]: pgmap v280: 129 pgs: 14 creating+peering, 36 unknown, 79 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-09-10T01:00:49.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:49 smithi092 ceph-mon[134067]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-09-10T01:00:49.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:49 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1598759752' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-09-10T01:00:49.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:49 smithi092 ceph-mon[134067]: osdmap e85: 12 total, 12 up, 12 in 2023-09-10T01:00:49.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:49 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1596659725' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-09-10T01:00:49.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 -- 172.21.15.3:0/1898114357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d8102250 msgr2=0x7f76d8108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 --2- 172.21.15.3:0/1898114357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d8102250 0x7f76d8108b00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f76d000b0d0 tx=0x7f76d000b3e0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 -- 172.21.15.3:0/1898114357 shutdown_connections 2023-09-10T01:00:49.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 --2- 172.21.15.3:0/1898114357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d8102250 0x7f76d8108b00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 --2- 172.21.15.3:0/1898114357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76d81006c0 0x7f76d8100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 --2- 172.21.15.3:0/1898114357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76d81018b0 0x7f76d8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 -- 172.21.15.3:0/1898114357 >> 172.21.15.3:0/1898114357 conn(0x7f76d80fbc30 msgr2=0x7f76d80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:49.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.273+0000 7f76df3ff700 1 -- 172.21.15.3:0/1898114357 shutdown_connections 2023-09-10T01:00:49.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.274+0000 7f76df3ff700 1 -- 172.21.15.3:0/1898114357 wait complete. 2023-09-10T01:00:49.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.274+0000 7f76df3ff700 1 Processor -- start 2023-09-10T01:00:49.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76df3ff700 1 -- start start 2023-09-10T01:00:49.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76df3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76d81006c0 0x7f76d8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76df3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d81018b0 0x7f76d8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76df3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76d8102250 0x7f76d819fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76df3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d81a02a0 con 0x7f76d81018b0 2023-09-10T01:00:49.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76df3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d81a03e0 con 0x7f76d8102250 2023-09-10T01:00:49.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76df3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d81a0570 con 0x7f76d81006c0 2023-09-10T01:00:49.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76dc99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d81018b0 0x7f76d8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76dc99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d81018b0 0x7f76d8198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44656/0 (socket says 172.21.15.3:44656) 2023-09-10T01:00:49.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76dc99a700 1 -- 172.21.15.3:0/4235578162 learned_addr learned my addr 172.21.15.3:0/4235578162 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:49.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76dd19b700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76d81006c0 0x7f76d8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.275+0000 7f76dd99c700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76d8102250 0x7f76d819fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76dc99a700 1 -- 172.21.15.3:0/4235578162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76d81006c0 msgr2=0x7f76d8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76dc99a700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76d81006c0 0x7f76d8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76dc99a700 1 -- 172.21.15.3:0/4235578162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76d8102250 msgr2=0x7f76d819fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76dc99a700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76d8102250 0x7f76d819fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76dc99a700 1 -- 172.21.15.3:0/4235578162 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0009d00 con 0x7f76d81018b0 2023-09-10T01:00:49.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76dc99a700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d81018b0 0x7f76d8198c60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f76c800e9a0 tx=0x7f76c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:49.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76ce7fc700 1 -- 172.21.15.3:0/4235578162 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8009790 con 0x7f76d81018b0 2023-09-10T01:00:49.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d81a0850 con 0x7f76d81018b0 2023-09-10T01:00:49.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76ce7fc700 1 -- 172.21.15.3:0/4235578162 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76c8010430 con 0x7f76d81018b0 2023-09-10T01:00:49.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.276+0000 7f76ce7fc700 1 -- 172.21.15.3:0/4235578162 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8018680 con 0x7f76d81018b0 2023-09-10T01:00:49.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.277+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d81a0d60 con 0x7f76d81018b0 2023-09-10T01:00:49.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.278+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d810bf20 con 0x7f76d81018b0 2023-09-10T01:00:49.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.279+0000 7f76ce7fc700 1 -- 172.21.15.3:0/4235578162 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f76c80105a0 con 0x7f76d81018b0 2023-09-10T01:00:49.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.281+0000 7f76ce7fc700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76c40a0640 0x7f76c40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.281+0000 7f76ce7fc700 1 -- 172.21.15.3:0/4235578162 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f76c8014070 con 0x7f76d81018b0 2023-09-10T01:00:49.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.281+0000 7f76dd19b700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76c40a0640 0x7f76c40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.282+0000 7f76dd19b700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76c40a0640 0x7f76c40a2b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f76d8101710 tx=0x7f76d4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:49.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.284+0000 7f76ce7fc700 1 -- 172.21.15.3:0/4235578162 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f76c8051e10 con 0x7f76d81018b0 2023-09-10T01:00:49.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:49 smithi003 ceph-mon[97654]: pgmap v280: 129 pgs: 14 creating+peering, 36 unknown, 79 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-09-10T01:00:49.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:49 smithi003 ceph-mon[97654]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-09-10T01:00:49.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:49 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1598759752' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-09-10T01:00:49.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:49 smithi003 ceph-mon[97654]: osdmap e85: 12 total, 12 up, 12 in 2023-09-10T01:00:49.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:49 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1596659725' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-09-10T01:00:49.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:49 smithi136 ceph-mon[133708]: pgmap v280: 129 pgs: 14 creating+peering, 36 unknown, 79 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-09-10T01:00:49.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:49 smithi136 ceph-mon[133708]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-09-10T01:00:49.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:49 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1598759752' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-09-10T01:00:49.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:49 smithi136 ceph-mon[133708]: osdmap e85: 12 total, 12 up, 12 in 2023-09-10T01:00:49.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:49 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1596659725' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-09-10T01:00:49.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.550+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d81a0f80 con 0x7f76d81018b0 2023-09-10T01:00:49.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.551+0000 7f76ce7fc700 1 -- 172.21.15.3:0/4235578162 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+11231 (secure 0 0 0) 0x7f76c80957d0 con 0x7f76d81018b0 2023-09-10T01:00:49.552 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:49.552 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24634,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24647,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:00:48.560325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-09-10T01:00:49.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76c40a0640 msgr2=0x7f76c40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76c40a0640 0x7f76c40a2b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f76d8101710 tx=0x7f76d4009260 comp rx=0 tx=0).stop 2023-09-10T01:00:49.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d81018b0 msgr2=0x7f76d8198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d81018b0 0x7f76d8198c60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f76c800e9a0 tx=0x7f76c800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 shutdown_connections 2023-09-10T01:00:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f76c40a0640 0x7f76c40a2b00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76d81006c0 0x7f76d8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76d8102250 0x7f76d819fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 --2- 172.21.15.3:0/4235578162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76d81018b0 0x7f76d8198c60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 >> 172.21.15.3:0/4235578162 conn(0x7f76d80fbc30 msgr2=0x7f76d8107150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 shutdown_connections 2023-09-10T01:00:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.554+0000 7f76df3ff700 1 -- 172.21.15.3:0/4235578162 wait complete. 2023-09-10T01:00:49.557 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 16 2023-09-10T01:00:49.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:00:49.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 -- 172.21.15.3:0/3342580828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5280745e0 msgr2=0x7fa528072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 --2- 172.21.15.3:0/3342580828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5280745e0 0x7fa528072a30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa524009990 tx=0x7fa524009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 -- 172.21.15.3:0/3342580828 shutdown_connections 2023-09-10T01:00:49.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 --2- 172.21.15.3:0/3342580828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 0x7fa52810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 --2- 172.21.15.3:0/3342580828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5280745e0 0x7fa528072a30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 --2- 172.21.15.3:0/3342580828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa528073000 0x7fa528073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 -- 172.21.15.3:0/3342580828 >> 172.21.15.3:0/3342580828 conn(0x7fa5280fb980 msgr2=0x7fa5280fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:49.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 -- 172.21.15.3:0/3342580828 shutdown_connections 2023-09-10T01:00:49.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.777+0000 7fa52f59d700 1 -- 172.21.15.3:0/3342580828 wait complete. 2023-09-10T01:00:49.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.778+0000 7fa52f59d700 1 Processor -- start 2023-09-10T01:00:49.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.778+0000 7fa52f59d700 1 -- start start 2023-09-10T01:00:49.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52f59d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa528073000 0x7fa528198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52f59d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5280745e0 0x7fa528198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52f59d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 0x7fa52819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52f59d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52819fe80 con 0x7fa5281082e0 2023-09-10T01:00:49.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52f59d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52819ffc0 con 0x7fa528073000 2023-09-10T01:00:49.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52f59d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5281a0150 con 0x7fa5280745e0 2023-09-10T01:00:49.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52ed9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 0x7fa52819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52ed9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 0x7fa52819f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44672/0 (socket says 172.21.15.3:44672) 2023-09-10T01:00:49.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52ed9c700 1 -- 172.21.15.3:0/460943808 learned_addr learned my addr 172.21.15.3:0/460943808 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:49.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52dd9a700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5280745e0 0x7fa528198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.779+0000 7fa52e59b700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa528073000 0x7fa528198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52ed9c700 1 -- 172.21.15.3:0/460943808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5280745e0 msgr2=0x7fa528198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52ed9c700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5280745e0 0x7fa528198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52ed9c700 1 -- 172.21.15.3:0/460943808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa528073000 msgr2=0x7fa528198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:49.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52ed9c700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa528073000 0x7fa528198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:49.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52ed9c700 1 -- 172.21.15.3:0/460943808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518009520 con 0x7fa5281082e0 2023-09-10T01:00:49.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52ed9c700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 0x7fa52819f8b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa52000ef30 tx=0x7fa52000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:49.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa51f7fe700 1 -- 172.21.15.3:0/460943808 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520018070 con 0x7fa5281082e0 2023-09-10T01:00:49.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa51f7fe700 1 -- 172.21.15.3:0/460943808 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa52000f040 con 0x7fa5281082e0 2023-09-10T01:00:49.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa51f7fe700 1 -- 172.21.15.3:0/460943808 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52000cd00 con 0x7fa5281082e0 2023-09-10T01:00:49.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5240095f0 con 0x7fa5281082e0 2023-09-10T01:00:49.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.780+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5281a0700 con 0x7fa5281082e0 2023-09-10T01:00:49.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.781+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5281069e0 con 0x7fa5281082e0 2023-09-10T01:00:49.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.783+0000 7fa51f7fe700 1 -- 172.21.15.3:0/460943808 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fa520003a00 con 0x7fa5281082e0 2023-09-10T01:00:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.785+0000 7fa51f7fe700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa5140a06c0 0x7fa5140a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.785+0000 7fa51f7fe700 1 -- 172.21.15.3:0/460943808 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa5200cc250 con 0x7fa5281082e0 2023-09-10T01:00:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.785+0000 7fa52e59b700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa5140a06c0 0x7fa5140a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.786+0000 7fa52e59b700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa5140a06c0 0x7fa5140a2b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa52400b680 tx=0x7fa524005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:49.787+0000 7fa51f7fe700 1 -- 172.21.15.3:0/460943808 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa520095bc0 con 0x7fa5281082e0 2023-09-10T01:00:50.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.001+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa52810dfe0 con 0x7fa5281082e0 2023-09-10T01:00:50.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.003+0000 7fa51f7fe700 1 -- 172.21.15.3:0/460943808 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23849 (secure 0 0 0) 0x7fa5200991e0 con 0x7fa5281082e0 2023-09-10T01:00:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:50.005 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":86,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:48.560093+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:00:50.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa5140a06c0 msgr2=0x7fa5140a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa5140a06c0 0x7fa5140a2b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa52400b680 tx=0x7fa524005fd0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 msgr2=0x7fa52819f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 0x7fa52819f8b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa52000ef30 tx=0x7fa52000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 shutdown_connections 2023-09-10T01:00:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5281082e0 0x7fa52819f8b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa528073000 0x7fa528198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa5140a06c0 0x7fa5140a2b80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 --2- 172.21.15.3:0/460943808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5280745e0 0x7fa528198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 >> 172.21.15.3:0/460943808 conn(0x7fa5280fb980 msgr2=0x7fa5280fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 shutdown_connections 2023-09-10T01:00:50.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.005+0000 7fa52f59d700 1 -- 172.21.15.3:0/460943808 wait complete. 2023-09-10T01:00:50.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:00:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 -- 172.21.15.3:0/3940582223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc0745c0 msgr2=0x7f82cc072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3940582223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc0745c0 0x7f82cc072a10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009990 tx=0x7f82bc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 -- 172.21.15.3:0/3940582223 shutdown_connections 2023-09-10T01:00:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3940582223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc1085b0 0x7f82cc10aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3940582223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc0745c0 0x7f82cc072a10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3940582223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82cc072fe0 0x7f82cc073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 -- 172.21.15.3:0/3940582223 >> 172.21.15.3:0/3940582223 conn(0x7f82cc0fbc30 msgr2=0x7f82cc0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.229+0000 7f82d13ff700 1 -- 172.21.15.3:0/3940582223 shutdown_connections 2023-09-10T01:00:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.230+0000 7f82d13ff700 1 -- 172.21.15.3:0/3940582223 wait complete. 2023-09-10T01:00:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.230+0000 7f82d13ff700 1 Processor -- start 2023-09-10T01:00:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82d13ff700 1 -- start start 2023-09-10T01:00:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82d13ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc072fe0 0x7f82cc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82d13ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82cc0745c0 0x7f82cc198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82d13ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc1085b0 0x7f82cc19fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82d13ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc1a0140 con 0x7f82cc072fe0 2023-09-10T01:00:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82d13ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc1a0280 con 0x7f82cc0745c0 2023-09-10T01:00:50.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82d13ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc1a0410 con 0x7f82cc1085b0 2023-09-10T01:00:50.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82ca7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82cc0745c0 0x7f82cc198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc072fe0 0x7f82cc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc072fe0 0x7f82cc1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44688/0 (socket says 172.21.15.3:44688) 2023-09-10T01:00:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82ca7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82cc0745c0 0x7f82cc198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39386/0 (socket says 172.21.15.3:39386) 2023-09-10T01:00:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82caffd700 1 -- 172.21.15.3:0/3180587904 learned_addr learned my addr 172.21.15.3:0/3180587904 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.231+0000 7f82cb7fe700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc1085b0 0x7f82cc19fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82caffd700 1 -- 172.21.15.3:0/3180587904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc1085b0 msgr2=0x7f82cc19fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82caffd700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc1085b0 0x7f82cc19fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82caffd700 1 -- 172.21.15.3:0/3180587904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82cc0745c0 msgr2=0x7f82cc198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82caffd700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82cc0745c0 0x7f82cc198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82caffd700 1 -- 172.21.15.3:0/3180587904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0095f0 con 0x7f82cc072fe0 2023-09-10T01:00:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82caffd700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc072fe0 0x7f82cc1985c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00b680 tx=0x7f82bc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82b3fff700 1 -- 172.21.15.3:0/3180587904 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc01d070 con 0x7f82cc072fe0 2023-09-10T01:00:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82cc1a0690 con 0x7f82cc072fe0 2023-09-10T01:00:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82b3fff700 1 -- 172.21.15.3:0/3180587904 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82bc022470 con 0x7f82cc072fe0 2023-09-10T01:00:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82b3fff700 1 -- 172.21.15.3:0/3180587904 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc017650 con 0x7f82cc072fe0 2023-09-10T01:00:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.232+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82cc1a0ac0 con 0x7f82cc072fe0 2023-09-10T01:00:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.233+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82cc106cb0 con 0x7f82cc072fe0 2023-09-10T01:00:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.235+0000 7f82b3fff700 1 -- 172.21.15.3:0/3180587904 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f82bc0225e0 con 0x7f82cc072fe0 2023-09-10T01:00:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.237+0000 7f82b3fff700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f82b40a0640 0x7f82b40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.237+0000 7f82b3fff700 1 -- 172.21.15.3:0/3180587904 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f82bc0cd3a0 con 0x7f82cc072fe0 2023-09-10T01:00:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.237+0000 7f82ca7fc700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f82b40a0640 0x7f82b40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.237+0000 7f82ca7fc700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f82b40a0640 0x7f82b40a2b00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009c60 tx=0x7f82c0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.241+0000 7f82b3fff700 1 -- 172.21.15.3:0/3180587904 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f82bc096e30 con 0x7f82cc072fe0 2023-09-10T01:00:50.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1596659725' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-09-10T01:00:50.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:50 smithi003 ceph-mon[97654]: osdmap e86: 12 total, 12 up, 12 in 2023-09-10T01:00:50.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:50 smithi003 ceph-mon[97654]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:00:50.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4235578162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:50.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/460943808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:50.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1596659725' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-09-10T01:00:50.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:50 smithi136 ceph-mon[133708]: osdmap e86: 12 total, 12 up, 12 in 2023-09-10T01:00:50.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:50 smithi136 ceph-mon[133708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:00:50.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4235578162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:50.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/460943808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:50.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.451+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82cc10e100 con 0x7f82cc072fe0 2023-09-10T01:00:50.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.452+0000 7f82b3fff700 1 -- 172.21.15.3:0/3180587904 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23849 (secure 0 0 0) 0x7f82bc027090 con 0x7f82cc072fe0 2023-09-10T01:00:50.455 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:50.456 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":86,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:48.560093+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:00:50.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f82b40a0640 msgr2=0x7f82b40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f82b40a0640 0x7f82b40a2b00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009c60 tx=0x7f82c0009260 comp rx=0 tx=0).stop 2023-09-10T01:00:50.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc072fe0 msgr2=0x7f82cc1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc072fe0 0x7f82cc1985c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00b680 tx=0x7f82bc005740 comp rx=0 tx=0).stop 2023-09-10T01:00:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 shutdown_connections 2023-09-10T01:00:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f82b40a0640 0x7f82b40a2b00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc1085b0 0x7f82cc19fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82cc072fe0 0x7f82cc1985c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 --2- 172.21.15.3:0/3180587904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82cc0745c0 0x7f82cc198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 >> 172.21.15.3:0/3180587904 conn(0x7f82cc0fbc30 msgr2=0x7f82cc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:50.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 shutdown_connections 2023-09-10T01:00:50.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.455+0000 7f82d13ff700 1 -- 172.21.15.3:0/3180587904 wait complete. 2023-09-10T01:00:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:50 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1596659725' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-09-10T01:00:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:50 smithi092 ceph-mon[134067]: osdmap e86: 12 total, 12 up, 12 in 2023-09-10T01:00:50.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:50 smithi092 ceph-mon[134067]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:00:50.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:50 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4235578162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:50.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:50 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/460943808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:50.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:00:50.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.701+0000 7fcff19ff700 1 -- 172.21.15.3:0/3220002804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1006d0 msgr2=0x7fcfec100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.701+0000 7fcff19ff700 1 --2- 172.21.15.3:0/3220002804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1006d0 0x7fcfec100af0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0099c0 tx=0x7fcfdc009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.701+0000 7fcff19ff700 1 -- 172.21.15.3:0/3220002804 shutdown_connections 2023-09-10T01:00:50.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.702+0000 7fcff19ff700 1 --2- 172.21.15.3:0/3220002804 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfec102260 0x7fcfec108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.702+0000 7fcff19ff700 1 --2- 172.21.15.3:0/3220002804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1006d0 0x7fcfec100af0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.702+0000 7fcff19ff700 1 --2- 172.21.15.3:0/3220002804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1018c0 0x7fcfec101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.702+0000 7fcff19ff700 1 -- 172.21.15.3:0/3220002804 >> 172.21.15.3:0/3220002804 conn(0x7fcfec0fbc60 msgr2=0x7fcfec0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:50.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.702+0000 7fcff19ff700 1 -- 172.21.15.3:0/3220002804 shutdown_connections 2023-09-10T01:00:50.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.702+0000 7fcff19ff700 1 -- 172.21.15.3:0/3220002804 wait complete. 2023-09-10T01:00:50.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 Processor -- start 2023-09-10T01:00:50.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 -- start start 2023-09-10T01:00:50.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1006d0 0x7fcfec198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1018c0 0x7fcfec198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfec102260 0x7fcfec19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec1a01c0 con 0x7fcfec1006d0 2023-09-10T01:00:50.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec1a0300 con 0x7fcfec1018c0 2023-09-10T01:00:50.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcff19ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec1a0490 con 0x7fcfec102260 2023-09-10T01:00:50.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcfeaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1006d0 0x7fcfec198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcfeaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1006d0 0x7fcfec198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44706/0 (socket says 172.21.15.3:44706) 2023-09-10T01:00:50.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcfeaffd700 1 -- 172.21.15.3:0/2497328364 learned_addr learned my addr 172.21.15.3:0/2497328364 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:50.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcfeb7fe700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfec102260 0x7fcfec19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.703+0000 7fcfea7fc700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1018c0 0x7fcfec198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcfeaffd700 1 -- 172.21.15.3:0/2497328364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfec102260 msgr2=0x7fcfec19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcfeaffd700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfec102260 0x7fcfec19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcfeaffd700 1 -- 172.21.15.3:0/2497328364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1018c0 msgr2=0x7fcfec198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcfeaffd700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1018c0 0x7fcfec198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcfeaffd700 1 -- 172.21.15.3:0/2497328364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfdc0095f0 con 0x7fcfec1006d0 2023-09-10T01:00:50.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcfeaffd700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1006d0 0x7fcfec198640 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009fd0 tx=0x7fcfdc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:50.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcff09fd700 1 -- 172.21.15.3:0/2497328364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc01d070 con 0x7fcfec1006d0 2023-09-10T01:00:50.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcff09fd700 1 -- 172.21.15.3:0/2497328364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfdc005030 con 0x7fcfec1006d0 2023-09-10T01:00:50.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec1a0710 con 0x7fcfec1006d0 2023-09-10T01:00:50.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcff09fd700 1 -- 172.21.15.3:0/2497328364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc004aa0 con 0x7fcfec1006d0 2023-09-10T01:00:50.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.704+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec1a0b30 con 0x7fcfec1006d0 2023-09-10T01:00:50.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.705+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec192cd0 con 0x7fcfec1006d0 2023-09-10T01:00:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.707+0000 7fcff09fd700 1 -- 172.21.15.3:0/2497328364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fcfdc005410 con 0x7fcfec1006d0 2023-09-10T01:00:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.710+0000 7fcff09fd700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcfd40a0640 0x7fcfd40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.710+0000 7fcff09fd700 1 -- 172.21.15.3:0/2497328364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcfdc0ccbe0 con 0x7fcfec1006d0 2023-09-10T01:00:50.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.710+0000 7fcfea7fc700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcfd40a0640 0x7fcfd40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:50.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.711+0000 7fcfea7fc700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcfd40a0640 0x7fcfd40a2b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009c60 tx=0x7fcfe0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:50.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.714+0000 7fcff09fd700 1 -- 172.21.15.3:0/2497328364 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fcfdc096550 con 0x7fcfec1006d0 2023-09-10T01:00:50.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.972+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfec10bee0 con 0x7fcfec1006d0 2023-09-10T01:00:50.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.973+0000 7fcff09fd700 1 -- 172.21.15.3:0/2497328364 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+11231 (secure 0 0 0) 0x7fcfdc099b70 con 0x7fcfec1006d0 2023-09-10T01:00:50.975 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:50.975 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24604,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24610,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24620,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24634,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24647,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:00:48.560325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-09-10T01:00:50.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcfd40a0640 msgr2=0x7fcfd40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcfd40a0640 0x7fcfd40a2b00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009c60 tx=0x7fcfe0009260 comp rx=0 tx=0).stop 2023-09-10T01:00:50.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1006d0 msgr2=0x7fcfec198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:50.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1006d0 0x7fcfec198640 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009fd0 tx=0x7fcfdc00f690 comp rx=0 tx=0).stop 2023-09-10T01:00:50.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 shutdown_connections 2023-09-10T01:00:50.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcfd40a0640 0x7fcfd40a2b00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfec102260 0x7fcfec19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec1006d0 0x7fcfec198640 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 --2- 172.21.15.3:0/2497328364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfec1018c0 0x7fcfec198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:50.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 >> 172.21.15.3:0/2497328364 conn(0x7fcfec0fbc60 msgr2=0x7fcfec107200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:50.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 shutdown_connections 2023-09-10T01:00:50.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:50.976+0000 7fcff19ff700 1 -- 172.21.15.3:0/2497328364 wait complete. 2023-09-10T01:00:50.982 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 16 2023-09-10T01:00:50.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 -- 172.21.15.3:0/2045074512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be81006d0 msgr2=0x7f3be8100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/2045074512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be81006d0 0x7f3be8100af0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3be40099c0 tx=0x7f3be4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 -- 172.21.15.3:0/2045074512 shutdown_connections 2023-09-10T01:00:51.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/2045074512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be8102260 0x7f3be8108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/2045074512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be81006d0 0x7f3be8100af0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/2045074512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81018c0 0x7f3be8101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 -- 172.21.15.3:0/2045074512 >> 172.21.15.3:0/2045074512 conn(0x7f3be80fbc60 msgr2=0x7f3be80fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 -- 172.21.15.3:0/2045074512 shutdown_connections 2023-09-10T01:00:51.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.203+0000 7f3bef3ff700 1 -- 172.21.15.3:0/2045074512 wait complete. 2023-09-10T01:00:51.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.204+0000 7f3bef3ff700 1 Processor -- start 2023-09-10T01:00:51.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.204+0000 7f3bef3ff700 1 -- start start 2023-09-10T01:00:51.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.204+0000 7f3bef3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81006d0 0x7f3be81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.204+0000 7f3bef3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be81018c0 0x7f3be8198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bef3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be8102260 0x7f3be819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bef3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be81a0170 con 0x7f3be81006d0 2023-09-10T01:00:51.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bef3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be81a02b0 con 0x7f3be8102260 2023-09-10T01:00:51.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bef3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be81a0440 con 0x7f3be81018c0 2023-09-10T01:00:51.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81006d0 0x7f3be81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81006d0 0x7f3be81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44722/0 (socket says 172.21.15.3:44722) 2023-09-10T01:00:51.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 -- 172.21.15.3:0/4239424435 learned_addr learned my addr 172.21.15.3:0/4239424435 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:51.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed99c700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be8102260 0x7f3be819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bec99a700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be81018c0 0x7f3be8198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 -- 172.21.15.3:0/4239424435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be81018c0 msgr2=0x7f3be8198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be81018c0 0x7f3be8198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 -- 172.21.15.3:0/4239424435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be8102260 msgr2=0x7f3be819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be8102260 0x7f3be819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 -- 172.21.15.3:0/4239424435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be40095f0 con 0x7f3be81006d0 2023-09-10T01:00:51.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bec99a700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be81018c0 0x7f3be8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:51.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.205+0000 7f3bed19b700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81006d0 0x7f3be81985f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009fd0 tx=0x7f3be400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.206+0000 7f3bde7fc700 1 -- 172.21.15.3:0/4239424435 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be401d070 con 0x7f3be81006d0 2023-09-10T01:00:51.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.206+0000 7f3bde7fc700 1 -- 172.21.15.3:0/4239424435 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3be4005510 con 0x7f3be81006d0 2023-09-10T01:00:51.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.206+0000 7f3bde7fc700 1 -- 172.21.15.3:0/4239424435 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4004b00 con 0x7f3be81006d0 2023-09-10T01:00:51.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.206+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be81a06c0 con 0x7f3be81006d0 2023-09-10T01:00:51.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.206+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be81a0a90 con 0x7f3be81006d0 2023-09-10T01:00:51.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.207+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be810bf30 con 0x7f3be81006d0 2023-09-10T01:00:51.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.209+0000 7f3bde7fc700 1 -- 172.21.15.3:0/4239424435 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f3be4004ca0 con 0x7f3be81006d0 2023-09-10T01:00:51.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.210+0000 7f3bde7fc700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3bd40a0640 0x7f3bd40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.210+0000 7f3bde7fc700 1 -- 172.21.15.3:0/4239424435 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3be40ccc60 con 0x7f3be81006d0 2023-09-10T01:00:51.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.210+0000 7f3bec99a700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3bd40a0640 0x7f3bd40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.211+0000 7f3bec99a700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3bd40a0640 0x7f3bd40a2b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8009cb0 tx=0x7f3bd8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.214+0000 7f3bde7fc700 1 -- 172.21.15.3:0/4239424435 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3be40965d0 con 0x7f3be81006d0 2023-09-10T01:00:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:51 smithi003 ceph-mon[97654]: pgmap v283: 129 pgs: 5 creating+peering, 21 unknown, 103 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2023-09-10T01:00:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:51 smithi003 ceph-mon[97654]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-09-10T01:00:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:51 smithi003 ceph-mon[97654]: Cluster is now healthy 2023-09-10T01:00:51.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:51 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3180587904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:51.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:51 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2497328364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:51.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:51 smithi136 ceph-mon[133708]: pgmap v283: 129 pgs: 5 creating+peering, 21 unknown, 103 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2023-09-10T01:00:51.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:51 smithi136 ceph-mon[133708]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-09-10T01:00:51.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:51 smithi136 ceph-mon[133708]: Cluster is now healthy 2023-09-10T01:00:51.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:51 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3180587904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:51.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:51 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2497328364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:51.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.427+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3be81a1090 con 0x7f3be81006d0 2023-09-10T01:00:51.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.428+0000 7f3bde7fc700 1 -- 172.21.15.3:0/4239424435 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23849 (secure 0 0 0) 0x7f3be4099bf0 con 0x7f3be81006d0 2023-09-10T01:00:51.430 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:51.430 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":86,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:48.560093+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:00:51.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3bd40a0640 msgr2=0x7f3bd40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3bd40a0640 0x7f3bd40a2b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8009cb0 tx=0x7f3bd8009260 comp rx=0 tx=0).stop 2023-09-10T01:00:51.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81006d0 msgr2=0x7f3be81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81006d0 0x7f3be81985f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009fd0 tx=0x7f3be400f690 comp rx=0 tx=0).stop 2023-09-10T01:00:51.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 shutdown_connections 2023-09-10T01:00:51.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3be8102260 0x7f3be819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3be81006d0 0x7f3be81985f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3bd40a0640 0x7f3bd40a2b00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 --2- 172.21.15.3:0/4239424435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be81018c0 0x7f3be8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 >> 172.21.15.3:0/4239424435 conn(0x7f3be80fbc60 msgr2=0x7f3be80fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 shutdown_connections 2023-09-10T01:00:51.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.431+0000 7f3bef3ff700 1 -- 172.21.15.3:0/4239424435 wait complete. 2023-09-10T01:00:51.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-09-10T01:00:43.531603+0000', 'modified': '2023-09-10T01:00:48.560325+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24698}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24698': {'gid': 24698, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6839/149607042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 149607042}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 149607042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-09-10T01:00:51.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-09-10T01:00:51.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-09-10T01:00:51.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:51 smithi092 ceph-mon[134067]: pgmap v283: 129 pgs: 5 creating+peering, 21 unknown, 103 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2023-09-10T01:00:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:51 smithi092 ceph-mon[134067]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-09-10T01:00:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:51 smithi092 ceph-mon[134067]: Cluster is now healthy 2023-09-10T01:00:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:51 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3180587904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:51 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2497328364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:00:51.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 -- 172.21.15.3:0/3245682674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201006e0 msgr2=0x7fdd20100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 --2- 172.21.15.3:0/3245682674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201006e0 0x7fdd20100b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009990 tx=0x7fdd10009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 -- 172.21.15.3:0/3245682674 shutdown_connections 2023-09-10T01:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 --2- 172.21.15.3:0/3245682674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201006e0 0x7fdd20100b00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 --2- 172.21.15.3:0/3245682674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd20102270 0x7fdd20108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 --2- 172.21.15.3:0/3245682674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201018d0 0x7fdd20101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 -- 172.21.15.3:0/3245682674 >> 172.21.15.3:0/3245682674 conn(0x7fdd200fbc50 msgr2=0x7fdd200fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 -- 172.21.15.3:0/3245682674 shutdown_connections 2023-09-10T01:00:51.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.657+0000 7fdd24fff700 1 -- 172.21.15.3:0/3245682674 wait complete. 2023-09-10T01:00:51.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.658+0000 7fdd24fff700 1 Processor -- start 2023-09-10T01:00:51.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.658+0000 7fdd24fff700 1 -- start start 2023-09-10T01:00:51.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd24fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201006e0 0x7fdd201963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd24fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201018d0 0x7fdd20196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd24fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd20102270 0x7fdd2019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd24fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2019df70 con 0x7fdd20102270 2023-09-10T01:00:51.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd24fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2019e0b0 con 0x7fdd201006e0 2023-09-10T01:00:51.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd24fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2019e240 con 0x7fdd201018d0 2023-09-10T01:00:51.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201006e0 0x7fdd201963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201006e0 0x7fdd201963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39442/0 (socket says 172.21.15.3:39442) 2023-09-10T01:00:51.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1e59c700 1 -- 172.21.15.3:0/1896405498 learned_addr learned my addr 172.21.15.3:0/1896405498 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:51.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd20102270 0x7fdd2019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd17fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201018d0 0x7fdd20196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1ed9d700 1 -- 172.21.15.3:0/1896405498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201018d0 msgr2=0x7fdd20196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201018d0 0x7fdd20196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1ed9d700 1 -- 172.21.15.3:0/1896405498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201006e0 msgr2=0x7fdd201963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201006e0 0x7fdd201963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1ed9d700 1 -- 172.21.15.3:0/1896405498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd100095f0 con 0x7fdd20102270 2023-09-10T01:00:51.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.659+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201006e0 0x7fdd201963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:51.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.660+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd20102270 0x7fdd2019d9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800efc0 tx=0x7fdd1800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.660+0000 7fdd177fe700 1 -- 172.21.15.3:0/1896405498 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18018070 con 0x7fdd20102270 2023-09-10T01:00:51.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.660+0000 7fdd177fe700 1 -- 172.21.15.3:0/1896405498 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd1800f040 con 0x7fdd20102270 2023-09-10T01:00:51.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.660+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2019e4c0 con 0x7fdd20102270 2023-09-10T01:00:51.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.660+0000 7fdd177fe700 1 -- 172.21.15.3:0/1896405498 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800cdf0 con 0x7fdd20102270 2023-09-10T01:00:51.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.660+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2019e980 con 0x7fdd20102270 2023-09-10T01:00:51.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.663+0000 7fdd177fe700 1 -- 172.21.15.3:0/1896405498 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fdd18003a00 con 0x7fdd20102270 2023-09-10T01:00:51.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.665+0000 7fdd388f0b80 1 Processor -- start 2023-09-10T01:00:51.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.665+0000 7fdd388f0b80 1 -- start start 2023-09-10T01:00:51.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.665+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a369f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.665+0000 7fdd177fe700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdd0c0a0640 0x7fdd0c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.665+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a369dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.665+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a369e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a35f26e0 con 0x55c9a369e1e0 2023-09-10T01:00:51.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a35f0000 con 0x55c9a369f980 2023-09-10T01:00:51.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a35eb8f0 con 0x55c9a369f020 2023-09-10T01:00:51.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd177fe700 1 -- 172.21.15.3:0/1896405498 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdd180cbaa0 con 0x7fdd20102270 2023-09-10T01:00:51.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a369e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd17fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdd0c0a0640 0x7fdd0c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a369e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44744/0 (socket says 172.21.15.3:44744) 2023-09-10T01:00:51.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 -- 172.21.15.3:0/2381366420 learned_addr learned my addr 172.21.15.3:0/2381366420 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:51.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a369f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd17fff700 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a369dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 -- 172.21.15.3:0/2381366420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 msgr2=0x55c9a369f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a369f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 -- 172.21.15.3:0/2381366420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 msgr2=0x55c9a369dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a369dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd1ed9d700 1 -- 172.21.15.3:0/2381366420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9a3417d90 con 0x55c9a369e1e0 2023-09-10T01:00:51.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.666+0000 7fdd17fff700 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a369dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:00:51.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.667+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a369e660 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd18098dd0 tx=0x7fdd18098c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0638e9a8ed4493 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.667+0000 7fdd17fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdd0c0a0640 0x7fdd0c0a2b00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55c9a369db90 tx=0x7fdd08009850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.667+0000 7fdd14ff9700 1 -- 172.21.15.3:0/2381366420 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1801b320 con 0x55c9a369e1e0 2023-09-10T01:00:51.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.667+0000 7fdd14ff9700 1 -- 172.21.15.3:0/2381366420 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd180997d0 con 0x55c9a369e1e0 2023-09-10T01:00:51.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.667+0000 7fdd14ff9700 1 -- 172.21.15.3:0/2381366420 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18097500 con 0x55c9a369e1e0 2023-09-10T01:00:51.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.667+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2381366420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 msgr2=0x55c9a369e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.667+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a369e660 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd18098dd0 tx=0x7fdd18098c90 comp rx=0 tx=0).stop 2023-09-10T01:00:51.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2381366420 shutdown_connections 2023-09-10T01:00:51.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a369f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a369e660 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2381366420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a369dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2381366420 >> 172.21.15.3:0/2381366420 conn(0x55c9a360e040 msgr2=0x55c9a36946d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2381366420 shutdown_connections 2023-09-10T01:00:51.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2381366420 wait complete. 2023-09-10T01:00:51.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 -- start start 2023-09-10T01:00:51.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.668+0000 7fdd388f0b80 1 Processor -- start 2023-09-10T01:00:51.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f020 0x55c9a36a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a3589b20 con 0x55c9a369e1e0 2023-09-10T01:00:51.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a3589820 con 0x55c9a369f020 2023-09-10T01:00:51.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a346d8d0 con 0x55c9a369f980 2023-09-10T01:00:51.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd388f0b80 10 client.0 _ll_register_callbacks cb 0x55c9a34d3a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:00:51.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd17fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f020 0x55c9a36a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36a79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44758/0 (socket says 172.21.15.3:44758) 2023-09-10T01:00:51.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd1e59c700 1 -- 172.21.15.3:0/3253224887 learned_addr learned my addr 172.21.15.3:0/3253224887 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:51.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.669+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd1e59c700 1 -- 172.21.15.3:0/3253224887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 msgr2=0x55c9a36aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd1e59c700 1 -- 172.21.15.3:0/3253224887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f020 msgr2=0x55c9a36a7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f020 0x55c9a36a7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd1e59c700 1 -- 172.21.15.3:0/3253224887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2019e980 con 0x55c9a369e1e0 2023-09-10T01:00:51.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36a79f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005250 tx=0x7fdd10005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d9eab9401b1c23 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd05ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:00:51.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd05ffb700 1 -- 172.21.15.3:0/3253224887 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1001d070 con 0x55c9a369e1e0 2023-09-10T01:00:51.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c9a35434a0 con 0x55c9a369e1e0 2023-09-10T01:00:51.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd05ffb700 1 -- 172.21.15.3:0/3253224887 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdd10004910 con 0x55c9a369e1e0 2023-09-10T01:00:51.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd05ffb700 1 -- 172.21.15.3:0/3253224887 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd10004ad0 con 0x55c9a369e1e0 2023-09-10T01:00:51.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd388f0b80 10 client.14760 fetch_fsmap learned FSMap version 16 2023-09-10T01:00:51.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.670+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55c9a3400c10 con 0x55c9a369e1e0 2023-09-10T01:00:51.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd05ffb700 1 -- 172.21.15.3:0/3253224887 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c9a35434a0 con 0x55c9a369e1e0 2023-09-10T01:00:51.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd05ffb700 1 -- 172.21.15.3:0/3253224887 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 16) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdd10024450 con 0x55c9a369e1e0 2023-09-10T01:00:51.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd388f0b80 10 client.14760 fetch_fsmap finished waiting for FSMap version 16 2023-09-10T01:00:51.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd388f0b80 10 client.14760 resolve_mds: resolved name 'b' to daemon mds.b 2023-09-10T01:00:51.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd388f0b80 20 client.14760 populate_metadata read hostname 'smithi003' 2023-09-10T01:00:51.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a36b3070 0x55c9a36b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd388f0b80 4 client.14760 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:00:51.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c9a3600850 con 0x55c9a36b3070 2023-09-10T01:00:51.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.671+0000 7fdd17fff700 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a36b3070 0x55c9a36b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.672+0000 7fdd17fff700 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a36b3070 0x55c9a36b1fe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2530e73ec69fb94 server_cookie=a8b41ea5efc28839 in_seq=0 out_seq=0 2023-09-10T01:00:51.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.672+0000 7fdd05ffb700 10 client.14760 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:00:51.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd05ffb700 1 -- 172.21.15.3:0/3253224887 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c9a3600850 con 0x55c9a36b3070 2023-09-10T01:00:51.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd05ffb700 10 client.14760 handle_command_reply: tid=0 2023-09-10T01:00:51.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 client.14760 shutdown 2023-09-10T01:00:51.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 msgr2=0x55c9a36a79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36a79f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005250 tx=0x7fdd10005280 comp rx=0 tx=0).stop 2023-09-10T01:00:51.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 shutdown_connections 2023-09-10T01:00:51.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a36b3070 0x55c9a36b1fe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36a79f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd05ffb700 0 client.14760 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:00:51.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/3253224887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f020 0x55c9a36a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 >> 172.21.15.3:0/3253224887 conn(0x55c9a360e040 msgr2=0x55c9a3693d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 shutdown_connections 2023-09-10T01:00:51.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.673+0000 7fdd388f0b80 1 -- 172.21.15.3:0/3253224887 wait complete. 2023-09-10T01:00:51.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.674+0000 7fdd388f0b80 20 client.14760 trim_cache size 0 max 16384 2023-09-10T01:00:51.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 Processor -- start 2023-09-10T01:00:51.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 -- start start 2023-09-10T01:00:51.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36af800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a36aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a36a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a3695f90 con 0x55c9a369e1e0 2023-09-10T01:00:51.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a3600850 con 0x55c9a369f980 2023-09-10T01:00:51.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a3479de0 con 0x55c9a369f020 2023-09-10T01:00:51.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44764/0 (socket says 172.21.15.3:44764) 2023-09-10T01:00:51.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.692+0000 7fdd1e59c700 1 -- 172.21.15.3:0/769834967 learned_addr learned my addr 172.21.15.3:0/769834967 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:51.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a36a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd17fff700 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a36aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd1e59c700 1 -- 172.21.15.3:0/769834967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 msgr2=0x55c9a36aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a36aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd1e59c700 1 -- 172.21.15.3:0/769834967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 msgr2=0x55c9a36a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a36a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd1e59c700 1 -- 172.21.15.3:0/769834967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9a343e200 con 0x55c9a369e1e0 2023-09-10T01:00:51.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36af800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005d10 tx=0x7fdd10005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d535da8a307f42e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd157fa700 1 -- 172.21.15.3:0/769834967 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1001d040 con 0x55c9a369e1e0 2023-09-10T01:00:51.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd157fa700 1 -- 172.21.15.3:0/769834967 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd1001e040 con 0x55c9a369e1e0 2023-09-10T01:00:51.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.693+0000 7fdd157fa700 1 -- 172.21.15.3:0/769834967 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1000f6c0 con 0x55c9a369e1e0 2023-09-10T01:00:51.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 -- 172.21.15.3:0/769834967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 msgr2=0x55c9a36af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36af800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005d10 tx=0x7fdd10005e80 comp rx=0 tx=0).stop 2023-09-10T01:00:51.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 -- 172.21.15.3:0/769834967 shutdown_connections 2023-09-10T01:00:51.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369f980 0x55c9a36a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369e1e0 0x55c9a36af800 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/769834967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f020 0x55c9a36aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 -- 172.21.15.3:0/769834967 >> 172.21.15.3:0/769834967 conn(0x55c9a360e040 msgr2=0x55c9a36bc030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 -- 172.21.15.3:0/769834967 shutdown_connections 2023-09-10T01:00:51.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 -- 172.21.15.3:0/769834967 wait complete. 2023-09-10T01:00:51.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.694+0000 7fdd388f0b80 1 -- start start 2023-09-10T01:00:51.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 1 Processor -- start 2023-09-10T01:00:51.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369e1e0 0x55c9a36a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369f020 0x55c9a36aee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36a9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a36aa4d0 con 0x55c9a369f020 2023-09-10T01:00:51.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a37070c0 con 0x55c9a369e1e0 2023-09-10T01:00:51.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9a3707230 con 0x55c9a369f980 2023-09-10T01:00:51.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd388f0b80 10 client.0 _ll_register_callbacks cb 0x7fdd1000bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:00:51.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369f020 0x55c9a36aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369f020 0x55c9a36aee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44770/0 (socket says 172.21.15.3:44770) 2023-09-10T01:00:51.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd17fff700 1 -- 172.21.15.3:0/2169434809 learned_addr learned my addr 172.21.15.3:0/2169434809 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:51.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369e1e0 0x55c9a36a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.695+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36a9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd17fff700 1 -- 172.21.15.3:0/2169434809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 msgr2=0x55c9a36a9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd17fff700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36a9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd17fff700 1 -- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369e1e0 msgr2=0x55c9a36a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:00:51.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd17fff700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369e1e0 0x55c9a36a98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd17fff700 1 -- 172.21.15.3:0/2169434809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9a3400c10 con 0x55c9a369f020 2023-09-10T01:00:51.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd1ed9d700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36a9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:51.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd17fff700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369f020 0x55c9a36aee20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55c9a3593560 tx=0x7fdd08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78cecd76bf22a62 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:51.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd05ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:00:51.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd05ffb700 1 -- 172.21.15.3:0/2169434809 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08020450 con 0x55c9a369f020 2023-09-10T01:00:51.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd180997d0 con 0x55c9a369f020 2023-09-10T01:00:51.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.696+0000 7fdd05ffb700 1 -- 172.21.15.3:0/2169434809 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdd080205b0 con 0x55c9a369f020 2023-09-10T01:00:51.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd05ffb700 1 -- 172.21.15.3:0/2169434809 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd080207b0 con 0x55c9a369f020 2023-09-10T01:00:51.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 10 client.14766 fetch_fsmap learned FSMap version 16 2023-09-10T01:00:51.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55c9a3416260 con 0x55c9a369f020 2023-09-10T01:00:51.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd05ffb700 1 -- 172.21.15.3:0/2169434809 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd180997d0 con 0x55c9a369f020 2023-09-10T01:00:51.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd05ffb700 1 -- 172.21.15.3:0/2169434809 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 16) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdd080109d0 con 0x55c9a369f020 2023-09-10T01:00:51.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 10 client.14766 fetch_fsmap finished waiting for FSMap version 16 2023-09-10T01:00:51.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 10 client.14766 resolve_mds: resolved name 'b' to daemon mds.b 2023-09-10T01:00:51.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 20 client.14766 populate_metadata read hostname 'smithi003' 2023-09-10T01:00:51.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a370c610 0x55c9a3708a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:51.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 4 client.14766 mds_command: new command op to 24698 tid=0[{"prefix": "status"}] 2023-09-10T01:00:51.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.697+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c9a35e48d0 con 0x55c9a370c610 2023-09-10T01:00:51.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.698+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a370c610 0x55c9a3708a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:51.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.698+0000 7fdd1e59c700 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a370c610 0x55c9a3708a70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21560780a8b59223 server_cookie=8dce4f6b104912ee in_seq=0 out_seq=0 2023-09-10T01:00:51.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.698+0000 7fdd05ffb700 10 client.14766 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:00:51.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.698+0000 7fdd05ffb700 1 -- 172.21.15.3:0/2169434809 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55c9a35e48d0 con 0x55c9a370c610 2023-09-10T01:00:51.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.698+0000 7fdd05ffb700 10 client.14766 handle_command_reply: tid=0 2023-09-10T01:00:51.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.698+0000 7fdd388f0b80 1 client.14766 shutdown 2023-09-10T01:00:51.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369f020 msgr2=0x55c9a36aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369f020 0x55c9a36aee20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55c9a3593560 tx=0x7fdd08004750 comp rx=0 tx=0).stop 2023-09-10T01:00:51.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 shutdown_connections 2023-09-10T01:00:51.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9a370c610 0x55c9a3708a70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9a369f980 0x55c9a36a9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9a369e1e0 0x55c9a36a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 --2- 172.21.15.3:0/2169434809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9a369f020 0x55c9a36aee20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd05ffb700 0 client.14766 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:00:51.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 >> 172.21.15.3:0/2169434809 conn(0x55c9a360e040 msgr2=0x55c9a36a57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 shutdown_connections 2023-09-10T01:00:51.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 1 -- 172.21.15.3:0/2169434809 wait complete. 2023-09-10T01:00:51.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd388f0b80 20 client.14766 trim_cache size 0 max 16384 2023-09-10T01:00:51.714 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:00:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T01:00:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2023-09-10T01:00:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24698, 2023-09-10T01:00:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2023-09-10T01:00:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2023-09-10T01:00:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2023-09-10T01:00:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 7.6558625969999996, 2023-09-10T01:00:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 16, 2023-09-10T01:00:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 86, 2023-09-10T01:00:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 85, 2023-09-10T01:00:51.717 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 12.892451713 2023-09-10T01:00:51.717 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:00:51.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdd0c0a0640 msgr2=0x7fdd0c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd24fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdd0c0a0640 0x7fdd0c0a2b00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55c9a369db90 tx=0x7fdd08009850 comp rx=0 tx=0).stop 2023-09-10T01:00:51.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd20102270 msgr2=0x7fdd2019d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:51.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.699+0000 7fdd24fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd20102270 0x7fdd2019d9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800efc0 tx=0x7fdd1800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 shutdown_connections 2023-09-10T01:00:51.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd20102270 0x7fdd2019d9a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd201006e0 0x7fdd201963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdd0c0a0640 0x7fdd0c0a2b00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 --2- 172.21.15.3:0/1896405498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd201018d0 0x7fdd20196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:51.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 >> 172.21.15.3:0/1896405498 conn(0x7fdd200fbc50 msgr2=0x7fdd200fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:51.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 shutdown_connections 2023-09-10T01:00:51.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:51.700+0000 7fdd24fff700 1 -- 172.21.15.3:0/1896405498 wait complete. 2023-09-10T01:00:51.721 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-09-10T01:00:51.722 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:00:51.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph fs ls 2023-09-10T01:00:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:52 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4239424435' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:52 smithi136 ceph-mon[133708]: pgmap v284: 129 pgs: 5 creating+peering, 10 unknown, 114 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 878 B/s wr, 3 op/s 2023-09-10T01:00:52.450 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:00:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:52 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4239424435' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:52.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:52 smithi092 ceph-mon[134067]: pgmap v284: 129 pgs: 5 creating+peering, 10 unknown, 114 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 878 B/s wr, 3 op/s 2023-09-10T01:00:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:52 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4239424435' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:00:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:52 smithi003 ceph-mon[97654]: pgmap v284: 129 pgs: 5 creating+peering, 10 unknown, 114 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 878 B/s wr, 3 op/s 2023-09-10T01:00:54.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 -- 172.21.15.3:0/1708654342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100620 msgr2=0x7f3670100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:54.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 --2- 172.21.15.3:0/1708654342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100620 0x7f3670100aa0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3660009990 tx=0x7f3660009ca0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 -- 172.21.15.3:0/1708654342 shutdown_connections 2023-09-10T01:00:54.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 --2- 172.21.15.3:0/1708654342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3670100fe0 0x7f367010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 --2- 172.21.15.3:0/1708654342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3670106620 0x7f3670106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 --2- 172.21.15.3:0/1708654342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100620 0x7f3670100aa0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 -- 172.21.15.3:0/1708654342 >> 172.21.15.3:0/1708654342 conn(0x7f36700fc0f0 msgr2=0x7f36700fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:54.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 -- 172.21.15.3:0/1708654342 shutdown_connections 2023-09-10T01:00:54.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.030+0000 7f3677e9a700 1 -- 172.21.15.3:0/1708654342 wait complete. 2023-09-10T01:00:54.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.031+0000 7f3677e9a700 1 Processor -- start 2023-09-10T01:00:54.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.031+0000 7f3677e9a700 1 -- start start 2023-09-10T01:00:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3677e9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3670100620 0x7f3670198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3677e9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100fe0 0x7f3670198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3677e9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3670106620 0x7f367019d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3677e9a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367019d930 con 0x7f3670100fe0 2023-09-10T01:00:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3677e9a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367019daa0 con 0x7f3670106620 2023-09-10T01:00:54.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3677e9a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367019dc60 con 0x7f3670100620 2023-09-10T01:00:54.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100fe0 0x7f3670198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:54.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100fe0 0x7f3670198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44798/0 (socket says 172.21.15.3:44798) 2023-09-10T01:00:54.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 -- 172.21.15.3:0/585774839 learned_addr learned my addr 172.21.15.3:0/585774839 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:54.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3676437700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3670106620 0x7f367019d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:54.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675c36700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3670100620 0x7f3670198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:54.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 -- 172.21.15.3:0/585774839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3670100620 msgr2=0x7f3670198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3670100620 0x7f3670198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 -- 172.21.15.3:0/585774839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3670106620 msgr2=0x7f367019d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3670106620 0x7f367019d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3675435700 1 -- 172.21.15.3:0/585774839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36600095f0 con 0x7f3670100fe0 2023-09-10T01:00:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.032+0000 7f3676437700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3670106620 0x7f367019d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:00:54.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.033+0000 7f3675435700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100fe0 0x7f3670198ec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3660005910 tx=0x7f3660004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:54.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.033+0000 7f3666ffd700 1 -- 172.21.15.3:0/585774839 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366001d070 con 0x7f3670100fe0 2023-09-10T01:00:54.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.033+0000 7f3666ffd700 1 -- 172.21.15.3:0/585774839 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3660022470 con 0x7f3670100fe0 2023-09-10T01:00:54.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.033+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367019dee0 con 0x7f3670100fe0 2023-09-10T01:00:54.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.033+0000 7f3666ffd700 1 -- 172.21.15.3:0/585774839 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3660017650 con 0x7f3670100fe0 2023-09-10T01:00:54.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.033+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36701a57e0 con 0x7f3670100fe0 2023-09-10T01:00:54.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.034+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367004e300 con 0x7f3670100fe0 2023-09-10T01:00:54.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.036+0000 7f3666ffd700 1 -- 172.21.15.3:0/585774839 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f36600225e0 con 0x7f3670100fe0 2023-09-10T01:00:54.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.038+0000 7f3666ffd700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f365c0a06c0 0x7f365c0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:54.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.039+0000 7f3666ffd700 1 -- 172.21.15.3:0/585774839 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f36600cd060 con 0x7f3670100fe0 2023-09-10T01:00:54.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.039+0000 7f3675c36700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f365c0a06c0 0x7f365c0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:54.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.039+0000 7f3675c36700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f365c0a06c0 0x7f365c0a2b80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f366c009be0 tx=0x7f366c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:54.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.042+0000 7f3666ffd700 1 -- 172.21.15.3:0/585774839 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f366005ce10 con 0x7f3670100fe0 2023-09-10T01:00:54.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.300+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3670110670 con 0x7f3670100fe0 2023-09-10T01:00:54.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.301+0000 7f3666ffd700 1 -- 172.21.15.3:0/585774839 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3660027020 con 0x7f3670100fe0 2023-09-10T01:00:54.303 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-09-10T01:00:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f365c0a06c0 msgr2=0x7f365c0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f365c0a06c0 0x7f365c0a2b80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f366c009be0 tx=0x7f366c009260 comp rx=0 tx=0).stop 2023-09-10T01:00:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100fe0 msgr2=0x7f3670198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100fe0 0x7f3670198ec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3660005910 tx=0x7f3660004710 comp rx=0 tx=0).stop 2023-09-10T01:00:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 shutdown_connections 2023-09-10T01:00:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f365c0a06c0 0x7f365c0a2b80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3670100620 0x7f3670198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3670106620 0x7f367019d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 --2- 172.21.15.3:0/585774839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3670100fe0 0x7f3670198ec0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:54.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 >> 172.21.15.3:0/585774839 conn(0x7f36700fc0f0 msgr2=0x7f367010ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:54.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 shutdown_connections 2023-09-10T01:00:54.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:54.303+0000 7f3677e9a700 1 -- 172.21.15.3:0/585774839 wait complete. 2023-09-10T01:00:54.825 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-09-10T01:00:54.826 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-09-10T01:00:54.826 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2023-09-10T01:00:54.827 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-09-10T01:00:54.827 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:00:54.827 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-09-10T01:00:54.828 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-09-10T01:00:54.828 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-09-10T01:00:54.828 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-09-10T01:00:54.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:54.829 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2023-09-10T01:00:54.875 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-09-10T01:00:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-09-10T01:00:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2023-09-10T01:00:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2023-09-10T01:00:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2023-09-10T01:00:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2023-09-10T01:00:54.877 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-09-10T01:00:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2023-09-10T01:00:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2023-09-10T01:00:54.878 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-09-10T01:00:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2023-09-10T01:00:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2023-09-10T01:00:54.879 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-09-10T01:00:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2023-09-10T01:00:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2023-09-10T01:00:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-09-10T01:00:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 41730sec preferred_lft 41730sec 2023-09-10T01:00:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2023-09-10T01:00:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2023-09-10T01:00:54.881 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-09-10T01:00:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2023-09-10T01:00:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2023-09-10T01:00:54.882 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-09-10T01:00:54.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:54.883 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:00:54.883 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:00:54.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add name ceph-brx type bridge 2023-09-10T01:00:54.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr flush dev ceph-brx 2023-09-10T01:00:54.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set ceph-brx up 2023-09-10T01:00:54.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-09-10T01:00:54.886 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:00:55.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:54 smithi003 ceph-mon[97654]: pgmap v285: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 750 B/s wr, 2 op/s 2023-09-10T01:00:55.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:54 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/585774839' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-09-10T01:00:55.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:54 smithi136 ceph-mon[133708]: pgmap v285: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 750 B/s wr, 2 op/s 2023-09-10T01:00:55.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:54 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/585774839' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-09-10T01:00:55.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:54 smithi092 ceph-mon[134067]: pgmap v285: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 750 B/s wr, 2 op/s 2023-09-10T01:00:55.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:54 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/585774839' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-09-10T01:00:55.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:55.368 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-09-10T01:00:55.443 INFO:teuthology.orchestra.run.smithi003.stdout:1 2023-09-10T01:00:55.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:55.446 DEBUG:teuthology.orchestra.run.smithi003:> route 2023-09-10T01:00:55.508 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2023-09-10T01:00:55.509 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-09-10T01:00:55.509 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-09-10T01:00:55.509 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-09-10T01:00:55.510 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-09-10T01:00:55.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:55.513 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:00:55.518 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:00:55.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-09-10T01:00:55.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-09-10T01:00:55.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-09-10T01:00:55.520 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:00:56.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-09-10T01:00:56.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-09-10T01:00:56.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-09-10T01:00:56.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-09-10T01:00:56.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-09-10T01:00:56.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2023-09-10T01:00:56.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2023-09-10T01:00:56.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-09-10T01:00:56.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2023-09-10T01:00:56.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:55 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-09-10T01:00:56.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-09-10T01:00:56.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-09-10T01:00:56.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-09-10T01:00:56.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-09-10T01:00:56.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-09-10T01:00:56.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2023-09-10T01:00:56.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2023-09-10T01:00:56.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-09-10T01:00:56.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2023-09-10T01:00:56.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:55 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-09-10T01:00:56.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:56.230 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2023-09-10T01:00:56.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:56.275 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list-id 2023-09-10T01:00:56.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:56.361 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:00:56.361 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:00:56.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-09-10T01:00:56.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-09-10T01:00:56.362 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:00:56.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-09-10T01:00:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-09-10T01:00:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-09-10T01:00:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-09-10T01:00:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-09-10T01:00:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2023-09-10T01:00:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2023-09-10T01:00:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-09-10T01:00:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2023-09-10T01:00:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:55 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-09-10T01:00:56.554 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-09-10T01:00:56.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:56.556 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:00:56.556 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:00:56.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-09-10T01:00:56.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-09-10T01:00:56.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-09-10T01:00:56.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-09-10T01:00:56.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-09-10T01:00:56.558 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:00:56.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:56.845 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:00:56.845 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:00:56.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.0 up 2023-09-10T01:00:56.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set dev brx.0 master ceph-brx 2023-09-10T01:00:56.847 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:00:57.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-09-10T01:00:57.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-09-10T01:00:57.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-09-10T01:00:57.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-09-10T01:00:57.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-09-10T01:00:57.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-09-10T01:00:57.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2023-09-10T01:00:57.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2023-09-10T01:00:57.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-09-10T01:00:57.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]': finished 2023-09-10T01:00:57.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-09-10T01:00:57.029 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:56 smithi003 ceph-mon[97654]: osdmap e87: 12 total, 12 up, 12 in 2023-09-10T01:00:57.030 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-09-10T01:00:57.031 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-09-10T01:00:57.031 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:00:57.077 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-09-10T01:00:57.078 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-09-10T01:00:57.078 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:00:57.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo modprobe fuse 2023-09-10T01:00:57.272 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-09-10T01:00:57.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-09-10T01:00:57.344 INFO:teuthology.orchestra.run.smithi003.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-09-10T01:00:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-09-10T01:00:57.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:57.349 DEBUG:teuthology.orchestra.run.smithi003:> ls /sys/fs/fuse/connections 2023-09-10T01:00:57.393 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-09-10T01:00:57.394 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-09-10T01:00:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-09-10T01:00:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-09-10T01:00:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-09-10T01:00:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-09-10T01:00:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-09-10T01:00:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-09-10T01:00:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2023-09-10T01:00:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2023-09-10T01:00:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-09-10T01:00:57.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]': finished 2023-09-10T01:00:57.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-09-10T01:00:57.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:56 smithi136 ceph-mon[133708]: osdmap e87: 12 total, 12 up, 12 in 2023-09-10T01:00:57.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo modprobe fuse 2023-09-10T01:00:57.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-09-10T01:00:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-09-10T01:00:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-09-10T01:00:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-09-10T01:00:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-09-10T01:00:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-09-10T01:00:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2023-09-10T01:00:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2023-09-10T01:00:57.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-09-10T01:00:57.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]': finished 2023-09-10T01:00:57.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-09-10T01:00:57.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:56 smithi092 ceph-mon[134067]: osdmap e87: 12 total, 12 up, 12 in 2023-09-10T01:00:57.511 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-09-10T01:00:57.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-09-10T01:00:57.583 INFO:teuthology.orchestra.run.smithi003.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-09-10T01:00:57.585 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-09-10T01:00:57.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:57.587 DEBUG:teuthology.orchestra.run.smithi003:> ls /sys/fs/fuse/connections 2023-09-10T01:00:57.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi003.stderr:2023-09-10T01:00:57.639+0000 7ff94caeb5c0 -1 init, newargv = 0x559702f4c210 newargc=15 2023-09-10T01:00:57.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi003.stderr:ceph-fuse[155757]: starting ceph client 2023-09-10T01:00:57.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi003.stderr:ceph-fuse[155757]: starting fuse 2023-09-10T01:00:58.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:57 smithi003 ceph-mon[97654]: osdmap e88: 12 total, 12 up, 12 in 2023-09-10T01:00:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:57 smithi136 ceph-mon[133708]: osdmap e88: 12 total, 12 up, 12 in 2023-09-10T01:00:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:57 smithi092 ceph-mon[134067]: osdmap e88: 12 total, 12 up, 12 in 2023-09-10T01:00:58.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo modprobe fuse 2023-09-10T01:00:58.811 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-09-10T01:00:58.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-09-10T01:00:58.882 INFO:teuthology.orchestra.run.smithi003.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-09-10T01:00:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-09-10T01:00:58.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:58.886 DEBUG:teuthology.orchestra.run.smithi003:> ls /sys/fs/fuse/connections 2023-09-10T01:00:58.930 INFO:teuthology.orchestra.run.smithi003.stdout:121 2023-09-10T01:00:58.931 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2023-09-10T01:00:58.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo stdin-killer -- python3 -c ' 2023-09-10T01:00:58.931 DEBUG:teuthology.orchestra.run.smithi003:> import glob 2023-09-10T01:00:58.932 DEBUG:teuthology.orchestra.run.smithi003:> import re 2023-09-10T01:00:58.932 DEBUG:teuthology.orchestra.run.smithi003:> import os 2023-09-10T01:00:58.932 DEBUG:teuthology.orchestra.run.smithi003:> import subprocess 2023-09-10T01:00:58.933 DEBUG:teuthology.orchestra.run.smithi003:> 2023-09-10T01:00:58.933 DEBUG:teuthology.orchestra.run.smithi003:> def _find_admin_socket(client_name): 2023-09-10T01:00:58.933 DEBUG:teuthology.orchestra.run.smithi003:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-09-10T01:00:58.934 DEBUG:teuthology.orchestra.run.smithi003:> files = glob.glob(asok_path) 2023-09-10T01:00:58.934 DEBUG:teuthology.orchestra.run.smithi003:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-09-10T01:00:58.934 DEBUG:teuthology.orchestra.run.smithi003:> 2023-09-10T01:00:58.935 DEBUG:teuthology.orchestra.run.smithi003:> # Given a non-glob path, it better be there 2023-09-10T01:00:58.935 DEBUG:teuthology.orchestra.run.smithi003:> if "*" not in asok_path: 2023-09-10T01:00:58.935 DEBUG:teuthology.orchestra.run.smithi003:> assert(len(files) == 1) 2023-09-10T01:00:58.936 DEBUG:teuthology.orchestra.run.smithi003:> return files[0] 2023-09-10T01:00:58.936 DEBUG:teuthology.orchestra.run.smithi003:> 2023-09-10T01:00:58.936 DEBUG:teuthology.orchestra.run.smithi003:> for f in files: 2023-09-10T01:00:58.937 DEBUG:teuthology.orchestra.run.smithi003:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-09-10T01:00:58.937 DEBUG:teuthology.orchestra.run.smithi003:> if os.path.exists("/proc/{0}".format(pid)): 2023-09-10T01:00:58.938 DEBUG:teuthology.orchestra.run.smithi003:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-09-10T01:00:58.938 DEBUG:teuthology.orchestra.run.smithi003:> contents = proc_f.read() 2023-09-10T01:00:58.938 DEBUG:teuthology.orchestra.run.smithi003:> if mountpoint in contents: 2023-09-10T01:00:58.939 DEBUG:teuthology.orchestra.run.smithi003:> return f 2023-09-10T01:00:58.939 DEBUG:teuthology.orchestra.run.smithi003:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-09-10T01:00:58.939 DEBUG:teuthology.orchestra.run.smithi003:> 2023-09-10T01:00:58.940 DEBUG:teuthology.orchestra.run.smithi003:> print(_find_admin_socket("client.admin")) 2023-09-10T01:00:58.940 DEBUG:teuthology.orchestra.run.smithi003:> ' 2023-09-10T01:00:59.116 INFO:teuthology.orchestra.run.smithi003.stdout:/var/run/ceph/ceph-client.admin.155757.asok 2023-09-10T01:00:59.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-09-10T01:00:59.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:00:58 smithi003 ceph-mon[97654]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-09-10T01:00:59.130 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.155757.asok 2023-09-10T01:00:59.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:59.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.155757.asok status 2023-09-10T01:00:59.360 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:00:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: "metadata": { 2023-09-10T01:00:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_sha1": "9670cca5bc8e17916865dc84939edfc10b1f2c00", 2023-09-10T01:00:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 18.0.0-6025-g9670cca5 (9670cca5bc8e17916865dc84939edfc10b1f2c00) reef (dev)", 2023-09-10T01:00:59.361 INFO:teuthology.orchestra.run.smithi003.stdout: "entity_id": "admin", 2023-09-10T01:00:59.362 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": "smithi003", 2023-09-10T01:00:59.362 INFO:teuthology.orchestra.run.smithi003.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-09-10T01:00:59.362 INFO:teuthology.orchestra.run.smithi003.stdout: "pid": "155757", 2023-09-10T01:00:59.362 INFO:teuthology.orchestra.run.smithi003.stdout: "root": "/" 2023-09-10T01:00:59.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:00:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "dentry_count": 0, 2023-09-10T01:00:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "dentry_pinned_count": 0, 2023-09-10T01:00:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 14778, 2023-09-10T01:00:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "inst": { 2023-09-10T01:00:59.363 INFO:teuthology.orchestra.run.smithi003.stdout: "name": { 2023-09-10T01:00:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "client", 2023-09-10T01:00:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: "num": 14778 2023-09-10T01:00:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:00:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": { 2023-09-10T01:00:59.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2023-09-10T01:00:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:00:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 972297933 2023-09-10T01:00:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:00:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:00:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": { 2023-09-10T01:00:59.365 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2023-09-10T01:00:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:00:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 972297933 2023-09-10T01:00:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:00:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "inst_str": "client.14778 172.21.15.3:0/972297933", 2023-09-10T01:00:59.366 INFO:teuthology.orchestra.run.smithi003.stdout: "addr_str": "172.21.15.3:0/972297933", 2023-09-10T01:00:59.367 INFO:teuthology.orchestra.run.smithi003.stdout: "inode_count": 1, 2023-09-10T01:00:59.367 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_epoch": 16, 2023-09-10T01:00:59.367 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_epoch": 88, 2023-09-10T01:00:59.367 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_epoch_barrier": 0, 2023-09-10T01:00:59.368 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklisted": false, 2023-09-10T01:00:59.368 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs" 2023-09-10T01:00:59.368 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:00:59.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:59.376 DEBUG:teuthology.orchestra.run.smithi003:> stat --file-system '--printf=%T 2023-09-10T01:00:59.376 DEBUG:teuthology.orchestra.run.smithi003:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:00:59.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:00:58 smithi136 ceph-mon[133708]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-09-10T01:00:59.421 INFO:teuthology.orchestra.run.smithi003.stdout:fuseblk 2023-09-10T01:00:59.421 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:00:59.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:59.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:00:59.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:00:58 smithi092 ceph-mon[134067]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-09-10T01:00:59.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:00:59.534 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-09-10T01:00:59.655 INFO:teuthology.orchestra.run.smithi003.stdout:# file: . 2023-09-10T01:00:59.655 INFO:teuthology.orchestra.run.smithi003.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-09-10T01:00:59.656 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:00:59.657 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T01:00:59.658 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2023-09-10T01:00:59.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:00:59.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 -- 172.21.15.3:0/200991223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f8072fe0 msgr2=0x7f99f8073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:59.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/200991223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f8072fe0 0x7f99f8073460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0098f0 tx=0x7f99ec009c00 comp rx=0 tx=0).stop 2023-09-10T01:00:59.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 -- 172.21.15.3:0/200991223 shutdown_connections 2023-09-10T01:00:59.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/200991223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f81085c0 0x7f99f810aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:59.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/200991223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f80745c0 0x7f99f8072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:59.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/200991223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f8072fe0 0x7f99f8073460 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:59.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 -- 172.21.15.3:0/200991223 >> 172.21.15.3:0/200991223 conn(0x7f99f80fbc60 msgr2=0x7f99f80fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:00:59.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 -- 172.21.15.3:0/200991223 shutdown_connections 2023-09-10T01:00:59.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.946+0000 7f99fe7ff700 1 -- 172.21.15.3:0/200991223 wait complete. 2023-09-10T01:00:59.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.947+0000 7f99fe7ff700 1 Processor -- start 2023-09-10T01:00:59.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.947+0000 7f99fe7ff700 1 -- start start 2023-09-10T01:00:59.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99fe7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f8072fe0 0x7f99f8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:59.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99fe7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f80745c0 0x7f99f8198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:59.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99fe7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f81085c0 0x7f99f819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:59.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99fe7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f81a01b0 con 0x7f99f8072fe0 2023-09-10T01:00:59.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99fe7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f81a02f0 con 0x7f99f80745c0 2023-09-10T01:00:59.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99fe7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f81a0480 con 0x7f99f81085c0 2023-09-10T01:00:59.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f8072fe0 0x7f99f8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:59.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f8072fe0 0x7f99f8198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49530/0 (socket says 172.21.15.3:49530) 2023-09-10T01:00:59.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 -- 172.21.15.3:0/2748042147 learned_addr learned my addr 172.21.15.3:0/2748042147 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:00:59.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f77fe700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f80745c0 0x7f99f8198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:59.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99fcd9c700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f81085c0 0x7f99f819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:59.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 -- 172.21.15.3:0/2748042147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f81085c0 msgr2=0x7f99f819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:59.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f81085c0 0x7f99f819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:59.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 -- 172.21.15.3:0/2748042147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f80745c0 msgr2=0x7f99f8198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:00:59.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f80745c0 0x7f99f8198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:00:59.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f7fff700 1 -- 172.21.15.3:0/2748042147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8009520 con 0x7f99f8072fe0 2023-09-10T01:00:59.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.948+0000 7f99f77fe700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f80745c0 0x7f99f8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:00:59.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.949+0000 7f99f7fff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f8072fe0 0x7f99f8198630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f99e800eb30 tx=0x7f99e800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:59.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.949+0000 7f99f57fa700 1 -- 172.21.15.3:0/2748042147 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8009880 con 0x7f99f8072fe0 2023-09-10T01:00:59.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.949+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f0009d00 con 0x7f99f8072fe0 2023-09-10T01:00:59.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.949+0000 7f99f57fa700 1 -- 172.21.15.3:0/2748042147 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99e80044e0 con 0x7f99f8072fe0 2023-09-10T01:00:59.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.949+0000 7f99f57fa700 1 -- 172.21.15.3:0/2748042147 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8003d80 con 0x7f99f8072fe0 2023-09-10T01:00:59.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.949+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec009520 con 0x7f99f8072fe0 2023-09-10T01:00:59.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.950+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f8106cc0 con 0x7f99f8072fe0 2023-09-10T01:00:59.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.952+0000 7f99f57fa700 1 -- 172.21.15.3:0/2748042147 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f99e8003f00 con 0x7f99f8072fe0 2023-09-10T01:00:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.953+0000 7f99f57fa700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f99e00a0640 0x7f99e00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:00:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.954+0000 7f99f57fa700 1 -- 172.21.15.3:0/2748042147 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f99e8014070 con 0x7f99f8072fe0 2023-09-10T01:00:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.954+0000 7f99f77fe700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f99e00a0640 0x7f99e00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:00:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.954+0000 7f99f77fe700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f99e00a0640 0x7f99e00a2b00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f99ec005e90 tx=0x7f99ec019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:00:59.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:00:59.956+0000 7f99f57fa700 1 -- 172.21.15.3:0/2748042147 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f99e8095d50 con 0x7f99f8072fe0 2023-09-10T01:01:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.174+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99f810e110 con 0x7f99f8072fe0 2023-09-10T01:01:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.176+0000 7f99f57fa700 1 -- 172.21.15.3:0/2748042147 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24321 (secure 0 0 0) 0x7f99e800fe60 con 0x7f99f8072fe0 2023-09-10T01:01:00.177 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:00.177 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:56.950998+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:01:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f99e00a0640 msgr2=0x7f99e00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f99e00a0640 0x7f99e00a2b00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f99ec005e90 tx=0x7f99ec019040 comp rx=0 tx=0).stop 2023-09-10T01:01:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f8072fe0 msgr2=0x7f99f8198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f8072fe0 0x7f99f8198630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f99e800eb30 tx=0x7f99e800ee40 comp rx=0 tx=0).stop 2023-09-10T01:01:00.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 shutdown_connections 2023-09-10T01:01:00.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f99e00a0640 0x7f99e00a2b00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99f81085c0 0x7f99f819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99f8072fe0 0x7f99f8198630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 --2- 172.21.15.3:0/2748042147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f80745c0 0x7f99f8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 >> 172.21.15.3:0/2748042147 conn(0x7f99f80fbc60 msgr2=0x7f99f80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:00.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 shutdown_connections 2023-09-10T01:01:00.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.178+0000 7f99fe7ff700 1 -- 172.21.15.3:0/2748042147 wait complete. 2023-09-10T01:01:00.197 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:01:00.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:01:00.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 -- 172.21.15.3:0/542372453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c81018e0 msgr2=0x7f05c8101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 --2- 172.21.15.3:0/542372453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c81018e0 0x7f05c8101d40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0099c0 tx=0x7f05bc009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 -- 172.21.15.3:0/542372453 shutdown_connections 2023-09-10T01:01:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 --2- 172.21.15.3:0/542372453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05c81006f0 0x7f05c8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 --2- 172.21.15.3:0/542372453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c8102280 0x7f05c8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 --2- 172.21.15.3:0/542372453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c81018e0 0x7f05c8101d40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 -- 172.21.15.3:0/542372453 >> 172.21.15.3:0/542372453 conn(0x7f05c80fbc60 msgr2=0x7f05c80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:00.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 -- 172.21.15.3:0/542372453 shutdown_connections 2023-09-10T01:01:00.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.403+0000 7f05cdbff700 1 -- 172.21.15.3:0/542372453 wait complete. 2023-09-10T01:01:00.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.404+0000 7f05cdbff700 1 Processor -- start 2023-09-10T01:01:00.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.404+0000 7f05cdbff700 1 -- start start 2023-09-10T01:01:00.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05cdbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c81006f0 0x7f05c81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05cdbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05c81018e0 0x7f05c8198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05cdbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c8102280 0x7f05c819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05cdbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c81a0150 con 0x7f05c8102280 2023-09-10T01:01:00.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05cdbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c81a0290 con 0x7f05c81006f0 2023-09-10T01:01:00.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05cdbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c81a0420 con 0x7f05c81018e0 2023-09-10T01:01:00.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c81006f0 0x7f05c81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c81006f0 0x7f05c81985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59318/0 (socket says 172.21.15.3:59318) 2023-09-10T01:01:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 -- 172.21.15.3:0/2836286687 learned_addr learned my addr 172.21.15.3:0/2836286687 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c7fff700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c8102280 0x7f05c819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c6ffd700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05c81018e0 0x7f05c8198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 -- 172.21.15.3:0/2836286687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05c81018e0 msgr2=0x7f05c8198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05c81018e0 0x7f05c8198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 -- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c8102280 msgr2=0x7f05c819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c8102280 0x7f05c819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.405+0000 7f05c77fe700 1 -- 172.21.15.3:0/2836286687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc0095f0 con 0x7f05c81006f0 2023-09-10T01:01:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.406+0000 7f05c77fe700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c81006f0 0x7f05c81985d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f05b800e970 tx=0x7f05b800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.406+0000 7f05c4ff9700 1 -- 172.21.15.3:0/2836286687 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8009790 con 0x7f05c81006f0 2023-09-10T01:01:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.406+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c81a0700 con 0x7f05c81006f0 2023-09-10T01:01:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.406+0000 7f05c4ff9700 1 -- 172.21.15.3:0/2836286687 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05b800eec0 con 0x7f05c81006f0 2023-09-10T01:01:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.406+0000 7f05c4ff9700 1 -- 172.21.15.3:0/2836286687 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8003cf0 con 0x7f05c81006f0 2023-09-10T01:01:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.406+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c81a0c10 con 0x7f05c81006f0 2023-09-10T01:01:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.407+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c810bf60 con 0x7f05c81006f0 2023-09-10T01:01:00.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.409+0000 7f05c4ff9700 1 -- 172.21.15.3:0/2836286687 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f05b800ccd0 con 0x7f05c81006f0 2023-09-10T01:01:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.411+0000 7f05c4ff9700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05b00a0640 0x7f05b00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.411+0000 7f05c4ff9700 1 -- 172.21.15.3:0/2836286687 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f05b8014070 con 0x7f05c81006f0 2023-09-10T01:01:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.411+0000 7f05c6ffd700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05b00a0640 0x7f05b00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.411+0000 7f05c6ffd700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05b00a0640 0x7f05b00a2b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00b580 tx=0x7f05bc01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:00.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.413+0000 7f05c4ff9700 1 -- 172.21.15.3:0/2836286687 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f05b8095b90 con 0x7f05c81006f0 2023-09-10T01:01:00.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.627+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05c81a0e90 con 0x7f05c81006f0 2023-09-10T01:01:00.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.627+0000 7f05c4ff9700 1 -- 172.21.15.3:0/2836286687 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05b80991b0 con 0x7f05c81006f0 2023-09-10T01:01:00.628 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:01:00.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05b00a0640 msgr2=0x7f05b00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05b00a0640 0x7f05b00a2b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00b580 tx=0x7f05bc01a040 comp rx=0 tx=0).stop 2023-09-10T01:01:00.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c81006f0 msgr2=0x7f05c81985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c81006f0 0x7f05c81985d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f05b800e970 tx=0x7f05b800ec80 comp rx=0 tx=0).stop 2023-09-10T01:01:00.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 shutdown_connections 2023-09-10T01:01:00.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05c8102280 0x7f05c819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c81006f0 0x7f05c81985d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05b00a0640 0x7f05b00a2b00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 --2- 172.21.15.3:0/2836286687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05c81018e0 0x7f05c8198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 >> 172.21.15.3:0/2836286687 conn(0x7f05c80fbc60 msgr2=0x7f05c80fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:00.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 shutdown_connections 2023-09-10T01:01:00.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.630+0000 7f05cdbff700 1 -- 172.21.15.3:0/2836286687 wait complete. 2023-09-10T01:01:00.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:01:00.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 -- 172.21.15.3:0/4238785591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb81006f0 msgr2=0x7efcb8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 --2- 172.21.15.3:0/4238785591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb81006f0 0x7efcb8100b10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efcac009990 tx=0x7efcac009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 -- 172.21.15.3:0/4238785591 shutdown_connections 2023-09-10T01:01:00.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 --2- 172.21.15.3:0/4238785591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb81006f0 0x7efcb8100b10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 --2- 172.21.15.3:0/4238785591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8102280 0x7efcb8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 --2- 172.21.15.3:0/4238785591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb81018e0 0x7efcb8101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 -- 172.21.15.3:0/4238785591 >> 172.21.15.3:0/4238785591 conn(0x7efcb80fbc60 msgr2=0x7efcb80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:00.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 -- 172.21.15.3:0/4238785591 shutdown_connections 2023-09-10T01:01:00.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.858+0000 7efcbedff700 1 -- 172.21.15.3:0/4238785591 wait complete. 2023-09-10T01:01:00.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 Processor -- start 2023-09-10T01:01:00.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 -- start start 2023-09-10T01:01:00.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb81006f0 0x7efcb8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb81018e0 0x7efcb8198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb8102280 0x7efcb819fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb81a0090 con 0x7efcb81006f0 2023-09-10T01:01:00.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb81a01d0 con 0x7efcb81018e0 2023-09-10T01:01:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.859+0000 7efcbedff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb81a0360 con 0x7efcb8102280 2023-09-10T01:01:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcbcb9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb81006f0 0x7efcb8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb81018e0 0x7efcb8198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb81018e0 0x7efcb8198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59344/0 (socket says 172.21.15.3:59344) 2023-09-10T01:01:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 -- 172.21.15.3:0/2535744498 learned_addr learned my addr 172.21.15.3:0/2535744498 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:00.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcbd39c700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb8102280 0x7efcb819fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 -- 172.21.15.3:0/2535744498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb8102280 msgr2=0x7efcb819fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb8102280 0x7efcb819fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 -- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb81006f0 msgr2=0x7efcb8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:00.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb81006f0 0x7efcb8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:00.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 -- 172.21.15.3:0/2535744498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac0095f0 con 0x7efcb81018e0 2023-09-10T01:01:00.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb7fff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb81018e0 0x7efcb8198a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efca400e9a0 tx=0x7efca400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:00.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.860+0000 7efcb5ffb700 1 -- 172.21.15.3:0/2535744498 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4009790 con 0x7efcb81018e0 2023-09-10T01:01:00.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.861+0000 7efcb5ffb700 1 -- 172.21.15.3:0/2535744498 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efca4010430 con 0x7efcb81018e0 2023-09-10T01:01:00.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.861+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb81a0640 con 0x7efcb81018e0 2023-09-10T01:01:00.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.861+0000 7efcb5ffb700 1 -- 172.21.15.3:0/2535744498 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4018680 con 0x7efcb81018e0 2023-09-10T01:01:00.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.861+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb81a0ba0 con 0x7efcb81018e0 2023-09-10T01:01:00.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.862+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb8192cf0 con 0x7efcb81018e0 2023-09-10T01:01:00.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.864+0000 7efcb5ffb700 1 -- 172.21.15.3:0/2535744498 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7efca40105a0 con 0x7efcb81018e0 2023-09-10T01:01:00.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.865+0000 7efcb5ffb700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efca80a06c0 0x7efca80a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:00.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.865+0000 7efcb5ffb700 1 -- 172.21.15.3:0/2535744498 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efca4014070 con 0x7efcb81018e0 2023-09-10T01:01:00.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.865+0000 7efcbcb9b700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efca80a06c0 0x7efca80a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:00.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.866+0000 7efcbcb9b700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efca80a06c0 0x7efca80a2b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efcac00b680 tx=0x7efcac005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:00.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:00.867+0000 7efcb5ffb700 1 -- 172.21.15.3:0/2535744498 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7efca4095b20 con 0x7efcb81018e0 2023-09-10T01:01:01.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.081+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcb810c090 con 0x7efcb81018e0 2023-09-10T01:01:01.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.081+0000 7efcb5ffb700 1 -- 172.21.15.3:0/2535744498 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7efca4099140 con 0x7efcb81018e0 2023-09-10T01:01:01.082 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:01:01.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efca80a06c0 msgr2=0x7efca80a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efca80a06c0 0x7efca80a2b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efcac00b680 tx=0x7efcac005fd0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb81018e0 msgr2=0x7efcb8198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb81018e0 0x7efcb8198a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efca400e9a0 tx=0x7efca400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 shutdown_connections 2023-09-10T01:01:01.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efca80a06c0 0x7efca80a2b80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb8102280 0x7efcb819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb81006f0 0x7efcb8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 --2- 172.21.15.3:0/2535744498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb81018e0 0x7efcb8198a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 >> 172.21.15.3:0/2535744498 conn(0x7efcb80fbc60 msgr2=0x7efcb81071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:01.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 shutdown_connections 2023-09-10T01:01:01.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.084+0000 7efcbedff700 1 -- 172.21.15.3:0/2535744498 wait complete. 2023-09-10T01:01:01.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:01:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:00 smithi003 ceph-mon[97654]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:01:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:00 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2748042147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:00 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2836286687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:01:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 -- 172.21.15.3:0/3655969623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 msgr2=0x7f0578072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 --2- 172.21.15.3:0/3655969623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578072a10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f05680098f0 tx=0x7f0568009c00 comp rx=0 tx=0).stop 2023-09-10T01:01:01.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 -- 172.21.15.3:0/3655969623 shutdown_connections 2023-09-10T01:01:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 --2- 172.21.15.3:0/3655969623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05781085e0 0x7f057810aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 --2- 172.21.15.3:0/3655969623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578072a10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 --2- 172.21.15.3:0/3655969623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0578072fe0 0x7f0578073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 -- 172.21.15.3:0/3655969623 >> 172.21.15.3:0/3655969623 conn(0x7f05780fbc60 msgr2=0x7f05780fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 -- 172.21.15.3:0/3655969623 shutdown_connections 2023-09-10T01:01:01.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.311+0000 7f057d3ff700 1 -- 172.21.15.3:0/3655969623 wait complete. 2023-09-10T01:01:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.312+0000 7f057d3ff700 1 Processor -- start 2023-09-10T01:01:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f057d3ff700 1 -- start start 2023-09-10T01:01:01.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f057d3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0578072fe0 0x7f05781985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f057d3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f057d3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05781085e0 0x7f057819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f057d3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05781a0130 con 0x7f05781085e0 2023-09-10T01:01:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f057d3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05781a0270 con 0x7f05780745c0 2023-09-10T01:01:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f057d3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05781a0400 con 0x7f0578072fe0 2023-09-10T01:01:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f05777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05781085e0 0x7f057819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f05767fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f0576ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0578072fe0 0x7f05781985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f05777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05781085e0 0x7f057819fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49598/0 (socket says 172.21.15.3:49598) 2023-09-10T01:01:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f05767fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59362/0 (socket says 172.21.15.3:59362) 2023-09-10T01:01:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.313+0000 7f05777fe700 1 -- 172.21.15.3:0/1884044499 learned_addr learned my addr 172.21.15.3:0/1884044499 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f05767fc700 1 -- 172.21.15.3:0/1884044499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0578072fe0 msgr2=0x7f05781985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f05767fc700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0578072fe0 0x7f05781985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f05767fc700 1 -- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05781085e0 msgr2=0x7f057819fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f05767fc700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05781085e0 0x7f057819fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f05767fc700 1 -- 172.21.15.3:0/1884044499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0568009520 con 0x7f05780745c0 2023-09-10T01:01:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f05777fe700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05781085e0 0x7f057819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f05767fc700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578198af0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f056c00e9c0 tx=0x7f056c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f055ffff700 1 -- 172.21.15.3:0/1884044499 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00c9c0 con 0x7f05780745c0 2023-09-10T01:01:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05781a06e0 con 0x7f05780745c0 2023-09-10T01:01:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f055ffff700 1 -- 172.21.15.3:0/1884044499 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f056c010430 con 0x7f05780745c0 2023-09-10T01:01:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f055ffff700 1 -- 172.21.15.3:0/1884044499 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c018680 con 0x7f05780745c0 2023-09-10T01:01:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.314+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05781a0c40 con 0x7f05780745c0 2023-09-10T01:01:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.315+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0578106ce0 con 0x7f05780745c0 2023-09-10T01:01:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.317+0000 7f055ffff700 1 -- 172.21.15.3:0/1884044499 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f056c010860 con 0x7f05780745c0 2023-09-10T01:01:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.319+0000 7f055ffff700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05600a0640 0x7f05600a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.319+0000 7f055ffff700 1 -- 172.21.15.3:0/1884044499 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f056c014070 con 0x7f05780745c0 2023-09-10T01:01:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.319+0000 7f0576ffd700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05600a0640 0x7f05600a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.320+0000 7f0576ffd700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05600a0640 0x7f05600a2b00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f056800fa40 tx=0x7f0568019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.320+0000 7f055ffff700 1 -- 172.21.15.3:0/1884044499 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f056c095950 con 0x7f05780745c0 2023-09-10T01:01:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:00 smithi136 ceph-mon[133708]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:01:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:00 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2748042147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:00 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2836286687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:01:01.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:00 smithi092 ceph-mon[134067]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:01:01.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:00 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2748042147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:01.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:00 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2836286687' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:01:01.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.528+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f057810e0e0 con 0x7f05780745c0 2023-09-10T01:01:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.529+0000 7f055ffff700 1 -- 172.21.15.3:0/1884044499 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f056c082e60 con 0x7f05780745c0 2023-09-10T01:01:01.530 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:01:01.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05600a0640 msgr2=0x7f05600a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05600a0640 0x7f05600a2b00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f056800fa40 tx=0x7f0568019040 comp rx=0 tx=0).stop 2023-09-10T01:01:01.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 msgr2=0x7f0578198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578198af0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f056c00e9c0 tx=0x7f056c00ecd0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 shutdown_connections 2023-09-10T01:01:01.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f05600a0640 0x7f05600a2b00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0578072fe0 0x7f05781985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05781085e0 0x7f057819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 --2- 172.21.15.3:0/1884044499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05780745c0 0x7f0578198af0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 >> 172.21.15.3:0/1884044499 conn(0x7f05780fbc60 msgr2=0x7f05780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:01.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 shutdown_connections 2023-09-10T01:01:01.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.532+0000 7f057d3ff700 1 -- 172.21.15.3:0/1884044499 wait complete. 2023-09-10T01:01:01.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:01:01.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 -- 172.21.15.3:0/2447935138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c073060 msgr2=0x7f0f8c0734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 --2- 172.21.15.3:0/2447935138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c073060 0x7f0f8c0734e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009990 tx=0x7f0f74009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 -- 172.21.15.3:0/2447935138 shutdown_connections 2023-09-10T01:01:01.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 --2- 172.21.15.3:0/2447935138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f8c074640 0x7f0f8c072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 --2- 172.21.15.3:0/2447935138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f8c108570 0x7f0f8c10aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 --2- 172.21.15.3:0/2447935138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c073060 0x7f0f8c0734e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 -- 172.21.15.3:0/2447935138 >> 172.21.15.3:0/2447935138 conn(0x7f0f8c0fbc20 msgr2=0x7f0f8c0fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:01.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 -- 172.21.15.3:0/2447935138 shutdown_connections 2023-09-10T01:01:01.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.758+0000 7f0f921ff700 1 -- 172.21.15.3:0/2447935138 wait complete. 2023-09-10T01:01:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.759+0000 7f0f921ff700 1 Processor -- start 2023-09-10T01:01:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f921ff700 1 -- start start 2023-09-10T01:01:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f921ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f8c073060 0x7f0f8c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f921ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c074640 0x7f0f8c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f921ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f8c108570 0x7f0f8c19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f921ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c1a0190 con 0x7f0f8c074640 2023-09-10T01:01:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f921ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c1a02d0 con 0x7f0f8c108570 2023-09-10T01:01:01.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f921ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c1a0460 con 0x7f0f8c073060 2023-09-10T01:01:01.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f83fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c074640 0x7f0f8c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f83fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c074640 0x7f0f8c198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49606/0 (socket says 172.21.15.3:49606) 2023-09-10T01:01:01.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f83fff700 1 -- 172.21.15.3:0/798425558 learned_addr learned my addr 172.21.15.3:0/798425558 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:01.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f8bfff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f8c108570 0x7f0f8c19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.760+0000 7f0f8b7fe700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f8c073060 0x7f0f8c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f83fff700 1 -- 172.21.15.3:0/798425558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f8c073060 msgr2=0x7f0f8c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f83fff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f8c073060 0x7f0f8c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f83fff700 1 -- 172.21.15.3:0/798425558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f8c108570 msgr2=0x7f0f8c19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f83fff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f8c108570 0x7f0f8c19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f83fff700 1 -- 172.21.15.3:0/798425558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f740095f0 con 0x7f0f8c074640 2023-09-10T01:01:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f83fff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c074640 0x7f0f8c198b50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0f74004710 tx=0x7f0f74009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f897fa700 1 -- 172.21.15.3:0/798425558 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7401d070 con 0x7f0f8c074640 2023-09-10T01:01:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c1a06e0 con 0x7f0f8c074640 2023-09-10T01:01:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f897fa700 1 -- 172.21.15.3:0/798425558 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f74021410 con 0x7f0f8c074640 2023-09-10T01:01:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f897fa700 1 -- 172.21.15.3:0/798425558 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f74017650 con 0x7f0f8c074640 2023-09-10T01:01:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.761+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c1a0b00 con 0x7f0f8c074640 2023-09-10T01:01:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.763+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c106c20 con 0x7f0f8c074640 2023-09-10T01:01:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.764+0000 7f0f897fa700 1 -- 172.21.15.3:0/798425558 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f0f74021580 con 0x7f0f8c074640 2023-09-10T01:01:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.767+0000 7f0f897fa700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f780a0640 0x7f0f780a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.767+0000 7f0f897fa700 1 -- 172.21.15.3:0/798425558 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0f740cd2f0 con 0x7f0f8c074640 2023-09-10T01:01:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.767+0000 7f0f8b7fe700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f780a0640 0x7f0f780a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.768+0000 7f0f8b7fe700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f780a0640 0x7f0f780a2b00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c009c60 tx=0x7f0f7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:01.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.771+0000 7f0f897fa700 1 -- 172.21.15.3:0/798425558 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0f74096b40 con 0x7f0f8c074640 2023-09-10T01:01:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.991+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0f8c02ce60 con 0x7f0f8c074640 2023-09-10T01:01:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.991+0000 7f0f897fa700 1 -- 172.21.15.3:0/798425558 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0f7409a160 con 0x7f0f8c074640 2023-09-10T01:01:01.993 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:01:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f780a0640 msgr2=0x7f0f780a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f780a0640 0x7f0f780a2b00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c009c60 tx=0x7f0f7c009260 comp rx=0 tx=0).stop 2023-09-10T01:01:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c074640 msgr2=0x7f0f8c198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c074640 0x7f0f8c198b50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0f74004710 tx=0x7f0f74009580 comp rx=0 tx=0).stop 2023-09-10T01:01:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 shutdown_connections 2023-09-10T01:01:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f780a0640 0x7f0f780a2b00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f8c073060 0x7f0f8c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f8c108570 0x7f0f8c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 --2- 172.21.15.3:0/798425558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f8c074640 0x7f0f8c198b50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:01.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 >> 172.21.15.3:0/798425558 conn(0x7f0f8c0fbc20 msgr2=0x7f0f8c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:01.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 shutdown_connections 2023-09-10T01:01:01.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:01.994+0000 7f0f921ff700 1 -- 172.21.15.3:0/798425558 wait complete. 2023-09-10T01:01:02.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-09-10T01:01:02.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:01 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2535744498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:01:02.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:01 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1884044499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:01:02.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.217+0000 7fb5135ff700 1 -- 172.21.15.3:0/396058457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c1018c0 msgr2=0x7fb50c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.217+0000 7fb5135ff700 1 --2- 172.21.15.3:0/396058457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c1018c0 0x7fb50c101d20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0099c0 tx=0x7fb4fc009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.217+0000 7fb5135ff700 1 -- 172.21.15.3:0/396058457 shutdown_connections 2023-09-10T01:01:02.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.217+0000 7fb5135ff700 1 --2- 172.21.15.3:0/396058457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c1006d0 0x7fb50c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.217+0000 7fb5135ff700 1 --2- 172.21.15.3:0/396058457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c102260 0x7fb50c108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.217+0000 7fb5135ff700 1 --2- 172.21.15.3:0/396058457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c1018c0 0x7fb50c101d20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.217+0000 7fb5135ff700 1 -- 172.21.15.3:0/396058457 >> 172.21.15.3:0/396058457 conn(0x7fb50c0fbc20 msgr2=0x7fb50c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:02.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.218+0000 7fb5135ff700 1 -- 172.21.15.3:0/396058457 shutdown_connections 2023-09-10T01:01:02.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.218+0000 7fb5135ff700 1 -- 172.21.15.3:0/396058457 wait complete. 2023-09-10T01:01:02.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.218+0000 7fb5135ff700 1 Processor -- start 2023-09-10T01:01:02.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb5135ff700 1 -- start start 2023-09-10T01:01:02.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb5135ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c1006d0 0x7fb50c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb5135ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c1018c0 0x7fb50c1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb5135ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c102260 0x7fb50c19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb5135ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c19df30 con 0x7fb50c102260 2023-09-10T01:01:02.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb5135ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c19e070 con 0x7fb50c1018c0 2023-09-10T01:01:02.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb5135ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50c19e200 con 0x7fb50c1006d0 2023-09-10T01:01:02.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb510b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c1018c0 0x7fb50c1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb510b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c1018c0 0x7fb50c1968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59402/0 (socket says 172.21.15.3:59402) 2023-09-10T01:01:02.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb510b9a700 1 -- 172.21.15.3:0/545347870 learned_addr learned my addr 172.21.15.3:0/545347870 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:02.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb511b9c700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c102260 0x7fb50c19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.219+0000 7fb51139b700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c1006d0 0x7fb50c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb510b9a700 1 -- 172.21.15.3:0/545347870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c1006d0 msgr2=0x7fb50c1963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb510b9a700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c1006d0 0x7fb50c1963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb510b9a700 1 -- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c102260 msgr2=0x7fb50c19d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb510b9a700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c102260 0x7fb50c19d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb510b9a700 1 -- 172.21.15.3:0/545347870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc0095f0 con 0x7fb50c1018c0 2023-09-10T01:01:02.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb511b9c700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c102260 0x7fb50c19d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:02.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb510b9a700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c1018c0 0x7fb50c1968f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009990 tx=0x7fb4fc00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:02.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb5027fc700 1 -- 172.21.15.3:0/545347870 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc01d070 con 0x7fb50c1018c0 2023-09-10T01:01:02.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb5027fc700 1 -- 172.21.15.3:0/545347870 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4fc005030 con 0x7fb50c1018c0 2023-09-10T01:01:02.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50c19e480 con 0x7fb50c1018c0 2023-09-10T01:01:02.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb5027fc700 1 -- 172.21.15.3:0/545347870 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc004a70 con 0x7fb50c1018c0 2023-09-10T01:01:02.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.220+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50c19e8a0 con 0x7fb50c1018c0 2023-09-10T01:01:02.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.221+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50c107600 con 0x7fb50c1018c0 2023-09-10T01:01:02.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.223+0000 7fb5027fc700 1 -- 172.21.15.3:0/545347870 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb4fc005410 con 0x7fb50c1018c0 2023-09-10T01:01:02.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.225+0000 7fb5027fc700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb4f80a0640 0x7fb4f80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.225+0000 7fb5027fc700 1 -- 172.21.15.3:0/545347870 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb4fc0cd160 con 0x7fb50c1018c0 2023-09-10T01:01:02.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.225+0000 7fb51139b700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb4f80a0640 0x7fb4f80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.225+0000 7fb51139b700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb4f80a0640 0x7fb4f80a2b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb50c101720 tx=0x7fb508009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:02.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.227+0000 7fb5027fc700 1 -- 172.21.15.3:0/545347870 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb4fc0969b0 con 0x7fb50c1018c0 2023-09-10T01:01:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2535744498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:01:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1884044499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:01:02.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.452+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb50c02cfc0 con 0x7fb50c1018c0 2023-09-10T01:01:02.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.453+0000 7fb5027fc700 1 -- 172.21.15.3:0/545347870 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) v1 ==== 81+0+575 (secure 0 0 0) 0x7fb4fc026250 con 0x7fb50c1018c0 2023-09-10T01:01:02.454 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/1916209618 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:02.455 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/70910038 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:02.455 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/71687535 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:02.455 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/3193301911 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:02.455 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1972995090 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:02.456 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/4153284543 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:02.456 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1689603322 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:02.456 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/1916209618 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:02.456 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/70910038 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:02.456 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/886409476 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.455+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb4f80a0640 msgr2=0x7fb4f80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.455+0000 7fb5135ff700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb4f80a0640 0x7fb4f80a2b00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb50c101720 tx=0x7fb508009260 comp rx=0 tx=0).stop 2023-09-10T01:01:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c1018c0 msgr2=0x7fb50c1968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c1018c0 0x7fb50c1968f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009990 tx=0x7fb4fc00f690 comp rx=0 tx=0).stop 2023-09-10T01:01:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 shutdown_connections 2023-09-10T01:01:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb4f80a0640 0x7fb4f80a2b00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c1006d0 0x7fb50c1963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb50c102260 0x7fb50c19d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 --2- 172.21.15.3:0/545347870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50c1018c0 0x7fb50c1968f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 >> 172.21.15.3:0/545347870 conn(0x7fb50c0fbc20 msgr2=0x7fb50c0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 shutdown_connections 2023-09-10T01:01:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.456+0000 7fb5135ff700 1 -- 172.21.15.3:0/545347870 wait complete. 2023-09-10T01:01:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:listed 10 entries 2023-09-10T01:01:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:01 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2535744498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:01:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:01 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1884044499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:01:02.475 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T01:01:02.475 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2023-09-10T01:01:02.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:01:02.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 -- 172.21.15.3:0/1010441685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81018c0 msgr2=0x7f68b8101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 --2- 172.21.15.3:0/1010441685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81018c0 0x7f68b8101d20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009920 tx=0x7f68a8009c30 comp rx=0 tx=0).stop 2023-09-10T01:01:02.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 -- 172.21.15.3:0/1010441685 shutdown_connections 2023-09-10T01:01:02.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 --2- 172.21.15.3:0/1010441685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68b8102260 0x7f68b8108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 --2- 172.21.15.3:0/1010441685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81006d0 0x7f68b8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 --2- 172.21.15.3:0/1010441685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81018c0 0x7f68b8101d20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 -- 172.21.15.3:0/1010441685 >> 172.21.15.3:0/1010441685 conn(0x7f68b80fbc60 msgr2=0x7f68b80fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:02.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 -- 172.21.15.3:0/1010441685 shutdown_connections 2023-09-10T01:01:02.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.762+0000 7f68c05ff700 1 -- 172.21.15.3:0/1010441685 wait complete. 2023-09-10T01:01:02.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.763+0000 7f68c05ff700 1 Processor -- start 2023-09-10T01:01:02.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68c05ff700 1 -- start start 2023-09-10T01:01:02.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68c05ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81006d0 0x7f68b8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68c05ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81018c0 0x7f68b8198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68c05ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68b8102260 0x7f68b819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68c05ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81a01b0 con 0x7f68b81018c0 2023-09-10T01:01:02.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68c05ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81a02f0 con 0x7f68b81006d0 2023-09-10T01:01:02.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68c05ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81a0480 con 0x7f68b8102260 2023-09-10T01:01:02.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68bdb9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81018c0 0x7f68b8198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68bdb9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81018c0 0x7f68b8198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49654/0 (socket says 172.21.15.3:49654) 2023-09-10T01:01:02.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68bdb9a700 1 -- 172.21.15.3:0/2476627129 learned_addr learned my addr 172.21.15.3:0/2476627129 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:02.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68be39b700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81006d0 0x7f68b8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.764+0000 7f68beb9c700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68b8102260 0x7f68b819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68bdb9a700 1 -- 172.21.15.3:0/2476627129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68b8102260 msgr2=0x7f68b819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68bdb9a700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68b8102260 0x7f68b819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68bdb9a700 1 -- 172.21.15.3:0/2476627129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81006d0 msgr2=0x7f68b8198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68bdb9a700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81006d0 0x7f68b8198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68bdb9a700 1 -- 172.21.15.3:0/2476627129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b40095f0 con 0x7f68b81018c0 2023-09-10T01:01:02.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68bdb9a700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81018c0 0x7f68b8198b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009230 tx=0x7f68a800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:02.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68af7fe700 1 -- 172.21.15.3:0/2476627129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a801c070 con 0x7f68b81018c0 2023-09-10T01:01:02.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a8009520 con 0x7f68b81018c0 2023-09-10T01:01:02.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68af7fe700 1 -- 172.21.15.3:0/2476627129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68a800bba0 con 0x7f68b81018c0 2023-09-10T01:01:02.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68af7fe700 1 -- 172.21.15.3:0/2476627129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8003a40 con 0x7f68b81018c0 2023-09-10T01:01:02.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.765+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b81a09f0 con 0x7f68b81018c0 2023-09-10T01:01:02.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.766+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b810bee0 con 0x7f68b81018c0 2023-09-10T01:01:02.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.770+0000 7f68af7fe700 1 -- 172.21.15.3:0/2476627129 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f68a8020030 con 0x7f68b81018c0 2023-09-10T01:01:02.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.771+0000 7f68af7fe700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68a40a0640 0x7f68a40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.771+0000 7f68af7fe700 1 -- 172.21.15.3:0/2476627129 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f68a80cd290 con 0x7f68b81018c0 2023-09-10T01:01:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.771+0000 7f68be39b700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68a40a0640 0x7f68a40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.772+0000 7f68be39b700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68a40a0640 0x7f68a40a2b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f68b8101720 tx=0x7f68b4009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:02.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.773+0000 7f68af7fe700 1 -- 172.21.15.3:0/2476627129 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f68a8096b40 con 0x7f68b81018c0 2023-09-10T01:01:02.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.985+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68b810c0a0 con 0x7f68b81018c0 2023-09-10T01:01:02.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.988+0000 7f68af7fe700 1 -- 172.21.15.3:0/2476627129 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24321 (secure 0 0 0) 0x7f68a8005040 con 0x7f68b81018c0 2023-09-10T01:01:02.989 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:02.990 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:56.950998+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:01:02.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68a40a0640 msgr2=0x7f68a40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68a40a0640 0x7f68a40a2b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f68b8101720 tx=0x7f68b4009310 comp rx=0 tx=0).stop 2023-09-10T01:01:02.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81018c0 msgr2=0x7f68b8198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:02.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81018c0 0x7f68b8198b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009230 tx=0x7f68a800bf80 comp rx=0 tx=0).stop 2023-09-10T01:01:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 shutdown_connections 2023-09-10T01:01:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68a40a0640 0x7f68a40a2b00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68b8102260 0x7f68b819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68b81006d0 0x7f68b8198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 --2- 172.21.15.3:0/2476627129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68b81018c0 0x7f68b8198b70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:02.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.990+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 >> 172.21.15.3:0/2476627129 conn(0x7f68b80fbc60 msgr2=0x7f68b80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:02.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.991+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 shutdown_connections 2023-09-10T01:01:02.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:02.991+0000 7f68c05ff700 1 -- 172.21.15.3:0/2476627129 wait complete. 2023-09-10T01:01:03.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:01:03.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:01:03.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 -- 172.21.15.3:0/2180832286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6598102280 msgr2=0x7f6598108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 --2- 172.21.15.3:0/2180832286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6598102280 0x7f6598108b30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f659000b0d0 tx=0x7f659000b3e0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 -- 172.21.15.3:0/2180832286 shutdown_connections 2023-09-10T01:01:03.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 --2- 172.21.15.3:0/2180832286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65981006f0 0x7f6598100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 --2- 172.21.15.3:0/2180832286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6598102280 0x7f6598108b30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 --2- 172.21.15.3:0/2180832286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65981018e0 0x7f6598101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 -- 172.21.15.3:0/2180832286 >> 172.21.15.3:0/2180832286 conn(0x7f65980fbc60 msgr2=0x7f65980fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:03.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 -- 172.21.15.3:0/2180832286 shutdown_connections 2023-09-10T01:01:03.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.220+0000 7f659f7ff700 1 -- 172.21.15.3:0/2180832286 wait complete. 2023-09-10T01:01:03.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 Processor -- start 2023-09-10T01:01:03.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 -- start start 2023-09-10T01:01:03.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65981006f0 0x7f65981986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65981018e0 0x7f6598198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6598102280 0x7f659819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65981a0240 con 0x7f65981006f0 2023-09-10T01:01:03.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65981a0380 con 0x7f6598102280 2023-09-10T01:01:03.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.221+0000 7f659f7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65981a0510 con 0x7f65981018e0 2023-09-10T01:01:03.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65981006f0 0x7f65981986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65981006f0 0x7f65981986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49676/0 (socket says 172.21.15.3:49676) 2023-09-10T01:01:03.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 -- 172.21.15.3:0/3350735119 learned_addr learned my addr 172.21.15.3:0/3350735119 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:03.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659dd9c700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6598102280 0x7f659819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659cd9a700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65981018e0 0x7f6598198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 -- 172.21.15.3:0/3350735119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65981018e0 msgr2=0x7f6598198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65981018e0 0x7f6598198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 -- 172.21.15.3:0/3350735119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6598102280 msgr2=0x7f659819fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6598102280 0x7f659819fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 -- 172.21.15.3:0/3350735119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6590009d00 con 0x7f65981006f0 2023-09-10T01:01:03.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659cd9a700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65981018e0 0x7f6598198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:03.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.222+0000 7f659d59b700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65981006f0 0x7f65981986c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f659400e970 tx=0x7f659400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:03.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.223+0000 7f658e7fc700 1 -- 172.21.15.3:0/3350735119 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594009790 con 0x7f65981006f0 2023-09-10T01:01:03.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.223+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65981a07f0 con 0x7f65981006f0 2023-09-10T01:01:03.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.223+0000 7f658e7fc700 1 -- 172.21.15.3:0/3350735119 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f659400eec0 con 0x7f65981006f0 2023-09-10T01:01:03.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.223+0000 7f658e7fc700 1 -- 172.21.15.3:0/3350735119 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594003cf0 con 0x7f65981006f0 2023-09-10T01:01:03.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.223+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65981a0d50 con 0x7f65981006f0 2023-09-10T01:01:03.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.224+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659810bf10 con 0x7f65981006f0 2023-09-10T01:01:03.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.226+0000 7f658e7fc700 1 -- 172.21.15.3:0/3350735119 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f659400ccd0 con 0x7f65981006f0 2023-09-10T01:01:03.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.227+0000 7f658e7fc700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65840a0640 0x7f65840a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.227+0000 7f658e7fc700 1 -- 172.21.15.3:0/3350735119 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6594014070 con 0x7f65981006f0 2023-09-10T01:01:03.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.227+0000 7f659cd9a700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65840a0640 0x7f65840a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.228+0000 7f659cd9a700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65840a0640 0x7f65840a2b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6588009c90 tx=0x7f6588009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:03.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.229+0000 7f658e7fc700 1 -- 172.21.15.3:0/3350735119 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6594095a70 con 0x7f65981006f0 2023-09-10T01:01:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:02 smithi003 ceph-mon[97654]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 72 B/s, 0 objects/s recovering 2023-09-10T01:01:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:02 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/798425558' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:01:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:02 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/545347870' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:01:03.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:02 smithi136 ceph-mon[133708]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 72 B/s, 0 objects/s recovering 2023-09-10T01:01:03.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:02 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/798425558' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:01:03.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:02 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/545347870' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:01:03.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.440+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65981a0fd0 con 0x7f65981006f0 2023-09-10T01:01:03.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.440+0000 7f658e7fc700 1 -- 172.21.15.3:0/3350735119 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65981a0fd0 con 0x7f65981006f0 2023-09-10T01:01:03.441 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:01:03.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65840a0640 msgr2=0x7f65840a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65840a0640 0x7f65840a2b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6588009c90 tx=0x7f6588009260 comp rx=0 tx=0).stop 2023-09-10T01:01:03.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65981006f0 msgr2=0x7f65981986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65981006f0 0x7f65981986c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f659400e970 tx=0x7f659400ec80 comp rx=0 tx=0).stop 2023-09-10T01:01:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 shutdown_connections 2023-09-10T01:01:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6598102280 0x7f659819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65981006f0 0x7f65981986c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65840a0640 0x7f65840a2b00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 --2- 172.21.15.3:0/3350735119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65981018e0 0x7f6598198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 >> 172.21.15.3:0/3350735119 conn(0x7f65980fbc60 msgr2=0x7f6598107130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 shutdown_connections 2023-09-10T01:01:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.443+0000 7f659f7ff700 1 -- 172.21.15.3:0/3350735119 wait complete. 2023-09-10T01:01:03.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:01:03.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:02 smithi092 ceph-mon[134067]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 72 B/s, 0 objects/s recovering 2023-09-10T01:01:03.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:02 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/798425558' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:01:03.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:02 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/545347870' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:01:03.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 -- 172.21.15.3:0/1362114310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 msgr2=0x7fd1e4100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/1362114310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 0x7fd1e4100ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009920 tx=0x7fd1e0009c30 comp rx=0 tx=0).stop 2023-09-10T01:01:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 -- 172.21.15.3:0/1362114310 shutdown_connections 2023-09-10T01:01:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/1362114310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e4102240 0x7fd1e4108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/1362114310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 0x7fd1e4100ad0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/1362114310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 0x7fd1e4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 -- 172.21.15.3:0/1362114310 >> 172.21.15.3:0/1362114310 conn(0x7fd1e40fbc20 msgr2=0x7fd1e40fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 -- 172.21.15.3:0/1362114310 shutdown_connections 2023-09-10T01:01:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.667+0000 7fd1ebfff700 1 -- 172.21.15.3:0/1362114310 wait complete. 2023-09-10T01:01:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.668+0000 7fd1ebfff700 1 Processor -- start 2023-09-10T01:01:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.668+0000 7fd1ebfff700 1 -- start start 2023-09-10T01:01:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1ebfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 0x7fd1e4196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1ebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 0x7fd1e41969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1ebfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e4102240 0x7fd1e419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1ebfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e419dff0 con 0x7fd1e41018a0 2023-09-10T01:01:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1ebfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e419e130 con 0x7fd1e41006b0 2023-09-10T01:01:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1ebfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e419e2c0 con 0x7fd1e4102240 2023-09-10T01:01:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1e959a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 0x7fd1e41969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1e959a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 0x7fd1e41969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49706/0 (socket says 172.21.15.3:49706) 2023-09-10T01:01:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1e959a700 1 -- 172.21.15.3:0/3544659479 learned_addr learned my addr 172.21.15.3:0/3544659479 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1e9d9b700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 0x7fd1e4196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.669+0000 7fd1ea59c700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e4102240 0x7fd1e419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1e959a700 1 -- 172.21.15.3:0/3544659479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e4102240 msgr2=0x7fd1e419da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1e959a700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e4102240 0x7fd1e419da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1e959a700 1 -- 172.21.15.3:0/3544659479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 msgr2=0x7fd1e4196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1e959a700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 0x7fd1e4196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1e959a700 1 -- 172.21.15.3:0/3544659479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40095f0 con 0x7fd1e41018a0 2023-09-10T01:01:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1e959a700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 0x7fd1e41969b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400ec10 tx=0x7fd1d400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1daffd700 1 -- 172.21.15.3:0/3544659479 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d400cc90 con 0x7fd1e41018a0 2023-09-10T01:01:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e0009520 con 0x7fd1e41018a0 2023-09-10T01:01:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1daffd700 1 -- 172.21.15.3:0/3544659479 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1d40044e0 con 0x7fd1e41018a0 2023-09-10T01:01:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1daffd700 1 -- 172.21.15.3:0/3544659479 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4010430 con 0x7fd1e41018a0 2023-09-10T01:01:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.670+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e419e920 con 0x7fd1e41018a0 2023-09-10T01:01:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.671+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e4107720 con 0x7fd1e41018a0 2023-09-10T01:01:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.673+0000 7fd1daffd700 1 -- 172.21.15.3:0/3544659479 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fd1d4003fb0 con 0x7fd1e41018a0 2023-09-10T01:01:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.675+0000 7fd1daffd700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd1d00a0640 0x7fd1d00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.675+0000 7fd1daffd700 1 -- 172.21.15.3:0/3544659479 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd1d4014070 con 0x7fd1e41018a0 2023-09-10T01:01:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.675+0000 7fd1e9d9b700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd1d00a0640 0x7fd1d00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.676+0000 7fd1e9d9b700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd1d00a0640 0x7fd1d00a2b00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00098f0 tx=0x7fd1e0019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.677+0000 7fd1daffd700 1 -- 172.21.15.3:0/3544659479 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd1d4095e70 con 0x7fd1e41018a0 2023-09-10T01:01:03.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.887+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1e410c160 con 0x7fd1e41018a0 2023-09-10T01:01:03.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.888+0000 7fd1daffd700 1 -- 172.21.15.3:0/3544659479 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd1d4017e90 con 0x7fd1e41018a0 2023-09-10T01:01:03.889 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:01:03.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.890+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd1d00a0640 msgr2=0x7fd1d00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.890+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd1d00a0640 0x7fd1d00a2b00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00098f0 tx=0x7fd1e0019040 comp rx=0 tx=0).stop 2023-09-10T01:01:03.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.890+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 msgr2=0x7fd1e41969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:03.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.890+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 0x7fd1e41969b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400ec10 tx=0x7fd1d400ef20 comp rx=0 tx=0).stop 2023-09-10T01:01:03.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 shutdown_connections 2023-09-10T01:01:03.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd1d00a0640 0x7fd1d00a2b00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e4102240 0x7fd1e419da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1e41006b0 0x7fd1e4196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 --2- 172.21.15.3:0/3544659479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1e41018a0 0x7fd1e41969b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:03.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 >> 172.21.15.3:0/3544659479 conn(0x7fd1e40fbc20 msgr2=0x7fd1e40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:03.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 shutdown_connections 2023-09-10T01:01:03.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:03.891+0000 7fd1ebfff700 1 -- 172.21.15.3:0/3544659479 wait complete. 2023-09-10T01:01:03.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:01:04.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 -- 172.21.15.3:0/3883742002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 msgr2=0x7fccf0101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 --2- 172.21.15.3:0/3883742002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0101d30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcce40099c0 tx=0x7fcce4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 -- 172.21.15.3:0/3883742002 shutdown_connections 2023-09-10T01:01:04.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 --2- 172.21.15.3:0/3883742002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 0x7fccf0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 --2- 172.21.15.3:0/3883742002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccf0102270 0x7fccf0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 --2- 172.21.15.3:0/3883742002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0101d30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 -- 172.21.15.3:0/3883742002 >> 172.21.15.3:0/3883742002 conn(0x7fccf00fbc50 msgr2=0x7fccf00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:04.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 -- 172.21.15.3:0/3883742002 shutdown_connections 2023-09-10T01:01:04.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.119+0000 7fccf55ff700 1 -- 172.21.15.3:0/3883742002 wait complete. 2023-09-10T01:01:04.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 Processor -- start 2023-09-10T01:01:04.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 -- start start 2023-09-10T01:01:04.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 0x7fccf0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccf0102270 0x7fccf019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01a0180 con 0x7fccf01018d0 2023-09-10T01:01:04.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01a02c0 con 0x7fccf0102270 2023-09-10T01:01:04.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.121+0000 7fccf55ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01a0450 con 0x7fccf01006e0 2023-09-10T01:01:04.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fcceeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 0x7fccf0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49720/0 (socket says 172.21.15.3:49720) 2023-09-10T01:01:04.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 -- 172.21.15.3:0/879423659 learned_addr learned my addr 172.21.15.3:0/879423659 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:04.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fcceeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 0x7fccf0198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58528/0 (socket says 172.21.15.3:58528) 2023-09-10T01:01:04.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccef7fe700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccf0102270 0x7fccf019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 -- 172.21.15.3:0/879423659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 msgr2=0x7fccf0198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 0x7fccf0198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 -- 172.21.15.3:0/879423659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccf0102270 msgr2=0x7fccf019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccf0102270 0x7fccf019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 -- 172.21.15.3:0/879423659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40095f0 con 0x7fccf01018d0 2023-09-10T01:01:04.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fcceeffd700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 0x7fccf0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:04.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.122+0000 7fccee7fc700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0198b40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcce400bf40 tx=0x7fcce400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:04.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.123+0000 7fccd7fff700 1 -- 172.21.15.3:0/879423659 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce401d070 con 0x7fccf01018d0 2023-09-10T01:01:04.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.123+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf01a06d0 con 0x7fccf01018d0 2023-09-10T01:01:04.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.123+0000 7fccd7fff700 1 -- 172.21.15.3:0/879423659 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcce4005050 con 0x7fccf01018d0 2023-09-10T01:01:04.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.123+0000 7fccd7fff700 1 -- 172.21.15.3:0/879423659 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4004b40 con 0x7fccf01018d0 2023-09-10T01:01:04.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.123+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf01a0b50 con 0x7fccf01018d0 2023-09-10T01:01:04.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.124+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf010bf50 con 0x7fccf01018d0 2023-09-10T01:01:04.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.126+0000 7fccd7fff700 1 -- 172.21.15.3:0/879423659 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fcce4004ce0 con 0x7fccf01018d0 2023-09-10T01:01:04.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.127+0000 7fccd7fff700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fccd80a0640 0x7fccd80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.127+0000 7fccd7fff700 1 -- 172.21.15.3:0/879423659 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcce40ccd90 con 0x7fccf01018d0 2023-09-10T01:01:04.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.128+0000 7fcceeffd700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fccd80a0640 0x7fccd80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.128+0000 7fcceeffd700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fccd80a0640 0x7fccd80a2b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fccf0101730 tx=0x7fcce0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:04.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.131+0000 7fccd7fff700 1 -- 172.21.15.3:0/879423659 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fcce4096700 con 0x7fccf01018d0 2023-09-10T01:01:04.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2476627129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3350735119' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:01:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3544659479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:01:04.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.342+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fccf01a1010 con 0x7fccf01018d0 2023-09-10T01:01:04.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.343+0000 7fccd7fff700 1 -- 172.21.15.3:0/879423659 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcce4099d20 con 0x7fccf01018d0 2023-09-10T01:01:04.344 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:01:04.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fccd80a0640 msgr2=0x7fccd80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fccd80a0640 0x7fccd80a2b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fccf0101730 tx=0x7fcce0009260 comp rx=0 tx=0).stop 2023-09-10T01:01:04.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 msgr2=0x7fccf0198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0198b40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcce400bf40 tx=0x7fcce400f690 comp rx=0 tx=0).stop 2023-09-10T01:01:04.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 shutdown_connections 2023-09-10T01:01:04.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fccd80a0640 0x7fccd80a2b00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccf01006e0 0x7fccf0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccf0102270 0x7fccf019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 --2- 172.21.15.3:0/879423659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fccf01018d0 0x7fccf0198b40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 >> 172.21.15.3:0/879423659 conn(0x7fccf00fbc50 msgr2=0x7fccf00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:04.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 shutdown_connections 2023-09-10T01:01:04.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.345+0000 7fccf55ff700 1 -- 172.21.15.3:0/879423659 wait complete. 2023-09-10T01:01:04.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:01:04.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:03 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2476627129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:04.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:03 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3350735119' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:01:04.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:03 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3544659479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:01:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:03 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2476627129' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:03 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3350735119' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:01:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:03 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3544659479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:01:04.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 -- 172.21.15.3:0/2979102693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 msgr2=0x7fd0d0108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/2979102693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 0x7fd0d0108b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800b0d0 tx=0x7fd0c800b3e0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 -- 172.21.15.3:0/2979102693 shutdown_connections 2023-09-10T01:01:04.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/2979102693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d01006e0 0x7fd0d0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/2979102693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 0x7fd0d0108b20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/2979102693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01018d0 0x7fd0d0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 -- 172.21.15.3:0/2979102693 >> 172.21.15.3:0/2979102693 conn(0x7fd0d00fbc50 msgr2=0x7fd0d00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:04.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 -- 172.21.15.3:0/2979102693 shutdown_connections 2023-09-10T01:01:04.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.572+0000 7fd0d65ff700 1 -- 172.21.15.3:0/2979102693 wait complete. 2023-09-10T01:01:04.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.573+0000 7fd0d65ff700 1 Processor -- start 2023-09-10T01:01:04.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.573+0000 7fd0d65ff700 1 -- start start 2023-09-10T01:01:04.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0d65ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01006e0 0x7fd0d0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0d65ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d01018d0 0x7fd0d0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0d65ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 0x7fd0d019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0d65ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d01a0290 con 0x7fd0d0102270 2023-09-10T01:01:04.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0d65ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d01a03d0 con 0x7fd0d01006e0 2023-09-10T01:01:04.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0d65ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d01a0560 con 0x7fd0d01018d0 2023-09-10T01:01:04.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01006e0 0x7fd0d0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cf7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d01018d0 0x7fd0d0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01006e0 0x7fd0d0198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59488/0 (socket says 172.21.15.3:59488) 2023-09-10T01:01:04.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 -- 172.21.15.3:0/4095506672 learned_addr learned my addr 172.21.15.3:0/4095506672 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:04.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0d4b9c700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 0x7fd0d019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 -- 172.21.15.3:0/4095506672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d01018d0 msgr2=0x7fd0d0198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d01018d0 0x7fd0d0198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 -- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 msgr2=0x7fd0d019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 0x7fd0d019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cffff700 1 -- 172.21.15.3:0/4095506672 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8009d00 con 0x7fd0d01006e0 2023-09-10T01:01:04.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.574+0000 7fd0cf7fe700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d01018d0 0x7fd0d0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:04.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.575+0000 7fd0cffff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01006e0 0x7fd0d0198710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000e970 tx=0x7fd0c000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:04.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.575+0000 7fd0cd7fa700 1 -- 172.21.15.3:0/4095506672 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0009790 con 0x7fd0d01006e0 2023-09-10T01:01:04.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.575+0000 7fd0cd7fa700 1 -- 172.21.15.3:0/4095506672 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0c000c7e0 con 0x7fd0d01006e0 2023-09-10T01:01:04.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.575+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d01a0840 con 0x7fd0d01006e0 2023-09-10T01:01:04.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.575+0000 7fd0cd7fa700 1 -- 172.21.15.3:0/4095506672 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0003c70 con 0x7fd0d01006e0 2023-09-10T01:01:04.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.575+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d01a0d50 con 0x7fd0d01006e0 2023-09-10T01:01:04.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.576+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d010bf40 con 0x7fd0d01006e0 2023-09-10T01:01:04.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.578+0000 7fd0cd7fa700 1 -- 172.21.15.3:0/4095506672 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fd0c000cc50 con 0x7fd0d01006e0 2023-09-10T01:01:04.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.579+0000 7fd0cd7fa700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd0b80a0640 0x7fd0b80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:04.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.580+0000 7fd0cd7fa700 1 -- 172.21.15.3:0/4095506672 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd0c0014070 con 0x7fd0d01006e0 2023-09-10T01:01:04.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.580+0000 7fd0cf7fe700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd0b80a0640 0x7fd0b80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:04.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.580+0000 7fd0cf7fe700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd0b80a0640 0x7fd0b80a2b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009fd0 tx=0x7fd0c4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:04.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.581+0000 7fd0cd7fa700 1 -- 172.21.15.3:0/4095506672 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd0c0095a10 con 0x7fd0d01006e0 2023-09-10T01:01:04.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.796+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd0d01a0fd0 con 0x7fd0d01006e0 2023-09-10T01:01:04.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.797+0000 7fd0cd7fa700 1 -- 172.21.15.3:0/4095506672 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd0c0003dd0 con 0x7fd0d01006e0 2023-09-10T01:01:04.797 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:01:04.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd0b80a0640 msgr2=0x7fd0b80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd0b80a0640 0x7fd0b80a2b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009fd0 tx=0x7fd0c4009260 comp rx=0 tx=0).stop 2023-09-10T01:01:04.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01006e0 msgr2=0x7fd0d0198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:04.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01006e0 0x7fd0d0198710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000e970 tx=0x7fd0c000ec80 comp rx=0 tx=0).stop 2023-09-10T01:01:04.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 shutdown_connections 2023-09-10T01:01:04.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0d0102270 0x7fd0d019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0d01006e0 0x7fd0d0198710 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd0b80a0640 0x7fd0b80a2b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 --2- 172.21.15.3:0/4095506672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d01018d0 0x7fd0d0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:04.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 >> 172.21.15.3:0/4095506672 conn(0x7fd0d00fbc50 msgr2=0x7fd0d0107140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:04.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 shutdown_connections 2023-09-10T01:01:04.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:04.799+0000 7fd0d65ff700 1 -- 172.21.15.3:0/4095506672 wait complete. 2023-09-10T01:01:04.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-09-10T01:01:05.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 -- 172.21.15.3:0/2246582827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc1018e0 msgr2=0x7f5efc101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 --2- 172.21.15.3:0/2246582827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc1018e0 0x7f5efc101d40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00099c0 tx=0x7f5ef0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 -- 172.21.15.3:0/2246582827 shutdown_connections 2023-09-10T01:01:05.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 --2- 172.21.15.3:0/2246582827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5efc102280 0x7f5efc108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 --2- 172.21.15.3:0/2246582827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5efc1006f0 0x7f5efc100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 --2- 172.21.15.3:0/2246582827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc1018e0 0x7f5efc101d40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 -- 172.21.15.3:0/2246582827 >> 172.21.15.3:0/2246582827 conn(0x7f5efc0fbc60 msgr2=0x7f5efc0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:05.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.026+0000 7f5f019ff700 1 -- 172.21.15.3:0/2246582827 shutdown_connections 2023-09-10T01:01:05.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.027+0000 7f5f019ff700 1 -- 172.21.15.3:0/2246582827 wait complete. 2023-09-10T01:01:05.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.027+0000 7f5f019ff700 1 Processor -- start 2023-09-10T01:01:05.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5f019ff700 1 -- start start 2023-09-10T01:01:05.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5f019ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5efc1006f0 0x7f5efc198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5f019ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5efc1018e0 0x7f5efc198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5f019ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc102280 0x7f5efc19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5f019ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc1a0100 con 0x7f5efc102280 2023-09-10T01:01:05.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5f019ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc1a0240 con 0x7f5efc1018e0 2023-09-10T01:01:05.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5f019ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc1a03d0 con 0x7f5efc1006f0 2023-09-10T01:01:05.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5efb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc102280 0x7f5efc19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5efb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc102280 0x7f5efc19fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48264/0 (socket says 172.21.15.3:48264) 2023-09-10T01:01:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5efb7fe700 1 -- 172.21.15.3:0/1071774215 learned_addr learned my addr 172.21.15.3:0/1071774215 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5efa7fc700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5efc1018e0 0x7f5efc198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.028+0000 7f5efaffd700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5efc1006f0 0x7f5efc198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5efb7fe700 1 -- 172.21.15.3:0/1071774215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5efc1006f0 msgr2=0x7f5efc198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5efb7fe700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5efc1006f0 0x7f5efc198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5efb7fe700 1 -- 172.21.15.3:0/1071774215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5efc1018e0 msgr2=0x7f5efc198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5efb7fe700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5efc1018e0 0x7f5efc198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5efb7fe700 1 -- 172.21.15.3:0/1071774215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00095f0 con 0x7f5efc102280 2023-09-10T01:01:05.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5efa7fc700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5efc1018e0 0x7f5efc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:05.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5efb7fe700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc102280 0x7f5efc19fb30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400ef30 tx=0x7f5ef400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:05.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5f009fd700 1 -- 172.21.15.3:0/1071774215 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4018070 con 0x7f5efc102280 2023-09-10T01:01:05.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc1a0650 con 0x7f5efc102280 2023-09-10T01:01:05.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5f009fd700 1 -- 172.21.15.3:0/1071774215 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ef400f040 con 0x7f5efc102280 2023-09-10T01:01:05.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5f009fd700 1 -- 172.21.15.3:0/1071774215 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400cd00 con 0x7f5efc102280 2023-09-10T01:01:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.029+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc1a0b60 con 0x7f5efc102280 2023-09-10T01:01:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.031+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5efc10bf00 con 0x7f5efc102280 2023-09-10T01:01:05.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.033+0000 7f5f009fd700 1 -- 172.21.15.3:0/1071774215 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f5ef4003a00 con 0x7f5efc102280 2023-09-10T01:01:05.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.034+0000 7f5f009fd700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5ee40a0640 0x7f5ee40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.034+0000 7f5f009fd700 1 -- 172.21.15.3:0/1071774215 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5ef40cba70 con 0x7f5efc102280 2023-09-10T01:01:05.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.034+0000 7f5efaffd700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5ee40a0640 0x7f5ee40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.035+0000 7f5efaffd700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5ee40a0640 0x7f5ee40a2b00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5efc101740 tx=0x7f5eec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:05.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.038+0000 7f5f009fd700 1 -- 172.21.15.3:0/1071774215 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5ef40952c0 con 0x7f5efc102280 2023-09-10T01:01:05.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.264+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5efc10c0c0 con 0x7f5efc102280 2023-09-10T01:01:05.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.265+0000 7f5f009fd700 1 -- 172.21.15.3:0/1071774215 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) v1 ==== 81+0+575 (secure 0 0 0) 0x7f5efc10c0c0 con 0x7f5efc102280 2023-09-10T01:01:05.266 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/1916209618 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:05.266 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/70910038 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:05.266 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/71687535 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:05.267 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/3193301911 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:05.267 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1972995090 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:05.267 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/4153284543 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:05.267 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1689603322 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:05.267 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/1916209618 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:05.268 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/70910038 2023-09-11T00:53:28.332636+0000 2023-09-10T01:01:05.268 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/886409476 2023-09-11T00:53:55.752989+0000 2023-09-10T01:01:05.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5ee40a0640 msgr2=0x7f5ee40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5ee40a0640 0x7f5ee40a2b00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5efc101740 tx=0x7f5eec009260 comp rx=0 tx=0).stop 2023-09-10T01:01:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc102280 msgr2=0x7f5efc19fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc102280 0x7f5efc19fb30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400ef30 tx=0x7f5ef400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 shutdown_connections 2023-09-10T01:01:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5ee40a0640 0x7f5ee40a2b00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5efc1006f0 0x7f5efc198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5efc102280 0x7f5efc19fb30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 --2- 172.21.15.3:0/1071774215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5efc1018e0 0x7f5efc198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 >> 172.21.15.3:0/1071774215 conn(0x7f5efc0fbc60 msgr2=0x7f5efc107160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 shutdown_connections 2023-09-10T01:01:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.267+0000 7f5f019ff700 1 -- 172.21.15.3:0/1071774215 wait complete. 2023-09-10T01:01:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:listed 10 entries 2023-09-10T01:01:05.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:04 smithi003 ceph-mon[97654]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2023-09-10T01:01:05.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:04 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/879423659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:01:05.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:04 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4095506672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:01:05.286 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi003.front.sepia.ceph.com... 2023-09-10T01:01:05.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:01:05.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-09-10T01:01:05.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi003.stderr:ceph-fuse[155757]: fuse finished with error 0 and tester_r 0 2023-09-10T01:01:05.358 INFO:teuthology.orchestra.run:waiting for 300 2023-09-10T01:01:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:04 smithi136 ceph-mon[133708]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2023-09-10T01:01:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:04 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/879423659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:01:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:04 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4095506672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:01:05.425 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:01:05.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:01:05.426 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-09-10T01:01:05.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:04 smithi092 ceph-mon[134067]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2023-09-10T01:01:05.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:04 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/879423659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:01:05.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:04 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4095506672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:01:05.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:01:05.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 -- 172.21.15.3:0/3643357124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b401006d0 msgr2=0x7f5b40100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3643357124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b401006d0 0x7f5b40100af0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5b300099c0 tx=0x7f5b30009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 -- 172.21.15.3:0/3643357124 shutdown_connections 2023-09-10T01:01:05.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3643357124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b40102260 0x7f5b40108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3643357124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b401006d0 0x7f5b40100af0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3643357124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b401018c0 0x7f5b40101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 -- 172.21.15.3:0/3643357124 >> 172.21.15.3:0/3643357124 conn(0x7f5b400fbc60 msgr2=0x7f5b400fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:05.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 -- 172.21.15.3:0/3643357124 shutdown_connections 2023-09-10T01:01:05.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.692+0000 7f5b44fff700 1 -- 172.21.15.3:0/3643357124 wait complete. 2023-09-10T01:01:05.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.693+0000 7f5b44fff700 1 Processor -- start 2023-09-10T01:01:05.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.693+0000 7f5b44fff700 1 -- start start 2023-09-10T01:01:05.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.693+0000 7f5b44fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b401006d0 0x7f5b40198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.693+0000 7f5b44fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b401018c0 0x7f5b40198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.693+0000 7f5b44fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b40102260 0x7f5b4019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.693+0000 7f5b44fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b401a0210 con 0x7f5b401006d0 2023-09-10T01:01:05.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b44fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b401a0350 con 0x7f5b40102260 2023-09-10T01:01:05.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b44fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b401a04e0 con 0x7f5b401018c0 2023-09-10T01:01:05.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b401018c0 0x7f5b40198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3ed9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b40102260 0x7f5b4019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b401018c0 0x7f5b40198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47086/0 (socket says 172.21.15.3:47086) 2023-09-10T01:01:05.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3dd9b700 1 -- 172.21.15.3:0/3235714650 learned_addr learned my addr 172.21.15.3:0/3235714650 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:05.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b401006d0 0x7f5b40198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3ed9d700 1 -- 172.21.15.3:0/3235714650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b401018c0 msgr2=0x7f5b40198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3ed9d700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b401018c0 0x7f5b40198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3ed9d700 1 -- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b401006d0 msgr2=0x7f5b40198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3ed9d700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b401006d0 0x7f5b40198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3ed9d700 1 -- 172.21.15.3:0/3235714650 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300095f0 con 0x7f5b40102260 2023-09-10T01:01:05.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3dd9b700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b401018c0 0x7f5b40198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:05.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3e59c700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b401006d0 0x7f5b40198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:05.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.694+0000 7f5b3ed9d700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b40102260 0x7f5b4019fc40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800efc0 tx=0x7f5b3800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:05.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.695+0000 7f5b2f7fe700 1 -- 172.21.15.3:0/3235714650 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38018070 con 0x7f5b40102260 2023-09-10T01:01:05.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.695+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b401a0760 con 0x7f5b40102260 2023-09-10T01:01:05.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.695+0000 7f5b2f7fe700 1 -- 172.21.15.3:0/3235714650 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b3800f040 con 0x7f5b40102260 2023-09-10T01:01:05.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.695+0000 7f5b2f7fe700 1 -- 172.21.15.3:0/3235714650 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3800cd40 con 0x7f5b40102260 2023-09-10T01:01:05.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.695+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b401a0c70 con 0x7f5b40102260 2023-09-10T01:01:05.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.696+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b40192d20 con 0x7f5b40102260 2023-09-10T01:01:05.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.698+0000 7f5b2f7fe700 1 -- 172.21.15.3:0/3235714650 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f5b38003a00 con 0x7f5b40102260 2023-09-10T01:01:05.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.699+0000 7f5b2f7fe700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5b280a0640 0x7f5b280a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:05.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.700+0000 7f5b2f7fe700 1 -- 172.21.15.3:0/3235714650 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5b380cc360 con 0x7f5b40102260 2023-09-10T01:01:05.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.700+0000 7f5b3e59c700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5b280a0640 0x7f5b280a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:05.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.700+0000 7f5b3e59c700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5b280a0640 0x7f5b280a2b00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009990 tx=0x7f5b30005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:05.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.701+0000 7f5b2f7fe700 1 -- 172.21.15.3:0/3235714650 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5b38095c10 con 0x7f5b40102260 2023-09-10T01:01:05.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.913+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b4010bf30 con 0x7f5b40102260 2023-09-10T01:01:05.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.915+0000 7f5b2f7fe700 1 -- 172.21.15.3:0/3235714650 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24321 (secure 0 0 0) 0x7f5b38099230 con 0x7f5b40102260 2023-09-10T01:01:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:05.917 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:56.950998+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:01:05.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.917+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5b280a0640 msgr2=0x7f5b280a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.917+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5b280a0640 0x7f5b280a2b00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009990 tx=0x7f5b30005fb0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.917+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b40102260 msgr2=0x7f5b4019fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:05.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.917+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b40102260 0x7f5b4019fc40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800efc0 tx=0x7f5b3800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 shutdown_connections 2023-09-10T01:01:05.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b40102260 0x7f5b4019fc40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b401006d0 0x7f5b40198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5b280a0640 0x7f5b280a2b00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 --2- 172.21.15.3:0/3235714650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b401018c0 0x7f5b40198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:05.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 >> 172.21.15.3:0/3235714650 conn(0x7f5b400fbc60 msgr2=0x7f5b40107200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:05.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 shutdown_connections 2023-09-10T01:01:05.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:05.918+0000 7f5b44fff700 1 -- 172.21.15.3:0/3235714650 wait complete. 2023-09-10T01:01:05.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:01:06.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/2685138439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a341006e0 msgr2=0x7f3a34100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/2685138439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a341006e0 0x7f3a34100b00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3a300099c0 tx=0x7f3a30009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/2685138439 shutdown_connections 2023-09-10T01:01:06.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/2685138439 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a34102270 0x7f3a34108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/2685138439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a341006e0 0x7f3a34100b00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/2685138439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a341018d0 0x7f3a34101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/2685138439 >> 172.21.15.3:0/2685138439 conn(0x7f3a340fbc50 msgr2=0x7f3a340fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:06.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/2685138439 shutdown_connections 2023-09-10T01:01:06.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.143+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/2685138439 wait complete. 2023-09-10T01:01:06.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.144+0000 7f3a3b5ff700 1 Processor -- start 2023-09-10T01:01:06.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.144+0000 7f3a3b5ff700 1 -- start start 2023-09-10T01:01:06.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.144+0000 7f3a3b5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a341006e0 0x7f3a340743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.144+0000 7f3a3b5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a341018d0 0x7f3a340729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.144+0000 7f3a3b5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a34102270 0x7f3a34072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a3b5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a340736d0 con 0x7f3a34102270 2023-09-10T01:01:06.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a3b5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a34073840 con 0x7f3a341006e0 2023-09-10T01:01:06.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a3b5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a341a0440 con 0x7f3a341018d0 2023-09-10T01:01:06.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a3939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a341006e0 0x7f3a340743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a39b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a34102270 0x7f3a34072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a3939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a341006e0 0x7f3a340743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56208/0 (socket says 172.21.15.3:56208) 2023-09-10T01:01:06.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a3939b700 1 -- 172.21.15.3:0/10273005 learned_addr learned my addr 172.21.15.3:0/10273005 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:06.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a38b9a700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a341018d0 0x7f3a340729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a39b9c700 1 -- 172.21.15.3:0/10273005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a341018d0 msgr2=0x7f3a340729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a39b9c700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a341018d0 0x7f3a340729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a39b9c700 1 -- 172.21.15.3:0/10273005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a341006e0 msgr2=0x7f3a340743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a39b9c700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a341006e0 0x7f3a340743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a39b9c700 1 -- 172.21.15.3:0/10273005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a300095f0 con 0x7f3a34102270 2023-09-10T01:01:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a38b9a700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a341018d0 0x7f3a340729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.145+0000 7f3a39b9c700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a34102270 0x7f3a34072f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c007f10 tx=0x7f3a2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.146+0000 7f3a2a7fc700 1 -- 172.21.15.3:0/10273005 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c00ddd0 con 0x7f3a34102270 2023-09-10T01:01:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.146+0000 7f3a2a7fc700 1 -- 172.21.15.3:0/10273005 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a2c00f040 con 0x7f3a34102270 2023-09-10T01:01:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.146+0000 7f3a2a7fc700 1 -- 172.21.15.3:0/10273005 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c005a90 con 0x7f3a34102270 2023-09-10T01:01:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.146+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a341a0720 con 0x7f3a34102270 2023-09-10T01:01:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.146+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a341a0be0 con 0x7f3a34102270 2023-09-10T01:01:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.147+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a34107610 con 0x7f3a34102270 2023-09-10T01:01:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.149+0000 7f3a2a7fc700 1 -- 172.21.15.3:0/10273005 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f3a2c004ad0 con 0x7f3a34102270 2023-09-10T01:01:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.150+0000 7f3a2a7fc700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3a200a0640 0x7f3a200a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.151+0000 7f3a2a7fc700 1 -- 172.21.15.3:0/10273005 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3a2c0cb7d0 con 0x7f3a34102270 2023-09-10T01:01:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.151+0000 7f3a3939b700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3a200a0640 0x7f3a200a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.152+0000 7f3a3939b700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3a200a0640 0x7f3a200a2b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3a30006010 tx=0x7f3a30009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.155+0000 7f3a2a7fc700 1 -- 172.21.15.3:0/10273005 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3a2c04ee10 con 0x7f3a34102270 2023-09-10T01:01:06.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:05 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1071774215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:01:06.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:05 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3235714650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:06.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.365+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a3410bf50 con 0x7f3a34102270 2023-09-10T01:01:06.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.366+0000 7f3a2a7fc700 1 -- 172.21.15.3:0/10273005 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24321 (secure 0 0 0) 0x7f3a2c094f90 con 0x7f3a34102270 2023-09-10T01:01:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:06.368 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:56.950998+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:01:06.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3a200a0640 msgr2=0x7f3a200a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3a200a0640 0x7f3a200a2b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3a30006010 tx=0x7f3a30009f10 comp rx=0 tx=0).stop 2023-09-10T01:01:06.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a34102270 msgr2=0x7f3a34072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a34102270 0x7f3a34072f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c007f10 tx=0x7f3a2c00d3b0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 shutdown_connections 2023-09-10T01:01:06.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a34102270 0x7f3a34072f30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a341006e0 0x7f3a340743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3a200a0640 0x7f3a200a2b00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 --2- 172.21.15.3:0/10273005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a341018d0 0x7f3a340729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 >> 172.21.15.3:0/10273005 conn(0x7f3a340fbc50 msgr2=0x7f3a340fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:06.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 shutdown_connections 2023-09-10T01:01:06.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.369+0000 7f3a3b5ff700 1 -- 172.21.15.3:0/10273005 wait complete. 2023-09-10T01:01:06.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-09-10T01:01:06.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:05 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1071774215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:01:06.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:05 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3235714650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:05 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1071774215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:01:06.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:05 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3235714650' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/791897852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8101150 msgr2=0x7fb5b8103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/791897852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8101150 0x7fb5b8103540 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009990 tx=0x7fb5a8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/791897852 shutdown_connections 2023-09-10T01:01:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/791897852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5b81063b0 0x7fb5b8108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/791897852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8101150 0x7fb5b8103540 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/791897852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5b8103a80 0x7fb5b8105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/791897852 >> 172.21.15.3:0/791897852 conn(0x7fb5b80fab80 msgr2=0x7fb5b80fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/791897852 shutdown_connections 2023-09-10T01:01:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.596+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/791897852 wait complete. 2023-09-10T01:01:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.597+0000 7fb5bd5ff700 1 Processor -- start 2023-09-10T01:01:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5bd5ff700 1 -- start start 2023-09-10T01:01:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5bd5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5b8101150 0x7fb5b81963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5bd5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8103a80 0x7fb5b8196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5bd5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5b81063b0 0x7fb5b819d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5bd5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b819df40 con 0x7fb5b8103a80 2023-09-10T01:01:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5bd5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b819e080 con 0x7fb5b8101150 2023-09-10T01:01:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5bd5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b819e210 con 0x7fb5b81063b0 2023-09-10T01:01:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5b67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8103a80 0x7fb5b8196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5b67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8103a80 0x7fb5b8196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48334/0 (socket says 172.21.15.3:48334) 2023-09-10T01:01:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5b67fc700 1 -- 172.21.15.3:0/203590253 learned_addr learned my addr 172.21.15.3:0/203590253 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5b6ffd700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5b8101150 0x7fb5b81963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.598+0000 7fb5b77fe700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5b81063b0 0x7fb5b819d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b67fc700 1 -- 172.21.15.3:0/203590253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5b81063b0 msgr2=0x7fb5b819d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b67fc700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5b81063b0 0x7fb5b819d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b67fc700 1 -- 172.21.15.3:0/203590253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5b8101150 msgr2=0x7fb5b81963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b67fc700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5b8101150 0x7fb5b81963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b67fc700 1 -- 172.21.15.3:0/203590253 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b0009c40 con 0x7fb5b8103a80 2023-09-10T01:01:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b77fe700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5b81063b0 0x7fb5b819d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b6ffd700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5b8101150 0x7fb5b81963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5b67fc700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8103a80 0x7fb5b8196900 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00e9a0 tx=0x7fb5ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb59ffff700 1 -- 172.21.15.3:0/203590253 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac009790 con 0x7fb5b8103a80 2023-09-10T01:01:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a80095f0 con 0x7fb5b8103a80 2023-09-10T01:01:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb59ffff700 1 -- 172.21.15.3:0/203590253 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5ac00cbd0 con 0x7fb5b8103a80 2023-09-10T01:01:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb59ffff700 1 -- 172.21.15.3:0/203590253 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac018680 con 0x7fb5b8103a80 2023-09-10T01:01:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.599+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b819e8c0 con 0x7fb5b8103a80 2023-09-10T01:01:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.600+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b80fc700 con 0x7fb5b8103a80 2023-09-10T01:01:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.602+0000 7fb59ffff700 1 -- 172.21.15.3:0/203590253 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb5ac00cd40 con 0x7fb5b8103a80 2023-09-10T01:01:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.604+0000 7fb59ffff700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb5a00a0640 0x7fb5a00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.604+0000 7fb59ffff700 1 -- 172.21.15.3:0/203590253 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb5ac014070 con 0x7fb5b8103a80 2023-09-10T01:01:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.604+0000 7fb5b6ffd700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb5a00a0640 0x7fb5a00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.605+0000 7fb5b6ffd700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb5a00a0640 0x7fb5a00a2b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800b2e0 tx=0x7fb5a8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.608+0000 7fb59ffff700 1 -- 172.21.15.3:0/203590253 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb5ac051e10 con 0x7fb5b8103a80 2023-09-10T01:01:06.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:06.865+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb5b810bf30 con 0x7fb5b8103a80 2023-09-10T01:01:07.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.008+0000 7fb59ffff700 1 -- 172.21.15.3:0/203590253 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb5ac095990 con 0x7fb5b8103a80 2023-09-10T01:01:07.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb5a00a0640 msgr2=0x7fb5a00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:07.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb5a00a0640 0x7fb5a00a2b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800b2e0 tx=0x7fb5a8005fb0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8103a80 msgr2=0x7fb5b8196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:07.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8103a80 0x7fb5b8196900 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00e9a0 tx=0x7fb5ac00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 shutdown_connections 2023-09-10T01:01:07.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb5a00a0640 0x7fb5a00a2b00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5b81063b0 0x7fb5b819d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5b8101150 0x7fb5b81963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 --2- 172.21.15.3:0/203590253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5b8103a80 0x7fb5b8196900 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 >> 172.21.15.3:0/203590253 conn(0x7fb5b80fab80 msgr2=0x7fb5b8101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:07.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 shutdown_connections 2023-09-10T01:01:07.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.011+0000 7fb5bd5ff700 1 -- 172.21.15.3:0/203590253 wait complete. 2023-09-10T01:01:07.031 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-09-10T01:01:07.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-09-10T01:01:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 -- 172.21.15.3:0/1455327200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 msgr2=0x7f9f14100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 --2- 172.21.15.3:0/1455327200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14100b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f080099c0 tx=0x7f9f08009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 -- 172.21.15.3:0/1455327200 shutdown_connections 2023-09-10T01:01:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 --2- 172.21.15.3:0/1455327200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f14102280 0x7f9f14108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 --2- 172.21.15.3:0/1455327200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14100b10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 --2- 172.21.15.3:0/1455327200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f141018e0 0x7f9f14101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 -- 172.21.15.3:0/1455327200 >> 172.21.15.3:0/1455327200 conn(0x7f9f140fbc60 msgr2=0x7f9f140fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 -- 172.21.15.3:0/1455327200 shutdown_connections 2023-09-10T01:01:07.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.238+0000 7f9f1abff700 1 -- 172.21.15.3:0/1455327200 wait complete. 2023-09-10T01:01:07.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.239+0000 7f9f1abff700 1 Processor -- start 2023-09-10T01:01:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.239+0000 7f9f1abff700 1 -- start start 2023-09-10T01:01:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.239+0000 7f9f1abff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.239+0000 7f9f1abff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f141018e0 0x7f9f14198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1abff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f14102280 0x7f9f1419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1abff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f141a01f0 con 0x7f9f141006f0 2023-09-10T01:01:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1abff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f141a0330 con 0x7f9f141018e0 2023-09-10T01:01:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1abff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f141a04c0 con 0x7f9f14102280 2023-09-10T01:01:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48348/0 (socket says 172.21.15.3:48348) 2023-09-10T01:01:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f13fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f141018e0 0x7f9f14198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 -- 172.21.15.3:0/213868421 learned_addr learned my addr 172.21.15.3:0/213868421 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1919c700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f14102280 0x7f9f1419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 -- 172.21.15.3:0/213868421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f14102280 msgr2=0x7f9f1419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f14102280 0x7f9f1419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 -- 172.21.15.3:0/213868421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f141018e0 msgr2=0x7f9f14198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f141018e0 0x7f9f14198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 -- 172.21.15.3:0/213868421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f080095f0 con 0x7f9f141006f0 2023-09-10T01:01:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1919c700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f14102280 0x7f9f1419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f13fff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f141018e0 0x7f9f14198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.240+0000 7f9f1899b700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14198670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009990 tx=0x7f9f0800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.241+0000 7f9f11ffb700 1 -- 172.21.15.3:0/213868421 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0801d070 con 0x7f9f141006f0 2023-09-10T01:01:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.241+0000 7f9f11ffb700 1 -- 172.21.15.3:0/213868421 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f080054f0 con 0x7f9f141006f0 2023-09-10T01:01:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.241+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f141a0740 con 0x7f9f141006f0 2023-09-10T01:01:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.241+0000 7f9f11ffb700 1 -- 172.21.15.3:0/213868421 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08004bd0 con 0x7f9f141006f0 2023-09-10T01:01:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.241+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f141a0bc0 con 0x7f9f141006f0 2023-09-10T01:01:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.242+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f14192d40 con 0x7f9f141006f0 2023-09-10T01:01:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.244+0000 7f9f11ffb700 1 -- 172.21.15.3:0/213868421 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f9f0800fb80 con 0x7f9f141006f0 2023-09-10T01:01:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.245+0000 7f9f11ffb700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9f040a06c0 0x7f9f040a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.245+0000 7f9f11ffb700 1 -- 172.21.15.3:0/213868421 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9f080cccd0 con 0x7f9f141006f0 2023-09-10T01:01:07.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.245+0000 7f9f13fff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9f040a06c0 0x7f9f040a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:07.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.246+0000 7f9f13fff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9f040a06c0 0x7f9f040a2b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9f00011490 tx=0x7f9f00009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:07.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.249+0000 7f9f11ffb700 1 -- 172.21.15.3:0/213868421 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9f08096520 con 0x7f9f141006f0 2023-09-10T01:01:07.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:07 smithi003 ceph-mon[97654]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 413 B/s wr, 0 op/s; 58 B/s, 0 objects/s recovering 2023-09-10T01:01:07.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:07 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/10273005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:07.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:07 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/203590253' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-09-10T01:01:07.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:07 smithi136 ceph-mon[133708]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 413 B/s wr, 0 op/s; 58 B/s, 0 objects/s recovering 2023-09-10T01:01:07.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:07 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/10273005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:07.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:07 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/203590253' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-09-10T01:01:07.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:07 smithi092 ceph-mon[134067]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 413 B/s wr, 0 op/s; 58 B/s, 0 objects/s recovering 2023-09-10T01:01:07.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:07 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/10273005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:07.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:07 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/203590253' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-09-10T01:01:07.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:07.506+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f9f1402ce20 con 0x7f9f141006f0 2023-09-10T01:01:08.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.014+0000 7f9f11ffb700 1 -- 172.21.15.3:0/213868421 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9f08099b40 con 0x7f9f141006f0 2023-09-10T01:01:08.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9f040a06c0 msgr2=0x7f9f040a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:08.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9f040a06c0 0x7f9f040a2b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9f00011490 tx=0x7f9f00009260 comp rx=0 tx=0).stop 2023-09-10T01:01:08.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 msgr2=0x7f9f14198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:08.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14198670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009990 tx=0x7f9f0800f690 comp rx=0 tx=0).stop 2023-09-10T01:01:08.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 shutdown_connections 2023-09-10T01:01:08.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9f040a06c0 0x7f9f040a2b80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f14102280 0x7f9f1419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9f141006f0 0x7f9f14198670 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 --2- 172.21.15.3:0/213868421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9f141018e0 0x7f9f14198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.017+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 >> 172.21.15.3:0/213868421 conn(0x7f9f140fbc60 msgr2=0x7f9f14107210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.018+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 shutdown_connections 2023-09-10T01:01:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.018+0000 7f9f1abff700 1 -- 172.21.15.3:0/213868421 wait complete. 2023-09-10T01:01:08.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-09-10T01:01:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 -- 172.21.15.3:0/1750843380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 msgr2=0x7f4ca0108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/1750843380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 0x7f4ca0108b20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800b0d0 tx=0x7f4c9800b3e0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 -- 172.21.15.3:0/1750843380 shutdown_connections 2023-09-10T01:01:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/1750843380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 0x7f4ca0108b20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/1750843380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca01006e0 0x7f4ca0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/1750843380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ca01018d0 0x7f4ca0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 -- 172.21.15.3:0/1750843380 >> 172.21.15.3:0/1750843380 conn(0x7f4ca00fbc50 msgr2=0x7f4ca00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 -- 172.21.15.3:0/1750843380 shutdown_connections 2023-09-10T01:01:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.258+0000 7f4ca83ff700 1 -- 172.21.15.3:0/1750843380 wait complete. 2023-09-10T01:01:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 Processor -- start 2023-09-10T01:01:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 -- start start 2023-09-10T01:01:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ca01006e0 0x7f4ca01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca01018d0 0x7f4ca0198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 0x7f4ca019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01a0260 con 0x7f4ca0102270 2023-09-10T01:01:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01a03a0 con 0x7f4ca01018d0 2023-09-10T01:01:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.259+0000 7f4ca83ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01a0530 con 0x7f4ca01006e0 2023-09-10T01:01:08.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca699c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 0x7f4ca019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:08.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca619b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ca01006e0 0x7f4ca01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:08.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca619b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ca01006e0 0x7f4ca01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47164/0 (socket says 172.21.15.3:47164) 2023-09-10T01:01:08.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca01018d0 0x7f4ca0198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca619b700 1 -- 172.21.15.3:0/2955608466 learned_addr learned my addr 172.21.15.3:0/2955608466 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca699c700 1 -- 172.21.15.3:0/2955608466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ca01006e0 msgr2=0x7f4ca01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca699c700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ca01006e0 0x7f4ca01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca699c700 1 -- 172.21.15.3:0/2955608466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca01018d0 msgr2=0x7f4ca0198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:08.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca699c700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca01018d0 0x7f4ca0198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:08.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca699c700 1 -- 172.21.15.3:0/2955608466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98009d00 con 0x7f4ca0102270 2023-09-10T01:01:08.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4ca699c700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 0x7f4ca019fc90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007aa0 tx=0x7f4c98009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:08.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.260+0000 7f4c977fe700 1 -- 172.21.15.3:0/2955608466 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9800e040 con 0x7f4ca0102270 2023-09-10T01:01:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.261+0000 7f4c977fe700 1 -- 172.21.15.3:0/2955608466 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c9801dd50 con 0x7f4ca0102270 2023-09-10T01:01:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.261+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca01a07b0 con 0x7f4ca0102270 2023-09-10T01:01:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.261+0000 7f4c977fe700 1 -- 172.21.15.3:0/2955608466 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9801bd20 con 0x7f4ca0102270 2023-09-10T01:01:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.261+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca01a0c30 con 0x7f4ca0102270 2023-09-10T01:01:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.262+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca010bf50 con 0x7f4ca0102270 2023-09-10T01:01:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.264+0000 7f4c977fe700 1 -- 172.21.15.3:0/2955608466 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f4c9801b450 con 0x7f4ca0102270 2023-09-10T01:01:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.266+0000 7f4c977fe700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c8c0a0640 0x7f4c8c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.266+0000 7f4c977fe700 1 -- 172.21.15.3:0/2955608466 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4c980cdb30 con 0x7f4ca0102270 2023-09-10T01:01:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.266+0000 7f4ca619b700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c8c0a0640 0x7f4c8c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.267+0000 7f4ca619b700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c8c0a0640 0x7f4c8c0a2b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00bd70 tx=0x7f4c9c00b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:08.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.270+0000 7f4c977fe700 1 -- 172.21.15.3:0/2955608466 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f4c98097380 con 0x7f4ca0102270 2023-09-10T01:01:08.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:08 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/203590253' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-09-10T01:01:08.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:08 smithi003 ceph-mon[97654]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:01:08.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:08 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/213868421' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-09-10T01:01:08.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:08 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/203590253' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-09-10T01:01:08.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:08 smithi136 ceph-mon[133708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:01:08.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:08 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/213868421' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-09-10T01:01:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:08 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/203590253' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-09-10T01:01:08.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:08 smithi092 ceph-mon[134067]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:01:08.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:08 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/213868421' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-09-10T01:01:08.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:08.529+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f4ca01a0e50 con 0x7f4ca0102270 2023-09-10T01:01:09.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.022+0000 7f4c977fe700 1 -- 172.21.15.3:0/2955608466 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f4c9809a9a0 con 0x7f4ca0102270 2023-09-10T01:01:09.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c8c0a0640 msgr2=0x7f4c8c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:09.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c8c0a0640 0x7f4c8c0a2b00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00bd70 tx=0x7f4c9c00b4a0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 msgr2=0x7f4ca019fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:09.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 0x7f4ca019fc90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007aa0 tx=0x7f4c98009440 comp rx=0 tx=0).stop 2023-09-10T01:01:09.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 shutdown_connections 2023-09-10T01:01:09.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4c8c0a0640 0x7f4c8c0a2b00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ca01006e0 0x7f4ca01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ca0102270 0x7f4ca019fc90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 --2- 172.21.15.3:0/2955608466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca01018d0 0x7f4ca0198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 >> 172.21.15.3:0/2955608466 conn(0x7f4ca00fbc50 msgr2=0x7f4ca0107160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:09.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 shutdown_connections 2023-09-10T01:01:09.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.025+0000 7f4ca83ff700 1 -- 172.21.15.3:0/2955608466 wait complete. 2023-09-10T01:01:09.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-09-10T01:01:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1077215238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb4102280 msgr2=0x7fcdb4108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1077215238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb4102280 0x7fcdb4108b30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00b0d0 tx=0x7fcdac00b3e0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1077215238 shutdown_connections 2023-09-10T01:01:09.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1077215238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdb41006f0 0x7fcdb4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1077215238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb4102280 0x7fcdb4108b30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1077215238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdb41018e0 0x7fcdb4101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1077215238 >> 172.21.15.3:0/1077215238 conn(0x7fcdb40fbc60 msgr2=0x7fcdb40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:09.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1077215238 shutdown_connections 2023-09-10T01:01:09.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.262+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1077215238 wait complete. 2023-09-10T01:01:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.263+0000 7fcdb93ff700 1 Processor -- start 2023-09-10T01:01:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.263+0000 7fcdb93ff700 1 -- start start 2023-09-10T01:01:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.263+0000 7fcdb93ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdb41006f0 0x7fcdb41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.263+0000 7fcdb93ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb41018e0 0x7fcdb4198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.263+0000 7fcdb93ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdb4102280 0x7fcdb419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdb93ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb41a0230 con 0x7fcdb41018e0 2023-09-10T01:01:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdb93ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb41a0370 con 0x7fcdb41006f0 2023-09-10T01:01:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdb93ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb41a0500 con 0x7fcdb4102280 2023-09-10T01:01:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb41018e0 0x7fcdb4198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb41018e0 0x7fcdb4198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48396/0 (socket says 172.21.15.3:48396) 2023-09-10T01:01:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 -- 172.21.15.3:0/1823751905 learned_addr learned my addr 172.21.15.3:0/1823751905 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdb2ffd700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdb41006f0 0x7fcdb41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdb37fe700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdb4102280 0x7fcdb419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 -- 172.21.15.3:0/1823751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdb4102280 msgr2=0x7fcdb419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdb4102280 0x7fcdb419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 -- 172.21.15.3:0/1823751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdb41006f0 msgr2=0x7fcdb41986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdb41006f0 0x7fcdb41986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 -- 172.21.15.3:0/1823751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac009d00 con 0x7fcdb41018e0 2023-09-10T01:01:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.264+0000 7fcdabfff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb41018e0 0x7fcdb4198bf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00e9a0 tx=0x7fcd9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.265+0000 7fcdb0ff9700 1 -- 172.21.15.3:0/1823751905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c009790 con 0x7fcdb41018e0 2023-09-10T01:01:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.265+0000 7fcdb0ff9700 1 -- 172.21.15.3:0/1823751905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd9c00cbd0 con 0x7fcdb41018e0 2023-09-10T01:01:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.265+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb41a07e0 con 0x7fcdb41018e0 2023-09-10T01:01:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.265+0000 7fcdb0ff9700 1 -- 172.21.15.3:0/1823751905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c018680 con 0x7fcdb41018e0 2023-09-10T01:01:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.265+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb41a0d40 con 0x7fcdb41018e0 2023-09-10T01:01:09.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.266+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb410bf50 con 0x7fcdb41018e0 2023-09-10T01:01:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.268+0000 7fcdb0ff9700 1 -- 172.21.15.3:0/1823751905 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fcd9c00cd40 con 0x7fcdb41018e0 2023-09-10T01:01:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.269+0000 7fcdb0ff9700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcda00a06c0 0x7fcda00a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.269+0000 7fcdb0ff9700 1 -- 172.21.15.3:0/1823751905 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcd9c014070 con 0x7fcdb41018e0 2023-09-10T01:01:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.269+0000 7fcdb2ffd700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcda00a06c0 0x7fcda00a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.270+0000 7fcdb2ffd700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcda00a06c0 0x7fcda00a2b80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4101740 tx=0x7fcda4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:09.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.273+0000 7fcdb0ff9700 1 -- 172.21.15.3:0/1823751905 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fcd9c051e10 con 0x7fcdb41018e0 2023-09-10T01:01:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 469 B/s wr, 0 op/s; 53 B/s, 0 objects/s recovering 2023-09-10T01:01:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-09-10T01:01:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/213868421' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-09-10T01:01:09.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-09-10T01:01:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:01:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-09-10T01:01:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-09-10T01:01:09.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:09 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2955608466' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-09-10T01:01:09.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 469 B/s wr, 0 op/s; 53 B/s, 0 objects/s recovering 2023-09-10T01:01:09.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-09-10T01:01:09.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/213868421' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-09-10T01:01:09.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-09-10T01:01:09.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:01:09.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-09-10T01:01:09.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-09-10T01:01:09.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:09 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2955608466' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-09-10T01:01:09.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 469 B/s wr, 0 op/s; 53 B/s, 0 objects/s recovering 2023-09-10T01:01:09.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-09-10T01:01:09.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/213868421' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-09-10T01:01:09.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-09-10T01:01:09.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-09-10T01:01:09.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-09-10T01:01:09.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-09-10T01:01:09.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:09 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2955608466' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-09-10T01:01:09.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.536+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fcdb410c110 con 0x7fcdb41018e0 2023-09-10T01:01:09.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.941+0000 7fcdb0ff9700 1 -- 172.21.15.3:0/1823751905 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcd9c095930 con 0x7fcdb41018e0 2023-09-10T01:01:09.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcda00a06c0 msgr2=0x7fcda00a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:09.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcda00a06c0 0x7fcda00a2b80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4101740 tx=0x7fcda4009260 comp rx=0 tx=0).stop 2023-09-10T01:01:09.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb41018e0 msgr2=0x7fcdb4198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:09.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb41018e0 0x7fcdb4198bf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00e9a0 tx=0x7fcd9c00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 shutdown_connections 2023-09-10T01:01:09.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcda00a06c0 0x7fcda00a2b80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdb4102280 0x7fcdb419fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdb41006f0 0x7fcdb41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 --2- 172.21.15.3:0/1823751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdb41018e0 0x7fcdb4198bf0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:09.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 >> 172.21.15.3:0/1823751905 conn(0x7fcdb40fbc60 msgr2=0x7fcdb4107170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:09.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 shutdown_connections 2023-09-10T01:01:09.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:09.944+0000 7fcdb93ff700 1 -- 172.21.15.3:0/1823751905 wait complete. 2023-09-10T01:01:09.966 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2023-09-10T01:01:09.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-09-10T01:01:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 -- 172.21.15.3:0/476120733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec102280 msgr2=0x7ff5ec108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/476120733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec102280 0x7ff5ec108b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5e400b0d0 tx=0x7ff5e400b3e0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 -- 172.21.15.3:0/476120733 shutdown_connections 2023-09-10T01:01:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/476120733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5ec1006f0 0x7ff5ec100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/476120733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec102280 0x7ff5ec108b30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/476120733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 0x7ff5ec101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 -- 172.21.15.3:0/476120733 >> 172.21.15.3:0/476120733 conn(0x7ff5ec0fbc60 msgr2=0x7ff5ec0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:10.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 -- 172.21.15.3:0/476120733 shutdown_connections 2023-09-10T01:01:10.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.179+0000 7ff5f09ff700 1 -- 172.21.15.3:0/476120733 wait complete. 2023-09-10T01:01:10.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.180+0000 7ff5f09ff700 1 Processor -- start 2023-09-10T01:01:10.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.180+0000 7ff5f09ff700 1 -- start start 2023-09-10T01:01:10.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.180+0000 7ff5f09ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec1006f0 0x7ff5ec196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.180+0000 7ff5f09ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 0x7ff5ec1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5f09ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5ec102280 0x7ff5ec19da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5f09ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19dfe0 con 0x7ff5ec1018e0 2023-09-10T01:01:10.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5f09ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19e120 con 0x7ff5ec1006f0 2023-09-10T01:01:10.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5f09ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec19e2b0 con 0x7ff5ec102280 2023-09-10T01:01:10.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec1006f0 0x7ff5ec196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 0x7ff5ec1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec1006f0 0x7ff5ec196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56294/0 (socket says 172.21.15.3:56294) 2023-09-10T01:01:10.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 -- 172.21.15.3:0/2280096122 learned_addr learned my addr 172.21.15.3:0/2280096122 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:10.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 0x7ff5ec1969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48406/0 (socket says 172.21.15.3:48406) 2023-09-10T01:01:10.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ead9d700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5ec102280 0x7ff5ec19da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 -- 172.21.15.3:0/2280096122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5ec102280 msgr2=0x7ff5ec19da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5ec102280 0x7ff5ec19da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 -- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 msgr2=0x7ff5ec1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 0x7ff5ec1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5ea59c700 1 -- 172.21.15.3:0/2280096122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d4009520 con 0x7ff5ec1006f0 2023-09-10T01:01:10.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.181+0000 7ff5e3fff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 0x7ff5ec1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:10.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.182+0000 7ff5ea59c700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec1006f0 0x7ff5ec196460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00e970 tx=0x7ff5dc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:10.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.182+0000 7ff5e37fe700 1 -- 172.21.15.3:0/2280096122 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc009790 con 0x7ff5ec1006f0 2023-09-10T01:01:10.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.182+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e4009d00 con 0x7ff5ec1006f0 2023-09-10T01:01:10.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.182+0000 7ff5e37fe700 1 -- 172.21.15.3:0/2280096122 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5dc00c7e0 con 0x7ff5ec1006f0 2023-09-10T01:01:10.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.182+0000 7ff5e37fe700 1 -- 172.21.15.3:0/2280096122 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc003c70 con 0x7ff5ec1006f0 2023-09-10T01:01:10.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.182+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ec19e960 con 0x7ff5ec1006f0 2023-09-10T01:01:10.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.183+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec107b90 con 0x7ff5ec1006f0 2023-09-10T01:01:10.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.185+0000 7ff5e37fe700 1 -- 172.21.15.3:0/2280096122 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7ff5dc00cc50 con 0x7ff5ec1006f0 2023-09-10T01:01:10.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.186+0000 7ff5e37fe700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff5d80a0640 0x7ff5d80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.186+0000 7ff5e37fe700 1 -- 172.21.15.3:0/2280096122 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff5dc014070 con 0x7ff5ec1006f0 2023-09-10T01:01:10.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.187+0000 7ff5e3fff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff5d80a0640 0x7ff5d80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.187+0000 7ff5e3fff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff5d80a0640 0x7ff5d80a2b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009fd0 tx=0x7ff5d4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:10.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.189+0000 7ff5e37fe700 1 -- 172.21.15.3:0/2280096122 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff5dc095b00 con 0x7ff5ec1006f0 2023-09-10T01:01:10.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2955608466' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-09-10T01:01:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] up:active 2023-09-10T01:01:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-09-10T01:01:10.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-09-10T01:01:10.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: Cluster is now healthy 2023-09-10T01:01:10.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-09-10T01:01:10.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-09-10T01:01:10.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-09-10T01:01:10.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1823751905' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-09-10T01:01:10.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1823751905' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-09-10T01:01:10.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] up:active 2023-09-10T01:01:10.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:01:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2955608466' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-09-10T01:01:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] up:active 2023-09-10T01:01:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-09-10T01:01:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-09-10T01:01:10.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: Cluster is now healthy 2023-09-10T01:01:10.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-09-10T01:01:10.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-09-10T01:01:10.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-09-10T01:01:10.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1823751905' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-09-10T01:01:10.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1823751905' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-09-10T01:01:10.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] up:active 2023-09-10T01:01:10.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:10 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:01:10.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.433+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5ec02ce90 con 0x7ff5d80a0640 2023-09-10T01:01:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2955608466' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-09-10T01:01:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] up:active 2023-09-10T01:01:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-09-10T01:01:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-09-10T01:01:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: Cluster is now healthy 2023-09-10T01:01:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-09-10T01:01:10.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-09-10T01:01:10.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-09-10T01:01:10.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1823751905' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-09-10T01:01:10.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1823751905' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-09-10T01:01:10.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] up:active 2023-09-10T01:01:10.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:10 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:01:10.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.478+0000 7ff5e37fe700 1 -- 172.21.15.3:0/2280096122 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff5ec02ce90 con 0x7ff5d80a0640 2023-09-10T01:01:10.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff5d80a0640 msgr2=0x7ff5d80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff5d80a0640 0x7ff5d80a2b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009fd0 tx=0x7ff5d4009260 comp rx=0 tx=0).stop 2023-09-10T01:01:10.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec1006f0 msgr2=0x7ff5ec196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec1006f0 0x7ff5ec196460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00e970 tx=0x7ff5dc00ec80 comp rx=0 tx=0).stop 2023-09-10T01:01:10.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 shutdown_connections 2023-09-10T01:01:10.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff5d80a0640 0x7ff5d80a2b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5ec102280 0x7ff5ec19da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5ec1006f0 0x7ff5ec196460 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 --2- 172.21.15.3:0/2280096122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5ec1018e0 0x7ff5ec1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.480+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 >> 172.21.15.3:0/2280096122 conn(0x7ff5ec0fbc60 msgr2=0x7ff5ec104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:10.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.481+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 shutdown_connections 2023-09-10T01:01:10.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.481+0000 7ff5f09ff700 1 -- 172.21.15.3:0/2280096122 wait complete. 2023-09-10T01:01:10.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-09-10T01:01:10.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 -- 172.21.15.3:0/1931535649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 msgr2=0x7f50b4108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1931535649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 0x7f50b4108af0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00b0d0 tx=0x7f50ac00b3e0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 -- 172.21.15.3:0/1931535649 shutdown_connections 2023-09-10T01:01:10.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1931535649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 0x7f50b4108af0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1931535649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b41006b0 0x7f50b4100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1931535649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50b41018a0 0x7f50b4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 -- 172.21.15.3:0/1931535649 >> 172.21.15.3:0/1931535649 conn(0x7f50b40fbc20 msgr2=0x7f50b40fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:10.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 -- 172.21.15.3:0/1931535649 shutdown_connections 2023-09-10T01:01:10.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.707+0000 7f50b91ff700 1 -- 172.21.15.3:0/1931535649 wait complete. 2023-09-10T01:01:10.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 Processor -- start 2023-09-10T01:01:10.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 -- start start 2023-09-10T01:01:10.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50b41006b0 0x7f50b40743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b41018a0 0x7f50b40729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 0x7f50b4072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40736d0 con 0x7f50b41018a0 2023-09-10T01:01:10.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4073840 con 0x7f50b4102240 2023-09-10T01:01:10.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.708+0000 7f50b91ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b41a04e0 con 0x7f50b41006b0 2023-09-10T01:01:10.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50b41006b0 0x7f50b40743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b41018a0 0x7f50b40729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50b41006b0 0x7f50b40743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47268/0 (socket says 172.21.15.3:47268) 2023-09-10T01:01:10.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b2d9d700 1 -- 172.21.15.3:0/1847277629 learned_addr learned my addr 172.21.15.3:0/1847277629 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:10.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b359e700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 0x7f50b4072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b359e700 1 -- 172.21.15.3:0/1847277629 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50b41006b0 msgr2=0x7f50b40743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b359e700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50b41006b0 0x7f50b40743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b359e700 1 -- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b41018a0 msgr2=0x7f50b40729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b359e700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b41018a0 0x7f50b40729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b359e700 1 -- 172.21.15.3:0/1847277629 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009d00 con 0x7f50b4102240 2023-09-10T01:01:10.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.709+0000 7f50b359e700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 0x7f50b4072f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00b770 tx=0x7f50ac008ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:10.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.710+0000 7f509bfff700 1 -- 172.21.15.3:0/1847277629 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac00e040 con 0x7f50b4102240 2023-09-10T01:01:10.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.710+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b41a0760 con 0x7f50b4102240 2023-09-10T01:01:10.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.710+0000 7f509bfff700 1 -- 172.21.15.3:0/1847277629 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50ac01d930 con 0x7f50b4102240 2023-09-10T01:01:10.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.710+0000 7f509bfff700 1 -- 172.21.15.3:0/1847277629 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac01b8f0 con 0x7f50b4102240 2023-09-10T01:01:10.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.710+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b41a0b90 con 0x7f50b4102240 2023-09-10T01:01:10.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.711+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b4107bf0 con 0x7f50b4102240 2023-09-10T01:01:10.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.713+0000 7f509bfff700 1 -- 172.21.15.3:0/1847277629 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f50ac01daa0 con 0x7f50b4102240 2023-09-10T01:01:10.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.714+0000 7f509bfff700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f509c0a0640 0x7f509c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:10.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.714+0000 7f509bfff700 1 -- 172.21.15.3:0/1847277629 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f50ac0cd780 con 0x7f50b4102240 2023-09-10T01:01:10.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.715+0000 7f50b2d9d700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f509c0a0640 0x7f509c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:10.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.715+0000 7f50b2d9d700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f509c0a0640 0x7f509c0a2b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f50a4005950 tx=0x7f50a4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:10.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.716+0000 7f509bfff700 1 -- 172.21.15.3:0/1847277629 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f50ac096fd0 con 0x7f50b4102240 2023-09-10T01:01:10.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:01:10.440+0000 7f61633f9700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:01:10.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:01:10.440+0000 7f61633f9700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:01:10.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:01:10.440+0000 7f61633f9700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:01:10.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:01:10.440+0000 7f61633f9700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:01:10.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:01:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:01:10.440+0000 7f61633f9700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:01:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.962+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f50b402cf00 con 0x7f509c0a0640 2023-09-10T01:01:10.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.967+0000 7f509bfff700 1 -- 172.21.15.3:0/1847277629 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f50b402cf00 con 0x7f509c0a0640 2023-09-10T01:01:10.968 INFO:teuthology.orchestra.run.smithi003.stdout:/volumes/_nogroup/sv_0/ed9eeaa3-7a42-4070-ab85-a1bb387f8146 2023-09-10T01:01:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f509c0a0640 msgr2=0x7f509c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f509c0a0640 0x7f509c0a2b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f50a4005950 tx=0x7f50a4009260 comp rx=0 tx=0).stop 2023-09-10T01:01:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 msgr2=0x7f50b4072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 0x7f50b4072f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00b770 tx=0x7f50ac008ec0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 shutdown_connections 2023-09-10T01:01:10.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f509c0a0640 0x7f509c0a2b00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50b41006b0 0x7f50b40743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50b4102240 0x7f50b4072f30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 --2- 172.21.15.3:0/1847277629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b41018a0 0x7f50b40729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 >> 172.21.15.3:0/1847277629 conn(0x7f50b40fbc20 msgr2=0x7f50b4104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:10.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 shutdown_connections 2023-09-10T01:01:10.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:10.969+0000 7f50b91ff700 1 -- 172.21.15.3:0/1847277629 wait complete. 2023-09-10T01:01:10.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-09-10T01:01:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 -- 172.21.15.3:0/1528018667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 msgr2=0x7fbbb8101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/1528018667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 0x7fbbb8101d10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbba00099c0 tx=0x7fbba0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 -- 172.21.15.3:0/1528018667 shutdown_connections 2023-09-10T01:01:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/1528018667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb8102250 0x7fbbb8108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/1528018667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbb81006c0 0x7fbbb8100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/1528018667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 0x7fbbb8101d10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 -- 172.21.15.3:0/1528018667 >> 172.21.15.3:0/1528018667 conn(0x7fbbb80fbc10 msgr2=0x7fbbb80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 -- 172.21.15.3:0/1528018667 shutdown_connections 2023-09-10T01:01:11.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.198+0000 7fbbbcdff700 1 -- 172.21.15.3:0/1528018667 wait complete. 2023-09-10T01:01:11.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.199+0000 7fbbbcdff700 1 Processor -- start 2023-09-10T01:01:11.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.199+0000 7fbbbcdff700 1 -- start start 2023-09-10T01:01:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbbcdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb81006c0 0x7fbbb8196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbbcdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 0x7fbbb81969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbbcdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbb8102250 0x7fbbb819da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbbcdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb819dff0 con 0x7fbbb8102250 2023-09-10T01:01:11.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbbcdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb819e130 con 0x7fbbb81018b0 2023-09-10T01:01:11.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbbcdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb819e2c0 con 0x7fbbb81006c0 2023-09-10T01:01:11.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbb6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbb8102250 0x7fbbb819da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbb659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb81006c0 0x7fbbb8196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbb659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb81006c0 0x7fbbb8196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47282/0 (socket says 172.21.15.3:47282) 2023-09-10T01:01:11.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbb659c700 1 -- 172.21.15.3:0/4100239777 learned_addr learned my addr 172.21.15.3:0/4100239777 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:11.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.200+0000 7fbbaffff700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 0x7fbbb81969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbb6d9d700 1 -- 172.21.15.3:0/4100239777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb81006c0 msgr2=0x7fbbb8196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbb6d9d700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb81006c0 0x7fbbb8196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbb6d9d700 1 -- 172.21.15.3:0/4100239777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 msgr2=0x7fbbb81969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbb6d9d700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 0x7fbbb81969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbb6d9d700 1 -- 172.21.15.3:0/4100239777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba00095f0 con 0x7fbbb8102250 2023-09-10T01:01:11.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbb6d9d700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbb8102250 0x7fbbb819da20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000efc0 tx=0x7fbbb000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:11.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbaf7fe700 1 -- 172.21.15.3:0/4100239777 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0018070 con 0x7fbbb8102250 2023-09-10T01:01:11.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb819e540 con 0x7fbbb8102250 2023-09-10T01:01:11.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbaf7fe700 1 -- 172.21.15.3:0/4100239777 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbb000f040 con 0x7fbbb8102250 2023-09-10T01:01:11.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbaf7fe700 1 -- 172.21.15.3:0/4100239777 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb000cd40 con 0x7fbbb8102250 2023-09-10T01:01:11.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.201+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb819eaa0 con 0x7fbbb8102250 2023-09-10T01:01:11.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.202+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb810bf20 con 0x7fbbb8102250 2023-09-10T01:01:11.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.204+0000 7fbbaf7fe700 1 -- 172.21.15.3:0/4100239777 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fbbb0003a00 con 0x7fbbb8102250 2023-09-10T01:01:11.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.206+0000 7fbbaf7fe700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbba40a06c0 0x7fbba40a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.206+0000 7fbbaf7fe700 1 -- 172.21.15.3:0/4100239777 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbbb00cb9a0 con 0x7fbbb8102250 2023-09-10T01:01:11.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.206+0000 7fbbb659c700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbba40a06c0 0x7fbba40a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.207+0000 7fbbb659c700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbba40a06c0 0x7fbba40a2b80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbba8005950 tx=0x7fbba8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:11.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.210+0000 7fbbaf7fe700 1 -- 172.21.15.3:0/4100239777 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbbb00951f0 con 0x7fbbb8102250 2023-09-10T01:01:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:11 smithi003 ceph-mon[97654]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s; 48 B/s, 0 objects/s recovering 2023-09-10T01:01:11.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:11 smithi136 ceph-mon[133708]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s; 48 B/s, 0 objects/s recovering 2023-09-10T01:01:11.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.456+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbb8060a40 con 0x7fbba40a06c0 2023-09-10T01:01:11.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:11 smithi092 ceph-mon[134067]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s; 48 B/s, 0 objects/s recovering 2023-09-10T01:01:11.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.483+0000 7fbbaf7fe700 1 -- 172.21.15.3:0/4100239777 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbbb8060a40 con 0x7fbba40a06c0 2023-09-10T01:01:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbba40a06c0 msgr2=0x7fbba40a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbba40a06c0 0x7fbba40a2b80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbba8005950 tx=0x7fbba8009260 comp rx=0 tx=0).stop 2023-09-10T01:01:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbb8102250 msgr2=0x7fbbb819da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbb8102250 0x7fbbb819da20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000efc0 tx=0x7fbbb000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 shutdown_connections 2023-09-10T01:01:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbba40a06c0 0x7fbba40a2b80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb81006c0 0x7fbbb8196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbb8102250 0x7fbbb819da20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 --2- 172.21.15.3:0/4100239777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb81018b0 0x7fbbb81969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 >> 172.21.15.3:0/4100239777 conn(0x7fbbb80fbc10 msgr2=0x7fbbb80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 shutdown_connections 2023-09-10T01:01:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.485+0000 7fbbbcdff700 1 -- 172.21.15.3:0/4100239777 wait complete. 2023-09-10T01:01:11.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-09-10T01:01:11.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 -- 172.21.15.3:0/2218717349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c24102270 msgr2=0x7f0c24108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 --2- 172.21.15.3:0/2218717349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c24102270 0x7f0c24108b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009fc0 tx=0x7f0c1c00b330 comp rx=0 tx=0).stop 2023-09-10T01:01:11.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 -- 172.21.15.3:0/2218717349 shutdown_connections 2023-09-10T01:01:11.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 --2- 172.21.15.3:0/2218717349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c24102270 0x7f0c24108b20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 --2- 172.21.15.3:0/2218717349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c241006e0 0x7f0c24100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 --2- 172.21.15.3:0/2218717349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241018d0 0x7f0c24101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 -- 172.21.15.3:0/2218717349 >> 172.21.15.3:0/2218717349 conn(0x7f0c240fbc50 msgr2=0x7f0c240fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:11.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 -- 172.21.15.3:0/2218717349 shutdown_connections 2023-09-10T01:01:11.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.711+0000 7f0c28dff700 1 -- 172.21.15.3:0/2218717349 wait complete. 2023-09-10T01:01:11.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 Processor -- start 2023-09-10T01:01:11.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 -- start start 2023-09-10T01:01:11.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c241006e0 0x7f0c24196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c241018d0 0x7f0c241969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c24102270 0x7f0c2419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2419dfe0 con 0x7f0c241018d0 2023-09-10T01:01:11.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2419e120 con 0x7f0c241006e0 2023-09-10T01:01:11.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.712+0000 7f0c28dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2419e2b0 con 0x7f0c24102270 2023-09-10T01:01:11.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c241018d0 0x7f0c241969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c241018d0 0x7f0c241969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48450/0 (socket says 172.21.15.3:48450) 2023-09-10T01:01:11.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 -- 172.21.15.3:0/3806158092 learned_addr learned my addr 172.21.15.3:0/3806158092 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:11.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c2259c700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c241006e0 0x7f0c24196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c22d9d700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c24102270 0x7f0c2419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 -- 172.21.15.3:0/3806158092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c24102270 msgr2=0x7f0c2419da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c24102270 0x7f0c2419da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 -- 172.21.15.3:0/3806158092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c241006e0 msgr2=0x7f0c24196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c241006e0 0x7f0c24196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 -- 172.21.15.3:0/3806158092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c14009520 con 0x7f0c241018d0 2023-09-10T01:01:11.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1bfff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c241018d0 0x7f0c241969a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00e9c0 tx=0x7f0c0c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:11.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1b7fe700 1 -- 172.21.15.3:0/3806158092 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c004a30 con 0x7f0c241018d0 2023-09-10T01:01:11.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1b7fe700 1 -- 172.21.15.3:0/3806158092 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c0c00cc80 con 0x7f0c241018d0 2023-09-10T01:01:11.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.713+0000 7f0c1b7fe700 1 -- 172.21.15.3:0/3806158092 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0054e0 con 0x7f0c241018d0 2023-09-10T01:01:11.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.714+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c009c40 con 0x7f0c241018d0 2023-09-10T01:01:11.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.714+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2419e910 con 0x7f0c241018d0 2023-09-10T01:01:11.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.715+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2410bf00 con 0x7f0c241018d0 2023-09-10T01:01:11.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.717+0000 7f0c1b7fe700 1 -- 172.21.15.3:0/3806158092 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f0c0c004b90 con 0x7f0c241018d0 2023-09-10T01:01:11.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.719+0000 7f0c1b7fe700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0c100a06c0 0x7f0c100a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:11.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.719+0000 7f0c1b7fe700 1 -- 172.21.15.3:0/3806158092 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0c0c014070 con 0x7f0c241018d0 2023-09-10T01:01:11.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.719+0000 7f0c2259c700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0c100a06c0 0x7f0c100a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:11.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.720+0000 7f0c2259c700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0c100a06c0 0x7f0c100a2b80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0c24101730 tx=0x7f0c14009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:11.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.723+0000 7f0c1b7fe700 1 -- 172.21.15.3:0/3806158092 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0c0c095710 con 0x7f0c241018d0 2023-09-10T01:01:11.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.973+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c2419eb90 con 0x7f0c100a06c0 2023-09-10T01:01:11.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.977+0000 7f0c1b7fe700 1 -- 172.21.15.3:0/3806158092 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f0c2419eb90 con 0x7f0c100a06c0 2023-09-10T01:01:11.978 INFO:teuthology.orchestra.run.smithi003.stdout:/volumes/_nogroup/sv_1/18c4d3b9-37ec-4fad-8baf-bf37c9efeddc 2023-09-10T01:01:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.979+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0c100a06c0 msgr2=0x7f0c100a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.979+0000 7f0c28dff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0c100a06c0 0x7f0c100a2b80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0c24101730 tx=0x7f0c14009260 comp rx=0 tx=0).stop 2023-09-10T01:01:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c241018d0 msgr2=0x7f0c241969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c241018d0 0x7f0c241969a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00e9c0 tx=0x7f0c0c00ecd0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 shutdown_connections 2023-09-10T01:01:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0c100a06c0 0x7f0c100a2b80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c24102270 0x7f0c2419da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c241006e0 0x7f0c24196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 --2- 172.21.15.3:0/3806158092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c241018d0 0x7f0c241969a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 >> 172.21.15.3:0/3806158092 conn(0x7f0c240fbc50 msgr2=0x7f0c240fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:11.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 shutdown_connections 2023-09-10T01:01:11.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:11.980+0000 7f0c28dff700 1 -- 172.21.15.3:0/3806158092 wait complete. 2023-09-10T01:01:11.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:01:12.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 -- 172.21.15.3:0/1632724917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8074640 msgr2=0x7f3ff8072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1632724917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8074640 0x7f3ff8072a90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009990 tx=0x7f3fec009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 -- 172.21.15.3:0/1632724917 shutdown_connections 2023-09-10T01:01:12.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1632724917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ff8108590 0x7f3ff810aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1632724917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8074640 0x7f3ff8072a90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1632724917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ff8073060 0x7f3ff80734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 -- 172.21.15.3:0/1632724917 >> 172.21.15.3:0/1632724917 conn(0x7f3ff80fbc60 msgr2=0x7f3ff80fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:12.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 -- 172.21.15.3:0/1632724917 shutdown_connections 2023-09-10T01:01:12.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.211+0000 7f3ffefff700 1 -- 172.21.15.3:0/1632724917 wait complete. 2023-09-10T01:01:12.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.213+0000 7f3ffefff700 1 Processor -- start 2023-09-10T01:01:12.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.213+0000 7f3ffefff700 1 -- start start 2023-09-10T01:01:12.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.213+0000 7f3ffefff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8073060 0x7f3ff81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.213+0000 7f3ffefff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ff8074640 0x7f3ff8198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.213+0000 7f3ffefff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ff8108590 0x7f3ff819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffefff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff81a0170 con 0x7f3ff8073060 2023-09-10T01:01:12.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffefff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff81a02b0 con 0x7f3ff8108590 2023-09-10T01:01:12.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffefff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff81a0440 con 0x7f3ff8074640 2023-09-10T01:01:12.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8073060 0x7f3ff81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8073060 0x7f3ff81985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48462/0 (socket says 172.21.15.3:48462) 2023-09-10T01:01:12.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 -- 172.21.15.3:0/1356273486 learned_addr learned my addr 172.21.15.3:0/1356273486 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:12.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffd59c700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ff8108590 0x7f3ff819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ff7fff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ff8074640 0x7f3ff8198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 -- 172.21.15.3:0/1356273486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ff8074640 msgr2=0x7f3ff8198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ff8074640 0x7f3ff8198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 -- 172.21.15.3:0/1356273486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ff8108590 msgr2=0x7f3ff819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ff8108590 0x7f3ff819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 -- 172.21.15.3:0/1356273486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec0095f0 con 0x7f3ff8073060 2023-09-10T01:01:12.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ff7fff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ff8074640 0x7f3ff8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:12.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.214+0000 7f3ffcd9b700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8073060 0x7f3ff81985f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b680 tx=0x7f3fec005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:12.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.215+0000 7f3ff5ffb700 1 -- 172.21.15.3:0/1356273486 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec01d070 con 0x7f3ff8073060 2023-09-10T01:01:12.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.215+0000 7f3ff5ffb700 1 -- 172.21.15.3:0/1356273486 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fec022470 con 0x7f3ff8073060 2023-09-10T01:01:12.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.215+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff81a06c0 con 0x7f3ff8073060 2023-09-10T01:01:12.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.215+0000 7f3ff5ffb700 1 -- 172.21.15.3:0/1356273486 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec017650 con 0x7f3ff8073060 2023-09-10T01:01:12.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.215+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff81a0b40 con 0x7f3ff8073060 2023-09-10T01:01:12.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.216+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff8106c40 con 0x7f3ff8073060 2023-09-10T01:01:12.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.218+0000 7f3ff5ffb700 1 -- 172.21.15.3:0/1356273486 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f3fec0225e0 con 0x7f3ff8073060 2023-09-10T01:01:12.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.219+0000 7f3ff5ffb700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3fe80a0750 0x7f3fe80a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.219+0000 7f3ff5ffb700 1 -- 172.21.15.3:0/1356273486 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3fec0cd530 con 0x7f3ff8073060 2023-09-10T01:01:12.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.219+0000 7f3ff7fff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3fe80a0750 0x7f3fe80a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.220+0000 7f3ff7fff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3fe80a0750 0x7f3fe80a2c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8074360 tx=0x7f3fe4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:12.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.223+0000 7f3ff5ffb700 1 -- 172.21.15.3:0/1356273486 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3fec096d40 con 0x7f3ff8073060 2023-09-10T01:01:12.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:12 smithi003 ceph-mon[97654]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:12.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:12 smithi003 ceph-mon[97654]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:12 smithi136 ceph-mon[133708]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:12 smithi136 ceph-mon[133708]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:12.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:12 smithi092 ceph-mon[134067]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:12.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:12 smithi092 ceph-mon[134067]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:12.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.481+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ff810e110 con 0x7f3ff8073060 2023-09-10T01:01:12.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.482+0000 7f3ff5ffb700 1 -- 172.21.15.3:0/1356273486 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f3fec027090 con 0x7f3ff8073060 2023-09-10T01:01:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:12.483 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:01:12.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3fe80a0750 msgr2=0x7f3fe80a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3fe80a0750 0x7f3fe80a2c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8074360 tx=0x7f3fe4009260 comp rx=0 tx=0).stop 2023-09-10T01:01:12.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8073060 msgr2=0x7f3ff81985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8073060 0x7f3ff81985f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b680 tx=0x7f3fec005740 comp rx=0 tx=0).stop 2023-09-10T01:01:12.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 shutdown_connections 2023-09-10T01:01:12.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ff8108590 0x7f3ff819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ff8073060 0x7f3ff81985f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3fe80a0750 0x7f3fe80a2c10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 --2- 172.21.15.3:0/1356273486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ff8074640 0x7f3ff8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 >> 172.21.15.3:0/1356273486 conn(0x7f3ff80fbc60 msgr2=0x7f3ff80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:12.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 shutdown_connections 2023-09-10T01:01:12.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.484+0000 7f3ffefff700 1 -- 172.21.15.3:0/1356273486 wait complete. 2023-09-10T01:01:12.488 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:01:12.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:01:12.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2799652503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f98102260 msgr2=0x7f4f98108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2799652503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f98102260 0x7f4f98108b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000b0d0 tx=0x7f4f9000b3e0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2799652503 shutdown_connections 2023-09-10T01:01:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2799652503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f98102260 0x7f4f98108b10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2799652503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f981006d0 0x7f4f98100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2799652503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f981018c0 0x7f4f98101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2799652503 >> 172.21.15.3:0/2799652503 conn(0x7f4f980fbc60 msgr2=0x7f4f980fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:12.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2799652503 shutdown_connections 2023-09-10T01:01:12.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.708+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2799652503 wait complete. 2023-09-10T01:01:12.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.709+0000 7f4f9dbff700 1 Processor -- start 2023-09-10T01:01:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f9dbff700 1 -- start start 2023-09-10T01:01:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f9dbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f981006d0 0x7f4f980743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f9dbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f981018c0 0x7f4f980729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f9dbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f98102260 0x7f4f98072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f9dbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f980736d0 con 0x7f4f981018c0 2023-09-10T01:01:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f9dbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f98073840 con 0x7f4f98102260 2023-09-10T01:01:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f9dbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f981a0490 con 0x7f4f981006d0 2023-09-10T01:01:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f977fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f981006d0 0x7f4f980743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.710+0000 7f4f96ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f981018c0 0x7f4f980729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f98102260 0x7f4f98072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f96ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f981018c0 0x7f4f980729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48484/0 (socket says 172.21.15.3:48484) 2023-09-10T01:01:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f98102260 0x7f4f98072f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56424/0 (socket says 172.21.15.3:56424) 2023-09-10T01:01:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f96ffd700 1 -- 172.21.15.3:0/2784492294 learned_addr learned my addr 172.21.15.3:0/2784492294 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:12.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f977fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f981006d0 0x7f4f980743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47324/0 (socket says 172.21.15.3:47324) 2023-09-10T01:01:12.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 -- 172.21.15.3:0/2784492294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f981006d0 msgr2=0x7f4f980743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f981006d0 0x7f4f980743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 -- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f981018c0 msgr2=0x7f4f980729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f981018c0 0x7f4f980729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 -- 172.21.15.3:0/2784492294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f88009520 con 0x7f4f98102260 2023-09-10T01:01:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f96ffd700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f981018c0 0x7f4f980729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f97fff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f98102260 0x7f4f98072f30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000b770 tx=0x7f4f90008ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f94ff9700 1 -- 172.21.15.3:0/2784492294 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9000e040 con 0x7f4f98102260 2023-09-10T01:01:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f94ff9700 1 -- 172.21.15.3:0/2784492294 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f9001d930 con 0x7f4f98102260 2023-09-10T01:01:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.711+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f90009d00 con 0x7f4f98102260 2023-09-10T01:01:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.712+0000 7f4f94ff9700 1 -- 172.21.15.3:0/2784492294 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9001b8f0 con 0x7f4f98102260 2023-09-10T01:01:12.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.712+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f981a09b0 con 0x7f4f98102260 2023-09-10T01:01:12.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.712+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9810bef0 con 0x7f4f98102260 2023-09-10T01:01:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.715+0000 7f4f94ff9700 1 -- 172.21.15.3:0/2784492294 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f4f9001daa0 con 0x7f4f98102260 2023-09-10T01:01:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.716+0000 7f4f94ff9700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f800a06d0 0x7f4f800a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.716+0000 7f4f94ff9700 1 -- 172.21.15.3:0/2784492294 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4f900cd910 con 0x7f4f98102260 2023-09-10T01:01:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.716+0000 7f4f977fe700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f800a06d0 0x7f4f800a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.717+0000 7f4f977fe700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f800a06d0 0x7f4f800a2b90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f98101720 tx=0x7f4f88009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.718+0000 7f4f94ff9700 1 -- 172.21.15.3:0/2784492294 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f4f90097000 con 0x7f4f98102260 2023-09-10T01:01:12.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.930+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f9810c110 con 0x7f4f98102260 2023-09-10T01:01:12.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.932+0000 7f4f94ff9700 1 -- 172.21.15.3:0/2784492294 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24321 (secure 0 0 0) 0x7f4f9009a620 con 0x7f4f98102260 2023-09-10T01:01:12.934 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:12.934 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:56.950998+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:01:12.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f800a06d0 msgr2=0x7f4f800a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f800a06d0 0x7f4f800a2b90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f98101720 tx=0x7f4f88009260 comp rx=0 tx=0).stop 2023-09-10T01:01:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f98102260 msgr2=0x7f4f98072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f98102260 0x7f4f98072f30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000b770 tx=0x7f4f90008ec0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 shutdown_connections 2023-09-10T01:01:12.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f800a06d0 0x7f4f800a2b90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f981006d0 0x7f4f980743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f98102260 0x7f4f98072f30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 --2- 172.21.15.3:0/2784492294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f981018c0 0x7f4f980729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:12.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 >> 172.21.15.3:0/2784492294 conn(0x7f4f980fbc60 msgr2=0x7f4f980fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:12.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 shutdown_connections 2023-09-10T01:01:12.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:12.935+0000 7f4f9dbff700 1 -- 172.21.15.3:0/2784492294 wait complete. 2023-09-10T01:01:12.955 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-09-10T01:01:12.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-10T01:01:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 -- 172.21.15.3:0/652624170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 msgr2=0x7f88ec100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 --2- 172.21.15.3:0/652624170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec100af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f88e80099c0 tx=0x7f88e8009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 -- 172.21.15.3:0/652624170 shutdown_connections 2023-09-10T01:01:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 --2- 172.21.15.3:0/652624170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88ec102260 0x7f88ec108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 --2- 172.21.15.3:0/652624170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec100af0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 --2- 172.21.15.3:0/652624170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ec1018c0 0x7f88ec101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 -- 172.21.15.3:0/652624170 >> 172.21.15.3:0/652624170 conn(0x7f88ec0fbc60 msgr2=0x7f88ec0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:13.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 -- 172.21.15.3:0/652624170 shutdown_connections 2023-09-10T01:01:13.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.160+0000 7f88f39ff700 1 -- 172.21.15.3:0/652624170 wait complete. 2023-09-10T01:01:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.161+0000 7f88f39ff700 1 Processor -- start 2023-09-10T01:01:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.161+0000 7f88f39ff700 1 -- start start 2023-09-10T01:01:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.161+0000 7f88f39ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f39ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88ec1018c0 0x7f88ec198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f39ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ec102260 0x7f88ec19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f39ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1a0150 con 0x7f88ec1006d0 2023-09-10T01:01:13.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f39ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1a0290 con 0x7f88ec102260 2023-09-10T01:01:13.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f39ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1a0420 con 0x7f88ec1018c0 2023-09-10T01:01:13.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48494/0 (socket says 172.21.15.3:48494) 2023-09-10T01:01:13.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 -- 172.21.15.3:0/3102401429 learned_addr learned my addr 172.21.15.3:0/3102401429 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:13.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f1f9c700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ec102260 0x7f88ec19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f0f9a700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88ec1018c0 0x7f88ec198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 -- 172.21.15.3:0/3102401429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88ec1018c0 msgr2=0x7f88ec198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88ec1018c0 0x7f88ec198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 -- 172.21.15.3:0/3102401429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ec102260 msgr2=0x7f88ec19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ec102260 0x7f88ec19fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.162+0000 7f88f179b700 1 -- 172.21.15.3:0/3102401429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e80095f0 con 0x7f88ec1006d0 2023-09-10T01:01:13.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.163+0000 7f88f179b700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec1985d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009990 tx=0x7f88e800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:13.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.163+0000 7f88e27fc700 1 -- 172.21.15.3:0/3102401429 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e801d070 con 0x7f88ec1006d0 2023-09-10T01:01:13.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.163+0000 7f88e27fc700 1 -- 172.21.15.3:0/3102401429 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88e8005030 con 0x7f88ec1006d0 2023-09-10T01:01:13.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.163+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec1a06a0 con 0x7f88ec1006d0 2023-09-10T01:01:13.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.163+0000 7f88e27fc700 1 -- 172.21.15.3:0/3102401429 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8004a70 con 0x7f88ec1006d0 2023-09-10T01:01:13.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.163+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec1a0b20 con 0x7f88ec1006d0 2023-09-10T01:01:13.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.164+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec192d20 con 0x7f88ec1006d0 2023-09-10T01:01:13.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.166+0000 7f88e27fc700 1 -- 172.21.15.3:0/3102401429 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f88e8005410 con 0x7f88ec1006d0 2023-09-10T01:01:13.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.167+0000 7f88e27fc700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88d80a0750 0x7f88d80a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.167+0000 7f88e27fc700 1 -- 172.21.15.3:0/3102401429 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88e80ccd40 con 0x7f88ec1006d0 2023-09-10T01:01:13.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.168+0000 7f88f0f9a700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88d80a0750 0x7f88d80a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.168+0000 7f88f0f9a700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88d80a0750 0x7f88d80a2c10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009c60 tx=0x7f88dc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:13.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.171+0000 7f88e27fc700 1 -- 172.21.15.3:0/3102401429 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f88e8096430 con 0x7f88ec1006d0 2023-09-10T01:01:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:13 smithi003 ceph-mon[97654]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:13 smithi003 ceph-mon[97654]: pgmap v296: 129 pgs: 129 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.3 KiB/s wr, 3 op/s; 48 B/s, 0 objects/s recovering 2023-09-10T01:01:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:13 smithi003 ceph-mon[97654]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:13 smithi003 ceph-mon[97654]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-09-10T01:01:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:13 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1356273486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:01:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:13 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2784492294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:13.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.381+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88ec02ce60 con 0x7f88d80a0750 2023-09-10T01:01:13.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.397+0000 7f88e27fc700 1 -- 172.21.15.3:0/3102401429 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422710 (secure 0 0 0) 0x7f88ec02ce60 con 0x7f88d80a0750 2023-09-10T01:01:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88d80a0750 msgr2=0x7f88d80a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88d80a0750 0x7f88d80a2c10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009c60 tx=0x7f88dc009260 comp rx=0 tx=0).stop 2023-09-10T01:01:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 msgr2=0x7f88ec1985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec1985d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009990 tx=0x7f88e800f690 comp rx=0 tx=0).stop 2023-09-10T01:01:13.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 shutdown_connections 2023-09-10T01:01:13.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88ec102260 0x7f88ec19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88ec1006d0 0x7f88ec1985d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88d80a0750 0x7f88d80a2c10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 --2- 172.21.15.3:0/3102401429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88ec1018c0 0x7f88ec198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 >> 172.21.15.3:0/3102401429 conn(0x7f88ec0fbc60 msgr2=0x7f88ec107190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:13.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 shutdown_connections 2023-09-10T01:01:13.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.402+0000 7f88f39ff700 1 -- 172.21.15.3:0/3102401429 wait complete. 2023-09-10T01:01:13.412 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-09-10T01:01:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:13 smithi136 ceph-mon[133708]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:13 smithi136 ceph-mon[133708]: pgmap v296: 129 pgs: 129 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.3 KiB/s wr, 3 op/s; 48 B/s, 0 objects/s recovering 2023-09-10T01:01:13.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:13 smithi136 ceph-mon[133708]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:13.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:13 smithi136 ceph-mon[133708]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-09-10T01:01:13.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:13 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1356273486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:01:13.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:13 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2784492294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:13.422 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2023-09-10T01:01:11.853431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595814,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":139,"num_read_kb":122,"num_write":270,"num_write_kb":2968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335744,"kb_used_data":11516,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124606080,"statfs":{"total":1151940427776,"available":1151596625920,"internally_reserved":0,"allocated":11792384,"data_stored":7572747,"data_compressed":320148,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3036,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":44,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003565"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962125+0000","last_change":"2023-09-10T01:00:44.026223+0000","last_active":"2023-09-10T01:00:55.962125+0000","last_peered":"2023-09-10T01:00:55.962125+0000","last_clean":"2023-09-10T01:00:55.962125+0000","last_became_active":"2023-09-10T01:00:44.025731+0000","last_became_peered":"2023-09-10T01:00:44.025731+0000","last_unstale":"2023-09-10T01:00:55.962125+0000","last_undegraded":"2023-09-10T01:00:55.962125+0000","last_fullsized":"2023-09-10T01:00:55.962125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:51:48.949903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957342+0000","last_change":"2023-09-10T01:00:44.038916+0000","last_active":"2023-09-10T01:00:55.957342+0000","last_peered":"2023-09-10T01:00:55.957342+0000","last_clean":"2023-09-10T01:00:55.957342+0000","last_became_active":"2023-09-10T01:00:44.038546+0000","last_became_peered":"2023-09-10T01:00:44.038546+0000","last_unstale":"2023-09-10T01:00:55.957342+0000","last_undegraded":"2023-09-10T01:00:55.957342+0000","last_fullsized":"2023-09-10T01:00:55.957342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:05:56.096857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957670+0000","last_change":"2023-09-10T01:00:44.032942+0000","last_active":"2023-09-10T01:00:55.957670+0000","last_peered":"2023-09-10T01:00:55.957670+0000","last_clean":"2023-09-10T01:00:55.957670+0000","last_became_active":"2023-09-10T01:00:44.032546+0000","last_became_peered":"2023-09-10T01:00:44.032546+0000","last_unstale":"2023-09-10T01:00:55.957670+0000","last_undegraded":"2023-09-10T01:00:55.957670+0000","last_fullsized":"2023-09-10T01:00:55.957670+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:46.613487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961438+0000","last_change":"2023-09-10T01:00:44.031411+0000","last_active":"2023-09-10T01:00:55.961438+0000","last_peered":"2023-09-10T01:00:55.961438+0000","last_clean":"2023-09-10T01:00:55.961438+0000","last_became_active":"2023-09-10T01:00:44.031121+0000","last_became_peered":"2023-09-10T01:00:44.031121+0000","last_unstale":"2023-09-10T01:00:55.961438+0000","last_undegraded":"2023-09-10T01:00:55.961438+0000","last_fullsized":"2023-09-10T01:00:55.961438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:47.658116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954364+0000","last_change":"2023-09-10T01:00:44.037351+0000","last_active":"2023-09-10T01:00:55.954364+0000","last_peered":"2023-09-10T01:00:55.954364+0000","last_clean":"2023-09-10T01:00:55.954364+0000","last_became_active":"2023-09-10T01:00:44.037005+0000","last_became_peered":"2023-09-10T01:00:44.037005+0000","last_unstale":"2023-09-10T01:00:55.954364+0000","last_undegraded":"2023-09-10T01:00:55.954364+0000","last_fullsized":"2023-09-10T01:00:55.954364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:10:45.694056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959048+0000","last_change":"2023-09-10T01:00:44.029543+0000","last_active":"2023-09-10T01:00:55.959048+0000","last_peered":"2023-09-10T01:00:55.959048+0000","last_clean":"2023-09-10T01:00:55.959048+0000","last_became_active":"2023-09-10T01:00:44.029210+0000","last_became_peered":"2023-09-10T01:00:44.029210+0000","last_unstale":"2023-09-10T01:00:55.959048+0000","last_undegraded":"2023-09-10T01:00:55.959048+0000","last_fullsized":"2023-09-10T01:00:55.959048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:08:56.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957131+0000","last_change":"2023-09-10T01:00:44.033328+0000","last_active":"2023-09-10T01:00:55.957131+0000","last_peered":"2023-09-10T01:00:55.957131+0000","last_clean":"2023-09-10T01:00:55.957131+0000","last_became_active":"2023-09-10T01:00:44.033007+0000","last_became_peered":"2023-09-10T01:00:44.033007+0000","last_unstale":"2023-09-10T01:00:55.957131+0000","last_undegraded":"2023-09-10T01:00:55.957131+0000","last_fullsized":"2023-09-10T01:00:55.957131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:59:14.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954756+0000","last_change":"2023-09-10T01:00:44.025611+0000","last_active":"2023-09-10T01:00:55.954756+0000","last_peered":"2023-09-10T01:00:55.954756+0000","last_clean":"2023-09-10T01:00:55.954756+0000","last_became_active":"2023-09-10T01:00:44.025249+0000","last_became_peered":"2023-09-10T01:00:44.025249+0000","last_unstale":"2023-09-10T01:00:55.954756+0000","last_undegraded":"2023-09-10T01:00:55.954756+0000","last_fullsized":"2023-09-10T01:00:55.954756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:55:04.320356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954201+0000","last_change":"2023-09-10T01:00:44.024207+0000","last_active":"2023-09-10T01:00:55.954201+0000","last_peered":"2023-09-10T01:00:55.954201+0000","last_clean":"2023-09-10T01:00:55.954201+0000","last_became_active":"2023-09-10T01:00:44.023866+0000","last_became_peered":"2023-09-10T01:00:44.023866+0000","last_unstale":"2023-09-10T01:00:55.954201+0000","last_undegraded":"2023-09-10T01:00:55.954201+0000","last_fullsized":"2023-09-10T01:00:55.954201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:53:39.845384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960173+0000","last_change":"2023-09-10T01:00:44.031241+0000","last_active":"2023-09-10T01:00:55.960173+0000","last_peered":"2023-09-10T01:00:55.960173+0000","last_clean":"2023-09-10T01:00:55.960173+0000","last_became_active":"2023-09-10T01:00:44.030392+0000","last_became_peered":"2023-09-10T01:00:44.030392+0000","last_unstale":"2023-09-10T01:00:55.960173+0000","last_undegraded":"2023-09-10T01:00:55.960173+0000","last_fullsized":"2023-09-10T01:00:55.960173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:19:39.786995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957143+0000","last_change":"2023-09-10T01:00:44.026983+0000","last_active":"2023-09-10T01:00:55.957143+0000","last_peered":"2023-09-10T01:00:55.957143+0000","last_clean":"2023-09-10T01:00:55.957143+0000","last_became_active":"2023-09-10T01:00:44.026695+0000","last_became_peered":"2023-09-10T01:00:44.026695+0000","last_unstale":"2023-09-10T01:00:55.957143+0000","last_undegraded":"2023-09-10T01:00:55.957143+0000","last_fullsized":"2023-09-10T01:00:55.957143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:37.809144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961340+0000","last_change":"2023-09-10T01:00:44.030787+0000","last_active":"2023-09-10T01:00:55.961340+0000","last_peered":"2023-09-10T01:00:55.961340+0000","last_clean":"2023-09-10T01:00:55.961340+0000","last_became_active":"2023-09-10T01:00:44.028579+0000","last_became_peered":"2023-09-10T01:00:44.028579+0000","last_unstale":"2023-09-10T01:00:55.961340+0000","last_undegraded":"2023-09-10T01:00:55.961340+0000","last_fullsized":"2023-09-10T01:00:55.961340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:04:26.446378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962635+0000","last_change":"2023-09-10T01:00:44.029905+0000","last_active":"2023-09-10T01:00:55.962635+0000","last_peered":"2023-09-10T01:00:55.962635+0000","last_clean":"2023-09-10T01:00:55.962635+0000","last_became_active":"2023-09-10T01:00:44.029561+0000","last_became_peered":"2023-09-10T01:00:44.029561+0000","last_unstale":"2023-09-10T01:00:55.962635+0000","last_undegraded":"2023-09-10T01:00:55.962635+0000","last_fullsized":"2023-09-10T01:00:55.962635+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:17:50.839001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957833+0000","last_change":"2023-09-10T01:00:44.037864+0000","last_active":"2023-09-10T01:00:55.957833+0000","last_peered":"2023-09-10T01:00:55.957833+0000","last_clean":"2023-09-10T01:00:55.957833+0000","last_became_active":"2023-09-10T01:00:44.037533+0000","last_became_peered":"2023-09-10T01:00:44.037533+0000","last_unstale":"2023-09-10T01:00:55.957833+0000","last_undegraded":"2023-09-10T01:00:55.957833+0000","last_fullsized":"2023-09-10T01:00:55.957833+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:29:26.768570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957576+0000","last_change":"2023-09-10T01:00:44.033396+0000","last_active":"2023-09-10T01:00:55.957576+0000","last_peered":"2023-09-10T01:00:55.957576+0000","last_clean":"2023-09-10T01:00:55.957576+0000","last_became_active":"2023-09-10T01:00:44.033056+0000","last_became_peered":"2023-09-10T01:00:44.033056+0000","last_unstale":"2023-09-10T01:00:55.957576+0000","last_undegraded":"2023-09-10T01:00:55.957576+0000","last_fullsized":"2023-09-10T01:00:55.957576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:00:03.590555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.953671+0000","last_change":"2023-09-10T01:00:44.027729+0000","last_active":"2023-09-10T01:00:55.953671+0000","last_peered":"2023-09-10T01:00:55.953671+0000","last_clean":"2023-09-10T01:00:55.953671+0000","last_became_active":"2023-09-10T01:00:44.027447+0000","last_became_peered":"2023-09-10T01:00:44.027447+0000","last_unstale":"2023-09-10T01:00:55.953671+0000","last_undegraded":"2023-09-10T01:00:55.953671+0000","last_fullsized":"2023-09-10T01:00:55.953671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:06:09.391443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961881+0000","last_change":"2023-09-10T01:00:44.026042+0000","last_active":"2023-09-10T01:00:55.961881+0000","last_peered":"2023-09-10T01:00:55.961881+0000","last_clean":"2023-09-10T01:00:55.961881+0000","last_became_active":"2023-09-10T01:00:44.025309+0000","last_became_peered":"2023-09-10T01:00:44.025309+0000","last_unstale":"2023-09-10T01:00:55.961881+0000","last_undegraded":"2023-09-10T01:00:55.961881+0000","last_fullsized":"2023-09-10T01:00:55.961881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:51:29.038717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958982+0000","last_change":"2023-09-10T01:00:44.036761+0000","last_active":"2023-09-10T01:00:55.958982+0000","last_peered":"2023-09-10T01:00:55.958982+0000","last_clean":"2023-09-10T01:00:55.958982+0000","last_became_active":"2023-09-10T01:00:44.036333+0000","last_became_peered":"2023-09-10T01:00:44.036333+0000","last_unstale":"2023-09-10T01:00:55.958982+0000","last_undegraded":"2023-09-10T01:00:55.958982+0000","last_fullsized":"2023-09-10T01:00:55.958982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:37:09.042178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956974+0000","last_change":"2023-09-10T01:00:44.027304+0000","last_active":"2023-09-10T01:00:55.956974+0000","last_peered":"2023-09-10T01:00:55.956974+0000","last_clean":"2023-09-10T01:00:55.956974+0000","last_became_active":"2023-09-10T01:00:44.026994+0000","last_became_peered":"2023-09-10T01:00:44.026994+0000","last_unstale":"2023-09-10T01:00:55.956974+0000","last_undegraded":"2023-09-10T01:00:55.956974+0000","last_fullsized":"2023-09-10T01:00:55.956974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:11:04.824687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956637+0000","last_change":"2023-09-10T01:00:48.056098+0000","last_active":"2023-09-10T01:00:55.956637+0000","last_peered":"2023-09-10T01:00:55.956637+0000","last_clean":"2023-09-10T01:00:55.956637+0000","last_became_active":"2023-09-10T01:00:48.055694+0000","last_became_peered":"2023-09-10T01:00:48.055694+0000","last_unstale":"2023-09-10T01:00:55.956637+0000","last_undegraded":"2023-09-10T01:00:55.956637+0000","last_fullsized":"2023-09-10T01:00:55.956637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:30:48.711613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.040267+0000","last_change":"2023-09-10T01:00:56.971435+0000","last_active":"2023-09-10T01:01:09.040267+0000","last_peered":"2023-09-10T01:01:09.040267+0000","last_clean":"2023-09-10T01:01:09.040267+0000","last_became_active":"2023-09-10T01:00:56.965158+0000","last_became_peered":"2023-09-10T01:00:56.965158+0000","last_unstale":"2023-09-10T01:01:09.040267+0000","last_undegraded":"2023-09-10T01:01:09.040267+0000","last_fullsized":"2023-09-10T01:01:09.040267+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:09:20.910597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955501+0000","last_change":"2023-09-10T01:00:44.023469+0000","last_active":"2023-09-10T01:00:55.955501+0000","last_peered":"2023-09-10T01:00:55.955501+0000","last_clean":"2023-09-10T01:00:55.955501+0000","last_became_active":"2023-09-10T01:00:44.023232+0000","last_became_peered":"2023-09-10T01:00:44.023232+0000","last_unstale":"2023-09-10T01:00:55.955501+0000","last_undegraded":"2023-09-10T01:00:55.955501+0000","last_fullsized":"2023-09-10T01:00:55.955501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:33:22.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960543+0000","last_change":"2023-09-10T01:00:48.063565+0000","last_active":"2023-09-10T01:00:55.960543+0000","last_peered":"2023-09-10T01:00:55.960543+0000","last_clean":"2023-09-10T01:00:55.960543+0000","last_became_active":"2023-09-10T01:00:48.063200+0000","last_became_peered":"2023-09-10T01:00:48.063200+0000","last_unstale":"2023-09-10T01:00:55.960543+0000","last_undegraded":"2023-09-10T01:00:55.960543+0000","last_fullsized":"2023-09-10T01:00:55.960543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:50:16.624296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.962186+0000","last_change":"2023-09-10T01:00:43.010566+0000","last_active":"2023-09-10T01:00:56.962186+0000","last_peered":"2023-09-10T01:00:56.962186+0000","last_clean":"2023-09-10T01:00:56.962186+0000","last_became_active":"2023-09-10T01:00:43.010277+0000","last_became_peered":"2023-09-10T01:00:43.010277+0000","last_unstale":"2023-09-10T01:00:56.962186+0000","last_undegraded":"2023-09-10T01:00:56.962186+0000","last_fullsized":"2023-09-10T01:00:56.962186+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:37.808085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956906+0000","last_change":"2023-09-10T01:00:44.026219+0000","last_active":"2023-09-10T01:00:55.956906+0000","last_peered":"2023-09-10T01:00:55.956906+0000","last_clean":"2023-09-10T01:00:55.956906+0000","last_became_active":"2023-09-10T01:00:44.025906+0000","last_became_peered":"2023-09-10T01:00:44.025906+0000","last_unstale":"2023-09-10T01:00:55.956906+0000","last_undegraded":"2023-09-10T01:00:55.956906+0000","last_fullsized":"2023-09-10T01:00:55.956906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:21:48.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960992+0000","last_change":"2023-09-10T01:00:48.069671+0000","last_active":"2023-09-10T01:00:55.960992+0000","last_peered":"2023-09-10T01:00:55.960992+0000","last_clean":"2023-09-10T01:00:55.960992+0000","last_became_active":"2023-09-10T01:00:48.069382+0000","last_became_peered":"2023-09-10T01:00:48.069382+0000","last_unstale":"2023-09-10T01:00:55.960992+0000","last_undegraded":"2023-09-10T01:00:55.960992+0000","last_fullsized":"2023-09-10T01:00:55.960992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:06:20.526398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.038678+0000","last_change":"2023-09-10T01:00:43.005098+0000","last_active":"2023-09-10T01:01:09.038678+0000","last_peered":"2023-09-10T01:01:09.038678+0000","last_clean":"2023-09-10T01:01:09.038678+0000","last_became_active":"2023-09-10T01:00:43.004556+0000","last_became_peered":"2023-09-10T01:00:43.004556+0000","last_unstale":"2023-09-10T01:01:09.038678+0000","last_undegraded":"2023-09-10T01:01:09.038678+0000","last_fullsized":"2023-09-10T01:01:09.038678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:27:43.538936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954321+0000","last_change":"2023-09-10T01:00:44.034451+0000","last_active":"2023-09-10T01:00:55.954321+0000","last_peered":"2023-09-10T01:00:55.954321+0000","last_clean":"2023-09-10T01:00:55.954321+0000","last_became_active":"2023-09-10T01:00:44.034131+0000","last_became_peered":"2023-09-10T01:00:44.034131+0000","last_unstale":"2023-09-10T01:00:55.954321+0000","last_undegraded":"2023-09-10T01:00:55.954321+0000","last_fullsized":"2023-09-10T01:00:55.954321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:50.643215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959977+0000","last_change":"2023-09-10T01:00:48.060114+0000","last_active":"2023-09-10T01:00:55.959977+0000","last_peered":"2023-09-10T01:00:55.959977+0000","last_clean":"2023-09-10T01:00:55.959977+0000","last_became_active":"2023-09-10T01:00:48.059531+0000","last_became_peered":"2023-09-10T01:00:48.059531+0000","last_unstale":"2023-09-10T01:00:55.959977+0000","last_undegraded":"2023-09-10T01:00:55.959977+0000","last_fullsized":"2023-09-10T01:00:55.959977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:28:01.462247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954478+0000","last_change":"2023-09-10T01:00:43.007196+0000","last_active":"2023-09-10T01:00:55.954478+0000","last_peered":"2023-09-10T01:00:55.954478+0000","last_clean":"2023-09-10T01:00:55.954478+0000","last_became_active":"2023-09-10T01:00:43.006795+0000","last_became_peered":"2023-09-10T01:00:43.006795+0000","last_unstale":"2023-09-10T01:00:55.954478+0000","last_undegraded":"2023-09-10T01:00:55.954478+0000","last_fullsized":"2023-09-10T01:00:55.954478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:26:13.146244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960525+0000","last_change":"2023-09-10T01:00:44.027809+0000","last_active":"2023-09-10T01:00:55.960525+0000","last_peered":"2023-09-10T01:00:55.960525+0000","last_clean":"2023-09-10T01:00:55.960525+0000","last_became_active":"2023-09-10T01:00:44.027499+0000","last_became_peered":"2023-09-10T01:00:44.027499+0000","last_unstale":"2023-09-10T01:00:55.960525+0000","last_undegraded":"2023-09-10T01:00:55.960525+0000","last_fullsized":"2023-09-10T01:00:55.960525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:17:07.138807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956798+0000","last_change":"2023-09-10T01:00:48.057645+0000","last_active":"2023-09-10T01:00:55.956798+0000","last_peered":"2023-09-10T01:00:55.956798+0000","last_clean":"2023-09-10T01:00:55.956798+0000","last_became_active":"2023-09-10T01:00:48.057312+0000","last_became_peered":"2023-09-10T01:00:48.057312+0000","last_unstale":"2023-09-10T01:00:55.956798+0000","last_undegraded":"2023-09-10T01:00:55.956798+0000","last_fullsized":"2023-09-10T01:00:55.956798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:28.904572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.965677+0000","last_change":"2023-09-10T01:00:56.965677+0000","last_active":"2023-09-10T01:00:56.965677+0000","last_peered":"2023-09-10T01:00:56.965677+0000","last_clean":"2023-09-10T01:00:56.965677+0000","last_became_active":"2023-09-10T01:00:56.965248+0000","last_became_peered":"2023-09-10T01:00:56.965248+0000","last_unstale":"2023-09-10T01:00:56.965677+0000","last_undegraded":"2023-09-10T01:00:56.965677+0000","last_fullsized":"2023-09-10T01:00:56.965677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:39:44.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954486+0000","last_change":"2023-09-10T01:00:44.026340+0000","last_active":"2023-09-10T01:00:55.954486+0000","last_peered":"2023-09-10T01:00:55.954486+0000","last_clean":"2023-09-10T01:00:55.954486+0000","last_became_active":"2023-09-10T01:00:44.026024+0000","last_became_peered":"2023-09-10T01:00:44.026024+0000","last_unstale":"2023-09-10T01:00:55.954486+0000","last_undegraded":"2023-09-10T01:00:55.954486+0000","last_fullsized":"2023-09-10T01:00:55.954486+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:58:18.942980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957190+0000","last_change":"2023-09-10T01:00:48.057876+0000","last_active":"2023-09-10T01:00:55.957190+0000","last_peered":"2023-09-10T01:00:55.957190+0000","last_clean":"2023-09-10T01:00:55.957190+0000","last_became_active":"2023-09-10T01:00:48.056686+0000","last_became_peered":"2023-09-10T01:00:48.056686+0000","last_unstale":"2023-09-10T01:00:55.957190+0000","last_undegraded":"2023-09-10T01:00:55.957190+0000","last_fullsized":"2023-09-10T01:00:55.957190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:10:15.995463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959038+0000","last_change":"2023-09-10T01:00:43.007811+0000","last_active":"2023-09-10T01:00:55.959038+0000","last_peered":"2023-09-10T01:00:55.959038+0000","last_clean":"2023-09-10T01:00:55.959038+0000","last_became_active":"2023-09-10T01:00:43.007205+0000","last_became_peered":"2023-09-10T01:00:43.007205+0000","last_unstale":"2023-09-10T01:00:55.959038+0000","last_undegraded":"2023-09-10T01:00:55.959038+0000","last_fullsized":"2023-09-10T01:00:55.959038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:27:38.114614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958958+0000","last_change":"2023-09-10T01:00:44.027954+0000","last_active":"2023-09-10T01:00:55.958958+0000","last_peered":"2023-09-10T01:00:55.958958+0000","last_clean":"2023-09-10T01:00:55.958958+0000","last_became_active":"2023-09-10T01:00:44.027643+0000","last_became_peered":"2023-09-10T01:00:44.027643+0000","last_unstale":"2023-09-10T01:00:55.958958+0000","last_undegraded":"2023-09-10T01:00:55.958958+0000","last_fullsized":"2023-09-10T01:00:55.958958+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:21.253013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961779+0000","last_change":"2023-09-10T01:00:48.062519+0000","last_active":"2023-09-10T01:00:55.961779+0000","last_peered":"2023-09-10T01:00:55.961779+0000","last_clean":"2023-09-10T01:00:55.961779+0000","last_became_active":"2023-09-10T01:00:48.061985+0000","last_became_peered":"2023-09-10T01:00:48.061985+0000","last_unstale":"2023-09-10T01:00:55.961779+0000","last_undegraded":"2023-09-10T01:00:55.961779+0000","last_fullsized":"2023-09-10T01:00:55.961779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:47:36.002325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962069+0000","last_change":"2023-09-10T01:00:42.999601+0000","last_active":"2023-09-10T01:00:55.962069+0000","last_peered":"2023-09-10T01:00:55.962069+0000","last_clean":"2023-09-10T01:00:55.962069+0000","last_became_active":"2023-09-10T01:00:42.999243+0000","last_became_peered":"2023-09-10T01:00:42.999243+0000","last_unstale":"2023-09-10T01:00:55.962069+0000","last_undegraded":"2023-09-10T01:00:55.962069+0000","last_fullsized":"2023-09-10T01:00:55.962069+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:49.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955499+0000","last_change":"2023-09-10T01:00:44.027136+0000","last_active":"2023-09-10T01:00:55.955499+0000","last_peered":"2023-09-10T01:00:55.955499+0000","last_clean":"2023-09-10T01:00:55.955499+0000","last_became_active":"2023-09-10T01:00:44.026784+0000","last_became_peered":"2023-09-10T01:00:44.026784+0000","last_unstale":"2023-09-10T01:00:55.955499+0000","last_undegraded":"2023-09-10T01:00:55.955499+0000","last_fullsized":"2023-09-10T01:00:55.955499+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:51:51.271658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957259+0000","last_change":"2023-09-10T01:00:48.066906+0000","last_active":"2023-09-10T01:00:55.957259+0000","last_peered":"2023-09-10T01:00:55.957259+0000","last_clean":"2023-09-10T01:00:55.957259+0000","last_became_active":"2023-09-10T01:00:48.066474+0000","last_became_peered":"2023-09-10T01:00:48.066474+0000","last_unstale":"2023-09-10T01:00:55.957259+0000","last_undegraded":"2023-09-10T01:00:55.957259+0000","last_fullsized":"2023-09-10T01:00:55.957259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:48:26.827857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957761+0000","last_change":"2023-09-10T01:00:43.005922+0000","last_active":"2023-09-10T01:00:55.957761+0000","last_peered":"2023-09-10T01:00:55.957761+0000","last_clean":"2023-09-10T01:00:55.957761+0000","last_became_active":"2023-09-10T01:00:43.005517+0000","last_became_peered":"2023-09-10T01:00:43.005517+0000","last_unstale":"2023-09-10T01:00:55.957761+0000","last_undegraded":"2023-09-10T01:00:55.957761+0000","last_fullsized":"2023-09-10T01:00:55.957761+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:32:12.619034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958744+0000","last_change":"2023-09-10T01:00:44.036766+0000","last_active":"2023-09-10T01:00:55.958744+0000","last_peered":"2023-09-10T01:00:55.958744+0000","last_clean":"2023-09-10T01:00:55.958744+0000","last_became_active":"2023-09-10T01:00:44.036316+0000","last_became_peered":"2023-09-10T01:00:44.036316+0000","last_unstale":"2023-09-10T01:00:55.958744+0000","last_undegraded":"2023-09-10T01:00:55.958744+0000","last_fullsized":"2023-09-10T01:00:55.958744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:17:51.875838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.051716+0000","last_change":"2023-09-10T01:00:57.051716+0000","last_active":"2023-09-10T01:00:57.051716+0000","last_peered":"2023-09-10T01:00:57.051716+0000","last_clean":"2023-09-10T01:00:57.051716+0000","last_became_active":"2023-09-10T01:00:57.051334+0000","last_became_peered":"2023-09-10T01:00:57.051334+0000","last_unstale":"2023-09-10T01:00:57.051716+0000","last_undegraded":"2023-09-10T01:00:57.051716+0000","last_fullsized":"2023-09-10T01:00:57.051716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:30:52.598639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959459+0000","last_change":"2023-09-10T01:00:43.008695+0000","last_active":"2023-09-10T01:00:55.959459+0000","last_peered":"2023-09-10T01:00:55.959459+0000","last_clean":"2023-09-10T01:00:55.959459+0000","last_became_active":"2023-09-10T01:00:43.008278+0000","last_became_peered":"2023-09-10T01:00:43.008278+0000","last_unstale":"2023-09-10T01:00:55.959459+0000","last_undegraded":"2023-09-10T01:00:55.959459+0000","last_fullsized":"2023-09-10T01:00:55.959459+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:46:38.370582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954339+0000","last_change":"2023-09-10T01:00:44.030540+0000","last_active":"2023-09-10T01:00:55.954339+0000","last_peered":"2023-09-10T01:00:55.954339+0000","last_clean":"2023-09-10T01:00:55.954339+0000","last_became_active":"2023-09-10T01:00:44.030212+0000","last_became_peered":"2023-09-10T01:00:44.030212+0000","last_unstale":"2023-09-10T01:00:55.954339+0000","last_undegraded":"2023-09-10T01:00:55.954339+0000","last_fullsized":"2023-09-10T01:00:55.954339+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:57:06.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.053566+0000","last_change":"2023-09-10T01:00:57.052197+0000","last_active":"2023-09-10T01:00:57.053566+0000","last_peered":"2023-09-10T01:00:57.053566+0000","last_clean":"2023-09-10T01:00:57.053566+0000","last_became_active":"2023-09-10T01:00:57.051782+0000","last_became_peered":"2023-09-10T01:00:57.051782+0000","last_unstale":"2023-09-10T01:00:57.053566+0000","last_undegraded":"2023-09-10T01:00:57.053566+0000","last_fullsized":"2023-09-10T01:00:57.053566+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:36:49.680123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957143+0000","last_change":"2023-09-10T01:00:43.007242+0000","last_active":"2023-09-10T01:00:55.957143+0000","last_peered":"2023-09-10T01:00:55.957143+0000","last_clean":"2023-09-10T01:00:55.957143+0000","last_became_active":"2023-09-10T01:00:43.006912+0000","last_became_peered":"2023-09-10T01:00:43.006912+0000","last_unstale":"2023-09-10T01:00:55.957143+0000","last_undegraded":"2023-09-10T01:00:55.957143+0000","last_fullsized":"2023-09-10T01:00:55.957143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:48:22.783656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957014+0000","last_change":"2023-09-10T01:00:44.032747+0000","last_active":"2023-09-10T01:00:55.957014+0000","last_peered":"2023-09-10T01:00:55.957014+0000","last_clean":"2023-09-10T01:00:55.957014+0000","last_became_active":"2023-09-10T01:00:44.032416+0000","last_became_peered":"2023-09-10T01:00:44.032416+0000","last_unstale":"2023-09-10T01:00:55.957014+0000","last_undegraded":"2023-09-10T01:00:55.957014+0000","last_fullsized":"2023-09-10T01:00:55.957014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:59:04.713333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961279+0000","last_change":"2023-09-10T01:00:48.062520+0000","last_active":"2023-09-10T01:00:55.961279+0000","last_peered":"2023-09-10T01:00:55.961279+0000","last_clean":"2023-09-10T01:00:55.961279+0000","last_became_active":"2023-09-10T01:00:48.061870+0000","last_became_peered":"2023-09-10T01:00:48.061870+0000","last_unstale":"2023-09-10T01:00:55.961279+0000","last_undegraded":"2023-09-10T01:00:55.961279+0000","last_fullsized":"2023-09-10T01:00:55.961279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:24:38.798775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'3","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960431+0000","last_change":"2023-09-10T01:00:43.006513+0000","last_active":"2023-09-10T01:00:55.960431+0000","last_peered":"2023-09-10T01:00:55.960431+0000","last_clean":"2023-09-10T01:00:55.960431+0000","last_became_active":"2023-09-10T01:00:43.006166+0000","last_became_peered":"2023-09-10T01:00:43.006166+0000","last_unstale":"2023-09-10T01:00:55.960431+0000","last_undegraded":"2023-09-10T01:00:55.960431+0000","last_fullsized":"2023-09-10T01:00:55.960431+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:52:32.167513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954478+0000","last_change":"2023-09-10T01:00:44.031264+0000","last_active":"2023-09-10T01:00:55.954478+0000","last_peered":"2023-09-10T01:00:55.954478+0000","last_clean":"2023-09-10T01:00:55.954478+0000","last_became_active":"2023-09-10T01:00:44.030993+0000","last_became_peered":"2023-09-10T01:00:44.030993+0000","last_unstale":"2023-09-10T01:00:55.954478+0000","last_undegraded":"2023-09-10T01:00:55.954478+0000","last_fullsized":"2023-09-10T01:00:55.954478+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:19:27.658594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958114+0000","last_change":"2023-09-10T01:00:48.056886+0000","last_active":"2023-09-10T01:00:55.958114+0000","last_peered":"2023-09-10T01:00:55.958114+0000","last_clean":"2023-09-10T01:00:55.958114+0000","last_became_active":"2023-09-10T01:00:48.056411+0000","last_became_peered":"2023-09-10T01:00:48.056411+0000","last_unstale":"2023-09-10T01:00:55.958114+0000","last_undegraded":"2023-09-10T01:00:55.958114+0000","last_fullsized":"2023-09-10T01:00:55.958114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:56:42.062702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:59.651537+0000","last_change":"2023-09-10T01:00:43.009113+0000","last_active":"2023-09-10T01:00:59.651537+0000","last_peered":"2023-09-10T01:00:59.651537+0000","last_clean":"2023-09-10T01:00:59.651537+0000","last_became_active":"2023-09-10T01:00:43.008757+0000","last_became_peered":"2023-09-10T01:00:43.008757+0000","last_unstale":"2023-09-10T01:00:59.651537+0000","last_undegraded":"2023-09-10T01:00:59.651537+0000","last_fullsized":"2023-09-10T01:00:59.651537+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:29:24.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954805+0000","last_change":"2023-09-10T01:00:44.027127+0000","last_active":"2023-09-10T01:00:55.954805+0000","last_peered":"2023-09-10T01:00:55.954805+0000","last_clean":"2023-09-10T01:00:55.954805+0000","last_became_active":"2023-09-10T01:00:44.026852+0000","last_became_peered":"2023-09-10T01:00:44.026852+0000","last_unstale":"2023-09-10T01:00:55.954805+0000","last_undegraded":"2023-09-10T01:00:55.954805+0000","last_fullsized":"2023-09-10T01:00:55.954805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:14:20.805300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960892+0000","last_change":"2023-09-10T01:00:48.069337+0000","last_active":"2023-09-10T01:00:55.960892+0000","last_peered":"2023-09-10T01:00:55.960892+0000","last_clean":"2023-09-10T01:00:55.960892+0000","last_became_active":"2023-09-10T01:00:48.068893+0000","last_became_peered":"2023-09-10T01:00:48.068893+0000","last_unstale":"2023-09-10T01:00:55.960892+0000","last_undegraded":"2023-09-10T01:00:55.960892+0000","last_fullsized":"2023-09-10T01:00:55.960892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:24:40.074551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.038363+0000","last_change":"2023-09-10T01:00:43.010128+0000","last_active":"2023-09-10T01:01:09.038363+0000","last_peered":"2023-09-10T01:01:09.038363+0000","last_clean":"2023-09-10T01:01:09.038363+0000","last_became_active":"2023-09-10T01:00:43.009808+0000","last_became_peered":"2023-09-10T01:00:43.009808+0000","last_unstale":"2023-09-10T01:01:09.038363+0000","last_undegraded":"2023-09-10T01:01:09.038363+0000","last_fullsized":"2023-09-10T01:01:09.038363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:27.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962222+0000","last_change":"2023-09-10T01:00:44.038665+0000","last_active":"2023-09-10T01:00:55.962222+0000","last_peered":"2023-09-10T01:00:55.962222+0000","last_clean":"2023-09-10T01:00:55.962222+0000","last_became_active":"2023-09-10T01:00:44.037944+0000","last_became_peered":"2023-09-10T01:00:44.037944+0000","last_unstale":"2023-09-10T01:00:55.962222+0000","last_undegraded":"2023-09-10T01:00:55.962222+0000","last_fullsized":"2023-09-10T01:00:55.962222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:11:34.018249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.965594+0000","last_change":"2023-09-10T01:00:56.965594+0000","last_active":"2023-09-10T01:00:56.965594+0000","last_peered":"2023-09-10T01:00:56.965594+0000","last_clean":"2023-09-10T01:00:56.965594+0000","last_became_active":"2023-09-10T01:00:56.965006+0000","last_became_peered":"2023-09-10T01:00:56.965006+0000","last_unstale":"2023-09-10T01:00:56.965594+0000","last_undegraded":"2023-09-10T01:00:56.965594+0000","last_fullsized":"2023-09-10T01:00:56.965594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:13:16.946697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958099+0000","last_change":"2023-09-10T01:00:43.004855+0000","last_active":"2023-09-10T01:00:55.958099+0000","last_peered":"2023-09-10T01:00:55.958099+0000","last_clean":"2023-09-10T01:00:55.958099+0000","last_became_active":"2023-09-10T01:00:43.004464+0000","last_became_peered":"2023-09-10T01:00:43.004464+0000","last_unstale":"2023-09-10T01:00:55.958099+0000","last_undegraded":"2023-09-10T01:00:55.958099+0000","last_fullsized":"2023-09-10T01:00:55.958099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:29.517290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957252+0000","last_change":"2023-09-10T01:00:44.037273+0000","last_active":"2023-09-10T01:00:55.957252+0000","last_peered":"2023-09-10T01:00:55.957252+0000","last_clean":"2023-09-10T01:00:55.957252+0000","last_became_active":"2023-09-10T01:00:44.036965+0000","last_became_peered":"2023-09-10T01:00:44.036965+0000","last_unstale":"2023-09-10T01:00:55.957252+0000","last_undegraded":"2023-09-10T01:00:55.957252+0000","last_fullsized":"2023-09-10T01:00:55.957252+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:47:20.149002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958612+0000","last_change":"2023-09-10T01:00:48.058574+0000","last_active":"2023-09-10T01:00:55.958612+0000","last_peered":"2023-09-10T01:00:55.958612+0000","last_clean":"2023-09-10T01:00:55.958612+0000","last_became_active":"2023-09-10T01:00:48.058190+0000","last_became_peered":"2023-09-10T01:00:48.058190+0000","last_unstale":"2023-09-10T01:00:55.958612+0000","last_undegraded":"2023-09-10T01:00:55.958612+0000","last_fullsized":"2023-09-10T01:00:55.958612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:46:42.177996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960206+0000","last_change":"2023-09-10T01:00:43.003816+0000","last_active":"2023-09-10T01:00:55.960206+0000","last_peered":"2023-09-10T01:00:55.960206+0000","last_clean":"2023-09-10T01:00:55.960206+0000","last_became_active":"2023-09-10T01:00:43.003404+0000","last_became_peered":"2023-09-10T01:00:43.003404+0000","last_unstale":"2023-09-10T01:00:55.960206+0000","last_undegraded":"2023-09-10T01:00:55.960206+0000","last_fullsized":"2023-09-10T01:00:55.960206+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:55.208302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957538+0000","last_change":"2023-09-10T01:00:44.034607+0000","last_active":"2023-09-10T01:00:55.957538+0000","last_peered":"2023-09-10T01:00:55.957538+0000","last_clean":"2023-09-10T01:00:55.957538+0000","last_became_active":"2023-09-10T01:00:44.034244+0000","last_became_peered":"2023-09-10T01:00:44.034244+0000","last_unstale":"2023-09-10T01:00:55.957538+0000","last_undegraded":"2023-09-10T01:00:55.957538+0000","last_fullsized":"2023-09-10T01:00:55.957538+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:06:13.214160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962896+0000","last_change":"2023-09-10T01:00:48.058789+0000","last_active":"2023-09-10T01:00:55.962896+0000","last_peered":"2023-09-10T01:00:55.962896+0000","last_clean":"2023-09-10T01:00:55.962896+0000","last_became_active":"2023-09-10T01:00:48.058220+0000","last_became_peered":"2023-09-10T01:00:48.058220+0000","last_unstale":"2023-09-10T01:00:55.962896+0000","last_undegraded":"2023-09-10T01:00:55.962896+0000","last_fullsized":"2023-09-10T01:00:55.962896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:01:12.291739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.040113+0000","last_change":"2023-09-10T01:00:43.006109+0000","last_active":"2023-09-10T01:01:09.040113+0000","last_peered":"2023-09-10T01:01:09.040113+0000","last_clean":"2023-09-10T01:01:09.040113+0000","last_became_active":"2023-09-10T01:00:43.005753+0000","last_became_peered":"2023-09-10T01:00:43.005753+0000","last_unstale":"2023-09-10T01:01:09.040113+0000","last_undegraded":"2023-09-10T01:01:09.040113+0000","last_fullsized":"2023-09-10T01:01:09.040113+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:45:47.083991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.963038+0000","last_change":"2023-09-10T01:00:44.028576+0000","last_active":"2023-09-10T01:00:55.963038+0000","last_peered":"2023-09-10T01:00:55.963038+0000","last_clean":"2023-09-10T01:00:55.963038+0000","last_became_active":"2023-09-10T01:00:44.028305+0000","last_became_peered":"2023-09-10T01:00:44.028305+0000","last_unstale":"2023-09-10T01:00:55.963038+0000","last_undegraded":"2023-09-10T01:00:55.963038+0000","last_fullsized":"2023-09-10T01:00:55.963038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:59:12.866257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.963016+0000","last_change":"2023-09-10T01:00:48.069318+0000","last_active":"2023-09-10T01:00:55.963016+0000","last_peered":"2023-09-10T01:00:55.963016+0000","last_clean":"2023-09-10T01:00:55.963016+0000","last_became_active":"2023-09-10T01:00:48.068876+0000","last_became_peered":"2023-09-10T01:00:48.068876+0000","last_unstale":"2023-09-10T01:00:55.963016+0000","last_undegraded":"2023-09-10T01:00:55.963016+0000","last_fullsized":"2023-09-10T01:00:55.963016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:06:22.366530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954808+0000","last_change":"2023-09-10T01:00:43.004164+0000","last_active":"2023-09-10T01:00:55.954808+0000","last_peered":"2023-09-10T01:00:55.954808+0000","last_clean":"2023-09-10T01:00:55.954808+0000","last_became_active":"2023-09-10T01:00:43.003781+0000","last_became_peered":"2023-09-10T01:00:43.003781+0000","last_unstale":"2023-09-10T01:00:55.954808+0000","last_undegraded":"2023-09-10T01:00:55.954808+0000","last_fullsized":"2023-09-10T01:00:55.954808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:38:18.559712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961744+0000","last_change":"2023-09-10T01:00:44.028206+0000","last_active":"2023-09-10T01:00:55.961744+0000","last_peered":"2023-09-10T01:00:55.961744+0000","last_clean":"2023-09-10T01:00:55.961744+0000","last_became_active":"2023-09-10T01:00:44.027914+0000","last_became_peered":"2023-09-10T01:00:44.027914+0000","last_unstale":"2023-09-10T01:00:55.961744+0000","last_undegraded":"2023-09-10T01:00:55.961744+0000","last_fullsized":"2023-09-10T01:00:55.961744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:41:12.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954406+0000","last_change":"2023-09-10T01:00:48.428613+0000","last_active":"2023-09-10T01:00:55.954406+0000","last_peered":"2023-09-10T01:00:55.954406+0000","last_clean":"2023-09-10T01:00:55.954406+0000","last_became_active":"2023-09-10T01:00:48.428059+0000","last_became_peered":"2023-09-10T01:00:48.428059+0000","last_unstale":"2023-09-10T01:00:55.954406+0000","last_undegraded":"2023-09-10T01:00:55.954406+0000","last_fullsized":"2023-09-10T01:00:55.954406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:46:03.683279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958500+0000","last_change":"2023-09-10T01:00:43.006061+0000","last_active":"2023-09-10T01:00:55.958500+0000","last_peered":"2023-09-10T01:00:55.958500+0000","last_clean":"2023-09-10T01:00:55.958500+0000","last_became_active":"2023-09-10T01:00:43.005392+0000","last_became_peered":"2023-09-10T01:00:43.005392+0000","last_unstale":"2023-09-10T01:00:55.958500+0000","last_undegraded":"2023-09-10T01:00:55.958500+0000","last_fullsized":"2023-09-10T01:00:55.958500+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:01:45.062338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959373+0000","last_change":"2023-09-10T01:00:44.026684+0000","last_active":"2023-09-10T01:00:55.959373+0000","last_peered":"2023-09-10T01:00:55.959373+0000","last_clean":"2023-09-10T01:00:55.959373+0000","last_became_active":"2023-09-10T01:00:44.026387+0000","last_became_peered":"2023-09-10T01:00:44.026387+0000","last_unstale":"2023-09-10T01:00:55.959373+0000","last_undegraded":"2023-09-10T01:00:55.959373+0000","last_fullsized":"2023-09-10T01:00:55.959373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:23:03.662543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962732+0000","last_change":"2023-09-10T01:00:48.071589+0000","last_active":"2023-09-10T01:00:55.962732+0000","last_peered":"2023-09-10T01:00:55.962732+0000","last_clean":"2023-09-10T01:00:55.962732+0000","last_became_active":"2023-09-10T01:00:48.071118+0000","last_became_peered":"2023-09-10T01:00:48.071118+0000","last_unstale":"2023-09-10T01:00:55.962732+0000","last_undegraded":"2023-09-10T01:00:55.962732+0000","last_fullsized":"2023-09-10T01:00:55.962732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:27.194868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956703+0000","last_change":"2023-09-10T01:00:43.009738+0000","last_active":"2023-09-10T01:00:55.956703+0000","last_peered":"2023-09-10T01:00:55.956703+0000","last_clean":"2023-09-10T01:00:55.956703+0000","last_became_active":"2023-09-10T01:00:43.009376+0000","last_became_peered":"2023-09-10T01:00:43.009376+0000","last_unstale":"2023-09-10T01:00:55.956703+0000","last_undegraded":"2023-09-10T01:00:55.956703+0000","last_fullsized":"2023-09-10T01:00:55.956703+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:08.761672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959197+0000","last_change":"2023-09-10T01:00:44.030175+0000","last_active":"2023-09-10T01:00:55.959197+0000","last_peered":"2023-09-10T01:00:55.959197+0000","last_clean":"2023-09-10T01:00:55.959197+0000","last_became_active":"2023-09-10T01:00:44.029748+0000","last_became_peered":"2023-09-10T01:00:44.029748+0000","last_unstale":"2023-09-10T01:00:55.959197+0000","last_undegraded":"2023-09-10T01:00:55.959197+0000","last_fullsized":"2023-09-10T01:00:55.959197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:51:15.149610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.963010+0000","last_change":"2023-09-10T01:00:48.428008+0000","last_active":"2023-09-10T01:00:55.963010+0000","last_peered":"2023-09-10T01:00:55.963010+0000","last_clean":"2023-09-10T01:00:55.963010+0000","last_became_active":"2023-09-10T01:00:48.427498+0000","last_became_peered":"2023-09-10T01:00:48.427498+0000","last_unstale":"2023-09-10T01:00:55.963010+0000","last_undegraded":"2023-09-10T01:00:55.963010+0000","last_fullsized":"2023-09-10T01:00:55.963010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:16:48.938006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.051488+0000","last_change":"2023-09-10T01:00:57.051488+0000","last_active":"2023-09-10T01:00:57.051488+0000","last_peered":"2023-09-10T01:00:57.051488+0000","last_clean":"2023-09-10T01:00:57.051488+0000","last_became_active":"2023-09-10T01:00:57.051060+0000","last_became_peered":"2023-09-10T01:00:57.051060+0000","last_unstale":"2023-09-10T01:00:57.051488+0000","last_undegraded":"2023-09-10T01:00:57.051488+0000","last_fullsized":"2023-09-10T01:00:57.051488+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:29:15.803867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961991+0000","last_change":"2023-09-10T01:00:43.001047+0000","last_active":"2023-09-10T01:00:55.961991+0000","last_peered":"2023-09-10T01:00:55.961991+0000","last_clean":"2023-09-10T01:00:55.961991+0000","last_became_active":"2023-09-10T01:00:43.000521+0000","last_became_peered":"2023-09-10T01:00:43.000521+0000","last_unstale":"2023-09-10T01:00:55.961991+0000","last_undegraded":"2023-09-10T01:00:55.961991+0000","last_fullsized":"2023-09-10T01:00:55.961991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:15:34.509964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956641+0000","last_change":"2023-09-10T01:00:48.062148+0000","last_active":"2023-09-10T01:00:55.956641+0000","last_peered":"2023-09-10T01:00:55.956641+0000","last_clean":"2023-09-10T01:00:55.956641+0000","last_became_active":"2023-09-10T01:00:48.061746+0000","last_became_peered":"2023-09-10T01:00:48.061746+0000","last_unstale":"2023-09-10T01:00:55.956641+0000","last_undegraded":"2023-09-10T01:00:55.956641+0000","last_fullsized":"2023-09-10T01:00:55.956641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:40.087012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954815+0000","last_change":"2023-09-10T01:00:44.030500+0000","last_active":"2023-09-10T01:00:55.954815+0000","last_peered":"2023-09-10T01:00:55.954815+0000","last_clean":"2023-09-10T01:00:55.954815+0000","last_became_active":"2023-09-10T01:00:44.030209+0000","last_became_peered":"2023-09-10T01:00:44.030209+0000","last_unstale":"2023-09-10T01:00:55.954815+0000","last_undegraded":"2023-09-10T01:00:55.954815+0000","last_fullsized":"2023-09-10T01:00:55.954815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:53:30.421493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961161+0000","last_change":"2023-09-10T01:00:43.001102+0000","last_active":"2023-09-10T01:00:55.961161+0000","last_peered":"2023-09-10T01:00:55.961161+0000","last_clean":"2023-09-10T01:00:55.961161+0000","last_became_active":"2023-09-10T01:00:43.000715+0000","last_became_peered":"2023-09-10T01:00:43.000715+0000","last_unstale":"2023-09-10T01:00:55.961161+0000","last_undegraded":"2023-09-10T01:00:55.961161+0000","last_fullsized":"2023-09-10T01:00:55.961161+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:43:37.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":324,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957221+0000","last_change":"2023-09-10T00:59:40.261244+0000","last_active":"2023-09-10T01:00:55.957221+0000","last_peered":"2023-09-10T01:00:55.957221+0000","last_clean":"2023-09-10T01:00:55.957221+0000","last_became_active":"2023-09-10T00:59:40.260673+0000","last_became_peered":"2023-09-10T00:59:40.260673+0000","last_unstale":"2023-09-10T01:00:55.957221+0000","last_undegraded":"2023-09-10T01:00:55.957221+0000","last_fullsized":"2023-09-10T01:00:55.957221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_clean_scrub_stamp":"2023-09-10T00:56:36.374323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:04:03.614104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.052019+0000","last_change":"2023-09-10T01:00:57.052019+0000","last_active":"2023-09-10T01:00:57.052019+0000","last_peered":"2023-09-10T01:00:57.052019+0000","last_clean":"2023-09-10T01:00:57.052019+0000","last_became_active":"2023-09-10T01:00:57.051527+0000","last_became_peered":"2023-09-10T01:00:57.051527+0000","last_unstale":"2023-09-10T01:00:57.052019+0000","last_undegraded":"2023-09-10T01:00:57.052019+0000","last_fullsized":"2023-09-10T01:00:57.052019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:10:43.756761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961967+0000","last_change":"2023-09-10T01:00:44.033214+0000","last_active":"2023-09-10T01:00:55.961967+0000","last_peered":"2023-09-10T01:00:55.961967+0000","last_clean":"2023-09-10T01:00:55.961967+0000","last_became_active":"2023-09-10T01:00:44.032828+0000","last_became_peered":"2023-09-10T01:00:44.032828+0000","last_unstale":"2023-09-10T01:00:55.961967+0000","last_undegraded":"2023-09-10T01:00:55.961967+0000","last_fullsized":"2023-09-10T01:00:55.961967+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:01:22.417216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.965456+0000","last_change":"2023-09-10T01:00:56.965456+0000","last_active":"2023-09-10T01:00:56.965456+0000","last_peered":"2023-09-10T01:00:56.965456+0000","last_clean":"2023-09-10T01:00:56.965456+0000","last_became_active":"2023-09-10T01:00:56.965021+0000","last_became_peered":"2023-09-10T01:00:56.965021+0000","last_unstale":"2023-09-10T01:00:56.965456+0000","last_undegraded":"2023-09-10T01:00:56.965456+0000","last_fullsized":"2023-09-10T01:00:56.965456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:19.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957045+0000","last_change":"2023-09-10T01:00:48.066811+0000","last_active":"2023-09-10T01:00:55.957045+0000","last_peered":"2023-09-10T01:00:55.957045+0000","last_clean":"2023-09-10T01:00:55.957045+0000","last_became_active":"2023-09-10T01:00:48.066421+0000","last_became_peered":"2023-09-10T01:00:48.066421+0000","last_unstale":"2023-09-10T01:00:55.957045+0000","last_undegraded":"2023-09-10T01:00:55.957045+0000","last_fullsized":"2023-09-10T01:00:55.957045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:48:01.459070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.958768+0000","last_change":"2023-09-10T01:00:43.011134+0000","last_active":"2023-09-10T01:00:56.958768+0000","last_peered":"2023-09-10T01:00:56.958768+0000","last_clean":"2023-09-10T01:00:56.958768+0000","last_became_active":"2023-09-10T01:00:43.010871+0000","last_became_peered":"2023-09-10T01:00:43.010871+0000","last_unstale":"2023-09-10T01:00:56.958768+0000","last_undegraded":"2023-09-10T01:00:56.958768+0000","last_fullsized":"2023-09-10T01:00:56.958768+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:56.903745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962891+0000","last_change":"2023-09-10T01:00:44.031104+0000","last_active":"2023-09-10T01:00:55.962891+0000","last_peered":"2023-09-10T01:00:55.962891+0000","last_clean":"2023-09-10T01:00:55.962891+0000","last_became_active":"2023-09-10T01:00:44.030745+0000","last_became_peered":"2023-09-10T01:00:44.030745+0000","last_unstale":"2023-09-10T01:00:55.962891+0000","last_undegraded":"2023-09-10T01:00:55.962891+0000","last_fullsized":"2023-09-10T01:00:55.962891+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:53.885886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954861+0000","last_change":"2023-09-10T01:00:48.427735+0000","last_active":"2023-09-10T01:00:55.954861+0000","last_peered":"2023-09-10T01:00:55.954861+0000","last_clean":"2023-09-10T01:00:55.954861+0000","last_became_active":"2023-09-10T01:00:48.427342+0000","last_became_peered":"2023-09-10T01:00:48.427342+0000","last_unstale":"2023-09-10T01:00:55.954861+0000","last_undegraded":"2023-09-10T01:00:55.954861+0000","last_fullsized":"2023-09-10T01:00:55.954861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:01:52.007325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.039077+0000","last_change":"2023-09-10T01:00:43.006314+0000","last_active":"2023-09-10T01:01:09.039077+0000","last_peered":"2023-09-10T01:01:09.039077+0000","last_clean":"2023-09-10T01:01:09.039077+0000","last_became_active":"2023-09-10T01:00:43.004835+0000","last_became_peered":"2023-09-10T01:00:43.004835+0000","last_unstale":"2023-09-10T01:01:09.039077+0000","last_undegraded":"2023-09-10T01:01:09.039077+0000","last_fullsized":"2023-09-10T01:01:09.039077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:37:09.359846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955280+0000","last_change":"2023-09-10T01:00:44.029839+0000","last_active":"2023-09-10T01:00:55.955280+0000","last_peered":"2023-09-10T01:00:55.955280+0000","last_clean":"2023-09-10T01:00:55.955280+0000","last_became_active":"2023-09-10T01:00:44.029492+0000","last_became_peered":"2023-09-10T01:00:44.029492+0000","last_unstale":"2023-09-10T01:00:55.955280+0000","last_undegraded":"2023-09-10T01:00:55.955280+0000","last_fullsized":"2023-09-10T01:00:55.955280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:14.357386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956421+0000","last_change":"2023-09-10T01:00:48.054536+0000","last_active":"2023-09-10T01:00:55.956421+0000","last_peered":"2023-09-10T01:00:55.956421+0000","last_clean":"2023-09-10T01:00:55.956421+0000","last_became_active":"2023-09-10T01:00:48.053938+0000","last_became_peered":"2023-09-10T01:00:48.053938+0000","last_unstale":"2023-09-10T01:00:55.956421+0000","last_undegraded":"2023-09-10T01:00:55.956421+0000","last_fullsized":"2023-09-10T01:00:55.956421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:45:12.703592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962156+0000","last_change":"2023-09-10T01:00:43.002313+0000","last_active":"2023-09-10T01:00:55.962156+0000","last_peered":"2023-09-10T01:00:55.962156+0000","last_clean":"2023-09-10T01:00:55.962156+0000","last_became_active":"2023-09-10T01:00:43.001839+0000","last_became_peered":"2023-09-10T01:00:43.001839+0000","last_unstale":"2023-09-10T01:00:55.962156+0000","last_undegraded":"2023-09-10T01:00:55.962156+0000","last_fullsized":"2023-09-10T01:00:55.962156+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:51.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957427+0000","last_change":"2023-09-10T01:00:44.027525+0000","last_active":"2023-09-10T01:00:55.957427+0000","last_peered":"2023-09-10T01:00:55.957427+0000","last_clean":"2023-09-10T01:00:55.957427+0000","last_became_active":"2023-09-10T01:00:44.027132+0000","last_became_peered":"2023-09-10T01:00:44.027132+0000","last_unstale":"2023-09-10T01:00:55.957427+0000","last_undegraded":"2023-09-10T01:00:55.957427+0000","last_fullsized":"2023-09-10T01:00:55.957427+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:44.197242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961729+0000","last_change":"2023-09-10T01:00:48.428406+0000","last_active":"2023-09-10T01:00:55.961729+0000","last_peered":"2023-09-10T01:00:55.961729+0000","last_clean":"2023-09-10T01:00:55.961729+0000","last_became_active":"2023-09-10T01:00:48.427938+0000","last_became_peered":"2023-09-10T01:00:48.427938+0000","last_unstale":"2023-09-10T01:00:55.961729+0000","last_undegraded":"2023-09-10T01:00:55.961729+0000","last_fullsized":"2023-09-10T01:00:55.961729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:52.316235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.033706+0000","last_change":"2023-09-10T01:00:56.963755+0000","last_active":"2023-09-10T01:01:08.033706+0000","last_peered":"2023-09-10T01:01:08.033706+0000","last_clean":"2023-09-10T01:01:08.033706+0000","last_became_active":"2023-09-10T01:00:56.963283+0000","last_became_peered":"2023-09-10T01:00:56.963283+0000","last_unstale":"2023-09-10T01:01:08.033706+0000","last_undegraded":"2023-09-10T01:01:08.033706+0000","last_fullsized":"2023-09-10T01:01:08.033706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:13:32.213393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954232+0000","last_change":"2023-09-10T01:00:44.025284+0000","last_active":"2023-09-10T01:00:55.954232+0000","last_peered":"2023-09-10T01:00:55.954232+0000","last_clean":"2023-09-10T01:00:55.954232+0000","last_became_active":"2023-09-10T01:00:44.024965+0000","last_became_peered":"2023-09-10T01:00:44.024965+0000","last_unstale":"2023-09-10T01:00:55.954232+0000","last_undegraded":"2023-09-10T01:00:55.954232+0000","last_fullsized":"2023-09-10T01:00:55.954232+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:19:08.620325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958759+0000","last_change":"2023-09-10T01:00:48.070843+0000","last_active":"2023-09-10T01:00:55.958759+0000","last_peered":"2023-09-10T01:00:55.958759+0000","last_clean":"2023-09-10T01:00:55.958759+0000","last_became_active":"2023-09-10T01:00:48.070350+0000","last_became_peered":"2023-09-10T01:00:48.070350+0000","last_unstale":"2023-09-10T01:00:55.958759+0000","last_undegraded":"2023-09-10T01:00:55.958759+0000","last_fullsized":"2023-09-10T01:00:55.958759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:35:25.196036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.035280+0000","last_change":"2023-09-10T01:00:43.005398+0000","last_active":"2023-09-10T01:01:08.035280+0000","last_peered":"2023-09-10T01:01:08.035280+0000","last_clean":"2023-09-10T01:01:08.035280+0000","last_became_active":"2023-09-10T01:00:43.005087+0000","last_became_peered":"2023-09-10T01:00:43.005087+0000","last_unstale":"2023-09-10T01:01:08.035280+0000","last_undegraded":"2023-09-10T01:01:08.035280+0000","last_fullsized":"2023-09-10T01:01:08.035280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:29:04.042683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958684+0000","last_change":"2023-09-10T01:00:44.024451+0000","last_active":"2023-09-10T01:00:55.958684+0000","last_peered":"2023-09-10T01:00:55.958684+0000","last_clean":"2023-09-10T01:00:55.958684+0000","last_became_active":"2023-09-10T01:00:44.024045+0000","last_became_peered":"2023-09-10T01:00:44.024045+0000","last_unstale":"2023-09-10T01:00:55.958684+0000","last_undegraded":"2023-09-10T01:00:55.958684+0000","last_fullsized":"2023-09-10T01:00:55.958684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:55:50.750994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961033+0000","last_change":"2023-09-10T01:00:48.062247+0000","last_active":"2023-09-10T01:00:55.961033+0000","last_peered":"2023-09-10T01:00:55.961033+0000","last_clean":"2023-09-10T01:00:55.961033+0000","last_became_active":"2023-09-10T01:00:48.061812+0000","last_became_peered":"2023-09-10T01:00:48.061812+0000","last_unstale":"2023-09-10T01:00:55.961033+0000","last_undegraded":"2023-09-10T01:00:55.961033+0000","last_fullsized":"2023-09-10T01:00:55.961033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:44:13.596534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956724+0000","last_change":"2023-09-10T01:00:43.010845+0000","last_active":"2023-09-10T01:00:55.956724+0000","last_peered":"2023-09-10T01:00:55.956724+0000","last_clean":"2023-09-10T01:00:55.956724+0000","last_became_active":"2023-09-10T01:00:43.010511+0000","last_became_peered":"2023-09-10T01:00:43.010511+0000","last_unstale":"2023-09-10T01:00:55.956724+0000","last_undegraded":"2023-09-10T01:00:55.956724+0000","last_fullsized":"2023-09-10T01:00:55.956724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:33:07.229023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955056+0000","last_change":"2023-09-10T01:00:44.029025+0000","last_active":"2023-09-10T01:00:55.955056+0000","last_peered":"2023-09-10T01:00:55.955056+0000","last_clean":"2023-09-10T01:00:55.955056+0000","last_became_active":"2023-09-10T01:00:44.028713+0000","last_became_peered":"2023-09-10T01:00:44.028713+0000","last_unstale":"2023-09-10T01:00:55.955056+0000","last_undegraded":"2023-09-10T01:00:55.955056+0000","last_fullsized":"2023-09-10T01:00:55.955056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:40:22.315742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961553+0000","last_change":"2023-09-10T01:00:48.427294+0000","last_active":"2023-09-10T01:00:55.961553+0000","last_peered":"2023-09-10T01:00:55.961553+0000","last_clean":"2023-09-10T01:00:55.961553+0000","last_became_active":"2023-09-10T01:00:48.426882+0000","last_became_peered":"2023-09-10T01:00:48.426882+0000","last_unstale":"2023-09-10T01:00:55.961553+0000","last_undegraded":"2023-09-10T01:00:55.961553+0000","last_fullsized":"2023-09-10T01:00:55.961553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:54:38.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.031610+0000","last_change":"2023-09-10T01:00:43.006175+0000","last_active":"2023-09-10T01:01:08.031610+0000","last_peered":"2023-09-10T01:01:08.031610+0000","last_clean":"2023-09-10T01:01:08.031610+0000","last_became_active":"2023-09-10T01:00:43.005786+0000","last_became_peered":"2023-09-10T01:00:43.005786+0000","last_unstale":"2023-09-10T01:01:08.031610+0000","last_undegraded":"2023-09-10T01:01:08.031610+0000","last_fullsized":"2023-09-10T01:01:08.031610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:14:47.074866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956801+0000","last_change":"2023-09-10T01:00:44.041303+0000","last_active":"2023-09-10T01:00:55.956801+0000","last_peered":"2023-09-10T01:00:55.956801+0000","last_clean":"2023-09-10T01:00:55.956801+0000","last_became_active":"2023-09-10T01:00:44.041009+0000","last_became_peered":"2023-09-10T01:00:44.041009+0000","last_unstale":"2023-09-10T01:00:55.956801+0000","last_undegraded":"2023-09-10T01:00:55.956801+0000","last_fullsized":"2023-09-10T01:00:55.956801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:53:23.001700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954255+0000","last_change":"2023-09-10T01:00:48.064314+0000","last_active":"2023-09-10T01:00:55.954255+0000","last_peered":"2023-09-10T01:00:55.954255+0000","last_clean":"2023-09-10T01:00:55.954255+0000","last_became_active":"2023-09-10T01:00:48.063813+0000","last_became_peered":"2023-09-10T01:00:48.063813+0000","last_unstale":"2023-09-10T01:00:55.954255+0000","last_undegraded":"2023-09-10T01:00:55.954255+0000","last_fullsized":"2023-09-10T01:00:55.954255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:00:10.002006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.039829+0000","last_change":"2023-09-10T01:00:43.007579+0000","last_active":"2023-09-10T01:01:09.039829+0000","last_peered":"2023-09-10T01:01:09.039829+0000","last_clean":"2023-09-10T01:01:09.039829+0000","last_became_active":"2023-09-10T01:00:43.007200+0000","last_became_peered":"2023-09-10T01:00:43.007200+0000","last_unstale":"2023-09-10T01:01:09.039829+0000","last_undegraded":"2023-09-10T01:01:09.039829+0000","last_fullsized":"2023-09-10T01:01:09.039829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:20:08.688531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957330+0000","last_change":"2023-09-10T01:00:44.033511+0000","last_active":"2023-09-10T01:00:55.957330+0000","last_peered":"2023-09-10T01:00:55.957330+0000","last_clean":"2023-09-10T01:00:55.957330+0000","last_became_active":"2023-09-10T01:00:44.033170+0000","last_became_peered":"2023-09-10T01:00:44.033170+0000","last_unstale":"2023-09-10T01:00:55.957330+0000","last_undegraded":"2023-09-10T01:00:55.957330+0000","last_fullsized":"2023-09-10T01:00:55.957330+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:36:12.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957931+0000","last_change":"2023-09-10T01:00:48.067436+0000","last_active":"2023-09-10T01:00:55.957931+0000","last_peered":"2023-09-10T01:00:55.957931+0000","last_clean":"2023-09-10T01:00:55.957931+0000","last_became_active":"2023-09-10T01:00:48.067038+0000","last_became_peered":"2023-09-10T01:00:48.067038+0000","last_unstale":"2023-09-10T01:00:55.957931+0000","last_undegraded":"2023-09-10T01:00:55.957931+0000","last_fullsized":"2023-09-10T01:00:55.957931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:26:41.210698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960632+0000","last_change":"2023-09-10T01:00:43.006934+0000","last_active":"2023-09-10T01:00:55.960632+0000","last_peered":"2023-09-10T01:00:55.960632+0000","last_clean":"2023-09-10T01:00:55.960632+0000","last_became_active":"2023-09-10T01:00:43.005787+0000","last_became_peered":"2023-09-10T01:00:43.005787+0000","last_unstale":"2023-09-10T01:00:55.960632+0000","last_undegraded":"2023-09-10T01:00:55.960632+0000","last_fullsized":"2023-09-10T01:00:55.960632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:48:27.403220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956948+0000","last_change":"2023-09-10T01:00:44.033812+0000","last_active":"2023-09-10T01:00:55.956948+0000","last_peered":"2023-09-10T01:00:55.956948+0000","last_clean":"2023-09-10T01:00:55.956948+0000","last_became_active":"2023-09-10T01:00:44.033496+0000","last_became_peered":"2023-09-10T01:00:44.033496+0000","last_unstale":"2023-09-10T01:00:55.956948+0000","last_undegraded":"2023-09-10T01:00:55.956948+0000","last_fullsized":"2023-09-10T01:00:55.956948+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:12.679368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957698+0000","last_change":"2023-09-10T01:00:48.060358+0000","last_active":"2023-09-10T01:00:55.957698+0000","last_peered":"2023-09-10T01:00:55.957698+0000","last_clean":"2023-09-10T01:00:55.957698+0000","last_became_active":"2023-09-10T01:00:48.059853+0000","last_became_peered":"2023-09-10T01:00:48.059853+0000","last_unstale":"2023-09-10T01:00:55.957698+0000","last_undegraded":"2023-09-10T01:00:55.957698+0000","last_fullsized":"2023-09-10T01:00:55.957698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:31:56.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962921+0000","last_change":"2023-09-10T01:00:43.002820+0000","last_active":"2023-09-10T01:00:55.962921+0000","last_peered":"2023-09-10T01:00:55.962921+0000","last_clean":"2023-09-10T01:00:55.962921+0000","last_became_active":"2023-09-10T01:00:43.002456+0000","last_became_peered":"2023-09-10T01:00:43.002456+0000","last_unstale":"2023-09-10T01:00:55.962921+0000","last_undegraded":"2023-09-10T01:00:55.962921+0000","last_fullsized":"2023-09-10T01:00:55.962921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:40:10.189942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957369+0000","last_change":"2023-09-10T01:00:44.032655+0000","last_active":"2023-09-10T01:00:55.957369+0000","last_peered":"2023-09-10T01:00:55.957369+0000","last_clean":"2023-09-10T01:00:55.957369+0000","last_became_active":"2023-09-10T01:00:44.032157+0000","last_became_peered":"2023-09-10T01:00:44.032157+0000","last_unstale":"2023-09-10T01:00:55.957369+0000","last_undegraded":"2023-09-10T01:00:55.957369+0000","last_fullsized":"2023-09-10T01:00:55.957369+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:31.176768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958791+0000","last_change":"2023-09-10T01:00:44.022285+0000","last_active":"2023-09-10T01:00:55.958791+0000","last_peered":"2023-09-10T01:00:55.958791+0000","last_clean":"2023-09-10T01:00:55.958791+0000","last_became_active":"2023-09-10T01:00:44.021959+0000","last_became_peered":"2023-09-10T01:00:44.021959+0000","last_unstale":"2023-09-10T01:00:55.958791+0000","last_undegraded":"2023-09-10T01:00:55.958791+0000","last_fullsized":"2023-09-10T01:00:55.958791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:13:07.072249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957475+0000","last_change":"2023-09-10T01:00:44.029722+0000","last_active":"2023-09-10T01:00:55.957475+0000","last_peered":"2023-09-10T01:00:55.957475+0000","last_clean":"2023-09-10T01:00:55.957475+0000","last_became_active":"2023-09-10T01:00:44.029424+0000","last_became_peered":"2023-09-10T01:00:44.029424+0000","last_unstale":"2023-09-10T01:00:55.957475+0000","last_undegraded":"2023-09-10T01:00:55.957475+0000","last_fullsized":"2023-09-10T01:00:55.957475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:24:16.655538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956754+0000","last_change":"2023-09-10T01:00:44.026026+0000","last_active":"2023-09-10T01:00:55.956754+0000","last_peered":"2023-09-10T01:00:55.956754+0000","last_clean":"2023-09-10T01:00:55.956754+0000","last_became_active":"2023-09-10T01:00:44.025727+0000","last_became_peered":"2023-09-10T01:00:44.025727+0000","last_unstale":"2023-09-10T01:00:55.956754+0000","last_undegraded":"2023-09-10T01:00:55.956754+0000","last_fullsized":"2023-09-10T01:00:55.956754+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:59.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.965355+0000","last_change":"2023-09-10T01:00:56.965355+0000","last_active":"2023-09-10T01:00:56.965355+0000","last_peered":"2023-09-10T01:00:56.965355+0000","last_clean":"2023-09-10T01:00:56.965355+0000","last_became_active":"2023-09-10T01:00:56.964903+0000","last_became_peered":"2023-09-10T01:00:56.964903+0000","last_unstale":"2023-09-10T01:00:56.965355+0000","last_undegraded":"2023-09-10T01:00:56.965355+0000","last_fullsized":"2023-09-10T01:00:56.965355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:49:21.940159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962913+0000","last_change":"2023-09-10T01:00:44.029525+0000","last_active":"2023-09-10T01:00:55.962913+0000","last_peered":"2023-09-10T01:00:55.962913+0000","last_clean":"2023-09-10T01:00:55.962913+0000","last_became_active":"2023-09-10T01:00:44.029139+0000","last_became_peered":"2023-09-10T01:00:44.029139+0000","last_unstale":"2023-09-10T01:00:55.962913+0000","last_undegraded":"2023-09-10T01:00:55.962913+0000","last_fullsized":"2023-09-10T01:00:55.962913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:20:24.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959105+0000","last_change":"2023-09-10T01:00:44.022299+0000","last_active":"2023-09-10T01:00:55.959105+0000","last_peered":"2023-09-10T01:00:55.959105+0000","last_clean":"2023-09-10T01:00:55.959105+0000","last_became_active":"2023-09-10T01:00:44.021992+0000","last_became_peered":"2023-09-10T01:00:44.021992+0000","last_unstale":"2023-09-10T01:00:55.959105+0000","last_undegraded":"2023-09-10T01:00:55.959105+0000","last_fullsized":"2023-09-10T01:00:55.959105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:50:30.766442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957191+0000","last_change":"2023-09-10T01:00:44.027799+0000","last_active":"2023-09-10T01:00:55.957191+0000","last_peered":"2023-09-10T01:00:55.957191+0000","last_clean":"2023-09-10T01:00:55.957191+0000","last_became_active":"2023-09-10T01:00:44.027518+0000","last_became_peered":"2023-09-10T01:00:44.027518+0000","last_unstale":"2023-09-10T01:00:55.957191+0000","last_undegraded":"2023-09-10T01:00:55.957191+0000","last_fullsized":"2023-09-10T01:00:55.957191+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:00:20.315436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955430+0000","last_change":"2023-09-10T01:00:44.035921+0000","last_active":"2023-09-10T01:00:55.955430+0000","last_peered":"2023-09-10T01:00:55.955430+0000","last_clean":"2023-09-10T01:00:55.955430+0000","last_became_active":"2023-09-10T01:00:44.035634+0000","last_became_peered":"2023-09-10T01:00:44.035634+0000","last_unstale":"2023-09-10T01:00:55.955430+0000","last_undegraded":"2023-09-10T01:00:55.955430+0000","last_fullsized":"2023-09-10T01:00:55.955430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:44:29.069789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957822+0000","last_change":"2023-09-10T01:00:44.025405+0000","last_active":"2023-09-10T01:00:55.957822+0000","last_peered":"2023-09-10T01:00:55.957822+0000","last_clean":"2023-09-10T01:00:55.957822+0000","last_became_active":"2023-09-10T01:00:44.025040+0000","last_became_peered":"2023-09-10T01:00:44.025040+0000","last_unstale":"2023-09-10T01:00:55.957822+0000","last_undegraded":"2023-09-10T01:00:55.957822+0000","last_fullsized":"2023-09-10T01:00:55.957822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:16:46.504902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955771+0000","last_change":"2023-09-10T01:00:44.023603+0000","last_active":"2023-09-10T01:00:55.955771+0000","last_peered":"2023-09-10T01:00:55.955771+0000","last_clean":"2023-09-10T01:00:55.955771+0000","last_became_active":"2023-09-10T01:00:44.023279+0000","last_became_peered":"2023-09-10T01:00:44.023279+0000","last_unstale":"2023-09-10T01:00:55.955771+0000","last_undegraded":"2023-09-10T01:00:55.955771+0000","last_fullsized":"2023-09-10T01:00:55.955771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:18:05.737675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954289+0000","last_change":"2023-09-10T01:00:44.035653+0000","last_active":"2023-09-10T01:00:55.954289+0000","last_peered":"2023-09-10T01:00:55.954289+0000","last_clean":"2023-09-10T01:00:55.954289+0000","last_became_active":"2023-09-10T01:00:44.035374+0000","last_became_peered":"2023-09-10T01:00:44.035374+0000","last_unstale":"2023-09-10T01:00:55.954289+0000","last_undegraded":"2023-09-10T01:00:55.954289+0000","last_fullsized":"2023-09-10T01:00:55.954289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:14:45.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962663+0000","last_change":"2023-09-10T01:00:44.035805+0000","last_active":"2023-09-10T01:00:55.962663+0000","last_peered":"2023-09-10T01:00:55.962663+0000","last_clean":"2023-09-10T01:00:55.962663+0000","last_became_active":"2023-09-10T01:00:44.035479+0000","last_became_peered":"2023-09-10T01:00:44.035479+0000","last_unstale":"2023-09-10T01:00:55.962663+0000","last_undegraded":"2023-09-10T01:00:55.962663+0000","last_fullsized":"2023-09-10T01:00:55.962663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:36.662975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958379+0000","last_change":"2023-09-10T01:00:44.025418+0000","last_active":"2023-09-10T01:00:55.958379+0000","last_peered":"2023-09-10T01:00:55.958379+0000","last_clean":"2023-09-10T01:00:55.958379+0000","last_became_active":"2023-09-10T01:00:44.025104+0000","last_became_peered":"2023-09-10T01:00:44.025104+0000","last_unstale":"2023-09-10T01:00:55.958379+0000","last_undegraded":"2023-09-10T01:00:55.958379+0000","last_fullsized":"2023-09-10T01:00:55.958379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:05:13.050781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282844,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5446,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":89,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":51314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481808,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":479869,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.51100000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.435}]},{"osd":2,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.65000000000000002}]},{"osd":5,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48099999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.501},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66500000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.52500000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.75},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.69999999999999996}]},{"osd":10,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.71399999999999997}]}]},{"osd":10,"up_from":70,"seq":300647710739,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1073691,"data_compressed":53658,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":1,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.501}]},{"osd":2,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64000000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.7519999999999998,"5min":6.7519999999999998,"15min":6.7519999999999998},"last":0.81200000000000006}]},{"osd":4,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.61199999999999999}]},{"osd":5,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.59699999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.70399999999999996}]},{"osd":9,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.83299999999999996}]},{"osd":11,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.88}]}]},{"osd":9,"up_from":65,"seq":279172874265,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":480105,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.40999999999999998}]},{"osd":1,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.71199999999999997}]},{"osd":2,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.63},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.52100000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.73499999999999999}]},{"osd":5,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.68500000000000005}]},{"osd":6,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.569,"5min":1.569,"15min":1.569},"last":0.74299999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.83899999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.79300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135902,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":479694,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.58899999999999997,"15min":0.59999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.58099999999999996,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58799999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54100000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55500000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74399999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58299999999999996,"15min":0.57999999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81200000000000006,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.78700000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.45300000000000001,"15min":0.44700000000000001},"min":{"1min":0.309,"5min":0.254,"15min":0.254},"max":{"1min":0.64600000000000002,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.71599999999999997}]},{"osd":3,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68700000000000006,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.623,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.432,"15min":0.432},"max":{"1min":0.71599999999999997,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60599999999999998}]},{"osd":4,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79500000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78300000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.82799999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.41999999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76200000000000001,"5min":0.87,"15min":0.87},"last":0.81200000000000006}]},{"osd":6,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68200000000000005,"15min":0.68799999999999994},"min":{"1min":0.46300000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.873,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96799999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.56999999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84599999999999997,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.878}]},{"osd":9,"last update":"Sun Sep 10 01:00:17 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.94099999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.90100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364834,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":1069655,"data_compressed":53658,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91400000000000003,"5min":1.367,"15min":1.367},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.44400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.44400000000000001,"15min":0.434},"min":{"1min":0.29399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.58599999999999997,"15min":0.58099999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.61399999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.441,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85499999999999998,"5min":1.409,"15min":1.409},"last":0.435},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84799999999999998,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.56699999999999995}]},{"osd":4,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.505},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.626,"15min":0.61799999999999999},"min":{"1min":0.47499999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.77700000000000002}]},{"osd":6,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.93300000000000005,"5min":1.343,"15min":1.343},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0249999999999999,"5min":1.319,"15min":1.319},"last":0.59599999999999997}]},{"osd":8,"last update":"Sun Sep 10 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.69099999999999995,"15min":0.68300000000000005},"min":{"1min":0.495,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.68799999999999994}]},{"osd":9,"last update":"Sun Sep 10 01:00:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67500000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.64800000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":479287,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.56699999999999995}]},{"osd":1,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63400000000000001,"15min":0.61699999999999999},"min":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.91500000000000004,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.878}]},{"osd":2,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96999999999999997,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.749,"15min":0.753},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.59499999999999997}]},{"osd":3,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.48299999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.628},{"interface":"front","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0389999999999999,"5min":1.425,"15min":1.425},"last":0.67800000000000005}]},{"osd":4,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.93899999999999995,"5min":1.111,"15min":1.111},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.74399999999999999,"15min":0.73099999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.745}]},{"osd":5,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.85699999999999998,"15min":0.85999999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.3149999999999999,"5min":1.53,"15min":1.53},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.85199999999999998,"15min":0.84799999999999998},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.23,"5min":1.353,"15min":1.353},"last":0.72599999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.82499999999999996,"15min":0.81999999999999995},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0600000000000001,"5min":1.179,"15min":1.179},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.83199999999999996,"15min":0.82599999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1759999999999999,"5min":1.258,"15min":1.258},"last":0.81899999999999995}]},{"osd":8,"last update":"Sun Sep 10 01:00:57 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90900000000000003,"15min":0.91100000000000003},"min":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1319999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88600000000000001,"15min":0.88400000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2010000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.82999999999999996}]},{"osd":9,"last update":"Sun Sep 10 01:00:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.89200000000000002}]},{"osd":10,"last update":"Sun Sep 10 01:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.96799999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":500510,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54900000000000004,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.312,"15min":0.312},"max":{"1min":0.70899999999999996,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.55100000000000005},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":5.1580000000000004,"15min":5.1580000000000004},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.42799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.39400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.68899999999999995}]},{"osd":3,"last update":"Sun Sep 10 01:01:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.752},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.755,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.78000000000000003},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84399999999999997,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.622}]},{"osd":4,"last update":"Sun Sep 10 01:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.36499999999999999},"min":{"1min":0.32800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40200000000000002,"15min":0.376},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.66400000000000003}]},{"osd":5,"last update":"Sun Sep 10 01:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56699999999999995,"15min":0.54900000000000004},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.70699999999999996}]},{"osd":6,"last update":"Sun Sep 10 01:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.66200000000000003,"15min":0.66800000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82699999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":1.018}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57499999999999996,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.91400000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.871,"5min":0.874,"15min":0.874},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65300000000000002,"15min":0.64300000000000002},"min":{"1min":0.54900000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.95999999999999996}]},{"osd":9,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":1.2370000000000001}]},{"osd":10,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.86499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]}]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":479981,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 10 01:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.60299999999999998,"15min":0.58299999999999996},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57399999999999995,"15min":0.55200000000000005},"min":{"1min":0.48699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.81499999999999995}]},{"osd":2,"last update":"Sun Sep 10 01:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.627,"15min":0.59299999999999997},"min":{"1min":0.46100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.61399999999999999},"min":{"1min":0.50800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89700000000000002,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.72199999999999998}]},{"osd":3,"last update":"Sun Sep 10 00:59:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.40300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56299999999999994,"15min":0.55400000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47499999999999998}]},{"osd":4,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.60999999999999999,"15min":0.57499999999999996},"min":{"1min":0.56599999999999995,"5min":0.308,"15min":0.308},"max":{"1min":0.88800000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.63800000000000001,"15min":0.59899999999999998},"min":{"1min":0.55500000000000005,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72599999999999998,"15min":0.70499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1,"5min":1.125,"15min":1.125},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68200000000000005,"15min":0.66000000000000003},"min":{"1min":0.47799999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94399999999999995,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.78300000000000003}]},{"osd":6,"last update":"Sun Sep 10 01:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.68899999999999995,"15min":0.66700000000000004},"min":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.73899999999999999,"15min":0.71299999999999997},"min":{"1min":0.46999999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.89800000000000002}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.877,"15min":0.86699999999999999},"min":{"1min":0.627,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.83099999999999996,"15min":0.82499999999999996},"min":{"1min":0.49299999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.86299999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.93600000000000005,"15min":0.94499999999999995},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.1619999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.873},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.089,"5min":1.24,"15min":1.24},"last":0.89100000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:00:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.79700000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.71399999999999997}]},{"osd":11,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.91900000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":501238,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.54900000000000004,"15min":0.51100000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.54900000000000004,"15min":0.52900000000000003},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.55300000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56299999999999994,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82499999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56200000000000006,"15min":0.53600000000000003},"min":{"1min":0.40000000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.90300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.90600000000000003}]},{"osd":3,"last update":"Sun Sep 10 01:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.56899999999999995},"min":{"1min":0.49399999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.036,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.48099999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.89400000000000002,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.505}]},{"osd":4,"last update":"Sun Sep 10 01:00:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73799999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.55800000000000005},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.82799999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.432},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.81399999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60699999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.46200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94799999999999995,"5min":1.079,"15min":1.079},"last":0.78900000000000003}]},{"osd":7,"last update":"Sun Sep 10 01:00:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.879,"15min":0.879},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88100000000000001,"5min":1.01,"15min":1.01},"last":0.876}]},{"osd":8,"last update":"Sun Sep 10 01:00:53 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.36799999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"min":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92800000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.85999999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.68300000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.746},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.92900000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":479571,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41999999999999998,"15min":0.39000000000000001},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65000000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.42699999999999999},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48499999999999999}]},{"osd":1,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.52700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.117,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56100000000000005},"min":{"1min":0.40999999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.90100000000000002,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.90100000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68300000000000005,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.012,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68999999999999995,"15min":0.63600000000000001},"min":{"1min":0.499,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0820000000000001,"5min":1.226,"15min":1.226},"last":0.59399999999999997}]},{"osd":4,"last update":"Sun Sep 10 01:00:28 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.70699999999999996,"15min":0.67500000000000004},"min":{"1min":0.59599999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.751,"15min":0.71199999999999997},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.96599999999999997}]},{"osd":5,"last update":"Sun Sep 10 01:00:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0209999999999999,"5min":1.345,"15min":1.345},"last":0.63},{"interface":"front","average":{"1min":0.754,"5min":0.73599999999999999,"15min":0.71299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.129,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.64500000000000002,"15min":0.62},"min":{"1min":0.47999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.68400000000000005,"15min":0.65200000000000002},"min":{"1min":0.58599999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.81799999999999995}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78200000000000003,"15min":0.79300000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.093,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.53600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":1.3480000000000001,"5min":1.827,"15min":1.827},"last":0.77400000000000002}]},{"osd":8,"last update":"Sun Sep 10 00:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.88700000000000001}]},{"osd":10,"last update":"Sun Sep 10 01:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.745}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953520,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":479309,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.372,"15min":0.35599999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.79000000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.38200000000000001,"15min":0.372},"min":{"1min":0.27100000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.441}]},{"osd":2,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.434,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.621},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.81699999999999995}]},{"osd":3,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84699999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51300000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.623,"15min":0.60099999999999998},"min":{"1min":0.501,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.60799999999999998,"15min":0.58299999999999996},"min":{"1min":0.47999999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.72799999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.68200000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.54800000000000004,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.58699999999999997,"15min":0.57999999999999996},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.69899999999999995}]},{"osd":8,"last update":"Sun Sep 10 01:00:52 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.66000000000000003,"15min":0.64400000000000002},"min":{"1min":0.51400000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.74099999999999999,"15min":0.72899999999999998},"min":{"1min":0.58599999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95499999999999996,"5min":0.996,"15min":0.996},"last":0.73799999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:00:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.66600000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.76600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]}]},{"osd":5,"up_from":37,"seq":158913789995,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1069837,"data_compressed":53658,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.52600000000000002,"15min":0.502},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51900000000000002,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79400000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.79800000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56200000000000006,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56699999999999995,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.55600000000000005}]},{"osd":2,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39600000000000002,"15min":0.378},"min":{"1min":0.29199999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65000000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41999999999999998,"15min":0.40400000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.45300000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.89500000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59599999999999997,"15min":0.58299999999999996},"min":{"1min":0.41299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.60199999999999998,"15min":0.59199999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.51300000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:00:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.629,"15min":0.624},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.82999999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65100000000000002,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.97299999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.68600000000000005,"15min":0.68999999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.74399999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63700000000000001,"15min":0.64000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.61299999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:00:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.996}]},{"osd":10,"last update":"Sun Sep 10 01:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.92400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":21951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-10T01:01:13.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-10T01:01:13.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:13 smithi092 ceph-mon[134067]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:13.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:13 smithi092 ceph-mon[134067]: pgmap v296: 129 pgs: 129 active+clean; 582 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.3 KiB/s wr, 3 op/s; 48 B/s, 0 objects/s recovering 2023-09-10T01:01:13.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:13 smithi092 ceph-mon[134067]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:01:13.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:13 smithi092 ceph-mon[134067]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-09-10T01:01:13.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:13 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1356273486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:01:13.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:13 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2784492294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:13.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 -- 172.21.15.3:0/2216411428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab81018c0 msgr2=0x7feab8101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 --2- 172.21.15.3:0/2216411428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab81018c0 0x7feab8101d20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feaac0099c0 tx=0x7feaac009cd0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 -- 172.21.15.3:0/2216411428 shutdown_connections 2023-09-10T01:01:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 --2- 172.21.15.3:0/2216411428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feab81006d0 0x7feab8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 --2- 172.21.15.3:0/2216411428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feab8102260 0x7feab8108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 --2- 172.21.15.3:0/2216411428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab81018c0 0x7feab8101d20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 -- 172.21.15.3:0/2216411428 >> 172.21.15.3:0/2216411428 conn(0x7feab80fbc60 msgr2=0x7feab80fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:13.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 -- 172.21.15.3:0/2216411428 shutdown_connections 2023-09-10T01:01:13.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.665+0000 7feabcdff700 1 -- 172.21.15.3:0/2216411428 wait complete. 2023-09-10T01:01:13.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.666+0000 7feabcdff700 1 Processor -- start 2023-09-10T01:01:13.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.666+0000 7feabcdff700 1 -- start start 2023-09-10T01:01:13.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feabcdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feab81006d0 0x7feab8196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feabcdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feab81018c0 0x7feab8196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feabcdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab8102260 0x7feab819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feabcdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab819df80 con 0x7feab8102260 2023-09-10T01:01:13.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feabcdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab819e0c0 con 0x7feab81006d0 2023-09-10T01:01:13.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feabcdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab819e250 con 0x7feab81018c0 2023-09-10T01:01:13.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feab6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab8102260 0x7feab819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feab6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab8102260 0x7feab819d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48506/0 (socket says 172.21.15.3:48506) 2023-09-10T01:01:13.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feab6d9d700 1 -- 172.21.15.3:0/3992247585 learned_addr learned my addr 172.21.15.3:0/3992247585 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:13.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feab659c700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feab81006d0 0x7feab8196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feab5d9b700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feab81018c0 0x7feab8196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.667+0000 7feab6d9d700 1 -- 172.21.15.3:0/3992247585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feab81018c0 msgr2=0x7feab8196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feab6d9d700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feab81018c0 0x7feab8196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feab6d9d700 1 -- 172.21.15.3:0/3992247585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feab81006d0 msgr2=0x7feab8196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feab6d9d700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feab81006d0 0x7feab8196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feab6d9d700 1 -- 172.21.15.3:0/3992247585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0095f0 con 0x7feab8102260 2023-09-10T01:01:13.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feab659c700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feab81006d0 0x7feab8196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:13.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feab6d9d700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab8102260 0x7feab819d9b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feab000ef30 tx=0x7feab000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:13.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feaa77fe700 1 -- 172.21.15.3:0/3992247585 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0018070 con 0x7feab8102260 2023-09-10T01:01:13.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab819e4d0 con 0x7feab8102260 2023-09-10T01:01:13.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feaa77fe700 1 -- 172.21.15.3:0/3992247585 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feab000f040 con 0x7feab8102260 2023-09-10T01:01:13.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feaa77fe700 1 -- 172.21.15.3:0/3992247585 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab000cd00 con 0x7feab8102260 2023-09-10T01:01:13.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.668+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab819ea30 con 0x7feab8102260 2023-09-10T01:01:13.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.669+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab810bf40 con 0x7feab8102260 2023-09-10T01:01:13.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.671+0000 7feaa77fe700 1 -- 172.21.15.3:0/3992247585 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7feab0003a00 con 0x7feab8102260 2023-09-10T01:01:13.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.674+0000 7feaa77fe700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feaa00a06d0 0x7feaa00a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:13.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.674+0000 7feaa77fe700 1 -- 172.21.15.3:0/3992247585 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feab00cbbc0 con 0x7feab8102260 2023-09-10T01:01:13.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.674+0000 7feab659c700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feaa00a06d0 0x7feaa00a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:13.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.674+0000 7feab659c700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feaa00a06d0 0x7feaa00a2b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feab8101720 tx=0x7feaa8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:13.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.678+0000 7feaa77fe700 1 -- 172.21.15.3:0/3992247585 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7feab00952b0 con 0x7feab8102260 2023-09-10T01:01:13.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.885+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feab810c0d0 con 0x7feaa00a06d0 2023-09-10T01:01:13.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.898+0000 7feaa77fe700 1 -- 172.21.15.3:0/3992247585 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422846 (secure 0 0 0) 0x7feab810c0d0 con 0x7feaa00a06d0 2023-09-10T01:01:13.904 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:13.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feaa00a06d0 msgr2=0x7feaa00a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feaa00a06d0 0x7feaa00a2b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feab8101720 tx=0x7feaa8009260 comp rx=0 tx=0).stop 2023-09-10T01:01:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab8102260 msgr2=0x7feab819d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab8102260 0x7feab819d9b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feab000ef30 tx=0x7feab000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 shutdown_connections 2023-09-10T01:01:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feab8102260 0x7feab819d9b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feab81006d0 0x7feab8196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7feaa00a06d0 0x7feaa00a2b90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 --2- 172.21.15.3:0/3992247585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feab81018c0 0x7feab8196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 >> 172.21.15.3:0/3992247585 conn(0x7feab80fbc60 msgr2=0x7feab80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 shutdown_connections 2023-09-10T01:01:13.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:13.903+0000 7feabcdff700 1 -- 172.21.15.3:0/3992247585 wait complete. 2023-09-10T01:01:13.910 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-09-10T01:01:13.923 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2023-09-10T01:01:13.853874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633177,"num_objects":62,"num_object_clones":0,"num_object_copies":188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":171,"num_read_kb":180,"num_write":321,"num_write_kb":3021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335832,"kb_used_data":11640,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605992,"statfs":{"total":1151940427776,"available":1151596535808,"internally_reserved":0,"allocated":11919360,"data_stored":7630826,"data_compressed":320148,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40399,"num_objects":38,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":35,"num_read_kb":61,"num_write":95,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003501"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962125+0000","last_change":"2023-09-10T01:00:44.026223+0000","last_active":"2023-09-10T01:00:55.962125+0000","last_peered":"2023-09-10T01:00:55.962125+0000","last_clean":"2023-09-10T01:00:55.962125+0000","last_became_active":"2023-09-10T01:00:44.025731+0000","last_became_peered":"2023-09-10T01:00:44.025731+0000","last_unstale":"2023-09-10T01:00:55.962125+0000","last_undegraded":"2023-09-10T01:00:55.962125+0000","last_fullsized":"2023-09-10T01:00:55.962125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:51:48.949903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957342+0000","last_change":"2023-09-10T01:00:44.038916+0000","last_active":"2023-09-10T01:00:55.957342+0000","last_peered":"2023-09-10T01:00:55.957342+0000","last_clean":"2023-09-10T01:00:55.957342+0000","last_became_active":"2023-09-10T01:00:44.038546+0000","last_became_peered":"2023-09-10T01:00:44.038546+0000","last_unstale":"2023-09-10T01:00:55.957342+0000","last_undegraded":"2023-09-10T01:00:55.957342+0000","last_fullsized":"2023-09-10T01:00:55.957342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:05:56.096857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957670+0000","last_change":"2023-09-10T01:00:44.032942+0000","last_active":"2023-09-10T01:00:55.957670+0000","last_peered":"2023-09-10T01:00:55.957670+0000","last_clean":"2023-09-10T01:00:55.957670+0000","last_became_active":"2023-09-10T01:00:44.032546+0000","last_became_peered":"2023-09-10T01:00:44.032546+0000","last_unstale":"2023-09-10T01:00:55.957670+0000","last_undegraded":"2023-09-10T01:00:55.957670+0000","last_fullsized":"2023-09-10T01:00:55.957670+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:46.613487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961438+0000","last_change":"2023-09-10T01:00:44.031411+0000","last_active":"2023-09-10T01:00:55.961438+0000","last_peered":"2023-09-10T01:00:55.961438+0000","last_clean":"2023-09-10T01:00:55.961438+0000","last_became_active":"2023-09-10T01:00:44.031121+0000","last_became_peered":"2023-09-10T01:00:44.031121+0000","last_unstale":"2023-09-10T01:00:55.961438+0000","last_undegraded":"2023-09-10T01:00:55.961438+0000","last_fullsized":"2023-09-10T01:00:55.961438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:47.658116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954364+0000","last_change":"2023-09-10T01:00:44.037351+0000","last_active":"2023-09-10T01:00:55.954364+0000","last_peered":"2023-09-10T01:00:55.954364+0000","last_clean":"2023-09-10T01:00:55.954364+0000","last_became_active":"2023-09-10T01:00:44.037005+0000","last_became_peered":"2023-09-10T01:00:44.037005+0000","last_unstale":"2023-09-10T01:00:55.954364+0000","last_undegraded":"2023-09-10T01:00:55.954364+0000","last_fullsized":"2023-09-10T01:00:55.954364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:10:45.694056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959048+0000","last_change":"2023-09-10T01:00:44.029543+0000","last_active":"2023-09-10T01:00:55.959048+0000","last_peered":"2023-09-10T01:00:55.959048+0000","last_clean":"2023-09-10T01:00:55.959048+0000","last_became_active":"2023-09-10T01:00:44.029210+0000","last_became_peered":"2023-09-10T01:00:44.029210+0000","last_unstale":"2023-09-10T01:00:55.959048+0000","last_undegraded":"2023-09-10T01:00:55.959048+0000","last_fullsized":"2023-09-10T01:00:55.959048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:08:56.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957131+0000","last_change":"2023-09-10T01:00:44.033328+0000","last_active":"2023-09-10T01:00:55.957131+0000","last_peered":"2023-09-10T01:00:55.957131+0000","last_clean":"2023-09-10T01:00:55.957131+0000","last_became_active":"2023-09-10T01:00:44.033007+0000","last_became_peered":"2023-09-10T01:00:44.033007+0000","last_unstale":"2023-09-10T01:00:55.957131+0000","last_undegraded":"2023-09-10T01:00:55.957131+0000","last_fullsized":"2023-09-10T01:00:55.957131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:59:14.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954756+0000","last_change":"2023-09-10T01:00:44.025611+0000","last_active":"2023-09-10T01:00:55.954756+0000","last_peered":"2023-09-10T01:00:55.954756+0000","last_clean":"2023-09-10T01:00:55.954756+0000","last_became_active":"2023-09-10T01:00:44.025249+0000","last_became_peered":"2023-09-10T01:00:44.025249+0000","last_unstale":"2023-09-10T01:00:55.954756+0000","last_undegraded":"2023-09-10T01:00:55.954756+0000","last_fullsized":"2023-09-10T01:00:55.954756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:55:04.320356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954201+0000","last_change":"2023-09-10T01:00:44.024207+0000","last_active":"2023-09-10T01:00:55.954201+0000","last_peered":"2023-09-10T01:00:55.954201+0000","last_clean":"2023-09-10T01:00:55.954201+0000","last_became_active":"2023-09-10T01:00:44.023866+0000","last_became_peered":"2023-09-10T01:00:44.023866+0000","last_unstale":"2023-09-10T01:00:55.954201+0000","last_undegraded":"2023-09-10T01:00:55.954201+0000","last_fullsized":"2023-09-10T01:00:55.954201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:53:39.845384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960173+0000","last_change":"2023-09-10T01:00:44.031241+0000","last_active":"2023-09-10T01:00:55.960173+0000","last_peered":"2023-09-10T01:00:55.960173+0000","last_clean":"2023-09-10T01:00:55.960173+0000","last_became_active":"2023-09-10T01:00:44.030392+0000","last_became_peered":"2023-09-10T01:00:44.030392+0000","last_unstale":"2023-09-10T01:00:55.960173+0000","last_undegraded":"2023-09-10T01:00:55.960173+0000","last_fullsized":"2023-09-10T01:00:55.960173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:19:39.786995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957143+0000","last_change":"2023-09-10T01:00:44.026983+0000","last_active":"2023-09-10T01:00:55.957143+0000","last_peered":"2023-09-10T01:00:55.957143+0000","last_clean":"2023-09-10T01:00:55.957143+0000","last_became_active":"2023-09-10T01:00:44.026695+0000","last_became_peered":"2023-09-10T01:00:44.026695+0000","last_unstale":"2023-09-10T01:00:55.957143+0000","last_undegraded":"2023-09-10T01:00:55.957143+0000","last_fullsized":"2023-09-10T01:00:55.957143+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:37.809144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961340+0000","last_change":"2023-09-10T01:00:44.030787+0000","last_active":"2023-09-10T01:00:55.961340+0000","last_peered":"2023-09-10T01:00:55.961340+0000","last_clean":"2023-09-10T01:00:55.961340+0000","last_became_active":"2023-09-10T01:00:44.028579+0000","last_became_peered":"2023-09-10T01:00:44.028579+0000","last_unstale":"2023-09-10T01:00:55.961340+0000","last_undegraded":"2023-09-10T01:00:55.961340+0000","last_fullsized":"2023-09-10T01:00:55.961340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:04:26.446378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962635+0000","last_change":"2023-09-10T01:00:44.029905+0000","last_active":"2023-09-10T01:00:55.962635+0000","last_peered":"2023-09-10T01:00:55.962635+0000","last_clean":"2023-09-10T01:00:55.962635+0000","last_became_active":"2023-09-10T01:00:44.029561+0000","last_became_peered":"2023-09-10T01:00:44.029561+0000","last_unstale":"2023-09-10T01:00:55.962635+0000","last_undegraded":"2023-09-10T01:00:55.962635+0000","last_fullsized":"2023-09-10T01:00:55.962635+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:17:50.839001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957833+0000","last_change":"2023-09-10T01:00:44.037864+0000","last_active":"2023-09-10T01:00:55.957833+0000","last_peered":"2023-09-10T01:00:55.957833+0000","last_clean":"2023-09-10T01:00:55.957833+0000","last_became_active":"2023-09-10T01:00:44.037533+0000","last_became_peered":"2023-09-10T01:00:44.037533+0000","last_unstale":"2023-09-10T01:00:55.957833+0000","last_undegraded":"2023-09-10T01:00:55.957833+0000","last_fullsized":"2023-09-10T01:00:55.957833+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:29:26.768570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957576+0000","last_change":"2023-09-10T01:00:44.033396+0000","last_active":"2023-09-10T01:00:55.957576+0000","last_peered":"2023-09-10T01:00:55.957576+0000","last_clean":"2023-09-10T01:00:55.957576+0000","last_became_active":"2023-09-10T01:00:44.033056+0000","last_became_peered":"2023-09-10T01:00:44.033056+0000","last_unstale":"2023-09-10T01:00:55.957576+0000","last_undegraded":"2023-09-10T01:00:55.957576+0000","last_fullsized":"2023-09-10T01:00:55.957576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:00:03.590555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.953671+0000","last_change":"2023-09-10T01:00:44.027729+0000","last_active":"2023-09-10T01:00:55.953671+0000","last_peered":"2023-09-10T01:00:55.953671+0000","last_clean":"2023-09-10T01:00:55.953671+0000","last_became_active":"2023-09-10T01:00:44.027447+0000","last_became_peered":"2023-09-10T01:00:44.027447+0000","last_unstale":"2023-09-10T01:00:55.953671+0000","last_undegraded":"2023-09-10T01:00:55.953671+0000","last_fullsized":"2023-09-10T01:00:55.953671+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:06:09.391443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961881+0000","last_change":"2023-09-10T01:00:44.026042+0000","last_active":"2023-09-10T01:00:55.961881+0000","last_peered":"2023-09-10T01:00:55.961881+0000","last_clean":"2023-09-10T01:00:55.961881+0000","last_became_active":"2023-09-10T01:00:44.025309+0000","last_became_peered":"2023-09-10T01:00:44.025309+0000","last_unstale":"2023-09-10T01:00:55.961881+0000","last_undegraded":"2023-09-10T01:00:55.961881+0000","last_fullsized":"2023-09-10T01:00:55.961881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:51:29.038717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958982+0000","last_change":"2023-09-10T01:00:44.036761+0000","last_active":"2023-09-10T01:00:55.958982+0000","last_peered":"2023-09-10T01:00:55.958982+0000","last_clean":"2023-09-10T01:00:55.958982+0000","last_became_active":"2023-09-10T01:00:44.036333+0000","last_became_peered":"2023-09-10T01:00:44.036333+0000","last_unstale":"2023-09-10T01:00:55.958982+0000","last_undegraded":"2023-09-10T01:00:55.958982+0000","last_fullsized":"2023-09-10T01:00:55.958982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:37:09.042178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956974+0000","last_change":"2023-09-10T01:00:44.027304+0000","last_active":"2023-09-10T01:00:55.956974+0000","last_peered":"2023-09-10T01:00:55.956974+0000","last_clean":"2023-09-10T01:00:55.956974+0000","last_became_active":"2023-09-10T01:00:44.026994+0000","last_became_peered":"2023-09-10T01:00:44.026994+0000","last_unstale":"2023-09-10T01:00:55.956974+0000","last_undegraded":"2023-09-10T01:00:55.956974+0000","last_fullsized":"2023-09-10T01:00:55.956974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:11:04.824687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956637+0000","last_change":"2023-09-10T01:00:48.056098+0000","last_active":"2023-09-10T01:00:55.956637+0000","last_peered":"2023-09-10T01:00:55.956637+0000","last_clean":"2023-09-10T01:00:55.956637+0000","last_became_active":"2023-09-10T01:00:48.055694+0000","last_became_peered":"2023-09-10T01:00:48.055694+0000","last_unstale":"2023-09-10T01:00:55.956637+0000","last_undegraded":"2023-09-10T01:00:55.956637+0000","last_fullsized":"2023-09-10T01:00:55.956637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:30:48.711613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.040267+0000","last_change":"2023-09-10T01:00:56.971435+0000","last_active":"2023-09-10T01:01:09.040267+0000","last_peered":"2023-09-10T01:01:09.040267+0000","last_clean":"2023-09-10T01:01:09.040267+0000","last_became_active":"2023-09-10T01:00:56.965158+0000","last_became_peered":"2023-09-10T01:00:56.965158+0000","last_unstale":"2023-09-10T01:01:09.040267+0000","last_undegraded":"2023-09-10T01:01:09.040267+0000","last_fullsized":"2023-09-10T01:01:09.040267+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:09:20.910597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955501+0000","last_change":"2023-09-10T01:00:44.023469+0000","last_active":"2023-09-10T01:00:55.955501+0000","last_peered":"2023-09-10T01:00:55.955501+0000","last_clean":"2023-09-10T01:00:55.955501+0000","last_became_active":"2023-09-10T01:00:44.023232+0000","last_became_peered":"2023-09-10T01:00:44.023232+0000","last_unstale":"2023-09-10T01:00:55.955501+0000","last_undegraded":"2023-09-10T01:00:55.955501+0000","last_fullsized":"2023-09-10T01:00:55.955501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:33:22.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960543+0000","last_change":"2023-09-10T01:00:48.063565+0000","last_active":"2023-09-10T01:00:55.960543+0000","last_peered":"2023-09-10T01:00:55.960543+0000","last_clean":"2023-09-10T01:00:55.960543+0000","last_became_active":"2023-09-10T01:00:48.063200+0000","last_became_peered":"2023-09-10T01:00:48.063200+0000","last_unstale":"2023-09-10T01:00:55.960543+0000","last_undegraded":"2023-09-10T01:00:55.960543+0000","last_fullsized":"2023-09-10T01:00:55.960543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:50:16.624296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.962186+0000","last_change":"2023-09-10T01:00:43.010566+0000","last_active":"2023-09-10T01:00:56.962186+0000","last_peered":"2023-09-10T01:00:56.962186+0000","last_clean":"2023-09-10T01:00:56.962186+0000","last_became_active":"2023-09-10T01:00:43.010277+0000","last_became_peered":"2023-09-10T01:00:43.010277+0000","last_unstale":"2023-09-10T01:00:56.962186+0000","last_undegraded":"2023-09-10T01:00:56.962186+0000","last_fullsized":"2023-09-10T01:00:56.962186+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:37.808085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956906+0000","last_change":"2023-09-10T01:00:44.026219+0000","last_active":"2023-09-10T01:00:55.956906+0000","last_peered":"2023-09-10T01:00:55.956906+0000","last_clean":"2023-09-10T01:00:55.956906+0000","last_became_active":"2023-09-10T01:00:44.025906+0000","last_became_peered":"2023-09-10T01:00:44.025906+0000","last_unstale":"2023-09-10T01:00:55.956906+0000","last_undegraded":"2023-09-10T01:00:55.956906+0000","last_fullsized":"2023-09-10T01:00:55.956906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:21:48.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960992+0000","last_change":"2023-09-10T01:00:48.069671+0000","last_active":"2023-09-10T01:00:55.960992+0000","last_peered":"2023-09-10T01:00:55.960992+0000","last_clean":"2023-09-10T01:00:55.960992+0000","last_became_active":"2023-09-10T01:00:48.069382+0000","last_became_peered":"2023-09-10T01:00:48.069382+0000","last_unstale":"2023-09-10T01:00:55.960992+0000","last_undegraded":"2023-09-10T01:00:55.960992+0000","last_fullsized":"2023-09-10T01:00:55.960992+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:06:20.526398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.038678+0000","last_change":"2023-09-10T01:00:43.005098+0000","last_active":"2023-09-10T01:01:09.038678+0000","last_peered":"2023-09-10T01:01:09.038678+0000","last_clean":"2023-09-10T01:01:09.038678+0000","last_became_active":"2023-09-10T01:00:43.004556+0000","last_became_peered":"2023-09-10T01:00:43.004556+0000","last_unstale":"2023-09-10T01:01:09.038678+0000","last_undegraded":"2023-09-10T01:01:09.038678+0000","last_fullsized":"2023-09-10T01:01:09.038678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:27:43.538936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954321+0000","last_change":"2023-09-10T01:00:44.034451+0000","last_active":"2023-09-10T01:00:55.954321+0000","last_peered":"2023-09-10T01:00:55.954321+0000","last_clean":"2023-09-10T01:00:55.954321+0000","last_became_active":"2023-09-10T01:00:44.034131+0000","last_became_peered":"2023-09-10T01:00:44.034131+0000","last_unstale":"2023-09-10T01:00:55.954321+0000","last_undegraded":"2023-09-10T01:00:55.954321+0000","last_fullsized":"2023-09-10T01:00:55.954321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:50.643215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959977+0000","last_change":"2023-09-10T01:00:48.060114+0000","last_active":"2023-09-10T01:00:55.959977+0000","last_peered":"2023-09-10T01:00:55.959977+0000","last_clean":"2023-09-10T01:00:55.959977+0000","last_became_active":"2023-09-10T01:00:48.059531+0000","last_became_peered":"2023-09-10T01:00:48.059531+0000","last_unstale":"2023-09-10T01:00:55.959977+0000","last_undegraded":"2023-09-10T01:00:55.959977+0000","last_fullsized":"2023-09-10T01:00:55.959977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:28:01.462247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954478+0000","last_change":"2023-09-10T01:00:43.007196+0000","last_active":"2023-09-10T01:00:55.954478+0000","last_peered":"2023-09-10T01:00:55.954478+0000","last_clean":"2023-09-10T01:00:55.954478+0000","last_became_active":"2023-09-10T01:00:43.006795+0000","last_became_peered":"2023-09-10T01:00:43.006795+0000","last_unstale":"2023-09-10T01:00:55.954478+0000","last_undegraded":"2023-09-10T01:00:55.954478+0000","last_fullsized":"2023-09-10T01:00:55.954478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:26:13.146244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960525+0000","last_change":"2023-09-10T01:00:44.027809+0000","last_active":"2023-09-10T01:00:55.960525+0000","last_peered":"2023-09-10T01:00:55.960525+0000","last_clean":"2023-09-10T01:00:55.960525+0000","last_became_active":"2023-09-10T01:00:44.027499+0000","last_became_peered":"2023-09-10T01:00:44.027499+0000","last_unstale":"2023-09-10T01:00:55.960525+0000","last_undegraded":"2023-09-10T01:00:55.960525+0000","last_fullsized":"2023-09-10T01:00:55.960525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:17:07.138807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956798+0000","last_change":"2023-09-10T01:00:48.057645+0000","last_active":"2023-09-10T01:00:55.956798+0000","last_peered":"2023-09-10T01:00:55.956798+0000","last_clean":"2023-09-10T01:00:55.956798+0000","last_became_active":"2023-09-10T01:00:48.057312+0000","last_became_peered":"2023-09-10T01:00:48.057312+0000","last_unstale":"2023-09-10T01:00:55.956798+0000","last_undegraded":"2023-09-10T01:00:55.956798+0000","last_fullsized":"2023-09-10T01:00:55.956798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:28.904572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:10.954867+0000","last_change":"2023-09-10T01:00:56.965677+0000","last_active":"2023-09-10T01:01:10.954867+0000","last_peered":"2023-09-10T01:01:10.954867+0000","last_clean":"2023-09-10T01:01:10.954867+0000","last_became_active":"2023-09-10T01:00:56.965248+0000","last_became_peered":"2023-09-10T01:00:56.965248+0000","last_unstale":"2023-09-10T01:01:10.954867+0000","last_undegraded":"2023-09-10T01:01:10.954867+0000","last_fullsized":"2023-09-10T01:01:10.954867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:39:44.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954486+0000","last_change":"2023-09-10T01:00:44.026340+0000","last_active":"2023-09-10T01:00:55.954486+0000","last_peered":"2023-09-10T01:00:55.954486+0000","last_clean":"2023-09-10T01:00:55.954486+0000","last_became_active":"2023-09-10T01:00:44.026024+0000","last_became_peered":"2023-09-10T01:00:44.026024+0000","last_unstale":"2023-09-10T01:00:55.954486+0000","last_undegraded":"2023-09-10T01:00:55.954486+0000","last_fullsized":"2023-09-10T01:00:55.954486+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:58:18.942980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957190+0000","last_change":"2023-09-10T01:00:48.057876+0000","last_active":"2023-09-10T01:00:55.957190+0000","last_peered":"2023-09-10T01:00:55.957190+0000","last_clean":"2023-09-10T01:00:55.957190+0000","last_became_active":"2023-09-10T01:00:48.056686+0000","last_became_peered":"2023-09-10T01:00:48.056686+0000","last_unstale":"2023-09-10T01:00:55.957190+0000","last_undegraded":"2023-09-10T01:00:55.957190+0000","last_fullsized":"2023-09-10T01:00:55.957190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:10:15.995463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.040152+0000","last_change":"2023-09-10T01:00:43.007811+0000","last_active":"2023-09-10T01:01:09.040152+0000","last_peered":"2023-09-10T01:01:09.040152+0000","last_clean":"2023-09-10T01:01:09.040152+0000","last_became_active":"2023-09-10T01:00:43.007205+0000","last_became_peered":"2023-09-10T01:00:43.007205+0000","last_unstale":"2023-09-10T01:01:09.040152+0000","last_undegraded":"2023-09-10T01:01:09.040152+0000","last_fullsized":"2023-09-10T01:01:09.040152+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:27:38.114614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958958+0000","last_change":"2023-09-10T01:00:44.027954+0000","last_active":"2023-09-10T01:00:55.958958+0000","last_peered":"2023-09-10T01:00:55.958958+0000","last_clean":"2023-09-10T01:00:55.958958+0000","last_became_active":"2023-09-10T01:00:44.027643+0000","last_became_peered":"2023-09-10T01:00:44.027643+0000","last_unstale":"2023-09-10T01:00:55.958958+0000","last_undegraded":"2023-09-10T01:00:55.958958+0000","last_fullsized":"2023-09-10T01:00:55.958958+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:21.253013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961779+0000","last_change":"2023-09-10T01:00:48.062519+0000","last_active":"2023-09-10T01:00:55.961779+0000","last_peered":"2023-09-10T01:00:55.961779+0000","last_clean":"2023-09-10T01:00:55.961779+0000","last_became_active":"2023-09-10T01:00:48.061985+0000","last_became_peered":"2023-09-10T01:00:48.061985+0000","last_unstale":"2023-09-10T01:00:55.961779+0000","last_undegraded":"2023-09-10T01:00:55.961779+0000","last_fullsized":"2023-09-10T01:00:55.961779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:47:36.002325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.040274+0000","last_change":"2023-09-10T01:00:42.999601+0000","last_active":"2023-09-10T01:01:09.040274+0000","last_peered":"2023-09-10T01:01:09.040274+0000","last_clean":"2023-09-10T01:01:09.040274+0000","last_became_active":"2023-09-10T01:00:42.999243+0000","last_became_peered":"2023-09-10T01:00:42.999243+0000","last_unstale":"2023-09-10T01:01:09.040274+0000","last_undegraded":"2023-09-10T01:01:09.040274+0000","last_fullsized":"2023-09-10T01:01:09.040274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:49.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955499+0000","last_change":"2023-09-10T01:00:44.027136+0000","last_active":"2023-09-10T01:00:55.955499+0000","last_peered":"2023-09-10T01:00:55.955499+0000","last_clean":"2023-09-10T01:00:55.955499+0000","last_became_active":"2023-09-10T01:00:44.026784+0000","last_became_peered":"2023-09-10T01:00:44.026784+0000","last_unstale":"2023-09-10T01:00:55.955499+0000","last_undegraded":"2023-09-10T01:00:55.955499+0000","last_fullsized":"2023-09-10T01:00:55.955499+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:51:51.271658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957259+0000","last_change":"2023-09-10T01:00:48.066906+0000","last_active":"2023-09-10T01:00:55.957259+0000","last_peered":"2023-09-10T01:00:55.957259+0000","last_clean":"2023-09-10T01:00:55.957259+0000","last_became_active":"2023-09-10T01:00:48.066474+0000","last_became_peered":"2023-09-10T01:00:48.066474+0000","last_unstale":"2023-09-10T01:00:55.957259+0000","last_undegraded":"2023-09-10T01:00:55.957259+0000","last_fullsized":"2023-09-10T01:00:55.957259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:48:26.827857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.957929+0000","last_change":"2023-09-10T01:00:43.005922+0000","last_active":"2023-09-10T01:00:56.957929+0000","last_peered":"2023-09-10T01:00:56.957929+0000","last_clean":"2023-09-10T01:00:56.957929+0000","last_became_active":"2023-09-10T01:00:43.005517+0000","last_became_peered":"2023-09-10T01:00:43.005517+0000","last_unstale":"2023-09-10T01:00:56.957929+0000","last_undegraded":"2023-09-10T01:00:56.957929+0000","last_fullsized":"2023-09-10T01:00:56.957929+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:32:12.619034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958744+0000","last_change":"2023-09-10T01:00:44.036766+0000","last_active":"2023-09-10T01:00:55.958744+0000","last_peered":"2023-09-10T01:00:55.958744+0000","last_clean":"2023-09-10T01:00:55.958744+0000","last_became_active":"2023-09-10T01:00:44.036316+0000","last_became_peered":"2023-09-10T01:00:44.036316+0000","last_unstale":"2023-09-10T01:00:55.958744+0000","last_undegraded":"2023-09-10T01:00:55.958744+0000","last_fullsized":"2023-09-10T01:00:55.958744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:17:51.875838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.051716+0000","last_change":"2023-09-10T01:00:57.051716+0000","last_active":"2023-09-10T01:00:57.051716+0000","last_peered":"2023-09-10T01:00:57.051716+0000","last_clean":"2023-09-10T01:00:57.051716+0000","last_became_active":"2023-09-10T01:00:57.051334+0000","last_became_peered":"2023-09-10T01:00:57.051334+0000","last_unstale":"2023-09-10T01:00:57.051716+0000","last_undegraded":"2023-09-10T01:00:57.051716+0000","last_fullsized":"2023-09-10T01:00:57.051716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:30:52.598639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:10.953766+0000","last_change":"2023-09-10T01:00:43.008695+0000","last_active":"2023-09-10T01:01:10.953766+0000","last_peered":"2023-09-10T01:01:10.953766+0000","last_clean":"2023-09-10T01:01:10.953766+0000","last_became_active":"2023-09-10T01:00:43.008278+0000","last_became_peered":"2023-09-10T01:00:43.008278+0000","last_unstale":"2023-09-10T01:01:10.953766+0000","last_undegraded":"2023-09-10T01:01:10.953766+0000","last_fullsized":"2023-09-10T01:01:10.953766+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:46:38.370582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954339+0000","last_change":"2023-09-10T01:00:44.030540+0000","last_active":"2023-09-10T01:00:55.954339+0000","last_peered":"2023-09-10T01:00:55.954339+0000","last_clean":"2023-09-10T01:00:55.954339+0000","last_became_active":"2023-09-10T01:00:44.030212+0000","last_became_peered":"2023-09-10T01:00:44.030212+0000","last_unstale":"2023-09-10T01:00:55.954339+0000","last_undegraded":"2023-09-10T01:00:55.954339+0000","last_fullsized":"2023-09-10T01:00:55.954339+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:57:06.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.053566+0000","last_change":"2023-09-10T01:00:57.052197+0000","last_active":"2023-09-10T01:00:57.053566+0000","last_peered":"2023-09-10T01:00:57.053566+0000","last_clean":"2023-09-10T01:00:57.053566+0000","last_became_active":"2023-09-10T01:00:57.051782+0000","last_became_peered":"2023-09-10T01:00:57.051782+0000","last_unstale":"2023-09-10T01:00:57.053566+0000","last_undegraded":"2023-09-10T01:00:57.053566+0000","last_fullsized":"2023-09-10T01:00:57.053566+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:36:49.680123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.034151+0000","last_change":"2023-09-10T01:00:43.007242+0000","last_active":"2023-09-10T01:01:08.034151+0000","last_peered":"2023-09-10T01:01:08.034151+0000","last_clean":"2023-09-10T01:01:08.034151+0000","last_became_active":"2023-09-10T01:00:43.006912+0000","last_became_peered":"2023-09-10T01:00:43.006912+0000","last_unstale":"2023-09-10T01:01:08.034151+0000","last_undegraded":"2023-09-10T01:01:08.034151+0000","last_fullsized":"2023-09-10T01:01:08.034151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:48:22.783656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957014+0000","last_change":"2023-09-10T01:00:44.032747+0000","last_active":"2023-09-10T01:00:55.957014+0000","last_peered":"2023-09-10T01:00:55.957014+0000","last_clean":"2023-09-10T01:00:55.957014+0000","last_became_active":"2023-09-10T01:00:44.032416+0000","last_became_peered":"2023-09-10T01:00:44.032416+0000","last_unstale":"2023-09-10T01:00:55.957014+0000","last_undegraded":"2023-09-10T01:00:55.957014+0000","last_fullsized":"2023-09-10T01:00:55.957014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:59:04.713333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961279+0000","last_change":"2023-09-10T01:00:48.062520+0000","last_active":"2023-09-10T01:00:55.961279+0000","last_peered":"2023-09-10T01:00:55.961279+0000","last_clean":"2023-09-10T01:00:55.961279+0000","last_became_active":"2023-09-10T01:00:48.061870+0000","last_became_peered":"2023-09-10T01:00:48.061870+0000","last_unstale":"2023-09-10T01:00:55.961279+0000","last_undegraded":"2023-09-10T01:00:55.961279+0000","last_fullsized":"2023-09-10T01:00:55.961279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:24:38.798775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.030571+0000","last_change":"2023-09-10T01:00:43.006513+0000","last_active":"2023-09-10T01:01:08.030571+0000","last_peered":"2023-09-10T01:01:08.030571+0000","last_clean":"2023-09-10T01:01:08.030571+0000","last_became_active":"2023-09-10T01:00:43.006166+0000","last_became_peered":"2023-09-10T01:00:43.006166+0000","last_unstale":"2023-09-10T01:01:08.030571+0000","last_undegraded":"2023-09-10T01:01:08.030571+0000","last_fullsized":"2023-09-10T01:01:08.030571+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:52:32.167513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954478+0000","last_change":"2023-09-10T01:00:44.031264+0000","last_active":"2023-09-10T01:00:55.954478+0000","last_peered":"2023-09-10T01:00:55.954478+0000","last_clean":"2023-09-10T01:00:55.954478+0000","last_became_active":"2023-09-10T01:00:44.030993+0000","last_became_peered":"2023-09-10T01:00:44.030993+0000","last_unstale":"2023-09-10T01:00:55.954478+0000","last_undegraded":"2023-09-10T01:00:55.954478+0000","last_fullsized":"2023-09-10T01:00:55.954478+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:19:27.658594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958114+0000","last_change":"2023-09-10T01:00:48.056886+0000","last_active":"2023-09-10T01:00:55.958114+0000","last_peered":"2023-09-10T01:00:55.958114+0000","last_clean":"2023-09-10T01:00:55.958114+0000","last_became_active":"2023-09-10T01:00:48.056411+0000","last_became_peered":"2023-09-10T01:00:48.056411+0000","last_unstale":"2023-09-10T01:00:55.958114+0000","last_undegraded":"2023-09-10T01:00:55.958114+0000","last_fullsized":"2023-09-10T01:00:55.958114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:56:42.062702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'22","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:11.957270+0000","last_change":"2023-09-10T01:00:43.009113+0000","last_active":"2023-09-10T01:01:11.957270+0000","last_peered":"2023-09-10T01:01:11.957270+0000","last_clean":"2023-09-10T01:01:11.957270+0000","last_became_active":"2023-09-10T01:00:43.008757+0000","last_became_peered":"2023-09-10T01:00:43.008757+0000","last_unstale":"2023-09-10T01:01:11.957270+0000","last_undegraded":"2023-09-10T01:01:11.957270+0000","last_fullsized":"2023-09-10T01:01:11.957270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:29:24.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38563,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":42,"num_write":23,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954805+0000","last_change":"2023-09-10T01:00:44.027127+0000","last_active":"2023-09-10T01:00:55.954805+0000","last_peered":"2023-09-10T01:00:55.954805+0000","last_clean":"2023-09-10T01:00:55.954805+0000","last_became_active":"2023-09-10T01:00:44.026852+0000","last_became_peered":"2023-09-10T01:00:44.026852+0000","last_unstale":"2023-09-10T01:00:55.954805+0000","last_undegraded":"2023-09-10T01:00:55.954805+0000","last_fullsized":"2023-09-10T01:00:55.954805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:14:20.805300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.960892+0000","last_change":"2023-09-10T01:00:48.069337+0000","last_active":"2023-09-10T01:00:55.960892+0000","last_peered":"2023-09-10T01:00:55.960892+0000","last_clean":"2023-09-10T01:00:55.960892+0000","last_became_active":"2023-09-10T01:00:48.068893+0000","last_became_peered":"2023-09-10T01:00:48.068893+0000","last_unstale":"2023-09-10T01:00:55.960892+0000","last_undegraded":"2023-09-10T01:00:55.960892+0000","last_fullsized":"2023-09-10T01:00:55.960892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:24:40.074551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.038363+0000","last_change":"2023-09-10T01:00:43.010128+0000","last_active":"2023-09-10T01:01:09.038363+0000","last_peered":"2023-09-10T01:01:09.038363+0000","last_clean":"2023-09-10T01:01:09.038363+0000","last_became_active":"2023-09-10T01:00:43.009808+0000","last_became_peered":"2023-09-10T01:00:43.009808+0000","last_unstale":"2023-09-10T01:01:09.038363+0000","last_undegraded":"2023-09-10T01:01:09.038363+0000","last_fullsized":"2023-09-10T01:01:09.038363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:27.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962222+0000","last_change":"2023-09-10T01:00:44.038665+0000","last_active":"2023-09-10T01:00:55.962222+0000","last_peered":"2023-09-10T01:00:55.962222+0000","last_clean":"2023-09-10T01:00:55.962222+0000","last_became_active":"2023-09-10T01:00:44.037944+0000","last_became_peered":"2023-09-10T01:00:44.037944+0000","last_unstale":"2023-09-10T01:00:55.962222+0000","last_undegraded":"2023-09-10T01:00:55.962222+0000","last_fullsized":"2023-09-10T01:00:55.962222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:11:34.018249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.965594+0000","last_change":"2023-09-10T01:00:56.965594+0000","last_active":"2023-09-10T01:00:56.965594+0000","last_peered":"2023-09-10T01:00:56.965594+0000","last_clean":"2023-09-10T01:00:56.965594+0000","last_became_active":"2023-09-10T01:00:56.965006+0000","last_became_peered":"2023-09-10T01:00:56.965006+0000","last_unstale":"2023-09-10T01:00:56.965594+0000","last_undegraded":"2023-09-10T01:00:56.965594+0000","last_fullsized":"2023-09-10T01:00:56.965594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:13:16.946697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.035422+0000","last_change":"2023-09-10T01:00:43.004855+0000","last_active":"2023-09-10T01:01:08.035422+0000","last_peered":"2023-09-10T01:01:08.035422+0000","last_clean":"2023-09-10T01:01:08.035422+0000","last_became_active":"2023-09-10T01:00:43.004464+0000","last_became_peered":"2023-09-10T01:00:43.004464+0000","last_unstale":"2023-09-10T01:01:08.035422+0000","last_undegraded":"2023-09-10T01:01:08.035422+0000","last_fullsized":"2023-09-10T01:01:08.035422+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:29.517290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957252+0000","last_change":"2023-09-10T01:00:44.037273+0000","last_active":"2023-09-10T01:00:55.957252+0000","last_peered":"2023-09-10T01:00:55.957252+0000","last_clean":"2023-09-10T01:00:55.957252+0000","last_became_active":"2023-09-10T01:00:44.036965+0000","last_became_peered":"2023-09-10T01:00:44.036965+0000","last_unstale":"2023-09-10T01:00:55.957252+0000","last_undegraded":"2023-09-10T01:00:55.957252+0000","last_fullsized":"2023-09-10T01:00:55.957252+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:47:20.149002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958612+0000","last_change":"2023-09-10T01:00:48.058574+0000","last_active":"2023-09-10T01:00:55.958612+0000","last_peered":"2023-09-10T01:00:55.958612+0000","last_clean":"2023-09-10T01:00:55.958612+0000","last_became_active":"2023-09-10T01:00:48.058190+0000","last_became_peered":"2023-09-10T01:00:48.058190+0000","last_unstale":"2023-09-10T01:00:55.958612+0000","last_undegraded":"2023-09-10T01:00:55.958612+0000","last_fullsized":"2023-09-10T01:00:55.958612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:46:42.177996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:10.954891+0000","last_change":"2023-09-10T01:00:43.003816+0000","last_active":"2023-09-10T01:01:10.954891+0000","last_peered":"2023-09-10T01:01:10.954891+0000","last_clean":"2023-09-10T01:01:10.954891+0000","last_became_active":"2023-09-10T01:00:43.003404+0000","last_became_peered":"2023-09-10T01:00:43.003404+0000","last_unstale":"2023-09-10T01:01:10.954891+0000","last_undegraded":"2023-09-10T01:01:10.954891+0000","last_fullsized":"2023-09-10T01:01:10.954891+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:55.208302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957538+0000","last_change":"2023-09-10T01:00:44.034607+0000","last_active":"2023-09-10T01:00:55.957538+0000","last_peered":"2023-09-10T01:00:55.957538+0000","last_clean":"2023-09-10T01:00:55.957538+0000","last_became_active":"2023-09-10T01:00:44.034244+0000","last_became_peered":"2023-09-10T01:00:44.034244+0000","last_unstale":"2023-09-10T01:00:55.957538+0000","last_undegraded":"2023-09-10T01:00:55.957538+0000","last_fullsized":"2023-09-10T01:00:55.957538+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:06:13.214160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962896+0000","last_change":"2023-09-10T01:00:48.058789+0000","last_active":"2023-09-10T01:00:55.962896+0000","last_peered":"2023-09-10T01:00:55.962896+0000","last_clean":"2023-09-10T01:00:55.962896+0000","last_became_active":"2023-09-10T01:00:48.058220+0000","last_became_peered":"2023-09-10T01:00:48.058220+0000","last_unstale":"2023-09-10T01:00:55.962896+0000","last_undegraded":"2023-09-10T01:00:55.962896+0000","last_fullsized":"2023-09-10T01:00:55.962896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:01:12.291739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.040113+0000","last_change":"2023-09-10T01:00:43.006109+0000","last_active":"2023-09-10T01:01:09.040113+0000","last_peered":"2023-09-10T01:01:09.040113+0000","last_clean":"2023-09-10T01:01:09.040113+0000","last_became_active":"2023-09-10T01:00:43.005753+0000","last_became_peered":"2023-09-10T01:00:43.005753+0000","last_unstale":"2023-09-10T01:01:09.040113+0000","last_undegraded":"2023-09-10T01:01:09.040113+0000","last_fullsized":"2023-09-10T01:01:09.040113+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:45:47.083991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.963038+0000","last_change":"2023-09-10T01:00:44.028576+0000","last_active":"2023-09-10T01:00:55.963038+0000","last_peered":"2023-09-10T01:00:55.963038+0000","last_clean":"2023-09-10T01:00:55.963038+0000","last_became_active":"2023-09-10T01:00:44.028305+0000","last_became_peered":"2023-09-10T01:00:44.028305+0000","last_unstale":"2023-09-10T01:00:55.963038+0000","last_undegraded":"2023-09-10T01:00:55.963038+0000","last_fullsized":"2023-09-10T01:00:55.963038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:59:12.866257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.963016+0000","last_change":"2023-09-10T01:00:48.069318+0000","last_active":"2023-09-10T01:00:55.963016+0000","last_peered":"2023-09-10T01:00:55.963016+0000","last_clean":"2023-09-10T01:00:55.963016+0000","last_became_active":"2023-09-10T01:00:48.068876+0000","last_became_peered":"2023-09-10T01:00:48.068876+0000","last_unstale":"2023-09-10T01:00:55.963016+0000","last_undegraded":"2023-09-10T01:00:55.963016+0000","last_fullsized":"2023-09-10T01:00:55.963016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:06:22.366530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954808+0000","last_change":"2023-09-10T01:00:43.004164+0000","last_active":"2023-09-10T01:00:55.954808+0000","last_peered":"2023-09-10T01:00:55.954808+0000","last_clean":"2023-09-10T01:00:55.954808+0000","last_became_active":"2023-09-10T01:00:43.003781+0000","last_became_peered":"2023-09-10T01:00:43.003781+0000","last_unstale":"2023-09-10T01:00:55.954808+0000","last_undegraded":"2023-09-10T01:00:55.954808+0000","last_fullsized":"2023-09-10T01:00:55.954808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:38:18.559712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961744+0000","last_change":"2023-09-10T01:00:44.028206+0000","last_active":"2023-09-10T01:00:55.961744+0000","last_peered":"2023-09-10T01:00:55.961744+0000","last_clean":"2023-09-10T01:00:55.961744+0000","last_became_active":"2023-09-10T01:00:44.027914+0000","last_became_peered":"2023-09-10T01:00:44.027914+0000","last_unstale":"2023-09-10T01:00:55.961744+0000","last_undegraded":"2023-09-10T01:00:55.961744+0000","last_fullsized":"2023-09-10T01:00:55.961744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:41:12.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954406+0000","last_change":"2023-09-10T01:00:48.428613+0000","last_active":"2023-09-10T01:00:55.954406+0000","last_peered":"2023-09-10T01:00:55.954406+0000","last_clean":"2023-09-10T01:00:55.954406+0000","last_became_active":"2023-09-10T01:00:48.428059+0000","last_became_peered":"2023-09-10T01:00:48.428059+0000","last_unstale":"2023-09-10T01:00:55.954406+0000","last_undegraded":"2023-09-10T01:00:55.954406+0000","last_fullsized":"2023-09-10T01:00:55.954406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:46:03.683279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958500+0000","last_change":"2023-09-10T01:00:43.006061+0000","last_active":"2023-09-10T01:00:55.958500+0000","last_peered":"2023-09-10T01:00:55.958500+0000","last_clean":"2023-09-10T01:00:55.958500+0000","last_became_active":"2023-09-10T01:00:43.005392+0000","last_became_peered":"2023-09-10T01:00:43.005392+0000","last_unstale":"2023-09-10T01:00:55.958500+0000","last_undegraded":"2023-09-10T01:00:55.958500+0000","last_fullsized":"2023-09-10T01:00:55.958500+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:01:45.062338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959373+0000","last_change":"2023-09-10T01:00:44.026684+0000","last_active":"2023-09-10T01:00:55.959373+0000","last_peered":"2023-09-10T01:00:55.959373+0000","last_clean":"2023-09-10T01:00:55.959373+0000","last_became_active":"2023-09-10T01:00:44.026387+0000","last_became_peered":"2023-09-10T01:00:44.026387+0000","last_unstale":"2023-09-10T01:00:55.959373+0000","last_undegraded":"2023-09-10T01:00:55.959373+0000","last_fullsized":"2023-09-10T01:00:55.959373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:23:03.662543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962732+0000","last_change":"2023-09-10T01:00:48.071589+0000","last_active":"2023-09-10T01:00:55.962732+0000","last_peered":"2023-09-10T01:00:55.962732+0000","last_clean":"2023-09-10T01:00:55.962732+0000","last_became_active":"2023-09-10T01:00:48.071118+0000","last_became_peered":"2023-09-10T01:00:48.071118+0000","last_unstale":"2023-09-10T01:00:55.962732+0000","last_undegraded":"2023-09-10T01:00:55.962732+0000","last_fullsized":"2023-09-10T01:00:55.962732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:27.194868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.962388+0000","last_change":"2023-09-10T01:00:43.009738+0000","last_active":"2023-09-10T01:00:56.962388+0000","last_peered":"2023-09-10T01:00:56.962388+0000","last_clean":"2023-09-10T01:00:56.962388+0000","last_became_active":"2023-09-10T01:00:43.009376+0000","last_became_peered":"2023-09-10T01:00:43.009376+0000","last_unstale":"2023-09-10T01:00:56.962388+0000","last_undegraded":"2023-09-10T01:00:56.962388+0000","last_fullsized":"2023-09-10T01:00:56.962388+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:08.761672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959197+0000","last_change":"2023-09-10T01:00:44.030175+0000","last_active":"2023-09-10T01:00:55.959197+0000","last_peered":"2023-09-10T01:00:55.959197+0000","last_clean":"2023-09-10T01:00:55.959197+0000","last_became_active":"2023-09-10T01:00:44.029748+0000","last_became_peered":"2023-09-10T01:00:44.029748+0000","last_unstale":"2023-09-10T01:00:55.959197+0000","last_undegraded":"2023-09-10T01:00:55.959197+0000","last_fullsized":"2023-09-10T01:00:55.959197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:51:15.149610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.963010+0000","last_change":"2023-09-10T01:00:48.428008+0000","last_active":"2023-09-10T01:00:55.963010+0000","last_peered":"2023-09-10T01:00:55.963010+0000","last_clean":"2023-09-10T01:00:55.963010+0000","last_became_active":"2023-09-10T01:00:48.427498+0000","last_became_peered":"2023-09-10T01:00:48.427498+0000","last_unstale":"2023-09-10T01:00:55.963010+0000","last_undegraded":"2023-09-10T01:00:55.963010+0000","last_fullsized":"2023-09-10T01:00:55.963010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:16:48.938006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.051488+0000","last_change":"2023-09-10T01:00:57.051488+0000","last_active":"2023-09-10T01:00:57.051488+0000","last_peered":"2023-09-10T01:00:57.051488+0000","last_clean":"2023-09-10T01:00:57.051488+0000","last_became_active":"2023-09-10T01:00:57.051060+0000","last_became_peered":"2023-09-10T01:00:57.051060+0000","last_unstale":"2023-09-10T01:00:57.051488+0000","last_undegraded":"2023-09-10T01:00:57.051488+0000","last_fullsized":"2023-09-10T01:00:57.051488+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:29:15.803867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.950760+0000","last_change":"2023-09-10T01:00:43.001047+0000","last_active":"2023-09-10T01:01:09.950760+0000","last_peered":"2023-09-10T01:01:09.950760+0000","last_clean":"2023-09-10T01:01:09.950760+0000","last_became_active":"2023-09-10T01:00:43.000521+0000","last_became_peered":"2023-09-10T01:00:43.000521+0000","last_unstale":"2023-09-10T01:01:09.950760+0000","last_undegraded":"2023-09-10T01:01:09.950760+0000","last_fullsized":"2023-09-10T01:01:09.950760+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:15:34.509964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.964282+0000","last_change":"2023-09-10T01:00:48.062148+0000","last_active":"2023-09-10T01:00:56.964282+0000","last_peered":"2023-09-10T01:00:56.964282+0000","last_clean":"2023-09-10T01:00:56.964282+0000","last_became_active":"2023-09-10T01:00:48.061746+0000","last_became_peered":"2023-09-10T01:00:48.061746+0000","last_unstale":"2023-09-10T01:00:56.964282+0000","last_undegraded":"2023-09-10T01:00:56.964282+0000","last_fullsized":"2023-09-10T01:00:56.964282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:40.087012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954815+0000","last_change":"2023-09-10T01:00:44.030500+0000","last_active":"2023-09-10T01:00:55.954815+0000","last_peered":"2023-09-10T01:00:55.954815+0000","last_clean":"2023-09-10T01:00:55.954815+0000","last_became_active":"2023-09-10T01:00:44.030209+0000","last_became_peered":"2023-09-10T01:00:44.030209+0000","last_unstale":"2023-09-10T01:00:55.954815+0000","last_undegraded":"2023-09-10T01:00:55.954815+0000","last_fullsized":"2023-09-10T01:00:55.954815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:53:30.421493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.948393+0000","last_change":"2023-09-10T01:00:43.001102+0000","last_active":"2023-09-10T01:01:09.948393+0000","last_peered":"2023-09-10T01:01:09.948393+0000","last_clean":"2023-09-10T01:01:09.948393+0000","last_became_active":"2023-09-10T01:00:43.000715+0000","last_became_peered":"2023-09-10T01:00:43.000715+0000","last_unstale":"2023-09-10T01:01:09.948393+0000","last_undegraded":"2023-09-10T01:01:09.948393+0000","last_fullsized":"2023-09-10T01:01:09.948393+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:43:37.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":324,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957221+0000","last_change":"2023-09-10T00:59:40.261244+0000","last_active":"2023-09-10T01:00:55.957221+0000","last_peered":"2023-09-10T01:00:55.957221+0000","last_clean":"2023-09-10T01:00:55.957221+0000","last_became_active":"2023-09-10T00:59:40.260673+0000","last_became_peered":"2023-09-10T00:59:40.260673+0000","last_unstale":"2023-09-10T01:00:55.957221+0000","last_undegraded":"2023-09-10T01:00:55.957221+0000","last_fullsized":"2023-09-10T01:00:55.957221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_clean_scrub_stamp":"2023-09-10T00:56:36.374323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:04:03.614104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:57.052019+0000","last_change":"2023-09-10T01:00:57.052019+0000","last_active":"2023-09-10T01:00:57.052019+0000","last_peered":"2023-09-10T01:00:57.052019+0000","last_clean":"2023-09-10T01:00:57.052019+0000","last_became_active":"2023-09-10T01:00:57.051527+0000","last_became_peered":"2023-09-10T01:00:57.051527+0000","last_unstale":"2023-09-10T01:00:57.052019+0000","last_undegraded":"2023-09-10T01:00:57.052019+0000","last_fullsized":"2023-09-10T01:00:57.052019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:10:43.756761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961967+0000","last_change":"2023-09-10T01:00:44.033214+0000","last_active":"2023-09-10T01:00:55.961967+0000","last_peered":"2023-09-10T01:00:55.961967+0000","last_clean":"2023-09-10T01:00:55.961967+0000","last_became_active":"2023-09-10T01:00:44.032828+0000","last_became_peered":"2023-09-10T01:00:44.032828+0000","last_unstale":"2023-09-10T01:00:55.961967+0000","last_undegraded":"2023-09-10T01:00:55.961967+0000","last_fullsized":"2023-09-10T01:00:55.961967+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:01:22.417216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.038169+0000","last_change":"2023-09-10T01:00:56.965456+0000","last_active":"2023-09-10T01:01:09.038169+0000","last_peered":"2023-09-10T01:01:09.038169+0000","last_clean":"2023-09-10T01:01:09.038169+0000","last_became_active":"2023-09-10T01:00:56.965021+0000","last_became_peered":"2023-09-10T01:00:56.965021+0000","last_unstale":"2023-09-10T01:01:09.038169+0000","last_undegraded":"2023-09-10T01:01:09.038169+0000","last_fullsized":"2023-09-10T01:01:09.038169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:19.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957045+0000","last_change":"2023-09-10T01:00:48.066811+0000","last_active":"2023-09-10T01:00:55.957045+0000","last_peered":"2023-09-10T01:00:55.957045+0000","last_clean":"2023-09-10T01:00:55.957045+0000","last_became_active":"2023-09-10T01:00:48.066421+0000","last_became_peered":"2023-09-10T01:00:48.066421+0000","last_unstale":"2023-09-10T01:00:55.957045+0000","last_undegraded":"2023-09-10T01:00:55.957045+0000","last_fullsized":"2023-09-10T01:00:55.957045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:48:01.459070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.958768+0000","last_change":"2023-09-10T01:00:43.011134+0000","last_active":"2023-09-10T01:00:56.958768+0000","last_peered":"2023-09-10T01:00:56.958768+0000","last_clean":"2023-09-10T01:00:56.958768+0000","last_became_active":"2023-09-10T01:00:43.010871+0000","last_became_peered":"2023-09-10T01:00:43.010871+0000","last_unstale":"2023-09-10T01:00:56.958768+0000","last_undegraded":"2023-09-10T01:00:56.958768+0000","last_fullsized":"2023-09-10T01:00:56.958768+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:56.903745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962891+0000","last_change":"2023-09-10T01:00:44.031104+0000","last_active":"2023-09-10T01:00:55.962891+0000","last_peered":"2023-09-10T01:00:55.962891+0000","last_clean":"2023-09-10T01:00:55.962891+0000","last_became_active":"2023-09-10T01:00:44.030745+0000","last_became_peered":"2023-09-10T01:00:44.030745+0000","last_unstale":"2023-09-10T01:00:55.962891+0000","last_undegraded":"2023-09-10T01:00:55.962891+0000","last_fullsized":"2023-09-10T01:00:55.962891+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:53.885886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954861+0000","last_change":"2023-09-10T01:00:48.427735+0000","last_active":"2023-09-10T01:00:55.954861+0000","last_peered":"2023-09-10T01:00:55.954861+0000","last_clean":"2023-09-10T01:00:55.954861+0000","last_became_active":"2023-09-10T01:00:48.427342+0000","last_became_peered":"2023-09-10T01:00:48.427342+0000","last_unstale":"2023-09-10T01:00:55.954861+0000","last_undegraded":"2023-09-10T01:00:55.954861+0000","last_fullsized":"2023-09-10T01:00:55.954861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:01:52.007325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.039077+0000","last_change":"2023-09-10T01:00:43.006314+0000","last_active":"2023-09-10T01:01:09.039077+0000","last_peered":"2023-09-10T01:01:09.039077+0000","last_clean":"2023-09-10T01:01:09.039077+0000","last_became_active":"2023-09-10T01:00:43.004835+0000","last_became_peered":"2023-09-10T01:00:43.004835+0000","last_unstale":"2023-09-10T01:01:09.039077+0000","last_undegraded":"2023-09-10T01:01:09.039077+0000","last_fullsized":"2023-09-10T01:01:09.039077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:37:09.359846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955280+0000","last_change":"2023-09-10T01:00:44.029839+0000","last_active":"2023-09-10T01:00:55.955280+0000","last_peered":"2023-09-10T01:00:55.955280+0000","last_clean":"2023-09-10T01:00:55.955280+0000","last_became_active":"2023-09-10T01:00:44.029492+0000","last_became_peered":"2023-09-10T01:00:44.029492+0000","last_unstale":"2023-09-10T01:00:55.955280+0000","last_undegraded":"2023-09-10T01:00:55.955280+0000","last_fullsized":"2023-09-10T01:00:55.955280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:14.357386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956421+0000","last_change":"2023-09-10T01:00:48.054536+0000","last_active":"2023-09-10T01:00:55.956421+0000","last_peered":"2023-09-10T01:00:55.956421+0000","last_clean":"2023-09-10T01:00:55.956421+0000","last_became_active":"2023-09-10T01:00:48.053938+0000","last_became_peered":"2023-09-10T01:00:48.053938+0000","last_unstale":"2023-09-10T01:00:55.956421+0000","last_undegraded":"2023-09-10T01:00:55.956421+0000","last_fullsized":"2023-09-10T01:00:55.956421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:45:12.703592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.035987+0000","last_change":"2023-09-10T01:00:43.002313+0000","last_active":"2023-09-10T01:01:08.035987+0000","last_peered":"2023-09-10T01:01:08.035987+0000","last_clean":"2023-09-10T01:01:08.035987+0000","last_became_active":"2023-09-10T01:00:43.001839+0000","last_became_peered":"2023-09-10T01:00:43.001839+0000","last_unstale":"2023-09-10T01:01:08.035987+0000","last_undegraded":"2023-09-10T01:01:08.035987+0000","last_fullsized":"2023-09-10T01:01:08.035987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:51.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957427+0000","last_change":"2023-09-10T01:00:44.027525+0000","last_active":"2023-09-10T01:00:55.957427+0000","last_peered":"2023-09-10T01:00:55.957427+0000","last_clean":"2023-09-10T01:00:55.957427+0000","last_became_active":"2023-09-10T01:00:44.027132+0000","last_became_peered":"2023-09-10T01:00:44.027132+0000","last_unstale":"2023-09-10T01:00:55.957427+0000","last_undegraded":"2023-09-10T01:00:55.957427+0000","last_fullsized":"2023-09-10T01:00:55.957427+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:44.197242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961729+0000","last_change":"2023-09-10T01:00:48.428406+0000","last_active":"2023-09-10T01:00:55.961729+0000","last_peered":"2023-09-10T01:00:55.961729+0000","last_clean":"2023-09-10T01:00:55.961729+0000","last_became_active":"2023-09-10T01:00:48.427938+0000","last_became_peered":"2023-09-10T01:00:48.427938+0000","last_unstale":"2023-09-10T01:00:55.961729+0000","last_undegraded":"2023-09-10T01:00:55.961729+0000","last_fullsized":"2023-09-10T01:00:55.961729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:52.316235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.033706+0000","last_change":"2023-09-10T01:00:56.963755+0000","last_active":"2023-09-10T01:01:08.033706+0000","last_peered":"2023-09-10T01:01:08.033706+0000","last_clean":"2023-09-10T01:01:08.033706+0000","last_became_active":"2023-09-10T01:00:56.963283+0000","last_became_peered":"2023-09-10T01:00:56.963283+0000","last_unstale":"2023-09-10T01:01:08.033706+0000","last_undegraded":"2023-09-10T01:01:08.033706+0000","last_fullsized":"2023-09-10T01:01:08.033706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:13:32.213393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954232+0000","last_change":"2023-09-10T01:00:44.025284+0000","last_active":"2023-09-10T01:00:55.954232+0000","last_peered":"2023-09-10T01:00:55.954232+0000","last_clean":"2023-09-10T01:00:55.954232+0000","last_became_active":"2023-09-10T01:00:44.024965+0000","last_became_peered":"2023-09-10T01:00:44.024965+0000","last_unstale":"2023-09-10T01:00:55.954232+0000","last_undegraded":"2023-09-10T01:00:55.954232+0000","last_fullsized":"2023-09-10T01:00:55.954232+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:19:08.620325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958759+0000","last_change":"2023-09-10T01:00:48.070843+0000","last_active":"2023-09-10T01:00:55.958759+0000","last_peered":"2023-09-10T01:00:55.958759+0000","last_clean":"2023-09-10T01:00:55.958759+0000","last_became_active":"2023-09-10T01:00:48.070350+0000","last_became_peered":"2023-09-10T01:00:48.070350+0000","last_unstale":"2023-09-10T01:00:55.958759+0000","last_undegraded":"2023-09-10T01:00:55.958759+0000","last_fullsized":"2023-09-10T01:00:55.958759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:35:25.196036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.035280+0000","last_change":"2023-09-10T01:00:43.005398+0000","last_active":"2023-09-10T01:01:08.035280+0000","last_peered":"2023-09-10T01:01:08.035280+0000","last_clean":"2023-09-10T01:01:08.035280+0000","last_became_active":"2023-09-10T01:00:43.005087+0000","last_became_peered":"2023-09-10T01:00:43.005087+0000","last_unstale":"2023-09-10T01:01:08.035280+0000","last_undegraded":"2023-09-10T01:01:08.035280+0000","last_fullsized":"2023-09-10T01:01:08.035280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:29:04.042683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958684+0000","last_change":"2023-09-10T01:00:44.024451+0000","last_active":"2023-09-10T01:00:55.958684+0000","last_peered":"2023-09-10T01:00:55.958684+0000","last_clean":"2023-09-10T01:00:55.958684+0000","last_became_active":"2023-09-10T01:00:44.024045+0000","last_became_peered":"2023-09-10T01:00:44.024045+0000","last_unstale":"2023-09-10T01:00:55.958684+0000","last_undegraded":"2023-09-10T01:00:55.958684+0000","last_fullsized":"2023-09-10T01:00:55.958684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:55:50.750994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.955577+0000","last_change":"2023-09-10T01:00:48.062247+0000","last_active":"2023-09-10T01:00:56.955577+0000","last_peered":"2023-09-10T01:00:56.955577+0000","last_clean":"2023-09-10T01:00:56.955577+0000","last_became_active":"2023-09-10T01:00:48.061812+0000","last_became_peered":"2023-09-10T01:00:48.061812+0000","last_unstale":"2023-09-10T01:00:56.955577+0000","last_undegraded":"2023-09-10T01:00:56.955577+0000","last_fullsized":"2023-09-10T01:00:56.955577+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:44:13.596534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956724+0000","last_change":"2023-09-10T01:00:43.010845+0000","last_active":"2023-09-10T01:00:55.956724+0000","last_peered":"2023-09-10T01:00:55.956724+0000","last_clean":"2023-09-10T01:00:55.956724+0000","last_became_active":"2023-09-10T01:00:43.010511+0000","last_became_peered":"2023-09-10T01:00:43.010511+0000","last_unstale":"2023-09-10T01:00:55.956724+0000","last_undegraded":"2023-09-10T01:00:55.956724+0000","last_fullsized":"2023-09-10T01:00:55.956724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:33:07.229023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955056+0000","last_change":"2023-09-10T01:00:44.029025+0000","last_active":"2023-09-10T01:00:55.955056+0000","last_peered":"2023-09-10T01:00:55.955056+0000","last_clean":"2023-09-10T01:00:55.955056+0000","last_became_active":"2023-09-10T01:00:44.028713+0000","last_became_peered":"2023-09-10T01:00:44.028713+0000","last_unstale":"2023-09-10T01:00:55.955056+0000","last_undegraded":"2023-09-10T01:00:55.955056+0000","last_fullsized":"2023-09-10T01:00:55.955056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:40:22.315742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.961553+0000","last_change":"2023-09-10T01:00:48.427294+0000","last_active":"2023-09-10T01:00:55.961553+0000","last_peered":"2023-09-10T01:00:55.961553+0000","last_clean":"2023-09-10T01:00:55.961553+0000","last_became_active":"2023-09-10T01:00:48.426882+0000","last_became_peered":"2023-09-10T01:00:48.426882+0000","last_unstale":"2023-09-10T01:00:55.961553+0000","last_undegraded":"2023-09-10T01:00:55.961553+0000","last_fullsized":"2023-09-10T01:00:55.961553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:54:38.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:08.031610+0000","last_change":"2023-09-10T01:00:43.006175+0000","last_active":"2023-09-10T01:01:08.031610+0000","last_peered":"2023-09-10T01:01:08.031610+0000","last_clean":"2023-09-10T01:01:08.031610+0000","last_became_active":"2023-09-10T01:00:43.005786+0000","last_became_peered":"2023-09-10T01:00:43.005786+0000","last_unstale":"2023-09-10T01:01:08.031610+0000","last_undegraded":"2023-09-10T01:01:08.031610+0000","last_fullsized":"2023-09-10T01:01:08.031610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:14:47.074866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956801+0000","last_change":"2023-09-10T01:00:44.041303+0000","last_active":"2023-09-10T01:00:55.956801+0000","last_peered":"2023-09-10T01:00:55.956801+0000","last_clean":"2023-09-10T01:00:55.956801+0000","last_became_active":"2023-09-10T01:00:44.041009+0000","last_became_peered":"2023-09-10T01:00:44.041009+0000","last_unstale":"2023-09-10T01:00:55.956801+0000","last_undegraded":"2023-09-10T01:00:55.956801+0000","last_fullsized":"2023-09-10T01:00:55.956801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:53:23.001700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954255+0000","last_change":"2023-09-10T01:00:48.064314+0000","last_active":"2023-09-10T01:00:55.954255+0000","last_peered":"2023-09-10T01:00:55.954255+0000","last_clean":"2023-09-10T01:00:55.954255+0000","last_became_active":"2023-09-10T01:00:48.063813+0000","last_became_peered":"2023-09-10T01:00:48.063813+0000","last_unstale":"2023-09-10T01:00:55.954255+0000","last_undegraded":"2023-09-10T01:00:55.954255+0000","last_fullsized":"2023-09-10T01:00:55.954255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:00:10.002006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:01:09.039829+0000","last_change":"2023-09-10T01:00:43.007579+0000","last_active":"2023-09-10T01:01:09.039829+0000","last_peered":"2023-09-10T01:01:09.039829+0000","last_clean":"2023-09-10T01:01:09.039829+0000","last_became_active":"2023-09-10T01:00:43.007200+0000","last_became_peered":"2023-09-10T01:00:43.007200+0000","last_unstale":"2023-09-10T01:01:09.039829+0000","last_undegraded":"2023-09-10T01:01:09.039829+0000","last_fullsized":"2023-09-10T01:01:09.039829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:20:08.688531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957330+0000","last_change":"2023-09-10T01:00:44.033511+0000","last_active":"2023-09-10T01:00:55.957330+0000","last_peered":"2023-09-10T01:00:55.957330+0000","last_clean":"2023-09-10T01:00:55.957330+0000","last_became_active":"2023-09-10T01:00:44.033170+0000","last_became_peered":"2023-09-10T01:00:44.033170+0000","last_unstale":"2023-09-10T01:00:55.957330+0000","last_undegraded":"2023-09-10T01:00:55.957330+0000","last_fullsized":"2023-09-10T01:00:55.957330+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:36:12.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957931+0000","last_change":"2023-09-10T01:00:48.067436+0000","last_active":"2023-09-10T01:00:55.957931+0000","last_peered":"2023-09-10T01:00:55.957931+0000","last_clean":"2023-09-10T01:00:55.957931+0000","last_became_active":"2023-09-10T01:00:48.067038+0000","last_became_peered":"2023-09-10T01:00:48.067038+0000","last_unstale":"2023-09-10T01:00:55.957931+0000","last_undegraded":"2023-09-10T01:00:55.957931+0000","last_fullsized":"2023-09-10T01:00:55.957931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:26:41.210698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.960037+0000","last_change":"2023-09-10T01:00:43.006934+0000","last_active":"2023-09-10T01:00:56.960037+0000","last_peered":"2023-09-10T01:00:56.960037+0000","last_clean":"2023-09-10T01:00:56.960037+0000","last_became_active":"2023-09-10T01:00:43.005787+0000","last_became_peered":"2023-09-10T01:00:43.005787+0000","last_unstale":"2023-09-10T01:00:56.960037+0000","last_undegraded":"2023-09-10T01:00:56.960037+0000","last_fullsized":"2023-09-10T01:00:56.960037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:48:27.403220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956948+0000","last_change":"2023-09-10T01:00:44.033812+0000","last_active":"2023-09-10T01:00:55.956948+0000","last_peered":"2023-09-10T01:00:55.956948+0000","last_clean":"2023-09-10T01:00:55.956948+0000","last_became_active":"2023-09-10T01:00:44.033496+0000","last_became_peered":"2023-09-10T01:00:44.033496+0000","last_unstale":"2023-09-10T01:00:55.956948+0000","last_undegraded":"2023-09-10T01:00:55.956948+0000","last_fullsized":"2023-09-10T01:00:55.956948+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:12.679368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957698+0000","last_change":"2023-09-10T01:00:48.060358+0000","last_active":"2023-09-10T01:00:55.957698+0000","last_peered":"2023-09-10T01:00:55.957698+0000","last_clean":"2023-09-10T01:00:55.957698+0000","last_became_active":"2023-09-10T01:00:48.059853+0000","last_became_peered":"2023-09-10T01:00:48.059853+0000","last_unstale":"2023-09-10T01:00:55.957698+0000","last_undegraded":"2023-09-10T01:00:55.957698+0000","last_fullsized":"2023-09-10T01:00:55.957698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:31:56.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962921+0000","last_change":"2023-09-10T01:00:43.002820+0000","last_active":"2023-09-10T01:00:55.962921+0000","last_peered":"2023-09-10T01:00:55.962921+0000","last_clean":"2023-09-10T01:00:55.962921+0000","last_became_active":"2023-09-10T01:00:43.002456+0000","last_became_peered":"2023-09-10T01:00:43.002456+0000","last_unstale":"2023-09-10T01:00:55.962921+0000","last_undegraded":"2023-09-10T01:00:55.962921+0000","last_fullsized":"2023-09-10T01:00:55.962921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:40:10.189942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957369+0000","last_change":"2023-09-10T01:00:44.032655+0000","last_active":"2023-09-10T01:00:55.957369+0000","last_peered":"2023-09-10T01:00:55.957369+0000","last_clean":"2023-09-10T01:00:55.957369+0000","last_became_active":"2023-09-10T01:00:44.032157+0000","last_became_peered":"2023-09-10T01:00:44.032157+0000","last_unstale":"2023-09-10T01:00:55.957369+0000","last_undegraded":"2023-09-10T01:00:55.957369+0000","last_fullsized":"2023-09-10T01:00:55.957369+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:31.176768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958791+0000","last_change":"2023-09-10T01:00:44.022285+0000","last_active":"2023-09-10T01:00:55.958791+0000","last_peered":"2023-09-10T01:00:55.958791+0000","last_clean":"2023-09-10T01:00:55.958791+0000","last_became_active":"2023-09-10T01:00:44.021959+0000","last_became_peered":"2023-09-10T01:00:44.021959+0000","last_unstale":"2023-09-10T01:00:55.958791+0000","last_undegraded":"2023-09-10T01:00:55.958791+0000","last_fullsized":"2023-09-10T01:00:55.958791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:13:07.072249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957475+0000","last_change":"2023-09-10T01:00:44.029722+0000","last_active":"2023-09-10T01:00:55.957475+0000","last_peered":"2023-09-10T01:00:55.957475+0000","last_clean":"2023-09-10T01:00:55.957475+0000","last_became_active":"2023-09-10T01:00:44.029424+0000","last_became_peered":"2023-09-10T01:00:44.029424+0000","last_unstale":"2023-09-10T01:00:55.957475+0000","last_undegraded":"2023-09-10T01:00:55.957475+0000","last_fullsized":"2023-09-10T01:00:55.957475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:24:16.655538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.956754+0000","last_change":"2023-09-10T01:00:44.026026+0000","last_active":"2023-09-10T01:00:55.956754+0000","last_peered":"2023-09-10T01:00:55.956754+0000","last_clean":"2023-09-10T01:00:55.956754+0000","last_became_active":"2023-09-10T01:00:44.025727+0000","last_became_peered":"2023-09-10T01:00:44.025727+0000","last_unstale":"2023-09-10T01:00:55.956754+0000","last_undegraded":"2023-09-10T01:00:55.956754+0000","last_fullsized":"2023-09-10T01:00:55.956754+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:59.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-09-10T01:00:56.965355+0000","last_change":"2023-09-10T01:00:56.965355+0000","last_active":"2023-09-10T01:00:56.965355+0000","last_peered":"2023-09-10T01:00:56.965355+0000","last_clean":"2023-09-10T01:00:56.965355+0000","last_became_active":"2023-09-10T01:00:56.964903+0000","last_became_peered":"2023-09-10T01:00:56.964903+0000","last_unstale":"2023-09-10T01:00:56.965355+0000","last_undegraded":"2023-09-10T01:00:56.965355+0000","last_fullsized":"2023-09-10T01:00:56.965355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:49:21.940159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962913+0000","last_change":"2023-09-10T01:00:44.029525+0000","last_active":"2023-09-10T01:00:55.962913+0000","last_peered":"2023-09-10T01:00:55.962913+0000","last_clean":"2023-09-10T01:00:55.962913+0000","last_became_active":"2023-09-10T01:00:44.029139+0000","last_became_peered":"2023-09-10T01:00:44.029139+0000","last_unstale":"2023-09-10T01:00:55.962913+0000","last_undegraded":"2023-09-10T01:00:55.962913+0000","last_fullsized":"2023-09-10T01:00:55.962913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:20:24.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.959105+0000","last_change":"2023-09-10T01:00:44.022299+0000","last_active":"2023-09-10T01:00:55.959105+0000","last_peered":"2023-09-10T01:00:55.959105+0000","last_clean":"2023-09-10T01:00:55.959105+0000","last_became_active":"2023-09-10T01:00:44.021992+0000","last_became_peered":"2023-09-10T01:00:44.021992+0000","last_unstale":"2023-09-10T01:00:55.959105+0000","last_undegraded":"2023-09-10T01:00:55.959105+0000","last_fullsized":"2023-09-10T01:00:55.959105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:50:30.766442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957191+0000","last_change":"2023-09-10T01:00:44.027799+0000","last_active":"2023-09-10T01:00:55.957191+0000","last_peered":"2023-09-10T01:00:55.957191+0000","last_clean":"2023-09-10T01:00:55.957191+0000","last_became_active":"2023-09-10T01:00:44.027518+0000","last_became_peered":"2023-09-10T01:00:44.027518+0000","last_unstale":"2023-09-10T01:00:55.957191+0000","last_undegraded":"2023-09-10T01:00:55.957191+0000","last_fullsized":"2023-09-10T01:00:55.957191+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:00:20.315436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955430+0000","last_change":"2023-09-10T01:00:44.035921+0000","last_active":"2023-09-10T01:00:55.955430+0000","last_peered":"2023-09-10T01:00:55.955430+0000","last_clean":"2023-09-10T01:00:55.955430+0000","last_became_active":"2023-09-10T01:00:44.035634+0000","last_became_peered":"2023-09-10T01:00:44.035634+0000","last_unstale":"2023-09-10T01:00:55.955430+0000","last_undegraded":"2023-09-10T01:00:55.955430+0000","last_fullsized":"2023-09-10T01:00:55.955430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:44:29.069789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.957822+0000","last_change":"2023-09-10T01:00:44.025405+0000","last_active":"2023-09-10T01:00:55.957822+0000","last_peered":"2023-09-10T01:00:55.957822+0000","last_clean":"2023-09-10T01:00:55.957822+0000","last_became_active":"2023-09-10T01:00:44.025040+0000","last_became_peered":"2023-09-10T01:00:44.025040+0000","last_unstale":"2023-09-10T01:00:55.957822+0000","last_undegraded":"2023-09-10T01:00:55.957822+0000","last_fullsized":"2023-09-10T01:00:55.957822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:16:46.504902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.955771+0000","last_change":"2023-09-10T01:00:44.023603+0000","last_active":"2023-09-10T01:00:55.955771+0000","last_peered":"2023-09-10T01:00:55.955771+0000","last_clean":"2023-09-10T01:00:55.955771+0000","last_became_active":"2023-09-10T01:00:44.023279+0000","last_became_peered":"2023-09-10T01:00:44.023279+0000","last_unstale":"2023-09-10T01:00:55.955771+0000","last_undegraded":"2023-09-10T01:00:55.955771+0000","last_fullsized":"2023-09-10T01:00:55.955771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:18:05.737675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.954289+0000","last_change":"2023-09-10T01:00:44.035653+0000","last_active":"2023-09-10T01:00:55.954289+0000","last_peered":"2023-09-10T01:00:55.954289+0000","last_clean":"2023-09-10T01:00:55.954289+0000","last_became_active":"2023-09-10T01:00:44.035374+0000","last_became_peered":"2023-09-10T01:00:44.035374+0000","last_unstale":"2023-09-10T01:00:55.954289+0000","last_undegraded":"2023-09-10T01:00:55.954289+0000","last_fullsized":"2023-09-10T01:00:55.954289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:14:45.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.962663+0000","last_change":"2023-09-10T01:00:44.035805+0000","last_active":"2023-09-10T01:00:55.962663+0000","last_peered":"2023-09-10T01:00:55.962663+0000","last_clean":"2023-09-10T01:00:55.962663+0000","last_became_active":"2023-09-10T01:00:44.035479+0000","last_became_peered":"2023-09-10T01:00:44.035479+0000","last_unstale":"2023-09-10T01:00:55.962663+0000","last_undegraded":"2023-09-10T01:00:55.962663+0000","last_fullsized":"2023-09-10T01:00:55.962663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:36.662975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-09-10T01:00:55.958379+0000","last_change":"2023-09-10T01:00:44.025418+0000","last_active":"2023-09-10T01:00:55.958379+0000","last_peered":"2023-09-10T01:00:55.958379+0000","last_clean":"2023-09-10T01:00:55.958379+0000","last_became_active":"2023-09-10T01:00:44.025104+0000","last_became_peered":"2023-09-10T01:00:44.025104+0000","last_unstale":"2023-09-10T01:00:55.958379+0000","last_undegraded":"2023-09-10T01:00:55.958379+0000","last_fullsized":"2023-09-10T01:00:55.958379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:05:13.050781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282844,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":42563,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":35,"num_read_kb":61,"num_write":138,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":93009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481809,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":484547,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.495}]},{"osd":1,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.64900000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.318},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.70399999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53100000000000003}]},{"osd":6,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66600000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68000000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.69299999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.746}]}]},{"osd":10,"up_from":70,"seq":300647710740,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1109796,"data_compressed":53658,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":1,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.68799999999999994}]},{"osd":2,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.751}]},{"osd":3,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.7519999999999998,"5min":6.7519999999999998,"15min":6.7519999999999998},"last":0.65300000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63300000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.79900000000000004}]},{"osd":6,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.67800000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.83199999999999996}]},{"osd":9,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.81799999999999995}]},{"osd":11,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.84399999999999997}]}]},{"osd":9,"up_from":65,"seq":279172874265,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":480105,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.40999999999999998}]},{"osd":1,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.71199999999999997}]},{"osd":2,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.63},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":0.52100000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.73499999999999999}]},{"osd":5,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.68500000000000005}]},{"osd":6,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.569,"5min":1.569,"15min":1.569},"last":0.74299999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.83899999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:00:38 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.79300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135902,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":479694,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.58899999999999997,"15min":0.59999999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.58099999999999996,"15min":0.60099999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58799999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54100000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55500000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74399999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58299999999999996,"15min":0.57999999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81200000000000006,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.78700000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.45300000000000001,"15min":0.44700000000000001},"min":{"1min":0.309,"5min":0.254,"15min":0.254},"max":{"1min":0.64600000000000002,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.71599999999999997}]},{"osd":3,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68700000000000006,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.623,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.432,"15min":0.432},"max":{"1min":0.71599999999999997,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60599999999999998}]},{"osd":4,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79500000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78300000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.82799999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58399999999999996,"15min":0.58199999999999996},"min":{"1min":0.41999999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76200000000000001,"5min":0.87,"15min":0.87},"last":0.81200000000000006}]},{"osd":6,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68200000000000005,"15min":0.68799999999999994},"min":{"1min":0.46300000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.873,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96799999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94399999999999995,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.56999999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84599999999999997,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.878}]},{"osd":9,"last update":"Sun Sep 10 01:00:17 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.94099999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.90100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364835,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1079447,"data_compressed":53658,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91400000000000003,"5min":1.367,"15min":1.367},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.44400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.44400000000000001,"15min":0.434},"min":{"1min":0.29399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.38100000000000001}]},{"osd":2,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.58599999999999997,"15min":0.58099999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.441,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85499999999999998,"5min":1.409,"15min":1.409},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84799999999999998,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.443}]},{"osd":4,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.34000000000000002}]},{"osd":5,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.626,"15min":0.61799999999999999},"min":{"1min":0.47499999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65900000000000003,"15min":0.65300000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.5}]},{"osd":6,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.93300000000000005,"5min":1.343,"15min":1.343},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0249999999999999,"5min":1.319,"15min":1.319},"last":0.55500000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.69099999999999995,"15min":0.68300000000000005},"min":{"1min":0.495,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.66300000000000003}]},{"osd":9,"last update":"Sun Sep 10 01:00:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59499999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]},{"osd":6,"up_from":45,"seq":193273528358,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":479377,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63400000000000001,"15min":0.61699999999999999},"min":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.91500000000000004,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.86099999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96999999999999997,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.749,"15min":0.753},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.81499999999999995}]},{"osd":3,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.48299999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0389999999999999,"5min":1.425,"15min":1.425},"last":0.70999999999999996}]},{"osd":4,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.93899999999999995,"5min":1.111,"15min":1.111},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.74399999999999999,"15min":0.73099999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.94699999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.85699999999999998,"15min":0.85999999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.3149999999999999,"5min":1.53,"15min":1.53},"last":0.754},{"interface":"front","average":{"1min":0.876,"5min":0.85199999999999998,"15min":0.84799999999999998},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.23,"5min":1.353,"15min":1.353},"last":0.77300000000000002}]},{"osd":7,"last update":"Sun Sep 10 01:00:31 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.82499999999999996,"15min":0.81999999999999995},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0600000000000001,"5min":1.179,"15min":1.179},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.83199999999999996,"15min":0.82599999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1759999999999999,"5min":1.258,"15min":1.258},"last":0.92300000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:00:57 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90900000000000003,"15min":0.91100000000000003},"min":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1319999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88600000000000001,"15min":0.88400000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2010000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":1.0409999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:00:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.0600000000000001}]},{"osd":10,"last update":"Sun Sep 10 01:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.96899999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":500510,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54900000000000004,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.312,"15min":0.312},"max":{"1min":0.70899999999999996,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.55100000000000005},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85799999999999998,"5min":5.1580000000000004,"15min":5.1580000000000004},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.42799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.39400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.68899999999999995}]},{"osd":3,"last update":"Sun Sep 10 01:01:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.752},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.755,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.78000000000000003},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84399999999999997,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.622}]},{"osd":4,"last update":"Sun Sep 10 01:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.39100000000000001,"15min":0.36499999999999999},"min":{"1min":0.32800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40200000000000002,"15min":0.376},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.66400000000000003}]},{"osd":5,"last update":"Sun Sep 10 01:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56699999999999995,"15min":0.54900000000000004},"min":{"1min":0.40699999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.46200000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.70699999999999996}]},{"osd":6,"last update":"Sun Sep 10 01:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.622,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.66200000000000003,"15min":0.66800000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82699999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":1.018}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57499999999999996,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.91400000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.871,"5min":0.874,"15min":0.874},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65300000000000002,"15min":0.64300000000000002},"min":{"1min":0.54900000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.95999999999999996}]},{"osd":9,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":1.2370000000000001}]},{"osd":10,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.86499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]}]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":479981,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 10 01:00:26 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.60299999999999998,"15min":0.58299999999999996},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57399999999999995,"15min":0.55200000000000005},"min":{"1min":0.48699999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.81499999999999995}]},{"osd":2,"last update":"Sun Sep 10 01:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.627,"15min":0.59299999999999997},"min":{"1min":0.46100000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.61399999999999999},"min":{"1min":0.50800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89700000000000002,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.72199999999999998}]},{"osd":3,"last update":"Sun Sep 10 00:59:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.40300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56299999999999994,"15min":0.55400000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47499999999999998}]},{"osd":4,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.60999999999999999,"15min":0.57499999999999996},"min":{"1min":0.56599999999999995,"5min":0.308,"15min":0.308},"max":{"1min":0.88800000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.63800000000000001,"15min":0.59899999999999998},"min":{"1min":0.55500000000000005,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.60199999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72599999999999998,"15min":0.70499999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1,"5min":1.125,"15min":1.125},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68200000000000005,"15min":0.66000000000000003},"min":{"1min":0.47799999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.94399999999999995,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.78300000000000003}]},{"osd":6,"last update":"Sun Sep 10 01:00:08 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.68899999999999995,"15min":0.66700000000000004},"min":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.73899999999999999,"15min":0.71299999999999997},"min":{"1min":0.46999999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.89800000000000002}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.877,"15min":0.86699999999999999},"min":{"1min":0.627,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.83099999999999996,"15min":0.82499999999999996},"min":{"1min":0.49299999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.86299999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:00:54 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.93600000000000005,"15min":0.94499999999999995},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.1619999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.873},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.089,"5min":1.24,"15min":1.24},"last":0.89100000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:00:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.79700000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.71399999999999997}]},{"osd":11,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.91900000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":501238,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.54900000000000004,"15min":0.51100000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.54900000000000004,"15min":0.52900000000000003},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.55300000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56299999999999994,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82499999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56200000000000006,"15min":0.53600000000000003},"min":{"1min":0.40000000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.90300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.90600000000000003}]},{"osd":3,"last update":"Sun Sep 10 01:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.56899999999999995},"min":{"1min":0.49399999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.036,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.48099999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.89400000000000002,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.505}]},{"osd":4,"last update":"Sun Sep 10 01:00:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73799999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.55800000000000005},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77700000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.82799999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.432},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.81399999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60699999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.67600000000000005,"15min":0.66200000000000003},"min":{"1min":0.46200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94799999999999995,"5min":1.079,"15min":1.079},"last":0.78900000000000003}]},{"osd":7,"last update":"Sun Sep 10 01:00:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.879,"15min":0.879},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88100000000000001,"5min":1.01,"15min":1.01},"last":0.876}]},{"osd":8,"last update":"Sun Sep 10 01:00:53 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.36799999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69699999999999995,"15min":0.70299999999999996},"min":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92800000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.85999999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:00:15 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.68300000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.746},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.92900000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":479571,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41999999999999998,"15min":0.39000000000000001},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65000000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.42699999999999999},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48499999999999999}]},{"osd":1,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.52700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.117,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56100000000000005},"min":{"1min":0.40999999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.90100000000000002,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.90100000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68300000000000005,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.012,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68999999999999995,"15min":0.63600000000000001},"min":{"1min":0.499,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0820000000000001,"5min":1.226,"15min":1.226},"last":0.59399999999999997}]},{"osd":4,"last update":"Sun Sep 10 01:00:28 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.70699999999999996,"15min":0.67500000000000004},"min":{"1min":0.59599999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.751,"15min":0.71199999999999997},"min":{"1min":0.63100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":0.96599999999999997}]},{"osd":5,"last update":"Sun Sep 10 01:00:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0209999999999999,"5min":1.345,"15min":1.345},"last":0.63},{"interface":"front","average":{"1min":0.754,"5min":0.73599999999999999,"15min":0.71299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.129,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.64500000000000002,"15min":0.62},"min":{"1min":0.47999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.68400000000000005,"15min":0.65200000000000002},"min":{"1min":0.58599999999999997,"5min":0.443,"15min":0.443},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.81799999999999995}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78200000000000003,"15min":0.79300000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.093,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.53600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":1.3480000000000001,"5min":1.827,"15min":1.827},"last":0.77400000000000002}]},{"osd":8,"last update":"Sun Sep 10 00:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.88700000000000001}]},{"osd":10,"last update":"Sun Sep 10 01:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.745}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953521,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":480943,"data_compressed":17686,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.372,"15min":0.35599999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.79000000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.318},{"interface":"front","average":{"1min":0.435,"5min":0.38200000000000001,"15min":0.372},"min":{"1min":0.27100000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.39400000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.434,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.70599999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84699999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.379}]},{"osd":5,"last update":"Sun Sep 10 01:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.623,"15min":0.60099999999999998},"min":{"1min":0.501,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.60799999999999998,"15min":0.58299999999999996},"min":{"1min":0.47999999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.64900000000000002}]},{"osd":6,"last update":"Sun Sep 10 01:00:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.53200000000000003}]},{"osd":7,"last update":"Sun Sep 10 01:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.54800000000000004,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.432},{"interface":"front","average":{"1min":0.628,"5min":0.58699999999999997,"15min":0.57999999999999996},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48799999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:00:52 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.66000000000000003,"15min":0.64400000000000002},"min":{"1min":0.51400000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.74099999999999999,"15min":0.72899999999999998},"min":{"1min":0.58599999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95499999999999996,"5min":0.996,"15min":0.996},"last":0.73999999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:00:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63200000000000001}]},{"osd":10,"last update":"Sun Sep 10 01:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.75700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":5,"up_from":37,"seq":158913789996,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1075617,"data_compressed":53658,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.52600000000000002,"15min":0.502},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51900000000000002,"15min":0.503},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79400000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52600000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56200000000000006,"15min":0.54300000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56699999999999995,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.57999999999999996}]},{"osd":2,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39600000000000002,"15min":0.378},"min":{"1min":0.29199999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65000000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.433},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.41999999999999998,"15min":0.40400000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.41299999999999998}]},{"osd":3,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.42899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.378},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69599999999999995}]},{"osd":4,"last update":"Sun Sep 10 01:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59599999999999997,"15min":0.58299999999999996},"min":{"1min":0.41299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.60199999999999998,"15min":0.59199999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61699999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:00:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.629,"15min":0.624},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.82999999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65100000000000002,"15min":0.64900000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63300000000000001}]},{"osd":7,"last update":"Sun Sep 10 01:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.68600000000000005,"15min":0.68999999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.749}]},{"osd":8,"last update":"Sun Sep 10 01:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63700000000000001,"15min":0.64000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.60899999999999999,"15min":0.60699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56699999999999995}]},{"osd":9,"last update":"Sun Sep 10 01:00:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59799999999999998}]},{"osd":10,"last update":"Sun Sep 10 01:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.78500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":21951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":40141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-10T01:01:13.932 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-09-10T01:01:13.933 INFO:tasks.cephadm:Setting up client nodes... 2023-09-10T01:01:13.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-09-10T01:01:14.215 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:01:13 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: 2023-09-10T01:01:13.956+0000 7f1e948f8700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-09-10T01:01:14.699 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:15.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:15 smithi003 ceph-mon[97654]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:01:15.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:15 smithi003 ceph-mon[97654]: pgmap v297: 129 pgs: 129 active+clean; 618 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2023-09-10T01:01:15.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:15 smithi003 ceph-mon[97654]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:01:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:15 smithi136 ceph-mon[133708]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:01:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:15 smithi136 ceph-mon[133708]: pgmap v297: 129 pgs: 129 active+clean; 618 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2023-09-10T01:01:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:15 smithi136 ceph-mon[133708]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:01:15.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:15 smithi092 ceph-mon[134067]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:01:15.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:15 smithi092 ceph-mon[134067]: pgmap v297: 129 pgs: 129 active+clean; 618 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2023-09-10T01:01:15.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:15 smithi092 ceph-mon[134067]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:01:16.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 -- 172.21.15.3:0/2603880397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c41016f0 msgr2=0x7f97c4101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:16.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 --2- 172.21.15.3:0/2603880397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c41016f0 0x7f97c4101b50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009990 tx=0x7f97b8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 -- 172.21.15.3:0/2603880397 shutdown_connections 2023-09-10T01:01:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 --2- 172.21.15.3:0/2603880397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c4102090 0x7f97c4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 --2- 172.21.15.3:0/2603880397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 0x7f97c4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 --2- 172.21.15.3:0/2603880397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c41016f0 0x7f97c4101b50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 -- 172.21.15.3:0/2603880397 >> 172.21.15.3:0/2603880397 conn(0x7f97c40fbab0 msgr2=0x7f97c40fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:16.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.275+0000 7f97c915c700 1 -- 172.21.15.3:0/2603880397 shutdown_connections 2023-09-10T01:01:16.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.276+0000 7f97c915c700 1 -- 172.21.15.3:0/2603880397 wait complete. 2023-09-10T01:01:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.276+0000 7f97c915c700 1 Processor -- start 2023-09-10T01:01:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c915c700 1 -- start start 2023-09-10T01:01:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c915c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 0x7f97c4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c915c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c41016f0 0x7f97c4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c915c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c4102090 0x7f97c419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c915c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c419ffc0 con 0x7f97c41016f0 2023-09-10T01:01:16.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c915c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c41a0100 con 0x7f97c41004f0 2023-09-10T01:01:16.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c915c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c41a0290 con 0x7f97c4102090 2023-09-10T01:01:16.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 0x7f97c4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:16.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c41016f0 0x7f97c4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:16.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 0x7f97c4198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33954/0 (socket says 172.21.15.3:33954) 2023-09-10T01:01:16.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c3fff700 1 -- 172.21.15.3:0/3814924848 learned_addr learned my addr 172.21.15.3:0/3814924848 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:16.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.277+0000 7f97c895b700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c4102090 0x7f97c419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:16.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c3fff700 1 -- 172.21.15.3:0/3814924848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c4102090 msgr2=0x7f97c419f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:16.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c3fff700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c4102090 0x7f97c419f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c3fff700 1 -- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c41016f0 msgr2=0x7f97c4198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:16.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c3fff700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c41016f0 0x7f97c4198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c3fff700 1 -- 172.21.15.3:0/3814924848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc009c40 con 0x7f97c41004f0 2023-09-10T01:01:16.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c3fff700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 0x7f97c4198430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f97b400e970 tx=0x7f97b400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:16.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c17fa700 1 -- 172.21.15.3:0/3814924848 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4009790 con 0x7f97c41004f0 2023-09-10T01:01:16.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b80095f0 con 0x7f97c41004f0 2023-09-10T01:01:16.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c17fa700 1 -- 172.21.15.3:0/3814924848 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b400c7c0 con 0x7f97c41004f0 2023-09-10T01:01:16.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c17fa700 1 -- 172.21.15.3:0/3814924848 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4003c70 con 0x7f97c41004f0 2023-09-10T01:01:16.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.278+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c41a08e0 con 0x7f97c41004f0 2023-09-10T01:01:16.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.280+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c410bd50 con 0x7f97c41004f0 2023-09-10T01:01:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.281+0000 7f97c17fa700 1 -- 172.21.15.3:0/3814924848 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f97b400cc30 con 0x7f97c41004f0 2023-09-10T01:01:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.283+0000 7f97c17fa700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f97ac0a06d0 0x7f97ac0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.283+0000 7f97c17fa700 1 -- 172.21.15.3:0/3814924848 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f97b4014070 con 0x7f97c41004f0 2023-09-10T01:01:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.283+0000 7f97c37fe700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f97ac0a06d0 0x7f97ac0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.284+0000 7f97c37fe700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f97ac0a06d0 0x7f97ac0a2b90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f97b800b2e0 tx=0x7f97b8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.285+0000 7f97c17fa700 1 -- 172.21.15.3:0/3814924848 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f97b4095af0 con 0x7f97c41004f0 2023-09-10T01:01:16.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.577+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f97c410bfc0 con 0x7f97c41004f0 2023-09-10T01:01:16.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.584+0000 7f97c17fa700 1 -- 172.21.15.3:0/3814924848 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f97b4003dd0 con 0x7f97c41004f0 2023-09-10T01:01:16.592 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2023-09-10T01:01:16.592 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBcFf1ku5CUIhAA8Dq4JZ+vVrf/q2LN8ke67g== 2023-09-10T01:01:16.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f97ac0a06d0 msgr2=0x7f97ac0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:16.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f97ac0a06d0 0x7f97ac0a2b90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f97b800b2e0 tx=0x7f97b8005fb0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 msgr2=0x7f97c4198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:16.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 0x7f97c4198430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f97b400e970 tx=0x7f97b400ec80 comp rx=0 tx=0).stop 2023-09-10T01:01:16.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 shutdown_connections 2023-09-10T01:01:16.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f97ac0a06d0 0x7f97ac0a2b90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97c4102090 0x7f97c419f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97c41004f0 0x7f97c4198430 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 --2- 172.21.15.3:0/3814924848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97c41016f0 0x7f97c4198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:16.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.586+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 >> 172.21.15.3:0/3814924848 conn(0x7f97c40fbab0 msgr2=0x7f97c40fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:16.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.587+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 shutdown_connections 2023-09-10T01:01:16.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:16.587+0000 7f97c915c700 1 -- 172.21.15.3:0/3814924848 wait complete. 2023-09-10T01:01:17.096 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T01:01:17.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-09-10T01:01:17.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-10T01:01:17.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph config log 1 --format=json 2023-09-10T01:01:17.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:17 smithi003 ceph-mon[97654]: pgmap v298: 129 pgs: 129 active+clean; 618 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.5 KiB/s wr, 11 op/s 2023-09-10T01:01:17.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:17 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3814924848' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-09-10T01:01:17.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:17 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-09-10T01:01:17.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:17 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-09-10T01:01:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:17 smithi136 ceph-mon[133708]: pgmap v298: 129 pgs: 129 active+clean; 618 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.5 KiB/s wr, 11 op/s 2023-09-10T01:01:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:17 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3814924848' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-09-10T01:01:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:17 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-09-10T01:01:17.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:17 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-09-10T01:01:17.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:17 smithi092 ceph-mon[134067]: pgmap v298: 129 pgs: 129 active+clean; 618 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 5.5 KiB/s wr, 11 op/s 2023-09-10T01:01:17.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:17 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3814924848' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-09-10T01:01:17.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:17 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-09-10T01:01:17.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:17 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-09-10T01:01:17.972 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:19 smithi003 ceph-mon[97654]: pgmap v299: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-09-10T01:01:19.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:19 smithi136 ceph-mon[133708]: pgmap v299: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-09-10T01:01:19.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:19 smithi092 ceph-mon[134067]: pgmap v299: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-09-10T01:01:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.658+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3957800301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90106620 msgr2=0x7fbd90106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.658+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3957800301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90106620 0x7fbd90106a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbd800098f0 tx=0x7fbd80009c00 comp rx=0 tx=0).stop 2023-09-10T01:01:19.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.658+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3957800301 shutdown_connections 2023-09-10T01:01:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.658+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3957800301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd90100fe0 0x7fbd9010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.658+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3957800301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90106620 0x7fbd90106a00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.658+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3957800301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd90100620 0x7fbd90100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.658+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3957800301 >> 172.21.15.3:0/3957800301 conn(0x7fbd900fc0f0 msgr2=0x7fbd900fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.659+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3957800301 shutdown_connections 2023-09-10T01:01:19.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.659+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3957800301 wait complete. 2023-09-10T01:01:19.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 Processor -- start 2023-09-10T01:01:19.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 -- start start 2023-09-10T01:01:19.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90100620 0x7fbd90072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:19.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd90100fe0 0x7fbd90072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:19.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd90106620 0x7fbd900733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd901a4b90 con 0x7fbd90106620 2023-09-10T01:01:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd901a4d00 con 0x7fbd90100620 2023-09-10T01:01:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd94d7f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd901a4ec0 con 0x7fbd90100fe0 2023-09-10T01:01:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd8e59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90100620 0x7fbd90072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:19.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.660+0000 7fbd8ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd90106620 0x7fbd900733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd90100fe0 0x7fbd90072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90100620 0x7fbd90072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33964/0 (socket says 172.21.15.3:33964) 2023-09-10T01:01:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 -- 172.21.15.3:0/3384137687 learned_addr learned my addr 172.21.15.3:0/3384137687 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 -- 172.21.15.3:0/3384137687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd90100fe0 msgr2=0x7fbd90072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:19.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd90100fe0 0x7fbd90072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 -- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd90106620 msgr2=0x7fbd900733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd90106620 0x7fbd900733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 -- 172.21.15.3:0/3384137687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80009520 con 0x7fbd90100620 2023-09-10T01:01:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8ed9d700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd90106620 0x7fbd900733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:19.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd8e59c700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90100620 0x7fbd90072900 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009230 tx=0x7fbd8000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.661+0000 7fbd7f7fe700 1 -- 172.21.15.3:0/3384137687 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8001c070 con 0x7fbd90100620 2023-09-10T01:01:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.662+0000 7fbd7f7fe700 1 -- 172.21.15.3:0/3384137687 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd800058e0 con 0x7fbd90100620 2023-09-10T01:01:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.662+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd901a5140 con 0x7fbd90100620 2023-09-10T01:01:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.662+0000 7fbd7f7fe700 1 -- 172.21.15.3:0/3384137687 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd800038e0 con 0x7fbd90100620 2023-09-10T01:01:19.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.662+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd901a5630 con 0x7fbd90100620 2023-09-10T01:01:19.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.663+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9004e300 con 0x7fbd90100620 2023-09-10T01:01:19.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.665+0000 7fbd7f7fe700 1 -- 172.21.15.3:0/3384137687 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fbd80020030 con 0x7fbd90100620 2023-09-10T01:01:19.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.666+0000 7fbd7f7fe700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd780a0750 0x7fbd780a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:19.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.666+0000 7fbd7f7fe700 1 -- 172.21.15.3:0/3384137687 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbd800cd220 con 0x7fbd90100620 2023-09-10T01:01:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.667+0000 7fbd8dd9b700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd780a0750 0x7fbd780a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.667+0000 7fbd8dd9b700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd780a0750 0x7fbd780a2c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbd900740f0 tx=0x7fbd84009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.668+0000 7fbd7f7fe700 1 -- 172.21.15.3:0/3384137687 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbd80096910 con 0x7fbd90100620 2023-09-10T01:01:19.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.878+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) v1 -- 0x7fbd90110780 con 0x7fbd90100620 2023-09-10T01:01:19.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.879+0000 7fbd7f7fe700 1 -- 172.21.15.3:0/3384137687 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v43) v1 ==== 86+0+133 (secure 0 0 0) 0x7fbd80099f30 con 0x7fbd90100620 2023-09-10T01:01:19.880 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:19.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd780a0750 msgr2=0x7fbd780a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:19.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd780a0750 0x7fbd780a2c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbd900740f0 tx=0x7fbd84009310 comp rx=0 tx=0).stop 2023-09-10T01:01:19.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90100620 msgr2=0x7fbd90072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:19.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90100620 0x7fbd90072900 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009230 tx=0x7fbd8000fa00 comp rx=0 tx=0).stop 2023-09-10T01:01:19.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 shutdown_connections 2023-09-10T01:01:19.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd90106620 0x7fbd900733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd90100620 0x7fbd90072900 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd780a0750 0x7fbd780a2c10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 --2- 172.21.15.3:0/3384137687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd90100fe0 0x7fbd90072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:19.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 >> 172.21.15.3:0/3384137687 conn(0x7fbd900fc0f0 msgr2=0x7fbd90104130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:19.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 shutdown_connections 2023-09-10T01:01:19.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:19.881+0000 7fbd94d7f700 1 -- 172.21.15.3:0/3384137687 wait complete. 2023-09-10T01:01:20.157 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:20 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3384137687' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2023-09-10T01:01:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:20 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3384137687' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2023-09-10T01:01:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:20 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3384137687' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2023-09-10T01:01:20.473 INFO:teuthology.orchestra.run.smithi003.stdout:[{"version":43,"timestamp":"2023-09-10T01:00:38.922675+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2023-09-10T01:01:20.473 INFO:tasks.ceph_manager:config epoch is 43 2023-09-10T01:01:20.474 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-10T01:01:20.475 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-09-10T01:01:20.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph mgr dump --format=json 2023-09-10T01:01:21.209 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:21 smithi003 ceph-mon[97654]: pgmap v300: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2023-09-10T01:01:21.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:21 smithi136 ceph-mon[133708]: pgmap v300: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2023-09-10T01:01:21.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:21 smithi092 ceph-mon[134067]: pgmap v300: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2023-09-10T01:01:22.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 -- 172.21.15.3:0/2153244394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84106620 msgr2=0x7f7b84106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:22.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/2153244394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84106620 0x7f7b84106a00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009990 tx=0x7f7b80009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:22.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 -- 172.21.15.3:0/2153244394 shutdown_connections 2023-09-10T01:01:22.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/2153244394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b84100fe0 0x7f7b8410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:22.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/2153244394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84106620 0x7f7b84106a00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:22.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/2153244394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b84100620 0x7f7b84100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:22.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 -- 172.21.15.3:0/2153244394 >> 172.21.15.3:0/2153244394 conn(0x7f7b840fc0f0 msgr2=0x7f7b840fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:22.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 -- 172.21.15.3:0/2153244394 shutdown_connections 2023-09-10T01:01:22.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.757+0000 7f7b8c78b700 1 -- 172.21.15.3:0/2153244394 wait complete. 2023-09-10T01:01:22.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.758+0000 7f7b8c78b700 1 Processor -- start 2023-09-10T01:01:22.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8c78b700 1 -- start start 2023-09-10T01:01:22.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8c78b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84100620 0x7f7b84196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:22.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8c78b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b84100fe0 0x7f7b84196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:22.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8c78b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b84106620 0x7f7b8419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:22.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8c78b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8419a600 con 0x7f7b84100620 2023-09-10T01:01:22.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8c78b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8419a770 con 0x7f7b84106620 2023-09-10T01:01:22.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8c78b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8419a930 con 0x7f7b84100fe0 2023-09-10T01:01:22.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8a527700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84100620 0x7f7b84196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:22.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8a527700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84100620 0x7f7b84196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49876/0 (socket says 172.21.15.3:49876) 2023-09-10T01:01:22.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8a527700 1 -- 172.21.15.3:0/1774257659 learned_addr learned my addr 172.21.15.3:0/1774257659 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:22.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b89d26700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b84100fe0 0x7f7b84196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:22.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.759+0000 7f7b8ad28700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b84106620 0x7f7b8419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:22.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b8a527700 1 -- 172.21.15.3:0/1774257659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b84100fe0 msgr2=0x7f7b84196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:22.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b8a527700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b84100fe0 0x7f7b84196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:22.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b8a527700 1 -- 172.21.15.3:0/1774257659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b84106620 msgr2=0x7f7b8419a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:22.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b8a527700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b84106620 0x7f7b8419a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:22.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b8a527700 1 -- 172.21.15.3:0/1774257659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74009520 con 0x7f7b84100620 2023-09-10T01:01:22.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b89d26700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b84100fe0 0x7f7b84196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:22.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b8a527700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84100620 0x7f7b84196700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000c00 tx=0x7f7b80004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:22.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b7b7fe700 1 -- 172.21.15.3:0/1774257659 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8001d070 con 0x7f7b84100620 2023-09-10T01:01:22.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b7b7fe700 1 -- 172.21.15.3:0/1774257659 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b80021410 con 0x7f7b84100620 2023-09-10T01:01:22.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b7b7fe700 1 -- 172.21.15.3:0/1774257659 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80017650 con 0x7f7b84100620 2023-09-10T01:01:22.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.760+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b800095f0 con 0x7f7b84100620 2023-09-10T01:01:22.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.761+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b84072830 con 0x7f7b84100620 2023-09-10T01:01:22.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.762+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8404e300 con 0x7f7b84100620 2023-09-10T01:01:22.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.763+0000 7f7b7b7fe700 1 -- 172.21.15.3:0/1774257659 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f7b80021580 con 0x7f7b84100620 2023-09-10T01:01:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.765+0000 7f7b7b7fe700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b700a0750 0x7f7b700a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.765+0000 7f7b7b7fe700 1 -- 172.21.15.3:0/1774257659 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7b800cd2c0 con 0x7f7b84100620 2023-09-10T01:01:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.765+0000 7f7b89d26700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b700a0750 0x7f7b700a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.766+0000 7f7b89d26700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b700a0750 0x7f7b700a2c10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009fd0 tx=0x7f7b74009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:22.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:22.769+0000 7f7b7b7fe700 1 -- 172.21.15.3:0/1774257659 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7b800969b0 con 0x7f7b84100620 2023-09-10T01:01:23.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.041+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7b84110550 con 0x7f7b84100620 2023-09-10T01:01:23.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.047+0000 7f7b7b7fe700 1 -- 172.21.15.3:0/1774257659 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+259003 (secure 0 0 0) 0x7f7b80099fd0 con 0x7f7b84100620 2023-09-10T01:01:23.052 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:23.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b700a0750 msgr2=0x7f7b700a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:23.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b700a0750 0x7f7b700a2c10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009fd0 tx=0x7f7b74009260 comp rx=0 tx=0).stop 2023-09-10T01:01:23.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84100620 msgr2=0x7f7b84196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:23.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84100620 0x7f7b84196700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000c00 tx=0x7f7b80004790 comp rx=0 tx=0).stop 2023-09-10T01:01:23.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 shutdown_connections 2023-09-10T01:01:23.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b84106620 0x7f7b8419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:23.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b84100620 0x7f7b84196700 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:23.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b700a0750 0x7f7b700a2c10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:23.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 --2- 172.21.15.3:0/1774257659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b84100fe0 0x7f7b84196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:23.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 >> 172.21.15.3:0/1774257659 conn(0x7f7b840fc0f0 msgr2=0x7f7b84104130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:23.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.051+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 shutdown_connections 2023-09-10T01:01:23.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:23.052+0000 7f7b8c78b700 1 -- 172.21.15.3:0/1774257659 wait complete. 2023-09-10T01:01:23.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:23 smithi136 ceph-mon[133708]: pgmap v301: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2023-09-10T01:01:23.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:23 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1774257659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-09-10T01:01:23.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:23 smithi092 ceph-mon[134067]: pgmap v301: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2023-09-10T01:01:23.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:23 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1774257659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-09-10T01:01:23.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:23 smithi003 ceph-mon[97654]: pgmap v301: 129 pgs: 129 active+clean; 623 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2023-09-10T01:01:23.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:23 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1774257659' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-09-10T01:01:23.607 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":17,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":91094391},{"type":"v1","addr":"172.21.15.3:6801","nonce":91094391}]},"active_addr":"172.21.15.3:6801/91094391","active_change":"2023-09-10T00:53:55.753101+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24226,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24239,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2296857090}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3516755706}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1056299694}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2575312228}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":795902388}]}]} 2023-09-10T01:01:23.614 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-09-10T01:01:23.615 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-09-10T01:01:23.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd dump --format=json 2023-09-10T01:01:24.355 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:24.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:24 smithi136 ceph-mon[133708]: pgmap v302: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2023-09-10T01:01:24.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:24 smithi092 ceph-mon[134067]: pgmap v302: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2023-09-10T01:01:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:24 smithi003 ceph-mon[97654]: pgmap v302: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2023-09-10T01:01:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 -- 172.21.15.3:0/1979591108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac106770 msgr2=0x7f6eac106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/1979591108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac106770 0x7f6eac106b50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009990 tx=0x7f6ea8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 -- 172.21.15.3:0/1979591108 shutdown_connections 2023-09-10T01:01:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/1979591108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac068ae0 0x7f6eac10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:25.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/1979591108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac106770 0x7f6eac106b50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/1979591108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eac068120 0x7f6eac0685a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 -- 172.21.15.3:0/1979591108 >> 172.21.15.3:0/1979591108 conn(0x7f6eac076600 msgr2=0x7f6eac076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 -- 172.21.15.3:0/1979591108 shutdown_connections 2023-09-10T01:01:25.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.990+0000 7f6eb3e00700 1 -- 172.21.15.3:0/1979591108 wait complete. 2023-09-10T01:01:25.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.991+0000 7f6eb3e00700 1 Processor -- start 2023-09-10T01:01:25.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.991+0000 7f6eb3e00700 1 -- start start 2023-09-10T01:01:25.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb3e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac068120 0x7f6eac19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:25.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb3e00700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac068ae0 0x7f6eac19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:25.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb3e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eac106770 0x7f6eac196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:25.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb3e00700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac196cb0 con 0x7f6eac068ae0 2023-09-10T01:01:25.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb3e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac196e20 con 0x7f6eac106770 2023-09-10T01:01:25.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb3e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac196fe0 con 0x7f6eac068120 2023-09-10T01:01:25.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb1b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac068120 0x7f6eac19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:25.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb1b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac068120 0x7f6eac19c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39588/0 (socket says 172.21.15.3:39588) 2023-09-10T01:01:25.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb139b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac068ae0 0x7f6eac19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:25.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb1b9c700 1 -- 172.21.15.3:0/851022666 learned_addr learned my addr 172.21.15.3:0/851022666 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:25.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb239d700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eac106770 0x7f6eac196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:25.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb139b700 1 -- 172.21.15.3:0/851022666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac068120 msgr2=0x7f6eac19c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:25.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb139b700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac068120 0x7f6eac19c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:25.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb139b700 1 -- 172.21.15.3:0/851022666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eac106770 msgr2=0x7f6eac196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:25.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb139b700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eac106770 0x7f6eac196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:25.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.992+0000 7f6eb139b700 1 -- 172.21.15.3:0/851022666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea80095f0 con 0x7f6eac068ae0 2023-09-10T01:01:25.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.993+0000 7f6eb139b700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac068ae0 0x7f6eac19cab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d750 tx=0x7f6e9c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:25.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.993+0000 7f6ea2ffd700 1 -- 172.21.15.3:0/851022666 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c004020 con 0x7f6eac068ae0 2023-09-10T01:01:25.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.993+0000 7f6ea2ffd700 1 -- 172.21.15.3:0/851022666 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e9c004830 con 0x7f6eac068ae0 2023-09-10T01:01:25.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.993+0000 7f6ea2ffd700 1 -- 172.21.15.3:0/851022666 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0054c0 con 0x7f6eac068ae0 2023-09-10T01:01:26.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.993+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac1972c0 con 0x7f6eac068ae0 2023-09-10T01:01:26.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.993+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac0ff6b0 con 0x7f6eac068ae0 2023-09-10T01:01:26.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.994+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac04e300 con 0x7f6eac068ae0 2023-09-10T01:01:26.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.996+0000 7f6ea2ffd700 1 -- 172.21.15.3:0/851022666 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f6e9c0049a0 con 0x7f6eac068ae0 2023-09-10T01:01:26.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.998+0000 7f6ea2ffd700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e980a06d0 0x7f6e980a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:26.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.998+0000 7f6ea2ffd700 1 -- 172.21.15.3:0/851022666 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6e9c0cb990 con 0x7f6eac068ae0 2023-09-10T01:01:26.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.998+0000 7f6eb1b9c700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e980a06d0 0x7f6e980a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:26.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:25.998+0000 7f6eb1b9c700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e980a06d0 0x7f6e980a2b90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8006010 tx=0x7f6ea800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:26.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.002+0000 7f6ea2ffd700 1 -- 172.21.15.3:0/851022666 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6e9c094ff0 con 0x7f6eac068ae0 2023-09-10T01:01:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.212+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6eac197b00 con 0x7f6eac068ae0 2023-09-10T01:01:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.214+0000 7f6ea2ffd700 1 -- 172.21.15.3:0/851022666 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24321 (secure 0 0 0) 0x7f6e9c098610 con 0x7f6eac068ae0 2023-09-10T01:01:26.215 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:26.215 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:56.950998+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:01:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e980a06d0 msgr2=0x7f6e980a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e980a06d0 0x7f6e980a2b90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8006010 tx=0x7f6ea800b540 comp rx=0 tx=0).stop 2023-09-10T01:01:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac068ae0 msgr2=0x7f6eac19cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac068ae0 0x7f6eac19cab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00d750 tx=0x7f6e9c00da60 comp rx=0 tx=0).stop 2023-09-10T01:01:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 shutdown_connections 2023-09-10T01:01:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6e980a06d0 0x7f6e980a2b90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac068120 0x7f6eac19c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eac106770 0x7f6eac196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 --2- 172.21.15.3:0/851022666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6eac068ae0 0x7f6eac19cab0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 >> 172.21.15.3:0/851022666 conn(0x7f6eac076600 msgr2=0x7f6eac0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:26.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 shutdown_connections 2023-09-10T01:01:26.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:26.216+0000 7f6eb3e00700 1 -- 172.21.15.3:0/851022666 wait complete. 2023-09-10T01:01:26.755 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-09-10T01:01:26.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd dump --format=json 2023-09-10T01:01:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:26 smithi003 ceph-mon[97654]: pgmap v303: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2023-09-10T01:01:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:26 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/851022666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:27.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:26 smithi092 ceph-mon[134067]: pgmap v303: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2023-09-10T01:01:27.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:26 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/851022666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:26 smithi136 ceph-mon[133708]: pgmap v303: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2023-09-10T01:01:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:26 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/851022666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:27.494 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:29.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.039+0000 7fbf49a8e700 1 -- 172.21.15.3:0/356775911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf441005d0 msgr2=0x7fbf44100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:29.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.039+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/356775911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf441005d0 0x7fbf44100a50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009990 tx=0x7fbf38009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.039+0000 7fbf49a8e700 1 -- 172.21.15.3:0/356775911 shutdown_connections 2023-09-10T01:01:29.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.039+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/356775911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf44106660 0x7fbf44106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.039+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/356775911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf44100f90 0x7fbf4410d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.039+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/356775911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf441005d0 0x7fbf44100a50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.039+0000 7fbf49a8e700 1 -- 172.21.15.3:0/356775911 >> 172.21.15.3:0/356775911 conn(0x7fbf440fc100 msgr2=0x7fbf440fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.040+0000 7fbf49a8e700 1 -- 172.21.15.3:0/356775911 shutdown_connections 2023-09-10T01:01:29.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.040+0000 7fbf49a8e700 1 -- 172.21.15.3:0/356775911 wait complete. 2023-09-10T01:01:29.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 Processor -- start 2023-09-10T01:01:29.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 -- start start 2023-09-10T01:01:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf441005d0 0x7fbf4419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf44100f90 0x7fbf4419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf44106660 0x7fbf441986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf44198d60 con 0x7fbf44100f90 2023-09-10T01:01:29.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf44198ed0 con 0x7fbf44106660 2023-09-10T01:01:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.041+0000 7fbf49a8e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf44199090 con 0x7fbf441005d0 2023-09-10T01:01:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf437fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf441005d0 0x7fbf4419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf437fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf441005d0 0x7fbf4419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39604/0 (socket says 172.21.15.3:39604) 2023-09-10T01:01:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf42ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf44100f90 0x7fbf4419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:29.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf437fe700 1 -- 172.21.15.3:0/916924364 learned_addr learned my addr 172.21.15.3:0/916924364 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf43fff700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf44106660 0x7fbf441986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf42ffd700 1 -- 172.21.15.3:0/916924364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf441005d0 msgr2=0x7fbf4419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf42ffd700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf441005d0 0x7fbf4419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf42ffd700 1 -- 172.21.15.3:0/916924364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf44106660 msgr2=0x7fbf441986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:29.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf42ffd700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf44106660 0x7fbf441986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf42ffd700 1 -- 172.21.15.3:0/916924364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380095f0 con 0x7fbf44100f90 2023-09-10T01:01:29.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf43fff700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf44106660 0x7fbf441986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:29.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf42ffd700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf44100f90 0x7fbf4419ebb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009990 tx=0x7fbf38009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:29.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.042+0000 7fbf40ff9700 1 -- 172.21.15.3:0/916924364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3801d070 con 0x7fbf44100f90 2023-09-10T01:01:29.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.043+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf44199310 con 0x7fbf44100f90 2023-09-10T01:01:29.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.043+0000 7fbf40ff9700 1 -- 172.21.15.3:0/916924364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf38021410 con 0x7fbf44100f90 2023-09-10T01:01:29.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.043+0000 7fbf40ff9700 1 -- 172.21.15.3:0/916924364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38017650 con 0x7fbf44100f90 2023-09-10T01:01:29.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.043+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf441a5660 con 0x7fbf44100f90 2023-09-10T01:01:29.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.044+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4404e300 con 0x7fbf44100f90 2023-09-10T01:01:29.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.047+0000 7fbf40ff9700 1 -- 172.21.15.3:0/916924364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fbf380217a0 con 0x7fbf44100f90 2023-09-10T01:01:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.048+0000 7fbf40ff9700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf2c0a06d0 0x7fbf2c0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.049+0000 7fbf40ff9700 1 -- 172.21.15.3:0/916924364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbf38021e40 con 0x7fbf44100f90 2023-09-10T01:01:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.049+0000 7fbf437fe700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf2c0a06d0 0x7fbf2c0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:29.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.049+0000 7fbf437fe700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf2c0a06d0 0x7fbf2c0a2b90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbf34005950 tx=0x7fbf34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:29.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.051+0000 7fbf40ff9700 1 -- 172.21.15.3:0/916924364 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbf380972a0 con 0x7fbf44100f90 2023-09-10T01:01:29.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:28 smithi003 ceph-mon[97654]: pgmap v304: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-09-10T01:01:29.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:28 smithi092 ceph-mon[134067]: pgmap v304: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-09-10T01:01:29.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.263+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf44110550 con 0x7fbf44100f90 2023-09-10T01:01:29.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.264+0000 7fbf40ff9700 1 -- 172.21.15.3:0/916924364 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24321 (secure 0 0 0) 0x7fbf3809a8c0 con 0x7fbf44100f90 2023-09-10T01:01:29.266 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:01:29.266 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:00:56.950998+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:01:29.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf2c0a06d0 msgr2=0x7fbf2c0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:29.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf2c0a06d0 0x7fbf2c0a2b90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbf34005950 tx=0x7fbf34009260 comp rx=0 tx=0).stop 2023-09-10T01:01:29.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf44100f90 msgr2=0x7fbf4419ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:29.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf44100f90 0x7fbf4419ebb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009990 tx=0x7fbf38009580 comp rx=0 tx=0).stop 2023-09-10T01:01:29.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 shutdown_connections 2023-09-10T01:01:29.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf2c0a06d0 0x7fbf2c0a2b90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf441005d0 0x7fbf4419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf44106660 0x7fbf441986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 --2- 172.21.15.3:0/916924364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf44100f90 0x7fbf4419ebb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:29.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 >> 172.21.15.3:0/916924364 conn(0x7fbf440fc100 msgr2=0x7fbf44072830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:29.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 shutdown_connections 2023-09-10T01:01:29.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:29.267+0000 7fbf49a8e700 1 -- 172.21.15.3:0/916924364 wait complete. 2023-09-10T01:01:29.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:28 smithi136 ceph-mon[133708]: pgmap v304: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-09-10T01:01:29.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.0 flush_pg_stats 2023-09-10T01:01:29.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.1 flush_pg_stats 2023-09-10T01:01:29.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.2 flush_pg_stats 2023-09-10T01:01:29.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.3 flush_pg_stats 2023-09-10T01:01:29.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.4 flush_pg_stats 2023-09-10T01:01:29.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.5 flush_pg_stats 2023-09-10T01:01:29.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.6 flush_pg_stats 2023-09-10T01:01:29.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.7 flush_pg_stats 2023-09-10T01:01:29.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.8 flush_pg_stats 2023-09-10T01:01:29.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.9 flush_pg_stats 2023-09-10T01:01:29.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.10 flush_pg_stats 2023-09-10T01:01:29.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph tell osd.11 flush_pg_stats 2023-09-10T01:01:29.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:29 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/916924364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:30.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:29 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/916924364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:30.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:29 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/916924364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:01:31.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:30 smithi003 ceph-mon[97654]: pgmap v305: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-09-10T01:01:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:30 smithi136 ceph-mon[133708]: pgmap v305: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-09-10T01:01:31.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:30 smithi092 ceph-mon[134067]: pgmap v305: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-09-10T01:01:31.470 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:31.886 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:31.924 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:31.951 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:32.313 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:32.313 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:32.347 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:32.482 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:32.516 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:32.538 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:32.638 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:33.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:32 smithi003 ceph-mon[97654]: pgmap v306: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-09-10T01:01:33.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:32 smithi136 ceph-mon[133708]: pgmap v306: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-09-10T01:01:33.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:32 smithi092 ceph-mon[134067]: pgmap v306: 129 pgs: 129 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-09-10T01:01:33.655 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:35.202 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:34 smithi003 ceph-mon[97654]: pgmap v307: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-09-10T01:01:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:34 smithi136 ceph-mon[133708]: pgmap v307: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-09-10T01:01:35.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:34 smithi092 ceph-mon[134067]: pgmap v307: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-09-10T01:01:36.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.811+0000 7f2b56e40700 1 -- 172.21.15.3:0/652603862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b5010cee0 msgr2=0x7f2b5010f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.811+0000 7f2b56e40700 1 --2- 172.21.15.3:0/652603862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b5010cee0 0x7f2b5010f2d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009f90 tx=0x7f2b4800b330 comp rx=0 tx=0).stop 2023-09-10T01:01:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.811+0000 7f2b56e40700 1 -- 172.21.15.3:0/652603862 shutdown_connections 2023-09-10T01:01:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.812+0000 7f2b56e40700 1 --2- 172.21.15.3:0/652603862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b50072920 0x7f2b50072d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.812+0000 7f2b56e40700 1 --2- 172.21.15.3:0/652603862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b5010cee0 0x7f2b5010f2d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:36.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.812+0000 7f2b56e40700 1 --2- 172.21.15.3:0/652603862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b50073240 0x7f2b5010c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:36.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.812+0000 7f2b56e40700 1 -- 172.21.15.3:0/652603862 >> 172.21.15.3:0/652603862 conn(0x7f2b500fc0f0 msgr2=0x7f2b500fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:36.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.812+0000 7f2b56e40700 1 -- 172.21.15.3:0/652603862 shutdown_connections 2023-09-10T01:01:36.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.812+0000 7f2b56e40700 1 -- 172.21.15.3:0/652603862 wait complete. 2023-09-10T01:01:36.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.813+0000 7f2b56e40700 1 Processor -- start 2023-09-10T01:01:36.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.813+0000 7f2b56e40700 1 -- start start 2023-09-10T01:01:36.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b56e40700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b50072920 0x7f2b5019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:36.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b56e40700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b50073240 0x7f2b5019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:36.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b56e40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b5010cee0 0x7f2b50198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:36.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b56e40700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b50198e80 con 0x7f2b5010cee0 2023-09-10T01:01:36.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b56e40700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b50198ff0 con 0x7f2b50072920 2023-09-10T01:01:36.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b56e40700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b501991b0 con 0x7f2b50073240 2023-09-10T01:01:36.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b54bdc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b50072920 0x7f2b5019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:36.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b54bdc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b50072920 0x7f2b5019e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59226/0 (socket says 172.21.15.3:59226) 2023-09-10T01:01:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b54bdc700 1 -- 172.21.15.3:0/761455064 learned_addr learned my addr 172.21.15.3:0/761455064 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b4ffff700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b50073240 0x7f2b5019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.814+0000 7f2b553dd700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b5010cee0 0x7f2b50198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b54bdc700 1 -- 172.21.15.3:0/761455064 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b50073240 msgr2=0x7f2b5019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b54bdc700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b50073240 0x7f2b5019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b54bdc700 1 -- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b5010cee0 msgr2=0x7f2b50198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b54bdc700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b5010cee0 0x7f2b50198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b54bdc700 1 -- 172.21.15.3:0/761455064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c0095f0 con 0x7f2b50072920 2023-09-10T01:01:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b54bdc700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b50072920 0x7f2b5019e790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400e9a0 tx=0x7f2b4400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44009790 con 0x7f2b50072920 2023-09-10T01:01:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b48009c40 con 0x7f2b50072920 2023-09-10T01:01:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b44010430 con 0x7f2b50072920 2023-09-10T01:01:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.815+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44018680 con 0x7f2b50072920 2023-09-10T01:01:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.816+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b501a55b0 con 0x7f2b50072920 2023-09-10T01:01:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.816+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2b5004e300 con 0x7f2b50072920 2023-09-10T01:01:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.819+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f2b440105a0 con 0x7f2b50072920 2023-09-10T01:01:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.820+0000 7f2b4dffb700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2b400a06d0 0x7f2b400a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.820+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2b44014070 con 0x7f2b50072920 2023-09-10T01:01:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.820+0000 7f2b4ffff700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2b400a06d0 0x7f2b400a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.821+0000 7f2b4dffb700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] conn(0x7f2b400ab6f0 0x7f2b400adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.821+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 --> [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b400ae260 con 0x7f2b400ab6f0 2023-09-10T01:01:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.821+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b440cc8b0 con 0x7f2b50072920 2023-09-10T01:01:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.821+0000 7f2b553dd700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] conn(0x7f2b400ab6f0 0x7f2b400adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.821+0000 7f2b4ffff700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2b400a06d0 0x7f2b400a2b90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2b5010e4e0 tx=0x7f2b3c009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.821+0000 7f2b553dd700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] conn(0x7f2b400ab6f0 0x7f2b400adb90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.823+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== osd.3 v2:172.21.15.3:6810/434967290 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f2b400ae260 con 0x7f2b400ab6f0 2023-09-10T01:01:36.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.850+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 --> [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2b50112a50 con 0x7f2b400ab6f0 2023-09-10T01:01:36.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b4dffb700 1 -- 172.21.15.3:0/761455064 <== osd.3 v2:172.21.15.3:6810/434967290 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2b48009ac0 con 0x7f2b400ab6f0 2023-09-10T01:01:36.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] conn(0x7f2b400ab6f0 msgr2=0x7f2b400adb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:36.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] conn(0x7f2b400ab6f0 0x7f2b400adb90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:36.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2b400a06d0 msgr2=0x7f2b400a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:36.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2b400a06d0 0x7f2b400a2b90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2b5010e4e0 tx=0x7f2b3c009310 comp rx=0 tx=0).stop 2023-09-10T01:01:36.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b50072920 msgr2=0x7f2b5019e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:36.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 --2- 172.21.15.3:0/761455064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b50072920 0x7f2b5019e790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400e9a0 tx=0x7f2b4400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:01:36.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b553dd700 1 -- 172.21.15.3:0/761455064 reap_dead start 2023-09-10T01:01:36.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 shutdown_connections 2023-09-10T01:01:36.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.851+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 >> 172.21.15.3:0/761455064 conn(0x7f2b500fc0f0 msgr2=0x7f2b50104940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:36.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.852+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 shutdown_connections 2023-09-10T01:01:36.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:36.852+0000 7f2b56e40700 1 -- 172.21.15.3:0/761455064 wait complete. 2023-09-10T01:01:37.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:36 smithi003 ceph-mon[97654]: pgmap v308: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:37.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:36 smithi136 ceph-mon[133708]: pgmap v308: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:37.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:36 smithi092 ceph-mon[134067]: pgmap v308: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:38.375 INFO:teuthology.orchestra.run.smithi003.stdout:107374182458 2023-09-10T01:01:38.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.3 2023-09-10T01:01:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 -- 172.21.15.3:0/1377333778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068120 msgr2=0x7f9be80685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 --2- 172.21.15.3:0/1377333778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068120 0x7f9be80685a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0098f0 tx=0x7f9bdc009c00 comp rx=0 tx=0).stop 2023-09-10T01:01:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 -- 172.21.15.3:0/1377333778 shutdown_connections 2023-09-10T01:01:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 --2- 172.21.15.3:0/1377333778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9be8068ae0 0x7f9be810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 --2- 172.21.15.3:0/1377333778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9be8106770 0x7f9be8106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 --2- 172.21.15.3:0/1377333778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068120 0x7f9be80685a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 -- 172.21.15.3:0/1377333778 >> 172.21.15.3:0/1377333778 conn(0x7f9be8076600 msgr2=0x7f9be8076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 -- 172.21.15.3:0/1377333778 shutdown_connections 2023-09-10T01:01:38.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.525+0000 7f9bed963700 1 -- 172.21.15.3:0/1377333778 wait complete. 2023-09-10T01:01:38.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 Processor -- start 2023-09-10T01:01:38.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 -- start start 2023-09-10T01:01:38.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9be8068120 0x7f9be8107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:38.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068ae0 0x7f9be8107990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:38.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9be8106770 0x7f9be8107ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:38.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be81a9020 con 0x7f9be8068120 2023-09-10T01:01:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be81a9190 con 0x7f9be8068ae0 2023-09-10T01:01:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.527+0000 7f9bed963700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be81a9350 con 0x7f9be8106770 2023-09-10T01:01:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be67fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068ae0 0x7f9be8107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9be8068120 0x7f9be8107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9be8068120 0x7f9be8107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58742/0 (socket says 172.21.15.3:58742) 2023-09-10T01:01:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be67fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068ae0 0x7f9be8107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59242/0 (socket says 172.21.15.3:59242) 2023-09-10T01:01:38.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be6ffd700 1 -- 172.21.15.3:0/3879547568 learned_addr learned my addr 172.21.15.3:0/3879547568 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be77fe700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9be8106770 0x7f9be8107ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be67fc700 1 -- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9be8106770 msgr2=0x7f9be8107ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:38.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be67fc700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9be8106770 0x7f9be8107ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be67fc700 1 -- 172.21.15.3:0/3879547568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9be8068120 msgr2=0x7f9be8107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.528+0000 7f9be67fc700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9be8068120 0x7f9be8107450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.529+0000 7f9be67fc700 1 -- 172.21.15.3:0/3879547568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc009520 con 0x7f9be8068ae0 2023-09-10T01:01:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.529+0000 7f9be67fc700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068ae0 0x7f9be8107990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc009810 tx=0x7f9bdc00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:38.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.529+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc01c070 con 0x7f9be8068ae0 2023-09-10T01:01:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.529+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bdc00bba0 con 0x7f9be8068ae0 2023-09-10T01:01:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.529+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc003970 con 0x7f9be8068ae0 2023-09-10T01:01:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.530+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be81a95d0 con 0x7f9be8068ae0 2023-09-10T01:01:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.530+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be81a9990 con 0x7f9be8068ae0 2023-09-10T01:01:38.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.532+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9be81108e0 con 0x7f9be8068ae0 2023-09-10T01:01:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.535+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f9bdc020030 con 0x7f9be8068ae0 2023-09-10T01:01:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.536+0000 7f9bec961700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9bd00a0750 0x7f9bd00a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:38.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.537+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9bdc0cd510 con 0x7f9be8068ae0 2023-09-10T01:01:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.537+0000 7f9be6ffd700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9bd00a0750 0x7f9bd00a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.537+0000 7f9bec961700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] conn(0x7f9bd00ab770 0x7f9bd00adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.537+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 --> [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bd00ae2e0 con 0x7f9bd00ab770 2023-09-10T01:01:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.537+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bdc0cd880 con 0x7f9be8068ae0 2023-09-10T01:01:38.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.537+0000 7f9be77fe700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] conn(0x7f9bd00ab770 0x7f9bd00adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.538+0000 7f9be6ffd700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9bd00a0750 0x7f9bd00a2c10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8005950 tx=0x7f9bd8009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.560+0000 7f9be77fe700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] conn(0x7f9bd00ab770 0x7f9bd00adc10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.561+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== osd.2 v2:172.21.15.136:6800/781257073 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f9bd00ae2e0 con 0x7f9bd00ab770 2023-09-10T01:01:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.585+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 --> [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9be804e300 con 0x7f9bd00ab770 2023-09-10T01:01:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.586+0000 7f9bec961700 1 -- 172.21.15.3:0/3879547568 <== osd.2 v2:172.21.15.136:6800/781257073 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9be804e300 con 0x7f9bd00ab770 2023-09-10T01:01:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] conn(0x7f9bd00ab770 msgr2=0x7f9bd00adc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] conn(0x7f9bd00ab770 0x7f9bd00adc10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9bd00a0750 msgr2=0x7f9bd00a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9bd00a0750 0x7f9bd00a2c10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8005950 tx=0x7f9bd8009310 comp rx=0 tx=0).stop 2023-09-10T01:01:38.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068ae0 msgr2=0x7f9be8107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 --2- 172.21.15.3:0/3879547568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9be8068ae0 0x7f9be8107990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc009810 tx=0x7f9bdc00fa00 comp rx=0 tx=0).stop 2023-09-10T01:01:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9be77fe700 1 -- 172.21.15.3:0/3879547568 reap_dead start 2023-09-10T01:01:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 shutdown_connections 2023-09-10T01:01:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 >> 172.21.15.3:0/3879547568 conn(0x7f9be8076600 msgr2=0x7f9be80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 shutdown_connections 2023-09-10T01:01:38.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:38.587+0000 7f9bed963700 1 -- 172.21.15.3:0/3879547568 wait complete. 2023-09-10T01:01:39.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:38 smithi003 ceph-mon[97654]: pgmap v309: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2023-09-10T01:01:39.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:38 smithi136 ceph-mon[133708]: pgmap v309: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2023-09-10T01:01:39.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:38 smithi092 ceph-mon[134067]: pgmap v309: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2023-09-10T01:01:40.481 INFO:teuthology.orchestra.run.smithi003.stdout:77309411391 2023-09-10T01:01:40.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.2 2023-09-10T01:01:40.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 -- 172.21.15.3:0/2511748248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 msgr2=0x7f27e4106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:40.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 --2- 172.21.15.3:0/2511748248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 0x7f27e4106a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009990 tx=0x7f27d4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 -- 172.21.15.3:0/2511748248 shutdown_connections 2023-09-10T01:01:40.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 --2- 172.21.15.3:0/2511748248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27e4100fe0 0x7f27e410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 --2- 172.21.15.3:0/2511748248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 0x7f27e4106a00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 --2- 172.21.15.3:0/2511748248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e4100620 0x7f27e4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 -- 172.21.15.3:0/2511748248 >> 172.21.15.3:0/2511748248 conn(0x7f27e40fc0f0 msgr2=0x7f27e40fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:40.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 -- 172.21.15.3:0/2511748248 shutdown_connections 2023-09-10T01:01:40.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.348+0000 7f27e9248700 1 -- 172.21.15.3:0/2511748248 wait complete. 2023-09-10T01:01:40.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.349+0000 7f27e9248700 1 Processor -- start 2023-09-10T01:01:40.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.349+0000 7f27e9248700 1 -- start start 2023-09-10T01:01:40.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e9248700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e4100620 0x7f27e4072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:40.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e9248700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27e4100fe0 0x7f27e4072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:40.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e9248700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 0x7f27e40733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:40.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e9248700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e41a4b80 con 0x7f27e4106620 2023-09-10T01:01:40.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e9248700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e41a4cf0 con 0x7f27e4100fe0 2023-09-10T01:01:40.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e9248700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e41a4eb0 con 0x7f27e4100620 2023-09-10T01:01:40.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 0x7f27e40733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:40.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 0x7f27e40733c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58762/0 (socket says 172.21.15.3:58762) 2023-09-10T01:01:40.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e359e700 1 -- 172.21.15.3:0/2010118264 learned_addr learned my addr 172.21.15.3:0/2010118264 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:40.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e259c700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27e4100fe0 0x7f27e4072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:40.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e2d9d700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e4100620 0x7f27e4072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:40.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e259c700 1 -- 172.21.15.3:0/2010118264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e4100620 msgr2=0x7f27e4072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:40.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e259c700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e4100620 0x7f27e4072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e259c700 1 -- 172.21.15.3:0/2010118264 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 msgr2=0x7f27e40733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:40.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.350+0000 7f27e259c700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27e4106620 0x7f27e40733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.351+0000 7f27e259c700 1 -- 172.21.15.3:0/2010118264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d40095f0 con 0x7f27e4100fe0 2023-09-10T01:01:40.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.351+0000 7f27e259c700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27e4100fe0 0x7f27e4072e40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f27d800e9f0 tx=0x7f27d800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:40.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.351+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d800c9c0 con 0x7f27e4100fe0 2023-09-10T01:01:40.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.351+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27d8010430 con 0x7f27e4100fe0 2023-09-10T01:01:40.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.351+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e41a5190 con 0x7f27e4100fe0 2023-09-10T01:01:40.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.351+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d8018680 con 0x7f27e4100fe0 2023-09-10T01:01:40.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.351+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e41a5690 con 0x7f27e4100fe0 2023-09-10T01:01:40.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.352+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f27e404e300 con 0x7f27e4100fe0 2023-09-10T01:01:40.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.354+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f27d8010860 con 0x7f27e4100fe0 2023-09-10T01:01:40.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.356+0000 7f27cbfff700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27cc0a06d0 0x7f27cc0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:40.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.356+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f27d8014070 con 0x7f27e4100fe0 2023-09-10T01:01:40.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.356+0000 7f27e2d9d700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27cc0a06d0 0x7f27cc0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:40.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.356+0000 7f27cbfff700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] conn(0x7f27cc0ab6f0 0x7f27cc0adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:40.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.356+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 --> [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27cc0ae260 con 0x7f27cc0ab6f0 2023-09-10T01:01:40.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.356+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27d80cc7e0 con 0x7f27e4100fe0 2023-09-10T01:01:40.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.357+0000 7f27e2d9d700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27cc0a06d0 0x7f27cc0a2b90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f27d400b2e0 tx=0x7f27d4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:40.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.357+0000 7f27e359e700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] conn(0x7f27cc0ab6f0 0x7f27cc0adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:40.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.357+0000 7f27e359e700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] conn(0x7f27cc0ab6f0 0x7f27cc0adb90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:40.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.358+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== osd.4 v2:172.21.15.92:6808/2892874326 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f27cc0ae260 con 0x7f27cc0ab6f0 2023-09-10T01:01:40.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.375+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 --> [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f27e4110850 con 0x7f27cc0ab6f0 2023-09-10T01:01:40.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.376+0000 7f27cbfff700 1 -- 172.21.15.3:0/2010118264 <== osd.4 v2:172.21.15.92:6808/2892874326 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f27dc00f040 con 0x7f27cc0ab6f0 2023-09-10T01:01:40.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.376+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] conn(0x7f27cc0ab6f0 msgr2=0x7f27cc0adb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:40.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.376+0000 7f27e9248700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] conn(0x7f27cc0ab6f0 0x7f27cc0adb90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.376+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27cc0a06d0 msgr2=0x7f27cc0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:40.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.376+0000 7f27e9248700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27cc0a06d0 0x7f27cc0a2b90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f27d400b2e0 tx=0x7f27d4005fd0 comp rx=0 tx=0).stop 2023-09-10T01:01:40.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.376+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27e4100fe0 msgr2=0x7f27e4072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:40.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.376+0000 7f27e9248700 1 --2- 172.21.15.3:0/2010118264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27e4100fe0 0x7f27e4072e40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f27d800e9f0 tx=0x7f27d800ed00 comp rx=0 tx=0).stop 2023-09-10T01:01:40.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.377+0000 7f27e359e700 1 -- 172.21.15.3:0/2010118264 reap_dead start 2023-09-10T01:01:40.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.377+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 shutdown_connections 2023-09-10T01:01:40.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.377+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 >> 172.21.15.3:0/2010118264 conn(0x7f27e40fc0f0 msgr2=0x7f27e40fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.377+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 shutdown_connections 2023-09-10T01:01:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:39.377+0000 7f27e9248700 1 -- 172.21.15.3:0/2010118264 wait complete. 2023-09-10T01:01:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 -- 172.21.15.3:0/556250325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c073350 msgr2=0x7fa32c10c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 --2- 172.21.15.3:0/556250325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c073350 0x7fa32c10c920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa320009990 tx=0x7fa320009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 -- 172.21.15.3:0/556250325 shutdown_connections 2023-09-10T01:01:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 --2- 172.21.15.3:0/556250325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa32c072a30 0x7fa32c072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 --2- 172.21.15.3:0/556250325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa32c10ce60 0x7fa32c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 --2- 172.21.15.3:0/556250325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c073350 0x7fa32c10c920 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 -- 172.21.15.3:0/556250325 >> 172.21.15.3:0/556250325 conn(0x7fa32c0fc0f0 msgr2=0x7fa32c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 -- 172.21.15.3:0/556250325 shutdown_connections 2023-09-10T01:01:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.034+0000 7fa3327f0700 1 -- 172.21.15.3:0/556250325 wait complete. 2023-09-10T01:01:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 Processor -- start 2023-09-10T01:01:41.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 -- start start 2023-09-10T01:01:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c072a30 0x7fa32c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa32c073350 0x7fa32c198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa32c10ce60 0x7fa32c19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c19d900 con 0x7fa32c072a30 2023-09-10T01:01:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c19da70 con 0x7fa32c10ce60 2023-09-10T01:01:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.035+0000 7fa3327f0700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c19dc30 con 0x7fa32c073350 2023-09-10T01:01:41.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c072a30 0x7fa32c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c072a30 0x7fa32c198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58808/0 (socket says 172.21.15.3:58808) 2023-09-10T01:01:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 -- 172.21.15.3:0/3144436689 learned_addr learned my addr 172.21.15.3:0/3144436689 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa330d8d700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa32c10ce60 0x7fa32c19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32b7fe700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa32c073350 0x7fa32c198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 -- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa32c073350 msgr2=0x7fa32c198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa32c073350 0x7fa32c198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 -- 172.21.15.3:0/3144436689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa32c10ce60 msgr2=0x7fa32c19d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa32c10ce60 0x7fa32c19d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 -- 172.21.15.3:0/3144436689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3200095f0 con 0x7fa32c072a30 2023-09-10T01:01:41.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa330d8d700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa32c10ce60 0x7fa32c19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:41.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa32bfff700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c072a30 0x7fa32c198950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00d980 tx=0x7fa31c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c004220 con 0x7fa32c072a30 2023-09-10T01:01:41.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa31c007640 con 0x7fa32c072a30 2023-09-10T01:01:41.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.036+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c19df10 con 0x7fa32c072a30 2023-09-10T01:01:41.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.037+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c005610 con 0x7fa32c072a30 2023-09-10T01:01:41.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.037+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c1a57b0 con 0x7fa32c072a30 2023-09-10T01:01:41.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.038+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa32c04e300 con 0x7fa32c072a30 2023-09-10T01:01:41.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.039+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fa31c004750 con 0x7fa32c072a30 2023-09-10T01:01:41.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.041+0000 7fa3297fa700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa3140a06d0 0x7fa3140a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.041+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa31c0cc510 con 0x7fa32c072a30 2023-09-10T01:01:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.041+0000 7fa32b7fe700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa3140a06d0 0x7fa3140a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.041+0000 7fa3297fa700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] conn(0x7fa3140ab6f0 0x7fa3140adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.041+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 --> [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3140ae260 con 0x7fa3140ab6f0 2023-09-10T01:01:41.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.041+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa31c0cc920 con 0x7fa32c072a30 2023-09-10T01:01:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.042+0000 7fa32b7fe700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa3140a06d0 0x7fa3140a2b90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa32c199f70 tx=0x7fa320005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.042+0000 7fa330d8d700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] conn(0x7fa3140ab6f0 0x7fa3140adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.042+0000 7fa330d8d700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] conn(0x7fa3140ab6f0 0x7fa3140adb90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.043+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== osd.5 v2:172.21.15.136:6808/1094245369 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fa3140ae260 con 0x7fa3140ab6f0 2023-09-10T01:01:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.060+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 --> [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa32c112ac0 con 0x7fa3140ab6f0 2023-09-10T01:01:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3297fa700 1 -- 172.21.15.3:0/3144436689 <== osd.5 v2:172.21.15.136:6808/1094245369 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa324009d90 con 0x7fa3140ab6f0 2023-09-10T01:01:41.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] conn(0x7fa3140ab6f0 msgr2=0x7fa3140adb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] conn(0x7fa3140ab6f0 0x7fa3140adb90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa3140a06d0 msgr2=0x7fa3140a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa3140a06d0 0x7fa3140a2b90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa32c199f70 tx=0x7fa320005fb0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c072a30 msgr2=0x7fa32c198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 --2- 172.21.15.3:0/3144436689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa32c072a30 0x7fa32c198950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00d980 tx=0x7fa31c00dc90 comp rx=0 tx=0).stop 2023-09-10T01:01:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa330d8d700 1 -- 172.21.15.3:0/3144436689 reap_dead start 2023-09-10T01:01:41.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 shutdown_connections 2023-09-10T01:01:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.061+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 >> 172.21.15.3:0/3144436689 conn(0x7fa32c0fc0f0 msgr2=0x7fa32c10f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.062+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 shutdown_connections 2023-09-10T01:01:41.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.062+0000 7fa3327f0700 1 -- 172.21.15.3:0/3144436689 wait complete. 2023-09-10T01:01:41.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:40 smithi003 ceph-mon[97654]: pgmap v310: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:41.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 -- 172.21.15.3:0/1558381079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054106770 msgr2=0x7f1054106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 --2- 172.21.15.3:0/1558381079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054106770 0x7f1054106b50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1050009990 tx=0x7f1050009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 -- 172.21.15.3:0/1558381079 shutdown_connections 2023-09-10T01:01:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 --2- 172.21.15.3:0/1558381079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1054068ae0 0x7f105410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 --2- 172.21.15.3:0/1558381079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054106770 0x7f1054106b50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 --2- 172.21.15.3:0/1558381079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1054068120 0x7f10540685a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 -- 172.21.15.3:0/1558381079 >> 172.21.15.3:0/1558381079 conn(0x7f1054076600 msgr2=0x7f1054076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 -- 172.21.15.3:0/1558381079 shutdown_connections 2023-09-10T01:01:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.231+0000 7f105bbbf700 1 -- 172.21.15.3:0/1558381079 wait complete. 2023-09-10T01:01:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.232+0000 7f105bbbf700 1 Processor -- start 2023-09-10T01:01:41.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.233+0000 7f105bbbf700 1 -- start start 2023-09-10T01:01:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.233+0000 7f105bbbf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1054068120 0x7f10540ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.233+0000 7f105bbbf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054068ae0 0x7f10540ffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.233+0000 7f105bbbf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1054106770 0x7f1054100030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.233+0000 7f105bbbf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10541a9140 con 0x7f1054106770 2023-09-10T01:01:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.233+0000 7f105bbbf700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105419a900 con 0x7f1054068ae0 2023-09-10T01:01:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.233+0000 7f105bbbf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105419aac0 con 0x7f1054068120 2023-09-10T01:01:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054068ae0 0x7f10540ffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054068ae0 0x7f10540ffaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59300/0 (socket says 172.21.15.3:59300) 2023-09-10T01:01:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 -- 172.21.15.3:0/1694853987 learned_addr learned my addr 172.21.15.3:0/1694853987 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 -- 172.21.15.3:0/1694853987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1054068120 msgr2=0x7f10540ff5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:01:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1054068120 0x7f10540ff5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 -- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1054106770 msgr2=0x7f1054100030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105a15c700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1054106770 0x7f1054100030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1054106770 0x7f1054100030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105915a700 1 -- 172.21.15.3:0/1694853987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10500095f0 con 0x7f1054068ae0 2023-09-10T01:01:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.234+0000 7f105a15c700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1054106770 0x7f1054100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:01:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.235+0000 7f105915a700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054068ae0 0x7f10540ffaf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f104400e9a0 tx=0x7f104400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.235+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044009790 con 0x7f1054068ae0 2023-09-10T01:01:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.235+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105419ada0 con 0x7f1054068ae0 2023-09-10T01:01:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.235+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f104400cbd0 con 0x7f1054068ae0 2023-09-10T01:01:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.235+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044018680 con 0x7f1054068ae0 2023-09-10T01:01:41.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.235+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105419b2f0 con 0x7f1054068ae0 2023-09-10T01:01:41.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.236+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10541108c0 con 0x7f1054068ae0 2023-09-10T01:01:41.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.239+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f104400cd40 con 0x7f1054068ae0 2023-09-10T01:01:41.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.241+0000 7f104affd700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f10400a0750 0x7f10400a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.241+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1044014070 con 0x7f1054068ae0 2023-09-10T01:01:41.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.241+0000 7f105995b700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f10400a0750 0x7f10400a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.241+0000 7f104affd700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] conn(0x7f10400ab770 0x7f10400adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.241+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 --> [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10400ae2e0 con 0x7f10400ab770 2023-09-10T01:01:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.241+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10440ccda0 con 0x7f1054068ae0 2023-09-10T01:01:41.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.242+0000 7f105a15c700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] conn(0x7f10400ab770 0x7f10400adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.242+0000 7f105995b700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f10400a0750 0x7f10400a2c10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f10500098e0 tx=0x7f1050000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.242+0000 7f105a15c700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] conn(0x7f10400ab770 0x7f10400adc10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.243+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== osd.1 v2:172.21.15.92:6800/4007866052 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f10400ae2e0 con 0x7f10400ab770 2023-09-10T01:01:41.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.261+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 --> [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f105404e300 con 0x7f10400ab770 2023-09-10T01:01:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.262+0000 7f104affd700 1 -- 172.21.15.3:0/1694853987 <== osd.1 v2:172.21.15.92:6800/4007866052 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f105404e300 con 0x7f10400ab770 2023-09-10T01:01:41.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] conn(0x7f10400ab770 msgr2=0x7f10400adc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] conn(0x7f10400ab770 0x7f10400adc10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f10400a0750 msgr2=0x7f10400a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f10400a0750 0x7f10400a2c10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f10500098e0 tx=0x7f1050000bc0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054068ae0 msgr2=0x7f10540ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 --2- 172.21.15.3:0/1694853987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1054068ae0 0x7f10540ffaf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f104400e9a0 tx=0x7f104400ed60 comp rx=0 tx=0).stop 2023-09-10T01:01:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105a15c700 1 -- 172.21.15.3:0/1694853987 reap_dead start 2023-09-10T01:01:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 shutdown_connections 2023-09-10T01:01:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 >> 172.21.15.3:0/1694853987 conn(0x7f1054076600 msgr2=0x7f10540fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:41.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 shutdown_connections 2023-09-10T01:01:41.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:41.263+0000 7f105bbbf700 1 -- 172.21.15.3:0/1694853987 wait complete. 2023-09-10T01:01:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:40 smithi136 ceph-mon[133708]: pgmap v310: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:41.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:40 smithi092 ceph-mon[134067]: pgmap v310: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 -- 172.21.15.3:0/3730914940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888106770 msgr2=0x7f4888106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 --2- 172.21.15.3:0/3730914940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888106770 0x7f4888106b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4878009990 tx=0x7f4878009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 -- 172.21.15.3:0/3730914940 shutdown_connections 2023-09-10T01:01:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 --2- 172.21.15.3:0/3730914940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888106770 0x7f4888106b50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 --2- 172.21.15.3:0/3730914940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4888068ae0 0x7f488810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 --2- 172.21.15.3:0/3730914940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4888068120 0x7f48880685a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 -- 172.21.15.3:0/3730914940 >> 172.21.15.3:0/3730914940 conn(0x7f4888076600 msgr2=0x7f4888076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 -- 172.21.15.3:0/3730914940 shutdown_connections 2023-09-10T01:01:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.540+0000 7f488df61700 1 -- 172.21.15.3:0/3730914940 wait complete. 2023-09-10T01:01:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.542+0000 7f488df61700 1 Processor -- start 2023-09-10T01:01:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.542+0000 7f488df61700 1 -- start start 2023-09-10T01:01:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f488df61700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888068120 0x7f488819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f488df61700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4888068ae0 0x7f488819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f488df61700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4888106770 0x7f48881965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f488df61700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888196c20 con 0x7f4888068ae0 2023-09-10T01:01:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f488df61700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888196d90 con 0x7f4888106770 2023-09-10T01:01:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f488df61700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888196f50 con 0x7f4888068120 2023-09-10T01:01:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f4886ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4888068ae0 0x7f488819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f4887fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4888106770 0x7f48881965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f4887fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4888106770 0x7f48881965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59260/0 (socket says 172.21.15.3:59260) 2023-09-10T01:01:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f4887fff700 1 -- 172.21.15.3:0/1594482031 learned_addr learned my addr 172.21.15.3:0/1594482031 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f4886ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4888068ae0 0x7f488819ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58788/0 (socket says 172.21.15.3:58788) 2023-09-10T01:01:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.543+0000 7f48877fe700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888068120 0x7f488819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f48877fe700 1 -- 172.21.15.3:0/1594482031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4888106770 msgr2=0x7f48881965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f48877fe700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4888106770 0x7f48881965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f48877fe700 1 -- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4888068ae0 msgr2=0x7f488819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f48877fe700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4888068ae0 0x7f488819ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f48877fe700 1 -- 172.21.15.3:0/1594482031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48780095f0 con 0x7f4888068120 2023-09-10T01:01:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f4887fff700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4888106770 0x7f48881965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f48877fe700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888068120 0x7f488819c530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f48780098e0 tx=0x7f4878004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487801d070 con 0x7f4888068120 2023-09-10T01:01:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.544+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48881971d0 con 0x7f4888068120 2023-09-10T01:01:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.545+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48880ff6b0 con 0x7f4888068120 2023-09-10T01:01:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.545+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4878005030 con 0x7f4888068120 2023-09-10T01:01:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.545+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4878004ab0 con 0x7f4888068120 2023-09-10T01:01:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.546+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4874000ff0 con 0x7f4888068120 2023-09-10T01:01:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.549+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f4878004c10 con 0x7f4888068120 2023-09-10T01:01:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.550+0000 7f4884ff9700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f48700a0750 0x7f48700a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.550+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48780cd090 con 0x7f4888068120 2023-09-10T01:01:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.551+0000 7f4884ff9700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] conn(0x7f48700ab770 0x7f48700adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.551+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 --> [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48700ae2e0 con 0x7f48700ab770 2023-09-10T01:01:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.551+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48780cd400 con 0x7f4888068120 2023-09-10T01:01:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.551+0000 7f4886ffd700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f48700a0750 0x7f48700a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.551+0000 7f4887fff700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] conn(0x7f48700ab770 0x7f48700adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.552+0000 7f4886ffd700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f48700a0750 0x7f48700a2c10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4888076f20 tx=0x7f487c012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.552+0000 7f4887fff700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] conn(0x7f48700ab770 0x7f48700adc10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.553+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== osd.0 v2:172.21.15.3:6802/3367767236 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f48700ae2e0 con 0x7f48700ab770 2023-09-10T01:01:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.574+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 --> [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4874002d70 con 0x7f48700ab770 2023-09-10T01:01:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.575+0000 7f4884ff9700 1 -- 172.21.15.3:0/1594482031 <== osd.0 v2:172.21.15.3:6802/3367767236 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4874002d70 con 0x7f48700ab770 2023-09-10T01:01:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] conn(0x7f48700ab770 msgr2=0x7f48700adc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] conn(0x7f48700ab770 0x7f48700adc10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f48700a0750 msgr2=0x7f48700a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f48700a0750 0x7f48700a2c10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4888076f20 tx=0x7f487c012040 comp rx=0 tx=0).stop 2023-09-10T01:01:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888068120 msgr2=0x7f488819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 --2- 172.21.15.3:0/1594482031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4888068120 0x7f488819c530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f48780098e0 tx=0x7f4878004730 comp rx=0 tx=0).stop 2023-09-10T01:01:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f4887fff700 1 -- 172.21.15.3:0/1594482031 reap_dead start 2023-09-10T01:01:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 shutdown_connections 2023-09-10T01:01:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 >> 172.21.15.3:0/1594482031 conn(0x7f4888076600 msgr2=0x7f4888104390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.576+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 shutdown_connections 2023-09-10T01:01:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:40.577+0000 7f488df61700 1 -- 172.21.15.3:0/1594482031 wait complete. 2023-09-10T01:01:42.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.055+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/1070342833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0106620 msgr2=0x7f7bb0106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:42.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.055+0000 7f7bb7a8b700 1 --2- 172.21.15.3:0/1070342833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0106620 0x7f7bb0106a00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009990 tx=0x7f7bac009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.055+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/1070342833 shutdown_connections 2023-09-10T01:01:42.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.056+0000 7f7bb7a8b700 1 --2- 172.21.15.3:0/1070342833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb0100fe0 0x7f7bb010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.056+0000 7f7bb7a8b700 1 --2- 172.21.15.3:0/1070342833 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0106620 0x7f7bb0106a00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.056+0000 7f7bb7a8b700 1 --2- 172.21.15.3:0/1070342833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bb0100620 0x7f7bb0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.056+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/1070342833 >> 172.21.15.3:0/1070342833 conn(0x7f7bb00fc0f0 msgr2=0x7f7bb00fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:42.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.056+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/1070342833 shutdown_connections 2023-09-10T01:01:42.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.056+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/1070342833 wait complete. 2023-09-10T01:01:42.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 Processor -- start 2023-09-10T01:01:42.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 -- start start 2023-09-10T01:01:42.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb0100620 0x7f7bb019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:42.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0100fe0 0x7f7bb019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:42.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bb0106620 0x7f7bb0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:42.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0198df0 con 0x7f7bb0100fe0 2023-09-10T01:01:42.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0198f60 con 0x7f7bb0100620 2023-09-10T01:01:42.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.057+0000 7f7bb7a8b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0199120 con 0x7f7bb0106620 2023-09-10T01:01:42.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0100fe0 0x7f7bb019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:42.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0100fe0 0x7f7bb019ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58832/0 (socket says 172.21.15.3:58832) 2023-09-10T01:01:42.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 -- 172.21.15.3:0/3076279614 learned_addr learned my addr 172.21.15.3:0/3076279614 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:42.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5827700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb0100620 0x7f7bb019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:42.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb6028700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bb0106620 0x7f7bb0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:42.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 -- 172.21.15.3:0/3076279614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bb0106620 msgr2=0x7f7bb0198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:42.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bb0106620 0x7f7bb0198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 -- 172.21.15.3:0/3076279614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb0100620 msgr2=0x7f7bb019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:42.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bb0100620 0x7f7bb019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 -- 172.21.15.3:0/3076279614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac0095f0 con 0x7f7bb0100fe0 2023-09-10T01:01:42.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb6028700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bb0106620 0x7f7bb0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:42.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.058+0000 7f7bb5026700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0100fe0 0x7f7bb019ec40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e9a0 tx=0x7f7ba000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:42.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.059+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0009790 con 0x7f7bb0100fe0 2023-09-10T01:01:42.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.059+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb0199400 con 0x7f7bb0100fe0 2023-09-10T01:01:42.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.059+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ba0010430 con 0x7f7bb0100fe0 2023-09-10T01:01:42.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.059+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0018680 con 0x7f7bb0100fe0 2023-09-10T01:01:42.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.059+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb01a56f0 con 0x7f7bb0100fe0 2023-09-10T01:01:42.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.060+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7bb0110810 con 0x7f7bb0100fe0 2023-09-10T01:01:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.062+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f7ba00105a0 con 0x7f7bb0100fe0 2023-09-10T01:01:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.063+0000 7f7ba6ffd700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b9c0a06d0 0x7f7b9c0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:42.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.063+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7ba0014070 con 0x7f7bb0100fe0 2023-09-10T01:01:42.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.063+0000 7f7bb5827700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b9c0a06d0 0x7f7b9c0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:42.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.063+0000 7f7ba6ffd700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] conn(0x7f7b9c0ab6f0 0x7f7b9c0adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:42.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.063+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 --> [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b9c0ae260 con 0x7f7b9c0ab6f0 2023-09-10T01:01:42.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.063+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ba00cc8b0 con 0x7f7bb0100fe0 2023-09-10T01:01:42.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.064+0000 7f7bb6028700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] conn(0x7f7b9c0ab6f0 0x7f7b9c0adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:42.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.064+0000 7f7bb5827700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b9c0a06d0 0x7f7b9c0a2b90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00b2e0 tx=0x7f7bac005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:42.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.064+0000 7f7bb6028700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] conn(0x7f7b9c0ab6f0 0x7f7b9c0adb90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:42.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.065+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== osd.6 v2:172.21.15.3:6818/989795778 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f7b9c0ae260 con 0x7f7b9c0ab6f0 2023-09-10T01:01:42.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.082+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 --> [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7bb004e300 con 0x7f7b9c0ab6f0 2023-09-10T01:01:42.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7ba6ffd700 1 -- 172.21.15.3:0/3076279614 <== osd.6 v2:172.21.15.3:6818/989795778 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7bb004e300 con 0x7f7b9c0ab6f0 2023-09-10T01:01:42.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] conn(0x7f7b9c0ab6f0 msgr2=0x7f7b9c0adb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:42.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] conn(0x7f7b9c0ab6f0 0x7f7b9c0adb90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b9c0a06d0 msgr2=0x7f7b9c0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:42.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7b9c0a06d0 0x7f7b9c0a2b90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00b2e0 tx=0x7f7bac005960 comp rx=0 tx=0).stop 2023-09-10T01:01:42.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0100fe0 msgr2=0x7f7bb019ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:42.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 --2- 172.21.15.3:0/3076279614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bb0100fe0 0x7f7bb019ec40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e9a0 tx=0x7f7ba000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:01:42.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb6028700 1 -- 172.21.15.3:0/3076279614 reap_dead start 2023-09-10T01:01:42.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 shutdown_connections 2023-09-10T01:01:42.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.083+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 >> 172.21.15.3:0/3076279614 conn(0x7f7bb00fc0f0 msgr2=0x7f7bb010bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:42.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.084+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 shutdown_connections 2023-09-10T01:01:42.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.084+0000 7f7bb7a8b700 1 -- 172.21.15.3:0/3076279614 wait complete. 2023-09-10T01:01:43.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.643+0000 7fbf55aab700 1 -- 172.21.15.3:0/2355937262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf50100620 msgr2=0x7fbf50100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.643+0000 7fbf55aab700 1 --2- 172.21.15.3:0/2355937262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf50100620 0x7fbf50100aa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009990 tx=0x7fbf44009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.643+0000 7fbf55aab700 1 -- 172.21.15.3:0/2355937262 shutdown_connections 2023-09-10T01:01:43.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.643+0000 7fbf55aab700 1 --2- 172.21.15.3:0/2355937262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf50106620 0x7fbf50106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.643+0000 7fbf55aab700 1 --2- 172.21.15.3:0/2355937262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf50100fe0 0x7fbf5010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.643+0000 7fbf55aab700 1 --2- 172.21.15.3:0/2355937262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf50100620 0x7fbf50100aa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.643+0000 7fbf55aab700 1 -- 172.21.15.3:0/2355937262 >> 172.21.15.3:0/2355937262 conn(0x7fbf500fc0f0 msgr2=0x7fbf500fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:43.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.644+0000 7fbf55aab700 1 -- 172.21.15.3:0/2355937262 shutdown_connections 2023-09-10T01:01:43.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.644+0000 7fbf55aab700 1 -- 172.21.15.3:0/2355937262 wait complete. 2023-09-10T01:01:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.645+0000 7fbf55aab700 1 Processor -- start 2023-09-10T01:01:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.645+0000 7fbf55aab700 1 -- start start 2023-09-10T01:01:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf55aab700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf50100620 0x7fbf50072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf55aab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf50100fe0 0x7fbf50072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf55aab700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf50106620 0x7fbf500733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf55aab700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501a4bc0 con 0x7fbf50100fe0 2023-09-10T01:01:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf55aab700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501a4d30 con 0x7fbf50100620 2023-09-10T01:01:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf55aab700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501a4ef0 con 0x7fbf50106620 2023-09-10T01:01:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf50106620 0x7fbf500733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf50106620 0x7fbf500733c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46562/0 (socket says 172.21.15.3:46562) 2023-09-10T01:01:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf4effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf50100fe0 0x7fbf50072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf4ffff700 1 -- 172.21.15.3:0/849386686 learned_addr learned my addr 172.21.15.3:0/849386686 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.646+0000 7fbf4f7fe700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf50100620 0x7fbf50072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4ffff700 1 -- 172.21.15.3:0/849386686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf50100620 msgr2=0x7fbf50072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4ffff700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf50100620 0x7fbf50072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4ffff700 1 -- 172.21.15.3:0/849386686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf50100fe0 msgr2=0x7fbf50072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4ffff700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf50100fe0 0x7fbf50072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4ffff700 1 -- 172.21.15.3:0/849386686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40009520 con 0x7fbf50106620 2023-09-10T01:01:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4f7fe700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf50100620 0x7fbf50072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:01:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4ffff700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf50106620 0x7fbf500733c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800ef30 tx=0x7fbf4800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48018070 con 0x7fbf50106620 2023-09-10T01:01:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.647+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf440095f0 con 0x7fbf50106620 2023-09-10T01:01:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.648+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf4800f040 con 0x7fbf50106620 2023-09-10T01:01:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.648+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800cdb0 con 0x7fbf50106620 2023-09-10T01:01:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.648+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf501a54e0 con 0x7fbf50106620 2023-09-10T01:01:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.649+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbf5004e300 con 0x7fbf50106620 2023-09-10T01:01:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.651+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fbf48003a00 con 0x7fbf50106620 2023-09-10T01:01:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.652+0000 7fbf4cff9700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf380a06d0 0x7fbf380a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.652+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbf480cc3e0 con 0x7fbf50106620 2023-09-10T01:01:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.652+0000 7fbf4f7fe700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf380a06d0 0x7fbf380a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.653+0000 7fbf4cff9700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] conn(0x7fbf380ab6f0 0x7fbf380adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.653+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 --> [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf380ae260 con 0x7fbf380ab6f0 2023-09-10T01:01:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.653+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf480cc750 con 0x7fbf50106620 2023-09-10T01:01:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.653+0000 7fbf4effd700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] conn(0x7fbf380ab6f0 0x7fbf380adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.653+0000 7fbf4f7fe700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf380a06d0 0x7fbf380a2b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e20 tx=0x7fbf40009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.654+0000 7fbf4effd700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] conn(0x7fbf380ab6f0 0x7fbf380adb90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.654+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== osd.8 v2:172.21.15.136:6816/1553122381 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fbf380ae260 con 0x7fbf380ab6f0 2023-09-10T01:01:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.673+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 --> [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbf501108b0 con 0x7fbf380ab6f0 2023-09-10T01:01:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.674+0000 7fbf4cff9700 1 -- 172.21.15.3:0/849386686 <== osd.8 v2:172.21.15.136:6816/1553122381 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbf4401e040 con 0x7fbf380ab6f0 2023-09-10T01:01:43.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.674+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] conn(0x7fbf380ab6f0 msgr2=0x7fbf380adb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.674+0000 7fbf55aab700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] conn(0x7fbf380ab6f0 0x7fbf380adb90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.674+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf380a06d0 msgr2=0x7fbf380a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.674+0000 7fbf55aab700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbf380a06d0 0x7fbf380a2b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e20 tx=0x7fbf40009260 comp rx=0 tx=0).stop 2023-09-10T01:01:43.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.674+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf50106620 msgr2=0x7fbf500733c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.674+0000 7fbf55aab700 1 --2- 172.21.15.3:0/849386686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf50106620 0x7fbf500733c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800ef30 tx=0x7fbf4800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.675+0000 7fbf4ffff700 1 -- 172.21.15.3:0/849386686 reap_dead start 2023-09-10T01:01:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.675+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 shutdown_connections 2023-09-10T01:01:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.675+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 >> 172.21.15.3:0/849386686 conn(0x7fbf500fc0f0 msgr2=0x7fbf500fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.675+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 shutdown_connections 2023-09-10T01:01:43.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:42.675+0000 7fbf55aab700 1 -- 172.21.15.3:0/849386686 wait complete. 2023-09-10T01:01:43.199 INFO:teuthology.orchestra.run.smithi003.stdout:137438953527 2023-09-10T01:01:43.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.4 2023-09-10T01:01:43.265 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:42 smithi003 ceph-mon[97654]: pgmap v311: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:43.412 INFO:teuthology.orchestra.run.smithi003.stdout:193273528365 2023-09-10T01:01:43.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.6 2023-09-10T01:01:43.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:42 smithi136 ceph-mon[133708]: pgmap v311: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:43.425 INFO:teuthology.orchestra.run.smithi003.stdout:34359738441 2023-09-10T01:01:43.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.0 2023-09-10T01:01:43.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:42 smithi092 ceph-mon[134067]: pgmap v311: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:01:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 -- 172.21.15.3:0/3886604894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 msgr2=0x7f08ec0685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 --2- 172.21.15.3:0/3886604894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 0x7f08ec0685a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009990 tx=0x7f08e0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 -- 172.21.15.3:0/3886604894 shutdown_connections 2023-09-10T01:01:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 --2- 172.21.15.3:0/3886604894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08ec068ae0 0x7f08ec10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 --2- 172.21.15.3:0/3886604894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08ec106770 0x7f08ec106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 --2- 172.21.15.3:0/3886604894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 0x7f08ec0685a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 -- 172.21.15.3:0/3886604894 >> 172.21.15.3:0/3886604894 conn(0x7f08ec076600 msgr2=0x7f08ec076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 -- 172.21.15.3:0/3886604894 shutdown_connections 2023-09-10T01:01:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.504+0000 7f08f2607700 1 -- 172.21.15.3:0/3886604894 wait complete. 2023-09-10T01:01:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.506+0000 7f08f2607700 1 Processor -- start 2023-09-10T01:01:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.506+0000 7f08f2607700 1 -- start start 2023-09-10T01:01:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08f2607700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 0x7f08ec0ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08f2607700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08ec068ae0 0x7f08ec0ffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08f2607700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08ec106770 0x7f08ec100070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08f2607700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1a4b40 con 0x7f08ec068ae0 2023-09-10T01:01:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08f2607700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1a4cb0 con 0x7f08ec106770 2023-09-10T01:01:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08f2607700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec1a4e70 con 0x7f08ec068120 2023-09-10T01:01:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08ebfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 0x7f08ec0ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08ebfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 0x7f08ec0ff5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46576/0 (socket says 172.21.15.3:46576) 2023-09-10T01:01:43.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.507+0000 7f08ebfff700 1 -- 172.21.15.3:0/1903584722 learned_addr learned my addr 172.21.15.3:0/1903584722 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:43.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08eb7fe700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08ec068ae0 0x7f08ec0ffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08f0ba4700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08ec106770 0x7f08ec100070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08ebfff700 1 -- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08ec106770 msgr2=0x7f08ec100070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08ebfff700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08ec106770 0x7f08ec100070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08ebfff700 1 -- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08ec068ae0 msgr2=0x7f08ec0ffaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08ebfff700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08ec068ae0 0x7f08ec0ffaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08ebfff700 1 -- 172.21.15.3:0/1903584722 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e00095f0 con 0x7f08ec068120 2023-09-10T01:01:43.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.508+0000 7f08ebfff700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 0x7f08ec0ff5b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00d950 tx=0x7f08dc00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:43.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.509+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc009940 con 0x7f08ec068120 2023-09-10T01:01:43.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.509+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08dc010460 con 0x7f08ec068120 2023-09-10T01:01:43.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.509+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec1a5150 con 0x7f08ec068120 2023-09-10T01:01:43.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.509+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc00f620 con 0x7f08ec068120 2023-09-10T01:01:43.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.509+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec1a5670 con 0x7f08ec068120 2023-09-10T01:01:43.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.510+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f08ec1108e0 con 0x7f08ec068120 2023-09-10T01:01:43.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.513+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f08dc0108d0 con 0x7f08ec068120 2023-09-10T01:01:43.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.514+0000 7f08e97fa700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08d40a0750 0x7f08d40a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.514+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f08dc0cbfa0 con 0x7f08ec068120 2023-09-10T01:01:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.514+0000 7f08e97fa700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] conn(0x7f08d40ab770 0x7f08d40adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.514+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 --> [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08d40ae2e0 con 0x7f08d40ab770 2023-09-10T01:01:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.514+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08dc0cc310 con 0x7f08ec068120 2023-09-10T01:01:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.515+0000 7f08eb7fe700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08d40a0750 0x7f08d40a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.515+0000 7f08f0ba4700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] conn(0x7f08d40ab770 0x7f08d40adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.515+0000 7f08eb7fe700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08d40a0750 0x7f08d40a2c10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f08e000b2e0 tx=0x7f08e0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.516+0000 7f08f0ba4700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] conn(0x7f08d40ab770 0x7f08d40adc10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.517+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== osd.7 v2:172.21.15.92:6816/4114652830 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f08d40ae2e0 con 0x7f08d40ab770 2023-09-10T01:01:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.534+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 --> [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f08ec04e300 con 0x7f08d40ab770 2023-09-10T01:01:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.535+0000 7f08e97fa700 1 -- 172.21.15.3:0/1903584722 <== osd.7 v2:172.21.15.92:6816/4114652830 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f08ec04e300 con 0x7f08d40ab770 2023-09-10T01:01:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] conn(0x7f08d40ab770 msgr2=0x7f08d40adc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] conn(0x7f08d40ab770 0x7f08d40adc10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08d40a0750 msgr2=0x7f08d40a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08d40a0750 0x7f08d40a2c10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f08e000b2e0 tx=0x7f08e0005960 comp rx=0 tx=0).stop 2023-09-10T01:01:43.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 msgr2=0x7f08ec0ff5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:43.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 --2- 172.21.15.3:0/1903584722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08ec068120 0x7f08ec0ff5b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00d950 tx=0x7f08dc00dc60 comp rx=0 tx=0).stop 2023-09-10T01:01:43.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f0ba4700 1 -- 172.21.15.3:0/1903584722 reap_dead start 2023-09-10T01:01:43.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 shutdown_connections 2023-09-10T01:01:43.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 >> 172.21.15.3:0/1903584722 conn(0x7f08ec076600 msgr2=0x7f08ec0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:43.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 shutdown_connections 2023-09-10T01:01:43.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:43.536+0000 7f08f2607700 1 -- 172.21.15.3:0/1903584722 wait complete. 2023-09-10T01:01:43.832 INFO:teuthology.orchestra.run.smithi003.stdout:55834574917 2023-09-10T01:01:43.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.1 2023-09-10T01:01:44.209 INFO:teuthology.orchestra.run.smithi003.stdout:158913790002 2023-09-10T01:01:44.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.5 2023-09-10T01:01:44.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.302+0000 7f4617b9a700 1 -- 172.21.15.3:0/369766040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100729a0 msgr2=0x7f4610072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.302+0000 7f4617b9a700 1 --2- 172.21.15.3:0/369766040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100729a0 0x7f4610072d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f460c009990 tx=0x7f460c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.302+0000 7f4617b9a700 1 -- 172.21.15.3:0/369766040 shutdown_connections 2023-09-10T01:01:44.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.302+0000 7f4617b9a700 1 --2- 172.21.15.3:0/369766040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4610111630 0x7f4610113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.302+0000 7f4617b9a700 1 --2- 172.21.15.3:0/369766040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100729a0 0x7f4610072d80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.302+0000 7f4617b9a700 1 --2- 172.21.15.3:0/369766040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46100732c0 0x7f46101110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.302+0000 7f4617b9a700 1 -- 172.21.15.3:0/369766040 >> 172.21.15.3:0/369766040 conn(0x7f46100fc0f0 msgr2=0x7f46100fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:44.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.303+0000 7f4617b9a700 1 -- 172.21.15.3:0/369766040 shutdown_connections 2023-09-10T01:01:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.304+0000 7f4617b9a700 1 -- 172.21.15.3:0/369766040 wait complete. 2023-09-10T01:01:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 Processor -- start 2023-09-10T01:01:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 -- start start 2023-09-10T01:01:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100732c0 0x7f461019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4610111630 0x7f461019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46101a1900 0x7f46101a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46101a22f0 con 0x7f46100732c0 2023-09-10T01:01:44.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46101a2460 con 0x7f4610111630 2023-09-10T01:01:44.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.305+0000 7f4617b9a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46101a2620 con 0x7f46101a1900 2023-09-10T01:01:44.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100732c0 0x7f461019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615135700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4610111630 0x7f461019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100732c0 0x7f461019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58882/0 (socket says 172.21.15.3:58882) 2023-09-10T01:01:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 -- 172.21.15.3:0/2079005467 learned_addr learned my addr 172.21.15.3:0/2079005467 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615135700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4610111630 0x7f461019d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59386/0 (socket says 172.21.15.3:59386) 2023-09-10T01:01:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4616137700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46101a1900 0x7f46101a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 -- 172.21.15.3:0/2079005467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46101a1900 msgr2=0x7f46101a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46101a1900 0x7f46101a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 -- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4610111630 msgr2=0x7f461019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4610111630 0x7f461019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4615936700 1 -- 172.21.15.3:0/2079005467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0095f0 con 0x7f46100732c0 2023-09-10T01:01:44.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.306+0000 7f4616137700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46101a1900 0x7f46101a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:01:44.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.307+0000 7f4615936700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100732c0 0x7f461019cf50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f460c006010 tx=0x7f460c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.307+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c01d070 con 0x7f46100732c0 2023-09-10T01:01:44.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.307+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f460c021410 con 0x7f46100732c0 2023-09-10T01:01:44.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.307+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c017650 con 0x7f46100732c0 2023-09-10T01:01:44.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.307+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46101a9bb0 con 0x7f46100732c0 2023-09-10T01:01:44.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.307+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46101a9fd0 con 0x7f46100732c0 2023-09-10T01:01:44.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.308+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4610116e60 con 0x7f46100732c0 2023-09-10T01:01:44.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.312+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f460c00fad0 con 0x7f46100732c0 2023-09-10T01:01:44.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.313+0000 7f4606ffd700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f45fc0a06d0 0x7f45fc0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.313+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f460c0cd8f0 con 0x7f46100732c0 2023-09-10T01:01:44.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.313+0000 7f4615135700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f45fc0a06d0 0x7f45fc0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.313+0000 7f4606ffd700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] conn(0x7f45fc0ab670 0x7f45fc0adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.313+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 --> [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45fc0ae1e0 con 0x7f45fc0ab670 2023-09-10T01:01:44.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.313+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f460c0cdd00 con 0x7f46100732c0 2023-09-10T01:01:44.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.314+0000 7f4616137700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] conn(0x7f45fc0ab670 0x7f45fc0adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.314+0000 7f4615135700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f45fc0a06d0 0x7f45fc0a2b90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f461019e570 tx=0x7f4600009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.314+0000 7f4616137700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] conn(0x7f45fc0ab670 0x7f45fc0adb10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.316+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== osd.10 v2:172.21.15.92:6824/2881136596 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f45fc0ae1e0 con 0x7f45fc0ab670 2023-09-10T01:01:44.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.333+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 --> [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f461004e300 con 0x7f45fc0ab670 2023-09-10T01:01:44.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.333+0000 7f4606ffd700 1 -- 172.21.15.3:0/2079005467 <== osd.10 v2:172.21.15.92:6824/2881136596 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f461004e300 con 0x7f45fc0ab670 2023-09-10T01:01:44.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] conn(0x7f45fc0ab670 msgr2=0x7f45fc0adb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] conn(0x7f45fc0ab670 0x7f45fc0adb10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f45fc0a06d0 msgr2=0x7f45fc0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f45fc0a06d0 0x7f45fc0a2b90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f461019e570 tx=0x7f4600009260 comp rx=0 tx=0).stop 2023-09-10T01:01:44.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100732c0 msgr2=0x7f461019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 --2- 172.21.15.3:0/2079005467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46100732c0 0x7f461019cf50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f460c006010 tx=0x7f460c005720 comp rx=0 tx=0).stop 2023-09-10T01:01:44.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4616137700 1 -- 172.21.15.3:0/2079005467 reap_dead start 2023-09-10T01:01:44.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 shutdown_connections 2023-09-10T01:01:44.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 >> 172.21.15.3:0/2079005467 conn(0x7f46100fc0f0 msgr2=0x7f46100fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:44.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 shutdown_connections 2023-09-10T01:01:44.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.334+0000 7f4617b9a700 1 -- 172.21.15.3:0/2079005467 wait complete. 2023-09-10T01:01:44.540 INFO:teuthology.orchestra.run.smithi003.stdout:244813135909 2023-09-10T01:01:44.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.8 2023-09-10T01:01:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 -- 172.21.15.3:0/1699341658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6020100020 msgr2=0x7f6020100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 --2- 172.21.15.3:0/1699341658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6020100020 0x7f6020100400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f60100098f0 tx=0x7f6010009c00 comp rx=0 tx=0).stop 2023-09-10T01:01:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 -- 172.21.15.3:0/1699341658 shutdown_connections 2023-09-10T01:01:44.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 --2- 172.21.15.3:0/1699341658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602010d090 0x7f602010f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 --2- 172.21.15.3:0/1699341658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6020100020 0x7f6020100400 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 --2- 172.21.15.3:0/1699341658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6020100940 0x7f602010cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 -- 172.21.15.3:0/1699341658 >> 172.21.15.3:0/1699341658 conn(0x7f6020076600 msgr2=0x7f6020076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 -- 172.21.15.3:0/1699341658 shutdown_connections 2023-09-10T01:01:44.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.629+0000 7f6026a26700 1 -- 172.21.15.3:0/1699341658 wait complete. 2023-09-10T01:01:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.631+0000 7f6026a26700 1 Processor -- start 2023-09-10T01:01:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.631+0000 7f6026a26700 1 -- start start 2023-09-10T01:01:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f6026a26700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6020100020 0x7f6020198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f6026a26700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6020100940 0x7f6020198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f6026a26700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602010d090 0x7f602019d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f6026a26700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602019d910 con 0x7f6020100940 2023-09-10T01:01:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f6026a26700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602019da80 con 0x7f6020100020 2023-09-10T01:01:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f6026a26700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602019dc40 con 0x7f602010d090 2023-09-10T01:01:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f601ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6020100020 0x7f6020198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f601ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6020100020 0x7f6020198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59398/0 (socket says 172.21.15.3:59398) 2023-09-10T01:01:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f601f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6020100940 0x7f6020198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f601ffff700 1 -- 172.21.15.3:0/503596417 learned_addr learned my addr 172.21.15.3:0/503596417 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.632+0000 7f6024fc3700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602010d090 0x7f602019d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601ffff700 1 -- 172.21.15.3:0/503596417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602010d090 msgr2=0x7f602019d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601ffff700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602010d090 0x7f602019d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601ffff700 1 -- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6020100940 msgr2=0x7f6020198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601ffff700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6020100940 0x7f6020198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601ffff700 1 -- 172.21.15.3:0/503596417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6010009520 con 0x7f6020100020 2023-09-10T01:01:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601ffff700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6020100020 0x7f6020198960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6010009810 tx=0x7f601000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601001c070 con 0x7f6020100020 2023-09-10T01:01:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f601000bb20 con 0x7f6020100020 2023-09-10T01:01:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.633+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010004930 con 0x7f6020100020 2023-09-10T01:01:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.634+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602019dec0 con 0x7f6020100020 2023-09-10T01:01:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.634+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60201a55f0 con 0x7f6020100020 2023-09-10T01:01:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.635+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f602004e300 con 0x7f6020100020 2023-09-10T01:01:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.637+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f6010020070 con 0x7f6020100020 2023-09-10T01:01:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.639+0000 7f601d7fa700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60080a06d0 0x7f60080a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.639+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f60100cd470 con 0x7f6020100020 2023-09-10T01:01:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.639+0000 7f601f7fe700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60080a06d0 0x7f60080a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.639+0000 7f601d7fa700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] conn(0x7f60080ab6f0 0x7f60080adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.639+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 --> [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60080ae260 con 0x7f60080ab6f0 2023-09-10T01:01:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.639+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60100cd7e0 con 0x7f6020100020 2023-09-10T01:01:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.639+0000 7f6024fc3700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] conn(0x7f60080ab6f0 0x7f60080adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.640+0000 7f601f7fe700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60080a06d0 0x7f60080a2b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6020199f80 tx=0x7f6014009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.640+0000 7f6024fc3700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] conn(0x7f60080ab6f0 0x7f60080adb90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.641+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== osd.9 v2:172.21.15.3:6826/2161930524 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f60080ae260 con 0x7f60080ab6f0 2023-09-10T01:01:44.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.678+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 --> [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6020112ae0 con 0x7f60080ab6f0 2023-09-10T01:01:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.678+0000 7f601d7fa700 1 -- 172.21.15.3:0/503596417 <== osd.9 v2:172.21.15.3:6826/2161930524 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f601800f040 con 0x7f60080ab6f0 2023-09-10T01:01:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.679+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] conn(0x7f60080ab6f0 msgr2=0x7f60080adb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.679+0000 7f6026a26700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] conn(0x7f60080ab6f0 0x7f60080adb90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.679+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60080a06d0 msgr2=0x7f60080a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.679+0000 7f6026a26700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60080a06d0 0x7f60080a2b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6020199f80 tx=0x7f6014009260 comp rx=0 tx=0).stop 2023-09-10T01:01:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.679+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6020100020 msgr2=0x7f6020198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.679+0000 7f6026a26700 1 --2- 172.21.15.3:0/503596417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6020100020 0x7f6020198960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6010009810 tx=0x7f601000fa00 comp rx=0 tx=0).stop 2023-09-10T01:01:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.679+0000 7f6024fc3700 1 -- 172.21.15.3:0/503596417 reap_dead start 2023-09-10T01:01:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.680+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 shutdown_connections 2023-09-10T01:01:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.680+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 >> 172.21.15.3:0/503596417 conn(0x7f6020076600 msgr2=0x7f602010e640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.680+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 shutdown_connections 2023-09-10T01:01:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.680+0000 7f6026a26700 1 -- 172.21.15.3:0/503596417 wait complete. 2023-09-10T01:01:44.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.831+0000 7f96e8c93700 1 -- 172.21.15.3:0/331302434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4106770 msgr2=0x7f96e4106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.831+0000 7f96e8c93700 1 --2- 172.21.15.3:0/331302434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4106770 0x7f96e4106b50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009990 tx=0x7f96d4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.831+0000 7f96e8c93700 1 -- 172.21.15.3:0/331302434 shutdown_connections 2023-09-10T01:01:44.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.831+0000 7f96e8c93700 1 --2- 172.21.15.3:0/331302434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e4068ae0 0x7f96e410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.831+0000 7f96e8c93700 1 --2- 172.21.15.3:0/331302434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4106770 0x7f96e4106b50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.831+0000 7f96e8c93700 1 --2- 172.21.15.3:0/331302434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96e4068120 0x7f96e40685a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.831+0000 7f96e8c93700 1 -- 172.21.15.3:0/331302434 >> 172.21.15.3:0/331302434 conn(0x7f96e4076600 msgr2=0x7f96e4076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:44.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.832+0000 7f96e8c93700 1 -- 172.21.15.3:0/331302434 shutdown_connections 2023-09-10T01:01:44.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.832+0000 7f96e8c93700 1 -- 172.21.15.3:0/331302434 wait complete. 2023-09-10T01:01:44.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 Processor -- start 2023-09-10T01:01:44.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 -- start start 2023-09-10T01:01:44.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4068120 0x7f96e40ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e4068ae0 0x7f96e40ffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96e4106770 0x7f96e4100070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e41a4bd0 con 0x7f96e4106770 2023-09-10T01:01:44.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e41a4d40 con 0x7f96e4068120 2023-09-10T01:01:44.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.833+0000 7f96e8c93700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e41a4f00 con 0x7f96e4068ae0 2023-09-10T01:01:44.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4068120 0x7f96e40ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4068120 0x7f96e40ff5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56986/0 (socket says 172.21.15.3:56986) 2023-09-10T01:01:44.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 -- 172.21.15.3:0/3931002132 learned_addr learned my addr 172.21.15.3:0/3931002132 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:44.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e1d9b700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e4068ae0 0x7f96e40ffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e2d9d700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96e4106770 0x7f96e4100070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 -- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e4068ae0 msgr2=0x7f96e40ffaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e4068ae0 0x7f96e40ffaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 -- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96e4106770 msgr2=0x7f96e4100070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96e4106770 0x7f96e4100070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.834+0000 7f96e259c700 1 -- 172.21.15.3:0/3931002132 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d8009520 con 0x7f96e4068120 2023-09-10T01:01:44.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.835+0000 7f96e259c700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4068120 0x7f96e40ff5b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006010 tx=0x7f96d4005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.835+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d401d070 con 0x7f96e4068120 2023-09-10T01:01:44.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.835+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d40095f0 con 0x7f96e4068120 2023-09-10T01:01:44.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.835+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96d4022470 con 0x7f96e4068120 2023-09-10T01:01:44.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.835+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4017650 con 0x7f96e4068120 2023-09-10T01:01:44.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.835+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e41a5400 con 0x7f96e4068120 2023-09-10T01:01:44.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.836+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f96e41108c0 con 0x7f96e4068120 2023-09-10T01:01:44.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.838+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f96d4022850 con 0x7f96e4068120 2023-09-10T01:01:44.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.840+0000 7f96d37fe700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f96cc0a0750 0x7f96cc0a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.840+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f96d40cdb80 con 0x7f96e4068120 2023-09-10T01:01:44.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.840+0000 7f96d37fe700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] conn(0x7f96cc0ab770 0x7f96cc0adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:44.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.840+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 --> [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96cc0ae2e0 con 0x7f96cc0ab770 2023-09-10T01:01:44.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.840+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96d40cdef0 con 0x7f96e4068120 2023-09-10T01:01:44.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.840+0000 7f96e1d9b700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f96cc0a0750 0x7f96cc0a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.841+0000 7f96e2d9d700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] conn(0x7f96cc0ab770 0x7f96cc0adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:44.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.841+0000 7f96e1d9b700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f96cc0a0750 0x7f96cc0a2c10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f96e4100da0 tx=0x7f96d8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.841+0000 7f96e2d9d700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] conn(0x7f96cc0ab770 0x7f96cc0adc10 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:44.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.842+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== osd.11 v2:172.21.15.136:6824/2854427876 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f96cc0ae2e0 con 0x7f96cc0ab770 2023-09-10T01:01:44.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.864+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 --> [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f96e404e300 con 0x7f96cc0ab770 2023-09-10T01:01:44.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.864+0000 7f96d37fe700 1 -- 172.21.15.3:0/3931002132 <== osd.11 v2:172.21.15.136:6824/2854427876 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f96e404e300 con 0x7f96cc0ab770 2023-09-10T01:01:44.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.865+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] conn(0x7f96cc0ab770 msgr2=0x7f96cc0adc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.865+0000 7f96e8c93700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] conn(0x7f96cc0ab770 0x7f96cc0adc10 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:44.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.865+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f96cc0a0750 msgr2=0x7f96cc0a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.865+0000 7f96e8c93700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f96cc0a0750 0x7f96cc0a2c10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f96e4100da0 tx=0x7f96d8009260 comp rx=0 tx=0).stop 2023-09-10T01:01:44.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.865+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4068120 msgr2=0x7f96e40ff5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:44.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.865+0000 7f96e8c93700 1 --2- 172.21.15.3:0/3931002132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96e4068120 0x7f96e40ff5b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006010 tx=0x7f96d4005720 comp rx=0 tx=0).stop 2023-09-10T01:01:44.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.865+0000 7f96e2d9d700 1 -- 172.21.15.3:0/3931002132 reap_dead start 2023-09-10T01:01:44.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.866+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 shutdown_connections 2023-09-10T01:01:44.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.866+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 >> 172.21.15.3:0/3931002132 conn(0x7f96e4076600 msgr2=0x7f96e4104390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:44.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.866+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 shutdown_connections 2023-09-10T01:01:44.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:44.866+0000 7f96e8c93700 1 -- 172.21.15.3:0/3931002132 wait complete. 2023-09-10T01:01:45.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:44 smithi003 ceph-mon[97654]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 170 B/s wr, 7 op/s 2023-09-10T01:01:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:44 smithi136 ceph-mon[133708]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 170 B/s wr, 7 op/s 2023-09-10T01:01:45.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:44 smithi092 ceph-mon[134067]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 170 B/s wr, 7 op/s 2023-09-10T01:01:45.548 INFO:teuthology.orchestra.run.smithi003.stdout:300647710747 2023-09-10T01:01:45.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.10 2023-09-10T01:01:45.973 INFO:teuthology.orchestra.run.smithi003.stdout:214748364842 2023-09-10T01:01:45.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.7 2023-09-10T01:01:46.026 INFO:teuthology.orchestra.run.smithi003.stdout:279172874272 2023-09-10T01:01:46.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.9 2023-09-10T01:01:46.162 INFO:teuthology.orchestra.run.smithi003.stdout:330712481816 2023-09-10T01:01:46.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph osd last-stat-seq osd.11 2023-09-10T01:01:46.296 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:46.498 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:47.167 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:47.183 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:47.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:46 smithi003 ceph-mon[97654]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-09-10T01:01:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:46 smithi136 ceph-mon[133708]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-09-10T01:01:47.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:46 smithi092 ceph-mon[134067]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-09-10T01:01:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:48.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:48 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:01:48.281 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:48 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:01:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:48 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:01:48.627 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:48.686 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:49.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:49 smithi003 ceph-mon[97654]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:01:49.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.117+0000 7f32bfb64700 1 -- 172.21.15.3:0/3722717911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8102870 msgr2=0x7f32b8102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:49.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.117+0000 7f32bfb64700 1 --2- 172.21.15.3:0/3722717911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8102870 0x7f32b8102cf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009990 tx=0x7f32a8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.118+0000 7f32bfb64700 1 -- 172.21.15.3:0/3722717911 shutdown_connections 2023-09-10T01:01:49.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.118+0000 7f32bfb64700 1 --2- 172.21.15.3:0/3722717911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b8103230 0x7f32b810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.118+0000 7f32bfb64700 1 --2- 172.21.15.3:0/3722717911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8108870 0x7f32b8108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.118+0000 7f32bfb64700 1 --2- 172.21.15.3:0/3722717911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8102870 0x7f32b8102cf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.118+0000 7f32bfb64700 1 -- 172.21.15.3:0/3722717911 >> 172.21.15.3:0/3722717911 conn(0x7f32b80fe340 msgr2=0x7f32b8100760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:49.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.118+0000 7f32bfb64700 1 -- 172.21.15.3:0/3722717911 shutdown_connections 2023-09-10T01:01:49.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.118+0000 7f32bfb64700 1 -- 172.21.15.3:0/3722717911 wait complete. 2023-09-10T01:01:49.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 Processor -- start 2023-09-10T01:01:49.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 -- start start 2023-09-10T01:01:49.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b8102870 0x7f32b8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:49.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8103230 0x7f32b8075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:49.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8108870 0x7f32b8075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:49.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b81a8ff0 con 0x7f32b8103230 2023-09-10T01:01:49.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b81a9160 con 0x7f32b8108870 2023-09-10T01:01:49.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.119+0000 7f32bfb64700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b81a9320 con 0x7f32b8102870 2023-09-10T01:01:49.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32bd0ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8103230 0x7f32b8075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:49.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32bd0ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8103230 0x7f32b8075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51682/0 (socket says 172.21.15.3:51682) 2023-09-10T01:01:49.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32bd900700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b8102870 0x7f32b8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:49.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32bd0ff700 1 -- 172.21.15.3:0/1058045598 learned_addr learned my addr 172.21.15.3:0/1058045598 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:49.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32be101700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8108870 0x7f32b8075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:49.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32be101700 1 -- 172.21.15.3:0/1058045598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b8102870 msgr2=0x7f32b8074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:49.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32be101700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b8102870 0x7f32b8074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32be101700 1 -- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8103230 msgr2=0x7f32b8075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:49.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32be101700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8103230 0x7f32b8075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32be101700 1 -- 172.21.15.3:0/1058045598 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b0009c40 con 0x7f32b8108870 2023-09-10T01:01:49.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32bd0ff700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8103230 0x7f32b8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:49.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.120+0000 7f32be101700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8108870 0x7f32b8075590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f32b000efb0 tx=0x7f32b000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:49.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.121+0000 7f32aeffd700 1 -- 172.21.15.3:0/1058045598 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0003a00 con 0x7f32b8108870 2023-09-10T01:01:49.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.121+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a80095f0 con 0x7f32b8108870 2023-09-10T01:01:49.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.121+0000 7f32aeffd700 1 -- 172.21.15.3:0/1058045598 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32b000f040 con 0x7f32b8108870 2023-09-10T01:01:49.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.121+0000 7f32aeffd700 1 -- 172.21.15.3:0/1058045598 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0004540 con 0x7f32b8108870 2023-09-10T01:01:49.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.121+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b81a9970 con 0x7f32b8108870 2023-09-10T01:01:49.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.122+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b804e300 con 0x7f32b8108870 2023-09-10T01:01:49.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.124+0000 7f32aeffd700 1 -- 172.21.15.3:0/1058045598 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f32b0003bf0 con 0x7f32b8108870 2023-09-10T01:01:49.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.125+0000 7f32aeffd700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f32a40a06d0 0x7f32a40a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:49.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.126+0000 7f32aeffd700 1 -- 172.21.15.3:0/1058045598 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f32b00cbf40 con 0x7f32b8108870 2023-09-10T01:01:49.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.126+0000 7f32bd900700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f32a40a06d0 0x7f32a40a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:49.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.126+0000 7f32bd900700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f32a40a06d0 0x7f32a40a2b90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009be0 tx=0x7f32b4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:49.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.127+0000 7f32aeffd700 1 -- 172.21.15.3:0/1058045598 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f32b0095630 con 0x7f32b8108870 2023-09-10T01:01:49.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.352+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f32b8112ab0 con 0x7f32b8108870 2023-09-10T01:01:49.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.352+0000 7f32aeffd700 1 -- 172.21.15.3:0/1058045598 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f32b009de60 con 0x7f32b8108870 2023-09-10T01:01:49.355 INFO:teuthology.orchestra.run.smithi003.stdout:107374182460 2023-09-10T01:01:49.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f32a40a06d0 msgr2=0x7f32a40a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:49.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f32a40a06d0 0x7f32a40a2b90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009be0 tx=0x7f32b4009260 comp rx=0 tx=0).stop 2023-09-10T01:01:49.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8108870 msgr2=0x7f32b8075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:49.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8108870 0x7f32b8075590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f32b000efb0 tx=0x7f32b000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 shutdown_connections 2023-09-10T01:01:49.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f32a40a06d0 0x7f32a40a2b90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b8102870 0x7f32b8074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32b8108870 0x7f32b8075590 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 --2- 172.21.15.3:0/1058045598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f32b8103230 0x7f32b8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:49.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 >> 172.21.15.3:0/1058045598 conn(0x7f32b80fe340 msgr2=0x7f32b80fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:49.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 shutdown_connections 2023-09-10T01:01:49.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:49.355+0000 7f32bfb64700 1 -- 172.21.15.3:0/1058045598 wait complete. 2023-09-10T01:01:49.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:49 smithi136 ceph-mon[133708]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:01:49.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:49 smithi092 ceph-mon[134067]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:01:49.827 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:50.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1058045598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-09-10T01:01:50.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:50 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1058045598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-09-10T01:01:50.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:50 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:50 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1058045598' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-09-10T01:01:50.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:50 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:01:50.588 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:50.901 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182458 got 107374182460 for osd.3 2023-09-10T01:01:50.901 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:51.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:51 smithi003 ceph-mon[97654]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:01:51.244 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:01:51.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:51 smithi136 ceph-mon[133708]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:01:51.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:51 smithi092 ceph-mon[134067]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:01:51.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/3932607690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0100620 msgr2=0x7f0ce0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:51.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/3932607690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0100620 0x7f0ce0100aa0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009990 tx=0x7f0cd4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/3932607690 shutdown_connections 2023-09-10T01:01:51.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/3932607690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce0100fe0 0x7f0ce010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/3932607690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ce0106620 0x7f0ce0106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/3932607690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0100620 0x7f0ce0100aa0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/3932607690 >> 172.21.15.3:0/3932607690 conn(0x7f0ce00fc0f0 msgr2=0x7f0ce00fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/3932607690 shutdown_connections 2023-09-10T01:01:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.716+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/3932607690 wait complete. 2023-09-10T01:01:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.717+0000 7f0ce5a3b700 1 Processor -- start 2023-09-10T01:01:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.717+0000 7f0ce5a3b700 1 -- start start 2023-09-10T01:01:51.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0ce5a3b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ce0100620 0x7f0ce019c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0ce5a3b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce0100fe0 0x7f0ce019cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0ce5a3b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0106620 0x7f0ce01965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0ce5a3b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce0196c60 con 0x7f0ce0106620 2023-09-10T01:01:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0ce5a3b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce0196dd0 con 0x7f0ce0100620 2023-09-10T01:01:51.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0ce5a3b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce0196f90 con 0x7f0ce0100fe0 2023-09-10T01:01:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0106620 0x7f0ce01965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0106620 0x7f0ce01965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51710/0 (socket says 172.21.15.3:51710) 2023-09-10T01:01:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cde7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce0100fe0 0x7f0ce019cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 -- 172.21.15.3:0/719483294 learned_addr learned my addr 172.21.15.3:0/719483294 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:51.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdeffd700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ce0100620 0x7f0ce019c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 -- 172.21.15.3:0/719483294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce0100fe0 msgr2=0x7f0ce019cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce0100fe0 0x7f0ce019cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 -- 172.21.15.3:0/719483294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ce0100620 msgr2=0x7f0ce019c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ce0100620 0x7f0ce019c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.718+0000 7f0cdf7fe700 1 -- 172.21.15.3:0/719483294 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40095f0 con 0x7f0ce0106620 2023-09-10T01:01:51.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.719+0000 7f0cdf7fe700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0106620 0x7f0ce01965f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8007ee0 tx=0x7f0cd800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:51.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.719+0000 7f0ce4a39700 1 -- 172.21.15.3:0/719483294 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd800dd20 con 0x7f0ce0106620 2023-09-10T01:01:51.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.719+0000 7f0ce4a39700 1 -- 172.21.15.3:0/719483294 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0cd800f040 con 0x7f0ce0106620 2023-09-10T01:01:51.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.719+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce0197270 con 0x7f0ce0106620 2023-09-10T01:01:51.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.719+0000 7f0ce4a39700 1 -- 172.21.15.3:0/719483294 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd80048a0 con 0x7f0ce0106620 2023-09-10T01:01:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.719+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce01a3560 con 0x7f0ce0106620 2023-09-10T01:01:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.720+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce004e300 con 0x7f0ce0106620 2023-09-10T01:01:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.722+0000 7f0ce4a39700 1 -- 172.21.15.3:0/719483294 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f0cd8003a00 con 0x7f0ce0106620 2023-09-10T01:01:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.724+0000 7f0ce4a39700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0cc80a0750 0x7f0cc80a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.724+0000 7f0ce4a39700 1 -- 172.21.15.3:0/719483294 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0cd80cb880 con 0x7f0ce0106620 2023-09-10T01:01:51.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.724+0000 7f0cdeffd700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0cc80a0750 0x7f0cc80a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:51.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.724+0000 7f0cdeffd700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0cc80a0750 0x7f0cc80a2c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009be0 tx=0x7f0cd0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:51.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.728+0000 7f0ce4a39700 1 -- 172.21.15.3:0/719483294 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0cd804ee10 con 0x7f0ce0106620 2023-09-10T01:01:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.952+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0ce0110750 con 0x7f0ce0106620 2023-09-10T01:01:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.952+0000 7f0ce4a39700 1 -- 172.21.15.3:0/719483294 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0cd8094ee0 con 0x7f0ce0106620 2023-09-10T01:01:51.956 INFO:teuthology.orchestra.run.smithi003.stdout:137438953529 2023-09-10T01:01:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0cc80a0750 msgr2=0x7f0cc80a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0cc80a0750 0x7f0cc80a2c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009be0 tx=0x7f0cd0009260 comp rx=0 tx=0).stop 2023-09-10T01:01:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0106620 msgr2=0x7f0ce01965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0106620 0x7f0ce01965f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8007ee0 tx=0x7f0cd800d3b0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 shutdown_connections 2023-09-10T01:01:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ce0106620 0x7f0ce01965f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ce0100620 0x7f0ce019c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0cc80a0750 0x7f0cc80a2c10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 --2- 172.21.15.3:0/719483294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce0100fe0 0x7f0ce019cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:51.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 >> 172.21.15.3:0/719483294 conn(0x7f0ce00fc0f0 msgr2=0x7f0ce00fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:51.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 shutdown_connections 2023-09-10T01:01:51.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:51.955+0000 7f0ce5a3b700 1 -- 172.21.15.3:0/719483294 wait complete. 2023-09-10T01:01:52.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:52 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/719483294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-09-10T01:01:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:52 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/719483294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-09-10T01:01:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:52 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/719483294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-09-10T01:01:53.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2102713451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 msgr2=0x7f5bcc10a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:53.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2102713451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc10a420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009990 tx=0x7f5bbc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2102713451 shutdown_connections 2023-09-10T01:01:53.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2102713451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bcc10aa40 0x7f5bcc10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2102713451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bcc101610 0x7f5bcc1019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2102713451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc10a420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2102713451 >> 172.21.15.3:0/2102713451 conn(0x7f5bcc0faea0 msgr2=0x7f5bcc0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:53.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2102713451 shutdown_connections 2023-09-10T01:01:53.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2102713451 wait complete. 2023-09-10T01:01:53.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.373+0000 7f5bd2e42700 1 Processor -- start 2023-09-10T01:01:53.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.374+0000 7f5bd2e42700 1 -- start start 2023-09-10T01:01:53.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd2e42700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bcc101610 0x7f5bcc0ff270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:53.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd2e42700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc0ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:53.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd2e42700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bcc10aa40 0x7f5bcc0ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:53.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd2e42700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc1a49e0 con 0x7f5bcc10aa40 2023-09-10T01:01:53.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd2e42700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc1a4b50 con 0x7f5bcc101f30 2023-09-10T01:01:53.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd2e42700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc1a4d10 con 0x7f5bcc101610 2023-09-10T01:01:53.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd163f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc0ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:53.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd2641700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bcc10aa40 0x7f5bcc0ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:53.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd163f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc0ff7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57048/0 (socket says 172.21.15.3:57048) 2023-09-10T01:01:53.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.375+0000 7f5bd163f700 1 -- 172.21.15.3:0/2299164703 learned_addr learned my addr 172.21.15.3:0/2299164703 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:53.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.376+0000 7f5bd1e40700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bcc101610 0x7f5bcc0ff270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:53.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.376+0000 7f5bd163f700 1 -- 172.21.15.3:0/2299164703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bcc101610 msgr2=0x7f5bcc0ff270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:53.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.376+0000 7f5bd163f700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bcc101610 0x7f5bcc0ff270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.136 INFO:teuthology.orchestra.run.smithi003.stdout:77309411394 2023-09-10T01:01:53.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.376+0000 7f5bd163f700 1 -- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bcc10aa40 msgr2=0x7f5bcc0ffd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:53.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.376+0000 7f5bd163f700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bcc10aa40 0x7f5bcc0ffd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.376+0000 7f5bd163f700 1 -- 172.21.15.3:0/2299164703 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc0095f0 con 0x7f5bcc101f30 2023-09-10T01:01:53.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.376+0000 7f5bd163f700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc0ff7b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc000c00 tx=0x7f5bbc005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:53.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.377+0000 7f5bc2ffd700 1 -- 172.21.15.3:0/2299164703 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc01d070 con 0x7f5bcc101f30 2023-09-10T01:01:53.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.377+0000 7f5bc2ffd700 1 -- 172.21.15.3:0/2299164703 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bbc022470 con 0x7f5bcc101f30 2023-09-10T01:01:53.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.377+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc1a4f90 con 0x7f5bcc101f30 2023-09-10T01:01:53.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.377+0000 7f5bc2ffd700 1 -- 172.21.15.3:0/2299164703 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc017650 con 0x7f5bcc101f30 2023-09-10T01:01:53.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.377+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc1a53a0 con 0x7f5bcc101f30 2023-09-10T01:01:53.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.378+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bcc04eb50 con 0x7f5bcc101f30 2023-09-10T01:01:53.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.380+0000 7f5bc2ffd700 1 -- 172.21.15.3:0/2299164703 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f5bbc0225e0 con 0x7f5bcc101f30 2023-09-10T01:01:53.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.382+0000 7f5bc2ffd700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5bb80a06d0 0x7f5bb80a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:53.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.382+0000 7f5bc2ffd700 1 -- 172.21.15.3:0/2299164703 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5bbc0cdc50 con 0x7f5bcc101f30 2023-09-10T01:01:53.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.382+0000 7f5bd1e40700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5bb80a06d0 0x7f5bb80a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:53.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.383+0000 7f5bd1e40700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5bb80a06d0 0x7f5bb80a2b90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009c60 tx=0x7f5bc8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:53.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.384+0000 7f5bc2ffd700 1 -- 172.21.15.3:0/2299164703 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5bbc097340 con 0x7f5bcc101f30 2023-09-10T01:01:53.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.610+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5bcc1105f0 con 0x7f5bcc101f30 2023-09-10T01:01:53.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.611+0000 7f5bc2ffd700 1 -- 172.21.15.3:0/2299164703 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bbc09a960 con 0x7f5bcc101f30 2023-09-10T01:01:53.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5bb80a06d0 msgr2=0x7f5bb80a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:53.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5bb80a06d0 0x7f5bb80a2b90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009c60 tx=0x7f5bc8009260 comp rx=0 tx=0).stop 2023-09-10T01:01:53.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 msgr2=0x7f5bcc0ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:53.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc0ff7b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc000c00 tx=0x7f5bbc005740 comp rx=0 tx=0).stop 2023-09-10T01:01:53.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 shutdown_connections 2023-09-10T01:01:53.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5bb80a06d0 0x7f5bb80a2b90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bcc101610 0x7f5bcc0ff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5bcc10aa40 0x7f5bcc0ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 --2- 172.21.15.3:0/2299164703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bcc101f30 0x7f5bcc0ff7b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:53.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.613+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 >> 172.21.15.3:0/2299164703 conn(0x7f5bcc0faea0 msgr2=0x7f5bcc10cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:53.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.614+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 shutdown_connections 2023-09-10T01:01:53.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:52.614+0000 7f5bd2e42700 1 -- 172.21.15.3:0/2299164703 wait complete. 2023-09-10T01:01:53.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:53 smithi003 ceph-mon[97654]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:01:53.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:53 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2299164703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-09-10T01:01:53.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:53 smithi136 ceph-mon[133708]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:01:53.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:53 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2299164703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-09-10T01:01:53.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:53 smithi092 ceph-mon[134067]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:01:53.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:53 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2299164703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-09-10T01:01:53.903 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953527 got 137438953529 for osd.4 2023-09-10T01:01:53.904 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:54.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 -- 172.21.15.3:0/176609555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa658100970 msgr2=0x7fa65810cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:54.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 --2- 172.21.15.3:0/176609555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa658100970 0x7fa65810cb50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009990 tx=0x7fa64c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 -- 172.21.15.3:0/176609555 shutdown_connections 2023-09-10T01:01:54.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 --2- 172.21.15.3:0/176609555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa65810d090 0x7fa65810f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 --2- 172.21.15.3:0/176609555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa658100050 0x7fa658100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 --2- 172.21.15.3:0/176609555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa658100970 0x7fa65810cb50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 -- 172.21.15.3:0/176609555 >> 172.21.15.3:0/176609555 conn(0x7fa6580765f0 msgr2=0x7fa658076a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:54.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 -- 172.21.15.3:0/176609555 shutdown_connections 2023-09-10T01:01:54.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.223+0000 7fa65e43b700 1 -- 172.21.15.3:0/176609555 wait complete. 2023-09-10T01:01:54.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.224+0000 7fa65e43b700 1 Processor -- start 2023-09-10T01:01:54.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.224+0000 7fa65e43b700 1 -- start start 2023-09-10T01:01:54.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa65e43b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa658100050 0x7fa65819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:54.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa65e43b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa658100970 0x7fa65819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:54.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa65e43b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa65810d090 0x7fa6581988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:54.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa65e43b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658198f20 con 0x7fa65810d090 2023-09-10T01:01:54.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa65e43b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658199090 con 0x7fa658100050 2023-09-10T01:01:54.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa65e43b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa658199250 con 0x7fa658100970 2023-09-10T01:01:54.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa657fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa658100050 0x7fa65819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:54.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa657fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa658100050 0x7fa65819e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57062/0 (socket says 172.21.15.3:57062) 2023-09-10T01:01:54.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa657fff700 1 -- 172.21.15.3:0/327369334 learned_addr learned my addr 172.21.15.3:0/327369334 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:54.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa6577fe700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa658100970 0x7fa65819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:54.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.225+0000 7fa65c9d8700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa65810d090 0x7fa6581988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:54.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6577fe700 1 -- 172.21.15.3:0/327369334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa658100050 msgr2=0x7fa65819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:54.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6577fe700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa658100050 0x7fa65819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6577fe700 1 -- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa65810d090 msgr2=0x7fa6581988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:54.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6577fe700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa65810d090 0x7fa6581988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6577fe700 1 -- 172.21.15.3:0/327369334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0095f0 con 0x7fa658100970 2023-09-10T01:01:54.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6577fe700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa658100970 0x7fa65819ed70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009990 tx=0x7fa64c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:54.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6557fa700 1 -- 172.21.15.3:0/327369334 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c01d070 con 0x7fa658100970 2023-09-10T01:01:54.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6581994d0 con 0x7fa658100970 2023-09-10T01:01:54.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6557fa700 1 -- 172.21.15.3:0/327369334 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa64c021410 con 0x7fa658100970 2023-09-10T01:01:54.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa6557fa700 1 -- 172.21.15.3:0/327369334 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c017650 con 0x7fa658100970 2023-09-10T01:01:54.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.226+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6581a5820 con 0x7fa658100970 2023-09-10T01:01:54.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.228+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65804e300 con 0x7fa658100970 2023-09-10T01:01:54.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.229+0000 7fa6557fa700 1 -- 172.21.15.3:0/327369334 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fa64c021810 con 0x7fa658100970 2023-09-10T01:01:54.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.231+0000 7fa6557fa700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa6400a06d0 0x7fa6400a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:54.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.231+0000 7fa6557fa700 1 -- 172.21.15.3:0/327369334 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa64c0cdc80 con 0x7fa658100970 2023-09-10T01:01:54.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.231+0000 7fa657fff700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa6400a06d0 0x7fa6400a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:54.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.232+0000 7fa657fff700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa6400a06d0 0x7fa6400a2b90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa648009d20 tx=0x7fa648009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:54.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.233+0000 7fa6557fa700 1 -- 172.21.15.3:0/327369334 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa64c097370 con 0x7fa658100970 2023-09-10T01:01:54.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411391 got 77309411394 for osd.2 2023-09-10T01:01:54.465 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:54.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.511+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa6581128f0 con 0x7fa658100970 2023-09-10T01:01:54.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.512+0000 7fa6557fa700 1 -- 172.21.15.3:0/327369334 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa64c09a990 con 0x7fa658100970 2023-09-10T01:01:54.513 INFO:teuthology.orchestra.run.smithi003.stdout:193273528367 2023-09-10T01:01:54.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa6400a06d0 msgr2=0x7fa6400a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:54.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa6400a06d0 0x7fa6400a2b90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa648009d20 tx=0x7fa648009260 comp rx=0 tx=0).stop 2023-09-10T01:01:54.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa658100970 msgr2=0x7fa65819ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:54.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa658100970 0x7fa65819ed70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009990 tx=0x7fa64c009580 comp rx=0 tx=0).stop 2023-09-10T01:01:54.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 shutdown_connections 2023-09-10T01:01:54.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa65810d090 0x7fa6581988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa658100050 0x7fa65819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa6400a06d0 0x7fa6400a2b90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 --2- 172.21.15.3:0/327369334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa658100970 0x7fa65819ed70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:54.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 >> 172.21.15.3:0/327369334 conn(0x7fa6580765f0 msgr2=0x7fa65810e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:54.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 shutdown_connections 2023-09-10T01:01:54.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:54.515+0000 7fa65e43b700 1 -- 172.21.15.3:0/327369334 wait complete. 2023-09-10T01:01:55.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:55 smithi003 ceph-mon[97654]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-09-10T01:01:55.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/327369334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-09-10T01:01:55.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:55 smithi136 ceph-mon[133708]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-09-10T01:01:55.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/327369334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-09-10T01:01:55.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:55 smithi092 ceph-mon[134067]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-09-10T01:01:55.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/327369334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-09-10T01:01:55.762 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528365 got 193273528367 for osd.6 2023-09-10T01:01:55.762 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]: dispatch 2023-09-10T01:01:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2023-09-10T01:01:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2023-09-10T01:01:56.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]: dispatch 2023-09-10T01:01:56.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.26", "id": [1, 9]}]: dispatch 2023-09-10T01:01:56.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:56 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2023-09-10T01:01:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]: dispatch 2023-09-10T01:01:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2023-09-10T01:01:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2023-09-10T01:01:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]: dispatch 2023-09-10T01:01:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.26", "id": [1, 9]}]: dispatch 2023-09-10T01:01:56.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:56 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2023-09-10T01:01:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]: dispatch 2023-09-10T01:01:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2023-09-10T01:01:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2023-09-10T01:01:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]: dispatch 2023-09-10T01:01:56.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.26", "id": [1, 9]}]: dispatch 2023-09-10T01:01:56.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:56 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2023-09-10T01:01:56.645 INFO:teuthology.orchestra.run.smithi003.stdout:34359738444 2023-09-10T01:01:56.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.355+0000 7fa465143700 1 -- 172.21.15.3:0/4275324442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100fe0 msgr2=0x7fa46010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.355+0000 7fa465143700 1 --2- 172.21.15.3:0/4275324442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100fe0 0x7fa46010d4f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa45800b0a0 tx=0x7fa45800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.355+0000 7fa465143700 1 -- 172.21.15.3:0/4275324442 shutdown_connections 2023-09-10T01:01:56.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.355+0000 7fa465143700 1 --2- 172.21.15.3:0/4275324442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460106620 0x7fa460106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.355+0000 7fa465143700 1 --2- 172.21.15.3:0/4275324442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100fe0 0x7fa46010d4f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.355+0000 7fa465143700 1 --2- 172.21.15.3:0/4275324442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100620 0x7fa460100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.355+0000 7fa465143700 1 -- 172.21.15.3:0/4275324442 >> 172.21.15.3:0/4275324442 conn(0x7fa4600fc0f0 msgr2=0x7fa4600fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:56.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.356+0000 7fa465143700 1 -- 172.21.15.3:0/4275324442 shutdown_connections 2023-09-10T01:01:56.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.356+0000 7fa465143700 1 -- 172.21.15.3:0/4275324442 wait complete. 2023-09-10T01:01:56.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 Processor -- start 2023-09-10T01:01:56.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 -- start start 2023-09-10T01:01:56.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100620 0x7fa46019e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100fe0 0x7fa46019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460106620 0x7fa460198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460198e90 con 0x7fa460100fe0 2023-09-10T01:01:56.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460199000 con 0x7fa460100620 2023-09-10T01:01:56.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa465143700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4601991c0 con 0x7fa460106620 2023-09-10T01:01:56.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa45e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100fe0 0x7fa46019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.357+0000 7fa45e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100fe0 0x7fa46019ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60402/0 (socket says 172.21.15.3:60402) 2023-09-10T01:01:56.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45e59c700 1 -- 172.21.15.3:0/2569407050 learned_addr learned my addr 172.21.15.3:0/2569407050 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:56.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45ed9d700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100620 0x7fa46019e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45f59e700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460106620 0x7fa460198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45e59c700 1 -- 172.21.15.3:0/2569407050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460106620 msgr2=0x7fa460198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45e59c700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460106620 0x7fa460198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45e59c700 1 -- 172.21.15.3:0/2569407050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100620 msgr2=0x7fa46019e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45e59c700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100620 0x7fa46019e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45e59c700 1 -- 172.21.15.3:0/2569407050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa458009d00 con 0x7fa460100fe0 2023-09-10T01:01:56.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45ed9d700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100620 0x7fa46019e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:56.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa45e59c700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100fe0 0x7fa46019ece0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa45400e9a0 tx=0x7fa45400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa447fff700 1 -- 172.21.15.3:0/2569407050 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454009790 con 0x7fa460100fe0 2023-09-10T01:01:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa447fff700 1 -- 172.21.15.3:0/2569407050 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa454010430 con 0x7fa460100fe0 2023-09-10T01:01:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.358+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4601994a0 con 0x7fa460100fe0 2023-09-10T01:01:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.359+0000 7fa447fff700 1 -- 172.21.15.3:0/2569407050 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454018680 con 0x7fa460100fe0 2023-09-10T01:01:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.359+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4601a5790 con 0x7fa460100fe0 2023-09-10T01:01:56.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.360+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46004e300 con 0x7fa460100fe0 2023-09-10T01:01:56.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.362+0000 7fa447fff700 1 -- 172.21.15.3:0/2569407050 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fa4540105a0 con 0x7fa460100fe0 2023-09-10T01:01:56.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.363+0000 7fa447fff700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa4480a06d0 0x7fa4480a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.363+0000 7fa447fff700 1 -- 172.21.15.3:0/2569407050 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fa454014070 con 0x7fa460100fe0 2023-09-10T01:01:56.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.363+0000 7fa45ed9d700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa4480a06d0 0x7fa4480a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.364+0000 7fa45ed9d700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa4480a06d0 0x7fa4480a2b90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa450009fd0 tx=0x7fa450009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:56.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.367+0000 7fa447fff700 1 -- 172.21.15.3:0/2569407050 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa454051e10 con 0x7fa460100fe0 2023-09-10T01:01:56.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.616+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa460110680 con 0x7fa460100fe0 2023-09-10T01:01:56.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.617+0000 7fa447fff700 1 -- 172.21.15.3:0/2569407050 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa454095070 con 0x7fa460100fe0 2023-09-10T01:01:56.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.619+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa4480a06d0 msgr2=0x7fa4480a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.619+0000 7fa465143700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa4480a06d0 0x7fa4480a2b90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa450009fd0 tx=0x7fa450009260 comp rx=0 tx=0).stop 2023-09-10T01:01:56.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.619+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100fe0 msgr2=0x7fa46019ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.619+0000 7fa465143700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100fe0 0x7fa46019ece0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa45400e9a0 tx=0x7fa45400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.620+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 shutdown_connections 2023-09-10T01:01:56.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.620+0000 7fa465143700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa4480a06d0 0x7fa4480a2b90 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa450009fd0 tx=0x7fa450009260 comp rx=0 tx=0).stop 2023-09-10T01:01:56.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.620+0000 7fa465143700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460106620 0x7fa460198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.620+0000 7fa465143700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa460100620 0x7fa46019e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.620+0000 7fa465143700 1 --2- 172.21.15.3:0/2569407050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa460100fe0 0x7fa46019ece0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.620+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 >> 172.21.15.3:0/2569407050 conn(0x7fa4600fc0f0 msgr2=0x7fa4600fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:56.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.621+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 shutdown_connections 2023-09-10T01:01:56.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.621+0000 7fa465143700 1 -- 172.21.15.3:0/2569407050 wait complete. 2023-09-10T01:01:56.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.826+0000 7fa578328700 1 -- 172.21.15.3:0/2226022936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570102870 msgr2=0x7fa570102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.826+0000 7fa578328700 1 --2- 172.21.15.3:0/2226022936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570102870 0x7fa570102cf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa560009990 tx=0x7fa560009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.826+0000 7fa578328700 1 -- 172.21.15.3:0/2226022936 shutdown_connections 2023-09-10T01:01:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.826+0000 7fa578328700 1 --2- 172.21.15.3:0/2226022936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa570108870 0x7fa570108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.826+0000 7fa578328700 1 --2- 172.21.15.3:0/2226022936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa570103230 0x7fa57010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.826+0000 7fa578328700 1 --2- 172.21.15.3:0/2226022936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570102870 0x7fa570102cf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.826+0000 7fa578328700 1 -- 172.21.15.3:0/2226022936 >> 172.21.15.3:0/2226022936 conn(0x7fa5700fe340 msgr2=0x7fa570100760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:56.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.828+0000 7fa578328700 1 -- 172.21.15.3:0/2226022936 shutdown_connections 2023-09-10T01:01:56.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.829+0000 7fa578328700 1 -- 172.21.15.3:0/2226022936 wait complete. 2023-09-10T01:01:56.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.830+0000 7fa578328700 1 Processor -- start 2023-09-10T01:01:56.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.830+0000 7fa578328700 1 -- start start 2023-09-10T01:01:56.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa578328700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa570102870 0x7fa57010f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa578328700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570103230 0x7fa57010c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa578328700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57010c7a0 0x7fa57010cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa578328700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57010fa40 con 0x7fa57010c7a0 2023-09-10T01:01:56.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa578328700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5701a9750 con 0x7fa570103230 2023-09-10T01:01:56.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa578328700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5701a9890 con 0x7fa570102870 2023-09-10T01:01:56.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa5758c3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570103230 0x7fa57010c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa5758c3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570103230 0x7fa57010c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56744/0 (socket says 172.21.15.3:56744) 2023-09-10T01:01:56.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.831+0000 7fa5758c3700 1 -- 172.21.15.3:0/1032030343 learned_addr learned my addr 172.21.15.3:0/1032030343 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:56.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5760c4700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa570102870 0x7fa57010f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5768c5700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57010c7a0 0x7fa57010cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5758c3700 1 -- 172.21.15.3:0/1032030343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa570102870 msgr2=0x7fa57010f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5758c3700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa570102870 0x7fa57010f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5758c3700 1 -- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57010c7a0 msgr2=0x7fa57010cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:56.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5758c3700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57010c7a0 0x7fa57010cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5758c3700 1 -- 172.21.15.3:0/1032030343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5600095f0 con 0x7fa570103230 2023-09-10T01:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.832+0000 7fa5758c3700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570103230 0x7fa57010c260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa56000b5c0 tx=0x7fa560003c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.833+0000 7fa5677fe700 1 -- 172.21.15.3:0/1032030343 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56001d070 con 0x7fa570103230 2023-09-10T01:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.833+0000 7fa5677fe700 1 -- 172.21.15.3:0/1032030343 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa56000f460 con 0x7fa570103230 2023-09-10T01:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.833+0000 7fa5677fe700 1 -- 172.21.15.3:0/1032030343 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560017650 con 0x7fa570103230 2023-09-10T01:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.833+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5701a9ac0 con 0x7fa570103230 2023-09-10T01:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.833+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5701a9e80 con 0x7fa570103230 2023-09-10T01:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.835+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57004e300 con 0x7fa570103230 2023-09-10T01:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.837+0000 7fa5677fe700 1 -- 172.21.15.3:0/1032030343 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fa56000f5d0 con 0x7fa570103230 2023-09-10T01:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.839+0000 7fa5677fe700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa55c0a08a0 0x7fa55c0a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.839+0000 7fa5677fe700 1 -- 172.21.15.3:0/1032030343 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fa5600cd3d0 con 0x7fa570103230 2023-09-10T01:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.839+0000 7fa5760c4700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa55c0a08a0 0x7fa55c0a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.840+0000 7fa5760c4700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa55c0a08a0 0x7fa55c0a2d60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa570103d00 tx=0x7fa56c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:56.840+0000 7fa5677fe700 1 -- 172.21.15.3:0/1032030343 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa560096aa0 con 0x7fa570103230 2023-09-10T01:01:57.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.107+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fa57010d790 con 0x7fa570103230 2023-09-10T01:01:57.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.108+0000 7fa5677fe700 1 -- 172.21.15.3:0/1032030343 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa56009a0c0 con 0x7fa570103230 2023-09-10T01:01:57.109 INFO:teuthology.orchestra.run.smithi003.stdout:244813135912 2023-09-10T01:01:57.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa55c0a08a0 msgr2=0x7fa55c0a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa55c0a08a0 0x7fa55c0a2d60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa570103d00 tx=0x7fa56c008040 comp rx=0 tx=0).stop 2023-09-10T01:01:57.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570103230 msgr2=0x7fa57010c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570103230 0x7fa57010c260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa56000b5c0 tx=0x7fa560003c40 comp rx=0 tx=0).stop 2023-09-10T01:01:57.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 shutdown_connections 2023-09-10T01:01:57.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa55c0a08a0 0x7fa55c0a2d60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa570102870 0x7fa57010f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa57010c7a0 0x7fa57010cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 --2- 172.21.15.3:0/1032030343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa570103230 0x7fa57010c260 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 >> 172.21.15.3:0/1032030343 conn(0x7fa5700fe340 msgr2=0x7fa5700fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:57.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 shutdown_connections 2023-09-10T01:01:57.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.111+0000 7fa578328700 1 -- 172.21.15.3:0/1032030343 wait complete. 2023-09-10T01:01:57.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 -- 172.21.15.3:0/3607379956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488106770 msgr2=0x7f7488106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3607379956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488106770 0x7f7488106b50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7480009990 tx=0x7f7480009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 -- 172.21.15.3:0/3607379956 shutdown_connections 2023-09-10T01:01:57.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3607379956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488106770 0x7f7488106b50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3607379956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488068ae0 0x7f748810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3607379956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7488068120 0x7f74880685a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 -- 172.21.15.3:0/3607379956 >> 172.21.15.3:0/3607379956 conn(0x7f7488076600 msgr2=0x7f7488076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:57.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 -- 172.21.15.3:0/3607379956 shutdown_connections 2023-09-10T01:01:57.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.219+0000 7f748c9a7700 1 -- 172.21.15.3:0/3607379956 wait complete. 2023-09-10T01:01:57.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.221+0000 7f748c9a7700 1 Processor -- start 2023-09-10T01:01:57.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.221+0000 7f748c9a7700 1 -- start start 2023-09-10T01:01:57.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f748c9a7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488068120 0x7f748819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f748c9a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7488068ae0 0x7f748819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f748c9a7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488106770 0x7f7488196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f748c9a7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488196c00 con 0x7f7488068ae0 2023-09-10T01:01:57.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f748c9a7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488196d70 con 0x7f7488106770 2023-09-10T01:01:57.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f748c9a7700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488196f30 con 0x7f7488068120 2023-09-10T01:01:57.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f7485d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7488068ae0 0x7f748819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f7485d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7488068ae0 0x7f748819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60426/0 (socket says 172.21.15.3:60426) 2023-09-10T01:01:57.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f7485d9b700 1 -- 172.21.15.3:0/3482415852 learned_addr learned my addr 172.21.15.3:0/3482415852 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.222+0000 7f748659c700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488068120 0x7f748819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.223+0000 7f7486d9d700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488106770 0x7f7488196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.223+0000 7f748659c700 1 -- 172.21.15.3:0/3482415852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488106770 msgr2=0x7f7488196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.223+0000 7f748659c700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488106770 0x7f7488196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.223+0000 7f748659c700 1 -- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7488068ae0 msgr2=0x7f748819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.223+0000 7f748659c700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7488068ae0 0x7f748819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.223+0000 7f748659c700 1 -- 172.21.15.3:0/3482415852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7470009520 con 0x7f7488068120 2023-09-10T01:01:57.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.223+0000 7f748659c700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488068120 0x7f748819c510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7480009990 tx=0x7f7480009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:57.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.224+0000 7f747f7fe700 1 -- 172.21.15.3:0/3482415852 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748001d070 con 0x7f7488068120 2023-09-10T01:01:57.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.224+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74800095f0 con 0x7f7488068120 2023-09-10T01:01:57.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.224+0000 7f747f7fe700 1 -- 172.21.15.3:0/3482415852 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7480021410 con 0x7f7488068120 2023-09-10T01:01:57.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.224+0000 7f747f7fe700 1 -- 172.21.15.3:0/3482415852 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7480017650 con 0x7f7488068120 2023-09-10T01:01:57.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.224+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74880ff4e0 con 0x7f7488068120 2023-09-10T01:01:57.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.225+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748804e300 con 0x7f7488068120 2023-09-10T01:01:57.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.228+0000 7f747f7fe700 1 -- 172.21.15.3:0/3482415852 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f7480021810 con 0x7f7488068120 2023-09-10T01:01:57.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.230+0000 7f747f7fe700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f74740a07d0 0x7f74740a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.230+0000 7f747f7fe700 1 -- 172.21.15.3:0/3482415852 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f74800cda00 con 0x7f7488068120 2023-09-10T01:01:57.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.231+0000 7f7485d9b700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f74740a07d0 0x7f74740a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.231+0000 7f747f7fe700 1 -- 172.21.15.3:0/3482415852 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7480097050 con 0x7f7488068120 2023-09-10T01:01:57.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.232+0000 7f7485d9b700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f74740a07d0 0x7f74740a2c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f74880fdcd0 tx=0x7f7470009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:57.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-09-10T01:01:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]': finished 2023-09-10T01:01:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2023-09-10T01:01:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2023-09-10T01:01:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]': finished 2023-09-10T01:01:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.26", "id": [1, 9]}]': finished 2023-09-10T01:01:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2023-09-10T01:01:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: osdmap e89: 12 total, 12 up, 12 in 2023-09-10T01:01:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:57 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2569407050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-09-10T01:01:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 -- 172.21.15.3:0/189106332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ff400 msgr2=0x7ff3cc0ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 --2- 172.21.15.3:0/189106332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ff400 0x7ff3cc0ff7e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009990 tx=0x7ff3c8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 -- 172.21.15.3:0/189106332 shutdown_connections 2023-09-10T01:01:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 --2- 172.21.15.3:0/189106332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ff400 0x7ff3cc0ff7e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 --2- 172.21.15.3:0/189106332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc104810 0x7ff3cc104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 --2- 172.21.15.3:0/189106332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3cc0ffd20 0x7ff3cc1041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 -- 172.21.15.3:0/189106332 >> 172.21.15.3:0/189106332 conn(0x7ff3cc0faea0 msgr2=0x7ff3cc0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 -- 172.21.15.3:0/189106332 shutdown_connections 2023-09-10T01:01:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.311+0000 7ff3d287d700 1 -- 172.21.15.3:0/189106332 wait complete. 2023-09-10T01:01:57.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.312+0000 7ff3d287d700 1 Processor -- start 2023-09-10T01:01:57.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.312+0000 7ff3d287d700 1 -- start start 2023-09-10T01:01:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.313+0000 7ff3d287d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3cc0ff400 0x7ff3cc1a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.313+0000 7ff3d287d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ffd20 0x7ff3cc1a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.313+0000 7ff3d287d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc104810 0x7ff3cc19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.313+0000 7ff3d287d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc19d0a0 con 0x7ff3cc104810 2023-09-10T01:01:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.313+0000 7ff3d287d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc19d210 con 0x7ff3cc0ff400 2023-09-10T01:01:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.313+0000 7ff3d287d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc19d3d0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d187b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3cc0ff400 0x7ff3cc1a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ffd20 0x7ff3cc1a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ffd20 0x7ff3cc1a2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39878/0 (socket says 172.21.15.3:39878) 2023-09-10T01:01:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 -- 172.21.15.3:0/781528795 learned_addr learned my addr 172.21.15.3:0/781528795 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d207c700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc104810 0x7ff3cc19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d187b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3cc0ff400 0x7ff3cc1a29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56770/0 (socket says 172.21.15.3:56770) 2023-09-10T01:01:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 -- 172.21.15.3:0/781528795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3cc0ff400 msgr2=0x7ff3cc1a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3cc0ff400 0x7ff3cc1a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 -- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc104810 msgr2=0x7ff3cc19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc104810 0x7ff3cc19ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 -- 172.21.15.3:0/781528795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c80095f0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.314+0000 7ff3d107a700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ffd20 0x7ff3cc1a2ef0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00e9c0 tx=0x7ff3bc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.315+0000 7ff3c2ffd700 1 -- 172.21.15.3:0/781528795 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc0096e0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.315+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc19d6b0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.315+0000 7ff3c2ffd700 1 -- 172.21.15.3:0/781528795 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3bc00c7c0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.315+0000 7ff3c2ffd700 1 -- 172.21.15.3:0/781528795 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc0053c0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.315+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc1a99a0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.316+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc04eb50 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.318+0000 7ff3c2ffd700 1 -- 172.21.15.3:0/781528795 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7ff3bc004a30 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.320+0000 7ff3c2ffd700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff3b80a0750 0x7ff3b80a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.321+0000 7ff3c2ffd700 1 -- 172.21.15.3:0/781528795 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7ff3bc014070 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.321+0000 7ff3c2ffd700 1 -- 172.21.15.3:0/781528795 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff3bc095b70 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.322+0000 7ff3d187b700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff3b80a0750 0x7ff3b80a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.324+0000 7ff3d187b700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff3b80a0750 0x7ff3b80a2c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b2e0 tx=0x7ff3c8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-09-10T01:01:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]': finished 2023-09-10T01:01:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2023-09-10T01:01:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2023-09-10T01:01:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]': finished 2023-09-10T01:01:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.26", "id": [1, 9]}]': finished 2023-09-10T01:01:57.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2023-09-10T01:01:57.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: osdmap e89: 12 total, 12 up, 12 in 2023-09-10T01:01:57.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:57 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2569407050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-09-10T01:01:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 -- 172.21.15.3:0/668150271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683410aa40 msgr2=0x7f683410cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 --2- 172.21.15.3:0/668150271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683410aa40 0x7f683410cf00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f682c009f90 tx=0x7f682c00b330 comp rx=0 tx=0).stop 2023-09-10T01:01:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 -- 172.21.15.3:0/668150271 shutdown_connections 2023-09-10T01:01:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 --2- 172.21.15.3:0/668150271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6834101610 0x7f68341019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 --2- 172.21.15.3:0/668150271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683410aa40 0x7f683410cf00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 --2- 172.21.15.3:0/668150271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6834101f30 0x7f683410a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 -- 172.21.15.3:0/668150271 >> 172.21.15.3:0/668150271 conn(0x7f68340faea0 msgr2=0x7f68340fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 -- 172.21.15.3:0/668150271 shutdown_connections 2023-09-10T01:01:57.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.429+0000 7f683a1a2700 1 -- 172.21.15.3:0/668150271 wait complete. 2023-09-10T01:01:57.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.430+0000 7f683a1a2700 1 Processor -- start 2023-09-10T01:01:57.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.430+0000 7f683a1a2700 1 -- start start 2023-09-10T01:01:57.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683a1a2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6834101610 0x7f683419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683a1a2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6834101f30 0x7f683419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683a1a2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683410aa40 0x7f6834198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683a1a2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834198da0 con 0x7f6834101610 2023-09-10T01:01:57.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683a1a2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834198f10 con 0x7f6834101f30 2023-09-10T01:01:57.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683a1a2700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68341990d0 con 0x7f683410aa40 2023-09-10T01:01:57.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f68391a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6834101610 0x7f683419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f68399a1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683410aa40 0x7f6834198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683899f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6834101f30 0x7f683419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683899f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6834101f30 0x7f683419ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56794/0 (socket says 172.21.15.3:56794) 2023-09-10T01:01:57.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f683899f700 1 -- 172.21.15.3:0/2705876294 learned_addr learned my addr 172.21.15.3:0/2705876294 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:57.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f68391a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6834101610 0x7f683419e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60462/0 (socket says 172.21.15.3:60462) 2023-09-10T01:01:57.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.431+0000 7f68399a1700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683410aa40 0x7f6834198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39898/0 (socket says 172.21.15.3:39898) 2023-09-10T01:01:57.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f683899f700 1 -- 172.21.15.3:0/2705876294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683410aa40 msgr2=0x7f6834198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f683899f700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683410aa40 0x7f6834198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f683899f700 1 -- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6834101610 msgr2=0x7f683419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f683899f700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6834101610 0x7f683419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f683899f700 1 -- 172.21.15.3:0/2705876294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c009c40 con 0x7f6834101f30 2023-09-10T01:01:57.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f683899f700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6834101f30 0x7f683419ebf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f682400e7e0 tx=0x7f682400eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:57.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f682a7fc700 1 -- 172.21.15.3:0/2705876294 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824010900 con 0x7f6834101f30 2023-09-10T01:01:57.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f682a7fc700 1 -- 172.21.15.3:0/2705876294 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6824018420 con 0x7f6834101f30 2023-09-10T01:01:57.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68341993b0 con 0x7f6834101f30 2023-09-10T01:01:57.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.432+0000 7f682a7fc700 1 -- 172.21.15.3:0/2705876294 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682400f680 con 0x7f6834101f30 2023-09-10T01:01:57.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.433+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68341a56a0 con 0x7f6834101f30 2023-09-10T01:01:57.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.434+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6834107f70 con 0x7f6834101f30 2023-09-10T01:01:57.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.438+0000 7f682a7fc700 1 -- 172.21.15.3:0/2705876294 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f6824018590 con 0x7f6834101f30 2023-09-10T01:01:57.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.440+0000 7f682a7fc700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68200a0750 0x7f68200a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:57.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.440+0000 7f682a7fc700 1 -- 172.21.15.3:0/2705876294 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f6824014070 con 0x7f6834101f30 2023-09-10T01:01:57.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.440+0000 7f68391a0700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68200a0750 0x7f68200a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:57.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.441+0000 7f682a7fc700 1 -- 172.21.15.3:0/2705876294 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f68240cc480 con 0x7f6834101f30 2023-09-10T01:01:57.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.441+0000 7f68391a0700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68200a0750 0x7f68200a2c10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6830009c60 tx=0x7f6830009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-09-10T01:01:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]': finished 2023-09-10T01:01:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2023-09-10T01:01:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2023-09-10T01:01:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]': finished 2023-09-10T01:01:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.26", "id": [1, 9]}]': finished 2023-09-10T01:01:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2023-09-10T01:01:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: osdmap e89: 12 total, 12 up, 12 in 2023-09-10T01:01:57.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:57 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2569407050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-09-10T01:01:57.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.505+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7488110600 con 0x7f7488068120 2023-09-10T01:01:57.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.506+0000 7f747f7fe700 1 -- 172.21.15.3:0/3482415852 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f748009a670 con 0x7f7488068120 2023-09-10T01:01:57.507 INFO:teuthology.orchestra.run.smithi003.stdout:158913790005 2023-09-10T01:01:57.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f74740a07d0 msgr2=0x7f74740a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f74740a07d0 0x7f74740a2c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f74880fdcd0 tx=0x7f7470009260 comp rx=0 tx=0).stop 2023-09-10T01:01:57.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488068120 msgr2=0x7f748819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488068120 0x7f748819c510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7480009990 tx=0x7f7480009580 comp rx=0 tx=0).stop 2023-09-10T01:01:57.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 shutdown_connections 2023-09-10T01:01:57.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f74740a07d0 0x7f74740a2c90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7488068120 0x7f748819c510 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7488106770 0x7f7488196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 --2- 172.21.15.3:0/3482415852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7488068ae0 0x7f748819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 >> 172.21.15.3:0/3482415852 conn(0x7f7488076600 msgr2=0x7f74880fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:57.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.508+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 shutdown_connections 2023-09-10T01:01:57.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.509+0000 7f748c9a7700 1 -- 172.21.15.3:0/3482415852 wait complete. 2023-09-10T01:01:57.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.595+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7ff3cc0008d0 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.596+0000 7ff3c2ffd700 1 -- 172.21.15.3:0/781528795 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7ff3bc099190 con 0x7ff3cc0ffd20 2023-09-10T01:01:57.597 INFO:teuthology.orchestra.run.smithi003.stdout:300647710749 2023-09-10T01:01:57.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.598+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff3b80a0750 msgr2=0x7ff3b80a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.598+0000 7ff3d287d700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff3b80a0750 0x7ff3b80a2c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b2e0 tx=0x7ff3c8005fd0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.598+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ffd20 msgr2=0x7ff3cc1a2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ffd20 0x7ff3cc1a2ef0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00e9c0 tx=0x7ff3bc00ecd0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 shutdown_connections 2023-09-10T01:01:57.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc104810 0x7ff3cc19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3cc0ff400 0x7ff3cc1a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff3b80a0750 0x7ff3b80a2c10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 --2- 172.21.15.3:0/781528795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3cc0ffd20 0x7ff3cc1a2ef0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 >> 172.21.15.3:0/781528795 conn(0x7ff3cc0faea0 msgr2=0x7ff3cc102310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:57.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 shutdown_connections 2023-09-10T01:01:57.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.599+0000 7ff3d287d700 1 -- 172.21.15.3:0/781528795 wait complete. 2023-09-10T01:01:57.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.678+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f683404eb50 con 0x7f6834101f30 2023-09-10T01:01:57.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.678+0000 7f682a7fc700 1 -- 172.21.15.3:0/2705876294 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6824095870 con 0x7f6834101f30 2023-09-10T01:01:57.679 INFO:teuthology.orchestra.run.smithi003.stdout:55834574919 2023-09-10T01:01:57.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68200a0750 msgr2=0x7f68200a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68200a0750 0x7f68200a2c10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6830009c60 tx=0x7f6830009260 comp rx=0 tx=0).stop 2023-09-10T01:01:57.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6834101f30 msgr2=0x7f683419ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:57.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6834101f30 0x7f683419ebf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f682400e7e0 tx=0x7f682400eaf0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 shutdown_connections 2023-09-10T01:01:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f68200a0750 0x7f68200a2c10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683410aa40 0x7f6834198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6834101610 0x7f683419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 --2- 172.21.15.3:0/2705876294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6834101f30 0x7f683419ebf0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:57.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.681+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 >> 172.21.15.3:0/2705876294 conn(0x7f68340faea0 msgr2=0x7f683410cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:57.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.682+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 shutdown_connections 2023-09-10T01:01:57.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:57.682+0000 7f683a1a2700 1 -- 172.21.15.3:0/2705876294 wait complete. 2023-09-10T01:01:58.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:58 smithi003 ceph-mon[97654]: osdmap e90: 12 total, 12 up, 12 in 2023-09-10T01:01:58.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:58 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1032030343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-09-10T01:01:58.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:58 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3482415852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-09-10T01:01:58.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:58 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/781528795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-09-10T01:01:58.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:58 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2705876294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-09-10T01:01:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:58 smithi136 ceph-mon[133708]: osdmap e90: 12 total, 12 up, 12 in 2023-09-10T01:01:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:58 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1032030343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-09-10T01:01:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:58 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3482415852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-09-10T01:01:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:58 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/781528795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-09-10T01:01:58.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:58 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2705876294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-09-10T01:01:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:58 smithi092 ceph-mon[134067]: osdmap e90: 12 total, 12 up, 12 in 2023-09-10T01:01:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:58 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1032030343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-09-10T01:01:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:58 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3482415852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-09-10T01:01:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:58 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/781528795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-09-10T01:01:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:58 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2705876294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-09-10T01:01:58.627 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790002 got 158913790005 for osd.5 2023-09-10T01:01:58.627 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:58.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738441 got 34359738444 for osd.0 2023-09-10T01:01:58.833 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:59.159 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574917 got 55834574919 for osd.1 2023-09-10T01:01:59.159 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:59.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710747 got 300647710749 for osd.10 2023-09-10T01:01:59.206 DEBUG:teuthology.parallel:result is None 2023-09-10T01:01:59.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:01:59 smithi003 ceph-mon[97654]: pgmap v321: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:01:59.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/2046235038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8100fe0 msgr2=0x7f1bc810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/2046235038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8100fe0 0x7f1bc810d4f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000b0a0 tx=0x7f1bc000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/2046235038 shutdown_connections 2023-09-10T01:01:59.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/2046235038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8100fe0 0x7f1bc810d4f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/2046235038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8106620 0x7f1bc8106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/2046235038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bc8100620 0x7f1bc8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/2046235038 >> 172.21.15.3:0/2046235038 conn(0x7f1bc80fc0f0 msgr2=0x7f1bc80fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:59.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.396+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/2046235038 shutdown_connections 2023-09-10T01:01:59.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.397+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/2046235038 wait complete. 2023-09-10T01:01:59.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 Processor -- start 2023-09-10T01:01:59.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 -- start start 2023-09-10T01:01:59.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8100620 0x7f1bc8199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bc8100fe0 0x7f1bc8199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8106620 0x7f1bc810a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc810a8c0 con 0x7f1bc8106620 2023-09-10T01:01:59.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc810aa30 con 0x7f1bc8100620 2023-09-10T01:01:59.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.398+0000 7f1bcf2d5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc810abf0 con 0x7f1bc8100fe0 2023-09-10T01:01:59.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd872700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8106620 0x7f1bc810a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8100620 0x7f1bc8199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8100620 0x7f1bc8199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56808/0 (socket says 172.21.15.3:56808) 2023-09-10T01:01:59.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 -- 172.21.15.3:0/1949786511 learned_addr learned my addr 172.21.15.3:0/1949786511 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:59.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcc870700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bc8100fe0 0x7f1bc8199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 -- 172.21.15.3:0/1949786511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bc8100fe0 msgr2=0x7f1bc8199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bc8100fe0 0x7f1bc8199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 -- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8106620 msgr2=0x7f1bc810a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8106620 0x7f1bc810a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 -- 172.21.15.3:0/1949786511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc0009d00 con 0x7f1bc8100620 2023-09-10T01:01:59.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.399+0000 7f1bcd071700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8100620 0x7f1bc8199410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400e9a0 tx=0x7f1bc400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:59.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.400+0000 7f1bbe7fc700 1 -- 172.21.15.3:0/1949786511 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4009790 con 0x7f1bc8100620 2023-09-10T01:01:59.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.400+0000 7f1bbe7fc700 1 -- 172.21.15.3:0/1949786511 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bc4010430 con 0x7f1bc8100620 2023-09-10T01:01:59.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.400+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc810aed0 con 0x7f1bc8100620 2023-09-10T01:01:59.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.400+0000 7f1bbe7fc700 1 -- 172.21.15.3:0/1949786511 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4018680 con 0x7f1bc8100620 2023-09-10T01:01:59.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.400+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc819d470 con 0x7f1bc8100620 2023-09-10T01:01:59.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.401+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc804e300 con 0x7f1bc8100620 2023-09-10T01:01:59.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.403+0000 7f1bbe7fc700 1 -- 172.21.15.3:0/1949786511 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f1bc40105a0 con 0x7f1bc8100620 2023-09-10T01:01:59.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.405+0000 7f1bbe7fc700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bb40a06d0 0x7f1bb40a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.405+0000 7f1bbe7fc700 1 -- 172.21.15.3:0/1949786511 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f1bc4014070 con 0x7f1bc8100620 2023-09-10T01:01:59.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.405+0000 7f1bcc870700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bb40a06d0 0x7f1bb40a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.406+0000 7f1bcc870700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bb40a06d0 0x7f1bb40a2b90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8105d70 tx=0x7f1bb80073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:59.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.406+0000 7f1bbe7fc700 1 -- 172.21.15.3:0/1949786511 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1bc4095af0 con 0x7f1bc8100620 2023-09-10T01:01:59.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:01:59 smithi136 ceph-mon[133708]: pgmap v321: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:01:59.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:01:59 smithi092 ceph-mon[134067]: pgmap v321: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:01:59.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.632+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f1bc810b620 con 0x7f1bc8100620 2023-09-10T01:01:59.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.633+0000 7f1bbe7fc700 1 -- 172.21.15.3:0/1949786511 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1bc4099110 con 0x7f1bc8100620 2023-09-10T01:01:59.635 INFO:teuthology.orchestra.run.smithi003.stdout:279172874275 2023-09-10T01:01:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bb40a06d0 msgr2=0x7f1bb40a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bb40a06d0 0x7f1bb40a2b90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8105d70 tx=0x7f1bb80073d0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8100620 msgr2=0x7f1bc8199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8100620 0x7f1bc8199410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400e9a0 tx=0x7f1bc400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 shutdown_connections 2023-09-10T01:01:59.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc8106620 0x7f1bc810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bc8100620 0x7f1bc8199410 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1bb40a06d0 0x7f1bb40a2b90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 --2- 172.21.15.3:0/1949786511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bc8100fe0 0x7f1bc8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 >> 172.21.15.3:0/1949786511 conn(0x7f1bc80fc0f0 msgr2=0x7f1bc810b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:59.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 shutdown_connections 2023-09-10T01:01:59.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.637+0000 7f1bcf2d5700 1 -- 172.21.15.3:0/1949786511 wait complete. 2023-09-10T01:01:59.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 -- 172.21.15.3:0/560721310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 msgr2=0x7ff058106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/560721310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 0x7ff058106a00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff048009990 tx=0x7ff048009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 -- 172.21.15.3:0/560721310 shutdown_connections 2023-09-10T01:01:59.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/560721310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff058100fe0 0x7ff05810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/560721310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 0x7ff058106a00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/560721310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 0x7ff058100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 -- 172.21.15.3:0/560721310 >> 172.21.15.3:0/560721310 conn(0x7ff0580fc0f0 msgr2=0x7ff0580fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:59.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 -- 172.21.15.3:0/560721310 shutdown_connections 2023-09-10T01:01:59.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.864+0000 7ff05d2f6700 1 -- 172.21.15.3:0/560721310 wait complete. 2023-09-10T01:01:59.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.865+0000 7ff05d2f6700 1 Processor -- start 2023-09-10T01:01:59.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.866+0000 7ff05d2f6700 1 -- start start 2023-09-10T01:01:59.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.866+0000 7ff05d2f6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 0x7ff05819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.866+0000 7ff05d2f6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff058100fe0 0x7ff05819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.866+0000 7ff05d2f6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 0x7ff0581965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.866+0000 7ff05d2f6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058196c10 con 0x7ff058106620 2023-09-10T01:01:59.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.866+0000 7ff05d2f6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058196d80 con 0x7ff058100620 2023-09-10T01:01:59.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.866+0000 7ff05d2f6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058196f40 con 0x7ff058100fe0 2023-09-10T01:01:59.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff0577fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 0x7ff0581965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 0x7ff05819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 0x7ff05819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56814/0 (socket says 172.21.15.3:56814) 2023-09-10T01:01:59.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 -- 172.21.15.3:0/1128924857 learned_addr learned my addr 172.21.15.3:0/1128924857 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:59.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff0567fc700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff058100fe0 0x7ff05819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 -- 172.21.15.3:0/1128924857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff058100fe0 msgr2=0x7ff05819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff058100fe0 0x7ff05819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 -- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 msgr2=0x7ff0581965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 0x7ff0581965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 -- 172.21.15.3:0/1128924857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0480095f0 con 0x7ff058100620 2023-09-10T01:01:59.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.867+0000 7ff056ffd700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 0x7ff05819c520 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff048005910 tx=0x7ff048004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:59.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.868+0000 7ff03ffff700 1 -- 172.21.15.3:0/1128924857 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04801d070 con 0x7ff058100620 2023-09-10T01:01:59.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.868+0000 7ff03ffff700 1 -- 172.21.15.3:0/1128924857 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff048022470 con 0x7ff058100620 2023-09-10T01:01:59.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.868+0000 7ff03ffff700 1 -- 172.21.15.3:0/1128924857 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048017650 con 0x7ff058100620 2023-09-10T01:01:59.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.868+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0581971c0 con 0x7ff058100620 2023-09-10T01:01:59.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.868+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0581a3510 con 0x7ff058100620 2023-09-10T01:01:59.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.869+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05804e300 con 0x7ff058100620 2023-09-10T01:01:59.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.871+0000 7ff03ffff700 1 -- 172.21.15.3:0/1128924857 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7ff04800fad0 con 0x7ff058100620 2023-09-10T01:01:59.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.872+0000 7ff03ffff700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff0400a06d0 0x7ff0400a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.873+0000 7ff03ffff700 1 -- 172.21.15.3:0/1128924857 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7ff0480cda30 con 0x7ff058100620 2023-09-10T01:01:59.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.873+0000 7ff0567fc700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff0400a06d0 0x7ff0400a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.873+0000 7ff0567fc700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff0400a06d0 0x7ff0400a2b90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff0580fe9f0 tx=0x7ff04c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:59.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.874+0000 7ff03ffff700 1 -- 172.21.15.3:0/1128924857 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff048097080 con 0x7ff058100620 2023-09-10T01:01:59.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 -- 172.21.15.3:0/3524154487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 msgr2=0x7fd964072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 --2- 172.21.15.3:0/3524154487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964072db0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd954009990 tx=0x7fd954009ca0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 -- 172.21.15.3:0/3524154487 shutdown_connections 2023-09-10T01:01:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 --2- 172.21.15.3:0/3524154487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd96410cec0 0x7fd96410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 --2- 172.21.15.3:0/3524154487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964072db0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 --2- 172.21.15.3:0/3524154487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9640732f0 0x7fd96410c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 -- 172.21.15.3:0/3524154487 >> 172.21.15.3:0/3524154487 conn(0x7fd9640fc0f0 msgr2=0x7fd9640fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:01:59.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 -- 172.21.15.3:0/3524154487 shutdown_connections 2023-09-10T01:01:59.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.967+0000 7fd9698a4700 1 -- 172.21.15.3:0/3524154487 wait complete. 2023-09-10T01:01:59.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.968+0000 7fd9698a4700 1 Processor -- start 2023-09-10T01:01:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.968+0000 7fd9698a4700 1 -- start start 2023-09-10T01:01:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9698a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9698a4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9640732f0 0x7fd964198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9698a4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd96410cec0 0x7fd96419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9698a4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96419d910 con 0x7fd9640729d0 2023-09-10T01:01:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9698a4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96419da80 con 0x7fd9640732f0 2023-09-10T01:01:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9698a4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96419dc40 con 0x7fd96410cec0 2023-09-10T01:01:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd962ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9627fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9640732f0 0x7fd964198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd962ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60524/0 (socket says 172.21.15.3:60524) 2023-09-10T01:01:59.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd962ffd700 1 -- 172.21.15.3:0/2231679049 learned_addr learned my addr 172.21.15.3:0/2231679049 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:01:59.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.969+0000 7fd9637fe700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd96410cec0 0x7fd96419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9627fc700 1 -- 172.21.15.3:0/2231679049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd96410cec0 msgr2=0x7fd96419d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9627fc700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd96410cec0 0x7fd96419d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9627fc700 1 -- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 msgr2=0x7fd964198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:01:59.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9627fc700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:01:59.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9627fc700 1 -- 172.21.15.3:0/2231679049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540095f0 con 0x7fd9640732f0 2023-09-10T01:01:59.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd962ffd700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:01:59.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9627fc700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9640732f0 0x7fd964198ea0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd95800e9f0 tx=0x7fd95800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9688a2700 1 -- 172.21.15.3:0/2231679049 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95800c9c0 con 0x7fd9640732f0 2023-09-10T01:01:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9688a2700 1 -- 172.21.15.3:0/2231679049 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd958010430 con 0x7fd9640732f0 2023-09-10T01:01:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96419df20 con 0x7fd9640732f0 2023-09-10T01:01:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.970+0000 7fd9688a2700 1 -- 172.21.15.3:0/2231679049 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958018680 con 0x7fd9640732f0 2023-09-10T01:01:59.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.971+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9641a57c0 con 0x7fd9640732f0 2023-09-10T01:01:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.972+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96404e300 con 0x7fd9640732f0 2023-09-10T01:01:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.973+0000 7fd9688a2700 1 -- 172.21.15.3:0/2231679049 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fd958010860 con 0x7fd9640732f0 2023-09-10T01:01:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.975+0000 7fd9688a2700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd94c0a0750 0x7fd94c0a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:01:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.975+0000 7fd9688a2700 1 -- 172.21.15.3:0/2231679049 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fd958014070 con 0x7fd9640732f0 2023-09-10T01:01:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.975+0000 7fd962ffd700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd94c0a0750 0x7fd94c0a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:01:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.976+0000 7fd962ffd700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd94c0a0750 0x7fd94c0a2c10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd95400b6f0 tx=0x7fd954005bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:01:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:01:59.979+0000 7fd9688a2700 1 -- 172.21.15.3:0/2231679049 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd958051e10 con 0x7fd9640732f0 2023-09-10T01:02:00.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.109+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7ff058197350 con 0x7ff058100620 2023-09-10T01:02:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.110+0000 7ff03ffff700 1 -- 172.21.15.3:0/1128924857 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7ff04809a6a0 con 0x7ff058100620 2023-09-10T01:02:00.112 INFO:teuthology.orchestra.run.smithi003.stdout:330712481819 2023-09-10T01:02:00.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff0400a06d0 msgr2=0x7ff0400a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:00.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff0400a06d0 0x7ff0400a2b90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff0580fe9f0 tx=0x7ff04c009260 comp rx=0 tx=0).stop 2023-09-10T01:02:00.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 msgr2=0x7ff05819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:00.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 0x7ff05819c520 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff048005910 tx=0x7ff048004730 comp rx=0 tx=0).stop 2023-09-10T01:02:00.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 shutdown_connections 2023-09-10T01:02:00.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff058106620 0x7ff0581965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff058100620 0x7ff05819c520 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff0400a06d0 0x7ff0400a2b90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 --2- 172.21.15.3:0/1128924857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff058100fe0 0x7ff05819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 >> 172.21.15.3:0/1128924857 conn(0x7ff0580fc0f0 msgr2=0x7ff0580fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:00.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 shutdown_connections 2023-09-10T01:02:00.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.113+0000 7ff05d2f6700 1 -- 172.21.15.3:0/1128924857 wait complete. 2023-09-10T01:02:00.215 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135909 got 244813135912 for osd.8 2023-09-10T01:02:00.215 DEBUG:teuthology.parallel:result is None 2023-09-10T01:02:00.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.242+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd964112790 con 0x7fd9640732f0 2023-09-10T01:02:00.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.243+0000 7fd9688a2700 1 -- 172.21.15.3:0/2231679049 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd958095800 con 0x7fd9640732f0 2023-09-10T01:02:00.244 INFO:teuthology.orchestra.run.smithi003.stdout:214748364845 2023-09-10T01:02:00.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd94c0a0750 msgr2=0x7fd94c0a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:00.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd94c0a0750 0x7fd94c0a2c10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd95400b6f0 tx=0x7fd954005bc0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9640732f0 msgr2=0x7fd964198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:00.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9640732f0 0x7fd964198ea0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd95800e9f0 tx=0x7fd95800ed00 comp rx=0 tx=0).stop 2023-09-10T01:02:00.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 shutdown_connections 2023-09-10T01:02:00.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd94c0a0750 0x7fd94c0a2c10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd96410cec0 0x7fd96419d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9640729d0 0x7fd964198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 --2- 172.21.15.3:0/2231679049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9640732f0 0x7fd964198ea0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:00.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.245+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 >> 172.21.15.3:0/2231679049 conn(0x7fd9640fc0f0 msgr2=0x7fd96410f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:00.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.246+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 shutdown_connections 2023-09-10T01:02:00.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:00.246+0000 7fd9698a4700 1 -- 172.21.15.3:0/2231679049 wait complete. 2023-09-10T01:02:00.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:00 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1949786511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-09-10T01:02:00.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:00 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1949786511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-09-10T01:02:00.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:00 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1949786511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-09-10T01:02:00.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481816 got 330712481819 for osd.11 2023-09-10T01:02:00.936 DEBUG:teuthology.parallel:result is None 2023-09-10T01:02:00.938 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874272 got 279172874275 for osd.9 2023-09-10T01:02:00.938 DEBUG:teuthology.parallel:result is None 2023-09-10T01:02:01.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:01 smithi003 ceph-mon[97654]: pgmap v322: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:01.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1128924857' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-09-10T01:02:01.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2231679049' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-09-10T01:02:01.324 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364842 got 214748364845 for osd.7 2023-09-10T01:02:01.325 DEBUG:teuthology.parallel:result is None 2023-09-10T01:02:01.325 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-09-10T01:02:01.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph pg dump --format=json 2023-09-10T01:02:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:01 smithi136 ceph-mon[133708]: pgmap v322: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:01 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1128924857' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-09-10T01:02:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:01 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2231679049' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-09-10T01:02:01.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:01 smithi092 ceph-mon[134067]: pgmap v322: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:01.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:01 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1128924857' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-09-10T01:02:01.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:01 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2231679049' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-09-10T01:02:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:02.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:02 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:02 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:02 smithi003 ceph-mon[97654]: pgmap v323: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2023-09-10T01:02:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:02 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:02:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:02 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:02:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:02 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:02 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:02 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:02 smithi136 ceph-mon[133708]: pgmap v323: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2023-09-10T01:02:02.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:02 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:02:02.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:02 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:02:02.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:02 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:02 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:02 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:02.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:02 smithi092 ceph-mon[134067]: pgmap v323: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2023-09-10T01:02:02.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:02 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:02:02.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:02 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:02:02.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:02 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' 2023-09-10T01:02:03.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 -- 172.21.15.3:0/4060041690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c100fe0 msgr2=0x7f819c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 --2- 172.21.15.3:0/4060041690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c100fe0 0x7f819c10d4f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8194009f90 tx=0x7f819400b330 comp rx=0 tx=0).stop 2023-09-10T01:02:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 -- 172.21.15.3:0/4060041690 shutdown_connections 2023-09-10T01:02:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 --2- 172.21.15.3:0/4060041690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c100fe0 0x7f819c10d4f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 --2- 172.21.15.3:0/4060041690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f819c106620 0x7f819c106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 --2- 172.21.15.3:0/4060041690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f819c100620 0x7f819c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 -- 172.21.15.3:0/4060041690 >> 172.21.15.3:0/4060041690 conn(0x7f819c0fc0f0 msgr2=0x7f819c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 -- 172.21.15.3:0/4060041690 shutdown_connections 2023-09-10T01:02:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.684+0000 7f81a40f6700 1 -- 172.21.15.3:0/4060041690 wait complete. 2023-09-10T01:02:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.685+0000 7f81a40f6700 1 Processor -- start 2023-09-10T01:02:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a40f6700 1 -- start start 2023-09-10T01:02:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a40f6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f819c100620 0x7f819c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a40f6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f819c100fe0 0x7f819c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a40f6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c106620 0x7f819c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a40f6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c198eb0 con 0x7f819c106620 2023-09-10T01:02:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a40f6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c199020 con 0x7f819c100fe0 2023-09-10T01:02:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a40f6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1991e0 con 0x7f819c100620 2023-09-10T01:02:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.686+0000 7f81a2693700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c106620 0x7f819c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a1691700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f819c100fe0 0x7f819c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a1e92700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f819c100620 0x7f819c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a1691700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f819c100fe0 0x7f819c19ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:56838/0 (socket says 172.21.15.3:56838) 2023-09-10T01:02:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a1691700 1 -- 172.21.15.3:0/2594229096 learned_addr learned my addr 172.21.15.3:0/2594229096 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a2693700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c106620 0x7f819c198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60544/0 (socket says 172.21.15.3:60544) 2023-09-10T01:02:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a2693700 1 -- 172.21.15.3:0/2594229096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f819c100620 msgr2=0x7f819c19e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a2693700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f819c100620 0x7f819c19e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a2693700 1 -- 172.21.15.3:0/2594229096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f819c100fe0 msgr2=0x7f819c19ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a2693700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f819c100fe0 0x7f819c19ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a2693700 1 -- 172.21.15.3:0/2594229096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c0095f0 con 0x7f819c106620 2023-09-10T01:02:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a1e92700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f819c100620 0x7f819c19e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f81a2693700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c106620 0x7f819c198840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f819400b6c0 tx=0x7f8194009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.687+0000 7f8192ffd700 1 -- 172.21.15.3:0/2594229096 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819400e040 con 0x7f819c106620 2023-09-10T01:02:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.688+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8194009c40 con 0x7f819c106620 2023-09-10T01:02:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.688+0000 7f8192ffd700 1 -- 172.21.15.3:0/2594229096 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f819401d470 con 0x7f819c106620 2023-09-10T01:02:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.688+0000 7f8192ffd700 1 -- 172.21.15.3:0/2594229096 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194005280 con 0x7f819c106620 2023-09-10T01:02:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.688+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c1a55e0 con 0x7f819c106620 2023-09-10T01:02:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.689+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c04e300 con 0x7f819c106620 2023-09-10T01:02:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.691+0000 7f8192ffd700 1 -- 172.21.15.3:0/2594229096 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f819401d5e0 con 0x7f819c106620 2023-09-10T01:02:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.692+0000 7f8192ffd700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81880a06d0 0x7f81880a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.692+0000 7f8192ffd700 1 -- 172.21.15.3:0/2594229096 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f81940cd790 con 0x7f819c106620 2023-09-10T01:02:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.693+0000 7f81a1e92700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81880a06d0 0x7f81880a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.693+0000 7f81a1e92700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81880a06d0 0x7f81880a2b90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8198009fd0 tx=0x7f8198009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.694+0000 7f8192ffd700 1 -- 172.21.15.3:0/2594229096 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f8194096de0 con 0x7f819c106620 2023-09-10T01:02:03.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.900+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f819c110680 con 0x7f81880a06d0 2023-09-10T01:02:03.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.914+0000 7f8192ffd700 1 -- 172.21.15.3:0/2594229096 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427656 (secure 0 0 0) 0x7f819c110680 con 0x7f81880a06d0 2023-09-10T01:02:03.923 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:03.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81880a06d0 msgr2=0x7f81880a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:03.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81880a06d0 0x7f81880a2b90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8198009fd0 tx=0x7f8198009260 comp rx=0 tx=0).stop 2023-09-10T01:02:03.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c106620 msgr2=0x7f819c198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:03.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c106620 0x7f819c198840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f819400b6c0 tx=0x7f8194009490 comp rx=0 tx=0).stop 2023-09-10T01:02:03.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 shutdown_connections 2023-09-10T01:02:03.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f81880a06d0 0x7f81880a2b90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f819c100620 0x7f819c19e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f819c106620 0x7f819c198840 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 --2- 172.21.15.3:0/2594229096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f819c100fe0 0x7f819c19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:03.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 >> 172.21.15.3:0/2594229096 conn(0x7f819c0fc0f0 msgr2=0x7f819c10b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:03.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 shutdown_connections 2023-09-10T01:02:03.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:03.920+0000 7f81a40f6700 1 -- 172.21.15.3:0/2594229096 wait complete. 2023-09-10T01:02:03.929 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-09-10T01:02:04.371 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2023-09-10T01:02:03.868314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654503,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":478,"num_read_kb":358,"num_write":335,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336312,"kb_used_data":12052,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605512,"statfs":{"total":1151940427776,"available":1151596044288,"internally_reserved":0,"allocated":12341248,"data_stored":7983004,"data_compressed":391800,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002133"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.128038+0000","last_change":"2023-09-10T01:00:44.026223+0000","last_active":"2023-09-10T01:01:56.128038+0000","last_peered":"2023-09-10T01:01:56.128038+0000","last_clean":"2023-09-10T01:01:56.128038+0000","last_became_active":"2023-09-10T01:00:44.025731+0000","last_became_peered":"2023-09-10T01:00:44.025731+0000","last_unstale":"2023-09-10T01:01:56.128038+0000","last_undegraded":"2023-09-10T01:01:56.128038+0000","last_fullsized":"2023-09-10T01:01:56.128038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:51:48.949903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039896+0000","last_change":"2023-09-10T01:00:44.038916+0000","last_active":"2023-09-10T01:01:56.039896+0000","last_peered":"2023-09-10T01:01:56.039896+0000","last_clean":"2023-09-10T01:01:56.039896+0000","last_became_active":"2023-09-10T01:00:44.038546+0000","last_became_peered":"2023-09-10T01:00:44.038546+0000","last_unstale":"2023-09-10T01:01:56.039896+0000","last_undegraded":"2023-09-10T01:01:56.039896+0000","last_fullsized":"2023-09-10T01:01:56.039896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:05:56.096857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.038734+0000","last_change":"2023-09-10T01:00:44.032942+0000","last_active":"2023-09-10T01:01:56.038734+0000","last_peered":"2023-09-10T01:01:56.038734+0000","last_clean":"2023-09-10T01:01:56.038734+0000","last_became_active":"2023-09-10T01:00:44.032546+0000","last_became_peered":"2023-09-10T01:00:44.032546+0000","last_unstale":"2023-09-10T01:01:56.038734+0000","last_undegraded":"2023-09-10T01:01:56.038734+0000","last_fullsized":"2023-09-10T01:01:56.038734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:46.613487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036467+0000","last_change":"2023-09-10T01:00:44.031411+0000","last_active":"2023-09-10T01:01:56.036467+0000","last_peered":"2023-09-10T01:01:56.036467+0000","last_clean":"2023-09-10T01:01:56.036467+0000","last_became_active":"2023-09-10T01:00:44.031121+0000","last_became_peered":"2023-09-10T01:00:44.031121+0000","last_unstale":"2023-09-10T01:01:56.036467+0000","last_undegraded":"2023-09-10T01:01:56.036467+0000","last_fullsized":"2023-09-10T01:01:56.036467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:47.658116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035816+0000","last_change":"2023-09-10T01:00:44.037351+0000","last_active":"2023-09-10T01:01:56.035816+0000","last_peered":"2023-09-10T01:01:56.035816+0000","last_clean":"2023-09-10T01:01:56.035816+0000","last_became_active":"2023-09-10T01:00:44.037005+0000","last_became_peered":"2023-09-10T01:00:44.037005+0000","last_unstale":"2023-09-10T01:01:56.035816+0000","last_undegraded":"2023-09-10T01:01:56.035816+0000","last_fullsized":"2023-09-10T01:01:56.035816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:10:45.694056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127691+0000","last_change":"2023-09-10T01:00:44.029543+0000","last_active":"2023-09-10T01:01:56.127691+0000","last_peered":"2023-09-10T01:01:56.127691+0000","last_clean":"2023-09-10T01:01:56.127691+0000","last_became_active":"2023-09-10T01:00:44.029210+0000","last_became_peered":"2023-09-10T01:00:44.029210+0000","last_unstale":"2023-09-10T01:01:56.127691+0000","last_undegraded":"2023-09-10T01:01:56.127691+0000","last_fullsized":"2023-09-10T01:01:56.127691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:08:56.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042147+0000","last_change":"2023-09-10T01:00:44.033328+0000","last_active":"2023-09-10T01:01:56.042147+0000","last_peered":"2023-09-10T01:01:56.042147+0000","last_clean":"2023-09-10T01:01:56.042147+0000","last_became_active":"2023-09-10T01:00:44.033007+0000","last_became_peered":"2023-09-10T01:00:44.033007+0000","last_unstale":"2023-09-10T01:01:56.042147+0000","last_undegraded":"2023-09-10T01:01:56.042147+0000","last_fullsized":"2023-09-10T01:01:56.042147+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:59:14.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035843+0000","last_change":"2023-09-10T01:00:44.025611+0000","last_active":"2023-09-10T01:01:56.035843+0000","last_peered":"2023-09-10T01:01:56.035843+0000","last_clean":"2023-09-10T01:01:56.035843+0000","last_became_active":"2023-09-10T01:00:44.025249+0000","last_became_peered":"2023-09-10T01:00:44.025249+0000","last_unstale":"2023-09-10T01:01:56.035843+0000","last_undegraded":"2023-09-10T01:01:56.035843+0000","last_fullsized":"2023-09-10T01:01:56.035843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:55:04.320356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036684+0000","last_change":"2023-09-10T01:00:44.024207+0000","last_active":"2023-09-10T01:01:56.036684+0000","last_peered":"2023-09-10T01:01:56.036684+0000","last_clean":"2023-09-10T01:01:56.036684+0000","last_became_active":"2023-09-10T01:00:44.023866+0000","last_became_peered":"2023-09-10T01:00:44.023866+0000","last_unstale":"2023-09-10T01:01:56.036684+0000","last_undegraded":"2023-09-10T01:01:56.036684+0000","last_fullsized":"2023-09-10T01:01:56.036684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:53:39.845384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127301+0000","last_change":"2023-09-10T01:00:44.031241+0000","last_active":"2023-09-10T01:01:56.127301+0000","last_peered":"2023-09-10T01:01:56.127301+0000","last_clean":"2023-09-10T01:01:56.127301+0000","last_became_active":"2023-09-10T01:00:44.030392+0000","last_became_peered":"2023-09-10T01:00:44.030392+0000","last_unstale":"2023-09-10T01:01:56.127301+0000","last_undegraded":"2023-09-10T01:01:56.127301+0000","last_fullsized":"2023-09-10T01:01:56.127301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:19:39.786995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042893+0000","last_change":"2023-09-10T01:00:44.026983+0000","last_active":"2023-09-10T01:01:56.042893+0000","last_peered":"2023-09-10T01:01:56.042893+0000","last_clean":"2023-09-10T01:01:56.042893+0000","last_became_active":"2023-09-10T01:00:44.026695+0000","last_became_peered":"2023-09-10T01:00:44.026695+0000","last_unstale":"2023-09-10T01:01:56.042893+0000","last_undegraded":"2023-09-10T01:01:56.042893+0000","last_fullsized":"2023-09-10T01:01:56.042893+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:37.809144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037500+0000","last_change":"2023-09-10T01:00:44.030787+0000","last_active":"2023-09-10T01:01:56.037500+0000","last_peered":"2023-09-10T01:01:56.037500+0000","last_clean":"2023-09-10T01:01:56.037500+0000","last_became_active":"2023-09-10T01:00:44.028579+0000","last_became_peered":"2023-09-10T01:00:44.028579+0000","last_unstale":"2023-09-10T01:01:56.037500+0000","last_undegraded":"2023-09-10T01:01:56.037500+0000","last_fullsized":"2023-09-10T01:01:56.037500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:04:26.446378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030849+0000","last_change":"2023-09-10T01:00:44.029905+0000","last_active":"2023-09-10T01:01:56.030849+0000","last_peered":"2023-09-10T01:01:56.030849+0000","last_clean":"2023-09-10T01:01:56.030849+0000","last_became_active":"2023-09-10T01:00:44.029561+0000","last_became_peered":"2023-09-10T01:00:44.029561+0000","last_unstale":"2023-09-10T01:01:56.030849+0000","last_undegraded":"2023-09-10T01:01:56.030849+0000","last_fullsized":"2023-09-10T01:01:56.030849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:17:50.839001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039075+0000","last_change":"2023-09-10T01:00:44.037864+0000","last_active":"2023-09-10T01:01:56.039075+0000","last_peered":"2023-09-10T01:01:56.039075+0000","last_clean":"2023-09-10T01:01:56.039075+0000","last_became_active":"2023-09-10T01:00:44.037533+0000","last_became_peered":"2023-09-10T01:00:44.037533+0000","last_unstale":"2023-09-10T01:01:56.039075+0000","last_undegraded":"2023-09-10T01:01:56.039075+0000","last_fullsized":"2023-09-10T01:01:56.039075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:29:26.768570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039763+0000","last_change":"2023-09-10T01:00:44.033396+0000","last_active":"2023-09-10T01:01:56.039763+0000","last_peered":"2023-09-10T01:01:56.039763+0000","last_clean":"2023-09-10T01:01:56.039763+0000","last_became_active":"2023-09-10T01:00:44.033056+0000","last_became_peered":"2023-09-10T01:00:44.033056+0000","last_unstale":"2023-09-10T01:01:56.039763+0000","last_undegraded":"2023-09-10T01:01:56.039763+0000","last_fullsized":"2023-09-10T01:01:56.039763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:00:03.590555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036777+0000","last_change":"2023-09-10T01:00:44.027729+0000","last_active":"2023-09-10T01:01:56.036777+0000","last_peered":"2023-09-10T01:01:56.036777+0000","last_clean":"2023-09-10T01:01:56.036777+0000","last_became_active":"2023-09-10T01:00:44.027447+0000","last_became_peered":"2023-09-10T01:00:44.027447+0000","last_unstale":"2023-09-10T01:01:56.036777+0000","last_undegraded":"2023-09-10T01:01:56.036777+0000","last_fullsized":"2023-09-10T01:01:56.036777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:06:09.391443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127910+0000","last_change":"2023-09-10T01:00:44.026042+0000","last_active":"2023-09-10T01:01:56.127910+0000","last_peered":"2023-09-10T01:01:56.127910+0000","last_clean":"2023-09-10T01:01:56.127910+0000","last_became_active":"2023-09-10T01:00:44.025309+0000","last_became_peered":"2023-09-10T01:00:44.025309+0000","last_unstale":"2023-09-10T01:01:56.127910+0000","last_undegraded":"2023-09-10T01:01:56.127910+0000","last_fullsized":"2023-09-10T01:01:56.127910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:51:29.038717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033488+0000","last_change":"2023-09-10T01:00:44.036761+0000","last_active":"2023-09-10T01:01:56.033488+0000","last_peered":"2023-09-10T01:01:56.033488+0000","last_clean":"2023-09-10T01:01:56.033488+0000","last_became_active":"2023-09-10T01:00:44.036333+0000","last_became_peered":"2023-09-10T01:00:44.036333+0000","last_unstale":"2023-09-10T01:01:56.033488+0000","last_undegraded":"2023-09-10T01:01:56.033488+0000","last_fullsized":"2023-09-10T01:01:56.033488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:37:09.042178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033082+0000","last_change":"2023-09-10T01:00:44.027304+0000","last_active":"2023-09-10T01:01:56.033082+0000","last_peered":"2023-09-10T01:01:56.033082+0000","last_clean":"2023-09-10T01:01:56.033082+0000","last_became_active":"2023-09-10T01:00:44.026994+0000","last_became_peered":"2023-09-10T01:00:44.026994+0000","last_unstale":"2023-09-10T01:01:56.033082+0000","last_undegraded":"2023-09-10T01:01:56.033082+0000","last_fullsized":"2023-09-10T01:01:56.033082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:11:04.824687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032471+0000","last_change":"2023-09-10T01:00:48.056098+0000","last_active":"2023-09-10T01:01:56.032471+0000","last_peered":"2023-09-10T01:01:56.032471+0000","last_clean":"2023-09-10T01:01:56.032471+0000","last_became_active":"2023-09-10T01:00:48.055694+0000","last_became_peered":"2023-09-10T01:00:48.055694+0000","last_unstale":"2023-09-10T01:01:56.032471+0000","last_undegraded":"2023-09-10T01:01:56.032471+0000","last_fullsized":"2023-09-10T01:01:56.032471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:30:48.711613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033599+0000","last_change":"2023-09-10T01:00:56.971435+0000","last_active":"2023-09-10T01:01:56.033599+0000","last_peered":"2023-09-10T01:01:56.033599+0000","last_clean":"2023-09-10T01:01:56.033599+0000","last_became_active":"2023-09-10T01:00:56.965158+0000","last_became_peered":"2023-09-10T01:00:56.965158+0000","last_unstale":"2023-09-10T01:01:56.033599+0000","last_undegraded":"2023-09-10T01:01:56.033599+0000","last_fullsized":"2023-09-10T01:01:56.033599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:09:20.910597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036174+0000","last_change":"2023-09-10T01:00:44.023469+0000","last_active":"2023-09-10T01:01:56.036174+0000","last_peered":"2023-09-10T01:01:56.036174+0000","last_clean":"2023-09-10T01:01:56.036174+0000","last_became_active":"2023-09-10T01:00:44.023232+0000","last_became_peered":"2023-09-10T01:00:44.023232+0000","last_unstale":"2023-09-10T01:01:56.036174+0000","last_undegraded":"2023-09-10T01:01:56.036174+0000","last_fullsized":"2023-09-10T01:01:56.036174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:33:22.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037995+0000","last_change":"2023-09-10T01:00:48.063565+0000","last_active":"2023-09-10T01:01:56.037995+0000","last_peered":"2023-09-10T01:01:56.037995+0000","last_clean":"2023-09-10T01:01:56.037995+0000","last_became_active":"2023-09-10T01:00:48.063200+0000","last_became_peered":"2023-09-10T01:00:48.063200+0000","last_unstale":"2023-09-10T01:01:56.037995+0000","last_undegraded":"2023-09-10T01:01:56.037995+0000","last_fullsized":"2023-09-10T01:01:56.037995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:50:16.624296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036729+0000","last_change":"2023-09-10T01:00:43.010566+0000","last_active":"2023-09-10T01:01:56.036729+0000","last_peered":"2023-09-10T01:01:56.036729+0000","last_clean":"2023-09-10T01:01:56.036729+0000","last_became_active":"2023-09-10T01:00:43.010277+0000","last_became_peered":"2023-09-10T01:00:43.010277+0000","last_unstale":"2023-09-10T01:01:56.036729+0000","last_undegraded":"2023-09-10T01:01:56.036729+0000","last_fullsized":"2023-09-10T01:01:56.036729+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:37.808085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039865+0000","last_change":"2023-09-10T01:00:44.026219+0000","last_active":"2023-09-10T01:01:56.039865+0000","last_peered":"2023-09-10T01:01:56.039865+0000","last_clean":"2023-09-10T01:01:56.039865+0000","last_became_active":"2023-09-10T01:00:44.025906+0000","last_became_peered":"2023-09-10T01:00:44.025906+0000","last_unstale":"2023-09-10T01:01:56.039865+0000","last_undegraded":"2023-09-10T01:01:56.039865+0000","last_fullsized":"2023-09-10T01:01:56.039865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:21:48.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037347+0000","last_change":"2023-09-10T01:00:48.069671+0000","last_active":"2023-09-10T01:01:56.037347+0000","last_peered":"2023-09-10T01:01:56.037347+0000","last_clean":"2023-09-10T01:01:56.037347+0000","last_became_active":"2023-09-10T01:00:48.069382+0000","last_became_peered":"2023-09-10T01:00:48.069382+0000","last_unstale":"2023-09-10T01:01:56.037347+0000","last_undegraded":"2023-09-10T01:01:56.037347+0000","last_fullsized":"2023-09-10T01:01:56.037347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:06:20.526398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036543+0000","last_change":"2023-09-10T01:00:43.005098+0000","last_active":"2023-09-10T01:01:56.036543+0000","last_peered":"2023-09-10T01:01:56.036543+0000","last_clean":"2023-09-10T01:01:56.036543+0000","last_became_active":"2023-09-10T01:00:43.004556+0000","last_became_peered":"2023-09-10T01:00:43.004556+0000","last_unstale":"2023-09-10T01:01:56.036543+0000","last_undegraded":"2023-09-10T01:01:56.036543+0000","last_fullsized":"2023-09-10T01:01:56.036543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:27:43.538936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037176+0000","last_change":"2023-09-10T01:00:44.034451+0000","last_active":"2023-09-10T01:01:56.037176+0000","last_peered":"2023-09-10T01:01:56.037176+0000","last_clean":"2023-09-10T01:01:56.037176+0000","last_became_active":"2023-09-10T01:00:44.034131+0000","last_became_peered":"2023-09-10T01:00:44.034131+0000","last_unstale":"2023-09-10T01:01:56.037176+0000","last_undegraded":"2023-09-10T01:01:56.037176+0000","last_fullsized":"2023-09-10T01:01:56.037176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:50.643215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036946+0000","last_change":"2023-09-10T01:00:48.060114+0000","last_active":"2023-09-10T01:01:56.036946+0000","last_peered":"2023-09-10T01:01:56.036946+0000","last_clean":"2023-09-10T01:01:56.036946+0000","last_became_active":"2023-09-10T01:00:48.059531+0000","last_became_peered":"2023-09-10T01:00:48.059531+0000","last_unstale":"2023-09-10T01:01:56.036946+0000","last_undegraded":"2023-09-10T01:01:56.036946+0000","last_fullsized":"2023-09-10T01:01:56.036946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:28:01.462247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035950+0000","last_change":"2023-09-10T01:00:43.007196+0000","last_active":"2023-09-10T01:01:56.035950+0000","last_peered":"2023-09-10T01:01:56.035950+0000","last_clean":"2023-09-10T01:01:56.035950+0000","last_became_active":"2023-09-10T01:00:43.006795+0000","last_became_peered":"2023-09-10T01:00:43.006795+0000","last_unstale":"2023-09-10T01:01:56.035950+0000","last_undegraded":"2023-09-10T01:01:56.035950+0000","last_fullsized":"2023-09-10T01:01:56.035950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:26:13.146244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037180+0000","last_change":"2023-09-10T01:00:44.027809+0000","last_active":"2023-09-10T01:01:56.037180+0000","last_peered":"2023-09-10T01:01:56.037180+0000","last_clean":"2023-09-10T01:01:56.037180+0000","last_became_active":"2023-09-10T01:00:44.027499+0000","last_became_peered":"2023-09-10T01:00:44.027499+0000","last_unstale":"2023-09-10T01:01:56.037180+0000","last_undegraded":"2023-09-10T01:01:56.037180+0000","last_fullsized":"2023-09-10T01:01:56.037180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:17:07.138807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032849+0000","last_change":"2023-09-10T01:00:48.057645+0000","last_active":"2023-09-10T01:01:56.032849+0000","last_peered":"2023-09-10T01:01:56.032849+0000","last_clean":"2023-09-10T01:01:56.032849+0000","last_became_active":"2023-09-10T01:00:48.057312+0000","last_became_peered":"2023-09-10T01:00:48.057312+0000","last_unstale":"2023-09-10T01:01:56.032849+0000","last_undegraded":"2023-09-10T01:01:56.032849+0000","last_fullsized":"2023-09-10T01:01:56.032849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:28.904572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:01.029848+0000","last_change":"2023-09-10T01:00:56.965677+0000","last_active":"2023-09-10T01:02:01.029848+0000","last_peered":"2023-09-10T01:02:01.029848+0000","last_clean":"2023-09-10T01:02:01.029848+0000","last_became_active":"2023-09-10T01:00:56.965248+0000","last_became_peered":"2023-09-10T01:00:56.965248+0000","last_unstale":"2023-09-10T01:02:01.029848+0000","last_undegraded":"2023-09-10T01:02:01.029848+0000","last_fullsized":"2023-09-10T01:02:01.029848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:39:44.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035997+0000","last_change":"2023-09-10T01:00:44.026340+0000","last_active":"2023-09-10T01:01:56.035997+0000","last_peered":"2023-09-10T01:01:56.035997+0000","last_clean":"2023-09-10T01:01:56.035997+0000","last_became_active":"2023-09-10T01:00:44.026024+0000","last_became_peered":"2023-09-10T01:00:44.026024+0000","last_unstale":"2023-09-10T01:01:56.035997+0000","last_undegraded":"2023-09-10T01:01:56.035997+0000","last_fullsized":"2023-09-10T01:01:56.035997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:58:18.942980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032668+0000","last_change":"2023-09-10T01:00:48.057876+0000","last_active":"2023-09-10T01:01:56.032668+0000","last_peered":"2023-09-10T01:01:56.032668+0000","last_clean":"2023-09-10T01:01:56.032668+0000","last_became_active":"2023-09-10T01:00:48.056686+0000","last_became_peered":"2023-09-10T01:00:48.056686+0000","last_unstale":"2023-09-10T01:01:56.032668+0000","last_undegraded":"2023-09-10T01:01:56.032668+0000","last_fullsized":"2023-09-10T01:01:56.032668+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:10:15.995463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127691+0000","last_change":"2023-09-10T01:00:43.007811+0000","last_active":"2023-09-10T01:01:56.127691+0000","last_peered":"2023-09-10T01:01:56.127691+0000","last_clean":"2023-09-10T01:01:56.127691+0000","last_became_active":"2023-09-10T01:00:43.007205+0000","last_became_peered":"2023-09-10T01:00:43.007205+0000","last_unstale":"2023-09-10T01:01:56.127691+0000","last_undegraded":"2023-09-10T01:01:56.127691+0000","last_fullsized":"2023-09-10T01:01:56.127691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:27:38.114614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037376+0000","last_change":"2023-09-10T01:00:44.027954+0000","last_active":"2023-09-10T01:01:56.037376+0000","last_peered":"2023-09-10T01:01:56.037376+0000","last_clean":"2023-09-10T01:01:56.037376+0000","last_became_active":"2023-09-10T01:00:44.027643+0000","last_became_peered":"2023-09-10T01:00:44.027643+0000","last_unstale":"2023-09-10T01:01:56.037376+0000","last_undegraded":"2023-09-10T01:01:56.037376+0000","last_fullsized":"2023-09-10T01:01:56.037376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:21.253013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127896+0000","last_change":"2023-09-10T01:00:48.062519+0000","last_active":"2023-09-10T01:01:56.127896+0000","last_peered":"2023-09-10T01:01:56.127896+0000","last_clean":"2023-09-10T01:01:56.127896+0000","last_became_active":"2023-09-10T01:00:48.061985+0000","last_became_peered":"2023-09-10T01:00:48.061985+0000","last_unstale":"2023-09-10T01:01:56.127896+0000","last_undegraded":"2023-09-10T01:01:56.127896+0000","last_fullsized":"2023-09-10T01:01:56.127896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:47:36.002325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030435+0000","last_change":"2023-09-10T01:00:42.999601+0000","last_active":"2023-09-10T01:01:56.030435+0000","last_peered":"2023-09-10T01:01:56.030435+0000","last_clean":"2023-09-10T01:01:56.030435+0000","last_became_active":"2023-09-10T01:00:42.999243+0000","last_became_peered":"2023-09-10T01:00:42.999243+0000","last_unstale":"2023-09-10T01:01:56.030435+0000","last_undegraded":"2023-09-10T01:01:56.030435+0000","last_fullsized":"2023-09-10T01:01:56.030435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:49.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035937+0000","last_change":"2023-09-10T01:00:44.027136+0000","last_active":"2023-09-10T01:01:56.035937+0000","last_peered":"2023-09-10T01:01:56.035937+0000","last_clean":"2023-09-10T01:01:56.035937+0000","last_became_active":"2023-09-10T01:00:44.026784+0000","last_became_peered":"2023-09-10T01:00:44.026784+0000","last_unstale":"2023-09-10T01:01:56.035937+0000","last_undegraded":"2023-09-10T01:01:56.035937+0000","last_fullsized":"2023-09-10T01:01:56.035937+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:51:51.271658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039808+0000","last_change":"2023-09-10T01:00:48.066906+0000","last_active":"2023-09-10T01:01:56.039808+0000","last_peered":"2023-09-10T01:01:56.039808+0000","last_clean":"2023-09-10T01:01:56.039808+0000","last_became_active":"2023-09-10T01:00:48.066474+0000","last_became_peered":"2023-09-10T01:00:48.066474+0000","last_unstale":"2023-09-10T01:01:56.039808+0000","last_undegraded":"2023-09-10T01:01:56.039808+0000","last_fullsized":"2023-09-10T01:01:56.039808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:48:26.827857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036365+0000","last_change":"2023-09-10T01:00:43.005922+0000","last_active":"2023-09-10T01:01:56.036365+0000","last_peered":"2023-09-10T01:01:56.036365+0000","last_clean":"2023-09-10T01:01:56.036365+0000","last_became_active":"2023-09-10T01:00:43.005517+0000","last_became_peered":"2023-09-10T01:00:43.005517+0000","last_unstale":"2023-09-10T01:01:56.036365+0000","last_undegraded":"2023-09-10T01:01:56.036365+0000","last_fullsized":"2023-09-10T01:01:56.036365+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:32:12.619034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031897+0000","last_change":"2023-09-10T01:00:44.036766+0000","last_active":"2023-09-10T01:01:56.031897+0000","last_peered":"2023-09-10T01:01:56.031897+0000","last_clean":"2023-09-10T01:01:56.031897+0000","last_became_active":"2023-09-10T01:00:44.036316+0000","last_became_peered":"2023-09-10T01:00:44.036316+0000","last_unstale":"2023-09-10T01:01:56.031897+0000","last_undegraded":"2023-09-10T01:01:56.031897+0000","last_fullsized":"2023-09-10T01:01:56.031897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:17:51.875838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036026+0000","last_change":"2023-09-10T01:00:57.051716+0000","last_active":"2023-09-10T01:01:56.036026+0000","last_peered":"2023-09-10T01:01:56.036026+0000","last_clean":"2023-09-10T01:01:56.036026+0000","last_became_active":"2023-09-10T01:00:57.051334+0000","last_became_peered":"2023-09-10T01:00:57.051334+0000","last_unstale":"2023-09-10T01:01:56.036026+0000","last_undegraded":"2023-09-10T01:01:56.036026+0000","last_fullsized":"2023-09-10T01:01:56.036026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:30:52.598639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:01.029158+0000","last_change":"2023-09-10T01:00:43.008695+0000","last_active":"2023-09-10T01:02:01.029158+0000","last_peered":"2023-09-10T01:02:01.029158+0000","last_clean":"2023-09-10T01:02:01.029158+0000","last_became_active":"2023-09-10T01:00:43.008278+0000","last_became_peered":"2023-09-10T01:00:43.008278+0000","last_unstale":"2023-09-10T01:02:01.029158+0000","last_undegraded":"2023-09-10T01:02:01.029158+0000","last_fullsized":"2023-09-10T01:02:01.029158+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:46:38.370582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036171+0000","last_change":"2023-09-10T01:00:44.030540+0000","last_active":"2023-09-10T01:01:56.036171+0000","last_peered":"2023-09-10T01:01:56.036171+0000","last_clean":"2023-09-10T01:01:56.036171+0000","last_became_active":"2023-09-10T01:00:44.030212+0000","last_became_peered":"2023-09-10T01:00:44.030212+0000","last_unstale":"2023-09-10T01:01:56.036171+0000","last_undegraded":"2023-09-10T01:01:56.036171+0000","last_fullsized":"2023-09-10T01:01:56.036171+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:57:06.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036424+0000","last_change":"2023-09-10T01:00:57.052197+0000","last_active":"2023-09-10T01:01:56.036424+0000","last_peered":"2023-09-10T01:01:56.036424+0000","last_clean":"2023-09-10T01:01:56.036424+0000","last_became_active":"2023-09-10T01:00:57.051782+0000","last_became_peered":"2023-09-10T01:00:57.051782+0000","last_unstale":"2023-09-10T01:01:56.036424+0000","last_undegraded":"2023-09-10T01:01:56.036424+0000","last_fullsized":"2023-09-10T01:01:56.036424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:36:49.680123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042966+0000","last_change":"2023-09-10T01:00:43.007242+0000","last_active":"2023-09-10T01:01:56.042966+0000","last_peered":"2023-09-10T01:01:56.042966+0000","last_clean":"2023-09-10T01:01:56.042966+0000","last_became_active":"2023-09-10T01:00:43.006912+0000","last_became_peered":"2023-09-10T01:00:43.006912+0000","last_unstale":"2023-09-10T01:01:56.042966+0000","last_undegraded":"2023-09-10T01:01:56.042966+0000","last_fullsized":"2023-09-10T01:01:56.042966+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:48:22.783656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.026024+0000","last_change":"2023-09-10T01:01:58.026024+0000","last_active":"2023-09-10T01:01:58.026024+0000","last_peered":"2023-09-10T01:01:58.026024+0000","last_clean":"2023-09-10T01:01:58.026024+0000","last_became_active":"2023-09-10T01:01:58.025647+0000","last_became_peered":"2023-09-10T01:01:58.025647+0000","last_unstale":"2023-09-10T01:01:58.026024+0000","last_undegraded":"2023-09-10T01:01:58.026024+0000","last_fullsized":"2023-09-10T01:01:58.026024+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:58:11.916233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037579+0000","last_change":"2023-09-10T01:00:48.062520+0000","last_active":"2023-09-10T01:01:56.037579+0000","last_peered":"2023-09-10T01:01:56.037579+0000","last_clean":"2023-09-10T01:01:56.037579+0000","last_became_active":"2023-09-10T01:00:48.061870+0000","last_became_peered":"2023-09-10T01:00:48.061870+0000","last_unstale":"2023-09-10T01:01:56.037579+0000","last_undegraded":"2023-09-10T01:01:56.037579+0000","last_fullsized":"2023-09-10T01:01:56.037579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:24:38.798775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037030+0000","last_change":"2023-09-10T01:00:43.006513+0000","last_active":"2023-09-10T01:01:56.037030+0000","last_peered":"2023-09-10T01:01:56.037030+0000","last_clean":"2023-09-10T01:01:56.037030+0000","last_became_active":"2023-09-10T01:00:43.006166+0000","last_became_peered":"2023-09-10T01:00:43.006166+0000","last_unstale":"2023-09-10T01:01:56.037030+0000","last_undegraded":"2023-09-10T01:01:56.037030+0000","last_fullsized":"2023-09-10T01:01:56.037030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:52:32.167513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037335+0000","last_change":"2023-09-10T01:00:44.031264+0000","last_active":"2023-09-10T01:01:56.037335+0000","last_peered":"2023-09-10T01:01:56.037335+0000","last_clean":"2023-09-10T01:01:56.037335+0000","last_became_active":"2023-09-10T01:00:44.030993+0000","last_became_peered":"2023-09-10T01:00:44.030993+0000","last_unstale":"2023-09-10T01:01:56.037335+0000","last_undegraded":"2023-09-10T01:01:56.037335+0000","last_fullsized":"2023-09-10T01:01:56.037335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:19:27.658594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032496+0000","last_change":"2023-09-10T01:00:48.056886+0000","last_active":"2023-09-10T01:01:56.032496+0000","last_peered":"2023-09-10T01:01:56.032496+0000","last_clean":"2023-09-10T01:01:56.032496+0000","last_became_active":"2023-09-10T01:00:48.056411+0000","last_became_peered":"2023-09-10T01:00:48.056411+0000","last_unstale":"2023-09-10T01:01:56.032496+0000","last_undegraded":"2023-09-10T01:01:56.032496+0000","last_fullsized":"2023-09-10T01:01:56.032496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:56:42.062702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'30","reported_seq":164,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:02.030460+0000","last_change":"2023-09-10T01:00:43.009113+0000","last_active":"2023-09-10T01:02:02.030460+0000","last_peered":"2023-09-10T01:02:02.030460+0000","last_clean":"2023-09-10T01:02:02.030460+0000","last_became_active":"2023-09-10T01:00:43.008757+0000","last_became_peered":"2023-09-10T01:00:43.008757+0000","last_unstale":"2023-09-10T01:02:02.030460+0000","last_undegraded":"2023-09-10T01:02:02.030460+0000","last_fullsized":"2023-09-10T01:02:02.030460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:29:24.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59889,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":113,"num_read_kb":115,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036282+0000","last_change":"2023-09-10T01:00:44.027127+0000","last_active":"2023-09-10T01:01:56.036282+0000","last_peered":"2023-09-10T01:01:56.036282+0000","last_clean":"2023-09-10T01:01:56.036282+0000","last_became_active":"2023-09-10T01:00:44.026852+0000","last_became_peered":"2023-09-10T01:00:44.026852+0000","last_unstale":"2023-09-10T01:01:56.036282+0000","last_undegraded":"2023-09-10T01:01:56.036282+0000","last_fullsized":"2023-09-10T01:01:56.036282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:14:20.805300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037605+0000","last_change":"2023-09-10T01:00:48.069337+0000","last_active":"2023-09-10T01:01:56.037605+0000","last_peered":"2023-09-10T01:01:56.037605+0000","last_clean":"2023-09-10T01:01:56.037605+0000","last_became_active":"2023-09-10T01:00:48.068893+0000","last_became_peered":"2023-09-10T01:00:48.068893+0000","last_unstale":"2023-09-10T01:01:56.037605+0000","last_undegraded":"2023-09-10T01:01:56.037605+0000","last_fullsized":"2023-09-10T01:01:56.037605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:24:40.074551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033160+0000","last_change":"2023-09-10T01:00:43.010128+0000","last_active":"2023-09-10T01:01:56.033160+0000","last_peered":"2023-09-10T01:01:56.033160+0000","last_clean":"2023-09-10T01:01:56.033160+0000","last_became_active":"2023-09-10T01:00:43.009808+0000","last_became_peered":"2023-09-10T01:00:43.009808+0000","last_unstale":"2023-09-10T01:01:56.033160+0000","last_undegraded":"2023-09-10T01:01:56.033160+0000","last_fullsized":"2023-09-10T01:01:56.033160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:27.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030524+0000","last_change":"2023-09-10T01:00:44.038665+0000","last_active":"2023-09-10T01:01:56.030524+0000","last_peered":"2023-09-10T01:01:56.030524+0000","last_clean":"2023-09-10T01:01:56.030524+0000","last_became_active":"2023-09-10T01:00:44.037944+0000","last_became_peered":"2023-09-10T01:00:44.037944+0000","last_unstale":"2023-09-10T01:01:56.030524+0000","last_undegraded":"2023-09-10T01:01:56.030524+0000","last_fullsized":"2023-09-10T01:01:56.030524+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:11:34.018249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127649+0000","last_change":"2023-09-10T01:00:56.965594+0000","last_active":"2023-09-10T01:01:56.127649+0000","last_peered":"2023-09-10T01:01:56.127649+0000","last_clean":"2023-09-10T01:01:56.127649+0000","last_became_active":"2023-09-10T01:00:56.965006+0000","last_became_peered":"2023-09-10T01:00:56.965006+0000","last_unstale":"2023-09-10T01:01:56.127649+0000","last_undegraded":"2023-09-10T01:01:56.127649+0000","last_fullsized":"2023-09-10T01:01:56.127649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:13:16.946697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036734+0000","last_change":"2023-09-10T01:00:43.004855+0000","last_active":"2023-09-10T01:01:56.036734+0000","last_peered":"2023-09-10T01:01:56.036734+0000","last_clean":"2023-09-10T01:01:56.036734+0000","last_became_active":"2023-09-10T01:00:43.004464+0000","last_became_peered":"2023-09-10T01:00:43.004464+0000","last_unstale":"2023-09-10T01:01:56.036734+0000","last_undegraded":"2023-09-10T01:01:56.036734+0000","last_fullsized":"2023-09-10T01:01:56.036734+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:29.517290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042188+0000","last_change":"2023-09-10T01:00:44.037273+0000","last_active":"2023-09-10T01:01:56.042188+0000","last_peered":"2023-09-10T01:01:56.042188+0000","last_clean":"2023-09-10T01:01:56.042188+0000","last_became_active":"2023-09-10T01:00:44.036965+0000","last_became_peered":"2023-09-10T01:00:44.036965+0000","last_unstale":"2023-09-10T01:01:56.042188+0000","last_undegraded":"2023-09-10T01:01:56.042188+0000","last_fullsized":"2023-09-10T01:01:56.042188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:47:20.149002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033203+0000","last_change":"2023-09-10T01:00:48.058574+0000","last_active":"2023-09-10T01:01:56.033203+0000","last_peered":"2023-09-10T01:01:56.033203+0000","last_clean":"2023-09-10T01:01:56.033203+0000","last_became_active":"2023-09-10T01:00:48.058190+0000","last_became_peered":"2023-09-10T01:00:48.058190+0000","last_unstale":"2023-09-10T01:01:56.033203+0000","last_undegraded":"2023-09-10T01:01:56.033203+0000","last_fullsized":"2023-09-10T01:01:56.033203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:46:42.177996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:00.045177+0000","last_change":"2023-09-10T01:00:43.003816+0000","last_active":"2023-09-10T01:02:00.045177+0000","last_peered":"2023-09-10T01:02:00.045177+0000","last_clean":"2023-09-10T01:02:00.045177+0000","last_became_active":"2023-09-10T01:00:43.003404+0000","last_became_peered":"2023-09-10T01:00:43.003404+0000","last_unstale":"2023-09-10T01:02:00.045177+0000","last_undegraded":"2023-09-10T01:02:00.045177+0000","last_fullsized":"2023-09-10T01:02:00.045177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:55.208302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042150+0000","last_change":"2023-09-10T01:00:44.034607+0000","last_active":"2023-09-10T01:01:56.042150+0000","last_peered":"2023-09-10T01:01:56.042150+0000","last_clean":"2023-09-10T01:01:56.042150+0000","last_became_active":"2023-09-10T01:00:44.034244+0000","last_became_peered":"2023-09-10T01:00:44.034244+0000","last_unstale":"2023-09-10T01:01:56.042150+0000","last_undegraded":"2023-09-10T01:01:56.042150+0000","last_fullsized":"2023-09-10T01:01:56.042150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:06:13.214160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031918+0000","last_change":"2023-09-10T01:00:48.058789+0000","last_active":"2023-09-10T01:01:56.031918+0000","last_peered":"2023-09-10T01:01:56.031918+0000","last_clean":"2023-09-10T01:01:56.031918+0000","last_became_active":"2023-09-10T01:00:48.058220+0000","last_became_peered":"2023-09-10T01:00:48.058220+0000","last_unstale":"2023-09-10T01:01:56.031918+0000","last_undegraded":"2023-09-10T01:01:56.031918+0000","last_fullsized":"2023-09-10T01:01:56.031918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:01:12.291739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036527+0000","last_change":"2023-09-10T01:00:43.006109+0000","last_active":"2023-09-10T01:01:56.036527+0000","last_peered":"2023-09-10T01:01:56.036527+0000","last_clean":"2023-09-10T01:01:56.036527+0000","last_became_active":"2023-09-10T01:00:43.005753+0000","last_became_peered":"2023-09-10T01:00:43.005753+0000","last_unstale":"2023-09-10T01:01:56.036527+0000","last_undegraded":"2023-09-10T01:01:56.036527+0000","last_fullsized":"2023-09-10T01:01:56.036527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:45:47.083991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.139294+0000","last_change":"2023-09-10T01:01:57.139294+0000","last_active":"2023-09-10T01:01:57.139294+0000","last_peered":"2023-09-10T01:01:57.139294+0000","last_clean":"2023-09-10T01:01:57.139294+0000","last_became_active":"2023-09-10T01:01:57.138930+0000","last_became_peered":"2023-09-10T01:01:57.138930+0000","last_unstale":"2023-09-10T01:01:57.139294+0000","last_undegraded":"2023-09-10T01:01:57.139294+0000","last_fullsized":"2023-09-10T01:01:57.139294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:21.143499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036928+0000","last_change":"2023-09-10T01:00:48.069318+0000","last_active":"2023-09-10T01:01:56.036928+0000","last_peered":"2023-09-10T01:01:56.036928+0000","last_clean":"2023-09-10T01:01:56.036928+0000","last_became_active":"2023-09-10T01:00:48.068876+0000","last_became_peered":"2023-09-10T01:00:48.068876+0000","last_unstale":"2023-09-10T01:01:56.036928+0000","last_undegraded":"2023-09-10T01:01:56.036928+0000","last_fullsized":"2023-09-10T01:01:56.036928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:06:22.366530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036038+0000","last_change":"2023-09-10T01:00:43.004164+0000","last_active":"2023-09-10T01:01:56.036038+0000","last_peered":"2023-09-10T01:01:56.036038+0000","last_clean":"2023-09-10T01:01:56.036038+0000","last_became_active":"2023-09-10T01:00:43.003781+0000","last_became_peered":"2023-09-10T01:00:43.003781+0000","last_unstale":"2023-09-10T01:01:56.036038+0000","last_undegraded":"2023-09-10T01:01:56.036038+0000","last_fullsized":"2023-09-10T01:01:56.036038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:38:18.559712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127889+0000","last_change":"2023-09-10T01:00:44.028206+0000","last_active":"2023-09-10T01:01:56.127889+0000","last_peered":"2023-09-10T01:01:56.127889+0000","last_clean":"2023-09-10T01:01:56.127889+0000","last_became_active":"2023-09-10T01:00:44.027914+0000","last_became_peered":"2023-09-10T01:00:44.027914+0000","last_unstale":"2023-09-10T01:01:56.127889+0000","last_undegraded":"2023-09-10T01:01:56.127889+0000","last_fullsized":"2023-09-10T01:01:56.127889+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:41:12.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035899+0000","last_change":"2023-09-10T01:00:48.428613+0000","last_active":"2023-09-10T01:01:56.035899+0000","last_peered":"2023-09-10T01:01:56.035899+0000","last_clean":"2023-09-10T01:01:56.035899+0000","last_became_active":"2023-09-10T01:00:48.428059+0000","last_became_peered":"2023-09-10T01:00:48.428059+0000","last_unstale":"2023-09-10T01:01:56.035899+0000","last_undegraded":"2023-09-10T01:01:56.035899+0000","last_fullsized":"2023-09-10T01:01:56.035899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:46:03.683279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032873+0000","last_change":"2023-09-10T01:00:43.006061+0000","last_active":"2023-09-10T01:01:56.032873+0000","last_peered":"2023-09-10T01:01:56.032873+0000","last_clean":"2023-09-10T01:01:56.032873+0000","last_became_active":"2023-09-10T01:00:43.005392+0000","last_became_peered":"2023-09-10T01:00:43.005392+0000","last_unstale":"2023-09-10T01:01:56.032873+0000","last_undegraded":"2023-09-10T01:01:56.032873+0000","last_fullsized":"2023-09-10T01:01:56.032873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:01:45.062338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127818+0000","last_change":"2023-09-10T01:00:44.026684+0000","last_active":"2023-09-10T01:01:56.127818+0000","last_peered":"2023-09-10T01:01:56.127818+0000","last_clean":"2023-09-10T01:01:56.127818+0000","last_became_active":"2023-09-10T01:00:44.026387+0000","last_became_peered":"2023-09-10T01:00:44.026387+0000","last_unstale":"2023-09-10T01:01:56.127818+0000","last_undegraded":"2023-09-10T01:01:56.127818+0000","last_fullsized":"2023-09-10T01:01:56.127818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:23:03.662543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031165+0000","last_change":"2023-09-10T01:00:48.071589+0000","last_active":"2023-09-10T01:01:56.031165+0000","last_peered":"2023-09-10T01:01:56.031165+0000","last_clean":"2023-09-10T01:01:56.031165+0000","last_became_active":"2023-09-10T01:00:48.071118+0000","last_became_peered":"2023-09-10T01:00:48.071118+0000","last_unstale":"2023-09-10T01:01:56.031165+0000","last_undegraded":"2023-09-10T01:01:56.031165+0000","last_fullsized":"2023-09-10T01:01:56.031165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:27.194868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042525+0000","last_change":"2023-09-10T01:00:43.009738+0000","last_active":"2023-09-10T01:01:56.042525+0000","last_peered":"2023-09-10T01:01:56.042525+0000","last_clean":"2023-09-10T01:01:56.042525+0000","last_became_active":"2023-09-10T01:00:43.009376+0000","last_became_peered":"2023-09-10T01:00:43.009376+0000","last_unstale":"2023-09-10T01:01:56.042525+0000","last_undegraded":"2023-09-10T01:01:56.042525+0000","last_fullsized":"2023-09-10T01:01:56.042525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:08.761672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127701+0000","last_change":"2023-09-10T01:00:44.030175+0000","last_active":"2023-09-10T01:01:56.127701+0000","last_peered":"2023-09-10T01:01:56.127701+0000","last_clean":"2023-09-10T01:01:56.127701+0000","last_became_active":"2023-09-10T01:00:44.029748+0000","last_became_peered":"2023-09-10T01:00:44.029748+0000","last_unstale":"2023-09-10T01:01:56.127701+0000","last_undegraded":"2023-09-10T01:01:56.127701+0000","last_fullsized":"2023-09-10T01:01:56.127701+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:51:15.149610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032036+0000","last_change":"2023-09-10T01:00:48.428008+0000","last_active":"2023-09-10T01:01:56.032036+0000","last_peered":"2023-09-10T01:01:56.032036+0000","last_clean":"2023-09-10T01:01:56.032036+0000","last_became_active":"2023-09-10T01:00:48.427498+0000","last_became_peered":"2023-09-10T01:00:48.427498+0000","last_unstale":"2023-09-10T01:01:56.032036+0000","last_undegraded":"2023-09-10T01:01:56.032036+0000","last_fullsized":"2023-09-10T01:01:56.032036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:16:48.938006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.109029+0000","last_change":"2023-09-10T01:01:57.107918+0000","last_active":"2023-09-10T01:01:57.109029+0000","last_peered":"2023-09-10T01:01:57.109029+0000","last_clean":"2023-09-10T01:01:57.109029+0000","last_became_active":"2023-09-10T01:01:57.107413+0000","last_became_peered":"2023-09-10T01:01:57.107413+0000","last_unstale":"2023-09-10T01:01:57.109029+0000","last_undegraded":"2023-09-10T01:01:57.109029+0000","last_fullsized":"2023-09-10T01:01:57.109029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:16:33.836871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:00.044207+0000","last_change":"2023-09-10T01:00:43.001047+0000","last_active":"2023-09-10T01:02:00.044207+0000","last_peered":"2023-09-10T01:02:00.044207+0000","last_clean":"2023-09-10T01:02:00.044207+0000","last_became_active":"2023-09-10T01:00:43.000521+0000","last_became_peered":"2023-09-10T01:00:43.000521+0000","last_unstale":"2023-09-10T01:02:00.044207+0000","last_undegraded":"2023-09-10T01:02:00.044207+0000","last_fullsized":"2023-09-10T01:02:00.044207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:15:34.509964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042829+0000","last_change":"2023-09-10T01:00:48.062148+0000","last_active":"2023-09-10T01:01:56.042829+0000","last_peered":"2023-09-10T01:01:56.042829+0000","last_clean":"2023-09-10T01:01:56.042829+0000","last_became_active":"2023-09-10T01:00:48.061746+0000","last_became_peered":"2023-09-10T01:00:48.061746+0000","last_unstale":"2023-09-10T01:01:56.042829+0000","last_undegraded":"2023-09-10T01:01:56.042829+0000","last_fullsized":"2023-09-10T01:01:56.042829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:40.087012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036350+0000","last_change":"2023-09-10T01:00:44.030500+0000","last_active":"2023-09-10T01:01:56.036350+0000","last_peered":"2023-09-10T01:01:56.036350+0000","last_clean":"2023-09-10T01:01:56.036350+0000","last_became_active":"2023-09-10T01:00:44.030209+0000","last_became_peered":"2023-09-10T01:00:44.030209+0000","last_unstale":"2023-09-10T01:01:56.036350+0000","last_undegraded":"2023-09-10T01:01:56.036350+0000","last_fullsized":"2023-09-10T01:01:56.036350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:53:30.421493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037460+0000","last_change":"2023-09-10T01:00:43.001102+0000","last_active":"2023-09-10T01:01:56.037460+0000","last_peered":"2023-09-10T01:01:56.037460+0000","last_clean":"2023-09-10T01:01:56.037460+0000","last_became_active":"2023-09-10T01:00:43.000715+0000","last_became_peered":"2023-09-10T01:00:43.000715+0000","last_unstale":"2023-09-10T01:01:56.037460+0000","last_undegraded":"2023-09-10T01:01:56.037460+0000","last_fullsized":"2023-09-10T01:01:56.037460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:43:37.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":326,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042414+0000","last_change":"2023-09-10T00:59:40.261244+0000","last_active":"2023-09-10T01:01:56.042414+0000","last_peered":"2023-09-10T01:01:56.042414+0000","last_clean":"2023-09-10T01:01:56.042414+0000","last_became_active":"2023-09-10T00:59:40.260673+0000","last_became_peered":"2023-09-10T00:59:40.260673+0000","last_unstale":"2023-09-10T01:01:56.042414+0000","last_undegraded":"2023-09-10T01:01:56.042414+0000","last_fullsized":"2023-09-10T01:01:56.042414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_clean_scrub_stamp":"2023-09-10T00:56:36.374323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:04:03.614104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037213+0000","last_change":"2023-09-10T01:00:57.052019+0000","last_active":"2023-09-10T01:01:56.037213+0000","last_peered":"2023-09-10T01:01:56.037213+0000","last_clean":"2023-09-10T01:01:56.037213+0000","last_became_active":"2023-09-10T01:00:57.051527+0000","last_became_peered":"2023-09-10T01:00:57.051527+0000","last_unstale":"2023-09-10T01:01:56.037213+0000","last_undegraded":"2023-09-10T01:01:56.037213+0000","last_fullsized":"2023-09-10T01:01:56.037213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:10:43.756761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030326+0000","last_change":"2023-09-10T01:00:44.033214+0000","last_active":"2023-09-10T01:01:56.030326+0000","last_peered":"2023-09-10T01:01:56.030326+0000","last_clean":"2023-09-10T01:01:56.030326+0000","last_became_active":"2023-09-10T01:00:44.032828+0000","last_became_peered":"2023-09-10T01:00:44.032828+0000","last_unstale":"2023-09-10T01:01:56.030326+0000","last_undegraded":"2023-09-10T01:01:56.030326+0000","last_fullsized":"2023-09-10T01:01:56.030326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:01:22.417216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042638+0000","last_change":"2023-09-10T01:00:56.965456+0000","last_active":"2023-09-10T01:01:56.042638+0000","last_peered":"2023-09-10T01:01:56.042638+0000","last_clean":"2023-09-10T01:01:56.042638+0000","last_became_active":"2023-09-10T01:00:56.965021+0000","last_became_peered":"2023-09-10T01:00:56.965021+0000","last_unstale":"2023-09-10T01:01:56.042638+0000","last_undegraded":"2023-09-10T01:01:56.042638+0000","last_fullsized":"2023-09-10T01:01:56.042638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:19.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127643+0000","last_change":"2023-09-10T01:00:48.066811+0000","last_active":"2023-09-10T01:01:56.127643+0000","last_peered":"2023-09-10T01:01:56.127643+0000","last_clean":"2023-09-10T01:01:56.127643+0000","last_became_active":"2023-09-10T01:00:48.066421+0000","last_became_peered":"2023-09-10T01:00:48.066421+0000","last_unstale":"2023-09-10T01:01:56.127643+0000","last_undegraded":"2023-09-10T01:01:56.127643+0000","last_fullsized":"2023-09-10T01:01:56.127643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:48:01.459070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036384+0000","last_change":"2023-09-10T01:00:43.011134+0000","last_active":"2023-09-10T01:01:56.036384+0000","last_peered":"2023-09-10T01:01:56.036384+0000","last_clean":"2023-09-10T01:01:56.036384+0000","last_became_active":"2023-09-10T01:00:43.010871+0000","last_became_peered":"2023-09-10T01:00:43.010871+0000","last_unstale":"2023-09-10T01:01:56.036384+0000","last_undegraded":"2023-09-10T01:01:56.036384+0000","last_fullsized":"2023-09-10T01:01:56.036384+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:56.903745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031298+0000","last_change":"2023-09-10T01:00:44.031104+0000","last_active":"2023-09-10T01:01:56.031298+0000","last_peered":"2023-09-10T01:01:56.031298+0000","last_clean":"2023-09-10T01:01:56.031298+0000","last_became_active":"2023-09-10T01:00:44.030745+0000","last_became_peered":"2023-09-10T01:00:44.030745+0000","last_unstale":"2023-09-10T01:01:56.031298+0000","last_undegraded":"2023-09-10T01:01:56.031298+0000","last_fullsized":"2023-09-10T01:01:56.031298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:53.885886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035909+0000","last_change":"2023-09-10T01:00:48.427735+0000","last_active":"2023-09-10T01:01:56.035909+0000","last_peered":"2023-09-10T01:01:56.035909+0000","last_clean":"2023-09-10T01:01:56.035909+0000","last_became_active":"2023-09-10T01:00:48.427342+0000","last_became_peered":"2023-09-10T01:00:48.427342+0000","last_unstale":"2023-09-10T01:01:56.035909+0000","last_undegraded":"2023-09-10T01:01:56.035909+0000","last_fullsized":"2023-09-10T01:01:56.035909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:01:52.007325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033359+0000","last_change":"2023-09-10T01:00:43.006314+0000","last_active":"2023-09-10T01:01:56.033359+0000","last_peered":"2023-09-10T01:01:56.033359+0000","last_clean":"2023-09-10T01:01:56.033359+0000","last_became_active":"2023-09-10T01:00:43.004835+0000","last_became_peered":"2023-09-10T01:00:43.004835+0000","last_unstale":"2023-09-10T01:01:56.033359+0000","last_undegraded":"2023-09-10T01:01:56.033359+0000","last_fullsized":"2023-09-10T01:01:56.033359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:37:09.359846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036353+0000","last_change":"2023-09-10T01:00:44.029839+0000","last_active":"2023-09-10T01:01:56.036353+0000","last_peered":"2023-09-10T01:01:56.036353+0000","last_clean":"2023-09-10T01:01:56.036353+0000","last_became_active":"2023-09-10T01:00:44.029492+0000","last_became_peered":"2023-09-10T01:00:44.029492+0000","last_unstale":"2023-09-10T01:01:56.036353+0000","last_undegraded":"2023-09-10T01:01:56.036353+0000","last_fullsized":"2023-09-10T01:01:56.036353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:14.357386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032332+0000","last_change":"2023-09-10T01:00:48.054536+0000","last_active":"2023-09-10T01:01:56.032332+0000","last_peered":"2023-09-10T01:01:56.032332+0000","last_clean":"2023-09-10T01:01:56.032332+0000","last_became_active":"2023-09-10T01:00:48.053938+0000","last_became_peered":"2023-09-10T01:00:48.053938+0000","last_unstale":"2023-09-10T01:01:56.032332+0000","last_undegraded":"2023-09-10T01:01:56.032332+0000","last_fullsized":"2023-09-10T01:01:56.032332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:45:12.703592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037909+0000","last_change":"2023-09-10T01:00:43.002313+0000","last_active":"2023-09-10T01:01:56.037909+0000","last_peered":"2023-09-10T01:01:56.037909+0000","last_clean":"2023-09-10T01:01:56.037909+0000","last_became_active":"2023-09-10T01:00:43.001839+0000","last_became_peered":"2023-09-10T01:00:43.001839+0000","last_unstale":"2023-09-10T01:01:56.037909+0000","last_undegraded":"2023-09-10T01:01:56.037909+0000","last_fullsized":"2023-09-10T01:01:56.037909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:51.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.108269+0000","last_change":"2023-09-10T01:01:57.108269+0000","last_active":"2023-09-10T01:01:57.108269+0000","last_peered":"2023-09-10T01:01:57.108269+0000","last_clean":"2023-09-10T01:01:57.108269+0000","last_became_active":"2023-09-10T01:01:57.107879+0000","last_became_peered":"2023-09-10T01:01:57.107879+0000","last_unstale":"2023-09-10T01:01:57.108269+0000","last_undegraded":"2023-09-10T01:01:57.108269+0000","last_fullsized":"2023-09-10T01:01:57.108269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:47:09.135978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037126+0000","last_change":"2023-09-10T01:00:48.428406+0000","last_active":"2023-09-10T01:01:56.037126+0000","last_peered":"2023-09-10T01:01:56.037126+0000","last_clean":"2023-09-10T01:01:56.037126+0000","last_became_active":"2023-09-10T01:00:48.427938+0000","last_became_peered":"2023-09-10T01:00:48.427938+0000","last_unstale":"2023-09-10T01:01:56.037126+0000","last_undegraded":"2023-09-10T01:01:56.037126+0000","last_fullsized":"2023-09-10T01:01:56.037126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:52.316235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036897+0000","last_change":"2023-09-10T01:00:56.963755+0000","last_active":"2023-09-10T01:01:56.036897+0000","last_peered":"2023-09-10T01:01:56.036897+0000","last_clean":"2023-09-10T01:01:56.036897+0000","last_became_active":"2023-09-10T01:00:56.963283+0000","last_became_peered":"2023-09-10T01:00:56.963283+0000","last_unstale":"2023-09-10T01:01:56.036897+0000","last_undegraded":"2023-09-10T01:01:56.036897+0000","last_fullsized":"2023-09-10T01:01:56.036897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:13:32.213393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035575+0000","last_change":"2023-09-10T01:00:44.025284+0000","last_active":"2023-09-10T01:01:56.035575+0000","last_peered":"2023-09-10T01:01:56.035575+0000","last_clean":"2023-09-10T01:01:56.035575+0000","last_became_active":"2023-09-10T01:00:44.024965+0000","last_became_peered":"2023-09-10T01:00:44.024965+0000","last_unstale":"2023-09-10T01:01:56.035575+0000","last_undegraded":"2023-09-10T01:01:56.035575+0000","last_fullsized":"2023-09-10T01:01:56.035575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:19:08.620325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033359+0000","last_change":"2023-09-10T01:00:48.070843+0000","last_active":"2023-09-10T01:01:56.033359+0000","last_peered":"2023-09-10T01:01:56.033359+0000","last_clean":"2023-09-10T01:01:56.033359+0000","last_became_active":"2023-09-10T01:00:48.070350+0000","last_became_peered":"2023-09-10T01:00:48.070350+0000","last_unstale":"2023-09-10T01:01:56.033359+0000","last_undegraded":"2023-09-10T01:01:56.033359+0000","last_fullsized":"2023-09-10T01:01:56.033359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:35:25.196036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036081+0000","last_change":"2023-09-10T01:00:43.005398+0000","last_active":"2023-09-10T01:01:56.036081+0000","last_peered":"2023-09-10T01:01:56.036081+0000","last_clean":"2023-09-10T01:01:56.036081+0000","last_became_active":"2023-09-10T01:00:43.005087+0000","last_became_peered":"2023-09-10T01:00:43.005087+0000","last_unstale":"2023-09-10T01:01:56.036081+0000","last_undegraded":"2023-09-10T01:01:56.036081+0000","last_fullsized":"2023-09-10T01:01:56.036081+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:29:04.042683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127507+0000","last_change":"2023-09-10T01:00:44.024451+0000","last_active":"2023-09-10T01:01:56.127507+0000","last_peered":"2023-09-10T01:01:56.127507+0000","last_clean":"2023-09-10T01:01:56.127507+0000","last_became_active":"2023-09-10T01:00:44.024045+0000","last_became_peered":"2023-09-10T01:00:44.024045+0000","last_unstale":"2023-09-10T01:01:56.127507+0000","last_undegraded":"2023-09-10T01:01:56.127507+0000","last_fullsized":"2023-09-10T01:01:56.127507+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:55:50.750994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037240+0000","last_change":"2023-09-10T01:00:48.062247+0000","last_active":"2023-09-10T01:01:56.037240+0000","last_peered":"2023-09-10T01:01:56.037240+0000","last_clean":"2023-09-10T01:01:56.037240+0000","last_became_active":"2023-09-10T01:00:48.061812+0000","last_became_peered":"2023-09-10T01:00:48.061812+0000","last_unstale":"2023-09-10T01:01:56.037240+0000","last_undegraded":"2023-09-10T01:01:56.037240+0000","last_fullsized":"2023-09-10T01:01:56.037240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:44:13.596534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032713+0000","last_change":"2023-09-10T01:00:43.010845+0000","last_active":"2023-09-10T01:01:56.032713+0000","last_peered":"2023-09-10T01:01:56.032713+0000","last_clean":"2023-09-10T01:01:56.032713+0000","last_became_active":"2023-09-10T01:00:43.010511+0000","last_became_peered":"2023-09-10T01:00:43.010511+0000","last_unstale":"2023-09-10T01:01:56.032713+0000","last_undegraded":"2023-09-10T01:01:56.032713+0000","last_fullsized":"2023-09-10T01:01:56.032713+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:33:07.229023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036175+0000","last_change":"2023-09-10T01:00:44.029025+0000","last_active":"2023-09-10T01:01:56.036175+0000","last_peered":"2023-09-10T01:01:56.036175+0000","last_clean":"2023-09-10T01:01:56.036175+0000","last_became_active":"2023-09-10T01:00:44.028713+0000","last_became_peered":"2023-09-10T01:00:44.028713+0000","last_unstale":"2023-09-10T01:01:56.036175+0000","last_undegraded":"2023-09-10T01:01:56.036175+0000","last_fullsized":"2023-09-10T01:01:56.036175+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:40:22.315742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037817+0000","last_change":"2023-09-10T01:00:48.427294+0000","last_active":"2023-09-10T01:01:56.037817+0000","last_peered":"2023-09-10T01:01:56.037817+0000","last_clean":"2023-09-10T01:01:56.037817+0000","last_became_active":"2023-09-10T01:00:48.426882+0000","last_became_peered":"2023-09-10T01:00:48.426882+0000","last_unstale":"2023-09-10T01:01:56.037817+0000","last_undegraded":"2023-09-10T01:01:56.037817+0000","last_fullsized":"2023-09-10T01:01:56.037817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:54:38.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035921+0000","last_change":"2023-09-10T01:00:43.006175+0000","last_active":"2023-09-10T01:01:56.035921+0000","last_peered":"2023-09-10T01:01:56.035921+0000","last_clean":"2023-09-10T01:01:56.035921+0000","last_became_active":"2023-09-10T01:00:43.005786+0000","last_became_peered":"2023-09-10T01:00:43.005786+0000","last_unstale":"2023-09-10T01:01:56.035921+0000","last_undegraded":"2023-09-10T01:01:56.035921+0000","last_fullsized":"2023-09-10T01:01:56.035921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:14:47.074866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042604+0000","last_change":"2023-09-10T01:00:44.041303+0000","last_active":"2023-09-10T01:01:56.042604+0000","last_peered":"2023-09-10T01:01:56.042604+0000","last_clean":"2023-09-10T01:01:56.042604+0000","last_became_active":"2023-09-10T01:00:44.041009+0000","last_became_peered":"2023-09-10T01:00:44.041009+0000","last_unstale":"2023-09-10T01:01:56.042604+0000","last_undegraded":"2023-09-10T01:01:56.042604+0000","last_fullsized":"2023-09-10T01:01:56.042604+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:53:23.001700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036842+0000","last_change":"2023-09-10T01:00:48.064314+0000","last_active":"2023-09-10T01:01:56.036842+0000","last_peered":"2023-09-10T01:01:56.036842+0000","last_clean":"2023-09-10T01:01:56.036842+0000","last_became_active":"2023-09-10T01:00:48.063813+0000","last_became_peered":"2023-09-10T01:00:48.063813+0000","last_unstale":"2023-09-10T01:01:56.036842+0000","last_undegraded":"2023-09-10T01:01:56.036842+0000","last_fullsized":"2023-09-10T01:01:56.036842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:00:10.002006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033268+0000","last_change":"2023-09-10T01:00:43.007579+0000","last_active":"2023-09-10T01:01:56.033268+0000","last_peered":"2023-09-10T01:01:56.033268+0000","last_clean":"2023-09-10T01:01:56.033268+0000","last_became_active":"2023-09-10T01:00:43.007200+0000","last_became_peered":"2023-09-10T01:00:43.007200+0000","last_unstale":"2023-09-10T01:01:56.033268+0000","last_undegraded":"2023-09-10T01:01:56.033268+0000","last_fullsized":"2023-09-10T01:01:56.033268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:20:08.688531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042996+0000","last_change":"2023-09-10T01:00:44.033511+0000","last_active":"2023-09-10T01:01:56.042996+0000","last_peered":"2023-09-10T01:01:56.042996+0000","last_clean":"2023-09-10T01:01:56.042996+0000","last_became_active":"2023-09-10T01:00:44.033170+0000","last_became_peered":"2023-09-10T01:00:44.033170+0000","last_unstale":"2023-09-10T01:01:56.042996+0000","last_undegraded":"2023-09-10T01:01:56.042996+0000","last_fullsized":"2023-09-10T01:01:56.042996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:36:12.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039255+0000","last_change":"2023-09-10T01:00:48.067436+0000","last_active":"2023-09-10T01:01:56.039255+0000","last_peered":"2023-09-10T01:01:56.039255+0000","last_clean":"2023-09-10T01:01:56.039255+0000","last_became_active":"2023-09-10T01:00:48.067038+0000","last_became_peered":"2023-09-10T01:00:48.067038+0000","last_unstale":"2023-09-10T01:01:56.039255+0000","last_undegraded":"2023-09-10T01:01:56.039255+0000","last_fullsized":"2023-09-10T01:01:56.039255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:26:41.210698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037337+0000","last_change":"2023-09-10T01:00:43.006934+0000","last_active":"2023-09-10T01:01:56.037337+0000","last_peered":"2023-09-10T01:01:56.037337+0000","last_clean":"2023-09-10T01:01:56.037337+0000","last_became_active":"2023-09-10T01:00:43.005787+0000","last_became_peered":"2023-09-10T01:00:43.005787+0000","last_unstale":"2023-09-10T01:01:56.037337+0000","last_undegraded":"2023-09-10T01:01:56.037337+0000","last_fullsized":"2023-09-10T01:01:56.037337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:48:27.403220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042682+0000","last_change":"2023-09-10T01:00:44.033812+0000","last_active":"2023-09-10T01:01:56.042682+0000","last_peered":"2023-09-10T01:01:56.042682+0000","last_clean":"2023-09-10T01:01:56.042682+0000","last_became_active":"2023-09-10T01:00:44.033496+0000","last_became_peered":"2023-09-10T01:00:44.033496+0000","last_unstale":"2023-09-10T01:01:56.042682+0000","last_undegraded":"2023-09-10T01:01:56.042682+0000","last_fullsized":"2023-09-10T01:01:56.042682+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:12.679368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036166+0000","last_change":"2023-09-10T01:00:48.060358+0000","last_active":"2023-09-10T01:01:56.036166+0000","last_peered":"2023-09-10T01:01:56.036166+0000","last_clean":"2023-09-10T01:01:56.036166+0000","last_became_active":"2023-09-10T01:00:48.059853+0000","last_became_peered":"2023-09-10T01:00:48.059853+0000","last_unstale":"2023-09-10T01:01:56.036166+0000","last_undegraded":"2023-09-10T01:01:56.036166+0000","last_fullsized":"2023-09-10T01:01:56.036166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:31:56.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031904+0000","last_change":"2023-09-10T01:00:43.002820+0000","last_active":"2023-09-10T01:01:56.031904+0000","last_peered":"2023-09-10T01:01:56.031904+0000","last_clean":"2023-09-10T01:01:56.031904+0000","last_became_active":"2023-09-10T01:00:43.002456+0000","last_became_peered":"2023-09-10T01:00:43.002456+0000","last_unstale":"2023-09-10T01:01:56.031904+0000","last_undegraded":"2023-09-10T01:01:56.031904+0000","last_fullsized":"2023-09-10T01:01:56.031904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:40:10.189942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.043029+0000","last_change":"2023-09-10T01:00:44.032655+0000","last_active":"2023-09-10T01:01:56.043029+0000","last_peered":"2023-09-10T01:01:56.043029+0000","last_clean":"2023-09-10T01:01:56.043029+0000","last_became_active":"2023-09-10T01:00:44.032157+0000","last_became_peered":"2023-09-10T01:00:44.032157+0000","last_unstale":"2023-09-10T01:01:56.043029+0000","last_undegraded":"2023-09-10T01:01:56.043029+0000","last_fullsized":"2023-09-10T01:01:56.043029+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:31.176768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127525+0000","last_change":"2023-09-10T01:00:44.022285+0000","last_active":"2023-09-10T01:01:56.127525+0000","last_peered":"2023-09-10T01:01:56.127525+0000","last_clean":"2023-09-10T01:01:56.127525+0000","last_became_active":"2023-09-10T01:00:44.021959+0000","last_became_peered":"2023-09-10T01:00:44.021959+0000","last_unstale":"2023-09-10T01:01:56.127525+0000","last_undegraded":"2023-09-10T01:01:56.127525+0000","last_fullsized":"2023-09-10T01:01:56.127525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:13:07.072249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032781+0000","last_change":"2023-09-10T01:00:44.029722+0000","last_active":"2023-09-10T01:01:56.032781+0000","last_peered":"2023-09-10T01:01:56.032781+0000","last_clean":"2023-09-10T01:01:56.032781+0000","last_became_active":"2023-09-10T01:00:44.029424+0000","last_became_peered":"2023-09-10T01:00:44.029424+0000","last_unstale":"2023-09-10T01:01:56.032781+0000","last_undegraded":"2023-09-10T01:01:56.032781+0000","last_fullsized":"2023-09-10T01:01:56.032781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:24:16.655538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032601+0000","last_change":"2023-09-10T01:00:44.026026+0000","last_active":"2023-09-10T01:01:56.032601+0000","last_peered":"2023-09-10T01:01:56.032601+0000","last_clean":"2023-09-10T01:01:56.032601+0000","last_became_active":"2023-09-10T01:00:44.025727+0000","last_became_peered":"2023-09-10T01:00:44.025727+0000","last_unstale":"2023-09-10T01:01:56.032601+0000","last_undegraded":"2023-09-10T01:01:56.032601+0000","last_fullsized":"2023-09-10T01:01:56.032601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:59.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039452+0000","last_change":"2023-09-10T01:00:56.965355+0000","last_active":"2023-09-10T01:01:56.039452+0000","last_peered":"2023-09-10T01:01:56.039452+0000","last_clean":"2023-09-10T01:01:56.039452+0000","last_became_active":"2023-09-10T01:00:56.964903+0000","last_became_peered":"2023-09-10T01:00:56.964903+0000","last_unstale":"2023-09-10T01:01:56.039452+0000","last_undegraded":"2023-09-10T01:01:56.039452+0000","last_fullsized":"2023-09-10T01:01:56.039452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:49:21.940159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031377+0000","last_change":"2023-09-10T01:00:44.029525+0000","last_active":"2023-09-10T01:01:56.031377+0000","last_peered":"2023-09-10T01:01:56.031377+0000","last_clean":"2023-09-10T01:01:56.031377+0000","last_became_active":"2023-09-10T01:00:44.029139+0000","last_became_peered":"2023-09-10T01:00:44.029139+0000","last_unstale":"2023-09-10T01:01:56.031377+0000","last_undegraded":"2023-09-10T01:01:56.031377+0000","last_fullsized":"2023-09-10T01:01:56.031377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:20:24.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127877+0000","last_change":"2023-09-10T01:00:44.022299+0000","last_active":"2023-09-10T01:01:56.127877+0000","last_peered":"2023-09-10T01:01:56.127877+0000","last_clean":"2023-09-10T01:01:56.127877+0000","last_became_active":"2023-09-10T01:00:44.021992+0000","last_became_peered":"2023-09-10T01:00:44.021992+0000","last_unstale":"2023-09-10T01:01:56.127877+0000","last_undegraded":"2023-09-10T01:01:56.127877+0000","last_fullsized":"2023-09-10T01:01:56.127877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:50:30.766442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.108199+0000","last_change":"2023-09-10T01:01:57.108199+0000","last_active":"2023-09-10T01:01:57.108199+0000","last_peered":"2023-09-10T01:01:57.108199+0000","last_clean":"2023-09-10T01:01:57.108199+0000","last_became_active":"2023-09-10T01:01:57.107825+0000","last_became_peered":"2023-09-10T01:01:57.107825+0000","last_unstale":"2023-09-10T01:01:57.108199+0000","last_undegraded":"2023-09-10T01:01:57.108199+0000","last_fullsized":"2023-09-10T01:01:57.108199+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:31:15.499277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,9],"acting":[2,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.050651+0000","last_change":"2023-09-10T01:01:57.049936+0000","last_active":"2023-09-10T01:01:57.050651+0000","last_peered":"2023-09-10T01:01:57.050651+0000","last_clean":"2023-09-10T01:01:57.050651+0000","last_became_active":"2023-09-10T01:01:57.049624+0000","last_became_peered":"2023-09-10T01:01:57.049624+0000","last_unstale":"2023-09-10T01:01:57.050651+0000","last_undegraded":"2023-09-10T01:01:57.050651+0000","last_fullsized":"2023-09-10T01:01:57.050651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:57:17.164558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036462+0000","last_change":"2023-09-10T01:00:44.025405+0000","last_active":"2023-09-10T01:01:56.036462+0000","last_peered":"2023-09-10T01:01:56.036462+0000","last_clean":"2023-09-10T01:01:56.036462+0000","last_became_active":"2023-09-10T01:00:44.025040+0000","last_became_peered":"2023-09-10T01:00:44.025040+0000","last_unstale":"2023-09-10T01:01:56.036462+0000","last_undegraded":"2023-09-10T01:01:56.036462+0000","last_fullsized":"2023-09-10T01:01:56.036462+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:16:46.504902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036192+0000","last_change":"2023-09-10T01:00:44.023603+0000","last_active":"2023-09-10T01:01:56.036192+0000","last_peered":"2023-09-10T01:01:56.036192+0000","last_clean":"2023-09-10T01:01:56.036192+0000","last_became_active":"2023-09-10T01:00:44.023279+0000","last_became_peered":"2023-09-10T01:00:44.023279+0000","last_unstale":"2023-09-10T01:01:56.036192+0000","last_undegraded":"2023-09-10T01:01:56.036192+0000","last_fullsized":"2023-09-10T01:01:56.036192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:18:05.737675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037149+0000","last_change":"2023-09-10T01:00:44.035653+0000","last_active":"2023-09-10T01:01:56.037149+0000","last_peered":"2023-09-10T01:01:56.037149+0000","last_clean":"2023-09-10T01:01:56.037149+0000","last_became_active":"2023-09-10T01:00:44.035374+0000","last_became_peered":"2023-09-10T01:00:44.035374+0000","last_unstale":"2023-09-10T01:01:56.037149+0000","last_undegraded":"2023-09-10T01:01:56.037149+0000","last_fullsized":"2023-09-10T01:01:56.037149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:14:45.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030922+0000","last_change":"2023-09-10T01:00:44.035805+0000","last_active":"2023-09-10T01:01:56.030922+0000","last_peered":"2023-09-10T01:01:56.030922+0000","last_clean":"2023-09-10T01:01:56.030922+0000","last_became_active":"2023-09-10T01:00:44.035479+0000","last_became_peered":"2023-09-10T01:00:44.035479+0000","last_unstale":"2023-09-10T01:01:56.030922+0000","last_undegraded":"2023-09-10T01:01:56.030922+0000","last_fullsized":"2023-09-10T01:01:56.030922+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:36.662975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127692+0000","last_change":"2023-09-10T01:00:44.025418+0000","last_active":"2023-09-10T01:01:56.127692+0000","last_peered":"2023-09-10T01:01:56.127692+0000","last_clean":"2023-09-10T01:01:56.127692+0000","last_became_active":"2023-09-10T01:00:44.025104+0000","last_became_peered":"2023-09-10T01:00:44.025104+0000","last_unstale":"2023-09-10T01:01:56.127692+0000","last_undegraded":"2023-09-10T01:01:56.127692+0000","last_fullsized":"2023-09-10T01:01:56.127692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:05:13.050781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282844,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63889,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":342,"num_read_kb":239,"num_write":152,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":191667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481820,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":504243,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69199999999999995,"5min":0.87,"15min":0.87},"last":0.498},{"interface":"front","average":{"1min":0.874,"5min":0.60799999999999998,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.54000000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.626,"15min":0.625},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82299999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.76100000000000001}]},{"osd":2,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45500000000000002,"15min":0.45600000000000002},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.74399999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.53000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63200000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.74399999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.73799999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81799999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.69799999999999995},"min":{"1min":0.495,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.83299999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.84699999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69399999999999995,"15min":0.69199999999999995},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.85599999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.877}]},{"osd":8,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.91500000000000004}]},{"osd":9,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.748,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.92300000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.93300000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93100000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.55900000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.95999999999999996}]}]},{"osd":10,"up_from":70,"seq":300647710751,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":1150818,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.502},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44600000000000001,"15min":0.441},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.57199999999999995}]},{"osd":2,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.747},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.628,"15min":0.63200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.82899999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.626}]},{"osd":3,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67700000000000005,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.84399999999999997,"15min":0.874},"min":{"1min":0.46300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81200000000000006,"5min":6.7519999999999998,"15min":6.7519999999999998},"last":0.78000000000000003}]},{"osd":4,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73499999999999999,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55600000000000005,"15min":0.55500000000000005},"min":{"1min":0.36299999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80400000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68899999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69599999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83999999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.443,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.89400000000000002}]},{"osd":6,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.71699999999999997,"15min":0.71499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95299999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.47999999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91300000000000003,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.91300000000000003}]},{"osd":7,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89300000000000002,"5min":1.534,"15min":1.534},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.52600000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.96599999999999997,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.76300000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.78500000000000003,"15min":0.78800000000000003},"min":{"1min":0.57499999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92000000000000004,"5min":1.304,"15min":1.304},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":0.85499999999999998,"15min":0.81699999999999995},"min":{"1min":0.51400000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":7.0510000000000002,"5min":7.0510000000000002,"15min":7.0510000000000002},"last":0.94299999999999995}]},{"osd":11,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.84299999999999997}]}]},{"osd":9,"up_from":65,"seq":279172874276,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":507993,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.32100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.39600000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.42599999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.73099999999999998,"15min":0.74199999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93200000000000005,"5min":1.349,"15min":1.349},"last":0.46899999999999997}]},{"osd":2,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.70699999999999996,"15min":0.70799999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.038,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.81200000000000006,"15min":0.82599999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.96399999999999997,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.67600000000000005}]},{"osd":3,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.69799999999999995,"15min":0.71099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2010000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.073,"5min":1.647,"15min":1.647},"last":0.66400000000000003}]},{"osd":4,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.82999999999999996,"15min":0.84499999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.052,"5min":1.595,"15min":1.595},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.85399999999999998,"15min":0.86799999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3080000000000001,"5min":1.492,"15min":1.492},"last":0.60099999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.91700000000000004,"15min":0.93799999999999994},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1339999999999999,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.88100000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.179,"5min":1.714,"15min":1.714},"last":0.75600000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.95999999999999996,"15min":0.98299999999999998},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.226,"5min":1.982,"15min":1.982},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.92000000000000004,"15min":0.93600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.3280000000000001,"5min":1.569,"15min":1.569},"last":0.79200000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.98899999999999999,"15min":1.0129999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2050000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.874,"5min":1,"15min":1.0209999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.274,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.85099999999999998}]},{"osd":10,"last update":"Sun Sep 10 01:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.91000000000000003,"15min":0.92900000000000005},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.151,"5min":1.363,"15min":1.363},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":0.95199999999999996},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.121,"5min":1.319,"15min":1.319},"last":0.77200000000000002}]},{"osd":11,"last update":"Sun Sep 10 01:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.82599999999999996}]}]},{"osd":8,"up_from":57,"seq":244813135913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":503486,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80200000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68700000000000006,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.66700000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86499999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.753},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.878,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.39400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.627},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.47399999999999998,"15min":0.45400000000000001},"min":{"1min":0.35999999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.73099999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46800000000000003}]},{"osd":3,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.69099999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.504}]},{"osd":4,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.56100000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97199999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82799999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81699999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.60399999999999998,"15min":0.58799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.79900000000000004}]},{"osd":6,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.52100000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73999999999999999,"15min":0.73299999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.004,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.72899999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.70499999999999996,"15min":0.67000000000000004},"min":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.879},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.72899999999999998,"15min":0.70899999999999996},"min":{"1min":0.60099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.89300000000000002}]},{"osd":9,"last update":"Sun Sep 10 01:01:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.76100000000000001,"15min":0.75800000000000001},"min":{"1min":0.55800000000000005,"5min":0.505,"15min":0.505},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75600000000000001},"min":{"1min":0.64200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.94099999999999995,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.85099999999999998}]},{"osd":10,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.72599999999999998,"15min":0.71199999999999997},"min":{"1min":0.69599999999999995,"5min":0.44,"15min":0.44},"max":{"1min":0.96999999999999997,"5min":1.01,"15min":1.01},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.76200000000000001,"15min":0.74199999999999999},"min":{"1min":0.70699999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.94999999999999996}]},{"osd":11,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.97599999999999998}]}]},{"osd":7,"up_from":50,"seq":214748364846,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":1099143,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91400000000000003,"5min":1.367,"15min":1.367},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.44400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.43099999999999999,"15min":0.41999999999999998},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45100000000000001,"15min":0.437},"min":{"1min":0.317,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76100000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.67900000000000005}]},{"osd":2,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59799999999999998,"15min":0.58499999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85999999999999999,"5min":1.409,"15min":1.409},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.71499999999999997,"15min":0.68300000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":7.1929999999999996,"5min":7.1929999999999996,"15min":7.1929999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.41699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69799999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.94899999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63100000000000001,"15min":0.62},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85799999999999998,"5min":1.038,"15min":1.038},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.64100000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.81799999999999995,"15min":0.75800000000000001},"min":{"1min":0.505,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":9.3919999999999995,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.82499999999999996,"15min":0.79000000000000004},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":7.1760000000000002,"5min":7.1760000000000002,"15min":7.1760000000000002},"last":0.77900000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.012,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70099999999999996,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98399999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.81100000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77300000000000002,"15min":0.77200000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.79000000000000004,"15min":0.78600000000000003},"min":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.875}]},{"osd":10,"last update":"Sun Sep 10 01:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.749,"15min":0.747},"min":{"1min":0.56899999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.90600000000000003}]},{"osd":11,"last update":"Sun Sep 10 01:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.94699999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528369,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":499073,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52800000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88400000000000001,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54000000000000004,"15min":0.52300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86599999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.65600000000000003,"15min":0.624},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98199999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91200000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.73499999999999999,"15min":0.749},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.044,"5min":1.069,"15min":1.069},"last":0.73499999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65400000000000003,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.014,"5min":1.147,"15min":1.147},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88400000000000001,"5min":1.425,"15min":1.425},"last":0.56599999999999995}]},{"osd":4,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.73799999999999999,"15min":0.73999999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95099999999999996,"5min":1.111,"15min":1.111},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.753,"15min":0.73399999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.68300000000000005}]},{"osd":5,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.84099999999999997,"15min":0.85499999999999998},"min":{"1min":0.55300000000000005,"5min":0.503,"15min":0.503},"max":{"1min":0.93700000000000006,"5min":1.53,"15min":1.53},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.83799999999999997,"15min":0.84299999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1,"5min":1.353,"15min":1.353},"last":0.61399999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":0.94699999999999995,"15min":0.86099999999999999},"min":{"1min":0.68100000000000005,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":10.081,"5min":10.081,"15min":10.081},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.004,"5min":0.86799999999999999,"15min":0.83799999999999997},"min":{"1min":0.70299999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.55100000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:01:58 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.92900000000000005,"15min":0.91700000000000004},"min":{"1min":0.64900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89200000000000002,"15min":0.88600000000000001},"min":{"1min":0.65500000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.79200000000000004}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.97799999999999998,"15min":0.99399999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1699999999999999,"5min":1.831,"15min":1.831},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0109999999999999,"15min":1.0329999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1539999999999999,"5min":1.617,"15min":1.617},"last":0.82899999999999996}]},{"osd":10,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96099999999999997,"15min":0.95799999999999996},"min":{"1min":0.75700000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.069,"15min":0.99299999999999999},"min":{"1min":0.78700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":10.170999999999999,"5min":10.170999999999999,"15min":10.170999999999999},"last":0.80500000000000005}]},{"osd":11,"last update":"Sun Sep 10 01:01:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":10.247,"5min":10.247,"15min":10.247},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3660000000000001,"5min":2.3660000000000001,"15min":2.3660000000000001},"last":0.75600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":558872,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56499999999999995,"15min":0.52700000000000002},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.82199999999999995,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58999999999999997,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72599999999999998,"5min":5.1580000000000004,"15min":5.1580000000000004},"last":0.48399999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.56899999999999995,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.42999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.58299999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:01:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.752},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.755,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.78000000000000003},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84399999999999997,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.78900000000000003}]},{"osd":4,"last update":"Sun Sep 10 01:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.41999999999999998,"15min":0.375},"min":{"1min":0.35399999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.42099999999999999,"15min":0.38200000000000001},"min":{"1min":0.30399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.371}]},{"osd":5,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.65400000000000003,"15min":0.57799999999999996},"min":{"1min":0.42199999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.9740000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.028,"5min":0.68899999999999995,"15min":0.60499999999999998},"min":{"1min":0.54600000000000004,"5min":0.38,"15min":0.38},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.63400000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.47699999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.60999999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56599999999999995,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.64500000000000002}]},{"osd":8,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.68000000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.44,"15min":0.44},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81599999999999995}]},{"osd":9,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.747,"15min":0.73699999999999999},"min":{"1min":0.63600000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78400000000000003,"15min":0.77900000000000003},"min":{"1min":0.63900000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.87}]},{"osd":10,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69699999999999995,"15min":0.68899999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.80300000000000005}]},{"osd":11,"last update":"Sun Sep 10 01:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.68799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500461,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 10 01:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.622,"15min":0.58899999999999997},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97299999999999998,"5min":1.069,"15min":1.069},"last":1.597},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58799999999999997,"15min":0.55600000000000005},"min":{"1min":0.45400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81499999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":1.4410000000000001}]},{"osd":2,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.66700000000000004,"15min":0.60599999999999998},"min":{"1min":0.39600000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.626},"min":{"1min":0.51000000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98399999999999999,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":1.8999999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.439},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.92200000000000004}]},{"osd":4,"last update":"Sun Sep 10 01:01:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.63800000000000001,"15min":0.58399999999999996},"min":{"1min":0.57699999999999996,"5min":0.308,"15min":0.308},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":1.397},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66500000000000004,"15min":0.60799999999999998},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.83099999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.747,"15min":0.71199999999999997},"min":{"1min":0.59799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.028,"5min":1.125,"15min":1.125},"last":1.643},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70699999999999996,"15min":0.66800000000000004},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":1.2969999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68999999999999995,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.876},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.71399999999999997},"min":{"1min":0.496,"5min":0.376,"15min":0.376},"max":{"1min":1.0720000000000001,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":1.7470000000000001}]},{"osd":7,"last update":"Sun Sep 10 01:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":1.863},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":1.482}]},{"osd":8,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.66400000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.0840000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":1.7130000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83299999999999996,"15min":0.84299999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1499999999999999,"5min":1.24,"15min":1.24},"last":1.3560000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.91300000000000003,"15min":0.92900000000000005},"min":{"1min":0.64300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1519999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.999,"15min":1.0149999999999999},"min":{"1min":0.76900000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1870000000000001,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":1.778}]},{"osd":10,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.91400000000000003,"15min":0.92600000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.117,"5min":1.405,"15min":1.405},"last":1.5209999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.93500000000000005,"15min":0.94199999999999995},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.319,"5min":1.444,"15min":1.444},"last":0.97499999999999998}]},{"osd":11,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":1.006},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":1.6779999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":559600,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.64400000000000002,"15min":0.54300000000000004},"min":{"1min":0.46999999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.7199999999999998,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.63600000000000001,"15min":0.55800000000000005},"min":{"1min":0.51200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":6.6639999999999997,"5min":6.6639999999999997,"15min":6.6639999999999997},"last":0.66400000000000003}]},{"osd":1,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60399999999999998,"15min":0.54700000000000004},"min":{"1min":0.48499999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.61199999999999999,"15min":0.55200000000000005},"min":{"1min":0.61099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.88800000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.56899999999999995},"min":{"1min":0.49399999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.036,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.48099999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.89400000000000002,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.83499999999999996}]},{"osd":4,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.91200000000000003}]},{"osd":5,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.498,"15min":0.44600000000000001},"min":{"1min":0.434,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.34200000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.86899999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:01:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68899999999999995,"15min":0.66600000000000004},"min":{"1min":0.42599999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.073,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.68799999999999994,"15min":0.67200000000000004},"min":{"1min":0.59899999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81999999999999995}]},{"osd":7,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":1.105},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.95799999999999996}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56399999999999995,"15min":0.54500000000000004},"min":{"1min":0.51000000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":1.05},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.73299999999999998,"15min":0.71499999999999997},"min":{"1min":0.59499999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.97099999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.79900000000000004,"15min":0.78800000000000003},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.72899999999999998,"15min":0.72099999999999997},"min":{"1min":0.622,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.91700000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.1539999999999999}]},{"osd":10,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.68600000000000005,"15min":0.66600000000000004},"min":{"1min":0.59299999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":1.117},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.79500000000000004,"15min":0.77400000000000002},"min":{"1min":0.68100000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.98599999999999999}]},{"osd":11,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.016}]}]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":499267,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41999999999999998,"15min":0.39000000000000001},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65000000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.42699999999999999},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.49399999999999999}]},{"osd":1,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.52700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.117,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56100000000000005},"min":{"1min":0.40999999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.90100000000000002,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65900000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68300000000000005,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.012,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68999999999999995,"15min":0.63600000000000001},"min":{"1min":0.499,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0820000000000001,"5min":1.226,"15min":1.226},"last":0.88}]},{"osd":4,"last update":"Sun Sep 10 01:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.72699999999999998,"15min":0.68200000000000005},"min":{"1min":0.45200000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.036,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.77500000000000002,"15min":0.71999999999999997},"min":{"1min":0.57199999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.0309999999999999,"5min":2.52,"15min":2.52},"last":0.82199999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.79500000000000004,"15min":0.73299999999999998},"min":{"1min":0.51600000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":0.73299999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:01:13 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65400000000000003,"15min":0.623},"min":{"1min":0.5,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.70899999999999996,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.127,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:01:32 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78100000000000003,"15min":0.79300000000000004},"min":{"1min":0.499,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7969999999999999,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.046,"5min":0.876,"15min":0.83799999999999997},"min":{"1min":0.55400000000000005,"5min":0.434,"15min":0.434},"max":{"1min":4.7699999999999996,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.69299999999999995}]},{"osd":8,"last update":"Sun Sep 10 00:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.94799999999999995,"15min":0.97599999999999998},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0549999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.87,"5min":0.92200000000000004,"15min":0.93100000000000005},"min":{"1min":0.61399999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.26,"5min":1.458,"15min":1.458},"last":0.92600000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:01:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.95999999999999996,"15min":0.96599999999999997},"min":{"1min":0.70699999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82599999999999996,"15min":0.82199999999999995},"min":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.048}]},{"osd":11,"last update":"Sun Sep 10 01:01:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":1.03},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":4.9139999999999997},"last":0.85699999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":504735,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.38700000000000001,"15min":0.36099999999999999},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.73899999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.39700000000000002,"15min":0.377},"min":{"1min":0.29099999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64200000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.56100000000000005},"min":{"1min":0.40200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":1.607,"15min":1.607},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.52700000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88900000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.76100000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59199999999999997,"15min":0.56999999999999995},"min":{"1min":0.45600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sun Sep 10 01:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.64300000000000002,"15min":0.60699999999999998},"min":{"1min":0.55000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":1.621,"15min":1.621},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.63300000000000001,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71899999999999997}]},{"osd":6,"last update":"Sun Sep 10 01:01:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80400000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93100000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.70499999999999996}]},{"osd":7,"last update":"Sun Sep 10 01:01:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.57399999999999995,"15min":0.54100000000000004},"min":{"1min":0.41199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.91400000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.77900000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91100000000000003,"5min":1.802,"15min":1.802},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.755,"15min":0.73299999999999998},"min":{"1min":0.56100000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.81000000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:01:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76200000000000001,"15min":0.77000000000000002},"min":{"1min":0.505,"5min":0.495,"15min":0.495},"max":{"1min":0.88400000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.79400000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78100000000000003,"15min":0.77800000000000002},"min":{"1min":0.64800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.65700000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84699999999999998}]},{"osd":11,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.874}]}]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1095313,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54000000000000004,"15min":0.50600000000000001},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81399999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.879},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.53100000000000003,"15min":0.50700000000000001},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79800000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.75900000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.55500000000000005},"min":{"1min":0.48199999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86099999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.59899999999999998,"15min":0.55700000000000005},"min":{"1min":0.55600000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.42199999999999999,"15min":0.38700000000000001},"min":{"1min":0.36699999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67300000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.442,"15min":0.41199999999999998},"min":{"1min":0.374,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44800000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.378,"5min":0.376,"15min":0.376},"max":{"1min":0.752,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.60199999999999998,"15min":0.55800000000000005},"min":{"1min":0.501,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.75}]},{"osd":4,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.622,"15min":0.59199999999999997},"min":{"1min":0.53500000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.627,"15min":0.59999999999999998},"min":{"1min":0.51300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.627}]},{"osd":6,"last update":"Sun Sep 10 01:01:16 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.64400000000000002,"15min":0.629},"min":{"1min":0.53300000000000003,"5min":0.437,"15min":0.437},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.54200000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.80300000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:01:33 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.68000000000000005,"15min":0.67000000000000004},"min":{"1min":0.48499999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91400000000000003,"5min":1.042,"15min":1.042},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0549999999999999,"5min":1.081,"15min":1.081},"last":0.52600000000000002}]},{"osd":8,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55300000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69999999999999996,"15min":0.68999999999999995},"min":{"1min":0.55900000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.93799999999999994,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77100000000000002,"15min":0.76100000000000001},"min":{"1min":0.59799999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 10 01:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0720000000000001,"5min":1.107,"15min":1.107},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.78500000000000003,"15min":0.76700000000000002},"min":{"1min":0.54500000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.64300000000000002}]},{"osd":11,"last update":"Sun Sep 10 01:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-10T01:02:04.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph pg dump --format=json 2023-09-10T01:02:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:05.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:04 smithi092 ceph-mon[134067]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:05.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:04 smithi092 ceph-mon[134067]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:05.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:04 smithi003 ceph-mon[97654]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:05.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:04 smithi003 ceph-mon[97654]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:04 smithi136 ceph-mon[133708]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:04 smithi136 ceph-mon[133708]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:06.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 -- 172.21.15.3:0/1413573106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 msgr2=0x7f9fd010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:06.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/1413573106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd010d4f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9fc800b0a0 tx=0x7f9fc800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 -- 172.21.15.3:0/1413573106 shutdown_connections 2023-09-10T01:02:06.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/1413573106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd010d4f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/1413573106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fd0106620 0x7f9fd0106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/1413573106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd0100620 0x7f9fd0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 -- 172.21.15.3:0/1413573106 >> 172.21.15.3:0/1413573106 conn(0x7f9fd00fc0f0 msgr2=0x7f9fd00fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:06.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 -- 172.21.15.3:0/1413573106 shutdown_connections 2023-09-10T01:02:06.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.751+0000 7f9fd87c3700 1 -- 172.21.15.3:0/1413573106 wait complete. 2023-09-10T01:02:06.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.752+0000 7f9fd87c3700 1 Processor -- start 2023-09-10T01:02:06.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.752+0000 7f9fd87c3700 1 -- start start 2023-09-10T01:02:06.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd87c3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd0100620 0x7f9fd019e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:06.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd87c3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:06.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd87c3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fd0106620 0x7f9fd0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:06.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd87c3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd0198ef0 con 0x7f9fd0100fe0 2023-09-10T01:02:06.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd87c3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd0199060 con 0x7f9fd0106620 2023-09-10T01:02:06.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd87c3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd0199220 con 0x7f9fd0100620 2023-09-10T01:02:06.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd5d5e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:06.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd5d5e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd019ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53918/0 (socket says 172.21.15.3:53918) 2023-09-10T01:02:06.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd5d5e700 1 -- 172.21.15.3:0/3493718281 learned_addr learned my addr 172.21.15.3:0/3493718281 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:06.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd655f700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd0100620 0x7f9fd019e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:06.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.753+0000 7f9fd6d60700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fd0106620 0x7f9fd0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:06.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd5d5e700 1 -- 172.21.15.3:0/3493718281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd0100620 msgr2=0x7f9fd019e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:06.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd5d5e700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd0100620 0x7f9fd019e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd5d5e700 1 -- 172.21.15.3:0/3493718281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fd0106620 msgr2=0x7f9fd0198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:06.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd5d5e700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fd0106620 0x7f9fd0198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd5d5e700 1 -- 172.21.15.3:0/3493718281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc8009d00 con 0x7f9fd0100fe0 2023-09-10T01:02:06.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd655f700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd0100620 0x7f9fd019e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:06.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd5d5e700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd019ed40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000d950 tx=0x7f9fc000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:06.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fc77fe700 1 -- 172.21.15.3:0/3493718281 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc000f890 con 0x7f9fd0100fe0 2023-09-10T01:02:06.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fc77fe700 1 -- 172.21.15.3:0/3493718281 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fc0010460 con 0x7f9fd0100fe0 2023-09-10T01:02:06.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd0199500 con 0x7f9fd0100fe0 2023-09-10T01:02:06.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fc77fe700 1 -- 172.21.15.3:0/3493718281 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc000e610 con 0x7f9fd0100fe0 2023-09-10T01:02:06.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.754+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd01a57f0 con 0x7f9fd0100fe0 2023-09-10T01:02:06.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.755+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd004e300 con 0x7f9fd0100fe0 2023-09-10T01:02:06.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.758+0000 7f9fc77fe700 1 -- 172.21.15.3:0/3493718281 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f9fc00108d0 con 0x7f9fd0100fe0 2023-09-10T01:02:06.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.760+0000 7f9fc77fe700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9fbc0a0750 0x7f9fbc0a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:06.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.760+0000 7f9fc77fe700 1 -- 172.21.15.3:0/3493718281 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f9fc00cbea0 con 0x7f9fd0100fe0 2023-09-10T01:02:06.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.760+0000 7f9fd655f700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9fbc0a0750 0x7f9fbc0a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:06.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.761+0000 7f9fd655f700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9fbc0a0750 0x7f9fbc0a2c10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc007f50 tx=0x7f9fcc0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:06.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.764+0000 7f9fc77fe700 1 -- 172.21.15.3:0/3493718281 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9fc0095460 con 0x7f9fd0100fe0 2023-09-10T01:02:06.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.968+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9fd00008d0 con 0x7f9fbc0a0750 2023-09-10T01:02:06.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.982+0000 7f9fc77fe700 1 -- 172.21.15.3:0/3493718281 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427720 (secure 0 0 0) 0x7f9fd00008d0 con 0x7f9fbc0a0750 2023-09-10T01:02:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:06.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.987+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9fbc0a0750 msgr2=0x7f9fbc0a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:06.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.987+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9fbc0a0750 0x7f9fbc0a2c10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc007f50 tx=0x7f9fcc0073d0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.987+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 msgr2=0x7f9fd019ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.987+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd019ed40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000d950 tx=0x7f9fc000dc60 comp rx=0 tx=0).stop 2023-09-10T01:02:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 shutdown_connections 2023-09-10T01:02:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f9fbc0a0750 0x7f9fbc0a2c10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd0100620 0x7f9fd019e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fd0106620 0x7f9fd0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 --2- 172.21.15.3:0/3493718281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9fd0100fe0 0x7f9fd019ed40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 >> 172.21.15.3:0/3493718281 conn(0x7f9fd00fc0f0 msgr2=0x7f9fd010b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 shutdown_connections 2023-09-10T01:02:06.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:06.988+0000 7f9fd87c3700 1 -- 172.21.15.3:0/3493718281 wait complete. 2023-09-10T01:02:06.994 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-09-10T01:02:07.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:06 smithi003 ceph-mon[97654]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-09-10T01:02:07.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:06 smithi136 ceph-mon[133708]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-09-10T01:02:07.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:06 smithi092 ceph-mon[134067]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-09-10T01:02:07.509 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2023-09-10T01:02:05.868903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654503,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":478,"num_read_kb":358,"num_write":335,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336312,"kb_used_data":12052,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605512,"statfs":{"total":1151940427776,"available":1151596044288,"internally_reserved":0,"allocated":12341248,"data_stored":7983004,"data_compressed":391800,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.840186"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.128038+0000","last_change":"2023-09-10T01:00:44.026223+0000","last_active":"2023-09-10T01:01:56.128038+0000","last_peered":"2023-09-10T01:01:56.128038+0000","last_clean":"2023-09-10T01:01:56.128038+0000","last_became_active":"2023-09-10T01:00:44.025731+0000","last_became_peered":"2023-09-10T01:00:44.025731+0000","last_unstale":"2023-09-10T01:01:56.128038+0000","last_undegraded":"2023-09-10T01:01:56.128038+0000","last_fullsized":"2023-09-10T01:01:56.128038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:51:48.949903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039896+0000","last_change":"2023-09-10T01:00:44.038916+0000","last_active":"2023-09-10T01:01:56.039896+0000","last_peered":"2023-09-10T01:01:56.039896+0000","last_clean":"2023-09-10T01:01:56.039896+0000","last_became_active":"2023-09-10T01:00:44.038546+0000","last_became_peered":"2023-09-10T01:00:44.038546+0000","last_unstale":"2023-09-10T01:01:56.039896+0000","last_undegraded":"2023-09-10T01:01:56.039896+0000","last_fullsized":"2023-09-10T01:01:56.039896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:05:56.096857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.038734+0000","last_change":"2023-09-10T01:00:44.032942+0000","last_active":"2023-09-10T01:01:56.038734+0000","last_peered":"2023-09-10T01:01:56.038734+0000","last_clean":"2023-09-10T01:01:56.038734+0000","last_became_active":"2023-09-10T01:00:44.032546+0000","last_became_peered":"2023-09-10T01:00:44.032546+0000","last_unstale":"2023-09-10T01:01:56.038734+0000","last_undegraded":"2023-09-10T01:01:56.038734+0000","last_fullsized":"2023-09-10T01:01:56.038734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:46.613487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036467+0000","last_change":"2023-09-10T01:00:44.031411+0000","last_active":"2023-09-10T01:01:56.036467+0000","last_peered":"2023-09-10T01:01:56.036467+0000","last_clean":"2023-09-10T01:01:56.036467+0000","last_became_active":"2023-09-10T01:00:44.031121+0000","last_became_peered":"2023-09-10T01:00:44.031121+0000","last_unstale":"2023-09-10T01:01:56.036467+0000","last_undegraded":"2023-09-10T01:01:56.036467+0000","last_fullsized":"2023-09-10T01:01:56.036467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:47.658116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035816+0000","last_change":"2023-09-10T01:00:44.037351+0000","last_active":"2023-09-10T01:01:56.035816+0000","last_peered":"2023-09-10T01:01:56.035816+0000","last_clean":"2023-09-10T01:01:56.035816+0000","last_became_active":"2023-09-10T01:00:44.037005+0000","last_became_peered":"2023-09-10T01:00:44.037005+0000","last_unstale":"2023-09-10T01:01:56.035816+0000","last_undegraded":"2023-09-10T01:01:56.035816+0000","last_fullsized":"2023-09-10T01:01:56.035816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:10:45.694056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127691+0000","last_change":"2023-09-10T01:00:44.029543+0000","last_active":"2023-09-10T01:01:56.127691+0000","last_peered":"2023-09-10T01:01:56.127691+0000","last_clean":"2023-09-10T01:01:56.127691+0000","last_became_active":"2023-09-10T01:00:44.029210+0000","last_became_peered":"2023-09-10T01:00:44.029210+0000","last_unstale":"2023-09-10T01:01:56.127691+0000","last_undegraded":"2023-09-10T01:01:56.127691+0000","last_fullsized":"2023-09-10T01:01:56.127691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:08:56.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042147+0000","last_change":"2023-09-10T01:00:44.033328+0000","last_active":"2023-09-10T01:01:56.042147+0000","last_peered":"2023-09-10T01:01:56.042147+0000","last_clean":"2023-09-10T01:01:56.042147+0000","last_became_active":"2023-09-10T01:00:44.033007+0000","last_became_peered":"2023-09-10T01:00:44.033007+0000","last_unstale":"2023-09-10T01:01:56.042147+0000","last_undegraded":"2023-09-10T01:01:56.042147+0000","last_fullsized":"2023-09-10T01:01:56.042147+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:59:14.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035843+0000","last_change":"2023-09-10T01:00:44.025611+0000","last_active":"2023-09-10T01:01:56.035843+0000","last_peered":"2023-09-10T01:01:56.035843+0000","last_clean":"2023-09-10T01:01:56.035843+0000","last_became_active":"2023-09-10T01:00:44.025249+0000","last_became_peered":"2023-09-10T01:00:44.025249+0000","last_unstale":"2023-09-10T01:01:56.035843+0000","last_undegraded":"2023-09-10T01:01:56.035843+0000","last_fullsized":"2023-09-10T01:01:56.035843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:55:04.320356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036684+0000","last_change":"2023-09-10T01:00:44.024207+0000","last_active":"2023-09-10T01:01:56.036684+0000","last_peered":"2023-09-10T01:01:56.036684+0000","last_clean":"2023-09-10T01:01:56.036684+0000","last_became_active":"2023-09-10T01:00:44.023866+0000","last_became_peered":"2023-09-10T01:00:44.023866+0000","last_unstale":"2023-09-10T01:01:56.036684+0000","last_undegraded":"2023-09-10T01:01:56.036684+0000","last_fullsized":"2023-09-10T01:01:56.036684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:53:39.845384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127301+0000","last_change":"2023-09-10T01:00:44.031241+0000","last_active":"2023-09-10T01:01:56.127301+0000","last_peered":"2023-09-10T01:01:56.127301+0000","last_clean":"2023-09-10T01:01:56.127301+0000","last_became_active":"2023-09-10T01:00:44.030392+0000","last_became_peered":"2023-09-10T01:00:44.030392+0000","last_unstale":"2023-09-10T01:01:56.127301+0000","last_undegraded":"2023-09-10T01:01:56.127301+0000","last_fullsized":"2023-09-10T01:01:56.127301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:19:39.786995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042893+0000","last_change":"2023-09-10T01:00:44.026983+0000","last_active":"2023-09-10T01:01:56.042893+0000","last_peered":"2023-09-10T01:01:56.042893+0000","last_clean":"2023-09-10T01:01:56.042893+0000","last_became_active":"2023-09-10T01:00:44.026695+0000","last_became_peered":"2023-09-10T01:00:44.026695+0000","last_unstale":"2023-09-10T01:01:56.042893+0000","last_undegraded":"2023-09-10T01:01:56.042893+0000","last_fullsized":"2023-09-10T01:01:56.042893+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:37.809144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037500+0000","last_change":"2023-09-10T01:00:44.030787+0000","last_active":"2023-09-10T01:01:56.037500+0000","last_peered":"2023-09-10T01:01:56.037500+0000","last_clean":"2023-09-10T01:01:56.037500+0000","last_became_active":"2023-09-10T01:00:44.028579+0000","last_became_peered":"2023-09-10T01:00:44.028579+0000","last_unstale":"2023-09-10T01:01:56.037500+0000","last_undegraded":"2023-09-10T01:01:56.037500+0000","last_fullsized":"2023-09-10T01:01:56.037500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:04:26.446378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030849+0000","last_change":"2023-09-10T01:00:44.029905+0000","last_active":"2023-09-10T01:01:56.030849+0000","last_peered":"2023-09-10T01:01:56.030849+0000","last_clean":"2023-09-10T01:01:56.030849+0000","last_became_active":"2023-09-10T01:00:44.029561+0000","last_became_peered":"2023-09-10T01:00:44.029561+0000","last_unstale":"2023-09-10T01:01:56.030849+0000","last_undegraded":"2023-09-10T01:01:56.030849+0000","last_fullsized":"2023-09-10T01:01:56.030849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:17:50.839001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039075+0000","last_change":"2023-09-10T01:00:44.037864+0000","last_active":"2023-09-10T01:01:56.039075+0000","last_peered":"2023-09-10T01:01:56.039075+0000","last_clean":"2023-09-10T01:01:56.039075+0000","last_became_active":"2023-09-10T01:00:44.037533+0000","last_became_peered":"2023-09-10T01:00:44.037533+0000","last_unstale":"2023-09-10T01:01:56.039075+0000","last_undegraded":"2023-09-10T01:01:56.039075+0000","last_fullsized":"2023-09-10T01:01:56.039075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:29:26.768570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039763+0000","last_change":"2023-09-10T01:00:44.033396+0000","last_active":"2023-09-10T01:01:56.039763+0000","last_peered":"2023-09-10T01:01:56.039763+0000","last_clean":"2023-09-10T01:01:56.039763+0000","last_became_active":"2023-09-10T01:00:44.033056+0000","last_became_peered":"2023-09-10T01:00:44.033056+0000","last_unstale":"2023-09-10T01:01:56.039763+0000","last_undegraded":"2023-09-10T01:01:56.039763+0000","last_fullsized":"2023-09-10T01:01:56.039763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:00:03.590555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036777+0000","last_change":"2023-09-10T01:00:44.027729+0000","last_active":"2023-09-10T01:01:56.036777+0000","last_peered":"2023-09-10T01:01:56.036777+0000","last_clean":"2023-09-10T01:01:56.036777+0000","last_became_active":"2023-09-10T01:00:44.027447+0000","last_became_peered":"2023-09-10T01:00:44.027447+0000","last_unstale":"2023-09-10T01:01:56.036777+0000","last_undegraded":"2023-09-10T01:01:56.036777+0000","last_fullsized":"2023-09-10T01:01:56.036777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:06:09.391443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127910+0000","last_change":"2023-09-10T01:00:44.026042+0000","last_active":"2023-09-10T01:01:56.127910+0000","last_peered":"2023-09-10T01:01:56.127910+0000","last_clean":"2023-09-10T01:01:56.127910+0000","last_became_active":"2023-09-10T01:00:44.025309+0000","last_became_peered":"2023-09-10T01:00:44.025309+0000","last_unstale":"2023-09-10T01:01:56.127910+0000","last_undegraded":"2023-09-10T01:01:56.127910+0000","last_fullsized":"2023-09-10T01:01:56.127910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:51:29.038717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033488+0000","last_change":"2023-09-10T01:00:44.036761+0000","last_active":"2023-09-10T01:01:56.033488+0000","last_peered":"2023-09-10T01:01:56.033488+0000","last_clean":"2023-09-10T01:01:56.033488+0000","last_became_active":"2023-09-10T01:00:44.036333+0000","last_became_peered":"2023-09-10T01:00:44.036333+0000","last_unstale":"2023-09-10T01:01:56.033488+0000","last_undegraded":"2023-09-10T01:01:56.033488+0000","last_fullsized":"2023-09-10T01:01:56.033488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:37:09.042178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033082+0000","last_change":"2023-09-10T01:00:44.027304+0000","last_active":"2023-09-10T01:01:56.033082+0000","last_peered":"2023-09-10T01:01:56.033082+0000","last_clean":"2023-09-10T01:01:56.033082+0000","last_became_active":"2023-09-10T01:00:44.026994+0000","last_became_peered":"2023-09-10T01:00:44.026994+0000","last_unstale":"2023-09-10T01:01:56.033082+0000","last_undegraded":"2023-09-10T01:01:56.033082+0000","last_fullsized":"2023-09-10T01:01:56.033082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:11:04.824687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032471+0000","last_change":"2023-09-10T01:00:48.056098+0000","last_active":"2023-09-10T01:01:56.032471+0000","last_peered":"2023-09-10T01:01:56.032471+0000","last_clean":"2023-09-10T01:01:56.032471+0000","last_became_active":"2023-09-10T01:00:48.055694+0000","last_became_peered":"2023-09-10T01:00:48.055694+0000","last_unstale":"2023-09-10T01:01:56.032471+0000","last_undegraded":"2023-09-10T01:01:56.032471+0000","last_fullsized":"2023-09-10T01:01:56.032471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:30:48.711613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033599+0000","last_change":"2023-09-10T01:00:56.971435+0000","last_active":"2023-09-10T01:01:56.033599+0000","last_peered":"2023-09-10T01:01:56.033599+0000","last_clean":"2023-09-10T01:01:56.033599+0000","last_became_active":"2023-09-10T01:00:56.965158+0000","last_became_peered":"2023-09-10T01:00:56.965158+0000","last_unstale":"2023-09-10T01:01:56.033599+0000","last_undegraded":"2023-09-10T01:01:56.033599+0000","last_fullsized":"2023-09-10T01:01:56.033599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:09:20.910597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036174+0000","last_change":"2023-09-10T01:00:44.023469+0000","last_active":"2023-09-10T01:01:56.036174+0000","last_peered":"2023-09-10T01:01:56.036174+0000","last_clean":"2023-09-10T01:01:56.036174+0000","last_became_active":"2023-09-10T01:00:44.023232+0000","last_became_peered":"2023-09-10T01:00:44.023232+0000","last_unstale":"2023-09-10T01:01:56.036174+0000","last_undegraded":"2023-09-10T01:01:56.036174+0000","last_fullsized":"2023-09-10T01:01:56.036174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:33:22.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037995+0000","last_change":"2023-09-10T01:00:48.063565+0000","last_active":"2023-09-10T01:01:56.037995+0000","last_peered":"2023-09-10T01:01:56.037995+0000","last_clean":"2023-09-10T01:01:56.037995+0000","last_became_active":"2023-09-10T01:00:48.063200+0000","last_became_peered":"2023-09-10T01:00:48.063200+0000","last_unstale":"2023-09-10T01:01:56.037995+0000","last_undegraded":"2023-09-10T01:01:56.037995+0000","last_fullsized":"2023-09-10T01:01:56.037995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:50:16.624296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036729+0000","last_change":"2023-09-10T01:00:43.010566+0000","last_active":"2023-09-10T01:01:56.036729+0000","last_peered":"2023-09-10T01:01:56.036729+0000","last_clean":"2023-09-10T01:01:56.036729+0000","last_became_active":"2023-09-10T01:00:43.010277+0000","last_became_peered":"2023-09-10T01:00:43.010277+0000","last_unstale":"2023-09-10T01:01:56.036729+0000","last_undegraded":"2023-09-10T01:01:56.036729+0000","last_fullsized":"2023-09-10T01:01:56.036729+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:37.808085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039865+0000","last_change":"2023-09-10T01:00:44.026219+0000","last_active":"2023-09-10T01:01:56.039865+0000","last_peered":"2023-09-10T01:01:56.039865+0000","last_clean":"2023-09-10T01:01:56.039865+0000","last_became_active":"2023-09-10T01:00:44.025906+0000","last_became_peered":"2023-09-10T01:00:44.025906+0000","last_unstale":"2023-09-10T01:01:56.039865+0000","last_undegraded":"2023-09-10T01:01:56.039865+0000","last_fullsized":"2023-09-10T01:01:56.039865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:21:48.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037347+0000","last_change":"2023-09-10T01:00:48.069671+0000","last_active":"2023-09-10T01:01:56.037347+0000","last_peered":"2023-09-10T01:01:56.037347+0000","last_clean":"2023-09-10T01:01:56.037347+0000","last_became_active":"2023-09-10T01:00:48.069382+0000","last_became_peered":"2023-09-10T01:00:48.069382+0000","last_unstale":"2023-09-10T01:01:56.037347+0000","last_undegraded":"2023-09-10T01:01:56.037347+0000","last_fullsized":"2023-09-10T01:01:56.037347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:06:20.526398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036543+0000","last_change":"2023-09-10T01:00:43.005098+0000","last_active":"2023-09-10T01:01:56.036543+0000","last_peered":"2023-09-10T01:01:56.036543+0000","last_clean":"2023-09-10T01:01:56.036543+0000","last_became_active":"2023-09-10T01:00:43.004556+0000","last_became_peered":"2023-09-10T01:00:43.004556+0000","last_unstale":"2023-09-10T01:01:56.036543+0000","last_undegraded":"2023-09-10T01:01:56.036543+0000","last_fullsized":"2023-09-10T01:01:56.036543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:27:43.538936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037176+0000","last_change":"2023-09-10T01:00:44.034451+0000","last_active":"2023-09-10T01:01:56.037176+0000","last_peered":"2023-09-10T01:01:56.037176+0000","last_clean":"2023-09-10T01:01:56.037176+0000","last_became_active":"2023-09-10T01:00:44.034131+0000","last_became_peered":"2023-09-10T01:00:44.034131+0000","last_unstale":"2023-09-10T01:01:56.037176+0000","last_undegraded":"2023-09-10T01:01:56.037176+0000","last_fullsized":"2023-09-10T01:01:56.037176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:50.643215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036946+0000","last_change":"2023-09-10T01:00:48.060114+0000","last_active":"2023-09-10T01:01:56.036946+0000","last_peered":"2023-09-10T01:01:56.036946+0000","last_clean":"2023-09-10T01:01:56.036946+0000","last_became_active":"2023-09-10T01:00:48.059531+0000","last_became_peered":"2023-09-10T01:00:48.059531+0000","last_unstale":"2023-09-10T01:01:56.036946+0000","last_undegraded":"2023-09-10T01:01:56.036946+0000","last_fullsized":"2023-09-10T01:01:56.036946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:28:01.462247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035950+0000","last_change":"2023-09-10T01:00:43.007196+0000","last_active":"2023-09-10T01:01:56.035950+0000","last_peered":"2023-09-10T01:01:56.035950+0000","last_clean":"2023-09-10T01:01:56.035950+0000","last_became_active":"2023-09-10T01:00:43.006795+0000","last_became_peered":"2023-09-10T01:00:43.006795+0000","last_unstale":"2023-09-10T01:01:56.035950+0000","last_undegraded":"2023-09-10T01:01:56.035950+0000","last_fullsized":"2023-09-10T01:01:56.035950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:26:13.146244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037180+0000","last_change":"2023-09-10T01:00:44.027809+0000","last_active":"2023-09-10T01:01:56.037180+0000","last_peered":"2023-09-10T01:01:56.037180+0000","last_clean":"2023-09-10T01:01:56.037180+0000","last_became_active":"2023-09-10T01:00:44.027499+0000","last_became_peered":"2023-09-10T01:00:44.027499+0000","last_unstale":"2023-09-10T01:01:56.037180+0000","last_undegraded":"2023-09-10T01:01:56.037180+0000","last_fullsized":"2023-09-10T01:01:56.037180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:17:07.138807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032849+0000","last_change":"2023-09-10T01:00:48.057645+0000","last_active":"2023-09-10T01:01:56.032849+0000","last_peered":"2023-09-10T01:01:56.032849+0000","last_clean":"2023-09-10T01:01:56.032849+0000","last_became_active":"2023-09-10T01:00:48.057312+0000","last_became_peered":"2023-09-10T01:00:48.057312+0000","last_unstale":"2023-09-10T01:01:56.032849+0000","last_undegraded":"2023-09-10T01:01:56.032849+0000","last_fullsized":"2023-09-10T01:01:56.032849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:28.904572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:01.029848+0000","last_change":"2023-09-10T01:00:56.965677+0000","last_active":"2023-09-10T01:02:01.029848+0000","last_peered":"2023-09-10T01:02:01.029848+0000","last_clean":"2023-09-10T01:02:01.029848+0000","last_became_active":"2023-09-10T01:00:56.965248+0000","last_became_peered":"2023-09-10T01:00:56.965248+0000","last_unstale":"2023-09-10T01:02:01.029848+0000","last_undegraded":"2023-09-10T01:02:01.029848+0000","last_fullsized":"2023-09-10T01:02:01.029848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:39:44.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035997+0000","last_change":"2023-09-10T01:00:44.026340+0000","last_active":"2023-09-10T01:01:56.035997+0000","last_peered":"2023-09-10T01:01:56.035997+0000","last_clean":"2023-09-10T01:01:56.035997+0000","last_became_active":"2023-09-10T01:00:44.026024+0000","last_became_peered":"2023-09-10T01:00:44.026024+0000","last_unstale":"2023-09-10T01:01:56.035997+0000","last_undegraded":"2023-09-10T01:01:56.035997+0000","last_fullsized":"2023-09-10T01:01:56.035997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:58:18.942980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032668+0000","last_change":"2023-09-10T01:00:48.057876+0000","last_active":"2023-09-10T01:01:56.032668+0000","last_peered":"2023-09-10T01:01:56.032668+0000","last_clean":"2023-09-10T01:01:56.032668+0000","last_became_active":"2023-09-10T01:00:48.056686+0000","last_became_peered":"2023-09-10T01:00:48.056686+0000","last_unstale":"2023-09-10T01:01:56.032668+0000","last_undegraded":"2023-09-10T01:01:56.032668+0000","last_fullsized":"2023-09-10T01:01:56.032668+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:10:15.995463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127691+0000","last_change":"2023-09-10T01:00:43.007811+0000","last_active":"2023-09-10T01:01:56.127691+0000","last_peered":"2023-09-10T01:01:56.127691+0000","last_clean":"2023-09-10T01:01:56.127691+0000","last_became_active":"2023-09-10T01:00:43.007205+0000","last_became_peered":"2023-09-10T01:00:43.007205+0000","last_unstale":"2023-09-10T01:01:56.127691+0000","last_undegraded":"2023-09-10T01:01:56.127691+0000","last_fullsized":"2023-09-10T01:01:56.127691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:27:38.114614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037376+0000","last_change":"2023-09-10T01:00:44.027954+0000","last_active":"2023-09-10T01:01:56.037376+0000","last_peered":"2023-09-10T01:01:56.037376+0000","last_clean":"2023-09-10T01:01:56.037376+0000","last_became_active":"2023-09-10T01:00:44.027643+0000","last_became_peered":"2023-09-10T01:00:44.027643+0000","last_unstale":"2023-09-10T01:01:56.037376+0000","last_undegraded":"2023-09-10T01:01:56.037376+0000","last_fullsized":"2023-09-10T01:01:56.037376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:21.253013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127896+0000","last_change":"2023-09-10T01:00:48.062519+0000","last_active":"2023-09-10T01:01:56.127896+0000","last_peered":"2023-09-10T01:01:56.127896+0000","last_clean":"2023-09-10T01:01:56.127896+0000","last_became_active":"2023-09-10T01:00:48.061985+0000","last_became_peered":"2023-09-10T01:00:48.061985+0000","last_unstale":"2023-09-10T01:01:56.127896+0000","last_undegraded":"2023-09-10T01:01:56.127896+0000","last_fullsized":"2023-09-10T01:01:56.127896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:47:36.002325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030435+0000","last_change":"2023-09-10T01:00:42.999601+0000","last_active":"2023-09-10T01:01:56.030435+0000","last_peered":"2023-09-10T01:01:56.030435+0000","last_clean":"2023-09-10T01:01:56.030435+0000","last_became_active":"2023-09-10T01:00:42.999243+0000","last_became_peered":"2023-09-10T01:00:42.999243+0000","last_unstale":"2023-09-10T01:01:56.030435+0000","last_undegraded":"2023-09-10T01:01:56.030435+0000","last_fullsized":"2023-09-10T01:01:56.030435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:49.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035937+0000","last_change":"2023-09-10T01:00:44.027136+0000","last_active":"2023-09-10T01:01:56.035937+0000","last_peered":"2023-09-10T01:01:56.035937+0000","last_clean":"2023-09-10T01:01:56.035937+0000","last_became_active":"2023-09-10T01:00:44.026784+0000","last_became_peered":"2023-09-10T01:00:44.026784+0000","last_unstale":"2023-09-10T01:01:56.035937+0000","last_undegraded":"2023-09-10T01:01:56.035937+0000","last_fullsized":"2023-09-10T01:01:56.035937+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:51:51.271658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039808+0000","last_change":"2023-09-10T01:00:48.066906+0000","last_active":"2023-09-10T01:01:56.039808+0000","last_peered":"2023-09-10T01:01:56.039808+0000","last_clean":"2023-09-10T01:01:56.039808+0000","last_became_active":"2023-09-10T01:00:48.066474+0000","last_became_peered":"2023-09-10T01:00:48.066474+0000","last_unstale":"2023-09-10T01:01:56.039808+0000","last_undegraded":"2023-09-10T01:01:56.039808+0000","last_fullsized":"2023-09-10T01:01:56.039808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:48:26.827857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036365+0000","last_change":"2023-09-10T01:00:43.005922+0000","last_active":"2023-09-10T01:01:56.036365+0000","last_peered":"2023-09-10T01:01:56.036365+0000","last_clean":"2023-09-10T01:01:56.036365+0000","last_became_active":"2023-09-10T01:00:43.005517+0000","last_became_peered":"2023-09-10T01:00:43.005517+0000","last_unstale":"2023-09-10T01:01:56.036365+0000","last_undegraded":"2023-09-10T01:01:56.036365+0000","last_fullsized":"2023-09-10T01:01:56.036365+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:32:12.619034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031897+0000","last_change":"2023-09-10T01:00:44.036766+0000","last_active":"2023-09-10T01:01:56.031897+0000","last_peered":"2023-09-10T01:01:56.031897+0000","last_clean":"2023-09-10T01:01:56.031897+0000","last_became_active":"2023-09-10T01:00:44.036316+0000","last_became_peered":"2023-09-10T01:00:44.036316+0000","last_unstale":"2023-09-10T01:01:56.031897+0000","last_undegraded":"2023-09-10T01:01:56.031897+0000","last_fullsized":"2023-09-10T01:01:56.031897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:17:51.875838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036026+0000","last_change":"2023-09-10T01:00:57.051716+0000","last_active":"2023-09-10T01:01:56.036026+0000","last_peered":"2023-09-10T01:01:56.036026+0000","last_clean":"2023-09-10T01:01:56.036026+0000","last_became_active":"2023-09-10T01:00:57.051334+0000","last_became_peered":"2023-09-10T01:00:57.051334+0000","last_unstale":"2023-09-10T01:01:56.036026+0000","last_undegraded":"2023-09-10T01:01:56.036026+0000","last_fullsized":"2023-09-10T01:01:56.036026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:30:52.598639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:01.029158+0000","last_change":"2023-09-10T01:00:43.008695+0000","last_active":"2023-09-10T01:02:01.029158+0000","last_peered":"2023-09-10T01:02:01.029158+0000","last_clean":"2023-09-10T01:02:01.029158+0000","last_became_active":"2023-09-10T01:00:43.008278+0000","last_became_peered":"2023-09-10T01:00:43.008278+0000","last_unstale":"2023-09-10T01:02:01.029158+0000","last_undegraded":"2023-09-10T01:02:01.029158+0000","last_fullsized":"2023-09-10T01:02:01.029158+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:46:38.370582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036171+0000","last_change":"2023-09-10T01:00:44.030540+0000","last_active":"2023-09-10T01:01:56.036171+0000","last_peered":"2023-09-10T01:01:56.036171+0000","last_clean":"2023-09-10T01:01:56.036171+0000","last_became_active":"2023-09-10T01:00:44.030212+0000","last_became_peered":"2023-09-10T01:00:44.030212+0000","last_unstale":"2023-09-10T01:01:56.036171+0000","last_undegraded":"2023-09-10T01:01:56.036171+0000","last_fullsized":"2023-09-10T01:01:56.036171+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:57:06.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036424+0000","last_change":"2023-09-10T01:00:57.052197+0000","last_active":"2023-09-10T01:01:56.036424+0000","last_peered":"2023-09-10T01:01:56.036424+0000","last_clean":"2023-09-10T01:01:56.036424+0000","last_became_active":"2023-09-10T01:00:57.051782+0000","last_became_peered":"2023-09-10T01:00:57.051782+0000","last_unstale":"2023-09-10T01:01:56.036424+0000","last_undegraded":"2023-09-10T01:01:56.036424+0000","last_fullsized":"2023-09-10T01:01:56.036424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:36:49.680123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042966+0000","last_change":"2023-09-10T01:00:43.007242+0000","last_active":"2023-09-10T01:01:56.042966+0000","last_peered":"2023-09-10T01:01:56.042966+0000","last_clean":"2023-09-10T01:01:56.042966+0000","last_became_active":"2023-09-10T01:00:43.006912+0000","last_became_peered":"2023-09-10T01:00:43.006912+0000","last_unstale":"2023-09-10T01:01:56.042966+0000","last_undegraded":"2023-09-10T01:01:56.042966+0000","last_fullsized":"2023-09-10T01:01:56.042966+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:48:22.783656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.026024+0000","last_change":"2023-09-10T01:01:58.026024+0000","last_active":"2023-09-10T01:01:58.026024+0000","last_peered":"2023-09-10T01:01:58.026024+0000","last_clean":"2023-09-10T01:01:58.026024+0000","last_became_active":"2023-09-10T01:01:58.025647+0000","last_became_peered":"2023-09-10T01:01:58.025647+0000","last_unstale":"2023-09-10T01:01:58.026024+0000","last_undegraded":"2023-09-10T01:01:58.026024+0000","last_fullsized":"2023-09-10T01:01:58.026024+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:58:11.916233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037579+0000","last_change":"2023-09-10T01:00:48.062520+0000","last_active":"2023-09-10T01:01:56.037579+0000","last_peered":"2023-09-10T01:01:56.037579+0000","last_clean":"2023-09-10T01:01:56.037579+0000","last_became_active":"2023-09-10T01:00:48.061870+0000","last_became_peered":"2023-09-10T01:00:48.061870+0000","last_unstale":"2023-09-10T01:01:56.037579+0000","last_undegraded":"2023-09-10T01:01:56.037579+0000","last_fullsized":"2023-09-10T01:01:56.037579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:24:38.798775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037030+0000","last_change":"2023-09-10T01:00:43.006513+0000","last_active":"2023-09-10T01:01:56.037030+0000","last_peered":"2023-09-10T01:01:56.037030+0000","last_clean":"2023-09-10T01:01:56.037030+0000","last_became_active":"2023-09-10T01:00:43.006166+0000","last_became_peered":"2023-09-10T01:00:43.006166+0000","last_unstale":"2023-09-10T01:01:56.037030+0000","last_undegraded":"2023-09-10T01:01:56.037030+0000","last_fullsized":"2023-09-10T01:01:56.037030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:52:32.167513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037335+0000","last_change":"2023-09-10T01:00:44.031264+0000","last_active":"2023-09-10T01:01:56.037335+0000","last_peered":"2023-09-10T01:01:56.037335+0000","last_clean":"2023-09-10T01:01:56.037335+0000","last_became_active":"2023-09-10T01:00:44.030993+0000","last_became_peered":"2023-09-10T01:00:44.030993+0000","last_unstale":"2023-09-10T01:01:56.037335+0000","last_undegraded":"2023-09-10T01:01:56.037335+0000","last_fullsized":"2023-09-10T01:01:56.037335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:19:27.658594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032496+0000","last_change":"2023-09-10T01:00:48.056886+0000","last_active":"2023-09-10T01:01:56.032496+0000","last_peered":"2023-09-10T01:01:56.032496+0000","last_clean":"2023-09-10T01:01:56.032496+0000","last_became_active":"2023-09-10T01:00:48.056411+0000","last_became_peered":"2023-09-10T01:00:48.056411+0000","last_unstale":"2023-09-10T01:01:56.032496+0000","last_undegraded":"2023-09-10T01:01:56.032496+0000","last_fullsized":"2023-09-10T01:01:56.032496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:56:42.062702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'30","reported_seq":164,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:02.030460+0000","last_change":"2023-09-10T01:00:43.009113+0000","last_active":"2023-09-10T01:02:02.030460+0000","last_peered":"2023-09-10T01:02:02.030460+0000","last_clean":"2023-09-10T01:02:02.030460+0000","last_became_active":"2023-09-10T01:00:43.008757+0000","last_became_peered":"2023-09-10T01:00:43.008757+0000","last_unstale":"2023-09-10T01:02:02.030460+0000","last_undegraded":"2023-09-10T01:02:02.030460+0000","last_fullsized":"2023-09-10T01:02:02.030460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:29:24.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59889,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":113,"num_read_kb":115,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036282+0000","last_change":"2023-09-10T01:00:44.027127+0000","last_active":"2023-09-10T01:01:56.036282+0000","last_peered":"2023-09-10T01:01:56.036282+0000","last_clean":"2023-09-10T01:01:56.036282+0000","last_became_active":"2023-09-10T01:00:44.026852+0000","last_became_peered":"2023-09-10T01:00:44.026852+0000","last_unstale":"2023-09-10T01:01:56.036282+0000","last_undegraded":"2023-09-10T01:01:56.036282+0000","last_fullsized":"2023-09-10T01:01:56.036282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:14:20.805300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037605+0000","last_change":"2023-09-10T01:00:48.069337+0000","last_active":"2023-09-10T01:01:56.037605+0000","last_peered":"2023-09-10T01:01:56.037605+0000","last_clean":"2023-09-10T01:01:56.037605+0000","last_became_active":"2023-09-10T01:00:48.068893+0000","last_became_peered":"2023-09-10T01:00:48.068893+0000","last_unstale":"2023-09-10T01:01:56.037605+0000","last_undegraded":"2023-09-10T01:01:56.037605+0000","last_fullsized":"2023-09-10T01:01:56.037605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:24:40.074551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033160+0000","last_change":"2023-09-10T01:00:43.010128+0000","last_active":"2023-09-10T01:01:56.033160+0000","last_peered":"2023-09-10T01:01:56.033160+0000","last_clean":"2023-09-10T01:01:56.033160+0000","last_became_active":"2023-09-10T01:00:43.009808+0000","last_became_peered":"2023-09-10T01:00:43.009808+0000","last_unstale":"2023-09-10T01:01:56.033160+0000","last_undegraded":"2023-09-10T01:01:56.033160+0000","last_fullsized":"2023-09-10T01:01:56.033160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:27.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030524+0000","last_change":"2023-09-10T01:00:44.038665+0000","last_active":"2023-09-10T01:01:56.030524+0000","last_peered":"2023-09-10T01:01:56.030524+0000","last_clean":"2023-09-10T01:01:56.030524+0000","last_became_active":"2023-09-10T01:00:44.037944+0000","last_became_peered":"2023-09-10T01:00:44.037944+0000","last_unstale":"2023-09-10T01:01:56.030524+0000","last_undegraded":"2023-09-10T01:01:56.030524+0000","last_fullsized":"2023-09-10T01:01:56.030524+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:11:34.018249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127649+0000","last_change":"2023-09-10T01:00:56.965594+0000","last_active":"2023-09-10T01:01:56.127649+0000","last_peered":"2023-09-10T01:01:56.127649+0000","last_clean":"2023-09-10T01:01:56.127649+0000","last_became_active":"2023-09-10T01:00:56.965006+0000","last_became_peered":"2023-09-10T01:00:56.965006+0000","last_unstale":"2023-09-10T01:01:56.127649+0000","last_undegraded":"2023-09-10T01:01:56.127649+0000","last_fullsized":"2023-09-10T01:01:56.127649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:13:16.946697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036734+0000","last_change":"2023-09-10T01:00:43.004855+0000","last_active":"2023-09-10T01:01:56.036734+0000","last_peered":"2023-09-10T01:01:56.036734+0000","last_clean":"2023-09-10T01:01:56.036734+0000","last_became_active":"2023-09-10T01:00:43.004464+0000","last_became_peered":"2023-09-10T01:00:43.004464+0000","last_unstale":"2023-09-10T01:01:56.036734+0000","last_undegraded":"2023-09-10T01:01:56.036734+0000","last_fullsized":"2023-09-10T01:01:56.036734+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:29.517290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042188+0000","last_change":"2023-09-10T01:00:44.037273+0000","last_active":"2023-09-10T01:01:56.042188+0000","last_peered":"2023-09-10T01:01:56.042188+0000","last_clean":"2023-09-10T01:01:56.042188+0000","last_became_active":"2023-09-10T01:00:44.036965+0000","last_became_peered":"2023-09-10T01:00:44.036965+0000","last_unstale":"2023-09-10T01:01:56.042188+0000","last_undegraded":"2023-09-10T01:01:56.042188+0000","last_fullsized":"2023-09-10T01:01:56.042188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:47:20.149002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033203+0000","last_change":"2023-09-10T01:00:48.058574+0000","last_active":"2023-09-10T01:01:56.033203+0000","last_peered":"2023-09-10T01:01:56.033203+0000","last_clean":"2023-09-10T01:01:56.033203+0000","last_became_active":"2023-09-10T01:00:48.058190+0000","last_became_peered":"2023-09-10T01:00:48.058190+0000","last_unstale":"2023-09-10T01:01:56.033203+0000","last_undegraded":"2023-09-10T01:01:56.033203+0000","last_fullsized":"2023-09-10T01:01:56.033203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:46:42.177996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:00.045177+0000","last_change":"2023-09-10T01:00:43.003816+0000","last_active":"2023-09-10T01:02:00.045177+0000","last_peered":"2023-09-10T01:02:00.045177+0000","last_clean":"2023-09-10T01:02:00.045177+0000","last_became_active":"2023-09-10T01:00:43.003404+0000","last_became_peered":"2023-09-10T01:00:43.003404+0000","last_unstale":"2023-09-10T01:02:00.045177+0000","last_undegraded":"2023-09-10T01:02:00.045177+0000","last_fullsized":"2023-09-10T01:02:00.045177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:55.208302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042150+0000","last_change":"2023-09-10T01:00:44.034607+0000","last_active":"2023-09-10T01:01:56.042150+0000","last_peered":"2023-09-10T01:01:56.042150+0000","last_clean":"2023-09-10T01:01:56.042150+0000","last_became_active":"2023-09-10T01:00:44.034244+0000","last_became_peered":"2023-09-10T01:00:44.034244+0000","last_unstale":"2023-09-10T01:01:56.042150+0000","last_undegraded":"2023-09-10T01:01:56.042150+0000","last_fullsized":"2023-09-10T01:01:56.042150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:06:13.214160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031918+0000","last_change":"2023-09-10T01:00:48.058789+0000","last_active":"2023-09-10T01:01:56.031918+0000","last_peered":"2023-09-10T01:01:56.031918+0000","last_clean":"2023-09-10T01:01:56.031918+0000","last_became_active":"2023-09-10T01:00:48.058220+0000","last_became_peered":"2023-09-10T01:00:48.058220+0000","last_unstale":"2023-09-10T01:01:56.031918+0000","last_undegraded":"2023-09-10T01:01:56.031918+0000","last_fullsized":"2023-09-10T01:01:56.031918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:01:12.291739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036527+0000","last_change":"2023-09-10T01:00:43.006109+0000","last_active":"2023-09-10T01:01:56.036527+0000","last_peered":"2023-09-10T01:01:56.036527+0000","last_clean":"2023-09-10T01:01:56.036527+0000","last_became_active":"2023-09-10T01:00:43.005753+0000","last_became_peered":"2023-09-10T01:00:43.005753+0000","last_unstale":"2023-09-10T01:01:56.036527+0000","last_undegraded":"2023-09-10T01:01:56.036527+0000","last_fullsized":"2023-09-10T01:01:56.036527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:45:47.083991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.139294+0000","last_change":"2023-09-10T01:01:57.139294+0000","last_active":"2023-09-10T01:01:57.139294+0000","last_peered":"2023-09-10T01:01:57.139294+0000","last_clean":"2023-09-10T01:01:57.139294+0000","last_became_active":"2023-09-10T01:01:57.138930+0000","last_became_peered":"2023-09-10T01:01:57.138930+0000","last_unstale":"2023-09-10T01:01:57.139294+0000","last_undegraded":"2023-09-10T01:01:57.139294+0000","last_fullsized":"2023-09-10T01:01:57.139294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:21.143499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036928+0000","last_change":"2023-09-10T01:00:48.069318+0000","last_active":"2023-09-10T01:01:56.036928+0000","last_peered":"2023-09-10T01:01:56.036928+0000","last_clean":"2023-09-10T01:01:56.036928+0000","last_became_active":"2023-09-10T01:00:48.068876+0000","last_became_peered":"2023-09-10T01:00:48.068876+0000","last_unstale":"2023-09-10T01:01:56.036928+0000","last_undegraded":"2023-09-10T01:01:56.036928+0000","last_fullsized":"2023-09-10T01:01:56.036928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:06:22.366530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036038+0000","last_change":"2023-09-10T01:00:43.004164+0000","last_active":"2023-09-10T01:01:56.036038+0000","last_peered":"2023-09-10T01:01:56.036038+0000","last_clean":"2023-09-10T01:01:56.036038+0000","last_became_active":"2023-09-10T01:00:43.003781+0000","last_became_peered":"2023-09-10T01:00:43.003781+0000","last_unstale":"2023-09-10T01:01:56.036038+0000","last_undegraded":"2023-09-10T01:01:56.036038+0000","last_fullsized":"2023-09-10T01:01:56.036038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:38:18.559712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127889+0000","last_change":"2023-09-10T01:00:44.028206+0000","last_active":"2023-09-10T01:01:56.127889+0000","last_peered":"2023-09-10T01:01:56.127889+0000","last_clean":"2023-09-10T01:01:56.127889+0000","last_became_active":"2023-09-10T01:00:44.027914+0000","last_became_peered":"2023-09-10T01:00:44.027914+0000","last_unstale":"2023-09-10T01:01:56.127889+0000","last_undegraded":"2023-09-10T01:01:56.127889+0000","last_fullsized":"2023-09-10T01:01:56.127889+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:41:12.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035899+0000","last_change":"2023-09-10T01:00:48.428613+0000","last_active":"2023-09-10T01:01:56.035899+0000","last_peered":"2023-09-10T01:01:56.035899+0000","last_clean":"2023-09-10T01:01:56.035899+0000","last_became_active":"2023-09-10T01:00:48.428059+0000","last_became_peered":"2023-09-10T01:00:48.428059+0000","last_unstale":"2023-09-10T01:01:56.035899+0000","last_undegraded":"2023-09-10T01:01:56.035899+0000","last_fullsized":"2023-09-10T01:01:56.035899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:46:03.683279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032873+0000","last_change":"2023-09-10T01:00:43.006061+0000","last_active":"2023-09-10T01:01:56.032873+0000","last_peered":"2023-09-10T01:01:56.032873+0000","last_clean":"2023-09-10T01:01:56.032873+0000","last_became_active":"2023-09-10T01:00:43.005392+0000","last_became_peered":"2023-09-10T01:00:43.005392+0000","last_unstale":"2023-09-10T01:01:56.032873+0000","last_undegraded":"2023-09-10T01:01:56.032873+0000","last_fullsized":"2023-09-10T01:01:56.032873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:01:45.062338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127818+0000","last_change":"2023-09-10T01:00:44.026684+0000","last_active":"2023-09-10T01:01:56.127818+0000","last_peered":"2023-09-10T01:01:56.127818+0000","last_clean":"2023-09-10T01:01:56.127818+0000","last_became_active":"2023-09-10T01:00:44.026387+0000","last_became_peered":"2023-09-10T01:00:44.026387+0000","last_unstale":"2023-09-10T01:01:56.127818+0000","last_undegraded":"2023-09-10T01:01:56.127818+0000","last_fullsized":"2023-09-10T01:01:56.127818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:23:03.662543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031165+0000","last_change":"2023-09-10T01:00:48.071589+0000","last_active":"2023-09-10T01:01:56.031165+0000","last_peered":"2023-09-10T01:01:56.031165+0000","last_clean":"2023-09-10T01:01:56.031165+0000","last_became_active":"2023-09-10T01:00:48.071118+0000","last_became_peered":"2023-09-10T01:00:48.071118+0000","last_unstale":"2023-09-10T01:01:56.031165+0000","last_undegraded":"2023-09-10T01:01:56.031165+0000","last_fullsized":"2023-09-10T01:01:56.031165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:27.194868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042525+0000","last_change":"2023-09-10T01:00:43.009738+0000","last_active":"2023-09-10T01:01:56.042525+0000","last_peered":"2023-09-10T01:01:56.042525+0000","last_clean":"2023-09-10T01:01:56.042525+0000","last_became_active":"2023-09-10T01:00:43.009376+0000","last_became_peered":"2023-09-10T01:00:43.009376+0000","last_unstale":"2023-09-10T01:01:56.042525+0000","last_undegraded":"2023-09-10T01:01:56.042525+0000","last_fullsized":"2023-09-10T01:01:56.042525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:08.761672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127701+0000","last_change":"2023-09-10T01:00:44.030175+0000","last_active":"2023-09-10T01:01:56.127701+0000","last_peered":"2023-09-10T01:01:56.127701+0000","last_clean":"2023-09-10T01:01:56.127701+0000","last_became_active":"2023-09-10T01:00:44.029748+0000","last_became_peered":"2023-09-10T01:00:44.029748+0000","last_unstale":"2023-09-10T01:01:56.127701+0000","last_undegraded":"2023-09-10T01:01:56.127701+0000","last_fullsized":"2023-09-10T01:01:56.127701+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:51:15.149610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032036+0000","last_change":"2023-09-10T01:00:48.428008+0000","last_active":"2023-09-10T01:01:56.032036+0000","last_peered":"2023-09-10T01:01:56.032036+0000","last_clean":"2023-09-10T01:01:56.032036+0000","last_became_active":"2023-09-10T01:00:48.427498+0000","last_became_peered":"2023-09-10T01:00:48.427498+0000","last_unstale":"2023-09-10T01:01:56.032036+0000","last_undegraded":"2023-09-10T01:01:56.032036+0000","last_fullsized":"2023-09-10T01:01:56.032036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:16:48.938006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.109029+0000","last_change":"2023-09-10T01:01:57.107918+0000","last_active":"2023-09-10T01:01:57.109029+0000","last_peered":"2023-09-10T01:01:57.109029+0000","last_clean":"2023-09-10T01:01:57.109029+0000","last_became_active":"2023-09-10T01:01:57.107413+0000","last_became_peered":"2023-09-10T01:01:57.107413+0000","last_unstale":"2023-09-10T01:01:57.109029+0000","last_undegraded":"2023-09-10T01:01:57.109029+0000","last_fullsized":"2023-09-10T01:01:57.109029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:16:33.836871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:00.044207+0000","last_change":"2023-09-10T01:00:43.001047+0000","last_active":"2023-09-10T01:02:00.044207+0000","last_peered":"2023-09-10T01:02:00.044207+0000","last_clean":"2023-09-10T01:02:00.044207+0000","last_became_active":"2023-09-10T01:00:43.000521+0000","last_became_peered":"2023-09-10T01:00:43.000521+0000","last_unstale":"2023-09-10T01:02:00.044207+0000","last_undegraded":"2023-09-10T01:02:00.044207+0000","last_fullsized":"2023-09-10T01:02:00.044207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:15:34.509964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042829+0000","last_change":"2023-09-10T01:00:48.062148+0000","last_active":"2023-09-10T01:01:56.042829+0000","last_peered":"2023-09-10T01:01:56.042829+0000","last_clean":"2023-09-10T01:01:56.042829+0000","last_became_active":"2023-09-10T01:00:48.061746+0000","last_became_peered":"2023-09-10T01:00:48.061746+0000","last_unstale":"2023-09-10T01:01:56.042829+0000","last_undegraded":"2023-09-10T01:01:56.042829+0000","last_fullsized":"2023-09-10T01:01:56.042829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:40.087012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036350+0000","last_change":"2023-09-10T01:00:44.030500+0000","last_active":"2023-09-10T01:01:56.036350+0000","last_peered":"2023-09-10T01:01:56.036350+0000","last_clean":"2023-09-10T01:01:56.036350+0000","last_became_active":"2023-09-10T01:00:44.030209+0000","last_became_peered":"2023-09-10T01:00:44.030209+0000","last_unstale":"2023-09-10T01:01:56.036350+0000","last_undegraded":"2023-09-10T01:01:56.036350+0000","last_fullsized":"2023-09-10T01:01:56.036350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:53:30.421493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037460+0000","last_change":"2023-09-10T01:00:43.001102+0000","last_active":"2023-09-10T01:01:56.037460+0000","last_peered":"2023-09-10T01:01:56.037460+0000","last_clean":"2023-09-10T01:01:56.037460+0000","last_became_active":"2023-09-10T01:00:43.000715+0000","last_became_peered":"2023-09-10T01:00:43.000715+0000","last_unstale":"2023-09-10T01:01:56.037460+0000","last_undegraded":"2023-09-10T01:01:56.037460+0000","last_fullsized":"2023-09-10T01:01:56.037460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:43:37.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":326,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042414+0000","last_change":"2023-09-10T00:59:40.261244+0000","last_active":"2023-09-10T01:01:56.042414+0000","last_peered":"2023-09-10T01:01:56.042414+0000","last_clean":"2023-09-10T01:01:56.042414+0000","last_became_active":"2023-09-10T00:59:40.260673+0000","last_became_peered":"2023-09-10T00:59:40.260673+0000","last_unstale":"2023-09-10T01:01:56.042414+0000","last_undegraded":"2023-09-10T01:01:56.042414+0000","last_fullsized":"2023-09-10T01:01:56.042414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_clean_scrub_stamp":"2023-09-10T00:56:36.374323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:04:03.614104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037213+0000","last_change":"2023-09-10T01:00:57.052019+0000","last_active":"2023-09-10T01:01:56.037213+0000","last_peered":"2023-09-10T01:01:56.037213+0000","last_clean":"2023-09-10T01:01:56.037213+0000","last_became_active":"2023-09-10T01:00:57.051527+0000","last_became_peered":"2023-09-10T01:00:57.051527+0000","last_unstale":"2023-09-10T01:01:56.037213+0000","last_undegraded":"2023-09-10T01:01:56.037213+0000","last_fullsized":"2023-09-10T01:01:56.037213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:10:43.756761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030326+0000","last_change":"2023-09-10T01:00:44.033214+0000","last_active":"2023-09-10T01:01:56.030326+0000","last_peered":"2023-09-10T01:01:56.030326+0000","last_clean":"2023-09-10T01:01:56.030326+0000","last_became_active":"2023-09-10T01:00:44.032828+0000","last_became_peered":"2023-09-10T01:00:44.032828+0000","last_unstale":"2023-09-10T01:01:56.030326+0000","last_undegraded":"2023-09-10T01:01:56.030326+0000","last_fullsized":"2023-09-10T01:01:56.030326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:01:22.417216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042638+0000","last_change":"2023-09-10T01:00:56.965456+0000","last_active":"2023-09-10T01:01:56.042638+0000","last_peered":"2023-09-10T01:01:56.042638+0000","last_clean":"2023-09-10T01:01:56.042638+0000","last_became_active":"2023-09-10T01:00:56.965021+0000","last_became_peered":"2023-09-10T01:00:56.965021+0000","last_unstale":"2023-09-10T01:01:56.042638+0000","last_undegraded":"2023-09-10T01:01:56.042638+0000","last_fullsized":"2023-09-10T01:01:56.042638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:19.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127643+0000","last_change":"2023-09-10T01:00:48.066811+0000","last_active":"2023-09-10T01:01:56.127643+0000","last_peered":"2023-09-10T01:01:56.127643+0000","last_clean":"2023-09-10T01:01:56.127643+0000","last_became_active":"2023-09-10T01:00:48.066421+0000","last_became_peered":"2023-09-10T01:00:48.066421+0000","last_unstale":"2023-09-10T01:01:56.127643+0000","last_undegraded":"2023-09-10T01:01:56.127643+0000","last_fullsized":"2023-09-10T01:01:56.127643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:48:01.459070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036384+0000","last_change":"2023-09-10T01:00:43.011134+0000","last_active":"2023-09-10T01:01:56.036384+0000","last_peered":"2023-09-10T01:01:56.036384+0000","last_clean":"2023-09-10T01:01:56.036384+0000","last_became_active":"2023-09-10T01:00:43.010871+0000","last_became_peered":"2023-09-10T01:00:43.010871+0000","last_unstale":"2023-09-10T01:01:56.036384+0000","last_undegraded":"2023-09-10T01:01:56.036384+0000","last_fullsized":"2023-09-10T01:01:56.036384+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:56.903745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031298+0000","last_change":"2023-09-10T01:00:44.031104+0000","last_active":"2023-09-10T01:01:56.031298+0000","last_peered":"2023-09-10T01:01:56.031298+0000","last_clean":"2023-09-10T01:01:56.031298+0000","last_became_active":"2023-09-10T01:00:44.030745+0000","last_became_peered":"2023-09-10T01:00:44.030745+0000","last_unstale":"2023-09-10T01:01:56.031298+0000","last_undegraded":"2023-09-10T01:01:56.031298+0000","last_fullsized":"2023-09-10T01:01:56.031298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:53.885886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035909+0000","last_change":"2023-09-10T01:00:48.427735+0000","last_active":"2023-09-10T01:01:56.035909+0000","last_peered":"2023-09-10T01:01:56.035909+0000","last_clean":"2023-09-10T01:01:56.035909+0000","last_became_active":"2023-09-10T01:00:48.427342+0000","last_became_peered":"2023-09-10T01:00:48.427342+0000","last_unstale":"2023-09-10T01:01:56.035909+0000","last_undegraded":"2023-09-10T01:01:56.035909+0000","last_fullsized":"2023-09-10T01:01:56.035909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:01:52.007325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033359+0000","last_change":"2023-09-10T01:00:43.006314+0000","last_active":"2023-09-10T01:01:56.033359+0000","last_peered":"2023-09-10T01:01:56.033359+0000","last_clean":"2023-09-10T01:01:56.033359+0000","last_became_active":"2023-09-10T01:00:43.004835+0000","last_became_peered":"2023-09-10T01:00:43.004835+0000","last_unstale":"2023-09-10T01:01:56.033359+0000","last_undegraded":"2023-09-10T01:01:56.033359+0000","last_fullsized":"2023-09-10T01:01:56.033359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:37:09.359846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036353+0000","last_change":"2023-09-10T01:00:44.029839+0000","last_active":"2023-09-10T01:01:56.036353+0000","last_peered":"2023-09-10T01:01:56.036353+0000","last_clean":"2023-09-10T01:01:56.036353+0000","last_became_active":"2023-09-10T01:00:44.029492+0000","last_became_peered":"2023-09-10T01:00:44.029492+0000","last_unstale":"2023-09-10T01:01:56.036353+0000","last_undegraded":"2023-09-10T01:01:56.036353+0000","last_fullsized":"2023-09-10T01:01:56.036353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:14.357386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032332+0000","last_change":"2023-09-10T01:00:48.054536+0000","last_active":"2023-09-10T01:01:56.032332+0000","last_peered":"2023-09-10T01:01:56.032332+0000","last_clean":"2023-09-10T01:01:56.032332+0000","last_became_active":"2023-09-10T01:00:48.053938+0000","last_became_peered":"2023-09-10T01:00:48.053938+0000","last_unstale":"2023-09-10T01:01:56.032332+0000","last_undegraded":"2023-09-10T01:01:56.032332+0000","last_fullsized":"2023-09-10T01:01:56.032332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:45:12.703592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037909+0000","last_change":"2023-09-10T01:00:43.002313+0000","last_active":"2023-09-10T01:01:56.037909+0000","last_peered":"2023-09-10T01:01:56.037909+0000","last_clean":"2023-09-10T01:01:56.037909+0000","last_became_active":"2023-09-10T01:00:43.001839+0000","last_became_peered":"2023-09-10T01:00:43.001839+0000","last_unstale":"2023-09-10T01:01:56.037909+0000","last_undegraded":"2023-09-10T01:01:56.037909+0000","last_fullsized":"2023-09-10T01:01:56.037909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:51.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.108269+0000","last_change":"2023-09-10T01:01:57.108269+0000","last_active":"2023-09-10T01:01:57.108269+0000","last_peered":"2023-09-10T01:01:57.108269+0000","last_clean":"2023-09-10T01:01:57.108269+0000","last_became_active":"2023-09-10T01:01:57.107879+0000","last_became_peered":"2023-09-10T01:01:57.107879+0000","last_unstale":"2023-09-10T01:01:57.108269+0000","last_undegraded":"2023-09-10T01:01:57.108269+0000","last_fullsized":"2023-09-10T01:01:57.108269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:47:09.135978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037126+0000","last_change":"2023-09-10T01:00:48.428406+0000","last_active":"2023-09-10T01:01:56.037126+0000","last_peered":"2023-09-10T01:01:56.037126+0000","last_clean":"2023-09-10T01:01:56.037126+0000","last_became_active":"2023-09-10T01:00:48.427938+0000","last_became_peered":"2023-09-10T01:00:48.427938+0000","last_unstale":"2023-09-10T01:01:56.037126+0000","last_undegraded":"2023-09-10T01:01:56.037126+0000","last_fullsized":"2023-09-10T01:01:56.037126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:52.316235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036897+0000","last_change":"2023-09-10T01:00:56.963755+0000","last_active":"2023-09-10T01:01:56.036897+0000","last_peered":"2023-09-10T01:01:56.036897+0000","last_clean":"2023-09-10T01:01:56.036897+0000","last_became_active":"2023-09-10T01:00:56.963283+0000","last_became_peered":"2023-09-10T01:00:56.963283+0000","last_unstale":"2023-09-10T01:01:56.036897+0000","last_undegraded":"2023-09-10T01:01:56.036897+0000","last_fullsized":"2023-09-10T01:01:56.036897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:13:32.213393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035575+0000","last_change":"2023-09-10T01:00:44.025284+0000","last_active":"2023-09-10T01:01:56.035575+0000","last_peered":"2023-09-10T01:01:56.035575+0000","last_clean":"2023-09-10T01:01:56.035575+0000","last_became_active":"2023-09-10T01:00:44.024965+0000","last_became_peered":"2023-09-10T01:00:44.024965+0000","last_unstale":"2023-09-10T01:01:56.035575+0000","last_undegraded":"2023-09-10T01:01:56.035575+0000","last_fullsized":"2023-09-10T01:01:56.035575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:19:08.620325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033359+0000","last_change":"2023-09-10T01:00:48.070843+0000","last_active":"2023-09-10T01:01:56.033359+0000","last_peered":"2023-09-10T01:01:56.033359+0000","last_clean":"2023-09-10T01:01:56.033359+0000","last_became_active":"2023-09-10T01:00:48.070350+0000","last_became_peered":"2023-09-10T01:00:48.070350+0000","last_unstale":"2023-09-10T01:01:56.033359+0000","last_undegraded":"2023-09-10T01:01:56.033359+0000","last_fullsized":"2023-09-10T01:01:56.033359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:35:25.196036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036081+0000","last_change":"2023-09-10T01:00:43.005398+0000","last_active":"2023-09-10T01:01:56.036081+0000","last_peered":"2023-09-10T01:01:56.036081+0000","last_clean":"2023-09-10T01:01:56.036081+0000","last_became_active":"2023-09-10T01:00:43.005087+0000","last_became_peered":"2023-09-10T01:00:43.005087+0000","last_unstale":"2023-09-10T01:01:56.036081+0000","last_undegraded":"2023-09-10T01:01:56.036081+0000","last_fullsized":"2023-09-10T01:01:56.036081+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:29:04.042683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127507+0000","last_change":"2023-09-10T01:00:44.024451+0000","last_active":"2023-09-10T01:01:56.127507+0000","last_peered":"2023-09-10T01:01:56.127507+0000","last_clean":"2023-09-10T01:01:56.127507+0000","last_became_active":"2023-09-10T01:00:44.024045+0000","last_became_peered":"2023-09-10T01:00:44.024045+0000","last_unstale":"2023-09-10T01:01:56.127507+0000","last_undegraded":"2023-09-10T01:01:56.127507+0000","last_fullsized":"2023-09-10T01:01:56.127507+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:55:50.750994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037240+0000","last_change":"2023-09-10T01:00:48.062247+0000","last_active":"2023-09-10T01:01:56.037240+0000","last_peered":"2023-09-10T01:01:56.037240+0000","last_clean":"2023-09-10T01:01:56.037240+0000","last_became_active":"2023-09-10T01:00:48.061812+0000","last_became_peered":"2023-09-10T01:00:48.061812+0000","last_unstale":"2023-09-10T01:01:56.037240+0000","last_undegraded":"2023-09-10T01:01:56.037240+0000","last_fullsized":"2023-09-10T01:01:56.037240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:44:13.596534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032713+0000","last_change":"2023-09-10T01:00:43.010845+0000","last_active":"2023-09-10T01:01:56.032713+0000","last_peered":"2023-09-10T01:01:56.032713+0000","last_clean":"2023-09-10T01:01:56.032713+0000","last_became_active":"2023-09-10T01:00:43.010511+0000","last_became_peered":"2023-09-10T01:00:43.010511+0000","last_unstale":"2023-09-10T01:01:56.032713+0000","last_undegraded":"2023-09-10T01:01:56.032713+0000","last_fullsized":"2023-09-10T01:01:56.032713+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:33:07.229023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036175+0000","last_change":"2023-09-10T01:00:44.029025+0000","last_active":"2023-09-10T01:01:56.036175+0000","last_peered":"2023-09-10T01:01:56.036175+0000","last_clean":"2023-09-10T01:01:56.036175+0000","last_became_active":"2023-09-10T01:00:44.028713+0000","last_became_peered":"2023-09-10T01:00:44.028713+0000","last_unstale":"2023-09-10T01:01:56.036175+0000","last_undegraded":"2023-09-10T01:01:56.036175+0000","last_fullsized":"2023-09-10T01:01:56.036175+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:40:22.315742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037817+0000","last_change":"2023-09-10T01:00:48.427294+0000","last_active":"2023-09-10T01:01:56.037817+0000","last_peered":"2023-09-10T01:01:56.037817+0000","last_clean":"2023-09-10T01:01:56.037817+0000","last_became_active":"2023-09-10T01:00:48.426882+0000","last_became_peered":"2023-09-10T01:00:48.426882+0000","last_unstale":"2023-09-10T01:01:56.037817+0000","last_undegraded":"2023-09-10T01:01:56.037817+0000","last_fullsized":"2023-09-10T01:01:56.037817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:54:38.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035921+0000","last_change":"2023-09-10T01:00:43.006175+0000","last_active":"2023-09-10T01:01:56.035921+0000","last_peered":"2023-09-10T01:01:56.035921+0000","last_clean":"2023-09-10T01:01:56.035921+0000","last_became_active":"2023-09-10T01:00:43.005786+0000","last_became_peered":"2023-09-10T01:00:43.005786+0000","last_unstale":"2023-09-10T01:01:56.035921+0000","last_undegraded":"2023-09-10T01:01:56.035921+0000","last_fullsized":"2023-09-10T01:01:56.035921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:14:47.074866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042604+0000","last_change":"2023-09-10T01:00:44.041303+0000","last_active":"2023-09-10T01:01:56.042604+0000","last_peered":"2023-09-10T01:01:56.042604+0000","last_clean":"2023-09-10T01:01:56.042604+0000","last_became_active":"2023-09-10T01:00:44.041009+0000","last_became_peered":"2023-09-10T01:00:44.041009+0000","last_unstale":"2023-09-10T01:01:56.042604+0000","last_undegraded":"2023-09-10T01:01:56.042604+0000","last_fullsized":"2023-09-10T01:01:56.042604+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:53:23.001700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036842+0000","last_change":"2023-09-10T01:00:48.064314+0000","last_active":"2023-09-10T01:01:56.036842+0000","last_peered":"2023-09-10T01:01:56.036842+0000","last_clean":"2023-09-10T01:01:56.036842+0000","last_became_active":"2023-09-10T01:00:48.063813+0000","last_became_peered":"2023-09-10T01:00:48.063813+0000","last_unstale":"2023-09-10T01:01:56.036842+0000","last_undegraded":"2023-09-10T01:01:56.036842+0000","last_fullsized":"2023-09-10T01:01:56.036842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:00:10.002006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033268+0000","last_change":"2023-09-10T01:00:43.007579+0000","last_active":"2023-09-10T01:01:56.033268+0000","last_peered":"2023-09-10T01:01:56.033268+0000","last_clean":"2023-09-10T01:01:56.033268+0000","last_became_active":"2023-09-10T01:00:43.007200+0000","last_became_peered":"2023-09-10T01:00:43.007200+0000","last_unstale":"2023-09-10T01:01:56.033268+0000","last_undegraded":"2023-09-10T01:01:56.033268+0000","last_fullsized":"2023-09-10T01:01:56.033268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:20:08.688531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042996+0000","last_change":"2023-09-10T01:00:44.033511+0000","last_active":"2023-09-10T01:01:56.042996+0000","last_peered":"2023-09-10T01:01:56.042996+0000","last_clean":"2023-09-10T01:01:56.042996+0000","last_became_active":"2023-09-10T01:00:44.033170+0000","last_became_peered":"2023-09-10T01:00:44.033170+0000","last_unstale":"2023-09-10T01:01:56.042996+0000","last_undegraded":"2023-09-10T01:01:56.042996+0000","last_fullsized":"2023-09-10T01:01:56.042996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:36:12.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039255+0000","last_change":"2023-09-10T01:00:48.067436+0000","last_active":"2023-09-10T01:01:56.039255+0000","last_peered":"2023-09-10T01:01:56.039255+0000","last_clean":"2023-09-10T01:01:56.039255+0000","last_became_active":"2023-09-10T01:00:48.067038+0000","last_became_peered":"2023-09-10T01:00:48.067038+0000","last_unstale":"2023-09-10T01:01:56.039255+0000","last_undegraded":"2023-09-10T01:01:56.039255+0000","last_fullsized":"2023-09-10T01:01:56.039255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:26:41.210698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037337+0000","last_change":"2023-09-10T01:00:43.006934+0000","last_active":"2023-09-10T01:01:56.037337+0000","last_peered":"2023-09-10T01:01:56.037337+0000","last_clean":"2023-09-10T01:01:56.037337+0000","last_became_active":"2023-09-10T01:00:43.005787+0000","last_became_peered":"2023-09-10T01:00:43.005787+0000","last_unstale":"2023-09-10T01:01:56.037337+0000","last_undegraded":"2023-09-10T01:01:56.037337+0000","last_fullsized":"2023-09-10T01:01:56.037337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:48:27.403220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042682+0000","last_change":"2023-09-10T01:00:44.033812+0000","last_active":"2023-09-10T01:01:56.042682+0000","last_peered":"2023-09-10T01:01:56.042682+0000","last_clean":"2023-09-10T01:01:56.042682+0000","last_became_active":"2023-09-10T01:00:44.033496+0000","last_became_peered":"2023-09-10T01:00:44.033496+0000","last_unstale":"2023-09-10T01:01:56.042682+0000","last_undegraded":"2023-09-10T01:01:56.042682+0000","last_fullsized":"2023-09-10T01:01:56.042682+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:12.679368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036166+0000","last_change":"2023-09-10T01:00:48.060358+0000","last_active":"2023-09-10T01:01:56.036166+0000","last_peered":"2023-09-10T01:01:56.036166+0000","last_clean":"2023-09-10T01:01:56.036166+0000","last_became_active":"2023-09-10T01:00:48.059853+0000","last_became_peered":"2023-09-10T01:00:48.059853+0000","last_unstale":"2023-09-10T01:01:56.036166+0000","last_undegraded":"2023-09-10T01:01:56.036166+0000","last_fullsized":"2023-09-10T01:01:56.036166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:31:56.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031904+0000","last_change":"2023-09-10T01:00:43.002820+0000","last_active":"2023-09-10T01:01:56.031904+0000","last_peered":"2023-09-10T01:01:56.031904+0000","last_clean":"2023-09-10T01:01:56.031904+0000","last_became_active":"2023-09-10T01:00:43.002456+0000","last_became_peered":"2023-09-10T01:00:43.002456+0000","last_unstale":"2023-09-10T01:01:56.031904+0000","last_undegraded":"2023-09-10T01:01:56.031904+0000","last_fullsized":"2023-09-10T01:01:56.031904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:40:10.189942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.043029+0000","last_change":"2023-09-10T01:00:44.032655+0000","last_active":"2023-09-10T01:01:56.043029+0000","last_peered":"2023-09-10T01:01:56.043029+0000","last_clean":"2023-09-10T01:01:56.043029+0000","last_became_active":"2023-09-10T01:00:44.032157+0000","last_became_peered":"2023-09-10T01:00:44.032157+0000","last_unstale":"2023-09-10T01:01:56.043029+0000","last_undegraded":"2023-09-10T01:01:56.043029+0000","last_fullsized":"2023-09-10T01:01:56.043029+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:31.176768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127525+0000","last_change":"2023-09-10T01:00:44.022285+0000","last_active":"2023-09-10T01:01:56.127525+0000","last_peered":"2023-09-10T01:01:56.127525+0000","last_clean":"2023-09-10T01:01:56.127525+0000","last_became_active":"2023-09-10T01:00:44.021959+0000","last_became_peered":"2023-09-10T01:00:44.021959+0000","last_unstale":"2023-09-10T01:01:56.127525+0000","last_undegraded":"2023-09-10T01:01:56.127525+0000","last_fullsized":"2023-09-10T01:01:56.127525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:13:07.072249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032781+0000","last_change":"2023-09-10T01:00:44.029722+0000","last_active":"2023-09-10T01:01:56.032781+0000","last_peered":"2023-09-10T01:01:56.032781+0000","last_clean":"2023-09-10T01:01:56.032781+0000","last_became_active":"2023-09-10T01:00:44.029424+0000","last_became_peered":"2023-09-10T01:00:44.029424+0000","last_unstale":"2023-09-10T01:01:56.032781+0000","last_undegraded":"2023-09-10T01:01:56.032781+0000","last_fullsized":"2023-09-10T01:01:56.032781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:24:16.655538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032601+0000","last_change":"2023-09-10T01:00:44.026026+0000","last_active":"2023-09-10T01:01:56.032601+0000","last_peered":"2023-09-10T01:01:56.032601+0000","last_clean":"2023-09-10T01:01:56.032601+0000","last_became_active":"2023-09-10T01:00:44.025727+0000","last_became_peered":"2023-09-10T01:00:44.025727+0000","last_unstale":"2023-09-10T01:01:56.032601+0000","last_undegraded":"2023-09-10T01:01:56.032601+0000","last_fullsized":"2023-09-10T01:01:56.032601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:59.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039452+0000","last_change":"2023-09-10T01:00:56.965355+0000","last_active":"2023-09-10T01:01:56.039452+0000","last_peered":"2023-09-10T01:01:56.039452+0000","last_clean":"2023-09-10T01:01:56.039452+0000","last_became_active":"2023-09-10T01:00:56.964903+0000","last_became_peered":"2023-09-10T01:00:56.964903+0000","last_unstale":"2023-09-10T01:01:56.039452+0000","last_undegraded":"2023-09-10T01:01:56.039452+0000","last_fullsized":"2023-09-10T01:01:56.039452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:49:21.940159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031377+0000","last_change":"2023-09-10T01:00:44.029525+0000","last_active":"2023-09-10T01:01:56.031377+0000","last_peered":"2023-09-10T01:01:56.031377+0000","last_clean":"2023-09-10T01:01:56.031377+0000","last_became_active":"2023-09-10T01:00:44.029139+0000","last_became_peered":"2023-09-10T01:00:44.029139+0000","last_unstale":"2023-09-10T01:01:56.031377+0000","last_undegraded":"2023-09-10T01:01:56.031377+0000","last_fullsized":"2023-09-10T01:01:56.031377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:20:24.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127877+0000","last_change":"2023-09-10T01:00:44.022299+0000","last_active":"2023-09-10T01:01:56.127877+0000","last_peered":"2023-09-10T01:01:56.127877+0000","last_clean":"2023-09-10T01:01:56.127877+0000","last_became_active":"2023-09-10T01:00:44.021992+0000","last_became_peered":"2023-09-10T01:00:44.021992+0000","last_unstale":"2023-09-10T01:01:56.127877+0000","last_undegraded":"2023-09-10T01:01:56.127877+0000","last_fullsized":"2023-09-10T01:01:56.127877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:50:30.766442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.108199+0000","last_change":"2023-09-10T01:01:57.108199+0000","last_active":"2023-09-10T01:01:57.108199+0000","last_peered":"2023-09-10T01:01:57.108199+0000","last_clean":"2023-09-10T01:01:57.108199+0000","last_became_active":"2023-09-10T01:01:57.107825+0000","last_became_peered":"2023-09-10T01:01:57.107825+0000","last_unstale":"2023-09-10T01:01:57.108199+0000","last_undegraded":"2023-09-10T01:01:57.108199+0000","last_fullsized":"2023-09-10T01:01:57.108199+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:31:15.499277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,9],"acting":[2,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.050651+0000","last_change":"2023-09-10T01:01:57.049936+0000","last_active":"2023-09-10T01:01:57.050651+0000","last_peered":"2023-09-10T01:01:57.050651+0000","last_clean":"2023-09-10T01:01:57.050651+0000","last_became_active":"2023-09-10T01:01:57.049624+0000","last_became_peered":"2023-09-10T01:01:57.049624+0000","last_unstale":"2023-09-10T01:01:57.050651+0000","last_undegraded":"2023-09-10T01:01:57.050651+0000","last_fullsized":"2023-09-10T01:01:57.050651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:57:17.164558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036462+0000","last_change":"2023-09-10T01:00:44.025405+0000","last_active":"2023-09-10T01:01:56.036462+0000","last_peered":"2023-09-10T01:01:56.036462+0000","last_clean":"2023-09-10T01:01:56.036462+0000","last_became_active":"2023-09-10T01:00:44.025040+0000","last_became_peered":"2023-09-10T01:00:44.025040+0000","last_unstale":"2023-09-10T01:01:56.036462+0000","last_undegraded":"2023-09-10T01:01:56.036462+0000","last_fullsized":"2023-09-10T01:01:56.036462+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:16:46.504902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036192+0000","last_change":"2023-09-10T01:00:44.023603+0000","last_active":"2023-09-10T01:01:56.036192+0000","last_peered":"2023-09-10T01:01:56.036192+0000","last_clean":"2023-09-10T01:01:56.036192+0000","last_became_active":"2023-09-10T01:00:44.023279+0000","last_became_peered":"2023-09-10T01:00:44.023279+0000","last_unstale":"2023-09-10T01:01:56.036192+0000","last_undegraded":"2023-09-10T01:01:56.036192+0000","last_fullsized":"2023-09-10T01:01:56.036192+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:18:05.737675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037149+0000","last_change":"2023-09-10T01:00:44.035653+0000","last_active":"2023-09-10T01:01:56.037149+0000","last_peered":"2023-09-10T01:01:56.037149+0000","last_clean":"2023-09-10T01:01:56.037149+0000","last_became_active":"2023-09-10T01:00:44.035374+0000","last_became_peered":"2023-09-10T01:00:44.035374+0000","last_unstale":"2023-09-10T01:01:56.037149+0000","last_undegraded":"2023-09-10T01:01:56.037149+0000","last_fullsized":"2023-09-10T01:01:56.037149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:14:45.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030922+0000","last_change":"2023-09-10T01:00:44.035805+0000","last_active":"2023-09-10T01:01:56.030922+0000","last_peered":"2023-09-10T01:01:56.030922+0000","last_clean":"2023-09-10T01:01:56.030922+0000","last_became_active":"2023-09-10T01:00:44.035479+0000","last_became_peered":"2023-09-10T01:00:44.035479+0000","last_unstale":"2023-09-10T01:01:56.030922+0000","last_undegraded":"2023-09-10T01:01:56.030922+0000","last_fullsized":"2023-09-10T01:01:56.030922+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:36.662975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127692+0000","last_change":"2023-09-10T01:00:44.025418+0000","last_active":"2023-09-10T01:01:56.127692+0000","last_peered":"2023-09-10T01:01:56.127692+0000","last_clean":"2023-09-10T01:01:56.127692+0000","last_became_active":"2023-09-10T01:00:44.025104+0000","last_became_peered":"2023-09-10T01:00:44.025104+0000","last_unstale":"2023-09-10T01:01:56.127692+0000","last_undegraded":"2023-09-10T01:01:56.127692+0000","last_fullsized":"2023-09-10T01:01:56.127692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:05:13.050781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282844,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63889,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":342,"num_read_kb":239,"num_write":152,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":191667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481820,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":504243,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69199999999999995,"5min":0.87,"15min":0.87},"last":0.498},{"interface":"front","average":{"1min":0.874,"5min":0.60799999999999998,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.54000000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.626,"15min":0.625},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82299999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.76100000000000001}]},{"osd":2,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45500000000000002,"15min":0.45600000000000002},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.74399999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.53000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63200000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.74399999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.73799999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81799999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.69799999999999995},"min":{"1min":0.495,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.83299999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.84699999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69399999999999995,"15min":0.69199999999999995},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.85599999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.877}]},{"osd":8,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.91500000000000004}]},{"osd":9,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.748,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.92300000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.93300000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93100000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.55900000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.95999999999999996}]}]},{"osd":10,"up_from":70,"seq":300647710751,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":1150818,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.502},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44600000000000001,"15min":0.441},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.57199999999999995}]},{"osd":2,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.747},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.628,"15min":0.63200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.82899999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.626}]},{"osd":3,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.62,"15min":0.63100000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67700000000000005,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.84399999999999997,"15min":0.874},"min":{"1min":0.46300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81200000000000006,"5min":6.7519999999999998,"15min":6.7519999999999998},"last":0.78000000000000003}]},{"osd":4,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73499999999999999,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55600000000000005,"15min":0.55500000000000005},"min":{"1min":0.36299999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80400000000000005,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68899999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69599999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83999999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.443,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88700000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.89400000000000002}]},{"osd":6,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.71699999999999997,"15min":0.71499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95299999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.47999999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91300000000000003,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.91300000000000003}]},{"osd":7,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.76700000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89300000000000002,"5min":1.534,"15min":1.534},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.77000000000000002,"15min":0.77900000000000003},"min":{"1min":0.52600000000000002,"5min":0.439,"15min":0.439},"max":{"1min":0.96599999999999997,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.76300000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.78500000000000003,"15min":0.78800000000000003},"min":{"1min":0.57499999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.92000000000000004,"5min":1.304,"15min":1.304},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":0.85499999999999998,"15min":0.81699999999999995},"min":{"1min":0.51400000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":7.0510000000000002,"5min":7.0510000000000002,"15min":7.0510000000000002},"last":0.94299999999999995}]},{"osd":11,"last update":"Sun Sep 10 01:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.84299999999999997}]}]},{"osd":9,"up_from":65,"seq":279172874277,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":507993,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.32100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.625}]},{"osd":1,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.42599999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.73099999999999998,"15min":0.74199999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93200000000000005,"5min":1.349,"15min":1.349},"last":0.76000000000000001}]},{"osd":2,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.70699999999999996,"15min":0.70799999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.038,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.81200000000000006,"15min":0.82599999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.96399999999999997,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.72799999999999998}]},{"osd":3,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.69799999999999995,"15min":0.71099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2010000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.72799999999999998,"15min":0.748},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.073,"5min":1.647,"15min":1.647},"last":0.71399999999999997}]},{"osd":4,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.82999999999999996,"15min":0.84499999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.052,"5min":1.595,"15min":1.595},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.85399999999999998,"15min":0.86799999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.3080000000000001,"5min":1.492,"15min":1.492},"last":0.96999999999999997}]},{"osd":5,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.91700000000000004,"15min":0.93799999999999994},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1339999999999999,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.88100000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.179,"5min":1.714,"15min":1.714},"last":0.77500000000000002}]},{"osd":6,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.95999999999999996,"15min":0.98299999999999998},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.226,"5min":1.982,"15min":1.982},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.92000000000000004,"15min":0.93600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.3280000000000001,"5min":1.569,"15min":1.569},"last":0.79900000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.98899999999999999,"15min":1.0129999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2050000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.873},{"interface":"front","average":{"1min":0.874,"5min":1,"15min":1.0209999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.274,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.0620000000000001}]},{"osd":10,"last update":"Sun Sep 10 01:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.91000000000000003,"15min":0.92900000000000005},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.151,"5min":1.363,"15min":1.363},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":0.95199999999999996},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.121,"5min":1.319,"15min":1.319},"last":1.149}]},{"osd":11,"last update":"Sun Sep 10 01:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":1.232},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":1.0229999999999999}]}]},{"osd":8,"up_from":57,"seq":244813135914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":503486,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80200000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68700000000000006,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.58699999999999997}]},{"osd":1,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86499999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.878,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.54100000000000004}]},{"osd":2,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.39400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.47399999999999998,"15min":0.45400000000000001},"min":{"1min":0.35999999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.73099999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.47499999999999998}]},{"osd":3,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.754},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.69099999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57199999999999995}]},{"osd":4,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.56100000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97199999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82799999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.45100000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.60399999999999998,"15min":0.58799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.78400000000000003}]},{"osd":6,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.52100000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.627},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73999999999999999,"15min":0.73299999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.004,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.85999999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.70499999999999996,"15min":0.67000000000000004},"min":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.72899999999999998,"15min":0.70899999999999996},"min":{"1min":0.60099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.80500000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:01:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.76100000000000001,"15min":0.75800000000000001},"min":{"1min":0.55800000000000005,"5min":0.505,"15min":0.505},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.75800000000000001,"15min":0.75600000000000001},"min":{"1min":0.64200000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.94099999999999995,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.92800000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.72599999999999998,"15min":0.71199999999999997},"min":{"1min":0.69599999999999995,"5min":0.44,"15min":0.44},"max":{"1min":0.96999999999999997,"5min":1.01,"15min":1.01},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.76200000000000001,"15min":0.74199999999999999},"min":{"1min":0.70699999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.96499999999999997}]},{"osd":11,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":1.0069999999999999}]}]},{"osd":7,"up_from":50,"seq":214748364846,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":1099143,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91400000000000003,"5min":1.367,"15min":1.367},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.44400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.43099999999999999,"15min":0.41999999999999998},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45100000000000001,"15min":0.437},"min":{"1min":0.317,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76100000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.67900000000000005}]},{"osd":2,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59799999999999998,"15min":0.58499999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.42099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85999999999999999,"5min":1.409,"15min":1.409},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.71499999999999997,"15min":0.68300000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":7.1929999999999996,"5min":7.1929999999999996,"15min":7.1929999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.41699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69799999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.94899999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63100000000000001,"15min":0.62},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85799999999999998,"5min":1.038,"15min":1.038},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.64100000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:01:34 2023","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.81799999999999995,"15min":0.75800000000000001},"min":{"1min":0.505,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":9.3919999999999995,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.82499999999999996,"15min":0.79000000000000004},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":7.1760000000000002,"5min":7.1760000000000002,"15min":7.1760000000000002},"last":0.77900000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.012,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70099999999999996,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98399999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.81100000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77300000000000002,"15min":0.77200000000000002},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.79000000000000004,"15min":0.78600000000000003},"min":{"1min":0.59499999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.875}]},{"osd":10,"last update":"Sun Sep 10 01:01:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.749,"15min":0.747},"min":{"1min":0.56899999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.77800000000000002,"15min":0.77700000000000002},"min":{"1min":0.54100000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.90600000000000003}]},{"osd":11,"last update":"Sun Sep 10 01:01:44 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.94699999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528369,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":499073,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52800000000000002,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88400000000000001,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54000000000000004,"15min":0.52300000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86599999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.65600000000000003,"15min":0.624},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98199999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91200000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.73499999999999999,"15min":0.749},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.044,"5min":1.069,"15min":1.069},"last":0.73499999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65400000000000003,"15min":0.64200000000000002},"min":{"1min":0.48899999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.014,"5min":1.147,"15min":1.147},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71899999999999997,"15min":0.72299999999999998},"min":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88400000000000001,"5min":1.425,"15min":1.425},"last":0.56599999999999995}]},{"osd":4,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.73799999999999999,"15min":0.73999999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.95099999999999996,"5min":1.111,"15min":1.111},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.753,"15min":0.73399999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.68300000000000005}]},{"osd":5,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.84099999999999997,"15min":0.85499999999999998},"min":{"1min":0.55300000000000005,"5min":0.503,"15min":0.503},"max":{"1min":0.93700000000000006,"5min":1.53,"15min":1.53},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.83799999999999997,"15min":0.84299999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1,"5min":1.353,"15min":1.353},"last":0.61399999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":0.94699999999999995,"15min":0.86099999999999999},"min":{"1min":0.68100000000000005,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":10.081,"5min":10.081,"15min":10.081},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.004,"5min":0.86799999999999999,"15min":0.83799999999999997},"min":{"1min":0.70299999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.55100000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:01:58 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.92900000000000005,"15min":0.91700000000000004},"min":{"1min":0.64900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89200000000000002,"15min":0.88600000000000001},"min":{"1min":0.65500000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.79200000000000004}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.97799999999999998,"15min":0.99399999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1699999999999999,"5min":1.831,"15min":1.831},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.0109999999999999,"15min":1.0329999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1539999999999999,"5min":1.617,"15min":1.617},"last":0.82899999999999996}]},{"osd":10,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96099999999999997,"15min":0.95799999999999996},"min":{"1min":0.75700000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.069,"15min":0.99299999999999999},"min":{"1min":0.78700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":10.170999999999999,"5min":10.170999999999999,"15min":10.170999999999999},"last":0.80500000000000005}]},{"osd":11,"last update":"Sun Sep 10 01:01:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":10.247,"5min":10.247,"15min":10.247},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3660000000000001,"5min":2.3660000000000001,"15min":2.3660000000000001},"last":0.75600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":558872,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56499999999999995,"15min":0.52700000000000002},"min":{"1min":0.372,"5min":0.312,"15min":0.312},"max":{"1min":0.82199999999999995,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58999999999999997,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72599999999999998,"5min":5.1580000000000004,"15min":5.1580000000000004},"last":0.48399999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.56899999999999995,"15min":0.54200000000000004},"min":{"1min":0.39400000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.42999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.58299999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:01:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.752},"min":{"1min":0.42399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.755,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.69799999999999995,"15min":0.78000000000000003},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84399999999999997,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.78900000000000003}]},{"osd":4,"last update":"Sun Sep 10 01:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.41999999999999998,"15min":0.375},"min":{"1min":0.35399999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.42099999999999999,"15min":0.38200000000000001},"min":{"1min":0.30399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67500000000000004,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.371}]},{"osd":5,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.65400000000000003,"15min":0.57799999999999996},"min":{"1min":0.42199999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":4.9740000000000002,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.028,"5min":0.68899999999999995,"15min":0.60499999999999998},"min":{"1min":0.54600000000000004,"5min":0.38,"15min":0.38},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.63400000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.47699999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.50800000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.60999999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56599999999999995,"15min":0.55400000000000005},"min":{"1min":0.41099999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.64500000000000002}]},{"osd":8,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.68000000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.44,"15min":0.44},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81599999999999995}]},{"osd":9,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.747,"15min":0.73699999999999999},"min":{"1min":0.63600000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78400000000000003,"15min":0.77900000000000003},"min":{"1min":0.63900000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.87}]},{"osd":10,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69699999999999995,"15min":0.68899999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.80300000000000005}]},{"osd":11,"last update":"Sun Sep 10 01:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.68799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500461,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 10 01:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.622,"15min":0.58899999999999997},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97299999999999998,"5min":1.069,"15min":1.069},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58799999999999997,"15min":0.55600000000000005},"min":{"1min":0.45400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81499999999999995,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67800000000000005}]},{"osd":2,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.66700000000000004,"15min":0.60599999999999998},"min":{"1min":0.39600000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.626},"min":{"1min":0.51000000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98399999999999999,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.60199999999999998}]},{"osd":3,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.439},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.92200000000000004}]},{"osd":4,"last update":"Sun Sep 10 01:01:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.63800000000000001,"15min":0.58399999999999996},"min":{"1min":0.57699999999999996,"5min":0.308,"15min":0.308},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.66500000000000004,"15min":0.60799999999999998},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.69099999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.747,"15min":0.71199999999999997},"min":{"1min":0.59799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.028,"5min":1.125,"15min":1.125},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70699999999999996,"15min":0.66800000000000004},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.47599999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68999999999999995,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.74099999999999999,"15min":0.71399999999999997},"min":{"1min":0.496,"5min":0.376,"15min":0.376},"max":{"1min":1.0720000000000001,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.755}]},{"osd":7,"last update":"Sun Sep 10 01:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.747}]},{"osd":8,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.66400000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.0840000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83299999999999996,"15min":0.84299999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1499999999999999,"5min":1.24,"15min":1.24},"last":0.72099999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.91300000000000003,"15min":0.92900000000000005},"min":{"1min":0.64300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1519999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.999,"15min":1.0149999999999999},"min":{"1min":0.76900000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1870000000000001,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.78300000000000003}]},{"osd":10,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.91400000000000003,"15min":0.92600000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.117,"5min":1.405,"15min":1.405},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.93500000000000005,"15min":0.94199999999999995},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.319,"5min":1.444,"15min":1.444},"last":0.79500000000000004}]},{"osd":11,"last update":"Sun Sep 10 01:02:03 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.92300000000000004,"15min":0.91800000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.002,"5min":0.97799999999999998,"15min":0.97399999999999998},"min":{"1min":0.76700000000000002,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.85699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":559600,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.64400000000000002,"15min":0.54300000000000004},"min":{"1min":0.46999999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.7199999999999998,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.63600000000000001,"15min":0.55800000000000005},"min":{"1min":0.51200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":6.6639999999999997,"5min":6.6639999999999997,"15min":6.6639999999999997},"last":0.66400000000000003}]},{"osd":1,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.60399999999999998,"15min":0.54700000000000004},"min":{"1min":0.48499999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.61199999999999999,"15min":0.55200000000000005},"min":{"1min":0.61099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.88800000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.56899999999999995},"min":{"1min":0.49399999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.036,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.59399999999999997},"min":{"1min":0.48099999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.89400000000000002,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.83499999999999996}]},{"osd":4,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.91200000000000003}]},{"osd":5,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.498,"15min":0.44600000000000001},"min":{"1min":0.434,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.34200000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.86899999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:01:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.68899999999999995,"15min":0.66600000000000004},"min":{"1min":0.42599999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.073,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.68799999999999994,"15min":0.67200000000000004},"min":{"1min":0.59899999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81999999999999995}]},{"osd":7,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":1.105},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.95799999999999996}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56399999999999995,"15min":0.54500000000000004},"min":{"1min":0.51000000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":1.05},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.73299999999999998,"15min":0.71499999999999997},"min":{"1min":0.59499999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.97099999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:01:18 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.79900000000000004,"15min":0.78800000000000003},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.72899999999999998,"15min":0.72099999999999997},"min":{"1min":0.622,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":0.91700000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":1.1539999999999999}]},{"osd":10,"last update":"Sun Sep 10 01:01:41 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.68600000000000005,"15min":0.66600000000000004},"min":{"1min":0.59299999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":1.117},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.79500000000000004,"15min":0.77400000000000002},"min":{"1min":0.68100000000000005,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.98599999999999999}]},{"osd":11,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.016}]}]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":499267,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41999999999999998,"15min":0.39000000000000001},"min":{"1min":0.32200000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65000000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.42699999999999999},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.628}]},{"osd":1,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.65100000000000002,"15min":0.60299999999999998},"min":{"1min":0.52700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.117,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56100000000000005},"min":{"1min":0.40999999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.90100000000000002,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69399999999999995}]},{"osd":2,"last update":"Sun Sep 10 01:01:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68300000000000005,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.012,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.68999999999999995,"15min":0.63600000000000001},"min":{"1min":0.499,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0820000000000001,"5min":1.226,"15min":1.226},"last":0.65100000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.72699999999999998,"15min":0.68200000000000005},"min":{"1min":0.45200000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.036,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.77500000000000002,"15min":0.71999999999999997},"min":{"1min":0.57199999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.0309999999999999,"5min":2.52,"15min":2.52},"last":0.879}]},{"osd":5,"last update":"Sun Sep 10 01:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.79500000000000004,"15min":0.73299999999999998},"min":{"1min":0.51600000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":0.57999999999999996}]},{"osd":6,"last update":"Sun Sep 10 01:01:13 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65400000000000003,"15min":0.623},"min":{"1min":0.5,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.70899999999999996,"15min":0.66100000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.127,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.54100000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:01:32 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.78100000000000003,"15min":0.79300000000000004},"min":{"1min":0.499,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.7969999999999999,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.046,"5min":0.876,"15min":0.83799999999999997},"min":{"1min":0.55400000000000005,"5min":0.434,"15min":0.434},"max":{"1min":4.7699999999999996,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.86699999999999999}]},{"osd":8,"last update":"Sun Sep 10 00:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Sun Sep 10 01:01:22 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.94799999999999995,"15min":0.97599999999999998},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0549999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.87,"5min":0.92200000000000004,"15min":0.93100000000000005},"min":{"1min":0.61399999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.26,"5min":1.458,"15min":1.458},"last":0.96899999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:01:38 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.95999999999999996,"15min":0.96599999999999997},"min":{"1min":0.70699999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82599999999999996,"15min":0.82199999999999995},"min":{"1min":0.63200000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.89500000000000002}]},{"osd":11,"last update":"Sun Sep 10 01:01:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":4.9139999999999997},"last":0.83699999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":504735,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.38700000000000001,"15min":0.36099999999999999},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.73899999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.39700000000000002,"15min":0.377},"min":{"1min":0.29099999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64200000000000002,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.56100000000000005},"min":{"1min":0.40200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.871,"5min":1.607,"15min":1.607},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.52700000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88900000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.76100000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:01:24 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59199999999999997,"15min":0.56999999999999995},"min":{"1min":0.45600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sun Sep 10 01:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.64300000000000002,"15min":0.60699999999999998},"min":{"1min":0.55000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":1.621,"15min":1.621},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.63300000000000001,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71899999999999997}]},{"osd":6,"last update":"Sun Sep 10 01:01:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80400000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.93100000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.70499999999999996}]},{"osd":7,"last update":"Sun Sep 10 01:01:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.57399999999999995,"15min":0.54100000000000004},"min":{"1min":0.41199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.91400000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.77900000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91100000000000003,"5min":1.802,"15min":1.802},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.755,"15min":0.73299999999999998},"min":{"1min":0.56100000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.81000000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:01:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76200000000000001,"15min":0.77000000000000002},"min":{"1min":0.505,"5min":0.495,"15min":0.495},"max":{"1min":0.88400000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.79400000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:01:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78100000000000003,"15min":0.77800000000000002},"min":{"1min":0.64800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.65700000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84699999999999998}]},{"osd":11,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.874}]}]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1095313,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54000000000000004,"15min":0.50600000000000001},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81399999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.879},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.53100000000000003,"15min":0.50700000000000001},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79800000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.75900000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.59899999999999998,"15min":0.55500000000000005},"min":{"1min":0.48199999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86099999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.59899999999999998,"15min":0.55700000000000005},"min":{"1min":0.55600000000000005,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.42199999999999999,"15min":0.38700000000000001},"min":{"1min":0.36699999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.67300000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.442,"15min":0.41199999999999998},"min":{"1min":0.374,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44800000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.378,"5min":0.376,"15min":0.376},"max":{"1min":0.752,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.60199999999999998,"15min":0.55800000000000005},"min":{"1min":0.501,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.75}]},{"osd":4,"last update":"Sun Sep 10 01:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.622,"15min":0.59199999999999997},"min":{"1min":0.53500000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.627,"15min":0.59999999999999998},"min":{"1min":0.51300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.627}]},{"osd":6,"last update":"Sun Sep 10 01:01:16 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.64400000000000002,"15min":0.629},"min":{"1min":0.53300000000000003,"5min":0.437,"15min":0.437},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.54200000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.80300000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:01:33 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.68000000000000005,"15min":0.67000000000000004},"min":{"1min":0.48499999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91400000000000003,"5min":1.042,"15min":1.042},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0549999999999999,"5min":1.081,"15min":1.081},"last":0.52600000000000002}]},{"osd":8,"last update":"Sun Sep 10 01:01:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.436,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55300000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69999999999999996,"15min":0.68999999999999995},"min":{"1min":0.55900000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.93799999999999994,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77100000000000002,"15min":0.76100000000000001},"min":{"1min":0.59799999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.86199999999999999}]},{"osd":10,"last update":"Sun Sep 10 01:01:39 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0720000000000001,"5min":1.107,"15min":1.107},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.78500000000000003,"15min":0.76700000000000002},"min":{"1min":0.54500000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.64300000000000002}]},{"osd":11,"last update":"Sun Sep 10 01:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-10T01:02:07.525 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-09-10T01:02:07.526 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-10T01:02:07.526 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-09-10T01:02:07.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph health --format=json 2023-09-10T01:02:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:08.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:07 smithi003 ceph-mon[97654]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:08.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:07 smithi136 ceph-mon[133708]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:07 smithi092 ceph-mon[134067]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:09.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:08 smithi003 ceph-mon[97654]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:09.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:08 smithi136 ceph-mon[133708]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:09.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:08 smithi092 ceph-mon[134067]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:09.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 -- 172.21.15.3:0/2096698324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad81114f0 msgr2=0x7f2ad81138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:09.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 --2- 172.21.15.3:0/2096698324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad81114f0 0x7f2ad81138e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000b0a0 tx=0x7f2ad000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:02:09.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 -- 172.21.15.3:0/2096698324 shutdown_connections 2023-09-10T01:02:09.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 --2- 172.21.15.3:0/2096698324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad81114f0 0x7f2ad81138e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:09.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 --2- 172.21.15.3:0/2096698324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ad8074b30 0x7f2ad8074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:09.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 --2- 172.21.15.3:0/2096698324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8075450 0x7f2ad8110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:09.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 -- 172.21.15.3:0/2096698324 >> 172.21.15.3:0/2096698324 conn(0x7f2ad80fe300 msgr2=0x7f2ad8100720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:09.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 -- 172.21.15.3:0/2096698324 shutdown_connections 2023-09-10T01:02:09.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.957+0000 7f2ade44f700 1 -- 172.21.15.3:0/2096698324 wait complete. 2023-09-10T01:02:09.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.958+0000 7f2ade44f700 1 Processor -- start 2023-09-10T01:02:09.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.958+0000 7f2ade44f700 1 -- start start 2023-09-10T01:02:09.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ade44f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad8074b30 0x7f2ad819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:09.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ade44f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8075450 0x7f2ad819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:09.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ade44f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ad81114f0 0x7f2ad81a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:09.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ade44f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad81a1e50 con 0x7f2ad8074b30 2023-09-10T01:02:09.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ade44f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad81a1fc0 con 0x7f2ad81114f0 2023-09-10T01:02:09.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ade44f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad81a2180 con 0x7f2ad8075450 2023-09-10T01:02:09.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad8074b30 0x7f2ad819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:09.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad8074b30 0x7f2ad819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53928/0 (socket says 172.21.15.3:53928) 2023-09-10T01:02:09.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 -- 172.21.15.3:0/157478545 learned_addr learned my addr 172.21.15.3:0/157478545 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:09.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad77fe700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8075450 0x7f2ad819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:09.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2adc9ec700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ad81114f0 0x7f2ad81a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:09.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 -- 172.21.15.3:0/157478545 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8075450 msgr2=0x7f2ad819d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:09.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8075450 0x7f2ad819d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:09.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 -- 172.21.15.3:0/157478545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ad81114f0 msgr2=0x7f2ad81a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:09.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ad81114f0 0x7f2ad81a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:09.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2ad7fff700 1 -- 172.21.15.3:0/157478545 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad0009d00 con 0x7f2ad8074b30 2023-09-10T01:02:09.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.959+0000 7f2adc9ec700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ad81114f0 0x7f2ad81a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:09.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.960+0000 7f2ad7fff700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad8074b30 0x7f2ad819ce90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800e9a0 tx=0x7f2ac800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:09.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.960+0000 7f2ad57fa700 1 -- 172.21.15.3:0/157478545 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8009790 con 0x7f2ad8074b30 2023-09-10T01:02:09.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.960+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad81a2460 con 0x7f2ad8074b30 2023-09-10T01:02:09.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.960+0000 7f2ad57fa700 1 -- 172.21.15.3:0/157478545 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ac800cbd0 con 0x7f2ad8074b30 2023-09-10T01:02:09.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.960+0000 7f2ad57fa700 1 -- 172.21.15.3:0/157478545 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8018680 con 0x7f2ad8074b30 2023-09-10T01:02:09.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.960+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad81a9cc0 con 0x7f2ad8074b30 2023-09-10T01:02:09.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.961+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad804e300 con 0x7f2ad8074b30 2023-09-10T01:02:09.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.963+0000 7f2ad57fa700 1 -- 172.21.15.3:0/157478545 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f2ac800cd40 con 0x7f2ad8074b30 2023-09-10T01:02:09.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.964+0000 7f2ad57fa700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2ac00a0750 0x7f2ac00a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:09.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.965+0000 7f2ad57fa700 1 -- 172.21.15.3:0/157478545 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f2ac8014070 con 0x7f2ad8074b30 2023-09-10T01:02:09.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.965+0000 7f2ad77fe700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2ac00a0750 0x7f2ac00a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:09.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.965+0000 7f2ad77fe700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2ac00a0750 0x7f2ac00a2c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2ad819e4b0 tx=0x7f2acc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:09.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:09.968+0000 7f2ad57fa700 1 -- 172.21.15.3:0/157478545 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f2ac8051e10 con 0x7f2ad8074b30 2023-09-10T01:02:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.223+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2ad819e1f0 con 0x7f2ad8074b30 2023-09-10T01:02:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.224+0000 7f2ad57fa700 1 -- 172.21.15.3:0/157478545 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2ac80959e0 con 0x7f2ad8074b30 2023-09-10T01:02:10.225 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:10.225 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-10T01:02:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2ac00a0750 msgr2=0x7f2ac00a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2ac00a0750 0x7f2ac00a2c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2ad819e4b0 tx=0x7f2acc009260 comp rx=0 tx=0).stop 2023-09-10T01:02:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad8074b30 msgr2=0x7f2ad819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad8074b30 0x7f2ad819ce90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800e9a0 tx=0x7f2ac800ed60 comp rx=0 tx=0).stop 2023-09-10T01:02:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 shutdown_connections 2023-09-10T01:02:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ad81114f0 0x7f2ad81a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad8074b30 0x7f2ad819ce90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2ac00a0750 0x7f2ac00a2c10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 --2- 172.21.15.3:0/157478545 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8075450 0x7f2ad819d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 >> 172.21.15.3:0/157478545 conn(0x7f2ad80fe300 msgr2=0x7f2ad81006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 shutdown_connections 2023-09-10T01:02:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:10.226+0000 7f2ade44f700 1 -- 172.21.15.3:0/157478545 wait complete. 2023-09-10T01:02:10.856 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-09-10T01:02:10.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:11.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:10 smithi003 ceph-mon[97654]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:11.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:10 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/157478545' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-09-10T01:02:11.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 -- 172.21.15.3:0/1965074397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341018c0 msgr2=0x7f4934101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 --2- 172.21.15.3:0/1965074397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341018c0 0x7f4934101d20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4924009990 tx=0x7f4924009ca0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 -- 172.21.15.3:0/1965074397 shutdown_connections 2023-09-10T01:02:11.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 --2- 172.21.15.3:0/1965074397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4934102260 0x7f4934108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 --2- 172.21.15.3:0/1965074397 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341006d0 0x7f4934100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 --2- 172.21.15.3:0/1965074397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341018c0 0x7f4934101d20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 -- 172.21.15.3:0/1965074397 >> 172.21.15.3:0/1965074397 conn(0x7f49340fbc60 msgr2=0x7f49340fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:11.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 -- 172.21.15.3:0/1965074397 shutdown_connections 2023-09-10T01:02:11.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.069+0000 7f493bfff700 1 -- 172.21.15.3:0/1965074397 wait complete. 2023-09-10T01:02:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.070+0000 7f493bfff700 1 Processor -- start 2023-09-10T01:02:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493bfff700 1 -- start start 2023-09-10T01:02:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341006d0 0x7f4934198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341018c0 0x7f4934198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4934102260 0x7f493419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49341a0100 con 0x7f49341018c0 2023-09-10T01:02:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493bfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49341a0240 con 0x7f4934102260 2023-09-10T01:02:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493bfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49341a03d0 con 0x7f49341006d0 2023-09-10T01:02:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493959a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341018c0 0x7f4934198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f4939d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341006d0 0x7f4934198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493959a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341018c0 0x7f4934198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53952/0 (socket says 172.21.15.3:53952) 2023-09-10T01:02:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493959a700 1 -- 172.21.15.3:0/700815006 learned_addr learned my addr 172.21.15.3:0/700815006 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.071+0000 7f493a59c700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4934102260 0x7f493419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f4939d9b700 1 -- 172.21.15.3:0/700815006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4934102260 msgr2=0x7f493419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f4939d9b700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4934102260 0x7f493419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f4939d9b700 1 -- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341018c0 msgr2=0x7f4934198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f4939d9b700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341018c0 0x7f4934198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f4939d9b700 1 -- 172.21.15.3:0/700815006 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930009520 con 0x7f49341006d0 2023-09-10T01:02:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f493959a700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341018c0 0x7f4934198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f4939d9b700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341006d0 0x7f4934198580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f493000e9c0 tx=0x7f493000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f492affd700 1 -- 172.21.15.3:0/700815006 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493000cb70 con 0x7f49341006d0 2023-09-10T01:02:11.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f492affd700 1 -- 172.21.15.3:0/700815006 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f493000ee60 con 0x7f49341006d0 2023-09-10T01:02:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49240095f0 con 0x7f49341006d0 2023-09-10T01:02:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f492affd700 1 -- 172.21.15.3:0/700815006 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4930018680 con 0x7f49341006d0 2023-09-10T01:02:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.072+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49341a0a30 con 0x7f49341006d0 2023-09-10T01:02:11.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.073+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4934192ce0 con 0x7f49341006d0 2023-09-10T01:02:11.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.076+0000 7f492affd700 1 -- 172.21.15.3:0/700815006 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f4930010430 con 0x7f49341006d0 2023-09-10T01:02:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.077+0000 7f492affd700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f49200a06d0 0x7f49200a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.077+0000 7f492affd700 1 -- 172.21.15.3:0/700815006 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f4930014070 con 0x7f49341006d0 2023-09-10T01:02:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.077+0000 7f493959a700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f49200a06d0 0x7f49200a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.078+0000 7f493959a700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f49200a06d0 0x7f49200a2b90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f492400b2e0 tx=0x7f4924005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:11.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.079+0000 7f492affd700 1 -- 172.21.15.3:0/700815006 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f4930096000 con 0x7f49341006d0 2023-09-10T01:02:11.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.289+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f493410bef0 con 0x7f49341006d0 2023-09-10T01:02:11.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.291+0000 7f492affd700 1 -- 172.21.15.3:0/700815006 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7f493000ce60 con 0x7f49341006d0 2023-09-10T01:02:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:11.292 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f49200a06d0 msgr2=0x7f49200a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f49200a06d0 0x7f49200a2b90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f492400b2e0 tx=0x7f4924005960 comp rx=0 tx=0).stop 2023-09-10T01:02:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341006d0 msgr2=0x7f4934198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341006d0 0x7f4934198580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f493000e9c0 tx=0x7f493000ecd0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 shutdown_connections 2023-09-10T01:02:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f49200a06d0 0x7f49200a2b90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49341006d0 0x7f4934198580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4934102260 0x7f493419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 --2- 172.21.15.3:0/700815006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49341018c0 0x7f4934198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.293+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 >> 172.21.15.3:0/700815006 conn(0x7f49340fbc60 msgr2=0x7f49341071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.294+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 shutdown_connections 2023-09-10T01:02:11.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.294+0000 7f493bfff700 1 -- 172.21.15.3:0/700815006 wait complete. 2023-09-10T01:02:11.313 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:02:11.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:02:11.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:10 smithi136 ceph-mon[133708]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:11.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:10 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/157478545' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-09-10T01:02:11.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:10 smithi092 ceph-mon[134067]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:11.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:10 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/157478545' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-09-10T01:02:11.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 -- 172.21.15.3:0/1834082350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 msgr2=0x7f56b8101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 --2- 172.21.15.3:0/1834082350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8101d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0099c0 tx=0x7f56ac009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 -- 172.21.15.3:0/1834082350 shutdown_connections 2023-09-10T01:02:11.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 --2- 172.21.15.3:0/1834082350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56b81006f0 0x7f56b8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 --2- 172.21.15.3:0/1834082350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56b8102280 0x7f56b8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 --2- 172.21.15.3:0/1834082350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8101d40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 -- 172.21.15.3:0/1834082350 >> 172.21.15.3:0/1834082350 conn(0x7f56b80fbc60 msgr2=0x7f56b80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:11.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 -- 172.21.15.3:0/1834082350 shutdown_connections 2023-09-10T01:02:11.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.520+0000 7f56bddff700 1 -- 172.21.15.3:0/1834082350 wait complete. 2023-09-10T01:02:11.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.521+0000 7f56bddff700 1 Processor -- start 2023-09-10T01:02:11.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.521+0000 7f56bddff700 1 -- start start 2023-09-10T01:02:11.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56bddff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56b81006f0 0x7f56b8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56bddff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56bddff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56b8102280 0x7f56b819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56bddff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b81a01a0 con 0x7f56b81018e0 2023-09-10T01:02:11.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56bddff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b81a02e0 con 0x7f56b8102280 2023-09-10T01:02:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56bddff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b81a0470 con 0x7f56b81006f0 2023-09-10T01:02:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53978/0 (socket says 172.21.15.3:53978) 2023-09-10T01:02:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 -- 172.21.15.3:0/1058200610 learned_addr learned my addr 172.21.15.3:0/1058200610 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b77fe700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56b81006f0 0x7f56b8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b7fff700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56b8102280 0x7f56b819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 -- 172.21.15.3:0/1058200610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56b81006f0 msgr2=0x7f56b8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56b81006f0 0x7f56b8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 -- 172.21.15.3:0/1058200610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56b8102280 msgr2=0x7f56b819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56b8102280 0x7f56b819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.522+0000 7f56b6ffd700 1 -- 172.21.15.3:0/1058200610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8009520 con 0x7f56b81018e0 2023-09-10T01:02:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.523+0000 7f56b6ffd700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8198b60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f56ac009fd0 tx=0x7f56ac00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.523+0000 7f56b4ff9700 1 -- 172.21.15.3:0/1058200610 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac01d070 con 0x7f56b81018e0 2023-09-10T01:02:11.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.523+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ac0095f0 con 0x7f56b81018e0 2023-09-10T01:02:11.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.523+0000 7f56b4ff9700 1 -- 172.21.15.3:0/1058200610 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56ac005510 con 0x7f56b81018e0 2023-09-10T01:02:11.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.523+0000 7f56b4ff9700 1 -- 172.21.15.3:0/1058200610 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac004ae0 con 0x7f56b81018e0 2023-09-10T01:02:11.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.523+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b81a09e0 con 0x7f56b81018e0 2023-09-10T01:02:11.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.524+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b8192d50 con 0x7f56b81018e0 2023-09-10T01:02:11.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.526+0000 7f56b4ff9700 1 -- 172.21.15.3:0/1058200610 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f56ac00fb80 con 0x7f56b81018e0 2023-09-10T01:02:11.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.527+0000 7f56b4ff9700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56a00a06d0 0x7f56a00a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.527+0000 7f56b4ff9700 1 -- 172.21.15.3:0/1058200610 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f56ac0cd3a0 con 0x7f56b81018e0 2023-09-10T01:02:11.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.528+0000 7f56b77fe700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56a00a06d0 0x7f56a00a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.528+0000 7f56b77fe700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56a00a06d0 0x7f56a00a2b90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56b8101740 tx=0x7f56a8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:11.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.530+0000 7f56b4ff9700 1 -- 172.21.15.3:0/1058200610 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f56ac0969f0 con 0x7f56b81018e0 2023-09-10T01:02:11.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.741+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f56b810c0c0 con 0x7f56b81018e0 2023-09-10T01:02:11.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.742+0000 7f56b4ff9700 1 -- 172.21.15.3:0/1058200610 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f56ac026020 con 0x7f56b81018e0 2023-09-10T01:02:11.743 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:02:11.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56a00a06d0 msgr2=0x7f56a00a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56a00a06d0 0x7f56a00a2b90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56b8101740 tx=0x7f56a8009260 comp rx=0 tx=0).stop 2023-09-10T01:02:11.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 msgr2=0x7f56b8198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8198b60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f56ac009fd0 tx=0x7f56ac00f690 comp rx=0 tx=0).stop 2023-09-10T01:02:11.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 shutdown_connections 2023-09-10T01:02:11.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f56a00a06d0 0x7f56a00a2b90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56b81006f0 0x7f56b8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56b8102280 0x7f56b819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 --2- 172.21.15.3:0/1058200610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56b81018e0 0x7f56b8198b60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 >> 172.21.15.3:0/1058200610 conn(0x7f56b80fbc60 msgr2=0x7f56b81071a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:11.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 shutdown_connections 2023-09-10T01:02:11.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.744+0000 7f56bddff700 1 -- 172.21.15.3:0/1058200610 wait complete. 2023-09-10T01:02:11.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:02:11.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 -- 172.21.15.3:0/924311673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641018d0 msgr2=0x7f6564101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 --2- 172.21.15.3:0/924311673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641018d0 0x7f6564101d30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f654c0099c0 tx=0x7f654c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 -- 172.21.15.3:0/924311673 shutdown_connections 2023-09-10T01:02:11.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 --2- 172.21.15.3:0/924311673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65641006e0 0x7f6564100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 --2- 172.21.15.3:0/924311673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6564102270 0x7f6564108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 --2- 172.21.15.3:0/924311673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641018d0 0x7f6564101d30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 -- 172.21.15.3:0/924311673 >> 172.21.15.3:0/924311673 conn(0x7f65640fbc50 msgr2=0x7f65640fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:11.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 -- 172.21.15.3:0/924311673 shutdown_connections 2023-09-10T01:02:11.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.972+0000 7f6568fff700 1 -- 172.21.15.3:0/924311673 wait complete. 2023-09-10T01:02:11.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.973+0000 7f6568fff700 1 Processor -- start 2023-09-10T01:02:11.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.973+0000 7f6568fff700 1 -- start start 2023-09-10T01:02:11.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f6568fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641006e0 0x7f65641963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f6568fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65641018d0 0x7f65641968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f6568fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6564102270 0x7f656419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f6568fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656419df20 con 0x7f65641006e0 2023-09-10T01:02:11.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f6568fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656419e060 con 0x7f65641018d0 2023-09-10T01:02:11.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f6568fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656419e1f0 con 0x7f6564102270 2023-09-10T01:02:11.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641006e0 0x7f65641963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641006e0 0x7f65641963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53992/0 (socket says 172.21.15.3:53992) 2023-09-10T01:02:11.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 -- 172.21.15.3:0/325806259 learned_addr learned my addr 172.21.15.3:0/325806259 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:11.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f655bfff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65641018d0 0x7f65641968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f6562d9d700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6564102270 0x7f656419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 -- 172.21.15.3:0/325806259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6564102270 msgr2=0x7f656419d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6564102270 0x7f656419d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 -- 172.21.15.3:0/325806259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65641018d0 msgr2=0x7f65641968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:11.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65641018d0 0x7f65641968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.974+0000 7f656259c700 1 -- 172.21.15.3:0/325806259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c0095f0 con 0x7f65641006e0 2023-09-10T01:02:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.975+0000 7f656259c700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641006e0 0x7f65641963a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f655400e970 tx=0x7f655400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.975+0000 7f655b7fe700 1 -- 172.21.15.3:0/325806259 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554009790 con 0x7f65641006e0 2023-09-10T01:02:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.975+0000 7f655b7fe700 1 -- 172.21.15.3:0/325806259 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f655400eec0 con 0x7f65641006e0 2023-09-10T01:02:11.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.975+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656419e4d0 con 0x7f65641006e0 2023-09-10T01:02:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.975+0000 7f655b7fe700 1 -- 172.21.15.3:0/325806259 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554003cf0 con 0x7f65641006e0 2023-09-10T01:02:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.975+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656419ea30 con 0x7f65641006e0 2023-09-10T01:02:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.976+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6564107610 con 0x7f65641006e0 2023-09-10T01:02:11.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.979+0000 7f655b7fe700 1 -- 172.21.15.3:0/325806259 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f655400ccd0 con 0x7f65641006e0 2023-09-10T01:02:11.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.981+0000 7f655b7fe700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65500a0750 0x7f65500a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:11.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.981+0000 7f655b7fe700 1 -- 172.21.15.3:0/325806259 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f6554014070 con 0x7f65641006e0 2023-09-10T01:02:11.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.981+0000 7f655bfff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65500a0750 0x7f65500a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:11.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.982+0000 7f655bfff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65500a0750 0x7f65500a2c10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f654c00b580 tx=0x7f654c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:11.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:11.983+0000 7f655b7fe700 1 -- 172.21.15.3:0/325806259 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6554095a50 con 0x7f65641006e0 2023-09-10T01:02:12.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:11 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/700815006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:12.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:11 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1058200610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:12.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.199+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f656410c0e0 con 0x7f65641006e0 2023-09-10T01:02:12.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.199+0000 7f655b7fe700 1 -- 172.21.15.3:0/325806259 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6554099070 con 0x7f65641006e0 2023-09-10T01:02:12.200 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65500a0750 msgr2=0x7f65500a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65500a0750 0x7f65500a2c10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f654c00b580 tx=0x7f654c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641006e0 msgr2=0x7f65641963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641006e0 0x7f65641963a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f655400e970 tx=0x7f655400ec80 comp rx=0 tx=0).stop 2023-09-10T01:02:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 shutdown_connections 2023-09-10T01:02:12.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f65500a0750 0x7f65500a2c10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6564102270 0x7f656419d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65641006e0 0x7f65641963a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 --2- 172.21.15.3:0/325806259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65641018d0 0x7f65641968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 >> 172.21.15.3:0/325806259 conn(0x7f65640fbc50 msgr2=0x7f65640fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:12.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 shutdown_connections 2023-09-10T01:02:12.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.202+0000 7f6568fff700 1 -- 172.21.15.3:0/325806259 wait complete. 2023-09-10T01:02:12.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:02:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:11 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/700815006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:11 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1058200610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:12.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 -- 172.21.15.3:0/874504576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141006e0 msgr2=0x7fcb14100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 --2- 172.21.15.3:0/874504576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141006e0 0x7fcb14100b00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcb040099c0 tx=0x7fcb04009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 -- 172.21.15.3:0/874504576 shutdown_connections 2023-09-10T01:02:12.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 --2- 172.21.15.3:0/874504576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb14102270 0x7fcb14108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 --2- 172.21.15.3:0/874504576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141006e0 0x7fcb14100b00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 --2- 172.21.15.3:0/874504576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb141018d0 0x7fcb14101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 -- 172.21.15.3:0/874504576 >> 172.21.15.3:0/874504576 conn(0x7fcb140fbc50 msgr2=0x7fcb140fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:12.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 -- 172.21.15.3:0/874504576 shutdown_connections 2023-09-10T01:02:12.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.428+0000 7fcb19bff700 1 -- 172.21.15.3:0/874504576 wait complete. 2023-09-10T01:02:12.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.429+0000 7fcb19bff700 1 Processor -- start 2023-09-10T01:02:12.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb19bff700 1 -- start start 2023-09-10T01:02:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb19bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb141006e0 0x7fcb141985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb19bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141018d0 0x7fcb14198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb19bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb14102270 0x7fcb1419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb19bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb141a0150 con 0x7fcb141018d0 2023-09-10T01:02:12.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb19bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb141a0290 con 0x7fcb141006e0 2023-09-10T01:02:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb19bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb141a0420 con 0x7fcb14102270 2023-09-10T01:02:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb12ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141018d0 0x7fcb14198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb12ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141018d0 0x7fcb14198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54006/0 (socket says 172.21.15.3:54006) 2023-09-10T01:02:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb12ffd700 1 -- 172.21.15.3:0/2954677635 learned_addr learned my addr 172.21.15.3:0/2954677635 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb137fe700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb141006e0 0x7fcb141985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.430+0000 7fcb13fff700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb14102270 0x7fcb1419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb12ffd700 1 -- 172.21.15.3:0/2954677635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb14102270 msgr2=0x7fcb1419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb12ffd700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb14102270 0x7fcb1419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb12ffd700 1 -- 172.21.15.3:0/2954677635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb141006e0 msgr2=0x7fcb141985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb12ffd700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb141006e0 0x7fcb141985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb12ffd700 1 -- 172.21.15.3:0/2954677635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb040095f0 con 0x7fcb141018d0 2023-09-10T01:02:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb137fe700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb141006e0 0x7fcb141985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb12ffd700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141018d0 0x7fcb14198b10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800e9a0 tx=0x7fcb0800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb10ff9700 1 -- 172.21.15.3:0/2954677635 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08009790 con 0x7fcb141018d0 2023-09-10T01:02:12.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb10ff9700 1 -- 172.21.15.3:0/2954677635 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb08010430 con 0x7fcb141018d0 2023-09-10T01:02:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb141a0700 con 0x7fcb141018d0 2023-09-10T01:02:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb10ff9700 1 -- 172.21.15.3:0/2954677635 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08018680 con 0x7fcb141018d0 2023-09-10T01:02:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.431+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb141a0c10 con 0x7fcb141018d0 2023-09-10T01:02:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.432+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1410bf40 con 0x7fcb141018d0 2023-09-10T01:02:12.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.434+0000 7fcb10ff9700 1 -- 172.21.15.3:0/2954677635 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fcb080105a0 con 0x7fcb141018d0 2023-09-10T01:02:12.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.436+0000 7fcb10ff9700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcafc0a06d0 0x7fcafc0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.436+0000 7fcb10ff9700 1 -- 172.21.15.3:0/2954677635 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fcb08014070 con 0x7fcb141018d0 2023-09-10T01:02:12.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.436+0000 7fcb137fe700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcafc0a06d0 0x7fcafc0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.437+0000 7fcb137fe700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcafc0a06d0 0x7fcafc0a2b90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009990 tx=0x7fcb04005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:12.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.440+0000 7fcb10ff9700 1 -- 172.21.15.3:0/2954677635 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fcb08051e10 con 0x7fcb141018d0 2023-09-10T01:02:12.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:11 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/700815006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:12.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:11 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1058200610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:12.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.650+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb141a0e30 con 0x7fcb141018d0 2023-09-10T01:02:12.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.650+0000 7fcb10ff9700 1 -- 172.21.15.3:0/2954677635 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb08095070 con 0x7fcb141018d0 2023-09-10T01:02:12.652 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:02:12.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcafc0a06d0 msgr2=0x7fcafc0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcafc0a06d0 0x7fcafc0a2b90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009990 tx=0x7fcb04005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141018d0 msgr2=0x7fcb14198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141018d0 0x7fcb14198b10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800e9a0 tx=0x7fcb0800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 shutdown_connections 2023-09-10T01:02:12.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcafc0a06d0 0x7fcafc0a2b90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb14102270 0x7fcb1419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb141006e0 0x7fcb141985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 --2- 172.21.15.3:0/2954677635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb141018d0 0x7fcb14198b10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 >> 172.21.15.3:0/2954677635 conn(0x7fcb140fbc50 msgr2=0x7fcb140fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:12.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 shutdown_connections 2023-09-10T01:02:12.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.653+0000 7fcb19bff700 1 -- 172.21.15.3:0/2954677635 wait complete. 2023-09-10T01:02:12.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:02:12.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 -- 172.21.15.3:0/2607478703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe41006f0 msgr2=0x7fbfe4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/2607478703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe41006f0 0x7fbfe4100b10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40099c0 tx=0x7fbfd4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 -- 172.21.15.3:0/2607478703 shutdown_connections 2023-09-10T01:02:12.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/2607478703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe4102280 0x7fbfe4108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/2607478703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe41006f0 0x7fbfe4100b10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/2607478703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41018e0 0x7fbfe4101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 -- 172.21.15.3:0/2607478703 >> 172.21.15.3:0/2607478703 conn(0x7fbfe40fbc60 msgr2=0x7fbfe40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:12.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 -- 172.21.15.3:0/2607478703 shutdown_connections 2023-09-10T01:02:12.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.878+0000 7fbfe89ff700 1 -- 172.21.15.3:0/2607478703 wait complete. 2023-09-10T01:02:12.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.879+0000 7fbfe89ff700 1 Processor -- start 2023-09-10T01:02:12.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.879+0000 7fbfe89ff700 1 -- start start 2023-09-10T01:02:12.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe89ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41006f0 0x7fbfe40743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe89ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe41018e0 0x7fbfe40729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe89ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe4102280 0x7fbfe4072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe89ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe419a250 con 0x7fbfe4102280 2023-09-10T01:02:12.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe89ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4073470 con 0x7fbfe41006f0 2023-09-10T01:02:12.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe89ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4073630 con 0x7fbfe41018e0 2023-09-10T01:02:12.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41006f0 0x7fbfe40743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe4102280 0x7fbfe4072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41006f0 0x7fbfe40743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45190/0 (socket says 172.21.15.3:45190) 2023-09-10T01:02:12.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 -- 172.21.15.3:0/4120007174 learned_addr learned my addr 172.21.15.3:0/4120007174 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:12.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe1d9b700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe41018e0 0x7fbfe40729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 -- 172.21.15.3:0/4120007174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe41018e0 msgr2=0x7fbfe40729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe41018e0 0x7fbfe40729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 -- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe4102280 msgr2=0x7fbfe4072f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:12.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe4102280 0x7fbfe4072f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe259c700 1 -- 172.21.15.3:0/4120007174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd8009520 con 0x7fbfe41006f0 2023-09-10T01:02:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.880+0000 7fbfe2d9d700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe4102280 0x7fbfe4072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.881+0000 7fbfe259c700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41006f0 0x7fbfe40743b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009fd0 tx=0x7fbfd400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.881+0000 7fbfd37fe700 1 -- 172.21.15.3:0/4120007174 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd401d070 con 0x7fbfe41006f0 2023-09-10T01:02:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.881+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd40095f0 con 0x7fbfe41006f0 2023-09-10T01:02:12.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.881+0000 7fbfd37fe700 1 -- 172.21.15.3:0/4120007174 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbfd4005050 con 0x7fbfe41006f0 2023-09-10T01:02:12.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.881+0000 7fbfd37fe700 1 -- 172.21.15.3:0/4120007174 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4004a70 con 0x7fbfe41006f0 2023-09-10T01:02:12.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.881+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe41a4de0 con 0x7fbfe41006f0 2023-09-10T01:02:12.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.882+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4107620 con 0x7fbfe41006f0 2023-09-10T01:02:12.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.884+0000 7fbfd37fe700 1 -- 172.21.15.3:0/4120007174 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fbfd4005450 con 0x7fbfe41006f0 2023-09-10T01:02:12.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.886+0000 7fbfd37fe700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfcc0a06d0 0x7fbfcc0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:12.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.886+0000 7fbfd37fe700 1 -- 172.21.15.3:0/4120007174 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fbfd40cd2d0 con 0x7fbfe41006f0 2023-09-10T01:02:12.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.886+0000 7fbfe1d9b700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfcc0a06d0 0x7fbfcc0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:12.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.887+0000 7fbfe1d9b700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfcc0a06d0 0x7fbfcc0a2b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8009df0 tx=0x7fbfd8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:12.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:12.888+0000 7fbfd37fe700 1 -- 172.21.15.3:0/4120007174 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbfd4096920 con 0x7fbfe41006f0 2023-09-10T01:02:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.101+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbfe410c180 con 0x7fbfe41006f0 2023-09-10T01:02:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.102+0000 7fbfd37fe700 1 -- 172.21.15.3:0/4120007174 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbfd4099f40 con 0x7fbfe41006f0 2023-09-10T01:02:13.103 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:13.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.104+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfcc0a06d0 msgr2=0x7fbfcc0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.104+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfcc0a06d0 0x7fbfcc0a2b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8009df0 tx=0x7fbfd8009260 comp rx=0 tx=0).stop 2023-09-10T01:02:13.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.104+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41006f0 msgr2=0x7fbfe40743b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.104+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41006f0 0x7fbfe40743b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009fd0 tx=0x7fbfd400f690 comp rx=0 tx=0).stop 2023-09-10T01:02:13.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 shutdown_connections 2023-09-10T01:02:13.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfe4102280 0x7fbfe4072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfe41006f0 0x7fbfe40743b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfcc0a06d0 0x7fbfcc0a2b90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 --2- 172.21.15.3:0/4120007174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe41018e0 0x7fbfe40729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 >> 172.21.15.3:0/4120007174 conn(0x7fbfe40fbc60 msgr2=0x7fbfe40fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:13.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 shutdown_connections 2023-09-10T01:02:13.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.105+0000 7fbfe89ff700 1 -- 172.21.15.3:0/4120007174 wait complete. 2023-09-10T01:02:13.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:02:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:12 smithi003 ceph-mon[97654]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:12 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/325806259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:12 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2954677635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:13.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.328+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/655905924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 msgr2=0x7f8a98108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.328+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/655905924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a98108b10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000b0d0 tx=0x7f8a9000b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.329+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/655905924 shutdown_connections 2023-09-10T01:02:13.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.329+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/655905924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a981006d0 0x7f8a98100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.329+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/655905924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a98108b10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.329+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/655905924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a981018c0 0x7f8a98101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.329+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/655905924 >> 172.21.15.3:0/655905924 conn(0x7f8a980fbc60 msgr2=0x7f8a980fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:13.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.329+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/655905924 shutdown_connections 2023-09-10T01:02:13.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.329+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/655905924 wait complete. 2023-09-10T01:02:13.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 Processor -- start 2023-09-10T01:02:13.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 -- start start 2023-09-10T01:02:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a981006d0 0x7f8a98198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a981018c0 0x7f8a98198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a9819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981a0210 con 0x7f8a98102260 2023-09-10T01:02:13.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981a0350 con 0x7f8a981018c0 2023-09-10T01:02:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9e3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981a04e0 con 0x7f8a981006d0 2023-09-10T01:02:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.330+0000 7f8a9c99c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a9819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a9819fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54044/0 (socket says 172.21.15.3:54044) 2023-09-10T01:02:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 -- 172.21.15.3:0/1073782726 learned_addr learned my addr 172.21.15.3:0/1073782726 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:13.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a977fe700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a981018c0 0x7f8a98198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a97fff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a981006d0 0x7f8a98198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 -- 172.21.15.3:0/1073782726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a981006d0 msgr2=0x7f8a98198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a981006d0 0x7f8a98198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 -- 172.21.15.3:0/1073782726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a981018c0 msgr2=0x7f8a98198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a981018c0 0x7f8a98198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 -- 172.21.15.3:0/1073782726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a90009d00 con 0x7f8a98102260 2023-09-10T01:02:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.331+0000 7f8a9c99c700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a9819fc40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8a90008eb0 tx=0x7f8a90008ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.332+0000 7f8a957fa700 1 -- 172.21.15.3:0/1073782726 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9000e040 con 0x7f8a98102260 2023-09-10T01:02:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.332+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a981a0760 con 0x7f8a98102260 2023-09-10T01:02:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.332+0000 7f8a957fa700 1 -- 172.21.15.3:0/1073782726 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a9000be30 con 0x7f8a98102260 2023-09-10T01:02:13.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.332+0000 7f8a957fa700 1 -- 172.21.15.3:0/1073782726 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9001d470 con 0x7f8a98102260 2023-09-10T01:02:13.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.332+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a981a0be0 con 0x7f8a98102260 2023-09-10T01:02:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.333+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9810bf30 con 0x7f8a98102260 2023-09-10T01:02:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.335+0000 7f8a957fa700 1 -- 172.21.15.3:0/1073782726 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f8a90005190 con 0x7f8a98102260 2023-09-10T01:02:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.336+0000 7f8a957fa700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a800a06d0 0x7f8a800a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.336+0000 7f8a957fa700 1 -- 172.21.15.3:0/1073782726 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f8a900cd480 con 0x7f8a98102260 2023-09-10T01:02:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.336+0000 7f8a97fff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a800a06d0 0x7f8a800a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.337+0000 7f8a97fff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a800a06d0 0x7f8a800a2b90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8a98101720 tx=0x7f8a88009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.340+0000 7f8a957fa700 1 -- 172.21.15.3:0/1073782726 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f8a90096ad0 con 0x7f8a98102260 2023-09-10T01:02:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:12 smithi136 ceph-mon[133708]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:12 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/325806259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:13.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:12 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2954677635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:13.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:12 smithi092 ceph-mon[134067]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:13.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:12 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/325806259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:13.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:12 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2954677635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:13.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.598+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a981a0e00 con 0x7f8a98102260 2023-09-10T01:02:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.599+0000 7f8a957fa700 1 -- 172.21.15.3:0/1073782726 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f8a9009a0f0 con 0x7f8a98102260 2023-09-10T01:02:13.600 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:13.600 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:02:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.601+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a800a06d0 msgr2=0x7f8a800a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.601+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a800a06d0 0x7f8a800a2b90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8a98101720 tx=0x7f8a88009260 comp rx=0 tx=0).stop 2023-09-10T01:02:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.601+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 msgr2=0x7f8a9819fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.601+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a9819fc40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8a90008eb0 tx=0x7f8a90008ee0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.601+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 shutdown_connections 2023-09-10T01:02:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.602+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a800a06d0 0x7f8a800a2b90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.602+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a981006d0 0x7f8a98198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.602+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a98102260 0x7f8a9819fc40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.602+0000 7f8a9e3ff700 1 --2- 172.21.15.3:0/1073782726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a981018c0 0x7f8a98198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.602+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 >> 172.21.15.3:0/1073782726 conn(0x7f8a980fbc60 msgr2=0x7f8a98107160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:13.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.602+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 shutdown_connections 2023-09-10T01:02:13.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.602+0000 7f8a9e3ff700 1 -- 172.21.15.3:0/1073782726 wait complete. 2023-09-10T01:02:13.605 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:02:13.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:13.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 -- 172.21.15.3:0/2047070607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 msgr2=0x7fab94101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2047070607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 0x7fab94101d20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fab840099c0 tx=0x7fab84009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 -- 172.21.15.3:0/2047070607 shutdown_connections 2023-09-10T01:02:13.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2047070607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 0x7fab94100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2047070607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 0x7fab94108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2047070607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 0x7fab94101d20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 -- 172.21.15.3:0/2047070607 >> 172.21.15.3:0/2047070607 conn(0x7fab940fbc60 msgr2=0x7fab940fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:13.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.824+0000 7fab9bfff700 1 -- 172.21.15.3:0/2047070607 shutdown_connections 2023-09-10T01:02:13.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.825+0000 7fab9bfff700 1 -- 172.21.15.3:0/2047070607 wait complete. 2023-09-10T01:02:13.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.825+0000 7fab9bfff700 1 Processor -- start 2023-09-10T01:02:13.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9bfff700 1 -- start start 2023-09-10T01:02:13.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 0x7fab940743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 0x7fab940729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 0x7fab94072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab940736d0 con 0x7fab94102260 2023-09-10T01:02:13.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9bfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94073840 con 0x7fab941018c0 2023-09-10T01:02:13.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9bfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941a03e0 con 0x7fab941006d0 2023-09-10T01:02:13.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 0x7fab94072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab99d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 0x7fab940743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab99d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 0x7fab940743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:52054/0 (socket says 172.21.15.3:52054) 2023-09-10T01:02:13.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab99d9b700 1 -- 172.21.15.3:0/2200280208 learned_addr learned my addr 172.21.15.3:0/2200280208 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:13.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 0x7fab94072f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54060/0 (socket says 172.21.15.3:54060) 2023-09-10T01:02:13.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.826+0000 7fab9959a700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 0x7fab940729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab99d9b700 1 -- 172.21.15.3:0/2200280208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 msgr2=0x7fab940729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab99d9b700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 0x7fab940729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab99d9b700 1 -- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 msgr2=0x7fab94072f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:13.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab99d9b700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 0x7fab94072f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:13.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab99d9b700 1 -- 172.21.15.3:0/2200280208 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab840095f0 con 0x7fab941006d0 2023-09-10T01:02:13.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab9a59c700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 0x7fab94072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:13.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab99d9b700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 0x7fab940743b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e970 tx=0x7fab9000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:13.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab8affd700 1 -- 172.21.15.3:0/2200280208 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90009790 con 0x7fab941006d0 2023-09-10T01:02:13.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab941a06c0 con 0x7fab941006d0 2023-09-10T01:02:13.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab8affd700 1 -- 172.21.15.3:0/2200280208 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab9000c7e0 con 0x7fab941006d0 2023-09-10T01:02:13.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab8affd700 1 -- 172.21.15.3:0/2200280208 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90003c70 con 0x7fab941006d0 2023-09-10T01:02:13.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.827+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab941a0bd0 con 0x7fab941006d0 2023-09-10T01:02:13.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.828+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab94107600 con 0x7fab941006d0 2023-09-10T01:02:13.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.831+0000 7fab8affd700 1 -- 172.21.15.3:0/2200280208 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fab9000cc50 con 0x7fab941006d0 2023-09-10T01:02:13.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.832+0000 7fab8affd700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fab800a06d0 0x7fab800a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:13.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.832+0000 7fab8affd700 1 -- 172.21.15.3:0/2200280208 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fab90014070 con 0x7fab941006d0 2023-09-10T01:02:13.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.832+0000 7fab9959a700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fab800a06d0 0x7fab800a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:13.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.833+0000 7fab9959a700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fab800a06d0 0x7fab800a2b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fab8400b580 tx=0x7fab84005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:13.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:13.834+0000 7fab8affd700 1 -- 172.21.15.3:0/2200280208 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fab90095a40 con 0x7fab941006d0 2023-09-10T01:02:14.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.049+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab9410c060 con 0x7fab941006d0 2023-09-10T01:02:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.051+0000 7fab8affd700 1 -- 172.21.15.3:0/2200280208 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7fab90003dd0 con 0x7fab941006d0 2023-09-10T01:02:14.052 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:14.053 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fab800a06d0 msgr2=0x7fab800a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fab800a06d0 0x7fab800a2b90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fab8400b580 tx=0x7fab84005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 msgr2=0x7fab940743b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 0x7fab940743b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fab9000e970 tx=0x7fab9000ec80 comp rx=0 tx=0).stop 2023-09-10T01:02:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 shutdown_connections 2023-09-10T01:02:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fab800a06d0 0x7fab800a2b90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab941006d0 0x7fab940743b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94102260 0x7fab94072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 --2- 172.21.15.3:0/2200280208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab941018c0 0x7fab940729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 >> 172.21.15.3:0/2200280208 conn(0x7fab940fbc60 msgr2=0x7fab940fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:14.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 shutdown_connections 2023-09-10T01:02:14.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.054+0000 7fab9bfff700 1 -- 172.21.15.3:0/2200280208 wait complete. 2023-09-10T01:02:14.074 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:02:14.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:02:14.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 -- 172.21.15.3:0/3632042890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c1018e0 msgr2=0x7f159c101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 --2- 172.21.15.3:0/3632042890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c1018e0 0x7f159c101d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1588009920 tx=0x7f1588009c30 comp rx=0 tx=0).stop 2023-09-10T01:02:14.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 -- 172.21.15.3:0/3632042890 shutdown_connections 2023-09-10T01:02:14.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 --2- 172.21.15.3:0/3632042890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f159c1006f0 0x7f159c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 --2- 172.21.15.3:0/3632042890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c102280 0x7f159c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 --2- 172.21.15.3:0/3632042890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c1018e0 0x7f159c101d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 -- 172.21.15.3:0/3632042890 >> 172.21.15.3:0/3632042890 conn(0x7f159c0fbc60 msgr2=0x7f159c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:14.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 -- 172.21.15.3:0/3632042890 shutdown_connections 2023-09-10T01:02:14.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.280+0000 7f15a31ff700 1 -- 172.21.15.3:0/3632042890 wait complete. 2023-09-10T01:02:14.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.281+0000 7f15a31ff700 1 Processor -- start 2023-09-10T01:02:14.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.281+0000 7f15a31ff700 1 -- start start 2023-09-10T01:02:14.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a31ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f159c1006f0 0x7f159c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a31ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c1018e0 0x7f159c198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a31ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c102280 0x7f159c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a31ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c1a0180 con 0x7f159c102280 2023-09-10T01:02:14.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a31ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c1a02c0 con 0x7f159c1018e0 2023-09-10T01:02:14.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a31ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c1a0450 con 0x7f159c1006f0 2023-09-10T01:02:14.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f159bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c1018e0 0x7f159c198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a179c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c102280 0x7f159c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f159bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c1018e0 0x7f159c198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45232/0 (socket says 172.21.15.3:45232) 2023-09-10T01:02:14.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f159bfff700 1 -- 172.21.15.3:0/871176830 learned_addr learned my addr 172.21.15.3:0/871176830 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:14.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a0f9b700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f159c1006f0 0x7f159c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a179c700 1 -- 172.21.15.3:0/871176830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f159c1006f0 msgr2=0x7f159c198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a179c700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f159c1006f0 0x7f159c198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a179c700 1 -- 172.21.15.3:0/871176830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c1018e0 msgr2=0x7f159c198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a179c700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c1018e0 0x7f159c198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f15a179c700 1 -- 172.21.15.3:0/871176830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1588009520 con 0x7f159c102280 2023-09-10T01:02:14.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.282+0000 7f159bfff700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c1018e0 0x7f159c198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:14.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.283+0000 7f15a179c700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c102280 0x7f159c19fbb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f159400efc0 tx=0x7f159400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:14.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.283+0000 7f1599ffb700 1 -- 172.21.15.3:0/871176830 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594018070 con 0x7f159c102280 2023-09-10T01:02:14.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.283+0000 7f1599ffb700 1 -- 172.21.15.3:0/871176830 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f159400f040 con 0x7f159c102280 2023-09-10T01:02:14.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.283+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c1a06d0 con 0x7f159c102280 2023-09-10T01:02:14.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.283+0000 7f1599ffb700 1 -- 172.21.15.3:0/871176830 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159400ccb0 con 0x7f159c102280 2023-09-10T01:02:14.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.283+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c1a0c30 con 0x7f159c102280 2023-09-10T01:02:14.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.284+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159c192d00 con 0x7f159c102280 2023-09-10T01:02:14.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.286+0000 7f1599ffb700 1 -- 172.21.15.3:0/871176830 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f1594003a00 con 0x7f159c102280 2023-09-10T01:02:14.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.287+0000 7f1599ffb700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f158c0a0750 0x7f158c0a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.287+0000 7f1599ffb700 1 -- 172.21.15.3:0/871176830 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f15940cbb10 con 0x7f159c102280 2023-09-10T01:02:14.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.287+0000 7f15a0f9b700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f158c0a0750 0x7f158c0a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.288+0000 7f15a0f9b700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f158c0a0750 0x7f158c0a2c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f159c101740 tx=0x7f1590009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:14.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.291+0000 7f1599ffb700 1 -- 172.21.15.3:0/871176830 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1594095160 con 0x7f159c102280 2023-09-10T01:02:14.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:13 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4120007174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:14.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:13 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1073782726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:14.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:13 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4120007174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:14.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:13 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1073782726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:14.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:13 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4120007174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:14.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:13 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1073782726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:14.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.499+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f159c02cdf0 con 0x7f159c102280 2023-09-10T01:02:14.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.499+0000 7f1599ffb700 1 -- 172.21.15.3:0/871176830 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1594098780 con 0x7f159c102280 2023-09-10T01:02:14.501 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:02:14.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f158c0a0750 msgr2=0x7f158c0a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f158c0a0750 0x7f158c0a2c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f159c101740 tx=0x7f1590009260 comp rx=0 tx=0).stop 2023-09-10T01:02:14.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c102280 msgr2=0x7f159c19fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c102280 0x7f159c19fbb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f159400efc0 tx=0x7f159400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 shutdown_connections 2023-09-10T01:02:14.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f158c0a0750 0x7f158c0a2c10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f159c1006f0 0x7f159c198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f159c102280 0x7f159c19fbb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 --2- 172.21.15.3:0/871176830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f159c1018e0 0x7f159c198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 >> 172.21.15.3:0/871176830 conn(0x7f159c0fbc60 msgr2=0x7f159c107270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:14.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 shutdown_connections 2023-09-10T01:02:14.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.502+0000 7f15a31ff700 1 -- 172.21.15.3:0/871176830 wait complete. 2023-09-10T01:02:14.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:02:14.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 -- 172.21.15.3:0/3649299067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581018e0 msgr2=0x7f1e58101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/3649299067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581018e0 0x7f1e58101d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009920 tx=0x7f1e48009c30 comp rx=0 tx=0).stop 2023-09-10T01:02:14.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 -- 172.21.15.3:0/3649299067 shutdown_connections 2023-09-10T01:02:14.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/3649299067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e581006f0 0x7f1e58100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/3649299067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e58102280 0x7f1e58108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/3649299067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581018e0 0x7f1e58101d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 -- 172.21.15.3:0/3649299067 >> 172.21.15.3:0/3649299067 conn(0x7f1e580fbc60 msgr2=0x7f1e580fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:14.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 -- 172.21.15.3:0/3649299067 shutdown_connections 2023-09-10T01:02:14.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.728+0000 7f1e5fdff700 1 -- 172.21.15.3:0/3649299067 wait complete. 2023-09-10T01:02:14.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.729+0000 7f1e5fdff700 1 Processor -- start 2023-09-10T01:02:14.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.729+0000 7f1e5fdff700 1 -- start start 2023-09-10T01:02:14.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5fdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581006f0 0x7f1e58198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5fdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e581018e0 0x7f1e58198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5fdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e58102280 0x7f1e5819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5fdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e581a01d0 con 0x7f1e58102280 2023-09-10T01:02:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5fdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e581a0310 con 0x7f1e581006f0 2023-09-10T01:02:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5fdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e581a04a0 con 0x7f1e581018e0 2023-09-10T01:02:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581006f0 0x7f1e58198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581006f0 0x7f1e58198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40626/0 (socket says 172.21.15.3:40626) 2023-09-10T01:02:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 -- 172.21.15.3:0/37219961 learned_addr learned my addr 172.21.15.3:0/37219961 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5e39c700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e58102280 0x7f1e5819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5d39a700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e581018e0 0x7f1e58198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 -- 172.21.15.3:0/37219961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e581018e0 msgr2=0x7f1e58198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e581018e0 0x7f1e58198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 -- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e58102280 msgr2=0x7f1e5819fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e58102280 0x7f1e5819fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.730+0000 7f1e5db9b700 1 -- 172.21.15.3:0/37219961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48009520 con 0x7f1e581006f0 2023-09-10T01:02:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.731+0000 7f1e5db9b700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581006f0 0x7f1e58198650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400e970 tx=0x7f1e5400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.731+0000 7f1e4effd700 1 -- 172.21.15.3:0/37219961 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54009790 con 0x7f1e581006f0 2023-09-10T01:02:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.731+0000 7f1e4effd700 1 -- 172.21.15.3:0/37219961 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e5400c7e0 con 0x7f1e581006f0 2023-09-10T01:02:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.731+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e581a0780 con 0x7f1e581006f0 2023-09-10T01:02:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.731+0000 7f1e4effd700 1 -- 172.21.15.3:0/37219961 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54003c70 con 0x7f1e581006f0 2023-09-10T01:02:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.731+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e581a0c90 con 0x7f1e581006f0 2023-09-10T01:02:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.732+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5810bf60 con 0x7f1e581006f0 2023-09-10T01:02:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.734+0000 7f1e4effd700 1 -- 172.21.15.3:0/37219961 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f1e5400cc50 con 0x7f1e581006f0 2023-09-10T01:02:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.735+0000 7f1e4effd700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1e440a06d0 0x7f1e440a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.736+0000 7f1e4effd700 1 -- 172.21.15.3:0/37219961 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f1e54014070 con 0x7f1e581006f0 2023-09-10T01:02:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.736+0000 7f1e5d39a700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1e440a06d0 0x7f1e440a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.736+0000 7f1e5d39a700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1e440a06d0 0x7f1e440a2b90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800fa40 tx=0x7f1e48019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.738+0000 7f1e4effd700 1 -- 172.21.15.3:0/37219961 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1e54095ba0 con 0x7f1e581006f0 2023-09-10T01:02:14.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.948+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e581a0f10 con 0x7f1e581006f0 2023-09-10T01:02:14.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.949+0000 7f1e4effd700 1 -- 172.21.15.3:0/37219961 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e54003dd0 con 0x7f1e581006f0 2023-09-10T01:02:14.950 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:14.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1e440a06d0 msgr2=0x7f1e440a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1e440a06d0 0x7f1e440a2b90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800fa40 tx=0x7f1e48019040 comp rx=0 tx=0).stop 2023-09-10T01:02:14.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581006f0 msgr2=0x7f1e58198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:14.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581006f0 0x7f1e58198650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400e970 tx=0x7f1e5400ec80 comp rx=0 tx=0).stop 2023-09-10T01:02:14.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 shutdown_connections 2023-09-10T01:02:14.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e58102280 0x7f1e5819fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e581006f0 0x7f1e58198650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1e440a06d0 0x7f1e440a2b90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 --2- 172.21.15.3:0/37219961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e581018e0 0x7f1e58198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:14.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.951+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 >> 172.21.15.3:0/37219961 conn(0x7f1e580fbc60 msgr2=0x7f1e580fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:14.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.952+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 shutdown_connections 2023-09-10T01:02:14.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:14.952+0000 7f1e5fdff700 1 -- 172.21.15.3:0/37219961 wait complete. 2023-09-10T01:02:14.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:02:15.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 -- 172.21.15.3:0/1477101333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1481006e0 msgr2=0x7fb148100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 --2- 172.21.15.3:0/1477101333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1481006e0 0x7fb148100b00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb1440099c0 tx=0x7fb144009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 -- 172.21.15.3:0/1477101333 shutdown_connections 2023-09-10T01:02:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 --2- 172.21.15.3:0/1477101333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb148102270 0x7fb148108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 --2- 172.21.15.3:0/1477101333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1481006e0 0x7fb148100b00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 --2- 172.21.15.3:0/1477101333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 0x7fb148101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 -- 172.21.15.3:0/1477101333 >> 172.21.15.3:0/1477101333 conn(0x7fb1480fbc50 msgr2=0x7fb1480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 -- 172.21.15.3:0/1477101333 shutdown_connections 2023-09-10T01:02:15.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.178+0000 7fb14fdff700 1 -- 172.21.15.3:0/1477101333 wait complete. 2023-09-10T01:02:15.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 Processor -- start 2023-09-10T01:02:15.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 -- start start 2023-09-10T01:02:15.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1481006e0 0x7fb1481985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 0x7fb148198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb148102270 0x7fb14819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1481a0170 con 0x7fb148102270 2023-09-10T01:02:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1481a02b0 con 0x7fb1481018d0 2023-09-10T01:02:15.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.179+0000 7fb14fdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1481a0440 con 0x7fb1481006e0 2023-09-10T01:02:15.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14d39a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 0x7fb148198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14e39c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb148102270 0x7fb14819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14d39a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 0x7fb148198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40646/0 (socket says 172.21.15.3:40646) 2023-09-10T01:02:15.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14d39a700 1 -- 172.21.15.3:0/2365456045 learned_addr learned my addr 172.21.15.3:0/2365456045 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:15.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14db9b700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1481006e0 0x7fb1481985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14e39c700 1 -- 172.21.15.3:0/2365456045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1481006e0 msgr2=0x7fb1481985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14e39c700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1481006e0 0x7fb1481985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14e39c700 1 -- 172.21.15.3:0/2365456045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 msgr2=0x7fb148198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14e39c700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 0x7fb148198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14e39c700 1 -- 172.21.15.3:0/2365456045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440095f0 con 0x7fb148102270 2023-09-10T01:02:15.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14d39a700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 0x7fb148198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:15.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.180+0000 7fb14e39c700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb148102270 0x7fb14819fba0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb14000efc0 tx=0x7fb14000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:15.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.181+0000 7fb13effd700 1 -- 172.21.15.3:0/2365456045 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140018070 con 0x7fb148102270 2023-09-10T01:02:15.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.181+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1481a06c0 con 0x7fb148102270 2023-09-10T01:02:15.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.181+0000 7fb13effd700 1 -- 172.21.15.3:0/2365456045 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb14000f040 con 0x7fb148102270 2023-09-10T01:02:15.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.181+0000 7fb13effd700 1 -- 172.21.15.3:0/2365456045 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14000cd40 con 0x7fb148102270 2023-09-10T01:02:15.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.181+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1481a0c20 con 0x7fb148102270 2023-09-10T01:02:15.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.182+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb148192d40 con 0x7fb148102270 2023-09-10T01:02:15.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.184+0000 7fb13effd700 1 -- 172.21.15.3:0/2365456045 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fb140003a00 con 0x7fb148102270 2023-09-10T01:02:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.185+0000 7fb13effd700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1340a0750 0x7fb1340a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.185+0000 7fb13effd700 1 -- 172.21.15.3:0/2365456045 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fb1400cbc30 con 0x7fb148102270 2023-09-10T01:02:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.185+0000 7fb14db9b700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1340a0750 0x7fb1340a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.186+0000 7fb14db9b700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1340a0750 0x7fb1340a2c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb144009990 tx=0x7fb144005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:15.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.189+0000 7fb13effd700 1 -- 172.21.15.3:0/2365456045 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb140095280 con 0x7fb148102270 2023-09-10T01:02:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:14 smithi003 ceph-mon[97654]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:14 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2200280208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:14 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/871176830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:15.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:14 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/37219961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:15.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.398+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb14802cdf0 con 0x7fb148102270 2023-09-10T01:02:15.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.398+0000 7fb13effd700 1 -- 172.21.15.3:0/2365456045 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb14001b320 con 0x7fb148102270 2023-09-10T01:02:15.400 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:02:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1340a0750 msgr2=0x7fb1340a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1340a0750 0x7fb1340a2c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb144009990 tx=0x7fb144005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb148102270 msgr2=0x7fb14819fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb148102270 0x7fb14819fba0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb14000efc0 tx=0x7fb14000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 shutdown_connections 2023-09-10T01:02:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1340a0750 0x7fb1340a2c10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1481006e0 0x7fb1481985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb148102270 0x7fb14819fba0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 --2- 172.21.15.3:0/2365456045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1481018d0 0x7fb148198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 >> 172.21.15.3:0/2365456045 conn(0x7fb1480fbc50 msgr2=0x7fb1481071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:15.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 shutdown_connections 2023-09-10T01:02:15.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.401+0000 7fb14fdff700 1 -- 172.21.15.3:0/2365456045 wait complete. 2023-09-10T01:02:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:14 smithi136 ceph-mon[133708]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:14 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2200280208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:15.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:14 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/871176830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:15.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:14 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/37219961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:15.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:02:15.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:15 smithi092 ceph-mon[134067]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:15.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:15 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2200280208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:15.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:15 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/871176830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:15.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:15 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/37219961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:15.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 -- 172.21.15.3:0/557235952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881006f0 msgr2=0x7f9088100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 --2- 172.21.15.3:0/557235952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881006f0 0x7f9088100b10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f90840099c0 tx=0x7f9084009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 -- 172.21.15.3:0/557235952 shutdown_connections 2023-09-10T01:02:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 --2- 172.21.15.3:0/557235952 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9088102280 0x7f9088108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 --2- 172.21.15.3:0/557235952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881006f0 0x7f9088100b10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 --2- 172.21.15.3:0/557235952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90881018e0 0x7f9088101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 -- 172.21.15.3:0/557235952 >> 172.21.15.3:0/557235952 conn(0x7f90880fbc60 msgr2=0x7f90880fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:15.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 -- 172.21.15.3:0/557235952 shutdown_connections 2023-09-10T01:02:15.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.627+0000 7f908f3ff700 1 -- 172.21.15.3:0/557235952 wait complete. 2023-09-10T01:02:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 Processor -- start 2023-09-10T01:02:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 -- start start 2023-09-10T01:02:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90881006f0 0x7f9088198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881018e0 0x7f9088198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9088102280 0x7f908819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90881a01e0 con 0x7f90881006f0 2023-09-10T01:02:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90881a0320 con 0x7f90881018e0 2023-09-10T01:02:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.628+0000 7f908f3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90881a04b0 con 0x7f9088102280 2023-09-10T01:02:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908d19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90881006f0 0x7f9088198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881018e0 0x7f9088198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881018e0 0x7f9088198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40676/0 (socket says 172.21.15.3:40676) 2023-09-10T01:02:15.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908d19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90881006f0 0x7f9088198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34460/0 (socket says 172.21.15.3:34460) 2023-09-10T01:02:15.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 -- 172.21.15.3:0/583233923 learned_addr learned my addr 172.21.15.3:0/583233923 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:15.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908d99c700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9088102280 0x7f908819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 -- 172.21.15.3:0/583233923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9088102280 msgr2=0x7f908819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9088102280 0x7f908819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 -- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90881006f0 msgr2=0x7f9088198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90881006f0 0x7f9088198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 -- 172.21.15.3:0/583233923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9078009520 con 0x7f90881018e0 2023-09-10T01:02:15.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.629+0000 7f908c99a700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881018e0 0x7f9088198ba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f907800eb30 tx=0x7f907800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:15.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.630+0000 7f907e7fc700 1 -- 172.21.15.3:0/583233923 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078009880 con 0x7f90881018e0 2023-09-10T01:02:15.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.630+0000 7f907e7fc700 1 -- 172.21.15.3:0/583233923 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90780044e0 con 0x7f90881018e0 2023-09-10T01:02:15.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.630+0000 7f907e7fc700 1 -- 172.21.15.3:0/583233923 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078003e90 con 0x7f90881018e0 2023-09-10T01:02:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.630+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90840095f0 con 0x7f90881018e0 2023-09-10T01:02:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.630+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90881a0ac0 con 0x7f90881018e0 2023-09-10T01:02:15.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.631+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9088192d40 con 0x7f90881018e0 2023-09-10T01:02:15.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.633+0000 7f907e7fc700 1 -- 172.21.15.3:0/583233923 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f9078004000 con 0x7f90881018e0 2023-09-10T01:02:15.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.634+0000 7f907e7fc700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90740a0750 0x7f90740a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:15.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.634+0000 7f907e7fc700 1 -- 172.21.15.3:0/583233923 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f9078014070 con 0x7f90881018e0 2023-09-10T01:02:15.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.635+0000 7f908d19b700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90740a0750 0x7f90740a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:15.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.635+0000 7f908d19b700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90740a0750 0x7f90740a2c10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9084009990 tx=0x7f9084005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:15.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.636+0000 7f907e7fc700 1 -- 172.21.15.3:0/583233923 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9078095c40 con 0x7f90881018e0 2023-09-10T01:02:15.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.846+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f908810c0e0 con 0x7f90881018e0 2023-09-10T01:02:15.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.847+0000 7f907e7fc700 1 -- 172.21.15.3:0/583233923 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9078099260 con 0x7f90881018e0 2023-09-10T01:02:15.848 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:15.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90740a0750 msgr2=0x7f90740a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90740a0750 0x7f90740a2c10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9084009990 tx=0x7f9084005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881018e0 msgr2=0x7f9088198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881018e0 0x7f9088198ba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f907800eb30 tx=0x7f907800ee40 comp rx=0 tx=0).stop 2023-09-10T01:02:15.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 shutdown_connections 2023-09-10T01:02:15.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90740a0750 0x7f90740a2c10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9088102280 0x7f908819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90881006f0 0x7f9088198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 --2- 172.21.15.3:0/583233923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90881018e0 0x7f9088198ba0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.849+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 >> 172.21.15.3:0/583233923 conn(0x7f90880fbc60 msgr2=0x7f90881071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.850+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 shutdown_connections 2023-09-10T01:02:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:15.850+0000 7f908f3ff700 1 -- 172.21.15.3:0/583233923 wait complete. 2023-09-10T01:02:15.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:02:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 -- 172.21.15.3:0/3480819963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 msgr2=0x7f7e8c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 --2- 172.21.15.3:0/3480819963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c101d20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7e800099c0 tx=0x7f7e80009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 -- 172.21.15.3:0/3480819963 shutdown_connections 2023-09-10T01:02:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 --2- 172.21.15.3:0/3480819963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e8c102260 0x7f7e8c108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 --2- 172.21.15.3:0/3480819963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e8c1006d0 0x7f7e8c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 --2- 172.21.15.3:0/3480819963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c101d20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 -- 172.21.15.3:0/3480819963 >> 172.21.15.3:0/3480819963 conn(0x7f7e8c0fbc60 msgr2=0x7f7e8c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 -- 172.21.15.3:0/3480819963 shutdown_connections 2023-09-10T01:02:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.078+0000 7f7e909ff700 1 -- 172.21.15.3:0/3480819963 wait complete. 2023-09-10T01:02:16.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 Processor -- start 2023-09-10T01:02:16.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 -- start start 2023-09-10T01:02:16.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e8c1006d0 0x7f7e8c196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c1968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e8c102260 0x7f7e8c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c19dee0 con 0x7f7e8c1018c0 2023-09-10T01:02:16.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c19e020 con 0x7f7e8c102260 2023-09-10T01:02:16.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.079+0000 7f7e909ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c19e1b0 con 0x7f7e8c1006d0 2023-09-10T01:02:16.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c1968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c1968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34486/0 (socket says 172.21.15.3:34486) 2023-09-10T01:02:16.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 -- 172.21.15.3:0/2864442739 learned_addr learned my addr 172.21.15.3:0/2864442739 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:16.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e8a59c700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e8c1006d0 0x7f7e8c196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e8ad9d700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e8c102260 0x7f7e8c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 -- 172.21.15.3:0/2864442739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e8c1006d0 msgr2=0x7f7e8c196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e8c1006d0 0x7f7e8c196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 -- 172.21.15.3:0/2864442739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e8c102260 msgr2=0x7f7e8c19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e8c102260 0x7f7e8c19d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 -- 172.21.15.3:0/2864442739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e800095f0 con 0x7f7e8c1018c0 2023-09-10T01:02:16.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e89d9b700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c1968a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009fd0 tx=0x7f7e8000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:16.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/2864442739 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8001d070 con 0x7f7e8c1018c0 2023-09-10T01:02:16.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/2864442739 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e80005030 con 0x7f7e8c1018c0 2023-09-10T01:02:16.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/2864442739 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80004b30 con 0x7f7e8c1018c0 2023-09-10T01:02:16.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.080+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c19e430 con 0x7f7e8c1018c0 2023-09-10T01:02:16.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.081+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c19e800 con 0x7f7e8c1018c0 2023-09-10T01:02:16.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.082+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c107600 con 0x7f7e8c1018c0 2023-09-10T01:02:16.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.083+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/2864442739 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f7e80004cd0 con 0x7f7e8c1018c0 2023-09-10T01:02:16.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.086+0000 7f7e7b7fe700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e740a0750 0x7f7e740a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.086+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/2864442739 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f7e800ccf50 con 0x7f7e8c1018c0 2023-09-10T01:02:16.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.086+0000 7f7e8a59c700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e740a0750 0x7f7e740a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.086+0000 7f7e8a59c700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e740a0750 0x7f7e740a2c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c101720 tx=0x7f7e7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:16.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.090+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/2864442739 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7e800965a0 con 0x7f7e8c1018c0 2023-09-10T01:02:16.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:16 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2365456045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:16.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:16 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/583233923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:16.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.345+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e8c10bee0 con 0x7f7e8c1018c0 2023-09-10T01:02:16.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.346+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/2864442739 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f7e80099bc0 con 0x7f7e8c1018c0 2023-09-10T01:02:16.347 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:16.347 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:02:16.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e740a0750 msgr2=0x7f7e740a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e740a0750 0x7f7e740a2c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c101720 tx=0x7f7e7c009260 comp rx=0 tx=0).stop 2023-09-10T01:02:16.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 msgr2=0x7f7e8c1968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c1968a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009fd0 tx=0x7f7e8000f690 comp rx=0 tx=0).stop 2023-09-10T01:02:16.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 shutdown_connections 2023-09-10T01:02:16.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e740a0750 0x7f7e740a2c10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e8c1006d0 0x7f7e8c196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e8c102260 0x7f7e8c19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 --2- 172.21.15.3:0/2864442739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c1018c0 0x7f7e8c1968a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.348+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 >> 172.21.15.3:0/2864442739 conn(0x7f7e8c0fbc60 msgr2=0x7f7e8c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:16.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.349+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 shutdown_connections 2023-09-10T01:02:16.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.349+0000 7f7e909ff700 1 -- 172.21.15.3:0/2864442739 wait complete. 2023-09-10T01:02:16.352 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:02:16.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:16.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:16 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2365456045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:16.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:16 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/583233923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:16.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:16 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2365456045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:16.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:16 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/583233923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:16.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 -- 172.21.15.3:0/436358199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1018b0 msgr2=0x7fc09c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/436358199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1018b0 0x7fc09c101d10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc0900099c0 tx=0x7fc090009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 -- 172.21.15.3:0/436358199 shutdown_connections 2023-09-10T01:02:16.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/436358199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc09c1006c0 0x7fc09c100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/436358199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc09c102250 0x7fc09c108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/436358199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1018b0 0x7fc09c101d10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 -- 172.21.15.3:0/436358199 >> 172.21.15.3:0/436358199 conn(0x7fc09c0fbc30 msgr2=0x7fc09c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:16.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 -- 172.21.15.3:0/436358199 shutdown_connections 2023-09-10T01:02:16.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.572+0000 7fc0a27ff700 1 -- 172.21.15.3:0/436358199 wait complete. 2023-09-10T01:02:16.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.573+0000 7fc0a27ff700 1 Processor -- start 2023-09-10T01:02:16.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.573+0000 7fc0a27ff700 1 -- start start 2023-09-10T01:02:16.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.573+0000 7fc0a27ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1006c0 0x7fc09c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a27ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc09c1018b0 0x7fc09c198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a27ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc09c102250 0x7fc09c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a27ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c1a0150 con 0x7fc09c102250 2023-09-10T01:02:16.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a27ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c1a0290 con 0x7fc09c1006c0 2023-09-10T01:02:16.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a27ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c1a0420 con 0x7fc09c1018b0 2023-09-10T01:02:16.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc09bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1006c0 0x7fc09c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a0d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc09c102250 0x7fc09c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc09bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1006c0 0x7fc09c1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40716/0 (socket says 172.21.15.3:40716) 2023-09-10T01:02:16.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc09bfff700 1 -- 172.21.15.3:0/3752494734 learned_addr learned my addr 172.21.15.3:0/3752494734 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:16.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc09b7fe700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc09c1018b0 0x7fc09c198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a0d9c700 1 -- 172.21.15.3:0/3752494734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc09c1018b0 msgr2=0x7fc09c198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a0d9c700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc09c1018b0 0x7fc09c198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a0d9c700 1 -- 172.21.15.3:0/3752494734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1006c0 msgr2=0x7fc09c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a0d9c700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1006c0 0x7fc09c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc0a0d9c700 1 -- 172.21.15.3:0/3752494734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c009520 con 0x7fc09c102250 2023-09-10T01:02:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc09b7fe700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc09c1018b0 0x7fc09c198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.574+0000 7fc09bfff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1006c0 0x7fc09c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.575+0000 7fc0a0d9c700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc09c102250 0x7fc09c19fb80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc09400efc0 tx=0x7fc09400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.575+0000 7fc0997fa700 1 -- 172.21.15.3:0/3752494734 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094018070 con 0x7fc09c102250 2023-09-10T01:02:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.575+0000 7fc0997fa700 1 -- 172.21.15.3:0/3752494734 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc09400f040 con 0x7fc09c102250 2023-09-10T01:02:16.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.575+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0900095f0 con 0x7fc09c102250 2023-09-10T01:02:16.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.575+0000 7fc0997fa700 1 -- 172.21.15.3:0/3752494734 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09400cd40 con 0x7fc09c102250 2023-09-10T01:02:16.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.575+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c1a0a20 con 0x7fc09c102250 2023-09-10T01:02:16.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.576+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c10bf20 con 0x7fc09c102250 2023-09-10T01:02:16.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.578+0000 7fc0997fa700 1 -- 172.21.15.3:0/3752494734 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fc094003a00 con 0x7fc09c102250 2023-09-10T01:02:16.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.579+0000 7fc0997fa700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fc0840a06d0 0x7fc0840a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:16.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.580+0000 7fc0997fa700 1 -- 172.21.15.3:0/3752494734 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fc0940cbca0 con 0x7fc09c102250 2023-09-10T01:02:16.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.580+0000 7fc09bfff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fc0840a06d0 0x7fc0840a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:16.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.580+0000 7fc09bfff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fc0840a06d0 0x7fc0840a2b90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009fd0 tx=0x7fc08c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:16.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.584+0000 7fc0997fa700 1 -- 172.21.15.3:0/3752494734 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fc0940952f0 con 0x7fc09c102250 2023-09-10T01:02:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.795+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc09c10c130 con 0x7fc09c102250 2023-09-10T01:02:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.797+0000 7fc0997fa700 1 -- 172.21.15.3:0/3752494734 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7fc0940b8020 con 0x7fc09c102250 2023-09-10T01:02:16.798 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:16.799 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:16.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.799+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fc0840a06d0 msgr2=0x7fc0840a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.799+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fc0840a06d0 0x7fc0840a2b90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009fd0 tx=0x7fc08c009260 comp rx=0 tx=0).stop 2023-09-10T01:02:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.799+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc09c102250 msgr2=0x7fc09c19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.799+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc09c102250 0x7fc09c19fb80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc09400efc0 tx=0x7fc09400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 shutdown_connections 2023-09-10T01:02:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc09c102250 0x7fc09c19fb80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc09c1006c0 0x7fc09c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fc0840a06d0 0x7fc0840a2b90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 --2- 172.21.15.3:0/3752494734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc09c1018b0 0x7fc09c198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:16.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 >> 172.21.15.3:0/3752494734 conn(0x7fc09c0fbc30 msgr2=0x7fc09c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:16.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 shutdown_connections 2023-09-10T01:02:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:16.800+0000 7fc0a27ff700 1 -- 172.21.15.3:0/3752494734 wait complete. 2023-09-10T01:02:16.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:02:17.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 -- 172.21.15.3:0/4134949096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c102260 msgr2=0x7f858c108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 --2- 172.21.15.3:0/4134949096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c102260 0x7f858c108b10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f858400b0d0 tx=0x7f858400b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 -- 172.21.15.3:0/4134949096 shutdown_connections 2023-09-10T01:02:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 --2- 172.21.15.3:0/4134949096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c1006d0 0x7f858c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 --2- 172.21.15.3:0/4134949096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c102260 0x7f858c108b10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 --2- 172.21.15.3:0/4134949096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f858c1018c0 0x7f858c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 -- 172.21.15.3:0/4134949096 >> 172.21.15.3:0/4134949096 conn(0x7f858c0fbc60 msgr2=0x7f858c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 -- 172.21.15.3:0/4134949096 shutdown_connections 2023-09-10T01:02:17.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.023+0000 7f8590bff700 1 -- 172.21.15.3:0/4134949096 wait complete. 2023-09-10T01:02:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.024+0000 7f8590bff700 1 Processor -- start 2023-09-10T01:02:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.025+0000 7f8590bff700 1 -- start start 2023-09-10T01:02:17.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.025+0000 7f8590bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f858c1006d0 0x7f858c196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.025+0000 7f8590bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c1018c0 0x7f858c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.025+0000 7f8590bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c102260 0x7f858c19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.025+0000 7f8590bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c19dff0 con 0x7f858c1018c0 2023-09-10T01:02:17.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.025+0000 7f8590bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c19e130 con 0x7f858c1006d0 2023-09-10T01:02:17.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.025+0000 7f8590bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c19e2c0 con 0x7f858c102260 2023-09-10T01:02:17.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c1018c0 0x7f858c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f858a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f858c1006d0 0x7f858c196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c1018c0 0x7f858c1969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34522/0 (socket says 172.21.15.3:34522) 2023-09-10T01:02:17.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 -- 172.21.15.3:0/3703030487 learned_addr learned my addr 172.21.15.3:0/3703030487 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f858ad9d700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c102260 0x7f858c19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 -- 172.21.15.3:0/3703030487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c102260 msgr2=0x7f858c19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c102260 0x7f858c19da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 -- 172.21.15.3:0/3703030487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f858c1006d0 msgr2=0x7f858c196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f858c1006d0 0x7f858c196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 -- 172.21.15.3:0/3703030487 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8584009d00 con 0x7f858c1018c0 2023-09-10T01:02:17.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f858a59c700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f858c1006d0 0x7f858c196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:17.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f8589d9b700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c1018c0 0x7f858c1969b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f858000e9a0 tx=0x7f858000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.026+0000 7f857b7fe700 1 -- 172.21.15.3:0/3703030487 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580009790 con 0x7f858c1018c0 2023-09-10T01:02:17.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.027+0000 7f857b7fe700 1 -- 172.21.15.3:0/3703030487 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8580010430 con 0x7f858c1018c0 2023-09-10T01:02:17.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.027+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c19e5a0 con 0x7f858c1018c0 2023-09-10T01:02:17.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.027+0000 7f857b7fe700 1 -- 172.21.15.3:0/3703030487 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580018680 con 0x7f858c1018c0 2023-09-10T01:02:17.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.027+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c19eab0 con 0x7f858c1018c0 2023-09-10T01:02:17.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.028+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c107530 con 0x7f858c1018c0 2023-09-10T01:02:17.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.030+0000 7f857b7fe700 1 -- 172.21.15.3:0/3703030487 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f85800105a0 con 0x7f858c1018c0 2023-09-10T01:02:17.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.031+0000 7f857b7fe700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85740a06d0 0x7f85740a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.031+0000 7f857b7fe700 1 -- 172.21.15.3:0/3703030487 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f8580014070 con 0x7f858c1018c0 2023-09-10T01:02:17.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.031+0000 7f858a59c700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85740a06d0 0x7f85740a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.032+0000 7f858a59c700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85740a06d0 0x7f85740a2b90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f858c101720 tx=0x7f857c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.035+0000 7f857b7fe700 1 -- 172.21.15.3:0/3703030487 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f8580051e10 con 0x7f858c1018c0 2023-09-10T01:02:17.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.289+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f858c10bef0 con 0x7f858c1018c0 2023-09-10T01:02:17.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.289+0000 7f857b7fe700 1 -- 172.21.15.3:0/3703030487 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f8580095070 con 0x7f858c1018c0 2023-09-10T01:02:17.290 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:17.291 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:02:17.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85740a06d0 msgr2=0x7f85740a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85740a06d0 0x7f85740a2b90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f858c101720 tx=0x7f857c00b410 comp rx=0 tx=0).stop 2023-09-10T01:02:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c1018c0 msgr2=0x7f858c1969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c1018c0 0x7f858c1969b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f858000e9a0 tx=0x7f858000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 shutdown_connections 2023-09-10T01:02:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85740a06d0 0x7f85740a2b90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c102260 0x7f858c19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f858c1006d0 0x7f858c196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 --2- 172.21.15.3:0/3703030487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c1018c0 0x7f858c1969b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 >> 172.21.15.3:0/3703030487 conn(0x7f858c0fbc60 msgr2=0x7f858c0fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 shutdown_connections 2023-09-10T01:02:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.292+0000 7f8590bff700 1 -- 172.21.15.3:0/3703030487 wait complete. 2023-09-10T01:02:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:02:17.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:17 smithi003 ceph-mon[97654]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:17 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2864442739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:17.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:17 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3752494734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:17.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-09-10T01:00:43.531603+0000', 'modified': '2023-09-10T01:01:09.940177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24698, 'mds_1': 24647, 'mds_2': 24634}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24604': {'gid': 24604, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6835/662717057', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6834', 'nonce': 662717057}, {'type': 'v1', 'addr': '172.21.15.136:6835', 'nonce': 662717057}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24610': {'gid': 24610, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6837/3841438021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6836', 'nonce': 3841438021}, {'type': 'v1', 'addr': '172.21.15.136:6837', 'nonce': 3841438021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24620': {'gid': 24620, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.92:6835/790897161', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 790897161}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 790897161}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24634': {'gid': 24634, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.136:6839/1259564848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 1259564848}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 1259564848}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24647': {'gid': 24647, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.92:6837/12010426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6836', 'nonce': 12010426}, {'type': 'v1', 'addr': '172.21.15.92:6837', 'nonce': 12010426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24698': {'gid': 24698, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6839/149607042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 149607042}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 149607042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-09-10T01:02:17.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-09-10T01:02:17.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-09-10T01:02:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:17 smithi136 ceph-mon[133708]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:17 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2864442739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:17.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:17 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3752494734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:17.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:17 smithi092 ceph-mon[134067]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:17.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:17 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2864442739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:17.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:17 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3752494734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:17.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.517+0000 7f94da9ff700 1 -- 172.21.15.3:0/3445200248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d4102270 msgr2=0x7f94d4108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.517+0000 7f94da9ff700 1 --2- 172.21.15.3:0/3445200248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d4102270 0x7f94d4108b20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00b0d0 tx=0x7f94cc00b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.518+0000 7f94da9ff700 1 -- 172.21.15.3:0/3445200248 shutdown_connections 2023-09-10T01:02:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.518+0000 7f94da9ff700 1 --2- 172.21.15.3:0/3445200248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d4102270 0x7f94d4108b20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.518+0000 7f94da9ff700 1 --2- 172.21.15.3:0/3445200248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41006e0 0x7f94d4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.518+0000 7f94da9ff700 1 --2- 172.21.15.3:0/3445200248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94d41018d0 0x7f94d4101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.518+0000 7f94da9ff700 1 -- 172.21.15.3:0/3445200248 >> 172.21.15.3:0/3445200248 conn(0x7f94d40fbc50 msgr2=0x7f94d40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.518+0000 7f94da9ff700 1 -- 172.21.15.3:0/3445200248 shutdown_connections 2023-09-10T01:02:17.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.518+0000 7f94da9ff700 1 -- 172.21.15.3:0/3445200248 wait complete. 2023-09-10T01:02:17.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.519+0000 7f94da9ff700 1 Processor -- start 2023-09-10T01:02:17.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.519+0000 7f94da9ff700 1 -- start start 2023-09-10T01:02:17.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94da9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d41006e0 0x7f94d4196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94da9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41018d0 0x7f94d41969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94da9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94d4102270 0x7f94d419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94da9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d419e010 con 0x7f94d41018d0 2023-09-10T01:02:17.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94da9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d419e150 con 0x7f94d41006e0 2023-09-10T01:02:17.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94da9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d419e2e0 con 0x7f94d4102270 2023-09-10T01:02:17.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41018d0 0x7f94d41969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41018d0 0x7f94d41969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34542/0 (socket says 172.21.15.3:34542) 2023-09-10T01:02:17.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94d37fe700 1 -- 172.21.15.3:0/2324368759 learned_addr learned my addr 172.21.15.3:0/2324368759 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94d3fff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d41006e0 0x7f94d4196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.520+0000 7f94d8f9c700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94d4102270 0x7f94d419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d37fe700 1 -- 172.21.15.3:0/2324368759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94d4102270 msgr2=0x7f94d419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d37fe700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94d4102270 0x7f94d419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d37fe700 1 -- 172.21.15.3:0/2324368759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d41006e0 msgr2=0x7f94d4196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d37fe700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d41006e0 0x7f94d4196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d37fe700 1 -- 172.21.15.3:0/2324368759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4009520 con 0x7f94d41018d0 2023-09-10T01:02:17.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d3fff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d41006e0 0x7f94d4196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:17.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d37fe700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41018d0 0x7f94d41969d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94c800e9a0 tx=0x7f94c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d17fa700 1 -- 172.21.15.3:0/2324368759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8009790 con 0x7f94d41018d0 2023-09-10T01:02:17.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d17fa700 1 -- 172.21.15.3:0/2324368759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94c8010430 con 0x7f94d41018d0 2023-09-10T01:02:17.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc009d00 con 0x7f94d41018d0 2023-09-10T01:02:17.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94d17fa700 1 -- 172.21.15.3:0/2324368759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8018680 con 0x7f94d41018d0 2023-09-10T01:02:17.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.521+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d419e940 con 0x7f94d41018d0 2023-09-10T01:02:17.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.526+0000 7f94d17fa700 1 -- 172.21.15.3:0/2324368759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f94c80105a0 con 0x7f94d41018d0 2023-09-10T01:02:17.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 Processor -- start 2023-09-10T01:02:17.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 -- start start 2023-09-10T01:02:17.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c03086e0 con 0x5639c03afdb0 2023-09-10T01:02:17.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c0306000 con 0x5639c03b0bb0 2023-09-10T01:02:17.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.527+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c03018f0 con 0x5639c03b1550 2023-09-10T01:02:17.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d8f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d8f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03b0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40770/0 (socket says 172.21.15.3:40770) 2023-09-10T01:02:17.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d8f9c700 1 -- 172.21.15.3:0/3135869178 learned_addr learned my addr 172.21.15.3:0/3135869178 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d3fff700 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d17fa700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f94bc0a06d0 0x7f94bc0a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d3fff700 1 -- 172.21.15.3:0/3135869178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 msgr2=0x5639c03b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d17fa700 1 -- 172.21.15.3:0/2324368759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f94c8014070 con 0x7f94d41018d0 2023-09-10T01:02:17.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d3fff700 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03b0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d3fff700 1 -- 172.21.15.3:0/3135869178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 msgr2=0x5639c03b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d3fff700 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d3fff700 1 -- 172.21.15.3:0/3135869178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639c012dd90 con 0x5639c03b1550 2023-09-10T01:02:17.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d8f9c700 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:17.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.528+0000 7f94d8f9c700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f94bc0a06d0 0x7f94bc0a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.529+0000 7f94d3fff700 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03af7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94c4011600 tx=0x7f94c4011910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d79b2362b75414 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.529+0000 7f94ba7fc700 1 -- 172.21.15.3:0/3135869178 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c400e470 con 0x5639c03b1550 2023-09-10T01:02:17.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.529+0000 7f94ba7fc700 1 -- 172.21.15.3:0/3135869178 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94c400ea90 con 0x5639c03b1550 2023-09-10T01:02:17.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.529+0000 7f94ba7fc700 1 -- 172.21.15.3:0/3135869178 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4019ab0 con 0x5639c03b1550 2023-09-10T01:02:17.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/3135869178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 msgr2=0x5639c03af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94d8f9c700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f94bc0a06d0 0x7f94bc0a2b90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f94cc015000 tx=0x7f94cc0099e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03af7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94c4011600 tx=0x7f94c4011910 comp rx=0 tx=0).stop 2023-09-10T01:02:17.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/3135869178 shutdown_connections 2023-09-10T01:02:17.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/3135869178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03af7b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/3135869178 >> 172.21.15.3:0/3135869178 conn(0x5639c0324040 msgr2=0x5639c03aa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/3135869178 shutdown_connections 2023-09-10T01:02:17.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.530+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/3135869178 wait complete. 2023-09-10T01:02:17.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 -- start start 2023-09-10T01:02:17.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 Processor -- start 2023-09-10T01:02:17.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b0bb0 0x5639c03b9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b1550 0x5639c03c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c029fb20 con 0x5639c03afdb0 2023-09-10T01:02:17.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c029f820 con 0x5639c03b1550 2023-09-10T01:02:17.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c01838d0 con 0x5639c03b0bb0 2023-09-10T01:02:17.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.531+0000 7f94ee2a3b80 10 client.0 _ll_register_callbacks cb 0x5639c01e9a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:17.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b0bb0 0x5639c03b9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b0bb0 0x5639c03b9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40036/0 (socket says 172.21.15.3:40036) 2023-09-10T01:02:17.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 -- 172.21.15.3:0/628150688 learned_addr learned my addr 172.21.15.3:0/628150688 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d3fff700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d8f9c700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b1550 0x5639c03c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 -- 172.21.15.3:0/628150688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b1550 msgr2=0x5639c03c0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b1550 0x5639c03c0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 -- 172.21.15.3:0/628150688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 msgr2=0x5639c03b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 -- 172.21.15.3:0/628150688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d4109470 con 0x5639c03b0bb0 2023-09-10T01:02:17.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d3fff700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:17.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94d37fe700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b0bb0 0x5639c03b9a80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f94c80c98b0 tx=0x7f94c80c9db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba61a393d6fc19b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94b37fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:02:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.532+0000 7f94b37fe700 1 -- 172.21.15.3:0/628150688 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8099c60 con 0x5639c03b0bb0 2023-09-10T01:02:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5639c02594a0 con 0x5639c03b0bb0 2023-09-10T01:02:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94b37fe700 1 -- 172.21.15.3:0/628150688 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f94c8099560 con 0x5639c03b0bb0 2023-09-10T01:02:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94b37fe700 1 -- 172.21.15.3:0/628150688 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94c809d160 con 0x5639c03b0bb0 2023-09-10T01:02:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 10 client.25228 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:17.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f94c4009e20 con 0x5639c03b0bb0 2023-09-10T01:02:17.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94b37fe700 1 -- 172.21.15.3:0/628150688 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94c80b0020 con 0x5639c03b0bb0 2023-09-10T01:02:17.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94b37fe700 1 -- 172.21.15.3:0/628150688 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f94c80ce0d0 con 0x5639c03b0bb0 2023-09-10T01:02:17.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 10 client.25228 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:17.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 10 client.25228 resolve_mds: resolved name 'i' to daemon mds.i 2023-09-10T01:02:17.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 20 client.25228 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:17.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c03c4ca0 0x5639c03c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 4 client.25228 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:02:17.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.533+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639c0316850 con 0x5639c03c4ca0 2023-09-10T01:02:17.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.534+0000 7f94d3fff700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c03c4ca0 0x5639c03c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.534+0000 7f94d3fff700 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c03c4ca0 0x5639c03c3c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac60ece96d42b601 server_cookie=ac71ab630391f1b1 in_seq=0 out_seq=0 2023-09-10T01:02:17.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.534+0000 7f94b37fe700 10 client.25228 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:17.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94b37fe700 1 -- 172.21.15.3:0/628150688 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5639c0316850 con 0x5639c03c4ca0 2023-09-10T01:02:17.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94b37fe700 10 client.25228 handle_command_reply: tid=0 2023-09-10T01:02:17.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 client.25228 shutdown 2023-09-10T01:02:17.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b0bb0 msgr2=0x5639c03b9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b0bb0 0x5639c03b9a80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f94c80c98b0 tx=0x7f94c80c9db0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 shutdown_connections 2023-09-10T01:02:17.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c03c4ca0 0x5639c03c3c20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b1550 0x5639c03c0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/628150688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b0bb0 0x5639c03b9a80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94b37fe700 0 client.25228 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:17.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 >> 172.21.15.3:0/628150688 conn(0x5639c0324040 msgr2=0x5639c03a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 shutdown_connections 2023-09-10T01:02:17.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.535+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/628150688 wait complete. 2023-09-10T01:02:17.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.536+0000 7f94ee2a3b80 20 client.25228 trim_cache size 0 max 16384 2023-09-10T01:02:17.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 Processor -- start 2023-09-10T01:02:17.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 -- start start 2023-09-10T01:02:17.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03c0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03b7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c02a94d0 con 0x5639c03afdb0 2023-09-10T01:02:17.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c03a4d20 con 0x5639c03b0bb0 2023-09-10T01:02:17.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.554+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c03b89f0 con 0x5639c03b1550 2023-09-10T01:02:17.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03c0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03c0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40786/0 (socket says 172.21.15.3:40786) 2023-09-10T01:02:17.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 -- 172.21.15.3:0/2657868518 learned_addr learned my addr 172.21.15.3:0/2657868518 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d8f9c700 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 -- 172.21.15.3:0/2657868518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 msgr2=0x5639c03b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03b7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 -- 172.21.15.3:0/2657868518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 msgr2=0x5639c03c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03c3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 -- 172.21.15.3:0/2657868518 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639c0116c10 con 0x5639c03b0bb0 2023-09-10T01:02:17.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.555+0000 7f94d37fe700 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03c0a80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94c800c2d0 tx=0x7f94c8010a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b8ce9f8f27867aa server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94baffd700 1 -- 172.21.15.3:0/2657868518 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c809d160 con 0x5639c03b0bb0 2023-09-10T01:02:17.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94baffd700 1 -- 172.21.15.3:0/2657868518 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94c80b0070 con 0x5639c03b0bb0 2023-09-10T01:02:17.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94baffd700 1 -- 172.21.15.3:0/2657868518 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c80a15d0 con 0x5639c03b0bb0 2023-09-10T01:02:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/2657868518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 msgr2=0x5639c03c0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03c0a80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94c800c2d0 tx=0x7f94c8010a90 comp rx=0 tx=0).stop 2023-09-10T01:02:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/2657868518 shutdown_connections 2023-09-10T01:02:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03b1550 0x5639c03b7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03afdb0 0x5639c03c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/2657868518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03c0a80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/2657868518 >> 172.21.15.3:0/2657868518 conn(0x5639c0324040 msgr2=0x5639c03ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/2657868518 shutdown_connections 2023-09-10T01:02:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.556+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/2657868518 wait complete. 2023-09-10T01:02:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.557+0000 7f94ee2a3b80 1 -- start start 2023-09-10T01:02:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.557+0000 7f94ee2a3b80 1 Processor -- start 2023-09-10T01:02:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03afdb0 0x5639c03bb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03bbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94ee2a3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03b1550 0x5639c0418cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c03c0d20 con 0x5639c03b1550 2023-09-10T01:02:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c03c0e90 con 0x5639c03b0bb0 2023-09-10T01:02:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94ee2a3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c0419230 con 0x5639c03afdb0 2023-09-10T01:02:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94ee2a3b80 10 client.0 _ll_register_callbacks cb 0x7f94c8095780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03afdb0 0x5639c03bb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d8f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03b1550 0x5639c0418cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03afdb0 0x5639c03bb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40066/0 (socket says 172.21.15.3:40066) 2023-09-10T01:02:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 -- 172.21.15.3:0/680647005 learned_addr learned my addr 172.21.15.3:0/680647005 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d37fe700 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03bbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 -- 172.21.15.3:0/680647005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 msgr2=0x5639c03bbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03bbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 -- 172.21.15.3:0/680647005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03b1550 msgr2=0x5639c0418cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03b1550 0x5639c0418cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.558+0000 7f94d3fff700 1 -- 172.21.15.3:0/680647005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c4009e20 con 0x5639c03afdb0 2023-09-10T01:02:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94d3fff700 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03afdb0 0x5639c03bb7b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009fd0 tx=0x7f94c40058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2f607d58126cfb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94b37fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:02:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94b37fe700 1 -- 172.21.15.3:0/680647005 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c400eda0 con 0x5639c03afdb0 2023-09-10T01:02:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94c400ea90 con 0x5639c03afdb0 2023-09-10T01:02:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94b37fe700 1 -- 172.21.15.3:0/680647005 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f94c400e800 con 0x5639c03afdb0 2023-09-10T01:02:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94b37fe700 1 -- 172.21.15.3:0/680647005 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94c4025a50 con 0x5639c03afdb0 2023-09-10T01:02:17.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94ee2a3b80 10 client.25234 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:17.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.559+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5639c0154200 con 0x5639c03afdb0 2023-09-10T01:02:17.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94b37fe700 1 -- 172.21.15.3:0/680647005 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94c400ea90 con 0x5639c03afdb0 2023-09-10T01:02:17.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94b37fe700 1 -- 172.21.15.3:0/680647005 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f94c4005140 con 0x5639c03afdb0 2023-09-10T01:02:17.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94ee2a3b80 10 client.25234 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:17.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94ee2a3b80 10 client.25234 resolve_mds: resolved name 'i' to daemon mds.i 2023-09-10T01:02:17.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94ee2a3b80 20 client.25234 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:17.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c041e340 0x5639c041a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94ee2a3b80 4 client.25234 mds_command: new command op to 24634 tid=0[{"prefix": "status"}] 2023-09-10T01:02:17.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5639c02fa8d0 con 0x5639c041e340 2023-09-10T01:02:17.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94d37fe700 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c041e340 0x5639c041a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94d37fe700 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c041e340 0x5639c041a7c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eaa4eb35b80c20c9 server_cookie=f4713236156df27b in_seq=0 out_seq=0 2023-09-10T01:02:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.560+0000 7f94b37fe700 10 client.25234 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94b37fe700 1 -- 172.21.15.3:0/680647005 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5639c02fa8d0 con 0x5639c041e340 2023-09-10T01:02:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94b37fe700 10 client.25234 handle_command_reply: tid=0 2023-09-10T01:02:17.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 client.25234 shutdown 2023-09-10T01:02:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03afdb0 msgr2=0x5639c03bb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03afdb0 0x5639c03bb7b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009fd0 tx=0x7f94c40058e0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 shutdown_connections 2023-09-10T01:02:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5639c041e340 0x5639c041a7c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5639c03afdb0 0x5639c03bb7b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c03b1550 0x5639c0418cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 --2- 172.21.15.3:0/680647005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5639c03b0bb0 0x5639c03bbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94b37fe700 0 client.25234 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 >> 172.21.15.3:0/680647005 conn(0x5639c0324040 msgr2=0x5639c03c3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 shutdown_connections 2023-09-10T01:02:17.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 1 -- 172.21.15.3:0/680647005 wait complete. 2023-09-10T01:02:17.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.561+0000 7f94ee2a3b80 20 client.25234 trim_cache size 0 max 16384 2023-09-10T01:02:17.576 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:02:17.576 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T01:02:17.576 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 2, 2023-09-10T01:02:17.577 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24634, 2023-09-10T01:02:17.577 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2023-09-10T01:02:17.577 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2023-09-10T01:02:17.577 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2023-09-10T01:02:17.577 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 68.532860029999995, 2023-09-10T01:02:17.578 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 22, 2023-09-10T01:02:17.578 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 90, 2023-09-10T01:02:17.578 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 88, 2023-09-10T01:02:17.578 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 107.309004965 2023-09-10T01:02:17.578 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:02:17.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f94bc0a06d0 msgr2=0x7f94bc0a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f94bc0a06d0 0x7f94bc0a2b90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f94cc015000 tx=0x7f94cc0099e0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41018d0 msgr2=0x7f94d41969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41018d0 0x7f94d41969d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94c800e9a0 tx=0x7f94c800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 shutdown_connections 2023-09-10T01:02:17.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f94bc0a06d0 0x7f94bc0a2b90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94d4102270 0x7f94d419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94d41006e0 0x7f94d4196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 --2- 172.21.15.3:0/2324368759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94d41018d0 0x7f94d41969d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 >> 172.21.15.3:0/2324368759 conn(0x7f94d40fbc50 msgr2=0x7f94d4104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 shutdown_connections 2023-09-10T01:02:17.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.562+0000 7f94da9ff700 1 -- 172.21.15.3:0/2324368759 wait complete. 2023-09-10T01:02:17.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-09-10T01:02:17.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/272365714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df41018d0 msgr2=0x7f2df4101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/272365714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df41018d0 0x7f2df4101d30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2de40099c0 tx=0x7f2de4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/272365714 shutdown_connections 2023-09-10T01:02:17.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/272365714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2df4102270 0x7f2df4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/272365714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2df41006e0 0x7f2df4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/272365714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df41018d0 0x7f2df4101d30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/272365714 >> 172.21.15.3:0/272365714 conn(0x7f2df40fbc50 msgr2=0x7f2df40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/272365714 shutdown_connections 2023-09-10T01:02:17.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.787+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/272365714 wait complete. 2023-09-10T01:02:17.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.788+0000 7f2dfc1ff700 1 Processor -- start 2023-09-10T01:02:17.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.788+0000 7f2dfc1ff700 1 -- start start 2023-09-10T01:02:17.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfc1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2df41006e0 0x7f2df40743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfc1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2df41018d0 0x7f2df40729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfc1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df4102270 0x7f2df4072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfc1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df40736d0 con 0x7f2df4102270 2023-09-10T01:02:17.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfc1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df4073840 con 0x7f2df41018d0 2023-09-10T01:02:17.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfc1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df41a0410 con 0x7f2df41006e0 2023-09-10T01:02:17.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df4102270 0x7f2df4072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2df979a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2df41018d0 0x7f2df40729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df4102270 0x7f2df4072f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34604/0 (socket says 172.21.15.3:34604) 2023-09-10T01:02:17.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 -- 172.21.15.3:0/536637725 learned_addr learned my addr 172.21.15.3:0/536637725 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2df41006e0 0x7f2df40743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 -- 172.21.15.3:0/536637725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2df41006e0 msgr2=0x7f2df40743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2df41006e0 0x7f2df40743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 -- 172.21.15.3:0/536637725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2df41018d0 msgr2=0x7f2df40729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2df41018d0 0x7f2df40729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2dfa79c700 1 -- 172.21.15.3:0/536637725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de40095f0 con 0x7f2df4102270 2023-09-10T01:02:17.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2df979a700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2df41018d0 0x7f2df40729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:17.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.789+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2df41006e0 0x7f2df40743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:17.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.790+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df4102270 0x7f2df4072f30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00ef30 tx=0x7f2dec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.790+0000 7f2deaffd700 1 -- 172.21.15.3:0/536637725 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec018070 con 0x7f2df4102270 2023-09-10T01:02:17.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.790+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df41a0690 con 0x7f2df4102270 2023-09-10T01:02:17.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.790+0000 7f2deaffd700 1 -- 172.21.15.3:0/536637725 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dec00f040 con 0x7f2df4102270 2023-09-10T01:02:17.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.790+0000 7f2deaffd700 1 -- 172.21.15.3:0/536637725 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec00cdb0 con 0x7f2df4102270 2023-09-10T01:02:17.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.790+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df41a0b50 con 0x7f2df4102270 2023-09-10T01:02:17.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.793+0000 7f2deaffd700 1 -- 172.21.15.3:0/536637725 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f2dec003a00 con 0x7f2df4102270 2023-09-10T01:02:17.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.794+0000 7f2deaffd700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2de00a06d0 0x7f2de00a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 Processor -- start 2023-09-10T01:02:17.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2deaffd700 1 -- 172.21.15.3:0/536637725 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f2dec0cbda0 con 0x7f2df4102270 2023-09-10T01:02:17.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2de00a06d0 0x7f2de00a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 -- start start 2023-09-10T01:02:17.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec8bb0 0x55cadcec8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadcec77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadcec8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadce206e0 con 0x55cadcec8bb0 2023-09-10T01:02:17.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadce1e000 con 0x55cadcec9550 2023-09-10T01:02:17.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.795+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadce198f0 con 0x55cadcec7db0 2023-09-10T01:02:17.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadcec8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2df9f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec8bb0 0x55cadcec8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2df979a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadcec77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadcec8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40102/0 (socket says 172.21.15.3:40102) 2023-09-10T01:02:17.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 -- 172.21.15.3:0/1148478893 learned_addr learned my addr 172.21.15.3:0/1148478893 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2de00a06d0 0x7f2de00a2b90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2df4101730 tx=0x7f2df00116c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 -- 172.21.15.3:0/1148478893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 msgr2=0x55cadcec77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadcec77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 -- 172.21.15.3:0/1148478893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec8bb0 msgr2=0x55cadcec8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec8bb0 0x55cadcec8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 -- 172.21.15.3:0/1148478893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cadcc45d90 con 0x55cadcec7db0 2023-09-10T01:02:17.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec8bb0 0x55cadcec8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:17.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadcec8260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0955b0 tx=0x7f2dec098cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965a81ed46a87cec server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2ddffff700 1 -- 172.21.15.3:0/1148478893 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec01b320 con 0x55cadcec7db0 2023-09-10T01:02:17.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2ddffff700 1 -- 172.21.15.3:0/1148478893 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dec0997b0 con 0x55cadcec7db0 2023-09-10T01:02:17.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.796+0000 7f2ddffff700 1 -- 172.21.15.3:0/1148478893 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec099310 con 0x55cadcec7db0 2023-09-10T01:02:17.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/1148478893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 msgr2=0x55cadcec8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadcec8260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0955b0 tx=0x7f2dec098cc0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/1148478893 shutdown_connections 2023-09-10T01:02:17.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadcec8260 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec8bb0 0x55cadcec8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/1148478893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadcec77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/1148478893 >> 172.21.15.3:0/1148478893 conn(0x55cadce3c040 msgr2=0x55cadcec26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/1148478893 shutdown_connections 2023-09-10T01:02:17.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.797+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/1148478893 wait complete. 2023-09-10T01:02:17.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 -- start start 2023-09-10T01:02:17.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 Processor -- start 2023-09-10T01:02:17.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec9550 0x55cadced8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadcdb7b20 con 0x55cadcec7db0 2023-09-10T01:02:17.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadcdb7820 con 0x55cadcec8bb0 2023-09-10T01:02:17.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.798+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadcc9b8d0 con 0x55cadcec9550 2023-09-10T01:02:17.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2e0fb7eb80 10 client.0 _ll_register_callbacks cb 0x55cadcd01a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:17.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34612/0 (socket says 172.21.15.3:34612) 2023-09-10T01:02:17.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 -- 172.21.15.3:0/336276415 learned_addr learned my addr 172.21.15.3:0/336276415 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df979a700 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec9550 0x55cadced8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 -- 172.21.15.3:0/336276415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec9550 msgr2=0x55cadced8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec9550 0x55cadced8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 -- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 msgr2=0x55cadced1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 -- 172.21.15.3:0/336276415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df0011c70 con 0x55cadcec7db0 2023-09-10T01:02:17.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced1580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2df0014c60 tx=0x7f2df00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449ef0cc2b2b4bf4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.799+0000 7f2ddd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:02:17.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.800+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/336276415 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0020a20 con 0x55cadcec7db0 2023-09-10T01:02:17.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.800+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cadcd714a0 con 0x55cadcec7db0 2023-09-10T01:02:17.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.800+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/336276415 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f2df001a070 con 0x55cadcec7db0 2023-09-10T01:02:17.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.800+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/336276415 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2df00142e0 con 0x55cadcec7db0 2023-09-10T01:02:17.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.800+0000 7f2e0fb7eb80 10 client.15240 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:17.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.800+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f2df41a0b50 con 0x55cadcec7db0 2023-09-10T01:02:17.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/336276415 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cadcd714a0 con 0x55cadcec7db0 2023-09-10T01:02:17.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/336276415 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2df0012cc0 con 0x55cadcec7db0 2023-09-10T01:02:17.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2e0fb7eb80 10 client.15240 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:17.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2e0fb7eb80 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2023-09-10T01:02:17.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2e0fb7eb80 20 client.15240 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:17.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcedcd10 0x55cadcedbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2e0fb7eb80 4 client.15240 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:02:17.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cadce2e850 con 0x55cadcedcd10 2023-09-10T01:02:17.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.801+0000 7f2df979a700 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcedcd10 0x55cadcedbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.802+0000 7f2df979a700 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcedcd10 0x55cadcedbc50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=59f88b65312e9eba server_cookie=8475617b5ad271f7 in_seq=0 out_seq=0 2023-09-10T01:02:17.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.802+0000 7f2ddd7fa700 10 client.15240 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:17.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.802+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/336276415 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cadce2e850 con 0x55cadcedcd10 2023-09-10T01:02:17.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.802+0000 7f2ddd7fa700 10 client.15240 handle_command_reply: tid=0 2023-09-10T01:02:17.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.802+0000 7f2e0fb7eb80 1 client.15240 shutdown 2023-09-10T01:02:17.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 msgr2=0x55cadced1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced1580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2df0014c60 tx=0x7f2df00047c0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 shutdown_connections 2023-09-10T01:02:17.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcedcd10 0x55cadcedbc50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec9550 0x55cadced8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced1580 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2ddd7fa700 0 client.15240 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:17.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/336276415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 >> 172.21.15.3:0/336276415 conn(0x55cadce3c040 msgr2=0x55cadcec1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 shutdown_connections 2023-09-10T01:02:17.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/336276415 wait complete. 2023-09-10T01:02:17.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.803+0000 7f2e0fb7eb80 20 client.15240 trim_cache size 0 max 16384 2023-09-10T01:02:17.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.821+0000 7f2e0fb7eb80 1 Processor -- start 2023-09-10T01:02:17.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.821+0000 7f2e0fb7eb80 1 -- start start 2023-09-10T01:02:17.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadced9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec9550 0x55cadced8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadcdd5ed0 con 0x55cadcec9550 2023-09-10T01:02:17.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadcdc14d0 con 0x55cadcec8bb0 2023-09-10T01:02:17.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadcecfeb0 con 0x55cadcec7db0 2023-09-10T01:02:17.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec9550 0x55cadced8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec9550 0x55cadced8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34622/0 (socket says 172.21.15.3:34622) 2023-09-10T01:02:17.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 -- 172.21.15.3:0/3651675819 learned_addr learned my addr 172.21.15.3:0/3651675819 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadced9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2df979a700 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 -- 172.21.15.3:0/3651675819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 msgr2=0x55cadced9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadced9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 -- 172.21.15.3:0/3651675819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 msgr2=0x55cadced0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2dfa79c700 1 -- 172.21.15.3:0/3651675819 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cadcc2ec10 con 0x55cadcec9550 2023-09-10T01:02:17.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.822+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadced9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:17.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec9550 0x55cadced8a90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2dec099040 tx=0x7f2dec004bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121f48f33f600521 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2de8ff9700 1 -- 172.21.15.3:0/3651675819 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec01b320 con 0x55cadcec9550 2023-09-10T01:02:17.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2de8ff9700 1 -- 172.21.15.3:0/3651675819 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dec0990b0 con 0x55cadcec9550 2023-09-10T01:02:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2de8ff9700 1 -- 172.21.15.3:0/3651675819 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec0985a0 con 0x55cadcec9550 2023-09-10T01:02:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/3651675819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec9550 msgr2=0x55cadced8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec9550 0x55cadced8a90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2dec099040 tx=0x7f2dec004bf0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/3651675819 shutdown_connections 2023-09-10T01:02:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec7db0 0x55cadced9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec9550 0x55cadced8a90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/3651675819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec8bb0 0x55cadced0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.823+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/3651675819 >> 172.21.15.3:0/3651675819 conn(0x55cadce3c040 msgr2=0x55cadcee5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.824+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/3651675819 shutdown_connections 2023-09-10T01:02:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.824+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/3651675819 wait complete. 2023-09-10T01:02:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.824+0000 7f2e0fb7eb80 1 -- start start 2023-09-10T01:02:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.824+0000 7f2e0fb7eb80 1 Processor -- start 2023-09-10T01:02:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec8bb0 0x55cadced37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2e0fb7eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadced8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadced8c50 con 0x55cadcec7db0 2023-09-10T01:02:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadced8dc0 con 0x55cadcec9550 2023-09-10T01:02:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2e0fb7eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cadcf30d60 con 0x55cadcec8bb0 2023-09-10T01:02:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2e0fb7eb80 10 client.0 _ll_register_callbacks cb 0x7f2df0014810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34634/0 (socket says 172.21.15.3:34634) 2023-09-10T01:02:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 -- 172.21.15.3:0/930347914 learned_addr learned my addr 172.21.15.3:0/930347914 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:17.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadced8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df979a700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec8bb0 0x55cadced37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 -- 172.21.15.3:0/930347914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec8bb0 msgr2=0x55cadced37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec8bb0 0x55cadced37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 -- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 msgr2=0x55cadced8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:17.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadced8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2df9f9b700 1 -- 172.21.15.3:0/930347914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cadcc45d90 con 0x55cadcec7db0 2023-09-10T01:02:17.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.825+0000 7f2dfa79c700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadced8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:17.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2df9f9b700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced3290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2df0014c30 tx=0x7f2df0029d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f115fe52f183c66c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:17.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2ddd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:02:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/930347914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0020b00 con 0x55cadcec7db0 2023-09-10T01:02:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/930347914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f2df001a050 con 0x55cadcec7db0 2023-09-10T01:02:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2dec0997b0 con 0x55cadcec7db0 2023-09-10T01:02:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/930347914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2df00142e0 con 0x55cadcec7db0 2023-09-10T01:02:17.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2e0fb7eb80 10 client.15246 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.826+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55cadcc6c200 con 0x55cadcec7db0 2023-09-10T01:02:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/930347914 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dec0997b0 con 0x55cadcec7db0 2023-09-10T01:02:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/930347914 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2df00129a0 con 0x55cadcec7db0 2023-09-10T01:02:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2e0fb7eb80 10 client.15246 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2e0fb7eb80 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2023-09-10T01:02:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2e0fb7eb80 20 client.15246 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcf35f90 0x55cadcf32410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2e0fb7eb80 4 client.15246 mds_command: new command op to 24647 tid=0[{"prefix": "status"}] 2023-09-10T01:02:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cadce128d0 con 0x55cadcf35f90 2023-09-10T01:02:17.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.827+0000 7f2df979a700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcf35f90 0x55cadcf32410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2df979a700 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcf35f90 0x55cadcf32410 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f72a6f82a9581489 server_cookie=2a843e0875f8f6de in_seq=0 out_seq=0 2023-09-10T01:02:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2ddd7fa700 10 client.15246 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2ddd7fa700 1 -- 172.21.15.3:0/930347914 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55cadce128d0 con 0x55cadcf35f90 2023-09-10T01:02:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2ddd7fa700 10 client.15246 handle_command_reply: tid=0 2023-09-10T01:02:17.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 client.15246 shutdown 2023-09-10T01:02:17.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 msgr2=0x55cadced3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced3290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2df0014c30 tx=0x7f2df0029d90 comp rx=0 tx=0).stop 2023-09-10T01:02:17.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 shutdown_connections 2023-09-10T01:02:17.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cadcec9550 0x55cadced8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cadcec7db0 0x55cadced3290 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55cadcf35f90 0x55cadcf32410 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 --2- 172.21.15.3:0/930347914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cadcec8bb0 0x55cadced37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 >> 172.21.15.3:0/930347914 conn(0x55cadce3c040 msgr2=0x55cadcecf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2ddd7fa700 0 client.15246 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:17.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 shutdown_connections 2023-09-10T01:02:17.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 1 -- 172.21.15.3:0/930347914 wait complete. 2023-09-10T01:02:17.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.828+0000 7f2e0fb7eb80 20 client.15246 trim_cache size 0 max 16384 2023-09-10T01:02:17.850 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:02:17.851 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T01:02:17.851 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 1, 2023-09-10T01:02:17.851 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24647, 2023-09-10T01:02:17.851 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2023-09-10T01:02:17.851 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2023-09-10T01:02:17.852 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2023-09-10T01:02:17.852 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 69.806456116999996, 2023-09-10T01:02:17.852 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 22, 2023-09-10T01:02:17.852 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 90, 2023-09-10T01:02:17.852 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 88, 2023-09-10T01:02:17.853 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 109.769399202 2023-09-10T01:02:17.853 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:02:17.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2de00a06d0 msgr2=0x7f2de00a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2de00a06d0 0x7f2de00a2b90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2df4101730 tx=0x7f2df00116c0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df4102270 msgr2=0x7f2df4072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df4102270 0x7f2df4072f30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00ef30 tx=0x7f2dec00c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 shutdown_connections 2023-09-10T01:02:17.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f2de00a06d0 0x7f2de00a2b90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2df41006e0 0x7f2df40743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2df4102270 0x7f2df4072f30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 --2- 172.21.15.3:0/536637725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2df41018d0 0x7f2df40729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 >> 172.21.15.3:0/536637725 conn(0x7f2df40fbc50 msgr2=0x7f2df40fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:17.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 shutdown_connections 2023-09-10T01:02:17.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:17.829+0000 7f2dfc1ff700 1 -- 172.21.15.3:0/536637725 wait complete. 2023-09-10T01:02:17.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-09-10T01:02:18.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 -- 172.21.15.3:0/3161346673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 msgr2=0x7f6d8c1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 --2- 172.21.15.3:0/3161346673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c1035a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009990 tx=0x7f6d88009ca0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 -- 172.21.15.3:0/3161346673 shutdown_connections 2023-09-10T01:02:18.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 --2- 172.21.15.3:0/3161346673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d8c106410 0x7f6d8c108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 --2- 172.21.15.3:0/3161346673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c1035a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 --2- 172.21.15.3:0/3161346673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d8c103ae0 0x7f6d8c105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 -- 172.21.15.3:0/3161346673 >> 172.21.15.3:0/3161346673 conn(0x7f6d8c0fab20 msgr2=0x7f6d8c0fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:18.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 -- 172.21.15.3:0/3161346673 shutdown_connections 2023-09-10T01:02:18.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.062+0000 7f6d947ff700 1 -- 172.21.15.3:0/3161346673 wait complete. 2023-09-10T01:02:18.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 Processor -- start 2023-09-10T01:02:18.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 -- start start 2023-09-10T01:02:18.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d8c103ae0 0x7f6d8c0ff160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d8c106410 0x7f6d8c0ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c0ffe40 con 0x7f6d8c103ae0 2023-09-10T01:02:18.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c0fffb0 con 0x7f6d8c1011b0 2023-09-10T01:02:18.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.063+0000 7f6d947ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c1a0400 con 0x7f6d8c106410 2023-09-10T01:02:18.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d91d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d8c103ae0 0x7f6d8c0ff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40862/0 (socket says 172.21.15.3:40862) 2023-09-10T01:02:18.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 -- 172.21.15.3:0/810715391 learned_addr learned my addr 172.21.15.3:0/810715391 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:18.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d8c106410 0x7f6d8c0ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 -- 172.21.15.3:0/810715391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d8c106410 msgr2=0x7f6d8c0ff6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d8c106410 0x7f6d8c0ff6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 -- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d8c103ae0 msgr2=0x7f6d8c0ff160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d8c103ae0 0x7f6d8c0ff160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 -- 172.21.15.3:0/810715391 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d880095f0 con 0x7f6d8c1011b0 2023-09-10T01:02:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d8c106410 0x7f6d8c0ff6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.064+0000 7f6d9259b700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c100b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800b680 tx=0x7f6d880057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.065+0000 7f6d837fe700 1 -- 172.21.15.3:0/810715391 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8801d070 con 0x7f6d8c1011b0 2023-09-10T01:02:18.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.065+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c1a0680 con 0x7f6d8c1011b0 2023-09-10T01:02:18.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.065+0000 7f6d837fe700 1 -- 172.21.15.3:0/810715391 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d8800fbc0 con 0x7f6d8c1011b0 2023-09-10T01:02:18.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.065+0000 7f6d837fe700 1 -- 172.21.15.3:0/810715391 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d88017650 con 0x7f6d8c1011b0 2023-09-10T01:02:18.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.065+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c1a0b00 con 0x7f6d8c1011b0 2023-09-10T01:02:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.068+0000 7f6d837fe700 1 -- 172.21.15.3:0/810715391 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7f6d88022470 con 0x7f6d8c1011b0 2023-09-10T01:02:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 Processor -- start 2023-09-10T01:02:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 -- start start 2023-09-10T01:02:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a571fcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a571fb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a571fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a571546e0 con 0x560a571fd550 2023-09-10T01:02:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57152000 con 0x560a571fcbb0 2023-09-10T01:02:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6da8179b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5714d8f0 con 0x560a571fbdb0 2023-09-10T01:02:18.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.069+0000 7f6d837fe700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6d780a06d0 0x7f6d780a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a571fb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a571fb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34654/0 (socket says 172.21.15.3:34654) 2023-09-10T01:02:18.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 -- 172.21.15.3:0/1989642719 learned_addr learned my addr 172.21.15.3:0/1989642719 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:18.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a571fcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6d780a06d0 0x7f6d780a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d837fe700 1 -- 172.21.15.3:0/810715391 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f6d880cd830 con 0x7f6d8c1011b0 2023-09-10T01:02:18.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d91d9a700 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a571fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 -- 172.21.15.3:0/1989642719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 msgr2=0x560a571fc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a571fc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 -- 172.21.15.3:0/1989642719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 msgr2=0x560a571fcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a571fcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.070+0000 7f6d9259b700 1 -- 172.21.15.3:0/1989642719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a56f79d90 con 0x560a571fd550 2023-09-10T01:02:18.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6d9259b700 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a571fb7b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6d880cad10 tx=0x7f6d88005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccee82a0c9cd4697 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:18.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6d780a06d0 0x7f6d780a2b90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6d84009fd0 tx=0x7f6d84009ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:18.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6d80ff9700 1 -- 172.21.15.3:0/1989642719 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8809b360 con 0x560a571fd550 2023-09-10T01:02:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6d80ff9700 1 -- 172.21.15.3:0/1989642719 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d880a2c40 con 0x560a571fd550 2023-09-10T01:02:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6d80ff9700 1 -- 172.21.15.3:0/1989642719 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d880871f0 con 0x560a571fd550 2023-09-10T01:02:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6da8179b80 1 -- 172.21.15.3:0/1989642719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 msgr2=0x560a571fb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a571fb7b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6d880cad10 tx=0x7f6d88005ad0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.071+0000 7f6da8179b80 1 -- 172.21.15.3:0/1989642719 shutdown_connections 2023-09-10T01:02:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a571fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a571fcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1989642719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a571fb7b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 -- 172.21.15.3:0/1989642719 >> 172.21.15.3:0/1989642719 conn(0x560a57170040 msgr2=0x560a571f66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:18.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 -- 172.21.15.3:0/1989642719 shutdown_connections 2023-09-10T01:02:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 -- 172.21.15.3:0/1989642719 wait complete. 2023-09-10T01:02:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 -- start start 2023-09-10T01:02:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.072+0000 7f6da8179b80 1 Processor -- start 2023-09-10T01:02:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a57205190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a572056d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a5720c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6da8179b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a570ebb20 con 0x560a571fd550 2023-09-10T01:02:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6da8179b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a570eb820 con 0x560a571fcbb0 2023-09-10T01:02:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6da8179b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a56fcf8d0 con 0x560a571fbdb0 2023-09-10T01:02:18.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6da8179b80 10 client.0 _ll_register_callbacks cb 0x560a57035a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6d92d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a5720c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6d92d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a5720c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34666/0 (socket says 172.21.15.3:34666) 2023-09-10T01:02:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6d92d9c700 1 -- 172.21.15.3:0/2597091626 learned_addr learned my addr 172.21.15.3:0/2597091626 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6d91d9a700 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a572056d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.073+0000 7f6d9259b700 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a57205190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d92d9c700 1 -- 172.21.15.3:0/2597091626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 msgr2=0x560a57205190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a57205190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d92d9c700 1 -- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 msgr2=0x560a572056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a572056d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d92d9c700 1 -- 172.21.15.3:0/2597091626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c1a0b00 con 0x560a571fd550 2023-09-10T01:02:18.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a5720c710 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d84008770 tx=0x7f6d84009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef019d88ea454c6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:18.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:02:18.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d75ffb700 1 -- 172.21.15.3:0/2597091626 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8400e040 con 0x560a571fd550 2023-09-10T01:02:18.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a570a54a0 con 0x560a571fd550 2023-09-10T01:02:18.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d75ffb700 1 -- 172.21.15.3:0/2597091626 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f6d84020070 con 0x560a571fd550 2023-09-10T01:02:18.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.074+0000 7f6d75ffb700 1 -- 172.21.15.3:0/2597091626 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d84008f10 con 0x560a571fd550 2023-09-10T01:02:18.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 10 client.15258 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x560a56f62c10 con 0x560a571fd550 2023-09-10T01:02:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6d75ffb700 1 -- 172.21.15.3:0/2597091626 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560a570a54a0 con 0x560a571fd550 2023-09-10T01:02:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6d75ffb700 1 -- 172.21.15.3:0/2597091626 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6d8401ab30 con 0x560a571fd550 2023-09-10T01:02:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 10 client.15258 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:18.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 10 client.15258 resolve_mds: resolved name 'b' to daemon mds.b 2023-09-10T01:02:18.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 20 client.15258 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:18.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a572108c0 0x560a5720f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 4 client.15258 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:02:18.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.075+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a57162850 con 0x560a572108c0 2023-09-10T01:02:18.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.076+0000 7f6d9259b700 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a572108c0 0x560a5720f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.076+0000 7f6d9259b700 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a572108c0 0x560a5720f810 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f967e42ac1cdc765 server_cookie=22cb9ad03f13d71d in_seq=0 out_seq=0 2023-09-10T01:02:18.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.076+0000 7f6d75ffb700 10 client.15258 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:18.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6d75ffb700 1 -- 172.21.15.3:0/2597091626 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560a57162850 con 0x560a572108c0 2023-09-10T01:02:18.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6d75ffb700 10 client.15258 handle_command_reply: tid=0 2023-09-10T01:02:18.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 client.15258 shutdown 2023-09-10T01:02:18.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 msgr2=0x560a5720c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a5720c710 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d84008770 tx=0x7f6d84009920 comp rx=0 tx=0).stop 2023-09-10T01:02:18.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 shutdown_connections 2023-09-10T01:02:18.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a572108c0 0x560a5720f810 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a57205190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fd550 0x560a5720c710 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6d75ffb700 0 client.15258 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:18.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 --2- 172.21.15.3:0/2597091626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fcbb0 0x560a572056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 >> 172.21.15.3:0/2597091626 conn(0x560a57170040 msgr2=0x560a571f5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:18.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 shutdown_connections 2023-09-10T01:02:18.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.077+0000 7f6da8179b80 1 -- 172.21.15.3:0/2597091626 wait complete. 2023-09-10T01:02:18.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.078+0000 7f6da8179b80 20 client.15258 trim_cache size 0 max 16384 2023-09-10T01:02:18.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 Processor -- start 2023-09-10T01:02:18.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 -- start start 2023-09-10T01:02:18.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fbdb0 0x560a571fb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fcbb0 0x560a5720c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a5720f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57162850 con 0x560a571fbdb0 2023-09-10T01:02:18.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a570a54a0 con 0x560a571fd550 2023-09-10T01:02:18.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6da8179b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57211cb0 con 0x560a571fcbb0 2023-09-10T01:02:18.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6d9259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fbdb0 0x560a571fb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6d9259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fbdb0 0x560a571fb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34682/0 (socket says 172.21.15.3:34682) 2023-09-10T01:02:18.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6d9259b700 1 -- 172.21.15.3:0/1121188746 learned_addr learned my addr 172.21.15.3:0/1121188746 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:18.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a5720f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6d91d9a700 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fcbb0 0x560a5720c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6d9259b700 1 -- 172.21.15.3:0/1121188746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fcbb0 msgr2=0x560a5720c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.096+0000 7f6d9259b700 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fcbb0 0x560a5720c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d9259b700 1 -- 172.21.15.3:0/1121188746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 msgr2=0x560a5720f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d9259b700 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a5720f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d9259b700 1 -- 172.21.15.3:0/1121188746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a56fa0200 con 0x560a571fbdb0 2023-09-10T01:02:18.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d91d9a700 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fcbb0 0x560a5720c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:02:18.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d9259b700 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fbdb0 0x560a571fb6c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6d8809a960 tx=0x7f6d8801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c780903474f7e9ce server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:18.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d817fa700 1 -- 172.21.15.3:0/1121188746 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8809a7a0 con 0x560a571fbdb0 2023-09-10T01:02:18.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d817fa700 1 -- 172.21.15.3:0/1121188746 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d880881c0 con 0x560a571fbdb0 2023-09-10T01:02:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.097+0000 7f6d817fa700 1 -- 172.21.15.3:0/1121188746 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8809a3b0 con 0x560a571fbdb0 2023-09-10T01:02:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 -- 172.21.15.3:0/1121188746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fbdb0 msgr2=0x560a571fb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fbdb0 0x560a571fb6c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6d8809a960 tx=0x7f6d8801a040 comp rx=0 tx=0).stop 2023-09-10T01:02:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 -- 172.21.15.3:0/1121188746 shutdown_connections 2023-09-10T01:02:18.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a5720f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fbdb0 0x560a571fb6c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 --2- 172.21.15.3:0/1121188746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fcbb0 0x560a5720c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 -- 172.21.15.3:0/1121188746 >> 172.21.15.3:0/1121188746 conn(0x560a57170040 msgr2=0x560a571fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 -- 172.21.15.3:0/1121188746 shutdown_connections 2023-09-10T01:02:18.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 -- 172.21.15.3:0/1121188746 wait complete. 2023-09-10T01:02:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 -- start start 2023-09-10T01:02:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.098+0000 7f6da8179b80 1 Processor -- start 2023-09-10T01:02:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a572078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fcbb0 0x560a57207de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6da8179b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a57264910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6da8179b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5720c820 con 0x560a571fcbb0 2023-09-10T01:02:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6da8179b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5720c990 con 0x560a571fd550 2023-09-10T01:02:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6da8179b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57264e50 con 0x560a571fbdb0 2023-09-10T01:02:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6da8179b80 10 client.0 _ll_register_callbacks cb 0x560a57035a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:18.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6d91d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fcbb0 0x560a57207de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6d91d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fcbb0 0x560a57207de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34686/0 (socket says 172.21.15.3:34686) 2023-09-10T01:02:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6d91d9a700 1 -- 172.21.15.3:0/3951168232 learned_addr learned my addr 172.21.15.3:0/3951168232 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6d9259b700 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a572078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.099+0000 7f6d92d9c700 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a57264910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d9259b700 1 -- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 msgr2=0x560a57264910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d9259b700 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a57264910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d9259b700 1 -- 172.21.15.3:0/3951168232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fcbb0 msgr2=0x560a57207de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d9259b700 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fcbb0 0x560a57207de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d9259b700 1 -- 172.21.15.3:0/3951168232 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a56f79d90 con 0x560a571fbdb0 2023-09-10T01:02:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d9259b700 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a572078a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6d8809abf0 tx=0x7f6d880990c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e49727cf6d1c1e9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:18.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:02:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d75ffb700 1 -- 172.21.15.3:0/3951168232 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8809a7a0 con 0x560a571fbdb0 2023-09-10T01:02:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d880a2c40 con 0x560a571fbdb0 2023-09-10T01:02:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d75ffb700 1 -- 172.21.15.3:0/3951168232 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f6d880c8040 con 0x560a571fbdb0 2023-09-10T01:02:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d75ffb700 1 -- 172.21.15.3:0/3951168232 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d8809ea90 con 0x560a571fbdb0 2023-09-10T01:02:18.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6da8179b80 10 client.25267 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:18.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x560a56f78260 con 0x560a571fbdb0 2023-09-10T01:02:18.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.100+0000 7f6d75ffb700 1 -- 172.21.15.3:0/3951168232 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d880a2c40 con 0x560a571fbdb0 2023-09-10T01:02:18.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6d75ffb700 1 -- 172.21.15.3:0/3951168232 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6d880a28d0 con 0x560a571fbdb0 2023-09-10T01:02:18.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6da8179b80 10 client.25267 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:18.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6da8179b80 10 client.25267 resolve_mds: resolved name 'b' to daemon mds.b 2023-09-10T01:02:18.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6da8179b80 20 client.25267 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:18.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6da8179b80 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a57269a30 0x560a57265ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:18.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6da8179b80 4 client.25267 mds_command: new command op to 24698 tid=0[{"prefix": "status"}] 2023-09-10T01:02:18.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560a571468d0 con 0x560a57269a30 2023-09-10T01:02:18.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.101+0000 7f6d91d9a700 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a57269a30 0x560a57265ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:18.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6d91d9a700 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a57269a30 0x560a57265ed0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f1071a0a4e6ff87 server_cookie=b04c49547f9631f4 in_seq=0 out_seq=0 2023-09-10T01:02:18.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6d75ffb700 10 client.25267 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:18.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6d75ffb700 1 -- 172.21.15.3:0/3951168232 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x560a571468d0 con 0x560a57269a30 2023-09-10T01:02:18.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6d75ffb700 10 client.25267 handle_command_reply: tid=0 2023-09-10T01:02:18.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 client.25267 shutdown 2023-09-10T01:02:18.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 msgr2=0x560a572078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a572078a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6d8809abf0 tx=0x7f6d880990c0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 shutdown_connections 2023-09-10T01:02:18.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560a57269a30 0x560a57265ed0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560a571fbdb0 0x560a572078a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6d75ffb700 0 client.25267 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:18.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a571fd550 0x560a57264910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 --2- 172.21.15.3:0/3951168232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a571fcbb0 0x560a57207de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 >> 172.21.15.3:0/3951168232 conn(0x560a57170040 msgr2=0x560a5720f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:18.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 shutdown_connections 2023-09-10T01:02:18.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 1 -- 172.21.15.3:0/3951168232 wait complete. 2023-09-10T01:02:18.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.102+0000 7f6da8179b80 20 client.25267 trim_cache size 0 max 16384 2023-09-10T01:02:18.118 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:02:18.118 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T01:02:18.118 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2023-09-10T01:02:18.119 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24698, 2023-09-10T01:02:18.119 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2023-09-10T01:02:18.119 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2023-09-10T01:02:18.119 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2023-09-10T01:02:18.119 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 94.059315983999994, 2023-09-10T01:02:18.120 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 22, 2023-09-10T01:02:18.120 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 90, 2023-09-10T01:02:18.120 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 88, 2023-09-10T01:02:18.120 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 99.295895251000005 2023-09-10T01:02:18.120 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:02:18.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6d780a06d0 msgr2=0x7f6d780a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6d780a06d0 0x7f6d780a2b90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6d84009fd0 tx=0x7f6d84009ee0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 msgr2=0x7f6d8c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:18.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c100b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800b680 tx=0x7f6d880057b0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 shutdown_connections 2023-09-10T01:02:18.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f6d780a06d0 0x7f6d780a2b90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d8c106410 0x7f6d8c0ff6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d8c1011b0 0x7f6d8c100b20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 --2- 172.21.15.3:0/810715391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d8c103ae0 0x7f6d8c0ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:18.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 >> 172.21.15.3:0/810715391 conn(0x7f6d8c0fab20 msgr2=0x7f6d8c102620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:18.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 shutdown_connections 2023-09-10T01:02:18.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:18.103+0000 7f6d947ff700 1 -- 172.21.15.3:0/810715391 wait complete. 2023-09-10T01:02:18.124 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:18 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3703030487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:18.125 INFO:tasks.cephadm:Setup complete, yielding 2023-09-10T01:02:18.125 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-09-10T01:02:18.135 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:02:18.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- bash -c 'ceph orch status' 2023-09-10T01:02:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:18 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3703030487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:18 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3703030487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:18.870 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:19.294 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:19 smithi003 ceph-mon[97654]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:19.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:19 smithi136 ceph-mon[133708]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:19.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:19 smithi092 ceph-mon[134067]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:20.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 -- 172.21.15.3:0/1634993328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 msgr2=0x7fb20c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:20.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 --2- 172.21.15.3:0/1634993328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c101cf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0099c0 tx=0x7fb1fc009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 -- 172.21.15.3:0/1634993328 shutdown_connections 2023-09-10T01:02:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 --2- 172.21.15.3:0/1634993328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c102230 0x7fb20c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 --2- 172.21.15.3:0/1634993328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb20c100690 0x7fb20c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 --2- 172.21.15.3:0/1634993328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c101cf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 -- 172.21.15.3:0/1634993328 >> 172.21.15.3:0/1634993328 conn(0x7fb20c0fbbf0 msgr2=0x7fb20c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 -- 172.21.15.3:0/1634993328 shutdown_connections 2023-09-10T01:02:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.425+0000 7fb213af3700 1 -- 172.21.15.3:0/1634993328 wait complete. 2023-09-10T01:02:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.426+0000 7fb213af3700 1 Processor -- start 2023-09-10T01:02:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.427+0000 7fb213af3700 1 -- start start 2023-09-10T01:02:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.427+0000 7fb213af3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c100690 0x7fb20c1963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.427+0000 7fb213af3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.427+0000 7fb213af3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb20c102230 0x7fb20c19d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.427+0000 7fb213af3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c19df60 con 0x7fb20c101890 2023-09-10T01:02:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.427+0000 7fb213af3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c19e0a0 con 0x7fb20c102230 2023-09-10T01:02:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.427+0000 7fb213af3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c19e230 con 0x7fb20c100690 2023-09-10T01:02:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21188f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c100690 0x7fb20c1963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34694/0 (socket says 172.21.15.3:34694) 2023-09-10T01:02:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 -- 172.21.15.3:0/264900001 learned_addr learned my addr 172.21.15.3:0/264900001 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb212090700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb20c102230 0x7fb20c19d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 -- 172.21.15.3:0/264900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c100690 msgr2=0x7fb20c1963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c100690 0x7fb20c1963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 -- 172.21.15.3:0/264900001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb20c102230 msgr2=0x7fb20c19d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb20c102230 0x7fb20c19d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 -- 172.21.15.3:0/264900001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0095f0 con 0x7fb20c101890 2023-09-10T01:02:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21188f700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c100690 0x7fb20c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb212090700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb20c102230 0x7fb20c19d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.428+0000 7fb21108e700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c196910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0095c0 tx=0x7fb1fc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.429+0000 7fb202ffd700 1 -- 172.21.15.3:0/264900001 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc01d070 con 0x7fb20c101890 2023-09-10T01:02:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.429+0000 7fb202ffd700 1 -- 172.21.15.3:0/264900001 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1fc005050 con 0x7fb20c101890 2023-09-10T01:02:20.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.429+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c19e4b0 con 0x7fb20c101890 2023-09-10T01:02:20.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.429+0000 7fb202ffd700 1 -- 172.21.15.3:0/264900001 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc004b40 con 0x7fb20c101890 2023-09-10T01:02:20.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.429+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c19e920 con 0x7fb20c101890 2023-09-10T01:02:20.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.430+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c1075d0 con 0x7fb20c101890 2023-09-10T01:02:20.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.432+0000 7fb202ffd700 1 -- 172.21.15.3:0/264900001 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fb1fc0051c0 con 0x7fb20c101890 2023-09-10T01:02:20.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.433+0000 7fb202ffd700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1f80a0750 0x7fb1f80a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:20.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.433+0000 7fb202ffd700 1 -- 172.21.15.3:0/264900001 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fb1fc0ccfe0 con 0x7fb20c101890 2023-09-10T01:02:20.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.434+0000 7fb21188f700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1f80a0750 0x7fb1f80a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:20.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.434+0000 7fb21188f700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1f80a0750 0x7fb1f80a2c10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb20c1016f0 tx=0x7fb208009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.438+0000 7fb202ffd700 1 -- 172.21.15.3:0/264900001 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb1fc096650 con 0x7fb20c101890 2023-09-10T01:02:20.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.666+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb20c02cff0 con 0x7fb1f80a0750 2023-09-10T01:02:20.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.667+0000 7fb202ffd700 1 -- 172.21.15.3:0/264900001 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb20c02cff0 con 0x7fb1f80a0750 2023-09-10T01:02:20.668 INFO:teuthology.orchestra.run.smithi003.stdout:Backend: cephadm 2023-09-10T01:02:20.668 INFO:teuthology.orchestra.run.smithi003.stdout:Available: Yes 2023-09-10T01:02:20.668 INFO:teuthology.orchestra.run.smithi003.stdout:Paused: No 2023-09-10T01:02:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1f80a0750 msgr2=0x7fb1f80a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1f80a0750 0x7fb1f80a2c10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb20c1016f0 tx=0x7fb208009260 comp rx=0 tx=0).stop 2023-09-10T01:02:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 msgr2=0x7fb20c196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c196910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc0095c0 tx=0x7fb1fc00f690 comp rx=0 tx=0).stop 2023-09-10T01:02:20.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 shutdown_connections 2023-09-10T01:02:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1f80a0750 0x7fb1f80a2c10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c100690 0x7fb20c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb20c102230 0x7fb20c19d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 --2- 172.21.15.3:0/264900001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb20c101890 0x7fb20c196910 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:20.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 >> 172.21.15.3:0/264900001 conn(0x7fb20c0fbbf0 msgr2=0x7fb20c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:20.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 shutdown_connections 2023-09-10T01:02:20.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:20.670+0000 7fb213af3700 1 -- 172.21.15.3:0/264900001 wait complete. 2023-09-10T01:02:21.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- bash -c 'ceph orch ps' 2023-09-10T01:02:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:21 smithi003 ceph-mon[97654]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:21.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:21 smithi136 ceph-mon[133708]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:21.574 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:21 smithi092 ceph-mon[134067]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:21.904 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:22.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:22 smithi003 ceph-mon[97654]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:22.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:22 smithi136 ceph-mon[133708]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:22 smithi092 ceph-mon[134067]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:23.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:23 smithi003 ceph-mon[97654]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:23.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:23 smithi136 ceph-mon[133708]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 -- 172.21.15.3:0/169257425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c100690 msgr2=0x7fa25c100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 --2- 172.21.15.3:0/169257425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c100690 0x7fa25c100ab0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa2580099c0 tx=0x7fa258009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 -- 172.21.15.3:0/169257425 shutdown_connections 2023-09-10T01:02:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 --2- 172.21.15.3:0/169257425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa25c102230 0x7fa25c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 --2- 172.21.15.3:0/169257425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c100690 0x7fa25c100ab0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 --2- 172.21.15.3:0/169257425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c101890 0x7fa25c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 -- 172.21.15.3:0/169257425 >> 172.21.15.3:0/169257425 conn(0x7fa25c0fbbf0 msgr2=0x7fa25c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 -- 172.21.15.3:0/169257425 shutdown_connections 2023-09-10T01:02:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.440+0000 7fa26412b700 1 -- 172.21.15.3:0/169257425 wait complete. 2023-09-10T01:02:23.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 Processor -- start 2023-09-10T01:02:23.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 -- start start 2023-09-10T01:02:23.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c100690 0x7fa25c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c101890 0x7fa25c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa25c102230 0x7fa25c19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c1a0120 con 0x7fa25c101890 2023-09-10T01:02:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c1a0260 con 0x7fa25c102230 2023-09-10T01:02:23.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.442+0000 7fa26412b700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c1a03f0 con 0x7fa25c100690 2023-09-10T01:02:23.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c100690 0x7fa25c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:23.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa2616c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c101890 0x7fa25c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:23.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c100690 0x7fa25c198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40206/0 (socket says 172.21.15.3:40206) 2023-09-10T01:02:23.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 -- 172.21.15.3:0/1325845108 learned_addr learned my addr 172.21.15.3:0/1325845108 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:23.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa2626c8700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa25c102230 0x7fa25c19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:23.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 -- 172.21.15.3:0/1325845108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa25c102230 msgr2=0x7fa25c19fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:23.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa25c102230 0x7fa25c19fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 -- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c101890 msgr2=0x7fa25c198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c101890 0x7fa25c198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 -- 172.21.15.3:0/1325845108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2580095f0 con 0x7fa25c100690 2023-09-10T01:02:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.443+0000 7fa261ec7700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c100690 0x7fa25c198590 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa2580098e0 tx=0x7fa25800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:23.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.444+0000 7fa252ffd700 1 -- 172.21.15.3:0/1325845108 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25801d070 con 0x7fa25c100690 2023-09-10T01:02:23.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.444+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c1a0670 con 0x7fa25c100690 2023-09-10T01:02:23.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.444+0000 7fa252ffd700 1 -- 172.21.15.3:0/1325845108 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa258005050 con 0x7fa25c100690 2023-09-10T01:02:23.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.444+0000 7fa252ffd700 1 -- 172.21.15.3:0/1325845108 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258004a70 con 0x7fa25c100690 2023-09-10T01:02:23.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.444+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c1a0a90 con 0x7fa25c100690 2023-09-10T01:02:23.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.445+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c192cd0 con 0x7fa25c100690 2023-09-10T01:02:23.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.447+0000 7fa252ffd700 1 -- 172.21.15.3:0/1325845108 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) v1 ==== 135309+0+0 (secure 0 0 0) 0x7fa2580051c0 con 0x7fa25c100690 2023-09-10T01:02:23.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.448+0000 7fa252ffd700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa2480a06d0 0x7fa2480a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:23.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.449+0000 7fa252ffd700 1 -- 172.21.15.3:0/1325845108 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fa2580cd360 con 0x7fa25c100690 2023-09-10T01:02:23.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.449+0000 7fa2616c6700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa2480a06d0 0x7fa2480a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:23.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.449+0000 7fa2616c6700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa2480a06d0 0x7fa2480a2b90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009c60 tx=0x7fa24c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:23.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.451+0000 7fa252ffd700 1 -- 172.21.15.3:0/1325845108 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa2580969b0 con 0x7fa25c100690 2023-09-10T01:02:23.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:23 smithi092 ceph-mon[134067]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:23.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.670+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa25c02cf50 con 0x7fa2480a06d0 2023-09-10T01:02:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.683+0000 7fa252ffd700 1 -- 172.21.15.3:0/1325845108 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fa25c02cf50 con 0x7fa2480a06d0 2023-09-10T01:02:23.684 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-09-10T01:02:23.684 INFO:teuthology.orchestra.run.smithi003.stdout:mds.a smithi003 running (107s) - 107s - - 18.0.0-6025-g9670cca5 20c3861ef6c9 ea54adaa57cd 2023-09-10T01:02:23.685 INFO:teuthology.orchestra.run.smithi003.stdout:mds.b smithi092 running (105s) - 104s 30.5M - 18.0.0-6025-g9670cca5 20c3861ef6c9 f8c49417c46d 2023-09-10T01:02:23.685 INFO:teuthology.orchestra.run.smithi003.stdout:mds.c smithi136 running (118s) - 117s 19.9M - 18.0.0-6025-g9670cca5 20c3861ef6c9 278e9befba9b 2023-09-10T01:02:23.685 INFO:teuthology.orchestra.run.smithi003.stdout:mds.d smithi003 running (2m) - 2m - - 18.0.0-6025-g9670cca5 20c3861ef6c9 7fc775c1b52e 2023-09-10T01:02:23.686 INFO:teuthology.orchestra.run.smithi003.stdout:mds.e smithi092 running (2m) - 2m 21.2M - 18.0.0-6025-g9670cca5 20c3861ef6c9 131439bf2762 2023-09-10T01:02:23.686 INFO:teuthology.orchestra.run.smithi003.stdout:mds.f smithi136 running (2m) - 119s 16.6M - 18.0.0-6025-g9670cca5 20c3861ef6c9 9826d3dc44ba 2023-09-10T01:02:23.686 INFO:teuthology.orchestra.run.smithi003.stdout:mds.g smithi003 running (110s) - 110s - - 18.0.0-6025-g9670cca5 20c3861ef6c9 e8f6624d6bb3 2023-09-10T01:02:23.687 INFO:teuthology.orchestra.run.smithi003.stdout:mds.h smithi092 running (115s) - 115s 18.7M - 18.0.0-6025-g9670cca5 20c3861ef6c9 34e948f895e7 2023-09-10T01:02:23.687 INFO:teuthology.orchestra.run.smithi003.stdout:mds.i smithi136 running (113s) - 113s 20.9M - 18.0.0-6025-g9670cca5 20c3861ef6c9 d359797a3930 2023-09-10T01:02:23.687 INFO:teuthology.orchestra.run.smithi003.stdout:mds.j smithi003 running (2m) - 2m - - 18.0.0-6025-g9670cca5 20c3861ef6c9 83347b1b1a0c 2023-09-10T01:02:23.687 INFO:teuthology.orchestra.run.smithi003.stdout:mds.k smithi092 running (2m) - 2m 14.2M - 18.0.0-6025-g9670cca5 20c3861ef6c9 f3c15383c9c4 2023-09-10T01:02:23.688 INFO:teuthology.orchestra.run.smithi003.stdout:mds.l smithi136 running (2m) - 2m 15.1M - 18.0.0-6025-g9670cca5 20c3861ef6c9 3e6969447a8f 2023-09-10T01:02:23.688 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi003 *:9283,8765 running (9m) - 9m - - 18.0.0-6025-g9670cca5 20c3861ef6c9 50ba7b729829 2023-09-10T01:02:23.688 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi092 *:8443,8765 running (6m) - 6m 450M - 18.0.0-6025-g9670cca5 20c3861ef6c9 8b62e4f31d91 2023-09-10T01:02:23.688 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.z smithi136 *:8443,8765 running (6m) - 6m 449M - 18.0.0-6025-g9670cca5 20c3861ef6c9 27520df7ae58 2023-09-10T01:02:23.689 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (9m) - 9m - 2048M 18.0.0-6025-g9670cca5 20c3861ef6c9 939386b04bd1 2023-09-10T01:02:23.689 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi092 running (7m) - 7m 69.5M 2048M 18.0.0-6025-g9670cca5 20c3861ef6c9 f9e8feb9fa28 2023-09-10T01:02:23.689 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi136 running (7m) - 7m 68.2M 2048M 18.0.0-6025-g9670cca5 20c3861ef6c9 e313c55c3f67 2023-09-10T01:02:23.689 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (6m) - 6m - 4318M 18.0.0-6025-g9670cca5 20c3861ef6c9 f772de8cacc0 2023-09-10T01:02:23.690 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi092 running (6m) - 6m 79.5M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 9315b9acf057 2023-09-10T01:02:23.690 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi136 running (5m) - 5m 74.2M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 8d033e996727 2023-09-10T01:02:23.690 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (5m) - 5m - 4318M 18.0.0-6025-g9670cca5 20c3861ef6c9 9c42b4b48903 2023-09-10T01:02:23.691 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi092 running (5m) - 5m 63.3M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 dfdb92ecf11d 2023-09-10T01:02:23.691 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi136 running (4m) - 4m 68.5M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 00352a2790ab 2023-09-10T01:02:23.691 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi003 running (4m) - 4m - 4318M 18.0.0-6025-g9670cca5 20c3861ef6c9 ae31f2b294f4 2023-09-10T01:02:23.692 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi092 running (4m) - 4m 65.9M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 f0c7c0df65db 2023-09-10T01:02:23.692 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8 smithi136 running (3m) - 3m 61.9M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 8568d1f88e75 2023-09-10T01:02:23.692 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9 smithi003 running (3m) - 3m - 4318M 18.0.0-6025-g9670cca5 20c3861ef6c9 9388cfd4ef6a 2023-09-10T01:02:23.692 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10 smithi092 running (2m) - 2m 64.0M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 e1f508d51c19 2023-09-10T01:02:23.693 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11 smithi136 running (2m) - 2m 58.3M 4292M 18.0.0-6025-g9670cca5 20c3861ef6c9 d5125fe79c06 2023-09-10T01:02:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa2480a06d0 msgr2=0x7fa2480a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa2480a06d0 0x7fa2480a2b90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009c60 tx=0x7fa24c009260 comp rx=0 tx=0).stop 2023-09-10T01:02:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c100690 msgr2=0x7fa25c198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c100690 0x7fa25c198590 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa2580098e0 tx=0x7fa25800f690 comp rx=0 tx=0).stop 2023-09-10T01:02:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 shutdown_connections 2023-09-10T01:02:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa2480a06d0 0x7fa2480a2b90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa25c100690 0x7fa25c198590 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa25c102230 0x7fa25c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 --2- 172.21.15.3:0/1325845108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa25c101890 0x7fa25c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 >> 172.21.15.3:0/1325845108 conn(0x7fa25c0fbbf0 msgr2=0x7fa25c1071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 shutdown_connections 2023-09-10T01:02:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:23.685+0000 7fa26412b700 1 -- 172.21.15.3:0/1325845108 wait complete. 2023-09-10T01:02:24.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- bash -c 'ceph orch ls' 2023-09-10T01:02:24.929 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:25.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:25 smithi003 ceph-mon[97654]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:25.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:25 smithi003 ceph-mon[97654]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:25.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:25 smithi136 ceph-mon[133708]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:25.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:25 smithi136 ceph-mon[133708]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:25.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:25 smithi092 ceph-mon[134067]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:25.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:25 smithi092 ceph-mon[134067]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 -- 172.21.15.3:0/2627552478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4101890 msgr2=0x7faaf4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/2627552478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4101890 0x7faaf4101cf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7faae8009920 tx=0x7faae8009c30 comp rx=0 tx=0).stop 2023-09-10T01:02:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 -- 172.21.15.3:0/2627552478 shutdown_connections 2023-09-10T01:02:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/2627552478 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf4100690 0x7faaf4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/2627552478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaf4102230 0x7faaf4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/2627552478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4101890 0x7faaf4101cf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 -- 172.21.15.3:0/2627552478 >> 172.21.15.3:0/2627552478 conn(0x7faaf40fbbf0 msgr2=0x7faaf40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 -- 172.21.15.3:0/2627552478 shutdown_connections 2023-09-10T01:02:26.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.532+0000 7faaf8b6f700 1 -- 172.21.15.3:0/2627552478 wait complete. 2023-09-10T01:02:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 Processor -- start 2023-09-10T01:02:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 -- start start 2023-09-10T01:02:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaf4100690 0x7faaf4196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf4101890 0x7faaf4196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4102230 0x7faaf419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:26.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf419dfc0 con 0x7faaf4102230 2023-09-10T01:02:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf419e100 con 0x7faaf4100690 2023-09-10T01:02:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.533+0000 7faaf8b6f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf419e290 con 0x7faaf4101890 2023-09-10T01:02:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4102230 0x7faaf419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:26.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaf4100690 0x7faaf4196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4102230 0x7faaf419d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60366/0 (socket says 172.21.15.3:60366) 2023-09-10T01:02:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 -- 172.21.15.3:0/1505658056 learned_addr learned my addr 172.21.15.3:0/1505658056 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf1d9b700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf4101890 0x7faaf4196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 -- 172.21.15.3:0/1505658056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf4101890 msgr2=0x7faaf4196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:26.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf4101890 0x7faaf4196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 -- 172.21.15.3:0/1505658056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaf4100690 msgr2=0x7faaf4196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaf4100690 0x7faaf4196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 -- 172.21.15.3:0/1505658056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec009d00 con 0x7faaf4102230 2023-09-10T01:02:26.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.534+0000 7faaf2d9d700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4102230 0x7faaf419d9f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faaec00e3f0 tx=0x7faaec00e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:26.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.535+0000 7faae37fe700 1 -- 172.21.15.3:0/1505658056 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec0090d0 con 0x7faaf4102230 2023-09-10T01:02:26.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.535+0000 7faae37fe700 1 -- 172.21.15.3:0/1505658056 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faaec00f040 con 0x7faaf4102230 2023-09-10T01:02:26.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.535+0000 7faae37fe700 1 -- 172.21.15.3:0/1505658056 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaec007c00 con 0x7faaf4102230 2023-09-10T01:02:26.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.535+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae8009520 con 0x7faaf4102230 2023-09-10T01:02:26.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.535+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf419e8e0 con 0x7faaf4102230 2023-09-10T01:02:26.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.536+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf4107d60 con 0x7faaf4102230 2023-09-10T01:02:26.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.538+0000 7faae37fe700 1 -- 172.21.15.3:0/1505658056 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7faaec003a00 con 0x7faaf4102230 2023-09-10T01:02:26.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.539+0000 7faae37fe700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7faadc0a06c0 0x7faadc0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:26.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.539+0000 7faae37fe700 1 -- 172.21.15.3:0/1505658056 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7faaec0cbc70 con 0x7faaf4102230 2023-09-10T01:02:26.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.539+0000 7faaf259c700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7faadc0a06c0 0x7faadc0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:26.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.540+0000 7faaf259c700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7faadc0a06c0 0x7faadc0a2b80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7faaf41016f0 tx=0x7faae4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:26.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.543+0000 7faae37fe700 1 -- 172.21.15.3:0/1505658056 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7faaec095420 con 0x7faaf4102230 2023-09-10T01:02:26.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.760+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7faaf402cfd0 con 0x7faadc0a06c0 2023-09-10T01:02:26.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.766+0000 7faae37fe700 1 -- 172.21.15.3:0/1505658056 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7faaf402cfd0 con 0x7faadc0a06c0 2023-09-10T01:02:26.767 INFO:teuthology.orchestra.run.smithi003.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-09-10T01:02:26.767 INFO:teuthology.orchestra.run.smithi003.stdout:mds.all 12/12 - 2m smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2023-09-10T01:02:26.767 INFO:teuthology.orchestra.run.smithi003.stdout:mgr 3/3 - 7m smithi003=x;smithi092=y;smithi136=z;count:3 2023-09-10T01:02:26.767 INFO:teuthology.orchestra.run.smithi003.stdout:mon 3/3 - 7m smithi003:172.21.15.3=a;smithi092:172.21.15.92=b;smithi136:172.21.15.136=c;count:3 2023-09-10T01:02:26.768 INFO:teuthology.orchestra.run.smithi003.stdout:osd 12 - - 2023-09-10T01:02:26.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7faadc0a06c0 msgr2=0x7faadc0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:26.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7faadc0a06c0 0x7faadc0a2b80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7faaf41016f0 tx=0x7faae4009260 comp rx=0 tx=0).stop 2023-09-10T01:02:26.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4102230 msgr2=0x7faaf419d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:26.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4102230 0x7faaf419d9f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faaec00e3f0 tx=0x7faaec00e700 comp rx=0 tx=0).stop 2023-09-10T01:02:26.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 shutdown_connections 2023-09-10T01:02:26.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faaf4102230 0x7faaf419d9f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faaf4100690 0x7faaf4196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7faadc0a06c0 0x7faadc0a2b80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 --2- 172.21.15.3:0/1505658056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf4101890 0x7faaf4196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:26.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.768+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 >> 172.21.15.3:0/1505658056 conn(0x7faaf40fbbf0 msgr2=0x7faaf4104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:26.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.769+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 shutdown_connections 2023-09-10T01:02:26.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:26.769+0000 7faaf8b6f700 1 -- 172.21.15.3:0/1505658056 wait complete. 2023-09-10T01:02:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:27 smithi003 ceph-mon[97654]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:27.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:27 smithi003 ceph-mon[97654]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-09-10T01:02:27.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- bash -c 'ceph orch host ls' 2023-09-10T01:02:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:27 smithi136 ceph-mon[133708]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:27.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:27 smithi136 ceph-mon[133708]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-09-10T01:02:27.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:27 smithi092 ceph-mon[134067]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:27.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:27 smithi092 ceph-mon[134067]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-09-10T01:02:27.990 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:28.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:28 smithi003 ceph-mon[97654]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:28.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:28 smithi136 ceph-mon[133708]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:28 smithi092 ceph-mon[134067]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:29.266 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:29 smithi003 ceph-mon[97654]: pgmap v336: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:29.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:29 smithi136 ceph-mon[133708]: pgmap v336: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:29.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:29 smithi092 ceph-mon[134067]: pgmap v336: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.524+0000 7fb08684f700 1 -- 172.21.15.3:0/129593054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080108560 msgr2=0x7fb08010a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.524+0000 7fb08684f700 1 --2- 172.21.15.3:0/129593054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080108560 0x7fb08010a9f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb07800b0a0 tx=0x7fb07800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.524+0000 7fb08684f700 1 -- 172.21.15.3:0/129593054 shutdown_connections 2023-09-10T01:02:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.524+0000 7fb08684f700 1 --2- 172.21.15.3:0/129593054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb080074630 0x7fb080072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.525+0000 7fb08684f700 1 --2- 172.21.15.3:0/129593054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080108560 0x7fb08010a9f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.525+0000 7fb08684f700 1 --2- 172.21.15.3:0/129593054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb080073060 0x7fb0800734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.525+0000 7fb08684f700 1 -- 172.21.15.3:0/129593054 >> 172.21.15.3:0/129593054 conn(0x7fb0800fbbf0 msgr2=0x7fb0800fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:29.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.525+0000 7fb08684f700 1 -- 172.21.15.3:0/129593054 shutdown_connections 2023-09-10T01:02:29.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.525+0000 7fb08684f700 1 -- 172.21.15.3:0/129593054 wait complete. 2023-09-10T01:02:29.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 Processor -- start 2023-09-10T01:02:29.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 -- start start 2023-09-10T01:02:29.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080073060 0x7fb080198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:29.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb080074630 0x7fb080198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:29.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb080108560 0x7fb08019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:29.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0801a0210 con 0x7fb080073060 2023-09-10T01:02:29.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0801a0350 con 0x7fb080074630 2023-09-10T01:02:29.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb08684f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0801a04e0 con 0x7fb080108560 2023-09-10T01:02:29.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb07ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080073060 0x7fb080198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:29.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb07ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080073060 0x7fb080198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60374/0 (socket says 172.21.15.3:60374) 2023-09-10T01:02:29.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.526+0000 7fb07ffff700 1 -- 172.21.15.3:0/1216126474 learned_addr learned my addr 172.21.15.3:0/1216126474 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:29.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07f7fe700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb080074630 0x7fb080198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:29.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb084dec700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb080108560 0x7fb08019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:29.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07ffff700 1 -- 172.21.15.3:0/1216126474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb080108560 msgr2=0x7fb08019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:29.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07ffff700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb080108560 0x7fb08019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07ffff700 1 -- 172.21.15.3:0/1216126474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb080074630 msgr2=0x7fb080198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07ffff700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb080074630 0x7fb080198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07ffff700 1 -- 172.21.15.3:0/1216126474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb078009d00 con 0x7fb080073060 2023-09-10T01:02:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07ffff700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080073060 0x7fb080198680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb07000e9a0 tx=0x7fb07000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07d7fa700 1 -- 172.21.15.3:0/1216126474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070009790 con 0x7fb080073060 2023-09-10T01:02:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07d7fa700 1 -- 172.21.15.3:0/1216126474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb07000c810 con 0x7fb080073060 2023-09-10T01:02:29.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0801a07c0 con 0x7fb080073060 2023-09-10T01:02:29.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.527+0000 7fb07d7fa700 1 -- 172.21.15.3:0/1216126474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0700054c0 con 0x7fb080073060 2023-09-10T01:02:29.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.528+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0801a0cc0 con 0x7fb080073060 2023-09-10T01:02:29.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.529+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb080106c10 con 0x7fb080073060 2023-09-10T01:02:29.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.530+0000 7fb07d7fa700 1 -- 172.21.15.3:0/1216126474 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb07000cc80 con 0x7fb080073060 2023-09-10T01:02:29.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.532+0000 7fb07d7fa700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0680a0640 0x7fb0680a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:29.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.532+0000 7fb07d7fa700 1 -- 172.21.15.3:0/1216126474 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fb070014070 con 0x7fb080073060 2023-09-10T01:02:29.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.532+0000 7fb07f7fe700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0680a0640 0x7fb0680a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:29.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.533+0000 7fb07f7fe700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0680a0640 0x7fb0680a2b00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb080074360 tx=0x7fb074009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:29.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.536+0000 7fb07d7fa700 1 -- 172.21.15.3:0/1216126474 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb0700958e0 con 0x7fb080073060 2023-09-10T01:02:29.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.754+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb08002d010 con 0x7fb0680a0640 2023-09-10T01:02:29.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.756+0000 7fb07d7fa700 1 -- 172.21.15.3:0/1216126474 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fb08002d010 con 0x7fb0680a0640 2023-09-10T01:02:29.757 INFO:teuthology.orchestra.run.smithi003.stdout:HOST ADDR LABELS STATUS 2023-09-10T01:02:29.757 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 172.21.15.3 2023-09-10T01:02:29.757 INFO:teuthology.orchestra.run.smithi003.stdout:smithi092 172.21.15.92 2023-09-10T01:02:29.757 INFO:teuthology.orchestra.run.smithi003.stdout:smithi136 172.21.15.136 2023-09-10T01:02:29.758 INFO:teuthology.orchestra.run.smithi003.stdout:3 hosts in cluster 2023-09-10T01:02:29.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.758+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0680a0640 msgr2=0x7fb0680a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:29.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.758+0000 7fb08684f700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0680a0640 0x7fb0680a2b00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb080074360 tx=0x7fb074009260 comp rx=0 tx=0).stop 2023-09-10T01:02:29.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.758+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080073060 msgr2=0x7fb080198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:29.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.758+0000 7fb08684f700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080073060 0x7fb080198680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb07000e9a0 tx=0x7fb07000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 shutdown_connections 2023-09-10T01:02:29.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb0680a0640 0x7fb0680a2b00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb080108560 0x7fb08019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb080073060 0x7fb080198680 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 --2- 172.21.15.3:0/1216126474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb080074630 0x7fb080198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:29.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 >> 172.21.15.3:0/1216126474 conn(0x7fb0800fbbf0 msgr2=0x7fb0800fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:29.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 shutdown_connections 2023-09-10T01:02:29.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:29.759+0000 7fb08684f700 1 -- 172.21.15.3:0/1216126474 wait complete. 2023-09-10T01:02:30.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- bash -c 'ceph orch device ls' 2023-09-10T01:02:31.024 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:31.307 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:31 smithi003 ceph-mon[97654]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:31.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:31 smithi003 ceph-mon[97654]: pgmap v337: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:31 smithi136 ceph-mon[133708]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:31 smithi136 ceph-mon[133708]: pgmap v337: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:31.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:31 smithi092 ceph-mon[134067]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:31.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:31 smithi092 ceph-mon[134067]: pgmap v337: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:32.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 -- 172.21.15.3:0/3440865018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 msgr2=0x7f3924100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 --2- 172.21.15.3:0/3440865018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f3924100ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f39140099c0 tx=0x7f3914009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 -- 172.21.15.3:0/3440865018 shutdown_connections 2023-09-10T01:02:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 --2- 172.21.15.3:0/3440865018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3924102230 0x7f3924108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 --2- 172.21.15.3:0/3440865018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f3924100ab0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 --2- 172.21.15.3:0/3440865018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3924101890 0x7f3924101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 -- 172.21.15.3:0/3440865018 >> 172.21.15.3:0/3440865018 conn(0x7f39240fbbf0 msgr2=0x7f39240fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 -- 172.21.15.3:0/3440865018 shutdown_connections 2023-09-10T01:02:32.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.576+0000 7f3929780700 1 -- 172.21.15.3:0/3440865018 wait complete. 2023-09-10T01:02:32.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 Processor -- start 2023-09-10T01:02:32.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 -- start start 2023-09-10T01:02:32.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f39241985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:32.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3924101890 0x7f3924198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:32.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3924102230 0x7f392419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:32.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a0160 con 0x7f3924100690 2023-09-10T01:02:32.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a02a0 con 0x7f3924102230 2023-09-10T01:02:32.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.578+0000 7f3929780700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a0430 con 0x7f3924101890 2023-09-10T01:02:32.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f39241985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:32.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f39241985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60386/0 (socket says 172.21.15.3:60386) 2023-09-10T01:02:32.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 -- 172.21.15.3:0/4245648831 learned_addr learned my addr 172.21.15.3:0/4245648831 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:32.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f39237fe700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3924102230 0x7f392419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:32.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f39227fc700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3924101890 0x7f3924198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:32.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 -- 172.21.15.3:0/4245648831 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3924101890 msgr2=0x7f3924198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:32.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3924101890 0x7f3924198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 -- 172.21.15.3:0/4245648831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3924102230 msgr2=0x7f392419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:32.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3924102230 0x7f392419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 -- 172.21.15.3:0/4245648831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39140095f0 con 0x7f3924100690 2023-09-10T01:02:32.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.579+0000 7f3922ffd700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f39241985d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f39140098e0 tx=0x7f391400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:32.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.580+0000 7f390bfff700 1 -- 172.21.15.3:0/4245648831 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391401d070 con 0x7f3924100690 2023-09-10T01:02:32.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.580+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39241a06b0 con 0x7f3924100690 2023-09-10T01:02:32.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.580+0000 7f390bfff700 1 -- 172.21.15.3:0/4245648831 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3914005050 con 0x7f3924100690 2023-09-10T01:02:32.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.580+0000 7f390bfff700 1 -- 172.21.15.3:0/4245648831 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914004a70 con 0x7f3924100690 2023-09-10T01:02:32.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.580+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39241a0ad0 con 0x7f3924100690 2023-09-10T01:02:32.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.581+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924106260 con 0x7f3924100690 2023-09-10T01:02:32.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.583+0000 7f390bfff700 1 -- 172.21.15.3:0/4245648831 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f39140051c0 con 0x7f3924100690 2023-09-10T01:02:32.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.584+0000 7f390bfff700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f390c0a0640 0x7f390c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:32.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.584+0000 7f390bfff700 1 -- 172.21.15.3:0/4245648831 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f39140ccda0 con 0x7f3924100690 2023-09-10T01:02:32.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.584+0000 7f39227fc700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f390c0a0640 0x7f390c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:32.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.585+0000 7f39227fc700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f390c0a0640 0x7f390c0a2b00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3918009c60 tx=0x7f3918009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:32.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.588+0000 7f390bfff700 1 -- 172.21.15.3:0/4245648831 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3914096550 con 0x7f3924100690 2023-09-10T01:02:32.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.803+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f392402cf50 con 0x7f390c0a0640 2023-09-10T01:02:32.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.805+0000 7f390bfff700 1 -- 172.21.15.3:0/4245648831 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f392402cf50 con 0x7f390c0a0640 2023-09-10T01:02:32.806 INFO:teuthology.orchestra.run.smithi003.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-09-10T01:02:32.806 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009Z400BGN 372G No 3m ago Has a FileSystem, LVM detected 2023-09-10T01:02:32.806 INFO:teuthology.orchestra.run.smithi003.stdout:smithi092 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QM400BGN 372G No 2m ago Has a FileSystem, LVM detected 2023-09-10T01:02:32.806 INFO:teuthology.orchestra.run.smithi003.stdout:smithi136 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD514100122P0SGN 1863G No 2m ago Has a FileSystem, LVM detected 2023-09-10T01:02:32.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f390c0a0640 msgr2=0x7f390c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:32.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f390c0a0640 0x7f390c0a2b00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3918009c60 tx=0x7f3918009260 comp rx=0 tx=0).stop 2023-09-10T01:02:32.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 msgr2=0x7f39241985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:32.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f39241985d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f39140098e0 tx=0x7f391400f690 comp rx=0 tx=0).stop 2023-09-10T01:02:32.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 shutdown_connections 2023-09-10T01:02:32.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3924102230 0x7f392419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3924100690 0x7f39241985d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f390c0a0640 0x7f390c0a2b00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 --2- 172.21.15.3:0/4245648831 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3924101890 0x7f3924198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 >> 172.21.15.3:0/4245648831 conn(0x7f39240fbbf0 msgr2=0x7f39240fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 shutdown_connections 2023-09-10T01:02:32.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:32.807+0000 7f3929780700 1 -- 172.21.15.3:0/4245648831 wait complete. 2023-09-10T01:02:33.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:33 smithi003 ceph-mon[97654]: pgmap v338: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:33.320 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-09-10T01:02:33.331 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:02:33.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- bash -c 'ceph fs dump' 2023-09-10T01:02:33.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:33 smithi136 ceph-mon[133708]: pgmap v338: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:33.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:33 smithi092 ceph-mon[134067]: pgmap v338: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:34.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:34 smithi003 ceph-mon[97654]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:34 smithi136 ceph-mon[133708]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:34 smithi092 ceph-mon[134067]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:02:35.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:35 smithi136 ceph-mon[133708]: pgmap v339: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:35.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:35 smithi003 ceph-mon[97654]: pgmap v339: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:35.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:35 smithi092 ceph-mon[134067]: pgmap v339: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:35.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 -- 172.21.15.3:0/421883270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 msgr2=0x7f1da8072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:35.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 --2- 172.21.15.3:0/421883270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 0x7f1da8072a10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009990 tx=0x7f1d98009ca0 comp rx=0 tx=0).stop 2023-09-10T01:02:35.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 -- 172.21.15.3:0/421883270 shutdown_connections 2023-09-10T01:02:35.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 --2- 172.21.15.3:0/421883270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8108590 0x7f1da810aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:35.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 --2- 172.21.15.3:0/421883270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 0x7f1da8072a10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:35.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 --2- 172.21.15.3:0/421883270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8072fe0 0x7f1da8073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:35.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 -- 172.21.15.3:0/421883270 >> 172.21.15.3:0/421883270 conn(0x7f1da80fbbf0 msgr2=0x7f1da80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:35.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 -- 172.21.15.3:0/421883270 shutdown_connections 2023-09-10T01:02:35.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.731+0000 7f1dadbce700 1 -- 172.21.15.3:0/421883270 wait complete. 2023-09-10T01:02:35.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.732+0000 7f1dadbce700 1 Processor -- start 2023-09-10T01:02:35.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.732+0000 7f1dadbce700 1 -- start start 2023-09-10T01:02:35.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.732+0000 7f1dadbce700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8072fe0 0x7f1da8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:35.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.732+0000 7f1dadbce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 0x7f1da8198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:35.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1dadbce700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8108590 0x7f1da819fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:35.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1dadbce700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da81a0100 con 0x7f1da80745b0 2023-09-10T01:02:35.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1dadbce700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da81a0240 con 0x7f1da8072fe0 2023-09-10T01:02:35.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1dadbce700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da81a03d0 con 0x7f1da8108590 2023-09-10T01:02:35.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8072fe0 0x7f1da8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:35.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 0x7f1da8198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:35.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8072fe0 0x7f1da8198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54452/0 (socket says 172.21.15.3:54452) 2023-09-10T01:02:35.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 -- 172.21.15.3:0/1338767649 learned_addr learned my addr 172.21.15.3:0/1338767649 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:35.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da7fff700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8108590 0x7f1da819fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:35.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 -- 172.21.15.3:0/1338767649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8108590 msgr2=0x7f1da819fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:35.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8108590 0x7f1da819fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:35.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 -- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 msgr2=0x7f1da8198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:35.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 0x7f1da8198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:35.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 -- 172.21.15.3:0/1338767649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d980095f0 con 0x7f1da8072fe0 2023-09-10T01:02:35.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.733+0000 7f1da77fe700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8072fe0 0x7f1da8198570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800b680 tx=0x7f1d980057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:35.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.734+0000 7f1da4ff9700 1 -- 172.21.15.3:0/1338767649 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9801d070 con 0x7f1da8072fe0 2023-09-10T01:02:35.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.734+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da81a0650 con 0x7f1da8072fe0 2023-09-10T01:02:35.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.734+0000 7f1da4ff9700 1 -- 172.21.15.3:0/1338767649 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d9800fbc0 con 0x7f1da8072fe0 2023-09-10T01:02:35.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.734+0000 7f1da4ff9700 1 -- 172.21.15.3:0/1338767649 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d98017650 con 0x7f1da8072fe0 2023-09-10T01:02:35.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.734+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da81a0ac0 con 0x7f1da8072fe0 2023-09-10T01:02:35.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.735+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da8106c90 con 0x7f1da8072fe0 2023-09-10T01:02:35.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.737+0000 7f1da4ff9700 1 -- 172.21.15.3:0/1338767649 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f1d98022470 con 0x7f1da8072fe0 2023-09-10T01:02:35.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.738+0000 7f1da4ff9700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1d900a0640 0x7f1d900a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:35.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.738+0000 7f1da4ff9700 1 -- 172.21.15.3:0/1338767649 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f1d980cdd70 con 0x7f1da8072fe0 2023-09-10T01:02:35.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.739+0000 7f1da6ffd700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1d900a0640 0x7f1d900a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:35.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.739+0000 7f1da6ffd700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1d900a0640 0x7f1d900a2b00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c005950 tx=0x7f1d9c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:35.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.740+0000 7f1da4ff9700 1 -- 172.21.15.3:0/1338767649 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1d98097520 con 0x7f1da8072fe0 2023-09-10T01:02:35.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.996+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1da810e1b0 con 0x7f1da8072fe0 2023-09-10T01:02:35.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:35.997+0000 7f1da4ff9700 1 -- 172.21.15.3:0/1338767649 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2925 (secure 0 0 0) 0x7f1d98027b80 con 0x7f1da8072fe0 2023-09-10T01:02:35.998 INFO:teuthology.orchestra.run.smithi003.stdout:e22 2023-09-10T01:02:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-09-10T01:02:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-09-10T01:02:35.999 INFO:teuthology.orchestra.run.smithi003.stdout:legacy client fscid: 1 2023-09-10T01:02:35.999 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:Filesystem 'cephfs' (1) 2023-09-10T01:02:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:fs_name cephfs 2023-09-10T01:02:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 22 2023-09-10T01:02:36.000 INFO:teuthology.orchestra.run.smithi003.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-09-10T01:02:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:created 2023-09-10T01:00:43.531603+0000 2023-09-10T01:02:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2023-09-10T01:01:09.940177+0000 2023-09-10T01:02:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:tableserver 0 2023-09-10T01:02:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:root 0 2023-09-10T01:02:36.001 INFO:teuthology.orchestra.run.smithi003.stdout:session_timeout 300 2023-09-10T01:02:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:session_autoclose 300 2023-09-10T01:02:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:max_file_size 1099511627776 2023-09-10T01:02:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:max_xattr_size 65536 2023-09-10T01:02:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:required_client_features {} 2023-09-10T01:02:36.002 INFO:teuthology.orchestra.run.smithi003.stdout:last_failure 0 2023-09-10T01:02:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:last_failure_osd_epoch 0 2023-09-10T01:02:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-09-10T01:02:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:max_mds 3 2023-09-10T01:02:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:in 0,1,2 2023-09-10T01:02:36.003 INFO:teuthology.orchestra.run.smithi003.stdout:up {0=24698,1=24647,2=24634} 2023-09-10T01:02:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:failed 2023-09-10T01:02:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:damaged 2023-09-10T01:02:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:stopped 2023-09-10T01:02:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:data_pools [3,4] 2023-09-10T01:02:36.004 INFO:teuthology.orchestra.run.smithi003.stdout:metadata_pool 2 2023-09-10T01:02:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:inline_data disabled 2023-09-10T01:02:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:balancer 2023-09-10T01:02:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:bal_rank_mask -1 2023-09-10T01:02:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:standby_count_wanted 0 2023-09-10T01:02:36.005 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b{0:24698} state up:active seq 3 addr [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.e{0:24620} state up:standby-replay seq 1 addr [v2:172.21.15.92:6834/790897161,v1:172.21.15.92:6835/790897161] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.h{1:24647} state up:active seq 11 addr [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c{1:24610} state up:standby-replay seq 1 addr [v2:172.21.15.136:6836/3841438021,v1:172.21.15.136:6837/3841438021] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.i{2:24634} state up:active seq 11 addr [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.006 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.f{2:24604} state up:standby-replay seq 1 addr [v2:172.21.15.136:6834/662717057,v1:172.21.15.136:6835/662717057] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.007 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:36.007 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:Standby daemons: 2023-09-10T01:02:36.007 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.j{-1:14574} state up:standby seq 1 addr [v2:172.21.15.3:6834/2642590493,v1:172.21.15.3:6835/2642590493] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.007 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d{-1:14586} state up:standby seq 1 addr [v2:172.21.15.3:6836/3588843640,v1:172.21.15.3:6837/3588843640] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.g{-1:14631} state up:standby seq 1 addr [v2:172.21.15.3:6838/640397108,v1:172.21.15.3:6839/640397108] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a{-1:14649} state up:standby seq 1 addr [v2:172.21.15.3:6840/1239897501,v1:172.21.15.3:6841/1239897501] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.l{-1:24562} state up:standby seq 1 addr [v2:172.21.15.136:6832/3536147716,v1:172.21.15.136:6833/3536147716] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.008 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.k{-1:24581} state up:standby seq 1 addr [v2:172.21.15.92:6832/1626913605,v1:172.21.15.92:6833/1626913605] compat {c=[1],r=[1],i=[fff]}] 2023-09-10T01:02:36.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1d900a0640 msgr2=0x7f1d900a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:36.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1d900a0640 0x7f1d900a2b00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c005950 tx=0x7f1d9c00b410 comp rx=0 tx=0).stop 2023-09-10T01:02:36.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8072fe0 msgr2=0x7f1da8198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:36.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8072fe0 0x7f1da8198570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800b680 tx=0x7f1d980057b0 comp rx=0 tx=0).stop 2023-09-10T01:02:36.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 shutdown_connections 2023-09-10T01:02:36.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1d900a0640 0x7f1d900a2b00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:36.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8108590 0x7f1da819fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:36.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1da8072fe0 0x7f1da8198570 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:36.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 --2- 172.21.15.3:0/1338767649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1da80745b0 0x7f1da8198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:36.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 >> 172.21.15.3:0/1338767649 conn(0x7f1da80fbbf0 msgr2=0x7f1da80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:36.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 shutdown_connections 2023-09-10T01:02:36.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:36.000+0000 7f1dadbce700 1 -- 172.21.15.3:0/1338767649 wait complete. 2023-09-10T01:02:36.012 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:02:36.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:36 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1338767649' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-09-10T01:02:36.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1338767649' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-09-10T01:02:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:36 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1338767649' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-09-10T01:02:36.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- bash -c 'ceph osd dump' 2023-09-10T01:02:37.304 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:37.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:37 smithi136 ceph-mon[133708]: pgmap v340: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:37.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:37 smithi092 ceph-mon[134067]: pgmap v340: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:37.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:37 smithi003 ceph-mon[97654]: pgmap v340: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-09-10T01:02:38.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:38 smithi003 ceph-mon[97654]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:38 smithi136 ceph-mon[133708]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:38 smithi092 ceph-mon[134067]: pgmap v341: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-09-10T01:02:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 -- 172.21.15.3:0/50208252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18101890 msgr2=0x7fee18101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 --2- 172.21.15.3:0/50208252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18101890 0x7fee18101cf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0099c0 tx=0x7fee0c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:38.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 -- 172.21.15.3:0/50208252 shutdown_connections 2023-09-10T01:02:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 --2- 172.21.15.3:0/50208252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee18102230 0x7fee18108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 --2- 172.21.15.3:0/50208252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee18100690 0x7fee18100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 --2- 172.21.15.3:0/50208252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18101890 0x7fee18101cf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:38.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 -- 172.21.15.3:0/50208252 >> 172.21.15.3:0/50208252 conn(0x7fee180fbbf0 msgr2=0x7fee180fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 -- 172.21.15.3:0/50208252 shutdown_connections 2023-09-10T01:02:38.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.840+0000 7fee1d925700 1 -- 172.21.15.3:0/50208252 wait complete. 2023-09-10T01:02:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.841+0000 7fee1d925700 1 Processor -- start 2023-09-10T01:02:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.841+0000 7fee1d925700 1 -- start start 2023-09-10T01:02:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.841+0000 7fee1d925700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18100690 0x7fee18198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:38.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.841+0000 7fee1d925700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee18101890 0x7fee18198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee1d925700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee18102230 0x7fee1819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee1d925700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee181a0120 con 0x7fee18100690 2023-09-10T01:02:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee1d925700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee181a0260 con 0x7fee18101890 2023-09-10T01:02:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee1d925700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee181a03f0 con 0x7fee18102230 2023-09-10T01:02:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee16ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18100690 0x7fee18198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee18101890 0x7fee18198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee16ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18100690 0x7fee18198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34076/0 (socket says 172.21.15.3:34076) 2023-09-10T01:02:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee16ffd700 1 -- 172.21.15.3:0/3546102836 learned_addr learned my addr 172.21.15.3:0/3546102836 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee177fe700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee18102230 0x7fee1819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:38.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee167fc700 1 -- 172.21.15.3:0/3546102836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee18102230 msgr2=0x7fee1819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee167fc700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee18102230 0x7fee1819fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee167fc700 1 -- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18100690 msgr2=0x7fee18198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee167fc700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18100690 0x7fee18198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:38.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee167fc700 1 -- 172.21.15.3:0/3546102836 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0095f0 con 0x7fee18101890 2023-09-10T01:02:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.842+0000 7fee177fe700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee18102230 0x7fee1819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.843+0000 7fee167fc700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee18101890 0x7fee18198ad0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0098e0 tx=0x7fee0c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.843+0000 7fee1c923700 1 -- 172.21.15.3:0/3546102836 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c01d070 con 0x7fee18101890 2023-09-10T01:02:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.843+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee181a0670 con 0x7fee18101890 2023-09-10T01:02:38.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.843+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee181a0ae0 con 0x7fee18101890 2023-09-10T01:02:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.843+0000 7fee1c923700 1 -- 172.21.15.3:0/3546102836 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee0c0054f0 con 0x7fee18101890 2023-09-10T01:02:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.843+0000 7fee1c923700 1 -- 172.21.15.3:0/3546102836 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c004b30 con 0x7fee18101890 2023-09-10T01:02:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.844+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1810bee0 con 0x7fee18101890 2023-09-10T01:02:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.846+0000 7fee1c923700 1 -- 172.21.15.3:0/3546102836 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fee0c00fb80 con 0x7fee18101890 2023-09-10T01:02:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.847+0000 7fee1c923700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fee000a06c0 0x7fee000a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.848+0000 7fee1c923700 1 -- 172.21.15.3:0/3546102836 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fee0c0ccdd0 con 0x7fee18101890 2023-09-10T01:02:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.848+0000 7fee16ffd700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fee000a06c0 0x7fee000a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.848+0000 7fee16ffd700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fee000a06c0 0x7fee000a2b80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fee08005950 tx=0x7fee08009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:38.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:38.850+0000 7fee1c923700 1 -- 172.21.15.3:0/3546102836 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fee0c096580 con 0x7fee18101890 2023-09-10T01:02:39.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.061+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fee1810c0d0 con 0x7fee18101890 2023-09-10T01:02:39.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.062+0000 7fee1c923700 1 -- 172.21.15.3:0/3546102836 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) v1 ==== 56+0+5692 (secure 0 0 0) 0x7fee0c099ba0 con 0x7fee18101890 2023-09-10T01:02:39.063 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 90 2023-09-10T01:02:39.064 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T01:02:39.064 INFO:teuthology.orchestra.run.smithi003.stdout:created 2023-09-10T00:52:53.067114+0000 2023-09-10T01:02:39.064 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2023-09-10T01:01:57.028004+0000 2023-09-10T01:02:39.064 INFO:teuthology.orchestra.run.smithi003.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-09-10T01:02:39.065 INFO:teuthology.orchestra.run.smithi003.stdout:crush_version 27 2023-09-10T01:02:39.065 INFO:teuthology.orchestra.run.smithi003.stdout:full_ratio 0.95 2023-09-10T01:02:39.065 INFO:teuthology.orchestra.run.smithi003.stdout:backfillfull_ratio 0.9 2023-09-10T01:02:39.065 INFO:teuthology.orchestra.run.smithi003.stdout:nearfull_ratio 0.85 2023-09-10T01:02:39.065 INFO:teuthology.orchestra.run.smithi003.stdout:require_min_compat_client luminous 2023-09-10T01:02:39.066 INFO:teuthology.orchestra.run.smithi003.stdout:min_compat_client luminous 2023-09-10T01:02:39.066 INFO:teuthology.orchestra.run.smithi003.stdout:require_osd_release reef 2023-09-10T01:02:39.066 INFO:teuthology.orchestra.run.smithi003.stdout:stretch_mode_enabled false 2023-09-10T01:02:39.066 INFO:teuthology.orchestra.run.smithi003.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-09-10T01:02:39.066 INFO:teuthology.orchestra.run.smithi003.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2023-09-10T01:02:39.067 INFO:teuthology.orchestra.run.smithi003.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-09-10T01:02:39.067 INFO:teuthology.orchestra.run.smithi003.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-09-10T01:02:39.067 INFO:teuthology.orchestra.run.smithi003.stdout:max_osd 12 2023-09-10T01:02:39.067 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6802/3367767236,v1:172.21.15.3:6803/3367767236] [v2:172.21.15.3:6804/3367767236,v1:172.21.15.3:6805/3367767236] exists,up aaca1361-1d53-4fff-b909-d973a1ce185a 2023-09-10T01:02:39.067 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6800/4007866052,v1:172.21.15.92:6801/4007866052] [v2:172.21.15.92:6802/4007866052,v1:172.21.15.92:6803/4007866052] exists,up e1f77daa-c19a-4eba-aebf-27ca13859ee5 2023-09-10T01:02:39.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/781257073,v1:172.21.15.136:6801/781257073] [v2:172.21.15.136:6802/781257073,v1:172.21.15.136:6803/781257073] exists,up 2c7848b8-ced5-42a8-b5e1-59a572c3fca3 2023-09-10T01:02:39.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6810/434967290,v1:172.21.15.3:6811/434967290] [v2:172.21.15.3:6812/434967290,v1:172.21.15.3:6813/434967290] exists,up c08fe6de-b501-43d2-bcf3-2ce57eb464f9 2023-09-10T01:02:39.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 up in weight 1 up_from 32 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6808/2892874326,v1:172.21.15.92:6809/2892874326] [v2:172.21.15.92:6810/2892874326,v1:172.21.15.92:6811/2892874326] exists,up b1cf565e-0d0a-49ec-b39b-6dbee04e586f 2023-09-10T01:02:39.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 up in weight 1 up_from 37 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6808/1094245369,v1:172.21.15.136:6809/1094245369] [v2:172.21.15.136:6810/1094245369,v1:172.21.15.136:6811/1094245369] exists,up eb4bd187-8a27-479f-a3c3-018993c8d986 2023-09-10T01:02:39.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 up in weight 1 up_from 45 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6818/989795778,v1:172.21.15.3:6819/989795778] [v2:172.21.15.3:6820/989795778,v1:172.21.15.3:6821/989795778] exists,up 6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1 2023-09-10T01:02:39.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 up in weight 1 up_from 50 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6816/4114652830,v1:172.21.15.92:6817/4114652830] [v2:172.21.15.92:6818/4114652830,v1:172.21.15.92:6819/4114652830] exists,up 23003010-ab55-4574-8464-0f56aec3ddbd 2023-09-10T01:02:39.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8 up in weight 1 up_from 57 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/1553122381,v1:172.21.15.136:6817/1553122381] [v2:172.21.15.136:6818/1553122381,v1:172.21.15.136:6819/1553122381] exists,up 77f3d09f-2450-4861-9c06-172488429b52 2023-09-10T01:02:39.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9 up in weight 1 up_from 65 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6826/2161930524,v1:172.21.15.3:6827/2161930524] [v2:172.21.15.3:6828/2161930524,v1:172.21.15.3:6829/2161930524] exists,up 8b3bd058-2c37-4910-b1b9-661e888ef9aa 2023-09-10T01:02:39.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10 up in weight 1 up_from 70 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6824/2881136596,v1:172.21.15.92:6825/2881136596] [v2:172.21.15.92:6826/2881136596,v1:172.21.15.92:6827/2881136596] exists,up b312bb63-21fa-4f9e-b676-c0cfca04749a 2023-09-10T01:02:39.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11 up in weight 1 up_from 77 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/2854427876,v1:172.21.15.136:6825/2854427876] [v2:172.21.15.136:6826/2854427876,v1:172.21.15.136:6827/2854427876] exists,up 6d479a7b-430c-4707-89c8-d20e7b47d68e 2023-09-10T01:02:39.070 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 2.0 [8,4] 2023-09-10T01:02:39.070 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 2.8 [2,11] 2023-09-10T01:02:39.070 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 2.1b [2,4] 2023-09-10T01:02:39.070 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 2.1f [7,11] 2023-09-10T01:02:39.070 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.0 [4,8,5,9] 2023-09-10T01:02:39.071 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.6 [3,8] 2023-09-10T01:02:39.071 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.11 [4,6] 2023-09-10T01:02:39.071 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.17 [4,6,5,7] 2023-09-10T01:02:39.071 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.23 [4,7] 2023-09-10T01:02:39.072 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.26 [1,9] 2023-09-10T01:02:39.072 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.27 [3,9] 2023-09-10T01:02:39.072 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.6 [9,11] 2023-09-10T01:02:39.072 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.10 [9,8] 2023-09-10T01:02:39.072 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.11 [5,10] 2023-09-10T01:02:39.072 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.14 [9,8] 2023-09-10T01:02:39.073 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/1916209618 expires 2023-09-11T00:53:55.752989+0000 2023-09-10T01:02:39.073 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/70910038 expires 2023-09-11T00:53:28.332636+0000 2023-09-10T01:02:39.073 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/71687535 expires 2023-09-11T00:53:28.332636+0000 2023-09-10T01:02:39.073 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3193301911 expires 2023-09-11T00:53:28.332636+0000 2023-09-10T01:02:39.073 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1972995090 expires 2023-09-11T00:53:55.752989+0000 2023-09-10T01:02:39.074 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/4153284543 expires 2023-09-11T00:53:28.332636+0000 2023-09-10T01:02:39.074 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1689603322 expires 2023-09-11T00:53:55.752989+0000 2023-09-10T01:02:39.074 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/1916209618 expires 2023-09-11T00:53:55.752989+0000 2023-09-10T01:02:39.074 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/70910038 expires 2023-09-11T00:53:28.332636+0000 2023-09-10T01:02:39.075 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/886409476 expires 2023-09-11T00:53:55.752989+0000 2023-09-10T01:02:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fee000a06c0 msgr2=0x7fee000a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fee000a06c0 0x7fee000a2b80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fee08005950 tx=0x7fee08009260 comp rx=0 tx=0).stop 2023-09-10T01:02:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee18101890 msgr2=0x7fee18198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:39.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee18101890 0x7fee18198ad0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0098e0 tx=0x7fee0c00f690 comp rx=0 tx=0).stop 2023-09-10T01:02:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 shutdown_connections 2023-09-10T01:02:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fee000a06c0 0x7fee000a2b80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee18102230 0x7fee1819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee18100690 0x7fee18198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 --2- 172.21.15.3:0/3546102836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee18101890 0x7fee18198ad0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 >> 172.21.15.3:0/3546102836 conn(0x7fee180fbbf0 msgr2=0x7fee180fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 shutdown_connections 2023-09-10T01:02:39.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.065+0000 7fee1d925700 1 -- 172.21.15.3:0/3546102836 wait complete. 2023-09-10T01:02:39.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:39 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3546102836' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-09-10T01:02:39.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:39 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3546102836' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-09-10T01:02:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:39 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3546102836' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-09-10T01:02:39.600 INFO:teuthology.run_tasks:Running task fs.ready... 2023-09-10T01:02:39.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.838+0000 7f0feb9ff700 1 -- 172.21.15.3:0/4046781504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41018c0 msgr2=0x7f0fe4101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.838+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/4046781504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41018c0 0x7f0fe4101d20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009920 tx=0x7f0fd4009c30 comp rx=0 tx=0).stop 2023-09-10T01:02:39.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.839+0000 7f0feb9ff700 1 -- 172.21.15.3:0/4046781504 shutdown_connections 2023-09-10T01:02:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.839+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/4046781504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe4102260 0x7f0fe4108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.839+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/4046781504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fe41006d0 0x7f0fe4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.839+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/4046781504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41018c0 0x7f0fe4101d20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.839+0000 7f0feb9ff700 1 -- 172.21.15.3:0/4046781504 >> 172.21.15.3:0/4046781504 conn(0x7f0fe40fbc60 msgr2=0x7f0fe40fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.839+0000 7f0feb9ff700 1 -- 172.21.15.3:0/4046781504 shutdown_connections 2023-09-10T01:02:39.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.839+0000 7f0feb9ff700 1 -- 172.21.15.3:0/4046781504 wait complete. 2023-09-10T01:02:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 Processor -- start 2023-09-10T01:02:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 -- start start 2023-09-10T01:02:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41006d0 0x7f0fe4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe41018c0 0x7f0fe4198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fe4102260 0x7f0fe419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41a01a0 con 0x7f0fe4102260 2023-09-10T01:02:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41a02e0 con 0x7f0fe41006d0 2023-09-10T01:02:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.840+0000 7f0feb9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41a0470 con 0x7f0fe41018c0 2023-09-10T01:02:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe41018c0 0x7f0fe4198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe979b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41006d0 0x7f0fe4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe41018c0 0x7f0fe4198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:53764/0 (socket says 172.21.15.3:53764) 2023-09-10T01:02:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe9f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fe4102260 0x7f0fe419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 -- 172.21.15.3:0/1499891744 learned_addr learned my addr 172.21.15.3:0/1499891744 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 -- 172.21.15.3:0/1499891744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41006d0 msgr2=0x7f0fe4198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41006d0 0x7f0fe4198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 -- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fe4102260 msgr2=0x7f0fe419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:39.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fe4102260 0x7f0fe419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 -- 172.21.15.3:0/1499891744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe00095f0 con 0x7f0fe41018c0 2023-09-10T01:02:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe979b700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41006d0 0x7f0fe4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe9f9c700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fe4102260 0x7f0fe419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.841+0000 7f0fe8f9a700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe41018c0 0x7f0fe4198b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009230 tx=0x7f0fd400bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.842+0000 7f0fda7fc700 1 -- 172.21.15.3:0/1499891744 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd401c070 con 0x7f0fe41018c0 2023-09-10T01:02:39.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.842+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd4009520 con 0x7f0fe41018c0 2023-09-10T01:02:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.842+0000 7f0fda7fc700 1 -- 172.21.15.3:0/1499891744 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fd400bba0 con 0x7f0fe41018c0 2023-09-10T01:02:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.842+0000 7f0fda7fc700 1 -- 172.21.15.3:0/1499891744 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4003a40 con 0x7f0fe41018c0 2023-09-10T01:02:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.842+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe41a09e0 con 0x7f0fe41018c0 2023-09-10T01:02:39.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.843+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe410bee0 con 0x7f0fe41018c0 2023-09-10T01:02:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.845+0000 7f0fda7fc700 1 -- 172.21.15.3:0/1499891744 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f0fd4020030 con 0x7f0fe41018c0 2023-09-10T01:02:39.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.846+0000 7f0fda7fc700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0fd00a0640 0x7f0fd00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.846+0000 7f0fda7fc700 1 -- 172.21.15.3:0/1499891744 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f0fd40cd480 con 0x7f0fe41018c0 2023-09-10T01:02:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.847+0000 7f0fe979b700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0fd00a0640 0x7f0fd00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.847+0000 7f0fe979b700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0fd00a0640 0x7f0fd00a2b00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4101720 tx=0x7f0fe0009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:39.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:39.849+0000 7f0fda7fc700 1 -- 172.21.15.3:0/1499891744 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0fd4096c30 con 0x7f0fe41018c0 2023-09-10T01:02:40.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.063+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fe41a0eb0 con 0x7f0fe41018c0 2023-09-10T01:02:40.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.065+0000 7f0fda7fc700 1 -- 172.21.15.3:0/1499891744 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7f0fd4005040 con 0x7f0fe41018c0 2023-09-10T01:02:40.067 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:40.067 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:40.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0fd00a0640 msgr2=0x7f0fd00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0fd00a0640 0x7f0fd00a2b00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4101720 tx=0x7f0fe0009310 comp rx=0 tx=0).stop 2023-09-10T01:02:40.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe41018c0 msgr2=0x7f0fe4198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe41018c0 0x7f0fe4198b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009230 tx=0x7f0fd400bf80 comp rx=0 tx=0).stop 2023-09-10T01:02:40.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 shutdown_connections 2023-09-10T01:02:40.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fe4102260 0x7f0fe419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fe41006d0 0x7f0fe4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0fd00a0640 0x7f0fd00a2b00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 --2- 172.21.15.3:0/1499891744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe41018c0 0x7f0fe4198b60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 >> 172.21.15.3:0/1499891744 conn(0x7f0fe40fbc60 msgr2=0x7f0fe40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:40.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 shutdown_connections 2023-09-10T01:02:40.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.068+0000 7f0feb9ff700 1 -- 172.21.15.3:0/1499891744 wait complete. 2023-09-10T01:02:40.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:02:40.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:02:40.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.297+0000 7f33767ff700 1 -- 172.21.15.3:0/3763885272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33701018d0 msgr2=0x7f3370101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.297+0000 7f33767ff700 1 --2- 172.21.15.3:0/3763885272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33701018d0 0x7f3370101d30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f33640099c0 tx=0x7f3364009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.298+0000 7f33767ff700 1 -- 172.21.15.3:0/3763885272 shutdown_connections 2023-09-10T01:02:40.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.298+0000 7f33767ff700 1 --2- 172.21.15.3:0/3763885272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33701006e0 0x7f3370100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.298+0000 7f33767ff700 1 --2- 172.21.15.3:0/3763885272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3370102270 0x7f3370108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.298+0000 7f33767ff700 1 --2- 172.21.15.3:0/3763885272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33701018d0 0x7f3370101d30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.298+0000 7f33767ff700 1 -- 172.21.15.3:0/3763885272 >> 172.21.15.3:0/3763885272 conn(0x7f33700fbc50 msgr2=0x7f33700fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:40.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.298+0000 7f33767ff700 1 -- 172.21.15.3:0/3763885272 shutdown_connections 2023-09-10T01:02:40.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.298+0000 7f33767ff700 1 -- 172.21.15.3:0/3763885272 wait complete. 2023-09-10T01:02:40.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 Processor -- start 2023-09-10T01:02:40.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 -- start start 2023-09-10T01:02:40.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33701006e0 0x7f33701985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33701018d0 0x7f3370198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3370102270 0x7f337019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33701a0130 con 0x7f3370102270 2023-09-10T01:02:40.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33701a0270 con 0x7f33701018d0 2023-09-10T01:02:40.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f33767ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33701a0400 con 0x7f33701006e0 2023-09-10T01:02:40.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.299+0000 7f3374d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3370102270 0x7f337019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f336ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33701006e0 0x7f33701985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3370102270 0x7f337019fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34108/0 (socket says 172.21.15.3:34108) 2023-09-10T01:02:40.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 -- 172.21.15.3:0/832975216 learned_addr learned my addr 172.21.15.3:0/832975216 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:40.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f336f7fe700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33701018d0 0x7f3370198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 -- 172.21.15.3:0/832975216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33701006e0 msgr2=0x7f33701985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33701006e0 0x7f33701985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 -- 172.21.15.3:0/832975216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33701018d0 msgr2=0x7f3370198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33701018d0 0x7f3370198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 -- 172.21.15.3:0/832975216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33640095f0 con 0x7f3370102270 2023-09-10T01:02:40.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f3374d9c700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3370102270 0x7f337019fb60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f336800ef30 tx=0x7f336800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:40.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f336d7fa700 1 -- 172.21.15.3:0/832975216 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368018070 con 0x7f3370102270 2023-09-10T01:02:40.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f336d7fa700 1 -- 172.21.15.3:0/832975216 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f336800f040 con 0x7f3370102270 2023-09-10T01:02:40.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33701a0680 con 0x7f3370102270 2023-09-10T01:02:40.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.300+0000 7f336d7fa700 1 -- 172.21.15.3:0/832975216 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336800cd00 con 0x7f3370102270 2023-09-10T01:02:40.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.301+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33701a0b90 con 0x7f3370102270 2023-09-10T01:02:40.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.302+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337010bf50 con 0x7f3370102270 2023-09-10T01:02:40.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.303+0000 7f336d7fa700 1 -- 172.21.15.3:0/832975216 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f3368003a00 con 0x7f3370102270 2023-09-10T01:02:40.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.305+0000 7f336d7fa700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33580a0640 0x7f33580a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.305+0000 7f336d7fa700 1 -- 172.21.15.3:0/832975216 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f33680cbad0 con 0x7f3370102270 2023-09-10T01:02:40.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.305+0000 7f336ffff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33580a0640 0x7f33580a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.306+0000 7f336ffff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33580a0640 0x7f33580a2b00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3370101730 tx=0x7f3360009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:40.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.309+0000 7f336d7fa700 1 -- 172.21.15.3:0/832975216 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3368095280 con 0x7f3370102270 2023-09-10T01:02:40.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.521+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33701a0db0 con 0x7f3370102270 2023-09-10T01:02:40.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.522+0000 7f336d7fa700 1 -- 172.21.15.3:0/832975216 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33680988a0 con 0x7f3370102270 2023-09-10T01:02:40.524 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:02:40.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.524+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33580a0640 msgr2=0x7f33580a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.524+0000 7f33767ff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33580a0640 0x7f33580a2b00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3370101730 tx=0x7f3360009260 comp rx=0 tx=0).stop 2023-09-10T01:02:40.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.524+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3370102270 msgr2=0x7f337019fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.524+0000 7f33767ff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3370102270 0x7f337019fb60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f336800ef30 tx=0x7f336800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 shutdown_connections 2023-09-10T01:02:40.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33580a0640 0x7f33580a2b00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33701006e0 0x7f33701985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3370102270 0x7f337019fb60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 --2- 172.21.15.3:0/832975216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33701018d0 0x7f3370198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 >> 172.21.15.3:0/832975216 conn(0x7f33700fbc50 msgr2=0x7f33700fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:40.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 shutdown_connections 2023-09-10T01:02:40.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.525+0000 7f33767ff700 1 -- 172.21.15.3:0/832975216 wait complete. 2023-09-10T01:02:40.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:02:40.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 -- 172.21.15.3:0/513295422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f01006e0 msgr2=0x7f88f0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 --2- 172.21.15.3:0/513295422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f01006e0 0x7f88f0100b00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0099c0 tx=0x7f88ec009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 -- 172.21.15.3:0/513295422 shutdown_connections 2023-09-10T01:02:40.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 --2- 172.21.15.3:0/513295422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88f0102270 0x7f88f0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 --2- 172.21.15.3:0/513295422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f01006e0 0x7f88f0100b00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 --2- 172.21.15.3:0/513295422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 0x7f88f0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 -- 172.21.15.3:0/513295422 >> 172.21.15.3:0/513295422 conn(0x7f88f00fbc50 msgr2=0x7f88f00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:40.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 -- 172.21.15.3:0/513295422 shutdown_connections 2023-09-10T01:02:40.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.750+0000 7f88f83ff700 1 -- 172.21.15.3:0/513295422 wait complete. 2023-09-10T01:02:40.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.751+0000 7f88f83ff700 1 Processor -- start 2023-09-10T01:02:40.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f83ff700 1 -- start start 2023-09-10T01:02:40.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f83ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88f01006e0 0x7f88f01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f83ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 0x7f88f0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f83ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0102270 0x7f88f019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f83ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f01a0160 con 0x7f88f0102270 2023-09-10T01:02:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f83ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f01a02a0 con 0x7f88f01018d0 2023-09-10T01:02:40.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f83ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f01a0430 con 0x7f88f01006e0 2023-09-10T01:02:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f699c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0102270 0x7f88f019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 0x7f88f0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f619b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88f01006e0 0x7f88f01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 0x7f88f0198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54532/0 (socket says 172.21.15.3:54532) 2023-09-10T01:02:40.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.752+0000 7f88f599a700 1 -- 172.21.15.3:0/307242715 learned_addr learned my addr 172.21.15.3:0/307242715 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f599a700 1 -- 172.21.15.3:0/307242715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88f01006e0 msgr2=0x7f88f01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f599a700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88f01006e0 0x7f88f01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f599a700 1 -- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0102270 msgr2=0x7f88f019fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f599a700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0102270 0x7f88f019fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f599a700 1 -- 172.21.15.3:0/307242715 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec0095f0 con 0x7f88f01018d0 2023-09-10T01:02:40.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f599a700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 0x7f88f0198b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88e000e9a0 tx=0x7f88e000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:40.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88e77fe700 1 -- 172.21.15.3:0/307242715 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0009790 con 0x7f88f01018d0 2023-09-10T01:02:40.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f01a0710 con 0x7f88f01018d0 2023-09-10T01:02:40.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88e77fe700 1 -- 172.21.15.3:0/307242715 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88e0010430 con 0x7f88f01018d0 2023-09-10T01:02:40.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88e77fe700 1 -- 172.21.15.3:0/307242715 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0018680 con 0x7f88f01018d0 2023-09-10T01:02:40.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.753+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f01a0c70 con 0x7f88f01018d0 2023-09-10T01:02:40.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.754+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f010bf40 con 0x7f88f01018d0 2023-09-10T01:02:40.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.757+0000 7f88e77fe700 1 -- 172.21.15.3:0/307242715 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f88e00105a0 con 0x7f88f01018d0 2023-09-10T01:02:40.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.758+0000 7f88e77fe700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88dc0a06c0 0x7f88dc0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:40.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.758+0000 7f88e77fe700 1 -- 172.21.15.3:0/307242715 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f88e0014070 con 0x7f88f01018d0 2023-09-10T01:02:40.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.758+0000 7f88f619b700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88dc0a06c0 0x7f88dc0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:40.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.759+0000 7f88f619b700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88dc0a06c0 0x7f88dc0a2b80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009990 tx=0x7f88ec005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:40.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.760+0000 7f88e77fe700 1 -- 172.21.15.3:0/307242715 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f88e0095a30 con 0x7f88f01018d0 2023-09-10T01:02:40.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.972+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88f01a0e90 con 0x7f88f01018d0 2023-09-10T01:02:40.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.972+0000 7f88e77fe700 1 -- 172.21.15.3:0/307242715 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88e00260a0 con 0x7f88f01018d0 2023-09-10T01:02:40.973 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:40.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88dc0a06c0 msgr2=0x7f88dc0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88dc0a06c0 0x7f88dc0a2b80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009990 tx=0x7f88ec005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 msgr2=0x7f88f0198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:40.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 0x7f88f0198b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88e000e9a0 tx=0x7f88e000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 shutdown_connections 2023-09-10T01:02:40.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f88dc0a06c0 0x7f88dc0a2b80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88f01006e0 0x7f88f01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88f0102270 0x7f88f019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 --2- 172.21.15.3:0/307242715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88f01018d0 0x7f88f0198b20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:40.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 >> 172.21.15.3:0/307242715 conn(0x7f88f00fbc50 msgr2=0x7f88f00fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:40.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 shutdown_connections 2023-09-10T01:02:40.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:40.975+0000 7f88f83ff700 1 -- 172.21.15.3:0/307242715 wait complete. 2023-09-10T01:02:40.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:02:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 -- 172.21.15.3:0/3064083923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c102270 msgr2=0x7fa23c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 --2- 172.21.15.3:0/3064083923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c102270 0x7fa23c108b20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa234009fc0 tx=0x7fa23400b330 comp rx=0 tx=0).stop 2023-09-10T01:02:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 -- 172.21.15.3:0/3064083923 shutdown_connections 2023-09-10T01:02:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 --2- 172.21.15.3:0/3064083923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c102270 0x7fa23c108b20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 --2- 172.21.15.3:0/3064083923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c1006e0 0x7fa23c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 --2- 172.21.15.3:0/3064083923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa23c1018d0 0x7fa23c101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 -- 172.21.15.3:0/3064083923 >> 172.21.15.3:0/3064083923 conn(0x7fa23c0fbc50 msgr2=0x7fa23c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 -- 172.21.15.3:0/3064083923 shutdown_connections 2023-09-10T01:02:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.202+0000 7fa2431ff700 1 -- 172.21.15.3:0/3064083923 wait complete. 2023-09-10T01:02:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.203+0000 7fa2431ff700 1 Processor -- start 2023-09-10T01:02:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa2431ff700 1 -- start start 2023-09-10T01:02:41.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa2431ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa23c1006e0 0x7fa23c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa2431ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c1018d0 0x7fa23c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa2431ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c102270 0x7fa23c19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa2431ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c1a02d0 con 0x7fa23c102270 2023-09-10T01:02:41.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa2431ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c1a0410 con 0x7fa23c1018d0 2023-09-10T01:02:41.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa2431ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c1a05a0 con 0x7fa23c1006e0 2023-09-10T01:02:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa24179c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c102270 0x7fa23c19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa24179c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c102270 0x7fa23c19fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34160/0 (socket says 172.21.15.3:34160) 2023-09-10T01:02:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa24179c700 1 -- 172.21.15.3:0/2775755495 learned_addr learned my addr 172.21.15.3:0/2775755495 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa23bfff700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c1018d0 0x7fa23c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.204+0000 7fa240f9b700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa23c1006e0 0x7fa23c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa24179c700 1 -- 172.21.15.3:0/2775755495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa23c1006e0 msgr2=0x7fa23c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa24179c700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa23c1006e0 0x7fa23c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa24179c700 1 -- 172.21.15.3:0/2775755495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c1018d0 msgr2=0x7fa23c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa24179c700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c1018d0 0x7fa23c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa24179c700 1 -- 172.21.15.3:0/2775755495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2300095f0 con 0x7fa23c102270 2023-09-10T01:02:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa24179c700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c102270 0x7fa23c19fd00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa234004c40 tx=0x7fa234004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa239ffb700 1 -- 172.21.15.3:0/2775755495 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23400e040 con 0x7fa23c102270 2023-09-10T01:02:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa234009c40 con 0x7fa23c102270 2023-09-10T01:02:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa239ffb700 1 -- 172.21.15.3:0/2775755495 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa23400bbf0 con 0x7fa23c102270 2023-09-10T01:02:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa239ffb700 1 -- 172.21.15.3:0/2775755495 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2340052a0 con 0x7fa23c102270 2023-09-10T01:02:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.205+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23c1a0b10 con 0x7fa23c102270 2023-09-10T01:02:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.206+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23c10bf50 con 0x7fa23c102270 2023-09-10T01:02:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.208+0000 7fa239ffb700 1 -- 172.21.15.3:0/2775755495 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fa23401d470 con 0x7fa23c102270 2023-09-10T01:02:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.209+0000 7fa239ffb700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa22c0a0640 0x7fa22c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.210+0000 7fa239ffb700 1 -- 172.21.15.3:0/2775755495 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fa2340cd610 con 0x7fa23c102270 2023-09-10T01:02:41.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.210+0000 7fa240f9b700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa22c0a0640 0x7fa22c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.210+0000 7fa240f9b700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa22c0a0640 0x7fa22c0a2b00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa23c101730 tx=0x7fa230009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:41.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.213+0000 7fa239ffb700 1 -- 172.21.15.3:0/2775755495 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa234096dc0 con 0x7fa23c102270 2023-09-10T01:02:41.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:40 smithi003 ceph-mon[97654]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-09-10T01:02:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1499891744' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/832975216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:41.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/307242715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:41.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:40 smithi136 ceph-mon[133708]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-09-10T01:02:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1499891744' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/832975216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:41.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/307242715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.423+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa23c10c110 con 0x7fa23c102270 2023-09-10T01:02:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.424+0000 7fa239ffb700 1 -- 172.21.15.3:0/2775755495 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa23c10c110 con 0x7fa23c102270 2023-09-10T01:02:41.425 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:02:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa22c0a0640 msgr2=0x7fa22c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa22c0a0640 0x7fa22c0a2b00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa23c101730 tx=0x7fa230009310 comp rx=0 tx=0).stop 2023-09-10T01:02:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c102270 msgr2=0x7fa23c19fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c102270 0x7fa23c19fd00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa234004c40 tx=0x7fa234004c70 comp rx=0 tx=0).stop 2023-09-10T01:02:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 shutdown_connections 2023-09-10T01:02:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fa22c0a0640 0x7fa22c0a2b00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa23c1006e0 0x7fa23c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa23c102270 0x7fa23c19fd00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 --2- 172.21.15.3:0/2775755495 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa23c1018d0 0x7fa23c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.426+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 >> 172.21.15.3:0/2775755495 conn(0x7fa23c0fbc50 msgr2=0x7fa23c107160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.427+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 shutdown_connections 2023-09-10T01:02:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.427+0000 7fa2431ff700 1 -- 172.21.15.3:0/2775755495 wait complete. 2023-09-10T01:02:41.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:02:41.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:40 smithi092 ceph-mon[134067]: pgmap v342: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-09-10T01:02:41.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1499891744' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:41.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/832975216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:41.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/307242715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:41.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 -- 172.21.15.3:0/1522660747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 msgr2=0x7f0f4c100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 --2- 172.21.15.3:0/1522660747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c100b10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0099c0 tx=0x7f0f3c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 -- 172.21.15.3:0/1522660747 shutdown_connections 2023-09-10T01:02:41.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 --2- 172.21.15.3:0/1522660747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c102280 0x7f0f4c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 --2- 172.21.15.3:0/1522660747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c100b10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 --2- 172.21.15.3:0/1522660747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f4c1018e0 0x7f0f4c101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 -- 172.21.15.3:0/1522660747 >> 172.21.15.3:0/1522660747 conn(0x7f0f4c0fbc60 msgr2=0x7f0f4c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 -- 172.21.15.3:0/1522660747 shutdown_connections 2023-09-10T01:02:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.652+0000 7f0f525ff700 1 -- 172.21.15.3:0/1522660747 wait complete. 2023-09-10T01:02:41.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 Processor -- start 2023-09-10T01:02:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 -- start start 2023-09-10T01:02:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c1018e0 0x7f0f4c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f4c102280 0x7f0f4c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c1a0130 con 0x7f0f4c1006f0 2023-09-10T01:02:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c1a0270 con 0x7f0f4c1018e0 2023-09-10T01:02:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.653+0000 7f0f525ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c1a0400 con 0x7f0f4c102280 2023-09-10T01:02:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34180/0 (socket says 172.21.15.3:34180) 2023-09-10T01:02:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 -- 172.21.15.3:0/137114707 learned_addr learned my addr 172.21.15.3:0/137114707 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f43fff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c1018e0 0x7f0f4c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f50b9c700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f4c102280 0x7f0f4c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 -- 172.21.15.3:0/137114707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f4c102280 msgr2=0x7f0f4c19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f4c102280 0x7f0f4c19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 -- 172.21.15.3:0/137114707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c1018e0 msgr2=0x7f0f4c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c1018e0 0x7f0f4c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 -- 172.21.15.3:0/137114707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c0095f0 con 0x7f0f4c1006f0 2023-09-10T01:02:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f4bfff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c1985b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009fd0 tx=0x7f0f3c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.654+0000 7f0f49ffb700 1 -- 172.21.15.3:0/137114707 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c01d070 con 0x7f0f4c1006f0 2023-09-10T01:02:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.655+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4c1a0680 con 0x7f0f4c1006f0 2023-09-10T01:02:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.655+0000 7f0f49ffb700 1 -- 172.21.15.3:0/137114707 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f3c005030 con 0x7f0f4c1006f0 2023-09-10T01:02:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.655+0000 7f0f49ffb700 1 -- 172.21.15.3:0/137114707 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c004a90 con 0x7f0f4c1006f0 2023-09-10T01:02:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.655+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c1a0b00 con 0x7f0f4c1006f0 2023-09-10T01:02:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.656+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4c10bf00 con 0x7f0f4c1006f0 2023-09-10T01:02:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.658+0000 7f0f49ffb700 1 -- 172.21.15.3:0/137114707 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f0f3c0051a0 con 0x7f0f4c1006f0 2023-09-10T01:02:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.660+0000 7f0f49ffb700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f380a0640 0x7f0f380a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.660+0000 7f0f49ffb700 1 -- 172.21.15.3:0/137114707 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f0f3c0cd120 con 0x7f0f4c1006f0 2023-09-10T01:02:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.661+0000 7f0f43fff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f380a0640 0x7f0f380a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.661+0000 7f0f43fff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f380a0640 0x7f0f380a2b00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0f34009c60 tx=0x7f0f34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:41.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.662+0000 7f0f49ffb700 1 -- 172.21.15.3:0/137114707 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0f3c0968d0 con 0x7f0f4c1006f0 2023-09-10T01:02:41.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.874+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0f4c1a0f90 con 0x7f0f4c1006f0 2023-09-10T01:02:41.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.874+0000 7f0f49ffb700 1 -- 172.21.15.3:0/137114707 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0f3c0260a0 con 0x7f0f4c1006f0 2023-09-10T01:02:41.876 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:41.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f380a0640 msgr2=0x7f0f380a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f380a0640 0x7f0f380a2b00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0f34009c60 tx=0x7f0f34009260 comp rx=0 tx=0).stop 2023-09-10T01:02:41.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 msgr2=0x7f0f4c1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:41.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c1985b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009fd0 tx=0x7f0f3c009580 comp rx=0 tx=0).stop 2023-09-10T01:02:41.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 shutdown_connections 2023-09-10T01:02:41.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0f380a0640 0x7f0f380a2b00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f4c102280 0x7f0f4c19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f4c1006f0 0x7f0f4c1985b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 --2- 172.21.15.3:0/137114707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c1018e0 0x7f0f4c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:41.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 >> 172.21.15.3:0/137114707 conn(0x7f0f4c0fbc60 msgr2=0x7f0f4c0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:41.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 shutdown_connections 2023-09-10T01:02:41.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:41.877+0000 7f0f525ff700 1 -- 172.21.15.3:0/137114707 wait complete. 2023-09-10T01:02:41.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:02:42.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 -- 172.21.15.3:0/3759730360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741006c0 msgr2=0x7fbd74100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 --2- 172.21.15.3:0/3759730360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741006c0 0x7fbd74100ae0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbd640099c0 tx=0x7fbd64009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 -- 172.21.15.3:0/3759730360 shutdown_connections 2023-09-10T01:02:42.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 --2- 172.21.15.3:0/3759730360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd74102250 0x7fbd74108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 --2- 172.21.15.3:0/3759730360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741006c0 0x7fbd74100ae0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 --2- 172.21.15.3:0/3759730360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd741018b0 0x7fbd74101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 -- 172.21.15.3:0/3759730360 >> 172.21.15.3:0/3759730360 conn(0x7fbd740fbc30 msgr2=0x7fbd740fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:42.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 -- 172.21.15.3:0/3759730360 shutdown_connections 2023-09-10T01:02:42.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.102+0000 7fbd79fff700 1 -- 172.21.15.3:0/3759730360 wait complete. 2023-09-10T01:02:42.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.103+0000 7fbd79fff700 1 Processor -- start 2023-09-10T01:02:42.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.103+0000 7fbd79fff700 1 -- start start 2023-09-10T01:02:42.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd79fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd741006c0 0x7fbd74196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd79fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741018b0 0x7fbd74196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd79fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd74102250 0x7fbd7419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd79fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7419df80 con 0x7fbd741018b0 2023-09-10T01:02:42.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd79fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7419e0c0 con 0x7fbd74102250 2023-09-10T01:02:42.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd79fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7419e250 con 0x7fbd741006c0 2023-09-10T01:02:42.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741018b0 0x7fbd74196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741018b0 0x7fbd74196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34198/0 (socket says 172.21.15.3:34198) 2023-09-10T01:02:42.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 -- 172.21.15.3:0/2172881893 learned_addr learned my addr 172.21.15.3:0/2172881893 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:42.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd737fe700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd741006c0 0x7fbd74196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd73fff700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd74102250 0x7fbd7419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 -- 172.21.15.3:0/2172881893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd741006c0 msgr2=0x7fbd74196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd741006c0 0x7fbd74196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 -- 172.21.15.3:0/2172881893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd74102250 msgr2=0x7fbd7419d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd74102250 0x7fbd7419d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.104+0000 7fbd72ffd700 1 -- 172.21.15.3:0/2172881893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640095f0 con 0x7fbd741018b0 2023-09-10T01:02:42.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.105+0000 7fbd72ffd700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741018b0 0x7fbd74196940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800e9a0 tx=0x7fbd6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:42.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.105+0000 7fbd70ff9700 1 -- 172.21.15.3:0/2172881893 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68009790 con 0x7fbd741018b0 2023-09-10T01:02:42.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.105+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7419e530 con 0x7fbd741018b0 2023-09-10T01:02:42.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.105+0000 7fbd70ff9700 1 -- 172.21.15.3:0/2172881893 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd68010430 con 0x7fbd741018b0 2023-09-10T01:02:42.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.105+0000 7fbd70ff9700 1 -- 172.21.15.3:0/2172881893 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68018680 con 0x7fbd741018b0 2023-09-10T01:02:42.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.105+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7419ea90 con 0x7fbd741018b0 2023-09-10T01:02:42.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.106+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd74107640 con 0x7fbd741018b0 2023-09-10T01:02:42.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.108+0000 7fbd70ff9700 1 -- 172.21.15.3:0/2172881893 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fbd680105a0 con 0x7fbd741018b0 2023-09-10T01:02:42.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.109+0000 7fbd70ff9700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd5c0a0640 0x7fbd5c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.109+0000 7fbd70ff9700 1 -- 172.21.15.3:0/2172881893 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fbd68014070 con 0x7fbd741018b0 2023-09-10T01:02:42.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.110+0000 7fbd737fe700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd5c0a0640 0x7fbd5c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.110+0000 7fbd737fe700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd5c0a0640 0x7fbd5c0a2b00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009990 tx=0x7fbd64005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:42.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.113+0000 7fbd70ff9700 1 -- 172.21.15.3:0/2172881893 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbd68051e10 con 0x7fbd741018b0 2023-09-10T01:02:42.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:41 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2775755495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:42.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:41 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/137114707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.366+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd7402ce20 con 0x7fbd741018b0 2023-09-10T01:02:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.367+0000 7fbd70ff9700 1 -- 172.21.15.3:0/2172881893 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7fbd68095870 con 0x7fbd741018b0 2023-09-10T01:02:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:42.368 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:02:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.369+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd5c0a0640 msgr2=0x7fbd5c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd5c0a0640 0x7fbd5c0a2b00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009990 tx=0x7fbd64005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741018b0 msgr2=0x7fbd74196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741018b0 0x7fbd74196940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800e9a0 tx=0x7fbd6800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 shutdown_connections 2023-09-10T01:02:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbd5c0a0640 0x7fbd5c0a2b00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd741006c0 0x7fbd74196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd74102250 0x7fbd7419d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 --2- 172.21.15.3:0/2172881893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd741018b0 0x7fbd74196940 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 >> 172.21.15.3:0/2172881893 conn(0x7fbd740fbc30 msgr2=0x7fbd740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:42.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 shutdown_connections 2023-09-10T01:02:42.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.370+0000 7fbd79fff700 1 -- 172.21.15.3:0/2172881893 wait complete. 2023-09-10T01:02:42.373 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:02:42.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:41 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2775755495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:42.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:41 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/137114707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:42.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:41 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2775755495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:42.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:41 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/137114707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:42.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 -- 172.21.15.3:0/3807518621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881006d0 msgr2=0x7f5c88100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/3807518621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881006d0 0x7f5c88100af0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5c780099c0 tx=0x7f5c78009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 -- 172.21.15.3:0/3807518621 shutdown_connections 2023-09-10T01:02:42.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/3807518621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c88102260 0x7f5c88108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/3807518621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881006d0 0x7f5c88100af0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/3807518621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c881018c0 0x7f5c88101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 -- 172.21.15.3:0/3807518621 >> 172.21.15.3:0/3807518621 conn(0x7f5c880fbc60 msgr2=0x7f5c880fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:42.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 -- 172.21.15.3:0/3807518621 shutdown_connections 2023-09-10T01:02:42.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.594+0000 7f5c8cdff700 1 -- 172.21.15.3:0/3807518621 wait complete. 2023-09-10T01:02:42.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.595+0000 7f5c8cdff700 1 Processor -- start 2023-09-10T01:02:42.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.595+0000 7f5c8cdff700 1 -- start start 2023-09-10T01:02:42.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c8cdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c881006d0 0x7f5c88198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c8cdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881018c0 0x7f5c88198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c8cdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c88102260 0x7f5c8819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c8cdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c881a00f0 con 0x7f5c881018c0 2023-09-10T01:02:42.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c8cdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c881a0230 con 0x7f5c881006d0 2023-09-10T01:02:42.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c8cdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c881a03c0 con 0x7f5c88102260 2023-09-10T01:02:42.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881018c0 0x7f5c88198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881018c0 0x7f5c88198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34214/0 (socket says 172.21.15.3:34214) 2023-09-10T01:02:42.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 -- 172.21.15.3:0/2351615620 learned_addr learned my addr 172.21.15.3:0/2351615620 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:42.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c86d9d700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c88102260 0x7f5c8819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c8659c700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c881006d0 0x7f5c88198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 -- 172.21.15.3:0/2351615620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c88102260 msgr2=0x7f5c8819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c88102260 0x7f5c8819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 -- 172.21.15.3:0/2351615620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c881006d0 msgr2=0x7f5c88198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c881006d0 0x7f5c88198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.596+0000 7f5c7ffff700 1 -- 172.21.15.3:0/2351615620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c780095f0 con 0x7f5c881018c0 2023-09-10T01:02:42.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.597+0000 7f5c7ffff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881018c0 0x7f5c88198ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5c7000e9a0 tx=0x7f5c7000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:42.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.597+0000 7f5c7f7fe700 1 -- 172.21.15.3:0/2351615620 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70009790 con 0x7f5c881018c0 2023-09-10T01:02:42.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.597+0000 7f5c7f7fe700 1 -- 172.21.15.3:0/2351615620 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c70010430 con 0x7f5c881018c0 2023-09-10T01:02:42.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.597+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c881a06a0 con 0x7f5c881018c0 2023-09-10T01:02:42.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.597+0000 7f5c7f7fe700 1 -- 172.21.15.3:0/2351615620 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70018680 con 0x7f5c881018c0 2023-09-10T01:02:42.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.597+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c881a0bb0 con 0x7f5c881018c0 2023-09-10T01:02:42.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.598+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8810bee0 con 0x7f5c881018c0 2023-09-10T01:02:42.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.600+0000 7f5c7f7fe700 1 -- 172.21.15.3:0/2351615620 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f5c700105a0 con 0x7f5c881018c0 2023-09-10T01:02:42.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.601+0000 7f5c7f7fe700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5c740a0640 0x7f5c740a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:42.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.601+0000 7f5c7f7fe700 1 -- 172.21.15.3:0/2351615620 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f5c70014070 con 0x7f5c881018c0 2023-09-10T01:02:42.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.602+0000 7f5c8659c700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5c740a0640 0x7f5c740a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:42.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.602+0000 7f5c8659c700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5c740a0640 0x7f5c740a2b00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009990 tx=0x7f5c78005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:42.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.605+0000 7f5c7f7fe700 1 -- 172.21.15.3:0/2351615620 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5c70051e10 con 0x7f5c881018c0 2023-09-10T01:02:42.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.816+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c8810c0f0 con 0x7f5c881018c0 2023-09-10T01:02:42.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.818+0000 7f5c7f7fe700 1 -- 172.21.15.3:0/2351615620 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7f5c70095870 con 0x7f5c881018c0 2023-09-10T01:02:42.819 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:42.820 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:42.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5c740a0640 msgr2=0x7f5c740a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5c740a0640 0x7f5c740a2b00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009990 tx=0x7f5c78005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881018c0 msgr2=0x7f5c88198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:42.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881018c0 0x7f5c88198ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5c7000e9a0 tx=0x7f5c7000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 shutdown_connections 2023-09-10T01:02:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5c740a0640 0x7f5c740a2b00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c88102260 0x7f5c8819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c881006d0 0x7f5c88198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 --2- 172.21.15.3:0/2351615620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5c881018c0 0x7f5c88198ab0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:42.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 >> 172.21.15.3:0/2351615620 conn(0x7f5c880fbc60 msgr2=0x7f5c880fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:42.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 shutdown_connections 2023-09-10T01:02:42.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:42.821+0000 7f5c8cdff700 1 -- 172.21.15.3:0/2351615620 wait complete. 2023-09-10T01:02:42.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:02:42.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:02:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 -- 172.21.15.3:0/2134952365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f01018e0 msgr2=0x7fb3f0101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/2134952365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f01018e0 0x7fb3f0101d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0009920 tx=0x7fb3e0009c30 comp rx=0 tx=0).stop 2023-09-10T01:02:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 -- 172.21.15.3:0/2134952365 shutdown_connections 2023-09-10T01:02:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/2134952365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f01006f0 0x7fb3f0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/2134952365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f0102280 0x7fb3f0108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/2134952365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f01018e0 0x7fb3f0101d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 -- 172.21.15.3:0/2134952365 >> 172.21.15.3:0/2134952365 conn(0x7fb3f00fbc60 msgr2=0x7fb3f00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 -- 172.21.15.3:0/2134952365 shutdown_connections 2023-09-10T01:02:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.054+0000 7fb3f7bff700 1 -- 172.21.15.3:0/2134952365 wait complete. 2023-09-10T01:02:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 Processor -- start 2023-09-10T01:02:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 -- start start 2023-09-10T01:02:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f01006f0 0x7fb3f0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f01018e0 0x7fb3f0198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f0102280 0x7fb3f019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f01a01d0 con 0x7fb3f01018e0 2023-09-10T01:02:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f01a0310 con 0x7fb3f0102280 2023-09-10T01:02:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.055+0000 7fb3f7bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f01a04a0 con 0x7fb3f01006f0 2023-09-10T01:02:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f01018e0 0x7fb3f0198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f01018e0 0x7fb3f0198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34228/0 (socket says 172.21.15.3:34228) 2023-09-10T01:02:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 -- 172.21.15.3:0/1389824672 learned_addr learned my addr 172.21.15.3:0/1389824672 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f619c700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f0102280 0x7fb3f019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f599b700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f01006f0 0x7fb3f0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 -- 172.21.15.3:0/1389824672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f01006f0 msgr2=0x7fb3f0198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f01006f0 0x7fb3f0198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 -- 172.21.15.3:0/1389824672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f0102280 msgr2=0x7fb3f019fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f0102280 0x7fb3f019fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 -- 172.21.15.3:0/1389824672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e0009520 con 0x7fb3f01018e0 2023-09-10T01:02:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3f519a700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f01018e0 0x7fb3f0198b90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0009230 tx=0x7fb3e000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.056+0000 7fb3e6ffd700 1 -- 172.21.15.3:0/1389824672 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e001c070 con 0x7fb3f01018e0 2023-09-10T01:02:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.057+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f01a0720 con 0x7fb3f01018e0 2023-09-10T01:02:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.057+0000 7fb3e6ffd700 1 -- 172.21.15.3:0/1389824672 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3e000bba0 con 0x7fb3f01018e0 2023-09-10T01:02:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.057+0000 7fb3e6ffd700 1 -- 172.21.15.3:0/1389824672 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0003a80 con 0x7fb3f01018e0 2023-09-10T01:02:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.057+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f01a0b40 con 0x7fb3f01018e0 2023-09-10T01:02:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.058+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f0192d50 con 0x7fb3f01018e0 2023-09-10T01:02:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.060+0000 7fb3e6ffd700 1 -- 172.21.15.3:0/1389824672 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb3e0020030 con 0x7fb3f01018e0 2023-09-10T01:02:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.062+0000 7fb3e6ffd700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb3dc0a06c0 0x7fb3dc0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.062+0000 7fb3e6ffd700 1 -- 172.21.15.3:0/1389824672 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fb3e00ccfe0 con 0x7fb3f01018e0 2023-09-10T01:02:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.062+0000 7fb3f599b700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb3dc0a06c0 0x7fb3dc0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.063+0000 7fb3f599b700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb3dc0a06c0 0x7fb3dc0a2b80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0101740 tx=0x7fb3ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.066+0000 7fb3e6ffd700 1 -- 172.21.15.3:0/1389824672 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb3e0096790 con 0x7fb3f01018e0 2023-09-10T01:02:43.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.276+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3f002cfc0 con 0x7fb3f01018e0 2023-09-10T01:02:43.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.276+0000 7fb3e6ffd700 1 -- 172.21.15.3:0/1389824672 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3e0005040 con 0x7fb3f01018e0 2023-09-10T01:02:43.278 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:02:43.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb3dc0a06c0 msgr2=0x7fb3dc0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb3dc0a06c0 0x7fb3dc0a2b80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0101740 tx=0x7fb3ec009260 comp rx=0 tx=0).stop 2023-09-10T01:02:43.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f01018e0 msgr2=0x7fb3f0198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f01018e0 0x7fb3f0198b90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0009230 tx=0x7fb3e000bf80 comp rx=0 tx=0).stop 2023-09-10T01:02:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 shutdown_connections 2023-09-10T01:02:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb3dc0a06c0 0x7fb3dc0a2b80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f01006f0 0x7fb3f0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3f0102280 0x7fb3f019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 --2- 172.21.15.3:0/1389824672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3f01018e0 0x7fb3f0198b90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 >> 172.21.15.3:0/1389824672 conn(0x7fb3f00fbc60 msgr2=0x7fb3f0107200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:43.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 shutdown_connections 2023-09-10T01:02:43.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.279+0000 7fb3f7bff700 1 -- 172.21.15.3:0/1389824672 wait complete. 2023-09-10T01:02:43.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:42 smithi003 ceph-mon[97654]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-09-10T01:02:43.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:42 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2172881893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:43.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:42 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2351615620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:43.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:02:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:42 smithi136 ceph-mon[133708]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-09-10T01:02:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:42 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2172881893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:43.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:42 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2351615620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:43.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:42 smithi092 ceph-mon[134067]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-09-10T01:02:43.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:42 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2172881893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:43.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:42 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2351615620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:43.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.506+0000 7f8a4dfff700 1 -- 172.21.15.3:0/1472064599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481018e0 msgr2=0x7f8a48101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.506+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/1472064599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481018e0 0x7f8a48101d40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009990 tx=0x7f8a3c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.506+0000 7f8a4dfff700 1 -- 172.21.15.3:0/1472064599 shutdown_connections 2023-09-10T01:02:43.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.506+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/1472064599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 0x7f8a48108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.506+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/1472064599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a481006f0 0x7f8a48100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.506+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/1472064599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481018e0 0x7f8a48101d40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.506+0000 7f8a4dfff700 1 -- 172.21.15.3:0/1472064599 >> 172.21.15.3:0/1472064599 conn(0x7f8a480fbc60 msgr2=0x7f8a480fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:43.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.507+0000 7f8a4dfff700 1 -- 172.21.15.3:0/1472064599 shutdown_connections 2023-09-10T01:02:43.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.507+0000 7f8a4dfff700 1 -- 172.21.15.3:0/1472064599 wait complete. 2023-09-10T01:02:43.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.507+0000 7f8a4dfff700 1 Processor -- start 2023-09-10T01:02:43.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a4dfff700 1 -- start start 2023-09-10T01:02:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a4dfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481006f0 0x7f8a481985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a4dfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a481018e0 0x7f8a48198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a4dfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 0x7f8a4819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a4dfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a481a0130 con 0x7f8a48102280 2023-09-10T01:02:43.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a4dfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a481a0270 con 0x7f8a481018e0 2023-09-10T01:02:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a4dfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a481a0400 con 0x7f8a481006f0 2023-09-10T01:02:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a47fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 0x7f8a4819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a47fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 0x7f8a4819fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34260/0 (socket says 172.21.15.3:34260) 2023-09-10T01:02:43.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a47fff700 1 -- 172.21.15.3:0/4010776474 learned_addr learned my addr 172.21.15.3:0/4010776474 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:43.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a477fe700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481006f0 0x7f8a481985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.508+0000 7f8a46ffd700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a481018e0 0x7f8a48198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a47fff700 1 -- 172.21.15.3:0/4010776474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481006f0 msgr2=0x7f8a481985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a47fff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481006f0 0x7f8a481985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a47fff700 1 -- 172.21.15.3:0/4010776474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a481018e0 msgr2=0x7f8a48198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a47fff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a481018e0 0x7f8a48198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a47fff700 1 -- 172.21.15.3:0/4010776474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38009520 con 0x7f8a48102280 2023-09-10T01:02:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a477fe700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481006f0 0x7f8a481985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a47fff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 0x7f8a4819fb60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000ef30 tx=0x7f8a4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:43.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a44ff9700 1 -- 172.21.15.3:0/4010776474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40018070 con 0x7f8a48102280 2023-09-10T01:02:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c0095f0 con 0x7f8a48102280 2023-09-10T01:02:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a44ff9700 1 -- 172.21.15.3:0/4010776474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a4000f040 con 0x7f8a48102280 2023-09-10T01:02:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a44ff9700 1 -- 172.21.15.3:0/4010776474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4000cdb0 con 0x7f8a48102280 2023-09-10T01:02:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.509+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a481a0a50 con 0x7f8a48102280 2023-09-10T01:02:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.510+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4810bf60 con 0x7f8a48102280 2023-09-10T01:02:43.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.512+0000 7f8a44ff9700 1 -- 172.21.15.3:0/4010776474 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f8a40003a00 con 0x7f8a48102280 2023-09-10T01:02:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.514+0000 7f8a44ff9700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a300a06c0 0x7f8a300a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.514+0000 7f8a44ff9700 1 -- 172.21.15.3:0/4010776474 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f8a400cbb00 con 0x7f8a48102280 2023-09-10T01:02:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.514+0000 7f8a477fe700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a300a06c0 0x7f8a300a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.515+0000 7f8a477fe700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a300a06c0 0x7f8a300a2b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8a48101740 tx=0x7f8a38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:43.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.518+0000 7f8a44ff9700 1 -- 172.21.15.3:0/4010776474 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f8a400cde60 con 0x7f8a48102280 2023-09-10T01:02:43.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.732+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a481a0c70 con 0x7f8a48102280 2023-09-10T01:02:43.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.732+0000 7f8a44ff9700 1 -- 172.21.15.3:0/4010776474 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a400952b0 con 0x7f8a48102280 2023-09-10T01:02:43.734 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:43.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a300a06c0 msgr2=0x7f8a300a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a300a06c0 0x7f8a300a2b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8a48101740 tx=0x7f8a38009260 comp rx=0 tx=0).stop 2023-09-10T01:02:43.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 msgr2=0x7f8a4819fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 0x7f8a4819fb60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000ef30 tx=0x7f8a4000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 shutdown_connections 2023-09-10T01:02:43.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8a300a06c0 0x7f8a300a2b80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a481006f0 0x7f8a481985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a48102280 0x7f8a4819fb60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 --2- 172.21.15.3:0/4010776474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a481018e0 0x7f8a48198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 >> 172.21.15.3:0/4010776474 conn(0x7f8a480fbc60 msgr2=0x7f8a48107160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:43.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 shutdown_connections 2023-09-10T01:02:43.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.735+0000 7f8a4dfff700 1 -- 172.21.15.3:0/4010776474 wait complete. 2023-09-10T01:02:43.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:02:43.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 -- 172.21.15.3:0/3542454767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b81018d0 msgr2=0x7f07b8101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/3542454767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b81018d0 0x7f07b8101d30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f07a8009920 tx=0x7f07a8009c30 comp rx=0 tx=0).stop 2023-09-10T01:02:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 -- 172.21.15.3:0/3542454767 shutdown_connections 2023-09-10T01:02:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/3542454767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b8102270 0x7f07b8108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/3542454767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07b81006e0 0x7f07b8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/3542454767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b81018d0 0x7f07b8101d30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 -- 172.21.15.3:0/3542454767 >> 172.21.15.3:0/3542454767 conn(0x7f07b80fbc50 msgr2=0x7f07b80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 -- 172.21.15.3:0/3542454767 shutdown_connections 2023-09-10T01:02:43.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.962+0000 7f07bf5ff700 1 -- 172.21.15.3:0/3542454767 wait complete. 2023-09-10T01:02:43.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.963+0000 7f07bf5ff700 1 Processor -- start 2023-09-10T01:02:43.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.963+0000 7f07bf5ff700 1 -- start start 2023-09-10T01:02:43.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bf5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b81006e0 0x7f07b81963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bf5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07b81018d0 0x7f07b8196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bf5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b8102270 0x7f07b819d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bf5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b819df70 con 0x7f07b81018d0 2023-09-10T01:02:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bf5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b819e0b0 con 0x7f07b8102270 2023-09-10T01:02:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bf5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b819e240 con 0x7f07b81006e0 2023-09-10T01:02:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bdb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b8102270 0x7f07b819d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bcb9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07b81018d0 0x7f07b8196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bdb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b8102270 0x7f07b819d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54632/0 (socket says 172.21.15.3:54632) 2023-09-10T01:02:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bdb9c700 1 -- 172.21.15.3:0/2172652139 learned_addr learned my addr 172.21.15.3:0/2172652139 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:43.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.964+0000 7f07bd39b700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b81006e0 0x7f07b81963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bdb9c700 1 -- 172.21.15.3:0/2172652139 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b81006e0 msgr2=0x7f07b81963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bdb9c700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b81006e0 0x7f07b81963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bdb9c700 1 -- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07b81018d0 msgr2=0x7f07b8196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bdb9c700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07b81018d0 0x7f07b8196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bdb9c700 1 -- 172.21.15.3:0/2172652139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8009520 con 0x7f07b8102270 2023-09-10T01:02:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bcb9a700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07b81018d0 0x7f07b8196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bd39b700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b81006e0 0x7f07b81963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bdb9c700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b8102270 0x7f07b819d9a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07b000ef30 tx=0x7f07b000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07ae7fc700 1 -- 172.21.15.3:0/2172652139 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0018070 con 0x7f07b8102270 2023-09-10T01:02:43.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b819e4c0 con 0x7f07b8102270 2023-09-10T01:02:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07ae7fc700 1 -- 172.21.15.3:0/2172652139 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07b000f040 con 0x7f07b8102270 2023-09-10T01:02:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07ae7fc700 1 -- 172.21.15.3:0/2172652139 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b000ccf0 con 0x7f07b8102270 2023-09-10T01:02:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.965+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b819e9d0 con 0x7f07b8102270 2023-09-10T01:02:43.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.966+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b810bef0 con 0x7f07b8102270 2023-09-10T01:02:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.969+0000 7f07ae7fc700 1 -- 172.21.15.3:0/2172652139 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f07b0003a00 con 0x7f07b8102270 2023-09-10T01:02:43.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.970+0000 7f07ae7fc700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f07a40a0640 0x7f07a40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.970+0000 7f07ae7fc700 1 -- 172.21.15.3:0/2172652139 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f07b00cc430 con 0x7f07b8102270 2023-09-10T01:02:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.970+0000 7f07bd39b700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f07a40a0640 0x7f07a40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.971+0000 7f07bd39b700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f07a40a0640 0x7f07a40a2b00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f07b8101730 tx=0x7f07b4009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:43.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:43.972+0000 7f07ae7fc700 1 -- 172.21.15.3:0/2172652139 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f07b0095be0 con 0x7f07b8102270 2023-09-10T01:02:44.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:43 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1389824672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:44.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:43 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4010776474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:44.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.190+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f07b8107650 con 0x7f07b8102270 2023-09-10T01:02:44.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.191+0000 7f07ae7fc700 1 -- 172.21.15.3:0/2172652139 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f07b0099200 con 0x7f07b8102270 2023-09-10T01:02:44.192 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:02:44.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f07a40a0640 msgr2=0x7f07a40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f07a40a0640 0x7f07a40a2b00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f07b8101730 tx=0x7f07b4009310 comp rx=0 tx=0).stop 2023-09-10T01:02:44.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b8102270 msgr2=0x7f07b819d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b8102270 0x7f07b819d9a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07b000ef30 tx=0x7f07b000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 shutdown_connections 2023-09-10T01:02:44.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f07a40a0640 0x7f07a40a2b00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b81006e0 0x7f07b81963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b8102270 0x7f07b819d9a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 --2- 172.21.15.3:0/2172652139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07b81018d0 0x7f07b8196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 >> 172.21.15.3:0/2172652139 conn(0x7f07b80fbc50 msgr2=0x7f07b80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:44.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 shutdown_connections 2023-09-10T01:02:44.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.194+0000 7f07bf5ff700 1 -- 172.21.15.3:0/2172652139 wait complete. 2023-09-10T01:02:44.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:02:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:43 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1389824672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:43 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4010776474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:44.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 -- 172.21.15.3:0/1700485089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601006f0 msgr2=0x7f0460100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 --2- 172.21.15.3:0/1700485089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601006f0 0x7f0460100b10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f045c009990 tx=0x7f045c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 -- 172.21.15.3:0/1700485089 shutdown_connections 2023-09-10T01:02:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 --2- 172.21.15.3:0/1700485089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601006f0 0x7f0460100b10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 --2- 172.21.15.3:0/1700485089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0460102280 0x7f0460108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 --2- 172.21.15.3:0/1700485089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601018e0 0x7f0460101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 -- 172.21.15.3:0/1700485089 >> 172.21.15.3:0/1700485089 conn(0x7f04600fbc60 msgr2=0x7f04600fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:44.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 -- 172.21.15.3:0/1700485089 shutdown_connections 2023-09-10T01:02:44.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.419+0000 7f04675ff700 1 -- 172.21.15.3:0/1700485089 wait complete. 2023-09-10T01:02:44.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.420+0000 7f04675ff700 1 Processor -- start 2023-09-10T01:02:44.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.420+0000 7f04675ff700 1 -- start start 2023-09-10T01:02:44.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f04675ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601006f0 0x7f04601963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f04675ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601018e0 0x7f0460196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f04675ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0460102280 0x7f046019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f04675ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046019df70 con 0x7f04601006f0 2023-09-10T01:02:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f04675ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046019e0b0 con 0x7f0460102280 2023-09-10T01:02:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f04675ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046019e240 con 0x7f04601018e0 2023-09-10T01:02:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601006f0 0x7f04601963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601006f0 0x7f04601963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34278/0 (socket says 172.21.15.3:34278) 2023-09-10T01:02:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 -- 172.21.15.3:0/108413806 learned_addr learned my addr 172.21.15.3:0/108413806 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f0464b9a700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601018e0 0x7f0460196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f0465b9c700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0460102280 0x7f046019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 -- 172.21.15.3:0/108413806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601018e0 msgr2=0x7f0460196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601018e0 0x7f0460196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 -- 172.21.15.3:0/108413806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0460102280 msgr2=0x7f046019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0460102280 0x7f046019d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.421+0000 7f046539b700 1 -- 172.21.15.3:0/108413806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450009520 con 0x7f04601006f0 2023-09-10T01:02:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.422+0000 7f0464b9a700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601018e0 0x7f0460196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.422+0000 7f046539b700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601006f0 0x7f04601963f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f045c000c00 tx=0x7f045c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:44.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.422+0000 7f04567fc700 1 -- 172.21.15.3:0/108413806 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c01d070 con 0x7f04601006f0 2023-09-10T01:02:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.422+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c0095f0 con 0x7f04601006f0 2023-09-10T01:02:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.422+0000 7f04567fc700 1 -- 172.21.15.3:0/108413806 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f045c022470 con 0x7f04601006f0 2023-09-10T01:02:44.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.422+0000 7f04567fc700 1 -- 172.21.15.3:0/108413806 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c017650 con 0x7f04601006f0 2023-09-10T01:02:44.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.422+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046019e7b0 con 0x7f04601006f0 2023-09-10T01:02:44.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.423+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0460107c80 con 0x7f04601006f0 2023-09-10T01:02:44.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.425+0000 7f04567fc700 1 -- 172.21.15.3:0/108413806 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f045c0225e0 con 0x7f04601006f0 2023-09-10T01:02:44.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.427+0000 7f04567fc700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f044c0a0640 0x7f044c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.427+0000 7f04567fc700 1 -- 172.21.15.3:0/108413806 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f045c0cd310 con 0x7f04601006f0 2023-09-10T01:02:44.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.427+0000 7f0464b9a700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f044c0a0640 0x7f044c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.427+0000 7f0464b9a700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f044c0a0640 0x7f044c0a2b00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e40 tx=0x7f0450009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:44.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.430+0000 7f04567fc700 1 -- 172.21.15.3:0/108413806 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f045c05ce10 con 0x7f04601006f0 2023-09-10T01:02:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:43 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1389824672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:43 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4010776474' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.640+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f046002cd50 con 0x7f04601006f0 2023-09-10T01:02:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.640+0000 7f04567fc700 1 -- 172.21.15.3:0/108413806 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f045c096ac0 con 0x7f04601006f0 2023-09-10T01:02:44.641 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:44.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.642+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f044c0a0640 msgr2=0x7f044c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.642+0000 7f04675ff700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f044c0a0640 0x7f044c0a2b00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e40 tx=0x7f0450009260 comp rx=0 tx=0).stop 2023-09-10T01:02:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.642+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601006f0 msgr2=0x7f04601963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.642+0000 7f04675ff700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601006f0 0x7f04601963f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f045c000c00 tx=0x7f045c005720 comp rx=0 tx=0).stop 2023-09-10T01:02:44.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 shutdown_connections 2023-09-10T01:02:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0460102280 0x7f046019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04601006f0 0x7f04601963f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f044c0a0640 0x7f044c0a2b00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 --2- 172.21.15.3:0/108413806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04601018e0 0x7f0460196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 >> 172.21.15.3:0/108413806 conn(0x7f04600fbc60 msgr2=0x7f0460104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 shutdown_connections 2023-09-10T01:02:44.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.643+0000 7f04675ff700 1 -- 172.21.15.3:0/108413806 wait complete. 2023-09-10T01:02:44.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:02:44.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 -- 172.21.15.3:0/348636891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 msgr2=0x7f332c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 --2- 172.21.15.3:0/348636891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c101d10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f33200099c0 tx=0x7f3320009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 -- 172.21.15.3:0/348636891 shutdown_connections 2023-09-10T01:02:44.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 --2- 172.21.15.3:0/348636891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c1006c0 0x7f332c100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 --2- 172.21.15.3:0/348636891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f332c102250 0x7f332c108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 --2- 172.21.15.3:0/348636891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c101d10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 -- 172.21.15.3:0/348636891 >> 172.21.15.3:0/348636891 conn(0x7f332c0fbc30 msgr2=0x7f332c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:44.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 -- 172.21.15.3:0/348636891 shutdown_connections 2023-09-10T01:02:44.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.865+0000 7f33321ff700 1 -- 172.21.15.3:0/348636891 wait complete. 2023-09-10T01:02:44.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.866+0000 7f33321ff700 1 Processor -- start 2023-09-10T01:02:44.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.866+0000 7f33321ff700 1 -- start start 2023-09-10T01:02:44.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.866+0000 7f33321ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f332c1006c0 0x7f332c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f33321ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f33321ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102250 0x7f332c19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f33321ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c19df30 con 0x7f332c1018b0 2023-09-10T01:02:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f33321ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c19e070 con 0x7f332c1006c0 2023-09-10T01:02:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f33321ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c19e200 con 0x7f332c102250 2023-09-10T01:02:44.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f332c1006c0 0x7f332c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c1968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41406/0 (socket says 172.21.15.3:41406) 2023-09-10T01:02:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f332c1006c0 0x7f332c1963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53114/0 (socket says 172.21.15.3:53114) 2023-09-10T01:02:44.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 -- 172.21.15.3:0/1663559319 learned_addr learned my addr 172.21.15.3:0/1663559319 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:44.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332bfff700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102250 0x7f332c19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 -- 172.21.15.3:0/1663559319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102250 msgr2=0x7f332c19d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102250 0x7f332c19d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 -- 172.21.15.3:0/1663559319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f332c1006c0 msgr2=0x7f332c1963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:44.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f332c1006c0 0x7f332c1963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:44.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.867+0000 7f332affd700 1 -- 172.21.15.3:0/1663559319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c009520 con 0x7f332c1018b0 2023-09-10T01:02:44.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.868+0000 7f332affd700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c1968f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f332000c010 tx=0x7f332000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:44.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.868+0000 7f3328ff9700 1 -- 172.21.15.3:0/1663559319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332001d070 con 0x7f332c1018b0 2023-09-10T01:02:44.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.868+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3324009c40 con 0x7f332c1018b0 2023-09-10T01:02:44.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.868+0000 7f3328ff9700 1 -- 172.21.15.3:0/1663559319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3320005050 con 0x7f332c1018b0 2023-09-10T01:02:44.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.868+0000 7f3328ff9700 1 -- 172.21.15.3:0/1663559319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320004ab0 con 0x7f332c1018b0 2023-09-10T01:02:44.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.868+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33200095f0 con 0x7f332c1018b0 2023-09-10T01:02:44.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.869+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c10bed0 con 0x7f332c1018b0 2023-09-10T01:02:44.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.871+0000 7f3328ff9700 1 -- 172.21.15.3:0/1663559319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f33200051c0 con 0x7f332c1018b0 2023-09-10T01:02:44.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.873+0000 7f3328ff9700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33140a0640 0x7f33140a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:44.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.874+0000 7f3328ff9700 1 -- 172.21.15.3:0/1663559319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f33200cce40 con 0x7f332c1018b0 2023-09-10T01:02:44.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.874+0000 7f332b7fe700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33140a0640 0x7f33140a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:44.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.874+0000 7f332b7fe700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33140a0640 0x7f33140a2b00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f332c101710 tx=0x7f331c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:44.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:44.877+0000 7f3328ff9700 1 -- 172.21.15.3:0/1663559319 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f33200965f0 con 0x7f332c1018b0 2023-09-10T01:02:45.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.133+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f332c10c0c0 con 0x7f332c1018b0 2023-09-10T01:02:45.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.134+0000 7f3328ff9700 1 -- 172.21.15.3:0/1663559319 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f3320099c10 con 0x7f332c1018b0 2023-09-10T01:02:45.135 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:45.135 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:02:45.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33140a0640 msgr2=0x7f33140a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33140a0640 0x7f33140a2b00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f332c101710 tx=0x7f331c009260 comp rx=0 tx=0).stop 2023-09-10T01:02:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 msgr2=0x7f332c1968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c1968f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f332000c010 tx=0x7f332000f690 comp rx=0 tx=0).stop 2023-09-10T01:02:45.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 shutdown_connections 2023-09-10T01:02:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f33140a0640 0x7f33140a2b00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102250 0x7f332c19d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f332c1006c0 0x7f332c1963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 --2- 172.21.15.3:0/1663559319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f332c1018b0 0x7f332c1968f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 >> 172.21.15.3:0/1663559319 conn(0x7f332c0fbc30 msgr2=0x7f332c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 shutdown_connections 2023-09-10T01:02:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.137+0000 7f33321ff700 1 -- 172.21.15.3:0/1663559319 wait complete. 2023-09-10T01:02:45.140 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:02:45.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:45 smithi003 ceph-mon[97654]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:45 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2172652139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:45 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/108413806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:45.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.359+0000 7f9085fff700 1 -- 172.21.15.3:0/1654273400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90801006c0 msgr2=0x7f9080100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.359+0000 7f9085fff700 1 --2- 172.21.15.3:0/1654273400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90801006c0 0x7f9080100ae0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f90700099c0 tx=0x7f9070009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.360+0000 7f9085fff700 1 -- 172.21.15.3:0/1654273400 shutdown_connections 2023-09-10T01:02:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.360+0000 7f9085fff700 1 --2- 172.21.15.3:0/1654273400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9080102250 0x7f9080108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.360+0000 7f9085fff700 1 --2- 172.21.15.3:0/1654273400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90801006c0 0x7f9080100ae0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.360+0000 7f9085fff700 1 --2- 172.21.15.3:0/1654273400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90801018b0 0x7f9080101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.360+0000 7f9085fff700 1 -- 172.21.15.3:0/1654273400 >> 172.21.15.3:0/1654273400 conn(0x7f90800fbc30 msgr2=0x7f90800fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.360+0000 7f9085fff700 1 -- 172.21.15.3:0/1654273400 shutdown_connections 2023-09-10T01:02:45.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.360+0000 7f9085fff700 1 -- 172.21.15.3:0/1654273400 wait complete. 2023-09-10T01:02:45.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 Processor -- start 2023-09-10T01:02:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 -- start start 2023-09-10T01:02:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90801006c0 0x7f9080198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90801018b0 0x7f9080198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9080102250 0x7f908019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90801a0180 con 0x7f9080102250 2023-09-10T01:02:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90801a02c0 con 0x7f90801006c0 2023-09-10T01:02:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f9085fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90801a0450 con 0x7f90801018b0 2023-09-10T01:02:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f907ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9080102250 0x7f908019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f907f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90801006c0 0x7f9080198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.361+0000 7f907f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90801006c0 0x7f9080198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53136/0 (socket says 172.21.15.3:53136) 2023-09-10T01:02:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907f7fe700 1 -- 172.21.15.3:0/298193382 learned_addr learned my addr 172.21.15.3:0/298193382 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907effd700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90801018b0 0x7f9080198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907ffff700 1 -- 172.21.15.3:0/298193382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90801018b0 msgr2=0x7f9080198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907ffff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90801018b0 0x7f9080198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907ffff700 1 -- 172.21.15.3:0/298193382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90801006c0 msgr2=0x7f9080198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907ffff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90801006c0 0x7f9080198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907ffff700 1 -- 172.21.15.3:0/298193382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700095f0 con 0x7f9080102250 2023-09-10T01:02:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907f7fe700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90801006c0 0x7f9080198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907ffff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9080102250 0x7f908019fbb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f907800efc0 tx=0x7f907800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907cff9700 1 -- 172.21.15.3:0/298193382 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078018070 con 0x7f9080102250 2023-09-10T01:02:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907cff9700 1 -- 172.21.15.3:0/298193382 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f907800f040 con 0x7f9080102250 2023-09-10T01:02:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90801a06d0 con 0x7f9080102250 2023-09-10T01:02:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.362+0000 7f907cff9700 1 -- 172.21.15.3:0/298193382 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907800cd40 con 0x7f9080102250 2023-09-10T01:02:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.363+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90801a0be0 con 0x7f9080102250 2023-09-10T01:02:45.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.364+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9080192d10 con 0x7f9080102250 2023-09-10T01:02:45.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.365+0000 7f907cff9700 1 -- 172.21.15.3:0/298193382 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f9078003a00 con 0x7f9080102250 2023-09-10T01:02:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.367+0000 7f907cff9700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90680a0640 0x7f90680a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.367+0000 7f907cff9700 1 -- 172.21.15.3:0/298193382 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f90780cbac0 con 0x7f9080102250 2023-09-10T01:02:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.367+0000 7f907f7fe700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90680a0640 0x7f90680a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.368+0000 7f907f7fe700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90680a0640 0x7f90680a2b00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9070000c00 tx=0x7f9070005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.372+0000 7f907cff9700 1 -- 172.21.15.3:0/298193382 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9078095270 con 0x7f9080102250 2023-09-10T01:02:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:45 smithi136 ceph-mon[133708]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:45 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2172652139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:45 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/108413806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:45.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:45 smithi092 ceph-mon[134067]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:45.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:45 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2172652139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:45.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:45 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/108413806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:45.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.583+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f908010c140 con 0x7f9080102250 2023-09-10T01:02:45.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.585+0000 7f907cff9700 1 -- 172.21.15.3:0/298193382 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7f907801b320 con 0x7f9080102250 2023-09-10T01:02:45.586 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:45.586 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:45.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90680a0640 msgr2=0x7f90680a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90680a0640 0x7f90680a2b00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9070000c00 tx=0x7f9070005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9080102250 msgr2=0x7f908019fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9080102250 0x7f908019fbb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f907800efc0 tx=0x7f907800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 shutdown_connections 2023-09-10T01:02:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9080102250 0x7f908019fbb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90801006c0 0x7f9080198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f90680a0640 0x7f90680a2b00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 --2- 172.21.15.3:0/298193382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90801018b0 0x7f9080198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 >> 172.21.15.3:0/298193382 conn(0x7f90800fbc30 msgr2=0x7f9080107200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 shutdown_connections 2023-09-10T01:02:45.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.587+0000 7f9085fff700 1 -- 172.21.15.3:0/298193382 wait complete. 2023-09-10T01:02:45.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-09-10T01:00:43.531603+0000', 'modified': '2023-09-10T01:01:09.940177+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24698, 'mds_1': 24647, 'mds_2': 24634}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24604': {'gid': 24604, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6835/662717057', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6834', 'nonce': 662717057}, {'type': 'v1', 'addr': '172.21.15.136:6835', 'nonce': 662717057}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24610': {'gid': 24610, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6837/3841438021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6836', 'nonce': 3841438021}, {'type': 'v1', 'addr': '172.21.15.136:6837', 'nonce': 3841438021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24620': {'gid': 24620, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.92:6835/790897161', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 790897161}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 790897161}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24634': {'gid': 24634, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.136:6839/1259564848', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 1259564848}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 1259564848}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24647': {'gid': 24647, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.92:6837/12010426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6836', 'nonce': 12010426}, {'type': 'v1', 'addr': '172.21.15.92:6837', 'nonce': 12010426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24698': {'gid': 24698, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6839/149607042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 149607042}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 149607042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-09-10T01:02:45.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-09-10T01:02:45.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-09-10T01:02:45.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 -- 172.21.15.3:0/946871405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1381006f0 msgr2=0x7ff138100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/946871405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1381006f0 0x7ff138100b10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff1280099c0 tx=0x7ff128009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 -- 172.21.15.3:0/946871405 shutdown_connections 2023-09-10T01:02:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/946871405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff138102280 0x7ff138108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/946871405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1381006f0 0x7ff138100b10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/946871405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1381018e0 0x7ff138101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 -- 172.21.15.3:0/946871405 >> 172.21.15.3:0/946871405 conn(0x7ff1380fbc60 msgr2=0x7ff1380fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 -- 172.21.15.3:0/946871405 shutdown_connections 2023-09-10T01:02:45.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.812+0000 7ff13d7ff700 1 -- 172.21.15.3:0/946871405 wait complete. 2023-09-10T01:02:45.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.813+0000 7ff13d7ff700 1 Processor -- start 2023-09-10T01:02:45.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.813+0000 7ff13d7ff700 1 -- start start 2023-09-10T01:02:45.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff13d7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1381006f0 0x7ff138198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff13d7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1381018e0 0x7ff138198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff13d7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff138102280 0x7ff13819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff13d7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1381a00d0 con 0x7ff138102280 2023-09-10T01:02:45.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff13d7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1381a0210 con 0x7ff1381006f0 2023-09-10T01:02:45.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff13d7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1381a03a0 con 0x7ff1381018e0 2023-09-10T01:02:45.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1381006f0 0x7ff138198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1381006f0 0x7ff138198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53166/0 (socket says 172.21.15.3:53166) 2023-09-10T01:02:45.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff136ffd700 1 -- 172.21.15.3:0/1638640715 learned_addr learned my addr 172.21.15.3:0/1638640715 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:45.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff1377fe700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff138102280 0x7ff13819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff1367fc700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1381018e0 0x7ff138198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff1377fe700 1 -- 172.21.15.3:0/1638640715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1381018e0 msgr2=0x7ff138198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff1377fe700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1381018e0 0x7ff138198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff1377fe700 1 -- 172.21.15.3:0/1638640715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1381006f0 msgr2=0x7ff138198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff1377fe700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1381006f0 0x7ff138198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.814+0000 7ff1377fe700 1 -- 172.21.15.3:0/1638640715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280095f0 con 0x7ff138102280 2023-09-10T01:02:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.815+0000 7ff1377fe700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff138102280 0x7ff13819fb00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff13000efc0 tx=0x7ff13000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.815+0000 7ff11ffff700 1 -- 172.21.15.3:0/1638640715 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff130018070 con 0x7ff138102280 2023-09-10T01:02:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.815+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1381a0620 con 0x7ff138102280 2023-09-10T01:02:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.815+0000 7ff11ffff700 1 -- 172.21.15.3:0/1638640715 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff13000f040 con 0x7ff138102280 2023-09-10T01:02:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.815+0000 7ff11ffff700 1 -- 172.21.15.3:0/1638640715 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13000cd40 con 0x7ff138102280 2023-09-10T01:02:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.815+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1381a0b80 con 0x7ff138102280 2023-09-10T01:02:45.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.818+0000 7ff11ffff700 1 -- 172.21.15.3:0/1638640715 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7ff130003a00 con 0x7ff138102280 2023-09-10T01:02:45.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.819+0000 7ff15107db80 1 Processor -- start 2023-09-10T01:02:45.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.819+0000 7ff11ffff700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1200a0640 0x7ff1200a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.819+0000 7ff15107db80 1 -- start start 2023-09-10T01:02:45.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff11ffff700 1 -- 172.21.15.3:0/1638640715 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7ff1300cbac0 con 0x7ff138102280 2023-09-10T01:02:45.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff15107db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f5c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff15107db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5cda0 0x5654c3f5afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff15107db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f5ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff15107db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3eab6e0 con 0x5654c3f5b590 2023-09-10T01:02:45.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff15107db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3ea9000 con 0x5654c3f5cda0 2023-09-10T01:02:45.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff15107db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3ea48f0 con 0x5654c3f5c420 2023-09-10T01:02:45.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff1367fc700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1200a0640 0x7ff1200a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff1367fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5cda0 0x5654c3f5afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f5c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff1377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f5ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f5c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37468/0 (socket says 172.21.15.3:37468) 2023-09-10T01:02:45.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff1367fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5cda0 0x5654c3f5afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53182/0 (socket says 172.21.15.3:53182) 2023-09-10T01:02:45.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.820+0000 7ff136ffd700 1 -- 172.21.15.3:0/3844162278 learned_addr learned my addr 172.21.15.3:0/3844162278 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:45.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff136ffd700 1 -- 172.21.15.3:0/3844162278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5cda0 msgr2=0x5654c3f5afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff136ffd700 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5cda0 0x5654c3f5afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff136ffd700 1 -- 172.21.15.3:0/3844162278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 msgr2=0x5654c3f5ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff136ffd700 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f5ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff136ffd700 1 -- 172.21.15.3:0/3844162278 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654c3cd0d90 con 0x5654c3f5c420 2023-09-10T01:02:45.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff1377fe700 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f5ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:45.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff1367fc700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1200a0640 0x7ff1200a2b00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff12c012500 tx=0x7ff12c007f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff136ffd700 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f5c840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff128005c70 tx=0x7ff128005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76cf90c18f149e8c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff11d7fa700 1 -- 172.21.15.3:0/3844162278 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12800ba90 con 0x5654c3f5c420 2023-09-10T01:02:45.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff11d7fa700 1 -- 172.21.15.3:0/3844162278 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff12801e040 con 0x5654c3f5c420 2023-09-10T01:02:45.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.821+0000 7ff11d7fa700 1 -- 172.21.15.3:0/3844162278 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128003b50 con 0x5654c3f5c420 2023-09-10T01:02:45.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 -- 172.21.15.3:0/3844162278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 msgr2=0x5654c3f5c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f5c840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff128005c70 tx=0x7ff128005f00 comp rx=0 tx=0).stop 2023-09-10T01:02:45.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 -- 172.21.15.3:0/3844162278 shutdown_connections 2023-09-10T01:02:45.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f5c840 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f5ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 --2- 172.21.15.3:0/3844162278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5cda0 0x5654c3f5afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 -- 172.21.15.3:0/3844162278 >> 172.21.15.3:0/3844162278 conn(0x5654c3ec7040 msgr2=0x5654c3f4d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 -- 172.21.15.3:0/3844162278 shutdown_connections 2023-09-10T01:02:45.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.822+0000 7ff15107db80 1 -- 172.21.15.3:0/3844162278 wait complete. 2023-09-10T01:02:45.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.823+0000 7ff15107db80 1 -- start start 2023-09-10T01:02:45.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.823+0000 7ff15107db80 1 Processor -- start 2023-09-10T01:02:45.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff15107db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f64d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff15107db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5c420 0x5654c3f652a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff15107db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5cda0 0x5654c3f6c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff15107db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3e42b20 con 0x5654c3f5b590 2023-09-10T01:02:45.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff15107db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3e42820 con 0x5654c3f5c420 2023-09-10T01:02:45.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff15107db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3d268d0 con 0x5654c3f5cda0 2023-09-10T01:02:45.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff15107db80 10 client.0 _ll_register_callbacks cb 0x5654c3d8ca30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:45.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f64d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f64d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41464/0 (socket says 172.21.15.3:41464) 2023-09-10T01:02:45.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff136ffd700 1 -- 172.21.15.3:0/2196065528 learned_addr learned my addr 172.21.15.3:0/2196065528 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:45.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff1367fc700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5c420 0x5654c3f652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.824+0000 7ff1377fe700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5cda0 0x5654c3f6c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff136ffd700 1 -- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5cda0 msgr2=0x5654c3f6c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff136ffd700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5cda0 0x5654c3f6c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff136ffd700 1 -- 172.21.15.3:0/2196065528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5c420 msgr2=0x5654c3f652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff136ffd700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5c420 0x5654c3f652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff136ffd700 1 -- 172.21.15.3:0/2196065528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1381a0b80 con 0x5654c3f5b590 2023-09-10T01:02:45.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff1377fe700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5cda0 0x5654c3f6c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:45.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff1367fc700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5c420 0x5654c3f652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:45.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff136ffd700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f64d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff128025040 tx=0x7ff128003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56049458b53dcb8f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff11aff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:02:45.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff11aff5700 1 -- 172.21.15.3:0/2196065528 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280052c0 con 0x5654c3f5b590 2023-09-10T01:02:45.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.825+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654c3dfc4a0 con 0x5654c3f5b590 2023-09-10T01:02:45.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff11aff5700 1 -- 172.21.15.3:0/2196065528 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7ff128005420 con 0x5654c3f5b590 2023-09-10T01:02:45.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff11aff5700 1 -- 172.21.15.3:0/2196065528 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff12800bc90 con 0x5654c3f5b590 2023-09-10T01:02:45.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 10 client.15396 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:45.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7ff138109480 con 0x5654c3f5b590 2023-09-10T01:02:45.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff11aff5700 1 -- 172.21.15.3:0/2196065528 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5654c3dfc4a0 con 0x5654c3f5b590 2023-09-10T01:02:45.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff11aff5700 1 -- 172.21.15.3:0/2196065528 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff12802b420 con 0x5654c3f5b590 2023-09-10T01:02:45.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 10 client.15396 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:45.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 10 client.15396 resolve_mds: resolved name 'i' to daemon mds.i 2023-09-10T01:02:45.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 20 client.15396 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3f6fed0 0x5654c3f74e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 4 client.15396 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:02:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.826+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654c3eb9850 con 0x5654c3f6fed0 2023-09-10T01:02:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.827+0000 7ff1367fc700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3f6fed0 0x5654c3f74e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.827+0000 7ff1367fc700 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3f6fed0 0x5654c3f74e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=33d76751b6238c2e server_cookie=b2efb80464784389 in_seq=0 out_seq=0 2023-09-10T01:02:45.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.827+0000 7ff11aff5700 10 client.15396 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:45.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.828+0000 7ff11aff5700 1 -- 172.21.15.3:0/2196065528 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5654c3eb9850 con 0x5654c3f6fed0 2023-09-10T01:02:45.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.828+0000 7ff11aff5700 10 client.15396 handle_command_reply: tid=0 2023-09-10T01:02:45.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.828+0000 7ff15107db80 1 client.15396 shutdown 2023-09-10T01:02:45.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.828+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 msgr2=0x5654c3f64d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.828+0000 7ff15107db80 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f64d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff128025040 tx=0x7ff128003b00 comp rx=0 tx=0).stop 2023-09-10T01:02:45.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.828+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 shutdown_connections 2023-09-10T01:02:45.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.828+0000 7ff15107db80 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3f6fed0 0x5654c3f74e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff15107db80 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5cda0 0x5654c3f6c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff15107db80 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5b590 0x5654c3f64d60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff15107db80 1 --2- 172.21.15.3:0/2196065528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5c420 0x5654c3f652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff11aff5700 0 client.15396 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:45.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 >> 172.21.15.3:0/2196065528 conn(0x5654c3ec7040 msgr2=0x5654c3f4cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 shutdown_connections 2023-09-10T01:02:45.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff15107db80 1 -- 172.21.15.3:0/2196065528 wait complete. 2023-09-10T01:02:45.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.829+0000 7ff15107db80 20 client.15396 trim_cache size 0 max 16384 2023-09-10T01:02:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 Processor -- start 2023-09-10T01:02:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 -- start start 2023-09-10T01:02:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f63a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f62bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3eb9850 con 0x5654c3f5cda0 2023-09-10T01:02:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3dfc4a0 con 0x5654c3f5b590 2023-09-10T01:02:45.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.847+0000 7ff15107db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3f712c0 con 0x5654c3f5c420 2023-09-10T01:02:45.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f62bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f62bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41476/0 (socket says 172.21.15.3:41476) 2023-09-10T01:02:45.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f63a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 -- 172.21.15.3:0/574175407 learned_addr learned my addr 172.21.15.3:0/574175407 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1367fc700 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 -- 172.21.15.3:0/574175407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 msgr2=0x5654c3f6c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 -- 172.21.15.3:0/574175407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 msgr2=0x5654c3f63a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f63a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 -- 172.21.15.3:0/574175407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654c3cb9c10 con 0x5654c3f5cda0 2023-09-10T01:02:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff136ffd700 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f63a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.848+0000 7ff1377fe700 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f62bc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff1300090a0 tx=0x7ff130098af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da474a4993a28622 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff11dffb700 1 -- 172.21.15.3:0/574175407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13001b320 con 0x5654c3f5cda0 2023-09-10T01:02:45.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff11dffb700 1 -- 172.21.15.3:0/574175407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff130004ad0 con 0x5654c3f5cda0 2023-09-10T01:02:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff11dffb700 1 -- 172.21.15.3:0/574175407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff130098420 con 0x5654c3f5cda0 2023-09-10T01:02:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 -- 172.21.15.3:0/574175407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 msgr2=0x5654c3f62bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f62bc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff1300090a0 tx=0x7ff130098af0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 -- 172.21.15.3:0/574175407 shutdown_connections 2023-09-10T01:02:45.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f62bc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f63a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 --2- 172.21.15.3:0/574175407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 -- 172.21.15.3:0/574175407 >> 172.21.15.3:0/574175407 conn(0x5654c3ec7040 msgr2=0x5654c3f4d640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 -- 172.21.15.3:0/574175407 shutdown_connections 2023-09-10T01:02:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.849+0000 7ff15107db80 1 -- 172.21.15.3:0/574175407 wait complete. 2023-09-10T01:02:45.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 -- start start 2023-09-10T01:02:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 Processor -- start 2023-09-10T01:02:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f692a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f6bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3f65690 con 0x5654c3f5cda0 2023-09-10T01:02:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3f6c270 con 0x5654c3f5b590 2023-09-10T01:02:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.850+0000 7ff15107db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c3f6c400 con 0x5654c3f5c420 2023-09-10T01:02:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff15107db80 10 client.0 _ll_register_callbacks cb 0x7ff130004cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f6bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff136ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f692a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f6bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41480/0 (socket says 172.21.15.3:41480) 2023-09-10T01:02:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1367fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 -- 172.21.15.3:0/4201427482 learned_addr learned my addr 172.21.15.3:0/4201427482 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 -- 172.21.15.3:0/4201427482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 msgr2=0x5654c3f6b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 -- 172.21.15.3:0/4201427482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 msgr2=0x5654c3f692a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f692a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 -- 172.21.15.3:0/4201427482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff138109480 con 0x5654c3f5cda0 2023-09-10T01:02:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff1377fe700 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f6bd30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff130099590 tx=0x7ff13001b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409b2f0115ea6228 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.851+0000 7ff11aff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:02:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff11aff5700 1 -- 172.21.15.3:0/4201427482 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1300a0c80 con 0x5654c3f5cda0 2023-09-10T01:02:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff12801e040 con 0x5654c3f5cda0 2023-09-10T01:02:45.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff11aff5700 1 -- 172.21.15.3:0/4201427482 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7ff130099630 con 0x5654c3f5cda0 2023-09-10T01:02:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff11aff5700 1 -- 172.21.15.3:0/4201427482 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff130098f30 con 0x5654c3f5cda0 2023-09-10T01:02:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 10 client.15402 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7ff1381a0620 con 0x5654c3f5cda0 2023-09-10T01:02:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff11aff5700 1 -- 172.21.15.3:0/4201427482 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff12801e040 con 0x5654c3f5cda0 2023-09-10T01:02:45.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff11aff5700 1 -- 172.21.15.3:0/4201427482 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff130098000 con 0x5654c3f5cda0 2023-09-10T01:02:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 10 client.15402 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 10 client.15402 resolve_mds: resolved name 'i' to daemon mds.i 2023-09-10T01:02:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 20 client.15402 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3fc8d80 0x5654c3fc5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:45.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 4 client.15402 mds_command: new command op to 24634 tid=0[{"prefix": "status"}] 2023-09-10T01:02:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.852+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5654c3e9d8d0 con 0x5654c3fc8d80 2023-09-10T01:02:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff136ffd700 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3fc8d80 0x5654c3fc5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff136ffd700 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3fc8d80 0x5654c3fc5260 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=569d0e2567a4a6c8 server_cookie=45ed7fd73f134779 in_seq=0 out_seq=0 2023-09-10T01:02:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff11aff5700 10 client.15402 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:45.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff11aff5700 1 -- 172.21.15.3:0/4201427482 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5654c3e9d8d0 con 0x5654c3fc8d80 2023-09-10T01:02:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff11aff5700 10 client.15402 handle_command_reply: tid=0 2023-09-10T01:02:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff15107db80 1 client.15402 shutdown 2023-09-10T01:02:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 msgr2=0x5654c3f6bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.853+0000 7ff15107db80 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f6bd30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff130099590 tx=0x7ff13001b320 comp rx=0 tx=0).stop 2023-09-10T01:02:45.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 shutdown_connections 2023-09-10T01:02:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5654c3fc8d80 0x5654c3fc5260 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5654c3f5cda0 0x5654c3f6bd30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5654c3f5b590 0x5654c3f692a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 --2- 172.21.15.3:0/4201427482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654c3f5c420 0x5654c3f6b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff11aff5700 0 client.15402 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:02:45.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 >> 172.21.15.3:0/4201427482 conn(0x5654c3ec7040 msgr2=0x5654c3f63a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 shutdown_connections 2023-09-10T01:02:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 1 -- 172.21.15.3:0/4201427482 wait complete. 2023-09-10T01:02:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff15107db80 20 client.15402 trim_cache size 0 max 16384 2023-09-10T01:02:45.869 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:02:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T01:02:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 2, 2023-09-10T01:02:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24634, 2023-09-10T01:02:45.870 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2023-09-10T01:02:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2023-09-10T01:02:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2023-09-10T01:02:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 96.825412227000001, 2023-09-10T01:02:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 22, 2023-09-10T01:02:45.871 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 90, 2023-09-10T01:02:45.872 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 88, 2023-09-10T01:02:45.872 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 135.601557337 2023-09-10T01:02:45.872 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:02:45.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1200a0640 msgr2=0x7ff1200a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1200a0640 0x7ff1200a2b00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff12c012500 tx=0x7ff12c007f90 comp rx=0 tx=0).stop 2023-09-10T01:02:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff138102280 msgr2=0x7ff13819fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.854+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff138102280 0x7ff13819fb00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff13000efc0 tx=0x7ff13000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 shutdown_connections 2023-09-10T01:02:45.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff138102280 0x7ff13819fb00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1381006f0 0x7ff138198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7ff1200a0640 0x7ff1200a2b00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 --2- 172.21.15.3:0/1638640715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1381018e0 0x7ff138198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 >> 172.21.15.3:0/1638640715 conn(0x7ff1380fbc60 msgr2=0x7ff138107160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 shutdown_connections 2023-09-10T01:02:45.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:45.855+0000 7ff13d7ff700 1 -- 172.21.15.3:0/1638640715 wait complete. 2023-09-10T01:02:45.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-09-10T01:02:46.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 -- 172.21.15.3:0/1003262949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea50102280 msgr2=0x7fea50108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 --2- 172.21.15.3:0/1003262949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea50102280 0x7fea50108b30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fea4800b0d0 tx=0x7fea4800b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 -- 172.21.15.3:0/1003262949 shutdown_connections 2023-09-10T01:02:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 --2- 172.21.15.3:0/1003262949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea501006f0 0x7fea50100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 --2- 172.21.15.3:0/1003262949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea50102280 0x7fea50108b30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 --2- 172.21.15.3:0/1003262949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea501018e0 0x7fea50101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 -- 172.21.15.3:0/1003262949 >> 172.21.15.3:0/1003262949 conn(0x7fea500fbc60 msgr2=0x7fea500fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 -- 172.21.15.3:0/1003262949 shutdown_connections 2023-09-10T01:02:46.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.081+0000 7fea569ff700 1 -- 172.21.15.3:0/1003262949 wait complete. 2023-09-10T01:02:46.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.082+0000 7fea569ff700 1 Processor -- start 2023-09-10T01:02:46.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.082+0000 7fea569ff700 1 -- start start 2023-09-10T01:02:46.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea569ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501006f0 0x7fea50198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea569ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea501018e0 0x7fea50198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea569ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea50102280 0x7fea5019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea569ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea501a02c0 con 0x7fea501006f0 2023-09-10T01:02:46.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea569ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea501a0400 con 0x7fea50102280 2023-09-10T01:02:46.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea569ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea501a0590 con 0x7fea501018e0 2023-09-10T01:02:46.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501006f0 0x7fea50198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501006f0 0x7fea50198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41490/0 (socket says 172.21.15.3:41490) 2023-09-10T01:02:46.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea4ffff700 1 -- 172.21.15.3:0/2774231599 learned_addr learned my addr 172.21.15.3:0/2774231599 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea501018e0 0x7fea50198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea54f9c700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea50102280 0x7fea5019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea4ffff700 1 -- 172.21.15.3:0/2774231599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea501018e0 msgr2=0x7fea50198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.083+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea501018e0 0x7fea50198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea4ffff700 1 -- 172.21.15.3:0/2774231599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea50102280 msgr2=0x7fea5019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea50102280 0x7fea5019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea4ffff700 1 -- 172.21.15.3:0/2774231599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea48009d00 con 0x7fea501006f0 2023-09-10T01:02:46.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501006f0 0x7fea50198740 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fea4000b500 tx=0x7fea4000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea4d7fa700 1 -- 172.21.15.3:0/2774231599 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40011830 con 0x7fea501006f0 2023-09-10T01:02:46.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea501a0870 con 0x7fea501006f0 2023-09-10T01:02:46.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea4d7fa700 1 -- 172.21.15.3:0/2774231599 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea40011e50 con 0x7fea501006f0 2023-09-10T01:02:46.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea4d7fa700 1 -- 172.21.15.3:0/2774231599 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40010dc0 con 0x7fea501006f0 2023-09-10T01:02:46.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.084+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea501a0e50 con 0x7fea501006f0 2023-09-10T01:02:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.087+0000 7fea4d7fa700 1 -- 172.21.15.3:0/2774231599 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fea40010380 con 0x7fea501006f0 2023-09-10T01:02:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.088+0000 7fea6a402b80 1 Processor -- start 2023-09-10T01:02:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea6a402b80 1 -- start start 2023-09-10T01:02:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea4d7fa700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fea380a0640 0x7fea380a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584074efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x55584074e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea4d7fa700 1 -- 172.21.15.3:0/2774231599 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fea400cb9f0 con 0x7fea501006f0 2023-09-10T01:02:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea6a402b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558406a66e0 con 0x55584074ebb0 2023-09-10T01:02:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea6a402b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558406a4000 con 0x55584074f550 2023-09-10T01:02:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea6a402b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55584069f8f0 con 0x55584074ddb0 2023-09-10T01:02:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584074efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584074efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41498/0 (socket says 172.21.15.3:41498) 2023-09-10T01:02:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea4ffff700 1 -- 172.21.15.3:0/2504528045 learned_addr learned my addr 172.21.15.3:0/2504528045 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.089+0000 7fea54f9c700 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x55584074e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea54f9c700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fea380a0640 0x7fea380a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea4ffff700 1 -- 172.21.15.3:0/2504528045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 msgr2=0x55584074e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x55584074e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea4ffff700 1 -- 172.21.15.3:0/2504528045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 msgr2=0x55584074d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea4ffff700 1 -- 172.21.15.3:0/2504528045 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558404cbd90 con 0x55584074ebb0 2023-09-10T01:02:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584074efd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fea400952d0 tx=0x7fea4000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5e53966c0d912d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea54f9c700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fea380a0640 0x7fea380a2b00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea48005bd0 tx=0x7fea4800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea367fc700 1 -- 172.21.15.3:0/2504528045 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40098dd0 con 0x55584074ebb0 2023-09-10T01:02:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea367fc700 1 -- 172.21.15.3:0/2504528045 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea4009cc00 con 0x55584074ebb0 2023-09-10T01:02:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.090+0000 7fea367fc700 1 -- 172.21.15.3:0/2504528045 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea400a09b0 con 0x55584074ebb0 2023-09-10T01:02:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 -- 172.21.15.3:0/2504528045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 msgr2=0x55584074efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584074efd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fea400952d0 tx=0x7fea4000bf10 comp rx=0 tx=0).stop 2023-09-10T01:02:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 -- 172.21.15.3:0/2504528045 shutdown_connections 2023-09-10T01:02:46.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x55584074e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584074efd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2504528045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 -- 172.21.15.3:0/2504528045 >> 172.21.15.3:0/2504528045 conn(0x5558406c2040 msgr2=0x5558407486d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 -- 172.21.15.3:0/2504528045 shutdown_connections 2023-09-10T01:02:46.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.091+0000 7fea6a402b80 1 -- 172.21.15.3:0/2504528045 wait complete. 2023-09-10T01:02:46.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.092+0000 7fea6a402b80 1 -- start start 2023-09-10T01:02:46.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.092+0000 7fea6a402b80 1 Processor -- start 2023-09-10T01:02:46.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.092+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x555840757560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.092+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ebb0 0x555840757aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074f550 0x55584075eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea6a402b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55584063db20 con 0x55584074f550 2023-09-10T01:02:46.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea6a402b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55584063d820 con 0x55584074ebb0 2023-09-10T01:02:46.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea6a402b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558405218d0 con 0x55584074ddb0 2023-09-10T01:02:46.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea6a402b80 10 client.0 _ll_register_callbacks cb 0x555840587a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:46.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ebb0 0x555840757aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea54f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074f550 0x55584075eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x555840757560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ebb0 0x555840757aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53266/0 (socket says 172.21.15.3:53266) 2023-09-10T01:02:46.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 -- 172.21.15.3:0/4204019916 learned_addr learned my addr 172.21.15.3:0/4204019916 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 -- 172.21.15.3:0/4204019916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 msgr2=0x555840757560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x555840757560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 -- 172.21.15.3:0/4204019916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074f550 msgr2=0x55584075eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074f550 0x55584075eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 -- 172.21.15.3:0/4204019916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea501a0e50 con 0x55584074ebb0 2023-09-10T01:02:46.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea54f9c700 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074f550 0x55584075eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ebb0 0x555840757aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea44012770 tx=0x7fea44012a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b79b196af0d9e374 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.093+0000 7fea2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:02:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea2f7fe700 1 -- 172.21.15.3:0/4204019916 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea44017070 con 0x55584074ebb0 2023-09-10T01:02:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558405f74a0 con 0x55584074ebb0 2023-09-10T01:02:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea2f7fe700 1 -- 172.21.15.3:0/4204019916 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fea4400f480 con 0x55584074ebb0 2023-09-10T01:02:46.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea2f7fe700 1 -- 172.21.15.3:0/4204019916 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea4400f640 con 0x55584074ebb0 2023-09-10T01:02:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea6a402b80 10 client.25454 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5558404b4c10 con 0x55584074ebb0 2023-09-10T01:02:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea2f7fe700 1 -- 172.21.15.3:0/4204019916 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5558405f74a0 con 0x55584074ebb0 2023-09-10T01:02:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea2f7fe700 1 -- 172.21.15.3:0/4204019916 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fea440107d0 con 0x55584074ebb0 2023-09-10T01:02:46.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.094+0000 7fea6a402b80 10 client.25454 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea6a402b80 10 client.25454 resolve_mds: resolved name 'h' to daemon mds.h 2023-09-10T01:02:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea6a402b80 20 client.25454 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea6a402b80 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x555840762d00 0x555840761c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea6a402b80 4 client.25454 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:02:46.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558406b4850 con 0x555840762d00 2023-09-10T01:02:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea4ffff700 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x555840762d00 0x555840761c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea4ffff700 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x555840762d00 0x555840761c40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25b990d35cc07ffc server_cookie=1bf88fdb4777464b in_seq=0 out_seq=0 2023-09-10T01:02:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.095+0000 7fea2f7fe700 10 client.25454 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea2f7fe700 1 -- 172.21.15.3:0/4204019916 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5558406b4850 con 0x555840762d00 2023-09-10T01:02:46.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea2f7fe700 10 client.25454 handle_command_reply: tid=0 2023-09-10T01:02:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 client.25454 shutdown 2023-09-10T01:02:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ebb0 msgr2=0x555840757aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ebb0 0x555840757aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea44012770 tx=0x7fea44012a80 comp rx=0 tx=0).stop 2023-09-10T01:02:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 shutdown_connections 2023-09-10T01:02:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x555840762d00 0x555840761c40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ddb0 0x555840757560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074f550 0x55584075eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 --2- 172.21.15.3:0/4204019916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ebb0 0x555840757aa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea2f7fe700 0 client.25454 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 >> 172.21.15.3:0/4204019916 conn(0x5558406c2040 msgr2=0x555840747d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.096+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 shutdown_connections 2023-09-10T01:02:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.097+0000 7fea6a402b80 1 -- 172.21.15.3:0/4204019916 wait complete. 2023-09-10T01:02:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.097+0000 7fea6a402b80 20 client.25454 trim_cache size 0 max 16384 2023-09-10T01:02:46.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 Processor -- start 2023-09-10T01:02:46.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 -- start start 2023-09-10T01:02:46.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ddb0 0x55584074d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555840757130 con 0x55584074ddb0 2023-09-10T01:02:46.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555840668f70 con 0x55584074f550 2023-09-10T01:02:46.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.115+0000 7fea6a402b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555840756a10 con 0x55584074ebb0 2023-09-10T01:02:46.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ddb0 0x55584074d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ddb0 0x55584074d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41510/0 (socket says 172.21.15.3:41510) 2023-09-10T01:02:46.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 -- 172.21.15.3:0/2407533697 learned_addr learned my addr 172.21.15.3:0/2407533697 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea54f9c700 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 -- 172.21.15.3:0/2407533697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ebb0 msgr2=0x55584075ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 -- 172.21.15.3:0/2407533697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 msgr2=0x55584074e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 -- 172.21.15.3:0/2407533697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558404f2200 con 0x55584074ddb0 2023-09-10T01:02:46.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea54f9c700 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:46.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:46.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea4ffff700 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ddb0 0x55584074d6c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fea40099410 tx=0x7fea40004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db884a7533831824 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.116+0000 7fea36ffd700 1 -- 172.21.15.3:0/2407533697 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea400b1200 con 0x55584074ddb0 2023-09-10T01:02:46.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea36ffd700 1 -- 172.21.15.3:0/2407533697 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea40098dd0 con 0x55584074ddb0 2023-09-10T01:02:46.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea36ffd700 1 -- 172.21.15.3:0/2407533697 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40094bc0 con 0x55584074ddb0 2023-09-10T01:02:46.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 -- 172.21.15.3:0/2407533697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ddb0 msgr2=0x55584074d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ddb0 0x55584074d6c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fea40099410 tx=0x7fea40004860 comp rx=0 tx=0).stop 2023-09-10T01:02:46.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 -- 172.21.15.3:0/2407533697 shutdown_connections 2023-09-10T01:02:46.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074f550 0x55584074e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ddb0 0x55584074d6c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 --2- 172.21.15.3:0/2407533697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 -- 172.21.15.3:0/2407533697 >> 172.21.15.3:0/2407533697 conn(0x5558406c2040 msgr2=0x5558407486a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 -- 172.21.15.3:0/2407533697 shutdown_connections 2023-09-10T01:02:46.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.117+0000 7fea6a402b80 1 -- 172.21.15.3:0/2407533697 wait complete. 2023-09-10T01:02:46.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 -- start start 2023-09-10T01:02:46.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 Processor -- start 2023-09-10T01:02:46.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ddb0 0x555840757b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074f550 0x555840753e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555840754540 con 0x55584074ebb0 2023-09-10T01:02:46.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558407546b0 con 0x55584074ddb0 2023-09-10T01:02:46.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555840754870 con 0x55584074f550 2023-09-10T01:02:46.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.118+0000 7fea6a402b80 10 client.0 _ll_register_callbacks cb 0x7fea44009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:46.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea54f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074f550 0x555840753e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41516/0 (socket says 172.21.15.3:41516) 2023-09-10T01:02:46.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ddb0 0x555840757b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 -- 172.21.15.3:0/576932561 learned_addr learned my addr 172.21.15.3:0/576932561 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 -- 172.21.15.3:0/576932561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074f550 msgr2=0x555840753e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074f550 0x555840753e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 -- 172.21.15.3:0/576932561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ddb0 msgr2=0x555840757b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ddb0 0x555840757b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 -- 172.21.15.3:0/576932561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558404b4c10 con 0x55584074ebb0 2023-09-10T01:02:46.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea4f7fe700 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584075ea00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fea44010b80 tx=0x7fea440096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f259125aa2276862 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:02:46.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.119+0000 7fea2f7fe700 1 -- 172.21.15.3:0/576932561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea44017070 con 0x55584074ebb0 2023-09-10T01:02:46.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea4009cc00 con 0x55584074ebb0 2023-09-10T01:02:46.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea2f7fe700 1 -- 172.21.15.3:0/576932561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fea44010950 con 0x55584074ebb0 2023-09-10T01:02:46.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea2f7fe700 1 -- 172.21.15.3:0/576932561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea44005b00 con 0x55584074ebb0 2023-09-10T01:02:46.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 10 client.15420 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:46.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5558404cbd90 con 0x55584074ebb0 2023-09-10T01:02:46.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea2f7fe700 1 -- 172.21.15.3:0/576932561 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea4009cc00 con 0x55584074ebb0 2023-09-10T01:02:46.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea2f7fe700 1 -- 172.21.15.3:0/576932561 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fea4402a450 con 0x55584074ebb0 2023-09-10T01:02:46.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 10 client.15420 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:46.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 10 client.15420 resolve_mds: resolved name 'h' to daemon mds.h 2023-09-10T01:02:46.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 20 client.15420 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:46.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558407bba90 0x5558407b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 4 client.15420 mds_command: new command op to 24647 tid=0[{"prefix": "status"}] 2023-09-10T01:02:46.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.120+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5558406988d0 con 0x5558407bba90 2023-09-10T01:02:46.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea4ffff700 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558407bba90 0x5558407b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea4ffff700 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558407bba90 0x5558407b7f10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e6d71296c8b221ec server_cookie=e1c0507d2acaf4d5 in_seq=0 out_seq=0 2023-09-10T01:02:46.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea2f7fe700 10 client.15420 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea2f7fe700 1 -- 172.21.15.3:0/576932561 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5558406988d0 con 0x5558407bba90 2023-09-10T01:02:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea2f7fe700 10 client.15420 handle_command_reply: tid=0 2023-09-10T01:02:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea6a402b80 1 client.15420 shutdown 2023-09-10T01:02:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 msgr2=0x55584075ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.121+0000 7fea6a402b80 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584075ea00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fea44010b80 tx=0x7fea440096d0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 shutdown_connections 2023-09-10T01:02:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558407bba90 0x5558407b7f10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55584074f550 0x555840753e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55584074ddb0 0x555840757b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea2f7fe700 0 client.15420 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:02:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 --2- 172.21.15.3:0/576932561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55584074ebb0 0x55584075ea00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 >> 172.21.15.3:0/576932561 conn(0x5558406c2040 msgr2=0x55584074e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 shutdown_connections 2023-09-10T01:02:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 1 -- 172.21.15.3:0/576932561 wait complete. 2023-09-10T01:02:46.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea6a402b80 20 client.15420 trim_cache size 0 max 16384 2023-09-10T01:02:46.138 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:02:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T01:02:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 1, 2023-09-10T01:02:46.138 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24647, 2023-09-10T01:02:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2023-09-10T01:02:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2023-09-10T01:02:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2023-09-10T01:02:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 98.099971054999997, 2023-09-10T01:02:46.139 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 22, 2023-09-10T01:02:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 90, 2023-09-10T01:02:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 88, 2023-09-10T01:02:46.140 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 138.062914103 2023-09-10T01:02:46.140 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:02:46.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fea380a0640 msgr2=0x7fea380a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea569ff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fea380a0640 0x7fea380a2b00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea48005bd0 tx=0x7fea4800b040 comp rx=0 tx=0).stop 2023-09-10T01:02:46.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501006f0 msgr2=0x7fea50198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.122+0000 7fea569ff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501006f0 0x7fea50198740 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fea4000b500 tx=0x7fea4000b810 comp rx=0 tx=0).stop 2023-09-10T01:02:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 shutdown_connections 2023-09-10T01:02:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea50102280 0x7fea5019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea501006f0 0x7fea50198740 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fea380a0640 0x7fea380a2b00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 --2- 172.21.15.3:0/2774231599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea501018e0 0x7fea50198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 >> 172.21.15.3:0/2774231599 conn(0x7fea500fbc60 msgr2=0x7fea50107130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 shutdown_connections 2023-09-10T01:02:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.123+0000 7fea569ff700 1 -- 172.21.15.3:0/2774231599 wait complete. 2023-09-10T01:02:46.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:46 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1663559319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:46.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:46 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/298193382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:46.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-09-10T01:02:46.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/1378480782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d68102280 msgr2=0x7f5d68108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/1378480782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d68102280 0x7f5d68108b30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000b0d0 tx=0x7f5d6000b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/1378480782 shutdown_connections 2023-09-10T01:02:46.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/1378480782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d68102280 0x7f5d68108b30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/1378480782 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d681006f0 0x7f5d68100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/1378480782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681018e0 0x7f5d68101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/1378480782 >> 172.21.15.3:0/1378480782 conn(0x7f5d680fbc60 msgr2=0x7f5d680fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/1378480782 shutdown_connections 2023-09-10T01:02:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.348+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/1378480782 wait complete. 2023-09-10T01:02:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.349+0000 7f5d6f5ff700 1 Processor -- start 2023-09-10T01:02:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.349+0000 7f5d6f5ff700 1 -- start start 2023-09-10T01:02:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6f5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d681006f0 0x7f5d681986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6f5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681018e0 0x7f5d68198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6f5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d68102280 0x7f5d6819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6f5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d681a0220 con 0x7f5d681006f0 2023-09-10T01:02:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6f5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d681a0360 con 0x7f5d68102280 2023-09-10T01:02:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6f5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d681a04f0 con 0x7f5d681018e0 2023-09-10T01:02:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d681006f0 0x7f5d681986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d681006f0 0x7f5d681986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41546/0 (socket says 172.21.15.3:41546) 2023-09-10T01:02:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 -- 172.21.15.3:0/286762306 learned_addr learned my addr 172.21.15.3:0/286762306 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d68102280 0x7f5d6819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681018e0 0x7f5d68198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 -- 172.21.15.3:0/286762306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681018e0 msgr2=0x7f5d68198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681018e0 0x7f5d68198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 -- 172.21.15.3:0/286762306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d68102280 msgr2=0x7f5d6819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d68102280 0x7f5d6819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6d39b700 1 -- 172.21.15.3:0/286762306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d64009520 con 0x7f5d681006f0 2023-09-10T01:02:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.350+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d68102280 0x7f5d6819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.351+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d681006f0 0x7f5d681986a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400ea50 tx=0x7f5d6400ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.351+0000 7f5d5e7fc700 1 -- 172.21.15.3:0/286762306 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6400cc90 con 0x7f5d681006f0 2023-09-10T01:02:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.351+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d60009d00 con 0x7f5d681006f0 2023-09-10T01:02:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.351+0000 7f5d5e7fc700 1 -- 172.21.15.3:0/286762306 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d6400cdf0 con 0x7f5d681006f0 2023-09-10T01:02:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.351+0000 7f5d5e7fc700 1 -- 172.21.15.3:0/286762306 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d640187a0 con 0x7f5d681006f0 2023-09-10T01:02:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.351+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d681a0b50 con 0x7f5d681006f0 2023-09-10T01:02:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.354+0000 7f5d5e7fc700 1 -- 172.21.15.3:0/286762306 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f5d64018900 con 0x7f5d681006f0 2023-09-10T01:02:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.355+0000 7f5d82fdcb80 1 Processor -- start 2023-09-10T01:02:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.355+0000 7f5d82fdcb80 1 -- start start 2023-09-10T01:02:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d5e7fc700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5d540a0640 0x7f5d540a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026ddfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026dc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026dd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d5e7fc700 1 -- 172.21.15.3:0/286762306 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f5d64014070 con 0x7f5d681006f0 2023-09-10T01:02:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575026356e0 con 0x5575026ddbb0 2023-09-10T01:02:46.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557502633000 con 0x5575026de550 2023-09-10T01:02:46.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750262e8f0 con 0x5575026dcdb0 2023-09-10T01:02:46.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5d540a0640 0x7f5d540a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d6d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026ddfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d6d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026ddfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41556/0 (socket says 172.21.15.3:41556) 2023-09-10T01:02:46.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d6d39b700 1 -- 172.21.15.3:0/4130371596 learned_addr learned my addr 172.21.15.3:0/4130371596 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026dc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.356+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026dd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6d39b700 1 -- 172.21.15.3:0/4130371596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 msgr2=0x5575026dd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026dd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6d39b700 1 -- 172.21.15.3:0/4130371596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 msgr2=0x5575026dc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026dc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6d39b700 1 -- 172.21.15.3:0/4130371596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55750245ad90 con 0x5575026ddbb0 2023-09-10T01:02:46.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026dd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:46.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026ddfd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5d64099760 tx=0x7f5d640996a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5415284421b9cc80 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5d540a0640 0x7f5d540a2b00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009610 tx=0x7f5d60009780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d537fe700 1 -- 172.21.15.3:0/4130371596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6409d890 con 0x5575026ddbb0 2023-09-10T01:02:46.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d537fe700 1 -- 172.21.15.3:0/4130371596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d640a1ad0 con 0x5575026ddbb0 2023-09-10T01:02:46.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.357+0000 7f5d537fe700 1 -- 172.21.15.3:0/4130371596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64099f20 con 0x5575026ddbb0 2023-09-10T01:02:46.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/4130371596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 msgr2=0x5575026ddfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026ddfd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5d64099760 tx=0x7f5d640996a0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/4130371596 shutdown_connections 2023-09-10T01:02:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026dd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026ddfd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/4130371596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026dc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/4130371596 >> 172.21.15.3:0/4130371596 conn(0x557502651040 msgr2=0x5575026d76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/4130371596 shutdown_connections 2023-09-10T01:02:46.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.358+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/4130371596 wait complete. 2023-09-10T01:02:46.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 -- start start 2023-09-10T01:02:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 Processor -- start 2023-09-10T01:02:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026e6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026edb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575025ccb20 con 0x5575026ddbb0 2023-09-10T01:02:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575025cc820 con 0x5575026de550 2023-09-10T01:02:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575024b08d0 con 0x5575026dcdb0 2023-09-10T01:02:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.359+0000 7f5d82fdcb80 10 client.0 _ll_register_callbacks cb 0x557502516a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:46.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6cb9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026e6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6cb9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026e6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41564/0 (socket says 172.21.15.3:41564) 2023-09-10T01:02:46.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6cb9a700 1 -- 172.21.15.3:0/1193020249 learned_addr learned my addr 172.21.15.3:0/1193020249 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6d39b700 1 -- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 msgr2=0x5575026edb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026edb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6d39b700 1 -- 172.21.15.3:0/1193020249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 msgr2=0x5575026e6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026e6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6d39b700 1 -- 172.21.15.3:0/1193020249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d681a0b50 con 0x5575026dcdb0 2023-09-10T01:02:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026e6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e6590 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5d64098050 tx=0x7f5d64099b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e03433bb861668 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d50ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:02:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.360+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1193020249 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64099ec0 con 0x5575026dcdb0 2023-09-10T01:02:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.361+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5575025864a0 con 0x5575026dcdb0 2023-09-10T01:02:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.361+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1193020249 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f5d6409d890 con 0x5575026dcdb0 2023-09-10T01:02:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.361+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1193020249 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d640a1640 con 0x5575026dcdb0 2023-09-10T01:02:46.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.361+0000 7f5d82fdcb80 10 client.25426 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:46.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.361+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557502443c10 con 0x5575026dcdb0 2023-09-10T01:02:46.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.361+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1193020249 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d6407c080 con 0x5575026dcdb0 2023-09-10T01:02:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.361+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1193020249 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5d640989c0 con 0x5575026dcdb0 2023-09-10T01:02:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d82fdcb80 10 client.25426 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d82fdcb80 10 client.25426 resolve_mds: resolved name 'b' to daemon mds.b 2023-09-10T01:02:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d82fdcb80 20 client.25426 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5575026f1870 0x5575026f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d82fdcb80 4 client.25426 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:02:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557502643850 con 0x5575026f1870 2023-09-10T01:02:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5575026f1870 0x5575026f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5575026f1870 0x5575026f0a40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c9c0ae47eb06f6 server_cookie=f00b7ffc45d4fcda in_seq=0 out_seq=0 2023-09-10T01:02:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.362+0000 7f5d50ff9700 10 client.25426 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.363+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1193020249 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557502643850 con 0x5575026f1870 2023-09-10T01:02:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.363+0000 7f5d50ff9700 10 client.25426 handle_command_reply: tid=0 2023-09-10T01:02:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.363+0000 7f5d82fdcb80 1 client.25426 shutdown 2023-09-10T01:02:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.363+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 msgr2=0x5575026e6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.363+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e6590 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5d64098050 tx=0x7f5d64099b90 comp rx=0 tx=0).stop 2023-09-10T01:02:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 shutdown_connections 2023-09-10T01:02:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5575026f1870 0x5575026f0a40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e6590 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026edb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1193020249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026e6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d50ff9700 0 client.25426 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 >> 172.21.15.3:0/1193020249 conn(0x557502651040 msgr2=0x5575026d6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 shutdown_connections 2023-09-10T01:02:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1193020249 wait complete. 2023-09-10T01:02:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.364+0000 7f5d82fdcb80 20 client.25426 trim_cache size 0 max 16384 2023-09-10T01:02:46.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 Processor -- start 2023-09-10T01:02:46.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 -- start start 2023-09-10T01:02:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026edad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026e0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575026e6160 con 0x5575026ddbb0 2023-09-10T01:02:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575026e7490 con 0x5575026de550 2023-09-10T01:02:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.382+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557502643850 con 0x5575026dcdb0 2023-09-10T01:02:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026edad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026edad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41568/0 (socket says 172.21.15.3:41568) 2023-09-10T01:02:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 -- 172.21.15.3:0/2929227407 learned_addr learned my addr 172.21.15.3:0/2929227407 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026e0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 -- 172.21.15.3:0/2929227407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 msgr2=0x5575026e4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 -- 172.21.15.3:0/2929227407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 msgr2=0x5575026e0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026e0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 -- 172.21.15.3:0/2929227407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557502481200 con 0x5575026ddbb0 2023-09-10T01:02:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026e0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026edad0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x5575026f2c00 tx=0x7f5d58009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10316b3ff8612ea server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.383+0000 7f5d53fff700 1 -- 172.21.15.3:0/2929227407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58017070 con 0x5575026ddbb0 2023-09-10T01:02:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d53fff700 1 -- 172.21.15.3:0/2929227407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d58018040 con 0x5575026ddbb0 2023-09-10T01:02:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d53fff700 1 -- 172.21.15.3:0/2929227407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58004500 con 0x5575026ddbb0 2023-09-10T01:02:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/2929227407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 msgr2=0x5575026edad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026edad0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x5575026f2c00 tx=0x7f5d58009260 comp rx=0 tx=0).stop 2023-09-10T01:02:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/2929227407 shutdown_connections 2023-09-10T01:02:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026dcdb0 0x5575026e4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026de550 0x5575026e0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/2929227407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026ddbb0 0x5575026edad0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/2929227407 >> 172.21.15.3:0/2929227407 conn(0x557502651040 msgr2=0x5575026e5280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/2929227407 shutdown_connections 2023-09-10T01:02:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.384+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/2929227407 wait complete. 2023-09-10T01:02:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 -- start start 2023-09-10T01:02:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 Processor -- start 2023-09-10T01:02:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026dcdb0 0x5575026e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026ddbb0 0x5575026e8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026de550 0x5575026ed760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575026edca0 con 0x5575026de550 2023-09-10T01:02:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575026ede10 con 0x5575026dcdb0 2023-09-10T01:02:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575027458c0 con 0x5575026ddbb0 2023-09-10T01:02:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.385+0000 7f5d82fdcb80 10 client.0 _ll_register_callbacks cb 0x7f5d58009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:02:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026de550 0x5575026ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026de550 0x5575026ed760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41584/0 (socket says 172.21.15.3:41584) 2023-09-10T01:02:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 -- 172.21.15.3:0/1114462894 learned_addr learned my addr 172.21.15.3:0/1114462894 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026dcdb0 0x5575026e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6cb9a700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026ddbb0 0x5575026e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 -- 172.21.15.3:0/1114462894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026ddbb0 msgr2=0x5575026e8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026ddbb0 0x5575026e8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 -- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026dcdb0 msgr2=0x5575026e82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:02:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026dcdb0 0x5575026e82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 -- 172.21.15.3:0/1114462894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d58020a60 con 0x5575026de550 2023-09-10T01:02:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026dcdb0 0x5575026e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d6db9c700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026de550 0x5575026ed760 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009fd0 tx=0x7f5d6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61a5076facadef8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d50ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:02:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.386+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1114462894 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60020860 con 0x5575026de550 2023-09-10T01:02:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d640a1ad0 con 0x5575026de550 2023-09-10T01:02:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1114462894 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f5d6000be30 con 0x5575026de550 2023-09-10T01:02:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1114462894 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d600209c0 con 0x5575026de550 2023-09-10T01:02:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 10 client.15438 fetch_fsmap learned FSMap version 22 2023-09-10T01:02:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557502459260 con 0x5575026de550 2023-09-10T01:02:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1114462894 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d640a1ad0 con 0x5575026de550 2023-09-10T01:02:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1114462894 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5d6002a070 con 0x5575026de550 2023-09-10T01:02:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 10 client.15438 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:02:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 10 client.15438 resolve_mds: resolved name 'b' to daemon mds.b 2023-09-10T01:02:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 20 client.15438 populate_metadata read hostname 'smithi003' 2023-09-10T01:02:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750274abd0 0x557502747010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 4 client.15438 mds_command: new command op to 24698 tid=0[{"prefix": "status"}] 2023-09-10T01:02:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.387+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5575026278d0 con 0x55750274abd0 2023-09-10T01:02:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.388+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750274abd0 0x557502747010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.388+0000 7f5d6d39b700 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750274abd0 0x557502747010 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac919b5364955ed7 server_cookie=6a8fee48f48bf1a7 in_seq=0 out_seq=0 2023-09-10T01:02:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.388+0000 7f5d50ff9700 10 client.15438 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.388+0000 7f5d50ff9700 1 -- 172.21.15.3:0/1114462894 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5575026278d0 con 0x55750274abd0 2023-09-10T01:02:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.388+0000 7f5d50ff9700 10 client.15438 handle_command_reply: tid=0 2023-09-10T01:02:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.388+0000 7f5d82fdcb80 1 client.15438 shutdown 2023-09-10T01:02:46.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026de550 msgr2=0x5575026ed760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026de550 0x5575026ed760 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009fd0 tx=0x7f5d6000b040 comp rx=0 tx=0).stop 2023-09-10T01:02:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 shutdown_connections 2023-09-10T01:02:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750274abd0 0x557502747010 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575026de550 0x5575026ed760 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5575026dcdb0 0x5575026e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d50ff9700 0 client.15438 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:02:46.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 --2- 172.21.15.3:0/1114462894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5575026ddbb0 0x5575026e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 >> 172.21.15.3:0/1114462894 conn(0x557502651040 msgr2=0x5575026e4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 shutdown_connections 2023-09-10T01:02:46.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 1 -- 172.21.15.3:0/1114462894 wait complete. 2023-09-10T01:02:46.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d82fdcb80 20 client.15438 trim_cache size 0 max 16384 2023-09-10T01:02:46.412 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:02:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "4260c16a-4f74-11ee-9ab7-7b867c8bd7da", 2023-09-10T01:02:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2023-09-10T01:02:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24698, 2023-09-10T01:02:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2023-09-10T01:02:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2023-09-10T01:02:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2023-09-10T01:02:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 122.345819703, 2023-09-10T01:02:46.414 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 22, 2023-09-10T01:02:46.414 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 90, 2023-09-10T01:02:46.414 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 88, 2023-09-10T01:02:46.414 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 127.582399071 2023-09-10T01:02:46.415 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:02:46.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5d540a0640 msgr2=0x7f5d540a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5d540a0640 0x7f5d540a2b00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009610 tx=0x7f5d60009780 comp rx=0 tx=0).stop 2023-09-10T01:02:46.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d681006f0 msgr2=0x7f5d681986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:46.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.389+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d681006f0 0x7f5d681986a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400ea50 tx=0x7f5d6400ee10 comp rx=0 tx=0).stop 2023-09-10T01:02:46.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 shutdown_connections 2023-09-10T01:02:46.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d68102280 0x7f5d6819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d681006f0 0x7f5d681986a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f5d540a0640 0x7f5d540a2b00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 --2- 172.21.15.3:0/286762306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681018e0 0x7f5d68198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:46.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 >> 172.21.15.3:0/286762306 conn(0x7f5d680fbc60 msgr2=0x7f5d68107130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:46.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 shutdown_connections 2023-09-10T01:02:46.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:46.390+0000 7f5d6f5ff700 1 -- 172.21.15.3:0/286762306 wait complete. 2023-09-10T01:02:46.419 INFO:teuthology.run_tasks:Running task kclient... 2023-09-10T01:02:46.435 INFO:tasks.kclient:Mounting kernel clients... 2023-09-10T01:02:46.436 INFO:tasks.kclient:config is {'client.0': {}} 2023-09-10T01:02:46.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:46.436 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2023-09-10T01:02:46.437 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:46 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1663559319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:46.437 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:46 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/298193382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:46 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1663559319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:46 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/298193382' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:46.481 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-09-10T01:02:46.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:46.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-09-10T01:02:46.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:46.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2023-09-10T01:02:46.713 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2023-09-10T01:02:46.713 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-09-10T01:02:46.714 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-09-10T01:02:46.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph fs ls 2023-09-10T01:02:47.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:47 smithi003 ceph-mon[97654]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:47.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:47 smithi136 ceph-mon[133708]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:47.449 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:02:47.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:47 smithi092 ceph-mon[134067]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:49.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.030+0000 7fe2f53fc700 1 -- 172.21.15.3:0/1878541746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f0073350 msgr2=0x7fe2f010c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:49.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.030+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/1878541746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f0073350 0x7fe2f010c920 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40098f0 tx=0x7fe2e4009c00 comp rx=0 tx=0).stop 2023-09-10T01:02:49.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.030+0000 7fe2f53fc700 1 -- 172.21.15.3:0/1878541746 shutdown_connections 2023-09-10T01:02:49.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.030+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/1878541746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f0072a30 0x7fe2f0072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.030+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/1878541746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f010ce60 0x7fe2f010f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.030+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/1878541746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f0073350 0x7fe2f010c920 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.030+0000 7fe2f53fc700 1 -- 172.21.15.3:0/1878541746 >> 172.21.15.3:0/1878541746 conn(0x7fe2f00fc0f0 msgr2=0x7fe2f00fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:49.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.031+0000 7fe2f53fc700 1 -- 172.21.15.3:0/1878541746 shutdown_connections 2023-09-10T01:02:49.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.031+0000 7fe2f53fc700 1 -- 172.21.15.3:0/1878541746 wait complete. 2023-09-10T01:02:49.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 Processor -- start 2023-09-10T01:02:49.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 -- start start 2023-09-10T01:02:49.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f0072a30 0x7fe2f0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:49.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f0073350 0x7fe2f0198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:49.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f010ce60 0x7fe2f019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:49.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f019d900 con 0x7fe2f010ce60 2023-09-10T01:02:49.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f019da70 con 0x7fe2f0073350 2023-09-10T01:02:49.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.032+0000 7fe2f53fc700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f019dc30 con 0x7fe2f0072a30 2023-09-10T01:02:49.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f0073350 0x7fe2f0198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:49.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f0073350 0x7fe2f0198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53356/0 (socket says 172.21.15.3:53356) 2023-09-10T01:02:49.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 -- 172.21.15.3:0/4195996257 learned_addr learned my addr 172.21.15.3:0/4195996257 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:49.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ef7fe700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f010ce60 0x7fe2f019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:49.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2eeffd700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f0072a30 0x7fe2f0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:49.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 -- 172.21.15.3:0/4195996257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f0072a30 msgr2=0x7fe2f0198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:49.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f0072a30 0x7fe2f0198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 -- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f010ce60 msgr2=0x7fe2f019d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:49.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f010ce60 0x7fe2f019d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 -- 172.21.15.3:0/4195996257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8009d00 con 0x7fe2f0073350 2023-09-10T01:02:49.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.033+0000 7fe2ee7fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f0073350 0x7fe2f0198e90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4009230 tx=0x7fe2e400fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:49.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.034+0000 7fe2d7fff700 1 -- 172.21.15.3:0/4195996257 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e401c070 con 0x7fe2f0073350 2023-09-10T01:02:49.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.034+0000 7fe2d7fff700 1 -- 172.21.15.3:0/4195996257 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2e4005040 con 0x7fe2f0073350 2023-09-10T01:02:49.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.034+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e4009520 con 0x7fe2f0073350 2023-09-10T01:02:49.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.034+0000 7fe2d7fff700 1 -- 172.21.15.3:0/4195996257 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4003920 con 0x7fe2f0073350 2023-09-10T01:02:49.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.034+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f01a55e0 con 0x7fe2f0073350 2023-09-10T01:02:49.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.035+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f004e300 con 0x7fe2f0073350 2023-09-10T01:02:49.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.037+0000 7fe2d7fff700 1 -- 172.21.15.3:0/4195996257 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fe2e4020030 con 0x7fe2f0073350 2023-09-10T01:02:49.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.038+0000 7fe2d7fff700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe2d80a06c0 0x7fe2d80a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:49.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.038+0000 7fe2d7fff700 1 -- 172.21.15.3:0/4195996257 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fe2e40ccf20 con 0x7fe2f0073350 2023-09-10T01:02:49.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.038+0000 7fe2eeffd700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe2d80a06c0 0x7fe2d80a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:49.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.039+0000 7fe2eeffd700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe2d80a06c0 0x7fe2d80a2b80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0007be0 tx=0x7fe2e00073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:49.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.042+0000 7fe2d7fff700 1 -- 172.21.15.3:0/4195996257 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fe2e40966d0 con 0x7fe2f0073350 2023-09-10T01:02:49.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:49 smithi003 ceph-mon[97654]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:49.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.300+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe2f0199cb0 con 0x7fe2f0073350 2023-09-10T01:02:49.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.300+0000 7fe2d7fff700 1 -- 172.21.15.3:0/4195996257 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe2e4099cf0 con 0x7fe2f0073350 2023-09-10T01:02:49.302 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-09-10T01:02:49.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe2d80a06c0 msgr2=0x7fe2d80a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:49.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe2d80a06c0 0x7fe2d80a2b80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0007be0 tx=0x7fe2e00073d0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f0073350 msgr2=0x7fe2f0198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:49.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f0073350 0x7fe2f0198e90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4009230 tx=0x7fe2e400fa70 comp rx=0 tx=0).stop 2023-09-10T01:02:49.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 shutdown_connections 2023-09-10T01:02:49.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe2d80a06c0 0x7fe2d80a2b80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f0072a30 0x7fe2f0198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2f010ce60 0x7fe2f019d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 --2- 172.21.15.3:0/4195996257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2f0073350 0x7fe2f0198e90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 >> 172.21.15.3:0/4195996257 conn(0x7fe2f00fc0f0 msgr2=0x7fe2f010f050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 shutdown_connections 2023-09-10T01:02:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:49.303+0000 7fe2f53fc700 1 -- 172.21.15.3:0/4195996257 wait complete. 2023-09-10T01:02:49.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:49 smithi136 ceph-mon[133708]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:49.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:49 smithi092 ceph-mon[134067]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-09-10T01:02:49.845 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-09-10T01:02:49.846 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-09-10T01:02:49.846 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2023-09-10T01:02:49.846 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-09-10T01:02:49.847 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-09-10T01:02:49.847 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-09-10T01:02:49.847 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-09-10T01:02:49.847 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-09-10T01:02:49.848 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-09-10T01:02:49.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:49.848 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2023-09-10T01:02:49.893 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-09-10T01:02:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-09-10T01:02:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2023-09-10T01:02:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2023-09-10T01:02:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2023-09-10T01:02:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2023-09-10T01:02:49.894 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-09-10T01:02:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2023-09-10T01:02:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2023-09-10T01:02:49.895 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-09-10T01:02:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2023-09-10T01:02:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2023-09-10T01:02:49.896 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-09-10T01:02:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2023-09-10T01:02:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2023-09-10T01:02:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-09-10T01:02:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 41615sec preferred_lft 41615sec 2023-09-10T01:02:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2023-09-10T01:02:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2023-09-10T01:02:49.897 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-09-10T01:02:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2023-09-10T01:02:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2023-09-10T01:02:49.899 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-09-10T01:02:49.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:49.900 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:02:49.900 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:02:49.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add name ceph-brx type bridge 2023-09-10T01:02:49.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr flush dev ceph-brx 2023-09-10T01:02:49.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set ceph-brx up 2023-09-10T01:02:49.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-09-10T01:02:49.901 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:02:50.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:50.155 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-09-10T01:02:50.227 INFO:teuthology.orchestra.run.smithi003.stdout:1 2023-09-10T01:02:50.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:50.231 DEBUG:teuthology.orchestra.run.smithi003:> route 2023-09-10T01:02:50.277 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2023-09-10T01:02:50.278 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-09-10T01:02:50.278 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-09-10T01:02:50.278 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-09-10T01:02:50.278 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-09-10T01:02:50.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:50.279 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:02:50.279 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:02:50.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-09-10T01:02:50.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-09-10T01:02:50.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-09-10T01:02:50.281 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:02:50.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4195996257' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-09-10T01:02:50.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4195996257' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-09-10T01:02:50.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:50.458 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2023-09-10T01:02:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:50 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4195996257' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-09-10T01:02:50.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:50.543 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list-id 2023-09-10T01:02:50.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:50.627 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:02:50.627 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:02:50.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-09-10T01:02:50.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-09-10T01:02:50.628 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:02:50.811 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-09-10T01:02:50.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:50.812 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:02:50.813 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:02:50.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-09-10T01:02:50.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-09-10T01:02:50.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-09-10T01:02:50.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-09-10T01:02:50.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-09-10T01:02:50.815 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:02:51.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:51.072 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2023-09-10T01:02:51.072 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2023-09-10T01:02:51.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.0 up 2023-09-10T01:02:51.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set dev brx.0 master ceph-brx 2023-09-10T01:02:51.074 DEBUG:teuthology.orchestra.run.smithi003:> ') 2023-09-10T01:02:51.251 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-09-10T01:02:51.251 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-09-10T01:02:51.297 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-09-10T01:02:51.297 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-09-10T01:02:51.298 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-09-10T01:02:51.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:51 smithi003 ceph-mon[97654]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:51.350 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-09-10T01:02:51.351 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-09-10T01:02:51.351 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/_nogroup/sv_1/18c4d3b9-37ec-4fad-8baf-bf37c9efeddc 2023-09-10T01:02:51.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:51.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/_nogroup/sv_1/18c4d3b9-37ec-4fad-8baf-bf37c9efeddc /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2023-09-10T01:02:51.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:51 smithi136 ceph-mon[133708]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:51.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:51 smithi092 ceph-mon[134067]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:51.497 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2023-09-10T01:02:51.498 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2023-09-10T01:02:51.986 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2023-09-10T01:02:51.986 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2023-09-10T01:02:51.987 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:6789,172.21.15.92:6789,172.21.15.136:6789" 2023-09-10T01:02:51.987 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: trying mount with new device syntax: 0@4260c16a-4f74-11ee-9ab7-7b867c8bd7da.cephfs=/volumes/_nogroup/sv_1/18c4d3b9-37ec-4fad-8baf-bf37c9efeddc 2023-09-10T01:02:51.987 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.3:6789/172.21.15.92:6789/172.21.15.136:6789" will pass to kernel 2023-09-10T01:02:51.993 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-09-10T01:02:51.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:51.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-09-10T01:02:52.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:52.075 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-09-10T01:02:52.162 INFO:teuthology.orchestra.run.smithi003.stdout:client15447 2023-09-10T01:02:52.163 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T01:02:52.163 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2023-09-10T01:02:52.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:52.209 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-09-10T01:02:52.335 INFO:teuthology.orchestra.run.smithi003.stdout:client15447 2023-09-10T01:02:52.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:52.336 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-09-10T01:02:52.423 INFO:teuthology.orchestra.run.smithi003.stdout:client15447 2023-09-10T01:02:52.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:02:52.424 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da.client15447/status') 2023-09-10T01:02:52.533 INFO:teuthology.orchestra.run.smithi003.stdout:instance: client.15447 (3)192.168.0.1:0/2874327242 2023-09-10T01:02:52.533 INFO:teuthology.orchestra.run.smithi003.stdout:blocklisted: false 2023-09-10T01:02:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records in 2023-09-10T01:02:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records out 2023-09-10T01:02:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:70 bytes copied, 5.557e-05 s, 1.3 MB/s 2023-09-10T01:02:52.537 INFO:teuthology.run_tasks:Running task check-counter... 2023-09-10T01:02:52.548 INFO:teuthology.run_tasks:Running task fwd_scrub... 2023-09-10T01:02:52.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:52.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.783+0000 7fade57ff700 1 -- 172.21.15.3:0/3701851324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade01018c0 msgr2=0x7fade0101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:52.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.783+0000 7fade57ff700 1 --2- 172.21.15.3:0/3701851324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade01018c0 0x7fade0101d20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fadd40099c0 tx=0x7fadd4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:52.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.783+0000 7fade57ff700 1 -- 172.21.15.3:0/3701851324 shutdown_connections 2023-09-10T01:02:52.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.783+0000 7fade57ff700 1 --2- 172.21.15.3:0/3701851324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fade0102260 0x7fade0108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:52.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.783+0000 7fade57ff700 1 --2- 172.21.15.3:0/3701851324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01006d0 0x7fade0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:52.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.783+0000 7fade57ff700 1 --2- 172.21.15.3:0/3701851324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade01018c0 0x7fade0101d20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:52.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.783+0000 7fade57ff700 1 -- 172.21.15.3:0/3701851324 >> 172.21.15.3:0/3701851324 conn(0x7fade00fbc60 msgr2=0x7fade00fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:52.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.784+0000 7fade57ff700 1 -- 172.21.15.3:0/3701851324 shutdown_connections 2023-09-10T01:02:52.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.784+0000 7fade57ff700 1 -- 172.21.15.3:0/3701851324 wait complete. 2023-09-10T01:02:52.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.784+0000 7fade57ff700 1 Processor -- start 2023-09-10T01:02:52.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fade57ff700 1 -- start start 2023-09-10T01:02:52.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fade57ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fade01006d0 0x7fade00743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:52.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fade57ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01018c0 0x7fade00729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:52.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fade57ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade0102260 0x7fade0072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:52.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fade57ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00736d0 con 0x7fade01018c0 2023-09-10T01:02:52.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fade57ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade0073840 con 0x7fade0102260 2023-09-10T01:02:52.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fade57ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01a0430 con 0x7fade01006d0 2023-09-10T01:02:52.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fadde7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01018c0 0x7fade00729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:52.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fadde7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01018c0 0x7fade00729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41626/0 (socket says 172.21.15.3:41626) 2023-09-10T01:02:52.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7fadde7fc700 1 -- 172.21.15.3:0/1869145268 learned_addr learned my addr 172.21.15.3:0/1869145268 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:52.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7faddeffd700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fade01006d0 0x7fade00743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:52.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.785+0000 7faddf7fe700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade0102260 0x7fade0072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:52.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadde7fc700 1 -- 172.21.15.3:0/1869145268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fade01006d0 msgr2=0x7fade00743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:52.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadde7fc700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fade01006d0 0x7fade00743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:52.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadde7fc700 1 -- 172.21.15.3:0/1869145268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade0102260 msgr2=0x7fade0072f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:52.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadde7fc700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade0102260 0x7fade0072f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:52.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadde7fc700 1 -- 172.21.15.3:0/1869145268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd0009520 con 0x7fade01018c0 2023-09-10T01:02:52.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7faddeffd700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fade01006d0 0x7fade00743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:52.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadde7fc700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01018c0 0x7fade00729f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009990 tx=0x7fadd400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:52.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadc7fff700 1 -- 172.21.15.3:0/1869145268 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd401d070 con 0x7fade01018c0 2023-09-10T01:02:52.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40095f0 con 0x7fade01018c0 2023-09-10T01:02:52.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadc7fff700 1 -- 172.21.15.3:0/1869145268 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadd4005030 con 0x7fade01018c0 2023-09-10T01:02:52.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fadc7fff700 1 -- 172.21.15.3:0/1869145268 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4004a70 con 0x7fade01018c0 2023-09-10T01:02:52.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.786+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade01a09a0 con 0x7fade01018c0 2023-09-10T01:02:52.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.787+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade010bf40 con 0x7fade01018c0 2023-09-10T01:02:52.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.789+0000 7fadc7fff700 1 -- 172.21.15.3:0/1869145268 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fadd4005410 con 0x7fade01018c0 2023-09-10T01:02:52.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.791+0000 7fadc7fff700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fadc80a0640 0x7fadc80a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:52.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.791+0000 7fadc7fff700 1 -- 172.21.15.3:0/1869145268 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fadd40cd120 con 0x7fade01018c0 2023-09-10T01:02:52.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.791+0000 7faddeffd700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fadc80a0640 0x7fadc80a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:52.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.792+0000 7faddeffd700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fadc80a0640 0x7fadc80a2b00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009fd0 tx=0x7fadd0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:52.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:52.793+0000 7fadc7fff700 1 -- 172.21.15.3:0/1869145268 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fadd40968d0 con 0x7fade01018c0 2023-09-10T01:02:53.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.009+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fade0107600 con 0x7fade01018c0 2023-09-10T01:02:53.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.010+0000 7fadc7fff700 1 -- 172.21.15.3:0/1869145268 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7fadd4099ef0 con 0x7fade01018c0 2023-09-10T01:02:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:53.019 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:53.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fadc80a0640 msgr2=0x7fadc80a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fadc80a0640 0x7fadc80a2b00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009fd0 tx=0x7fadd0009260 comp rx=0 tx=0).stop 2023-09-10T01:02:53.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01018c0 msgr2=0x7fade00729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01018c0 0x7fade00729f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009990 tx=0x7fadd400f690 comp rx=0 tx=0).stop 2023-09-10T01:02:53.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 shutdown_connections 2023-09-10T01:02:53.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fadc80a0640 0x7fadc80a2b00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fade01006d0 0x7fade00743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade0102260 0x7fade0072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 --2- 172.21.15.3:0/1869145268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fade01018c0 0x7fade00729f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 >> 172.21.15.3:0/1869145268 conn(0x7fade00fbc60 msgr2=0x7fade00fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:53.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 shutdown_connections 2023-09-10T01:02:53.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.013+0000 7fade57ff700 1 -- 172.21.15.3:0/1869145268 wait complete. 2023-09-10T01:02:53.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:02:53.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:02:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 -- 172.21.15.3:0/1497211767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 msgr2=0x7f085c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 --2- 172.21.15.3:0/1497211767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c100b00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f084c0099c0 tx=0x7f084c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 -- 172.21.15.3:0/1497211767 shutdown_connections 2023-09-10T01:02:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 --2- 172.21.15.3:0/1497211767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f085c102270 0x7f085c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 --2- 172.21.15.3:0/1497211767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c100b00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 --2- 172.21.15.3:0/1497211767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f085c1018d0 0x7f085c101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 -- 172.21.15.3:0/1497211767 >> 172.21.15.3:0/1497211767 conn(0x7f085c0fbc50 msgr2=0x7f085c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 -- 172.21.15.3:0/1497211767 shutdown_connections 2023-09-10T01:02:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.246+0000 7f08611ff700 1 -- 172.21.15.3:0/1497211767 wait complete. 2023-09-10T01:02:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.247+0000 7f08611ff700 1 Processor -- start 2023-09-10T01:02:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.247+0000 7f08611ff700 1 -- start start 2023-09-10T01:02:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.247+0000 7f08611ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.247+0000 7f08611ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f085c1018d0 0x7f085c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f08611ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f085c102270 0x7f085c19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f08611ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c1a0160 con 0x7f085c1006e0 2023-09-10T01:02:53.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f08611ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c1a02a0 con 0x7f085c102270 2023-09-10T01:02:53.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f08611ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c1a0430 con 0x7f085c1018d0 2023-09-10T01:02:53.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41646/0 (socket says 172.21.15.3:41646) 2023-09-10T01:02:53.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 -- 172.21.15.3:0/138723330 learned_addr learned my addr 172.21.15.3:0/138723330 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:53.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085b59e700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f085c102270 0x7f085c19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f0853fff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f085c1018d0 0x7f085c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 -- 172.21.15.3:0/138723330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f085c1018d0 msgr2=0x7f085c198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f085c1018d0 0x7f085c198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 -- 172.21.15.3:0/138723330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f085c102270 msgr2=0x7f085c19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f085c102270 0x7f085c19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 -- 172.21.15.3:0/138723330 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0095f0 con 0x7f085c1006e0 2023-09-10T01:02:53.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.248+0000 7f085ad9d700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c1985e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f084c009fd0 tx=0x7f084c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:53.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.249+0000 7f0858d99700 1 -- 172.21.15.3:0/138723330 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c01d070 con 0x7f085c1006e0 2023-09-10T01:02:53.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.249+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085c1a06b0 con 0x7f085c1006e0 2023-09-10T01:02:53.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.249+0000 7f0858d99700 1 -- 172.21.15.3:0/138723330 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f084c005510 con 0x7f085c1006e0 2023-09-10T01:02:53.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.249+0000 7f0858d99700 1 -- 172.21.15.3:0/138723330 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c004b00 con 0x7f085c1006e0 2023-09-10T01:02:53.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.249+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085c1a0b30 con 0x7f085c1006e0 2023-09-10T01:02:53.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.250+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085c10bf40 con 0x7f085c1006e0 2023-09-10T01:02:53.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.252+0000 7f0858d99700 1 -- 172.21.15.3:0/138723330 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f084c004ca0 con 0x7f085c1006e0 2023-09-10T01:02:53.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.253+0000 7f0858d99700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08480a0640 0x7f08480a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.253+0000 7f0858d99700 1 -- 172.21.15.3:0/138723330 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f084c0cce20 con 0x7f085c1006e0 2023-09-10T01:02:53.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.253+0000 7f0853fff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08480a0640 0x7f08480a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.254+0000 7f0853fff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08480a0640 0x7f08480a2b00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0844009bf0 tx=0x7f0844009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:53.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.257+0000 7f0858d99700 1 -- 172.21.15.3:0/138723330 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f084c0965d0 con 0x7f085c1006e0 2023-09-10T01:02:53.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:53 smithi003 ceph-mon[97654]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:53.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:53 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1869145268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:53.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:53 smithi136 ceph-mon[133708]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:53.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:53 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1869145268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:53.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:53 smithi092 ceph-mon[134067]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-09-10T01:02:53.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:53 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1869145268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:53.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.469+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f085c1a0fc0 con 0x7f085c1006e0 2023-09-10T01:02:53.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.469+0000 7f0858d99700 1 -- 172.21.15.3:0/138723330 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f084c099bf0 con 0x7f085c1006e0 2023-09-10T01:02:53.471 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:02:53.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.471+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08480a0640 msgr2=0x7f08480a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.471+0000 7f08611ff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08480a0640 0x7f08480a2b00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0844009bf0 tx=0x7f0844009260 comp rx=0 tx=0).stop 2023-09-10T01:02:53.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.471+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 msgr2=0x7f085c1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.471+0000 7f08611ff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c1985e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f084c009fd0 tx=0x7f084c00f690 comp rx=0 tx=0).stop 2023-09-10T01:02:53.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 shutdown_connections 2023-09-10T01:02:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f085c102270 0x7f085c19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c1006e0 0x7f085c1985e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f08480a0640 0x7f08480a2b00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 --2- 172.21.15.3:0/138723330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f085c1018d0 0x7f085c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 >> 172.21.15.3:0/138723330 conn(0x7f085c0fbc50 msgr2=0x7f085c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 shutdown_connections 2023-09-10T01:02:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.472+0000 7f08611ff700 1 -- 172.21.15.3:0/138723330 wait complete. 2023-09-10T01:02:53.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:02:53.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 -- 172.21.15.3:0/2598075119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 msgr2=0x7f1008101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 --2- 172.21.15.3:0/2598075119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 0x7f1008101d30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40099c0 tx=0x7f0ff4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 -- 172.21.15.3:0/2598075119 shutdown_connections 2023-09-10T01:02:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 --2- 172.21.15.3:0/2598075119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10081006e0 0x7f1008100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 --2- 172.21.15.3:0/2598075119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1008102270 0x7f1008108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 --2- 172.21.15.3:0/2598075119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 0x7f1008101d30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 -- 172.21.15.3:0/2598075119 >> 172.21.15.3:0/2598075119 conn(0x7f10080fbc50 msgr2=0x7f10080fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:53.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 -- 172.21.15.3:0/2598075119 shutdown_connections 2023-09-10T01:02:53.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.707+0000 7f100efff700 1 -- 172.21.15.3:0/2598075119 wait complete. 2023-09-10T01:02:53.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.708+0000 7f100efff700 1 Processor -- start 2023-09-10T01:02:53.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.708+0000 7f100efff700 1 -- start start 2023-09-10T01:02:53.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.708+0000 7f100efff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10081006e0 0x7f10081963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.708+0000 7f100efff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 0x7f1008196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.708+0000 7f100efff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1008102270 0x7f100819d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100efff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100819df40 con 0x7f10081006e0 2023-09-10T01:02:53.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100efff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100819e080 con 0x7f10081018d0 2023-09-10T01:02:53.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100efff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100819e210 con 0x7f1008102270 2023-09-10T01:02:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10081006e0 0x7f10081963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10081006e0 0x7f10081963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41670/0 (socket says 172.21.15.3:41670) 2023-09-10T01:02:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 -- 172.21.15.3:0/2608434586 learned_addr learned my addr 172.21.15.3:0/2608434586 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100d59c700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1008102270 0x7f100819d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f1007fff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 0x7f1008196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 -- 172.21.15.3:0/2608434586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1008102270 msgr2=0x7f100819d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1008102270 0x7f100819d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 -- 172.21.15.3:0/2608434586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 msgr2=0x7f1008196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 0x7f1008196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 -- 172.21.15.3:0/2608434586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1000009c40 con 0x7f10081006e0 2023-09-10T01:02:53.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.709+0000 7f100cd9b700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10081006e0 0x7f10081963c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00e970 tx=0x7f0ffc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.710+0000 7f1005ffb700 1 -- 172.21.15.3:0/2608434586 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc009790 con 0x7f10081006e0 2023-09-10T01:02:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.710+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff40095f0 con 0x7f10081006e0 2023-09-10T01:02:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.710+0000 7f1005ffb700 1 -- 172.21.15.3:0/2608434586 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ffc00c7c0 con 0x7f10081006e0 2023-09-10T01:02:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.710+0000 7f1005ffb700 1 -- 172.21.15.3:0/2608434586 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc003c20 con 0x7f10081006e0 2023-09-10T01:02:53.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.710+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100819e8c0 con 0x7f10081006e0 2023-09-10T01:02:53.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.711+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1008107610 con 0x7f10081006e0 2023-09-10T01:02:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.713+0000 7f1005ffb700 1 -- 172.21.15.3:0/2608434586 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f0ffc00c930 con 0x7f10081006e0 2023-09-10T01:02:53.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.714+0000 7f1005ffb700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0ff80a06c0 0x7f0ff80a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.714+0000 7f1005ffb700 1 -- 172.21.15.3:0/2608434586 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f0ffc014070 con 0x7f10081006e0 2023-09-10T01:02:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.715+0000 7f1007fff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0ff80a06c0 0x7f0ff80a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:53.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.715+0000 7f1007fff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0ff80a06c0 0x7f0ff80a2b80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400b580 tx=0x7f0ff4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:53.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.716+0000 7f1005ffb700 1 -- 172.21.15.3:0/2608434586 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0ffc0959a0 con 0x7f10081006e0 2023-09-10T01:02:53.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.927+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f100810c0e0 con 0x7f10081006e0 2023-09-10T01:02:53.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.927+0000 7f1005ffb700 1 -- 172.21.15.3:0/2608434586 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0ffc098fc0 con 0x7f10081006e0 2023-09-10T01:02:53.931 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:53.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0ff80a06c0 msgr2=0x7f0ff80a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0ff80a06c0 0x7f0ff80a2b80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400b580 tx=0x7f0ff4005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10081006e0 msgr2=0x7f10081963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:53.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10081006e0 0x7f10081963c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00e970 tx=0x7f0ffc00ec80 comp rx=0 tx=0).stop 2023-09-10T01:02:53.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 shutdown_connections 2023-09-10T01:02:53.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f0ff80a06c0 0x7f0ff80a2b80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1008102270 0x7f100819d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10081006e0 0x7f10081963c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 --2- 172.21.15.3:0/2608434586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10081018d0 0x7f1008196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:53.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 >> 172.21.15.3:0/2608434586 conn(0x7f10080fbc50 msgr2=0x7f10080fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:53.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 shutdown_connections 2023-09-10T01:02:53.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:53.930+0000 7f100efff700 1 -- 172.21.15.3:0/2608434586 wait complete. 2023-09-10T01:02:53.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:02:54.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.161+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1940333480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 msgr2=0x7fbfc0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.161+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1940333480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc0100b00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0099c0 tx=0x7fbfbc009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.161+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1940333480 shutdown_connections 2023-09-10T01:02:54.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.162+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1940333480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfc0102270 0x7fbfc0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.162+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1940333480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc0100b00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.162+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1940333480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc01018d0 0x7fbfc0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.162+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1940333480 >> 172.21.15.3:0/1940333480 conn(0x7fbfc00fbc50 msgr2=0x7fbfc00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:54.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.162+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1940333480 shutdown_connections 2023-09-10T01:02:54.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.162+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1940333480 wait complete. 2023-09-10T01:02:54.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 Processor -- start 2023-09-10T01:02:54.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 -- start start 2023-09-10T01:02:54.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc01018d0 0x7fbfc0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfc0102270 0x7fbfc019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc01a0160 con 0x7fbfc01006e0 2023-09-10T01:02:54.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc01a02a0 con 0x7fbfc01018d0 2023-09-10T01:02:54.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc7fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc01a0430 con 0x7fbfc0102270 2023-09-10T01:02:54.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc01985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41690/0 (socket says 172.21.15.3:41690) 2023-09-10T01:02:54.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.163+0000 7fbfc5d9b700 1 -- 172.21.15.3:0/1595012449 learned_addr learned my addr 172.21.15.3:0/1595012449 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:54.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc559a700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc01018d0 0x7fbfc0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc659c700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfc0102270 0x7fbfc019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc5d9b700 1 -- 172.21.15.3:0/1595012449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfc0102270 msgr2=0x7fbfc019fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc5d9b700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfc0102270 0x7fbfc019fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc5d9b700 1 -- 172.21.15.3:0/1595012449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc01018d0 msgr2=0x7fbfc0198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc5d9b700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc01018d0 0x7fbfc0198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc5d9b700 1 -- 172.21.15.3:0/1595012449 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc0095f0 con 0x7fbfc01006e0 2023-09-10T01:02:54.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc5d9b700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc01985e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0098e0 tx=0x7fbfbc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:54.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfb6ffd700 1 -- 172.21.15.3:0/1595012449 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc01d070 con 0x7fbfc01006e0 2023-09-10T01:02:54.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfb6ffd700 1 -- 172.21.15.3:0/1595012449 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbfbc005510 con 0x7fbfc01006e0 2023-09-10T01:02:54.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfb6ffd700 1 -- 172.21.15.3:0/1595012449 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc004c00 con 0x7fbfc01006e0 2023-09-10T01:02:54.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.164+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc01a06b0 con 0x7fbfc01006e0 2023-09-10T01:02:54.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.165+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc01a0a80 con 0x7fbfc01006e0 2023-09-10T01:02:54.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.166+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc0192d30 con 0x7fbfc01006e0 2023-09-10T01:02:54.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.167+0000 7fbfb6ffd700 1 -- 172.21.15.3:0/1595012449 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fbfbc00fb80 con 0x7fbfc01006e0 2023-09-10T01:02:54.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.169+0000 7fbfb6ffd700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfac0a0640 0x7fbfac0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.169+0000 7fbfb6ffd700 1 -- 172.21.15.3:0/1595012449 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fbfbc0cce70 con 0x7fbfc01006e0 2023-09-10T01:02:54.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.169+0000 7fbfc559a700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfac0a0640 0x7fbfac0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.170+0000 7fbfc559a700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfac0a0640 0x7fbfac0a2b00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009c60 tx=0x7fbfb0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:54.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.172+0000 7fbfb6ffd700 1 -- 172.21.15.3:0/1595012449 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbfbc096620 con 0x7fbfc01006e0 2023-09-10T01:02:54.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:54 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/138723330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:54.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:54 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2608434586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:54.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.385+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbfc002ced0 con 0x7fbfc01006e0 2023-09-10T01:02:54.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.385+0000 7fbfb6ffd700 1 -- 172.21.15.3:0/1595012449 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbfbc099c40 con 0x7fbfc01006e0 2023-09-10T01:02:54.386 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:02:54.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.387+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfac0a0640 msgr2=0x7fbfac0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.387+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfac0a0640 0x7fbfac0a2b00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009c60 tx=0x7fbfb0009260 comp rx=0 tx=0).stop 2023-09-10T01:02:54.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.387+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 msgr2=0x7fbfc01985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.387+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc01985e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0098e0 tx=0x7fbfbc00f690 comp rx=0 tx=0).stop 2023-09-10T01:02:54.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.387+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 shutdown_connections 2023-09-10T01:02:54.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.387+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fbfac0a0640 0x7fbfac0a2b00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.387+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfc0102270 0x7fbfc019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.388+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfc01006e0 0x7fbfc01985e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.388+0000 7fbfc7fff700 1 --2- 172.21.15.3:0/1595012449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfc01018d0 0x7fbfc0198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.388+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 >> 172.21.15.3:0/1595012449 conn(0x7fbfc00fbc50 msgr2=0x7fbfc01071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.388+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 shutdown_connections 2023-09-10T01:02:54.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.388+0000 7fbfc7fff700 1 -- 172.21.15.3:0/1595012449 wait complete. 2023-09-10T01:02:54.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:02:54.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:54 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/138723330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:54 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2608434586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:54 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/138723330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:54.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:54 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2608434586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.613+0000 7f84a5bff700 1 -- 172.21.15.3:0/1212055867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a0102270 msgr2=0x7f84a0108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.613+0000 7f84a5bff700 1 --2- 172.21.15.3:0/1212055867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a0102270 0x7f84a0108b20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f849800b0d0 tx=0x7f849800b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.614+0000 7f84a5bff700 1 -- 172.21.15.3:0/1212055867 shutdown_connections 2023-09-10T01:02:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.614+0000 7f84a5bff700 1 --2- 172.21.15.3:0/1212055867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84a01006e0 0x7f84a0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.614+0000 7f84a5bff700 1 --2- 172.21.15.3:0/1212055867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a0102270 0x7f84a0108b20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.614+0000 7f84a5bff700 1 --2- 172.21.15.3:0/1212055867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84a01018d0 0x7f84a0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.614+0000 7f84a5bff700 1 -- 172.21.15.3:0/1212055867 >> 172.21.15.3:0/1212055867 conn(0x7f84a00fbc50 msgr2=0x7f84a00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.614+0000 7f84a5bff700 1 -- 172.21.15.3:0/1212055867 shutdown_connections 2023-09-10T01:02:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.614+0000 7f84a5bff700 1 -- 172.21.15.3:0/1212055867 wait complete. 2023-09-10T01:02:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 Processor -- start 2023-09-10T01:02:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 -- start start 2023-09-10T01:02:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a01006e0 0x7f84a0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84a01018d0 0x7f84a0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84a0102270 0x7f84a019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01a0200 con 0x7f84a01006e0 2023-09-10T01:02:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01a0340 con 0x7f84a01018d0 2023-09-10T01:02:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.615+0000 7f84a5bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01a04d0 con 0x7f84a0102270 2023-09-10T01:02:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a01006e0 0x7f84a0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a01006e0 0x7f84a0198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41712/0 (socket says 172.21.15.3:41712) 2023-09-10T01:02:54.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 -- 172.21.15.3:0/3417197954 learned_addr learned my addr 172.21.15.3:0/3417197954 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:54.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849effd700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84a01018d0 0x7f84a0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849ffff700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84a0102270 0x7f84a019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 -- 172.21.15.3:0/3417197954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84a0102270 msgr2=0x7f84a019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84a0102270 0x7f84a019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 -- 172.21.15.3:0/3417197954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84a01018d0 msgr2=0x7f84a0198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84a01018d0 0x7f84a0198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 -- 172.21.15.3:0/3417197954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8494009520 con 0x7f84a01006e0 2023-09-10T01:02:54.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.616+0000 7f849f7fe700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a01006e0 0x7f84a0198680 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f849000d980 tx=0x7f849000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:54.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.617+0000 7f849cff9700 1 -- 172.21.15.3:0/3417197954 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849000f890 con 0x7f84a01006e0 2023-09-10T01:02:54.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.617+0000 7f849cff9700 1 -- 172.21.15.3:0/3417197954 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8490010460 con 0x7f84a01006e0 2023-09-10T01:02:54.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.617+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8498009d00 con 0x7f84a01006e0 2023-09-10T01:02:54.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.617+0000 7f849cff9700 1 -- 172.21.15.3:0/3417197954 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84900195c0 con 0x7f84a01006e0 2023-09-10T01:02:54.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.617+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a01a0b80 con 0x7f84a01006e0 2023-09-10T01:02:54.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.618+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a010bf00 con 0x7f84a01006e0 2023-09-10T01:02:54.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.620+0000 7f849cff9700 1 -- 172.21.15.3:0/3417197954 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f84900105d0 con 0x7f84a01006e0 2023-09-10T01:02:54.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.621+0000 7f849cff9700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f84880a06c0 0x7f84880a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:54.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.621+0000 7f849cff9700 1 -- 172.21.15.3:0/3417197954 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f84900cbc30 con 0x7f84a01006e0 2023-09-10T01:02:54.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.621+0000 7f849effd700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f84880a06c0 0x7f84880a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:54.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.622+0000 7f849effd700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f84880a06c0 0x7f84880a2b80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8494009d70 tx=0x7f8494009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:54.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.625+0000 7f849cff9700 1 -- 172.21.15.3:0/3417197954 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f84900953b0 con 0x7f84a01006e0 2023-09-10T01:02:54.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.837+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f84a01a0e00 con 0x7f84a01006e0 2023-09-10T01:02:54.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.838+0000 7f849cff9700 1 -- 172.21.15.3:0/3417197954 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8490020020 con 0x7f84a01006e0 2023-09-10T01:02:54.839 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:54.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.840+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f84880a06c0 msgr2=0x7f84880a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.840+0000 7f84a5bff700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f84880a06c0 0x7f84880a2b80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8494009d70 tx=0x7f8494009260 comp rx=0 tx=0).stop 2023-09-10T01:02:54.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.840+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a01006e0 msgr2=0x7f84a0198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:54.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a01006e0 0x7f84a0198680 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f849000d980 tx=0x7f849000dc90 comp rx=0 tx=0).stop 2023-09-10T01:02:54.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 shutdown_connections 2023-09-10T01:02:54.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f84880a06c0 0x7f84880a2b80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84a0102270 0x7f84a019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84a01006e0 0x7f84a0198680 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 --2- 172.21.15.3:0/3417197954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84a01018d0 0x7f84a0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:54.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 >> 172.21.15.3:0/3417197954 conn(0x7f84a00fbc50 msgr2=0x7f84a00fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:54.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 shutdown_connections 2023-09-10T01:02:54.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:54.841+0000 7f84a5bff700 1 -- 172.21.15.3:0/3417197954 wait complete. 2023-09-10T01:02:54.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:55.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 -- 172.21.15.3:0/2129077862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c141028e0 msgr2=0x7f8c14102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2129077862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c141028e0 0x7f8c14102d00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8c040099c0 tx=0x7f8c04009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 -- 172.21.15.3:0/2129077862 shutdown_connections 2023-09-10T01:02:55.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2129077862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c14104470 0x7f8c1410ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2129077862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c141028e0 0x7f8c14102d00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2129077862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 0x7f8c14103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 -- 172.21.15.3:0/2129077862 >> 172.21.15.3:0/2129077862 conn(0x7f8c140fde50 msgr2=0x7f8c14100290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:55.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 -- 172.21.15.3:0/2129077862 shutdown_connections 2023-09-10T01:02:55.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.072+0000 7f8c195ff700 1 -- 172.21.15.3:0/2129077862 wait complete. 2023-09-10T01:02:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.073+0000 7f8c195ff700 1 Processor -- start 2023-09-10T01:02:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.073+0000 7f8c195ff700 1 -- start start 2023-09-10T01:02:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c195ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c141028e0 0x7f8c141985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c195ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 0x7f8c14198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c195ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c14104470 0x7f8c1419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c195ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141a0150 con 0x7f8c14104470 2023-09-10T01:02:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c195ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141a0290 con 0x7f8c14103ad0 2023-09-10T01:02:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c195ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141a0420 con 0x7f8c141028e0 2023-09-10T01:02:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 0x7f8c14198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c12ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c141028e0 0x7f8c141985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 0x7f8c14198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50080/0 (socket says 172.21.15.3:50080) 2023-09-10T01:02:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 -- 172.21.15.3:0/2971283824 learned_addr learned my addr 172.21.15.3:0/2971283824 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c137fe700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c14104470 0x7f8c1419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 -- 172.21.15.3:0/2971283824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c141028e0 msgr2=0x7f8c141985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c141028e0 0x7f8c141985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 -- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c14104470 msgr2=0x7f8c1419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c14104470 0x7f8c1419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.074+0000 7f8c127fc700 1 -- 172.21.15.3:0/2971283824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040095f0 con 0x7f8c14103ad0 2023-09-10T01:02:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.075+0000 7f8c127fc700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 0x7f8c14198b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800e9a0 tx=0x7f8c0800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.075+0000 7f8bfbfff700 1 -- 172.21.15.3:0/2971283824 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08009790 con 0x7f8c14103ad0 2023-09-10T01:02:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.075+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c141a0700 con 0x7f8c14103ad0 2023-09-10T01:02:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.075+0000 7f8bfbfff700 1 -- 172.21.15.3:0/2971283824 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c08010430 con 0x7f8c14103ad0 2023-09-10T01:02:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.075+0000 7f8bfbfff700 1 -- 172.21.15.3:0/2971283824 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08018680 con 0x7f8c14103ad0 2023-09-10T01:02:55.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.075+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c141a0c60 con 0x7f8c14103ad0 2023-09-10T01:02:55.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.076+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c14109810 con 0x7f8c14103ad0 2023-09-10T01:02:55.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.078+0000 7f8bfbfff700 1 -- 172.21.15.3:0/2971283824 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f8c080105a0 con 0x7f8c14103ad0 2023-09-10T01:02:55.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.079+0000 7f8bfbfff700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8bfc0a06c0 0x7f8bfc0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.079+0000 7f8bfbfff700 1 -- 172.21.15.3:0/2971283824 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f8c08014070 con 0x7f8c14103ad0 2023-09-10T01:02:55.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.080+0000 7f8c12ffd700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8bfc0a06c0 0x7f8bfc0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.080+0000 7f8c12ffd700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8bfc0a06c0 0x7f8bfc0a2b80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009990 tx=0x7f8c04005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:55.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.081+0000 7f8bfbfff700 1 -- 172.21.15.3:0/2971283824 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f8c08095b20 con 0x7f8c14103ad0 2023-09-10T01:02:55.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.292+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c1410e140 con 0x7f8c14103ad0 2023-09-10T01:02:55.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.295+0000 7f8bfbfff700 1 -- 172.21.15.3:0/2971283824 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7f8c08099140 con 0x7f8c14103ad0 2023-09-10T01:02:55.296 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:55.296 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:55.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8bfc0a06c0 msgr2=0x7f8bfc0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8bfc0a06c0 0x7f8bfc0a2b80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009990 tx=0x7f8c04005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 msgr2=0x7f8c14198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 0x7f8c14198b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800e9a0 tx=0x7f8c0800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 shutdown_connections 2023-09-10T01:02:55.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f8bfc0a06c0 0x7f8bfc0a2b80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c141028e0 0x7f8c141985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c14104470 0x7f8c1419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 --2- 172.21.15.3:0/2971283824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c14103ad0 0x7f8c14198b10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 >> 172.21.15.3:0/2971283824 conn(0x7f8c140fde50 msgr2=0x7f8c14100140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:55.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.297+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 shutdown_connections 2023-09-10T01:02:55.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.298+0000 7f8c195ff700 1 -- 172.21.15.3:0/2971283824 wait complete. 2023-09-10T01:02:55.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:55 smithi003 ceph-mon[97654]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-09-10T01:02:55.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1595012449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:55.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3417197954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:55.318 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:02:55.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:02:55.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:55 smithi136 ceph-mon[133708]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-09-10T01:02:55.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1595012449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:55.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3417197954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:55.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:55 smithi092 ceph-mon[134067]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-09-10T01:02:55.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1595012449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:55.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3417197954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:55.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 -- 172.21.15.3:0/2390166875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510074640 msgr2=0x7f8510072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 --2- 172.21.15.3:0/2390166875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510074640 0x7f8510072a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8504009990 tx=0x7f8504009ca0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 -- 172.21.15.3:0/2390166875 shutdown_connections 2023-09-10T01:02:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 --2- 172.21.15.3:0/2390166875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85101085b0 0x7f851010aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 --2- 172.21.15.3:0/2390166875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510074640 0x7f8510072a90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 --2- 172.21.15.3:0/2390166875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8510073060 0x7f85100734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 -- 172.21.15.3:0/2390166875 >> 172.21.15.3:0/2390166875 conn(0x7f85100fbc60 msgr2=0x7f85100fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:55.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 -- 172.21.15.3:0/2390166875 shutdown_connections 2023-09-10T01:02:55.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.525+0000 7f8516fff700 1 -- 172.21.15.3:0/2390166875 wait complete. 2023-09-10T01:02:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.526+0000 7f8516fff700 1 Processor -- start 2023-09-10T01:02:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.526+0000 7f8516fff700 1 -- start start 2023-09-10T01:02:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.526+0000 7f8516fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510073060 0x7f85101985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8516fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8510074640 0x7f8510198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8516fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85101085b0 0x7f851019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8516fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101a0140 con 0x7f8510073060 2023-09-10T01:02:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8516fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101a0280 con 0x7f8510074640 2023-09-10T01:02:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8516fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101a0410 con 0x7f85101085b0 2023-09-10T01:02:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510073060 0x7f85101985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510073060 0x7f85101985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49152/0 (socket says 172.21.15.3:49152) 2023-09-10T01:02:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 -- 172.21.15.3:0/2335147914 learned_addr learned my addr 172.21.15.3:0/2335147914 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f850ffff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8510074640 0x7f8510198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f851559c700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85101085b0 0x7f851019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 -- 172.21.15.3:0/2335147914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85101085b0 msgr2=0x7f851019fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85101085b0 0x7f851019fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 -- 172.21.15.3:0/2335147914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8510074640 msgr2=0x7f8510198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8510074640 0x7f8510198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f8514d9b700 1 -- 172.21.15.3:0/2335147914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85040095f0 con 0x7f8510073060 2023-09-10T01:02:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.527+0000 7f851559c700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85101085b0 0x7f851019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.528+0000 7f8514d9b700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510073060 0x7f85101985c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f850400b680 tx=0x7f85040057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.528+0000 7f850dffb700 1 -- 172.21.15.3:0/2335147914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850401d070 con 0x7f8510073060 2023-09-10T01:02:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.528+0000 7f850dffb700 1 -- 172.21.15.3:0/2335147914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f850400fbc0 con 0x7f8510073060 2023-09-10T01:02:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.528+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85101a0690 con 0x7f8510073060 2023-09-10T01:02:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.528+0000 7f850dffb700 1 -- 172.21.15.3:0/2335147914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8504017650 con 0x7f8510073060 2023-09-10T01:02:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.528+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85101a0b10 con 0x7f8510073060 2023-09-10T01:02:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.529+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8510106c60 con 0x7f8510073060 2023-09-10T01:02:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.531+0000 7f850dffb700 1 -- 172.21.15.3:0/2335147914 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f8504022470 con 0x7f8510073060 2023-09-10T01:02:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.532+0000 7f850dffb700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85000a06c0 0x7f85000a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.532+0000 7f850dffb700 1 -- 172.21.15.3:0/2335147914 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f85040cdd80 con 0x7f8510073060 2023-09-10T01:02:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.532+0000 7f850ffff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85000a06c0 0x7f85000a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.533+0000 7f850ffff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85000a06c0 0x7f85000a2b80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8510074360 tx=0x7f84fc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.534+0000 7f850dffb700 1 -- 172.21.15.3:0/2335147914 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f8504097530 con 0x7f8510073060 2023-09-10T01:02:55.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.746+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f851010e2c0 con 0x7f8510073060 2023-09-10T01:02:55.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.747+0000 7f850dffb700 1 -- 172.21.15.3:0/2335147914 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8504027020 con 0x7f8510073060 2023-09-10T01:02:55.748 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:02:55.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85000a06c0 msgr2=0x7f85000a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85000a06c0 0x7f85000a2b80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8510074360 tx=0x7f84fc00b410 comp rx=0 tx=0).stop 2023-09-10T01:02:55.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510073060 msgr2=0x7f85101985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510073060 0x7f85101985c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f850400b680 tx=0x7f85040057b0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 shutdown_connections 2023-09-10T01:02:55.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f85000a06c0 0x7f85000a2b80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85101085b0 0x7f851019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510073060 0x7f85101985c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 --2- 172.21.15.3:0/2335147914 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8510074640 0x7f8510198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.749+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 >> 172.21.15.3:0/2335147914 conn(0x7f85100fbc60 msgr2=0x7f85100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:55.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.750+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 shutdown_connections 2023-09-10T01:02:55.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.750+0000 7f8516fff700 1 -- 172.21.15.3:0/2335147914 wait complete. 2023-09-10T01:02:55.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:02:55.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 -- 172.21.15.3:0/3382115913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001018e0 msgr2=0x7f1b00101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 --2- 172.21.15.3:0/3382115913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001018e0 0x7f1b00101d40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1af00099c0 tx=0x7f1af0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 -- 172.21.15.3:0/3382115913 shutdown_connections 2023-09-10T01:02:55.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 --2- 172.21.15.3:0/3382115913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b001006f0 0x7f1b00100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 --2- 172.21.15.3:0/3382115913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b00102280 0x7f1b00108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 --2- 172.21.15.3:0/3382115913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001018e0 0x7f1b00101d40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 -- 172.21.15.3:0/3382115913 >> 172.21.15.3:0/3382115913 conn(0x7f1b000fbc60 msgr2=0x7f1b000fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:55.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 -- 172.21.15.3:0/3382115913 shutdown_connections 2023-09-10T01:02:55.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.979+0000 7f1b083ff700 1 -- 172.21.15.3:0/3382115913 wait complete. 2023-09-10T01:02:55.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 Processor -- start 2023-09-10T01:02:55.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 -- start start 2023-09-10T01:02:55.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001006f0 0x7f1b001985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b001018e0 0x7f1b00198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00102280 0x7f1b0019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001a0130 con 0x7f1b001006f0 2023-09-10T01:02:55.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001a0270 con 0x7f1b001018e0 2023-09-10T01:02:55.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b083ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001a0400 con 0x7f1b00102280 2023-09-10T01:02:55.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b0619b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001006f0 0x7f1b001985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b0619b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001006f0 0x7f1b001985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49172/0 (socket says 172.21.15.3:49172) 2023-09-10T01:02:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b0599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b001018e0 0x7f1b00198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.981+0000 7f1b0619b700 1 -- 172.21.15.3:0/1882265125 learned_addr learned my addr 172.21.15.3:0/1882265125 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0699c700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00102280 0x7f1b0019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0619b700 1 -- 172.21.15.3:0/1882265125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00102280 msgr2=0x7f1b0019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0619b700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00102280 0x7f1b0019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0619b700 1 -- 172.21.15.3:0/1882265125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b001018e0 msgr2=0x7f1b00198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0619b700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b001018e0 0x7f1b00198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:55.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0619b700 1 -- 172.21.15.3:0/1882265125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00095f0 con 0x7f1b001006f0 2023-09-10T01:02:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0599a700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b001018e0 0x7f1b00198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b0619b700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001006f0 0x7f1b001985b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00e970 tx=0x7f1afc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1af77fe700 1 -- 172.21.15.3:0/1882265125 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc009790 con 0x7f1b001006f0 2023-09-10T01:02:55.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1af77fe700 1 -- 172.21.15.3:0/1882265125 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1afc00c7e0 con 0x7f1b001006f0 2023-09-10T01:02:55.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b001a06e0 con 0x7f1b001006f0 2023-09-10T01:02:55.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.982+0000 7f1af77fe700 1 -- 172.21.15.3:0/1882265125 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc003c70 con 0x7f1b001006f0 2023-09-10T01:02:55.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.983+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b001a0c40 con 0x7f1b001006f0 2023-09-10T01:02:55.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.984+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0010bf60 con 0x7f1b001006f0 2023-09-10T01:02:55.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.985+0000 7f1af77fe700 1 -- 172.21.15.3:0/1882265125 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f1afc00cc50 con 0x7f1b001006f0 2023-09-10T01:02:55.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.987+0000 7f1af77fe700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1aec0a06c0 0x7f1aec0a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:55.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.987+0000 7f1af77fe700 1 -- 172.21.15.3:0/1882265125 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f1afc014070 con 0x7f1b001006f0 2023-09-10T01:02:55.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.987+0000 7f1b0599a700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1aec0a06c0 0x7f1aec0a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:55.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.988+0000 7f1b0599a700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1aec0a06c0 0x7f1aec0a2b80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1af000b580 tx=0x7f1af0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:55.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:55.990+0000 7f1af77fe700 1 -- 172.21.15.3:0/1882265125 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1afc095830 con 0x7f1b001006f0 2023-09-10T01:02:56.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.206+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1b001a0ec0 con 0x7f1b001006f0 2023-09-10T01:02:56.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.206+0000 7f1af77fe700 1 -- 172.21.15.3:0/1882265125 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1afc003dd0 con 0x7f1b001006f0 2023-09-10T01:02:56.208 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:56.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1aec0a06c0 msgr2=0x7f1aec0a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1aec0a06c0 0x7f1aec0a2b80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1af000b580 tx=0x7f1af0005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001006f0 msgr2=0x7f1b001985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001006f0 0x7f1b001985b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00e970 tx=0x7f1afc00ec80 comp rx=0 tx=0).stop 2023-09-10T01:02:56.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 shutdown_connections 2023-09-10T01:02:56.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1aec0a06c0 0x7f1aec0a2b80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00102280 0x7f1b0019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b001006f0 0x7f1b001985b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 --2- 172.21.15.3:0/1882265125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b001018e0 0x7f1b00198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 >> 172.21.15.3:0/1882265125 conn(0x7f1b000fbc60 msgr2=0x7f1b000fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:56.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 shutdown_connections 2023-09-10T01:02:56.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.209+0000 7f1b083ff700 1 -- 172.21.15.3:0/1882265125 wait complete. 2023-09-10T01:02:56.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:02:56.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:56 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2971283824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:56.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:56 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2335147914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:56 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2971283824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:56 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2335147914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:56.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 -- 172.21.15.3:0/2719591944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe01006e0 msgr2=0x7fdbe0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/2719591944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe01006e0 0x7fdbe0100b00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009990 tx=0x7fdbd0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 -- 172.21.15.3:0/2719591944 shutdown_connections 2023-09-10T01:02:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/2719591944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe01006e0 0x7fdbe0100b00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/2719591944 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102270 0x7fdbe0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/2719591944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01018d0 0x7fdbe0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 -- 172.21.15.3:0/2719591944 >> 172.21.15.3:0/2719591944 conn(0x7fdbe00fbc50 msgr2=0x7fdbe00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:56.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 -- 172.21.15.3:0/2719591944 shutdown_connections 2023-09-10T01:02:56.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.436+0000 7fdbe69ff700 1 -- 172.21.15.3:0/2719591944 wait complete. 2023-09-10T01:02:56.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.437+0000 7fdbe69ff700 1 Processor -- start 2023-09-10T01:02:56.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.437+0000 7fdbe69ff700 1 -- start start 2023-09-10T01:02:56.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbe69ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01006e0 0x7fdbe0196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbe69ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe01018d0 0x7fdbe01968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbe69ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0102270 0x7fdbe019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbe69ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe019df00 con 0x7fdbe01006e0 2023-09-10T01:02:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbe69ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe019e040 con 0x7fdbe01018d0 2023-09-10T01:02:56.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbe69ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe019e1d0 con 0x7fdbe0102270 2023-09-10T01:02:56.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01006e0 0x7fdbe0196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe01018d0 0x7fdbe01968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01006e0 0x7fdbe0196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49188/0 (socket says 172.21.15.3:49188) 2023-09-10T01:02:56.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 -- 172.21.15.3:0/1088048912 learned_addr learned my addr 172.21.15.3:0/1088048912 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:56.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbe4f9c700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0102270 0x7fdbe019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 -- 172.21.15.3:0/1088048912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0102270 msgr2=0x7fdbe019d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0102270 0x7fdbe019d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 -- 172.21.15.3:0/1088048912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe01018d0 msgr2=0x7fdbe01968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe01018d0 0x7fdbe01968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdffff700 1 -- 172.21.15.3:0/1088048912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd00095f0 con 0x7fdbe01006e0 2023-09-10T01:02:56.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.438+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe01018d0 0x7fdbe01968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:56.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.439+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01006e0 0x7fdbe0196380 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0004710 tx=0x7fdbd0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:56.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.439+0000 7fdbdd7fa700 1 -- 172.21.15.3:0/1088048912 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd001d070 con 0x7fdbe01006e0 2023-09-10T01:02:56.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.439+0000 7fdbdd7fa700 1 -- 172.21.15.3:0/1088048912 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbd0021410 con 0x7fdbe01006e0 2023-09-10T01:02:56.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.439+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe019e450 con 0x7fdbe01006e0 2023-09-10T01:02:56.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.439+0000 7fdbdd7fa700 1 -- 172.21.15.3:0/1088048912 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0017650 con 0x7fdbe01006e0 2023-09-10T01:02:56.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.439+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe019e8d0 con 0x7fdbe01006e0 2023-09-10T01:02:56.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.440+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe010bef0 con 0x7fdbe01006e0 2023-09-10T01:02:56.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.442+0000 7fdbdd7fa700 1 -- 172.21.15.3:0/1088048912 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fdbd0021580 con 0x7fdbe01006e0 2023-09-10T01:02:56.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.443+0000 7fdbdd7fa700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdbc80a06c0 0x7fdbc80a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.444+0000 7fdbdd7fa700 1 -- 172.21.15.3:0/1088048912 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fdbd00cd590 con 0x7fdbe01006e0 2023-09-10T01:02:56.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.444+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdbc80a06c0 0x7fdbc80a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.445+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdbc80a06c0 0x7fdbc80a2b80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4011490 tx=0x7fdbd4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:56.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.446+0000 7fdbdd7fa700 1 -- 172.21.15.3:0/1088048912 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fdbd0096d40 con 0x7fdbe01006e0 2023-09-10T01:02:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:56 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2971283824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:02:56.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:56 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2335147914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:02:56.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.658+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdbe019eaf0 con 0x7fdbe01006e0 2023-09-10T01:02:56.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.659+0000 7fdbdd7fa700 1 -- 172.21.15.3:0/1088048912 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdbd009a360 con 0x7fdbe01006e0 2023-09-10T01:02:56.660 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:02:56.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.661+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdbc80a06c0 msgr2=0x7fdbc80a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.661+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdbc80a06c0 0x7fdbc80a2b80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4011490 tx=0x7fdbd4009260 comp rx=0 tx=0).stop 2023-09-10T01:02:56.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.661+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01006e0 msgr2=0x7fdbe0196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.661+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01006e0 0x7fdbe0196380 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0004710 tx=0x7fdbd0009580 comp rx=0 tx=0).stop 2023-09-10T01:02:56.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 shutdown_connections 2023-09-10T01:02:56.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fdbc80a06c0 0x7fdbc80a2b80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0102270 0x7fdbe019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe01006e0 0x7fdbe0196380 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 --2- 172.21.15.3:0/1088048912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe01018d0 0x7fdbe01968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 >> 172.21.15.3:0/1088048912 conn(0x7fdbe00fbc50 msgr2=0x7fdbe00fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:56.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 shutdown_connections 2023-09-10T01:02:56.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.662+0000 7fdbe69ff700 1 -- 172.21.15.3:0/1088048912 wait complete. 2023-09-10T01:02:56.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:02:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/3193215696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 msgr2=0x7fcd68101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/3193215696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68101d30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcd580099c0 tx=0x7fcd58009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/3193215696 shutdown_connections 2023-09-10T01:02:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/3193215696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd681006e0 0x7fcd68100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/3193215696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd68102270 0x7fcd68108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/3193215696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68101d30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/3193215696 >> 172.21.15.3:0/3193215696 conn(0x7fcd680fbc50 msgr2=0x7fcd680fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/3193215696 shutdown_connections 2023-09-10T01:02:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.888+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/3193215696 wait complete. 2023-09-10T01:02:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.889+0000 7fcd6f3ff700 1 Processor -- start 2023-09-10T01:02:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6f3ff700 1 -- start start 2023-09-10T01:02:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6f3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd681006e0 0x7fcd68198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6f3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6f3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd68102270 0x7fcd6819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6f3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd681a00f0 con 0x7fcd681018d0 2023-09-10T01:02:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6f3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd681a0230 con 0x7fcd68102270 2023-09-10T01:02:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6f3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd681a03c0 con 0x7fcd681006e0 2023-09-10T01:02:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6c99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6c99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49200/0 (socket says 172.21.15.3:49200) 2023-09-10T01:02:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6c99a700 1 -- 172.21.15.3:0/506468488 learned_addr learned my addr 172.21.15.3:0/506468488 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6d19b700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd681006e0 0x7fcd68198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.890+0000 7fcd6d99c700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd68102270 0x7fcd6819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6c99a700 1 -- 172.21.15.3:0/506468488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd681006e0 msgr2=0x7fcd68198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6c99a700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd681006e0 0x7fcd68198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6c99a700 1 -- 172.21.15.3:0/506468488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd68102270 msgr2=0x7fcd6819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:56.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6c99a700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd68102270 0x7fcd6819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:56.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6c99a700 1 -- 172.21.15.3:0/506468488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580095f0 con 0x7fcd681018d0 2023-09-10T01:02:56.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6d19b700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd681006e0 0x7fcd68198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:56.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6d99c700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd68102270 0x7fcd6819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:56.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6c99a700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68198ab0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c010 tx=0x7fcd5800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd5e7fc700 1 -- 172.21.15.3:0/506468488 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5801d070 con 0x7fcd681018d0 2023-09-10T01:02:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd5e7fc700 1 -- 172.21.15.3:0/506468488 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd58005050 con 0x7fcd681018d0 2023-09-10T01:02:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd681a0640 con 0x7fcd681018d0 2023-09-10T01:02:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd5e7fc700 1 -- 172.21.15.3:0/506468488 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58004b70 con 0x7fcd681018d0 2023-09-10T01:02:56.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.891+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd681a0ac0 con 0x7fcd681018d0 2023-09-10T01:02:56.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.893+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd68192ce0 con 0x7fcd681018d0 2023-09-10T01:02:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.894+0000 7fcd5e7fc700 1 -- 172.21.15.3:0/506468488 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fcd580051c0 con 0x7fcd681018d0 2023-09-10T01:02:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.896+0000 7fcd5e7fc700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcd540a06c0 0x7fcd540a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.896+0000 7fcd5e7fc700 1 -- 172.21.15.3:0/506468488 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fcd580cce00 con 0x7fcd681018d0 2023-09-10T01:02:56.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.896+0000 7fcd6d19b700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcd540a06c0 0x7fcd540a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:56.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.897+0000 7fcd6d19b700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcd540a06c0 0x7fcd540a2b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcd68101730 tx=0x7fcd64009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:56.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:56.900+0000 7fcd5e7fc700 1 -- 172.21.15.3:0/506468488 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fcd58096610 con 0x7fcd681018d0 2023-09-10T01:02:57.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.116+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcd6802ce90 con 0x7fcd681018d0 2023-09-10T01:02:57.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.116+0000 7fcd5e7fc700 1 -- 172.21.15.3:0/506468488 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcd58099c30 con 0x7fcd681018d0 2023-09-10T01:02:57.118 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:02:57.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcd540a06c0 msgr2=0x7fcd540a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcd540a06c0 0x7fcd540a2b80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcd68101730 tx=0x7fcd64009260 comp rx=0 tx=0).stop 2023-09-10T01:02:57.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 msgr2=0x7fcd68198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68198ab0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800c010 tx=0x7fcd5800f690 comp rx=0 tx=0).stop 2023-09-10T01:02:57.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 shutdown_connections 2023-09-10T01:02:57.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fcd540a06c0 0x7fcd540a2b80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd681006e0 0x7fcd68198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd68102270 0x7fcd6819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 --2- 172.21.15.3:0/506468488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd681018d0 0x7fcd68198ab0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 >> 172.21.15.3:0/506468488 conn(0x7fcd680fbc50 msgr2=0x7fcd681071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:57.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 shutdown_connections 2023-09-10T01:02:57.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.119+0000 7fcd6f3ff700 1 -- 172.21.15.3:0/506468488 wait complete. 2023-09-10T01:02:57.138 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2023-09-10T01:02:57.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:02:57.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:57 smithi003 ceph-mon[97654]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2023-09-10T01:02:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:57 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1882265125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:57.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:57 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1088048912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:57.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 -- 172.21.15.3:0/2313318506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 msgr2=0x7fe1fc108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 --2- 172.21.15.3:0/2313318506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 0x7fe1fc108b00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400b0d0 tx=0x7fe1f400b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 -- 172.21.15.3:0/2313318506 shutdown_connections 2023-09-10T01:02:57.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 --2- 172.21.15.3:0/2313318506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1fc1006c0 0x7fe1fc100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 --2- 172.21.15.3:0/2313318506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 0x7fe1fc108b00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 --2- 172.21.15.3:0/2313318506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1018b0 0x7fe1fc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 -- 172.21.15.3:0/2313318506 >> 172.21.15.3:0/2313318506 conn(0x7fe1fc0fbc10 msgr2=0x7fe1fc0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:57.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 -- 172.21.15.3:0/2313318506 shutdown_connections 2023-09-10T01:02:57.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.343+0000 7fe2013ff700 1 -- 172.21.15.3:0/2313318506 wait complete. 2023-09-10T01:02:57.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.344+0000 7fe2013ff700 1 Processor -- start 2023-09-10T01:02:57.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.344+0000 7fe2013ff700 1 -- start start 2023-09-10T01:02:57.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe2013ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1006c0 0x7fe1fc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe2013ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1fc1018b0 0x7fe1fc198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe2013ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 0x7fe1fc19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe2013ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc1a0230 con 0x7fe1fc102250 2023-09-10T01:02:57.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe2013ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc1a0370 con 0x7fe1fc1006c0 2023-09-10T01:02:57.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe2013ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc1a0500 con 0x7fe1fc1018b0 2023-09-10T01:02:57.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe1fb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 0x7fe1fc19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe1faffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1006c0 0x7fe1fc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe1faffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1006c0 0x7fe1fc1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50180/0 (socket says 172.21.15.3:50180) 2023-09-10T01:02:57.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe1faffd700 1 -- 172.21.15.3:0/1142957311 learned_addr learned my addr 172.21.15.3:0/1142957311 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:57.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.345+0000 7fe1fa7fc700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1fc1018b0 0x7fe1fc198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1faffd700 1 -- 172.21.15.3:0/1142957311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1fc1018b0 msgr2=0x7fe1fc198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1faffd700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1fc1018b0 0x7fe1fc198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1faffd700 1 -- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 msgr2=0x7fe1fc19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1faffd700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 0x7fe1fc19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1faffd700 1 -- 172.21.15.3:0/1142957311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f4009d00 con 0x7fe1fc1006c0 2023-09-10T01:02:57.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1fb7fe700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 0x7fe1fc19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:02:57.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1faffd700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1006c0 0x7fe1fc1986b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00e970 tx=0x7fe1ec00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:57.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1e3fff700 1 -- 172.21.15.3:0/1142957311 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec009790 con 0x7fe1fc1006c0 2023-09-10T01:02:57.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc1a07e0 con 0x7fe1fc1006c0 2023-09-10T01:02:57.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1e3fff700 1 -- 172.21.15.3:0/1142957311 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1ec00c7e0 con 0x7fe1fc1006c0 2023-09-10T01:02:57.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe1e3fff700 1 -- 172.21.15.3:0/1142957311 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec003c70 con 0x7fe1fc1006c0 2023-09-10T01:02:57.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.346+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc1a0d40 con 0x7fe1fc1006c0 2023-09-10T01:02:57.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.347+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc10bf20 con 0x7fe1fc1006c0 2023-09-10T01:02:57.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.349+0000 7fe1e3fff700 1 -- 172.21.15.3:0/1142957311 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fe1ec00cc50 con 0x7fe1fc1006c0 2023-09-10T01:02:57.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.351+0000 7fe1e3fff700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe1e40a0640 0x7fe1e40a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.351+0000 7fe1e3fff700 1 -- 172.21.15.3:0/1142957311 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fe1ec014070 con 0x7fe1fc1006c0 2023-09-10T01:02:57.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.351+0000 7fe1fa7fc700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe1e40a0640 0x7fe1e40a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.351+0000 7fe1fa7fc700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe1e40a0640 0x7fe1e40a2b00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009c60 tx=0x7fe1f0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:57.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.353+0000 7fe1e3fff700 1 -- 172.21.15.3:0/1142957311 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fe1ec095a10 con 0x7fe1fc1006c0 2023-09-10T01:02:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:57 smithi136 ceph-mon[133708]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2023-09-10T01:02:57.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:57 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1882265125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:57.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:57 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1088048912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:57.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:57 smithi092 ceph-mon[134067]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2023-09-10T01:02:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:57 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1882265125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:02:57.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:57 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1088048912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:02:57.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.612+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1fc1a0fc0 con 0x7fe1fc1006c0 2023-09-10T01:02:57.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.613+0000 7fe1e3fff700 1 -- 172.21.15.3:0/1142957311 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7fe1ec003dd0 con 0x7fe1fc1006c0 2023-09-10T01:02:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:57.615 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:02:57.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe1e40a0640 msgr2=0x7fe1e40a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe1e40a0640 0x7fe1e40a2b00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009c60 tx=0x7fe1f0009260 comp rx=0 tx=0).stop 2023-09-10T01:02:57.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1006c0 msgr2=0x7fe1fc1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1006c0 0x7fe1fc1986b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00e970 tx=0x7fe1ec00ec80 comp rx=0 tx=0).stop 2023-09-10T01:02:57.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 shutdown_connections 2023-09-10T01:02:57.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1fc102250 0x7fe1fc19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1fc1006c0 0x7fe1fc1986b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe1e40a0640 0x7fe1e40a2b00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 --2- 172.21.15.3:0/1142957311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1fc1018b0 0x7fe1fc198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 >> 172.21.15.3:0/1142957311 conn(0x7fe1fc0fbc10 msgr2=0x7fe1fc107140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:57.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 shutdown_connections 2023-09-10T01:02:57.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.616+0000 7fe2013ff700 1 -- 172.21.15.3:0/1142957311 wait complete. 2023-09-10T01:02:57.620 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:02:57.635 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2023-09-10T01:02:57.636 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2023-09-10T01:02:57.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-10T01:02:57.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.839+0000 7fb6cbbff700 1 -- 172.21.15.3:0/2984873680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c4102250 msgr2=0x7fb6c4108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.839+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/2984873680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c4102250 0x7fb6c4108b00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00b0d0 tx=0x7fb6bc00b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.839+0000 7fb6cbbff700 1 -- 172.21.15.3:0/2984873680 shutdown_connections 2023-09-10T01:02:57.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.839+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/2984873680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6c41006c0 0x7fb6c4100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.839+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/2984873680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c4102250 0x7fb6c4108b00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.839+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/2984873680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c41018b0 0x7fb6c4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.839+0000 7fb6cbbff700 1 -- 172.21.15.3:0/2984873680 >> 172.21.15.3:0/2984873680 conn(0x7fb6c40fbc30 msgr2=0x7fb6c40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:57.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.840+0000 7fb6cbbff700 1 -- 172.21.15.3:0/2984873680 shutdown_connections 2023-09-10T01:02:57.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.840+0000 7fb6cbbff700 1 -- 172.21.15.3:0/2984873680 wait complete. 2023-09-10T01:02:57.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 Processor -- start 2023-09-10T01:02:57.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 -- start start 2023-09-10T01:02:57.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6c41006c0 0x7fb6c41964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c41018b0 0x7fb6c4196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102250 0x7fb6c419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c419e050 con 0x7fb6c41018b0 2023-09-10T01:02:57.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c419e190 con 0x7fb6c4102250 2023-09-10T01:02:57.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6cbbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c419e320 con 0x7fb6c41006c0 2023-09-10T01:02:57.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6c919a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c41018b0 0x7fb6c4196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6c919a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c41018b0 0x7fb6c4196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49244/0 (socket says 172.21.15.3:49244) 2023-09-10T01:02:57.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6c919a700 1 -- 172.21.15.3:0/3528384210 learned_addr learned my addr 172.21.15.3:0/3528384210 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:57.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.841+0000 7fb6c999b700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6c41006c0 0x7fb6c41964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6ca19c700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102250 0x7fb6c419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6c919a700 1 -- 172.21.15.3:0/3528384210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6c41006c0 msgr2=0x7fb6c41964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6c919a700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6c41006c0 0x7fb6c41964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6c919a700 1 -- 172.21.15.3:0/3528384210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102250 msgr2=0x7fb6c419da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:57.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6c919a700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102250 0x7fb6c419da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:57.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6c919a700 1 -- 172.21.15.3:0/3528384210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc009d00 con 0x7fb6c41018b0 2023-09-10T01:02:57.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6ca19c700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102250 0x7fb6c419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6c919a700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c41018b0 0x7fb6c4196a10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb6b400e9a0 tx=0x7fb6b400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6baffd700 1 -- 172.21.15.3:0/3528384210 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4009790 con 0x7fb6c41018b0 2023-09-10T01:02:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6baffd700 1 -- 172.21.15.3:0/3528384210 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6b400cbd0 con 0x7fb6c41018b0 2023-09-10T01:02:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c419e600 con 0x7fb6c41018b0 2023-09-10T01:02:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6baffd700 1 -- 172.21.15.3:0/3528384210 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4018680 con 0x7fb6c41018b0 2023-09-10T01:02:57.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.842+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c419eb10 con 0x7fb6c41018b0 2023-09-10T01:02:57.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.844+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c4107c10 con 0x7fb6c41018b0 2023-09-10T01:02:57.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.845+0000 7fb6baffd700 1 -- 172.21.15.3:0/3528384210 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb6b400cd40 con 0x7fb6c41018b0 2023-09-10T01:02:57.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.847+0000 7fb6baffd700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb6b00a0640 0x7fb6b00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:57.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.847+0000 7fb6baffd700 1 -- 172.21.15.3:0/3528384210 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fb6b4014070 con 0x7fb6c41018b0 2023-09-10T01:02:57.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.847+0000 7fb6c999b700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb6b00a0640 0x7fb6b00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:57.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.848+0000 7fb6c999b700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb6b00a0640 0x7fb6b00a2b00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4101710 tx=0x7fb6c0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:57.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:57.850+0000 7fb6baffd700 1 -- 172.21.15.3:0/3528384210 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb6b4051e10 con 0x7fb6c41018b0 2023-09-10T01:02:58.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.062+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6c402cf00 con 0x7fb6b00a0640 2023-09-10T01:02:58.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.076+0000 7fb6baffd700 1 -- 172.21.15.3:0/3528384210 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427635 (secure 0 0 0) 0x7fb6c402cf00 con 0x7fb6b00a0640 2023-09-10T01:02:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:58.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb6b00a0640 msgr2=0x7fb6b00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb6b00a0640 0x7fb6b00a2b00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4101710 tx=0x7fb6c0009260 comp rx=0 tx=0).stop 2023-09-10T01:02:58.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c41018b0 msgr2=0x7fb6c4196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c41018b0 0x7fb6c4196a10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb6b400e9a0 tx=0x7fb6b400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 shutdown_connections 2023-09-10T01:02:58.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb6b00a0640 0x7fb6b00a2b00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6c41006c0 0x7fb6c41964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102250 0x7fb6c419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 --2- 172.21.15.3:0/3528384210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6c41018b0 0x7fb6c4196a10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 >> 172.21.15.3:0/3528384210 conn(0x7fb6c40fbc30 msgr2=0x7fb6c4104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:58.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 shutdown_connections 2023-09-10T01:02:58.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.082+0000 7fb6cbbff700 1 -- 172.21.15.3:0/3528384210 wait complete. 2023-09-10T01:02:58.091 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-09-10T01:02:58.102 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2023-09-10T01:02:57.882984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659845,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":813,"num_read_kb":532,"num_write":349,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336256,"kb_used_data":12064,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605568,"statfs":{"total":1151940427776,"available":1151596101632,"internally_reserved":0,"allocated":12353536,"data_stored":7999030,"data_compressed":391800,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":93,"num_read_kb":52,"num_write":7,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003042"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885419+0000","last_change":"2023-09-10T01:00:44.026223+0000","last_active":"2023-09-10T01:02:25.885419+0000","last_peered":"2023-09-10T01:02:25.885419+0000","last_clean":"2023-09-10T01:02:25.885419+0000","last_became_active":"2023-09-10T01:00:44.025731+0000","last_became_peered":"2023-09-10T01:00:44.025731+0000","last_unstale":"2023-09-10T01:02:25.885419+0000","last_undegraded":"2023-09-10T01:02:25.885419+0000","last_fullsized":"2023-09-10T01:02:25.885419+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:51:48.949903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871454+0000","last_change":"2023-09-10T01:00:44.038916+0000","last_active":"2023-09-10T01:02:25.871454+0000","last_peered":"2023-09-10T01:02:25.871454+0000","last_clean":"2023-09-10T01:02:25.871454+0000","last_became_active":"2023-09-10T01:00:44.038546+0000","last_became_peered":"2023-09-10T01:00:44.038546+0000","last_unstale":"2023-09-10T01:02:25.871454+0000","last_undegraded":"2023-09-10T01:02:25.871454+0000","last_fullsized":"2023-09-10T01:02:25.871454+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:05:56.096857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880464+0000","last_change":"2023-09-10T01:00:44.032942+0000","last_active":"2023-09-10T01:02:25.880464+0000","last_peered":"2023-09-10T01:02:25.880464+0000","last_clean":"2023-09-10T01:02:25.880464+0000","last_became_active":"2023-09-10T01:00:44.032546+0000","last_became_peered":"2023-09-10T01:00:44.032546+0000","last_unstale":"2023-09-10T01:02:25.880464+0000","last_undegraded":"2023-09-10T01:02:25.880464+0000","last_fullsized":"2023-09-10T01:02:25.880464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:46.613487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.874834+0000","last_change":"2023-09-10T01:00:44.031411+0000","last_active":"2023-09-10T01:02:25.874834+0000","last_peered":"2023-09-10T01:02:25.874834+0000","last_clean":"2023-09-10T01:02:25.874834+0000","last_became_active":"2023-09-10T01:00:44.031121+0000","last_became_peered":"2023-09-10T01:00:44.031121+0000","last_unstale":"2023-09-10T01:02:25.874834+0000","last_undegraded":"2023-09-10T01:02:25.874834+0000","last_fullsized":"2023-09-10T01:02:25.874834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:47.658116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887476+0000","last_change":"2023-09-10T01:00:44.037351+0000","last_active":"2023-09-10T01:02:25.887476+0000","last_peered":"2023-09-10T01:02:25.887476+0000","last_clean":"2023-09-10T01:02:25.887476+0000","last_became_active":"2023-09-10T01:00:44.037005+0000","last_became_peered":"2023-09-10T01:00:44.037005+0000","last_unstale":"2023-09-10T01:02:25.887476+0000","last_undegraded":"2023-09-10T01:02:25.887476+0000","last_fullsized":"2023-09-10T01:02:25.887476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:10:45.694056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882017+0000","last_change":"2023-09-10T01:00:44.029543+0000","last_active":"2023-09-10T01:02:25.882017+0000","last_peered":"2023-09-10T01:02:25.882017+0000","last_clean":"2023-09-10T01:02:25.882017+0000","last_became_active":"2023-09-10T01:00:44.029210+0000","last_became_peered":"2023-09-10T01:00:44.029210+0000","last_unstale":"2023-09-10T01:02:25.882017+0000","last_undegraded":"2023-09-10T01:02:25.882017+0000","last_fullsized":"2023-09-10T01:02:25.882017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:08:56.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879037+0000","last_change":"2023-09-10T01:00:44.033328+0000","last_active":"2023-09-10T01:02:25.879037+0000","last_peered":"2023-09-10T01:02:25.879037+0000","last_clean":"2023-09-10T01:02:25.879037+0000","last_became_active":"2023-09-10T01:00:44.033007+0000","last_became_peered":"2023-09-10T01:00:44.033007+0000","last_unstale":"2023-09-10T01:02:25.879037+0000","last_undegraded":"2023-09-10T01:02:25.879037+0000","last_fullsized":"2023-09-10T01:02:25.879037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:59:14.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891940+0000","last_change":"2023-09-10T01:00:44.025611+0000","last_active":"2023-09-10T01:02:25.891940+0000","last_peered":"2023-09-10T01:02:25.891940+0000","last_clean":"2023-09-10T01:02:25.891940+0000","last_became_active":"2023-09-10T01:00:44.025249+0000","last_became_peered":"2023-09-10T01:00:44.025249+0000","last_unstale":"2023-09-10T01:02:25.891940+0000","last_undegraded":"2023-09-10T01:02:25.891940+0000","last_fullsized":"2023-09-10T01:02:25.891940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:55:04.320356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878755+0000","last_change":"2023-09-10T01:00:44.024207+0000","last_active":"2023-09-10T01:02:25.878755+0000","last_peered":"2023-09-10T01:02:25.878755+0000","last_clean":"2023-09-10T01:02:25.878755+0000","last_became_active":"2023-09-10T01:00:44.023866+0000","last_became_peered":"2023-09-10T01:00:44.023866+0000","last_unstale":"2023-09-10T01:02:25.878755+0000","last_undegraded":"2023-09-10T01:02:25.878755+0000","last_fullsized":"2023-09-10T01:02:25.878755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:53:39.845384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883727+0000","last_change":"2023-09-10T01:00:44.031241+0000","last_active":"2023-09-10T01:02:25.883727+0000","last_peered":"2023-09-10T01:02:25.883727+0000","last_clean":"2023-09-10T01:02:25.883727+0000","last_became_active":"2023-09-10T01:00:44.030392+0000","last_became_peered":"2023-09-10T01:00:44.030392+0000","last_unstale":"2023-09-10T01:02:25.883727+0000","last_undegraded":"2023-09-10T01:02:25.883727+0000","last_fullsized":"2023-09-10T01:02:25.883727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:19:39.786995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883954+0000","last_change":"2023-09-10T01:00:44.026983+0000","last_active":"2023-09-10T01:02:25.883954+0000","last_peered":"2023-09-10T01:02:25.883954+0000","last_clean":"2023-09-10T01:02:25.883954+0000","last_became_active":"2023-09-10T01:00:44.026695+0000","last_became_peered":"2023-09-10T01:00:44.026695+0000","last_unstale":"2023-09-10T01:02:25.883954+0000","last_undegraded":"2023-09-10T01:02:25.883954+0000","last_fullsized":"2023-09-10T01:02:25.883954+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:37.809144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.874638+0000","last_change":"2023-09-10T01:00:44.030787+0000","last_active":"2023-09-10T01:02:25.874638+0000","last_peered":"2023-09-10T01:02:25.874638+0000","last_clean":"2023-09-10T01:02:25.874638+0000","last_became_active":"2023-09-10T01:00:44.028579+0000","last_became_peered":"2023-09-10T01:00:44.028579+0000","last_unstale":"2023-09-10T01:02:25.874638+0000","last_undegraded":"2023-09-10T01:02:25.874638+0000","last_fullsized":"2023-09-10T01:02:25.874638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:04:26.446378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.886732+0000","last_change":"2023-09-10T01:00:44.029905+0000","last_active":"2023-09-10T01:02:25.886732+0000","last_peered":"2023-09-10T01:02:25.886732+0000","last_clean":"2023-09-10T01:02:25.886732+0000","last_became_active":"2023-09-10T01:00:44.029561+0000","last_became_peered":"2023-09-10T01:00:44.029561+0000","last_unstale":"2023-09-10T01:02:25.886732+0000","last_undegraded":"2023-09-10T01:02:25.886732+0000","last_fullsized":"2023-09-10T01:02:25.886732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:17:50.839001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.877754+0000","last_change":"2023-09-10T01:00:44.037864+0000","last_active":"2023-09-10T01:02:25.877754+0000","last_peered":"2023-09-10T01:02:25.877754+0000","last_clean":"2023-09-10T01:02:25.877754+0000","last_became_active":"2023-09-10T01:00:44.037533+0000","last_became_peered":"2023-09-10T01:00:44.037533+0000","last_unstale":"2023-09-10T01:02:25.877754+0000","last_undegraded":"2023-09-10T01:02:25.877754+0000","last_fullsized":"2023-09-10T01:02:25.877754+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:29:26.768570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.869883+0000","last_change":"2023-09-10T01:00:44.033396+0000","last_active":"2023-09-10T01:02:25.869883+0000","last_peered":"2023-09-10T01:02:25.869883+0000","last_clean":"2023-09-10T01:02:25.869883+0000","last_became_active":"2023-09-10T01:00:44.033056+0000","last_became_peered":"2023-09-10T01:00:44.033056+0000","last_unstale":"2023-09-10T01:02:25.869883+0000","last_undegraded":"2023-09-10T01:02:25.869883+0000","last_fullsized":"2023-09-10T01:02:25.869883+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:00:03.590555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882604+0000","last_change":"2023-09-10T01:00:44.027729+0000","last_active":"2023-09-10T01:02:25.882604+0000","last_peered":"2023-09-10T01:02:25.882604+0000","last_clean":"2023-09-10T01:02:25.882604+0000","last_became_active":"2023-09-10T01:00:44.027447+0000","last_became_peered":"2023-09-10T01:00:44.027447+0000","last_unstale":"2023-09-10T01:02:25.882604+0000","last_undegraded":"2023-09-10T01:02:25.882604+0000","last_fullsized":"2023-09-10T01:02:25.882604+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:06:09.391443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885073+0000","last_change":"2023-09-10T01:00:44.026042+0000","last_active":"2023-09-10T01:02:25.885073+0000","last_peered":"2023-09-10T01:02:25.885073+0000","last_clean":"2023-09-10T01:02:25.885073+0000","last_became_active":"2023-09-10T01:00:44.025309+0000","last_became_peered":"2023-09-10T01:00:44.025309+0000","last_unstale":"2023-09-10T01:02:25.885073+0000","last_undegraded":"2023-09-10T01:02:25.885073+0000","last_fullsized":"2023-09-10T01:02:25.885073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:51:29.038717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.886092+0000","last_change":"2023-09-10T01:00:44.036761+0000","last_active":"2023-09-10T01:02:25.886092+0000","last_peered":"2023-09-10T01:02:25.886092+0000","last_clean":"2023-09-10T01:02:25.886092+0000","last_became_active":"2023-09-10T01:00:44.036333+0000","last_became_peered":"2023-09-10T01:00:44.036333+0000","last_unstale":"2023-09-10T01:02:25.886092+0000","last_undegraded":"2023-09-10T01:02:25.886092+0000","last_fullsized":"2023-09-10T01:02:25.886092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:37:09.042178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885302+0000","last_change":"2023-09-10T01:00:44.027304+0000","last_active":"2023-09-10T01:02:25.885302+0000","last_peered":"2023-09-10T01:02:25.885302+0000","last_clean":"2023-09-10T01:02:25.885302+0000","last_became_active":"2023-09-10T01:00:44.026994+0000","last_became_peered":"2023-09-10T01:00:44.026994+0000","last_unstale":"2023-09-10T01:02:25.885302+0000","last_undegraded":"2023-09-10T01:02:25.885302+0000","last_fullsized":"2023-09-10T01:02:25.885302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:11:04.824687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032471+0000","last_change":"2023-09-10T01:00:48.056098+0000","last_active":"2023-09-10T01:01:56.032471+0000","last_peered":"2023-09-10T01:01:56.032471+0000","last_clean":"2023-09-10T01:01:56.032471+0000","last_became_active":"2023-09-10T01:00:48.055694+0000","last_became_peered":"2023-09-10T01:00:48.055694+0000","last_unstale":"2023-09-10T01:01:56.032471+0000","last_undegraded":"2023-09-10T01:01:56.032471+0000","last_fullsized":"2023-09-10T01:01:56.032471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:30:48.711613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.046616+0000","last_change":"2023-09-10T01:00:56.971435+0000","last_active":"2023-09-10T01:01:57.046616+0000","last_peered":"2023-09-10T01:01:57.046616+0000","last_clean":"2023-09-10T01:01:57.046616+0000","last_became_active":"2023-09-10T01:00:56.965158+0000","last_became_peered":"2023-09-10T01:00:56.965158+0000","last_unstale":"2023-09-10T01:01:57.046616+0000","last_undegraded":"2023-09-10T01:01:57.046616+0000","last_fullsized":"2023-09-10T01:01:57.046616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:09:20.910597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880753+0000","last_change":"2023-09-10T01:00:44.023469+0000","last_active":"2023-09-10T01:02:25.880753+0000","last_peered":"2023-09-10T01:02:25.880753+0000","last_clean":"2023-09-10T01:02:25.880753+0000","last_became_active":"2023-09-10T01:00:44.023232+0000","last_became_peered":"2023-09-10T01:00:44.023232+0000","last_unstale":"2023-09-10T01:02:25.880753+0000","last_undegraded":"2023-09-10T01:02:25.880753+0000","last_fullsized":"2023-09-10T01:02:25.880753+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:33:22.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037995+0000","last_change":"2023-09-10T01:00:48.063565+0000","last_active":"2023-09-10T01:01:56.037995+0000","last_peered":"2023-09-10T01:01:56.037995+0000","last_clean":"2023-09-10T01:01:56.037995+0000","last_became_active":"2023-09-10T01:00:48.063200+0000","last_became_peered":"2023-09-10T01:00:48.063200+0000","last_unstale":"2023-09-10T01:01:56.037995+0000","last_undegraded":"2023-09-10T01:01:56.037995+0000","last_fullsized":"2023-09-10T01:01:56.037995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:50:16.624296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.471781+0000","last_change":"2023-09-10T01:00:43.010566+0000","last_active":"2023-09-10T01:01:57.471781+0000","last_peered":"2023-09-10T01:01:57.471781+0000","last_clean":"2023-09-10T01:01:57.471781+0000","last_became_active":"2023-09-10T01:00:43.010277+0000","last_became_peered":"2023-09-10T01:00:43.010277+0000","last_unstale":"2023-09-10T01:01:57.471781+0000","last_undegraded":"2023-09-10T01:01:57.471781+0000","last_fullsized":"2023-09-10T01:01:57.471781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:37.808085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.884035+0000","last_change":"2023-09-10T01:00:44.026219+0000","last_active":"2023-09-10T01:02:25.884035+0000","last_peered":"2023-09-10T01:02:25.884035+0000","last_clean":"2023-09-10T01:02:25.884035+0000","last_became_active":"2023-09-10T01:00:44.025906+0000","last_became_peered":"2023-09-10T01:00:44.025906+0000","last_unstale":"2023-09-10T01:02:25.884035+0000","last_undegraded":"2023-09-10T01:02:25.884035+0000","last_fullsized":"2023-09-10T01:02:25.884035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:21:48.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037347+0000","last_change":"2023-09-10T01:00:48.069671+0000","last_active":"2023-09-10T01:01:56.037347+0000","last_peered":"2023-09-10T01:01:56.037347+0000","last_clean":"2023-09-10T01:01:56.037347+0000","last_became_active":"2023-09-10T01:00:48.069382+0000","last_became_peered":"2023-09-10T01:00:48.069382+0000","last_unstale":"2023-09-10T01:01:56.037347+0000","last_undegraded":"2023-09-10T01:01:56.037347+0000","last_fullsized":"2023-09-10T01:01:56.037347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:06:20.526398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036543+0000","last_change":"2023-09-10T01:00:43.005098+0000","last_active":"2023-09-10T01:01:56.036543+0000","last_peered":"2023-09-10T01:01:56.036543+0000","last_clean":"2023-09-10T01:01:56.036543+0000","last_became_active":"2023-09-10T01:00:43.004556+0000","last_became_peered":"2023-09-10T01:00:43.004556+0000","last_unstale":"2023-09-10T01:01:56.036543+0000","last_undegraded":"2023-09-10T01:01:56.036543+0000","last_fullsized":"2023-09-10T01:01:56.036543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:27:43.538936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890280+0000","last_change":"2023-09-10T01:00:44.034451+0000","last_active":"2023-09-10T01:02:25.890280+0000","last_peered":"2023-09-10T01:02:25.890280+0000","last_clean":"2023-09-10T01:02:25.890280+0000","last_became_active":"2023-09-10T01:00:44.034131+0000","last_became_peered":"2023-09-10T01:00:44.034131+0000","last_unstale":"2023-09-10T01:02:25.890280+0000","last_undegraded":"2023-09-10T01:02:25.890280+0000","last_fullsized":"2023-09-10T01:02:25.890280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:50.643215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036946+0000","last_change":"2023-09-10T01:00:48.060114+0000","last_active":"2023-09-10T01:01:56.036946+0000","last_peered":"2023-09-10T01:01:56.036946+0000","last_clean":"2023-09-10T01:01:56.036946+0000","last_became_active":"2023-09-10T01:00:48.059531+0000","last_became_peered":"2023-09-10T01:00:48.059531+0000","last_unstale":"2023-09-10T01:01:56.036946+0000","last_undegraded":"2023-09-10T01:01:56.036946+0000","last_fullsized":"2023-09-10T01:01:56.036946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:28:01.462247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035950+0000","last_change":"2023-09-10T01:00:43.007196+0000","last_active":"2023-09-10T01:01:56.035950+0000","last_peered":"2023-09-10T01:01:56.035950+0000","last_clean":"2023-09-10T01:01:56.035950+0000","last_became_active":"2023-09-10T01:00:43.006795+0000","last_became_peered":"2023-09-10T01:00:43.006795+0000","last_unstale":"2023-09-10T01:01:56.035950+0000","last_undegraded":"2023-09-10T01:01:56.035950+0000","last_fullsized":"2023-09-10T01:01:56.035950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:26:13.146244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882156+0000","last_change":"2023-09-10T01:00:44.027809+0000","last_active":"2023-09-10T01:02:25.882156+0000","last_peered":"2023-09-10T01:02:25.882156+0000","last_clean":"2023-09-10T01:02:25.882156+0000","last_became_active":"2023-09-10T01:00:44.027499+0000","last_became_peered":"2023-09-10T01:00:44.027499+0000","last_unstale":"2023-09-10T01:02:25.882156+0000","last_undegraded":"2023-09-10T01:02:25.882156+0000","last_fullsized":"2023-09-10T01:02:25.882156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:17:07.138807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032849+0000","last_change":"2023-09-10T01:00:48.057645+0000","last_active":"2023-09-10T01:01:56.032849+0000","last_peered":"2023-09-10T01:01:56.032849+0000","last_clean":"2023-09-10T01:01:56.032849+0000","last_became_active":"2023-09-10T01:00:48.057312+0000","last_became_peered":"2023-09-10T01:00:48.057312+0000","last_unstale":"2023-09-10T01:01:56.032849+0000","last_undegraded":"2023-09-10T01:01:56.032849+0000","last_fullsized":"2023-09-10T01:01:56.032849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:28.904572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:56.110754+0000","last_change":"2023-09-10T01:00:56.965677+0000","last_active":"2023-09-10T01:02:56.110754+0000","last_peered":"2023-09-10T01:02:56.110754+0000","last_clean":"2023-09-10T01:02:56.110754+0000","last_became_active":"2023-09-10T01:00:56.965248+0000","last_became_peered":"2023-09-10T01:00:56.965248+0000","last_unstale":"2023-09-10T01:02:56.110754+0000","last_undegraded":"2023-09-10T01:02:56.110754+0000","last_fullsized":"2023-09-10T01:02:56.110754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:39:44.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.866210+0000","last_change":"2023-09-10T01:00:44.026340+0000","last_active":"2023-09-10T01:02:25.866210+0000","last_peered":"2023-09-10T01:02:25.866210+0000","last_clean":"2023-09-10T01:02:25.866210+0000","last_became_active":"2023-09-10T01:00:44.026024+0000","last_became_peered":"2023-09-10T01:00:44.026024+0000","last_unstale":"2023-09-10T01:02:25.866210+0000","last_undegraded":"2023-09-10T01:02:25.866210+0000","last_fullsized":"2023-09-10T01:02:25.866210+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:58:18.942980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032668+0000","last_change":"2023-09-10T01:00:48.057876+0000","last_active":"2023-09-10T01:01:56.032668+0000","last_peered":"2023-09-10T01:01:56.032668+0000","last_clean":"2023-09-10T01:01:56.032668+0000","last_became_active":"2023-09-10T01:00:48.056686+0000","last_became_peered":"2023-09-10T01:00:48.056686+0000","last_unstale":"2023-09-10T01:01:56.032668+0000","last_undegraded":"2023-09-10T01:01:56.032668+0000","last_fullsized":"2023-09-10T01:01:56.032668+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:10:15.995463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127691+0000","last_change":"2023-09-10T01:00:43.007811+0000","last_active":"2023-09-10T01:01:56.127691+0000","last_peered":"2023-09-10T01:01:56.127691+0000","last_clean":"2023-09-10T01:01:56.127691+0000","last_became_active":"2023-09-10T01:00:43.007205+0000","last_became_peered":"2023-09-10T01:00:43.007205+0000","last_unstale":"2023-09-10T01:01:56.127691+0000","last_undegraded":"2023-09-10T01:01:56.127691+0000","last_fullsized":"2023-09-10T01:01:56.127691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:27:38.114614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.888059+0000","last_change":"2023-09-10T01:00:44.027954+0000","last_active":"2023-09-10T01:02:25.888059+0000","last_peered":"2023-09-10T01:02:25.888059+0000","last_clean":"2023-09-10T01:02:25.888059+0000","last_became_active":"2023-09-10T01:00:44.027643+0000","last_became_peered":"2023-09-10T01:00:44.027643+0000","last_unstale":"2023-09-10T01:02:25.888059+0000","last_undegraded":"2023-09-10T01:02:25.888059+0000","last_fullsized":"2023-09-10T01:02:25.888059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:21.253013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.731942+0000","last_change":"2023-09-10T01:00:48.062519+0000","last_active":"2023-09-10T01:01:58.731942+0000","last_peered":"2023-09-10T01:01:58.731942+0000","last_clean":"2023-09-10T01:01:58.731942+0000","last_became_active":"2023-09-10T01:00:48.061985+0000","last_became_peered":"2023-09-10T01:00:48.061985+0000","last_unstale":"2023-09-10T01:01:58.731942+0000","last_undegraded":"2023-09-10T01:01:58.731942+0000","last_fullsized":"2023-09-10T01:01:58.731942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:47:36.002325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030435+0000","last_change":"2023-09-10T01:00:42.999601+0000","last_active":"2023-09-10T01:01:56.030435+0000","last_peered":"2023-09-10T01:01:56.030435+0000","last_clean":"2023-09-10T01:01:56.030435+0000","last_became_active":"2023-09-10T01:00:42.999243+0000","last_became_peered":"2023-09-10T01:00:42.999243+0000","last_unstale":"2023-09-10T01:01:56.030435+0000","last_undegraded":"2023-09-10T01:01:56.030435+0000","last_fullsized":"2023-09-10T01:01:56.030435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:49.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890496+0000","last_change":"2023-09-10T01:00:44.027136+0000","last_active":"2023-09-10T01:02:25.890496+0000","last_peered":"2023-09-10T01:02:25.890496+0000","last_clean":"2023-09-10T01:02:25.890496+0000","last_became_active":"2023-09-10T01:00:44.026784+0000","last_became_peered":"2023-09-10T01:00:44.026784+0000","last_unstale":"2023-09-10T01:02:25.890496+0000","last_undegraded":"2023-09-10T01:02:25.890496+0000","last_fullsized":"2023-09-10T01:02:25.890496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:51:51.271658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.055264+0000","last_change":"2023-09-10T01:00:48.066906+0000","last_active":"2023-09-10T01:01:57.055264+0000","last_peered":"2023-09-10T01:01:57.055264+0000","last_clean":"2023-09-10T01:01:57.055264+0000","last_became_active":"2023-09-10T01:00:48.066474+0000","last_became_peered":"2023-09-10T01:00:48.066474+0000","last_unstale":"2023-09-10T01:01:57.055264+0000","last_undegraded":"2023-09-10T01:01:57.055264+0000","last_fullsized":"2023-09-10T01:01:57.055264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:48:26.827857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036365+0000","last_change":"2023-09-10T01:00:43.005922+0000","last_active":"2023-09-10T01:01:56.036365+0000","last_peered":"2023-09-10T01:01:56.036365+0000","last_clean":"2023-09-10T01:01:56.036365+0000","last_became_active":"2023-09-10T01:00:43.005517+0000","last_became_peered":"2023-09-10T01:00:43.005517+0000","last_unstale":"2023-09-10T01:01:56.036365+0000","last_undegraded":"2023-09-10T01:01:56.036365+0000","last_fullsized":"2023-09-10T01:01:56.036365+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:32:12.619034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889259+0000","last_change":"2023-09-10T01:00:44.036766+0000","last_active":"2023-09-10T01:02:25.889259+0000","last_peered":"2023-09-10T01:02:25.889259+0000","last_clean":"2023-09-10T01:02:25.889259+0000","last_became_active":"2023-09-10T01:00:44.036316+0000","last_became_peered":"2023-09-10T01:00:44.036316+0000","last_unstale":"2023-09-10T01:02:25.889259+0000","last_undegraded":"2023-09-10T01:02:25.889259+0000","last_fullsized":"2023-09-10T01:02:25.889259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:17:51.875838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036026+0000","last_change":"2023-09-10T01:00:57.051716+0000","last_active":"2023-09-10T01:01:56.036026+0000","last_peered":"2023-09-10T01:01:56.036026+0000","last_clean":"2023-09-10T01:01:56.036026+0000","last_became_active":"2023-09-10T01:00:57.051334+0000","last_became_peered":"2023-09-10T01:00:57.051334+0000","last_unstale":"2023-09-10T01:01:56.036026+0000","last_undegraded":"2023-09-10T01:01:56.036026+0000","last_fullsized":"2023-09-10T01:01:56.036026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:30:52.598639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":133,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:56.109837+0000","last_change":"2023-09-10T01:00:43.008695+0000","last_active":"2023-09-10T01:02:56.109837+0000","last_peered":"2023-09-10T01:02:56.109837+0000","last_clean":"2023-09-10T01:02:56.109837+0000","last_became_active":"2023-09-10T01:00:43.008278+0000","last_became_peered":"2023-09-10T01:00:43.008278+0000","last_unstale":"2023-09-10T01:02:56.109837+0000","last_undegraded":"2023-09-10T01:02:56.109837+0000","last_fullsized":"2023-09-10T01:02:56.109837+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:46:38.370582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882691+0000","last_change":"2023-09-10T01:00:44.030540+0000","last_active":"2023-09-10T01:02:25.882691+0000","last_peered":"2023-09-10T01:02:25.882691+0000","last_clean":"2023-09-10T01:02:25.882691+0000","last_became_active":"2023-09-10T01:00:44.030212+0000","last_became_peered":"2023-09-10T01:00:44.030212+0000","last_unstale":"2023-09-10T01:02:25.882691+0000","last_undegraded":"2023-09-10T01:02:25.882691+0000","last_fullsized":"2023-09-10T01:02:25.882691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:57:06.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036424+0000","last_change":"2023-09-10T01:00:57.052197+0000","last_active":"2023-09-10T01:01:56.036424+0000","last_peered":"2023-09-10T01:01:56.036424+0000","last_clean":"2023-09-10T01:01:56.036424+0000","last_became_active":"2023-09-10T01:00:57.051782+0000","last_became_peered":"2023-09-10T01:00:57.051782+0000","last_unstale":"2023-09-10T01:01:56.036424+0000","last_undegraded":"2023-09-10T01:01:56.036424+0000","last_fullsized":"2023-09-10T01:01:56.036424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:36:49.680123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042966+0000","last_change":"2023-09-10T01:00:43.007242+0000","last_active":"2023-09-10T01:01:56.042966+0000","last_peered":"2023-09-10T01:01:56.042966+0000","last_clean":"2023-09-10T01:01:56.042966+0000","last_became_active":"2023-09-10T01:00:43.006912+0000","last_became_peered":"2023-09-10T01:00:43.006912+0000","last_unstale":"2023-09-10T01:01:56.042966+0000","last_undegraded":"2023-09-10T01:01:56.042966+0000","last_fullsized":"2023-09-10T01:01:56.042966+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:48:22.783656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.884684+0000","last_change":"2023-09-10T01:01:58.026024+0000","last_active":"2023-09-10T01:02:25.884684+0000","last_peered":"2023-09-10T01:02:25.884684+0000","last_clean":"2023-09-10T01:02:25.884684+0000","last_became_active":"2023-09-10T01:01:58.025647+0000","last_became_peered":"2023-09-10T01:01:58.025647+0000","last_unstale":"2023-09-10T01:02:25.884684+0000","last_undegraded":"2023-09-10T01:02:25.884684+0000","last_fullsized":"2023-09-10T01:02:25.884684+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:58:11.916233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037579+0000","last_change":"2023-09-10T01:00:48.062520+0000","last_active":"2023-09-10T01:01:56.037579+0000","last_peered":"2023-09-10T01:01:56.037579+0000","last_clean":"2023-09-10T01:01:56.037579+0000","last_became_active":"2023-09-10T01:00:48.061870+0000","last_became_peered":"2023-09-10T01:00:48.061870+0000","last_unstale":"2023-09-10T01:01:56.037579+0000","last_undegraded":"2023-09-10T01:01:56.037579+0000","last_fullsized":"2023-09-10T01:01:56.037579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:24:38.798775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037030+0000","last_change":"2023-09-10T01:00:43.006513+0000","last_active":"2023-09-10T01:01:56.037030+0000","last_peered":"2023-09-10T01:01:56.037030+0000","last_clean":"2023-09-10T01:01:56.037030+0000","last_became_active":"2023-09-10T01:00:43.006166+0000","last_became_peered":"2023-09-10T01:00:43.006166+0000","last_unstale":"2023-09-10T01:01:56.037030+0000","last_undegraded":"2023-09-10T01:01:56.037030+0000","last_fullsized":"2023-09-10T01:01:56.037030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:52:32.167513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880315+0000","last_change":"2023-09-10T01:00:44.031264+0000","last_active":"2023-09-10T01:02:25.880315+0000","last_peered":"2023-09-10T01:02:25.880315+0000","last_clean":"2023-09-10T01:02:25.880315+0000","last_became_active":"2023-09-10T01:00:44.030993+0000","last_became_peered":"2023-09-10T01:00:44.030993+0000","last_unstale":"2023-09-10T01:02:25.880315+0000","last_undegraded":"2023-09-10T01:02:25.880315+0000","last_fullsized":"2023-09-10T01:02:25.880315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:19:27.658594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032496+0000","last_change":"2023-09-10T01:00:48.056886+0000","last_active":"2023-09-10T01:01:56.032496+0000","last_peered":"2023-09-10T01:01:56.032496+0000","last_clean":"2023-09-10T01:01:56.032496+0000","last_became_active":"2023-09-10T01:00:48.056411+0000","last_became_peered":"2023-09-10T01:00:48.056411+0000","last_unstale":"2023-09-10T01:01:56.032496+0000","last_undegraded":"2023-09-10T01:01:56.032496+0000","last_fullsized":"2023-09-10T01:01:56.032496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:56:42.062702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":283,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:57.111487+0000","last_change":"2023-09-10T01:00:43.009113+0000","last_active":"2023-09-10T01:02:57.111487+0000","last_peered":"2023-09-10T01:02:57.111487+0000","last_clean":"2023-09-10T01:02:57.111487+0000","last_became_active":"2023-09-10T01:00:43.008757+0000","last_became_peered":"2023-09-10T01:00:43.008757+0000","last_unstale":"2023-09-10T01:02:57.111487+0000","last_undegraded":"2023-09-10T01:02:57.111487+0000","last_fullsized":"2023-09-10T01:02:57.111487+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:29:24.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64774,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":226,"num_read_kb":177,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.870186+0000","last_change":"2023-09-10T01:00:44.027127+0000","last_active":"2023-09-10T01:02:25.870186+0000","last_peered":"2023-09-10T01:02:25.870186+0000","last_clean":"2023-09-10T01:02:25.870186+0000","last_became_active":"2023-09-10T01:00:44.026852+0000","last_became_peered":"2023-09-10T01:00:44.026852+0000","last_unstale":"2023-09-10T01:02:25.870186+0000","last_undegraded":"2023-09-10T01:02:25.870186+0000","last_fullsized":"2023-09-10T01:02:25.870186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:14:20.805300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037605+0000","last_change":"2023-09-10T01:00:48.069337+0000","last_active":"2023-09-10T01:01:56.037605+0000","last_peered":"2023-09-10T01:01:56.037605+0000","last_clean":"2023-09-10T01:01:56.037605+0000","last_became_active":"2023-09-10T01:00:48.068893+0000","last_became_peered":"2023-09-10T01:00:48.068893+0000","last_unstale":"2023-09-10T01:01:56.037605+0000","last_undegraded":"2023-09-10T01:01:56.037605+0000","last_fullsized":"2023-09-10T01:01:56.037605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:24:40.074551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033160+0000","last_change":"2023-09-10T01:00:43.010128+0000","last_active":"2023-09-10T01:01:56.033160+0000","last_peered":"2023-09-10T01:01:56.033160+0000","last_clean":"2023-09-10T01:01:56.033160+0000","last_became_active":"2023-09-10T01:00:43.009808+0000","last_became_peered":"2023-09-10T01:00:43.009808+0000","last_unstale":"2023-09-10T01:01:56.033160+0000","last_undegraded":"2023-09-10T01:01:56.033160+0000","last_fullsized":"2023-09-10T01:01:56.033160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:27.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891792+0000","last_change":"2023-09-10T01:00:44.038665+0000","last_active":"2023-09-10T01:02:25.891792+0000","last_peered":"2023-09-10T01:02:25.891792+0000","last_clean":"2023-09-10T01:02:25.891792+0000","last_became_active":"2023-09-10T01:00:44.037944+0000","last_became_peered":"2023-09-10T01:00:44.037944+0000","last_unstale":"2023-09-10T01:02:25.891792+0000","last_undegraded":"2023-09-10T01:02:25.891792+0000","last_fullsized":"2023-09-10T01:02:25.891792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:11:34.018249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127649+0000","last_change":"2023-09-10T01:00:56.965594+0000","last_active":"2023-09-10T01:01:56.127649+0000","last_peered":"2023-09-10T01:01:56.127649+0000","last_clean":"2023-09-10T01:01:56.127649+0000","last_became_active":"2023-09-10T01:00:56.965006+0000","last_became_peered":"2023-09-10T01:00:56.965006+0000","last_unstale":"2023-09-10T01:01:56.127649+0000","last_undegraded":"2023-09-10T01:01:56.127649+0000","last_fullsized":"2023-09-10T01:01:56.127649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:13:16.946697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036734+0000","last_change":"2023-09-10T01:00:43.004855+0000","last_active":"2023-09-10T01:01:56.036734+0000","last_peered":"2023-09-10T01:01:56.036734+0000","last_clean":"2023-09-10T01:01:56.036734+0000","last_became_active":"2023-09-10T01:00:43.004464+0000","last_became_peered":"2023-09-10T01:00:43.004464+0000","last_unstale":"2023-09-10T01:01:56.036734+0000","last_undegraded":"2023-09-10T01:01:56.036734+0000","last_fullsized":"2023-09-10T01:01:56.036734+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:29.517290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889936+0000","last_change":"2023-09-10T01:00:44.037273+0000","last_active":"2023-09-10T01:02:25.889936+0000","last_peered":"2023-09-10T01:02:25.889936+0000","last_clean":"2023-09-10T01:02:25.889936+0000","last_became_active":"2023-09-10T01:00:44.036965+0000","last_became_peered":"2023-09-10T01:00:44.036965+0000","last_unstale":"2023-09-10T01:02:25.889936+0000","last_undegraded":"2023-09-10T01:02:25.889936+0000","last_fullsized":"2023-09-10T01:02:25.889936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:47:20.149002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033203+0000","last_change":"2023-09-10T01:00:48.058574+0000","last_active":"2023-09-10T01:01:56.033203+0000","last_peered":"2023-09-10T01:01:56.033203+0000","last_clean":"2023-09-10T01:01:56.033203+0000","last_became_active":"2023-09-10T01:00:48.058190+0000","last_became_peered":"2023-09-10T01:00:48.058190+0000","last_unstale":"2023-09-10T01:01:56.033203+0000","last_undegraded":"2023-09-10T01:01:56.033203+0000","last_fullsized":"2023-09-10T01:01:56.033203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:46:42.177996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":134,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:55.146304+0000","last_change":"2023-09-10T01:00:43.003816+0000","last_active":"2023-09-10T01:02:55.146304+0000","last_peered":"2023-09-10T01:02:55.146304+0000","last_clean":"2023-09-10T01:02:55.146304+0000","last_became_active":"2023-09-10T01:00:43.003404+0000","last_became_peered":"2023-09-10T01:00:43.003404+0000","last_unstale":"2023-09-10T01:02:55.146304+0000","last_undegraded":"2023-09-10T01:02:55.146304+0000","last_fullsized":"2023-09-10T01:02:55.146304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:55.208302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878740+0000","last_change":"2023-09-10T01:00:44.034607+0000","last_active":"2023-09-10T01:02:25.878740+0000","last_peered":"2023-09-10T01:02:25.878740+0000","last_clean":"2023-09-10T01:02:25.878740+0000","last_became_active":"2023-09-10T01:00:44.034244+0000","last_became_peered":"2023-09-10T01:00:44.034244+0000","last_unstale":"2023-09-10T01:02:25.878740+0000","last_undegraded":"2023-09-10T01:02:25.878740+0000","last_fullsized":"2023-09-10T01:02:25.878740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:06:13.214160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031918+0000","last_change":"2023-09-10T01:00:48.058789+0000","last_active":"2023-09-10T01:01:56.031918+0000","last_peered":"2023-09-10T01:01:56.031918+0000","last_clean":"2023-09-10T01:01:56.031918+0000","last_became_active":"2023-09-10T01:00:48.058220+0000","last_became_peered":"2023-09-10T01:00:48.058220+0000","last_unstale":"2023-09-10T01:01:56.031918+0000","last_undegraded":"2023-09-10T01:01:56.031918+0000","last_fullsized":"2023-09-10T01:01:56.031918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:01:12.291739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036527+0000","last_change":"2023-09-10T01:00:43.006109+0000","last_active":"2023-09-10T01:01:56.036527+0000","last_peered":"2023-09-10T01:01:56.036527+0000","last_clean":"2023-09-10T01:01:56.036527+0000","last_became_active":"2023-09-10T01:00:43.005753+0000","last_became_peered":"2023-09-10T01:00:43.005753+0000","last_unstale":"2023-09-10T01:01:56.036527+0000","last_undegraded":"2023-09-10T01:01:56.036527+0000","last_fullsized":"2023-09-10T01:01:56.036527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:45:47.083991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.881751+0000","last_change":"2023-09-10T01:01:57.139294+0000","last_active":"2023-09-10T01:02:25.881751+0000","last_peered":"2023-09-10T01:02:25.881751+0000","last_clean":"2023-09-10T01:02:25.881751+0000","last_became_active":"2023-09-10T01:01:57.138930+0000","last_became_peered":"2023-09-10T01:01:57.138930+0000","last_unstale":"2023-09-10T01:02:25.881751+0000","last_undegraded":"2023-09-10T01:02:25.881751+0000","last_fullsized":"2023-09-10T01:02:25.881751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:21.143499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.043801+0000","last_change":"2023-09-10T01:00:48.069318+0000","last_active":"2023-09-10T01:01:57.043801+0000","last_peered":"2023-09-10T01:01:57.043801+0000","last_clean":"2023-09-10T01:01:57.043801+0000","last_became_active":"2023-09-10T01:00:48.068876+0000","last_became_peered":"2023-09-10T01:00:48.068876+0000","last_unstale":"2023-09-10T01:01:57.043801+0000","last_undegraded":"2023-09-10T01:01:57.043801+0000","last_fullsized":"2023-09-10T01:01:57.043801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:06:22.366530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036038+0000","last_change":"2023-09-10T01:00:43.004164+0000","last_active":"2023-09-10T01:01:56.036038+0000","last_peered":"2023-09-10T01:01:56.036038+0000","last_clean":"2023-09-10T01:01:56.036038+0000","last_became_active":"2023-09-10T01:00:43.003781+0000","last_became_peered":"2023-09-10T01:00:43.003781+0000","last_unstale":"2023-09-10T01:01:56.036038+0000","last_undegraded":"2023-09-10T01:01:56.036038+0000","last_fullsized":"2023-09-10T01:01:56.036038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:38:18.559712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872316+0000","last_change":"2023-09-10T01:00:44.028206+0000","last_active":"2023-09-10T01:02:25.872316+0000","last_peered":"2023-09-10T01:02:25.872316+0000","last_clean":"2023-09-10T01:02:25.872316+0000","last_became_active":"2023-09-10T01:00:44.027914+0000","last_became_peered":"2023-09-10T01:00:44.027914+0000","last_unstale":"2023-09-10T01:02:25.872316+0000","last_undegraded":"2023-09-10T01:02:25.872316+0000","last_fullsized":"2023-09-10T01:02:25.872316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:41:12.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.106476+0000","last_change":"2023-09-10T01:00:48.428613+0000","last_active":"2023-09-10T01:01:57.106476+0000","last_peered":"2023-09-10T01:01:57.106476+0000","last_clean":"2023-09-10T01:01:57.106476+0000","last_became_active":"2023-09-10T01:00:48.428059+0000","last_became_peered":"2023-09-10T01:00:48.428059+0000","last_unstale":"2023-09-10T01:01:57.106476+0000","last_undegraded":"2023-09-10T01:01:57.106476+0000","last_fullsized":"2023-09-10T01:01:57.106476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:46:03.683279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.046289+0000","last_change":"2023-09-10T01:00:43.006061+0000","last_active":"2023-09-10T01:01:57.046289+0000","last_peered":"2023-09-10T01:01:57.046289+0000","last_clean":"2023-09-10T01:01:57.046289+0000","last_became_active":"2023-09-10T01:00:43.005392+0000","last_became_peered":"2023-09-10T01:00:43.005392+0000","last_unstale":"2023-09-10T01:01:57.046289+0000","last_undegraded":"2023-09-10T01:01:57.046289+0000","last_fullsized":"2023-09-10T01:01:57.046289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:01:45.062338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891410+0000","last_change":"2023-09-10T01:00:44.026684+0000","last_active":"2023-09-10T01:02:25.891410+0000","last_peered":"2023-09-10T01:02:25.891410+0000","last_clean":"2023-09-10T01:02:25.891410+0000","last_became_active":"2023-09-10T01:00:44.026387+0000","last_became_peered":"2023-09-10T01:00:44.026387+0000","last_unstale":"2023-09-10T01:02:25.891410+0000","last_undegraded":"2023-09-10T01:02:25.891410+0000","last_fullsized":"2023-09-10T01:02:25.891410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:23:03.662543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.041259+0000","last_change":"2023-09-10T01:00:48.071589+0000","last_active":"2023-09-10T01:01:57.041259+0000","last_peered":"2023-09-10T01:01:57.041259+0000","last_clean":"2023-09-10T01:01:57.041259+0000","last_became_active":"2023-09-10T01:00:48.071118+0000","last_became_peered":"2023-09-10T01:00:48.071118+0000","last_unstale":"2023-09-10T01:01:57.041259+0000","last_undegraded":"2023-09-10T01:01:57.041259+0000","last_fullsized":"2023-09-10T01:01:57.041259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:27.194868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024640+0000","last_change":"2023-09-10T01:00:43.009738+0000","last_active":"2023-09-10T01:01:58.024640+0000","last_peered":"2023-09-10T01:01:58.024640+0000","last_clean":"2023-09-10T01:01:58.024640+0000","last_became_active":"2023-09-10T01:00:43.009376+0000","last_became_peered":"2023-09-10T01:00:43.009376+0000","last_unstale":"2023-09-10T01:01:58.024640+0000","last_undegraded":"2023-09-10T01:01:58.024640+0000","last_fullsized":"2023-09-10T01:01:58.024640+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:08.761672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882436+0000","last_change":"2023-09-10T01:00:44.030175+0000","last_active":"2023-09-10T01:02:25.882436+0000","last_peered":"2023-09-10T01:02:25.882436+0000","last_clean":"2023-09-10T01:02:25.882436+0000","last_became_active":"2023-09-10T01:00:44.029748+0000","last_became_peered":"2023-09-10T01:00:44.029748+0000","last_unstale":"2023-09-10T01:02:25.882436+0000","last_undegraded":"2023-09-10T01:02:25.882436+0000","last_fullsized":"2023-09-10T01:02:25.882436+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:51:15.149610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032036+0000","last_change":"2023-09-10T01:00:48.428008+0000","last_active":"2023-09-10T01:01:56.032036+0000","last_peered":"2023-09-10T01:01:56.032036+0000","last_clean":"2023-09-10T01:01:56.032036+0000","last_became_active":"2023-09-10T01:00:48.427498+0000","last_became_peered":"2023-09-10T01:00:48.427498+0000","last_unstale":"2023-09-10T01:01:56.032036+0000","last_undegraded":"2023-09-10T01:01:56.032036+0000","last_fullsized":"2023-09-10T01:01:56.032036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:16:48.938006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872721+0000","last_change":"2023-09-10T01:01:57.107918+0000","last_active":"2023-09-10T01:02:25.872721+0000","last_peered":"2023-09-10T01:02:25.872721+0000","last_clean":"2023-09-10T01:02:25.872721+0000","last_became_active":"2023-09-10T01:01:57.107413+0000","last_became_peered":"2023-09-10T01:01:57.107413+0000","last_unstale":"2023-09-10T01:02:25.872721+0000","last_undegraded":"2023-09-10T01:02:25.872721+0000","last_fullsized":"2023-09-10T01:02:25.872721+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:16:33.836871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":134,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:55.145098+0000","last_change":"2023-09-10T01:00:43.001047+0000","last_active":"2023-09-10T01:02:55.145098+0000","last_peered":"2023-09-10T01:02:55.145098+0000","last_clean":"2023-09-10T01:02:55.145098+0000","last_became_active":"2023-09-10T01:00:43.000521+0000","last_became_peered":"2023-09-10T01:00:43.000521+0000","last_unstale":"2023-09-10T01:02:55.145098+0000","last_undegraded":"2023-09-10T01:02:55.145098+0000","last_fullsized":"2023-09-10T01:02:55.145098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:15:34.509964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024504+0000","last_change":"2023-09-10T01:00:48.062148+0000","last_active":"2023-09-10T01:01:58.024504+0000","last_peered":"2023-09-10T01:01:58.024504+0000","last_clean":"2023-09-10T01:01:58.024504+0000","last_became_active":"2023-09-10T01:00:48.061746+0000","last_became_peered":"2023-09-10T01:00:48.061746+0000","last_unstale":"2023-09-10T01:01:58.024504+0000","last_undegraded":"2023-09-10T01:01:58.024504+0000","last_fullsized":"2023-09-10T01:01:58.024504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:40.087012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889845+0000","last_change":"2023-09-10T01:00:44.030500+0000","last_active":"2023-09-10T01:02:25.889845+0000","last_peered":"2023-09-10T01:02:25.889845+0000","last_clean":"2023-09-10T01:02:25.889845+0000","last_became_active":"2023-09-10T01:00:44.030209+0000","last_became_peered":"2023-09-10T01:00:44.030209+0000","last_unstale":"2023-09-10T01:02:25.889845+0000","last_undegraded":"2023-09-10T01:02:25.889845+0000","last_fullsized":"2023-09-10T01:02:25.889845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:53:30.421493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.040869+0000","last_change":"2023-09-10T01:00:43.001102+0000","last_active":"2023-09-10T01:01:57.040869+0000","last_peered":"2023-09-10T01:01:57.040869+0000","last_clean":"2023-09-10T01:01:57.040869+0000","last_became_active":"2023-09-10T01:00:43.000715+0000","last_became_peered":"2023-09-10T01:00:43.000715+0000","last_unstale":"2023-09-10T01:01:57.040869+0000","last_undegraded":"2023-09-10T01:01:57.040869+0000","last_fullsized":"2023-09-10T01:01:57.040869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:43:37.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":327,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024171+0000","last_change":"2023-09-10T00:59:40.261244+0000","last_active":"2023-09-10T01:01:58.024171+0000","last_peered":"2023-09-10T01:01:58.024171+0000","last_clean":"2023-09-10T01:01:58.024171+0000","last_became_active":"2023-09-10T00:59:40.260673+0000","last_became_peered":"2023-09-10T00:59:40.260673+0000","last_unstale":"2023-09-10T01:01:58.024171+0000","last_undegraded":"2023-09-10T01:01:58.024171+0000","last_fullsized":"2023-09-10T01:01:58.024171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_clean_scrub_stamp":"2023-09-10T00:56:36.374323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:04:03.614104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.044220+0000","last_change":"2023-09-10T01:00:57.052019+0000","last_active":"2023-09-10T01:01:57.044220+0000","last_peered":"2023-09-10T01:01:57.044220+0000","last_clean":"2023-09-10T01:01:57.044220+0000","last_became_active":"2023-09-10T01:00:57.051527+0000","last_became_peered":"2023-09-10T01:00:57.051527+0000","last_unstale":"2023-09-10T01:01:57.044220+0000","last_undegraded":"2023-09-10T01:01:57.044220+0000","last_fullsized":"2023-09-10T01:01:57.044220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:10:43.756761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889771+0000","last_change":"2023-09-10T01:00:44.033214+0000","last_active":"2023-09-10T01:02:25.889771+0000","last_peered":"2023-09-10T01:02:25.889771+0000","last_clean":"2023-09-10T01:02:25.889771+0000","last_became_active":"2023-09-10T01:00:44.032828+0000","last_became_peered":"2023-09-10T01:00:44.032828+0000","last_unstale":"2023-09-10T01:02:25.889771+0000","last_undegraded":"2023-09-10T01:02:25.889771+0000","last_fullsized":"2023-09-10T01:02:25.889771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:01:22.417216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024928+0000","last_change":"2023-09-10T01:00:56.965456+0000","last_active":"2023-09-10T01:01:58.024928+0000","last_peered":"2023-09-10T01:01:58.024928+0000","last_clean":"2023-09-10T01:01:58.024928+0000","last_became_active":"2023-09-10T01:00:56.965021+0000","last_became_peered":"2023-09-10T01:00:56.965021+0000","last_unstale":"2023-09-10T01:01:58.024928+0000","last_undegraded":"2023-09-10T01:01:58.024928+0000","last_fullsized":"2023-09-10T01:01:58.024928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:19.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.731761+0000","last_change":"2023-09-10T01:00:48.066811+0000","last_active":"2023-09-10T01:01:58.731761+0000","last_peered":"2023-09-10T01:01:58.731761+0000","last_clean":"2023-09-10T01:01:58.731761+0000","last_became_active":"2023-09-10T01:00:48.066421+0000","last_became_peered":"2023-09-10T01:00:48.066421+0000","last_unstale":"2023-09-10T01:01:58.731761+0000","last_undegraded":"2023-09-10T01:01:58.731761+0000","last_fullsized":"2023-09-10T01:01:58.731761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:48:01.459070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.106555+0000","last_change":"2023-09-10T01:00:43.011134+0000","last_active":"2023-09-10T01:01:57.106555+0000","last_peered":"2023-09-10T01:01:57.106555+0000","last_clean":"2023-09-10T01:01:57.106555+0000","last_became_active":"2023-09-10T01:00:43.010871+0000","last_became_peered":"2023-09-10T01:00:43.010871+0000","last_unstale":"2023-09-10T01:01:57.106555+0000","last_undegraded":"2023-09-10T01:01:57.106555+0000","last_fullsized":"2023-09-10T01:01:57.106555+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:56.903745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890244+0000","last_change":"2023-09-10T01:00:44.031104+0000","last_active":"2023-09-10T01:02:25.890244+0000","last_peered":"2023-09-10T01:02:25.890244+0000","last_clean":"2023-09-10T01:02:25.890244+0000","last_became_active":"2023-09-10T01:00:44.030745+0000","last_became_peered":"2023-09-10T01:00:44.030745+0000","last_unstale":"2023-09-10T01:02:25.890244+0000","last_undegraded":"2023-09-10T01:02:25.890244+0000","last_fullsized":"2023-09-10T01:02:25.890244+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:53.885886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.470456+0000","last_change":"2023-09-10T01:00:48.427735+0000","last_active":"2023-09-10T01:01:57.470456+0000","last_peered":"2023-09-10T01:01:57.470456+0000","last_clean":"2023-09-10T01:01:57.470456+0000","last_became_active":"2023-09-10T01:00:48.427342+0000","last_became_peered":"2023-09-10T01:00:48.427342+0000","last_unstale":"2023-09-10T01:01:57.470456+0000","last_undegraded":"2023-09-10T01:01:57.470456+0000","last_fullsized":"2023-09-10T01:01:57.470456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:01:52.007325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:53.813562+0000","last_change":"2023-09-10T01:00:43.006314+0000","last_active":"2023-09-10T01:02:53.813562+0000","last_peered":"2023-09-10T01:02:53.813562+0000","last_clean":"2023-09-10T01:02:53.813562+0000","last_became_active":"2023-09-10T01:00:43.004835+0000","last_became_peered":"2023-09-10T01:00:43.004835+0000","last_unstale":"2023-09-10T01:02:53.813562+0000","last_undegraded":"2023-09-10T01:02:53.813562+0000","last_fullsized":"2023-09-10T01:02:53.813562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:37:09.359846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891331+0000","last_change":"2023-09-10T01:00:44.029839+0000","last_active":"2023-09-10T01:02:25.891331+0000","last_peered":"2023-09-10T01:02:25.891331+0000","last_clean":"2023-09-10T01:02:25.891331+0000","last_became_active":"2023-09-10T01:00:44.029492+0000","last_became_peered":"2023-09-10T01:00:44.029492+0000","last_unstale":"2023-09-10T01:02:25.891331+0000","last_undegraded":"2023-09-10T01:02:25.891331+0000","last_fullsized":"2023-09-10T01:02:25.891331+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:14.357386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.047012+0000","last_change":"2023-09-10T01:00:48.054536+0000","last_active":"2023-09-10T01:01:57.047012+0000","last_peered":"2023-09-10T01:01:57.047012+0000","last_clean":"2023-09-10T01:01:57.047012+0000","last_became_active":"2023-09-10T01:00:48.053938+0000","last_became_peered":"2023-09-10T01:00:48.053938+0000","last_unstale":"2023-09-10T01:01:57.047012+0000","last_undegraded":"2023-09-10T01:01:57.047012+0000","last_fullsized":"2023-09-10T01:01:57.047012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:45:12.703592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.041448+0000","last_change":"2023-09-10T01:00:43.002313+0000","last_active":"2023-09-10T01:01:57.041448+0000","last_peered":"2023-09-10T01:01:57.041448+0000","last_clean":"2023-09-10T01:01:57.041448+0000","last_became_active":"2023-09-10T01:00:43.001839+0000","last_became_peered":"2023-09-10T01:00:43.001839+0000","last_unstale":"2023-09-10T01:01:57.041448+0000","last_undegraded":"2023-09-10T01:01:57.041448+0000","last_fullsized":"2023-09-10T01:01:57.041448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:51.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880327+0000","last_change":"2023-09-10T01:01:57.108269+0000","last_active":"2023-09-10T01:02:25.880327+0000","last_peered":"2023-09-10T01:02:25.880327+0000","last_clean":"2023-09-10T01:02:25.880327+0000","last_became_active":"2023-09-10T01:01:57.107879+0000","last_became_peered":"2023-09-10T01:01:57.107879+0000","last_unstale":"2023-09-10T01:02:25.880327+0000","last_undegraded":"2023-09-10T01:02:25.880327+0000","last_fullsized":"2023-09-10T01:02:25.880327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:47:09.135978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037126+0000","last_change":"2023-09-10T01:00:48.428406+0000","last_active":"2023-09-10T01:01:56.037126+0000","last_peered":"2023-09-10T01:01:56.037126+0000","last_clean":"2023-09-10T01:01:56.037126+0000","last_became_active":"2023-09-10T01:00:48.427938+0000","last_became_peered":"2023-09-10T01:00:48.427938+0000","last_unstale":"2023-09-10T01:01:56.037126+0000","last_undegraded":"2023-09-10T01:01:56.037126+0000","last_fullsized":"2023-09-10T01:01:56.037126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:52.316235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.138071+0000","last_change":"2023-09-10T01:00:56.963755+0000","last_active":"2023-09-10T01:01:57.138071+0000","last_peered":"2023-09-10T01:01:57.138071+0000","last_clean":"2023-09-10T01:01:57.138071+0000","last_became_active":"2023-09-10T01:00:56.963283+0000","last_became_peered":"2023-09-10T01:00:56.963283+0000","last_unstale":"2023-09-10T01:01:57.138071+0000","last_undegraded":"2023-09-10T01:01:57.138071+0000","last_fullsized":"2023-09-10T01:01:57.138071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:13:32.213393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.864057+0000","last_change":"2023-09-10T01:00:44.025284+0000","last_active":"2023-09-10T01:02:25.864057+0000","last_peered":"2023-09-10T01:02:25.864057+0000","last_clean":"2023-09-10T01:02:25.864057+0000","last_became_active":"2023-09-10T01:00:44.024965+0000","last_became_peered":"2023-09-10T01:00:44.024965+0000","last_unstale":"2023-09-10T01:02:25.864057+0000","last_undegraded":"2023-09-10T01:02:25.864057+0000","last_fullsized":"2023-09-10T01:02:25.864057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:19:08.620325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033359+0000","last_change":"2023-09-10T01:00:48.070843+0000","last_active":"2023-09-10T01:01:56.033359+0000","last_peered":"2023-09-10T01:01:56.033359+0000","last_clean":"2023-09-10T01:01:56.033359+0000","last_became_active":"2023-09-10T01:00:48.070350+0000","last_became_peered":"2023-09-10T01:00:48.070350+0000","last_unstale":"2023-09-10T01:01:56.033359+0000","last_undegraded":"2023-09-10T01:01:56.033359+0000","last_fullsized":"2023-09-10T01:01:56.033359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:35:25.196036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.470530+0000","last_change":"2023-09-10T01:00:43.005398+0000","last_active":"2023-09-10T01:01:57.470530+0000","last_peered":"2023-09-10T01:01:57.470530+0000","last_clean":"2023-09-10T01:01:57.470530+0000","last_became_active":"2023-09-10T01:00:43.005087+0000","last_became_peered":"2023-09-10T01:00:43.005087+0000","last_unstale":"2023-09-10T01:01:57.470530+0000","last_undegraded":"2023-09-10T01:01:57.470530+0000","last_fullsized":"2023-09-10T01:01:57.470530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:29:04.042683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871590+0000","last_change":"2023-09-10T01:00:44.024451+0000","last_active":"2023-09-10T01:02:25.871590+0000","last_peered":"2023-09-10T01:02:25.871590+0000","last_clean":"2023-09-10T01:02:25.871590+0000","last_became_active":"2023-09-10T01:00:44.024045+0000","last_became_peered":"2023-09-10T01:00:44.024045+0000","last_unstale":"2023-09-10T01:02:25.871590+0000","last_undegraded":"2023-09-10T01:02:25.871590+0000","last_fullsized":"2023-09-10T01:02:25.871590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:55:50.750994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037240+0000","last_change":"2023-09-10T01:00:48.062247+0000","last_active":"2023-09-10T01:01:56.037240+0000","last_peered":"2023-09-10T01:01:56.037240+0000","last_clean":"2023-09-10T01:01:56.037240+0000","last_became_active":"2023-09-10T01:00:48.061812+0000","last_became_peered":"2023-09-10T01:00:48.061812+0000","last_unstale":"2023-09-10T01:01:56.037240+0000","last_undegraded":"2023-09-10T01:01:56.037240+0000","last_fullsized":"2023-09-10T01:01:56.037240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:44:13.596534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.047127+0000","last_change":"2023-09-10T01:00:43.010845+0000","last_active":"2023-09-10T01:01:57.047127+0000","last_peered":"2023-09-10T01:01:57.047127+0000","last_clean":"2023-09-10T01:01:57.047127+0000","last_became_active":"2023-09-10T01:00:43.010511+0000","last_became_peered":"2023-09-10T01:00:43.010511+0000","last_unstale":"2023-09-10T01:01:57.047127+0000","last_undegraded":"2023-09-10T01:01:57.047127+0000","last_fullsized":"2023-09-10T01:01:57.047127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:33:07.229023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891545+0000","last_change":"2023-09-10T01:00:44.029025+0000","last_active":"2023-09-10T01:02:25.891545+0000","last_peered":"2023-09-10T01:02:25.891545+0000","last_clean":"2023-09-10T01:02:25.891545+0000","last_became_active":"2023-09-10T01:00:44.028713+0000","last_became_peered":"2023-09-10T01:00:44.028713+0000","last_unstale":"2023-09-10T01:02:25.891545+0000","last_undegraded":"2023-09-10T01:02:25.891545+0000","last_fullsized":"2023-09-10T01:02:25.891545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:40:22.315742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037817+0000","last_change":"2023-09-10T01:00:48.427294+0000","last_active":"2023-09-10T01:01:56.037817+0000","last_peered":"2023-09-10T01:01:56.037817+0000","last_clean":"2023-09-10T01:01:56.037817+0000","last_became_active":"2023-09-10T01:00:48.426882+0000","last_became_peered":"2023-09-10T01:00:48.426882+0000","last_unstale":"2023-09-10T01:01:56.037817+0000","last_undegraded":"2023-09-10T01:01:56.037817+0000","last_fullsized":"2023-09-10T01:01:56.037817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:54:38.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035921+0000","last_change":"2023-09-10T01:00:43.006175+0000","last_active":"2023-09-10T01:01:56.035921+0000","last_peered":"2023-09-10T01:01:56.035921+0000","last_clean":"2023-09-10T01:01:56.035921+0000","last_became_active":"2023-09-10T01:00:43.005786+0000","last_became_peered":"2023-09-10T01:00:43.005786+0000","last_unstale":"2023-09-10T01:01:56.035921+0000","last_undegraded":"2023-09-10T01:01:56.035921+0000","last_fullsized":"2023-09-10T01:01:56.035921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:14:47.074866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887993+0000","last_change":"2023-09-10T01:00:44.041303+0000","last_active":"2023-09-10T01:02:25.887993+0000","last_peered":"2023-09-10T01:02:25.887993+0000","last_clean":"2023-09-10T01:02:25.887993+0000","last_became_active":"2023-09-10T01:00:44.041009+0000","last_became_peered":"2023-09-10T01:00:44.041009+0000","last_unstale":"2023-09-10T01:02:25.887993+0000","last_undegraded":"2023-09-10T01:02:25.887993+0000","last_fullsized":"2023-09-10T01:02:25.887993+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:53:23.001700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036842+0000","last_change":"2023-09-10T01:00:48.064314+0000","last_active":"2023-09-10T01:01:56.036842+0000","last_peered":"2023-09-10T01:01:56.036842+0000","last_clean":"2023-09-10T01:01:56.036842+0000","last_became_active":"2023-09-10T01:00:48.063813+0000","last_became_peered":"2023-09-10T01:00:48.063813+0000","last_unstale":"2023-09-10T01:01:56.036842+0000","last_undegraded":"2023-09-10T01:01:56.036842+0000","last_fullsized":"2023-09-10T01:01:56.036842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:00:10.002006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033268+0000","last_change":"2023-09-10T01:00:43.007579+0000","last_active":"2023-09-10T01:01:56.033268+0000","last_peered":"2023-09-10T01:01:56.033268+0000","last_clean":"2023-09-10T01:01:56.033268+0000","last_became_active":"2023-09-10T01:00:43.007200+0000","last_became_peered":"2023-09-10T01:00:43.007200+0000","last_unstale":"2023-09-10T01:01:56.033268+0000","last_undegraded":"2023-09-10T01:01:56.033268+0000","last_fullsized":"2023-09-10T01:01:56.033268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:20:08.688531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883972+0000","last_change":"2023-09-10T01:00:44.033511+0000","last_active":"2023-09-10T01:02:25.883972+0000","last_peered":"2023-09-10T01:02:25.883972+0000","last_clean":"2023-09-10T01:02:25.883972+0000","last_became_active":"2023-09-10T01:00:44.033170+0000","last_became_peered":"2023-09-10T01:00:44.033170+0000","last_unstale":"2023-09-10T01:02:25.883972+0000","last_undegraded":"2023-09-10T01:02:25.883972+0000","last_fullsized":"2023-09-10T01:02:25.883972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:36:12.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039255+0000","last_change":"2023-09-10T01:00:48.067436+0000","last_active":"2023-09-10T01:01:56.039255+0000","last_peered":"2023-09-10T01:01:56.039255+0000","last_clean":"2023-09-10T01:01:56.039255+0000","last_became_active":"2023-09-10T01:00:48.067038+0000","last_became_peered":"2023-09-10T01:00:48.067038+0000","last_unstale":"2023-09-10T01:01:56.039255+0000","last_undegraded":"2023-09-10T01:01:56.039255+0000","last_fullsized":"2023-09-10T01:01:56.039255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:26:41.210698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.044273+0000","last_change":"2023-09-10T01:00:43.006934+0000","last_active":"2023-09-10T01:01:57.044273+0000","last_peered":"2023-09-10T01:01:57.044273+0000","last_clean":"2023-09-10T01:01:57.044273+0000","last_became_active":"2023-09-10T01:00:43.005787+0000","last_became_peered":"2023-09-10T01:00:43.005787+0000","last_unstale":"2023-09-10T01:01:57.044273+0000","last_undegraded":"2023-09-10T01:01:57.044273+0000","last_fullsized":"2023-09-10T01:01:57.044273+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:48:27.403220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879574+0000","last_change":"2023-09-10T01:00:44.033812+0000","last_active":"2023-09-10T01:02:25.879574+0000","last_peered":"2023-09-10T01:02:25.879574+0000","last_clean":"2023-09-10T01:02:25.879574+0000","last_became_active":"2023-09-10T01:00:44.033496+0000","last_became_peered":"2023-09-10T01:00:44.033496+0000","last_unstale":"2023-09-10T01:02:25.879574+0000","last_undegraded":"2023-09-10T01:02:25.879574+0000","last_fullsized":"2023-09-10T01:02:25.879574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:12.679368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.043542+0000","last_change":"2023-09-10T01:00:48.060358+0000","last_active":"2023-09-10T01:01:57.043542+0000","last_peered":"2023-09-10T01:01:57.043542+0000","last_clean":"2023-09-10T01:01:57.043542+0000","last_became_active":"2023-09-10T01:00:48.059853+0000","last_became_peered":"2023-09-10T01:00:48.059853+0000","last_unstale":"2023-09-10T01:01:57.043542+0000","last_undegraded":"2023-09-10T01:01:57.043542+0000","last_fullsized":"2023-09-10T01:01:57.043542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:31:56.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031904+0000","last_change":"2023-09-10T01:00:43.002820+0000","last_active":"2023-09-10T01:01:56.031904+0000","last_peered":"2023-09-10T01:01:56.031904+0000","last_clean":"2023-09-10T01:01:56.031904+0000","last_became_active":"2023-09-10T01:00:43.002456+0000","last_became_peered":"2023-09-10T01:00:43.002456+0000","last_unstale":"2023-09-10T01:01:56.031904+0000","last_undegraded":"2023-09-10T01:01:56.031904+0000","last_fullsized":"2023-09-10T01:01:56.031904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:40:10.189942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879728+0000","last_change":"2023-09-10T01:00:44.032655+0000","last_active":"2023-09-10T01:02:25.879728+0000","last_peered":"2023-09-10T01:02:25.879728+0000","last_clean":"2023-09-10T01:02:25.879728+0000","last_became_active":"2023-09-10T01:00:44.032157+0000","last_became_peered":"2023-09-10T01:00:44.032157+0000","last_unstale":"2023-09-10T01:02:25.879728+0000","last_undegraded":"2023-09-10T01:02:25.879728+0000","last_fullsized":"2023-09-10T01:02:25.879728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:31.176768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872475+0000","last_change":"2023-09-10T01:00:44.022285+0000","last_active":"2023-09-10T01:02:25.872475+0000","last_peered":"2023-09-10T01:02:25.872475+0000","last_clean":"2023-09-10T01:02:25.872475+0000","last_became_active":"2023-09-10T01:00:44.021959+0000","last_became_peered":"2023-09-10T01:00:44.021959+0000","last_unstale":"2023-09-10T01:02:25.872475+0000","last_undegraded":"2023-09-10T01:02:25.872475+0000","last_fullsized":"2023-09-10T01:02:25.872475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:13:07.072249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.873581+0000","last_change":"2023-09-10T01:00:44.029722+0000","last_active":"2023-09-10T01:02:25.873581+0000","last_peered":"2023-09-10T01:02:25.873581+0000","last_clean":"2023-09-10T01:02:25.873581+0000","last_became_active":"2023-09-10T01:00:44.029424+0000","last_became_peered":"2023-09-10T01:00:44.029424+0000","last_unstale":"2023-09-10T01:02:25.873581+0000","last_undegraded":"2023-09-10T01:02:25.873581+0000","last_fullsized":"2023-09-10T01:02:25.873581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:24:16.655538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883202+0000","last_change":"2023-09-10T01:00:44.026026+0000","last_active":"2023-09-10T01:02:25.883202+0000","last_peered":"2023-09-10T01:02:25.883202+0000","last_clean":"2023-09-10T01:02:25.883202+0000","last_became_active":"2023-09-10T01:00:44.025727+0000","last_became_peered":"2023-09-10T01:00:44.025727+0000","last_unstale":"2023-09-10T01:02:25.883202+0000","last_undegraded":"2023-09-10T01:02:25.883202+0000","last_fullsized":"2023-09-10T01:02:25.883202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:59.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890507+0000","last_change":"2023-09-10T01:00:56.965355+0000","last_active":"2023-09-10T01:02:25.890507+0000","last_peered":"2023-09-10T01:02:25.890507+0000","last_clean":"2023-09-10T01:02:25.890507+0000","last_became_active":"2023-09-10T01:00:56.964903+0000","last_became_peered":"2023-09-10T01:00:56.964903+0000","last_unstale":"2023-09-10T01:02:25.890507+0000","last_undegraded":"2023-09-10T01:02:25.890507+0000","last_fullsized":"2023-09-10T01:02:25.890507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:49:21.940159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890130+0000","last_change":"2023-09-10T01:00:44.029525+0000","last_active":"2023-09-10T01:02:25.890130+0000","last_peered":"2023-09-10T01:02:25.890130+0000","last_clean":"2023-09-10T01:02:25.890130+0000","last_became_active":"2023-09-10T01:00:44.029139+0000","last_became_peered":"2023-09-10T01:00:44.029139+0000","last_unstale":"2023-09-10T01:02:25.890130+0000","last_undegraded":"2023-09-10T01:02:25.890130+0000","last_fullsized":"2023-09-10T01:02:25.890130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:20:24.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871853+0000","last_change":"2023-09-10T01:00:44.022299+0000","last_active":"2023-09-10T01:02:25.871853+0000","last_peered":"2023-09-10T01:02:25.871853+0000","last_clean":"2023-09-10T01:02:25.871853+0000","last_became_active":"2023-09-10T01:00:44.021992+0000","last_became_peered":"2023-09-10T01:00:44.021992+0000","last_unstale":"2023-09-10T01:02:25.871853+0000","last_undegraded":"2023-09-10T01:02:25.871853+0000","last_fullsized":"2023-09-10T01:02:25.871853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:50:30.766442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.870556+0000","last_change":"2023-09-10T01:01:57.108199+0000","last_active":"2023-09-10T01:02:25.870556+0000","last_peered":"2023-09-10T01:02:25.870556+0000","last_clean":"2023-09-10T01:02:25.870556+0000","last_became_active":"2023-09-10T01:01:57.107825+0000","last_became_peered":"2023-09-10T01:01:57.107825+0000","last_unstale":"2023-09-10T01:02:25.870556+0000","last_undegraded":"2023-09-10T01:02:25.870556+0000","last_fullsized":"2023-09-10T01:02:25.870556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:31:15.499277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,9],"acting":[2,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.876680+0000","last_change":"2023-09-10T01:01:57.049936+0000","last_active":"2023-09-10T01:02:25.876680+0000","last_peered":"2023-09-10T01:02:25.876680+0000","last_clean":"2023-09-10T01:02:25.876680+0000","last_became_active":"2023-09-10T01:01:57.049624+0000","last_became_peered":"2023-09-10T01:01:57.049624+0000","last_unstale":"2023-09-10T01:02:25.876680+0000","last_undegraded":"2023-09-10T01:02:25.876680+0000","last_fullsized":"2023-09-10T01:02:25.876680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:57:17.164558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887999+0000","last_change":"2023-09-10T01:00:44.025405+0000","last_active":"2023-09-10T01:02:25.887999+0000","last_peered":"2023-09-10T01:02:25.887999+0000","last_clean":"2023-09-10T01:02:25.887999+0000","last_became_active":"2023-09-10T01:00:44.025040+0000","last_became_peered":"2023-09-10T01:00:44.025040+0000","last_unstale":"2023-09-10T01:02:25.887999+0000","last_undegraded":"2023-09-10T01:02:25.887999+0000","last_fullsized":"2023-09-10T01:02:25.887999+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:16:46.504902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887060+0000","last_change":"2023-09-10T01:00:44.023603+0000","last_active":"2023-09-10T01:02:25.887060+0000","last_peered":"2023-09-10T01:02:25.887060+0000","last_clean":"2023-09-10T01:02:25.887060+0000","last_became_active":"2023-09-10T01:00:44.023279+0000","last_became_peered":"2023-09-10T01:00:44.023279+0000","last_unstale":"2023-09-10T01:02:25.887060+0000","last_undegraded":"2023-09-10T01:02:25.887060+0000","last_fullsized":"2023-09-10T01:02:25.887060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:18:05.737675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.892481+0000","last_change":"2023-09-10T01:00:44.035653+0000","last_active":"2023-09-10T01:02:25.892481+0000","last_peered":"2023-09-10T01:02:25.892481+0000","last_clean":"2023-09-10T01:02:25.892481+0000","last_became_active":"2023-09-10T01:00:44.035374+0000","last_became_peered":"2023-09-10T01:00:44.035374+0000","last_unstale":"2023-09-10T01:02:25.892481+0000","last_undegraded":"2023-09-10T01:02:25.892481+0000","last_fullsized":"2023-09-10T01:02:25.892481+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:14:45.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890327+0000","last_change":"2023-09-10T01:00:44.035805+0000","last_active":"2023-09-10T01:02:25.890327+0000","last_peered":"2023-09-10T01:02:25.890327+0000","last_clean":"2023-09-10T01:02:25.890327+0000","last_became_active":"2023-09-10T01:00:44.035479+0000","last_became_peered":"2023-09-10T01:00:44.035479+0000","last_unstale":"2023-09-10T01:02:25.890327+0000","last_undegraded":"2023-09-10T01:02:25.890327+0000","last_fullsized":"2023-09-10T01:02:25.890327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:36.662975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878041+0000","last_change":"2023-09-10T01:00:44.025418+0000","last_active":"2023-09-10T01:02:25.878041+0000","last_peered":"2023-09-10T01:02:25.878041+0000","last_clean":"2023-09-10T01:02:25.878041+0000","last_became_active":"2023-09-10T01:00:44.025104+0000","last_became_peered":"2023-09-10T01:00:44.025104+0000","last_unstale":"2023-09-10T01:02:25.878041+0000","last_undegraded":"2023-09-10T01:02:25.878041+0000","last_fullsized":"2023-09-10T01:02:25.878041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:05:13.050781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282844,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69231,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":677,"num_read_kb":413,"num_write":166,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":207693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481830,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":504243,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69199999999999995,"5min":0.87,"15min":0.87},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.60799999999999998,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.499}]},{"osd":1,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.626,"15min":0.625},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82299999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53100000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45500000000000002,"15min":0.45600000000000002},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.74399999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.53000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.74399999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.75}]},{"osd":6,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81799999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.69799999999999995},"min":{"1min":0.495,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.83299999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63600000000000001}]},{"osd":7,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69399999999999995,"15min":0.69199999999999995},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.85599999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.85899999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.46200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.74299999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.748,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.92300000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.873}]},{"osd":10,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93100000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.55900000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80100000000000005}]}]},{"osd":10,"up_from":70,"seq":300647710762,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1156019,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.65900000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55900000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.441,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58099999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.47199999999999998,"15min":0.47499999999999998},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.499}]},{"osd":2,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60699999999999998,"15min":0.60499999999999998},"min":{"1min":0.432,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80700000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61799999999999999,"15min":0.629},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60499999999999998,"15min":0.626},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.753,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.629},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.78900000000000003,"15min":0.85599999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78000000000000003,"5min":6.7519999999999998,"15min":6.7519999999999998},"last":0.67800000000000005}]},{"osd":4,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71299999999999997,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.36299999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.65900000000000003}]},{"osd":5,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.69499999999999995},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.99099999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.42199999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85799999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.87,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91300000000000003,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.82899999999999996}]},{"osd":7,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.76400000000000001,"15min":0.76800000000000002},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.3069999999999999,"5min":1.534,"15min":1.534},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75700000000000001,"15min":0.77500000000000002},"min":{"1min":0.59099999999999997,"5min":0.439,"15min":0.439},"max":{"1min":0.88800000000000001,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.79500000000000004}]},{"osd":9,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.58399999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.93200000000000005,"5min":1.304,"15min":1.304},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84899999999999998,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94299999999999995,"5min":7.0510000000000002,"15min":7.0510000000000002},"last":0.92100000000000004}]},{"osd":11,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73299999999999998,"15min":0.73099999999999998},"min":{"1min":0.50900000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93600000000000005,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76400000000000001,"15min":0.75800000000000001},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.84499999999999997}]}]},{"osd":9,"up_from":65,"seq":279172874287,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":507993,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55800000000000005,"15min":0.58399999999999996},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80200000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67500000000000004,"15min":0.68700000000000006},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":1.0069999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.73599999999999999},"min":{"1min":0.41199999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86499999999999999,"5min":1.349,"15min":1.349},"last":0.73499999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71299999999999997,"15min":0.70999999999999996},"min":{"1min":0.57199999999999995,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.069,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.90200000000000002}]},{"osd":3,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68300000000000005,"15min":0.70599999999999996},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.0860000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.751},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.42499999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.2729999999999999,"5min":1.647,"15min":1.647},"last":0.61299999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.80300000000000005,"15min":0.83599999999999997},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.369,"5min":1.595,"15min":1.595},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.85899999999999999},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0129999999999999,"5min":1.492,"15min":1.492},"last":0.79200000000000004}]},{"osd":5,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.88400000000000001,"15min":0.92700000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1890000000000001,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.873},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3460000000000001,"5min":1.714,"15min":1.714},"last":1.01}]},{"osd":6,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.97199999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.319,"5min":1.982,"15min":1.982},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.88900000000000001,"15min":0.92500000000000004},"min":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4330000000000001,"5min":1.569,"15min":1.569},"last":0.93100000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.94699999999999995,"15min":0.999},"min":{"1min":0.54500000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.294,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.97099999999999997,"15min":1.0109999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1759999999999999,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.96699999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.92200000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93300000000000005,"15min":0.94999999999999996},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.996}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.87,"15min":0.85699999999999998},"min":{"1min":0.69099999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84399999999999997,"15min":0.84199999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":1.0549999999999999}]}]},{"osd":8,"up_from":57,"seq":244813135924,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":503486,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80200000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68700000000000006,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51800000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86499999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.878,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.39400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.47399999999999998,"15min":0.45400000000000001},"min":{"1min":0.35999999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.73099999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.34499999999999997}]},{"osd":3,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.69099999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.624}]},{"osd":4,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.56100000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97199999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82799999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.66800000000000004}]},{"osd":5,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.60399999999999998,"15min":0.58799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995}]},{"osd":6,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.52100000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73999999999999999,"15min":0.73299999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.004,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.70499999999999996,"15min":0.67000000000000004},"min":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.72899999999999998,"15min":0.70899999999999996},"min":{"1min":0.60099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":9,"last update":"Sun Sep 10 01:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78900000000000003,"15min":0.76700000000000002},"min":{"1min":0.67100000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.78400000000000003,"15min":0.76400000000000001},"min":{"1min":0.63400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0249999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.81399999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.58499999999999996,"5min":0.44,"15min":0.44},"max":{"1min":0.99199999999999999,"5min":1.01,"15min":1.01},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.79000000000000004,"15min":0.751},"min":{"1min":0.65900000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.75}]},{"osd":11,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":1.071,"15min":1.071},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83399999999999996,"15min":0.83599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78900000000000003}]}]},{"osd":7,"up_from":50,"seq":214748364856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1099600,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91400000000000003,"5min":1.367,"15min":1.367},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.44400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.42099999999999999},"min":{"1min":0.34100000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.53000000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.378},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70299999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41999999999999998}]},{"osd":2,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.47699999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.82399999999999995,"5min":1.101,"15min":1.101},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80200000000000005,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63300000000000001,"15min":0.65300000000000002},"min":{"1min":0.42299999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.873,"5min":1.409,"15min":1.409},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.70999999999999996,"15min":0.68100000000000005},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88600000000000001,"5min":7.1929999999999996,"15min":7.1929999999999996},"last":0.51300000000000001}]},{"osd":4,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68200000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54500000000000004,"15min":0.53700000000000003},"min":{"1min":0.33000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82099999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.44900000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.79400000000000004,"5min":1.038,"15min":1.038},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88900000000000001,"5min":1.179,"15min":1.179},"last":0.75900000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.81299999999999994,"15min":0.75700000000000001},"min":{"1min":0.44500000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84799999999999998,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.80800000000000005,"15min":0.78400000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95099999999999996,"5min":7.1760000000000002,"15min":7.1760000000000002},"last":0.66600000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.012,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70099999999999996,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98399999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.64000000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.79600000000000004,"15min":0.78000000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":3.423,"5min":3.423,"15min":3.423},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.81699999999999995,"15min":0.79500000000000004},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.5249999999999999,"5min":3.5249999999999999,"15min":3.5249999999999999},"last":0.71999999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.746},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92900000000000005,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.62}]},{"osd":11,"last update":"Sun Sep 10 01:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.034,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85799999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0780000000000001,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.83499999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528380,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":499073,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.65100000000000002,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.9890000000000008,"5min":9.9890000000000008,"15min":9.9890000000000008},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.59099999999999997,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.363,"5min":2.363,"15min":2.363},"last":0.54300000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.67600000000000005,"15min":0.63100000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.577,"5min":0.89000000000000001,"15min":0.76800000000000002},"min":{"1min":0.50800000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.77900000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.748,"15min":0.72899999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.327,"5min":0.84899999999999998,"15min":0.78700000000000003},"min":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":10.608000000000001,"5min":10.608000000000001,"15min":10.608000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.67300000000000004,"15min":0.64900000000000002},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.9750000000000001,"5min":0.96899999999999997,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":23.449999999999999,"5min":23.449999999999999,"15min":23.449999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.76700000000000002,"15min":0.75},"min":{"1min":0.49399999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.77500000000000002,"15min":0.74099999999999999},"min":{"1min":0.60499999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.76200000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.94399999999999995,"15min":0.88900000000000001},"min":{"1min":0.62,"5min":0.503,"15min":0.503},"max":{"1min":10.132,"5min":10.132,"15min":10.132},"last":0.497},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.85499999999999998,"15min":0.84899999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.81200000000000006}]},{"osd":7,"last update":"Sun Sep 10 01:02:34 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.94499999999999995,"15min":0.85999999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7969999999999999,"5min":10.081,"15min":10.081},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.93600000000000005,"15min":0.85999999999999999},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":6.9009999999999998,"5min":6.9009999999999998,"15min":6.9009999999999998},"last":0.88200000000000001}]},{"osd":8,"last update":"Sun Sep 10 01:01:58 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.92900000000000005,"15min":0.91700000000000004},"min":{"1min":0.64900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.753},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89200000000000002,"15min":0.88600000000000001},"min":{"1min":0.65500000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.62}]},{"osd":9,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.093,"15min":1.0329999999999999},"min":{"1min":0.67300000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":12.683,"5min":12.683,"15min":12.683},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.0009999999999999,"15min":1.0289999999999999},"min":{"1min":0.66500000000000004,"5min":0.625,"15min":0.625},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.79400000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.94599999999999995,"15min":0.95299999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.056,"15min":0.98799999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.9450000000000001,"5min":10.170999999999999,"15min":10.170999999999999},"last":0.86699999999999999}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.4970000000000001,"15min":1.5389999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":6.9349999999999996,"5min":10.247,"15min":10.247},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.075,"15min":1.105},"min":{"1min":0.71699999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.379,"5min":2.3660000000000001,"15min":2.3660000000000001},"last":0.90400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":563757,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.627,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":5.3129999999999997,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.64700000000000002,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.2859999999999996,"5min":5.2859999999999996,"15min":5.2859999999999996},"last":0.46000000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.64800000000000002,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.9569999999999999,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.58499999999999996,"15min":0.53700000000000003},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.61799999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.66500000000000004,"15min":0.751},"min":{"1min":0.45200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.8919999999999999,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.69699999999999995,"15min":0.77900000000000003},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.875,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.75}]},{"osd":4,"last update":"Sun Sep 10 01:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45900000000000002,"15min":0.38800000000000001},"min":{"1min":0.36599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.46500000000000002,"15min":0.39700000000000002},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.69399999999999995,"15min":0.59099999999999997},"min":{"1min":0.626,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89000000000000001,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.62},"min":{"1min":0.57299999999999995,"5min":0.38,"15min":0.38},"max":{"1min":0.92500000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.67700000000000005}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.67900000000000005,"15min":0.64300000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.9769999999999999,"5min":2.9769999999999999,"15min":2.9769999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.012,"5min":0.73499999999999999,"15min":0.69299999999999995},"min":{"1min":0.53300000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":5.0970000000000004},"last":0.71599999999999997}]},{"osd":7,"last update":"Sun Sep 10 01:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55900000000000005},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60399999999999998,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.79000000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.68000000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.44,"15min":0.44},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.88}]},{"osd":9,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":0.81599999999999995,"15min":0.76000000000000001},"min":{"1min":0.60899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":4.9969999999999999,"5min":4.9969999999999999,"15min":4.9969999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.84699999999999998,"15min":0.80000000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":5.0380000000000003,"5min":5.0380000000000003,"15min":5.0380000000000003},"last":0.93899999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.71199999999999997,"15min":0.69399999999999995},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.73299999999999998,"15min":0.70499999999999996},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81200000000000006}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.86599999999999999,"15min":0.86399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0740000000000001,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.86799999999999999,"15min":0.86399999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1060000000000001,"5min":1.107,"15min":1.107},"last":0.89900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500461,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 10 01:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.67200000000000004,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.274,"5min":2.274,"15min":2.274},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.61799999999999999,"15min":0.56599999999999995},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.34399999999999997},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.996}]},{"osd":2,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69099999999999995,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.038,"5min":1.26,"15min":1.26},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.77200000000000002,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8999999999999999,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.90800000000000003}]},{"osd":3,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.439},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.92200000000000004}]},{"osd":4,"last update":"Sun Sep 10 01:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66000000000000003,"15min":0.59199999999999997},"min":{"1min":0.42699999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.70099999999999996,"15min":0.62},"min":{"1min":0.53000000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.167,"5min":1.302,"15min":1.302},"last":0.95699999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.747,"15min":0.71199999999999997},"min":{"1min":0.59799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.028,"5min":1.125,"15min":1.125},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70699999999999996,"15min":0.66800000000000004},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.752}]},{"osd":6,"last update":"Sun Sep 10 01:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69999999999999996,"15min":0.67100000000000004},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76100000000000001,"15min":0.72099999999999997},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":0.72199999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.79900000000000004},"min":{"1min":0.64300000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.58099999999999996,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.0329999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.66400000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.0840000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83299999999999996,"15min":0.84299999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1499999999999999,"5min":1.24,"15min":1.24},"last":0.92600000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:02:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.89100000000000001,"15min":0.92100000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3360000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.98399999999999999,"15min":1.01},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.046}]},{"osd":10,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.89000000000000001,"15min":0.91800000000000004},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.071},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.91300000000000003,"15min":0.93500000000000005},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.129,"5min":1.444,"15min":1.444},"last":1.0600000000000001}]},{"osd":11,"last update":"Sun Sep 10 01:02:03 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.92300000000000004,"15min":0.91800000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":1.093},{"interface":"front","average":{"1min":1.002,"5min":0.97799999999999998,"15min":0.97399999999999998},"min":{"1min":0.76700000000000002,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1020000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":564485,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67000000000000004,"15min":0.55200000000000005},"min":{"1min":0.434,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85999999999999999,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.56599999999999995},"min":{"1min":0.48099999999999998,"5min":0.36099999999999999,"15min":0.34399999999999997},"max":{"1min":0.86599999999999999,"5min":6.6639999999999997,"15min":6.6639999999999997},"last":0.51500000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.63400000000000001,"15min":0.55700000000000005},"min":{"1min":0.47699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82999999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.64600000000000002,"15min":0.56399999999999995},"min":{"1min":0.51300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88800000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.878}]},{"osd":3,"last update":"Sun Sep 10 01:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.64100000000000001,"15min":0.58299999999999996},"min":{"1min":0.47699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.63700000000000001,"15min":0.60099999999999998},"min":{"1min":0.505,"5min":0.379,"15min":0.379},"max":{"1min":1.6060000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.77800000000000002,"15min":0.78400000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93500000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.89200000000000002}]},{"osd":5,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.54300000000000004,"15min":0.46100000000000002},"min":{"1min":0.40899999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.60199999999999998,"15min":0.52800000000000002},"min":{"1min":0.41999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0029999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.86299999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.79300000000000004,"15min":0.70099999999999996},"min":{"1min":0.70199999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.6820000000000004,"5min":6.6820000000000004,"15min":6.6820000000000004},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.72799999999999998,"15min":0.68600000000000005},"min":{"1min":0.65500000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.65600000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.105,"5min":1.173,"15min":1.173},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.046,"5min":1.157,"15min":1.157},"last":0.97699999999999998}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56399999999999995,"15min":0.54500000000000004},"min":{"1min":0.51000000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.73299999999999998,"15min":0.71499999999999997},"min":{"1min":0.59499999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.96499999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:02:20 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.84699999999999998,"15min":0.80400000000000005},"min":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.113,"5min":0.80800000000000005,"15min":0.747},"min":{"1min":0.71699999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":4.577,"5min":4.577,"15min":4.577},"last":0.84599999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.71699999999999997,"15min":0.67600000000000005},"min":{"1min":0.59199999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.117,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.82399999999999995,"15min":0.78400000000000003},"min":{"1min":0.77500000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0649999999999999,"5min":1.099,"15min":1.099},"last":1.004}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.82699999999999996,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.621,"15min":0.621},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93899999999999995,"15min":0.93700000000000006},"min":{"1min":0.81599999999999995,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.097,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.0189999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":499267,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.47299999999999998,"15min":0.40699999999999997},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.54200000000000004,"15min":0.45100000000000001},"min":{"1min":0.33000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.5309999999999997,"5min":4.5309999999999997,"15min":4.5309999999999997},"last":0.40500000000000003}]},{"osd":1,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.69999999999999996,"15min":0.61899999999999999},"min":{"1min":0.49399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.63900000000000001,"15min":0.57499999999999996},"min":{"1min":0.48099999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.74399999999999999,"15min":0.64300000000000002},"min":{"1min":0.45700000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.819,"5min":4.819,"15min":4.819},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.72499999999999998,"15min":0.64700000000000002},"min":{"1min":0.53000000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.77200000000000002,"15min":0.69699999999999995},"min":{"1min":0.57899999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9139999999999999,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.82299999999999995,"15min":0.73599999999999999},"min":{"1min":0.66000000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.9450000000000001,"5min":2.52,"15min":2.52},"last":0.72199999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.79500000000000004,"15min":0.73299999999999998},"min":{"1min":0.51600000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":0.75800000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.71199999999999997,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.8609999999999998,"5min":4.8609999999999998,"15min":4.8609999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.745,"15min":0.67300000000000004},"min":{"1min":0.46100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.89900000000000002}]},{"osd":7,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.80100000000000005,"15min":0.79900000000000004},"min":{"1min":0.51300000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.437,"5min":0.434,"15min":0.434},"max":{"1min":0.99199999999999999,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.59999999999999998}]},{"osd":8,"last update":"Sun Sep 10 00:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Sun Sep 10 01:02:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":0.96199999999999997,"15min":0.97999999999999998},"min":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.012,"5min":0.93799999999999994,"15min":0.93600000000000005},"min":{"1min":0.53300000000000003,"5min":0.433,"15min":0.433},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.93600000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.97999999999999998,"15min":0.97199999999999998},"min":{"1min":0.71099999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.6680000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.84799999999999998,"15min":0.82899999999999996},"min":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.6280000000000001,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":0.66700000000000004}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.095,"15min":1.1259999999999999},"min":{"1min":0.73599999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.25,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0900000000000001,"15min":1.1020000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.6930000000000001,"5min":4.9139999999999997,"15min":4.9139999999999997},"last":0.879}]}]},{"osd":4,"up_from":32,"seq":137438953543,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":505192,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.36799999999999999},"min":{"1min":0.32900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41799999999999998,"15min":0.38400000000000001},"min":{"1min":0.34799999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.442}]},{"osd":2,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59399999999999997,"15min":0.56699999999999995},"min":{"1min":0.42699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0820000000000001,"5min":1.607,"15min":1.607},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.59099999999999997},"min":{"1min":0.50600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.079,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.61199999999999999,"15min":0.57699999999999996},"min":{"1min":0.46899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.86099999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.624},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.35999999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.495}]},{"osd":5,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.61199999999999999},"min":{"1min":0.47799999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1020000000000001,"5min":1.621,"15min":1.621},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66000000000000003,"15min":0.59999999999999998},"min":{"1min":0.56399999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81799999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.66900000000000004}]},{"osd":6,"last update":"Sun Sep 10 01:02:19 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.83399999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.44700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68500000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59399999999999997,"15min":0.54800000000000004},"min":{"1min":0.41599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.62,"15min":0.59099999999999997},"min":{"1min":0.42899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55500000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.67900000000000005,"15min":0.65100000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1200000000000001,"5min":1.802,"15min":1.802},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76700000000000002,"15min":0.73699999999999999},"min":{"1min":0.66300000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91100000000000003,"5min":0.997,"15min":0.997},"last":0.84399999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:02:22 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.58499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.73399999999999999},"min":{"1min":0.54600000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.751}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77800000000000002,"15min":0.77700000000000002},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95199999999999996,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.77400000000000002,"15min":0.76000000000000001},"min":{"1min":0.48699999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.76800000000000002}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.80400000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790018,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1095454,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56699999999999995,"15min":0.51500000000000001},"min":{"1min":0.38200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.55800000000000005,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.441}]},{"osd":1,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.63700000000000001,"15min":0.56799999999999995},"min":{"1min":0.47299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.622,"15min":0.56499999999999995},"min":{"1min":0.46700000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.39400000000000002},"min":{"1min":0.35099999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.623,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.46000000000000002,"15min":0.41799999999999998},"min":{"1min":0.316,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.38700000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.58199999999999996,"15min":0.55600000000000005},"min":{"1min":0.443,"5min":0.376,"15min":0.376},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.64000000000000001,"15min":0.56999999999999995},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.52500000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.65100000000000002,"15min":0.60099999999999998},"min":{"1min":0.47399999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92100000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.65700000000000003,"15min":0.60999999999999999},"min":{"1min":0.48499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1,"5min":1.02,"15min":1.02},"last":0.60699999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:02:17 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.67000000000000004,"15min":0.63800000000000001},"min":{"1min":0.58199999999999996,"5min":0.437,"15min":0.437},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.69599999999999995,"15min":0.66400000000000003},"min":{"1min":0.61799999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.89200000000000002,"5min":0.998,"15min":0.998},"last":0.623}]},{"osd":7,"last update":"Sun Sep 10 01:02:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.70299999999999996,"15min":0.67800000000000005},"min":{"1min":0.59599999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93000000000000005,"5min":1.042,"15min":1.042},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.73799999999999999,"15min":0.70699999999999996},"min":{"1min":0.52600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.77600000000000002}]},{"osd":8,"last update":"Sun Sep 10 01:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.66600000000000004,"15min":0.65000000000000002},"min":{"1min":0.438,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91000000000000003,"5min":1.036,"15min":1.036},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.64700000000000002,"15min":0.61899999999999999},"min":{"1min":0.45000000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.64300000000000002}]},{"osd":9,"last update":"Sun Sep 10 01:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.72999999999999998,"15min":0.69999999999999996},"min":{"1min":0.63300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.80100000000000005,"15min":0.77100000000000002},"min":{"1min":0.747,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.746}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.73699999999999999},"min":{"1min":0.59899999999999998,"5min":0.505,"15min":0.505},"max":{"1min":0.98999999999999999,"5min":1.107,"15min":1.107},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.81499999999999995,"15min":0.77700000000000002},"min":{"1min":0.64300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0860000000000001,"5min":1.171,"15min":1.171},"last":0.82899999999999996}]},{"osd":11,"last update":"Sun Sep 10 01:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.80400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-10T01:02:58.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-10T01:02:58.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:58 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/506468488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:58.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:58 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1142957311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.345+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/2202198470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41018c0 msgr2=0x7f1cc4101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.345+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/2202198470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41018c0 0x7f1cc4101d20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40099c0 tx=0x7f1cb4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.346+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/2202198470 shutdown_connections 2023-09-10T01:02:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.346+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/2202198470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc4102260 0x7f1cc4108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.346+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/2202198470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc41006d0 0x7f1cc4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.346+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/2202198470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41018c0 0x7f1cc4101d20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.346+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/2202198470 >> 172.21.15.3:0/2202198470 conn(0x7f1cc40fbc60 msgr2=0x7f1cc40fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.346+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/2202198470 shutdown_connections 2023-09-10T01:02:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.346+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/2202198470 wait complete. 2023-09-10T01:02:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 Processor -- start 2023-09-10T01:02:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 -- start start 2023-09-10T01:02:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41006d0 0x7f1cc41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc41018c0 0x7f1cc4198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc4102260 0x7f1cc419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc41a0130 con 0x7f1cc41006d0 2023-09-10T01:02:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc41a0270 con 0x7f1cc4102260 2023-09-10T01:02:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1ccc3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc41a0400 con 0x7f1cc41018c0 2023-09-10T01:02:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1cca19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41006d0 0x7f1cc41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1cca19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41006d0 0x7f1cc41985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49262/0 (socket says 172.21.15.3:49262) 2023-09-10T01:02:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1cca19b700 1 -- 172.21.15.3:0/4092472466 learned_addr learned my addr 172.21.15.3:0/4092472466 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.347+0000 7f1cca99c700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc4102260 0x7f1cc419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cc999a700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc41018c0 0x7f1cc4198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cca19b700 1 -- 172.21.15.3:0/4092472466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc41018c0 msgr2=0x7f1cc4198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cca19b700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc41018c0 0x7f1cc4198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cca19b700 1 -- 172.21.15.3:0/4092472466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc4102260 msgr2=0x7f1cc419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cca19b700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc4102260 0x7f1cc419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cca19b700 1 -- 172.21.15.3:0/4092472466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb40095f0 con 0x7f1cc41006d0 2023-09-10T01:02:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cca19b700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41006d0 0x7f1cc41985b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000e970 tx=0x7f1cc000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cbb7fe700 1 -- 172.21.15.3:0/4092472466 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0009790 con 0x7f1cc41006d0 2023-09-10T01:02:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc41a06e0 con 0x7f1cc41006d0 2023-09-10T01:02:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cbb7fe700 1 -- 172.21.15.3:0/4092472466 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cc000c7e0 con 0x7f1cc41006d0 2023-09-10T01:02:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.348+0000 7f1cbb7fe700 1 -- 172.21.15.3:0/4092472466 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0003c70 con 0x7f1cc41006d0 2023-09-10T01:02:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.349+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc41a0c40 con 0x7f1cc41006d0 2023-09-10T01:02:58.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.349+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc4192d30 con 0x7f1cc41006d0 2023-09-10T01:02:58.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.351+0000 7f1cbb7fe700 1 -- 172.21.15.3:0/4092472466 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f1cc000cc50 con 0x7f1cc41006d0 2023-09-10T01:02:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.353+0000 7f1cbb7fe700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1cb00a0640 0x7f1cb00a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.353+0000 7f1cbb7fe700 1 -- 172.21.15.3:0/4092472466 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f1cc0014070 con 0x7f1cc41006d0 2023-09-10T01:02:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.353+0000 7f1cc999a700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1cb00a0640 0x7f1cb00a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.354+0000 7f1cc999a700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1cb00a0640 0x7f1cb00a2b00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400b580 tx=0x7f1cb4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:58.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.357+0000 7f1cbb7fe700 1 -- 172.21.15.3:0/4092472466 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1cc0095880 con 0x7f1cc41006d0 2023-09-10T01:02:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:58 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/506468488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:58 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1142957311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:58 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/506468488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:02:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:58 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1142957311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:02:58.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.563+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1cc402cec0 con 0x7f1cb00a0640 2023-09-10T01:02:58.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.576+0000 7f1cbb7fe700 1 -- 172.21.15.3:0/4092472466 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427635 (secure 0 0 0) 0x7f1cc402cec0 con 0x7f1cb00a0640 2023-09-10T01:02:58.582 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:58.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1cb00a0640 msgr2=0x7f1cb00a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1cb00a0640 0x7f1cb00a2b00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400b580 tx=0x7f1cb4005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41006d0 msgr2=0x7f1cc41985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41006d0 0x7f1cc41985b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000e970 tx=0x7f1cc000ec80 comp rx=0 tx=0).stop 2023-09-10T01:02:58.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 shutdown_connections 2023-09-10T01:02:58.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cc4102260 0x7f1cc419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cc41006d0 0x7f1cc41985b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f1cb00a0640 0x7f1cb00a2b00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 --2- 172.21.15.3:0/4092472466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc41018c0 0x7f1cc4198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.581+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 >> 172.21.15.3:0/4092472466 conn(0x7f1cc40fbc60 msgr2=0x7f1cc41071a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:58.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.582+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 shutdown_connections 2023-09-10T01:02:58.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.582+0000 7f1ccc3ff700 1 -- 172.21.15.3:0/4092472466 wait complete. 2023-09-10T01:02:58.588 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-09-10T01:02:58.601 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2023-09-10T01:02:57.882984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659845,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":813,"num_read_kb":532,"num_write":349,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336256,"kb_used_data":12064,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605568,"statfs":{"total":1151940427776,"available":1151596101632,"internally_reserved":0,"allocated":12353536,"data_stored":7999030,"data_compressed":391800,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":93,"num_read_kb":52,"num_write":7,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003042"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885419+0000","last_change":"2023-09-10T01:00:44.026223+0000","last_active":"2023-09-10T01:02:25.885419+0000","last_peered":"2023-09-10T01:02:25.885419+0000","last_clean":"2023-09-10T01:02:25.885419+0000","last_became_active":"2023-09-10T01:00:44.025731+0000","last_became_peered":"2023-09-10T01:00:44.025731+0000","last_unstale":"2023-09-10T01:02:25.885419+0000","last_undegraded":"2023-09-10T01:02:25.885419+0000","last_fullsized":"2023-09-10T01:02:25.885419+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:51:48.949903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871454+0000","last_change":"2023-09-10T01:00:44.038916+0000","last_active":"2023-09-10T01:02:25.871454+0000","last_peered":"2023-09-10T01:02:25.871454+0000","last_clean":"2023-09-10T01:02:25.871454+0000","last_became_active":"2023-09-10T01:00:44.038546+0000","last_became_peered":"2023-09-10T01:00:44.038546+0000","last_unstale":"2023-09-10T01:02:25.871454+0000","last_undegraded":"2023-09-10T01:02:25.871454+0000","last_fullsized":"2023-09-10T01:02:25.871454+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:05:56.096857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880464+0000","last_change":"2023-09-10T01:00:44.032942+0000","last_active":"2023-09-10T01:02:25.880464+0000","last_peered":"2023-09-10T01:02:25.880464+0000","last_clean":"2023-09-10T01:02:25.880464+0000","last_became_active":"2023-09-10T01:00:44.032546+0000","last_became_peered":"2023-09-10T01:00:44.032546+0000","last_unstale":"2023-09-10T01:02:25.880464+0000","last_undegraded":"2023-09-10T01:02:25.880464+0000","last_fullsized":"2023-09-10T01:02:25.880464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:46.613487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.874834+0000","last_change":"2023-09-10T01:00:44.031411+0000","last_active":"2023-09-10T01:02:25.874834+0000","last_peered":"2023-09-10T01:02:25.874834+0000","last_clean":"2023-09-10T01:02:25.874834+0000","last_became_active":"2023-09-10T01:00:44.031121+0000","last_became_peered":"2023-09-10T01:00:44.031121+0000","last_unstale":"2023-09-10T01:02:25.874834+0000","last_undegraded":"2023-09-10T01:02:25.874834+0000","last_fullsized":"2023-09-10T01:02:25.874834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:47.658116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887476+0000","last_change":"2023-09-10T01:00:44.037351+0000","last_active":"2023-09-10T01:02:25.887476+0000","last_peered":"2023-09-10T01:02:25.887476+0000","last_clean":"2023-09-10T01:02:25.887476+0000","last_became_active":"2023-09-10T01:00:44.037005+0000","last_became_peered":"2023-09-10T01:00:44.037005+0000","last_unstale":"2023-09-10T01:02:25.887476+0000","last_undegraded":"2023-09-10T01:02:25.887476+0000","last_fullsized":"2023-09-10T01:02:25.887476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:10:45.694056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882017+0000","last_change":"2023-09-10T01:00:44.029543+0000","last_active":"2023-09-10T01:02:25.882017+0000","last_peered":"2023-09-10T01:02:25.882017+0000","last_clean":"2023-09-10T01:02:25.882017+0000","last_became_active":"2023-09-10T01:00:44.029210+0000","last_became_peered":"2023-09-10T01:00:44.029210+0000","last_unstale":"2023-09-10T01:02:25.882017+0000","last_undegraded":"2023-09-10T01:02:25.882017+0000","last_fullsized":"2023-09-10T01:02:25.882017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:08:56.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879037+0000","last_change":"2023-09-10T01:00:44.033328+0000","last_active":"2023-09-10T01:02:25.879037+0000","last_peered":"2023-09-10T01:02:25.879037+0000","last_clean":"2023-09-10T01:02:25.879037+0000","last_became_active":"2023-09-10T01:00:44.033007+0000","last_became_peered":"2023-09-10T01:00:44.033007+0000","last_unstale":"2023-09-10T01:02:25.879037+0000","last_undegraded":"2023-09-10T01:02:25.879037+0000","last_fullsized":"2023-09-10T01:02:25.879037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:59:14.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891940+0000","last_change":"2023-09-10T01:00:44.025611+0000","last_active":"2023-09-10T01:02:25.891940+0000","last_peered":"2023-09-10T01:02:25.891940+0000","last_clean":"2023-09-10T01:02:25.891940+0000","last_became_active":"2023-09-10T01:00:44.025249+0000","last_became_peered":"2023-09-10T01:00:44.025249+0000","last_unstale":"2023-09-10T01:02:25.891940+0000","last_undegraded":"2023-09-10T01:02:25.891940+0000","last_fullsized":"2023-09-10T01:02:25.891940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:55:04.320356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878755+0000","last_change":"2023-09-10T01:00:44.024207+0000","last_active":"2023-09-10T01:02:25.878755+0000","last_peered":"2023-09-10T01:02:25.878755+0000","last_clean":"2023-09-10T01:02:25.878755+0000","last_became_active":"2023-09-10T01:00:44.023866+0000","last_became_peered":"2023-09-10T01:00:44.023866+0000","last_unstale":"2023-09-10T01:02:25.878755+0000","last_undegraded":"2023-09-10T01:02:25.878755+0000","last_fullsized":"2023-09-10T01:02:25.878755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:53:39.845384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883727+0000","last_change":"2023-09-10T01:00:44.031241+0000","last_active":"2023-09-10T01:02:25.883727+0000","last_peered":"2023-09-10T01:02:25.883727+0000","last_clean":"2023-09-10T01:02:25.883727+0000","last_became_active":"2023-09-10T01:00:44.030392+0000","last_became_peered":"2023-09-10T01:00:44.030392+0000","last_unstale":"2023-09-10T01:02:25.883727+0000","last_undegraded":"2023-09-10T01:02:25.883727+0000","last_fullsized":"2023-09-10T01:02:25.883727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:19:39.786995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883954+0000","last_change":"2023-09-10T01:00:44.026983+0000","last_active":"2023-09-10T01:02:25.883954+0000","last_peered":"2023-09-10T01:02:25.883954+0000","last_clean":"2023-09-10T01:02:25.883954+0000","last_became_active":"2023-09-10T01:00:44.026695+0000","last_became_peered":"2023-09-10T01:00:44.026695+0000","last_unstale":"2023-09-10T01:02:25.883954+0000","last_undegraded":"2023-09-10T01:02:25.883954+0000","last_fullsized":"2023-09-10T01:02:25.883954+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:37.809144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.874638+0000","last_change":"2023-09-10T01:00:44.030787+0000","last_active":"2023-09-10T01:02:25.874638+0000","last_peered":"2023-09-10T01:02:25.874638+0000","last_clean":"2023-09-10T01:02:25.874638+0000","last_became_active":"2023-09-10T01:00:44.028579+0000","last_became_peered":"2023-09-10T01:00:44.028579+0000","last_unstale":"2023-09-10T01:02:25.874638+0000","last_undegraded":"2023-09-10T01:02:25.874638+0000","last_fullsized":"2023-09-10T01:02:25.874638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:04:26.446378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.886732+0000","last_change":"2023-09-10T01:00:44.029905+0000","last_active":"2023-09-10T01:02:25.886732+0000","last_peered":"2023-09-10T01:02:25.886732+0000","last_clean":"2023-09-10T01:02:25.886732+0000","last_became_active":"2023-09-10T01:00:44.029561+0000","last_became_peered":"2023-09-10T01:00:44.029561+0000","last_unstale":"2023-09-10T01:02:25.886732+0000","last_undegraded":"2023-09-10T01:02:25.886732+0000","last_fullsized":"2023-09-10T01:02:25.886732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:17:50.839001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.877754+0000","last_change":"2023-09-10T01:00:44.037864+0000","last_active":"2023-09-10T01:02:25.877754+0000","last_peered":"2023-09-10T01:02:25.877754+0000","last_clean":"2023-09-10T01:02:25.877754+0000","last_became_active":"2023-09-10T01:00:44.037533+0000","last_became_peered":"2023-09-10T01:00:44.037533+0000","last_unstale":"2023-09-10T01:02:25.877754+0000","last_undegraded":"2023-09-10T01:02:25.877754+0000","last_fullsized":"2023-09-10T01:02:25.877754+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:29:26.768570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.869883+0000","last_change":"2023-09-10T01:00:44.033396+0000","last_active":"2023-09-10T01:02:25.869883+0000","last_peered":"2023-09-10T01:02:25.869883+0000","last_clean":"2023-09-10T01:02:25.869883+0000","last_became_active":"2023-09-10T01:00:44.033056+0000","last_became_peered":"2023-09-10T01:00:44.033056+0000","last_unstale":"2023-09-10T01:02:25.869883+0000","last_undegraded":"2023-09-10T01:02:25.869883+0000","last_fullsized":"2023-09-10T01:02:25.869883+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:00:03.590555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882604+0000","last_change":"2023-09-10T01:00:44.027729+0000","last_active":"2023-09-10T01:02:25.882604+0000","last_peered":"2023-09-10T01:02:25.882604+0000","last_clean":"2023-09-10T01:02:25.882604+0000","last_became_active":"2023-09-10T01:00:44.027447+0000","last_became_peered":"2023-09-10T01:00:44.027447+0000","last_unstale":"2023-09-10T01:02:25.882604+0000","last_undegraded":"2023-09-10T01:02:25.882604+0000","last_fullsized":"2023-09-10T01:02:25.882604+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:06:09.391443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885073+0000","last_change":"2023-09-10T01:00:44.026042+0000","last_active":"2023-09-10T01:02:25.885073+0000","last_peered":"2023-09-10T01:02:25.885073+0000","last_clean":"2023-09-10T01:02:25.885073+0000","last_became_active":"2023-09-10T01:00:44.025309+0000","last_became_peered":"2023-09-10T01:00:44.025309+0000","last_unstale":"2023-09-10T01:02:25.885073+0000","last_undegraded":"2023-09-10T01:02:25.885073+0000","last_fullsized":"2023-09-10T01:02:25.885073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:51:29.038717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.886092+0000","last_change":"2023-09-10T01:00:44.036761+0000","last_active":"2023-09-10T01:02:25.886092+0000","last_peered":"2023-09-10T01:02:25.886092+0000","last_clean":"2023-09-10T01:02:25.886092+0000","last_became_active":"2023-09-10T01:00:44.036333+0000","last_became_peered":"2023-09-10T01:00:44.036333+0000","last_unstale":"2023-09-10T01:02:25.886092+0000","last_undegraded":"2023-09-10T01:02:25.886092+0000","last_fullsized":"2023-09-10T01:02:25.886092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:37:09.042178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885302+0000","last_change":"2023-09-10T01:00:44.027304+0000","last_active":"2023-09-10T01:02:25.885302+0000","last_peered":"2023-09-10T01:02:25.885302+0000","last_clean":"2023-09-10T01:02:25.885302+0000","last_became_active":"2023-09-10T01:00:44.026994+0000","last_became_peered":"2023-09-10T01:00:44.026994+0000","last_unstale":"2023-09-10T01:02:25.885302+0000","last_undegraded":"2023-09-10T01:02:25.885302+0000","last_fullsized":"2023-09-10T01:02:25.885302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:11:04.824687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032471+0000","last_change":"2023-09-10T01:00:48.056098+0000","last_active":"2023-09-10T01:01:56.032471+0000","last_peered":"2023-09-10T01:01:56.032471+0000","last_clean":"2023-09-10T01:01:56.032471+0000","last_became_active":"2023-09-10T01:00:48.055694+0000","last_became_peered":"2023-09-10T01:00:48.055694+0000","last_unstale":"2023-09-10T01:01:56.032471+0000","last_undegraded":"2023-09-10T01:01:56.032471+0000","last_fullsized":"2023-09-10T01:01:56.032471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:30:48.711613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.046616+0000","last_change":"2023-09-10T01:00:56.971435+0000","last_active":"2023-09-10T01:01:57.046616+0000","last_peered":"2023-09-10T01:01:57.046616+0000","last_clean":"2023-09-10T01:01:57.046616+0000","last_became_active":"2023-09-10T01:00:56.965158+0000","last_became_peered":"2023-09-10T01:00:56.965158+0000","last_unstale":"2023-09-10T01:01:57.046616+0000","last_undegraded":"2023-09-10T01:01:57.046616+0000","last_fullsized":"2023-09-10T01:01:57.046616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:09:20.910597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880753+0000","last_change":"2023-09-10T01:00:44.023469+0000","last_active":"2023-09-10T01:02:25.880753+0000","last_peered":"2023-09-10T01:02:25.880753+0000","last_clean":"2023-09-10T01:02:25.880753+0000","last_became_active":"2023-09-10T01:00:44.023232+0000","last_became_peered":"2023-09-10T01:00:44.023232+0000","last_unstale":"2023-09-10T01:02:25.880753+0000","last_undegraded":"2023-09-10T01:02:25.880753+0000","last_fullsized":"2023-09-10T01:02:25.880753+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:33:22.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037995+0000","last_change":"2023-09-10T01:00:48.063565+0000","last_active":"2023-09-10T01:01:56.037995+0000","last_peered":"2023-09-10T01:01:56.037995+0000","last_clean":"2023-09-10T01:01:56.037995+0000","last_became_active":"2023-09-10T01:00:48.063200+0000","last_became_peered":"2023-09-10T01:00:48.063200+0000","last_unstale":"2023-09-10T01:01:56.037995+0000","last_undegraded":"2023-09-10T01:01:56.037995+0000","last_fullsized":"2023-09-10T01:01:56.037995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:50:16.624296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.471781+0000","last_change":"2023-09-10T01:00:43.010566+0000","last_active":"2023-09-10T01:01:57.471781+0000","last_peered":"2023-09-10T01:01:57.471781+0000","last_clean":"2023-09-10T01:01:57.471781+0000","last_became_active":"2023-09-10T01:00:43.010277+0000","last_became_peered":"2023-09-10T01:00:43.010277+0000","last_unstale":"2023-09-10T01:01:57.471781+0000","last_undegraded":"2023-09-10T01:01:57.471781+0000","last_fullsized":"2023-09-10T01:01:57.471781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:37.808085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.884035+0000","last_change":"2023-09-10T01:00:44.026219+0000","last_active":"2023-09-10T01:02:25.884035+0000","last_peered":"2023-09-10T01:02:25.884035+0000","last_clean":"2023-09-10T01:02:25.884035+0000","last_became_active":"2023-09-10T01:00:44.025906+0000","last_became_peered":"2023-09-10T01:00:44.025906+0000","last_unstale":"2023-09-10T01:02:25.884035+0000","last_undegraded":"2023-09-10T01:02:25.884035+0000","last_fullsized":"2023-09-10T01:02:25.884035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:21:48.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037347+0000","last_change":"2023-09-10T01:00:48.069671+0000","last_active":"2023-09-10T01:01:56.037347+0000","last_peered":"2023-09-10T01:01:56.037347+0000","last_clean":"2023-09-10T01:01:56.037347+0000","last_became_active":"2023-09-10T01:00:48.069382+0000","last_became_peered":"2023-09-10T01:00:48.069382+0000","last_unstale":"2023-09-10T01:01:56.037347+0000","last_undegraded":"2023-09-10T01:01:56.037347+0000","last_fullsized":"2023-09-10T01:01:56.037347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:06:20.526398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036543+0000","last_change":"2023-09-10T01:00:43.005098+0000","last_active":"2023-09-10T01:01:56.036543+0000","last_peered":"2023-09-10T01:01:56.036543+0000","last_clean":"2023-09-10T01:01:56.036543+0000","last_became_active":"2023-09-10T01:00:43.004556+0000","last_became_peered":"2023-09-10T01:00:43.004556+0000","last_unstale":"2023-09-10T01:01:56.036543+0000","last_undegraded":"2023-09-10T01:01:56.036543+0000","last_fullsized":"2023-09-10T01:01:56.036543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:27:43.538936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890280+0000","last_change":"2023-09-10T01:00:44.034451+0000","last_active":"2023-09-10T01:02:25.890280+0000","last_peered":"2023-09-10T01:02:25.890280+0000","last_clean":"2023-09-10T01:02:25.890280+0000","last_became_active":"2023-09-10T01:00:44.034131+0000","last_became_peered":"2023-09-10T01:00:44.034131+0000","last_unstale":"2023-09-10T01:02:25.890280+0000","last_undegraded":"2023-09-10T01:02:25.890280+0000","last_fullsized":"2023-09-10T01:02:25.890280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:50.643215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036946+0000","last_change":"2023-09-10T01:00:48.060114+0000","last_active":"2023-09-10T01:01:56.036946+0000","last_peered":"2023-09-10T01:01:56.036946+0000","last_clean":"2023-09-10T01:01:56.036946+0000","last_became_active":"2023-09-10T01:00:48.059531+0000","last_became_peered":"2023-09-10T01:00:48.059531+0000","last_unstale":"2023-09-10T01:01:56.036946+0000","last_undegraded":"2023-09-10T01:01:56.036946+0000","last_fullsized":"2023-09-10T01:01:56.036946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:28:01.462247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035950+0000","last_change":"2023-09-10T01:00:43.007196+0000","last_active":"2023-09-10T01:01:56.035950+0000","last_peered":"2023-09-10T01:01:56.035950+0000","last_clean":"2023-09-10T01:01:56.035950+0000","last_became_active":"2023-09-10T01:00:43.006795+0000","last_became_peered":"2023-09-10T01:00:43.006795+0000","last_unstale":"2023-09-10T01:01:56.035950+0000","last_undegraded":"2023-09-10T01:01:56.035950+0000","last_fullsized":"2023-09-10T01:01:56.035950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:26:13.146244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882156+0000","last_change":"2023-09-10T01:00:44.027809+0000","last_active":"2023-09-10T01:02:25.882156+0000","last_peered":"2023-09-10T01:02:25.882156+0000","last_clean":"2023-09-10T01:02:25.882156+0000","last_became_active":"2023-09-10T01:00:44.027499+0000","last_became_peered":"2023-09-10T01:00:44.027499+0000","last_unstale":"2023-09-10T01:02:25.882156+0000","last_undegraded":"2023-09-10T01:02:25.882156+0000","last_fullsized":"2023-09-10T01:02:25.882156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:17:07.138807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032849+0000","last_change":"2023-09-10T01:00:48.057645+0000","last_active":"2023-09-10T01:01:56.032849+0000","last_peered":"2023-09-10T01:01:56.032849+0000","last_clean":"2023-09-10T01:01:56.032849+0000","last_became_active":"2023-09-10T01:00:48.057312+0000","last_became_peered":"2023-09-10T01:00:48.057312+0000","last_unstale":"2023-09-10T01:01:56.032849+0000","last_undegraded":"2023-09-10T01:01:56.032849+0000","last_fullsized":"2023-09-10T01:01:56.032849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:28.904572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:56.110754+0000","last_change":"2023-09-10T01:00:56.965677+0000","last_active":"2023-09-10T01:02:56.110754+0000","last_peered":"2023-09-10T01:02:56.110754+0000","last_clean":"2023-09-10T01:02:56.110754+0000","last_became_active":"2023-09-10T01:00:56.965248+0000","last_became_peered":"2023-09-10T01:00:56.965248+0000","last_unstale":"2023-09-10T01:02:56.110754+0000","last_undegraded":"2023-09-10T01:02:56.110754+0000","last_fullsized":"2023-09-10T01:02:56.110754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:39:44.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.866210+0000","last_change":"2023-09-10T01:00:44.026340+0000","last_active":"2023-09-10T01:02:25.866210+0000","last_peered":"2023-09-10T01:02:25.866210+0000","last_clean":"2023-09-10T01:02:25.866210+0000","last_became_active":"2023-09-10T01:00:44.026024+0000","last_became_peered":"2023-09-10T01:00:44.026024+0000","last_unstale":"2023-09-10T01:02:25.866210+0000","last_undegraded":"2023-09-10T01:02:25.866210+0000","last_fullsized":"2023-09-10T01:02:25.866210+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:58:18.942980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032668+0000","last_change":"2023-09-10T01:00:48.057876+0000","last_active":"2023-09-10T01:01:56.032668+0000","last_peered":"2023-09-10T01:01:56.032668+0000","last_clean":"2023-09-10T01:01:56.032668+0000","last_became_active":"2023-09-10T01:00:48.056686+0000","last_became_peered":"2023-09-10T01:00:48.056686+0000","last_unstale":"2023-09-10T01:01:56.032668+0000","last_undegraded":"2023-09-10T01:01:56.032668+0000","last_fullsized":"2023-09-10T01:01:56.032668+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:10:15.995463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127691+0000","last_change":"2023-09-10T01:00:43.007811+0000","last_active":"2023-09-10T01:01:56.127691+0000","last_peered":"2023-09-10T01:01:56.127691+0000","last_clean":"2023-09-10T01:01:56.127691+0000","last_became_active":"2023-09-10T01:00:43.007205+0000","last_became_peered":"2023-09-10T01:00:43.007205+0000","last_unstale":"2023-09-10T01:01:56.127691+0000","last_undegraded":"2023-09-10T01:01:56.127691+0000","last_fullsized":"2023-09-10T01:01:56.127691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:27:38.114614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.888059+0000","last_change":"2023-09-10T01:00:44.027954+0000","last_active":"2023-09-10T01:02:25.888059+0000","last_peered":"2023-09-10T01:02:25.888059+0000","last_clean":"2023-09-10T01:02:25.888059+0000","last_became_active":"2023-09-10T01:00:44.027643+0000","last_became_peered":"2023-09-10T01:00:44.027643+0000","last_unstale":"2023-09-10T01:02:25.888059+0000","last_undegraded":"2023-09-10T01:02:25.888059+0000","last_fullsized":"2023-09-10T01:02:25.888059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:21.253013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.731942+0000","last_change":"2023-09-10T01:00:48.062519+0000","last_active":"2023-09-10T01:01:58.731942+0000","last_peered":"2023-09-10T01:01:58.731942+0000","last_clean":"2023-09-10T01:01:58.731942+0000","last_became_active":"2023-09-10T01:00:48.061985+0000","last_became_peered":"2023-09-10T01:00:48.061985+0000","last_unstale":"2023-09-10T01:01:58.731942+0000","last_undegraded":"2023-09-10T01:01:58.731942+0000","last_fullsized":"2023-09-10T01:01:58.731942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:47:36.002325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030435+0000","last_change":"2023-09-10T01:00:42.999601+0000","last_active":"2023-09-10T01:01:56.030435+0000","last_peered":"2023-09-10T01:01:56.030435+0000","last_clean":"2023-09-10T01:01:56.030435+0000","last_became_active":"2023-09-10T01:00:42.999243+0000","last_became_peered":"2023-09-10T01:00:42.999243+0000","last_unstale":"2023-09-10T01:01:56.030435+0000","last_undegraded":"2023-09-10T01:01:56.030435+0000","last_fullsized":"2023-09-10T01:01:56.030435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:49.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890496+0000","last_change":"2023-09-10T01:00:44.027136+0000","last_active":"2023-09-10T01:02:25.890496+0000","last_peered":"2023-09-10T01:02:25.890496+0000","last_clean":"2023-09-10T01:02:25.890496+0000","last_became_active":"2023-09-10T01:00:44.026784+0000","last_became_peered":"2023-09-10T01:00:44.026784+0000","last_unstale":"2023-09-10T01:02:25.890496+0000","last_undegraded":"2023-09-10T01:02:25.890496+0000","last_fullsized":"2023-09-10T01:02:25.890496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:51:51.271658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.055264+0000","last_change":"2023-09-10T01:00:48.066906+0000","last_active":"2023-09-10T01:01:57.055264+0000","last_peered":"2023-09-10T01:01:57.055264+0000","last_clean":"2023-09-10T01:01:57.055264+0000","last_became_active":"2023-09-10T01:00:48.066474+0000","last_became_peered":"2023-09-10T01:00:48.066474+0000","last_unstale":"2023-09-10T01:01:57.055264+0000","last_undegraded":"2023-09-10T01:01:57.055264+0000","last_fullsized":"2023-09-10T01:01:57.055264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:48:26.827857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036365+0000","last_change":"2023-09-10T01:00:43.005922+0000","last_active":"2023-09-10T01:01:56.036365+0000","last_peered":"2023-09-10T01:01:56.036365+0000","last_clean":"2023-09-10T01:01:56.036365+0000","last_became_active":"2023-09-10T01:00:43.005517+0000","last_became_peered":"2023-09-10T01:00:43.005517+0000","last_unstale":"2023-09-10T01:01:56.036365+0000","last_undegraded":"2023-09-10T01:01:56.036365+0000","last_fullsized":"2023-09-10T01:01:56.036365+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:32:12.619034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889259+0000","last_change":"2023-09-10T01:00:44.036766+0000","last_active":"2023-09-10T01:02:25.889259+0000","last_peered":"2023-09-10T01:02:25.889259+0000","last_clean":"2023-09-10T01:02:25.889259+0000","last_became_active":"2023-09-10T01:00:44.036316+0000","last_became_peered":"2023-09-10T01:00:44.036316+0000","last_unstale":"2023-09-10T01:02:25.889259+0000","last_undegraded":"2023-09-10T01:02:25.889259+0000","last_fullsized":"2023-09-10T01:02:25.889259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:17:51.875838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036026+0000","last_change":"2023-09-10T01:00:57.051716+0000","last_active":"2023-09-10T01:01:56.036026+0000","last_peered":"2023-09-10T01:01:56.036026+0000","last_clean":"2023-09-10T01:01:56.036026+0000","last_became_active":"2023-09-10T01:00:57.051334+0000","last_became_peered":"2023-09-10T01:00:57.051334+0000","last_unstale":"2023-09-10T01:01:56.036026+0000","last_undegraded":"2023-09-10T01:01:56.036026+0000","last_fullsized":"2023-09-10T01:01:56.036026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:30:52.598639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":133,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:56.109837+0000","last_change":"2023-09-10T01:00:43.008695+0000","last_active":"2023-09-10T01:02:56.109837+0000","last_peered":"2023-09-10T01:02:56.109837+0000","last_clean":"2023-09-10T01:02:56.109837+0000","last_became_active":"2023-09-10T01:00:43.008278+0000","last_became_peered":"2023-09-10T01:00:43.008278+0000","last_unstale":"2023-09-10T01:02:56.109837+0000","last_undegraded":"2023-09-10T01:02:56.109837+0000","last_fullsized":"2023-09-10T01:02:56.109837+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:46:38.370582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882691+0000","last_change":"2023-09-10T01:00:44.030540+0000","last_active":"2023-09-10T01:02:25.882691+0000","last_peered":"2023-09-10T01:02:25.882691+0000","last_clean":"2023-09-10T01:02:25.882691+0000","last_became_active":"2023-09-10T01:00:44.030212+0000","last_became_peered":"2023-09-10T01:00:44.030212+0000","last_unstale":"2023-09-10T01:02:25.882691+0000","last_undegraded":"2023-09-10T01:02:25.882691+0000","last_fullsized":"2023-09-10T01:02:25.882691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:57:06.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036424+0000","last_change":"2023-09-10T01:00:57.052197+0000","last_active":"2023-09-10T01:01:56.036424+0000","last_peered":"2023-09-10T01:01:56.036424+0000","last_clean":"2023-09-10T01:01:56.036424+0000","last_became_active":"2023-09-10T01:00:57.051782+0000","last_became_peered":"2023-09-10T01:00:57.051782+0000","last_unstale":"2023-09-10T01:01:56.036424+0000","last_undegraded":"2023-09-10T01:01:56.036424+0000","last_fullsized":"2023-09-10T01:01:56.036424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:36:49.680123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042966+0000","last_change":"2023-09-10T01:00:43.007242+0000","last_active":"2023-09-10T01:01:56.042966+0000","last_peered":"2023-09-10T01:01:56.042966+0000","last_clean":"2023-09-10T01:01:56.042966+0000","last_became_active":"2023-09-10T01:00:43.006912+0000","last_became_peered":"2023-09-10T01:00:43.006912+0000","last_unstale":"2023-09-10T01:01:56.042966+0000","last_undegraded":"2023-09-10T01:01:56.042966+0000","last_fullsized":"2023-09-10T01:01:56.042966+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:48:22.783656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.884684+0000","last_change":"2023-09-10T01:01:58.026024+0000","last_active":"2023-09-10T01:02:25.884684+0000","last_peered":"2023-09-10T01:02:25.884684+0000","last_clean":"2023-09-10T01:02:25.884684+0000","last_became_active":"2023-09-10T01:01:58.025647+0000","last_became_peered":"2023-09-10T01:01:58.025647+0000","last_unstale":"2023-09-10T01:02:25.884684+0000","last_undegraded":"2023-09-10T01:02:25.884684+0000","last_fullsized":"2023-09-10T01:02:25.884684+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:58:11.916233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037579+0000","last_change":"2023-09-10T01:00:48.062520+0000","last_active":"2023-09-10T01:01:56.037579+0000","last_peered":"2023-09-10T01:01:56.037579+0000","last_clean":"2023-09-10T01:01:56.037579+0000","last_became_active":"2023-09-10T01:00:48.061870+0000","last_became_peered":"2023-09-10T01:00:48.061870+0000","last_unstale":"2023-09-10T01:01:56.037579+0000","last_undegraded":"2023-09-10T01:01:56.037579+0000","last_fullsized":"2023-09-10T01:01:56.037579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:24:38.798775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037030+0000","last_change":"2023-09-10T01:00:43.006513+0000","last_active":"2023-09-10T01:01:56.037030+0000","last_peered":"2023-09-10T01:01:56.037030+0000","last_clean":"2023-09-10T01:01:56.037030+0000","last_became_active":"2023-09-10T01:00:43.006166+0000","last_became_peered":"2023-09-10T01:00:43.006166+0000","last_unstale":"2023-09-10T01:01:56.037030+0000","last_undegraded":"2023-09-10T01:01:56.037030+0000","last_fullsized":"2023-09-10T01:01:56.037030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:52:32.167513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880315+0000","last_change":"2023-09-10T01:00:44.031264+0000","last_active":"2023-09-10T01:02:25.880315+0000","last_peered":"2023-09-10T01:02:25.880315+0000","last_clean":"2023-09-10T01:02:25.880315+0000","last_became_active":"2023-09-10T01:00:44.030993+0000","last_became_peered":"2023-09-10T01:00:44.030993+0000","last_unstale":"2023-09-10T01:02:25.880315+0000","last_undegraded":"2023-09-10T01:02:25.880315+0000","last_fullsized":"2023-09-10T01:02:25.880315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:19:27.658594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032496+0000","last_change":"2023-09-10T01:00:48.056886+0000","last_active":"2023-09-10T01:01:56.032496+0000","last_peered":"2023-09-10T01:01:56.032496+0000","last_clean":"2023-09-10T01:01:56.032496+0000","last_became_active":"2023-09-10T01:00:48.056411+0000","last_became_peered":"2023-09-10T01:00:48.056411+0000","last_unstale":"2023-09-10T01:01:56.032496+0000","last_undegraded":"2023-09-10T01:01:56.032496+0000","last_fullsized":"2023-09-10T01:01:56.032496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:56:42.062702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":283,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:57.111487+0000","last_change":"2023-09-10T01:00:43.009113+0000","last_active":"2023-09-10T01:02:57.111487+0000","last_peered":"2023-09-10T01:02:57.111487+0000","last_clean":"2023-09-10T01:02:57.111487+0000","last_became_active":"2023-09-10T01:00:43.008757+0000","last_became_peered":"2023-09-10T01:00:43.008757+0000","last_unstale":"2023-09-10T01:02:57.111487+0000","last_undegraded":"2023-09-10T01:02:57.111487+0000","last_fullsized":"2023-09-10T01:02:57.111487+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:29:24.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64774,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":226,"num_read_kb":177,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.870186+0000","last_change":"2023-09-10T01:00:44.027127+0000","last_active":"2023-09-10T01:02:25.870186+0000","last_peered":"2023-09-10T01:02:25.870186+0000","last_clean":"2023-09-10T01:02:25.870186+0000","last_became_active":"2023-09-10T01:00:44.026852+0000","last_became_peered":"2023-09-10T01:00:44.026852+0000","last_unstale":"2023-09-10T01:02:25.870186+0000","last_undegraded":"2023-09-10T01:02:25.870186+0000","last_fullsized":"2023-09-10T01:02:25.870186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:14:20.805300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037605+0000","last_change":"2023-09-10T01:00:48.069337+0000","last_active":"2023-09-10T01:01:56.037605+0000","last_peered":"2023-09-10T01:01:56.037605+0000","last_clean":"2023-09-10T01:01:56.037605+0000","last_became_active":"2023-09-10T01:00:48.068893+0000","last_became_peered":"2023-09-10T01:00:48.068893+0000","last_unstale":"2023-09-10T01:01:56.037605+0000","last_undegraded":"2023-09-10T01:01:56.037605+0000","last_fullsized":"2023-09-10T01:01:56.037605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:24:40.074551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033160+0000","last_change":"2023-09-10T01:00:43.010128+0000","last_active":"2023-09-10T01:01:56.033160+0000","last_peered":"2023-09-10T01:01:56.033160+0000","last_clean":"2023-09-10T01:01:56.033160+0000","last_became_active":"2023-09-10T01:00:43.009808+0000","last_became_peered":"2023-09-10T01:00:43.009808+0000","last_unstale":"2023-09-10T01:01:56.033160+0000","last_undegraded":"2023-09-10T01:01:56.033160+0000","last_fullsized":"2023-09-10T01:01:56.033160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:27.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891792+0000","last_change":"2023-09-10T01:00:44.038665+0000","last_active":"2023-09-10T01:02:25.891792+0000","last_peered":"2023-09-10T01:02:25.891792+0000","last_clean":"2023-09-10T01:02:25.891792+0000","last_became_active":"2023-09-10T01:00:44.037944+0000","last_became_peered":"2023-09-10T01:00:44.037944+0000","last_unstale":"2023-09-10T01:02:25.891792+0000","last_undegraded":"2023-09-10T01:02:25.891792+0000","last_fullsized":"2023-09-10T01:02:25.891792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:11:34.018249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127649+0000","last_change":"2023-09-10T01:00:56.965594+0000","last_active":"2023-09-10T01:01:56.127649+0000","last_peered":"2023-09-10T01:01:56.127649+0000","last_clean":"2023-09-10T01:01:56.127649+0000","last_became_active":"2023-09-10T01:00:56.965006+0000","last_became_peered":"2023-09-10T01:00:56.965006+0000","last_unstale":"2023-09-10T01:01:56.127649+0000","last_undegraded":"2023-09-10T01:01:56.127649+0000","last_fullsized":"2023-09-10T01:01:56.127649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:13:16.946697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036734+0000","last_change":"2023-09-10T01:00:43.004855+0000","last_active":"2023-09-10T01:01:56.036734+0000","last_peered":"2023-09-10T01:01:56.036734+0000","last_clean":"2023-09-10T01:01:56.036734+0000","last_became_active":"2023-09-10T01:00:43.004464+0000","last_became_peered":"2023-09-10T01:00:43.004464+0000","last_unstale":"2023-09-10T01:01:56.036734+0000","last_undegraded":"2023-09-10T01:01:56.036734+0000","last_fullsized":"2023-09-10T01:01:56.036734+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:29.517290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889936+0000","last_change":"2023-09-10T01:00:44.037273+0000","last_active":"2023-09-10T01:02:25.889936+0000","last_peered":"2023-09-10T01:02:25.889936+0000","last_clean":"2023-09-10T01:02:25.889936+0000","last_became_active":"2023-09-10T01:00:44.036965+0000","last_became_peered":"2023-09-10T01:00:44.036965+0000","last_unstale":"2023-09-10T01:02:25.889936+0000","last_undegraded":"2023-09-10T01:02:25.889936+0000","last_fullsized":"2023-09-10T01:02:25.889936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:47:20.149002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033203+0000","last_change":"2023-09-10T01:00:48.058574+0000","last_active":"2023-09-10T01:01:56.033203+0000","last_peered":"2023-09-10T01:01:56.033203+0000","last_clean":"2023-09-10T01:01:56.033203+0000","last_became_active":"2023-09-10T01:00:48.058190+0000","last_became_peered":"2023-09-10T01:00:48.058190+0000","last_unstale":"2023-09-10T01:01:56.033203+0000","last_undegraded":"2023-09-10T01:01:56.033203+0000","last_fullsized":"2023-09-10T01:01:56.033203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:46:42.177996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":134,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:55.146304+0000","last_change":"2023-09-10T01:00:43.003816+0000","last_active":"2023-09-10T01:02:55.146304+0000","last_peered":"2023-09-10T01:02:55.146304+0000","last_clean":"2023-09-10T01:02:55.146304+0000","last_became_active":"2023-09-10T01:00:43.003404+0000","last_became_peered":"2023-09-10T01:00:43.003404+0000","last_unstale":"2023-09-10T01:02:55.146304+0000","last_undegraded":"2023-09-10T01:02:55.146304+0000","last_fullsized":"2023-09-10T01:02:55.146304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:55.208302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878740+0000","last_change":"2023-09-10T01:00:44.034607+0000","last_active":"2023-09-10T01:02:25.878740+0000","last_peered":"2023-09-10T01:02:25.878740+0000","last_clean":"2023-09-10T01:02:25.878740+0000","last_became_active":"2023-09-10T01:00:44.034244+0000","last_became_peered":"2023-09-10T01:00:44.034244+0000","last_unstale":"2023-09-10T01:02:25.878740+0000","last_undegraded":"2023-09-10T01:02:25.878740+0000","last_fullsized":"2023-09-10T01:02:25.878740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:06:13.214160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031918+0000","last_change":"2023-09-10T01:00:48.058789+0000","last_active":"2023-09-10T01:01:56.031918+0000","last_peered":"2023-09-10T01:01:56.031918+0000","last_clean":"2023-09-10T01:01:56.031918+0000","last_became_active":"2023-09-10T01:00:48.058220+0000","last_became_peered":"2023-09-10T01:00:48.058220+0000","last_unstale":"2023-09-10T01:01:56.031918+0000","last_undegraded":"2023-09-10T01:01:56.031918+0000","last_fullsized":"2023-09-10T01:01:56.031918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:01:12.291739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036527+0000","last_change":"2023-09-10T01:00:43.006109+0000","last_active":"2023-09-10T01:01:56.036527+0000","last_peered":"2023-09-10T01:01:56.036527+0000","last_clean":"2023-09-10T01:01:56.036527+0000","last_became_active":"2023-09-10T01:00:43.005753+0000","last_became_peered":"2023-09-10T01:00:43.005753+0000","last_unstale":"2023-09-10T01:01:56.036527+0000","last_undegraded":"2023-09-10T01:01:56.036527+0000","last_fullsized":"2023-09-10T01:01:56.036527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:45:47.083991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.881751+0000","last_change":"2023-09-10T01:01:57.139294+0000","last_active":"2023-09-10T01:02:25.881751+0000","last_peered":"2023-09-10T01:02:25.881751+0000","last_clean":"2023-09-10T01:02:25.881751+0000","last_became_active":"2023-09-10T01:01:57.138930+0000","last_became_peered":"2023-09-10T01:01:57.138930+0000","last_unstale":"2023-09-10T01:02:25.881751+0000","last_undegraded":"2023-09-10T01:02:25.881751+0000","last_fullsized":"2023-09-10T01:02:25.881751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:21.143499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.043801+0000","last_change":"2023-09-10T01:00:48.069318+0000","last_active":"2023-09-10T01:01:57.043801+0000","last_peered":"2023-09-10T01:01:57.043801+0000","last_clean":"2023-09-10T01:01:57.043801+0000","last_became_active":"2023-09-10T01:00:48.068876+0000","last_became_peered":"2023-09-10T01:00:48.068876+0000","last_unstale":"2023-09-10T01:01:57.043801+0000","last_undegraded":"2023-09-10T01:01:57.043801+0000","last_fullsized":"2023-09-10T01:01:57.043801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:06:22.366530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036038+0000","last_change":"2023-09-10T01:00:43.004164+0000","last_active":"2023-09-10T01:01:56.036038+0000","last_peered":"2023-09-10T01:01:56.036038+0000","last_clean":"2023-09-10T01:01:56.036038+0000","last_became_active":"2023-09-10T01:00:43.003781+0000","last_became_peered":"2023-09-10T01:00:43.003781+0000","last_unstale":"2023-09-10T01:01:56.036038+0000","last_undegraded":"2023-09-10T01:01:56.036038+0000","last_fullsized":"2023-09-10T01:01:56.036038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:38:18.559712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872316+0000","last_change":"2023-09-10T01:00:44.028206+0000","last_active":"2023-09-10T01:02:25.872316+0000","last_peered":"2023-09-10T01:02:25.872316+0000","last_clean":"2023-09-10T01:02:25.872316+0000","last_became_active":"2023-09-10T01:00:44.027914+0000","last_became_peered":"2023-09-10T01:00:44.027914+0000","last_unstale":"2023-09-10T01:02:25.872316+0000","last_undegraded":"2023-09-10T01:02:25.872316+0000","last_fullsized":"2023-09-10T01:02:25.872316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:41:12.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.106476+0000","last_change":"2023-09-10T01:00:48.428613+0000","last_active":"2023-09-10T01:01:57.106476+0000","last_peered":"2023-09-10T01:01:57.106476+0000","last_clean":"2023-09-10T01:01:57.106476+0000","last_became_active":"2023-09-10T01:00:48.428059+0000","last_became_peered":"2023-09-10T01:00:48.428059+0000","last_unstale":"2023-09-10T01:01:57.106476+0000","last_undegraded":"2023-09-10T01:01:57.106476+0000","last_fullsized":"2023-09-10T01:01:57.106476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:46:03.683279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.046289+0000","last_change":"2023-09-10T01:00:43.006061+0000","last_active":"2023-09-10T01:01:57.046289+0000","last_peered":"2023-09-10T01:01:57.046289+0000","last_clean":"2023-09-10T01:01:57.046289+0000","last_became_active":"2023-09-10T01:00:43.005392+0000","last_became_peered":"2023-09-10T01:00:43.005392+0000","last_unstale":"2023-09-10T01:01:57.046289+0000","last_undegraded":"2023-09-10T01:01:57.046289+0000","last_fullsized":"2023-09-10T01:01:57.046289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:01:45.062338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891410+0000","last_change":"2023-09-10T01:00:44.026684+0000","last_active":"2023-09-10T01:02:25.891410+0000","last_peered":"2023-09-10T01:02:25.891410+0000","last_clean":"2023-09-10T01:02:25.891410+0000","last_became_active":"2023-09-10T01:00:44.026387+0000","last_became_peered":"2023-09-10T01:00:44.026387+0000","last_unstale":"2023-09-10T01:02:25.891410+0000","last_undegraded":"2023-09-10T01:02:25.891410+0000","last_fullsized":"2023-09-10T01:02:25.891410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:23:03.662543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.041259+0000","last_change":"2023-09-10T01:00:48.071589+0000","last_active":"2023-09-10T01:01:57.041259+0000","last_peered":"2023-09-10T01:01:57.041259+0000","last_clean":"2023-09-10T01:01:57.041259+0000","last_became_active":"2023-09-10T01:00:48.071118+0000","last_became_peered":"2023-09-10T01:00:48.071118+0000","last_unstale":"2023-09-10T01:01:57.041259+0000","last_undegraded":"2023-09-10T01:01:57.041259+0000","last_fullsized":"2023-09-10T01:01:57.041259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:27.194868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024640+0000","last_change":"2023-09-10T01:00:43.009738+0000","last_active":"2023-09-10T01:01:58.024640+0000","last_peered":"2023-09-10T01:01:58.024640+0000","last_clean":"2023-09-10T01:01:58.024640+0000","last_became_active":"2023-09-10T01:00:43.009376+0000","last_became_peered":"2023-09-10T01:00:43.009376+0000","last_unstale":"2023-09-10T01:01:58.024640+0000","last_undegraded":"2023-09-10T01:01:58.024640+0000","last_fullsized":"2023-09-10T01:01:58.024640+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:08.761672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882436+0000","last_change":"2023-09-10T01:00:44.030175+0000","last_active":"2023-09-10T01:02:25.882436+0000","last_peered":"2023-09-10T01:02:25.882436+0000","last_clean":"2023-09-10T01:02:25.882436+0000","last_became_active":"2023-09-10T01:00:44.029748+0000","last_became_peered":"2023-09-10T01:00:44.029748+0000","last_unstale":"2023-09-10T01:02:25.882436+0000","last_undegraded":"2023-09-10T01:02:25.882436+0000","last_fullsized":"2023-09-10T01:02:25.882436+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:51:15.149610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032036+0000","last_change":"2023-09-10T01:00:48.428008+0000","last_active":"2023-09-10T01:01:56.032036+0000","last_peered":"2023-09-10T01:01:56.032036+0000","last_clean":"2023-09-10T01:01:56.032036+0000","last_became_active":"2023-09-10T01:00:48.427498+0000","last_became_peered":"2023-09-10T01:00:48.427498+0000","last_unstale":"2023-09-10T01:01:56.032036+0000","last_undegraded":"2023-09-10T01:01:56.032036+0000","last_fullsized":"2023-09-10T01:01:56.032036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:16:48.938006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872721+0000","last_change":"2023-09-10T01:01:57.107918+0000","last_active":"2023-09-10T01:02:25.872721+0000","last_peered":"2023-09-10T01:02:25.872721+0000","last_clean":"2023-09-10T01:02:25.872721+0000","last_became_active":"2023-09-10T01:01:57.107413+0000","last_became_peered":"2023-09-10T01:01:57.107413+0000","last_unstale":"2023-09-10T01:02:25.872721+0000","last_undegraded":"2023-09-10T01:02:25.872721+0000","last_fullsized":"2023-09-10T01:02:25.872721+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:16:33.836871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":134,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:55.145098+0000","last_change":"2023-09-10T01:00:43.001047+0000","last_active":"2023-09-10T01:02:55.145098+0000","last_peered":"2023-09-10T01:02:55.145098+0000","last_clean":"2023-09-10T01:02:55.145098+0000","last_became_active":"2023-09-10T01:00:43.000521+0000","last_became_peered":"2023-09-10T01:00:43.000521+0000","last_unstale":"2023-09-10T01:02:55.145098+0000","last_undegraded":"2023-09-10T01:02:55.145098+0000","last_fullsized":"2023-09-10T01:02:55.145098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:15:34.509964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024504+0000","last_change":"2023-09-10T01:00:48.062148+0000","last_active":"2023-09-10T01:01:58.024504+0000","last_peered":"2023-09-10T01:01:58.024504+0000","last_clean":"2023-09-10T01:01:58.024504+0000","last_became_active":"2023-09-10T01:00:48.061746+0000","last_became_peered":"2023-09-10T01:00:48.061746+0000","last_unstale":"2023-09-10T01:01:58.024504+0000","last_undegraded":"2023-09-10T01:01:58.024504+0000","last_fullsized":"2023-09-10T01:01:58.024504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:40.087012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889845+0000","last_change":"2023-09-10T01:00:44.030500+0000","last_active":"2023-09-10T01:02:25.889845+0000","last_peered":"2023-09-10T01:02:25.889845+0000","last_clean":"2023-09-10T01:02:25.889845+0000","last_became_active":"2023-09-10T01:00:44.030209+0000","last_became_peered":"2023-09-10T01:00:44.030209+0000","last_unstale":"2023-09-10T01:02:25.889845+0000","last_undegraded":"2023-09-10T01:02:25.889845+0000","last_fullsized":"2023-09-10T01:02:25.889845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:53:30.421493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.040869+0000","last_change":"2023-09-10T01:00:43.001102+0000","last_active":"2023-09-10T01:01:57.040869+0000","last_peered":"2023-09-10T01:01:57.040869+0000","last_clean":"2023-09-10T01:01:57.040869+0000","last_became_active":"2023-09-10T01:00:43.000715+0000","last_became_peered":"2023-09-10T01:00:43.000715+0000","last_unstale":"2023-09-10T01:01:57.040869+0000","last_undegraded":"2023-09-10T01:01:57.040869+0000","last_fullsized":"2023-09-10T01:01:57.040869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:43:37.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":327,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024171+0000","last_change":"2023-09-10T00:59:40.261244+0000","last_active":"2023-09-10T01:01:58.024171+0000","last_peered":"2023-09-10T01:01:58.024171+0000","last_clean":"2023-09-10T01:01:58.024171+0000","last_became_active":"2023-09-10T00:59:40.260673+0000","last_became_peered":"2023-09-10T00:59:40.260673+0000","last_unstale":"2023-09-10T01:01:58.024171+0000","last_undegraded":"2023-09-10T01:01:58.024171+0000","last_fullsized":"2023-09-10T01:01:58.024171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_clean_scrub_stamp":"2023-09-10T00:56:36.374323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:04:03.614104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.044220+0000","last_change":"2023-09-10T01:00:57.052019+0000","last_active":"2023-09-10T01:01:57.044220+0000","last_peered":"2023-09-10T01:01:57.044220+0000","last_clean":"2023-09-10T01:01:57.044220+0000","last_became_active":"2023-09-10T01:00:57.051527+0000","last_became_peered":"2023-09-10T01:00:57.051527+0000","last_unstale":"2023-09-10T01:01:57.044220+0000","last_undegraded":"2023-09-10T01:01:57.044220+0000","last_fullsized":"2023-09-10T01:01:57.044220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:10:43.756761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889771+0000","last_change":"2023-09-10T01:00:44.033214+0000","last_active":"2023-09-10T01:02:25.889771+0000","last_peered":"2023-09-10T01:02:25.889771+0000","last_clean":"2023-09-10T01:02:25.889771+0000","last_became_active":"2023-09-10T01:00:44.032828+0000","last_became_peered":"2023-09-10T01:00:44.032828+0000","last_unstale":"2023-09-10T01:02:25.889771+0000","last_undegraded":"2023-09-10T01:02:25.889771+0000","last_fullsized":"2023-09-10T01:02:25.889771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:01:22.417216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024928+0000","last_change":"2023-09-10T01:00:56.965456+0000","last_active":"2023-09-10T01:01:58.024928+0000","last_peered":"2023-09-10T01:01:58.024928+0000","last_clean":"2023-09-10T01:01:58.024928+0000","last_became_active":"2023-09-10T01:00:56.965021+0000","last_became_peered":"2023-09-10T01:00:56.965021+0000","last_unstale":"2023-09-10T01:01:58.024928+0000","last_undegraded":"2023-09-10T01:01:58.024928+0000","last_fullsized":"2023-09-10T01:01:58.024928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:19.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.731761+0000","last_change":"2023-09-10T01:00:48.066811+0000","last_active":"2023-09-10T01:01:58.731761+0000","last_peered":"2023-09-10T01:01:58.731761+0000","last_clean":"2023-09-10T01:01:58.731761+0000","last_became_active":"2023-09-10T01:00:48.066421+0000","last_became_peered":"2023-09-10T01:00:48.066421+0000","last_unstale":"2023-09-10T01:01:58.731761+0000","last_undegraded":"2023-09-10T01:01:58.731761+0000","last_fullsized":"2023-09-10T01:01:58.731761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:48:01.459070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.106555+0000","last_change":"2023-09-10T01:00:43.011134+0000","last_active":"2023-09-10T01:01:57.106555+0000","last_peered":"2023-09-10T01:01:57.106555+0000","last_clean":"2023-09-10T01:01:57.106555+0000","last_became_active":"2023-09-10T01:00:43.010871+0000","last_became_peered":"2023-09-10T01:00:43.010871+0000","last_unstale":"2023-09-10T01:01:57.106555+0000","last_undegraded":"2023-09-10T01:01:57.106555+0000","last_fullsized":"2023-09-10T01:01:57.106555+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:56.903745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890244+0000","last_change":"2023-09-10T01:00:44.031104+0000","last_active":"2023-09-10T01:02:25.890244+0000","last_peered":"2023-09-10T01:02:25.890244+0000","last_clean":"2023-09-10T01:02:25.890244+0000","last_became_active":"2023-09-10T01:00:44.030745+0000","last_became_peered":"2023-09-10T01:00:44.030745+0000","last_unstale":"2023-09-10T01:02:25.890244+0000","last_undegraded":"2023-09-10T01:02:25.890244+0000","last_fullsized":"2023-09-10T01:02:25.890244+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:53.885886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.470456+0000","last_change":"2023-09-10T01:00:48.427735+0000","last_active":"2023-09-10T01:01:57.470456+0000","last_peered":"2023-09-10T01:01:57.470456+0000","last_clean":"2023-09-10T01:01:57.470456+0000","last_became_active":"2023-09-10T01:00:48.427342+0000","last_became_peered":"2023-09-10T01:00:48.427342+0000","last_unstale":"2023-09-10T01:01:57.470456+0000","last_undegraded":"2023-09-10T01:01:57.470456+0000","last_fullsized":"2023-09-10T01:01:57.470456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:01:52.007325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:53.813562+0000","last_change":"2023-09-10T01:00:43.006314+0000","last_active":"2023-09-10T01:02:53.813562+0000","last_peered":"2023-09-10T01:02:53.813562+0000","last_clean":"2023-09-10T01:02:53.813562+0000","last_became_active":"2023-09-10T01:00:43.004835+0000","last_became_peered":"2023-09-10T01:00:43.004835+0000","last_unstale":"2023-09-10T01:02:53.813562+0000","last_undegraded":"2023-09-10T01:02:53.813562+0000","last_fullsized":"2023-09-10T01:02:53.813562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:37:09.359846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891331+0000","last_change":"2023-09-10T01:00:44.029839+0000","last_active":"2023-09-10T01:02:25.891331+0000","last_peered":"2023-09-10T01:02:25.891331+0000","last_clean":"2023-09-10T01:02:25.891331+0000","last_became_active":"2023-09-10T01:00:44.029492+0000","last_became_peered":"2023-09-10T01:00:44.029492+0000","last_unstale":"2023-09-10T01:02:25.891331+0000","last_undegraded":"2023-09-10T01:02:25.891331+0000","last_fullsized":"2023-09-10T01:02:25.891331+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:14.357386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.047012+0000","last_change":"2023-09-10T01:00:48.054536+0000","last_active":"2023-09-10T01:01:57.047012+0000","last_peered":"2023-09-10T01:01:57.047012+0000","last_clean":"2023-09-10T01:01:57.047012+0000","last_became_active":"2023-09-10T01:00:48.053938+0000","last_became_peered":"2023-09-10T01:00:48.053938+0000","last_unstale":"2023-09-10T01:01:57.047012+0000","last_undegraded":"2023-09-10T01:01:57.047012+0000","last_fullsized":"2023-09-10T01:01:57.047012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:45:12.703592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.041448+0000","last_change":"2023-09-10T01:00:43.002313+0000","last_active":"2023-09-10T01:01:57.041448+0000","last_peered":"2023-09-10T01:01:57.041448+0000","last_clean":"2023-09-10T01:01:57.041448+0000","last_became_active":"2023-09-10T01:00:43.001839+0000","last_became_peered":"2023-09-10T01:00:43.001839+0000","last_unstale":"2023-09-10T01:01:57.041448+0000","last_undegraded":"2023-09-10T01:01:57.041448+0000","last_fullsized":"2023-09-10T01:01:57.041448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:51.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880327+0000","last_change":"2023-09-10T01:01:57.108269+0000","last_active":"2023-09-10T01:02:25.880327+0000","last_peered":"2023-09-10T01:02:25.880327+0000","last_clean":"2023-09-10T01:02:25.880327+0000","last_became_active":"2023-09-10T01:01:57.107879+0000","last_became_peered":"2023-09-10T01:01:57.107879+0000","last_unstale":"2023-09-10T01:02:25.880327+0000","last_undegraded":"2023-09-10T01:02:25.880327+0000","last_fullsized":"2023-09-10T01:02:25.880327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:47:09.135978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037126+0000","last_change":"2023-09-10T01:00:48.428406+0000","last_active":"2023-09-10T01:01:56.037126+0000","last_peered":"2023-09-10T01:01:56.037126+0000","last_clean":"2023-09-10T01:01:56.037126+0000","last_became_active":"2023-09-10T01:00:48.427938+0000","last_became_peered":"2023-09-10T01:00:48.427938+0000","last_unstale":"2023-09-10T01:01:56.037126+0000","last_undegraded":"2023-09-10T01:01:56.037126+0000","last_fullsized":"2023-09-10T01:01:56.037126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:52.316235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.138071+0000","last_change":"2023-09-10T01:00:56.963755+0000","last_active":"2023-09-10T01:01:57.138071+0000","last_peered":"2023-09-10T01:01:57.138071+0000","last_clean":"2023-09-10T01:01:57.138071+0000","last_became_active":"2023-09-10T01:00:56.963283+0000","last_became_peered":"2023-09-10T01:00:56.963283+0000","last_unstale":"2023-09-10T01:01:57.138071+0000","last_undegraded":"2023-09-10T01:01:57.138071+0000","last_fullsized":"2023-09-10T01:01:57.138071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:13:32.213393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.864057+0000","last_change":"2023-09-10T01:00:44.025284+0000","last_active":"2023-09-10T01:02:25.864057+0000","last_peered":"2023-09-10T01:02:25.864057+0000","last_clean":"2023-09-10T01:02:25.864057+0000","last_became_active":"2023-09-10T01:00:44.024965+0000","last_became_peered":"2023-09-10T01:00:44.024965+0000","last_unstale":"2023-09-10T01:02:25.864057+0000","last_undegraded":"2023-09-10T01:02:25.864057+0000","last_fullsized":"2023-09-10T01:02:25.864057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:19:08.620325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033359+0000","last_change":"2023-09-10T01:00:48.070843+0000","last_active":"2023-09-10T01:01:56.033359+0000","last_peered":"2023-09-10T01:01:56.033359+0000","last_clean":"2023-09-10T01:01:56.033359+0000","last_became_active":"2023-09-10T01:00:48.070350+0000","last_became_peered":"2023-09-10T01:00:48.070350+0000","last_unstale":"2023-09-10T01:01:56.033359+0000","last_undegraded":"2023-09-10T01:01:56.033359+0000","last_fullsized":"2023-09-10T01:01:56.033359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:35:25.196036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.470530+0000","last_change":"2023-09-10T01:00:43.005398+0000","last_active":"2023-09-10T01:01:57.470530+0000","last_peered":"2023-09-10T01:01:57.470530+0000","last_clean":"2023-09-10T01:01:57.470530+0000","last_became_active":"2023-09-10T01:00:43.005087+0000","last_became_peered":"2023-09-10T01:00:43.005087+0000","last_unstale":"2023-09-10T01:01:57.470530+0000","last_undegraded":"2023-09-10T01:01:57.470530+0000","last_fullsized":"2023-09-10T01:01:57.470530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:29:04.042683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871590+0000","last_change":"2023-09-10T01:00:44.024451+0000","last_active":"2023-09-10T01:02:25.871590+0000","last_peered":"2023-09-10T01:02:25.871590+0000","last_clean":"2023-09-10T01:02:25.871590+0000","last_became_active":"2023-09-10T01:00:44.024045+0000","last_became_peered":"2023-09-10T01:00:44.024045+0000","last_unstale":"2023-09-10T01:02:25.871590+0000","last_undegraded":"2023-09-10T01:02:25.871590+0000","last_fullsized":"2023-09-10T01:02:25.871590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:55:50.750994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037240+0000","last_change":"2023-09-10T01:00:48.062247+0000","last_active":"2023-09-10T01:01:56.037240+0000","last_peered":"2023-09-10T01:01:56.037240+0000","last_clean":"2023-09-10T01:01:56.037240+0000","last_became_active":"2023-09-10T01:00:48.061812+0000","last_became_peered":"2023-09-10T01:00:48.061812+0000","last_unstale":"2023-09-10T01:01:56.037240+0000","last_undegraded":"2023-09-10T01:01:56.037240+0000","last_fullsized":"2023-09-10T01:01:56.037240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:44:13.596534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.047127+0000","last_change":"2023-09-10T01:00:43.010845+0000","last_active":"2023-09-10T01:01:57.047127+0000","last_peered":"2023-09-10T01:01:57.047127+0000","last_clean":"2023-09-10T01:01:57.047127+0000","last_became_active":"2023-09-10T01:00:43.010511+0000","last_became_peered":"2023-09-10T01:00:43.010511+0000","last_unstale":"2023-09-10T01:01:57.047127+0000","last_undegraded":"2023-09-10T01:01:57.047127+0000","last_fullsized":"2023-09-10T01:01:57.047127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:33:07.229023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891545+0000","last_change":"2023-09-10T01:00:44.029025+0000","last_active":"2023-09-10T01:02:25.891545+0000","last_peered":"2023-09-10T01:02:25.891545+0000","last_clean":"2023-09-10T01:02:25.891545+0000","last_became_active":"2023-09-10T01:00:44.028713+0000","last_became_peered":"2023-09-10T01:00:44.028713+0000","last_unstale":"2023-09-10T01:02:25.891545+0000","last_undegraded":"2023-09-10T01:02:25.891545+0000","last_fullsized":"2023-09-10T01:02:25.891545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:40:22.315742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037817+0000","last_change":"2023-09-10T01:00:48.427294+0000","last_active":"2023-09-10T01:01:56.037817+0000","last_peered":"2023-09-10T01:01:56.037817+0000","last_clean":"2023-09-10T01:01:56.037817+0000","last_became_active":"2023-09-10T01:00:48.426882+0000","last_became_peered":"2023-09-10T01:00:48.426882+0000","last_unstale":"2023-09-10T01:01:56.037817+0000","last_undegraded":"2023-09-10T01:01:56.037817+0000","last_fullsized":"2023-09-10T01:01:56.037817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:54:38.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035921+0000","last_change":"2023-09-10T01:00:43.006175+0000","last_active":"2023-09-10T01:01:56.035921+0000","last_peered":"2023-09-10T01:01:56.035921+0000","last_clean":"2023-09-10T01:01:56.035921+0000","last_became_active":"2023-09-10T01:00:43.005786+0000","last_became_peered":"2023-09-10T01:00:43.005786+0000","last_unstale":"2023-09-10T01:01:56.035921+0000","last_undegraded":"2023-09-10T01:01:56.035921+0000","last_fullsized":"2023-09-10T01:01:56.035921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:14:47.074866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887993+0000","last_change":"2023-09-10T01:00:44.041303+0000","last_active":"2023-09-10T01:02:25.887993+0000","last_peered":"2023-09-10T01:02:25.887993+0000","last_clean":"2023-09-10T01:02:25.887993+0000","last_became_active":"2023-09-10T01:00:44.041009+0000","last_became_peered":"2023-09-10T01:00:44.041009+0000","last_unstale":"2023-09-10T01:02:25.887993+0000","last_undegraded":"2023-09-10T01:02:25.887993+0000","last_fullsized":"2023-09-10T01:02:25.887993+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:53:23.001700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036842+0000","last_change":"2023-09-10T01:00:48.064314+0000","last_active":"2023-09-10T01:01:56.036842+0000","last_peered":"2023-09-10T01:01:56.036842+0000","last_clean":"2023-09-10T01:01:56.036842+0000","last_became_active":"2023-09-10T01:00:48.063813+0000","last_became_peered":"2023-09-10T01:00:48.063813+0000","last_unstale":"2023-09-10T01:01:56.036842+0000","last_undegraded":"2023-09-10T01:01:56.036842+0000","last_fullsized":"2023-09-10T01:01:56.036842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:00:10.002006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033268+0000","last_change":"2023-09-10T01:00:43.007579+0000","last_active":"2023-09-10T01:01:56.033268+0000","last_peered":"2023-09-10T01:01:56.033268+0000","last_clean":"2023-09-10T01:01:56.033268+0000","last_became_active":"2023-09-10T01:00:43.007200+0000","last_became_peered":"2023-09-10T01:00:43.007200+0000","last_unstale":"2023-09-10T01:01:56.033268+0000","last_undegraded":"2023-09-10T01:01:56.033268+0000","last_fullsized":"2023-09-10T01:01:56.033268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:20:08.688531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883972+0000","last_change":"2023-09-10T01:00:44.033511+0000","last_active":"2023-09-10T01:02:25.883972+0000","last_peered":"2023-09-10T01:02:25.883972+0000","last_clean":"2023-09-10T01:02:25.883972+0000","last_became_active":"2023-09-10T01:00:44.033170+0000","last_became_peered":"2023-09-10T01:00:44.033170+0000","last_unstale":"2023-09-10T01:02:25.883972+0000","last_undegraded":"2023-09-10T01:02:25.883972+0000","last_fullsized":"2023-09-10T01:02:25.883972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:36:12.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039255+0000","last_change":"2023-09-10T01:00:48.067436+0000","last_active":"2023-09-10T01:01:56.039255+0000","last_peered":"2023-09-10T01:01:56.039255+0000","last_clean":"2023-09-10T01:01:56.039255+0000","last_became_active":"2023-09-10T01:00:48.067038+0000","last_became_peered":"2023-09-10T01:00:48.067038+0000","last_unstale":"2023-09-10T01:01:56.039255+0000","last_undegraded":"2023-09-10T01:01:56.039255+0000","last_fullsized":"2023-09-10T01:01:56.039255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:26:41.210698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.044273+0000","last_change":"2023-09-10T01:00:43.006934+0000","last_active":"2023-09-10T01:01:57.044273+0000","last_peered":"2023-09-10T01:01:57.044273+0000","last_clean":"2023-09-10T01:01:57.044273+0000","last_became_active":"2023-09-10T01:00:43.005787+0000","last_became_peered":"2023-09-10T01:00:43.005787+0000","last_unstale":"2023-09-10T01:01:57.044273+0000","last_undegraded":"2023-09-10T01:01:57.044273+0000","last_fullsized":"2023-09-10T01:01:57.044273+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:48:27.403220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879574+0000","last_change":"2023-09-10T01:00:44.033812+0000","last_active":"2023-09-10T01:02:25.879574+0000","last_peered":"2023-09-10T01:02:25.879574+0000","last_clean":"2023-09-10T01:02:25.879574+0000","last_became_active":"2023-09-10T01:00:44.033496+0000","last_became_peered":"2023-09-10T01:00:44.033496+0000","last_unstale":"2023-09-10T01:02:25.879574+0000","last_undegraded":"2023-09-10T01:02:25.879574+0000","last_fullsized":"2023-09-10T01:02:25.879574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:12.679368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.043542+0000","last_change":"2023-09-10T01:00:48.060358+0000","last_active":"2023-09-10T01:01:57.043542+0000","last_peered":"2023-09-10T01:01:57.043542+0000","last_clean":"2023-09-10T01:01:57.043542+0000","last_became_active":"2023-09-10T01:00:48.059853+0000","last_became_peered":"2023-09-10T01:00:48.059853+0000","last_unstale":"2023-09-10T01:01:57.043542+0000","last_undegraded":"2023-09-10T01:01:57.043542+0000","last_fullsized":"2023-09-10T01:01:57.043542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:31:56.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031904+0000","last_change":"2023-09-10T01:00:43.002820+0000","last_active":"2023-09-10T01:01:56.031904+0000","last_peered":"2023-09-10T01:01:56.031904+0000","last_clean":"2023-09-10T01:01:56.031904+0000","last_became_active":"2023-09-10T01:00:43.002456+0000","last_became_peered":"2023-09-10T01:00:43.002456+0000","last_unstale":"2023-09-10T01:01:56.031904+0000","last_undegraded":"2023-09-10T01:01:56.031904+0000","last_fullsized":"2023-09-10T01:01:56.031904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:40:10.189942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879728+0000","last_change":"2023-09-10T01:00:44.032655+0000","last_active":"2023-09-10T01:02:25.879728+0000","last_peered":"2023-09-10T01:02:25.879728+0000","last_clean":"2023-09-10T01:02:25.879728+0000","last_became_active":"2023-09-10T01:00:44.032157+0000","last_became_peered":"2023-09-10T01:00:44.032157+0000","last_unstale":"2023-09-10T01:02:25.879728+0000","last_undegraded":"2023-09-10T01:02:25.879728+0000","last_fullsized":"2023-09-10T01:02:25.879728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:31.176768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872475+0000","last_change":"2023-09-10T01:00:44.022285+0000","last_active":"2023-09-10T01:02:25.872475+0000","last_peered":"2023-09-10T01:02:25.872475+0000","last_clean":"2023-09-10T01:02:25.872475+0000","last_became_active":"2023-09-10T01:00:44.021959+0000","last_became_peered":"2023-09-10T01:00:44.021959+0000","last_unstale":"2023-09-10T01:02:25.872475+0000","last_undegraded":"2023-09-10T01:02:25.872475+0000","last_fullsized":"2023-09-10T01:02:25.872475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:13:07.072249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.873581+0000","last_change":"2023-09-10T01:00:44.029722+0000","last_active":"2023-09-10T01:02:25.873581+0000","last_peered":"2023-09-10T01:02:25.873581+0000","last_clean":"2023-09-10T01:02:25.873581+0000","last_became_active":"2023-09-10T01:00:44.029424+0000","last_became_peered":"2023-09-10T01:00:44.029424+0000","last_unstale":"2023-09-10T01:02:25.873581+0000","last_undegraded":"2023-09-10T01:02:25.873581+0000","last_fullsized":"2023-09-10T01:02:25.873581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:24:16.655538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883202+0000","last_change":"2023-09-10T01:00:44.026026+0000","last_active":"2023-09-10T01:02:25.883202+0000","last_peered":"2023-09-10T01:02:25.883202+0000","last_clean":"2023-09-10T01:02:25.883202+0000","last_became_active":"2023-09-10T01:00:44.025727+0000","last_became_peered":"2023-09-10T01:00:44.025727+0000","last_unstale":"2023-09-10T01:02:25.883202+0000","last_undegraded":"2023-09-10T01:02:25.883202+0000","last_fullsized":"2023-09-10T01:02:25.883202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:59.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890507+0000","last_change":"2023-09-10T01:00:56.965355+0000","last_active":"2023-09-10T01:02:25.890507+0000","last_peered":"2023-09-10T01:02:25.890507+0000","last_clean":"2023-09-10T01:02:25.890507+0000","last_became_active":"2023-09-10T01:00:56.964903+0000","last_became_peered":"2023-09-10T01:00:56.964903+0000","last_unstale":"2023-09-10T01:02:25.890507+0000","last_undegraded":"2023-09-10T01:02:25.890507+0000","last_fullsized":"2023-09-10T01:02:25.890507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:49:21.940159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890130+0000","last_change":"2023-09-10T01:00:44.029525+0000","last_active":"2023-09-10T01:02:25.890130+0000","last_peered":"2023-09-10T01:02:25.890130+0000","last_clean":"2023-09-10T01:02:25.890130+0000","last_became_active":"2023-09-10T01:00:44.029139+0000","last_became_peered":"2023-09-10T01:00:44.029139+0000","last_unstale":"2023-09-10T01:02:25.890130+0000","last_undegraded":"2023-09-10T01:02:25.890130+0000","last_fullsized":"2023-09-10T01:02:25.890130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:20:24.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871853+0000","last_change":"2023-09-10T01:00:44.022299+0000","last_active":"2023-09-10T01:02:25.871853+0000","last_peered":"2023-09-10T01:02:25.871853+0000","last_clean":"2023-09-10T01:02:25.871853+0000","last_became_active":"2023-09-10T01:00:44.021992+0000","last_became_peered":"2023-09-10T01:00:44.021992+0000","last_unstale":"2023-09-10T01:02:25.871853+0000","last_undegraded":"2023-09-10T01:02:25.871853+0000","last_fullsized":"2023-09-10T01:02:25.871853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:50:30.766442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.870556+0000","last_change":"2023-09-10T01:01:57.108199+0000","last_active":"2023-09-10T01:02:25.870556+0000","last_peered":"2023-09-10T01:02:25.870556+0000","last_clean":"2023-09-10T01:02:25.870556+0000","last_became_active":"2023-09-10T01:01:57.107825+0000","last_became_peered":"2023-09-10T01:01:57.107825+0000","last_unstale":"2023-09-10T01:02:25.870556+0000","last_undegraded":"2023-09-10T01:02:25.870556+0000","last_fullsized":"2023-09-10T01:02:25.870556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:31:15.499277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,9],"acting":[2,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.876680+0000","last_change":"2023-09-10T01:01:57.049936+0000","last_active":"2023-09-10T01:02:25.876680+0000","last_peered":"2023-09-10T01:02:25.876680+0000","last_clean":"2023-09-10T01:02:25.876680+0000","last_became_active":"2023-09-10T01:01:57.049624+0000","last_became_peered":"2023-09-10T01:01:57.049624+0000","last_unstale":"2023-09-10T01:02:25.876680+0000","last_undegraded":"2023-09-10T01:02:25.876680+0000","last_fullsized":"2023-09-10T01:02:25.876680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:57:17.164558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887999+0000","last_change":"2023-09-10T01:00:44.025405+0000","last_active":"2023-09-10T01:02:25.887999+0000","last_peered":"2023-09-10T01:02:25.887999+0000","last_clean":"2023-09-10T01:02:25.887999+0000","last_became_active":"2023-09-10T01:00:44.025040+0000","last_became_peered":"2023-09-10T01:00:44.025040+0000","last_unstale":"2023-09-10T01:02:25.887999+0000","last_undegraded":"2023-09-10T01:02:25.887999+0000","last_fullsized":"2023-09-10T01:02:25.887999+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:16:46.504902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887060+0000","last_change":"2023-09-10T01:00:44.023603+0000","last_active":"2023-09-10T01:02:25.887060+0000","last_peered":"2023-09-10T01:02:25.887060+0000","last_clean":"2023-09-10T01:02:25.887060+0000","last_became_active":"2023-09-10T01:00:44.023279+0000","last_became_peered":"2023-09-10T01:00:44.023279+0000","last_unstale":"2023-09-10T01:02:25.887060+0000","last_undegraded":"2023-09-10T01:02:25.887060+0000","last_fullsized":"2023-09-10T01:02:25.887060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:18:05.737675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.892481+0000","last_change":"2023-09-10T01:00:44.035653+0000","last_active":"2023-09-10T01:02:25.892481+0000","last_peered":"2023-09-10T01:02:25.892481+0000","last_clean":"2023-09-10T01:02:25.892481+0000","last_became_active":"2023-09-10T01:00:44.035374+0000","last_became_peered":"2023-09-10T01:00:44.035374+0000","last_unstale":"2023-09-10T01:02:25.892481+0000","last_undegraded":"2023-09-10T01:02:25.892481+0000","last_fullsized":"2023-09-10T01:02:25.892481+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:14:45.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890327+0000","last_change":"2023-09-10T01:00:44.035805+0000","last_active":"2023-09-10T01:02:25.890327+0000","last_peered":"2023-09-10T01:02:25.890327+0000","last_clean":"2023-09-10T01:02:25.890327+0000","last_became_active":"2023-09-10T01:00:44.035479+0000","last_became_peered":"2023-09-10T01:00:44.035479+0000","last_unstale":"2023-09-10T01:02:25.890327+0000","last_undegraded":"2023-09-10T01:02:25.890327+0000","last_fullsized":"2023-09-10T01:02:25.890327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:36.662975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878041+0000","last_change":"2023-09-10T01:00:44.025418+0000","last_active":"2023-09-10T01:02:25.878041+0000","last_peered":"2023-09-10T01:02:25.878041+0000","last_clean":"2023-09-10T01:02:25.878041+0000","last_became_active":"2023-09-10T01:00:44.025104+0000","last_became_peered":"2023-09-10T01:00:44.025104+0000","last_unstale":"2023-09-10T01:02:25.878041+0000","last_undegraded":"2023-09-10T01:02:25.878041+0000","last_fullsized":"2023-09-10T01:02:25.878041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:05:13.050781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282844,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69231,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":677,"num_read_kb":413,"num_write":166,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":207693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481830,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":504243,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69199999999999995,"5min":0.87,"15min":0.87},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.60799999999999998,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.499}]},{"osd":1,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.626,"15min":0.625},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82299999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53100000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45500000000000002,"15min":0.45600000000000002},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.74399999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.53000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.74399999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.75}]},{"osd":6,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81799999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.69799999999999995},"min":{"1min":0.495,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.83299999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63600000000000001}]},{"osd":7,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69399999999999995,"15min":0.69199999999999995},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.85599999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.85899999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.46200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.74299999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.748,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.92300000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.873}]},{"osd":10,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93100000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.55900000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80100000000000005}]}]},{"osd":10,"up_from":70,"seq":300647710762,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1156019,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.65900000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55900000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.441,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58099999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.47199999999999998,"15min":0.47499999999999998},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.499}]},{"osd":2,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60699999999999998,"15min":0.60499999999999998},"min":{"1min":0.432,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80700000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61799999999999999,"15min":0.629},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60499999999999998,"15min":0.626},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.753,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.629},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.78900000000000003,"15min":0.85599999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78000000000000003,"5min":6.7519999999999998,"15min":6.7519999999999998},"last":0.67800000000000005}]},{"osd":4,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71299999999999997,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.36299999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.65900000000000003}]},{"osd":5,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.69499999999999995},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.99099999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.42199999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85799999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.87,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91300000000000003,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.82899999999999996}]},{"osd":7,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.76400000000000001,"15min":0.76800000000000002},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.3069999999999999,"5min":1.534,"15min":1.534},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75700000000000001,"15min":0.77500000000000002},"min":{"1min":0.59099999999999997,"5min":0.439,"15min":0.439},"max":{"1min":0.88800000000000001,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.79500000000000004}]},{"osd":9,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.58399999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.93200000000000005,"5min":1.304,"15min":1.304},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84899999999999998,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94299999999999995,"5min":7.0510000000000002,"15min":7.0510000000000002},"last":0.92100000000000004}]},{"osd":11,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73299999999999998,"15min":0.73099999999999998},"min":{"1min":0.50900000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93600000000000005,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76400000000000001,"15min":0.75800000000000001},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.84499999999999997}]}]},{"osd":9,"up_from":65,"seq":279172874287,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":507993,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55800000000000005,"15min":0.58399999999999996},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80200000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67500000000000004,"15min":0.68700000000000006},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":1.0069999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.73599999999999999},"min":{"1min":0.41199999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86499999999999999,"5min":1.349,"15min":1.349},"last":0.73499999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71299999999999997,"15min":0.70999999999999996},"min":{"1min":0.57199999999999995,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.069,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.90200000000000002}]},{"osd":3,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68300000000000005,"15min":0.70599999999999996},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.0860000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.751},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.42499999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.2729999999999999,"5min":1.647,"15min":1.647},"last":0.61299999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.80300000000000005,"15min":0.83599999999999997},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.369,"5min":1.595,"15min":1.595},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.85899999999999999},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0129999999999999,"5min":1.492,"15min":1.492},"last":0.79200000000000004}]},{"osd":5,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.88400000000000001,"15min":0.92700000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1890000000000001,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.873},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3460000000000001,"5min":1.714,"15min":1.714},"last":1.01}]},{"osd":6,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.97199999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.319,"5min":1.982,"15min":1.982},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.88900000000000001,"15min":0.92500000000000004},"min":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4330000000000001,"5min":1.569,"15min":1.569},"last":0.93100000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.94699999999999995,"15min":0.999},"min":{"1min":0.54500000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.294,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.97099999999999997,"15min":1.0109999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1759999999999999,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.96699999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.92200000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93300000000000005,"15min":0.94999999999999996},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.996}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.87,"15min":0.85699999999999998},"min":{"1min":0.69099999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84399999999999997,"15min":0.84199999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":1.0549999999999999}]}]},{"osd":8,"up_from":57,"seq":244813135924,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":503486,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80200000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68700000000000006,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51800000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86499999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.878,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.39400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.47399999999999998,"15min":0.45400000000000001},"min":{"1min":0.35999999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.73099999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.34499999999999997}]},{"osd":3,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.69099999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.624}]},{"osd":4,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.56100000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97199999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82799999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.66800000000000004}]},{"osd":5,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.60399999999999998,"15min":0.58799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995}]},{"osd":6,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.52100000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73999999999999999,"15min":0.73299999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.004,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.70499999999999996,"15min":0.67000000000000004},"min":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.72899999999999998,"15min":0.70899999999999996},"min":{"1min":0.60099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":9,"last update":"Sun Sep 10 01:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78900000000000003,"15min":0.76700000000000002},"min":{"1min":0.67100000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.78400000000000003,"15min":0.76400000000000001},"min":{"1min":0.63400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0249999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.81399999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.58499999999999996,"5min":0.44,"15min":0.44},"max":{"1min":0.99199999999999999,"5min":1.01,"15min":1.01},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.79000000000000004,"15min":0.751},"min":{"1min":0.65900000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.75}]},{"osd":11,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":1.071,"15min":1.071},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83399999999999996,"15min":0.83599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78900000000000003}]}]},{"osd":7,"up_from":50,"seq":214748364856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1099600,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91400000000000003,"5min":1.367,"15min":1.367},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.44400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.42099999999999999},"min":{"1min":0.34100000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.53000000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.378},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70299999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41999999999999998}]},{"osd":2,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.47699999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.82399999999999995,"5min":1.101,"15min":1.101},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80200000000000005,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63300000000000001,"15min":0.65300000000000002},"min":{"1min":0.42299999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.873,"5min":1.409,"15min":1.409},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.70999999999999996,"15min":0.68100000000000005},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88600000000000001,"5min":7.1929999999999996,"15min":7.1929999999999996},"last":0.51300000000000001}]},{"osd":4,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68200000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54500000000000004,"15min":0.53700000000000003},"min":{"1min":0.33000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82099999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.44900000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.79400000000000004,"5min":1.038,"15min":1.038},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88900000000000001,"5min":1.179,"15min":1.179},"last":0.75900000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.81299999999999994,"15min":0.75700000000000001},"min":{"1min":0.44500000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84799999999999998,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.80800000000000005,"15min":0.78400000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95099999999999996,"5min":7.1760000000000002,"15min":7.1760000000000002},"last":0.66600000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.012,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70099999999999996,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98399999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.64000000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.79600000000000004,"15min":0.78000000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":3.423,"5min":3.423,"15min":3.423},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.81699999999999995,"15min":0.79500000000000004},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.5249999999999999,"5min":3.5249999999999999,"15min":3.5249999999999999},"last":0.71999999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.746},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92900000000000005,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.62}]},{"osd":11,"last update":"Sun Sep 10 01:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.034,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85799999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0780000000000001,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.83499999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528380,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":499073,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.65100000000000002,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.9890000000000008,"5min":9.9890000000000008,"15min":9.9890000000000008},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.59099999999999997,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.363,"5min":2.363,"15min":2.363},"last":0.54300000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.67600000000000005,"15min":0.63100000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.577,"5min":0.89000000000000001,"15min":0.76800000000000002},"min":{"1min":0.50800000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.77900000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.748,"15min":0.72899999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.327,"5min":0.84899999999999998,"15min":0.78700000000000003},"min":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":10.608000000000001,"5min":10.608000000000001,"15min":10.608000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.67300000000000004,"15min":0.64900000000000002},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.9750000000000001,"5min":0.96899999999999997,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":23.449999999999999,"5min":23.449999999999999,"15min":23.449999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.76700000000000002,"15min":0.75},"min":{"1min":0.49399999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.77500000000000002,"15min":0.74099999999999999},"min":{"1min":0.60499999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.76200000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.94399999999999995,"15min":0.88900000000000001},"min":{"1min":0.62,"5min":0.503,"15min":0.503},"max":{"1min":10.132,"5min":10.132,"15min":10.132},"last":0.497},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.85499999999999998,"15min":0.84899999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.81200000000000006}]},{"osd":7,"last update":"Sun Sep 10 01:02:34 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.94499999999999995,"15min":0.85999999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7969999999999999,"5min":10.081,"15min":10.081},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.93600000000000005,"15min":0.85999999999999999},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":6.9009999999999998,"5min":6.9009999999999998,"15min":6.9009999999999998},"last":0.88200000000000001}]},{"osd":8,"last update":"Sun Sep 10 01:01:58 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.92900000000000005,"15min":0.91700000000000004},"min":{"1min":0.64900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.753},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89200000000000002,"15min":0.88600000000000001},"min":{"1min":0.65500000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.62}]},{"osd":9,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.093,"15min":1.0329999999999999},"min":{"1min":0.67300000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":12.683,"5min":12.683,"15min":12.683},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.0009999999999999,"15min":1.0289999999999999},"min":{"1min":0.66500000000000004,"5min":0.625,"15min":0.625},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.79400000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.94599999999999995,"15min":0.95299999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.056,"15min":0.98799999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.9450000000000001,"5min":10.170999999999999,"15min":10.170999999999999},"last":0.86699999999999999}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.4970000000000001,"15min":1.5389999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":6.9349999999999996,"5min":10.247,"15min":10.247},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.075,"15min":1.105},"min":{"1min":0.71699999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.379,"5min":2.3660000000000001,"15min":2.3660000000000001},"last":0.90400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":563757,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.627,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":5.3129999999999997,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.64700000000000002,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.2859999999999996,"5min":5.2859999999999996,"15min":5.2859999999999996},"last":0.46000000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.64800000000000002,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.9569999999999999,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.58499999999999996,"15min":0.53700000000000003},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.61799999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.66500000000000004,"15min":0.751},"min":{"1min":0.45200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.8919999999999999,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.69699999999999995,"15min":0.77900000000000003},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.875,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.75}]},{"osd":4,"last update":"Sun Sep 10 01:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45900000000000002,"15min":0.38800000000000001},"min":{"1min":0.36599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.46500000000000002,"15min":0.39700000000000002},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.69399999999999995,"15min":0.59099999999999997},"min":{"1min":0.626,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89000000000000001,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.62},"min":{"1min":0.57299999999999995,"5min":0.38,"15min":0.38},"max":{"1min":0.92500000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.67700000000000005}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.67900000000000005,"15min":0.64300000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.9769999999999999,"5min":2.9769999999999999,"15min":2.9769999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.012,"5min":0.73499999999999999,"15min":0.69299999999999995},"min":{"1min":0.53300000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":5.0970000000000004},"last":0.71599999999999997}]},{"osd":7,"last update":"Sun Sep 10 01:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55900000000000005},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60399999999999998,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.79000000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.68000000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.44,"15min":0.44},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.88}]},{"osd":9,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":0.81599999999999995,"15min":0.76000000000000001},"min":{"1min":0.60899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":4.9969999999999999,"5min":4.9969999999999999,"15min":4.9969999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.84699999999999998,"15min":0.80000000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":5.0380000000000003,"5min":5.0380000000000003,"15min":5.0380000000000003},"last":0.93899999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.71199999999999997,"15min":0.69399999999999995},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.73299999999999998,"15min":0.70499999999999996},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81200000000000006}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.86599999999999999,"15min":0.86399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0740000000000001,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.86799999999999999,"15min":0.86399999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1060000000000001,"5min":1.107,"15min":1.107},"last":0.89900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500461,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 10 01:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.67200000000000004,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.274,"5min":2.274,"15min":2.274},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.61799999999999999,"15min":0.56599999999999995},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.34399999999999997},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.996}]},{"osd":2,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69099999999999995,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.038,"5min":1.26,"15min":1.26},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.77200000000000002,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8999999999999999,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.90800000000000003}]},{"osd":3,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.439},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.92200000000000004}]},{"osd":4,"last update":"Sun Sep 10 01:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66000000000000003,"15min":0.59199999999999997},"min":{"1min":0.42699999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.70099999999999996,"15min":0.62},"min":{"1min":0.53000000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.167,"5min":1.302,"15min":1.302},"last":0.95699999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.747,"15min":0.71199999999999997},"min":{"1min":0.59799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.028,"5min":1.125,"15min":1.125},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70699999999999996,"15min":0.66800000000000004},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.752}]},{"osd":6,"last update":"Sun Sep 10 01:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69999999999999996,"15min":0.67100000000000004},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76100000000000001,"15min":0.72099999999999997},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":0.72199999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.79900000000000004},"min":{"1min":0.64300000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.58099999999999996,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.0329999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.66400000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.0840000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83299999999999996,"15min":0.84299999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1499999999999999,"5min":1.24,"15min":1.24},"last":0.92600000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:02:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.89100000000000001,"15min":0.92100000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3360000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.98399999999999999,"15min":1.01},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.046}]},{"osd":10,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.89000000000000001,"15min":0.91800000000000004},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.071},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.91300000000000003,"15min":0.93500000000000005},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.129,"5min":1.444,"15min":1.444},"last":1.0600000000000001}]},{"osd":11,"last update":"Sun Sep 10 01:02:03 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.92300000000000004,"15min":0.91800000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":1.093},{"interface":"front","average":{"1min":1.002,"5min":0.97799999999999998,"15min":0.97399999999999998},"min":{"1min":0.76700000000000002,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1020000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":564485,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67000000000000004,"15min":0.55200000000000005},"min":{"1min":0.434,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85999999999999999,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.56599999999999995},"min":{"1min":0.48099999999999998,"5min":0.36099999999999999,"15min":0.34399999999999997},"max":{"1min":0.86599999999999999,"5min":6.6639999999999997,"15min":6.6639999999999997},"last":0.51500000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.63400000000000001,"15min":0.55700000000000005},"min":{"1min":0.47699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82999999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.64600000000000002,"15min":0.56399999999999995},"min":{"1min":0.51300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88800000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.878}]},{"osd":3,"last update":"Sun Sep 10 01:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.64100000000000001,"15min":0.58299999999999996},"min":{"1min":0.47699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.63700000000000001,"15min":0.60099999999999998},"min":{"1min":0.505,"5min":0.379,"15min":0.379},"max":{"1min":1.6060000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.77800000000000002,"15min":0.78400000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93500000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.89200000000000002}]},{"osd":5,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.54300000000000004,"15min":0.46100000000000002},"min":{"1min":0.40899999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.60199999999999998,"15min":0.52800000000000002},"min":{"1min":0.41999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0029999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.86299999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.79300000000000004,"15min":0.70099999999999996},"min":{"1min":0.70199999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.6820000000000004,"5min":6.6820000000000004,"15min":6.6820000000000004},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.72799999999999998,"15min":0.68600000000000005},"min":{"1min":0.65500000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.65600000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.105,"5min":1.173,"15min":1.173},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.046,"5min":1.157,"15min":1.157},"last":0.97699999999999998}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56399999999999995,"15min":0.54500000000000004},"min":{"1min":0.51000000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.73299999999999998,"15min":0.71499999999999997},"min":{"1min":0.59499999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.96499999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:02:20 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.84699999999999998,"15min":0.80400000000000005},"min":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.113,"5min":0.80800000000000005,"15min":0.747},"min":{"1min":0.71699999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":4.577,"5min":4.577,"15min":4.577},"last":0.84599999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.71699999999999997,"15min":0.67600000000000005},"min":{"1min":0.59199999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.117,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.82399999999999995,"15min":0.78400000000000003},"min":{"1min":0.77500000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0649999999999999,"5min":1.099,"15min":1.099},"last":1.004}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.82699999999999996,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.621,"15min":0.621},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93899999999999995,"15min":0.93700000000000006},"min":{"1min":0.81599999999999995,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.097,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.0189999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":499267,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.47299999999999998,"15min":0.40699999999999997},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.54200000000000004,"15min":0.45100000000000001},"min":{"1min":0.33000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.5309999999999997,"5min":4.5309999999999997,"15min":4.5309999999999997},"last":0.40500000000000003}]},{"osd":1,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.69999999999999996,"15min":0.61899999999999999},"min":{"1min":0.49399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.63900000000000001,"15min":0.57499999999999996},"min":{"1min":0.48099999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.74399999999999999,"15min":0.64300000000000002},"min":{"1min":0.45700000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.819,"5min":4.819,"15min":4.819},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.72499999999999998,"15min":0.64700000000000002},"min":{"1min":0.53000000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.77200000000000002,"15min":0.69699999999999995},"min":{"1min":0.57899999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9139999999999999,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.82299999999999995,"15min":0.73599999999999999},"min":{"1min":0.66000000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.9450000000000001,"5min":2.52,"15min":2.52},"last":0.72199999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.79500000000000004,"15min":0.73299999999999998},"min":{"1min":0.51600000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":0.75800000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.71199999999999997,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.8609999999999998,"5min":4.8609999999999998,"15min":4.8609999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.745,"15min":0.67300000000000004},"min":{"1min":0.46100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.89900000000000002}]},{"osd":7,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.80100000000000005,"15min":0.79900000000000004},"min":{"1min":0.51300000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.437,"5min":0.434,"15min":0.434},"max":{"1min":0.99199999999999999,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.59999999999999998}]},{"osd":8,"last update":"Sun Sep 10 00:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Sun Sep 10 01:02:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":0.96199999999999997,"15min":0.97999999999999998},"min":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.012,"5min":0.93799999999999994,"15min":0.93600000000000005},"min":{"1min":0.53300000000000003,"5min":0.433,"15min":0.433},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.93600000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.97999999999999998,"15min":0.97199999999999998},"min":{"1min":0.71099999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.6680000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.84799999999999998,"15min":0.82899999999999996},"min":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.6280000000000001,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":0.66700000000000004}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.095,"15min":1.1259999999999999},"min":{"1min":0.73599999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.25,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0900000000000001,"15min":1.1020000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.6930000000000001,"5min":4.9139999999999997,"15min":4.9139999999999997},"last":0.879}]}]},{"osd":4,"up_from":32,"seq":137438953543,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":505192,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.36799999999999999},"min":{"1min":0.32900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41799999999999998,"15min":0.38400000000000001},"min":{"1min":0.34799999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.442}]},{"osd":2,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59399999999999997,"15min":0.56699999999999995},"min":{"1min":0.42699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0820000000000001,"5min":1.607,"15min":1.607},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.59099999999999997},"min":{"1min":0.50600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.079,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.61199999999999999,"15min":0.57699999999999996},"min":{"1min":0.46899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.86099999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.624},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.35999999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.495}]},{"osd":5,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.61199999999999999},"min":{"1min":0.47799999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1020000000000001,"5min":1.621,"15min":1.621},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66000000000000003,"15min":0.59999999999999998},"min":{"1min":0.56399999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81799999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.66900000000000004}]},{"osd":6,"last update":"Sun Sep 10 01:02:19 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.83399999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.44700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68500000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59399999999999997,"15min":0.54800000000000004},"min":{"1min":0.41599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.62,"15min":0.59099999999999997},"min":{"1min":0.42899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55500000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.67900000000000005,"15min":0.65100000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1200000000000001,"5min":1.802,"15min":1.802},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76700000000000002,"15min":0.73699999999999999},"min":{"1min":0.66300000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91100000000000003,"5min":0.997,"15min":0.997},"last":0.84399999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:02:22 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.58499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.73399999999999999},"min":{"1min":0.54600000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.751}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77800000000000002,"15min":0.77700000000000002},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95199999999999996,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.77400000000000002,"15min":0.76000000000000001},"min":{"1min":0.48699999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.76800000000000002}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.80400000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790018,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1095454,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56699999999999995,"15min":0.51500000000000001},"min":{"1min":0.38200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.55800000000000005,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.441}]},{"osd":1,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.63700000000000001,"15min":0.56799999999999995},"min":{"1min":0.47299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.622,"15min":0.56499999999999995},"min":{"1min":0.46700000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.39400000000000002},"min":{"1min":0.35099999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.623,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.46000000000000002,"15min":0.41799999999999998},"min":{"1min":0.316,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.38700000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.58199999999999996,"15min":0.55600000000000005},"min":{"1min":0.443,"5min":0.376,"15min":0.376},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.64000000000000001,"15min":0.56999999999999995},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.52500000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.65100000000000002,"15min":0.60099999999999998},"min":{"1min":0.47399999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92100000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.65700000000000003,"15min":0.60999999999999999},"min":{"1min":0.48499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1,"5min":1.02,"15min":1.02},"last":0.60699999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:02:17 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.67000000000000004,"15min":0.63800000000000001},"min":{"1min":0.58199999999999996,"5min":0.437,"15min":0.437},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.69599999999999995,"15min":0.66400000000000003},"min":{"1min":0.61799999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.89200000000000002,"5min":0.998,"15min":0.998},"last":0.623}]},{"osd":7,"last update":"Sun Sep 10 01:02:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.70299999999999996,"15min":0.67800000000000005},"min":{"1min":0.59599999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93000000000000005,"5min":1.042,"15min":1.042},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.73799999999999999,"15min":0.70699999999999996},"min":{"1min":0.52600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.77600000000000002}]},{"osd":8,"last update":"Sun Sep 10 01:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.66600000000000004,"15min":0.65000000000000002},"min":{"1min":0.438,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91000000000000003,"5min":1.036,"15min":1.036},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.64700000000000002,"15min":0.61899999999999999},"min":{"1min":0.45000000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.64300000000000002}]},{"osd":9,"last update":"Sun Sep 10 01:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.72999999999999998,"15min":0.69999999999999996},"min":{"1min":0.63300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.80100000000000005,"15min":0.77100000000000002},"min":{"1min":0.747,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.746}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.73699999999999999},"min":{"1min":0.59899999999999998,"5min":0.505,"15min":0.505},"max":{"1min":0.98999999999999999,"5min":1.107,"15min":1.107},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.81499999999999995,"15min":0.77700000000000002},"min":{"1min":0.64300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0860000000000001,"5min":1.171,"15min":1.171},"last":0.82899999999999996}]},{"osd":11,"last update":"Sun Sep 10 01:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.80400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-10T01:02:58.610 INFO:tasks.fwd_scrub.ceph_manager:clean! 2023-09-10T01:02:58.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-10T01:02:58.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 -- 172.21.15.3:0/2143052204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f41018c0 msgr2=0x7f62f4101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/2143052204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f41018c0 0x7f62f4101d20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f62e00099c0 tx=0x7f62e0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 -- 172.21.15.3:0/2143052204 shutdown_connections 2023-09-10T01:02:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/2143052204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62f41006d0 0x7f62f4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/2143052204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f4102260 0x7f62f4108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/2143052204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f41018c0 0x7f62f4101d20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 -- 172.21.15.3:0/2143052204 >> 172.21.15.3:0/2143052204 conn(0x7f62f40fbc60 msgr2=0x7f62f40fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:58.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 -- 172.21.15.3:0/2143052204 shutdown_connections 2023-09-10T01:02:58.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.845+0000 7f62fb1ff700 1 -- 172.21.15.3:0/2143052204 wait complete. 2023-09-10T01:02:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.846+0000 7f62fb1ff700 1 Processor -- start 2023-09-10T01:02:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.846+0000 7f62fb1ff700 1 -- start start 2023-09-10T01:02:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62fb1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f41006d0 0x7f62f40743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62fb1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62f41018c0 0x7f62f40729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62fb1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f4102260 0x7f62f4072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62fb1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40736d0 con 0x7f62f4102260 2023-09-10T01:02:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62fb1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4073840 con 0x7f62f41006d0 2023-09-10T01:02:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62fb1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a03b0 con 0x7f62f41018c0 2023-09-10T01:02:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f4102260 0x7f62f4072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f8f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f41006d0 0x7f62f40743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f4102260 0x7f62f4072f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49276/0 (socket says 172.21.15.3:49276) 2023-09-10T01:02:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 -- 172.21.15.3:0/1925002588 learned_addr learned my addr 172.21.15.3:0/1925002588 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:58.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f3fff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62f41018c0 0x7f62f40729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 -- 172.21.15.3:0/1925002588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62f41018c0 msgr2=0x7f62f40729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62f41018c0 0x7f62f40729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 -- 172.21.15.3:0/1925002588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f41006d0 msgr2=0x7f62f40743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:58.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f41006d0 0x7f62f40743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:58.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f979c700 1 -- 172.21.15.3:0/1925002588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e00095f0 con 0x7f62f4102260 2023-09-10T01:02:58.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.847+0000 7f62f3fff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62f41018c0 0x7f62f40729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:02:58.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.848+0000 7f62f979c700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f4102260 0x7f62f4072f30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00ef30 tx=0x7f62ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:58.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.848+0000 7f62f1ffb700 1 -- 172.21.15.3:0/1925002588 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec018070 con 0x7f62f4102260 2023-09-10T01:02:58.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.848+0000 7f62f1ffb700 1 -- 172.21.15.3:0/1925002588 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62ec00f040 con 0x7f62f4102260 2023-09-10T01:02:58.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.848+0000 7f62f1ffb700 1 -- 172.21.15.3:0/1925002588 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec00cdb0 con 0x7f62f4102260 2023-09-10T01:02:58.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.848+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f41a0630 con 0x7f62f4102260 2023-09-10T01:02:58.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.848+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f41a0b40 con 0x7f62f4102260 2023-09-10T01:02:58.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.849+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4107600 con 0x7f62f4102260 2023-09-10T01:02:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.851+0000 7f62f1ffb700 1 -- 172.21.15.3:0/1925002588 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f62ec003a00 con 0x7f62f4102260 2023-09-10T01:02:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.852+0000 7f62f1ffb700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f62e40a06c0 0x7f62e40a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.853+0000 7f62f1ffb700 1 -- 172.21.15.3:0/1925002588 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f62ec0cc350 con 0x7f62f4102260 2023-09-10T01:02:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.853+0000 7f62f8f9b700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f62e40a06c0 0x7f62e40a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.853+0000 7f62f8f9b700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f62e40a06c0 0x7f62e40a2b80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f62f4101720 tx=0x7f62e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:58.855+0000 7f62f1ffb700 1 -- 172.21.15.3:0/1925002588 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f62ec095c00 con 0x7f62f4102260 2023-09-10T01:02:59.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.067+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62f402cf50 con 0x7f62e40a06c0 2023-09-10T01:02:59.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.080+0000 7f62f1ffb700 1 -- 172.21.15.3:0/1925002588 <== mgr.14150 v2:172.21.15.3:6800/91094391 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427635 (secure 0 0 0) 0x7f62f402cf50 con 0x7f62e40a06c0 2023-09-10T01:02:59.087 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:59.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f62e40a06c0 msgr2=0x7f62e40a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f62e40a06c0 0x7f62e40a2b80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f62f4101720 tx=0x7f62e8009260 comp rx=0 tx=0).stop 2023-09-10T01:02:59.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f4102260 msgr2=0x7f62f4072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f4102260 0x7f62f4072f30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00ef30 tx=0x7f62ec00c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 shutdown_connections 2023-09-10T01:02:59.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62f4102260 0x7f62f4072f30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62f41006d0 0x7f62f40743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f62e40a06c0 0x7f62e40a2b80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 --2- 172.21.15.3:0/1925002588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62f41018c0 0x7f62f40729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 >> 172.21.15.3:0/1925002588 conn(0x7f62f40fbc60 msgr2=0x7f62f40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:59.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.085+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 shutdown_connections 2023-09-10T01:02:59.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.086+0000 7f62fb1ff700 1 -- 172.21.15.3:0/1925002588 wait complete. 2023-09-10T01:02:59.093 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-09-10T01:02:59.105 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2023-09-10T01:02:57.882984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659845,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":813,"num_read_kb":532,"num_write":349,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336256,"kb_used_data":12064,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605568,"statfs":{"total":1151940427776,"available":1151596101632,"internally_reserved":0,"allocated":12353536,"data_stored":7999030,"data_compressed":391800,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":93,"num_read_kb":52,"num_write":7,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003042"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885419+0000","last_change":"2023-09-10T01:00:44.026223+0000","last_active":"2023-09-10T01:02:25.885419+0000","last_peered":"2023-09-10T01:02:25.885419+0000","last_clean":"2023-09-10T01:02:25.885419+0000","last_became_active":"2023-09-10T01:00:44.025731+0000","last_became_peered":"2023-09-10T01:00:44.025731+0000","last_unstale":"2023-09-10T01:02:25.885419+0000","last_undegraded":"2023-09-10T01:02:25.885419+0000","last_fullsized":"2023-09-10T01:02:25.885419+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:51:48.949903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871454+0000","last_change":"2023-09-10T01:00:44.038916+0000","last_active":"2023-09-10T01:02:25.871454+0000","last_peered":"2023-09-10T01:02:25.871454+0000","last_clean":"2023-09-10T01:02:25.871454+0000","last_became_active":"2023-09-10T01:00:44.038546+0000","last_became_peered":"2023-09-10T01:00:44.038546+0000","last_unstale":"2023-09-10T01:02:25.871454+0000","last_undegraded":"2023-09-10T01:02:25.871454+0000","last_fullsized":"2023-09-10T01:02:25.871454+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:05:56.096857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880464+0000","last_change":"2023-09-10T01:00:44.032942+0000","last_active":"2023-09-10T01:02:25.880464+0000","last_peered":"2023-09-10T01:02:25.880464+0000","last_clean":"2023-09-10T01:02:25.880464+0000","last_became_active":"2023-09-10T01:00:44.032546+0000","last_became_peered":"2023-09-10T01:00:44.032546+0000","last_unstale":"2023-09-10T01:02:25.880464+0000","last_undegraded":"2023-09-10T01:02:25.880464+0000","last_fullsized":"2023-09-10T01:02:25.880464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:46.613487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.874834+0000","last_change":"2023-09-10T01:00:44.031411+0000","last_active":"2023-09-10T01:02:25.874834+0000","last_peered":"2023-09-10T01:02:25.874834+0000","last_clean":"2023-09-10T01:02:25.874834+0000","last_became_active":"2023-09-10T01:00:44.031121+0000","last_became_peered":"2023-09-10T01:00:44.031121+0000","last_unstale":"2023-09-10T01:02:25.874834+0000","last_undegraded":"2023-09-10T01:02:25.874834+0000","last_fullsized":"2023-09-10T01:02:25.874834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:47.658116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887476+0000","last_change":"2023-09-10T01:00:44.037351+0000","last_active":"2023-09-10T01:02:25.887476+0000","last_peered":"2023-09-10T01:02:25.887476+0000","last_clean":"2023-09-10T01:02:25.887476+0000","last_became_active":"2023-09-10T01:00:44.037005+0000","last_became_peered":"2023-09-10T01:00:44.037005+0000","last_unstale":"2023-09-10T01:02:25.887476+0000","last_undegraded":"2023-09-10T01:02:25.887476+0000","last_fullsized":"2023-09-10T01:02:25.887476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:10:45.694056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882017+0000","last_change":"2023-09-10T01:00:44.029543+0000","last_active":"2023-09-10T01:02:25.882017+0000","last_peered":"2023-09-10T01:02:25.882017+0000","last_clean":"2023-09-10T01:02:25.882017+0000","last_became_active":"2023-09-10T01:00:44.029210+0000","last_became_peered":"2023-09-10T01:00:44.029210+0000","last_unstale":"2023-09-10T01:02:25.882017+0000","last_undegraded":"2023-09-10T01:02:25.882017+0000","last_fullsized":"2023-09-10T01:02:25.882017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:08:56.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879037+0000","last_change":"2023-09-10T01:00:44.033328+0000","last_active":"2023-09-10T01:02:25.879037+0000","last_peered":"2023-09-10T01:02:25.879037+0000","last_clean":"2023-09-10T01:02:25.879037+0000","last_became_active":"2023-09-10T01:00:44.033007+0000","last_became_peered":"2023-09-10T01:00:44.033007+0000","last_unstale":"2023-09-10T01:02:25.879037+0000","last_undegraded":"2023-09-10T01:02:25.879037+0000","last_fullsized":"2023-09-10T01:02:25.879037+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:59:14.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891940+0000","last_change":"2023-09-10T01:00:44.025611+0000","last_active":"2023-09-10T01:02:25.891940+0000","last_peered":"2023-09-10T01:02:25.891940+0000","last_clean":"2023-09-10T01:02:25.891940+0000","last_became_active":"2023-09-10T01:00:44.025249+0000","last_became_peered":"2023-09-10T01:00:44.025249+0000","last_unstale":"2023-09-10T01:02:25.891940+0000","last_undegraded":"2023-09-10T01:02:25.891940+0000","last_fullsized":"2023-09-10T01:02:25.891940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:55:04.320356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878755+0000","last_change":"2023-09-10T01:00:44.024207+0000","last_active":"2023-09-10T01:02:25.878755+0000","last_peered":"2023-09-10T01:02:25.878755+0000","last_clean":"2023-09-10T01:02:25.878755+0000","last_became_active":"2023-09-10T01:00:44.023866+0000","last_became_peered":"2023-09-10T01:00:44.023866+0000","last_unstale":"2023-09-10T01:02:25.878755+0000","last_undegraded":"2023-09-10T01:02:25.878755+0000","last_fullsized":"2023-09-10T01:02:25.878755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:53:39.845384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883727+0000","last_change":"2023-09-10T01:00:44.031241+0000","last_active":"2023-09-10T01:02:25.883727+0000","last_peered":"2023-09-10T01:02:25.883727+0000","last_clean":"2023-09-10T01:02:25.883727+0000","last_became_active":"2023-09-10T01:00:44.030392+0000","last_became_peered":"2023-09-10T01:00:44.030392+0000","last_unstale":"2023-09-10T01:02:25.883727+0000","last_undegraded":"2023-09-10T01:02:25.883727+0000","last_fullsized":"2023-09-10T01:02:25.883727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:19:39.786995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883954+0000","last_change":"2023-09-10T01:00:44.026983+0000","last_active":"2023-09-10T01:02:25.883954+0000","last_peered":"2023-09-10T01:02:25.883954+0000","last_clean":"2023-09-10T01:02:25.883954+0000","last_became_active":"2023-09-10T01:00:44.026695+0000","last_became_peered":"2023-09-10T01:00:44.026695+0000","last_unstale":"2023-09-10T01:02:25.883954+0000","last_undegraded":"2023-09-10T01:02:25.883954+0000","last_fullsized":"2023-09-10T01:02:25.883954+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:37.809144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.874638+0000","last_change":"2023-09-10T01:00:44.030787+0000","last_active":"2023-09-10T01:02:25.874638+0000","last_peered":"2023-09-10T01:02:25.874638+0000","last_clean":"2023-09-10T01:02:25.874638+0000","last_became_active":"2023-09-10T01:00:44.028579+0000","last_became_peered":"2023-09-10T01:00:44.028579+0000","last_unstale":"2023-09-10T01:02:25.874638+0000","last_undegraded":"2023-09-10T01:02:25.874638+0000","last_fullsized":"2023-09-10T01:02:25.874638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:04:26.446378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.886732+0000","last_change":"2023-09-10T01:00:44.029905+0000","last_active":"2023-09-10T01:02:25.886732+0000","last_peered":"2023-09-10T01:02:25.886732+0000","last_clean":"2023-09-10T01:02:25.886732+0000","last_became_active":"2023-09-10T01:00:44.029561+0000","last_became_peered":"2023-09-10T01:00:44.029561+0000","last_unstale":"2023-09-10T01:02:25.886732+0000","last_undegraded":"2023-09-10T01:02:25.886732+0000","last_fullsized":"2023-09-10T01:02:25.886732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:17:50.839001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.877754+0000","last_change":"2023-09-10T01:00:44.037864+0000","last_active":"2023-09-10T01:02:25.877754+0000","last_peered":"2023-09-10T01:02:25.877754+0000","last_clean":"2023-09-10T01:02:25.877754+0000","last_became_active":"2023-09-10T01:00:44.037533+0000","last_became_peered":"2023-09-10T01:00:44.037533+0000","last_unstale":"2023-09-10T01:02:25.877754+0000","last_undegraded":"2023-09-10T01:02:25.877754+0000","last_fullsized":"2023-09-10T01:02:25.877754+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:29:26.768570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.869883+0000","last_change":"2023-09-10T01:00:44.033396+0000","last_active":"2023-09-10T01:02:25.869883+0000","last_peered":"2023-09-10T01:02:25.869883+0000","last_clean":"2023-09-10T01:02:25.869883+0000","last_became_active":"2023-09-10T01:00:44.033056+0000","last_became_peered":"2023-09-10T01:00:44.033056+0000","last_unstale":"2023-09-10T01:02:25.869883+0000","last_undegraded":"2023-09-10T01:02:25.869883+0000","last_fullsized":"2023-09-10T01:02:25.869883+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:00:03.590555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882604+0000","last_change":"2023-09-10T01:00:44.027729+0000","last_active":"2023-09-10T01:02:25.882604+0000","last_peered":"2023-09-10T01:02:25.882604+0000","last_clean":"2023-09-10T01:02:25.882604+0000","last_became_active":"2023-09-10T01:00:44.027447+0000","last_became_peered":"2023-09-10T01:00:44.027447+0000","last_unstale":"2023-09-10T01:02:25.882604+0000","last_undegraded":"2023-09-10T01:02:25.882604+0000","last_fullsized":"2023-09-10T01:02:25.882604+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:06:09.391443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885073+0000","last_change":"2023-09-10T01:00:44.026042+0000","last_active":"2023-09-10T01:02:25.885073+0000","last_peered":"2023-09-10T01:02:25.885073+0000","last_clean":"2023-09-10T01:02:25.885073+0000","last_became_active":"2023-09-10T01:00:44.025309+0000","last_became_peered":"2023-09-10T01:00:44.025309+0000","last_unstale":"2023-09-10T01:02:25.885073+0000","last_undegraded":"2023-09-10T01:02:25.885073+0000","last_fullsized":"2023-09-10T01:02:25.885073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:51:29.038717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.886092+0000","last_change":"2023-09-10T01:00:44.036761+0000","last_active":"2023-09-10T01:02:25.886092+0000","last_peered":"2023-09-10T01:02:25.886092+0000","last_clean":"2023-09-10T01:02:25.886092+0000","last_became_active":"2023-09-10T01:00:44.036333+0000","last_became_peered":"2023-09-10T01:00:44.036333+0000","last_unstale":"2023-09-10T01:02:25.886092+0000","last_undegraded":"2023-09-10T01:02:25.886092+0000","last_fullsized":"2023-09-10T01:02:25.886092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:37:09.042178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.885302+0000","last_change":"2023-09-10T01:00:44.027304+0000","last_active":"2023-09-10T01:02:25.885302+0000","last_peered":"2023-09-10T01:02:25.885302+0000","last_clean":"2023-09-10T01:02:25.885302+0000","last_became_active":"2023-09-10T01:00:44.026994+0000","last_became_peered":"2023-09-10T01:00:44.026994+0000","last_unstale":"2023-09-10T01:02:25.885302+0000","last_undegraded":"2023-09-10T01:02:25.885302+0000","last_fullsized":"2023-09-10T01:02:25.885302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:11:04.824687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032471+0000","last_change":"2023-09-10T01:00:48.056098+0000","last_active":"2023-09-10T01:01:56.032471+0000","last_peered":"2023-09-10T01:01:56.032471+0000","last_clean":"2023-09-10T01:01:56.032471+0000","last_became_active":"2023-09-10T01:00:48.055694+0000","last_became_peered":"2023-09-10T01:00:48.055694+0000","last_unstale":"2023-09-10T01:01:56.032471+0000","last_undegraded":"2023-09-10T01:01:56.032471+0000","last_fullsized":"2023-09-10T01:01:56.032471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:30:48.711613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.046616+0000","last_change":"2023-09-10T01:00:56.971435+0000","last_active":"2023-09-10T01:01:57.046616+0000","last_peered":"2023-09-10T01:01:57.046616+0000","last_clean":"2023-09-10T01:01:57.046616+0000","last_became_active":"2023-09-10T01:00:56.965158+0000","last_became_peered":"2023-09-10T01:00:56.965158+0000","last_unstale":"2023-09-10T01:01:57.046616+0000","last_undegraded":"2023-09-10T01:01:57.046616+0000","last_fullsized":"2023-09-10T01:01:57.046616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:09:20.910597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880753+0000","last_change":"2023-09-10T01:00:44.023469+0000","last_active":"2023-09-10T01:02:25.880753+0000","last_peered":"2023-09-10T01:02:25.880753+0000","last_clean":"2023-09-10T01:02:25.880753+0000","last_became_active":"2023-09-10T01:00:44.023232+0000","last_became_peered":"2023-09-10T01:00:44.023232+0000","last_unstale":"2023-09-10T01:02:25.880753+0000","last_undegraded":"2023-09-10T01:02:25.880753+0000","last_fullsized":"2023-09-10T01:02:25.880753+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:33:22.605759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037995+0000","last_change":"2023-09-10T01:00:48.063565+0000","last_active":"2023-09-10T01:01:56.037995+0000","last_peered":"2023-09-10T01:01:56.037995+0000","last_clean":"2023-09-10T01:01:56.037995+0000","last_became_active":"2023-09-10T01:00:48.063200+0000","last_became_peered":"2023-09-10T01:00:48.063200+0000","last_unstale":"2023-09-10T01:01:56.037995+0000","last_undegraded":"2023-09-10T01:01:56.037995+0000","last_fullsized":"2023-09-10T01:01:56.037995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:50:16.624296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.471781+0000","last_change":"2023-09-10T01:00:43.010566+0000","last_active":"2023-09-10T01:01:57.471781+0000","last_peered":"2023-09-10T01:01:57.471781+0000","last_clean":"2023-09-10T01:01:57.471781+0000","last_became_active":"2023-09-10T01:00:43.010277+0000","last_became_peered":"2023-09-10T01:00:43.010277+0000","last_unstale":"2023-09-10T01:01:57.471781+0000","last_undegraded":"2023-09-10T01:01:57.471781+0000","last_fullsized":"2023-09-10T01:01:57.471781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:37.808085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.884035+0000","last_change":"2023-09-10T01:00:44.026219+0000","last_active":"2023-09-10T01:02:25.884035+0000","last_peered":"2023-09-10T01:02:25.884035+0000","last_clean":"2023-09-10T01:02:25.884035+0000","last_became_active":"2023-09-10T01:00:44.025906+0000","last_became_peered":"2023-09-10T01:00:44.025906+0000","last_unstale":"2023-09-10T01:02:25.884035+0000","last_undegraded":"2023-09-10T01:02:25.884035+0000","last_fullsized":"2023-09-10T01:02:25.884035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:21:48.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037347+0000","last_change":"2023-09-10T01:00:48.069671+0000","last_active":"2023-09-10T01:01:56.037347+0000","last_peered":"2023-09-10T01:01:56.037347+0000","last_clean":"2023-09-10T01:01:56.037347+0000","last_became_active":"2023-09-10T01:00:48.069382+0000","last_became_peered":"2023-09-10T01:00:48.069382+0000","last_unstale":"2023-09-10T01:01:56.037347+0000","last_undegraded":"2023-09-10T01:01:56.037347+0000","last_fullsized":"2023-09-10T01:01:56.037347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:06:20.526398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036543+0000","last_change":"2023-09-10T01:00:43.005098+0000","last_active":"2023-09-10T01:01:56.036543+0000","last_peered":"2023-09-10T01:01:56.036543+0000","last_clean":"2023-09-10T01:01:56.036543+0000","last_became_active":"2023-09-10T01:00:43.004556+0000","last_became_peered":"2023-09-10T01:00:43.004556+0000","last_unstale":"2023-09-10T01:01:56.036543+0000","last_undegraded":"2023-09-10T01:01:56.036543+0000","last_fullsized":"2023-09-10T01:01:56.036543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:27:43.538936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890280+0000","last_change":"2023-09-10T01:00:44.034451+0000","last_active":"2023-09-10T01:02:25.890280+0000","last_peered":"2023-09-10T01:02:25.890280+0000","last_clean":"2023-09-10T01:02:25.890280+0000","last_became_active":"2023-09-10T01:00:44.034131+0000","last_became_peered":"2023-09-10T01:00:44.034131+0000","last_unstale":"2023-09-10T01:02:25.890280+0000","last_undegraded":"2023-09-10T01:02:25.890280+0000","last_fullsized":"2023-09-10T01:02:25.890280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:50.643215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036946+0000","last_change":"2023-09-10T01:00:48.060114+0000","last_active":"2023-09-10T01:01:56.036946+0000","last_peered":"2023-09-10T01:01:56.036946+0000","last_clean":"2023-09-10T01:01:56.036946+0000","last_became_active":"2023-09-10T01:00:48.059531+0000","last_became_peered":"2023-09-10T01:00:48.059531+0000","last_unstale":"2023-09-10T01:01:56.036946+0000","last_undegraded":"2023-09-10T01:01:56.036946+0000","last_fullsized":"2023-09-10T01:01:56.036946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:28:01.462247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035950+0000","last_change":"2023-09-10T01:00:43.007196+0000","last_active":"2023-09-10T01:01:56.035950+0000","last_peered":"2023-09-10T01:01:56.035950+0000","last_clean":"2023-09-10T01:01:56.035950+0000","last_became_active":"2023-09-10T01:00:43.006795+0000","last_became_peered":"2023-09-10T01:00:43.006795+0000","last_unstale":"2023-09-10T01:01:56.035950+0000","last_undegraded":"2023-09-10T01:01:56.035950+0000","last_fullsized":"2023-09-10T01:01:56.035950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:26:13.146244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882156+0000","last_change":"2023-09-10T01:00:44.027809+0000","last_active":"2023-09-10T01:02:25.882156+0000","last_peered":"2023-09-10T01:02:25.882156+0000","last_clean":"2023-09-10T01:02:25.882156+0000","last_became_active":"2023-09-10T01:00:44.027499+0000","last_became_peered":"2023-09-10T01:00:44.027499+0000","last_unstale":"2023-09-10T01:02:25.882156+0000","last_undegraded":"2023-09-10T01:02:25.882156+0000","last_fullsized":"2023-09-10T01:02:25.882156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:17:07.138807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032849+0000","last_change":"2023-09-10T01:00:48.057645+0000","last_active":"2023-09-10T01:01:56.032849+0000","last_peered":"2023-09-10T01:01:56.032849+0000","last_clean":"2023-09-10T01:01:56.032849+0000","last_became_active":"2023-09-10T01:00:48.057312+0000","last_became_peered":"2023-09-10T01:00:48.057312+0000","last_unstale":"2023-09-10T01:01:56.032849+0000","last_undegraded":"2023-09-10T01:01:56.032849+0000","last_fullsized":"2023-09-10T01:01:56.032849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:34:28.904572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":147,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:56.110754+0000","last_change":"2023-09-10T01:00:56.965677+0000","last_active":"2023-09-10T01:02:56.110754+0000","last_peered":"2023-09-10T01:02:56.110754+0000","last_clean":"2023-09-10T01:02:56.110754+0000","last_became_active":"2023-09-10T01:00:56.965248+0000","last_became_peered":"2023-09-10T01:00:56.965248+0000","last_unstale":"2023-09-10T01:02:56.110754+0000","last_undegraded":"2023-09-10T01:02:56.110754+0000","last_fullsized":"2023-09-10T01:02:56.110754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:39:44.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.866210+0000","last_change":"2023-09-10T01:00:44.026340+0000","last_active":"2023-09-10T01:02:25.866210+0000","last_peered":"2023-09-10T01:02:25.866210+0000","last_clean":"2023-09-10T01:02:25.866210+0000","last_became_active":"2023-09-10T01:00:44.026024+0000","last_became_peered":"2023-09-10T01:00:44.026024+0000","last_unstale":"2023-09-10T01:02:25.866210+0000","last_undegraded":"2023-09-10T01:02:25.866210+0000","last_fullsized":"2023-09-10T01:02:25.866210+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:58:18.942980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032668+0000","last_change":"2023-09-10T01:00:48.057876+0000","last_active":"2023-09-10T01:01:56.032668+0000","last_peered":"2023-09-10T01:01:56.032668+0000","last_clean":"2023-09-10T01:01:56.032668+0000","last_became_active":"2023-09-10T01:00:48.056686+0000","last_became_peered":"2023-09-10T01:00:48.056686+0000","last_unstale":"2023-09-10T01:01:56.032668+0000","last_undegraded":"2023-09-10T01:01:56.032668+0000","last_fullsized":"2023-09-10T01:01:56.032668+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:10:15.995463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127691+0000","last_change":"2023-09-10T01:00:43.007811+0000","last_active":"2023-09-10T01:01:56.127691+0000","last_peered":"2023-09-10T01:01:56.127691+0000","last_clean":"2023-09-10T01:01:56.127691+0000","last_became_active":"2023-09-10T01:00:43.007205+0000","last_became_peered":"2023-09-10T01:00:43.007205+0000","last_unstale":"2023-09-10T01:01:56.127691+0000","last_undegraded":"2023-09-10T01:01:56.127691+0000","last_fullsized":"2023-09-10T01:01:56.127691+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:27:38.114614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.888059+0000","last_change":"2023-09-10T01:00:44.027954+0000","last_active":"2023-09-10T01:02:25.888059+0000","last_peered":"2023-09-10T01:02:25.888059+0000","last_clean":"2023-09-10T01:02:25.888059+0000","last_became_active":"2023-09-10T01:00:44.027643+0000","last_became_peered":"2023-09-10T01:00:44.027643+0000","last_unstale":"2023-09-10T01:02:25.888059+0000","last_undegraded":"2023-09-10T01:02:25.888059+0000","last_fullsized":"2023-09-10T01:02:25.888059+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:21.253013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.731942+0000","last_change":"2023-09-10T01:00:48.062519+0000","last_active":"2023-09-10T01:01:58.731942+0000","last_peered":"2023-09-10T01:01:58.731942+0000","last_clean":"2023-09-10T01:01:58.731942+0000","last_became_active":"2023-09-10T01:00:48.061985+0000","last_became_peered":"2023-09-10T01:00:48.061985+0000","last_unstale":"2023-09-10T01:01:58.731942+0000","last_undegraded":"2023-09-10T01:01:58.731942+0000","last_fullsized":"2023-09-10T01:01:58.731942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:47:36.002325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.030435+0000","last_change":"2023-09-10T01:00:42.999601+0000","last_active":"2023-09-10T01:01:56.030435+0000","last_peered":"2023-09-10T01:01:56.030435+0000","last_clean":"2023-09-10T01:01:56.030435+0000","last_became_active":"2023-09-10T01:00:42.999243+0000","last_became_peered":"2023-09-10T01:00:42.999243+0000","last_unstale":"2023-09-10T01:01:56.030435+0000","last_undegraded":"2023-09-10T01:01:56.030435+0000","last_fullsized":"2023-09-10T01:01:56.030435+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:49.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890496+0000","last_change":"2023-09-10T01:00:44.027136+0000","last_active":"2023-09-10T01:02:25.890496+0000","last_peered":"2023-09-10T01:02:25.890496+0000","last_clean":"2023-09-10T01:02:25.890496+0000","last_became_active":"2023-09-10T01:00:44.026784+0000","last_became_peered":"2023-09-10T01:00:44.026784+0000","last_unstale":"2023-09-10T01:02:25.890496+0000","last_undegraded":"2023-09-10T01:02:25.890496+0000","last_fullsized":"2023-09-10T01:02:25.890496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:51:51.271658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.055264+0000","last_change":"2023-09-10T01:00:48.066906+0000","last_active":"2023-09-10T01:01:57.055264+0000","last_peered":"2023-09-10T01:01:57.055264+0000","last_clean":"2023-09-10T01:01:57.055264+0000","last_became_active":"2023-09-10T01:00:48.066474+0000","last_became_peered":"2023-09-10T01:00:48.066474+0000","last_unstale":"2023-09-10T01:01:57.055264+0000","last_undegraded":"2023-09-10T01:01:57.055264+0000","last_fullsized":"2023-09-10T01:01:57.055264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:48:26.827857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036365+0000","last_change":"2023-09-10T01:00:43.005922+0000","last_active":"2023-09-10T01:01:56.036365+0000","last_peered":"2023-09-10T01:01:56.036365+0000","last_clean":"2023-09-10T01:01:56.036365+0000","last_became_active":"2023-09-10T01:00:43.005517+0000","last_became_peered":"2023-09-10T01:00:43.005517+0000","last_unstale":"2023-09-10T01:01:56.036365+0000","last_undegraded":"2023-09-10T01:01:56.036365+0000","last_fullsized":"2023-09-10T01:01:56.036365+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:32:12.619034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889259+0000","last_change":"2023-09-10T01:00:44.036766+0000","last_active":"2023-09-10T01:02:25.889259+0000","last_peered":"2023-09-10T01:02:25.889259+0000","last_clean":"2023-09-10T01:02:25.889259+0000","last_became_active":"2023-09-10T01:00:44.036316+0000","last_became_peered":"2023-09-10T01:00:44.036316+0000","last_unstale":"2023-09-10T01:02:25.889259+0000","last_undegraded":"2023-09-10T01:02:25.889259+0000","last_fullsized":"2023-09-10T01:02:25.889259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:17:51.875838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036026+0000","last_change":"2023-09-10T01:00:57.051716+0000","last_active":"2023-09-10T01:01:56.036026+0000","last_peered":"2023-09-10T01:01:56.036026+0000","last_clean":"2023-09-10T01:01:56.036026+0000","last_became_active":"2023-09-10T01:00:57.051334+0000","last_became_peered":"2023-09-10T01:00:57.051334+0000","last_unstale":"2023-09-10T01:01:56.036026+0000","last_undegraded":"2023-09-10T01:01:56.036026+0000","last_fullsized":"2023-09-10T01:01:56.036026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:30:52.598639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":133,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:56.109837+0000","last_change":"2023-09-10T01:00:43.008695+0000","last_active":"2023-09-10T01:02:56.109837+0000","last_peered":"2023-09-10T01:02:56.109837+0000","last_clean":"2023-09-10T01:02:56.109837+0000","last_became_active":"2023-09-10T01:00:43.008278+0000","last_became_peered":"2023-09-10T01:00:43.008278+0000","last_unstale":"2023-09-10T01:02:56.109837+0000","last_undegraded":"2023-09-10T01:02:56.109837+0000","last_fullsized":"2023-09-10T01:02:56.109837+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:46:38.370582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882691+0000","last_change":"2023-09-10T01:00:44.030540+0000","last_active":"2023-09-10T01:02:25.882691+0000","last_peered":"2023-09-10T01:02:25.882691+0000","last_clean":"2023-09-10T01:02:25.882691+0000","last_became_active":"2023-09-10T01:00:44.030212+0000","last_became_peered":"2023-09-10T01:00:44.030212+0000","last_unstale":"2023-09-10T01:02:25.882691+0000","last_undegraded":"2023-09-10T01:02:25.882691+0000","last_fullsized":"2023-09-10T01:02:25.882691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:57:06.220364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036424+0000","last_change":"2023-09-10T01:00:57.052197+0000","last_active":"2023-09-10T01:01:56.036424+0000","last_peered":"2023-09-10T01:01:56.036424+0000","last_clean":"2023-09-10T01:01:56.036424+0000","last_became_active":"2023-09-10T01:00:57.051782+0000","last_became_peered":"2023-09-10T01:00:57.051782+0000","last_unstale":"2023-09-10T01:01:56.036424+0000","last_undegraded":"2023-09-10T01:01:56.036424+0000","last_fullsized":"2023-09-10T01:01:56.036424+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:36:49.680123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.042966+0000","last_change":"2023-09-10T01:00:43.007242+0000","last_active":"2023-09-10T01:01:56.042966+0000","last_peered":"2023-09-10T01:01:56.042966+0000","last_clean":"2023-09-10T01:01:56.042966+0000","last_became_active":"2023-09-10T01:00:43.006912+0000","last_became_peered":"2023-09-10T01:00:43.006912+0000","last_unstale":"2023-09-10T01:01:56.042966+0000","last_undegraded":"2023-09-10T01:01:56.042966+0000","last_fullsized":"2023-09-10T01:01:56.042966+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:48:22.783656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.884684+0000","last_change":"2023-09-10T01:01:58.026024+0000","last_active":"2023-09-10T01:02:25.884684+0000","last_peered":"2023-09-10T01:02:25.884684+0000","last_clean":"2023-09-10T01:02:25.884684+0000","last_became_active":"2023-09-10T01:01:58.025647+0000","last_became_peered":"2023-09-10T01:01:58.025647+0000","last_unstale":"2023-09-10T01:02:25.884684+0000","last_undegraded":"2023-09-10T01:02:25.884684+0000","last_fullsized":"2023-09-10T01:02:25.884684+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:58:11.916233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037579+0000","last_change":"2023-09-10T01:00:48.062520+0000","last_active":"2023-09-10T01:01:56.037579+0000","last_peered":"2023-09-10T01:01:56.037579+0000","last_clean":"2023-09-10T01:01:56.037579+0000","last_became_active":"2023-09-10T01:00:48.061870+0000","last_became_peered":"2023-09-10T01:00:48.061870+0000","last_unstale":"2023-09-10T01:01:56.037579+0000","last_undegraded":"2023-09-10T01:01:56.037579+0000","last_fullsized":"2023-09-10T01:01:56.037579+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:24:38.798775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037030+0000","last_change":"2023-09-10T01:00:43.006513+0000","last_active":"2023-09-10T01:01:56.037030+0000","last_peered":"2023-09-10T01:01:56.037030+0000","last_clean":"2023-09-10T01:01:56.037030+0000","last_became_active":"2023-09-10T01:00:43.006166+0000","last_became_peered":"2023-09-10T01:00:43.006166+0000","last_unstale":"2023-09-10T01:01:56.037030+0000","last_undegraded":"2023-09-10T01:01:56.037030+0000","last_fullsized":"2023-09-10T01:01:56.037030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:52:32.167513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880315+0000","last_change":"2023-09-10T01:00:44.031264+0000","last_active":"2023-09-10T01:02:25.880315+0000","last_peered":"2023-09-10T01:02:25.880315+0000","last_clean":"2023-09-10T01:02:25.880315+0000","last_became_active":"2023-09-10T01:00:44.030993+0000","last_became_peered":"2023-09-10T01:00:44.030993+0000","last_unstale":"2023-09-10T01:02:25.880315+0000","last_undegraded":"2023-09-10T01:02:25.880315+0000","last_fullsized":"2023-09-10T01:02:25.880315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:19:27.658594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032496+0000","last_change":"2023-09-10T01:00:48.056886+0000","last_active":"2023-09-10T01:01:56.032496+0000","last_peered":"2023-09-10T01:01:56.032496+0000","last_clean":"2023-09-10T01:01:56.032496+0000","last_became_active":"2023-09-10T01:00:48.056411+0000","last_became_peered":"2023-09-10T01:00:48.056411+0000","last_unstale":"2023-09-10T01:01:56.032496+0000","last_undegraded":"2023-09-10T01:01:56.032496+0000","last_fullsized":"2023-09-10T01:01:56.032496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:56:42.062702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":283,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:57.111487+0000","last_change":"2023-09-10T01:00:43.009113+0000","last_active":"2023-09-10T01:02:57.111487+0000","last_peered":"2023-09-10T01:02:57.111487+0000","last_clean":"2023-09-10T01:02:57.111487+0000","last_became_active":"2023-09-10T01:00:43.008757+0000","last_became_peered":"2023-09-10T01:00:43.008757+0000","last_unstale":"2023-09-10T01:02:57.111487+0000","last_undegraded":"2023-09-10T01:02:57.111487+0000","last_fullsized":"2023-09-10T01:02:57.111487+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:29:24.402470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64774,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":226,"num_read_kb":177,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.870186+0000","last_change":"2023-09-10T01:00:44.027127+0000","last_active":"2023-09-10T01:02:25.870186+0000","last_peered":"2023-09-10T01:02:25.870186+0000","last_clean":"2023-09-10T01:02:25.870186+0000","last_became_active":"2023-09-10T01:00:44.026852+0000","last_became_peered":"2023-09-10T01:00:44.026852+0000","last_unstale":"2023-09-10T01:02:25.870186+0000","last_undegraded":"2023-09-10T01:02:25.870186+0000","last_fullsized":"2023-09-10T01:02:25.870186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:14:20.805300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037605+0000","last_change":"2023-09-10T01:00:48.069337+0000","last_active":"2023-09-10T01:01:56.037605+0000","last_peered":"2023-09-10T01:01:56.037605+0000","last_clean":"2023-09-10T01:01:56.037605+0000","last_became_active":"2023-09-10T01:00:48.068893+0000","last_became_peered":"2023-09-10T01:00:48.068893+0000","last_unstale":"2023-09-10T01:01:56.037605+0000","last_undegraded":"2023-09-10T01:01:56.037605+0000","last_fullsized":"2023-09-10T01:01:56.037605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:24:40.074551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033160+0000","last_change":"2023-09-10T01:00:43.010128+0000","last_active":"2023-09-10T01:01:56.033160+0000","last_peered":"2023-09-10T01:01:56.033160+0000","last_clean":"2023-09-10T01:01:56.033160+0000","last_became_active":"2023-09-10T01:00:43.009808+0000","last_became_peered":"2023-09-10T01:00:43.009808+0000","last_unstale":"2023-09-10T01:01:56.033160+0000","last_undegraded":"2023-09-10T01:01:56.033160+0000","last_fullsized":"2023-09-10T01:01:56.033160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:40:27.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891792+0000","last_change":"2023-09-10T01:00:44.038665+0000","last_active":"2023-09-10T01:02:25.891792+0000","last_peered":"2023-09-10T01:02:25.891792+0000","last_clean":"2023-09-10T01:02:25.891792+0000","last_became_active":"2023-09-10T01:00:44.037944+0000","last_became_peered":"2023-09-10T01:00:44.037944+0000","last_unstale":"2023-09-10T01:02:25.891792+0000","last_undegraded":"2023-09-10T01:02:25.891792+0000","last_fullsized":"2023-09-10T01:02:25.891792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:11:34.018249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.127649+0000","last_change":"2023-09-10T01:00:56.965594+0000","last_active":"2023-09-10T01:01:56.127649+0000","last_peered":"2023-09-10T01:01:56.127649+0000","last_clean":"2023-09-10T01:01:56.127649+0000","last_became_active":"2023-09-10T01:00:56.965006+0000","last_became_peered":"2023-09-10T01:00:56.965006+0000","last_unstale":"2023-09-10T01:01:56.127649+0000","last_undegraded":"2023-09-10T01:01:56.127649+0000","last_fullsized":"2023-09-10T01:01:56.127649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:13:16.946697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036734+0000","last_change":"2023-09-10T01:00:43.004855+0000","last_active":"2023-09-10T01:01:56.036734+0000","last_peered":"2023-09-10T01:01:56.036734+0000","last_clean":"2023-09-10T01:01:56.036734+0000","last_became_active":"2023-09-10T01:00:43.004464+0000","last_became_peered":"2023-09-10T01:00:43.004464+0000","last_unstale":"2023-09-10T01:01:56.036734+0000","last_undegraded":"2023-09-10T01:01:56.036734+0000","last_fullsized":"2023-09-10T01:01:56.036734+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:29.517290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889936+0000","last_change":"2023-09-10T01:00:44.037273+0000","last_active":"2023-09-10T01:02:25.889936+0000","last_peered":"2023-09-10T01:02:25.889936+0000","last_clean":"2023-09-10T01:02:25.889936+0000","last_became_active":"2023-09-10T01:00:44.036965+0000","last_became_peered":"2023-09-10T01:00:44.036965+0000","last_unstale":"2023-09-10T01:02:25.889936+0000","last_undegraded":"2023-09-10T01:02:25.889936+0000","last_fullsized":"2023-09-10T01:02:25.889936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:47:20.149002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033203+0000","last_change":"2023-09-10T01:00:48.058574+0000","last_active":"2023-09-10T01:01:56.033203+0000","last_peered":"2023-09-10T01:01:56.033203+0000","last_clean":"2023-09-10T01:01:56.033203+0000","last_became_active":"2023-09-10T01:00:48.058190+0000","last_became_peered":"2023-09-10T01:00:48.058190+0000","last_unstale":"2023-09-10T01:01:56.033203+0000","last_undegraded":"2023-09-10T01:01:56.033203+0000","last_fullsized":"2023-09-10T01:01:56.033203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:46:42.177996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":134,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:55.146304+0000","last_change":"2023-09-10T01:00:43.003816+0000","last_active":"2023-09-10T01:02:55.146304+0000","last_peered":"2023-09-10T01:02:55.146304+0000","last_clean":"2023-09-10T01:02:55.146304+0000","last_became_active":"2023-09-10T01:00:43.003404+0000","last_became_peered":"2023-09-10T01:00:43.003404+0000","last_unstale":"2023-09-10T01:02:55.146304+0000","last_undegraded":"2023-09-10T01:02:55.146304+0000","last_fullsized":"2023-09-10T01:02:55.146304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:55.208302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878740+0000","last_change":"2023-09-10T01:00:44.034607+0000","last_active":"2023-09-10T01:02:25.878740+0000","last_peered":"2023-09-10T01:02:25.878740+0000","last_clean":"2023-09-10T01:02:25.878740+0000","last_became_active":"2023-09-10T01:00:44.034244+0000","last_became_peered":"2023-09-10T01:00:44.034244+0000","last_unstale":"2023-09-10T01:02:25.878740+0000","last_undegraded":"2023-09-10T01:02:25.878740+0000","last_fullsized":"2023-09-10T01:02:25.878740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:06:13.214160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031918+0000","last_change":"2023-09-10T01:00:48.058789+0000","last_active":"2023-09-10T01:01:56.031918+0000","last_peered":"2023-09-10T01:01:56.031918+0000","last_clean":"2023-09-10T01:01:56.031918+0000","last_became_active":"2023-09-10T01:00:48.058220+0000","last_became_peered":"2023-09-10T01:00:48.058220+0000","last_unstale":"2023-09-10T01:01:56.031918+0000","last_undegraded":"2023-09-10T01:01:56.031918+0000","last_fullsized":"2023-09-10T01:01:56.031918+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:01:12.291739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036527+0000","last_change":"2023-09-10T01:00:43.006109+0000","last_active":"2023-09-10T01:01:56.036527+0000","last_peered":"2023-09-10T01:01:56.036527+0000","last_clean":"2023-09-10T01:01:56.036527+0000","last_became_active":"2023-09-10T01:00:43.005753+0000","last_became_peered":"2023-09-10T01:00:43.005753+0000","last_unstale":"2023-09-10T01:01:56.036527+0000","last_undegraded":"2023-09-10T01:01:56.036527+0000","last_fullsized":"2023-09-10T01:01:56.036527+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:45:47.083991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.881751+0000","last_change":"2023-09-10T01:01:57.139294+0000","last_active":"2023-09-10T01:02:25.881751+0000","last_peered":"2023-09-10T01:02:25.881751+0000","last_clean":"2023-09-10T01:02:25.881751+0000","last_became_active":"2023-09-10T01:01:57.138930+0000","last_became_peered":"2023-09-10T01:01:57.138930+0000","last_unstale":"2023-09-10T01:02:25.881751+0000","last_undegraded":"2023-09-10T01:02:25.881751+0000","last_fullsized":"2023-09-10T01:02:25.881751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:10:21.143499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.043801+0000","last_change":"2023-09-10T01:00:48.069318+0000","last_active":"2023-09-10T01:01:57.043801+0000","last_peered":"2023-09-10T01:01:57.043801+0000","last_clean":"2023-09-10T01:01:57.043801+0000","last_became_active":"2023-09-10T01:00:48.068876+0000","last_became_peered":"2023-09-10T01:00:48.068876+0000","last_unstale":"2023-09-10T01:01:57.043801+0000","last_undegraded":"2023-09-10T01:01:57.043801+0000","last_fullsized":"2023-09-10T01:01:57.043801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:06:22.366530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036038+0000","last_change":"2023-09-10T01:00:43.004164+0000","last_active":"2023-09-10T01:01:56.036038+0000","last_peered":"2023-09-10T01:01:56.036038+0000","last_clean":"2023-09-10T01:01:56.036038+0000","last_became_active":"2023-09-10T01:00:43.003781+0000","last_became_peered":"2023-09-10T01:00:43.003781+0000","last_unstale":"2023-09-10T01:01:56.036038+0000","last_undegraded":"2023-09-10T01:01:56.036038+0000","last_fullsized":"2023-09-10T01:01:56.036038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:38:18.559712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872316+0000","last_change":"2023-09-10T01:00:44.028206+0000","last_active":"2023-09-10T01:02:25.872316+0000","last_peered":"2023-09-10T01:02:25.872316+0000","last_clean":"2023-09-10T01:02:25.872316+0000","last_became_active":"2023-09-10T01:00:44.027914+0000","last_became_peered":"2023-09-10T01:00:44.027914+0000","last_unstale":"2023-09-10T01:02:25.872316+0000","last_undegraded":"2023-09-10T01:02:25.872316+0000","last_fullsized":"2023-09-10T01:02:25.872316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:41:12.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.106476+0000","last_change":"2023-09-10T01:00:48.428613+0000","last_active":"2023-09-10T01:01:57.106476+0000","last_peered":"2023-09-10T01:01:57.106476+0000","last_clean":"2023-09-10T01:01:57.106476+0000","last_became_active":"2023-09-10T01:00:48.428059+0000","last_became_peered":"2023-09-10T01:00:48.428059+0000","last_unstale":"2023-09-10T01:01:57.106476+0000","last_undegraded":"2023-09-10T01:01:57.106476+0000","last_fullsized":"2023-09-10T01:01:57.106476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:46:03.683279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.046289+0000","last_change":"2023-09-10T01:00:43.006061+0000","last_active":"2023-09-10T01:01:57.046289+0000","last_peered":"2023-09-10T01:01:57.046289+0000","last_clean":"2023-09-10T01:01:57.046289+0000","last_became_active":"2023-09-10T01:00:43.005392+0000","last_became_peered":"2023-09-10T01:00:43.005392+0000","last_unstale":"2023-09-10T01:01:57.046289+0000","last_undegraded":"2023-09-10T01:01:57.046289+0000","last_fullsized":"2023-09-10T01:01:57.046289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T02:01:45.062338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891410+0000","last_change":"2023-09-10T01:00:44.026684+0000","last_active":"2023-09-10T01:02:25.891410+0000","last_peered":"2023-09-10T01:02:25.891410+0000","last_clean":"2023-09-10T01:02:25.891410+0000","last_became_active":"2023-09-10T01:00:44.026387+0000","last_became_peered":"2023-09-10T01:00:44.026387+0000","last_unstale":"2023-09-10T01:02:25.891410+0000","last_undegraded":"2023-09-10T01:02:25.891410+0000","last_fullsized":"2023-09-10T01:02:25.891410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:23:03.662543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.041259+0000","last_change":"2023-09-10T01:00:48.071589+0000","last_active":"2023-09-10T01:01:57.041259+0000","last_peered":"2023-09-10T01:01:57.041259+0000","last_clean":"2023-09-10T01:01:57.041259+0000","last_became_active":"2023-09-10T01:00:48.071118+0000","last_became_peered":"2023-09-10T01:00:48.071118+0000","last_unstale":"2023-09-10T01:01:57.041259+0000","last_undegraded":"2023-09-10T01:01:57.041259+0000","last_fullsized":"2023-09-10T01:01:57.041259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:01:27.194868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024640+0000","last_change":"2023-09-10T01:00:43.009738+0000","last_active":"2023-09-10T01:01:58.024640+0000","last_peered":"2023-09-10T01:01:58.024640+0000","last_clean":"2023-09-10T01:01:58.024640+0000","last_became_active":"2023-09-10T01:00:43.009376+0000","last_became_peered":"2023-09-10T01:00:43.009376+0000","last_unstale":"2023-09-10T01:01:58.024640+0000","last_undegraded":"2023-09-10T01:01:58.024640+0000","last_fullsized":"2023-09-10T01:01:58.024640+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:08.761672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.882436+0000","last_change":"2023-09-10T01:00:44.030175+0000","last_active":"2023-09-10T01:02:25.882436+0000","last_peered":"2023-09-10T01:02:25.882436+0000","last_clean":"2023-09-10T01:02:25.882436+0000","last_became_active":"2023-09-10T01:00:44.029748+0000","last_became_peered":"2023-09-10T01:00:44.029748+0000","last_unstale":"2023-09-10T01:02:25.882436+0000","last_undegraded":"2023-09-10T01:02:25.882436+0000","last_fullsized":"2023-09-10T01:02:25.882436+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:51:15.149610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.032036+0000","last_change":"2023-09-10T01:00:48.428008+0000","last_active":"2023-09-10T01:01:56.032036+0000","last_peered":"2023-09-10T01:01:56.032036+0000","last_clean":"2023-09-10T01:01:56.032036+0000","last_became_active":"2023-09-10T01:00:48.427498+0000","last_became_peered":"2023-09-10T01:00:48.427498+0000","last_unstale":"2023-09-10T01:01:56.032036+0000","last_undegraded":"2023-09-10T01:01:56.032036+0000","last_fullsized":"2023-09-10T01:01:56.032036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:16:48.938006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872721+0000","last_change":"2023-09-10T01:01:57.107918+0000","last_active":"2023-09-10T01:02:25.872721+0000","last_peered":"2023-09-10T01:02:25.872721+0000","last_clean":"2023-09-10T01:02:25.872721+0000","last_became_active":"2023-09-10T01:01:57.107413+0000","last_became_peered":"2023-09-10T01:01:57.107413+0000","last_unstale":"2023-09-10T01:02:25.872721+0000","last_undegraded":"2023-09-10T01:02:25.872721+0000","last_fullsized":"2023-09-10T01:02:25.872721+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:16:33.836871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":134,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:55.145098+0000","last_change":"2023-09-10T01:00:43.001047+0000","last_active":"2023-09-10T01:02:55.145098+0000","last_peered":"2023-09-10T01:02:55.145098+0000","last_clean":"2023-09-10T01:02:55.145098+0000","last_became_active":"2023-09-10T01:00:43.000521+0000","last_became_peered":"2023-09-10T01:00:43.000521+0000","last_unstale":"2023-09-10T01:02:55.145098+0000","last_undegraded":"2023-09-10T01:02:55.145098+0000","last_fullsized":"2023-09-10T01:02:55.145098+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:15:34.509964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024504+0000","last_change":"2023-09-10T01:00:48.062148+0000","last_active":"2023-09-10T01:01:58.024504+0000","last_peered":"2023-09-10T01:01:58.024504+0000","last_clean":"2023-09-10T01:01:58.024504+0000","last_became_active":"2023-09-10T01:00:48.061746+0000","last_became_peered":"2023-09-10T01:00:48.061746+0000","last_unstale":"2023-09-10T01:01:58.024504+0000","last_undegraded":"2023-09-10T01:01:58.024504+0000","last_fullsized":"2023-09-10T01:01:58.024504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:12:40.087012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889845+0000","last_change":"2023-09-10T01:00:44.030500+0000","last_active":"2023-09-10T01:02:25.889845+0000","last_peered":"2023-09-10T01:02:25.889845+0000","last_clean":"2023-09-10T01:02:25.889845+0000","last_became_active":"2023-09-10T01:00:44.030209+0000","last_became_peered":"2023-09-10T01:00:44.030209+0000","last_unstale":"2023-09-10T01:02:25.889845+0000","last_undegraded":"2023-09-10T01:02:25.889845+0000","last_fullsized":"2023-09-10T01:02:25.889845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:53:30.421493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.040869+0000","last_change":"2023-09-10T01:00:43.001102+0000","last_active":"2023-09-10T01:01:57.040869+0000","last_peered":"2023-09-10T01:01:57.040869+0000","last_clean":"2023-09-10T01:01:57.040869+0000","last_became_active":"2023-09-10T01:00:43.000715+0000","last_became_peered":"2023-09-10T01:00:43.000715+0000","last_unstale":"2023-09-10T01:01:57.040869+0000","last_undegraded":"2023-09-10T01:01:57.040869+0000","last_fullsized":"2023-09-10T01:01:57.040869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:43:37.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":327,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024171+0000","last_change":"2023-09-10T00:59:40.261244+0000","last_active":"2023-09-10T01:01:58.024171+0000","last_peered":"2023-09-10T01:01:58.024171+0000","last_clean":"2023-09-10T01:01:58.024171+0000","last_became_active":"2023-09-10T00:59:40.260673+0000","last_became_peered":"2023-09-10T00:59:40.260673+0000","last_unstale":"2023-09-10T01:01:58.024171+0000","last_undegraded":"2023-09-10T01:01:58.024171+0000","last_fullsized":"2023-09-10T01:01:58.024171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T00:56:36.374323+0000","last_clean_scrub_stamp":"2023-09-10T00:56:36.374323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:04:03.614104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.044220+0000","last_change":"2023-09-10T01:00:57.052019+0000","last_active":"2023-09-10T01:01:57.044220+0000","last_peered":"2023-09-10T01:01:57.044220+0000","last_clean":"2023-09-10T01:01:57.044220+0000","last_became_active":"2023-09-10T01:00:57.051527+0000","last_became_peered":"2023-09-10T01:00:57.051527+0000","last_unstale":"2023-09-10T01:01:57.044220+0000","last_undegraded":"2023-09-10T01:01:57.044220+0000","last_fullsized":"2023-09-10T01:01:57.044220+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:10:43.756761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.889771+0000","last_change":"2023-09-10T01:00:44.033214+0000","last_active":"2023-09-10T01:02:25.889771+0000","last_peered":"2023-09-10T01:02:25.889771+0000","last_clean":"2023-09-10T01:02:25.889771+0000","last_became_active":"2023-09-10T01:00:44.032828+0000","last_became_peered":"2023-09-10T01:00:44.032828+0000","last_unstale":"2023-09-10T01:02:25.889771+0000","last_undegraded":"2023-09-10T01:02:25.889771+0000","last_fullsized":"2023-09-10T01:02:25.889771+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:01:22.417216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.024928+0000","last_change":"2023-09-10T01:00:56.965456+0000","last_active":"2023-09-10T01:01:58.024928+0000","last_peered":"2023-09-10T01:01:58.024928+0000","last_clean":"2023-09-10T01:01:58.024928+0000","last_became_active":"2023-09-10T01:00:56.965021+0000","last_became_peered":"2023-09-10T01:00:56.965021+0000","last_unstale":"2023-09-10T01:01:58.024928+0000","last_undegraded":"2023-09-10T01:01:58.024928+0000","last_fullsized":"2023-09-10T01:01:58.024928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:19:19.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:58.731761+0000","last_change":"2023-09-10T01:00:48.066811+0000","last_active":"2023-09-10T01:01:58.731761+0000","last_peered":"2023-09-10T01:01:58.731761+0000","last_clean":"2023-09-10T01:01:58.731761+0000","last_became_active":"2023-09-10T01:00:48.066421+0000","last_became_peered":"2023-09-10T01:00:48.066421+0000","last_unstale":"2023-09-10T01:01:58.731761+0000","last_undegraded":"2023-09-10T01:01:58.731761+0000","last_fullsized":"2023-09-10T01:01:58.731761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:48:01.459070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.106555+0000","last_change":"2023-09-10T01:00:43.011134+0000","last_active":"2023-09-10T01:01:57.106555+0000","last_peered":"2023-09-10T01:01:57.106555+0000","last_clean":"2023-09-10T01:01:57.106555+0000","last_became_active":"2023-09-10T01:00:43.010871+0000","last_became_peered":"2023-09-10T01:00:43.010871+0000","last_unstale":"2023-09-10T01:01:57.106555+0000","last_undegraded":"2023-09-10T01:01:57.106555+0000","last_fullsized":"2023-09-10T01:01:57.106555+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:08:56.903745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890244+0000","last_change":"2023-09-10T01:00:44.031104+0000","last_active":"2023-09-10T01:02:25.890244+0000","last_peered":"2023-09-10T01:02:25.890244+0000","last_clean":"2023-09-10T01:02:25.890244+0000","last_became_active":"2023-09-10T01:00:44.030745+0000","last_became_peered":"2023-09-10T01:00:44.030745+0000","last_unstale":"2023-09-10T01:02:25.890244+0000","last_undegraded":"2023-09-10T01:02:25.890244+0000","last_fullsized":"2023-09-10T01:02:25.890244+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:14:53.885886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.470456+0000","last_change":"2023-09-10T01:00:48.427735+0000","last_active":"2023-09-10T01:01:57.470456+0000","last_peered":"2023-09-10T01:01:57.470456+0000","last_clean":"2023-09-10T01:01:57.470456+0000","last_became_active":"2023-09-10T01:00:48.427342+0000","last_became_peered":"2023-09-10T01:00:48.427342+0000","last_unstale":"2023-09-10T01:01:57.470456+0000","last_undegraded":"2023-09-10T01:01:57.470456+0000","last_fullsized":"2023-09-10T01:01:57.470456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:01:52.007325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:53.813562+0000","last_change":"2023-09-10T01:00:43.006314+0000","last_active":"2023-09-10T01:02:53.813562+0000","last_peered":"2023-09-10T01:02:53.813562+0000","last_clean":"2023-09-10T01:02:53.813562+0000","last_became_active":"2023-09-10T01:00:43.004835+0000","last_became_peered":"2023-09-10T01:00:43.004835+0000","last_unstale":"2023-09-10T01:02:53.813562+0000","last_undegraded":"2023-09-10T01:02:53.813562+0000","last_fullsized":"2023-09-10T01:02:53.813562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:37:09.359846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891331+0000","last_change":"2023-09-10T01:00:44.029839+0000","last_active":"2023-09-10T01:02:25.891331+0000","last_peered":"2023-09-10T01:02:25.891331+0000","last_clean":"2023-09-10T01:02:25.891331+0000","last_became_active":"2023-09-10T01:00:44.029492+0000","last_became_peered":"2023-09-10T01:00:44.029492+0000","last_unstale":"2023-09-10T01:02:25.891331+0000","last_undegraded":"2023-09-10T01:02:25.891331+0000","last_fullsized":"2023-09-10T01:02:25.891331+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:14.357386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.047012+0000","last_change":"2023-09-10T01:00:48.054536+0000","last_active":"2023-09-10T01:01:57.047012+0000","last_peered":"2023-09-10T01:01:57.047012+0000","last_clean":"2023-09-10T01:01:57.047012+0000","last_became_active":"2023-09-10T01:00:48.053938+0000","last_became_peered":"2023-09-10T01:00:48.053938+0000","last_unstale":"2023-09-10T01:01:57.047012+0000","last_undegraded":"2023-09-10T01:01:57.047012+0000","last_fullsized":"2023-09-10T01:01:57.047012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:45:12.703592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.041448+0000","last_change":"2023-09-10T01:00:43.002313+0000","last_active":"2023-09-10T01:01:57.041448+0000","last_peered":"2023-09-10T01:01:57.041448+0000","last_clean":"2023-09-10T01:01:57.041448+0000","last_became_active":"2023-09-10T01:00:43.001839+0000","last_became_peered":"2023-09-10T01:00:43.001839+0000","last_unstale":"2023-09-10T01:01:57.041448+0000","last_undegraded":"2023-09-10T01:01:57.041448+0000","last_fullsized":"2023-09-10T01:01:57.041448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:51.421843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.880327+0000","last_change":"2023-09-10T01:01:57.108269+0000","last_active":"2023-09-10T01:02:25.880327+0000","last_peered":"2023-09-10T01:02:25.880327+0000","last_clean":"2023-09-10T01:02:25.880327+0000","last_became_active":"2023-09-10T01:01:57.107879+0000","last_became_peered":"2023-09-10T01:01:57.107879+0000","last_unstale":"2023-09-10T01:02:25.880327+0000","last_undegraded":"2023-09-10T01:02:25.880327+0000","last_fullsized":"2023-09-10T01:02:25.880327+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:47:09.135978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037126+0000","last_change":"2023-09-10T01:00:48.428406+0000","last_active":"2023-09-10T01:01:56.037126+0000","last_peered":"2023-09-10T01:01:56.037126+0000","last_clean":"2023-09-10T01:01:56.037126+0000","last_became_active":"2023-09-10T01:00:48.427938+0000","last_became_peered":"2023-09-10T01:00:48.427938+0000","last_unstale":"2023-09-10T01:01:56.037126+0000","last_undegraded":"2023-09-10T01:01:56.037126+0000","last_fullsized":"2023-09-10T01:01:56.037126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:41:52.316235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.138071+0000","last_change":"2023-09-10T01:00:56.963755+0000","last_active":"2023-09-10T01:01:57.138071+0000","last_peered":"2023-09-10T01:01:57.138071+0000","last_clean":"2023-09-10T01:01:57.138071+0000","last_became_active":"2023-09-10T01:00:56.963283+0000","last_became_peered":"2023-09-10T01:00:56.963283+0000","last_unstale":"2023-09-10T01:01:57.138071+0000","last_undegraded":"2023-09-10T01:01:57.138071+0000","last_fullsized":"2023-09-10T01:01:57.138071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:13:32.213393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.864057+0000","last_change":"2023-09-10T01:00:44.025284+0000","last_active":"2023-09-10T01:02:25.864057+0000","last_peered":"2023-09-10T01:02:25.864057+0000","last_clean":"2023-09-10T01:02:25.864057+0000","last_became_active":"2023-09-10T01:00:44.024965+0000","last_became_peered":"2023-09-10T01:00:44.024965+0000","last_unstale":"2023-09-10T01:02:25.864057+0000","last_undegraded":"2023-09-10T01:02:25.864057+0000","last_fullsized":"2023-09-10T01:02:25.864057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:19:08.620325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033359+0000","last_change":"2023-09-10T01:00:48.070843+0000","last_active":"2023-09-10T01:01:56.033359+0000","last_peered":"2023-09-10T01:01:56.033359+0000","last_clean":"2023-09-10T01:01:56.033359+0000","last_became_active":"2023-09-10T01:00:48.070350+0000","last_became_peered":"2023-09-10T01:00:48.070350+0000","last_unstale":"2023-09-10T01:01:56.033359+0000","last_undegraded":"2023-09-10T01:01:56.033359+0000","last_fullsized":"2023-09-10T01:01:56.033359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:35:25.196036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.470530+0000","last_change":"2023-09-10T01:00:43.005398+0000","last_active":"2023-09-10T01:01:57.470530+0000","last_peered":"2023-09-10T01:01:57.470530+0000","last_clean":"2023-09-10T01:01:57.470530+0000","last_became_active":"2023-09-10T01:00:43.005087+0000","last_became_peered":"2023-09-10T01:00:43.005087+0000","last_unstale":"2023-09-10T01:01:57.470530+0000","last_undegraded":"2023-09-10T01:01:57.470530+0000","last_fullsized":"2023-09-10T01:01:57.470530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:29:04.042683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871590+0000","last_change":"2023-09-10T01:00:44.024451+0000","last_active":"2023-09-10T01:02:25.871590+0000","last_peered":"2023-09-10T01:02:25.871590+0000","last_clean":"2023-09-10T01:02:25.871590+0000","last_became_active":"2023-09-10T01:00:44.024045+0000","last_became_peered":"2023-09-10T01:00:44.024045+0000","last_unstale":"2023-09-10T01:02:25.871590+0000","last_undegraded":"2023-09-10T01:02:25.871590+0000","last_fullsized":"2023-09-10T01:02:25.871590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:55:50.750994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037240+0000","last_change":"2023-09-10T01:00:48.062247+0000","last_active":"2023-09-10T01:01:56.037240+0000","last_peered":"2023-09-10T01:01:56.037240+0000","last_clean":"2023-09-10T01:01:56.037240+0000","last_became_active":"2023-09-10T01:00:48.061812+0000","last_became_peered":"2023-09-10T01:00:48.061812+0000","last_unstale":"2023-09-10T01:01:56.037240+0000","last_undegraded":"2023-09-10T01:01:56.037240+0000","last_fullsized":"2023-09-10T01:01:56.037240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:44:13.596534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.047127+0000","last_change":"2023-09-10T01:00:43.010845+0000","last_active":"2023-09-10T01:01:57.047127+0000","last_peered":"2023-09-10T01:01:57.047127+0000","last_clean":"2023-09-10T01:01:57.047127+0000","last_became_active":"2023-09-10T01:00:43.010511+0000","last_became_peered":"2023-09-10T01:00:43.010511+0000","last_unstale":"2023-09-10T01:01:57.047127+0000","last_undegraded":"2023-09-10T01:01:57.047127+0000","last_fullsized":"2023-09-10T01:01:57.047127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T10:33:07.229023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.891545+0000","last_change":"2023-09-10T01:00:44.029025+0000","last_active":"2023-09-10T01:02:25.891545+0000","last_peered":"2023-09-10T01:02:25.891545+0000","last_clean":"2023-09-10T01:02:25.891545+0000","last_became_active":"2023-09-10T01:00:44.028713+0000","last_became_peered":"2023-09-10T01:00:44.028713+0000","last_unstale":"2023-09-10T01:02:25.891545+0000","last_undegraded":"2023-09-10T01:02:25.891545+0000","last_fullsized":"2023-09-10T01:02:25.891545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:40:22.315742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.037817+0000","last_change":"2023-09-10T01:00:48.427294+0000","last_active":"2023-09-10T01:01:56.037817+0000","last_peered":"2023-09-10T01:01:56.037817+0000","last_clean":"2023-09-10T01:01:56.037817+0000","last_became_active":"2023-09-10T01:00:48.426882+0000","last_became_peered":"2023-09-10T01:00:48.426882+0000","last_unstale":"2023-09-10T01:01:56.037817+0000","last_undegraded":"2023-09-10T01:01:56.037817+0000","last_fullsized":"2023-09-10T01:01:56.037817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:54:38.325230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.035921+0000","last_change":"2023-09-10T01:00:43.006175+0000","last_active":"2023-09-10T01:01:56.035921+0000","last_peered":"2023-09-10T01:01:56.035921+0000","last_clean":"2023-09-10T01:01:56.035921+0000","last_became_active":"2023-09-10T01:00:43.005786+0000","last_became_peered":"2023-09-10T01:00:43.005786+0000","last_unstale":"2023-09-10T01:01:56.035921+0000","last_undegraded":"2023-09-10T01:01:56.035921+0000","last_fullsized":"2023-09-10T01:01:56.035921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:14:47.074866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887993+0000","last_change":"2023-09-10T01:00:44.041303+0000","last_active":"2023-09-10T01:02:25.887993+0000","last_peered":"2023-09-10T01:02:25.887993+0000","last_clean":"2023-09-10T01:02:25.887993+0000","last_became_active":"2023-09-10T01:00:44.041009+0000","last_became_peered":"2023-09-10T01:00:44.041009+0000","last_unstale":"2023-09-10T01:02:25.887993+0000","last_undegraded":"2023-09-10T01:02:25.887993+0000","last_fullsized":"2023-09-10T01:02:25.887993+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:53:23.001700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.036842+0000","last_change":"2023-09-10T01:00:48.064314+0000","last_active":"2023-09-10T01:01:56.036842+0000","last_peered":"2023-09-10T01:01:56.036842+0000","last_clean":"2023-09-10T01:01:56.036842+0000","last_became_active":"2023-09-10T01:00:48.063813+0000","last_became_peered":"2023-09-10T01:00:48.063813+0000","last_unstale":"2023-09-10T01:01:56.036842+0000","last_undegraded":"2023-09-10T01:01:56.036842+0000","last_fullsized":"2023-09-10T01:01:56.036842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:00:10.002006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.033268+0000","last_change":"2023-09-10T01:00:43.007579+0000","last_active":"2023-09-10T01:01:56.033268+0000","last_peered":"2023-09-10T01:01:56.033268+0000","last_clean":"2023-09-10T01:01:56.033268+0000","last_became_active":"2023-09-10T01:00:43.007200+0000","last_became_peered":"2023-09-10T01:00:43.007200+0000","last_unstale":"2023-09-10T01:01:56.033268+0000","last_undegraded":"2023-09-10T01:01:56.033268+0000","last_fullsized":"2023-09-10T01:01:56.033268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:20:08.688531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883972+0000","last_change":"2023-09-10T01:00:44.033511+0000","last_active":"2023-09-10T01:02:25.883972+0000","last_peered":"2023-09-10T01:02:25.883972+0000","last_clean":"2023-09-10T01:02:25.883972+0000","last_became_active":"2023-09-10T01:00:44.033170+0000","last_became_peered":"2023-09-10T01:00:44.033170+0000","last_unstale":"2023-09-10T01:02:25.883972+0000","last_undegraded":"2023-09-10T01:02:25.883972+0000","last_fullsized":"2023-09-10T01:02:25.883972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:36:12.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.039255+0000","last_change":"2023-09-10T01:00:48.067436+0000","last_active":"2023-09-10T01:01:56.039255+0000","last_peered":"2023-09-10T01:01:56.039255+0000","last_clean":"2023-09-10T01:01:56.039255+0000","last_became_active":"2023-09-10T01:00:48.067038+0000","last_became_peered":"2023-09-10T01:00:48.067038+0000","last_unstale":"2023-09-10T01:01:56.039255+0000","last_undegraded":"2023-09-10T01:01:56.039255+0000","last_fullsized":"2023-09-10T01:01:56.039255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:26:41.210698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.044273+0000","last_change":"2023-09-10T01:00:43.006934+0000","last_active":"2023-09-10T01:01:57.044273+0000","last_peered":"2023-09-10T01:01:57.044273+0000","last_clean":"2023-09-10T01:01:57.044273+0000","last_became_active":"2023-09-10T01:00:43.005787+0000","last_became_peered":"2023-09-10T01:00:43.005787+0000","last_unstale":"2023-09-10T01:01:57.044273+0000","last_undegraded":"2023-09-10T01:01:57.044273+0000","last_fullsized":"2023-09-10T01:01:57.044273+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T11:48:27.403220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879574+0000","last_change":"2023-09-10T01:00:44.033812+0000","last_active":"2023-09-10T01:02:25.879574+0000","last_peered":"2023-09-10T01:02:25.879574+0000","last_clean":"2023-09-10T01:02:25.879574+0000","last_became_active":"2023-09-10T01:00:44.033496+0000","last_became_peered":"2023-09-10T01:00:44.033496+0000","last_unstale":"2023-09-10T01:02:25.879574+0000","last_undegraded":"2023-09-10T01:02:25.879574+0000","last_fullsized":"2023-09-10T01:02:25.879574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T09:43:12.679368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:01:57.043542+0000","last_change":"2023-09-10T01:00:48.060358+0000","last_active":"2023-09-10T01:01:57.043542+0000","last_peered":"2023-09-10T01:01:57.043542+0000","last_clean":"2023-09-10T01:01:57.043542+0000","last_became_active":"2023-09-10T01:00:48.059853+0000","last_became_peered":"2023-09-10T01:00:48.059853+0000","last_unstale":"2023-09-10T01:01:57.043542+0000","last_undegraded":"2023-09-10T01:01:57.043542+0000","last_fullsized":"2023-09-10T01:01:57.043542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:47.021033+0000","last_clean_scrub_stamp":"2023-09-10T01:00:47.021033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T04:31:56.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-09-10T01:01:56.031904+0000","last_change":"2023-09-10T01:00:43.002820+0000","last_active":"2023-09-10T01:01:56.031904+0000","last_peered":"2023-09-10T01:01:56.031904+0000","last_clean":"2023-09-10T01:01:56.031904+0000","last_became_active":"2023-09-10T01:00:43.002456+0000","last_became_peered":"2023-09-10T01:00:43.002456+0000","last_unstale":"2023-09-10T01:01:56.031904+0000","last_undegraded":"2023-09-10T01:01:56.031904+0000","last_fullsized":"2023-09-10T01:01:56.031904+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:41.968928+0000","last_clean_scrub_stamp":"2023-09-10T01:00:41.968928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:40:10.189942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.879728+0000","last_change":"2023-09-10T01:00:44.032655+0000","last_active":"2023-09-10T01:02:25.879728+0000","last_peered":"2023-09-10T01:02:25.879728+0000","last_clean":"2023-09-10T01:02:25.879728+0000","last_became_active":"2023-09-10T01:00:44.032157+0000","last_became_peered":"2023-09-10T01:00:44.032157+0000","last_unstale":"2023-09-10T01:02:25.879728+0000","last_undegraded":"2023-09-10T01:02:25.879728+0000","last_fullsized":"2023-09-10T01:02:25.879728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:53:31.176768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.872475+0000","last_change":"2023-09-10T01:00:44.022285+0000","last_active":"2023-09-10T01:02:25.872475+0000","last_peered":"2023-09-10T01:02:25.872475+0000","last_clean":"2023-09-10T01:02:25.872475+0000","last_became_active":"2023-09-10T01:00:44.021959+0000","last_became_peered":"2023-09-10T01:00:44.021959+0000","last_unstale":"2023-09-10T01:02:25.872475+0000","last_undegraded":"2023-09-10T01:02:25.872475+0000","last_fullsized":"2023-09-10T01:02:25.872475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:13:07.072249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.873581+0000","last_change":"2023-09-10T01:00:44.029722+0000","last_active":"2023-09-10T01:02:25.873581+0000","last_peered":"2023-09-10T01:02:25.873581+0000","last_clean":"2023-09-10T01:02:25.873581+0000","last_became_active":"2023-09-10T01:00:44.029424+0000","last_became_peered":"2023-09-10T01:00:44.029424+0000","last_unstale":"2023-09-10T01:02:25.873581+0000","last_undegraded":"2023-09-10T01:02:25.873581+0000","last_fullsized":"2023-09-10T01:02:25.873581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:24:16.655538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.883202+0000","last_change":"2023-09-10T01:00:44.026026+0000","last_active":"2023-09-10T01:02:25.883202+0000","last_peered":"2023-09-10T01:02:25.883202+0000","last_clean":"2023-09-10T01:02:25.883202+0000","last_became_active":"2023-09-10T01:00:44.025727+0000","last_became_peered":"2023-09-10T01:00:44.025727+0000","last_unstale":"2023-09-10T01:02:25.883202+0000","last_undegraded":"2023-09-10T01:02:25.883202+0000","last_fullsized":"2023-09-10T01:02:25.883202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T12:41:59.379750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890507+0000","last_change":"2023-09-10T01:00:56.965355+0000","last_active":"2023-09-10T01:02:25.890507+0000","last_peered":"2023-09-10T01:02:25.890507+0000","last_clean":"2023-09-10T01:02:25.890507+0000","last_became_active":"2023-09-10T01:00:56.964903+0000","last_became_peered":"2023-09-10T01:00:56.964903+0000","last_unstale":"2023-09-10T01:02:25.890507+0000","last_undegraded":"2023-09-10T01:02:25.890507+0000","last_fullsized":"2023-09-10T01:02:25.890507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:49:21.940159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890130+0000","last_change":"2023-09-10T01:00:44.029525+0000","last_active":"2023-09-10T01:02:25.890130+0000","last_peered":"2023-09-10T01:02:25.890130+0000","last_clean":"2023-09-10T01:02:25.890130+0000","last_became_active":"2023-09-10T01:00:44.029139+0000","last_became_peered":"2023-09-10T01:00:44.029139+0000","last_unstale":"2023-09-10T01:02:25.890130+0000","last_undegraded":"2023-09-10T01:02:25.890130+0000","last_fullsized":"2023-09-10T01:02:25.890130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:20:24.855890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.871853+0000","last_change":"2023-09-10T01:00:44.022299+0000","last_active":"2023-09-10T01:02:25.871853+0000","last_peered":"2023-09-10T01:02:25.871853+0000","last_clean":"2023-09-10T01:02:25.871853+0000","last_became_active":"2023-09-10T01:00:44.021992+0000","last_became_peered":"2023-09-10T01:00:44.021992+0000","last_unstale":"2023-09-10T01:02:25.871853+0000","last_undegraded":"2023-09-10T01:02:25.871853+0000","last_fullsized":"2023-09-10T01:02:25.871853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T08:50:30.766442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.870556+0000","last_change":"2023-09-10T01:01:57.108199+0000","last_active":"2023-09-10T01:02:25.870556+0000","last_peered":"2023-09-10T01:02:25.870556+0000","last_clean":"2023-09-10T01:02:25.870556+0000","last_became_active":"2023-09-10T01:01:57.107825+0000","last_became_peered":"2023-09-10T01:01:57.107825+0000","last_unstale":"2023-09-10T01:02:25.870556+0000","last_undegraded":"2023-09-10T01:02:25.870556+0000","last_fullsized":"2023-09-10T01:02:25.870556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:31:15.499277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,9],"acting":[2,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":13,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.876680+0000","last_change":"2023-09-10T01:01:57.049936+0000","last_active":"2023-09-10T01:02:25.876680+0000","last_peered":"2023-09-10T01:02:25.876680+0000","last_clean":"2023-09-10T01:02:25.876680+0000","last_became_active":"2023-09-10T01:01:57.049624+0000","last_became_peered":"2023-09-10T01:01:57.049624+0000","last_unstale":"2023-09-10T01:02:25.876680+0000","last_undegraded":"2023-09-10T01:02:25.876680+0000","last_fullsized":"2023-09-10T01:02:25.876680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T03:57:17.164558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887999+0000","last_change":"2023-09-10T01:00:44.025405+0000","last_active":"2023-09-10T01:02:25.887999+0000","last_peered":"2023-09-10T01:02:25.887999+0000","last_clean":"2023-09-10T01:02:25.887999+0000","last_became_active":"2023-09-10T01:00:44.025040+0000","last_became_peered":"2023-09-10T01:00:44.025040+0000","last_unstale":"2023-09-10T01:02:25.887999+0000","last_undegraded":"2023-09-10T01:02:25.887999+0000","last_fullsized":"2023-09-10T01:02:25.887999+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:16:46.504902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.887060+0000","last_change":"2023-09-10T01:00:44.023603+0000","last_active":"2023-09-10T01:02:25.887060+0000","last_peered":"2023-09-10T01:02:25.887060+0000","last_clean":"2023-09-10T01:02:25.887060+0000","last_became_active":"2023-09-10T01:00:44.023279+0000","last_became_peered":"2023-09-10T01:00:44.023279+0000","last_unstale":"2023-09-10T01:02:25.887060+0000","last_undegraded":"2023-09-10T01:02:25.887060+0000","last_fullsized":"2023-09-10T01:02:25.887060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T07:18:05.737675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.892481+0000","last_change":"2023-09-10T01:00:44.035653+0000","last_active":"2023-09-10T01:02:25.892481+0000","last_peered":"2023-09-10T01:02:25.892481+0000","last_clean":"2023-09-10T01:02:25.892481+0000","last_became_active":"2023-09-10T01:00:44.035374+0000","last_became_peered":"2023-09-10T01:00:44.035374+0000","last_unstale":"2023-09-10T01:02:25.892481+0000","last_undegraded":"2023-09-10T01:02:25.892481+0000","last_fullsized":"2023-09-10T01:02:25.892481+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T06:14:45.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.890327+0000","last_change":"2023-09-10T01:00:44.035805+0000","last_active":"2023-09-10T01:02:25.890327+0000","last_peered":"2023-09-10T01:02:25.890327+0000","last_clean":"2023-09-10T01:02:25.890327+0000","last_became_active":"2023-09-10T01:00:44.035479+0000","last_became_peered":"2023-09-10T01:00:44.035479+0000","last_unstale":"2023-09-10T01:02:25.890327+0000","last_undegraded":"2023-09-10T01:02:25.890327+0000","last_fullsized":"2023-09-10T01:02:25.890327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T01:32:36.662975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-09-10T01:02:25.878041+0000","last_change":"2023-09-10T01:00:44.025418+0000","last_active":"2023-09-10T01:02:25.878041+0000","last_peered":"2023-09-10T01:02:25.878041+0000","last_clean":"2023-09-10T01:02:25.878041+0000","last_became_active":"2023-09-10T01:00:44.025104+0000","last_became_peered":"2023-09-10T01:00:44.025104+0000","last_unstale":"2023-09-10T01:02:25.878041+0000","last_undegraded":"2023-09-10T01:02:25.878041+0000","last_fullsized":"2023-09-10T01:02:25.878041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-10T01:00:42.983413+0000","last_clean_scrub_stamp":"2023-09-10T01:00:42.983413+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-11T05:05:13.050781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282844,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69231,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":677,"num_read_kb":413,"num_write":166,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":207693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481830,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":504243,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69199999999999995,"5min":0.87,"15min":0.87},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.60799999999999998,"15min":0.56399999999999995},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9509999999999996,"5min":5.9509999999999996,"15min":5.9509999999999996},"last":0.499}]},{"osd":1,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59599999999999997,"15min":0.58899999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.626,"15min":0.625},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82299999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53100000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45500000000000002,"15min":0.45600000000000002},"min":{"1min":0.27800000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.74399999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.65500000000000003,"15min":0.64900000000000002},"min":{"1min":0.53000000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73199999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.74399999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.75}]},{"osd":6,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66300000000000003,"15min":0.67100000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81799999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.69799999999999995},"min":{"1min":0.495,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.83299999999999996,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63600000000000001}]},{"osd":7,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69399999999999995,"15min":0.69199999999999995},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.85599999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.72699999999999998,"15min":0.72299999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.85899999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.65000000000000002},"min":{"1min":0.46200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.74299999999999999}]},{"osd":9,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.748,"15min":0.73899999999999999},"min":{"1min":0.52400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.747,"15min":0.74399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.92300000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.873}]},{"osd":10,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93100000000000005,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.77100000000000002,"15min":0.76600000000000001},"min":{"1min":0.55900000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80100000000000005}]}]},{"osd":10,"up_from":70,"seq":300647710762,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1156019,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.65900000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55900000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.441,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58099999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.47199999999999998,"15min":0.47499999999999998},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.499}]},{"osd":2,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60699999999999998,"15min":0.60499999999999998},"min":{"1min":0.432,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80700000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61799999999999999,"15min":0.629},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60499999999999998,"15min":0.626},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.753,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.629},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.78900000000000003,"15min":0.85599999999999998},"min":{"1min":0.44400000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78000000000000003,"5min":6.7519999999999998,"15min":6.7519999999999998},"last":0.67800000000000005}]},{"osd":4,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71299999999999997,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.36299999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.65900000000000003}]},{"osd":5,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.69499999999999995},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.99099999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.42199999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85799999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.87,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.71099999999999997},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91300000000000003,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.82899999999999996}]},{"osd":7,"last update":"Sun Sep 10 01:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]},{"osd":8,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.76400000000000001,"15min":0.76800000000000002},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.3069999999999999,"5min":1.534,"15min":1.534},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75700000000000001,"15min":0.77500000000000002},"min":{"1min":0.59099999999999997,"5min":0.439,"15min":0.439},"max":{"1min":0.88800000000000001,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.79500000000000004}]},{"osd":9,"last update":"Sun Sep 10 01:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.58399999999999996,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.93200000000000005,"5min":1.304,"15min":1.304},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84899999999999998,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94299999999999995,"5min":7.0510000000000002,"15min":7.0510000000000002},"last":0.92100000000000004}]},{"osd":11,"last update":"Sun Sep 10 01:02:02 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73299999999999998,"15min":0.73099999999999998},"min":{"1min":0.50900000000000001,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93600000000000005,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.76400000000000001,"15min":0.75800000000000001},"min":{"1min":0.629,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.84499999999999997}]}]},{"osd":9,"up_from":65,"seq":279172874287,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":507993,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55800000000000005,"15min":0.58399999999999996},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80200000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67500000000000004,"15min":0.68700000000000006},"min":{"1min":0.434,"5min":0.318,"15min":0.318},"max":{"1min":1.0069999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.73599999999999999},"min":{"1min":0.41199999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86499999999999999,"5min":1.349,"15min":1.349},"last":0.73499999999999999}]},{"osd":2,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71299999999999997,"15min":0.70999999999999996},"min":{"1min":0.57199999999999995,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.78100000000000003,"15min":0.81599999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.069,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.90200000000000002}]},{"osd":3,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68300000000000005,"15min":0.70599999999999996},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.0860000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.751},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.42499999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.2729999999999999,"5min":1.647,"15min":1.647},"last":0.61299999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.80300000000000005,"15min":0.83599999999999997},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.369,"5min":1.595,"15min":1.595},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.82699999999999996,"15min":0.85899999999999999},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0129999999999999,"5min":1.492,"15min":1.492},"last":0.79200000000000004}]},{"osd":5,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.88400000000000001,"15min":0.92700000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1890000000000001,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84099999999999997,"15min":0.873},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3460000000000001,"5min":1.714,"15min":1.714},"last":1.01}]},{"osd":6,"last update":"Sun Sep 10 01:00:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.97199999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.319,"5min":1.982,"15min":1.982},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.88900000000000001,"15min":0.92500000000000004},"min":{"1min":0.54700000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4330000000000001,"5min":1.569,"15min":1.569},"last":0.93100000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.94699999999999995,"15min":0.999},"min":{"1min":0.54500000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.294,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.97099999999999997,"15min":1.0109999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1759999999999999,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.96699999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.92200000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93300000000000005,"15min":0.94999999999999996},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.996}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.87,"15min":0.85699999999999998},"min":{"1min":0.69099999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.84399999999999997,"15min":0.84199999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":1.0549999999999999}]}]},{"osd":8,"up_from":57,"seq":244813135924,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":503486,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80200000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.68700000000000006,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51800000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.86499999999999999,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.878,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.39400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.47399999999999998,"15min":0.45400000000000001},"min":{"1min":0.35999999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.73099999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.34499999999999997}]},{"osd":3,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.627},"min":{"1min":0.443,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81699999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.69099999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.624}]},{"osd":4,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.56100000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.97199999999999998,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82799999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.66800000000000004}]},{"osd":5,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.60399999999999998,"15min":0.58799999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995}]},{"osd":6,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.52100000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73999999999999999,"15min":0.73299999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.004,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.70499999999999996,"15min":0.67000000000000004},"min":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.72899999999999998,"15min":0.70899999999999996},"min":{"1min":0.60099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":9,"last update":"Sun Sep 10 01:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78900000000000003,"15min":0.76700000000000002},"min":{"1min":0.67100000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.78400000000000003,"15min":0.76400000000000001},"min":{"1min":0.63400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0249999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.81399999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.74299999999999999,"15min":0.71699999999999997},"min":{"1min":0.58499999999999996,"5min":0.44,"15min":0.44},"max":{"1min":0.99199999999999999,"5min":1.01,"15min":1.01},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.79000000000000004,"15min":0.751},"min":{"1min":0.65900000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.75}]},{"osd":11,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":1.071,"15min":1.071},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83399999999999996,"15min":0.83599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0069999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78900000000000003}]}]},{"osd":7,"up_from":50,"seq":214748364856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1099600,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91400000000000003,"5min":1.367,"15min":1.367},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.44400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80400000000000005,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.434,"15min":0.42099999999999999},"min":{"1min":0.34100000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.53000000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.378},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45900000000000002,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70299999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41999999999999998}]},{"osd":2,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.621,"15min":0.60399999999999998},"min":{"1min":0.47699999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.82399999999999995,"5min":1.101,"15min":1.101},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60899999999999999,"15min":0.58899999999999997},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80200000000000005,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63300000000000001,"15min":0.65300000000000002},"min":{"1min":0.42299999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.873,"5min":1.409,"15min":1.409},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.70999999999999996,"15min":0.68100000000000005},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.88600000000000001,"5min":7.1929999999999996,"15min":7.1929999999999996},"last":0.51300000000000001}]},{"osd":4,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52100000000000002,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68200000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54500000000000004,"15min":0.53700000000000003},"min":{"1min":0.33000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82099999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.44900000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.79400000000000004,"5min":1.038,"15min":1.038},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68300000000000005,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88900000000000001,"5min":1.179,"15min":1.179},"last":0.75900000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.81299999999999994,"15min":0.75700000000000001},"min":{"1min":0.44500000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84799999999999998,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.80800000000000005,"15min":0.78400000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95099999999999996,"5min":7.1760000000000002,"15min":7.1760000000000002},"last":0.66600000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.70699999999999996,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.012,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.70099999999999996,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98399999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.64000000000000001}]},{"osd":9,"last update":"Sun Sep 10 01:02:20 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.79600000000000004,"15min":0.78000000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":3.423,"5min":3.423,"15min":3.423},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.81699999999999995,"15min":0.79500000000000004},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.5249999999999999,"5min":3.5249999999999999,"15min":3.5249999999999999},"last":0.71999999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.746},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92900000000000005,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.62}]},{"osd":11,"last update":"Sun Sep 10 01:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.034,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85799999999999998,"15min":0.85999999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0780000000000001,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.83499999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528380,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":499073,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.65100000000000002,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.9890000000000008,"5min":9.9890000000000008,"15min":9.9890000000000008},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.59099999999999997,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.363,"5min":2.363,"15min":2.363},"last":0.54300000000000004}]},{"osd":1,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.67600000000000005,"15min":0.63100000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.577,"5min":0.89000000000000001,"15min":0.76800000000000002},"min":{"1min":0.50800000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.77900000000000003}]},{"osd":2,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.748,"15min":0.72899999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.327,"5min":0.84899999999999998,"15min":0.78700000000000003},"min":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":10.608000000000001,"5min":10.608000000000001,"15min":10.608000000000001},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.67300000000000004,"15min":0.64900000000000002},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.9750000000000001,"5min":0.96899999999999997,"15min":0.80700000000000005},"min":{"1min":0.46700000000000003,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":23.449999999999999,"5min":23.449999999999999,"15min":23.449999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.76700000000000002,"15min":0.75},"min":{"1min":0.49399999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.4380000000000002,"5min":2.4380000000000002,"15min":2.4380000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.77500000000000002,"15min":0.74099999999999999},"min":{"1min":0.60499999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.76200000000000001}]},{"osd":5,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.94399999999999995,"15min":0.88900000000000001},"min":{"1min":0.62,"5min":0.503,"15min":0.503},"max":{"1min":10.132,"5min":10.132,"15min":10.132},"last":0.497},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.85499999999999998,"15min":0.84899999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.81200000000000006}]},{"osd":7,"last update":"Sun Sep 10 01:02:34 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.94499999999999995,"15min":0.85999999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7969999999999999,"5min":10.081,"15min":10.081},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.93600000000000005,"15min":0.85999999999999999},"min":{"1min":0.55100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":6.9009999999999998,"5min":6.9009999999999998,"15min":6.9009999999999998},"last":0.88200000000000001}]},{"osd":8,"last update":"Sun Sep 10 01:01:58 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.92900000000000005,"15min":0.91700000000000004},"min":{"1min":0.64900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.753},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.89200000000000002,"15min":0.88600000000000001},"min":{"1min":0.65500000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.62}]},{"osd":9,"last update":"Sun Sep 10 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.093,"15min":1.0329999999999999},"min":{"1min":0.67300000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":12.683,"5min":12.683,"15min":12.683},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.0009999999999999,"15min":1.0289999999999999},"min":{"1min":0.66500000000000004,"5min":0.625,"15min":0.625},"max":{"1min":1.9990000000000001,"5min":1.9990000000000001,"15min":1.9990000000000001},"last":0.79400000000000004}]},{"osd":10,"last update":"Sun Sep 10 01:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.94599999999999995,"15min":0.95299999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.056,"15min":0.98799999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.9450000000000001,"5min":10.170999999999999,"15min":10.170999999999999},"last":0.86699999999999999}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.4970000000000001,"15min":1.5389999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":6.9349999999999996,"5min":10.247,"15min":10.247},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.075,"15min":1.105},"min":{"1min":0.71699999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.379,"5min":2.3660000000000001,"15min":2.3660000000000001},"last":0.90400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":563757,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.627,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":5.3129999999999997,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.64700000000000002,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.2859999999999996,"5min":5.2859999999999996,"15min":5.2859999999999996},"last":0.46000000000000002}]},{"osd":2,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.64800000000000002,"15min":0.56799999999999995},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.9569999999999999,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.58499999999999996,"15min":0.53700000000000003},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.61799999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.66500000000000004,"15min":0.751},"min":{"1min":0.45200000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.8919999999999999,"5min":5.0830000000000002,"15min":5.0830000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.69699999999999995,"15min":0.77900000000000003},"min":{"1min":0.52200000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.875,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.75}]},{"osd":4,"last update":"Sun Sep 10 01:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45900000000000002,"15min":0.38800000000000001},"min":{"1min":0.36599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.46500000000000002,"15min":0.39700000000000002},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.69399999999999995,"15min":0.59099999999999997},"min":{"1min":0.626,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.89000000000000001,"5min":4.9740000000000002,"15min":4.9740000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.62},"min":{"1min":0.57299999999999995,"5min":0.38,"15min":0.38},"max":{"1min":0.92500000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.67700000000000005}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.67900000000000005,"15min":0.64300000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.9769999999999999,"5min":2.9769999999999999,"15min":2.9769999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.012,"5min":0.73499999999999999,"15min":0.69299999999999995},"min":{"1min":0.53300000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":5.0970000000000004,"5min":5.0970000000000004,"15min":5.0970000000000004},"last":0.71599999999999997}]},{"osd":7,"last update":"Sun Sep 10 01:02:39 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55900000000000005},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60399999999999998,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.79000000000000004}]},{"osd":8,"last update":"Sun Sep 10 01:01:57 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67600000000000005,"15min":0.65800000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.68000000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.44,"15min":0.44},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.88}]},{"osd":9,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":0.81599999999999995,"15min":0.76000000000000001},"min":{"1min":0.60899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":4.9969999999999999,"5min":4.9969999999999999,"15min":4.9969999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.84699999999999998,"15min":0.80000000000000004},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":5.0380000000000003,"5min":5.0380000000000003,"15min":5.0380000000000003},"last":0.93899999999999995}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.71199999999999997,"15min":0.69399999999999995},"min":{"1min":0.46500000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.73299999999999998,"15min":0.70499999999999996},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81200000000000006}]},{"osd":11,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.86599999999999999,"15min":0.86399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0740000000000001,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.86799999999999999,"15min":0.86399999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1060000000000001,"5min":1.107,"15min":1.107},"last":0.89900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500461,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 10 01:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.67200000000000004,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.274,"5min":2.274,"15min":2.274},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.61799999999999999,"15min":0.56599999999999995},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.34399999999999997},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.996}]},{"osd":2,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69099999999999995,"15min":0.61399999999999999},"min":{"1min":0.379,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.038,"5min":1.26,"15min":1.26},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.77200000000000002,"15min":0.63800000000000001},"min":{"1min":0.47499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8999999999999999,"5min":6.1900000000000004,"15min":6.1900000000000004},"last":0.90800000000000003}]},{"osd":3,"last update":"Sun Sep 10 01:01:43 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.439},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.92200000000000004}]},{"osd":4,"last update":"Sun Sep 10 01:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66000000000000003,"15min":0.59199999999999997},"min":{"1min":0.42699999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.70099999999999996,"15min":0.62},"min":{"1min":0.53000000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.167,"5min":1.302,"15min":1.302},"last":0.95699999999999996}]},{"osd":5,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.747,"15min":0.71199999999999997},"min":{"1min":0.59799999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.028,"5min":1.125,"15min":1.125},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70699999999999996,"15min":0.66800000000000004},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.752}]},{"osd":6,"last update":"Sun Sep 10 01:02:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69999999999999996,"15min":0.67100000000000004},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76100000000000001,"15min":0.72099999999999997},"min":{"1min":0.41099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":0.72199999999999998}]},{"osd":7,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.79900000000000004},"min":{"1min":0.64300000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.58099999999999996,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":1.0329999999999999}]},{"osd":8,"last update":"Sun Sep 10 01:01:56 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.66400000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.0840000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83299999999999996,"15min":0.84299999999999997},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1499999999999999,"5min":1.24,"15min":1.24},"last":0.92600000000000005}]},{"osd":9,"last update":"Sun Sep 10 01:02:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.89100000000000001,"15min":0.92100000000000004},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3360000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.98399999999999999,"15min":1.01},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.046}]},{"osd":10,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.89000000000000001,"15min":0.91800000000000004},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.071},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.91300000000000003,"15min":0.93500000000000005},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.129,"5min":1.444,"15min":1.444},"last":1.0600000000000001}]},{"osd":11,"last update":"Sun Sep 10 01:02:03 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.92300000000000004,"15min":0.91800000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":1.093},{"interface":"front","average":{"1min":1.002,"5min":0.97799999999999998,"15min":0.97399999999999998},"min":{"1min":0.76700000000000002,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1020000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":564485,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.67000000000000004,"15min":0.55200000000000005},"min":{"1min":0.434,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.85999999999999999,"5min":6.7199999999999998,"15min":6.7199999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.56599999999999995},"min":{"1min":0.48099999999999998,"5min":0.36099999999999999,"15min":0.34399999999999997},"max":{"1min":0.86599999999999999,"5min":6.6639999999999997,"15min":6.6639999999999997},"last":0.51500000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.63400000000000001,"15min":0.55700000000000005},"min":{"1min":0.47699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82999999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.64600000000000002,"15min":0.56399999999999995},"min":{"1min":0.51300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88800000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.878}]},{"osd":3,"last update":"Sun Sep 10 01:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.64100000000000001,"15min":0.58299999999999996},"min":{"1min":0.47699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.63700000000000001,"15min":0.60099999999999998},"min":{"1min":0.505,"5min":0.379,"15min":0.379},"max":{"1min":1.6060000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.77800000000000002,"15min":0.78400000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93500000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.89200000000000002}]},{"osd":5,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.54300000000000004,"15min":0.46100000000000002},"min":{"1min":0.40899999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.60199999999999998,"15min":0.52800000000000002},"min":{"1min":0.41999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0029999999999999,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.86299999999999999}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":0.79300000000000004,"15min":0.70099999999999996},"min":{"1min":0.70199999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":6.6820000000000004,"5min":6.6820000000000004,"15min":6.6820000000000004},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.72799999999999998,"15min":0.68600000000000005},"min":{"1min":0.65500000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Sep 10 01:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86099999999999999,"15min":0.85999999999999999},"min":{"1min":0.65600000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.105,"5min":1.173,"15min":1.173},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.046,"5min":1.157,"15min":1.157},"last":0.97699999999999998}]},{"osd":8,"last update":"Sun Sep 10 01:01:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.56399999999999995,"15min":0.54500000000000004},"min":{"1min":0.51000000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.73299999999999998,"15min":0.71499999999999997},"min":{"1min":0.59499999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.96499999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:02:20 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.84699999999999998,"15min":0.80400000000000005},"min":{"1min":0.79300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.113,"5min":0.80800000000000005,"15min":0.747},"min":{"1min":0.71699999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":4.577,"5min":4.577,"15min":4.577},"last":0.84599999999999997}]},{"osd":10,"last update":"Sun Sep 10 01:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.71699999999999997,"15min":0.67600000000000005},"min":{"1min":0.59199999999999997,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.117,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.82399999999999995,"15min":0.78400000000000003},"min":{"1min":0.77500000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0649999999999999,"5min":1.099,"15min":1.099},"last":1.004}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.82699999999999996,"15min":0.82199999999999995},"min":{"1min":0.625,"5min":0.621,"15min":0.621},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93899999999999995,"15min":0.93700000000000006},"min":{"1min":0.81599999999999995,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.097,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.0189999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":499267,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.47299999999999998,"15min":0.40699999999999997},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.54200000000000004,"15min":0.45100000000000001},"min":{"1min":0.33000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.5309999999999997,"5min":4.5309999999999997,"15min":4.5309999999999997},"last":0.40500000000000003}]},{"osd":1,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.69999999999999996,"15min":0.61899999999999999},"min":{"1min":0.49399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.63900000000000001,"15min":0.57499999999999996},"min":{"1min":0.48099999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.74399999999999999,"15min":0.64300000000000002},"min":{"1min":0.45700000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.819,"5min":4.819,"15min":4.819},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.72499999999999998,"15min":0.64700000000000002},"min":{"1min":0.53000000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Sep 10 01:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.77200000000000002,"15min":0.69699999999999995},"min":{"1min":0.57899999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.9139999999999999,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.82299999999999995,"15min":0.73599999999999999},"min":{"1min":0.66000000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.9450000000000001,"5min":2.52,"15min":2.52},"last":0.72199999999999998}]},{"osd":5,"last update":"Sun Sep 10 01:01:53 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69999999999999996,"15min":0.69199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.79500000000000004,"15min":0.73299999999999998},"min":{"1min":0.51600000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":4.9550000000000001},"last":0.75800000000000001}]},{"osd":6,"last update":"Sun Sep 10 01:02:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.71199999999999997,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.8609999999999998,"5min":4.8609999999999998,"15min":4.8609999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.745,"15min":0.67300000000000004},"min":{"1min":0.46100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.89900000000000002}]},{"osd":7,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.80100000000000005,"15min":0.79900000000000004},"min":{"1min":0.51300000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.437,"5min":0.434,"15min":0.434},"max":{"1min":0.99199999999999999,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":0.59999999999999998}]},{"osd":8,"last update":"Sun Sep 10 00:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Sun Sep 10 01:02:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":0.96199999999999997,"15min":0.97999999999999998},"min":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":5.1740000000000004,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.012,"5min":0.93799999999999994,"15min":0.93600000000000005},"min":{"1min":0.53300000000000003,"5min":0.433,"15min":0.433},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.93600000000000005}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.97999999999999998,"15min":0.97199999999999998},"min":{"1min":0.71099999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":3.6680000000000001,"5min":3.6680000000000001,"15min":3.6680000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.84799999999999998,"15min":0.82899999999999996},"min":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.6280000000000001,"5min":3.6280000000000001,"15min":3.6280000000000001},"last":0.66700000000000004}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.095,"15min":1.1259999999999999},"min":{"1min":0.73599999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.25,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0900000000000001,"15min":1.1020000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.6930000000000001,"5min":4.9139999999999997,"15min":4.9139999999999997},"last":0.879}]}]},{"osd":4,"up_from":32,"seq":137438953543,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":505192,"data_compressed":23657,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:00:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.435},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.52400000000000002},"min":{"1min":0.38100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.36799999999999999},"min":{"1min":0.32900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41799999999999998,"15min":0.38400000000000001},"min":{"1min":0.34799999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.442}]},{"osd":2,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59399999999999997,"15min":0.56699999999999995},"min":{"1min":0.42699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0820000000000001,"5min":1.607,"15min":1.607},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.59099999999999997},"min":{"1min":0.50600000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.079,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Sun Sep 10 01:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.61199999999999999,"15min":0.57699999999999996},"min":{"1min":0.46899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.86099999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.624},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.35999999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.495}]},{"osd":5,"last update":"Sun Sep 10 01:02:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.61199999999999999},"min":{"1min":0.47799999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1020000000000001,"5min":1.621,"15min":1.621},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66000000000000003,"15min":0.59999999999999998},"min":{"1min":0.56399999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81799999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.66900000000000004}]},{"osd":6,"last update":"Sun Sep 10 01:02:19 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.83399999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.44700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68500000000000005}]},{"osd":7,"last update":"Sun Sep 10 01:02:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59399999999999997,"15min":0.54800000000000004},"min":{"1min":0.41599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.62,"15min":0.59099999999999997},"min":{"1min":0.42899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55500000000000005}]},{"osd":8,"last update":"Sun Sep 10 01:02:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.67900000000000005,"15min":0.65100000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1200000000000001,"5min":1.802,"15min":1.802},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.76700000000000002,"15min":0.73699999999999999},"min":{"1min":0.66300000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91100000000000003,"5min":0.997,"15min":0.997},"last":0.84399999999999997}]},{"osd":9,"last update":"Sun Sep 10 01:02:22 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.58499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.73399999999999999},"min":{"1min":0.54600000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.751}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.77800000000000002,"15min":0.77700000000000002},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95199999999999996,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.77400000000000002,"15min":0.76000000000000001},"min":{"1min":0.48699999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.76800000000000002}]},{"osd":11,"last update":"Sun Sep 10 01:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.80400000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790018,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1095454,"data_compressed":59629,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56699999999999995,"15min":0.51500000000000001},"min":{"1min":0.38200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.55800000000000005,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.441}]},{"osd":1,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.63700000000000001,"15min":0.56799999999999995},"min":{"1min":0.47299999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.622,"15min":0.56499999999999995},"min":{"1min":0.46700000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.44400000000000001,"15min":0.39400000000000002},"min":{"1min":0.35099999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.623,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.46000000000000002,"15min":0.41799999999999998},"min":{"1min":0.316,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.38700000000000001}]},{"osd":3,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.58199999999999996,"15min":0.55600000000000005},"min":{"1min":0.443,"5min":0.376,"15min":0.376},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.64000000000000001,"15min":0.56999999999999995},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.52500000000000002}]},{"osd":4,"last update":"Sun Sep 10 01:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.65100000000000002,"15min":0.60099999999999998},"min":{"1min":0.47399999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92100000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.65700000000000003,"15min":0.60999999999999999},"min":{"1min":0.48499999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1,"5min":1.02,"15min":1.02},"last":0.60699999999999998}]},{"osd":6,"last update":"Sun Sep 10 01:02:17 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.67000000000000004,"15min":0.63800000000000001},"min":{"1min":0.58199999999999996,"5min":0.437,"15min":0.437},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.69599999999999995,"15min":0.66400000000000003},"min":{"1min":0.61799999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.89200000000000002,"5min":0.998,"15min":0.998},"last":0.623}]},{"osd":7,"last update":"Sun Sep 10 01:02:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.70299999999999996,"15min":0.67800000000000005},"min":{"1min":0.59599999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93000000000000005,"5min":1.042,"15min":1.042},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.73799999999999999,"15min":0.70699999999999996},"min":{"1min":0.52600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.77600000000000002}]},{"osd":8,"last update":"Sun Sep 10 01:02:54 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.66600000000000004,"15min":0.65000000000000002},"min":{"1min":0.438,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91000000000000003,"5min":1.036,"15min":1.036},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.64700000000000002,"15min":0.61899999999999999},"min":{"1min":0.45000000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.64300000000000002}]},{"osd":9,"last update":"Sun Sep 10 01:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.72999999999999998,"15min":0.69999999999999996},"min":{"1min":0.63300000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.80100000000000005,"15min":0.77100000000000002},"min":{"1min":0.747,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.746}]},{"osd":10,"last update":"Sun Sep 10 01:02:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.77100000000000002,"15min":0.73699999999999999},"min":{"1min":0.59899999999999998,"5min":0.505,"15min":0.505},"max":{"1min":0.98999999999999999,"5min":1.107,"15min":1.107},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.81499999999999995,"15min":0.77700000000000002},"min":{"1min":0.64300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0860000000000001,"5min":1.171,"15min":1.171},"last":0.82899999999999996}]},{"osd":11,"last update":"Sun Sep 10 01:01:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.80400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34328,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35972,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-10T01:02:59.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:02:59.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:59 smithi003 ceph-mon[97654]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2023-09-10T01:02:59.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:02:59 smithi003 ceph-mon[97654]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:59.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 -- 172.21.15.3:0/2298463154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2381006d0 msgr2=0x7fd238100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/2298463154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2381006d0 0x7fd238100af0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd2280099c0 tx=0x7fd228009cd0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 -- 172.21.15.3:0/2298463154 shutdown_connections 2023-09-10T01:02:59.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/2298463154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd238102260 0x7fd238108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/2298463154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2381006d0 0x7fd238100af0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/2298463154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2381018c0 0x7fd238101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 -- 172.21.15.3:0/2298463154 >> 172.21.15.3:0/2298463154 conn(0x7fd2380fbc60 msgr2=0x7fd2380fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:59.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 -- 172.21.15.3:0/2298463154 shutdown_connections 2023-09-10T01:02:59.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.348+0000 7fd23d5ff700 1 -- 172.21.15.3:0/2298463154 wait complete. 2023-09-10T01:02:59.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 Processor -- start 2023-09-10T01:02:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 -- start start 2023-09-10T01:02:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2381006d0 0x7fd2380743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2381018c0 0x7fd2380729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238102260 0x7fd238072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380736d0 con 0x7fd238102260 2023-09-10T01:02:59.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd238073840 con 0x7fd2381018c0 2023-09-10T01:02:59.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.349+0000 7fd23d5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2381a03e0 con 0x7fd2381006d0 2023-09-10T01:02:59.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd22ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2381018c0 0x7fd2380729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd236ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2381006d0 0x7fd2380743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd2377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238102260 0x7fd238072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd22ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2381018c0 0x7fd2380729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50258/0 (socket says 172.21.15.3:50258) 2023-09-10T01:02:59.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd22ffff700 1 -- 172.21.15.3:0/1497875089 learned_addr learned my addr 172.21.15.3:0/1497875089 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:59.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd2377fe700 1 -- 172.21.15.3:0/1497875089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2381006d0 msgr2=0x7fd2380743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd2377fe700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2381006d0 0x7fd2380743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd2377fe700 1 -- 172.21.15.3:0/1497875089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2381018c0 msgr2=0x7fd2380729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd2377fe700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2381018c0 0x7fd2380729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd2377fe700 1 -- 172.21.15.3:0/1497875089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2280095f0 con 0x7fd238102260 2023-09-10T01:02:59.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.350+0000 7fd2377fe700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238102260 0x7fd238072f30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd23000efc0 tx=0x7fd23000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:59.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.351+0000 7fd234ff9700 1 -- 172.21.15.3:0/1497875089 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230018070 con 0x7fd238102260 2023-09-10T01:02:59.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.351+0000 7fd234ff9700 1 -- 172.21.15.3:0/1497875089 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd23000f040 con 0x7fd238102260 2023-09-10T01:02:59.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.351+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2381a0660 con 0x7fd238102260 2023-09-10T01:02:59.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.351+0000 7fd234ff9700 1 -- 172.21.15.3:0/1497875089 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23000cd40 con 0x7fd238102260 2023-09-10T01:02:59.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.351+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2381a0bc0 con 0x7fd238102260 2023-09-10T01:02:59.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.352+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238107650 con 0x7fd238102260 2023-09-10T01:02:59.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.354+0000 7fd234ff9700 1 -- 172.21.15.3:0/1497875089 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fd230003a00 con 0x7fd238102260 2023-09-10T01:02:59.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.355+0000 7fd234ff9700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2240a06c0 0x7fd2240a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.355+0000 7fd234ff9700 1 -- 172.21.15.3:0/1497875089 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fd2300cc360 con 0x7fd238102260 2023-09-10T01:02:59.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.355+0000 7fd236ffd700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2240a06c0 0x7fd2240a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.356+0000 7fd236ffd700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2240a06c0 0x7fd2240a2b80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd228009990 tx=0x7fd228005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:59.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.357+0000 7fd234ff9700 1 -- 172.21.15.3:0/1497875089 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd230095c10 con 0x7fd238102260 2023-09-10T01:02:59.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:59 smithi136 ceph-mon[133708]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2023-09-10T01:02:59.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:02:59 smithi136 ceph-mon[133708]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:59.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:59 smithi092 ceph-mon[134067]: pgmap v351: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2023-09-10T01:02:59.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:02:59 smithi092 ceph-mon[134067]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:02:59.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.570+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd23810bee0 con 0x7fd238102260 2023-09-10T01:02:59.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.571+0000 7fd234ff9700 1 -- 172.21.15.3:0/1497875089 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7fd230099230 con 0x7fd238102260 2023-09-10T01:02:59.572 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:02:59.573 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:02:59.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2240a06c0 msgr2=0x7fd2240a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2240a06c0 0x7fd2240a2b80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd228009990 tx=0x7fd228005fb0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238102260 msgr2=0x7fd238072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238102260 0x7fd238072f30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd23000efc0 tx=0x7fd23000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 shutdown_connections 2023-09-10T01:02:59.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fd2240a06c0 0x7fd2240a2b80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2381006d0 0x7fd2380743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238102260 0x7fd238072f30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 --2- 172.21.15.3:0/1497875089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2381018c0 0x7fd2380729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 >> 172.21.15.3:0/1497875089 conn(0x7fd2380fbc60 msgr2=0x7fd2380fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:59.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 shutdown_connections 2023-09-10T01:02:59.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.574+0000 7fd23d5ff700 1 -- 172.21.15.3:0/1497875089 wait complete. 2023-09-10T01:02:59.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:02:59.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:02:59.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1309298029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc102270 msgr2=0x7fb1dc108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1309298029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc102270 0x7fb1dc108b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400b0d0 tx=0x7fb1d400b3e0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1309298029 shutdown_connections 2023-09-10T01:02:59.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1309298029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc102270 0x7fb1dc108b20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1309298029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1dc1006e0 0x7fb1dc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1309298029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc1018d0 0x7fb1dc101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1309298029 >> 172.21.15.3:0/1309298029 conn(0x7fb1dc0fbc50 msgr2=0x7fb1dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:02:59.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1309298029 shutdown_connections 2023-09-10T01:02:59.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.803+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1309298029 wait complete. 2023-09-10T01:02:59.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 Processor -- start 2023-09-10T01:02:59.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 -- start start 2023-09-10T01:02:59.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc1006e0 0x7fb1dc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc1018d0 0x7fb1dc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1dc102270 0x7fb1dc19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1a0290 con 0x7fb1dc1006e0 2023-09-10T01:02:59.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1a03d0 con 0x7fb1dc102270 2023-09-10T01:02:59.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.804+0000 7fb1e21ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1a0560 con 0x7fb1dc1018d0 2023-09-10T01:02:59.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc1006e0 0x7fb1dc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc1006e0 0x7fb1dc198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49308/0 (socket says 172.21.15.3:49308) 2023-09-10T01:02:59.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 -- 172.21.15.3:0/1064751905 learned_addr learned my addr 172.21.15.3:0/1064751905 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:02:59.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1dbfff700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1dc102270 0x7fb1dc19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1daffd700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc1018d0 0x7fb1dc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 -- 172.21.15.3:0/1064751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc1018d0 msgr2=0x7fb1dc198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc1018d0 0x7fb1dc198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 -- 172.21.15.3:0/1064751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1dc102270 msgr2=0x7fb1dc19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:02:59.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1dc102270 0x7fb1dc19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:02:59.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 -- 172.21.15.3:0/1064751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4009d00 con 0x7fb1dc1006e0 2023-09-10T01:02:59.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1daffd700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc1018d0 0x7fb1dc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:02:59.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.805+0000 7fb1db7fe700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc1006e0 0x7fb1dc198710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00e970 tx=0x7fb1cc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:59.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.806+0000 7fb1d8ff9700 1 -- 172.21.15.3:0/1064751905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc009790 con 0x7fb1dc1006e0 2023-09-10T01:02:59.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.806+0000 7fb1d8ff9700 1 -- 172.21.15.3:0/1064751905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1cc00eec0 con 0x7fb1dc1006e0 2023-09-10T01:02:59.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.806+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc1a0840 con 0x7fb1dc1006e0 2023-09-10T01:02:59.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.806+0000 7fb1d8ff9700 1 -- 172.21.15.3:0/1064751905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc003cf0 con 0x7fb1dc1006e0 2023-09-10T01:02:59.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.806+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc1a0da0 con 0x7fb1dc1006e0 2023-09-10T01:02:59.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.807+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc10bf40 con 0x7fb1dc1006e0 2023-09-10T01:02:59.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.809+0000 7fb1d8ff9700 1 -- 172.21.15.3:0/1064751905 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fb1cc00ccd0 con 0x7fb1dc1006e0 2023-09-10T01:02:59.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.810+0000 7fb1d8ff9700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1c40a06c0 0x7fb1c40a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:02:59.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.810+0000 7fb1d8ff9700 1 -- 172.21.15.3:0/1064751905 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fb1cc014070 con 0x7fb1dc1006e0 2023-09-10T01:02:59.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.810+0000 7fb1daffd700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1c40a06c0 0x7fb1c40a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:02:59.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.811+0000 7fb1daffd700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1c40a06c0 0x7fb1c40a2b80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009b50 tx=0x7fb1d0009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:02:59.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:02:59.814+0000 7fb1d8ff9700 1 -- 172.21.15.3:0/1064751905 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fb1cc0958b0 con 0x7fb1dc1006e0 2023-09-10T01:03:00.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.023+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1dc1a0fc0 con 0x7fb1dc1006e0 2023-09-10T01:03:00.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.024+0000 7fb1d8ff9700 1 -- 172.21.15.3:0/1064751905 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1cc098ed0 con 0x7fb1dc1006e0 2023-09-10T01:03:00.026 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:03:00.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1c40a06c0 msgr2=0x7fb1c40a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1c40a06c0 0x7fb1c40a2b80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009b50 tx=0x7fb1d0009550 comp rx=0 tx=0).stop 2023-09-10T01:03:00.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc1006e0 msgr2=0x7fb1dc198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc1006e0 0x7fb1dc198710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00e970 tx=0x7fb1cc00ec80 comp rx=0 tx=0).stop 2023-09-10T01:03:00.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 shutdown_connections 2023-09-10T01:03:00.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb1dc102270 0x7fb1dc19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1dc1006e0 0x7fb1dc198710 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fb1c40a06c0 0x7fb1c40a2b80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 --2- 172.21.15.3:0/1064751905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc1018d0 0x7fb1dc198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.026+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 >> 172.21.15.3:0/1064751905 conn(0x7fb1dc0fbc50 msgr2=0x7fb1dc107140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:00.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.027+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 shutdown_connections 2023-09-10T01:03:00.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.027+0000 7fb1e21ff700 1 -- 172.21.15.3:0/1064751905 wait complete. 2023-09-10T01:03:00.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:03:00.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.262+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3884982580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8641006e0 msgr2=0x7fe864100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.262+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3884982580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8641006e0 0x7fe864100b00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe8540099c0 tx=0x7fe854009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.263+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3884982580 shutdown_connections 2023-09-10T01:03:00.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.263+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3884982580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe864102270 0x7fe864108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.263+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3884982580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8641006e0 0x7fe864100b00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.263+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3884982580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641018d0 0x7fe864101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.263+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3884982580 >> 172.21.15.3:0/3884982580 conn(0x7fe8640fbc50 msgr2=0x7fe8640fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:00.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.263+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3884982580 shutdown_connections 2023-09-10T01:03:00.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.263+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3884982580 wait complete. 2023-09-10T01:03:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 Processor -- start 2023-09-10T01:03:00.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 -- start start 2023-09-10T01:03:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641006e0 0x7fe8640743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8641018d0 0x7fe8640729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe864102270 0x7fe864072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8640736d0 con 0x7fe864102270 2023-09-10T01:03:00.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864073840 con 0x7fe8641018d0 2023-09-10T01:03:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.264+0000 7fe86a9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8641a03f0 con 0x7fe8641006e0 2023-09-10T01:03:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641006e0 0x7fe8640743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe8637fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8641018d0 0x7fe8640729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641006e0 0x7fe8640743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51168/0 (socket says 172.21.15.3:51168) 2023-09-10T01:03:00.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 -- 172.21.15.3:0/3852057815 learned_addr learned my addr 172.21.15.3:0/3852057815 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:00.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe868f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe864102270 0x7fe864072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 -- 172.21.15.3:0/3852057815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8641018d0 msgr2=0x7fe8640729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8641018d0 0x7fe8640729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 -- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe864102270 msgr2=0x7fe864072f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe864102270 0x7fe864072f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe863fff700 1 -- 172.21.15.3:0/3852057815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8540095f0 con 0x7fe8641006e0 2023-09-10T01:03:00.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe868f9c700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe864102270 0x7fe864072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:00.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.265+0000 7fe8637fe700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8641018d0 0x7fe8640729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:00.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.266+0000 7fe863fff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641006e0 0x7fe8640743b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe8540058e0 tx=0x7fe85400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:00.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.266+0000 7fe8617fa700 1 -- 172.21.15.3:0/3852057815 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85401d070 con 0x7fe8641006e0 2023-09-10T01:03:00.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.266+0000 7fe8617fa700 1 -- 172.21.15.3:0/3852057815 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe854005050 con 0x7fe8641006e0 2023-09-10T01:03:00.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.266+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8641a0670 con 0x7fe8641006e0 2023-09-10T01:03:00.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.266+0000 7fe8617fa700 1 -- 172.21.15.3:0/3852057815 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854004b30 con 0x7fe8641006e0 2023-09-10T01:03:00.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.266+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8641a0a90 con 0x7fe8641006e0 2023-09-10T01:03:00.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.267+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe864107660 con 0x7fe8641006e0 2023-09-10T01:03:00.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.269+0000 7fe8617fa700 1 -- 172.21.15.3:0/3852057815 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fe8540051c0 con 0x7fe8641006e0 2023-09-10T01:03:00.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.271+0000 7fe8617fa700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe84c0a0640 0x7fe84c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.271+0000 7fe8617fa700 1 -- 172.21.15.3:0/3852057815 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fe8540cd190 con 0x7fe8641006e0 2023-09-10T01:03:00.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.271+0000 7fe8637fe700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe84c0a0640 0x7fe84c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.271+0000 7fe8637fe700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe84c0a0640 0x7fe84c0a2b00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe858011490 tx=0x7fe858009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:00.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.273+0000 7fe8617fa700 1 -- 172.21.15.3:0/3852057815 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fe854096940 con 0x7fe8641006e0 2023-09-10T01:03:00.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:00 smithi003 ceph-mon[97654]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:03:00.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:00 smithi003 ceph-mon[97654]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:03:00.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:00 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1497875089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:03:00.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:00 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1064751905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:03:00.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:00 smithi136 ceph-mon[133708]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:03:00.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:00 smithi136 ceph-mon[133708]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:03:00.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:00 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1497875089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:03:00.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:00 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1064751905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:03:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:00 smithi092 ceph-mon[134067]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:03:00.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:00 smithi092 ceph-mon[134067]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-09-10T01:03:00.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:00 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1497875089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:03:00.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:00 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1064751905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:03:00.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.483+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe86402cdf0 con 0x7fe8641006e0 2023-09-10T01:03:00.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.484+0000 7fe8617fa700 1 -- 172.21.15.3:0/3852057815 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe8540a0070 con 0x7fe8641006e0 2023-09-10T01:03:00.486 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:03:00.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe84c0a0640 msgr2=0x7fe84c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe84c0a0640 0x7fe84c0a2b00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe858011490 tx=0x7fe858009260 comp rx=0 tx=0).stop 2023-09-10T01:03:00.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641006e0 msgr2=0x7fe8640743b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641006e0 0x7fe8640743b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe8540058e0 tx=0x7fe85400f690 comp rx=0 tx=0).stop 2023-09-10T01:03:00.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 shutdown_connections 2023-09-10T01:03:00.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fe84c0a0640 0x7fe84c0a2b00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8641006e0 0x7fe8640743b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe864102270 0x7fe864072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.486+0000 7fe86a9ff700 1 --2- 172.21.15.3:0/3852057815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8641018d0 0x7fe8640729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.487+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 >> 172.21.15.3:0/3852057815 conn(0x7fe8640fbc50 msgr2=0x7fe8640fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:00.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.487+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 shutdown_connections 2023-09-10T01:03:00.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.487+0000 7fe86a9ff700 1 -- 172.21.15.3:0/3852057815 wait complete. 2023-09-10T01:03:00.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:03:00.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/1875407873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e781018e0 msgr2=0x7f7e78101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/1875407873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e781018e0 0x7f7e78101d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0099c0 tx=0x7f7e6c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/1875407873 shutdown_connections 2023-09-10T01:03:00.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/1875407873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e78102280 0x7f7e78108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/1875407873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781006f0 0x7f7e78100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/1875407873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e781018e0 0x7f7e78101d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/1875407873 >> 172.21.15.3:0/1875407873 conn(0x7f7e780fbc60 msgr2=0x7f7e780fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:00.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/1875407873 shutdown_connections 2023-09-10T01:03:00.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.714+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/1875407873 wait complete. 2023-09-10T01:03:00.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.715+0000 7f7e7e5ff700 1 Processor -- start 2023-09-10T01:03:00.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.715+0000 7f7e7e5ff700 1 -- start start 2023-09-10T01:03:00.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7e5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e781006f0 0x7f7e781985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7e5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781018e0 0x7f7e78198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7e5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e78102280 0x7f7e7819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7e5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e781a0130 con 0x7f7e781018e0 2023-09-10T01:03:00.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7e5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e781a0270 con 0x7f7e78102280 2023-09-10T01:03:00.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7e5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e781a0400 con 0x7f7e781006f0 2023-09-10T01:03:00.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781018e0 0x7f7e78198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e777fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781018e0 0x7f7e78198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49354/0 (socket says 172.21.15.3:49354) 2023-09-10T01:03:00.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e777fe700 1 -- 172.21.15.3:0/3942843761 learned_addr learned my addr 172.21.15.3:0/3942843761 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:00.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7cb9c700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e78102280 0x7f7e7819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e77fff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e781006f0 0x7f7e781985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7cb9c700 1 -- 172.21.15.3:0/3942843761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e781006f0 msgr2=0x7f7e781985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7cb9c700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e781006f0 0x7f7e781985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7cb9c700 1 -- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781018e0 msgr2=0x7f7e78198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7cb9c700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781018e0 0x7f7e78198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e7cb9c700 1 -- 172.21.15.3:0/3942843761 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0095f0 con 0x7f7e78102280 2023-09-10T01:03:00.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.716+0000 7f7e777fe700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781018e0 0x7f7e78198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:00.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.717+0000 7f7e7cb9c700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e78102280 0x7f7e7819fb60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000efc0 tx=0x7f7e7000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:00.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.717+0000 7f7e757fa700 1 -- 172.21.15.3:0/3942843761 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70018070 con 0x7f7e78102280 2023-09-10T01:03:00.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.717+0000 7f7e757fa700 1 -- 172.21.15.3:0/3942843761 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e7000f040 con 0x7f7e78102280 2023-09-10T01:03:00.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.717+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e781a0680 con 0x7f7e78102280 2023-09-10T01:03:00.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.717+0000 7f7e757fa700 1 -- 172.21.15.3:0/3942843761 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000cd40 con 0x7f7e78102280 2023-09-10T01:03:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.717+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e781a0b90 con 0x7f7e78102280 2023-09-10T01:03:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.718+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7810bf60 con 0x7f7e78102280 2023-09-10T01:03:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.720+0000 7f7e757fa700 1 -- 172.21.15.3:0/3942843761 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f7e70003a00 con 0x7f7e78102280 2023-09-10T01:03:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.721+0000 7f7e757fa700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e600a0640 0x7f7e600a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:00.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.722+0000 7f7e757fa700 1 -- 172.21.15.3:0/3942843761 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f7e700cc4e0 con 0x7f7e78102280 2023-09-10T01:03:00.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.722+0000 7f7e77fff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e600a0640 0x7f7e600a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:00.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.722+0000 7f7e77fff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e600a0640 0x7f7e600a2b00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e78101740 tx=0x7f7e68009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:00.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.724+0000 7f7e757fa700 1 -- 172.21.15.3:0/3942843761 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7e70095c90 con 0x7f7e78102280 2023-09-10T01:03:00.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.937+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e781a0db0 con 0x7f7e78102280 2023-09-10T01:03:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.938+0000 7f7e757fa700 1 -- 172.21.15.3:0/3942843761 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e700992b0 con 0x7f7e78102280 2023-09-10T01:03:00.939 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:03:00.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e600a0640 msgr2=0x7f7e600a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e600a0640 0x7f7e600a2b00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e78101740 tx=0x7f7e68009260 comp rx=0 tx=0).stop 2023-09-10T01:03:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e78102280 msgr2=0x7f7e7819fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e78102280 0x7f7e7819fb60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000efc0 tx=0x7f7e7000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 shutdown_connections 2023-09-10T01:03:00.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f7e600a0640 0x7f7e600a2b00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e781006f0 0x7f7e781985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e78102280 0x7f7e7819fb60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 --2- 172.21.15.3:0/3942843761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e781018e0 0x7f7e78198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.940+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 >> 172.21.15.3:0/3942843761 conn(0x7f7e780fbc60 msgr2=0x7f7e780fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:00.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.941+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 shutdown_connections 2023-09-10T01:03:00.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:00.941+0000 7f7e7e5ff700 1 -- 172.21.15.3:0/3942843761 wait complete. 2023-09-10T01:03:00.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:03:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 -- 172.21.15.3:0/2741566715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 msgr2=0x7f602c108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 --2- 172.21.15.3:0/2741566715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 0x7f602c108b30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f602400b0d0 tx=0x7f602400b3e0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 -- 172.21.15.3:0/2741566715 shutdown_connections 2023-09-10T01:03:01.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 --2- 172.21.15.3:0/2741566715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 0x7f602c108b30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 --2- 172.21.15.3:0/2741566715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 0x7f602c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 --2- 172.21.15.3:0/2741566715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602c1018e0 0x7f602c101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 -- 172.21.15.3:0/2741566715 >> 172.21.15.3:0/2741566715 conn(0x7f602c0fbc60 msgr2=0x7f602c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:01.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 -- 172.21.15.3:0/2741566715 shutdown_connections 2023-09-10T01:03:01.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.169+0000 7f60309ff700 1 -- 172.21.15.3:0/2741566715 wait complete. 2023-09-10T01:03:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.170+0000 7f60309ff700 1 Processor -- start 2023-09-10T01:03:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f60309ff700 1 -- start start 2023-09-10T01:03:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f60309ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 0x7f602c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f60309ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602c1018e0 0x7f602c198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f60309ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 0x7f602c19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f60309ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c1a01f0 con 0x7f602c102280 2023-09-10T01:03:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f60309ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c1a0330 con 0x7f602c1006f0 2023-09-10T01:03:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f60309ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c1a04c0 con 0x7f602c1018e0 2023-09-10T01:03:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f602a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 0x7f602c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f602a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 0x7f602c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50324/0 (socket says 172.21.15.3:50324) 2023-09-10T01:03:01.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f602a59c700 1 -- 172.21.15.3:0/3736488120 learned_addr learned my addr 172.21.15.3:0/3736488120 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:01.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f602ad9d700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 0x7f602c19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.171+0000 7f6029d9b700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602c1018e0 0x7f602c198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f602a59c700 1 -- 172.21.15.3:0/3736488120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602c1018e0 msgr2=0x7f602c198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f602a59c700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602c1018e0 0x7f602c198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f602a59c700 1 -- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 msgr2=0x7f602c19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f602a59c700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 0x7f602c19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f602a59c700 1 -- 172.21.15.3:0/3736488120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c009520 con 0x7f602c1006f0 2023-09-10T01:03:01.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f602a59c700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 0x7f602c198670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f601c00ea50 tx=0x7f601c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:01.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f601b7fe700 1 -- 172.21.15.3:0/3736488120 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00cbd0 con 0x7f602c1006f0 2023-09-10T01:03:01.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6024009d00 con 0x7f602c1006f0 2023-09-10T01:03:01.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f601b7fe700 1 -- 172.21.15.3:0/3736488120 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f601c00cd30 con 0x7f602c1006f0 2023-09-10T01:03:01.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f601b7fe700 1 -- 172.21.15.3:0/3736488120 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c018680 con 0x7f602c1006f0 2023-09-10T01:03:01.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.172+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c1a0b20 con 0x7f602c1006f0 2023-09-10T01:03:01.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.173+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c10bf10 con 0x7f602c1006f0 2023-09-10T01:03:01.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.175+0000 7f601b7fe700 1 -- 172.21.15.3:0/3736488120 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f601c010430 con 0x7f602c1006f0 2023-09-10T01:03:01.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.177+0000 7f601b7fe700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60140a0640 0x7f60140a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.177+0000 7f601b7fe700 1 -- 172.21.15.3:0/3736488120 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f601c014070 con 0x7f602c1006f0 2023-09-10T01:03:01.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.177+0000 7f6029d9b700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60140a0640 0x7f60140a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.178+0000 7f6029d9b700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60140a0640 0x7f60140a2b00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6020009c60 tx=0x7f6020009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:01.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.179+0000 7f601b7fe700 1 -- 172.21.15.3:0/3736488120 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f601c0961d0 con 0x7f602c1006f0 2023-09-10T01:03:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:01 smithi003 ceph-mon[97654]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:01.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3852057815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:03:01.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:01 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3942843761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:03:01.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.389+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f602c1a0d40 con 0x7f602c1006f0 2023-09-10T01:03:01.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.390+0000 7f601b7fe700 1 -- 172.21.15.3:0/3736488120 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f601c0997f0 con 0x7f602c1006f0 2023-09-10T01:03:01.393 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:03:01.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60140a0640 msgr2=0x7f60140a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60140a0640 0x7f60140a2b00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6020009c60 tx=0x7f6020009260 comp rx=0 tx=0).stop 2023-09-10T01:03:01.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 msgr2=0x7f602c198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 0x7f602c198670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f601c00ea50 tx=0x7f601c00ed60 comp rx=0 tx=0).stop 2023-09-10T01:03:01.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 shutdown_connections 2023-09-10T01:03:01.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f602c102280 0x7f602c19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f602c1006f0 0x7f602c198670 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f60140a0640 0x7f60140a2b00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 --2- 172.21.15.3:0/3736488120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f602c1018e0 0x7f602c198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 >> 172.21.15.3:0/3736488120 conn(0x7f602c0fbc60 msgr2=0x7f602c107170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:01.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 shutdown_connections 2023-09-10T01:03:01.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.393+0000 7f60309ff700 1 -- 172.21.15.3:0/3736488120 wait complete. 2023-09-10T01:03:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:01 smithi136 ceph-mon[133708]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:01 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3852057815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:03:01.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:01 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3942843761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:03:01.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:03:01.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:01 smithi092 ceph-mon[134067]: pgmap v352: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:01.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:01 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3852057815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:03:01.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:01 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3942843761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:03:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 -- 172.21.15.3:0/1922411802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15341006d0 msgr2=0x7f1534100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1922411802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15341006d0 0x7f1534100af0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f15280099c0 tx=0x7f1528009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 -- 172.21.15.3:0/1922411802 shutdown_connections 2023-09-10T01:03:01.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1922411802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1534102260 0x7f1534108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1922411802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15341006d0 0x7f1534100af0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1922411802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15341018c0 0x7f1534101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 -- 172.21.15.3:0/1922411802 >> 172.21.15.3:0/1922411802 conn(0x7f15340fbc60 msgr2=0x7f15340fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:01.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 -- 172.21.15.3:0/1922411802 shutdown_connections 2023-09-10T01:03:01.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.617+0000 7f153b1ff700 1 -- 172.21.15.3:0/1922411802 wait complete. 2023-09-10T01:03:01.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.618+0000 7f153b1ff700 1 Processor -- start 2023-09-10T01:03:01.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.618+0000 7f153b1ff700 1 -- start start 2023-09-10T01:03:01.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153b1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15341006d0 0x7f15340743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153b1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15341018c0 0x7f15340729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153b1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1534102260 0x7f1534072f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153b1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15340736d0 con 0x7f1534102260 2023-09-10T01:03:01.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153b1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1534073840 con 0x7f15341018c0 2023-09-10T01:03:01.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153b1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15341a0430 con 0x7f15341006d0 2023-09-10T01:03:01.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153979c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1534102260 0x7f1534072f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f1538f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15341006d0 0x7f15340743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153979c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1534102260 0x7f1534072f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49390/0 (socket says 172.21.15.3:49390) 2023-09-10T01:03:01.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f153979c700 1 -- 172.21.15.3:0/1043013910 learned_addr learned my addr 172.21.15.3:0/1043013910 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:01.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.619+0000 7f1533fff700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15341018c0 0x7f15340729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153979c700 1 -- 172.21.15.3:0/1043013910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15341006d0 msgr2=0x7f15340743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153979c700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15341006d0 0x7f15340743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153979c700 1 -- 172.21.15.3:0/1043013910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15341018c0 msgr2=0x7f15340729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153979c700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15341018c0 0x7f15340729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153979c700 1 -- 172.21.15.3:0/1043013910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15280095f0 con 0x7f1534102260 2023-09-10T01:03:01.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153979c700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1534102260 0x7f1534072f30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f152c00ef30 tx=0x7f152c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:01.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f1531ffb700 1 -- 172.21.15.3:0/1043013910 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c018070 con 0x7f1534102260 2023-09-10T01:03:01.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f1531ffb700 1 -- 172.21.15.3:0/1043013910 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f152c00f040 con 0x7f1534102260 2023-09-10T01:03:01.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15341a06b0 con 0x7f1534102260 2023-09-10T01:03:01.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f1531ffb700 1 -- 172.21.15.3:0/1043013910 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c00cd00 con 0x7f1534102260 2023-09-10T01:03:01.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.620+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15341a0c10 con 0x7f1534102260 2023-09-10T01:03:01.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.621+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153410bf30 con 0x7f1534102260 2023-09-10T01:03:01.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.623+0000 7f1531ffb700 1 -- 172.21.15.3:0/1043013910 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f152c003a00 con 0x7f1534102260 2023-09-10T01:03:01.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.625+0000 7f1531ffb700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15240a06c0 0x7f15240a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:01.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.625+0000 7f1531ffb700 1 -- 172.21.15.3:0/1043013910 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f152c0cba50 con 0x7f1534102260 2023-09-10T01:03:01.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.625+0000 7f1538f9b700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15240a06c0 0x7f15240a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:01.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.625+0000 7f1538f9b700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15240a06c0 0x7f15240a2b80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1528009990 tx=0x7f1528005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:01.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.629+0000 7f1531ffb700 1 -- 172.21.15.3:0/1043013910 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f152c095200 con 0x7f1534102260 2023-09-10T01:03:01.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.886+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f153410c120 con 0x7f1534102260 2023-09-10T01:03:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.887+0000 7f1531ffb700 1 -- 172.21.15.3:0/1043013910 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f152c098820 con 0x7f1534102260 2023-09-10T01:03:01.889 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:03:01.889 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:03:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15240a06c0 msgr2=0x7f15240a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15240a06c0 0x7f15240a2b80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1528009990 tx=0x7f1528005fb0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1534102260 msgr2=0x7f1534072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:01.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1534102260 0x7f1534072f30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f152c00ef30 tx=0x7f152c00c3a0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 shutdown_connections 2023-09-10T01:03:01.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f15240a06c0 0x7f15240a2b80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15341006d0 0x7f15340743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1534102260 0x7f1534072f30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 --2- 172.21.15.3:0/1043013910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15341018c0 0x7f15340729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:01.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 >> 172.21.15.3:0/1043013910 conn(0x7f15340fbc60 msgr2=0x7f15340fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:01.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.889+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 shutdown_connections 2023-09-10T01:03:01.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:01.890+0000 7f153b1ff700 1 -- 172.21.15.3:0/1043013910 wait complete. 2023-09-10T01:03:01.896 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:03:01.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:03:02.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.115+0000 7efd68dff700 1 -- 172.21.15.3:0/526227249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 msgr2=0x7efd64100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.115+0000 7efd68dff700 1 --2- 172.21.15.3:0/526227249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd64100af0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efd540099c0 tx=0x7efd54009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.116+0000 7efd68dff700 1 -- 172.21.15.3:0/526227249 shutdown_connections 2023-09-10T01:03:02.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.116+0000 7efd68dff700 1 --2- 172.21.15.3:0/526227249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd64102260 0x7efd64108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.116+0000 7efd68dff700 1 --2- 172.21.15.3:0/526227249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd64100af0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.116+0000 7efd68dff700 1 --2- 172.21.15.3:0/526227249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd641018c0 0x7efd64101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.116+0000 7efd68dff700 1 -- 172.21.15.3:0/526227249 >> 172.21.15.3:0/526227249 conn(0x7efd640fbc60 msgr2=0x7efd640fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.116+0000 7efd68dff700 1 -- 172.21.15.3:0/526227249 shutdown_connections 2023-09-10T01:03:02.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.116+0000 7efd68dff700 1 -- 172.21.15.3:0/526227249 wait complete. 2023-09-10T01:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.117+0000 7efd68dff700 1 Processor -- start 2023-09-10T01:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.117+0000 7efd68dff700 1 -- start start 2023-09-10T01:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd68dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd641963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd68dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd641018c0 0x7efd641968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd68dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd64102260 0x7efd6419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd68dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6419df20 con 0x7efd64102260 2023-09-10T01:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd68dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6419e060 con 0x7efd641006d0 2023-09-10T01:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd68dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6419e1f0 con 0x7efd641018c0 2023-09-10T01:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd6259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd641963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd5bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd641018c0 0x7efd641968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd6259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd641963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50356/0 (socket says 172.21.15.3:50356) 2023-09-10T01:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd6259c700 1 -- 172.21.15.3:0/3511567880 learned_addr learned my addr 172.21.15.3:0/3511567880 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.118+0000 7efd62d9d700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd64102260 0x7efd6419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd6259c700 1 -- 172.21.15.3:0/3511567880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd641018c0 msgr2=0x7efd641968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd6259c700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd641018c0 0x7efd641968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd6259c700 1 -- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd64102260 msgr2=0x7efd6419d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd6259c700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd64102260 0x7efd6419d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd6259c700 1 -- 172.21.15.3:0/3511567880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd540095f0 con 0x7efd641006d0 2023-09-10T01:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd6259c700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd641963a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd54009fd0 tx=0x7efd54009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd5b7fe700 1 -- 172.21.15.3:0/3511567880 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5401d070 con 0x7efd641006d0 2023-09-10T01:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd5b7fe700 1 -- 172.21.15.3:0/3511567880 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd54005030 con 0x7efd641006d0 2023-09-10T01:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd5b7fe700 1 -- 172.21.15.3:0/3511567880 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54004a90 con 0x7efd641006d0 2023-09-10T01:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.119+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6419e470 con 0x7efd641006d0 2023-09-10T01:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.120+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6419e840 con 0x7efd641006d0 2023-09-10T01:03:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.120+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd64107650 con 0x7efd641006d0 2023-09-10T01:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.123+0000 7efd5b7fe700 1 -- 172.21.15.3:0/3511567880 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7efd540051a0 con 0x7efd641006d0 2023-09-10T01:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.124+0000 7efd5b7fe700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efd500a0640 0x7efd500a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.124+0000 7efd5b7fe700 1 -- 172.21.15.3:0/3511567880 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7efd540cd200 con 0x7efd641006d0 2023-09-10T01:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.124+0000 7efd5bfff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efd500a0640 0x7efd500a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.125+0000 7efd5bfff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efd500a0640 0x7efd500a2b00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009c60 tx=0x7efd4c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.126+0000 7efd5b7fe700 1 -- 172.21.15.3:0/3511567880 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7efd54096ad0 con 0x7efd641006d0 2023-09-10T01:03:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:02 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3736488120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:03:02.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:02 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1043013910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:03:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.336+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd6410bee0 con 0x7efd641006d0 2023-09-10T01:03:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.339+0000 7efd5b7fe700 1 -- 172.21.15.3:0/3511567880 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24591 (secure 0 0 0) 0x7efd54026020 con 0x7efd641006d0 2023-09-10T01:03:02.341 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:03:02.341 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":90,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:01:57.028004+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:03:02.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efd500a0640 msgr2=0x7efd500a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efd500a0640 0x7efd500a2b00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009c60 tx=0x7efd4c009260 comp rx=0 tx=0).stop 2023-09-10T01:03:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 msgr2=0x7efd641963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd641963a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd54009fd0 tx=0x7efd54009580 comp rx=0 tx=0).stop 2023-09-10T01:03:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 shutdown_connections 2023-09-10T01:03:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd64102260 0x7efd6419d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd641006d0 0x7efd641963a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.340+0000 7efd68dff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7efd500a0640 0x7efd500a2b00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.341+0000 7efd68dff700 1 --2- 172.21.15.3:0/3511567880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd641018c0 0x7efd641968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.341+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 >> 172.21.15.3:0/3511567880 conn(0x7efd640fbc60 msgr2=0x7efd640fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.341+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 shutdown_connections 2023-09-10T01:03:02.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.341+0000 7efd68dff700 1 -- 172.21.15.3:0/3511567880 wait complete. 2023-09-10T01:03:02.361 DEBUG:tasks.fwd_scrub:Yielding 2023-09-10T01:03:02.362 INFO:teuthology.run_tasks:Running task exec... 2023-09-10T01:03:02.373 INFO:teuthology.task.exec:Executing custom commands... 2023-09-10T01:03:02.373 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:03:02.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-09-10T01:03:02.374 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2023-09-10T01:03:02.375 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:03:02.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:03:02.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:02 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3736488120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:03:02.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:02 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1043013910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:03:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:02 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3736488120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:03:02.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:02 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1043013910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:03:02.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 -- 172.21.15.3:0/3844135284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 msgr2=0x7fef80108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 --2- 172.21.15.3:0/3844135284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 0x7fef80108760 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fef7800b0a0 tx=0x7fef7800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 -- 172.21.15.3:0/3844135284 shutdown_connections 2023-09-10T01:03:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 --2- 172.21.15.3:0/3844135284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 0x7fef80108760 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 --2- 172.21.15.3:0/3844135284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 0x7fef80103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 --2- 172.21.15.3:0/3844135284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef801039c0 0x7fef80105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 -- 172.21.15.3:0/3844135284 >> 172.21.15.3:0/3844135284 conn(0x7fef800faac0 msgr2=0x7fef800fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 -- 172.21.15.3:0/3844135284 shutdown_connections 2023-09-10T01:03:02.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.577+0000 7fef85dff700 1 -- 172.21.15.3:0/3844135284 wait complete. 2023-09-10T01:03:02.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.578+0000 7fef85dff700 1 Processor -- start 2023-09-10T01:03:02.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.578+0000 7fef85dff700 1 -- start start 2023-09-10T01:03:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef85dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 0x7fef80196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef85dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef801039c0 0x7fef80196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef85dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 0x7fef8019d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef85dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8019df90 con 0x7fef801062f0 2023-09-10T01:03:02.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef85dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8019e0d0 con 0x7fef80101090 2023-09-10T01:03:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef85dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8019e260 con 0x7fef801039c0 2023-09-10T01:03:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef7f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 0x7fef80196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef7ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 0x7fef8019d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef7f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 0x7fef80196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50374/0 (socket says 172.21.15.3:50374) 2023-09-10T01:03:02.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef7f7fe700 1 -- 172.21.15.3:0/2003634857 learned_addr learned my addr 172.21.15.3:0/2003634857 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.579+0000 7fef7effd700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef801039c0 0x7fef80196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7f7fe700 1 -- 172.21.15.3:0/2003634857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef801039c0 msgr2=0x7fef80196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7f7fe700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef801039c0 0x7fef80196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7f7fe700 1 -- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 msgr2=0x7fef8019d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7f7fe700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 0x7fef8019d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7f7fe700 1 -- 172.21.15.3:0/2003634857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef70009520 con 0x7fef80101090 2023-09-10T01:03:02.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7f7fe700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 0x7fef80196410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fef7000eb30 tx=0x7fef7000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70009880 con 0x7fef80101090 2023-09-10T01:03:02.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef78009d00 con 0x7fef80101090 2023-09-10T01:03:02.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef700044e0 con 0x7fef80101090 2023-09-10T01:03:02.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70003d80 con 0x7fef80101090 2023-09-10T01:03:02.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.580+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8019e8c0 con 0x7fef80101090 2023-09-10T01:03:02.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.582+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef800fc640 con 0x7fef80101090 2023-09-10T01:03:02.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.583+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7fef70003f20 con 0x7fef80101090 2023-09-10T01:03:02.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.585+0000 7fef7cff9700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fef680a0640 0x7fef680a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.585+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7fef70014070 con 0x7fef80101090 2023-09-10T01:03:02.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.585+0000 7fef7effd700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fef680a0640 0x7fef680a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.586+0000 7fef7effd700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fef680a0640 0x7fef680a2b00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef74009c60 tx=0x7fef74009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.589+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fef70095c10 con 0x7fef80101090 2023-09-10T01:03:02.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.589+0000 7f4f55dff700 1 -- 172.21.15.3:0/3320972325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501018d0 msgr2=0x7f4f50101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.589+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3320972325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501018d0 0x7f4f50101d30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4f440099c0 tx=0x7f4f44009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.590+0000 7f4f55dff700 1 -- 172.21.15.3:0/3320972325 shutdown_connections 2023-09-10T01:03:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.590+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3320972325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f501006e0 0x7f4f50100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.590+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3320972325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f50102270 0x7f4f50108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.590+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3320972325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501018d0 0x7f4f50101d30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.590+0000 7f4f55dff700 1 -- 172.21.15.3:0/3320972325 >> 172.21.15.3:0/3320972325 conn(0x7f4f500fbc50 msgr2=0x7f4f500fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.590+0000 7f4f55dff700 1 -- 172.21.15.3:0/3320972325 shutdown_connections 2023-09-10T01:03:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.590+0000 7f4f55dff700 1 -- 172.21.15.3:0/3320972325 wait complete. 2023-09-10T01:03:02.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 Processor -- start 2023-09-10T01:03:02.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 -- start start 2023-09-10T01:03:02.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501006e0 0x7f4f50198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f501018d0 0x7f4f50198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f50102270 0x7f4f5019fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f501a00f0 con 0x7f4f501018d0 2023-09-10T01:03:02.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f501a0230 con 0x7f4f501006e0 2023-09-10T01:03:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.591+0000 7f4f55dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f501a03c0 con 0x7f4f50102270 2023-09-10T01:03:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501006e0 0x7f4f50198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f501018d0 0x7f4f50198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501006e0 0x7f4f50198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50390/0 (socket says 172.21.15.3:50390) 2023-09-10T01:03:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/3409282824 learned_addr learned my addr 172.21.15.3:0/3409282824 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4ffff700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f50102270 0x7f4f5019fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4effd700 1 -- 172.21.15.3:0/3409282824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f50102270 msgr2=0x7f4f5019fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4effd700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f50102270 0x7f4f5019fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4effd700 1 -- 172.21.15.3:0/3409282824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501006e0 msgr2=0x7f4f50198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4effd700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501006e0 0x7f4f50198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4effd700 1 -- 172.21.15.3:0/3409282824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f440095f0 con 0x7f4f501018d0 2023-09-10T01:03:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501006e0 0x7f4f50198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.592+0000 7f4f4effd700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f501018d0 0x7f4f50198ab0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009fd0 tx=0x7f4f44009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.593+0000 7f4f4cff9700 1 -- 172.21.15.3:0/3409282824 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4401d070 con 0x7f4f501018d0 2023-09-10T01:03:02.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.593+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f501a0640 con 0x7f4f501018d0 2023-09-10T01:03:02.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.593+0000 7f4f4cff9700 1 -- 172.21.15.3:0/3409282824 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f44005020 con 0x7f4f501018d0 2023-09-10T01:03:02.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.593+0000 7f4f4cff9700 1 -- 172.21.15.3:0/3409282824 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44004a90 con 0x7f4f501018d0 2023-09-10T01:03:02.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.593+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f501a0a60 con 0x7f4f501018d0 2023-09-10T01:03:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.596+0000 7f4f4cff9700 1 -- 172.21.15.3:0/3409282824 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f4f44005190 con 0x7f4f501018d0 2023-09-10T01:03:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.598+0000 7f4f4cff9700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f380a0640 0x7f4f380a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.598+0000 7f4f4cff9700 1 -- 172.21.15.3:0/3409282824 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f4f440cc9a0 con 0x7f4f501018d0 2023-09-10T01:03:02.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.598+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f380a0640 0x7f4f380a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.599+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f380a0640 0x7f4f380a2b00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f40005950 tx=0x7f4f40009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 Processor -- start 2023-09-10T01:03:02.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 -- start start 2023-09-10T01:03:02.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e8400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e83e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e7977b0 con 0x55e24e840620 2023-09-10T01:03:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e7950d0 con 0x55e24e83ee80 2023-09-10T01:03:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.600+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e7909c0 con 0x55e24e83fc80 2023-09-10T01:03:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e83e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e83e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49454/0 (socket says 172.21.15.3:49454) 2023-09-10T01:03:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e8400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 -- 172.21.15.3:0/1646867596 learned_addr learned my addr 172.21.15.3:0/1646867596 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4ffff700 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 -- 172.21.15.3:0/1646867596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 msgr2=0x55e24e8400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e8400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 -- 172.21.15.3:0/1646867596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 msgr2=0x55e24e83f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 -- 172.21.15.3:0/1646867596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24e5bce60 con 0x55e24e840620 2023-09-10T01:03:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.601+0000 7f4f4effd700 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e83e880 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4f44096310 tx=0x7f4f44009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc7015b5d3aa248 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f35ffb700 1 -- 172.21.15.3:0/1646867596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4409dc40 con 0x55e24e840620 2023-09-10T01:03:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f35ffb700 1 -- 172.21.15.3:0/1646867596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f440a1df0 con 0x55e24e840620 2023-09-10T01:03:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f35ffb700 1 -- 172.21.15.3:0/1646867596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44099300 con 0x55e24e840620 2023-09-10T01:03:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1646867596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 msgr2=0x55e24e83e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e83e880 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4f44096310 tx=0x7f4f44009310 comp rx=0 tx=0).stop 2023-09-10T01:03:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1646867596 shutdown_connections 2023-09-10T01:03:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e8400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1646867596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e83e880 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1646867596 >> 172.21.15.3:0/1646867596 conn(0x55e24e7b3110 msgr2=0x55e24e8397a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.602+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1646867596 shutdown_connections 2023-09-10T01:03:02.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.603+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1646867596 wait complete. 2023-09-10T01:03:02.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.603+0000 7f4f697e3b80 1 -- start start 2023-09-10T01:03:02.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.603+0000 7f4f697e3b80 1 Processor -- start 2023-09-10T01:03:02.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83ee80 0x55e24e848300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83fc80 0x55e24e848840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e84f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e72ebf0 con 0x55e24e83ee80 2023-09-10T01:03:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e72e8f0 con 0x55e24e83fc80 2023-09-10T01:03:02.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e6129a0 con 0x55e24e840620 2023-09-10T01:03:02.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f697e3b80 10 client.0 _ll_register_callbacks cb 0x55e24e678b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:02.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83ee80 0x55e24e848300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83fc80 0x55e24e848840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83fc80 0x55e24e848840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50400/0 (socket says 172.21.15.3:50400) 2023-09-10T01:03:02.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 -- 172.21.15.3:0/4100975026 learned_addr learned my addr 172.21.15.3:0/4100975026 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4ffff700 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e84f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 -- 172.21.15.3:0/4100975026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 msgr2=0x55e24e84f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e84f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 -- 172.21.15.3:0/4100975026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83ee80 msgr2=0x55e24e848300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83ee80 0x55e24e848300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.604+0000 7f4f4effd700 1 -- 172.21.15.3:0/4100975026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f501a0a60 con 0x55e24e83fc80 2023-09-10T01:03:02.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f4effd700 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83fc80 0x55e24e848840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4f440b7fd0 tx=0x7f4f44009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8f9d813d1ce834e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f2effd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:02.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f2effd700 1 -- 172.21.15.3:0/4100975026 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44099300 con 0x55e24e83fc80 2023-09-10T01:03:02.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e24e6e8570 con 0x55e24e83fc80 2023-09-10T01:03:02.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f2effd700 1 -- 172.21.15.3:0/4100975026 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f4f44099460 con 0x55e24e83fc80 2023-09-10T01:03:02.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f2effd700 1 -- 172.21.15.3:0/4100975026 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f440a1960 con 0x55e24e83fc80 2023-09-10T01:03:02.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f697e3b80 10 client.25628 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:02.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.605+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e24e5a5ce0 con 0x55e24e83fc80 2023-09-10T01:03:02.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f2effd700 1 -- 172.21.15.3:0/4100975026 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f44065020 con 0x55e24e83fc80 2023-09-10T01:03:02.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f2effd700 1 -- 172.21.15.3:0/4100975026 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4f44098d70 con 0x55e24e83fc80 2023-09-10T01:03:02.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f697e3b80 10 client.25628 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:02.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f697e3b80 10 client.25628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:02.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f697e3b80 20 client.25628 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:02.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e853bb0 0x55e24e855fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f697e3b80 4 client.25628 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:02.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e24e7a5920 con 0x55e24e853bb0 2023-09-10T01:03:02.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.606+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e853bb0 0x55e24e855fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.607+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e853bb0 0x55e24e855fd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a55171ac76cf279 server_cookie=c099145f01afd682 in_seq=0 out_seq=0 2023-09-10T01:03:02.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.607+0000 7f4f2effd700 10 client.25628 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f2effd700 1 -- 172.21.15.3:0/4100975026 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e24e7a5920 con 0x55e24e853bb0 2023-09-10T01:03:02.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f2effd700 10 client.25628 handle_command_reply: tid=0 2023-09-10T01:03:02.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 client.25628 shutdown 2023-09-10T01:03:02.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83fc80 msgr2=0x55e24e848840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83fc80 0x55e24e848840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4f440b7fd0 tx=0x7f4f44009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 shutdown_connections 2023-09-10T01:03:02.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e853bb0 0x55e24e855fd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e84f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83ee80 0x55e24e848300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/4100975026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83fc80 0x55e24e848840 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f2effd700 0 client.25628 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 >> 172.21.15.3:0/4100975026 conn(0x55e24e7b3110 msgr2=0x55e24e838e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 shutdown_connections 2023-09-10T01:03:02.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 1 -- 172.21.15.3:0/4100975026 wait complete. 2023-09-10T01:03:02.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.608+0000 7f4f697e3b80 20 client.25628 trim_cache size 0 max 16384 2023-09-10T01:03:02.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 Processor -- start 2023-09-10T01:03:02.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 -- start start 2023-09-10T01:03:02.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83fc80 0x55e24e84f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e83dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e83b090 con 0x55e24e83fc80 2023-09-10T01:03:02.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e849230 con 0x55e24e83ee80 2023-09-10T01:03:02.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.626+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e847e70 con 0x55e24e840620 2023-09-10T01:03:02.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83fc80 0x55e24e84f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50416/0 (socket says 172.21.15.3:50416) 2023-09-10T01:03:02.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e83dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/1599509411 learned_addr learned my addr 172.21.15.3:0/1599509411 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/1599509411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 msgr2=0x55e24e83dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/1599509411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e83dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/1599509411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83fc80 msgr2=0x55e24e84f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/1599509411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83fc80 0x55e24e84f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/1599509411 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24e847340 con 0x55e24e83ee80 2023-09-10T01:03:02.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/1599509411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55e24e8568c0 tx=0x7f4f40014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d61f0f6bc9e22558 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.627+0000 7f4f367fc700 1 -- 172.21.15.3:0/1599509411 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40022030 con 0x55e24e83ee80 2023-09-10T01:03:02.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f367fc700 1 -- 172.21.15.3:0/1599509411 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f40023040 con 0x55e24e83ee80 2023-09-10T01:03:02.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f367fc700 1 -- 172.21.15.3:0/1599509411 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f400043a0 con 0x55e24e83ee80 2023-09-10T01:03:02.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1599509411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 msgr2=0x55e24e83d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1599509411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55e24e8568c0 tx=0x7f4f40014000 comp rx=0 tx=0).stop 2023-09-10T01:03:02.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1599509411 shutdown_connections 2023-09-10T01:03:02.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1599509411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e840620 0x55e24e83dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1599509411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e83d780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/1599509411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e83fc80 0x55e24e84f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1599509411 >> 172.21.15.3:0/1599509411 conn(0x55e24e7b3110 msgr2=0x55e24e8463d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1599509411 shutdown_connections 2023-09-10T01:03:02.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.628+0000 7f4f697e3b80 1 -- 172.21.15.3:0/1599509411 wait complete. 2023-09-10T01:03:02.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.629+0000 7f4f697e3b80 1 -- start start 2023-09-10T01:03:02.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.629+0000 7f4f697e3b80 1 Processor -- start 2023-09-10T01:03:02.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.629+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e84ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.629+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e84d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.629+0000 7f4f697e3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e84e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e84e8a0 con 0x55e24e840620 2023-09-10T01:03:02.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e84ea10 con 0x55e24e83ee80 2023-09-10T01:03:02.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f697e3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e84ebd0 con 0x55e24e83fc80 2023-09-10T01:03:02.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f697e3b80 10 client.0 _ll_register_callbacks cb 0x7f4f48009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:02.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e84ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e84ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50430/0 (socket says 172.21.15.3:50430) 2023-09-10T01:03:02.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/2880703684 learned_addr learned my addr 172.21.15.3:0/2880703684 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4ffff700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e84e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4effd700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e84d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/2880703684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 msgr2=0x55e24e84d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e84d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/2880703684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 msgr2=0x55e24e84e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e84e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4f7fe700 1 -- 172.21.15.3:0/2880703684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f40017070 con 0x55e24e83ee80 2023-09-10T01:03:02.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.630+0000 7f4f4ffff700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e84e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:02.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f4f7fe700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e84ce10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f4002e040 tx=0x7f4f4000e590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2f28d143850b81d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f2effd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:02.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f2effd700 1 -- 172.21.15.3:0/2880703684 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40022040 con 0x55e24e83ee80 2023-09-10T01:03:02.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f440a1df0 con 0x55e24e83ee80 2023-09-10T01:03:02.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f2effd700 1 -- 172.21.15.3:0/2880703684 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f4f40004320 con 0x55e24e83ee80 2023-09-10T01:03:02.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f2effd700 1 -- 172.21.15.3:0/2880703684 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f400044e0 con 0x55e24e83ee80 2023-09-10T01:03:02.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f697e3b80 10 client.25634 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:02.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e24e5e32d0 con 0x55e24e83ee80 2023-09-10T01:03:02.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.631+0000 7f4f2effd700 1 -- 172.21.15.3:0/2880703684 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f440a1df0 con 0x55e24e83ee80 2023-09-10T01:03:02.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f2effd700 1 -- 172.21.15.3:0/2880703684 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4f40004750 con 0x55e24e83ee80 2023-09-10T01:03:02.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f697e3b80 10 client.25634 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:02.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f697e3b80 10 client.25634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:02.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f697e3b80 20 client.25634 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:02.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e8aa1c0 0x55e24e8ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f697e3b80 4 client.25634 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:03:02.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e24e7899a0 con 0x55e24e8aa1c0 2023-09-10T01:03:02.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f4effd700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e8aa1c0 0x55e24e8ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f4effd700 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e8aa1c0 0x55e24e8ac5e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6265a09a7290e7b0 server_cookie=8078a6fb605504e0 in_seq=0 out_seq=0 2023-09-10T01:03:02.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.632+0000 7f4f2effd700 10 client.25634 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f2effd700 1 -- 172.21.15.3:0/2880703684 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e24e7899a0 con 0x55e24e8aa1c0 2023-09-10T01:03:02.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f2effd700 10 client.25634 handle_command_reply: tid=0 2023-09-10T01:03:02.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 client.25634 shutdown 2023-09-10T01:03:02.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 msgr2=0x55e24e84ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e84ce10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f4002e040 tx=0x7f4f4000e590 comp rx=0 tx=0).stop 2023-09-10T01:03:02.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 shutdown_connections 2023-09-10T01:03:02.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e24e8aa1c0 0x55e24e8ac5e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e24e840620 0x55e24e84e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e24e83ee80 0x55e24e84ce10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f2effd700 0 client.25634 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 --2- 172.21.15.3:0/2880703684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e24e83fc80 0x55e24e84d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 >> 172.21.15.3:0/2880703684 conn(0x55e24e7b3110 msgr2=0x55e24e838e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 shutdown_connections 2023-09-10T01:03:02.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 1 -- 172.21.15.3:0/2880703684 wait complete. 2023-09-10T01:03:02.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.634+0000 7f4f697e3b80 20 client.25634 trim_cache size 0 max 16384 2023-09-10T01:03:02.649 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:03:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:03:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "c08fdf9e-a5d9-4a57-9374-4f40bcc0fc67", 2023-09-10T01:03:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:03:02.650 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:03:02.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f380a0640 msgr2=0x7f4f380a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f380a0640 0x7f4f380a2b00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4f40005950 tx=0x7f4f40009260 comp rx=0 tx=0).stop 2023-09-10T01:03:02.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f501018d0 msgr2=0x7f4f50198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f501018d0 0x7f4f50198ab0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009fd0 tx=0x7f4f44009580 comp rx=0 tx=0).stop 2023-09-10T01:03:02.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 shutdown_connections 2023-09-10T01:03:02.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f4f380a0640 0x7f4f380a2b00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f50102270 0x7f4f5019fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f501006e0 0x7f4f50198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 --2- 172.21.15.3:0/3409282824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f501018d0 0x7f4f50198ab0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 >> 172.21.15.3:0/3409282824 conn(0x7f4f500fbc50 msgr2=0x7f4f501071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 shutdown_connections 2023-09-10T01:03:02.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.635+0000 7f4f55dff700 1 -- 172.21.15.3:0/3409282824 wait complete. 2023-09-10T01:03:02.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:03:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.880+0000 7f27963ff700 1 -- 172.21.15.3:0/1376631787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27901006f0 msgr2=0x7f2790100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.880+0000 7f27963ff700 1 --2- 172.21.15.3:0/1376631787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27901006f0 0x7f2790100b10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2780009990 tx=0x7f2780009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.880+0000 7f27963ff700 1 -- 172.21.15.3:0/1376631787 shutdown_connections 2023-09-10T01:03:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.880+0000 7f27963ff700 1 --2- 172.21.15.3:0/1376631787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27901006f0 0x7f2790100b10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.880+0000 7f27963ff700 1 --2- 172.21.15.3:0/1376631787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2790102280 0x7f2790108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.880+0000 7f27963ff700 1 --2- 172.21.15.3:0/1376631787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901018e0 0x7f2790101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.880+0000 7f27963ff700 1 -- 172.21.15.3:0/1376631787 >> 172.21.15.3:0/1376631787 conn(0x7f27900fbc60 msgr2=0x7f27900fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.881+0000 7f27963ff700 1 -- 172.21.15.3:0/1376631787 shutdown_connections 2023-09-10T01:03:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.881+0000 7f27963ff700 1 -- 172.21.15.3:0/1376631787 wait complete. 2023-09-10T01:03:02.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 Processor -- start 2023-09-10T01:03:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 -- start start 2023-09-10T01:03:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901006f0 0x7f2790196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27901018e0 0x7f2790196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2790102280 0x7f279019d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279019dfc0 con 0x7f27901006f0 2023-09-10T01:03:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279019e100 con 0x7f27901018e0 2023-09-10T01:03:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.882+0000 7f27963ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279019e290 con 0x7f2790102280 2023-09-10T01:03:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901006f0 0x7f2790196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27901018e0 0x7f2790196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901006f0 0x7f2790196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49508/0 (socket says 172.21.15.3:49508) 2023-09-10T01:03:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 -- 172.21.15.3:0/1344280787 learned_addr learned my addr 172.21.15.3:0/1344280787 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f279499c700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2790102280 0x7f279019d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 -- 172.21.15.3:0/1344280787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2790102280 msgr2=0x7f279019d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2790102280 0x7f279019d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 -- 172.21.15.3:0/1344280787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27901018e0 msgr2=0x7f2790196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27901018e0 0x7f2790196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 -- 172.21.15.3:0/1344280787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800095f0 con 0x7f27901006f0 2023-09-10T01:03:02.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f279499c700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2790102280 0x7f279019d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:03:02.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.883+0000 7f278ffff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901006f0 0x7f2790196440 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2780004710 tx=0x7f2780009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.884+0000 7f278d7fa700 1 -- 172.21.15.3:0/1344280787 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278001d070 con 0x7f27901006f0 2023-09-10T01:03:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.884+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2790072830 con 0x7f27901006f0 2023-09-10T01:03:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.884+0000 7f278d7fa700 1 -- 172.21.15.3:0/1344280787 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2780021410 con 0x7f27901006f0 2023-09-10T01:03:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.884+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2790072d30 con 0x7f27901006f0 2023-09-10T01:03:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.884+0000 7f278d7fa700 1 -- 172.21.15.3:0/1344280787 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780017650 con 0x7f27901006f0 2023-09-10T01:03:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.888+0000 7f278d7fa700 1 -- 172.21.15.3:0/1344280787 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 135258+0+0 (secure 0 0 0) 0x7f2780021580 con 0x7f27901006f0 2023-09-10T01:03:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.889+0000 7f27a9c4fb80 1 Processor -- start 2023-09-10T01:03:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.889+0000 7f278d7fa700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27780a06c0 0x7f27780a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.889+0000 7f27a9c4fb80 1 -- start start 2023-09-10T01:03:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.889+0000 7f278d7fa700 1 -- 172.21.15.3:0/1344280787 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f27800cd250 con 0x7f27901006f0 2023-09-10T01:03:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcecc00 0x559befced020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befced5a0 0x559befceb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcec2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befc44730 con 0x559befcebe00 2023-09-10T01:03:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befc42050 con 0x559befcecc00 2023-09-10T01:03:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befc3d940 con 0x559befced5a0 2023-09-10T01:03:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f279499c700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27780a06c0 0x7f27780a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f278ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcecc00 0x559befced020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f278f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befced5a0 0x559befceb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f278f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befced5a0 0x559befceb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51354/0 (socket says 172.21.15.3:51354) 2023-09-10T01:03:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f278ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcecc00 0x559befced020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50460/0 (socket says 172.21.15.3:50460) 2023-09-10T01:03:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f278f7fe700 1 -- 172.21.15.3:0/4124977225 learned_addr learned my addr 172.21.15.3:0/4124977225 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.890+0000 7f279499c700 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcec2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f278ffff700 1 -- 172.21.15.3:0/4124977225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befced5a0 msgr2=0x559befceb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f278ffff700 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befced5a0 0x559befceb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f278ffff700 1 -- 172.21.15.3:0/4124977225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 msgr2=0x559befcec2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f278ffff700 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcec2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f278ffff700 1 -- 172.21.15.3:0/4124977225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559befa69de0 con 0x559befcecc00 2023-09-10T01:03:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f278ffff700 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcecc00 0x559befced020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f27800cad10 tx=0x7f2780096b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5be6dd5eb404ff54 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f279499c700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27780a06c0 0x7f27780a2b80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2788009fd0 tx=0x7f2788009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f27767fc700 1 -- 172.21.15.3:0/4124977225 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278009a720 con 0x559befcecc00 2023-09-10T01:03:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f27767fc700 1 -- 172.21.15.3:0/4124977225 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f278009e550 con 0x559befcecc00 2023-09-10T01:03:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.891+0000 7f27767fc700 1 -- 172.21.15.3:0/4124977225 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278009ab80 con 0x559befcecc00 2023-09-10T01:03:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/4124977225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcecc00 msgr2=0x559befced020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcecc00 0x559befced020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f27800cad10 tx=0x7f2780096b70 comp rx=0 tx=0).stop 2023-09-10T01:03:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/4124977225 shutdown_connections 2023-09-10T01:03:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcec2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcecc00 0x559befced020 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/4124977225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befced5a0 0x559befceb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/4124977225 >> 172.21.15.3:0/4124977225 conn(0x559befc60090 msgr2=0x559befce6720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/4124977225 shutdown_connections 2023-09-10T01:03:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.892+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/4124977225 wait complete. 2023-09-10T01:03:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.893+0000 7f27a9c4fb80 1 -- start start 2023-09-10T01:03:02.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.893+0000 7f27a9c4fb80 1 Processor -- start 2023-09-10T01:03:02.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcebe00 0x559befcf5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befced5a0 0x559befcfc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befbdbb70 con 0x559befced5a0 2023-09-10T01:03:02.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befbdb870 con 0x559befcebe00 2023-09-10T01:03:02.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befabf920 con 0x559befcecc00 2023-09-10T01:03:02.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f27a9c4fb80 10 client.0 _ll_register_callbacks cb 0x559befb25a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:02.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f279499c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befced5a0 0x559befcfc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f279499c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befced5a0 0x559befcfc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49530/0 (socket says 172.21.15.3:49530) 2023-09-10T01:03:02.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51366/0 (socket says 172.21.15.3:51366) 2023-09-10T01:03:02.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f279499c700 1 -- 172.21.15.3:0/2358917357 learned_addr learned my addr 172.21.15.3:0/2358917357 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278ffff700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcebe00 0x559befcf5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278f7fe700 1 -- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcebe00 msgr2=0x559befcf5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278f7fe700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcebe00 0x559befcf5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278f7fe700 1 -- 172.21.15.3:0/2358917357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befced5a0 msgr2=0x559befcfc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278f7fe700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befced5a0 0x559befcfc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278f7fe700 1 -- 172.21.15.3:0/2358917357 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2790072d30 con 0x559befcecc00 2023-09-10T01:03:02.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f278ffff700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcebe00 0x559befcf5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:02.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.894+0000 7f279499c700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befced5a0 0x559befcfc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:02.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f278f7fe700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf57a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2784005c50 tx=0x7f2784009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=837ec99563cfd67e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f276f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:03:02.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f276f7fe700 1 -- 172.21.15.3:0/2358917357 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784015030 con 0x559befcecc00 2023-09-10T01:03:02.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559befb954f0 con 0x559befcecc00 2023-09-10T01:03:02.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f276f7fe700 1 -- 172.21.15.3:0/2358917357 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f2784004460 con 0x559befcecc00 2023-09-10T01:03:02.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f276f7fe700 1 -- 172.21.15.3:0/2358917357 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2784010070 con 0x559befcecc00 2023-09-10T01:03:02.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f27a9c4fb80 10 client.25591 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:02.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x559befa52c60 con 0x559befcecc00 2023-09-10T01:03:02.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.895+0000 7f276f7fe700 1 -- 172.21.15.3:0/2358917357 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559befb954f0 con 0x559befcecc00 2023-09-10T01:03:02.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f276f7fe700 1 -- 172.21.15.3:0/2358917357 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f278400f450 con 0x559befcecc00 2023-09-10T01:03:02.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f27a9c4fb80 10 client.25591 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:02.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f27a9c4fb80 10 client.25591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:02.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f27a9c4fb80 20 client.25591 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:02.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd01060 0x559befd03480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f27a9c4fb80 4 client.25591 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:02.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559befc528a0 con 0x559befd01060 2023-09-10T01:03:02.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.896+0000 7f278ffff700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd01060 0x559befd03480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.897+0000 7f278ffff700 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd01060 0x559befd03480 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fef07ed842714bb9 server_cookie=6778e2f1c514e0b0 in_seq=0 out_seq=0 2023-09-10T01:03:02.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.897+0000 7f276f7fe700 10 client.25591 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f276f7fe700 1 -- 172.21.15.3:0/2358917357 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559befc528a0 con 0x559befd01060 2023-09-10T01:03:02.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f276f7fe700 10 client.25591 handle_command_reply: tid=0 2023-09-10T01:03:02.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 client.25591 shutdown 2023-09-10T01:03:02.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 msgr2=0x559befcf57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf57a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2784005c50 tx=0x7f2784009bb0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 shutdown_connections 2023-09-10T01:03:02.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd01060 0x559befd03480 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befced5a0 0x559befcfc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befcebe00 0x559befcf5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f276f7fe700 0 client.25591 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/2358917357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf57a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 >> 172.21.15.3:0/2358917357 conn(0x559befc60090 msgr2=0x559befce5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 shutdown_connections 2023-09-10T01:03:02.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/2358917357 wait complete. 2023-09-10T01:03:02.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.898+0000 7f27a9c4fb80 20 client.25591 trim_cache size 0 max 16384 2023-09-10T01:03:02.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.901+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fef8002cf30 con 0x7fef80101090 2023-09-10T01:03:02.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.916+0000 7f27a9c4fb80 1 Processor -- start 2023-09-10T01:03:02.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f27a9c4fb80 1 -- start start 2023-09-10T01:03:02.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcebe00 0x559befcefe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcecc00 0x559befcf03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befcf4710 con 0x559befcecc00 2023-09-10T01:03:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befcf4dd0 con 0x559befced5a0 2023-09-10T01:03:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befcec860 con 0x559befcebe00 2023-09-10T01:03:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f278ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcebe00 0x559befcefe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f279499c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f278ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcebe00 0x559befcefe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51380/0 (socket says 172.21.15.3:51380) 2023-09-10T01:03:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f278ffff700 1 -- 172.21.15.3:0/120754245 learned_addr learned my addr 172.21.15.3:0/120754245 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.917+0000 7f278f7fe700 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcecc00 0x559befcf03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f278ffff700 1 -- 172.21.15.3:0/120754245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 msgr2=0x559befcfc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f278ffff700 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f278ffff700 1 -- 172.21.15.3:0/120754245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcecc00 msgr2=0x559befcf03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f278ffff700 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcecc00 0x559befcf03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f278ffff700 1 -- 172.21.15.3:0/120754245 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559befa90250 con 0x559befcebe00 2023-09-10T01:03:02.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f279499c700 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:02.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f278ffff700 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcebe00 0x559befcefe80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2780098d10 tx=0x7f278009a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0f26c2fb3735d84 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f2776ffd700 1 -- 172.21.15.3:0/120754245 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780098b30 con 0x559befcebe00 2023-09-10T01:03:02.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f2776ffd700 1 -- 172.21.15.3:0/120754245 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f278009e370 con 0x559befcebe00 2023-09-10T01:03:02.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.918+0000 7f2776ffd700 1 -- 172.21.15.3:0/120754245 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278009abe0 con 0x559befcebe00 2023-09-10T01:03:02.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/120754245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcebe00 msgr2=0x559befcefe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcebe00 0x559befcefe80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2780098d10 tx=0x7f278009a1a0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/120754245 shutdown_connections 2023-09-10T01:03:02.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcebe00 0x559befcefe80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/120754245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcecc00 0x559befcf03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/120754245 >> 172.21.15.3:0/120754245 conn(0x559befc60090 msgr2=0x559befcfd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/120754245 shutdown_connections 2023-09-10T01:03:02.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/120754245 wait complete. 2023-09-10T01:03:02.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.919+0000 7f27a9c4fb80 1 -- start start 2023-09-10T01:03:02.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 1 Processor -- start 2023-09-10T01:03:02.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcf8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befcfc1c0 con 0x559befcebe00 2023-09-10T01:03:02.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befcfc330 con 0x559befced5a0 2023-09-10T01:03:02.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559befcfc4f0 con 0x559befcecc00 2023-09-10T01:03:02.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f27a9c4fb80 10 client.0 _ll_register_callbacks cb 0x7f2784010e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:02.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f278f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f278f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51394/0 (socket says 172.21.15.3:51394) 2023-09-10T01:03:02.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f278f7fe700 1 -- 172.21.15.3:0/1430247837 learned_addr learned my addr 172.21.15.3:0/1430247837 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:02.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f278ffff700 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcf8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.920+0000 7f279499c700 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f278f7fe700 1 -- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 msgr2=0x559befcfbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f278f7fe700 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f278f7fe700 1 -- 172.21.15.3:0/1430247837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 msgr2=0x559befcf8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:02.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f278f7fe700 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcf8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f278f7fe700 1 -- 172.21.15.3:0/1430247837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2788013040 con 0x559befcecc00 2023-09-10T01:03:02.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f278f7fe700 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf8c60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f278400f830 tx=0x7f278400f680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6146451993adb1e5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:02.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f276f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:03:02.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f276f7fe700 1 -- 172.21.15.3:0/1430247837 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784030070 con 0x559befcecc00 2023-09-10T01:03:02.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559befc36920 con 0x559befcecc00 2023-09-10T01:03:02.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.921+0000 7f276f7fe700 1 -- 172.21.15.3:0/1430247837 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f27840054f0 con 0x559befcecc00 2023-09-10T01:03:02.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f276f7fe700 1 -- 172.21.15.3:0/1430247837 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2784015040 con 0x559befcecc00 2023-09-10T01:03:02.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 10 client.25597 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:02.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x559befa69de0 con 0x559befcecc00 2023-09-10T01:03:02.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f276f7fe700 1 -- 172.21.15.3:0/1430247837 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559befc36920 con 0x559befcecc00 2023-09-10T01:03:02.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f276f7fe700 1 -- 172.21.15.3:0/1430247837 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f278401b5b0 con 0x559befcecc00 2023-09-10T01:03:02.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 10 client.25597 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:02.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 10 client.25597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 20 client.25597 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd55fe0 0x559befd58400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 4 client.25597 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:03:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559befbf70b0 con 0x559befd55fe0 2023-09-10T01:03:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.922+0000 7f278ffff700 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd55fe0 0x559befd58400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f278ffff700 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd55fe0 0x559befd58400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7899a3ef7b8a342 server_cookie=788f759642a5c4ea in_seq=0 out_seq=0 2023-09-10T01:03:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f276f7fe700 10 client.25597 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f276f7fe700 1 -- 172.21.15.3:0/1430247837 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559befbf70b0 con 0x559befd55fe0 2023-09-10T01:03:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f276f7fe700 10 client.25597 handle_command_reply: tid=0 2023-09-10T01:03:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 client.25597 shutdown 2023-09-10T01:03:02.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 msgr2=0x559befcf8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf8c60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f278400f830 tx=0x7f278400f680 comp rx=0 tx=0).stop 2023-09-10T01:03:02.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 shutdown_connections 2023-09-10T01:03:02.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559befd55fe0 0x559befd58400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559befced5a0 0x559befcfbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f276f7fe700 0 client.25597 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:02.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559befcebe00 0x559befcf8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.923+0000 7f27a9c4fb80 1 --2- 172.21.15.3:0/1430247837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559befcecc00 0x559befcf8c60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 >> 172.21.15.3:0/1430247837 conn(0x559befc60090 msgr2=0x559befcefe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 shutdown_connections 2023-09-10T01:03:02.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27a9c4fb80 1 -- 172.21.15.3:0/1430247837 wait complete. 2023-09-10T01:03:02.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27a9c4fb80 20 client.25597 trim_cache size 0 max 16384 2023-09-10T01:03:02.940 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:03:02.940 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:03:02.941 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:03:02.941 INFO:teuthology.orchestra.run.smithi003.stdout: "c08fdf9e-a5d9-4a57-9374-4f40bcc0fc67": { 2023-09-10T01:03:02.941 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:03:02.941 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "c08fdf9e-a5d9-4a57-9374-4f40bcc0fc67", 2023-09-10T01:03:02.941 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:03:02.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:03:02.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:03:02.942 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:03:02.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27780a06c0 msgr2=0x7f27780a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27963ff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27780a06c0 0x7f27780a2b80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2788009fd0 tx=0x7f2788009260 comp rx=0 tx=0).stop 2023-09-10T01:03:02.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901006f0 msgr2=0x7f2790196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:02.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.924+0000 7f27963ff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901006f0 0x7f2790196440 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2780004710 tx=0x7f2780009580 comp rx=0 tx=0).stop 2023-09-10T01:03:02.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 shutdown_connections 2023-09-10T01:03:02.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f27780a06c0 0x7f27780a2b80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2790102280 0x7f279019d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27901006f0 0x7f2790196440 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 --2- 172.21.15.3:0/1344280787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27901018e0 0x7f2790196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:02.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 >> 172.21.15.3:0/1344280787 conn(0x7f27900fbc60 msgr2=0x7f2790104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:02.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 shutdown_connections 2023-09-10T01:03:02.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:02.925+0000 7f27963ff700 1 -- 172.21.15.3:0/1344280787 wait complete. 2023-09-10T01:03:02.946 INFO:tasks.cephfs.filesystem:scrub status for tag:c08fdf9e-a5d9-4a57-9374-4f40bcc0fc67 - {'path': '/', 'tag': 'c08fdf9e-a5d9-4a57-9374-4f40bcc0fc67', 'options': 'recursive,force'} 2023-09-10T01:03:03.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.071+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 19) v1 ==== 135275+0+0 (secure 0 0 0) 0x7fef700041d0 con 0x7fef80101090 2023-09-10T01:03:03.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.072+0000 7fef7cff9700 1 -- 172.21.15.3:0/2003634857 <== mon.2 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7fef7005b390 con 0x7fef80101090 2023-09-10T01:03:03.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fef680a0640 msgr2=0x7fef680a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fef680a0640 0x7fef680a2b00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef74009c60 tx=0x7fef74009260 comp rx=0 tx=0).stop 2023-09-10T01:03:03.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 msgr2=0x7fef80196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 0x7fef80196410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fef7000eb30 tx=0x7fef7000ee40 comp rx=0 tx=0).stop 2023-09-10T01:03:03.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 shutdown_connections 2023-09-10T01:03:03.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef801062f0 0x7fef8019d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef80101090 0x7fef80196410 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7fef680a0640 0x7fef680a2b00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 --2- 172.21.15.3:0/2003634857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef801039c0 0x7fef80196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 >> 172.21.15.3:0/2003634857 conn(0x7fef800faac0 msgr2=0x7fef80101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:03.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 shutdown_connections 2023-09-10T01:03:03.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.076+0000 7fef85dff700 1 -- 172.21.15.3:0/2003634857 wait complete. 2023-09-10T01:03:03.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-09-10T01:03:03.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: ignoring --setuser ceph since I am not root 2023-09-10T01:03:03.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: ignoring --setgroup ceph since I am not root 2023-09-10T01:03:03.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:03.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3511567880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:03:03.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:03:03.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:03.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:03:03.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:03.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:03.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2003634857' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-09-10T01:03:03.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-09-10T01:03:03.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 -- 172.21.15.3:0/2622036564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c100630 msgr2=0x7f580c100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 --2- 172.21.15.3:0/2622036564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c100630 0x7f580c100a50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f58080099c0 tx=0x7f5808009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 -- 172.21.15.3:0/2622036564 shutdown_connections 2023-09-10T01:03:03.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 --2- 172.21.15.3:0/2622036564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c1021c0 0x7f580c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 --2- 172.21.15.3:0/2622036564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c100630 0x7f580c100a50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 --2- 172.21.15.3:0/2622036564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c101820 0x7f580c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 -- 172.21.15.3:0/2622036564 >> 172.21.15.3:0/2622036564 conn(0x7f580c0fbbc0 msgr2=0x7f580c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:03.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 -- 172.21.15.3:0/2622036564 shutdown_connections 2023-09-10T01:03:03.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.310+0000 7f58141ff700 1 -- 172.21.15.3:0/2622036564 wait complete. 2023-09-10T01:03:03.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.311+0000 7f58141ff700 1 Processor -- start 2023-09-10T01:03:03.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f58141ff700 1 -- start start 2023-09-10T01:03:03.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f58141ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c100630 0x7f580c1963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f58141ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c101820 0x7f580c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f58141ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c1021c0 0x7f580c19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f58141ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c19df50 con 0x7f580c100630 2023-09-10T01:03:03.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f58141ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c19e090 con 0x7f580c1021c0 2023-09-10T01:03:03.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f58141ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c19e220 con 0x7f580c101820 2023-09-10T01:03:03.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f5811f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c100630 0x7f580c1963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:03.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f581179a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c101820 0x7f580c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:03.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f581179a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c101820 0x7f580c196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51420/0 (socket says 172.21.15.3:51420) 2023-09-10T01:03:03.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f581179a700 1 -- 172.21.15.3:0/2442193810 learned_addr learned my addr 172.21.15.3:0/2442193810 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:03.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f5811f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c100630 0x7f580c1963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49594/0 (socket says 172.21.15.3:49594) 2023-09-10T01:03:03.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.312+0000 7f581279c700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c1021c0 0x7f580c19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:03.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f581179a700 1 -- 172.21.15.3:0/2442193810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c1021c0 msgr2=0x7f580c19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f581179a700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c1021c0 0x7f580c19d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f581179a700 1 -- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c100630 msgr2=0x7f580c1963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f581179a700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c100630 0x7f580c1963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f581179a700 1 -- 172.21.15.3:0/2442193810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5804009c40 con 0x7f580c101820 2023-09-10T01:03:03.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f581179a700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c101820 0x7f580c196910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00d980 tx=0x7f57fc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:03.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f5802ffd700 1 -- 172.21.15.3:0/2442193810 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc004220 con 0x7f580c101820 2023-09-10T01:03:03.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f5802ffd700 1 -- 172.21.15.3:0/2442193810 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57fc007640 con 0x7f580c101820 2023-09-10T01:03:03.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58080095f0 con 0x7f580c101820 2023-09-10T01:03:03.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f5802ffd700 1 -- 172.21.15.3:0/2442193810 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc005580 con 0x7f580c101820 2023-09-10T01:03:03.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.313+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c19e8d0 con 0x7f580c101820 2023-09-10T01:03:03.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.315+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c107c10 con 0x7f580c101820 2023-09-10T01:03:03.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.316+0000 7f5802ffd700 1 -- 172.21.15.3:0/2442193810 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 19) v1 ==== 135275+0+0 (secure 0 0 0) 0x7f57fc004750 con 0x7f580c101820 2023-09-10T01:03:03.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.318+0000 7f5802ffd700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 0x7f57f80a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.318+0000 7f5802ffd700 1 -- 172.21.15.3:0/2442193810 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f57fc0cbd10 con 0x7f580c101820 2023-09-10T01:03:03.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.318+0000 7f5811f9b700 1 -- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 msgr2=0x7f57f80a2bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:03.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.318+0000 7f5811f9b700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 0x7f57f80a2bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-10T01:03:03.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.322+0000 7f5802ffd700 1 -- 172.21.15.3:0/2442193810 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f57fc0d1020 con 0x7f580c101820 2023-09-10T01:03:03.330 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:03.330 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3511567880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:03:03.330 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:03:03.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:03.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:03:03.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:03.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:03.331 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2003634857' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-09-10T01:03:03.332 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-09-10T01:03:03.332 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: ignoring --setuser ceph since I am not root 2023-09-10T01:03:03.332 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: ignoring --setgroup ceph since I am not root 2023-09-10T01:03:03.332 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:03.245+0000 7f2cc90c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T01:03:03.333 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:03.329+0000 7f2cc90c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T01:03:03.333 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: ignoring --setuser ceph since I am not root 2023-09-10T01:03:03.334 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: ignoring --setgroup ceph since I am not root 2023-09-10T01:03:03.334 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mgr[135811]: -- 172.21.15.136:0/1045163192 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55584ddf71e0 con 0x55584ddccc00 2023-09-10T01:03:03.334 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:03.244+0000 7fc2eda81200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T01:03:03.334 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:03.333+0000 7fc2eda81200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T01:03:03.335 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: pgmap v353: 129 pgs: 129 active+clean; 644 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:03.335 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3511567880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:03:03.335 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:03:03.335 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:03.335 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:03:03.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:03.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: from='mgr.14150 172.21.15.3:0/3144999834' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:03.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2003634857' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-09-10T01:03:03.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-09-10T01:03:03.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.519+0000 7f5811f9b700 1 -- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 msgr2=0x7f57f80a2bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:03.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.519+0000 7f5811f9b700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 0x7f57f80a2bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-10T01:03:03.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.551+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f580c10bea0 con 0x7f580c101820 2023-09-10T01:03:03.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.560+0000 7f5802ffd700 1 -- 172.21.15.3:0/2442193810 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v44)=0 v44) v1 ==== 138+0+0 (secure 0 0 0) 0x7f57fc016020 con 0x7f580c101820 2023-09-10T01:03:03.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 msgr2=0x7f57f80a2bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:03:03.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 0x7f57f80a2bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c101820 msgr2=0x7f580c196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c101820 0x7f580c196910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00d980 tx=0x7f57fc00dc90 comp rx=0 tx=0).stop 2023-09-10T01:03:03.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 shutdown_connections 2023-09-10T01:03:03.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f580c1021c0 0x7f580c19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f580c100630 0x7f580c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f57f80a0710 0x7f57f80a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 --2- 172.21.15.3:0/2442193810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f580c101820 0x7f580c196910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.562+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 >> 172.21.15.3:0/2442193810 conn(0x7f580c0fbbc0 msgr2=0x7f580c104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:03.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.563+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 shutdown_connections 2023-09-10T01:03:03.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.563+0000 7f58141ff700 1 -- 172.21.15.3:0/2442193810 wait complete. 2023-09-10T01:03:03.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-09-10T01:03:03.625 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:03.363+0000 7f16bae43200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-10T01:03:03.625 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:03.473+0000 7f16bae43200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-10T01:03:03.661 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:03.429+0000 7fc2eda81200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T01:03:03.662 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:03.529+0000 7fc2eda81200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T01:03:03.699 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:03.420+0000 7f2cc90c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T01:03:03.699 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:03.524+0000 7f2cc90c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T01:03:03.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 -- 172.21.15.3:0/2796678673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25941084f0 msgr2=0x7f259410a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 --2- 172.21.15.3:0/2796678673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25941084f0 0x7f259410a980 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f258c00b0a0 tx=0x7f258c00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 -- 172.21.15.3:0/2796678673 shutdown_connections 2023-09-10T01:03:03.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 --2- 172.21.15.3:0/2796678673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2594074640 0x7f2594072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 --2- 172.21.15.3:0/2796678673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25941084f0 0x7f259410a980 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 --2- 172.21.15.3:0/2796678673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2594073060 0x7f25940734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 -- 172.21.15.3:0/2796678673 >> 172.21.15.3:0/2796678673 conn(0x7f25940fbbc0 msgr2=0x7f25940fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:03.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 -- 172.21.15.3:0/2796678673 shutdown_connections 2023-09-10T01:03:03.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.805+0000 7f259bbff700 1 -- 172.21.15.3:0/2796678673 wait complete. 2023-09-10T01:03:03.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.806+0000 7f259bbff700 1 Processor -- start 2023-09-10T01:03:03.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.807+0000 7f259bbff700 1 -- start start 2023-09-10T01:03:03.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.807+0000 7f259bbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2594073060 0x7f2594198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.807+0000 7f259bbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2594074640 0x7f2594198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.807+0000 7f259bbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25941084f0 0x7f259419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.807+0000 7f259bbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25941a01b0 con 0x7f2594074640 2023-09-10T01:03:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.807+0000 7f259bbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25941a02f0 con 0x7f2594073060 2023-09-10T01:03:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.807+0000 7f259bbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25941a0480 con 0x7f25941084f0 2023-09-10T01:03:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2594073060 0x7f2594198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2594073060 0x7f2594198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50538/0 (socket says 172.21.15.3:50538) 2023-09-10T01:03:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259919a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2594074640 0x7f2594198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 -- 172.21.15.3:0/2956109014 learned_addr learned my addr 172.21.15.3:0/2956109014 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259a19c700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25941084f0 0x7f259419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 -- 172.21.15.3:0/2956109014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25941084f0 msgr2=0x7f259419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25941084f0 0x7f259419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 -- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2594074640 msgr2=0x7f2594198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2594074640 0x7f2594198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 -- 172.21.15.3:0/2956109014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c009d00 con 0x7f2594073060 2023-09-10T01:03:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259999b700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2594073060 0x7f2594198630 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f259000e9a0 tx=0x7f259000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f258affd700 1 -- 172.21.15.3:0/2956109014 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590009790 con 0x7f2594073060 2023-09-10T01:03:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f258affd700 1 -- 172.21.15.3:0/2956109014 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2590010430 con 0x7f2594073060 2023-09-10T01:03:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25941a0760 con 0x7f2594073060 2023-09-10T01:03:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.808+0000 7f258affd700 1 -- 172.21.15.3:0/2956109014 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590018680 con 0x7f2594073060 2023-09-10T01:03:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.809+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25941a0cc0 con 0x7f2594073060 2023-09-10T01:03:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.810+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25941072b0 con 0x7f2594073060 2023-09-10T01:03:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.811+0000 7f258affd700 1 -- 172.21.15.3:0/2956109014 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 19) v1 ==== 135275+0+0 (secure 0 0 0) 0x7f2590048080 con 0x7f2594073060 2023-09-10T01:03:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.813+0000 7f258affd700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 0x7f25800a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.813+0000 7f259919a700 1 -- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 msgr2=0x7f25800a2bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.813+0000 7f258affd700 1 -- 172.21.15.3:0/2956109014 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f2590014070 con 0x7f2594073060 2023-09-10T01:03:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.813+0000 7f259919a700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 0x7f25800a2bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-10T01:03:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:03.815+0000 7f258affd700 1 -- 172.21.15.3:0/2956109014 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f25900958b0 con 0x7f2594073060 2023-09-10T01:03:03.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:03.698+0000 7f2cc90c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T01:03:04.015 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:03.623+0000 7f16bae43200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-10T01:03:04.015 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:03 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:03.773+0000 7f16bae43200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-10T01:03:04.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.014+0000 7f259919a700 1 -- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 msgr2=0x7f25800a2bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:04.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.014+0000 7f259919a700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 0x7f25800a2bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-10T01:03:04.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.040+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f259410e300 con 0x7f2594073060 2023-09-10T01:03:04.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.045+0000 7f258affd700 1 -- 172.21.15.3:0/2956109014 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v45)=0 v45) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2590098ed0 con 0x7f2594073060 2023-09-10T01:03:04.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 msgr2=0x7f25800a2bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:03:04.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 0x7f25800a2bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2594073060 msgr2=0x7f2594198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:04.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2594073060 0x7f2594198630 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f259000e9a0 tx=0x7f259000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 shutdown_connections 2023-09-10T01:03:04.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f25800a0710 0x7f25800a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25941084f0 0x7f259419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2594073060 0x7f2594198630 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 --2- 172.21.15.3:0/2956109014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2594074640 0x7f2594198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 >> 172.21.15.3:0/2956109014 conn(0x7f25940fbbc0 msgr2=0x7f25941043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:04.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 shutdown_connections 2023-09-10T01:03:04.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.048+0000 7f259bbff700 1 -- 172.21.15.3:0/2956109014 wait complete. 2023-09-10T01:03:04.067 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:03.713+0000 7fc2eda81200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T01:03:04.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-09-10T01:03:04.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:04.013+0000 7f16bae43200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-10T01:03:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-09-10T01:03:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-mon[97654]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-09-10T01:03:04.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T01:03:04.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' 2023-09-10T01:03:04.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-09-10T01:03:04.316 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-mon[134067]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-09-10T01:03:04.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T01:03:04.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' 2023-09-10T01:03:04.317 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:04.090+0000 7f2cc90c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T01:03:04.317 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:04.226+0000 7f2cc90c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T01:03:04.318 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:04.315+0000 7f2cc90c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T01:03:04.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 -- 172.21.15.3:0/2559856598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c240fee90 msgr2=0x7f3c241012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:04.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 --2- 172.21.15.3:0/2559856598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c240fee90 0x7f3c241012c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3c140098f0 tx=0x7f3c14009c00 comp rx=0 tx=0).stop 2023-09-10T01:03:04.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 -- 172.21.15.3:0/2559856598 shutdown_connections 2023-09-10T01:03:04.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 --2- 172.21.15.3:0/2559856598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c240fee90 0x7f3c241012c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 --2- 172.21.15.3:0/2559856598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c241041d0 0x7f3c24104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 --2- 172.21.15.3:0/2559856598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c24101800 0x7f3c24103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 -- 172.21.15.3:0/2559856598 >> 172.21.15.3:0/2559856598 conn(0x7f3c240faa90 msgr2=0x7f3c240fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:04.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 -- 172.21.15.3:0/2559856598 shutdown_connections 2023-09-10T01:03:04.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.331+0000 7f3c28dff700 1 -- 172.21.15.3:0/2559856598 wait complete. 2023-09-10T01:03:04.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 Processor -- start 2023-09-10T01:03:04.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 -- start start 2023-09-10T01:03:04.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c240fee90 0x7f3c2419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:04.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c24101800 0x7f3c2419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:04.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c241041d0 0x7f3c241a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:04.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c241a4510 con 0x7f3c240fee90 2023-09-10T01:03:04.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c241a4650 con 0x7f3c241041d0 2023-09-10T01:03:04.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.332+0000 7f3c28dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c241a47e0 con 0x7f3c24101800 2023-09-10T01:03:04.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c24101800 0x7f3c2419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:04.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c2259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c240fee90 0x7f3c2419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:04.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c24101800 0x7f3c2419ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51458/0 (socket says 172.21.15.3:51458) 2023-09-10T01:03:04.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 -- 172.21.15.3:0/414921993 learned_addr learned my addr 172.21.15.3:0/414921993 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:04.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c22d9d700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c241041d0 0x7f3c241a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:04.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c241041d0 msgr2=0x7f3c241a3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:04.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c241041d0 0x7f3c241a3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c240fee90 msgr2=0x7f3c2419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:04.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c240fee90 0x7f3c2419c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:04.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 -- 172.21.15.3:0/414921993 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c180095f0 con 0x7f3c24101800 2023-09-10T01:03:04.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.333+0000 7f3c21d9b700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c24101800 0x7f3c2419ced0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800ec10 tx=0x7f3c1800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:04.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.334+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1800cc90 con 0x7f3c24101800 2023-09-10T01:03:04.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.334+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c180044e0 con 0x7f3c24101800 2023-09-10T01:03:04.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.334+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c14009520 con 0x7f3c24101800 2023-09-10T01:03:04.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.334+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18010430 con 0x7f3c24101800 2023-09-10T01:03:04.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.334+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c241a4e40 con 0x7f3c24101800 2023-09-10T01:03:04.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.335+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c240fc610 con 0x7f3c24101800 2023-09-10T01:03:04.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.338+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 19) v1 ==== 135275+0+0 (secure 0 0 0) 0x7f3c18003fb0 con 0x7f3c24101800 2023-09-10T01:03:04.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.340+0000 7f3c137fe700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:04.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.340+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9181+0+0 (secure 0 0 0) 0x7f3c18014070 con 0x7f3c24101800 2023-09-10T01:03:04.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.340+0000 7f3c2259c700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 msgr2=0x7f3c0c0a2b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:04.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.340+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-10T01:03:04.346 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:04.107+0000 7fc2eda81200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T01:03:04.346 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:04.248+0000 7fc2eda81200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T01:03:04.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-09-10T01:03:04.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-mon[133708]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-09-10T01:03:04.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T01:03:04.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' 2023-09-10T01:03:04.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.344+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3c18095ca0 con 0x7f3c24101800 2023-09-10T01:03:04.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.541+0000 7f3c2259c700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 msgr2=0x7f3c0c0a2b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:04.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.541+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-10T01:03:04.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.614+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 --> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c2402d010 con 0x7f3c0c0a0690 2023-09-10T01:03:04.661 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:04.345+0000 7fc2eda81200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T01:03:04.662 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:04.444+0000 7fc2eda81200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T01:03:04.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:04.417+0000 7f2cc90c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T01:03:04.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:04.537+0000 7f16bae43200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-10T01:03:04.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:04.728+0000 7f16bae43200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-10T01:03:04.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.942+0000 7f3c2259c700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 msgr2=0x7f3c0c0a2b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:04.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:04.942+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-10T01:03:05.161 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:04.888+0000 7fc2eda81200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T01:03:05.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:04.861+0000 7f2cc90c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T01:03:05.308 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:04 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:04.868+0000 7f16bae43200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-10T01:03:05.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:05 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:05.015+0000 7f16bae43200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-10T01:03:05.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:05.744+0000 7f3c2259c700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 msgr2=0x7f3c0c0a2b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:05.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:05.744+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-10T01:03:05.911 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:05.588+0000 7fc2eda81200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T01:03:05.912 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:05.671+0000 7fc2eda81200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T01:03:05.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:05.547+0000 7f2cc90c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T01:03:05.966 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:05.630+0000 7f2cc90c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T01:03:06.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:05 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:05.598+0000 7f16bae43200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-10T01:03:06.665 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:06.401+0000 7f2cc90c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T01:03:06.666 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:06.491+0000 7f2cc90c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T01:03:06.713 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:06.448+0000 7fc2eda81200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T01:03:06.713 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:06.539+0000 7fc2eda81200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T01:03:06.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:06 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:06.519+0000 7f16bae43200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-10T01:03:06.809 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:06 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:06.630+0000 7f16bae43200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-10T01:03:06.946 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:06.664+0000 7f2cc90c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T01:03:06.993 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:06.712+0000 7fc2eda81200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T01:03:07.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:06.945+0000 7f2cc90c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T01:03:07.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:07.173+0000 7f2cc90c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T01:03:07.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:07.346+0000 7f3c2259c700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 msgr2=0x7f3c0c0a2b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:07.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:07.346+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-09-10T01:03:07.373 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:06.992+0000 7fc2eda81200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T01:03:07.373 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:07.225+0000 7fc2eda81200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T01:03:07.585 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:07.320+0000 7f2cc90c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T01:03:07.637 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:07.372+0000 7fc2eda81200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T01:03:07.911 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:07.637+0000 7fc2eda81200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T01:03:07.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:07.584+0000 7f2cc90c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T01:03:07.979 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:07 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:07.624+0000 7f16bae43200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-10T01:03:07.980 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:07 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:07.747+0000 7f16bae43200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-10T01:03:08.308 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:07 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:07.979+0000 7f16bae43200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-10T01:03:08.608 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:08.284+0000 7f2cc90c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T01:03:08.608 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:08.415+0000 7f2cc90c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T01:03:08.645 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:08 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:08.360+0000 7f16bae43200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-10T01:03:08.661 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:08.346+0000 7fc2eda81200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T01:03:08.662 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:08.476+0000 7fc2eda81200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T01:03:08.938 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:08.607+0000 7f2cc90c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T01:03:08.938 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:08.720+0000 7f2cc90c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T01:03:09.001 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:08.670+0000 7fc2eda81200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T01:03:09.001 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:08.783+0000 7fc2eda81200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T01:03:09.058 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:08 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:08.644+0000 7f16bae43200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-10T01:03:09.059 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:08 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:08.846+0000 7f16bae43200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-10T01:03:09.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:08.937+0000 7f2cc90c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T01:03:09.412 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:09.000+0000 7fc2eda81200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T01:03:09.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:09 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:09.204+0000 7f16bae43200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-10T01:03:09.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:09.870+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 20) v1 ==== 135275+0+0 (secure 0 0 0) 0x7f3c18096070 con 0x7f3c24101800 2023-09-10T01:03:10.162 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z[135733]: 2023-09-10T01:03:09.884+0000 7fc2eda81200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T01:03:10.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-mon[133708]: Standby manager daemon y restarted 2023-09-10T01:03:10.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-mon[133708]: Standby manager daemon y started 2023-09-10T01:03:10.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-09-10T01:03:10.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T01:03:10.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-09-10T01:03:10.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:09 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T01:03:10.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:09 smithi003 ceph-mon[97654]: Standby manager daemon y restarted 2023-09-10T01:03:10.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:09 smithi003 ceph-mon[97654]: Standby manager daemon y started 2023-09-10T01:03:10.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:09 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-09-10T01:03:10.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:09 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T01:03:10.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:09 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-09-10T01:03:10.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:09 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T01:03:10.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:03:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y[135640]: 2023-09-10T01:03:09.795+0000 7f2cc90c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T01:03:10.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:09 smithi092 ceph-mon[134067]: Standby manager daemon y restarted 2023-09-10T01:03:10.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:09 smithi092 ceph-mon[134067]: Standby manager daemon y started 2023-09-10T01:03:10.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:09 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-09-10T01:03:10.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:09 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T01:03:10.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:09 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-09-10T01:03:10.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:09 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.92:0/1800377339' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T01:03:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:10.549+0000 7f3c2259c700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 msgr2=0x7f3c0c0a2b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/91094391 2023-09-10T01:03:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:10.549+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-09-10T01:03:10.558 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:10.176+0000 7f16bae43200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-10T01:03:10.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:10.358+0000 7f16bae43200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-10T01:03:10.874 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:10.618+0000 7f16bae43200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-10T01:03:10.875 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:10.773+0000 7f16bae43200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-10T01:03:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:10.889+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 8 ==== mgrmap(e 21) v1 ==== 135275+0+0 (secure 0 0 0) 0x7f3c180c9cf0 con 0x7f3c24101800 2023-09-10T01:03:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:10 smithi136 ceph-mon[133708]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-09-10T01:03:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:10 smithi136 ceph-mon[133708]: Standby manager daemon z restarted 2023-09-10T01:03:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:10 smithi136 ceph-mon[133708]: Standby manager daemon z started 2023-09-10T01:03:11.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:10 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-09-10T01:03:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:10 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T01:03:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:10 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-09-10T01:03:11.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:10 smithi136 ceph-mon[133708]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T01:03:11.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:10 smithi092 ceph-mon[134067]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-09-10T01:03:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:10 smithi092 ceph-mon[134067]: Standby manager daemon z restarted 2023-09-10T01:03:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:10 smithi092 ceph-mon[134067]: Standby manager daemon z started 2023-09-10T01:03:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:10 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-09-10T01:03:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:10 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T01:03:11.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:10 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-09-10T01:03:11.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:10 smithi092 ceph-mon[134067]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T01:03:11.308 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:11 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:11.067+0000 7f16bae43200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-10T01:03:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-mon[97654]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-09-10T01:03:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-mon[97654]: Standby manager daemon z restarted 2023-09-10T01:03:11.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-mon[97654]: Standby manager daemon z started 2023-09-10T01:03:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-09-10T01:03:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-09-10T01:03:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-09-10T01:03:11.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:10 smithi003 ceph-mon[97654]: from='mgr.? 172.21.15.136:0/1989297289' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-09-10T01:03:12.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:11 smithi136 ceph-mon[133708]: mgrmap e21: x(active, since 9m), standbys: z, y 2023-09-10T01:03:12.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:11 smithi003 ceph-mon[97654]: mgrmap e21: x(active, since 9m), standbys: z, y 2023-09-10T01:03:12.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:12.204+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 9 ==== mgrmap(e 22) v1 ==== 134944+0+0 (secure 0 0 0) 0x7f3c18098e50 con 0x7f3c24101800 2023-09-10T01:03:12.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:12.205+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 msgr2=0x7f3c0c0a2b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:03:12.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:12.205+0000 7f3c137fe700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/91094391,v1:172.21.15.3:6801/91094391] conn(0x7f3c0c0a0690 0x7f3c0c0a2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:12.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:11 smithi092 ceph-mon[134067]: mgrmap e21: x(active, since 9m), standbys: z, y 2023-09-10T01:03:12.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.186+0000 7f16bae43200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-10T01:03:12.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.415+0000 7f161c77b700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.415+0000 7f161c77b700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.415+0000 7f161c77b700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.415+0000 7f161c77b700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.415+0000 7f161c77b700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.417+0000 7f1619775700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.560 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.417+0000 7f1619775700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.417+0000 7f1619775700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.417+0000 7f1619775700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.561 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:03:12.417+0000 7f1619775700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:03:12.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: Active manager daemon x restarted 2023-09-10T01:03:12.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: Activating manager daemon x 2023-09-10T01:03:12.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: osdmap e91: 12 total, 12 up, 12 in 2023-09-10T01:03:12.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: mgrmap e22: x(active, starting, since 0.0124585s), standbys: z, y 2023-09-10T01:03:13.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: Active manager daemon x restarted 2023-09-10T01:03:13.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: Activating manager daemon x 2023-09-10T01:03:13.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: osdmap e91: 12 total, 12 up, 12 in 2023-09-10T01:03:13.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: mgrmap e22: x(active, starting, since 0.0124585s), standbys: z, y 2023-09-10T01:03:13.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T01:03:13.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T01:03:13.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T01:03:13.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-09-10T01:03:13.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-09-10T01:03:13.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-09-10T01:03:13.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-09-10T01:03:13.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-09-10T01:03:13.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-09-10T01:03:13.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:03:13.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:03:13.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:03:13.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:03:13.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:03:13.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:03:13.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-09-10T01:03:13.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-09-10T01:03:13.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-09-10T01:03:13.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T01:03:13.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T01:03:13.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T01:03:13.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T01:03:13.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T01:03:13.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T01:03:13.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T01:03:13.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T01:03:13.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T01:03:13.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T01:03:13.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T01:03:13.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T01:03:13.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-09-10T01:03:13.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-09-10T01:03:13.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-09-10T01:03:13.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: Manager daemon x is now available 2023-09-10T01:03:13.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:13.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:13.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T01:03:13.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T01:03:13.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T01:03:13.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:12 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T01:03:13.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.209+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mon.1 v2:172.21.15.136:3300/0 10 ==== mgrmap(e 23) v1 ==== 135071+0+0 (secure 0 0 0) 0x7f3c18096940 con 0x7f3c24101800 2023-09-10T01:03:13.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.210+0000 7f3c137fe700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c0c04dba0 0x7f3c0c04df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:13.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.210+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 --> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c2402d010 con 0x7f3c0c04dba0 2023-09-10T01:03:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: Active manager daemon x restarted 2023-09-10T01:03:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: Activating manager daemon x 2023-09-10T01:03:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: osdmap e91: 12 total, 12 up, 12 in 2023-09-10T01:03:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: mgrmap e22: x(active, starting, since 0.0124585s), standbys: z, y 2023-09-10T01:03:13.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T01:03:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T01:03:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T01:03:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-09-10T01:03:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-09-10T01:03:13.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-09-10T01:03:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-09-10T01:03:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-09-10T01:03:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-09-10T01:03:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:03:13.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:03:13.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:03:13.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:03:13.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:03:13.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:03:13.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-09-10T01:03:13.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-09-10T01:03:13.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-09-10T01:03:13.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T01:03:13.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T01:03:13.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T01:03:13.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T01:03:13.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T01:03:13.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T01:03:13.221 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T01:03:13.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T01:03:13.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T01:03:13.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T01:03:13.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T01:03:13.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T01:03:13.222 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-09-10T01:03:13.223 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-09-10T01:03:13.223 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-09-10T01:03:13.223 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: Manager daemon x is now available 2023-09-10T01:03:13.223 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:13.223 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:13.224 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T01:03:13.224 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T01:03:13.224 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T01:03:13.224 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:12 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T01:03:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.216+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c0c04dba0 0x7f3c0c04df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:13.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.222+0000 7f3c2259c700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c0c04dba0 0x7f3c0c04df80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3c14005e90 tx=0x7f3c14005dc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:13.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.291+0000 7f3c137fe700 1 -- 172.21.15.3:0/414921993 <== mgr.25667 v2:172.21.15.3:6800/3336040255 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3c2402d010 con 0x7f3c0c04dba0 2023-09-10T01:03:13.292 INFO:teuthology.orchestra.run.smithi003.stdout:Schedule set for path / 2023-09-10T01:03:13.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c0c04dba0 msgr2=0x7f3c0c04df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:13.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c0c04dba0 0x7f3c0c04df80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3c14005e90 tx=0x7f3c14005dc0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c24101800 msgr2=0x7f3c2419ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:13.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 --2- 172.21.15.3:0/414921993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c24101800 0x7f3c2419ced0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800ec10 tx=0x7f3c1800ef20 comp rx=0 tx=0).stop 2023-09-10T01:03:13.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c22d9d700 1 -- 172.21.15.3:0/414921993 reap_dead start 2023-09-10T01:03:13.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 shutdown_connections 2023-09-10T01:03:13.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 >> 172.21.15.3:0/414921993 conn(0x7f3c240faa90 msgr2=0x7f3c24102510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:13.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 shutdown_connections 2023-09-10T01:03:13.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.293+0000 7f3c28dff700 1 -- 172.21.15.3:0/414921993 wait complete. 2023-09-10T01:03:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-09-10T01:03:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-09-10T01:03:13.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-09-10T01:03:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-09-10T01:03:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-09-10T01:03:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-09-10T01:03:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-09-10T01:03:13.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-09-10T01:03:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-09-10T01:03:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:03:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:03:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:03:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:03:13.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:03:13.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:03:13.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-09-10T01:03:13.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-09-10T01:03:13.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-09-10T01:03:13.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-09-10T01:03:13.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-09-10T01:03:13.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-09-10T01:03:13.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-09-10T01:03:13.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-09-10T01:03:13.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-09-10T01:03:13.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-09-10T01:03:13.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-09-10T01:03:13.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-09-10T01:03:13.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-09-10T01:03:13.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-09-10T01:03:13.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-09-10T01:03:13.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-09-10T01:03:13.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-09-10T01:03:13.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-09-10T01:03:13.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: Manager daemon x is now available 2023-09-10T01:03:13.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:13.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:13.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T01:03:13.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-09-10T01:03:13.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T01:03:13.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:12 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-09-10T01:03:13.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-09-10T01:03:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.532+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2161254321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88100f20 msgr2=0x7f5a88103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.532+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2161254321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88100f20 0x7f5a88103310 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009990 tx=0x7f5a7c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.532+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2161254321 shutdown_connections 2023-09-10T01:03:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.532+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2161254321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88106180 0x7f5a881085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.532+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2161254321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88100f20 0x7f5a88103310 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.532+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2161254321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a88103850 0x7f5a88105c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.533+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2161254321 >> 172.21.15.3:0/2161254321 conn(0x7f5a880fa910 msgr2=0x7f5a880fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:13.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.533+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2161254321 shutdown_connections 2023-09-10T01:03:13.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.533+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2161254321 wait complete. 2023-09-10T01:03:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.534+0000 7f5a8dd9d700 1 Processor -- start 2023-09-10T01:03:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.534+0000 7f5a8dd9d700 1 -- start start 2023-09-10T01:03:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8dd9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88100f20 0x7f5a88196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8dd9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88103850 0x7f5a88196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8dd9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a88106180 0x7f5a8819d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8dd9d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8819dcc0 con 0x7f5a88100f20 2023-09-10T01:03:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8dd9d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8819de00 con 0x7f5a88103850 2023-09-10T01:03:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8dd9d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8819df90 con 0x7f5a88106180 2023-09-10T01:03:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88100f20 0x7f5a88196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:13.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a87fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88103850 0x7f5a88196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:13.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88100f20 0x7f5a88196140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41084/0 (socket says 172.21.15.3:41084) 2023-09-10T01:03:13.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 -- 172.21.15.3:0/2642549363 learned_addr learned my addr 172.21.15.3:0/2642549363 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:13.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a87fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88103850 0x7f5a88196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:32830/0 (socket says 172.21.15.3:32830) 2023-09-10T01:03:13.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 -- 172.21.15.3:0/2642549363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a88106180 msgr2=0x7f5a8819d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:13.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a88106180 0x7f5a8819d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 -- 172.21.15.3:0/2642549363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88103850 msgr2=0x7f5a88196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:13.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88103850 0x7f5a88196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.535+0000 7f5a8cd9b700 1 -- 172.21.15.3:0/2642549363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a80009c40 con 0x7f5a88100f20 2023-09-10T01:03:13.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.536+0000 7f5a87fff700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88103850 0x7f5a88196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:13.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.536+0000 7f5a8cd9b700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88100f20 0x7f5a88196140 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c004730 tx=0x7f5a7c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:13.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.536+0000 7f5a85ffb700 1 -- 172.21.15.3:0/2642549363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c01d070 con 0x7f5a88100f20 2023-09-10T01:03:13.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.536+0000 7f5a85ffb700 1 -- 172.21.15.3:0/2642549363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a7c022470 con 0x7f5a88100f20 2023-09-10T01:03:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.536+0000 7f5a85ffb700 1 -- 172.21.15.3:0/2642549363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c017650 con 0x7f5a88100f20 2023-09-10T01:03:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.536+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c0095f0 con 0x7f5a88100f20 2023-09-10T01:03:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.536+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a880fef50 con 0x7f5a88100f20 2023-09-10T01:03:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.538+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a880fc490 con 0x7f5a88100f20 2023-09-10T01:03:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.539+0000 7f5a85ffb700 1 -- 172.21.15.3:0/2642549363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 23) v1 ==== 135071+0+0 (secure 0 0 0) 0x7f5a7c0225e0 con 0x7f5a88100f20 2023-09-10T01:03:13.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.542+0000 7f5a85ffb700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5a780a0440 0x7f5a780a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.542+0000 7f5a85ffb700 1 -- 172.21.15.3:0/2642549363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f5a7c0cd170 con 0x7f5a88100f20 2023-09-10T01:03:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.542+0000 7f5a87fff700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5a780a0440 0x7f5a780a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.543+0000 7f5a87fff700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5a780a0440 0x7f5a780a2900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400ffd0 tx=0x7f5a7400f820 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.546+0000 7f5a85ffb700 1 -- 172.21.15.3:0/2642549363 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5a7c096c50 con 0x7f5a88100f20 2023-09-10T01:03:13.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.806+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 --> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a8802d0d0 con 0x7f5a780a0440 2023-09-10T01:03:13.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.866+0000 7f5a85ffb700 1 -- 172.21.15.3:0/2642549363 <== mgr.25667 v2:172.21.15.3:6800/3336040255 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f5a8802d0d0 con 0x7f5a780a0440 2023-09-10T01:03:13.867 INFO:teuthology.orchestra.run.smithi003.stdout:Retention added to path / 2023-09-10T01:03:13.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.868+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5a780a0440 msgr2=0x7f5a780a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:13.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.868+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5a780a0440 0x7f5a780a2900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400ffd0 tx=0x7f5a7400f820 comp rx=0 tx=0).stop 2023-09-10T01:03:13.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.868+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88100f20 msgr2=0x7f5a88196140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:13.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.868+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88100f20 0x7f5a88196140 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c004730 tx=0x7f5a7c004760 comp rx=0 tx=0).stop 2023-09-10T01:03:13.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 shutdown_connections 2023-09-10T01:03:13.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5a780a0440 0x7f5a780a2900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a88106180 0x7f5a8819d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a88100f20 0x7f5a88196140 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 --2- 172.21.15.3:0/2642549363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a88103850 0x7f5a88196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:13.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 >> 172.21.15.3:0/2642549363 conn(0x7f5a880fa910 msgr2=0x7f5a88101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:13.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 shutdown_connections 2023-09-10T01:03:13.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:13.869+0000 7f5a8dd9d700 1 -- 172.21.15.3:0/2642549363 wait complete. 2023-09-10T01:03:13.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-09-10T01:03:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 -- 172.21.15.3:0/3675158937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c101820 msgr2=0x7f0b7c101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 --2- 172.21.15.3:0/3675158937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c101820 0x7f0b7c101c80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c0099c0 tx=0x7f0b6c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 -- 172.21.15.3:0/3675158937 shutdown_connections 2023-09-10T01:03:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 --2- 172.21.15.3:0/3675158937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b7c1021c0 0x7f0b7c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 --2- 172.21.15.3:0/3675158937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 0x7f0b7c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 --2- 172.21.15.3:0/3675158937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c101820 0x7f0b7c101c80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 -- 172.21.15.3:0/3675158937 >> 172.21.15.3:0/3675158937 conn(0x7f0b7c0fbbc0 msgr2=0x7f0b7c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:14.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 -- 172.21.15.3:0/3675158937 shutdown_connections 2023-09-10T01:03:14.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.102+0000 7f0b847ff700 1 -- 172.21.15.3:0/3675158937 wait complete. 2023-09-10T01:03:14.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.103+0000 7f0b847ff700 1 Processor -- start 2023-09-10T01:03:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b847ff700 1 -- start start 2023-09-10T01:03:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b847ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 0x7f0b7c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b847ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b7c101820 0x7f0b7c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b847ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c1021c0 0x7f0b7c19d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b847ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c19dea0 con 0x7f0b7c1021c0 2023-09-10T01:03:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b847ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c19dfe0 con 0x7f0b7c100630 2023-09-10T01:03:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b847ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c19e170 con 0x7f0b7c101820 2023-09-10T01:03:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b8259b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 0x7f0b7c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b8259b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 0x7f0b7c196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:32854/0 (socket says 172.21.15.3:32854) 2023-09-10T01:03:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b8259b700 1 -- 172.21.15.3:0/1122978507 learned_addr learned my addr 172.21.15.3:0/1122978507 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b81d9a700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b7c101820 0x7f0b7c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.104+0000 7f0b82d9c700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c1021c0 0x7f0b7c19d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b8259b700 1 -- 172.21.15.3:0/1122978507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b7c101820 msgr2=0x7f0b7c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b8259b700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b7c101820 0x7f0b7c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b8259b700 1 -- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c1021c0 msgr2=0x7f0b7c19d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b8259b700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c1021c0 0x7f0b7c19d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b8259b700 1 -- 172.21.15.3:0/1122978507 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009520 con 0x7f0b7c100630 2023-09-10T01:03:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b8259b700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 0x7f0b7c196320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800ea50 tx=0x7f0b7800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7800cbd0 con 0x7f0b7c100630 2023-09-10T01:03:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b7800cd30 con 0x7f0b7c100630 2023-09-10T01:03:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6c0095f0 con 0x7f0b7c100630 2023-09-10T01:03:14.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78018680 con 0x7f0b7c100630 2023-09-10T01:03:14.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.105+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c19e7d0 con 0x7f0b7c100630 2023-09-10T01:03:14.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.106+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c10be90 con 0x7f0b7c100630 2023-09-10T01:03:14.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.108+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 23) v1 ==== 135071+0+0 (secure 0 0 0) 0x7f0b78010430 con 0x7f0b7c100630 2023-09-10T01:03:14.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.110+0000 7f0b737fe700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b680a0440 0x7f0b680a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.111+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0b78014070 con 0x7f0b7c100630 2023-09-10T01:03:14.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.111+0000 7f0b81d9a700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b680a0440 0x7f0b680a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.112+0000 7f0b81d9a700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b680a0440 0x7f0b680a2900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00b580 tx=0x7f0b6c005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:14.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.112+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0b78095f80 con 0x7f0b7c100630 2023-09-10T01:03:14.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.217+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 24) v1 ==== 135378+0+0 (secure 0 0 0) 0x7f0b78096350 con 0x7f0b7c100630 2023-09-10T01:03:14.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.399+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 --> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b7c060a40 con 0x7f0b680a0440 2023-09-10T01:03:14.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.407+0000 7f0b737fe700 1 -- 172.21.15.3:0/1122978507 <== mgr.25667 v2:172.21.15.3:6800/3336040255 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f0b7c060a40 con 0x7f0b680a0440 2023-09-10T01:03:14.409 INFO:teuthology.orchestra.run.smithi003.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-09-10T00:00:00", "created": "2023-09-10T01:03:13", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-09-10T01:03:14.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b680a0440 msgr2=0x7f0b680a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b680a0440 0x7f0b680a2900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00b580 tx=0x7f0b6c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 msgr2=0x7f0b7c196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 0x7f0b7c196320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800ea50 tx=0x7f0b7800ed60 comp rx=0 tx=0).stop 2023-09-10T01:03:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 shutdown_connections 2023-09-10T01:03:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b680a0440 0x7f0b680a2900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b7c1021c0 0x7f0b7c19d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b7c100630 0x7f0b7c196320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 --2- 172.21.15.3:0/1122978507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b7c101820 0x7f0b7c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 >> 172.21.15.3:0/1122978507 conn(0x7f0b7c0fbbc0 msgr2=0x7f0b7c0fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:14.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 shutdown_connections 2023-09-10T01:03:14.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.410+0000 7f0b847ff700 1 -- 172.21.15.3:0/1122978507 wait complete. 2023-09-10T01:03:14.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-09-10T01:03:14.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:14 smithi092 ceph-mon[134067]: mgrmap e23: x(active, since 1.01865s), standbys: z, y 2023-09-10T01:03:14.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:14 smithi092 ceph-mon[134067]: [10/Sep/2023:01:03:13] ENGINE Bus STARTING 2023-09-10T01:03:14.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:14 smithi092 ceph-mon[134067]: [10/Sep/2023:01:03:13] ENGINE Serving on https://172.21.15.3:7150 2023-09-10T01:03:14.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:14 smithi092 ceph-mon[134067]: [10/Sep/2023:01:03:13] ENGINE Serving on http://172.21.15.3:8765 2023-09-10T01:03:14.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:14 smithi092 ceph-mon[134067]: [10/Sep/2023:01:03:13] ENGINE Bus STARTED 2023-09-10T01:03:14.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:14 smithi092 ceph-mon[134067]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:14.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:14 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:03:14.495 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:14 smithi003 ceph-mon[97654]: mgrmap e23: x(active, since 1.01865s), standbys: z, y 2023-09-10T01:03:14.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:14 smithi003 ceph-mon[97654]: [10/Sep/2023:01:03:13] ENGINE Bus STARTING 2023-09-10T01:03:14.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:14 smithi003 ceph-mon[97654]: [10/Sep/2023:01:03:13] ENGINE Serving on https://172.21.15.3:7150 2023-09-10T01:03:14.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:14 smithi003 ceph-mon[97654]: [10/Sep/2023:01:03:13] ENGINE Serving on http://172.21.15.3:8765 2023-09-10T01:03:14.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:14 smithi003 ceph-mon[97654]: [10/Sep/2023:01:03:13] ENGINE Bus STARTED 2023-09-10T01:03:14.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:14 smithi003 ceph-mon[97654]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:14.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:14 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:03:14.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 -- 172.21.15.3:0/2939578503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c074640 msgr2=0x7f036c072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 --2- 172.21.15.3:0/2939578503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c074640 0x7f036c072a90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0368009990 tx=0x7f0368009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 -- 172.21.15.3:0/2939578503 shutdown_connections 2023-09-10T01:03:14.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 --2- 172.21.15.3:0/2939578503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f036c1084f0 0x7f036c10a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 --2- 172.21.15.3:0/2939578503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c074640 0x7f036c072a90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 --2- 172.21.15.3:0/2939578503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f036c073060 0x7f036c0734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 -- 172.21.15.3:0/2939578503 >> 172.21.15.3:0/2939578503 conn(0x7f036c0fbb80 msgr2=0x7f036c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:14.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 -- 172.21.15.3:0/2939578503 shutdown_connections 2023-09-10T01:03:14.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.645+0000 7f0373fff700 1 -- 172.21.15.3:0/2939578503 wait complete. 2023-09-10T01:03:14.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 Processor -- start 2023-09-10T01:03:14.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 -- start start 2023-09-10T01:03:14.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c073060 0x7f036c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f036c074640 0x7f036c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f036c1084f0 0x7f036c19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1a0090 con 0x7f036c1084f0 2023-09-10T01:03:14.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1a01d0 con 0x7f036c073060 2023-09-10T01:03:14.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.646+0000 7f0373fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1a0360 con 0x7f036c074640 2023-09-10T01:03:14.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c073060 0x7f036c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c073060 0x7f036c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:32904/0 (socket says 172.21.15.3:32904) 2023-09-10T01:03:14.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 -- 172.21.15.3:0/1690549158 learned_addr learned my addr 172.21.15.3:0/1690549158 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:14.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f037159a700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f036c074640 0x7f036c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f037259c700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f036c1084f0 0x7f036c19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 -- 172.21.15.3:0/1690549158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f036c074640 msgr2=0x7f036c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f036c074640 0x7f036c198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 -- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f036c1084f0 msgr2=0x7f036c19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f036c1084f0 0x7f036c19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 -- 172.21.15.3:0/1690549158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680095f0 con 0x7f036c073060 2023-09-10T01:03:14.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0371d9b700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c073060 0x7f036c198510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f036800b680 tx=0x7f03680057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:14.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0362ffd700 1 -- 172.21.15.3:0/1690549158 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036801d070 con 0x7f036c073060 2023-09-10T01:03:14.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0362ffd700 1 -- 172.21.15.3:0/1690549158 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f036800fbc0 con 0x7f036c073060 2023-09-10T01:03:14.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.647+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c1a05e0 con 0x7f036c073060 2023-09-10T01:03:14.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.648+0000 7f0362ffd700 1 -- 172.21.15.3:0/1690549158 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368017650 con 0x7f036c073060 2023-09-10T01:03:14.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.648+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c1a0a60 con 0x7f036c073060 2023-09-10T01:03:14.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.649+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036c106ba0 con 0x7f036c073060 2023-09-10T01:03:14.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.650+0000 7f0362ffd700 1 -- 172.21.15.3:0/1690549158 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) v1 ==== 135378+0+0 (secure 0 0 0) 0x7f0368022470 con 0x7f036c073060 2023-09-10T01:03:14.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.652+0000 7f0362ffd700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03580a0830 0x7f03580a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:14.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.652+0000 7f0362ffd700 1 -- 172.21.15.3:0/1690549158 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f03680ce440 con 0x7f036c073060 2023-09-10T01:03:14.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.652+0000 7f037159a700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03580a0830 0x7f03580a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:14.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.653+0000 7f037159a700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03580a0830 0x7f03580a2cf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f036c074360 tx=0x7f035c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:14.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.654+0000 7f0362ffd700 1 -- 172.21.15.3:0/1690549158 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f03680978d0 con 0x7f036c073060 2023-09-10T01:03:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:14 smithi136 ceph-mon[133708]: mgrmap e23: x(active, since 1.01865s), standbys: z, y 2023-09-10T01:03:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:14 smithi136 ceph-mon[133708]: [10/Sep/2023:01:03:13] ENGINE Bus STARTING 2023-09-10T01:03:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:14 smithi136 ceph-mon[133708]: [10/Sep/2023:01:03:13] ENGINE Serving on https://172.21.15.3:7150 2023-09-10T01:03:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:14 smithi136 ceph-mon[133708]: [10/Sep/2023:01:03:13] ENGINE Serving on http://172.21.15.3:8765 2023-09-10T01:03:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:14 smithi136 ceph-mon[133708]: [10/Sep/2023:01:03:13] ENGINE Bus STARTED 2023-09-10T01:03:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:14 smithi136 ceph-mon[133708]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:14.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:14 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:03:14.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.908+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 --> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f036c02cf80 con 0x7f03580a0830 2023-09-10T01:03:14.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.915+0000 7f0362ffd700 1 -- 172.21.15.3:0/1690549158 <== mgr.25667 v2:172.21.15.3:6800/3336040255 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f036c02cf80 con 0x7f03580a0830 2023-09-10T01:03:14.916 INFO:teuthology.orchestra.run.smithi003.stdout:/ 1M 6M3h 2023-09-10T01:03:14.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03580a0830 msgr2=0x7f03580a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03580a0830 0x7f03580a2cf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f036c074360 tx=0x7f035c009260 comp rx=0 tx=0).stop 2023-09-10T01:03:14.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c073060 msgr2=0x7f036c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:14.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c073060 0x7f036c198510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f036800b680 tx=0x7f03680057b0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 shutdown_connections 2023-09-10T01:03:14.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03580a0830 0x7f03580a2cf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f036c1084f0 0x7f036c19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f036c073060 0x7f036c198510 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 --2- 172.21.15.3:0/1690549158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f036c074640 0x7f036c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:14.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 >> 172.21.15.3:0/1690549158 conn(0x7f036c0fbb80 msgr2=0x7f036c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:14.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 shutdown_connections 2023-09-10T01:03:14.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:14.918+0000 7f0373fff700 1 -- 172.21.15.3:0/1690549158 wait complete. 2023-09-10T01:03:14.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-09-10T01:03:15.022 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-09-10T01:03:15.074 INFO:teuthology.run_tasks:Running task background_exec... 2023-09-10T01:03:15.105 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:03:15.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-09-10T01:03:15.111 INFO:teuthology.run_tasks:Running task workunit... 2023-09-10T01:03:15.124 INFO:tasks.workunit:Pulling workunits from ref 9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T01:03:15.125 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-09-10T01:03:15.125 INFO:tasks.workunit:timeout=3h 2023-09-10T01:03:15.125 INFO:tasks.workunit:cleanup=True 2023-09-10T01:03:15.126 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2023-09-10T01:03:15.173 INFO:teuthology.orchestra.run.smithi003.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-09-10T01:03:15.173 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-09-10T01:03:15.173 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2023-09-10T01:03:15.173 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-10T01:03:15.177 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:cephfs_t:s0 2023-09-10T01:03:15.177 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-09-10 01:01:11.461018102 +0000 2023-09-10T01:03:15.177 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-09-10 01:01:11.461018102 +0000 2023-09-10T01:03:15.177 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-09-10 01:02:52.064114527 +0000 2023-09-10T01:03:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-09-10 01:01:11.461018102 +0000 2023-09-10T01:03:15.178 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-09-10T01:03:15.178 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-09-10T01:03:15.275 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9670cca5bc8e17916865dc84939edfc10b1f2c00 2023-09-10T01:03:15.324 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-09-10T01:03:15.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:15 smithi092 ceph-mon[134067]: mgrmap e24: x(active, since 2s), standbys: z, y 2023-09-10T01:03:15.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:15 smithi092 ceph-mon[134067]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:03:15.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:15 smithi092 ceph-mon[134067]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:15.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:15 smithi092 ceph-mon[134067]: from='client.25745 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:15.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:15 smithi003 ceph-mon[97654]: mgrmap e24: x(active, since 2s), standbys: z, y 2023-09-10T01:03:15.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:15 smithi003 ceph-mon[97654]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:03:15.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:15 smithi003 ceph-mon[97654]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:15.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:15 smithi003 ceph-mon[97654]: from='client.25745 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:15.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:15 smithi136 ceph-mon[133708]: mgrmap e24: x(active, since 2s), standbys: z, y 2023-09-10T01:03:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:15 smithi136 ceph-mon[133708]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:03:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:15 smithi136 ceph-mon[133708]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:15 smithi136 ceph-mon[133708]: from='client.25745 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:03:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi136 to 196.2M 2023-09-10T01:03:16.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:16 smithi003 ceph-mon[97654]: Unable to set osd_memory_target on smithi136 to 205750988: error parsing value: Value '205750988' is below minimum 939524096 2023-09-10T01:03:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi136 to 196.2M 2023-09-10T01:03:16.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:16 smithi136 ceph-mon[133708]: Unable to set osd_memory_target on smithi136 to 205750988: error parsing value: Value '205750988' is below minimum 939524096 2023-09-10T01:03:16.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:16.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi136 to 196.2M 2023-09-10T01:03:16.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:16 smithi092 ceph-mon[134067]: Unable to set osd_memory_target on smithi136 to 205750988: error parsing value: Value '205750988' is below minimum 939524096 2023-09-10T01:03:17.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:03:17.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi092 to 196.2M 2023-09-10T01:03:17.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: Unable to set osd_memory_target on smithi092 to 205766041: error parsing value: Value '205766041' is below minimum 939524096 2023-09-10T01:03:17.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:17 smithi003 ceph-mon[97654]: mgrmap e25: x(active, since 4s), standbys: z, y 2023-09-10T01:03:17.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:03:17.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi092 to 196.2M 2023-09-10T01:03:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: Unable to set osd_memory_target on smithi092 to 205766041: error parsing value: Value '205766041' is below minimum 939524096 2023-09-10T01:03:17.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:17 smithi136 ceph-mon[133708]: mgrmap e25: x(active, since 4s), standbys: z, y 2023-09-10T01:03:17.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:03:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:17.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi092 to 196.2M 2023-09-10T01:03:17.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: Unable to set osd_memory_target on smithi092 to 205766041: error parsing value: Value '205766041' is below minimum 939524096 2023-09-10T01:03:17.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:17.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:17 smithi092 ceph-mon[134067]: mgrmap e25: x(active, since 4s), standbys: z, y 2023-09-10T01:03:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi003 to 222.0M 2023-09-10T01:03:19.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: Unable to set osd_memory_target on smithi003 to 232784384: error parsing value: Value '232784384' is below minimum 939524096 2023-09-10T01:03:19.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:03:19.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:03:19.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: Updating smithi003:/etc/ceph/ceph.conf 2023-09-10T01:03:19.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T01:03:19.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T01:03:19.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:19.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:18 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.217 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi003 to 222.0M 2023-09-10T01:03:19.218 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: Unable to set osd_memory_target on smithi003 to 232784384: error parsing value: Value '232784384' is below minimum 939524096 2023-09-10T01:03:19.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:03:19.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:03:19.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: Updating smithi003:/etc/ceph/ceph.conf 2023-09-10T01:03:19.219 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T01:03:19.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T01:03:19.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:19.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.220 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:18 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:03:19.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi003 to 222.0M 2023-09-10T01:03:19.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: Unable to set osd_memory_target on smithi003 to 232784384: error parsing value: Value '232784384' is below minimum 939524096 2023-09-10T01:03:19.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:03:19.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:03:19.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: Updating smithi003:/etc/ceph/ceph.conf 2023-09-10T01:03:19.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: Updating smithi092:/etc/ceph/ceph.conf 2023-09-10T01:03:19.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: Updating smithi136:/etc/ceph/ceph.conf 2023-09-10T01:03:19.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:19.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:19.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:18 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:20.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 1 op/s 2023-09-10T01:03:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:20.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:20.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:20 smithi003 ceph-mon[97654]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 1 op/s 2023-09-10T01:03:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:20.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:20.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:20.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:20 smithi136 ceph-mon[133708]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 1 op/s 2023-09-10T01:03:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.conf 2023-09-10T01:03:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:20.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: Updating smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:20.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: Updating smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:20.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:20.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:20.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:20 smithi092 ceph-mon[134067]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2023-09-10T01:03:21.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:21.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:03:21.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:03:21.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:21 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:21.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:21.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:03:21.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:03:21.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:21 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: Updating smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/config/ceph.client.admin.keyring 2023-09-10T01:03:21.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:21.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:03:21.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:03:21.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:03:21.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:21 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:03:22.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:22 smithi136 ceph-mon[133708]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 132 KiB/s wr, 67 op/s 2023-09-10T01:03:22.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:22 smithi092 ceph-mon[134067]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 132 KiB/s wr, 67 op/s 2023-09-10T01:03:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:22 smithi003 ceph-mon[97654]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 132 KiB/s wr, 67 op/s 2023-09-10T01:03:24.417 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:24 smithi136 ceph-mon[133708]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 102 KiB/s wr, 52 op/s 2023-09-10T01:03:24.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:24 smithi092 ceph-mon[134067]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 102 KiB/s wr, 52 op/s 2023-09-10T01:03:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:24 smithi003 ceph-mon[97654]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 102 KiB/s wr, 52 op/s 2023-09-10T01:03:26.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:26 smithi136 ceph-mon[133708]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 84 KiB/s wr, 46 op/s 2023-09-10T01:03:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:26 smithi092 ceph-mon[134067]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 84 KiB/s wr, 46 op/s 2023-09-10T01:03:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:26 smithi003 ceph-mon[97654]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 84 KiB/s wr, 46 op/s 2023-09-10T01:03:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.380+0000 7fef47dff700 1 -- 172.21.15.3:0/3172226281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400729a0 msgr2=0x7fef40072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.380+0000 7fef47dff700 1 --2- 172.21.15.3:0/3172226281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400729a0 0x7fef40072d80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009990 tx=0x7fef3c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.380+0000 7fef47dff700 1 -- 172.21.15.3:0/3172226281 shutdown_connections 2023-09-10T01:03:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.380+0000 7fef47dff700 1 --2- 172.21.15.3:0/3172226281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef401115b0 0x7fef401139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.380+0000 7fef47dff700 1 --2- 172.21.15.3:0/3172226281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400729a0 0x7fef40072d80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.380+0000 7fef47dff700 1 --2- 172.21.15.3:0/3172226281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef400732c0 0x7fef40111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.380+0000 7fef47dff700 1 -- 172.21.15.3:0/3172226281 >> 172.21.15.3:0/3172226281 conn(0x7fef400fc0b0 msgr2=0x7fef400fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.381+0000 7fef47dff700 1 -- 172.21.15.3:0/3172226281 shutdown_connections 2023-09-10T01:03:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.381+0000 7fef47dff700 1 -- 172.21.15.3:0/3172226281 wait complete. 2023-09-10T01:03:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.381+0000 7fef47dff700 1 Processor -- start 2023-09-10T01:03:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef47dff700 1 -- start start 2023-09-10T01:03:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef47dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef400729a0 0x7fef4019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef47dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400732c0 0x7fef4019d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef47dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef401115b0 0x7fef401a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef47dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef401a0bd0 con 0x7fef400732c0 2023-09-10T01:03:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef47dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef401a0d40 con 0x7fef400729a0 2023-09-10T01:03:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef47dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef401a0f00 con 0x7fef401115b0 2023-09-10T01:03:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef4539a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400732c0 0x7fef4019d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef4539a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400732c0 0x7fef4019d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32944/0 (socket says 172.21.15.3:32944) 2023-09-10T01:03:28.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef4539a700 1 -- 172.21.15.3:0/3857905366 learned_addr learned my addr 172.21.15.3:0/3857905366 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.382+0000 7fef45b9b700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef400729a0 0x7fef4019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef4639c700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef401115b0 0x7fef401a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef45b9b700 1 -- 172.21.15.3:0/3857905366 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef401115b0 msgr2=0x7fef401a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef45b9b700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef401115b0 0x7fef401a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef45b9b700 1 -- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400732c0 msgr2=0x7fef4019d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef45b9b700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400732c0 0x7fef4019d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef45b9b700 1 -- 172.21.15.3:0/3857905366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0095f0 con 0x7fef400729a0 2023-09-10T01:03:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef4539a700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400732c0 0x7fef4019d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef45b9b700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef400729a0 0x7fef4019cce0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009990 tx=0x7fef3c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:28.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef36ffd700 1 -- 172.21.15.3:0/3857905366 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c01d070 con 0x7fef400729a0 2023-09-10T01:03:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef401a1180 con 0x7fef400729a0 2023-09-10T01:03:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef36ffd700 1 -- 172.21.15.3:0/3857905366 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef3c021410 con 0x7fef400729a0 2023-09-10T01:03:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.383+0000 7fef36ffd700 1 -- 172.21.15.3:0/3857905366 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c017650 con 0x7fef400729a0 2023-09-10T01:03:28.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.384+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef401a9b40 con 0x7fef400729a0 2023-09-10T01:03:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.387+0000 7fef36ffd700 1 -- 172.21.15.3:0/3857905366 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7fef3c0217a0 con 0x7fef400729a0 2023-09-10T01:03:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.388+0000 7fef5b629b80 1 Processor -- start 2023-09-10T01:03:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.388+0000 7fef5b629b80 1 -- start start 2023-09-10T01:03:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.388+0000 7fef36ffd700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef2c0a08c0 0x7fef2c0a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.388+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e051580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.388+0000 7fef36ffd700 1 -- 172.21.15.3:0/3857905366 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fef3c021e40 con 0x7fef400729a0 2023-09-10T01:03:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e04b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e057f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef5b629b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5dfa46d0 con 0x55fb5e0511a0 2023-09-10T01:03:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef5b629b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5dfa1ff0 con 0x55fb5e04b300 2023-09-10T01:03:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef5b629b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5df9d8e0 con 0x55fb5e04bcc0 2023-09-10T01:03:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef4639c700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef2c0a08c0 0x7fef2c0a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef45b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e051580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef4539a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e04b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef4539a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e04b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59860/0 (socket says 172.21.15.3:59860) 2023-09-10T01:03:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef4539a700 1 -- 172.21.15.3:0/1910622097 learned_addr learned my addr 172.21.15.3:0/1910622097 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef4639c700 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e057f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef45b9b700 1 -- 172.21.15.3:0/1910622097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 msgr2=0x55fb5e057f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef45b9b700 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e057f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef45b9b700 1 -- 172.21.15.3:0/1910622097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 msgr2=0x55fb5e04b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef45b9b700 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e04b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef45b9b700 1 -- 172.21.15.3:0/1910622097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb5ddc9d80 con 0x55fb5e0511a0 2023-09-10T01:03:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.389+0000 7fef4539a700 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e04b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.390+0000 7fef45b9b700 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e051580 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0cab20 tx=0x7fef3c096c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678537c0180b0fee server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:28.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.390+0000 7fef4639c700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef2c0a08c0 0x7fef2c0a2d80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef40069b60 tx=0x7fef38007a80 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:28.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.390+0000 7fef2bfff700 1 -- 172.21.15.3:0/1910622097 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c09a1c0 con 0x55fb5e0511a0 2023-09-10T01:03:28.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.390+0000 7fef2bfff700 1 -- 172.21.15.3:0/1910622097 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef3c09af60 con 0x55fb5e0511a0 2023-09-10T01:03:28.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.390+0000 7fef2bfff700 1 -- 172.21.15.3:0/1910622097 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0cd690 con 0x55fb5e0511a0 2023-09-10T01:03:28.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 -- 172.21.15.3:0/1910622097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 msgr2=0x55fb5e051580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e051580 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0cab20 tx=0x7fef3c096c20 comp rx=0 tx=0).stop 2023-09-10T01:03:28.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 -- 172.21.15.3:0/1910622097 shutdown_connections 2023-09-10T01:03:28.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e057f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e051580 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1910622097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e04b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 -- 172.21.15.3:0/1910622097 >> 172.21.15.3:0/1910622097 conn(0x55fb5dfc0030 msgr2=0x55fb5e046560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:28.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 -- 172.21.15.3:0/1910622097 shutdown_connections 2023-09-10T01:03:28.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 -- 172.21.15.3:0/1910622097 wait complete. 2023-09-10T01:03:28.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.391+0000 7fef5b629b80 1 -- start start 2023-09-10T01:03:28.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 1 Processor -- start 2023-09-10T01:03:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e05aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e0511a0 0x55fb5e054cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5df3bb10 con 0x55fb5e04b300 2023-09-10T01:03:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5df3b810 con 0x55fb5e0511a0 2023-09-10T01:03:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5de1f8c0 con 0x55fb5e04bcc0 2023-09-10T01:03:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef5b629b80 10 client.0 _ll_register_callbacks cb 0x55fb5de85a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef45b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e05aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef45b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e05aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32964/0 (socket says 172.21.15.3:32964) 2023-09-10T01:03:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef45b9b700 1 -- 172.21.15.3:0/1423788314 learned_addr learned my addr 172.21.15.3:0/1423788314 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:28.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.392+0000 7fef4539a700 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef4639c700 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e0511a0 0x55fb5e054cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef45b9b700 1 -- 172.21.15.3:0/1423788314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 msgr2=0x55fb5e05af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef45b9b700 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef45b9b700 1 -- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e0511a0 msgr2=0x55fb5e054cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef45b9b700 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e0511a0 0x55fb5e054cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef45b9b700 1 -- 172.21.15.3:0/1423788314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef30009e00 con 0x55fb5e04b300 2023-09-10T01:03:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef45b9b700 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e05aa30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fef3c000c00 tx=0x7fef3c0092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3666755a27e5a9b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef297fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef297fa700 1 -- 172.21.15.3:0/1423788314 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c09a860 con 0x55fb5e04b300 2023-09-10T01:03:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef297fa700 1 -- 172.21.15.3:0/1423788314 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fef3c0b0020 con 0x55fb5e04b300 2023-09-10T01:03:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb5def5490 con 0x55fb5e04b300 2023-09-10T01:03:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.393+0000 7fef297fa700 1 -- 172.21.15.3:0/1423788314 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef3c09e690 con 0x55fb5e04b300 2023-09-10T01:03:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 10 client.15714 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fef401a9b40 con 0x55fb5e04b300 2023-09-10T01:03:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef297fa700 1 -- 172.21.15.3:0/1423788314 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fb5def5490 con 0x55fb5e04b300 2023-09-10T01:03:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef297fa700 1 -- 172.21.15.3:0/1423788314 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fef3c099d50 con 0x55fb5e04b300 2023-09-10T01:03:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 10 client.15714 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 20 client.15714 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e064a90 0x55fb5e066e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 4 client.15714 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb5dfb2840 con 0x55fb5e064a90 2023-09-10T01:03:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.394+0000 7fef4539a700 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e064a90 0x55fb5e066e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.395+0000 7fef4539a700 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e064a90 0x55fb5e066e80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f1388681a74603e server_cookie=d56b54239d17e82a in_seq=0 out_seq=0 2023-09-10T01:03:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.395+0000 7fef297fa700 10 client.15714 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef297fa700 1 -- 172.21.15.3:0/1423788314 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fb5dfb2840 con 0x55fb5e064a90 2023-09-10T01:03:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef297fa700 10 client.15714 handle_command_reply: tid=0 2023-09-10T01:03:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 client.15714 shutdown 2023-09-10T01:03:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 msgr2=0x55fb5e05aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e05aa30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fef3c000c00 tx=0x7fef3c0092b0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 shutdown_connections 2023-09-10T01:03:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e064a90 0x55fb5e066e80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e0511a0 0x55fb5e054cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e05aa30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef297fa700 0 client.15714 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 --2- 172.21.15.3:0/1423788314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 >> 172.21.15.3:0/1423788314 conn(0x55fb5dfc0030 msgr2=0x55fb5e0457a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 shutdown_connections 2023-09-10T01:03:28.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 1 -- 172.21.15.3:0/1423788314 wait complete. 2023-09-10T01:03:28.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.396+0000 7fef5b629b80 20 client.15714 trim_cache size 0 max 16384 2023-09-10T01:03:28.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:28 smithi136 ceph-mon[133708]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 42 op/s 2023-09-10T01:03:28.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 Processor -- start 2023-09-10T01:03:28.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 -- start start 2023-09-10T01:03:28.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e054c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e05ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5e047c80 con 0x55fb5e0511a0 2023-09-10T01:03:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5e055cd0 con 0x55fb5e04b300 2023-09-10T01:03:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.412+0000 7fef5b629b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5e042fc0 con 0x55fb5e04bcc0 2023-09-10T01:03:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e054c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e054c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59882/0 (socket says 172.21.15.3:59882) 2023-09-10T01:03:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 -- 172.21.15.3:0/914472827 learned_addr learned my addr 172.21.15.3:0/914472827 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef4539a700 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef4639c700 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e05ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 -- 172.21.15.3:0/914472827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 msgr2=0x55fb5e05a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 -- 172.21.15.3:0/914472827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 msgr2=0x55fb5e05ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e05ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 -- 172.21.15.3:0/914472827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb5ddb2c00 con 0x55fb5e04b300 2023-09-10T01:03:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef4639c700 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e05ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef45b9b700 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e054c70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fef3c09a560 tx=0x7fef3c0ba010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5743194d0f59175d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef34bff700 1 -- 172.21.15.3:0/914472827 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c09aac0 con 0x55fb5e04b300 2023-09-10T01:03:28.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef34bff700 1 -- 172.21.15.3:0/914472827 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef3c0a25f0 con 0x55fb5e04b300 2023-09-10T01:03:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.413+0000 7fef34bff700 1 -- 172.21.15.3:0/914472827 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0a2840 con 0x55fb5e04b300 2023-09-10T01:03:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 -- 172.21.15.3:0/914472827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 msgr2=0x55fb5e054c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e054c70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fef3c09a560 tx=0x7fef3c0ba010 comp rx=0 tx=0).stop 2023-09-10T01:03:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 -- 172.21.15.3:0/914472827 shutdown_connections 2023-09-10T01:03:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e0511a0 0x55fb5e05ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04b300 0x55fb5e054c70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 --2- 172.21.15.3:0/914472827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e04bcc0 0x55fb5e05a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 -- 172.21.15.3:0/914472827 >> 172.21.15.3:0/914472827 conn(0x55fb5dfc0030 msgr2=0x55fb5e054560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 -- 172.21.15.3:0/914472827 shutdown_connections 2023-09-10T01:03:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.414+0000 7fef5b629b80 1 -- 172.21.15.3:0/914472827 wait complete. 2023-09-10T01:03:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 -- start start 2023-09-10T01:03:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 Processor -- start 2023-09-10T01:03:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e0b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04bcc0 0x55fb5e0b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e0511a0 0x55fb5e0ac820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5e0ace20 con 0x55fb5e04b300 2023-09-10T01:03:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5e0acf90 con 0x55fb5e04bcc0 2023-09-10T01:03:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5e0ad150 con 0x55fb5e0511a0 2023-09-10T01:03:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.415+0000 7fef5b629b80 10 client.0 _ll_register_callbacks cb 0x7fef30009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef45b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e0b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef4539a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04bcc0 0x55fb5e0b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef4539a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04bcc0 0x55fb5e0b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59888/0 (socket says 172.21.15.3:59888) 2023-09-10T01:03:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef4539a700 1 -- 172.21.15.3:0/2443292112 learned_addr learned my addr 172.21.15.3:0/2443292112 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef4639c700 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e0511a0 0x55fb5e0ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef45b9b700 1 -- 172.21.15.3:0/2443292112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e0511a0 msgr2=0x55fb5e0ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef45b9b700 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e0511a0 0x55fb5e0ac820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef45b9b700 1 -- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04bcc0 msgr2=0x55fb5e0b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef45b9b700 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04bcc0 0x55fb5e0b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef45b9b700 1 -- 172.21.15.3:0/2443292112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef401a9b40 con 0x55fb5e04b300 2023-09-10T01:03:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef4539a700 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04bcc0 0x55fb5e0b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef45b9b700 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e0b0cc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0a2310 tx=0x7fef3c021b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cd3667be29f97c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef297fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef297fa700 1 -- 172.21.15.3:0/2443292112 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c096ba0 con 0x55fb5e04b300 2023-09-10T01:03:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef297fa700 1 -- 172.21.15.3:0/2443292112 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fef3c0b0020 con 0x55fb5e04b300 2023-09-10T01:03:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.416+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef3c09af60 con 0x55fb5e04b300 2023-09-10T01:03:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef297fa700 1 -- 172.21.15.3:0/2443292112 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef3c0a2840 con 0x55fb5e04b300 2023-09-10T01:03:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 10 client.15720 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55fb5ddf01f0 con 0x55fb5e04b300 2023-09-10T01:03:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef297fa700 1 -- 172.21.15.3:0/2443292112 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef3c09af60 con 0x55fb5e04b300 2023-09-10T01:03:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef297fa700 1 -- 172.21.15.3:0/2443292112 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fef3c096160 con 0x55fb5e04b300 2023-09-10T01:03:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 10 client.15720 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 20 client.15720 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e0b9700 0x55fb5e0bbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 4 client.15720 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:03:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.417+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fb5df968c0 con 0x55fb5e0b9700 2023-09-10T01:03:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.418+0000 7fef4539a700 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e0b9700 0x55fb5e0bbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.418+0000 7fef4539a700 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e0b9700 0x55fb5e0bbaf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bb17597d07a2a97 server_cookie=a91e19d0c9ebe26e in_seq=0 out_seq=0 2023-09-10T01:03:28.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.418+0000 7fef297fa700 10 client.15720 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.431+0000 7fef297fa700 1 -- 172.21.15.3:0/2443292112 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fb5df968c0 con 0x55fb5e0b9700 2023-09-10T01:03:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.431+0000 7fef297fa700 10 client.15720 handle_command_reply: tid=0 2023-09-10T01:03:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.431+0000 7fef5b629b80 1 client.15720 shutdown 2023-09-10T01:03:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 msgr2=0x55fb5e0b0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e0b0cc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0a2310 tx=0x7fef3c021b40 comp rx=0 tx=0).stop 2023-09-10T01:03:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 shutdown_connections 2023-09-10T01:03:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb5e0b9700 0x55fb5e0bbaf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb5e0511a0 0x55fb5e0ac820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb5e04b300 0x55fb5e0b0cc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 --2- 172.21.15.3:0/2443292112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb5e04bcc0 0x55fb5e0b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 >> 172.21.15.3:0/2443292112 conn(0x55fb5dfc0030 msgr2=0x55fb5e044f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef297fa700 0 client.15720 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 shutdown_connections 2023-09-10T01:03:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.432+0000 7fef5b629b80 1 -- 172.21.15.3:0/2443292112 wait complete. 2023-09-10T01:03:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef5b629b80 20 client.15720 trim_cache size 0 max 16384 2023-09-10T01:03:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:03:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:03:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:03:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:03:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef2c0a08c0 msgr2=0x7fef2c0a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef2c0a08c0 0x7fef2c0a2d80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef40069b60 tx=0x7fef38007a80 comp rx=0 tx=0).stop 2023-09-10T01:03:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef400729a0 msgr2=0x7fef4019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef400729a0 0x7fef4019cce0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009990 tx=0x7fef3c009580 comp rx=0 tx=0).stop 2023-09-10T01:03:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 shutdown_connections 2023-09-10T01:03:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef2c0a08c0 0x7fef2c0a2d80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef401115b0 0x7fef401a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef400729a0 0x7fef4019cce0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 --2- 172.21.15.3:0/3857905366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef400732c0 0x7fef4019d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.433+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 >> 172.21.15.3:0/3857905366 conn(0x7fef400fc0b0 msgr2=0x7fef400fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.434+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 shutdown_connections 2023-09-10T01:03:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:28.434+0000 7fef47dff700 1 -- 172.21.15.3:0/3857905366 wait complete. 2023-09-10T01:03:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:28 smithi092 ceph-mon[134067]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 42 op/s 2023-09-10T01:03:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:28 smithi003 ceph-mon[97654]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 42 op/s 2023-09-10T01:03:30.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:30 smithi136 ceph-mon[133708]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 42 op/s 2023-09-10T01:03:30.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:30 smithi092 ceph-mon[134067]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 42 op/s 2023-09-10T01:03:30.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:30 smithi003 ceph-mon[97654]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 42 op/s 2023-09-10T01:03:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:32 smithi136 ceph-mon[133708]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 44 op/s 2023-09-10T01:03:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:32 smithi092 ceph-mon[134067]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 44 op/s 2023-09-10T01:03:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:32 smithi003 ceph-mon[97654]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 44 op/s 2023-09-10T01:03:32.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:03:33.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 -- 172.21.15.3:0/2713472987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec100640 msgr2=0x7f1dec100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 --2- 172.21.15.3:0/2713472987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec100640 0x7f1dec100ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009990 tx=0x7f1de0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 -- 172.21.15.3:0/2713472987 shutdown_connections 2023-09-10T01:03:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 --2- 172.21.15.3:0/2713472987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dec101000 0x7f1dec10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 --2- 172.21.15.3:0/2713472987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dec106650 0x7f1dec106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 --2- 172.21.15.3:0/2713472987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec100640 0x7f1dec100ac0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 -- 172.21.15.3:0/2713472987 >> 172.21.15.3:0/2713472987 conn(0x7f1dec0fc170 msgr2=0x7f1dec0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 -- 172.21.15.3:0/2713472987 shutdown_connections 2023-09-10T01:03:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.168+0000 7f1df09ff700 1 -- 172.21.15.3:0/2713472987 wait complete. 2023-09-10T01:03:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.169+0000 7f1df09ff700 1 Processor -- start 2023-09-10T01:03:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1df09ff700 1 -- start start 2023-09-10T01:03:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1df09ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dec100640 0x7f1dec196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1df09ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec101000 0x7f1dec196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1df09ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dec106650 0x7f1dec199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1df09ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19a5f0 con 0x7f1dec101000 2023-09-10T01:03:33.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1df09ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19a760 con 0x7f1dec100640 2023-09-10T01:03:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1df09ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19a920 con 0x7f1dec106650 2023-09-10T01:03:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1de9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec101000 0x7f1dec196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1de9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec101000 0x7f1dec196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33008/0 (socket says 172.21.15.3:33008) 2023-09-10T01:03:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1de9d9b700 1 -- 172.21.15.3:0/3360407390 learned_addr learned my addr 172.21.15.3:0/3360407390 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1dea59c700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dec100640 0x7f1dec196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.170+0000 7f1dead9d700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dec106650 0x7f1dec199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1de9d9b700 1 -- 172.21.15.3:0/3360407390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dec106650 msgr2=0x7f1dec199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dec106650 0x7f1dec199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1de9d9b700 1 -- 172.21.15.3:0/3360407390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dec100640 msgr2=0x7f1dec196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dec100640 0x7f1dec196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1de9d9b700 1 -- 172.21.15.3:0/3360407390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00095f0 con 0x7f1dec101000 2023-09-10T01:03:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec101000 0x7f1dec196c40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1de0005910 tx=0x7f1de0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1ddb7fe700 1 -- 172.21.15.3:0/3360407390 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de001d070 con 0x7f1dec101000 2023-09-10T01:03:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1ddb7fe700 1 -- 172.21.15.3:0/3360407390 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1de0022470 con 0x7f1dec101000 2023-09-10T01:03:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec19aba0 con 0x7f1dec101000 2023-09-10T01:03:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1ddb7fe700 1 -- 172.21.15.3:0/3360407390 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0017650 con 0x7f1dec101000 2023-09-10T01:03:33.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.171+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a3560 con 0x7f1dec101000 2023-09-10T01:03:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.174+0000 7f1ddb7fe700 1 -- 172.21.15.3:0/3360407390 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7f1de00225e0 con 0x7f1dec101000 2023-09-10T01:03:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1ddb7fe700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dd40a08c0 0x7f1dd40a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1e043f7b80 1 Processor -- start 2023-09-10T01:03:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1ddb7fe700 1 -- 172.21.15.3:0/3360407390 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1de00cd8c0 con 0x7f1dec101000 2023-09-10T01:03:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1e043f7b80 1 -- start start 2023-09-10T01:03:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1dea59c700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dd40a08c0 0x7f1dd40a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a4465eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a44669d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a4466c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a445b4730 con 0x557a4465eb10 2023-09-10T01:03:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.176+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a445b2050 con 0x557a4466a260 2023-09-10T01:03:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a445ad940 con 0x557a4465f430 2023-09-10T01:03:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dea59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a4465eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1de9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a44669d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dead9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a4466c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1de9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a44669d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49642/0 (socket says 172.21.15.3:49642) 2023-09-10T01:03:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dead9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a4466c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59900/0 (socket says 172.21.15.3:59900) 2023-09-10T01:03:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1de9d9b700 1 -- 172.21.15.3:0/280282009 learned_addr learned my addr 172.21.15.3:0/280282009 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dea59c700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dd40a08c0 0x7f1dd40a2d80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009c30 tx=0x7f1ddc011440 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dead9d700 1 -- 172.21.15.3:0/280282009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 msgr2=0x557a44669d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dead9d700 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a44669d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dead9d700 1 -- 172.21.15.3:0/280282009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 msgr2=0x557a4465eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dead9d700 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a4465eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dead9d700 1 -- 172.21.15.3:0/280282009 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a443d9de0 con 0x557a4466a260 2023-09-10T01:03:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.177+0000 7f1dea59c700 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a4465eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.178+0000 7f1dead9d700 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a4466c650 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1de4011580 tx=0x7f1de4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=286f7ebabe14be2f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.178+0000 7f1dd8ff9700 1 -- 172.21.15.3:0/280282009 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400e040 con 0x557a4466a260 2023-09-10T01:03:33.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.178+0000 7f1dd8ff9700 1 -- 172.21.15.3:0/280282009 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1de400f040 con 0x557a4466a260 2023-09-10T01:03:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.178+0000 7f1dd8ff9700 1 -- 172.21.15.3:0/280282009 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4004390 con 0x557a4466a260 2023-09-10T01:03:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.178+0000 7f1e043f7b80 1 -- 172.21.15.3:0/280282009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 msgr2=0x557a4466c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.178+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a4466c650 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1de4011580 tx=0x7f1de4009920 comp rx=0 tx=0).stop 2023-09-10T01:03:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 -- 172.21.15.3:0/280282009 shutdown_connections 2023-09-10T01:03:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a4466c650 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a4465eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/280282009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a44669d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 -- 172.21.15.3:0/280282009 >> 172.21.15.3:0/280282009 conn(0x557a445d0090 msgr2=0x557a44656a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 -- 172.21.15.3:0/280282009 shutdown_connections 2023-09-10T01:03:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 -- 172.21.15.3:0/280282009 wait complete. 2023-09-10T01:03:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 -- start start 2023-09-10T01:03:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.179+0000 7f1e043f7b80 1 Processor -- start 2023-09-10T01:03:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465eb10 0x557a44669c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465f430 0x557a44666e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a44667480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a4454bb70 con 0x557a4465f430 2023-09-10T01:03:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a4454b870 con 0x557a4466a260 2023-09-10T01:03:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a4442f920 con 0x557a4465eb10 2023-09-10T01:03:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1e043f7b80 10 client.0 _ll_register_callbacks cb 0x557a44495a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:33.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1de9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465f430 0x557a44666e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1dead9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a44667480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1dead9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a44667480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59908/0 (socket says 172.21.15.3:59908) 2023-09-10T01:03:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1dead9d700 1 -- 172.21.15.3:0/2442637447 learned_addr learned my addr 172.21.15.3:0/2442637447 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.180+0000 7f1dea59c700 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465eb10 0x557a44669c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dead9d700 1 -- 172.21.15.3:0/2442637447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465eb10 msgr2=0x557a44669c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dead9d700 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465eb10 0x557a44669c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dead9d700 1 -- 172.21.15.3:0/2442637447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465f430 msgr2=0x557a44666e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dead9d700 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465f430 0x557a44666e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dead9d700 1 -- 172.21.15.3:0/2442637447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc011b20 con 0x557a4466a260 2023-09-10T01:03:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dead9d700 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a44667480 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1de400b010 tx=0x7f1de4004dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c463b877cc5e719 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/2442637447 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400e040 con 0x557a4466a260 2023-09-10T01:03:33.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a445054f0 con 0x557a4466a260 2023-09-10T01:03:33.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/2442637447 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1de4003e30 con 0x557a4466a260 2023-09-10T01:03:33.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/2442637447 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1de4004240 con 0x557a4466a260 2023-09-10T01:03:33.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1e043f7b80 10 client.25775 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:33.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.181+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f1de4011230 con 0x557a4466a260 2023-09-10T01:03:33.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/2442637447 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557a445054f0 con 0x557a4466a260 2023-09-10T01:03:33.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/2442637447 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1de401b020 con 0x557a4466a260 2023-09-10T01:03:33.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1e043f7b80 10 client.25775 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:33.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1e043f7b80 10 client.25775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:33.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1e043f7b80 20 client.25775 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:33.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a44679b60 0x557a4467bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1e043f7b80 4 client.25775 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:33.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a445c28a0 con 0x557a44679b60 2023-09-10T01:03:33.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.182+0000 7f1dea59c700 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a44679b60 0x557a4467bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.183+0000 7f1dea59c700 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a44679b60 0x557a4467bf80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f8e673f7e2c161 server_cookie=a386331870f2c1dc in_seq=0 out_seq=0 2023-09-10T01:03:33.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.183+0000 7f1dd1ffb700 10 client.25775 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:33.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.183+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/2442637447 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557a445c28a0 con 0x557a44679b60 2023-09-10T01:03:33.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.183+0000 7f1dd1ffb700 10 client.25775 handle_command_reply: tid=0 2023-09-10T01:03:33.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.183+0000 7f1e043f7b80 1 client.25775 shutdown 2023-09-10T01:03:33.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 msgr2=0x557a44667480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a44667480 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1de400b010 tx=0x7f1de4004dd0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 shutdown_connections 2023-09-10T01:03:33.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a44679b60 0x557a4467bf80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465eb10 0x557a44669c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4466a260 0x557a44667480 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/2442637447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465f430 0x557a44666e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 >> 172.21.15.3:0/2442637447 conn(0x557a445d0090 msgr2=0x557a446566a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1dd1ffb700 0 client.25775 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:33.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 shutdown_connections 2023-09-10T01:03:33.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 1 -- 172.21.15.3:0/2442637447 wait complete. 2023-09-10T01:03:33.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.184+0000 7f1e043f7b80 20 client.25775 trim_cache size 0 max 16384 2023-09-10T01:03:33.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.201+0000 7f1e043f7b80 1 Processor -- start 2023-09-10T01:03:33.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.201+0000 7f1e043f7b80 1 -- start start 2023-09-10T01:03:33.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a44669790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465f430 0x557a44656530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4466a260 0x557a44656a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a44656fb0 con 0x557a4465eb10 2023-09-10T01:03:33.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a44667a60 con 0x557a4465f430 2023-09-10T01:03:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a44657850 con 0x557a4466a260 2023-09-10T01:03:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1dea59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a44669790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1dea59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a44669790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33044/0 (socket says 172.21.15.3:33044) 2023-09-10T01:03:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1dea59c700 1 -- 172.21.15.3:0/1218941550 learned_addr learned my addr 172.21.15.3:0/1218941550 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465f430 0x557a44656530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1dead9d700 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4466a260 0x557a44656a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1de9d9b700 1 -- 172.21.15.3:0/1218941550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4466a260 msgr2=0x557a44656a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4466a260 0x557a44656a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1de9d9b700 1 -- 172.21.15.3:0/1218941550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 msgr2=0x557a44669790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a44669790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1de9d9b700 1 -- 172.21.15.3:0/1218941550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a443c2c60 con 0x557a4465f430 2023-09-10T01:03:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.202+0000 7f1dea59c700 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a44669790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465f430 0x557a44656530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1de000f740 tx=0x7f1de0022f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db3e31b1336ba330 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1dd97fa700 1 -- 172.21.15.3:0/1218941550 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de009a500 con 0x557a4465f430 2023-09-10T01:03:33.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1dd97fa700 1 -- 172.21.15.3:0/1218941550 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1de009a320 con 0x557a4465f430 2023-09-10T01:03:33.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1dd97fa700 1 -- 172.21.15.3:0/1218941550 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de00cd450 con 0x557a4465f430 2023-09-10T01:03:33.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1218941550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465f430 msgr2=0x557a44656530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465f430 0x557a44656530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1de000f740 tx=0x7f1de0022f90 comp rx=0 tx=0).stop 2023-09-10T01:03:33.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1218941550 shutdown_connections 2023-09-10T01:03:33.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4466a260 0x557a44656a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4465eb10 0x557a44669790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1218941550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465f430 0x557a44656530 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1218941550 >> 172.21.15.3:0/1218941550 conn(0x557a445d0090 msgr2=0x557a44662180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.203+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1218941550 shutdown_connections 2023-09-10T01:03:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.204+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1218941550 wait complete. 2023-09-10T01:03:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.204+0000 7f1e043f7b80 1 -- start start 2023-09-10T01:03:33.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.204+0000 7f1e043f7b80 1 Processor -- start 2023-09-10T01:03:33.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465eb10 0x557a44670bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a4466ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1e043f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4466a260 0x557a4466f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a4466f8f0 con 0x557a4466a260 2023-09-10T01:03:33.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a4466fa60 con 0x557a4465eb10 2023-09-10T01:03:33.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1e043f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a4466fc20 con 0x557a4465f430 2023-09-10T01:03:33.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1e043f7b80 10 client.0 _ll_register_callbacks cb 0x7f1ddc0099d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:33.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465eb10 0x557a44670bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465eb10 0x557a44670bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59920/0 (socket says 172.21.15.3:59920) 2023-09-10T01:03:33.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 -- 172.21.15.3:0/1769988902 learned_addr learned my addr 172.21.15.3:0/1769988902 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dead9d700 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4466a260 0x557a4466f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a4466ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 -- 172.21.15.3:0/1769988902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 msgr2=0x557a4466ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a4466ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 -- 172.21.15.3:0/1769988902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4466a260 msgr2=0x557a4466f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4466a260 0x557a4466f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 -- 172.21.15.3:0/1769988902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc028900 con 0x557a4465eb10 2023-09-10T01:03:33.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dea59c700 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465eb10 0x557a44670bf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc028c50 tx=0x7f1ddc0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c520e9abd319a8c4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.205+0000 7f1dd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:33.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/1769988902 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc023040 con 0x557a4465eb10 2023-09-10T01:03:33.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1de400f040 con 0x557a4465eb10 2023-09-10T01:03:33.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/1769988902 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1ddc00fd60 con 0x557a4465eb10 2023-09-10T01:03:33.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/1769988902 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ddc0283a0 con 0x557a4465eb10 2023-09-10T01:03:33.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 10 client.25781 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:33.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f1dec1a3560 con 0x557a4465eb10 2023-09-10T01:03:33.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/1769988902 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1de400f040 con 0x557a4465eb10 2023-09-10T01:03:33.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/1769988902 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1ddc0051b0 con 0x557a4465eb10 2023-09-10T01:03:33.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 10 client.25781 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:33.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 10 client.25781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:33.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 20 client.25781 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:33.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a446ceec0 0x557a446d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 4 client.25781 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:03:33.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.206+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557a445a6920 con 0x557a446ceec0 2023-09-10T01:03:33.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.207+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a446ceec0 0x557a446d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.207+0000 7f1de9d9b700 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a446ceec0 0x557a446d12e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=138b555c72f557f0 server_cookie=cbab6fba19bddb2b in_seq=0 out_seq=0 2023-09-10T01:03:33.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.207+0000 7f1dd1ffb700 10 client.25781 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:33.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.207+0000 7f1dd1ffb700 1 -- 172.21.15.3:0/1769988902 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557a445a6920 con 0x557a446ceec0 2023-09-10T01:03:33.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.207+0000 7f1dd1ffb700 10 client.25781 handle_command_reply: tid=0 2023-09-10T01:03:33.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.207+0000 7f1e043f7b80 1 client.25781 shutdown 2023-09-10T01:03:33.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.207+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465eb10 msgr2=0x557a44670bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465eb10 0x557a44670bf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc028c50 tx=0x7f1ddc0047c0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 shutdown_connections 2023-09-10T01:03:33.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a446ceec0 0x557a446d12e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a4466a260 0x557a4466f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a4465eb10 0x557a44670bf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1dd1ffb700 0 client.25781 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:33.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 --2- 172.21.15.3:0/1769988902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a4465f430 0x557a4466ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 >> 172.21.15.3:0/1769988902 conn(0x557a445d0090 msgr2=0x557a44656470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 shutdown_connections 2023-09-10T01:03:33.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 1 -- 172.21.15.3:0/1769988902 wait complete. 2023-09-10T01:03:33.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1e043f7b80 20 client.25781 trim_cache size 0 max 16384 2023-09-10T01:03:33.226 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:03:33.226 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:03:33.227 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:03:33.227 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:03:33.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dd40a08c0 msgr2=0x7f1dd40a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1df09ff700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dd40a08c0 0x7f1dd40a2d80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009c30 tx=0x7f1ddc011440 comp rx=0 tx=0).stop 2023-09-10T01:03:33.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec101000 msgr2=0x7f1dec196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.208+0000 7f1df09ff700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec101000 0x7f1dec196c40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1de0005910 tx=0x7f1de0005720 comp rx=0 tx=0).stop 2023-09-10T01:03:33.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 shutdown_connections 2023-09-10T01:03:33.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dd40a08c0 0x7f1dd40a2d80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1dec106650 0x7f1dec199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1dec100640 0x7f1dec196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 --2- 172.21.15.3:0/3360407390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1dec101000 0x7f1dec196c40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 >> 172.21.15.3:0/3360407390 conn(0x7f1dec0fc170 msgr2=0x7f1dec0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 shutdown_connections 2023-09-10T01:03:33.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.209+0000 7f1df09ff700 1 -- 172.21.15.3:0/3360407390 wait complete. 2023-09-10T01:03:33.231 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:03:33.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:03:33.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 -- 172.21.15.3:0/1239687490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f648010ce60 msgr2=0x7f648010f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 --2- 172.21.15.3:0/1239687490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f648010ce60 0x7f648010f2d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f647800b0a0 tx=0x7f647800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 -- 172.21.15.3:0/1239687490 shutdown_connections 2023-09-10T01:03:33.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 --2- 172.21.15.3:0/1239687490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6480072a30 0x7f6480072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 --2- 172.21.15.3:0/1239687490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f648010ce60 0x7f648010f2d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 --2- 172.21.15.3:0/1239687490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6480073350 0x7f648010c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 -- 172.21.15.3:0/1239687490 >> 172.21.15.3:0/1239687490 conn(0x7f64800fc110 msgr2=0x7f64800fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 -- 172.21.15.3:0/1239687490 shutdown_connections 2023-09-10T01:03:33.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.436+0000 7f6485fff700 1 -- 172.21.15.3:0/1239687490 wait complete. 2023-09-10T01:03:33.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 Processor -- start 2023-09-10T01:03:33.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 -- start start 2023-09-10T01:03:33.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6480072a30 0x7f648019e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6480073350 0x7f648019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f648010ce60 0x7f6480198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480198e80 con 0x7f6480072a30 2023-09-10T01:03:33.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480198ff0 con 0x7f6480073350 2023-09-10T01:03:33.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.437+0000 7f6485fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64801991b0 con 0x7f648010ce60 2023-09-10T01:03:33.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6480072a30 0x7f648019e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6480072a30 0x7f648019e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33060/0 (socket says 172.21.15.3:33060) 2023-09-10T01:03:33.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 -- 172.21.15.3:0/2224333644 learned_addr learned my addr 172.21.15.3:0/2224333644 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647effd700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6480073350 0x7f648019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647ffff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f648010ce60 0x7f6480198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 -- 172.21.15.3:0/2224333644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f648010ce60 msgr2=0x7f6480198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f648010ce60 0x7f6480198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 -- 172.21.15.3:0/2224333644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6480073350 msgr2=0x7f648019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6480073350 0x7f648019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 -- 172.21.15.3:0/2224333644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6478009d00 con 0x7f6480072a30 2023-09-10T01:03:33.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647ffff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f648010ce60 0x7f6480198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:33.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647f7fe700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6480072a30 0x7f648019e7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f647000e9a0 tx=0x7f647000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.438+0000 7f647cff9700 1 -- 172.21.15.3:0/2224333644 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470009790 con 0x7f6480072a30 2023-09-10T01:03:33.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.439+0000 7f647cff9700 1 -- 172.21.15.3:0/2224333644 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f647000cbd0 con 0x7f6480072a30 2023-09-10T01:03:33.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.439+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6480199490 con 0x7f6480072a30 2023-09-10T01:03:33.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.439+0000 7f647cff9700 1 -- 172.21.15.3:0/2224333644 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470018680 con 0x7f6480072a30 2023-09-10T01:03:33.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.439+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64801a5790 con 0x7f6480072a30 2023-09-10T01:03:33.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.440+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480107780 con 0x7f6480072a30 2023-09-10T01:03:33.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.442+0000 7f647cff9700 1 -- 172.21.15.3:0/2224333644 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7f647000cd40 con 0x7f6480072a30 2023-09-10T01:03:33.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.443+0000 7f647cff9700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f64680a08c0 0x7f64680a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.443+0000 7f647cff9700 1 -- 172.21.15.3:0/2224333644 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6470014070 con 0x7f6480072a30 2023-09-10T01:03:33.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.443+0000 7f647effd700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f64680a08c0 0x7f64680a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.444+0000 7f647effd700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f64680a08c0 0x7f64680a2d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f648010e240 tx=0x7f6474009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.447+0000 7f647cff9700 1 -- 172.21.15.3:0/2224333644 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6470051e10 con 0x7f6480072a30 2023-09-10T01:03:33.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.704+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6480067480 con 0x7f6480072a30 2023-09-10T01:03:33.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.704+0000 7f647cff9700 1 -- 172.21.15.3:0/2224333644 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f6470095a30 con 0x7f6480072a30 2023-09-10T01:03:33.705 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:03:33.706 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:03:33.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f64680a08c0 msgr2=0x7f64680a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f64680a08c0 0x7f64680a2d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f648010e240 tx=0x7f6474009260 comp rx=0 tx=0).stop 2023-09-10T01:03:33.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6480072a30 msgr2=0x7f648019e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6480072a30 0x7f648019e7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f647000e9a0 tx=0x7f647000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 shutdown_connections 2023-09-10T01:03:33.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f64680a08c0 0x7f64680a2d80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f648010ce60 0x7f6480198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6480072a30 0x7f648019e7a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 --2- 172.21.15.3:0/2224333644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6480073350 0x7f648019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 >> 172.21.15.3:0/2224333644 conn(0x7f64800fc110 msgr2=0x7f64801048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 shutdown_connections 2023-09-10T01:03:33.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.707+0000 7f6485fff700 1 -- 172.21.15.3:0/2224333644 wait complete. 2023-09-10T01:03:33.711 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:03:33.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.932+0000 7f84dbbff700 1 -- 172.21.15.3:0/2934196930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4100640 msgr2=0x7f84d4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.932+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2934196930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4100640 0x7f84d4100ac0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009990 tx=0x7f84c4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.932+0000 7f84dbbff700 1 -- 172.21.15.3:0/2934196930 shutdown_connections 2023-09-10T01:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.932+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2934196930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4106650 0x7f84d4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.932+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2934196930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4101000 0x7f84d410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.932+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2934196930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4100640 0x7f84d4100ac0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.932+0000 7f84dbbff700 1 -- 172.21.15.3:0/2934196930 >> 172.21.15.3:0/2934196930 conn(0x7f84d40fc170 msgr2=0x7f84d40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.933+0000 7f84dbbff700 1 -- 172.21.15.3:0/2934196930 shutdown_connections 2023-09-10T01:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.933+0000 7f84dbbff700 1 -- 172.21.15.3:0/2934196930 wait complete. 2023-09-10T01:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 Processor -- start 2023-09-10T01:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 -- start start 2023-09-10T01:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4100640 0x7f84d4072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4101000 0x7f84d4072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4106650 0x7f84d41a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d4073670 con 0x7f84d4106650 2023-09-10T01:03:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d41a4e40 con 0x7f84d4100640 2023-09-10T01:03:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84dbbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d41a4f80 con 0x7f84d4101000 2023-09-10T01:03:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84da19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4106650 0x7f84d41a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84d919a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4101000 0x7f84d4072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.934+0000 7f84d999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4100640 0x7f84d4072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4101000 0x7f84d4072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49716/0 (socket says 172.21.15.3:49716) 2023-09-10T01:03:33.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 -- 172.21.15.3:0/2907335525 learned_addr learned my addr 172.21.15.3:0/2907335525 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4100640 0x7f84d4072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59966/0 (socket says 172.21.15.3:59966) 2023-09-10T01:03:33.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84da19c700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4106650 0x7f84d41a4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33078/0 (socket says 172.21.15.3:33078) 2023-09-10T01:03:33.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 -- 172.21.15.3:0/2907335525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4100640 msgr2=0x7f84d4072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4100640 0x7f84d4072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 -- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4106650 msgr2=0x7f84d41a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4106650 0x7f84d41a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 -- 172.21.15.3:0/2907335525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40095f0 con 0x7f84d4101000 2023-09-10T01:03:33.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d999b700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4100640 0x7f84d4072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:33.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84da19c700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4106650 0x7f84d41a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:33.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84d919a700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4101000 0x7f84d4072e40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84c4005910 tx=0x7f84c4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84caffd700 1 -- 172.21.15.3:0/2907335525 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c401d070 con 0x7f84d4101000 2023-09-10T01:03:33.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d41a5180 con 0x7f84d4101000 2023-09-10T01:03:33.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.935+0000 7f84caffd700 1 -- 172.21.15.3:0/2907335525 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84c4022470 con 0x7f84d4101000 2023-09-10T01:03:33.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.936+0000 7f84caffd700 1 -- 172.21.15.3:0/2907335525 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4017650 con 0x7f84d4101000 2023-09-10T01:03:33.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.936+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d41a5660 con 0x7f84d4101000 2023-09-10T01:03:33.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.939+0000 7f84caffd700 1 -- 172.21.15.3:0/2907335525 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7f84c400fad0 con 0x7f84d4101000 2023-09-10T01:03:33.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.940+0000 7f84ef606b80 1 Processor -- start 2023-09-10T01:03:33.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.940+0000 7f84ef606b80 1 -- start start 2023-09-10T01:03:33.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.940+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f5faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.940+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f69900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.940+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84ef606b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747eb8730 con 0x561747f69e40 2023-09-10T01:03:33.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84ef606b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747eb6050 con 0x561747f60030 2023-09-10T01:03:33.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84ef606b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747eb1940 con 0x561747f5f710 2023-09-10T01:03:33.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84caffd700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84c00a0840 0x7f84c00a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84d919a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f5faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84d999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f69900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f5faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49732/0 (socket says 172.21.15.3:49732) 2023-09-10T01:03:33.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 -- 172.21.15.3:0/2828972802 learned_addr learned my addr 172.21.15.3:0/2828972802 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84caffd700 1 -- 172.21.15.3:0/2907335525 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f84c40cdaf0 con 0x7f84d4101000 2023-09-10T01:03:33.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84c00a0840 0x7f84c00a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 -- 172.21.15.3:0/2828972802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 msgr2=0x561747f69900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f69900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 -- 172.21.15.3:0/2828972802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 msgr2=0x561747f6c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84da19c700 1 -- 172.21.15.3:0/2828972802 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561747cddde0 con 0x561747f5f710 2023-09-10T01:03:33.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.941+0000 7f84d999b700 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f69900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:33.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.942+0000 7f84da19c700 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f5faf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009fd0 tx=0x7f84cc008d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=572b81bff5fa2183 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.942+0000 7f84da19c700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84c00a0840 0x7f84c00a2d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f84cc011000 tx=0x7f84cc01e000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.942+0000 7f84bffff700 1 -- 172.21.15.3:0/2828972802 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc01a930 con 0x561747f5f710 2023-09-10T01:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.942+0000 7f84bffff700 1 -- 172.21.15.3:0/2828972802 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84cc00f040 con 0x561747f5f710 2023-09-10T01:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.942+0000 7f84bffff700 1 -- 172.21.15.3:0/2828972802 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc022b40 con 0x561747f5f710 2023-09-10T01:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 -- 172.21.15.3:0/2828972802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 msgr2=0x561747f5faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f5faf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009fd0 tx=0x7f84cc008d90 comp rx=0 tx=0).stop 2023-09-10T01:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 -- 172.21.15.3:0/2828972802 shutdown_connections 2023-09-10T01:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f5faf0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2828972802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f69900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 -- 172.21.15.3:0/2828972802 >> 172.21.15.3:0/2828972802 conn(0x561747ed4090 msgr2=0x561747f5abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 -- 172.21.15.3:0/2828972802 shutdown_connections 2023-09-10T01:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 -- 172.21.15.3:0/2828972802 wait complete. 2023-09-10T01:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.943+0000 7f84ef606b80 1 -- start start 2023-09-10T01:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 1 Processor -- start 2023-09-10T01:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f60030 0x561747f66b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f69e40 0x561747f67240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747e4fb70 con 0x561747f60030 2023-09-10T01:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747e4f870 con 0x561747f69e40 2023-09-10T01:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747d33920 con 0x561747f5f710 2023-09-10T01:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84ef606b80 10 client.0 _ll_register_callbacks cb 0x561747d99a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84d999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84d999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49740/0 (socket says 172.21.15.3:49740) 2023-09-10T01:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84d999b700 1 -- 172.21.15.3:0/513463695 learned_addr learned my addr 172.21.15.3:0/513463695 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.944+0000 7f84d919a700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f60030 0x561747f66b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84da19c700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f69e40 0x561747f67240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84d999b700 1 -- 172.21.15.3:0/513463695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f69e40 msgr2=0x561747f67240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84d999b700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f69e40 0x561747f67240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84d999b700 1 -- 172.21.15.3:0/513463695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f60030 msgr2=0x561747f66b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84d999b700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f60030 0x561747f66b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84d999b700 1 -- 172.21.15.3:0/513463695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d000bc20 con 0x561747f5f710 2023-09-10T01:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84da19c700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f69e40 0x561747f67240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84d999b700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f698f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84d00119d0 tx=0x7f84d0011ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c6a18dd6165dd82 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84bd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:03:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84bd7fa700 1 -- 172.21.15.3:0/513463695 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0016070 con 0x561747f5f710 2023-09-10T01:03:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561747e094f0 con 0x561747f5f710 2023-09-10T01:03:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.945+0000 7f84bd7fa700 1 -- 172.21.15.3:0/513463695 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f84d0010ab0 con 0x561747f5f710 2023-09-10T01:03:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84bd7fa700 1 -- 172.21.15.3:0/513463695 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84d0010c70 con 0x561747f5f710 2023-09-10T01:03:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 10 client.25732 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:33.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f84cc008690 con 0x561747f5f710 2023-09-10T01:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84bd7fa700 1 -- 172.21.15.3:0/513463695 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561747e094f0 con 0x561747f5f710 2023-09-10T01:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84bd7fa700 1 -- 172.21.15.3:0/513463695 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f84d0003b30 con 0x561747f5f710 2023-09-10T01:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 10 client.25732 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 10 client.25732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 20 client.25732 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747f79a60 0x561747f7be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 4 client.25732 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.946+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561747ec68a0 con 0x561747f79a60 2023-09-10T01:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.947+0000 7f84d919a700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747f79a60 0x561747f7be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.947+0000 7f84bd7fa700 10 client.25732 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:03:33.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.947+0000 7f84d919a700 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747f79a60 0x561747f7be80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e94e81a99d0ef8f2 server_cookie=2b139cd87ceecf91 in_seq=0 out_seq=0 2023-09-10T01:03:33.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84bd7fa700 1 -- 172.21.15.3:0/513463695 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561747ec68a0 con 0x561747f79a60 2023-09-10T01:03:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84bd7fa700 10 client.25732 handle_command_reply: tid=0 2023-09-10T01:03:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 client.25732 shutdown 2023-09-10T01:03:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 msgr2=0x561747f698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f698f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84d00119d0 tx=0x7f84d0011ce0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 shutdown_connections 2023-09-10T01:03:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747f79a60 0x561747f7be80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f698f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f69e40 0x561747f67240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 --2- 172.21.15.3:0/513463695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f60030 0x561747f66b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 >> 172.21.15.3:0/513463695 conn(0x561747ed4090 msgr2=0x561747f5a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84bd7fa700 0 client.25732 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:03:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 shutdown_connections 2023-09-10T01:03:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 1 -- 172.21.15.3:0/513463695 wait complete. 2023-09-10T01:03:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.948+0000 7f84ef606b80 20 client.25732 trim_cache size 0 max 16384 2023-09-10T01:03:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 Processor -- start 2023-09-10T01:03:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 -- start start 2023-09-10T01:03:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f69420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f61e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f62340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747f59a60 con 0x561747f69e40 2023-09-10T01:03:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747f5b1a0 con 0x561747f60030 2023-09-10T01:03:33.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.962+0000 7f84ef606b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747f67840 con 0x561747f5f710 2023-09-10T01:03:33.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84d999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f69420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f62340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84d999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f69420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49752/0 (socket says 172.21.15.3:49752) 2023-09-10T01:03:33.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f62340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33106/0 (socket says 172.21.15.3:33106) 2023-09-10T01:03:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84d919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f61e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84d999b700 1 -- 172.21.15.3:0/2643681600 learned_addr learned my addr 172.21.15.3:0/2643681600 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 -- 172.21.15.3:0/2643681600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 msgr2=0x561747f69420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 --2- 172.21.15.3:0/2643681600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f69420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 -- 172.21.15.3:0/2643681600 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 msgr2=0x561747f61e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 --2- 172.21.15.3:0/2643681600 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f61e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 -- 172.21.15.3:0/2643681600 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561747cc6c60 con 0x561747f69e40 2023-09-10T01:03:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.963+0000 7f84da19c700 1 --2- 172.21.15.3:0/2643681600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f62340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000f80 tx=0x7f84cc008570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1ec2f31eaac9a3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84c8bff700 1 -- 172.21.15.3:0/2643681600 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc004a90 con 0x561747f69e40 2023-09-10T01:03:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84c8bff700 1 -- 172.21.15.3:0/2643681600 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84cc004bf0 con 0x561747f69e40 2023-09-10T01:03:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84c8bff700 1 -- 172.21.15.3:0/2643681600 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc004040 con 0x561747f69e40 2023-09-10T01:03:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 -- 172.21.15.3:0/2643681600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 msgr2=0x561747f62340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2643681600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f62340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000f80 tx=0x7f84cc008570 comp rx=0 tx=0).stop 2023-09-10T01:03:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 -- 172.21.15.3:0/2643681600 shutdown_connections 2023-09-10T01:03:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2643681600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f69420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2643681600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f62340 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 --2- 172.21.15.3:0/2643681600 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f61e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 -- 172.21.15.3:0/2643681600 >> 172.21.15.3:0/2643681600 conn(0x561747ed4090 msgr2=0x561747f59320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 -- 172.21.15.3:0/2643681600 shutdown_connections 2023-09-10T01:03:33.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.964+0000 7f84ef606b80 1 -- 172.21.15.3:0/2643681600 wait complete. 2023-09-10T01:03:33.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 -- start start 2023-09-10T01:03:33.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 Processor -- start 2023-09-10T01:03:33.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f70920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f6eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747f6f6f0 con 0x561747f69e40 2023-09-10T01:03:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747f6f860 con 0x561747f60030 2023-09-10T01:03:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.965+0000 7f84ef606b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561747f6fa20 con 0x561747f5f710 2023-09-10T01:03:33.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84ef606b80 10 client.0 _ll_register_callbacks cb 0x561747d99a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84d919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f6eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33116/0 (socket says 172.21.15.3:33116) 2023-09-10T01:03:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 -- 172.21.15.3:0/1225368008 learned_addr learned my addr 172.21.15.3:0/1225368008 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:33.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84d999b700 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f70920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 -- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 msgr2=0x561747f70920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f70920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 -- 172.21.15.3:0/1225368008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 msgr2=0x561747f6eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f6eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 -- 172.21.15.3:0/1225368008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561747cddde0 con 0x561747f69e40 2023-09-10T01:03:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84d919a700 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f6eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84da19c700 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6f0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f84cc008dd0 tx=0x7f84cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2278320b785b007b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.966+0000 7f84bd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84bd7fa700 1 -- 172.21.15.3:0/1225368008 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc009530 con 0x561747f69e40 2023-09-10T01:03:33.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84cc00f040 con 0x561747f69e40 2023-09-10T01:03:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84bd7fa700 1 -- 172.21.15.3:0/1225368008 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f84cc003c10 con 0x561747f69e40 2023-09-10T01:03:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84bd7fa700 1 -- 172.21.15.3:0/1225368008 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84cc003dd0 con 0x561747f69e40 2023-09-10T01:03:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84ef606b80 10 client.15762 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x561747d04250 con 0x561747f69e40 2023-09-10T01:03:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84bd7fa700 1 -- 172.21.15.3:0/1225368008 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84cc00f040 con 0x561747f69e40 2023-09-10T01:03:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84bd7fa700 1 -- 172.21.15.3:0/1225368008 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f84cc029070 con 0x561747f69e40 2023-09-10T01:03:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84ef606b80 10 client.15762 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84ef606b80 10 client.15762 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:03:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84ef606b80 20 client.15762 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:33.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.967+0000 7f84ef606b80 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747fceb20 0x561747fd0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.968+0000 7f84ef606b80 4 client.15762 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:03:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.968+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561747eaa920 con 0x561747fceb20 2023-09-10T01:03:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.968+0000 7f84d999b700 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747fceb20 0x561747fd0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.968+0000 7f84d999b700 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747fceb20 0x561747fd0f10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f648154813d085f server_cookie=a4d346afc240f37d in_seq=0 out_seq=0 2023-09-10T01:03:33.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.968+0000 7f84bd7fa700 10 client.15762 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:03:33.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84bd7fa700 1 -- 172.21.15.3:0/1225368008 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561747eaa920 con 0x561747fceb20 2023-09-10T01:03:33.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84bd7fa700 10 client.15762 handle_command_reply: tid=0 2023-09-10T01:03:33.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 client.15762 shutdown 2023-09-10T01:03:33.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 msgr2=0x561747f6f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6f0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f84cc008dd0 tx=0x7f84cc004ad0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 shutdown_connections 2023-09-10T01:03:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x561747fceb20 0x561747fd0f10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561747f5f710 0x561747f70920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561747f69e40 0x561747f6f0e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84bd7fa700 0 client.15762 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:03:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 --2- 172.21.15.3:0/1225368008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561747f60030 0x561747f6eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 >> 172.21.15.3:0/1225368008 conn(0x561747ed4090 msgr2=0x561747f61e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 shutdown_connections 2023-09-10T01:03:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 1 -- 172.21.15.3:0/1225368008 wait complete. 2023-09-10T01:03:33.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.969+0000 7f84ef606b80 20 client.15762 trim_cache size 0 max 16384 2023-09-10T01:03:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:03:33.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84c00a0840 msgr2=0x7f84c00a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84c00a0840 0x7f84c00a2d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f84cc011000 tx=0x7f84cc01e000 comp rx=0 tx=0).stop 2023-09-10T01:03:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4101000 msgr2=0x7f84d4072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4101000 0x7f84d4072e40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84c4005910 tx=0x7f84c4004730 comp rx=0 tx=0).stop 2023-09-10T01:03:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 shutdown_connections 2023-09-10T01:03:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84c00a0840 0x7f84c00a2d00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84d4106650 0x7f84d41a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d4100640 0x7f84d4072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 --2- 172.21.15.3:0/2907335525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84d4101000 0x7f84d4072e40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 >> 172.21.15.3:0/2907335525 conn(0x7f84d40fc170 msgr2=0x7f84d40fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 shutdown_connections 2023-09-10T01:03:33.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:33.970+0000 7f84dbbff700 1 -- 172.21.15.3:0/2907335525 wait complete. 2023-09-10T01:03:33.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:03:34.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1556649033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ff420 msgr2=0x7f4b040ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1556649033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ff420 0x7f4b040ff800 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009990 tx=0x7f4af4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1556649033 shutdown_connections 2023-09-10T01:03:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1556649033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b0410cfd0 0x7f4b0410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1556649033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ff420 0x7f4b040ff800 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1556649033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b040ffdd0 0x7f4b0410ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1556649033 >> 172.21.15.3:0/1556649033 conn(0x7f4b04076710 msgr2=0x7f4b04076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.199+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1556649033 shutdown_connections 2023-09-10T01:03:34.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.200+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1556649033 wait complete. 2023-09-10T01:03:34.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.200+0000 7f4b0a1ff700 1 Processor -- start 2023-09-10T01:03:34.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b0a1ff700 1 -- start start 2023-09-10T01:03:34.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b0a1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b040ff420 0x7f4b0419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b0a1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ffdd0 0x7f4b0419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b0a1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b0410cfd0 0x7f4b041988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b0a1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04198f20 con 0x7f4b040ffdd0 2023-09-10T01:03:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b0a1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04199090 con 0x7f4b0410cfd0 2023-09-10T01:03:34.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b0a1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04199250 con 0x7f4b040ff420 2023-09-10T01:03:34.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ffdd0 0x7f4b0419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ffdd0 0x7f4b0419ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33132/0 (socket says 172.21.15.3:33132) 2023-09-10T01:03:34.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1358211909 learned_addr learned my addr 172.21.15.3:0/1358211909 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.201+0000 7f4b037fe700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b040ff420 0x7f4b0419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b0410cfd0 0x7f4b041988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1358211909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b040ff420 msgr2=0x7f4b0419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b040ff420 0x7f4b0419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1358211909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b0410cfd0 msgr2=0x7f4b041988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b0410cfd0 0x7f4b041988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1358211909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af40095f0 con 0x7f4b040ffdd0 2023-09-10T01:03:34.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b0410cfd0 0x7f4b041988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:34.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ffdd0 0x7f4b0419ed80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e9a0 tx=0x7f4af800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b00ff9700 1 -- 172.21.15.3:0/1358211909 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8009790 con 0x7f4b040ffdd0 2023-09-10T01:03:34.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b04199530 con 0x7f4b040ffdd0 2023-09-10T01:03:34.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b00ff9700 1 -- 172.21.15.3:0/1358211909 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4af800cbd0 con 0x7f4b040ffdd0 2023-09-10T01:03:34.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b00ff9700 1 -- 172.21.15.3:0/1358211909 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8018680 con 0x7f4b040ffdd0 2023-09-10T01:03:34.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.202+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b041a5830 con 0x7f4b040ffdd0 2023-09-10T01:03:34.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.205+0000 7f4b00ff9700 1 -- 172.21.15.3:0/1358211909 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7f4af800cd40 con 0x7f4b040ffdd0 2023-09-10T01:03:34.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 Processor -- start 2023-09-10T01:03:34.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 -- start start 2023-09-10T01:03:34.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b00ff9700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4aec0a0840 0x7f4aec0a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b00ff9700 1 -- 172.21.15.3:0/1358211909 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4af8014070 con 0x7f4b040ffdd0 2023-09-10T01:03:34.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b30af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4aec0a0840 0x7f4aec0a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6a89730 con 0x5570d6b3ae40 2023-09-10T01:03:34.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6a87050 con 0x5570d6b31030 2023-09-10T01:03:34.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.207+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6a82940 con 0x5570d6b30710 2023-09-10T01:03:34.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b03fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b03fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33144/0 (socket says 172.21.15.3:33144) 2023-09-10T01:03:34.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b03fff700 1 -- 172.21.15.3:0/2539922050 learned_addr learned my addr 172.21.15.3:0/2539922050 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b037fe700 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b30af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b02ffd700 1 -- 172.21.15.3:0/2539922050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 msgr2=0x5570d6b30af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b30af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b02ffd700 1 -- 172.21.15.3:0/2539922050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 msgr2=0x5570d6b3d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b02ffd700 1 -- 172.21.15.3:0/2539922050 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d68aede0 con 0x5570d6b31030 2023-09-10T01:03:34.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b03fff700 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:34.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4aec0a0840 0x7f4aec0a2d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4afc007770 tx=0x7f4afc007b30 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.208+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3a900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4af80c9aa0 tx=0x7f4af8095ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d259fe3c5b1fdb3f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.209+0000 7f4ae9ffb700 1 -- 172.21.15.3:0/2539922050 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8099e50 con 0x5570d6b31030 2023-09-10T01:03:34.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.209+0000 7f4ae9ffb700 1 -- 172.21.15.3:0/2539922050 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4af8099750 con 0x5570d6b31030 2023-09-10T01:03:34.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.209+0000 7f4ae9ffb700 1 -- 172.21.15.3:0/2539922050 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af80c9f90 con 0x5570d6b31030 2023-09-10T01:03:34.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.209+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/2539922050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 msgr2=0x5570d6b3a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.209+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3a900 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4af80c9aa0 tx=0x7f4af8095ba0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/2539922050 shutdown_connections 2023-09-10T01:03:34.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b30af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/2539922050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3a900 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/2539922050 >> 172.21.15.3:0/2539922050 conn(0x5570d6aa5090 msgr2=0x5570d6b2bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/2539922050 shutdown_connections 2023-09-10T01:03:34.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/2539922050 wait complete. 2023-09-10T01:03:34.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 -- start start 2023-09-10T01:03:34.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.210+0000 7f4b1db3ab80 1 Processor -- start 2023-09-10T01:03:34.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b31030 0x5570d6b37a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b38090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6a20b70 con 0x5570d6b3ae40 2023-09-10T01:03:34.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6a20870 con 0x5570d6b30710 2023-09-10T01:03:34.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6904920 con 0x5570d6b31030 2023-09-10T01:03:34.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b1db3ab80 10 client.0 _ll_register_callbacks cb 0x5570d696aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:34.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b037fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b037fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60040/0 (socket says 172.21.15.3:60040) 2023-09-10T01:03:34.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b037fe700 1 -- 172.21.15.3:0/3497951832 learned_addr learned my addr 172.21.15.3:0/3497951832 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b03fff700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b38090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.211+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b31030 0x5570d6b37a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b037fe700 1 -- 172.21.15.3:0/3497951832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b31030 msgr2=0x5570d6b37a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b037fe700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b31030 0x5570d6b37a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b037fe700 1 -- 172.21.15.3:0/3497951832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 msgr2=0x5570d6b38090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b037fe700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b38090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b037fe700 1 -- 172.21.15.3:0/3497951832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc0174b0 con 0x5570d6b30710 2023-09-10T01:03:34.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b03fff700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b38090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:34.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b037fe700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a850 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4af400bff0 tx=0x7f4af4005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af8f5f831d546623 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4ae2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:34.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/3497951832 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af401d070 con 0x5570d6b30710 2023-09-10T01:03:34.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570d69da4f0 con 0x5570d6b30710 2023-09-10T01:03:34.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.212+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/3497951832 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4af4004500 con 0x5570d6b30710 2023-09-10T01:03:34.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/3497951832 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4af40046c0 con 0x5570d6b30710 2023-09-10T01:03:34.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 10 client.25817 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:34.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f4b041a5830 con 0x5570d6b30710 2023-09-10T01:03:34.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/3497951832 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5570d69da4f0 con 0x5570d6b30710 2023-09-10T01:03:34.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/3497951832 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4af400f880 con 0x5570d6b30710 2023-09-10T01:03:34.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 10 client.25817 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:34.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 10 client.25817 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:03:34.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 20 client.25817 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:34.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b4a9b0 0x5570d6b4cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 4 client.25817 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:34.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.213+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570d6a978a0 con 0x5570d6b4a9b0 2023-09-10T01:03:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.214+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b4a9b0 0x5570d6b4cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.214+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b4a9b0 0x5570d6b4cdd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c66a5d86489df6bf server_cookie=e047470dcdfa2a97 in_seq=0 out_seq=0 2023-09-10T01:03:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.214+0000 7f4ae2ffd700 10 client.25817 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:03:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/3497951832 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5570d6a978a0 con 0x5570d6b4a9b0 2023-09-10T01:03:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4ae2ffd700 10 client.25817 handle_command_reply: tid=0 2023-09-10T01:03:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 client.25817 shutdown 2023-09-10T01:03:34.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 msgr2=0x5570d6b3a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a850 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4af400bff0 tx=0x7f4af4005e80 comp rx=0 tx=0).stop 2023-09-10T01:03:34.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 shutdown_connections 2023-09-10T01:03:34.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b38090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a850 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b4a9b0 0x5570d6b4cdd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/3497951832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b31030 0x5570d6b37a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4ae2ffd700 0 client.25817 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:03:34.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 >> 172.21.15.3:0/3497951832 conn(0x5570d6aa5090 msgr2=0x5570d6b2b780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 shutdown_connections 2023-09-10T01:03:34.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/3497951832 wait complete. 2023-09-10T01:03:34.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.215+0000 7f4b1db3ab80 20 client.25817 trim_cache size 0 max 16384 2023-09-10T01:03:34.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 Processor -- start 2023-09-10T01:03:34.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 -- start start 2023-09-10T01:03:34.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b31030 0x5570d6b32d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b3ae40 0x5570d6b33260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6b3d750 con 0x5570d6b31030 2023-09-10T01:03:34.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6b32280 con 0x5570d6b30710 2023-09-10T01:03:34.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.229+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6b23400 con 0x5570d6b3ae40 2023-09-10T01:03:34.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b31030 0x5570d6b32d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b31030 0x5570d6b32d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33172/0 (socket says 172.21.15.3:33172) 2023-09-10T01:03:34.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b037fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1875479863 learned_addr learned my addr 172.21.15.3:0/1875479863 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b3ae40 0x5570d6b33260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1875479863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b3ae40 msgr2=0x5570d6b33260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b3ae40 0x5570d6b33260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1875479863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 msgr2=0x5570d6b3a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 -- 172.21.15.3:0/1875479863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d6897c60 con 0x5570d6b31030 2023-09-10T01:03:34.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.230+0000 7f4b02ffd700 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b31030 0x5570d6b32d20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c5b0 tx=0x7f4af800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2775572cf387e5c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4aea7fc700 1 -- 172.21.15.3:0/1875479863 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af809d600 con 0x5570d6b31030 2023-09-10T01:03:34.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4aea7fc700 1 -- 172.21.15.3:0/1875479863 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4af80b5070 con 0x5570d6b31030 2023-09-10T01:03:34.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4aea7fc700 1 -- 172.21.15.3:0/1875479863 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af80cc5f0 con 0x5570d6b31030 2023-09-10T01:03:34.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1875479863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b31030 msgr2=0x5570d6b32d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b31030 0x5570d6b32d20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c5b0 tx=0x7f4af800eef0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1875479863 shutdown_connections 2023-09-10T01:03:34.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b3ae40 0x5570d6b33260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b30710 0x5570d6b3a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1875479863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b31030 0x5570d6b32d20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1875479863 >> 172.21.15.3:0/1875479863 conn(0x5570d6aa5090 msgr2=0x5570d6b46140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1875479863 shutdown_connections 2023-09-10T01:03:34.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.231+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1875479863 wait complete. 2023-09-10T01:03:34.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.232+0000 7f4b1db3ab80 1 -- start start 2023-09-10T01:03:34.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.232+0000 7f4b1db3ab80 1 Processor -- start 2023-09-10T01:03:34.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.232+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b3eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.232+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.232+0000 7f4b1db3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6b41fc0 con 0x5570d6b3ae40 2023-09-10T01:03:34.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6b9d7c0 con 0x5570d6b31030 2023-09-10T01:03:34.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b1db3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d6b9d950 con 0x5570d6b30710 2023-09-10T01:03:34.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b1db3ab80 10 client.0 _ll_register_callbacks cb 0x7f4af400bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:34.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33182/0 (socket says 172.21.15.3:33182) 2023-09-10T01:03:34.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 -- 172.21.15.3:0/1328819595 learned_addr learned my addr 172.21.15.3:0/1328819595 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b037fe700 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b3eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 -- 172.21.15.3:0/1328819595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 msgr2=0x5570d6b3eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b3eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 -- 172.21.15.3:0/1328819595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 msgr2=0x5570d6b3f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.233+0000 7f4b03fff700 1 -- 172.21.15.3:0/1328819595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570d68aede0 con 0x5570d6b3ae40 2023-09-10T01:03:34.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4b03fff700 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3f970 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4afc01a720 tx=0x7f4afc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d736c2ff3cf6e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4ae2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:34.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/1328819595 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc00e050 con 0x5570d6b3ae40 2023-09-10T01:03:34.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4af8099750 con 0x5570d6b3ae40 2023-09-10T01:03:34.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/1328819595 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4afc003a00 con 0x5570d6b3ae40 2023-09-10T01:03:34.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/1328819595 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4afc003bc0 con 0x5570d6b3ae40 2023-09-10T01:03:34.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4b1db3ab80 10 client.15780 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:34.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5570d68d5250 con 0x5570d6b3ae40 2023-09-10T01:03:34.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.234+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/1328819595 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4af8099750 con 0x5570d6b3ae40 2023-09-10T01:03:34.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/1328819595 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4afc026430 con 0x5570d6b3ae40 2023-09-10T01:03:34.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4b1db3ab80 10 client.15780 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:34.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4b1db3ab80 10 client.15780 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:03:34.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4b1db3ab80 20 client.15780 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:34.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b9f100 0x5570d6ba1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4b1db3ab80 4 client.15780 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:03:34.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5570d6a7b920 con 0x5570d6b9f100 2023-09-10T01:03:34.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.235+0000 7f4b037fe700 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b9f100 0x5570d6ba1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b037fe700 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b9f100 0x5570d6ba1520 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=adc6c0a3f00a5ad9 server_cookie=18384d2d4fc4f226 in_seq=0 out_seq=0 2023-09-10T01:03:34.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4ae2ffd700 10 client.15780 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:03:34.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4ae2ffd700 1 -- 172.21.15.3:0/1328819595 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5570d6a7b920 con 0x5570d6b9f100 2023-09-10T01:03:34.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4ae2ffd700 10 client.15780 handle_command_reply: tid=0 2023-09-10T01:03:34.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 client.15780 shutdown 2023-09-10T01:03:34.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 msgr2=0x5570d6b3f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3f970 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4afc01a720 tx=0x7f4afc004b40 comp rx=0 tx=0).stop 2023-09-10T01:03:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 shutdown_connections 2023-09-10T01:03:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5570d6b9f100 0x5570d6ba1520 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570d6b30710 0x5570d6b3eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4ae2ffd700 0 client.15780 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:03:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570d6b3ae40 0x5570d6b3f970 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 --2- 172.21.15.3:0/1328819595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570d6b31030 0x5570d6b3f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 >> 172.21.15.3:0/1328819595 conn(0x5570d6aa5090 msgr2=0x5570d6b32d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 shutdown_connections 2023-09-10T01:03:34.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.236+0000 7f4b1db3ab80 1 -- 172.21.15.3:0/1328819595 wait complete. 2023-09-10T01:03:34.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b1db3ab80 20 client.15780 trim_cache size 0 max 16384 2023-09-10T01:03:34.255 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:03:34.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4aec0a0840 msgr2=0x7f4aec0a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4aec0a0840 0x7f4aec0a2d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4afc007770 tx=0x7f4afc007b30 comp rx=0 tx=0).stop 2023-09-10T01:03:34.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ffdd0 msgr2=0x7f4b0419ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ffdd0 0x7f4b0419ed80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e9a0 tx=0x7f4af800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 shutdown_connections 2023-09-10T01:03:34.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4aec0a0840 0x7f4aec0a2d00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b040ff420 0x7f4b0419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b0410cfd0 0x7f4b041988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 --2- 172.21.15.3:0/1358211909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4b040ffdd0 0x7f4b0419ed80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 >> 172.21.15.3:0/1358211909 conn(0x7f4b04076710 msgr2=0x7f4b0410e300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 shutdown_connections 2023-09-10T01:03:34.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.237+0000 7f4b0a1ff700 1 -- 172.21.15.3:0/1358211909 wait complete. 2023-09-10T01:03:34.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:34 smithi003 ceph-mon[97654]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:03:34.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:34 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2224333644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:03:34.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:03:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:34 smithi136 ceph-mon[133708]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:03:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:34 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2224333644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:03:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:34 smithi092 ceph-mon[134067]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-09-10T01:03:34.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:34 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2224333644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:03:34.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 -- 172.21.15.3:0/137786066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 msgr2=0x7f3b6c101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 --2- 172.21.15.3:0/137786066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 0x7f3b6c101c80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009990 tx=0x7f3b5c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 -- 172.21.15.3:0/137786066 shutdown_connections 2023-09-10T01:03:34.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 --2- 172.21.15.3:0/137786066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b6c10ac70 0x7f3b6c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 --2- 172.21.15.3:0/137786066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 0x7f3b6c101c80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 --2- 172.21.15.3:0/137786066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b6c1021c0 0x7f3b6c10a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 -- 172.21.15.3:0/137786066 >> 172.21.15.3:0/137786066 conn(0x7f3b6c076710 msgr2=0x7f3b6c076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 -- 172.21.15.3:0/137786066 shutdown_connections 2023-09-10T01:03:34.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.465+0000 7f3b71fff700 1 -- 172.21.15.3:0/137786066 wait complete. 2023-09-10T01:03:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.467+0000 7f3b71fff700 1 Processor -- start 2023-09-10T01:03:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.467+0000 7f3b71fff700 1 -- start start 2023-09-10T01:03:34.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.467+0000 7f3b71fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 0x7f3b6c0ff4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.467+0000 7f3b71fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b6c1021c0 0x7f3b6c0ffa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.467+0000 7f3b71fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b6c10ac70 0x7f3b6c1a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.467+0000 7f3b71fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c100260 con 0x7f3b6c10ac70 2023-09-10T01:03:34.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.467+0000 7f3b71fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c1a4e80 con 0x7f3b6c1018a0 2023-09-10T01:03:34.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b71fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c1a4fc0 con 0x7f3b6c1021c0 2023-09-10T01:03:34.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 0x7f3b6c0ff4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b6c10ac70 0x7f3b6c1a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 0x7f3b6c0ff4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60060/0 (socket says 172.21.15.3:60060) 2023-09-10T01:03:34.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/2658290323 learned_addr learned my addr 172.21.15.3:0/2658290323 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6affd700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b6c1021c0 0x7f3b6c0ffa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6bfff700 1 -- 172.21.15.3:0/2658290323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b6c1021c0 msgr2=0x7f3b6c0ffa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6bfff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b6c1021c0 0x7f3b6c0ffa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6bfff700 1 -- 172.21.15.3:0/2658290323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 msgr2=0x7f3b6c0ff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6bfff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 0x7f3b6c0ff4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6bfff700 1 -- 172.21.15.3:0/2658290323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c0095f0 con 0x7f3b6c10ac70 2023-09-10T01:03:34.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.468+0000 7f3b6bfff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b6c10ac70 0x7f3b6c1a4940 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400efc0 tx=0x7f3b6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.469+0000 7f3b68ff9700 1 -- 172.21.15.3:0/2658290323 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b64018070 con 0x7f3b6c10ac70 2023-09-10T01:03:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.469+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c1a51c0 con 0x7f3b6c10ac70 2023-09-10T01:03:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.469+0000 7f3b68ff9700 1 -- 172.21.15.3:0/2658290323 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b6400f040 con 0x7f3b6c10ac70 2023-09-10T01:03:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.469+0000 7f3b68ff9700 1 -- 172.21.15.3:0/2658290323 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6400cd40 con 0x7f3b6c10ac70 2023-09-10T01:03:34.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.469+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c1a56f0 con 0x7f3b6c10ac70 2023-09-10T01:03:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.472+0000 7f3b68ff9700 1 -- 172.21.15.3:0/2658290323 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7f3b64003a00 con 0x7f3b6c10ac70 2023-09-10T01:03:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.474+0000 7f3b85908b80 1 Processor -- start 2023-09-10T01:03:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.474+0000 7f3b85908b80 1 -- start start 2023-09-10T01:03:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbdfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbec230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b85908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cb38730 con 0x55dd8cbe0030 2023-09-10T01:03:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b85908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cb36050 con 0x55dd8cbe9e40 2023-09-10T01:03:34.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b85908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cb31940 con 0x55dd8cbdf710 2023-09-10T01:03:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b68ff9700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3b540a0840 0x7f3b540a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbdfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33212/0 (socket says 172.21.15.3:33212) 2023-09-10T01:03:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/3634073826 learned_addr learned my addr 172.21.15.3:0/3634073826 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6affd700 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6bfff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3b540a0840 0x7f3b540a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b68ff9700 1 -- 172.21.15.3:0/2658290323 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3b640cbff0 con 0x7f3b6c10ac70 2023-09-10T01:03:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/3634073826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 msgr2=0x55dd8cbdfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbdfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/3634073826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 msgr2=0x55dd8cbec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.475+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/3634073826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd8c95dde0 con 0x55dd8cbe0030 2023-09-10T01:03:34.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.476+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe9900 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b640 tx=0x7f3b5c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6f9dba21fd25e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.476+0000 7f3b6bfff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3b540a0840 0x7f3b540a2d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b640c9350 tx=0x7f3b6400fdc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.476+0000 7f3b51ffb700 1 -- 172.21.15.3:0/3634073826 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c01d030 con 0x55dd8cbe0030 2023-09-10T01:03:34.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.476+0000 7f3b51ffb700 1 -- 172.21.15.3:0/3634073826 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b5c01e040 con 0x55dd8cbe0030 2023-09-10T01:03:34.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.476+0000 7f3b51ffb700 1 -- 172.21.15.3:0/3634073826 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c004a60 con 0x55dd8cbe0030 2023-09-10T01:03:34.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 -- 172.21.15.3:0/3634073826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe0030 msgr2=0x55dd8cbe9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe9900 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b640 tx=0x7f3b5c00b670 comp rx=0 tx=0).stop 2023-09-10T01:03:34.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 -- 172.21.15.3:0/3634073826 shutdown_connections 2023-09-10T01:03:34.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbdfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 --2- 172.21.15.3:0/3634073826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe9900 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 -- 172.21.15.3:0/3634073826 >> 172.21.15.3:0/3634073826 conn(0x55dd8cb54090 msgr2=0x55dd8cbdabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 -- 172.21.15.3:0/3634073826 shutdown_connections 2023-09-10T01:03:34.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.477+0000 7f3b85908b80 1 -- 172.21.15.3:0/3634073826 wait complete. 2023-09-10T01:03:34.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 -- start start 2023-09-10T01:03:34.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 Processor -- start 2023-09-10T01:03:34.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbe9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cacfb70 con 0x55dd8cbdf710 2023-09-10T01:03:34.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cacf870 con 0x55dd8cbe9e40 2023-09-10T01:03:34.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.478+0000 7f3b85908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c9b3920 con 0x55dd8cbe0030 2023-09-10T01:03:34.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b85908b80 10 client.0 _ll_register_callbacks cb 0x55dd8ca19a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:34.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbe9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbe9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33214/0 (socket says 172.21.15.3:33214) 2023-09-10T01:03:34.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6affd700 1 -- 172.21.15.3:0/231935231 learned_addr learned my addr 172.21.15.3:0/231935231 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6affd700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 msgr2=0x55dd8cbe7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/231935231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 msgr2=0x55dd8cbe9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbe9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/231935231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c10ddb0 con 0x55dd8cbe0030 2023-09-10T01:03:34.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.479+0000 7f3b6affd700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbe9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe6b00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00bb30 tx=0x7f3b5c005700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65524f046878dd5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b4affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b4affd700 1 -- 172.21.15.3:0/231935231 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c01d070 con 0x55dd8cbe0030 2023-09-10T01:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd8ca894f0 con 0x55dd8cbe0030 2023-09-10T01:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b4affd700 1 -- 172.21.15.3:0/231935231 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3b5c004840 con 0x55dd8cbe0030 2023-09-10T01:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b4affd700 1 -- 172.21.15.3:0/231935231 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b5c005bc0 con 0x55dd8cbe0030 2023-09-10T01:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b85908b80 10 client.25768 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.480+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55dd8c946c60 con 0x55dd8cbe0030 2023-09-10T01:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b4affd700 1 -- 172.21.15.3:0/231935231 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dd8ca894f0 con 0x55dd8cbe0030 2023-09-10T01:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b4affd700 1 -- 172.21.15.3:0/231935231 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3b5c00fa40 con 0x55dd8cbe0030 2023-09-10T01:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b85908b80 10 client.25768 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b85908b80 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b85908b80 20 client.25768 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b85908b80 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cbf9420 0x55dd8cbfb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b85908b80 4 client.25768 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.481+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd8cb468a0 con 0x55dd8cbf9420 2023-09-10T01:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.482+0000 7f3b6affd700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cbf9420 0x55dd8cbfb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.482+0000 7f3b6affd700 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cbf9420 0x55dd8cbfb810 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac1f2d4bb5337811 server_cookie=2ed3e2d776548c20 in_seq=0 out_seq=0 2023-09-10T01:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.482+0000 7f3b4affd700 10 client.25768 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b4affd700 1 -- 172.21.15.3:0/231935231 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dd8cb468a0 con 0x55dd8cbf9420 2023-09-10T01:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b4affd700 10 client.25768 handle_command_reply: tid=0 2023-09-10T01:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 client.25768 shutdown 2023-09-10T01:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 msgr2=0x55dd8cbe6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe6b00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00bb30 tx=0x7f3b5c005700 comp rx=0 tx=0).stop 2023-09-10T01:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 shutdown_connections 2023-09-10T01:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cbf9420 0x55dd8cbfb810 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b4affd700 0 client.25768 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbe9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 --2- 172.21.15.3:0/231935231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbe6b00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 >> 172.21.15.3:0/231935231 conn(0x55dd8cb54090 msgr2=0x55dd8cbda7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 shutdown_connections 2023-09-10T01:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.483+0000 7f3b85908b80 1 -- 172.21.15.3:0/231935231 wait complete. 2023-09-10T01:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.484+0000 7f3b85908b80 20 client.25768 trim_cache size 0 max 16384 2023-09-10T01:03:34.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.497+0000 7f3b85908b80 1 Processor -- start 2023-09-10T01:03:34.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.497+0000 7f3b85908b80 1 -- start start 2023-09-10T01:03:34.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.497+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbec230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.497+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbf5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.497+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b85908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cbe5060 con 0x55dd8cbdf710 2023-09-10T01:03:34.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b85908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cbe1890 con 0x55dd8cbe9e40 2023-09-10T01:03:34.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b85908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cbdb6e0 con 0x55dd8cbe0030 2023-09-10T01:03:34.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbec230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33230/0 (socket says 172.21.15.3:33230) 2023-09-10T01:03:34.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 -- 172.21.15.3:0/2941322351 learned_addr learned my addr 172.21.15.3:0/2941322351 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbf5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 -- 172.21.15.3:0/2941322351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 msgr2=0x55dd8cbf5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbf5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 -- 172.21.15.3:0/2941322351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 msgr2=0x55dd8cbe9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 -- 172.21.15.3:0/2941322351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd8c984250 con 0x55dd8cbdf710 2023-09-10T01:03:34.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbf5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:34.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.498+0000 7f3b6affd700 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbec230 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3b600046e0 tx=0x7f3b60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7e22aa6834fc51 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b527fc700 1 -- 172.21.15.3:0/2941322351 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60016040 con 0x55dd8cbdf710 2023-09-10T01:03:34.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b527fc700 1 -- 172.21.15.3:0/2941322351 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b60004cf0 con 0x55dd8cbdf710 2023-09-10T01:03:34.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b527fc700 1 -- 172.21.15.3:0/2941322351 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b600038e0 con 0x55dd8cbdf710 2023-09-10T01:03:34.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 -- 172.21.15.3:0/2941322351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 msgr2=0x55dd8cbec230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbec230 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3b600046e0 tx=0x7f3b60004750 comp rx=0 tx=0).stop 2023-09-10T01:03:34.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 -- 172.21.15.3:0/2941322351 shutdown_connections 2023-09-10T01:03:34.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe9e40 0x55dd8cbe9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbdf710 0x55dd8cbec230 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 --2- 172.21.15.3:0/2941322351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbe0030 0x55dd8cbf5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 -- 172.21.15.3:0/2941322351 >> 172.21.15.3:0/2941322351 conn(0x55dd8cb54090 msgr2=0x55dd8cbd9320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 -- 172.21.15.3:0/2941322351 shutdown_connections 2023-09-10T01:03:34.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.499+0000 7f3b85908b80 1 -- 172.21.15.3:0/2941322351 wait complete. 2023-09-10T01:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.500+0000 7f3b85908b80 1 -- start start 2023-09-10T01:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.500+0000 7f3b85908b80 1 Processor -- start 2023-09-10T01:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbee640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe0030 0x55dd8cbeeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b85908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe9e40 0x55dd8cc4c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b85908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cbef310 con 0x55dd8cbe9e40 2023-09-10T01:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b85908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cc4c7b0 con 0x55dd8cbe0030 2023-09-10T01:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b85908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8cc4c970 con 0x55dd8cbdf710 2023-09-10T01:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b85908b80 10 client.0 _ll_register_callbacks cb 0x7f3b60009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe9e40 0x55dd8cc4c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe9e40 0x55dd8cc4c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33246/0 (socket says 172.21.15.3:33246) 2023-09-10T01:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b6affd700 1 -- 172.21.15.3:0/848576163 learned_addr learned my addr 172.21.15.3:0/848576163 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe0030 0x55dd8cbeeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.501+0000 7f3b6affd700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/848576163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 msgr2=0x55dd8cbee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/848576163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe9e40 msgr2=0x55dd8cc4c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe9e40 0x55dd8cc4c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b6b7fe700 1 -- 172.21.15.3:0/848576163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd8c946c60 con 0x55dd8cbe0030 2023-09-10T01:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b6affd700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe9e40 0x55dd8cc4c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b6b7fe700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe0030 0x55dd8cbeeb80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00fdd0 tx=0x7f3b5c00b2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ae90bcf48695835 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b4affd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b4affd700 1 -- 172.21.15.3:0/848576163 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c043070 con 0x55dd8cbe0030 2023-09-10T01:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b5c01e040 con 0x55dd8cbe0030 2023-09-10T01:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.502+0000 7f3b4affd700 1 -- 172.21.15.3:0/848576163 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3b5c005260 con 0x55dd8cbe0030 2023-09-10T01:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b4affd700 1 -- 172.21.15.3:0/848576163 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b5c01d070 con 0x55dd8cbe0030 2023-09-10T01:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 10 client.25841 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f3b60019040 con 0x55dd8cbe0030 2023-09-10T01:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b4affd700 1 -- 172.21.15.3:0/848576163 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b5c01e040 con 0x55dd8cbe0030 2023-09-10T01:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b4affd700 1 -- 172.21.15.3:0/848576163 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3b5c046030 con 0x55dd8cbe0030 2023-09-10T01:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 10 client.25841 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 10 client.25841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 20 client.25841 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cc4dd50 0x55dd8cc50140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 4 client.25841 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd8cb2a920 con 0x55dd8cc4dd50 2023-09-10T01:03:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.503+0000 7f3b6affd700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cc4dd50 0x55dd8cc50140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b6affd700 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cc4dd50 0x55dd8cc50140 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f4ab517c64402b0 server_cookie=a2a5bd11d3ac97e in_seq=0 out_seq=0 2023-09-10T01:03:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b4affd700 10 client.25841 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b4affd700 1 -- 172.21.15.3:0/848576163 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd8cb2a920 con 0x55dd8cc4dd50 2023-09-10T01:03:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b4affd700 10 client.25841 handle_command_reply: tid=0 2023-09-10T01:03:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 client.25841 shutdown 2023-09-10T01:03:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe0030 msgr2=0x55dd8cbeeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe0030 0x55dd8cbeeb80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00fdd0 tx=0x7f3b5c00b2f0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 shutdown_connections 2023-09-10T01:03:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd8cc4dd50 0x55dd8cc50140 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd8cbdf710 0x55dd8cbee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b4affd700 0 client.25841 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd8cbe9e40 0x55dd8cc4c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 --2- 172.21.15.3:0/848576163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd8cbe0030 0x55dd8cbeeb80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 >> 172.21.15.3:0/848576163 conn(0x55dd8cb54090 msgr2=0x55dd8cbf5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 shutdown_connections 2023-09-10T01:03:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.504+0000 7f3b85908b80 1 -- 172.21.15.3:0/848576163 wait complete. 2023-09-10T01:03:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b85908b80 20 client.25841 trim_cache size 0 max 16384 2023-09-10T01:03:34.520 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:03:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3b540a0840 msgr2=0x7f3b540a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3b540a0840 0x7f3b540a2d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b640c9350 tx=0x7f3b6400fdc0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b6c10ac70 msgr2=0x7f3b6c1a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:34.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b6c10ac70 0x7f3b6c1a4940 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400efc0 tx=0x7f3b6400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 shutdown_connections 2023-09-10T01:03:34.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3b540a0840 0x7f3b540a2d00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b6c10ac70 0x7f3b6c1a4940 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b6c1018a0 0x7f3b6c0ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 --2- 172.21.15.3:0/2658290323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b6c1021c0 0x7f3b6c0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 >> 172.21.15.3:0/2658290323 conn(0x7f3b6c076710 msgr2=0x7f3b6c102600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 shutdown_connections 2023-09-10T01:03:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:34.505+0000 7f3b71fff700 1 -- 172.21.15.3:0/2658290323 wait complete. 2023-09-10T01:03:35.525 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:03:35.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:03:35.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 -- 172.21.15.3:0/2590101957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 msgr2=0x7fa974100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/2590101957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa974100ac0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa968009990 tx=0x7fa968009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 -- 172.21.15.3:0/2590101957 shutdown_connections 2023-09-10T01:03:35.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/2590101957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa974101000 0x7fa97410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/2590101957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa974106650 0x7fa974106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/2590101957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa974100ac0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 -- 172.21.15.3:0/2590101957 >> 172.21.15.3:0/2590101957 conn(0x7fa9740fc170 msgr2=0x7fa9740fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:35.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 -- 172.21.15.3:0/2590101957 shutdown_connections 2023-09-10T01:03:35.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.745+0000 7fa97a7ff700 1 -- 172.21.15.3:0/2590101957 wait complete. 2023-09-10T01:03:35.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.746+0000 7fa97a7ff700 1 Processor -- start 2023-09-10T01:03:35.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa97a7ff700 1 -- start start 2023-09-10T01:03:35.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa97a7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa97419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa97a7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa974101000 0x7fa97419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa97a7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa974106650 0x7fa974198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa97a7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa974198dc0 con 0x7fa974100640 2023-09-10T01:03:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa97a7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa974198f30 con 0x7fa974101000 2023-09-10T01:03:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa97a7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9741990f0 con 0x7fa974106650 2023-09-10T01:03:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa973fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa97419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa973fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa97419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39808/0 (socket says 172.21.15.3:39808) 2023-09-10T01:03:35.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa973fff700 1 -- 172.21.15.3:0/1009650461 learned_addr learned my addr 172.21.15.3:0/1009650461 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa978d9c700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa974106650 0x7fa974198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.747+0000 7fa9737fe700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa974101000 0x7fa97419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa973fff700 1 -- 172.21.15.3:0/1009650461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa974106650 msgr2=0x7fa974198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa973fff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa974106650 0x7fa974198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa973fff700 1 -- 172.21.15.3:0/1009650461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa974101000 msgr2=0x7fa97419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa973fff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa974101000 0x7fa97419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa973fff700 1 -- 172.21.15.3:0/1009650461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680095f0 con 0x7fa974100640 2023-09-10T01:03:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa973fff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa97419e6e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa96400e9a0 tx=0x7fa96400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa9717fa700 1 -- 172.21.15.3:0/1009650461 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964009790 con 0x7fa974100640 2023-09-10T01:03:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9741993d0 con 0x7fa974100640 2023-09-10T01:03:35.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa9717fa700 1 -- 172.21.15.3:0/1009650461 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa964010430 con 0x7fa974100640 2023-09-10T01:03:35.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa9717fa700 1 -- 172.21.15.3:0/1009650461 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa964018680 con 0x7fa974100640 2023-09-10T01:03:35.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.748+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9741a56d0 con 0x7fa974100640 2023-09-10T01:03:35.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.753+0000 7fa9717fa700 1 -- 172.21.15.3:0/1009650461 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7fa9640105a0 con 0x7fa974100640 2023-09-10T01:03:35.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.754+0000 7fa98e0ecb80 1 Processor -- start 2023-09-10T01:03:35.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.754+0000 7fa98e0ecb80 1 -- start start 2023-09-10T01:03:35.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dd4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04dde980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04d2d7b0 con 0x55af04dd50b0 2023-09-10T01:03:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04d2b0d0 con 0x55af04ddeec0 2023-09-10T01:03:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04d269c0 con 0x55af04dd4790 2023-09-10T01:03:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9717fa700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa95c0a08c0 0x7fa95c0a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9717fa700 1 -- 172.21.15.3:0/1009650461 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa964014070 con 0x7fa974100640 2023-09-10T01:03:35.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa973fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04dde980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:58074/0 (socket says 172.21.15.3:58074) 2023-09-10T01:03:35.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa973fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04dde980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39816/0 (socket says 172.21.15.3:39816) 2023-09-10T01:03:35.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 -- 172.21.15.3:0/494284897 learned_addr learned my addr 172.21.15.3:0/494284897 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:35.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa978d9c700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa95c0a08c0 0x7fa95c0a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa978d9c700 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dd4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 -- 172.21.15.3:0/494284897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 msgr2=0x55af04dd4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dd4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 -- 172.21.15.3:0/494284897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 msgr2=0x55af04dde980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04dde980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa9737fe700 1 -- 172.21.15.3:0/494284897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af04b52e60 con 0x55af04ddeec0 2023-09-10T01:03:35.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.755+0000 7fa973fff700 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04dde980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:35.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.756+0000 7fa9737fe700 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de12b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa968015000 tx=0x7fa96800b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fee16ed508ff49f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:35.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.756+0000 7fa978d9c700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa95c0a08c0 0x7fa95c0a2d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa97419f3e0 tx=0x7fa96c00b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.756+0000 7fa95a7fc700 1 -- 172.21.15.3:0/494284897 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96801d070 con 0x55af04ddeec0 2023-09-10T01:03:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.756+0000 7fa95a7fc700 1 -- 172.21.15.3:0/494284897 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa96801e040 con 0x55af04ddeec0 2023-09-10T01:03:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.756+0000 7fa95a7fc700 1 -- 172.21.15.3:0/494284897 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680049c0 con 0x55af04ddeec0 2023-09-10T01:03:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/494284897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 msgr2=0x55af04de12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de12b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa968015000 tx=0x7fa96800b610 comp rx=0 tx=0).stop 2023-09-10T01:03:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/494284897 shutdown_connections 2023-09-10T01:03:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dd4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de12b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/494284897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04dde980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/494284897 >> 172.21.15.3:0/494284897 conn(0x55af04d49110 msgr2=0x55af04dcfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/494284897 shutdown_connections 2023-09-10T01:03:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.757+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/494284897 wait complete. 2023-09-10T01:03:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 -- start start 2023-09-10T01:03:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 Processor -- start 2023-09-10T01:03:35.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dde820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04ddbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04ddc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04cc4bf0 con 0x55af04dd50b0 2023-09-10T01:03:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04cc48f0 con 0x55af04ddeec0 2023-09-10T01:03:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04ba89a0 con 0x55af04dd4790 2023-09-10T01:03:35.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.758+0000 7fa98e0ecb80 10 client.0 _ll_register_callbacks cb 0x55af04c0eb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:35.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04ddbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa973fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dde820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04ddbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39818/0 (socket says 172.21.15.3:39818) 2023-09-10T01:03:35.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 -- 172.21.15.3:0/1592522854 learned_addr learned my addr 172.21.15.3:0/1592522854 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:35.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa978d9c700 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04ddc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 -- 172.21.15.3:0/1592522854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 msgr2=0x55af04dde820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dde820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 -- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 msgr2=0x55af04ddc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04ddc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 -- 172.21.15.3:0/1592522854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9741a56d0 con 0x55af04dd50b0 2023-09-10T01:03:35.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9737fe700 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04ddbaa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa968009990 tx=0x7fa96800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572d3a5457ed79f2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:35.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.759+0000 7fa9537fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:35.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa9537fe700 1 -- 172.21.15.3:0/1592522854 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96801d040 con 0x55af04dd50b0 2023-09-10T01:03:35.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af04c7e570 con 0x55af04dd50b0 2023-09-10T01:03:35.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa9537fe700 1 -- 172.21.15.3:0/1592522854 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa968005980 con 0x55af04dd50b0 2023-09-10T01:03:35.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa9537fe700 1 -- 172.21.15.3:0/1592522854 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa968005bc0 con 0x55af04dd50b0 2023-09-10T01:03:35.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 10 client.15810 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:35.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fa97410de10 con 0x55af04dd50b0 2023-09-10T01:03:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa9537fe700 1 -- 172.21.15.3:0/1592522854 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55af04c7e570 con 0x55af04dd50b0 2023-09-10T01:03:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa9537fe700 1 -- 172.21.15.3:0/1592522854 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa96802b3f0 con 0x55af04dd50b0 2023-09-10T01:03:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 10 client.15810 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 10 client.15810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 20 client.15810 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04dee420 0x55af04df0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 4 client.15810 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.760+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af04d3b920 con 0x55af04dee420 2023-09-10T01:03:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.761+0000 7fa973fff700 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04dee420 0x55af04df0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.761+0000 7fa973fff700 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04dee420 0x55af04df0810 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b231fd551731a0 server_cookie=e0971bb63ee22c0 in_seq=0 out_seq=0 2023-09-10T01:03:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.761+0000 7fa9537fe700 10 client.15810 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa9537fe700 1 -- 172.21.15.3:0/1592522854 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55af04d3b920 con 0x55af04dee420 2023-09-10T01:03:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa9537fe700 10 client.15810 handle_command_reply: tid=0 2023-09-10T01:03:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 client.15810 shutdown 2023-09-10T01:03:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 msgr2=0x55af04ddbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04ddbaa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa968009990 tx=0x7fa96800f690 comp rx=0 tx=0).stop 2023-09-10T01:03:35.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 shutdown_connections 2023-09-10T01:03:35.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04dee420 0x55af04df0810 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd4790 0x55af04dde820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04ddc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa9537fe700 0 client.15810 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:35.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/1592522854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd50b0 0x55af04ddbaa0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 >> 172.21.15.3:0/1592522854 conn(0x55af04d49110 msgr2=0x55af04dcf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:35.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 shutdown_connections 2023-09-10T01:03:35.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.762+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/1592522854 wait complete. 2023-09-10T01:03:35.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.763+0000 7fa98e0ecb80 20 client.15810 trim_cache size 0 max 16384 2023-09-10T01:03:35.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.780+0000 7fa98e0ecb80 1 Processor -- start 2023-09-10T01:03:35.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.780+0000 7fa98e0ecb80 1 -- start start 2023-09-10T01:03:35.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04dd4790 0x55af04de5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04dd3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04ddeec0 0x55af04dd8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04dd5490 con 0x55af04ddeec0 2023-09-10T01:03:35.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04dcc450 con 0x55af04dd4790 2023-09-10T01:03:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04ddd080 con 0x55af04dd50b0 2023-09-10T01:03:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04ddeec0 0x55af04dd8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04ddeec0 0x55af04dd8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39834/0 (socket says 172.21.15.3:39834) 2023-09-10T01:03:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 -- 172.21.15.3:0/4006225294 learned_addr learned my addr 172.21.15.3:0/4006225294 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:35.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa973fff700 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04dd4790 0x55af04de5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa9737fe700 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04dd3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 -- 172.21.15.3:0/4006225294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 msgr2=0x55af04dd3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04dd3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 -- 172.21.15.3:0/4006225294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04dd4790 msgr2=0x55af04de5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04dd4790 0x55af04de5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa978d9c700 1 -- 172.21.15.3:0/4006225294 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af04b3bce0 con 0x55af04ddeec0 2023-09-10T01:03:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa9737fe700 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04dd3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.781+0000 7fa973fff700 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04dd4790 0x55af04de5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.782+0000 7fa978d9c700 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04ddeec0 0x55af04dd8d10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa96c019810 tx=0x7fa96c009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c77f5957d862ff server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:35.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.782+0000 7fa95affd700 1 -- 172.21.15.3:0/4006225294 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c02b070 con 0x55af04ddeec0 2023-09-10T01:03:35.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.782+0000 7fa95affd700 1 -- 172.21.15.3:0/4006225294 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa96c022040 con 0x55af04ddeec0 2023-09-10T01:03:35.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.782+0000 7fa95affd700 1 -- 172.21.15.3:0/4006225294 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c021050 con 0x55af04ddeec0 2023-09-10T01:03:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/4006225294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04ddeec0 msgr2=0x55af04dd8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04ddeec0 0x55af04dd8d10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa96c019810 tx=0x7fa96c009100 comp rx=0 tx=0).stop 2023-09-10T01:03:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/4006225294 shutdown_connections 2023-09-10T01:03:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04ddeec0 0x55af04dd8d10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04dd4790 0x55af04de5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/4006225294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04dd3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/4006225294 >> 172.21.15.3:0/4006225294 conn(0x55af04d49110 msgr2=0x55af04dd3780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/4006225294 shutdown_connections 2023-09-10T01:03:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/4006225294 wait complete. 2023-09-10T01:03:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 -- start start 2023-09-10T01:03:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.783+0000 7fa98e0ecb80 1 Processor -- start 2023-09-10T01:03:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd4790 0x55af04e3cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04e3d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa98e0ecb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04de5690 con 0x55af04dd4790 2023-09-10T01:03:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04de5800 con 0x55af04ddeec0 2023-09-10T01:03:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa98e0ecb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af04de59c0 con 0x55af04dd50b0 2023-09-10T01:03:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa98e0ecb80 10 client.0 _ll_register_callbacks cb 0x7fa9640c9770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa973fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd4790 0x55af04e3cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa973fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd4790 0x55af04e3cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39844/0 (socket says 172.21.15.3:39844) 2023-09-10T01:03:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa973fff700 1 -- 172.21.15.3:0/2225524123 learned_addr learned my addr 172.21.15.3:0/2225524123 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa978d9c700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.784+0000 7fa9737fe700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04e3d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa973fff700 1 -- 172.21.15.3:0/2225524123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 msgr2=0x55af04e3d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa973fff700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04e3d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa973fff700 1 -- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 msgr2=0x55af04de50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa973fff700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa973fff700 1 -- 172.21.15.3:0/2225524123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9640cc040 con 0x55af04dd4790 2023-09-10T01:03:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa978d9c700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa973fff700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd4790 0x55af04e3cd50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa964095a90 tx=0x7fa96401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554fabd16df53d2a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa9537fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa9537fe700 1 -- 172.21.15.3:0/2225524123 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96409d3c0 con 0x55af04dd4790 2023-09-10T01:03:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa9537fe700 1 -- 172.21.15.3:0/2225524123 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa9640a1600 con 0x55af04dd4790 2023-09-10T01:03:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa96801e040 con 0x55af04dd4790 2023-09-10T01:03:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa9537fe700 1 -- 172.21.15.3:0/2225524123 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa964010880 con 0x55af04dd4790 2023-09-10T01:03:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa98e0ecb80 10 client.15816 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fa96c007870 con 0x55af04dd4790 2023-09-10T01:03:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.785+0000 7fa9537fe700 1 -- 172.21.15.3:0/2225524123 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa96801e040 con 0x55af04dd4790 2023-09-10T01:03:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa9537fe700 1 -- 172.21.15.3:0/2225524123 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa9640997f0 con 0x55af04dd4790 2023-09-10T01:03:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa98e0ecb80 10 client.15816 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa98e0ecb80 10 client.15816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa98e0ecb80 20 client.15816 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04e44c60 0x55af04e47080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa98e0ecb80 4 client.15816 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:03:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55af04d1f9a0 con 0x55af04e44c60 2023-09-10T01:03:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.786+0000 7fa9737fe700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04e44c60 0x55af04e47080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.787+0000 7fa9737fe700 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04e44c60 0x55af04e47080 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c620cc708e1c05bd server_cookie=d66af21734cf6db9 in_seq=0 out_seq=0 2023-09-10T01:03:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.787+0000 7fa9537fe700 10 client.15816 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa9537fe700 1 -- 172.21.15.3:0/2225524123 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55af04d1f9a0 con 0x55af04e44c60 2023-09-10T01:03:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa9537fe700 10 client.15816 handle_command_reply: tid=0 2023-09-10T01:03:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 client.15816 shutdown 2023-09-10T01:03:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd4790 msgr2=0x55af04e3cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd4790 0x55af04e3cd50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa964095a90 tx=0x7fa96401a040 comp rx=0 tx=0).stop 2023-09-10T01:03:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 shutdown_connections 2023-09-10T01:03:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af04e44c60 0x55af04e47080 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af04ddeec0 0x55af04de50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af04dd4790 0x55af04e3cd50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 --2- 172.21.15.3:0/2225524123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af04dd50b0 0x55af04e3d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 >> 172.21.15.3:0/2225524123 conn(0x55af04d49110 msgr2=0x55af04dd4010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.788+0000 7fa9537fe700 0 client.15816 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 shutdown_connections 2023-09-10T01:03:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa98e0ecb80 1 -- 172.21.15.3:0/2225524123 wait complete. 2023-09-10T01:03:35.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa98e0ecb80 20 client.15816 trim_cache size 0 max 16384 2023-09-10T01:03:35.803 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:03:35.803 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:03:35.803 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "5a1d4086-0943-424a-938d-b6de8ef2f337", 2023-09-10T01:03:35.804 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:03:35.804 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:03:35.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa95c0a08c0 msgr2=0x7fa95c0a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa95c0a08c0 0x7fa95c0a2d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa97419f3e0 tx=0x7fa96c00b040 comp rx=0 tx=0).stop 2023-09-10T01:03:35.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 msgr2=0x7fa97419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:35.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa97419e6e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa96400e9a0 tx=0x7fa96400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 shutdown_connections 2023-09-10T01:03:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa95c0a08c0 0x7fa95c0a2d80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa974106650 0x7fa974198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa974100640 0x7fa97419e6e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 --2- 172.21.15.3:0/1009650461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa974101000 0x7fa97419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.789+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 >> 172.21.15.3:0/1009650461 conn(0x7fa9740fc170 msgr2=0x7fa9740728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.790+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 shutdown_connections 2023-09-10T01:03:35.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:35.790+0000 7fa97a7ff700 1 -- 172.21.15.3:0/1009650461 wait complete. 2023-09-10T01:03:35.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:03:36.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 -- 172.21.15.3:0/807161766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e5010ac70 msgr2=0x7f3e5010d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 --2- 172.21.15.3:0/807161766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e5010ac70 0x7f3e5010d0f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800b0a0 tx=0x7f3e4800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 -- 172.21.15.3:0/807161766 shutdown_connections 2023-09-10T01:03:36.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 --2- 172.21.15.3:0/807161766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e5010ac70 0x7f3e5010d0f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 --2- 172.21.15.3:0/807161766 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e501018a0 0x7f3e50101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 --2- 172.21.15.3:0/807161766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e501021c0 0x7f3e5010a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 -- 172.21.15.3:0/807161766 >> 172.21.15.3:0/807161766 conn(0x7f3e50076710 msgr2=0x7f3e50076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:36.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 -- 172.21.15.3:0/807161766 shutdown_connections 2023-09-10T01:03:36.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.035+0000 7f3e587ff700 1 -- 172.21.15.3:0/807161766 wait complete. 2023-09-10T01:03:36.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.036+0000 7f3e587ff700 1 Processor -- start 2023-09-10T01:03:36.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e587ff700 1 -- start start 2023-09-10T01:03:36.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e587ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e501018a0 0x7f3e5019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e587ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e501021c0 0x7f3e5019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e587ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e5010ac70 0x7f3e501966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e587ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e50196d30 con 0x7f3e501021c0 2023-09-10T01:03:36.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e587ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e50196ea0 con 0x7f3e501018a0 2023-09-10T01:03:36.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e587ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e50197060 con 0x7f3e5010ac70 2023-09-10T01:03:36.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e55d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e501021c0 0x7f3e5019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e55d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e501021c0 0x7f3e5019cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39866/0 (socket says 172.21.15.3:39866) 2023-09-10T01:03:36.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e55d9a700 1 -- 172.21.15.3:0/1515884648 learned_addr learned my addr 172.21.15.3:0/1515884648 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:36.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e5010ac70 0x7f3e501966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.037+0000 7f3e5659b700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e501018a0 0x7f3e5019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e55d9a700 1 -- 172.21.15.3:0/1515884648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e5010ac70 msgr2=0x7f3e501966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e5010ac70 0x7f3e501966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e55d9a700 1 -- 172.21.15.3:0/1515884648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e501018a0 msgr2=0x7f3e5019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e501018a0 0x7f3e5019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e55d9a700 1 -- 172.21.15.3:0/1515884648 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48009d00 con 0x7f3e501021c0 2023-09-10T01:03:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e5010ac70 0x7f3e501966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e5659b700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e501018a0 0x7f3e5019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e501021c0 0x7f3e5019cb90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e9a0 tx=0x7f3e4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e477fe700 1 -- 172.21.15.3:0/1515884648 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40009790 con 0x7f3e501021c0 2023-09-10T01:03:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e477fe700 1 -- 172.21.15.3:0/1515884648 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e40010430 con 0x7f3e501021c0 2023-09-10T01:03:36.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e50197340 con 0x7f3e501021c0 2023-09-10T01:03:36.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.038+0000 7f3e477fe700 1 -- 172.21.15.3:0/1515884648 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40018680 con 0x7f3e501021c0 2023-09-10T01:03:36.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.039+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e501a3640 con 0x7f3e501021c0 2023-09-10T01:03:36.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.044+0000 7f3e477fe700 1 -- 172.21.15.3:0/1515884648 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7f3e400105a0 con 0x7f3e501021c0 2023-09-10T01:03:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 Processor -- start 2023-09-10T01:03:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 -- start start 2023-09-10T01:03:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c65710 0x55e5a8c65af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c66030 0x55e5a8c73d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c74250 0x55e5a8c76640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8bbe730 con 0x55e5a8c74250 2023-09-10T01:03:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8bbc050 con 0x55e5a8c65710 2023-09-10T01:03:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.045+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8bb7940 con 0x55e5a8c66030 2023-09-10T01:03:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e477fe700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e3c0a0840 0x7f3e3c0a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c74250 0x55e5a8c76640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e56d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c65710 0x55e5a8c65af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c74250 0x55e5a8c76640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39870/0 (socket says 172.21.15.3:39870) 2023-09-10T01:03:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e56d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c65710 0x55e5a8c65af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:58140/0 (socket says 172.21.15.3:58140) 2023-09-10T01:03:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2896832576 learned_addr learned my addr 172.21.15.3:0/2896832576 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e477fe700 1 -- 172.21.15.3:0/1515884648 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3e40014070 con 0x7f3e501021c0 2023-09-10T01:03:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e3c0a0840 0x7f3e3c0a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e5659b700 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c66030 0x55e5a8c73d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2896832576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c66030 msgr2=0x55e5a8c73d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c66030 0x55e5a8c73d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2896832576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c65710 msgr2=0x55e5a8c65af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c65710 0x55e5a8c65af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.046+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2896832576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5a89e3de0 con 0x55e5a8c74250 2023-09-10T01:03:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.047+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c74250 0x55e5a8c76640 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3e40095a90 tx=0x7f3e4006c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93101381bcd0efc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.047+0000 7f3e44ff9700 1 -- 172.21.15.3:0/2896832576 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e400652d0 con 0x55e5a8c74250 2023-09-10T01:03:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.047+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e3c0a0840 0x7f3e3c0a2d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800b780 tx=0x7f3e48006230 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.047+0000 7f3e44ff9700 1 -- 172.21.15.3:0/2896832576 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e40099c90 con 0x55e5a8c74250 2023-09-10T01:03:36.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.047+0000 7f3e44ff9700 1 -- 172.21.15.3:0/2896832576 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e400780c0 con 0x55e5a8c74250 2023-09-10T01:03:36.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2896832576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c74250 msgr2=0x55e5a8c76640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c74250 0x55e5a8c76640 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3e40095a90 tx=0x7f3e4006c6b0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2896832576 shutdown_connections 2023-09-10T01:03:36.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c74250 0x55e5a8c76640 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c65710 0x55e5a8c65af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2896832576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c66030 0x55e5a8c73d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2896832576 >> 172.21.15.3:0/2896832576 conn(0x55e5a8bda090 msgr2=0x55e5a8c60bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:36.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2896832576 shutdown_connections 2023-09-10T01:03:36.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2896832576 wait complete. 2023-09-10T01:03:36.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.048+0000 7f3e6c0ffb80 1 -- start start 2023-09-10T01:03:36.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 1 Processor -- start 2023-09-10T01:03:36.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c65710 0x55e5a8c73c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c66030 0x55e5a8c70ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c74250 0x55e5a8c71540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8b55b70 con 0x55e5a8c65710 2023-09-10T01:03:36.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8b55870 con 0x55e5a8c66030 2023-09-10T01:03:36.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8a39920 con 0x55e5a8c74250 2023-09-10T01:03:36.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e6c0ffb80 10 client.0 _ll_register_callbacks cb 0x55e5a8a9fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:36.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e5659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c65710 0x55e5a8c73c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e56d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c74250 0x55e5a8c71540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e56d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c74250 0x55e5a8c71540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47910/0 (socket says 172.21.15.3:47910) 2023-09-10T01:03:36.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e55d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c66030 0x55e5a8c70ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.049+0000 7f3e56d9c700 1 -- 172.21.15.3:0/3531130614 learned_addr learned my addr 172.21.15.3:0/3531130614 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:36.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e5659b700 1 -- 172.21.15.3:0/3531130614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c74250 msgr2=0x55e5a8c71540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e5659b700 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c74250 0x55e5a8c71540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e5659b700 1 -- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c66030 msgr2=0x55e5a8c70ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e5659b700 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c66030 0x55e5a8c70ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e5659b700 1 -- 172.21.15.3:0/3531130614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e501a3640 con 0x55e5a8c65710 2023-09-10T01:03:36.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c66030 0x55e5a8c70ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:36.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c74250 0x55e5a8c71540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:36.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e5659b700 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c65710 0x55e5a8c73c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c011cc0 tx=0x7f3e4c009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2ff74c916af65c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:36.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:36.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3531130614 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c016070 con 0x55e5a8c65710 2023-09-10T01:03:36.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e5a8b0f4f0 con 0x55e5a8c65710 2023-09-10T01:03:36.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.050+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3531130614 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3e4c00fa10 con 0x55e5a8c65710 2023-09-10T01:03:36.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3531130614 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e4c00fbd0 con 0x55e5a8c65710 2023-09-10T01:03:36.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 10 client.15828 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:36.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e5a89ccc60 con 0x55e5a8c65710 2023-09-10T01:03:36.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3531130614 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e5a8b0f4f0 con 0x55e5a8c65710 2023-09-10T01:03:36.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3531130614 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3e4c005380 con 0x55e5a8c65710 2023-09-10T01:03:36.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 10 client.15828 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:36.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 20 client.15828 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8c83e40 0x55e5a8c86260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 4 client.15828 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.051+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5a8bcc8a0 con 0x55e5a8c83e40 2023-09-10T01:03:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.052+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8c83e40 0x55e5a8c86260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.052+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8c83e40 0x55e5a8c86260 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=374e1f5cf6a263e9 server_cookie=bfb5c42aedef8744 in_seq=0 out_seq=0 2023-09-10T01:03:36.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.052+0000 7f3e39ffb700 10 client.15828 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:36.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3531130614 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e5a8bcc8a0 con 0x55e5a8c83e40 2023-09-10T01:03:36.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e39ffb700 10 client.15828 handle_command_reply: tid=0 2023-09-10T01:03:36.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 client.15828 shutdown 2023-09-10T01:03:36.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c65710 msgr2=0x55e5a8c73c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c65710 0x55e5a8c73c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c011cc0 tx=0x7f3e4c009d40 comp rx=0 tx=0).stop 2023-09-10T01:03:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 shutdown_connections 2023-09-10T01:03:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8c83e40 0x55e5a8c86260 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c74250 0x55e5a8c71540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c65710 0x55e5a8c73c40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3531130614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c66030 0x55e5a8c70ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e39ffb700 0 client.15828 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 >> 172.21.15.3:0/3531130614 conn(0x55e5a8bda090 msgr2=0x55e5a8c607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.053+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 shutdown_connections 2023-09-10T01:03:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.054+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3531130614 wait complete. 2023-09-10T01:03:36.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.054+0000 7f3e6c0ffb80 20 client.15828 trim_cache size 0 max 16384 2023-09-10T01:03:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.076+0000 7f3e6c0ffb80 1 Processor -- start 2023-09-10T01:03:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.076+0000 7f3e6c0ffb80 1 -- start start 2023-09-10T01:03:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c737e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c70790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c70cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8c724d0 con 0x55e5a8c66030 2023-09-10T01:03:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8c81490 con 0x55e5a8c74250 2023-09-10T01:03:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8c67210 con 0x55e5a8c73d20 2023-09-10T01:03:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e5659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c70790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c70790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:58168/0 (socket says 172.21.15.3:58168) 2023-09-10T01:03:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e5659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c737e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39884/0 (socket says 172.21.15.3:39884) 2023-09-10T01:03:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2656333431 learned_addr learned my addr 172.21.15.3:0/2656333431 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c70cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2656333431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 msgr2=0x55e5a8c70cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c70cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2656333431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 msgr2=0x55e5a8c737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c737e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e55d9a700 1 -- 172.21.15.3:0/2656333431 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5a8a0a250 con 0x55e5a8c74250 2023-09-10T01:03:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.077+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c70cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:03:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.078+0000 7f3e5659b700 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.078+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c70790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3e400c9820 tx=0x7f3e400c3970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0e9a7cf546ed27 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.078+0000 7f3e457fa700 1 -- 172.21.15.3:0/2656333431 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e400a1600 con 0x55e5a8c74250 2023-09-10T01:03:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.078+0000 7f3e457fa700 1 -- 172.21.15.3:0/2656333431 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e40099590 con 0x55e5a8c74250 2023-09-10T01:03:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.078+0000 7f3e457fa700 1 -- 172.21.15.3:0/2656333431 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e400cc210 con 0x55e5a8c74250 2023-09-10T01:03:36.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2656333431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 msgr2=0x55e5a8c70790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c70790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3e400c9820 tx=0x7f3e400c3970 comp rx=0 tx=0).stop 2023-09-10T01:03:36.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2656333431 shutdown_connections 2023-09-10T01:03:36.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c70cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/2656333431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c70790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2656333431 >> 172.21.15.3:0/2656333431 conn(0x55e5a8bda090 msgr2=0x55e5a8c80070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2656333431 shutdown_connections 2023-09-10T01:03:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/2656333431 wait complete. 2023-09-10T01:03:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.079+0000 7f3e6c0ffb80 1 -- start start 2023-09-10T01:03:36.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.080+0000 7f3e6c0ffb80 1 Processor -- start 2023-09-10T01:03:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c7af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e6c0ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c79800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8c79ea0 con 0x55e5a8c66030 2023-09-10T01:03:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8c7a010 con 0x55e5a8c74250 2023-09-10T01:03:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e6c0ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a8cd7c80 con 0x55e5a8c73d20 2023-09-10T01:03:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e6c0ffb80 10 client.0 _ll_register_callbacks cb 0x7f3e4c0117e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e5659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c7af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e56d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c79800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e56d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c79800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:58178/0 (socket says 172.21.15.3:58178) 2023-09-10T01:03:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e56d9c700 1 -- 172.21.15.3:0/3412292217 learned_addr learned my addr 172.21.15.3:0/3412292217 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.081+0000 7f3e5659b700 1 -- 172.21.15.3:0/3412292217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 msgr2=0x55e5a8c791d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e5659b700 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c791d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e5659b700 1 -- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 msgr2=0x55e5a8c79800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e5659b700 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c79800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e5659b700 1 -- 172.21.15.3:0/3412292217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5a89ccc60 con 0x55e5a8c66030 2023-09-10T01:03:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e56d9c700 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c79800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e5659b700 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c7af20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c012000 tx=0x7f3e4c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51399c62f5af3788 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3412292217 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c016070 con 0x55e5a8c66030 2023-09-10T01:03:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e40099c90 con 0x55e5a8c66030 2023-09-10T01:03:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.082+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3412292217 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3e4c0052b0 con 0x55e5a8c66030 2023-09-10T01:03:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3412292217 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e4c005470 con 0x55e5a8c66030 2023-09-10T01:03:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 10 client.15834 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e5a89e3de0 con 0x55e5a8c66030 2023-09-10T01:03:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3412292217 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e40099c90 con 0x55e5a8c66030 2023-09-10T01:03:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3412292217 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3e4c011cf0 con 0x55e5a8c66030 2023-09-10T01:03:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 10 client.15834 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 10 client.15834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 20 client.15834 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8cd93c0 0x55e5a8cdb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 4 client.15834 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:03:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e5a8bb0920 con 0x55e5a8cd93c0 2023-09-10T01:03:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.083+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8cd93c0 0x55e5a8cdb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e55d9a700 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8cd93c0 0x55e5a8cdb7e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4f9747676b81b2c server_cookie=cf574bb49afca4d8 in_seq=0 out_seq=0 2023-09-10T01:03:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e39ffb700 10 client.15834 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e39ffb700 1 -- 172.21.15.3:0/3412292217 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e5a8bb0920 con 0x55e5a8cd93c0 2023-09-10T01:03:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e39ffb700 10 client.15834 handle_command_reply: tid=0 2023-09-10T01:03:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 client.15834 shutdown 2023-09-10T01:03:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 msgr2=0x55e5a8c7af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c7af20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c012000 tx=0x7f3e4c005d50 comp rx=0 tx=0).stop 2023-09-10T01:03:36.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 shutdown_connections 2023-09-10T01:03:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e5a8cd93c0 0x55e5a8cdb7e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5a8c74250 0x55e5a8c79800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5a8c66030 0x55e5a8c7af20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 --2- 172.21.15.3:0/3412292217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5a8c73d20 0x55e5a8c791d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 >> 172.21.15.3:0/3412292217 conn(0x55e5a8bda090 msgr2=0x55e5a8c70790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.084+0000 7f3e39ffb700 0 client.15834 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.085+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 shutdown_connections 2023-09-10T01:03:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.085+0000 7f3e6c0ffb80 1 -- 172.21.15.3:0/3412292217 wait complete. 2023-09-10T01:03:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.085+0000 7f3e6c0ffb80 20 client.15834 trim_cache size 0 max 16384 2023-09-10T01:03:36.103 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:03:36.103 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:03:36.103 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:03:36.103 INFO:teuthology.orchestra.run.smithi003.stdout: "5a1d4086-0943-424a-938d-b6de8ef2f337": { 2023-09-10T01:03:36.103 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:03:36.104 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "5a1d4086-0943-424a-938d-b6de8ef2f337", 2023-09-10T01:03:36.104 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:03:36.104 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:03:36.104 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:03:36.104 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:03:36.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.085+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e3c0a0840 msgr2=0x7f3e3c0a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.085+0000 7f3e587ff700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e3c0a0840 0x7f3e3c0a2d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800b780 tx=0x7f3e48006230 comp rx=0 tx=0).stop 2023-09-10T01:03:36.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e501021c0 msgr2=0x7f3e5019cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:36.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e501021c0 0x7f3e5019cb90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e9a0 tx=0x7f3e4000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 shutdown_connections 2023-09-10T01:03:36.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e3c0a0840 0x7f3e3c0a2d00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e5010ac70 0x7f3e501966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e501018a0 0x7f3e5019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 --2- 172.21.15.3:0/1515884648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e501021c0 0x7f3e5019cb90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:36.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 >> 172.21.15.3:0/1515884648 conn(0x7f3e50076710 msgr2=0x7f3e500fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:36.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 shutdown_connections 2023-09-10T01:03:36.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:03:36.086+0000 7f3e587ff700 1 -- 172.21.15.3:0/1515884648 wait complete. 2023-09-10T01:03:36.109 INFO:tasks.cephfs.filesystem:scrub status for tag:5a1d4086-0943-424a-938d-b6de8ef2f337 - {'path': '/', 'tag': '5a1d4086-0943-424a-938d-b6de8ef2f337', 'options': 'recursive,force'} 2023-09-10T01:03:36.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:36 smithi003 ceph-mon[97654]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2023-09-10T01:03:36.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:36 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:03:36.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:36 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:36.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:36 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:03:36.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:36 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:36.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:36 smithi136 ceph-mon[133708]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2023-09-10T01:03:36.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:36 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:03:36.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:36 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:36.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:36 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:03:36.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:36 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:36 smithi092 ceph-mon[134067]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2023-09-10T01:03:36.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:36 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:03:36.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:36 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:36.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:36 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:03:36.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:36 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:03:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:38 smithi136 ceph-mon[133708]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-09-10T01:03:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:38 smithi092 ceph-mon[134067]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-09-10T01:03:38.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:38 smithi003 ceph-mon[97654]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-09-10T01:03:39.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:39 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:03:39.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:39 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:03:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:39 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:03:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:40 smithi136 ceph-mon[133708]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-09-10T01:03:40.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:40 smithi092 ceph-mon[134067]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-09-10T01:03:40.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:40 smithi003 ceph-mon[97654]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-09-10T01:03:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.588+0000 7f1d4ffff700 1 -- 172.21.15.3:0/175202250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d50101550 msgr2=0x7f1d50101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.588+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/175202250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d50101550 0x7f1d50101930 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009990 tx=0x7f1d40009ca0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.588+0000 7f1d4ffff700 1 -- 172.21.15.3:0/175202250 shutdown_connections 2023-09-10T01:03:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.588+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/175202250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d5010a970 0x7f1d5010ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.588+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/175202250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d50101550 0x7f1d50101930 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.588+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/175202250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d50101e70 0x7f1d5010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.588+0000 7f1d4ffff700 1 -- 172.21.15.3:0/175202250 >> 172.21.15.3:0/175202250 conn(0x7f1d500fad90 msgr2=0x7f1d500fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.589+0000 7f1d4ffff700 1 -- 172.21.15.3:0/175202250 shutdown_connections 2023-09-10T01:03:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.589+0000 7f1d4ffff700 1 -- 172.21.15.3:0/175202250 wait complete. 2023-09-10T01:03:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 Processor -- start 2023-09-10T01:03:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 -- start start 2023-09-10T01:03:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d50101550 0x7f1d501986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d50101e70 0x7f1d50198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d5010a970 0x7f1d5019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5019c5c0 con 0x7f1d5010a970 2023-09-10T01:03:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5019c730 con 0x7f1d50101e70 2023-09-10T01:03:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.590+0000 7f1d4ffff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5019c8f0 con 0x7f1d50101550 2023-09-10T01:03:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d5010a970 0x7f1d5019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d5010a970 0x7f1d5019bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39914/0 (socket says 172.21.15.3:39914) 2023-09-10T01:03:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 -- 172.21.15.3:0/2710457014 learned_addr learned my addr 172.21.15.3:0/2710457014 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d50101e70 0x7f1d50198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4effd700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d50101550 0x7f1d501986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 -- 172.21.15.3:0/2710457014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d50101550 msgr2=0x7f1d501986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d50101550 0x7f1d501986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 -- 172.21.15.3:0/2710457014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d50101e70 msgr2=0x7f1d50198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d50101e70 0x7f1d50198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4f7fe700 1 -- 172.21.15.3:0/2710457014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400095f0 con 0x7f1d5010a970 2023-09-10T01:03:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.591+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d50101e70 0x7f1d50198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.592+0000 7f1d4f7fe700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d5010a970 0x7f1d5019bfc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800ef30 tx=0x7f1d4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.592+0000 7f1d37fff700 1 -- 172.21.15.3:0/2710457014 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48018070 con 0x7f1d5010a970 2023-09-10T01:03:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.592+0000 7f1d37fff700 1 -- 172.21.15.3:0/2710457014 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d4800f040 con 0x7f1d5010a970 2023-09-10T01:03:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.592+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5019cb70 con 0x7f1d5010a970 2023-09-10T01:03:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.592+0000 7f1d37fff700 1 -- 172.21.15.3:0/2710457014 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4800cd00 con 0x7f1d5010a970 2023-09-10T01:03:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.592+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d501a5530 con 0x7f1d5010a970 2023-09-10T01:03:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.596+0000 7f1d37fff700 1 -- 172.21.15.3:0/2710457014 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7f1d48003a00 con 0x7f1d5010a970 2023-09-10T01:03:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.597+0000 7f1d691bcb80 1 Processor -- start 2023-09-10T01:03:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.597+0000 7f1d691bcb80 1 -- start start 2023-09-10T01:03:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.597+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e33670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.597+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e2dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8d866d0 con 0x5620e8e33bb0 2023-09-10T01:03:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8d83ff0 con 0x5620e8e33290 2023-09-10T01:03:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8d7f8e0 con 0x5620e8e2d9a0 2023-09-10T01:03:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39926/0 (socket says 172.21.15.3:39926) 2023-09-10T01:03:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 -- 172.21.15.3:0/3101917066 learned_addr learned my addr 172.21.15.3:0/3101917066 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e33670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d37fff700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1d380a0840 0x7f1d380a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e2dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 -- 172.21.15.3:0/3101917066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 msgr2=0x5620e8e2dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e2dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 -- 172.21.15.3:0/3101917066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 msgr2=0x5620e8e33670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e33670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4effd700 1 -- 172.21.15.3:0/3101917066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620e8babd80 con 0x5620e8e33bb0 2023-09-10T01:03:41.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e2dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:03:41.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.598+0000 7f1d37fff700 1 -- 172.21.15.3:0/2710457014 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1d480cbdc0 con 0x7f1d5010a970 2023-09-10T01:03:41.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.599+0000 7f1d4f7fe700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1d380a0840 0x7f1d380a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.599+0000 7f1d4effd700 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2d460 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009990 tx=0x7f1d40005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c44b4293427854 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:41.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.599+0000 7f1d357fa700 1 -- 172.21.15.3:0/3101917066 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4001d070 con 0x5620e8e33bb0 2023-09-10T01:03:41.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.599+0000 7f1d357fa700 1 -- 172.21.15.3:0/3101917066 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d4001e040 con 0x5620e8e33bb0 2023-09-10T01:03:41.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.599+0000 7f1d4f7fe700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1d380a0840 0x7f1d380a2d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d480c90c0 tx=0x7f1d48003f50 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:41.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.599+0000 7f1d357fa700 1 -- 172.21.15.3:0/3101917066 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40004a60 con 0x5620e8e33bb0 2023-09-10T01:03:41.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 -- 172.21.15.3:0/3101917066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 msgr2=0x5620e8e2d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2d460 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009990 tx=0x7f1d40005f00 comp rx=0 tx=0).stop 2023-09-10T01:03:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 -- 172.21.15.3:0/3101917066 shutdown_connections 2023-09-10T01:03:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e2dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e33670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/3101917066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2d460 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 -- 172.21.15.3:0/3101917066 >> 172.21.15.3:0/3101917066 conn(0x5620e8da2030 msgr2=0x5620e8e286c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 -- 172.21.15.3:0/3101917066 shutdown_connections 2023-09-10T01:03:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.600+0000 7f1d691bcb80 1 -- 172.21.15.3:0/3101917066 wait complete. 2023-09-10T01:03:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.601+0000 7f1d691bcb80 1 -- start start 2023-09-10T01:03:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.601+0000 7f1d691bcb80 1 Processor -- start 2023-09-10T01:03:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.601+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e2d9a0 0x5620e8e39b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33290 0x5620e8e36df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e37530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8d1db10 con 0x5620e8e33bb0 2023-09-10T01:03:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8d1d810 con 0x5620e8e2d9a0 2023-09-10T01:03:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8c018c0 con 0x5620e8e33290 2023-09-10T01:03:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d691bcb80 10 client.0 _ll_register_callbacks cb 0x5620e8c67a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e37530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33290 0x5620e8e36df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e37530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39942/0 (socket says 172.21.15.3:39942) 2023-09-10T01:03:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1215457508 learned_addr learned my addr 172.21.15.3:0/1215457508 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e2d9a0 0x5620e8e39b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1215457508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33290 msgr2=0x5620e8e36df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33290 0x5620e8e36df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e2d9a0 msgr2=0x5620e8e39b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e2d9a0 0x5620e8e39b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1215457508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5010daf0 con 0x5620e8e33bb0 2023-09-10T01:03:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.602+0000 7f1d4effd700 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33290 0x5620e8e36df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e37530 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400f3c0 tx=0x7f1d440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451d979f1f257e74 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d32ff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1215457508 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4400fcb0 con 0x5620e8e33bb0 2023-09-10T01:03:41.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5620e8cd7490 con 0x5620e8e33bb0 2023-09-10T01:03:41.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1215457508 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1d4400fe10 con 0x5620e8e33bb0 2023-09-10T01:03:41.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1215457508 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d44004020 con 0x5620e8e33bb0 2023-09-10T01:03:41.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d691bcb80 10 client.15846 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:41.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.603+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5620e8b94c00 con 0x5620e8e33bb0 2023-09-10T01:03:41.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1215457508 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5620e8cd7490 con 0x5620e8e33bb0 2023-09-10T01:03:41.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1215457508 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1d44003c10 con 0x5620e8e33bb0 2023-09-10T01:03:41.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d691bcb80 10 client.15846 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:41.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d691bcb80 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d691bcb80 20 client.15846 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e47550 0x5620e8e49970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d691bcb80 4 client.15846 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620e8d94840 con 0x5620e8e47550 2023-09-10T01:03:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.604+0000 7f1d4effd700 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e47550 0x5620e8e49970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.605+0000 7f1d4effd700 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e47550 0x5620e8e49970 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c721e5c7e8652090 server_cookie=246e77998f22c78a in_seq=0 out_seq=0 2023-09-10T01:03:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.605+0000 7f1d32ff5700 10 client.15846 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1215457508 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5620e8d94840 con 0x5620e8e47550 2023-09-10T01:03:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d32ff5700 10 client.15846 handle_command_reply: tid=0 2023-09-10T01:03:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 client.15846 shutdown 2023-09-10T01:03:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 msgr2=0x5620e8e37530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e37530 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400f3c0 tx=0x7f1d440047c0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 shutdown_connections 2023-09-10T01:03:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e47550 0x5620e8e49970 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e37530 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e2d9a0 0x5620e8e39b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1215457508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33290 0x5620e8e36df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 >> 172.21.15.3:0/1215457508 conn(0x5620e8da2030 msgr2=0x5620e8e27900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d32ff5700 0 client.15846 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 shutdown_connections 2023-09-10T01:03:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.606+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1215457508 wait complete. 2023-09-10T01:03:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.607+0000 7f1d691bcb80 20 client.15846 trim_cache size 0 max 16384 2023-09-10T01:03:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.622+0000 7f1d691bcb80 1 Processor -- start 2023-09-10T01:03:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.622+0000 7f1d691bcb80 1 -- start start 2023-09-10T01:03:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e2d9a0 0x5620e8e3e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e30f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33bb0 0x5620e8e314a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8e3a110 con 0x5620e8e2d9a0 2023-09-10T01:03:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8e2e720 con 0x5620e8e33290 2023-09-10T01:03:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8e44bb0 con 0x5620e8e33bb0 2023-09-10T01:03:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d4e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e2d9a0 0x5620e8e3e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d4effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e30f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d4effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e30f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:58232/0 (socket says 172.21.15.3:58232) 2023-09-10T01:03:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d4effd700 1 -- 172.21.15.3:0/2986417373 learned_addr learned my addr 172.21.15.3:0/2986417373 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33bb0 0x5620e8e314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d4effd700 1 -- 172.21.15.3:0/2986417373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33bb0 msgr2=0x5620e8e314a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.623+0000 7f1d4effd700 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33bb0 0x5620e8e314a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.624+0000 7f1d4effd700 1 -- 172.21.15.3:0/2986417373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e2d9a0 msgr2=0x5620e8e3e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.624+0000 7f1d4effd700 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e2d9a0 0x5620e8e3e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.624+0000 7f1d4effd700 1 -- 172.21.15.3:0/2986417373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620e8e4e560 con 0x5620e8e33290 2023-09-10T01:03:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.624+0000 7f1d4effd700 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e30f60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009990 tx=0x7f1d400045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa464ef5d2be87f7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.624+0000 7f1d35ffb700 1 -- 172.21.15.3:0/2986417373 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4001d040 con 0x5620e8e33290 2023-09-10T01:03:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.624+0000 7f1d35ffb700 1 -- 172.21.15.3:0/2986417373 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d40005510 con 0x5620e8e33290 2023-09-10T01:03:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.624+0000 7f1d35ffb700 1 -- 172.21.15.3:0/2986417373 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40034e10 con 0x5620e8e33290 2023-09-10T01:03:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 -- 172.21.15.3:0/2986417373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 msgr2=0x5620e8e30f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e30f60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009990 tx=0x7f1d400045a0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 -- 172.21.15.3:0/2986417373 shutdown_connections 2023-09-10T01:03:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e33bb0 0x5620e8e314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e2d9a0 0x5620e8e3e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/2986417373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e30f60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 -- 172.21.15.3:0/2986417373 >> 172.21.15.3:0/2986417373 conn(0x5620e8da2030 msgr2=0x5620e8e42d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 -- 172.21.15.3:0/2986417373 shutdown_connections 2023-09-10T01:03:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 -- 172.21.15.3:0/2986417373 wait complete. 2023-09-10T01:03:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.625+0000 7f1d691bcb80 1 -- start start 2023-09-10T01:03:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 1 Processor -- start 2023-09-10T01:03:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e36bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e2ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8e2f770 con 0x5620e8e33bb0 2023-09-10T01:03:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8e2f8e0 con 0x5620e8e33290 2023-09-10T01:03:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620e8e2faa0 con 0x5620e8e2d9a0 2023-09-10T01:03:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d691bcb80 10 client.0 _ll_register_callbacks cb 0x7f1d44010ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.626+0000 7f1d4e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39956/0 (socket says 172.21.15.3:39956) 2023-09-10T01:03:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1500259349 learned_addr learned my addr 172.21.15.3:0/1500259349 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e36bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4effd700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e2ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1500259349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 msgr2=0x5620e8e36bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e36bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 msgr2=0x5620e8e2ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e2ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 -- 172.21.15.3:0/1500259349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620e8babd80 con 0x5620e8e33bb0 2023-09-10T01:03:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4effd700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e2ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d4e7fc700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2f230 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1d44010df0 tx=0x7f1d44000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3f6bdf4a8a52bf server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d32ff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:03:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1500259349 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4400fee0 con 0x5620e8e33bb0 2023-09-10T01:03:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d4001e040 con 0x5620e8e33bb0 2023-09-10T01:03:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.627+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1500259349 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1d44016070 con 0x5620e8e33bb0 2023-09-10T01:03:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1500259349 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d44020b50 con 0x5620e8e33bb0 2023-09-10T01:03:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 10 client.15852 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5620e8bd21f0 con 0x5620e8e33bb0 2023-09-10T01:03:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1500259349 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d4001e040 con 0x5620e8e33bb0 2023-09-10T01:03:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1500259349 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1d44003680 con 0x5620e8e33bb0 2023-09-10T01:03:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 10 client.15852 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 20 client.15852 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e9b710 0x5620e8e9db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 4 client.15852 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:03:41.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.628+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5620e8d788c0 con 0x5620e8e9b710 2023-09-10T01:03:41.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.629+0000 7f1d4effd700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e9b710 0x5620e8e9db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:41.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.629+0000 7f1d4effd700 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e9b710 0x5620e8e9db30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38e1a7d46a1cfe92 server_cookie=3993840221322e95 in_seq=0 out_seq=0 2023-09-10T01:03:41.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.629+0000 7f1d32ff5700 10 client.15852 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:41.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.636+0000 7f1d32ff5700 1 -- 172.21.15.3:0/1500259349 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5620e8d788c0 con 0x5620e8e9b710 2023-09-10T01:03:41.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.636+0000 7f1d32ff5700 10 client.15852 handle_command_reply: tid=0 2023-09-10T01:03:41.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.636+0000 7f1d691bcb80 1 client.15852 shutdown 2023-09-10T01:03:41.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.636+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 msgr2=0x5620e8e2f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.636+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2f230 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1d44010df0 tx=0x7f1d44000bc0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 shutdown_connections 2023-09-10T01:03:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5620e8e9b710 0x5620e8e9db30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5620e8e2d9a0 0x5620e8e36bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5620e8e33bb0 0x5620e8e2f230 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 --2- 172.21.15.3:0/1500259349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5620e8e33290 0x5620e8e2ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 >> 172.21.15.3:0/1500259349 conn(0x5620e8da2030 msgr2=0x5620e8e30f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d32ff5700 0 client.15852 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 shutdown_connections 2023-09-10T01:03:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 1 -- 172.21.15.3:0/1500259349 wait complete. 2023-09-10T01:03:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.637+0000 7f1d691bcb80 20 client.15852 trim_cache size 0 max 16384 2023-09-10T01:03:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:03:41.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:03:41.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:03:41.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:03:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1d380a0840 msgr2=0x7f1d380a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1d380a0840 0x7f1d380a2d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d480c90c0 tx=0x7f1d48003f50 comp rx=0 tx=0).stop 2023-09-10T01:03:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d5010a970 msgr2=0x7f1d5019bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d5010a970 0x7f1d5019bfc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800ef30 tx=0x7f1d4800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 shutdown_connections 2023-09-10T01:03:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1d380a0840 0x7f1d380a2d00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d50101550 0x7f1d501986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d5010a970 0x7f1d5019bfc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 --2- 172.21.15.3:0/2710457014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d50101e70 0x7f1d50198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 >> 172.21.15.3:0/2710457014 conn(0x7f1d500fad90 msgr2=0x7f1d5010cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 shutdown_connections 2023-09-10T01:03:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:41.638+0000 7f1d4ffff700 1 -- 172.21.15.3:0/2710457014 wait complete. 2023-09-10T01:03:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:42 smithi136 ceph-mon[133708]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2023-09-10T01:03:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:42 smithi092 ceph-mon[134067]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2023-09-10T01:03:42.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:42 smithi003 ceph-mon[97654]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2023-09-10T01:03:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:44 smithi136 ceph-mon[133708]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-09-10T01:03:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:44 smithi092 ceph-mon[134067]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-09-10T01:03:44.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:44 smithi003 ceph-mon[97654]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-09-10T01:03:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:46 smithi136 ceph-mon[133708]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2023-09-10T01:03:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:46 smithi092 ceph-mon[134067]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2023-09-10T01:03:46.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:46 smithi003 ceph-mon[97654]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2023-09-10T01:03:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:48 smithi136 ceph-mon[133708]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-09-10T01:03:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:48 smithi092 ceph-mon[134067]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-09-10T01:03:48.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:48 smithi003 ceph-mon[97654]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-09-10T01:03:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:50 smithi136 ceph-mon[133708]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-09-10T01:03:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:50 smithi092 ceph-mon[134067]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-09-10T01:03:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:50 smithi003 ceph-mon[97654]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-09-10T01:03:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:52 smithi136 ceph-mon[133708]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-09-10T01:03:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:52 smithi092 ceph-mon[134067]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-09-10T01:03:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:52 smithi003 ceph-mon[97654]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-09-10T01:03:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:54 smithi136 ceph-mon[133708]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-09-10T01:03:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:54 smithi092 ceph-mon[134067]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-09-10T01:03:54.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:54 smithi003 ceph-mon[97654]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-09-10T01:03:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 -- 172.21.15.3:0/1288651634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038100fb0 msgr2=0x7fa038104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 --2- 172.21.15.3:0/1288651634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038100fb0 0x7fa038104ff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa0280098f0 tx=0x7fa028009c00 comp rx=0 tx=0).stop 2023-09-10T01:03:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 -- 172.21.15.3:0/1288651634 shutdown_connections 2023-09-10T01:03:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 --2- 172.21.15.3:0/1288651634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 0x7fa0381009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 --2- 172.21.15.3:0/1288651634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa038105720 0x7fa03810d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 --2- 172.21.15.3:0/1288651634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038100fb0 0x7fa038104ff0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 -- 172.21.15.3:0/1288651634 >> 172.21.15.3:0/1288651634 conn(0x7fa0380fbec0 msgr2=0x7fa0380fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 -- 172.21.15.3:0/1288651634 shutdown_connections 2023-09-10T01:03:54.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.804+0000 7fa03f59d700 1 -- 172.21.15.3:0/1288651634 wait complete. 2023-09-10T01:03:54.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.805+0000 7fa03f59d700 1 Processor -- start 2023-09-10T01:03:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03f59d700 1 -- start start 2023-09-10T01:03:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03f59d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 0x7fa038072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03f59d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038105720 0x7fa038072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03f59d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa038073380 0x7fa0381a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03f59d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381a4e40 con 0x7fa038073380 2023-09-10T01:03:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03f59d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381a4fb0 con 0x7fa038105720 2023-09-10T01:03:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03f59d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381a5170 con 0x7fa038100600 2023-09-10T01:03:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03e59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 0x7fa038072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa038073380 0x7fa0381a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03e59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 0x7fa038072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:33120/0 (socket says 172.21.15.3:33120) 2023-09-10T01:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03e59b700 1 -- 172.21.15.3:0/3949029315 learned_addr learned my addr 172.21.15.3:0/3949029315 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.806+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038105720 0x7fa038072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.807+0000 7fa03e59b700 1 -- 172.21.15.3:0/3949029315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038105720 msgr2=0x7fa038072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.807+0000 7fa03e59b700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038105720 0x7fa038072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.807+0000 7fa03e59b700 1 -- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa038073380 msgr2=0x7fa0381a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.807+0000 7fa03e59b700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa038073380 0x7fa0381a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.807+0000 7fa03e59b700 1 -- 172.21.15.3:0/3949029315 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa030009d00 con 0x7fa038100600 2023-09-10T01:03:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.807+0000 7fa03e59b700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 0x7fa038072900 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa03400e9a0 tx=0x7fa03400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:54.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.807+0000 7fa02f7fe700 1 -- 172.21.15.3:0/3949029315 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa034009790 con 0x7fa038100600 2023-09-10T01:03:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.808+0000 7fa02f7fe700 1 -- 172.21.15.3:0/3949029315 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa034010430 con 0x7fa038100600 2023-09-10T01:03:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.808+0000 7fa02f7fe700 1 -- 172.21.15.3:0/3949029315 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa034018680 con 0x7fa038100600 2023-09-10T01:03:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.808+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa028009520 con 0x7fa038100600 2023-09-10T01:03:54.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.808+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0381a5750 con 0x7fa038100600 2023-09-10T01:03:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.813+0000 7fa054206b80 1 Processor -- start 2023-09-10T01:03:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.813+0000 7fa054206b80 1 -- start start 2023-09-10T01:03:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.813+0000 7fa02f7fe700 1 -- 172.21.15.3:0/3949029315 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 25) v1 ==== 135434+0+0 (secure 0 0 0) 0x7fa0340105a0 con 0x7fa038100600 2023-09-10T01:03:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa054206b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f00670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa054206b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 0x556f70efa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa054206b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70efade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa054206b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70e536d0 con 0x556f70efa9a0 2023-09-10T01:03:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa054206b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70e50ff0 con 0x556f70f00290 2023-09-10T01:03:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa054206b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70e4c8e0 con 0x556f70f00bb0 2023-09-10T01:03:54.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f00670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f00670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45126/0 (socket says 172.21.15.3:45126) 2023-09-10T01:03:54.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa03ed9c700 1 -- 172.21.15.3:0/4033408961 learned_addr learned my addr 172.21.15.3:0/4033408961 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:54.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.814+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70efade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa03ed9c700 1 -- 172.21.15.3:0/4033408961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 msgr2=0x556f70efa460 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:03:54.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 0x556f70efa460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa03ed9c700 1 -- 172.21.15.3:0/4033408961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 msgr2=0x556f70efade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:54.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70efade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa03ed9c700 1 -- 172.21.15.3:0/4033408961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f70c78d80 con 0x556f70f00290 2023-09-10T01:03:54.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f00670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa030009530 tx=0x7fa0300099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f56a6750657695 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:54.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa02cff9700 1 -- 172.21.15.3:0/4033408961 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03000e070 con 0x556f70f00290 2023-09-10T01:03:54.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa02cff9700 1 -- 172.21.15.3:0/4033408961 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa03000f040 con 0x556f70f00290 2023-09-10T01:03:54.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.815+0000 7fa02cff9700 1 -- 172.21.15.3:0/4033408961 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030017680 con 0x556f70f00290 2023-09-10T01:03:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.816+0000 7fa02f7fe700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0240a0940 0x7fa0240a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.816+0000 7fa02f7fe700 1 -- 172.21.15.3:0/3949029315 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa034014070 con 0x7fa038100600 2023-09-10T01:03:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.816+0000 7fa054206b80 1 -- 172.21.15.3:0/4033408961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 msgr2=0x556f70f00670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.816+0000 7fa054206b80 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f00670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa030009530 tx=0x7fa0300099e0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.816+0000 7fa03e59b700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0240a0940 0x7fa0240a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa054206b80 1 -- 172.21.15.3:0/4033408961 shutdown_connections 2023-09-10T01:03:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa054206b80 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70efade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa054206b80 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f00670 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa054206b80 1 --2- 172.21.15.3:0/4033408961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 0x556f70efa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa054206b80 1 -- 172.21.15.3:0/4033408961 >> 172.21.15.3:0/4033408961 conn(0x556f70e6f030 msgr2=0x556f70ef56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa054206b80 1 -- 172.21.15.3:0/4033408961 shutdown_connections 2023-09-10T01:03:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa03e59b700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0240a0940 0x7fa0240a2e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0340c9990 tx=0x7fa03400c480 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.817+0000 7fa054206b80 1 -- 172.21.15.3:0/4033408961 wait complete. 2023-09-10T01:03:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.818+0000 7fa054206b80 1 -- start start 2023-09-10T01:03:54.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.818+0000 7fa054206b80 1 Processor -- start 2023-09-10T01:03:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa054206b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f06cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa054206b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f03ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa054206b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f04620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa054206b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70deab10 con 0x556f70f00bb0 2023-09-10T01:03:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa054206b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70dea810 con 0x556f70efa9a0 2023-09-10T01:03:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa054206b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70cce8c0 con 0x556f70f00290 2023-09-10T01:03:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa054206b80 10 client.0 _ll_register_callbacks cb 0x556f70d34a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f06cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f06cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45138/0 (socket says 172.21.15.3:45138) 2023-09-10T01:03:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa03ed9c700 1 -- 172.21.15.3:0/448997317 learned_addr learned my addr 172.21.15.3:0/448997317 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.819+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f03ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f04620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa03ed9c700 1 -- 172.21.15.3:0/448997317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 msgr2=0x556f70f03ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f03ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa03ed9c700 1 -- 172.21.15.3:0/448997317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 msgr2=0x556f70f04620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f04620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa03ed9c700 1 -- 172.21.15.3:0/448997317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0300091e0 con 0x556f70efa9a0 2023-09-10T01:03:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f06cd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa0300062a0 tx=0x7fa0300061f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d6211c0cd8355a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa021ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa021ffb700 1 -- 172.21.15.3:0/448997317 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030017660 con 0x556f70efa9a0 2023-09-10T01:03:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556f70da4490 con 0x556f70efa9a0 2023-09-10T01:03:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.820+0000 7fa021ffb700 1 -- 172.21.15.3:0/448997317 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa0300177c0 con 0x556f70efa9a0 2023-09-10T01:03:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.821+0000 7fa021ffb700 1 -- 172.21.15.3:0/448997317 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa030017980 con 0x556f70efa9a0 2023-09-10T01:03:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.821+0000 7fa054206b80 10 client.25907 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.821+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556f70c61c00 con 0x556f70efa9a0 2023-09-10T01:03:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.821+0000 7fa021ffb700 1 -- 172.21.15.3:0/448997317 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556f70da4490 con 0x556f70efa9a0 2023-09-10T01:03:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.821+0000 7fa021ffb700 1 -- 172.21.15.3:0/448997317 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa03001e030 con 0x556f70efa9a0 2023-09-10T01:03:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.822+0000 7fa054206b80 10 client.25907 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.822+0000 7fa054206b80 10 client.25907 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.822+0000 7fa054206b80 20 client.25907 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.822+0000 7fa054206b80 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f14420 0x556f70f16810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.822+0000 7fa054206b80 4 client.25907 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:03:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.822+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f70e61840 con 0x556f70f14420 2023-09-10T01:03:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.822+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f14420 0x556f70f16810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.823+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f14420 0x556f70f16810 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efa4e5bb273ce101 server_cookie=419fc56aba35da32 in_seq=0 out_seq=0 2023-09-10T01:03:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.823+0000 7fa021ffb700 10 client.25907 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.823+0000 7fa021ffb700 1 -- 172.21.15.3:0/448997317 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556f70e61840 con 0x556f70f14420 2023-09-10T01:03:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.823+0000 7fa021ffb700 10 client.25907 handle_command_reply: tid=0 2023-09-10T01:03:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 client.25907 shutdown 2023-09-10T01:03:54.834 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 msgr2=0x556f70f06cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f06cd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa0300062a0 tx=0x7fa0300061f0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 shutdown_connections 2023-09-10T01:03:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f14420 0x556f70f16810 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f04620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f06cd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 --2- 172.21.15.3:0/448997317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f03ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 >> 172.21.15.3:0/448997317 conn(0x556f70e6f030 msgr2=0x556f70ef4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa021ffb700 0 client.25907 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 shutdown_connections 2023-09-10T01:03:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 1 -- 172.21.15.3:0/448997317 wait complete. 2023-09-10T01:03:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.824+0000 7fa054206b80 20 client.25907 trim_cache size 0 max 16384 2023-09-10T01:03:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.846+0000 7fa054206b80 1 Processor -- start 2023-09-10T01:03:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.846+0000 7fa054206b80 1 -- start start 2023-09-10T01:03:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa054206b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70f06800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa054206b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f06d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa054206b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 0x556f70ef4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa054206b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70efb720 con 0x556f70efa9a0 2023-09-10T01:03:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa054206b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70ef6d40 con 0x556f70f00290 2023-09-10T01:03:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa054206b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70f10f10 con 0x556f70f00bb0 2023-09-10T01:03:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70f06800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70f06800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46392/0 (socket says 172.21.15.3:46392) 2023-09-10T01:03:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa03ed9c700 1 -- 172.21.15.3:0/3909359232 learned_addr learned my addr 172.21.15.3:0/3909359232 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f06d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.847+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 0x556f70ef4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa03ed9c700 1 -- 172.21.15.3:0/3909359232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 msgr2=0x556f70ef4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 0x556f70ef4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa03ed9c700 1 -- 172.21.15.3:0/3909359232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 msgr2=0x556f70f06d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f06d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa03ed9c700 1 -- 172.21.15.3:0/3909359232 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f70c9f1f0 con 0x556f70efa9a0 2023-09-10T01:03:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f06d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:03:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70f06800 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa03002f040 tx=0x7fa0300239a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38b889bcd411d0c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa02d7fa700 1 -- 172.21.15.3:0/3909359232 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030009960 con 0x556f70efa9a0 2023-09-10T01:03:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa02d7fa700 1 -- 172.21.15.3:0/3909359232 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa03000e030 con 0x556f70efa9a0 2023-09-10T01:03:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.848+0000 7fa02d7fa700 1 -- 172.21.15.3:0/3909359232 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0300176c0 con 0x556f70efa9a0 2023-09-10T01:03:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 -- 172.21.15.3:0/3909359232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 msgr2=0x556f70f06800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70f06800 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa03002f040 tx=0x7fa0300239a0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 -- 172.21.15.3:0/3909359232 shutdown_connections 2023-09-10T01:03:54.856 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00bb0 0x556f70ef4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70efa9a0 0x556f70f06800 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 --2- 172.21.15.3:0/3909359232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70f00290 0x556f70f06d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 -- 172.21.15.3:0/3909359232 >> 172.21.15.3:0/3909359232 conn(0x556f70e6f030 msgr2=0x556f70efe040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 -- 172.21.15.3:0/3909359232 shutdown_connections 2023-09-10T01:03:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.849+0000 7fa054206b80 1 -- 172.21.15.3:0/3909359232 wait complete. 2023-09-10T01:03:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.850+0000 7fa054206b80 1 -- start start 2023-09-10T01:03:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.850+0000 7fa054206b80 1 Processor -- start 2023-09-10T01:03:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa054206b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f03b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa054206b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f02440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa054206b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f02980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa054206b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70f041e0 con 0x556f70f00bb0 2023-09-10T01:03:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa054206b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70f02ec0 con 0x556f70efa9a0 2023-09-10T01:03:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa054206b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f70f03080 con 0x556f70f00290 2023-09-10T01:03:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa054206b80 10 client.0 _ll_register_callbacks cb 0x7fa030009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:03:54.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f03b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa03ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f03b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45146/0 (socket says 172.21.15.3:45146) 2023-09-10T01:03:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa03ed9c700 1 -- 172.21.15.3:0/2156554160 learned_addr learned my addr 172.21.15.3:0/2156554160 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:03:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.851+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f02980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f02440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa03ed9c700 1 -- 172.21.15.3:0/2156554160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 msgr2=0x556f70f02440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f02440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa03ed9c700 1 -- 172.21.15.3:0/2156554160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 msgr2=0x556f70f02980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:03:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f02980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa03ed9c700 1 -- 172.21.15.3:0/2156554160 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03810db10 con 0x556f70efa9a0 2023-09-10T01:03:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa03ed9c700 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f03b90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x556f70eefd40 tx=0x7fa0300259b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c06c7d5f276612 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:03:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.852+0000 7fa021ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:03:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.853+0000 7fa021ffb700 1 -- 172.21.15.3:0/2156554160 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030004a10 con 0x556f70efa9a0 2023-09-10T01:03:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.853+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa03000f040 con 0x556f70efa9a0 2023-09-10T01:03:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.853+0000 7fa021ffb700 1 -- 172.21.15.3:0/2156554160 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa030004b70 con 0x556f70efa9a0 2023-09-10T01:03:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.853+0000 7fa021ffb700 1 -- 172.21.15.3:0/2156554160 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa030004d30 con 0x556f70efa9a0 2023-09-10T01:03:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.853+0000 7fa054206b80 10 client.25913 fetch_fsmap learned FSMap version 22 2023-09-10T01:03:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.853+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556f70c61c00 con 0x556f70efa9a0 2023-09-10T01:03:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.853+0000 7fa021ffb700 1 -- 172.21.15.3:0/2156554160 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa03004a020 con 0x556f70efa9a0 2023-09-10T01:03:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa021ffb700 1 -- 172.21.15.3:0/2156554160 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa03002d020 con 0x556f70efa9a0 2023-09-10T01:03:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa054206b80 10 client.25913 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:03:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa054206b80 10 client.25913 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:03:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa054206b80 20 client.25913 populate_metadata read hostname 'smithi003' 2023-09-10T01:03:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa054206b80 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f71950 0x556f70f73d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:03:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa054206b80 4 client.25913 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:03:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556f70e458c0 con 0x556f70f71950 2023-09-10T01:03:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.854+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f71950 0x556f70f73d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:03:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.855+0000 7fa03dd9a700 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f71950 0x556f70f73d70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1da0ec35e98c093 server_cookie=7b38be10a7ab6a8e in_seq=0 out_seq=0 2023-09-10T01:03:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.855+0000 7fa021ffb700 10 client.25913 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.863+0000 7fa021ffb700 1 -- 172.21.15.3:0/2156554160 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556f70e458c0 con 0x556f70f71950 2023-09-10T01:03:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.863+0000 7fa021ffb700 10 client.25913 handle_command_reply: tid=0 2023-09-10T01:03:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.863+0000 7fa054206b80 1 client.25913 shutdown 2023-09-10T01:03:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.863+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 msgr2=0x556f70f03b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.863+0000 7fa054206b80 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f03b90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x556f70eefd40 tx=0x7fa0300259b0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 shutdown_connections 2023-09-10T01:03:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556f70f71950 0x556f70f73d70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556f70f00bb0 0x556f70f02980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556f70efa9a0 0x556f70f03b90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 --2- 172.21.15.3:0/2156554160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f70f00290 0x556f70f02440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 >> 172.21.15.3:0/2156554160 conn(0x556f70e6f030 msgr2=0x556f70ef4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:54.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa021ffb700 0 client.25913 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:03:54.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 shutdown_connections 2023-09-10T01:03:54.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 1 -- 172.21.15.3:0/2156554160 wait complete. 2023-09-10T01:03:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.864+0000 7fa054206b80 20 client.25913 trim_cache size 0 max 16384 2023-09-10T01:03:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:03:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:03:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:03:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:03:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0240a0940 msgr2=0x7fa0240a2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0240a0940 0x7fa0240a2e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0340c9990 tx=0x7fa03400c480 comp rx=0 tx=0).stop 2023-09-10T01:03:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 msgr2=0x7fa038072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:03:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 0x7fa038072900 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa03400e9a0 tx=0x7fa03400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 shutdown_connections 2023-09-10T01:03:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0240a0940 0x7fa0240a2e00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa038100600 0x7fa038072900 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa038073380 0x7fa0381a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 --2- 172.21.15.3:0/3949029315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa038105720 0x7fa038072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:03:54.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.865+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 >> 172.21.15.3:0/3949029315 conn(0x7fa0380fbec0 msgr2=0x7fa0380fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:03:54.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.866+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 shutdown_connections 2023-09-10T01:03:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:03:54.866+0000 7fa03f59d700 1 -- 172.21.15.3:0/3949029315 wait complete. 2023-09-10T01:03:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:56 smithi136 ceph-mon[133708]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2023-09-10T01:03:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:56 smithi092 ceph-mon[134067]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2023-09-10T01:03:56.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:56 smithi003 ceph-mon[97654]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2023-09-10T01:03:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:03:58 smithi136 ceph-mon[133708]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:03:58 smithi092 ceph-mon[134067]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:03:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:03:58 smithi003 ceph-mon[97654]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2023-09-10T01:04:00.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:00 smithi136 ceph-mon[133708]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-09-10T01:04:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:00 smithi092 ceph-mon[134067]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-09-10T01:04:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:00 smithi003 ceph-mon[97654]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-09-10T01:04:01.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:04:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:04:00.882+0000 7f15fbb7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:04:01.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:04:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:04:00.882+0000 7f15fbb7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:04:01.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:04:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:04:00.882+0000 7f15fbb7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:04:01.309 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:04:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:04:00.882+0000 7f15fbb7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:04:01.310 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:04:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:04:00.882+0000 7f15fbb7a700 -1 client.0 error registering admin socket command: (17) File exists 2023-09-10T01:04:02.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:02 smithi136 ceph-mon[133708]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-09-10T01:04:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:02 smithi092 ceph-mon[134067]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-09-10T01:04:02.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:02 smithi003 ceph-mon[97654]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-09-10T01:04:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:04 smithi092 ceph-mon[134067]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 853 B/s wr, 6 op/s 2023-09-10T01:04:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:04 smithi092 ceph-mon[134067]: mgrmap e26: x(active, since 50s), standbys: z, y 2023-09-10T01:04:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:04 smithi003 ceph-mon[97654]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 853 B/s wr, 6 op/s 2023-09-10T01:04:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:04 smithi003 ceph-mon[97654]: mgrmap e26: x(active, since 50s), standbys: z, y 2023-09-10T01:04:04.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:04 smithi136 ceph-mon[133708]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 853 B/s wr, 6 op/s 2023-09-10T01:04:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:04 smithi136 ceph-mon[133708]: mgrmap e26: x(active, since 50s), standbys: z, y 2023-09-10T01:04:06.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:04:06.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 -- 172.21.15.3:0/1882298892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334101000 msgr2=0x7fd33410d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1882298892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334101000 0x7fd33410d510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00b0a0 tx=0x7fd32c00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 -- 172.21.15.3:0/1882298892 shutdown_connections 2023-09-10T01:04:06.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1882298892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334101000 0x7fd33410d510 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1882298892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd334106650 0x7fd334106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1882298892 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 0x7fd334100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 -- 172.21.15.3:0/1882298892 >> 172.21.15.3:0/1882298892 conn(0x7fd3340fc170 msgr2=0x7fd3340fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:06.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 -- 172.21.15.3:0/1882298892 shutdown_connections 2023-09-10T01:04:06.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.460+0000 7fd33bbff700 1 -- 172.21.15.3:0/1882298892 wait complete. 2023-09-10T01:04:06.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.462+0000 7fd33bbff700 1 Processor -- start 2023-09-10T01:04:06.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.462+0000 7fd33bbff700 1 -- start start 2023-09-10T01:04:06.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33bbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 0x7fd33419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33bbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd334101000 0x7fd33419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33bbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334106650 0x7fd334198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33bbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd334198db0 con 0x7fd334101000 2023-09-10T01:04:06.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33bbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd334198f20 con 0x7fd334100640 2023-09-10T01:04:06.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33bbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3341990e0 con 0x7fd334106650 2023-09-10T01:04:06.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 0x7fd33419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 0x7fd33419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55098/0 (socket says 172.21.15.3:55098) 2023-09-10T01:04:06.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33999b700 1 -- 172.21.15.3:0/1052692378 learned_addr learned my addr 172.21.15.3:0/1052692378 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:06.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33919a700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd334101000 0x7fd33419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.463+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334106650 0x7fd334198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:06 smithi092 ceph-mon[134067]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-09-10T01:04:06.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd33999b700 1 -- 172.21.15.3:0/1052692378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334106650 msgr2=0x7fd334198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd33999b700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334106650 0x7fd334198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd33999b700 1 -- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd334101000 msgr2=0x7fd33419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd33999b700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd334101000 0x7fd33419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd33999b700 1 -- 172.21.15.3:0/1052692378 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c009d00 con 0x7fd334100640 2023-09-10T01:04:06.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd33999b700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 0x7fd33419e630 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd33000d950 tx=0x7fd33000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:06.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd32affd700 1 -- 172.21.15.3:0/1052692378 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33000f890 con 0x7fd334100640 2023-09-10T01:04:06.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd32affd700 1 -- 172.21.15.3:0/1052692378 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd330010460 con 0x7fd334100640 2023-09-10T01:04:06.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd32affd700 1 -- 172.21.15.3:0/1052692378 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33000e610 con 0x7fd334100640 2023-09-10T01:04:06.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.464+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3341993c0 con 0x7fd334100640 2023-09-10T01:04:06.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.465+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3341a5620 con 0x7fd334100640 2023-09-10T01:04:06.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.470+0000 7fd32affd700 1 -- 172.21.15.3:0/1052692378 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7fd3300108d0 con 0x7fd334100640 2023-09-10T01:04:06.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.471+0000 7fd34f5f2b80 1 Processor -- start 2023-09-10T01:04:06.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.472+0000 7fd34f5f2b80 1 -- start start 2023-09-10T01:04:06.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.472+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c3f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.472+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.472+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c40ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.472+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c352730 con 0x55850c3fa030 2023-09-10T01:04:06.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.472+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c350050 con 0x55850c3f9710 2023-09-10T01:04:06.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.472+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c34b940 con 0x55850c40c650 2023-09-10T01:04:06.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33919a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c40ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33999b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33919a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c40ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58488/0 (socket says 172.21.15.3:58488) 2023-09-10T01:04:06.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33919a700 1 -- 172.21.15.3:0/1017560638 learned_addr learned my addr 172.21.15.3:0/1017560638 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:06.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33919a700 1 -- 172.21.15.3:0/1017560638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 msgr2=0x55850c3f9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33919a700 1 --2- 172.21.15.3:0/1017560638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c3f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33919a700 1 -- 172.21.15.3:0/1017560638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 msgr2=0x55850c40c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.473+0000 7fd33919a700 1 --2- 172.21.15.3:0/1017560638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd33919a700 1 -- 172.21.15.3:0/1017560638 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55850c177de0 con 0x55850c40c650 2023-09-10T01:04:06.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd32affd700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3200a0950 0x7fd3200a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd32affd700 1 -- 172.21.15.3:0/1052692378 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd3300cc310 con 0x7fd334100640 2023-09-10T01:04:06.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd33919a700 1 --2- 172.21.15.3:0/1017560638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c40ea40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd324009d20 tx=0x7fd324011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c18fe66696a68ca server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:06.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3200a0950 0x7fd3200a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd31ffff700 1 -- 172.21.15.3:0/1017560638 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd324019630 con 0x55850c40c650 2023-09-10T01:04:06.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd31ffff700 1 -- 172.21.15.3:0/1017560638 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd324019c50 con 0x55850c40c650 2023-09-10T01:04:06.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.474+0000 7fd31ffff700 1 -- 172.21.15.3:0/1017560638 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32400ec40 con 0x55850c40c650 2023-09-10T01:04:06.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.475+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1017560638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 msgr2=0x55850c40ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.475+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3200a0950 0x7fd3200a2e10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00b780 tx=0x7fd32c009550 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:06.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.475+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1017560638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c40ea40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd324009d20 tx=0x7fd324011720 comp rx=0 tx=0).stop 2023-09-10T01:04:06.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.476+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1017560638 shutdown_connections 2023-09-10T01:04:06.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.476+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1017560638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c40ea40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.476+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1017560638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c3f9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.476+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1017560638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.476+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1017560638 >> 172.21.15.3:0/1017560638 conn(0x55850c36e090 msgr2=0x55850c3f4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:06.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.476+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1017560638 shutdown_connections 2023-09-10T01:04:06.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.476+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1017560638 wait complete. 2023-09-10T01:04:06.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.477+0000 7fd34f5f2b80 1 -- start start 2023-09-10T01:04:06.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.477+0000 7fd34f5f2b80 1 Processor -- start 2023-09-10T01:04:06.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c40c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c3fa030 0x55850c409290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c40c650 0x55850c4099d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c2e9b70 con 0x55850c40c650 2023-09-10T01:04:06.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c2e9870 con 0x55850c3f9710 2023-09-10T01:04:06.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c1cd920 con 0x55850c3fa030 2023-09-10T01:04:06.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd34f5f2b80 10 client.0 _ll_register_callbacks cb 0x55850c233a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:06.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c40c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c40c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55118/0 (socket says 172.21.15.3:55118) 2023-09-10T01:04:06.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.478+0000 7fd33999b700 1 -- 172.21.15.3:0/1945844358 learned_addr learned my addr 172.21.15.3:0/1945844358 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:06.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33919a700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c3fa030 0x55850c409290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c40c650 0x55850c4099d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33a19c700 1 -- 172.21.15.3:0/1945844358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c3fa030 msgr2=0x55850c409290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c3fa030 0x55850c409290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33a19c700 1 -- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 msgr2=0x55850c40c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c40c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33a19c700 1 -- 172.21.15.3:0/1945844358 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd324009980 con 0x55850c40c650 2023-09-10T01:04:06.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.479+0000 7fd33999b700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c40c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:06.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.480+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c40c650 0x55850c4099d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0062a0 tx=0x7fd32c00b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec957bb161855a9f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:06.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.480+0000 7fd31d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:06.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.480+0000 7fd31d7fa700 1 -- 172.21.15.3:0/1945844358 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c00e040 con 0x55850c40c650 2023-09-10T01:04:06.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.480+0000 7fd31d7fa700 1 -- 172.21.15.3:0/1945844358 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd32c01c640 con 0x55850c40c650 2023-09-10T01:04:06.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.480+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55850c2a34f0 con 0x55850c40c650 2023-09-10T01:04:06.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.481+0000 7fd31d7fa700 1 -- 172.21.15.3:0/1945844358 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd32c01c800 con 0x55850c40c650 2023-09-10T01:04:06.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.481+0000 7fd34f5f2b80 10 client.15888 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:06.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.481+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55850c160c60 con 0x55850c40c650 2023-09-10T01:04:06.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.481+0000 7fd31d7fa700 1 -- 172.21.15.3:0/1945844358 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55850c2a34f0 con 0x55850c40c650 2023-09-10T01:04:06.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd31d7fa700 1 -- 172.21.15.3:0/1945844358 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd32c005710 con 0x55850c40c650 2023-09-10T01:04:06.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd34f5f2b80 10 client.15888 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd34f5f2b80 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd34f5f2b80 20 client.15888 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c41c130 0x55850c41e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd34f5f2b80 4 client.15888 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55850c3608a0 con 0x55850c41c130 2023-09-10T01:04:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.482+0000 7fd33999b700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c41c130 0x55850c41e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.483+0000 7fd33999b700 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c41c130 0x55850c41e550 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9675b9b7bc4df69 server_cookie=f23400049f0e1960 in_seq=0 out_seq=0 2023-09-10T01:04:06.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.483+0000 7fd31d7fa700 10 client.15888 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:06.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd31d7fa700 1 -- 172.21.15.3:0/1945844358 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55850c3608a0 con 0x55850c41c130 2023-09-10T01:04:06.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd31d7fa700 10 client.15888 handle_command_reply: tid=0 2023-09-10T01:04:06.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 client.15888 shutdown 2023-09-10T01:04:06.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c40c650 msgr2=0x55850c4099d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c40c650 0x55850c4099d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0062a0 tx=0x7fd32c00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 shutdown_connections 2023-09-10T01:04:06.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c41c130 0x55850c41e550 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c40c650 0x55850c4099d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c40c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd31d7fa700 0 client.15888 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:06.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1945844358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c3fa030 0x55850c409290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.484+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 >> 172.21.15.3:0/1945844358 conn(0x55850c36e090 msgr2=0x55850c3f4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:06.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.485+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 shutdown_connections 2023-09-10T01:04:06.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.485+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1945844358 wait complete. 2023-09-10T01:04:06.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.485+0000 7fd34f5f2b80 20 client.15888 trim_cache size 0 max 16384 2023-09-10T01:04:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 Processor -- start 2023-09-10T01:04:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 -- start start 2023-09-10T01:04:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c413200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c4046b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c3608a0 con 0x55850c3fa030 2023-09-10T01:04:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c2a34f0 con 0x55850c3f9710 2023-09-10T01:04:06.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.519+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c3fa7e0 con 0x55850c40c650 2023-09-10T01:04:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.520+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c413200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.520+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c413200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55132/0 (socket says 172.21.15.3:55132) 2023-09-10T01:04:06.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.520+0000 7fd33999b700 1 -- 172.21.15.3:0/1051199852 learned_addr learned my addr 172.21.15.3:0/1051199852 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.520+0000 7fd33919a700 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.520+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c4046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd33999b700 1 -- 172.21.15.3:0/1051199852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 msgr2=0x55850c4046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd33999b700 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c4046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd33999b700 1 -- 172.21.15.3:0/1051199852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 msgr2=0x55850c40bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd33999b700 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd33999b700 1 -- 172.21.15.3:0/1051199852 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55850c19e250 con 0x55850c3f9710 2023-09-10T01:04:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd33a19c700 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c4046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd33999b700 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c413200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd330095bf0 tx=0x7fd330000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e5285629e1ca8c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:06.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd328bff700 1 -- 172.21.15.3:0/1051199852 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330099880 con 0x55850c3f9710 2023-09-10T01:04:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd328bff700 1 -- 172.21.15.3:0/1051199852 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd330099180 con 0x55850c3f9710 2023-09-10T01:04:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.521+0000 7fd328bff700 1 -- 172.21.15.3:0/1051199852 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330099640 con 0x55850c3f9710 2023-09-10T01:04:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.522+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1051199852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 msgr2=0x55850c413200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.522+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c413200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd330095bf0 tx=0x7fd330000bc0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.522+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1051199852 shutdown_connections 2023-09-10T01:04:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.522+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c4046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.522+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c413200 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.522+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/1051199852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c40bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.522+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1051199852 >> 172.21.15.3:0/1051199852 conn(0x55850c36e090 msgr2=0x55850c418450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:06.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.523+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1051199852 shutdown_connections 2023-09-10T01:04:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.523+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/1051199852 wait complete. 2023-09-10T01:04:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.523+0000 7fd34f5f2b80 1 -- start start 2023-09-10T01:04:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 1 Processor -- start 2023-09-10T01:04:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c412f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c4111e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c411720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c411c60 con 0x55850c3fa030 2023-09-10T01:04:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c411dd0 con 0x55850c3f9710 2023-09-10T01:04:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55850c411f90 con 0x55850c40c650 2023-09-10T01:04:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.524+0000 7fd34f5f2b80 10 client.0 _ll_register_callbacks cb 0x7fd32c003d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c412f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33a19c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c411720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c412f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55148/0 (socket says 172.21.15.3:55148) 2023-09-10T01:04:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33999b700 1 -- 172.21.15.3:0/608503963 learned_addr learned my addr 172.21.15.3:0/608503963 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33919a700 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c4111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33a19c700 1 -- 172.21.15.3:0/608503963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 msgr2=0x55850c412f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33a19c700 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c412f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33a19c700 1 -- 172.21.15.3:0/608503963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 msgr2=0x55850c4111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33a19c700 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c4111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33a19c700 1 -- 172.21.15.3:0/608503963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55850c160c60 con 0x55850c40c650 2023-09-10T01:04:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.525+0000 7fd33919a700 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c4111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:04:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.526+0000 7fd33a19c700 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c411720 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009fd0 tx=0x7fd32c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2640e427c382d474 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:06.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.526+0000 7fd31d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.526+0000 7fd31d7fa700 1 -- 172.21.15.3:0/608503963 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c00e040 con 0x55850c40c650 2023-09-10T01:04:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.526+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd324019c50 con 0x55850c40c650 2023-09-10T01:04:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.526+0000 7fd31d7fa700 1 -- 172.21.15.3:0/608503963 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd32c0054d0 con 0x55850c40c650 2023-09-10T01:04:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.527+0000 7fd31d7fa700 1 -- 172.21.15.3:0/608503963 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd32c005690 con 0x55850c40c650 2023-09-10T01:04:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.527+0000 7fd34f5f2b80 10 client.25858 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.527+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fd3341a5620 con 0x55850c40c650 2023-09-10T01:04:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.527+0000 7fd31d7fa700 1 -- 172.21.15.3:0/608503963 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd324019c50 con 0x55850c40c650 2023-09-10T01:04:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.528+0000 7fd31d7fa700 1 -- 172.21.15.3:0/608503963 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd32c046450 con 0x55850c40c650 2023-09-10T01:04:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.528+0000 7fd34f5f2b80 10 client.25858 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:06.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.528+0000 7fd34f5f2b80 10 client.25858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:06.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.528+0000 7fd34f5f2b80 20 client.25858 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:06.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.528+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c470f20 0x55850c473310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.528+0000 7fd34f5f2b80 4 client.25858 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:04:06.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.528+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55850c344920 con 0x55850c470f20 2023-09-10T01:04:06.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.529+0000 7fd33999b700 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c470f20 0x55850c473310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.529+0000 7fd33999b700 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c470f20 0x55850c473310 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=428878f291585871 server_cookie=7c434de7dc5404c5 in_seq=0 out_seq=0 2023-09-10T01:04:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.529+0000 7fd31d7fa700 10 client.25858 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.529+0000 7fd31d7fa700 1 -- 172.21.15.3:0/608503963 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55850c344920 con 0x55850c470f20 2023-09-10T01:04:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.529+0000 7fd31d7fa700 10 client.25858 handle_command_reply: tid=0 2023-09-10T01:04:06.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.529+0000 7fd34f5f2b80 1 client.25858 shutdown 2023-09-10T01:04:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 msgr2=0x55850c411720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c411720 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009fd0 tx=0x7fd32c009780 comp rx=0 tx=0).stop 2023-09-10T01:04:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 shutdown_connections 2023-09-10T01:04:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55850c470f20 0x55850c473310 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55850c40c650 0x55850c411720 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd31d7fa700 0 client.25858 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:06.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55850c3f9710 0x55850c412f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 --2- 172.21.15.3:0/608503963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55850c3fa030 0x55850c4111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 >> 172.21.15.3:0/608503963 conn(0x55850c36e090 msgr2=0x55850c4046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 shutdown_connections 2023-09-10T01:04:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.530+0000 7fd34f5f2b80 1 -- 172.21.15.3:0/608503963 wait complete. 2023-09-10T01:04:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.531+0000 7fd34f5f2b80 20 client.25858 trim_cache size 0 max 16384 2023-09-10T01:04:06.542 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:04:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:04:06.542 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:04:06.542 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:04:06.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.531+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3200a0950 msgr2=0x7fd3200a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.531+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3200a0950 0x7fd3200a2e10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00b780 tx=0x7fd32c009550 comp rx=0 tx=0).stop 2023-09-10T01:04:06.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.531+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 msgr2=0x7fd33419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.531+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 0x7fd33419e630 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd33000d950 tx=0x7fd33000dc60 comp rx=0 tx=0).stop 2023-09-10T01:04:06.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 shutdown_connections 2023-09-10T01:04:06.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3200a0950 0x7fd3200a2e10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd334106650 0x7fd334198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd334100640 0x7fd33419e630 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 --2- 172.21.15.3:0/1052692378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd334101000 0x7fd33419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 >> 172.21.15.3:0/1052692378 conn(0x7fd3340fc170 msgr2=0x7fd33410bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 shutdown_connections 2023-09-10T01:04:06.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.532+0000 7fd33bbff700 1 -- 172.21.15.3:0/1052692378 wait complete. 2023-09-10T01:04:06.557 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:06 smithi003 ceph-mon[97654]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-09-10T01:04:06.560 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:04:06.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:04:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:06 smithi136 ceph-mon[133708]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-09-10T01:04:06.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.901+0000 7fdc2dfff700 1 -- 172.21.15.3:0/1263917552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 msgr2=0x7fdc2810c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.901+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/1263917552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc2810c920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009990 tx=0x7fdc1c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.902+0000 7fdc2dfff700 1 -- 172.21.15.3:0/1263917552 shutdown_connections 2023-09-10T01:04:06.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.902+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/1263917552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc28072a30 0x7fdc28072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.902+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/1263917552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc2810ce60 0x7fdc2810f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.902+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/1263917552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc2810c920 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.902+0000 7fdc2dfff700 1 -- 172.21.15.3:0/1263917552 >> 172.21.15.3:0/1263917552 conn(0x7fdc280fc110 msgr2=0x7fdc280fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:06.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.902+0000 7fdc2dfff700 1 -- 172.21.15.3:0/1263917552 shutdown_connections 2023-09-10T01:04:06.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.902+0000 7fdc2dfff700 1 -- 172.21.15.3:0/1263917552 wait complete. 2023-09-10T01:04:06.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.903+0000 7fdc2dfff700 1 Processor -- start 2023-09-10T01:04:06.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.904+0000 7fdc2dfff700 1 -- start start 2023-09-10T01:04:06.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.904+0000 7fdc2dfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc28072a30 0x7fdc28198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.904+0000 7fdc2dfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc28198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.904+0000 7fdc2dfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc2810ce60 0x7fdc2819d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.904+0000 7fdc2dfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2819d900 con 0x7fdc28073350 2023-09-10T01:04:06.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.904+0000 7fdc2dfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2819da70 con 0x7fdc28072a30 2023-09-10T01:04:06.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.904+0000 7fdc2dfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2819dc30 con 0x7fdc2810ce60 2023-09-10T01:04:06.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc26ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc28198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc26ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc28198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42060/0 (socket says 172.21.15.3:42060) 2023-09-10T01:04:06.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc26ffd700 1 -- 172.21.15.3:0/2805622463 learned_addr learned my addr 172.21.15.3:0/2805622463 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:06.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc27fff700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc2810ce60 0x7fdc2819d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc277fe700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc28072a30 0x7fdc28198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc277fe700 1 -- 172.21.15.3:0/2805622463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc2810ce60 msgr2=0x7fdc2819d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc277fe700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc2810ce60 0x7fdc2819d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc277fe700 1 -- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 msgr2=0x7fdc28198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:06.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc277fe700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc28198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:06.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc277fe700 1 -- 172.21.15.3:0/2805622463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20009c40 con 0x7fdc28072a30 2023-09-10T01:04:06.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.905+0000 7fdc26ffd700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc28198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:06.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.906+0000 7fdc277fe700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc28072a30 0x7fdc28198960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e9f0 tx=0x7fdc1800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:06.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.906+0000 7fdc24ff9700 1 -- 172.21.15.3:0/2805622463 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc180096d0 con 0x7fdc28072a30 2023-09-10T01:04:06.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.906+0000 7fdc24ff9700 1 -- 172.21.15.3:0/2805622463 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc1800cbd0 con 0x7fdc28072a30 2023-09-10T01:04:06.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.906+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c0095f0 con 0x7fdc28072a30 2023-09-10T01:04:06.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.906+0000 7fdc24ff9700 1 -- 172.21.15.3:0/2805622463 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18018680 con 0x7fdc28072a30 2023-09-10T01:04:06.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.906+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc281a55e0 con 0x7fdc28072a30 2023-09-10T01:04:06.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.908+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc28067480 con 0x7fdc28072a30 2023-09-10T01:04:06.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.912+0000 7fdc24ff9700 1 -- 172.21.15.3:0/2805622463 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7fdc1800cd40 con 0x7fdc28072a30 2023-09-10T01:04:06.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.914+0000 7fdc24ff9700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc100a0950 0x7fdc100a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:06.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.914+0000 7fdc24ff9700 1 -- 172.21.15.3:0/2805622463 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdc18014070 con 0x7fdc28072a30 2023-09-10T01:04:06.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.915+0000 7fdc26ffd700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc100a0950 0x7fdc100a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:06.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.915+0000 7fdc24ff9700 1 -- 172.21.15.3:0/2805622463 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fdc180f8e70 con 0x7fdc28072a30 2023-09-10T01:04:06.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:06.915+0000 7fdc26ffd700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc100a0950 0x7fdc100a2e10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdc28199f70 tx=0x7fdc1c00b260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:07.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.427+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc28199cb0 con 0x7fdc28072a30 2023-09-10T01:04:07.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.428+0000 7fdc24ff9700 1 -- 172.21.15.3:0/2805622463 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11292 (secure 0 0 0) 0x7fdc18095d80 con 0x7fdc28072a30 2023-09-10T01:04:07.429 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:04:07.429 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:01:09.940177+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:04:07.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.431+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc100a0950 msgr2=0x7fdc100a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.431+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc100a0950 0x7fdc100a2e10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdc28199f70 tx=0x7fdc1c00b260 comp rx=0 tx=0).stop 2023-09-10T01:04:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.431+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc28072a30 msgr2=0x7fdc28198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.431+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc28072a30 0x7fdc28198960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e9f0 tx=0x7fdc1800ed00 comp rx=0 tx=0).stop 2023-09-10T01:04:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 shutdown_connections 2023-09-10T01:04:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc100a0950 0x7fdc100a2e10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc2810ce60 0x7fdc2819d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc28072a30 0x7fdc28198960 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 --2- 172.21.15.3:0/2805622463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc28073350 0x7fdc28198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 >> 172.21.15.3:0/2805622463 conn(0x7fdc280fc110 msgr2=0x7fdc2810f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 shutdown_connections 2023-09-10T01:04:07.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.432+0000 7fdc2dfff700 1 -- 172.21.15.3:0/2805622463 wait complete. 2023-09-10T01:04:07.435 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 22 2023-09-10T01:04:07.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:04:07.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.811+0000 7f7d205ff700 1 -- 172.21.15.3:0/3761750822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 msgr2=0x7f7d18100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.811+0000 7f7d205ff700 1 --2- 172.21.15.3:0/3761750822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18100ac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009990 tx=0x7f7d08009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.812+0000 7f7d205ff700 1 -- 172.21.15.3:0/3761750822 shutdown_connections 2023-09-10T01:04:07.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.812+0000 7f7d205ff700 1 --2- 172.21.15.3:0/3761750822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18106650 0x7f7d18106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.812+0000 7f7d205ff700 1 --2- 172.21.15.3:0/3761750822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d18101000 0x7f7d1810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.812+0000 7f7d205ff700 1 --2- 172.21.15.3:0/3761750822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18100ac0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.812+0000 7f7d205ff700 1 -- 172.21.15.3:0/3761750822 >> 172.21.15.3:0/3761750822 conn(0x7f7d180fc170 msgr2=0x7f7d180fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:07.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.812+0000 7f7d205ff700 1 -- 172.21.15.3:0/3761750822 shutdown_connections 2023-09-10T01:04:07.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.812+0000 7f7d205ff700 1 -- 172.21.15.3:0/3761750822 wait complete. 2023-09-10T01:04:07.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.813+0000 7f7d205ff700 1 Processor -- start 2023-09-10T01:04:07.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.814+0000 7f7d205ff700 1 -- start start 2023-09-10T01:04:07.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.814+0000 7f7d205ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.814+0000 7f7d205ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18101000 0x7f7d18072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.814+0000 7f7d205ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d18106650 0x7f7d181a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.814+0000 7f7d205ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d18073670 con 0x7f7d18106650 2023-09-10T01:04:07.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.814+0000 7f7d205ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d181a4e00 con 0x7f7d18100640 2023-09-10T01:04:07.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.814+0000 7f7d205ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d181a4f40 con 0x7f7d18101000 2023-09-10T01:04:07.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1db9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18101000 0x7f7d18072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55186/0 (socket says 172.21.15.3:55186) 2023-09-10T01:04:07.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 -- 172.21.15.3:0/1209210558 learned_addr learned my addr 172.21.15.3:0/1209210558 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:07.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1db9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18101000 0x7f7d18072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58546/0 (socket says 172.21.15.3:58546) 2023-09-10T01:04:07.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d18106650 0x7f7d181a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 -- 172.21.15.3:0/1209210558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18101000 msgr2=0x7f7d18072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18101000 0x7f7d18072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 -- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d18106650 msgr2=0x7f7d181a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d18106650 0x7f7d181a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.815+0000 7f7d1e39b700 1 -- 172.21.15.3:0/1209210558 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d080095f0 con 0x7f7d18100640 2023-09-10T01:04:07.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.816+0000 7f7d1db9a700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18101000 0x7f7d18072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:07.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.816+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18072900 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400e9a0 tx=0x7f7d1400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:07.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.816+0000 7f7d0f7fe700 1 -- 172.21.15.3:0/1209210558 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1400ca10 con 0x7f7d18100640 2023-09-10T01:04:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.816+0000 7f7d0f7fe700 1 -- 172.21.15.3:0/1209210558 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d1400cb70 con 0x7f7d18100640 2023-09-10T01:04:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.816+0000 7f7d0f7fe700 1 -- 172.21.15.3:0/1209210558 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14018680 con 0x7f7d18100640 2023-09-10T01:04:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.816+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d181a51a0 con 0x7f7d18100640 2023-09-10T01:04:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.816+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d181a5630 con 0x7f7d18100640 2023-09-10T01:04:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.822+0000 7f7d0f7fe700 1 -- 172.21.15.3:0/1209210558 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f7d1400cce0 con 0x7f7d18100640 2023-09-10T01:04:07.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.823+0000 7f7d33e47b80 1 Processor -- start 2023-09-10T01:04:07.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d33e47b80 1 -- start start 2023-09-10T01:04:07.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 0x5634f2d6faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d79900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 0x5634f2d7c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2cc8730 con 0x5634f2d79e40 2023-09-10T01:04:07.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2cc6050 con 0x5634f2d6f710 2023-09-10T01:04:07.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2cc1940 con 0x5634f2d70030 2023-09-10T01:04:07.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.824+0000 7f7d0f7fe700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7d040a09d0 0x7f7d040a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d79900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d79900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58548/0 (socket says 172.21.15.3:58548) 2023-09-10T01:04:07.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 -- 172.21.15.3:0/515013741 learned_addr learned my addr 172.21.15.3:0/515013741 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:07.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d0f7fe700 1 -- 172.21.15.3:0/1209210558 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7d14014070 con 0x7f7d18100640 2023-09-10T01:04:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 -- 172.21.15.3:0/515013741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 msgr2=0x5634f2d6faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/515013741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 0x5634f2d6faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 -- 172.21.15.3:0/515013741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 msgr2=0x5634f2d7c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/515013741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 0x5634f2d7c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.825+0000 7f7d1e39b700 1 -- 172.21.15.3:0/515013741 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634f2aedde0 con 0x5634f2d70030 2023-09-10T01:04:07.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.826+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/515013741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d79900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d14096030 tx=0x7f7d14010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10a4a813119aa137 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:07.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.826+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7d040a09d0 0x7f7d040a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.826+0000 7f7d0cbfe700 1 -- 172.21.15.3:0/515013741 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14098f20 con 0x5634f2d70030 2023-09-10T01:04:07.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.826+0000 7f7d0cbfe700 1 -- 172.21.15.3:0/515013741 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d14099cc0 con 0x5634f2d70030 2023-09-10T01:04:07.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.826+0000 7f7d0cbfe700 1 -- 172.21.15.3:0/515013741 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14084070 con 0x5634f2d70030 2023-09-10T01:04:07.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 -- 172.21.15.3:0/515013741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 msgr2=0x5634f2d79900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7d040a09d0 0x7f7d040a2e90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d18074a10 tx=0x7f7d1000b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:07.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/515013741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d79900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d14096030 tx=0x7f7d14010c80 comp rx=0 tx=0).stop 2023-09-10T01:04:07.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 -- 172.21.15.3:0/515013741 shutdown_connections 2023-09-10T01:04:07.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/515013741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 0x5634f2d7c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/515013741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 0x5634f2d6faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/515013741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d79900 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 -- 172.21.15.3:0/515013741 >> 172.21.15.3:0/515013741 conn(0x5634f2ce4090 msgr2=0x5634f2d6abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:07.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 -- 172.21.15.3:0/515013741 shutdown_connections 2023-09-10T01:04:07.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.827+0000 7f7d33e47b80 1 -- 172.21.15.3:0/515013741 wait complete. 2023-09-10T01:04:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.828+0000 7f7d33e47b80 1 -- start start 2023-09-10T01:04:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 1 Processor -- start 2023-09-10T01:04:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d6f710 0x5634f2d76b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d70030 0x5634f2d770a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d775e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2c5fb70 con 0x5634f2d70030 2023-09-10T01:04:07.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2c5f870 con 0x5634f2d79e40 2023-09-10T01:04:07.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2b43920 con 0x5634f2d6f710 2023-09-10T01:04:07.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.829+0000 7f7d33e47b80 10 client.0 _ll_register_callbacks cb 0x5634f2ba9a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:07.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.830+0000 7f7d1eb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d775e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.830+0000 7f7d1eb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d775e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55202/0 (socket says 172.21.15.3:55202) 2023-09-10T01:04:07.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.830+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/412239014 learned_addr learned my addr 172.21.15.3:0/412239014 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:07.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/412239014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d6f710 msgr2=0x5634f2d76b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:04:07.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d6f710 0x5634f2d76b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/412239014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d70030 msgr2=0x5634f2d770a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:07.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d70030 0x5634f2d770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/412239014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d181a5630 con 0x5634f2d79e40 2023-09-10T01:04:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d775e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d10016a40 tx=0x7f7d10004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16375708bdb36c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d01ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.831+0000 7f7d01ffb700 1 -- 172.21.15.3:0/412239014 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10015640 con 0x5634f2d79e40 2023-09-10T01:04:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.832+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5634f2c194f0 con 0x5634f2d79e40 2023-09-10T01:04:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.832+0000 7f7d01ffb700 1 -- 172.21.15.3:0/412239014 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7d100157a0 con 0x5634f2d79e40 2023-09-10T01:04:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.832+0000 7f7d01ffb700 1 -- 172.21.15.3:0/412239014 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d10015960 con 0x5634f2d79e40 2023-09-10T01:04:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.832+0000 7f7d33e47b80 10 client.25949 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.832+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5634f2ad6c60 con 0x5634f2d79e40 2023-09-10T01:04:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.832+0000 7f7d01ffb700 1 -- 172.21.15.3:0/412239014 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5634f2c194f0 con 0x5634f2d79e40 2023-09-10T01:04:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.833+0000 7f7d01ffb700 1 -- 172.21.15.3:0/412239014 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7d10021ca0 con 0x5634f2d79e40 2023-09-10T01:04:07.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.833+0000 7f7d33e47b80 10 client.25949 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:07.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.833+0000 7f7d33e47b80 10 client.25949 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:04:07.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.833+0000 7f7d33e47b80 20 client.25949 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.833+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2d89a00 0x5634f2d8be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.833+0000 7f7d33e47b80 4 client.25949 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.833+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634f2cd68a0 con 0x5634f2d89a00 2023-09-10T01:04:07.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.834+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2d89a00 0x5634f2d8be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.834+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2d89a00 0x5634f2d8be40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e52925384c06dfc server_cookie=c17301cb452cbc99 in_seq=0 out_seq=0 2023-09-10T01:04:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.834+0000 7f7d01ffb700 10 client.25949 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d01ffb700 1 -- 172.21.15.3:0/412239014 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5634f2cd68a0 con 0x5634f2d89a00 2023-09-10T01:04:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d01ffb700 10 client.25949 handle_command_reply: tid=0 2023-09-10T01:04:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 client.25949 shutdown 2023-09-10T01:04:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 msgr2=0x5634f2d775e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d775e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d10016a40 tx=0x7f7d10004ad0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 shutdown_connections 2023-09-10T01:04:07.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2d89a00 0x5634f2d8be40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d6f710 0x5634f2d76b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d775e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/412239014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d70030 0x5634f2d770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 >> 172.21.15.3:0/412239014 conn(0x5634f2ce4090 msgr2=0x5634f2d6a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.835+0000 7f7d01ffb700 0 client.25949 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.836+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 shutdown_connections 2023-09-10T01:04:07.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.836+0000 7f7d33e47b80 1 -- 172.21.15.3:0/412239014 wait complete. 2023-09-10T01:04:07.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.836+0000 7f7d33e47b80 20 client.25949 trim_cache size 0 max 16384 2023-09-10T01:04:07.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.862+0000 7f7d33e47b80 1 Processor -- start 2023-09-10T01:04:07.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.863+0000 7f7d33e47b80 1 -- start start 2023-09-10T01:04:07.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.863+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d6f710 0x5634f2d77410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.863+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d6a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.863+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d71e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.863+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2d79c10 con 0x5634f2d6f710 2023-09-10T01:04:07.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.863+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2d6b170 con 0x5634f2d79e40 2023-09-10T01:04:07.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.863+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2d64d70 con 0x5634f2d70030 2023-09-10T01:04:07.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d71e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1e39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d6f710 0x5634f2d77410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d71e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55210/0 (socket says 172.21.15.3:55210) 2023-09-10T01:04:07.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/513737136 learned_addr learned my addr 172.21.15.3:0/513737136 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:07.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1db9a700 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d6a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/513737136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 msgr2=0x5634f2d6a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:07.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d6a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/513737136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d6f710 msgr2=0x5634f2d77410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:07.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d6f710 0x5634f2d77410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.864+0000 7f7d1eb9c700 1 -- 172.21.15.3:0/513737136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634f2b14250 con 0x5634f2d79e40 2023-09-10T01:04:07.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.865+0000 7f7d1eb9c700 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d71e40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d10016fb0 tx=0x7f7d10009270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ca5e35a2b650b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:07.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.865+0000 7f7d0d3ff700 1 -- 172.21.15.3:0/513737136 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1002a620 con 0x5634f2d79e40 2023-09-10T01:04:07.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.865+0000 7f7d0d3ff700 1 -- 172.21.15.3:0/513737136 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d1001a070 con 0x5634f2d79e40 2023-09-10T01:04:07.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.865+0000 7f7d0d3ff700 1 -- 172.21.15.3:0/513737136 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10015ce0 con 0x5634f2d79e40 2023-09-10T01:04:07.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 -- 172.21.15.3:0/513737136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 msgr2=0x5634f2d71e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d71e40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d10016fb0 tx=0x7f7d10009270 comp rx=0 tx=0).stop 2023-09-10T01:04:07.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 -- 172.21.15.3:0/513737136 shutdown_connections 2023-09-10T01:04:07.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d79e40 0x5634f2d71e40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d6f710 0x5634f2d77410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/513737136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2d6a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 -- 172.21.15.3:0/513737136 >> 172.21.15.3:0/513737136 conn(0x5634f2ce4090 msgr2=0x5634f2d850e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 -- 172.21.15.3:0/513737136 shutdown_connections 2023-09-10T01:04:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.866+0000 7f7d33e47b80 1 -- 172.21.15.3:0/513737136 wait complete. 2023-09-10T01:04:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.867+0000 7f7d33e47b80 1 -- start start 2023-09-10T01:04:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.867+0000 7f7d33e47b80 1 Processor -- start 2023-09-10T01:04:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 0x5634f2dd8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2dd91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d33e47b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 0x5634f2d7e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2d7eff0 con 0x5634f2d79e40 2023-09-10T01:04:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2d7f160 con 0x5634f2d6f710 2023-09-10T01:04:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d33e47b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f2d7f320 con 0x5634f2d70030 2023-09-10T01:04:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d33e47b80 10 client.0 _ll_register_callbacks cb 0x7f7d10005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d1db9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2dd91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d1eb9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 0x5634f2d7e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d1db9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2dd91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58576/0 (socket says 172.21.15.3:58576) 2023-09-10T01:04:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.868+0000 7f7d1db9a700 1 -- 172.21.15.3:0/1845064392 learned_addr learned my addr 172.21.15.3:0/1845064392 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.869+0000 7f7d1db9a700 1 -- 172.21.15.3:0/1845064392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 msgr2=0x5634f2dd8c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:04:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.869+0000 7f7d1db9a700 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 0x5634f2dd8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.869+0000 7f7d1db9a700 1 -- 172.21.15.3:0/1845064392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 msgr2=0x5634f2d7e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.869+0000 7f7d1db9a700 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 0x5634f2d7e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.869+0000 7f7d1db9a700 1 -- 172.21.15.3:0/1845064392 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634f2ad6c60 con 0x5634f2d70030 2023-09-10T01:04:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.869+0000 7f7d1db9a700 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2dd91a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d080098e0 tx=0x7f7d08005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d10321c90556dfa server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.869+0000 7f7d01ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.870+0000 7f7d01ffb700 1 -- 172.21.15.3:0/1845064392 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0801d070 con 0x5634f2d70030 2023-09-10T01:04:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.870+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d14099cc0 con 0x5634f2d70030 2023-09-10T01:04:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.870+0000 7f7d01ffb700 1 -- 172.21.15.3:0/1845064392 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7d08004910 con 0x5634f2d70030 2023-09-10T01:04:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.870+0000 7f7d01ffb700 1 -- 172.21.15.3:0/1845064392 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d08004ad0 con 0x5634f2d70030 2023-09-10T01:04:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.870+0000 7f7d33e47b80 10 client.25879 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.870+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5634f2aec2b0 con 0x5634f2d70030 2023-09-10T01:04:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.870+0000 7f7d01ffb700 1 -- 172.21.15.3:0/1845064392 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d14099cc0 con 0x5634f2d70030 2023-09-10T01:04:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d01ffb700 1 -- 172.21.15.3:0/1845064392 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7d0800fc80 con 0x5634f2d70030 2023-09-10T01:04:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d33e47b80 10 client.25879 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d33e47b80 10 client.25879 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:04:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d33e47b80 20 client.25879 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2dd5bd0 0x5634f2dd5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d33e47b80 4 client.25879 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:04:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5634f2cba920 con 0x5634f2dd5bd0 2023-09-10T01:04:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.871+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2dd5bd0 0x5634f2dd5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.872+0000 7f7d1e39b700 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2dd5bd0 0x5634f2dd5fe0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57276e7bdef0c3d7 server_cookie=1d9022534d3b069d in_seq=0 out_seq=0 2023-09-10T01:04:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.872+0000 7f7d01ffb700 10 client.25879 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.872+0000 7f7d01ffb700 1 -- 172.21.15.3:0/1845064392 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5634f2cba920 con 0x5634f2dd5bd0 2023-09-10T01:04:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.872+0000 7f7d01ffb700 10 client.25879 handle_command_reply: tid=0 2023-09-10T01:04:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.872+0000 7f7d33e47b80 1 client.25879 shutdown 2023-09-10T01:04:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.872+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 msgr2=0x5634f2dd91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.872+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2dd91a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d080098e0 tx=0x7f7d08005250 comp rx=0 tx=0).stop 2023-09-10T01:04:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 shutdown_connections 2023-09-10T01:04:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5634f2dd5bd0 0x5634f2dd5fe0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634f2d79e40 0x5634f2d7e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634f2d6f710 0x5634f2dd8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 --2- 172.21.15.3:0/1845064392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634f2d70030 0x5634f2dd91a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 >> 172.21.15.3:0/1845064392 conn(0x5634f2ce4090 msgr2=0x5634f2d6a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d01ffb700 0 client.25879 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 shutdown_connections 2023-09-10T01:04:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 1 -- 172.21.15.3:0/1845064392 wait complete. 2023-09-10T01:04:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.873+0000 7f7d33e47b80 20 client.25879 trim_cache size 0 max 16384 2023-09-10T01:04:07.885 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:04:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7d040a09d0 msgr2=0x7f7d040a2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7d040a09d0 0x7f7d040a2e90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7d18074a10 tx=0x7f7d1000b040 comp rx=0 tx=0).stop 2023-09-10T01:04:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 msgr2=0x7f7d18072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18072900 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400e9a0 tx=0x7f7d1400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 shutdown_connections 2023-09-10T01:04:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7d040a09d0 0x7f7d040a2e90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d18106650 0x7f7d181a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d18100640 0x7f7d18072900 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 --2- 172.21.15.3:0/1209210558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d18101000 0x7f7d18072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:07.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 >> 172.21.15.3:0/1209210558 conn(0x7f7d180fc170 msgr2=0x7f7d18104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:07.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 shutdown_connections 2023-09-10T01:04:07.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:07.874+0000 7f7d205ff700 1 -- 172.21.15.3:0/1209210558 wait complete. 2023-09-10T01:04:07.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:04:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.110+0000 7f13093ff700 1 -- 172.21.15.3:0/1862138780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 msgr2=0x7f130410f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.111 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.110+0000 7f13093ff700 1 --2- 172.21.15.3:0/1862138780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 0x7f130410f250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00b0a0 tx=0x7f12fc00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.110+0000 7f13093ff700 1 -- 172.21.15.3:0/1862138780 shutdown_connections 2023-09-10T01:04:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.110+0000 7f13093ff700 1 --2- 172.21.15.3:0/1862138780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 0x7f130410f250 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.110+0000 7f13093ff700 1 --2- 172.21.15.3:0/1862138780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1304072a30 0x7f1304072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.110+0000 7f13093ff700 1 --2- 172.21.15.3:0/1862138780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 0x7f130410c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.110+0000 7f13093ff700 1 -- 172.21.15.3:0/1862138780 >> 172.21.15.3:0/1862138780 conn(0x7f13040fc0b0 msgr2=0x7f13040fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.111+0000 7f13093ff700 1 -- 172.21.15.3:0/1862138780 shutdown_connections 2023-09-10T01:04:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.111+0000 7f13093ff700 1 -- 172.21.15.3:0/1862138780 wait complete. 2023-09-10T01:04:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.112+0000 7f13093ff700 1 Processor -- start 2023-09-10T01:04:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.112+0000 7f13093ff700 1 -- start start 2023-09-10T01:04:08.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.113+0000 7f13093ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1304072a30 0x7f130419e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.113+0000 7f13093ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 0x7f130419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.113+0000 7f13093ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 0x7f1304198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.113+0000 7f13093ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1304198e80 con 0x7f1304072a30 2023-09-10T01:04:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.113+0000 7f13093ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1304198ff0 con 0x7f1304073350 2023-09-10T01:04:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.113+0000 7f13093ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13041991b0 con 0x7f130410cde0 2023-09-10T01:04:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 0x7f130419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 0x7f130419ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55244/0 (socket says 172.21.15.3:55244) 2023-09-10T01:04:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 -- 172.21.15.3:0/2695718162 learned_addr learned my addr 172.21.15.3:0/2695718162 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13037fe700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 0x7f1304198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f1302ffd700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1304072a30 0x7f130419e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 -- 172.21.15.3:0/2695718162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 msgr2=0x7f1304198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 0x7f1304198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 -- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1304072a30 msgr2=0x7f130419e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1304072a30 0x7f130419e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.114+0000 7f13027fc700 1 -- 172.21.15.3:0/2695718162 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc009d00 con 0x7f1304073350 2023-09-10T01:04:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.115+0000 7f13027fc700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 0x7f130419ece0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e9a0 tx=0x7f12f800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.115+0000 7f12ebfff700 1 -- 172.21.15.3:0/2695718162 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f800cbd0 con 0x7f1304073350 2023-09-10T01:04:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.115+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1304199490 con 0x7f1304073350 2023-09-10T01:04:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.115+0000 7f12ebfff700 1 -- 172.21.15.3:0/2695718162 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12f800cd30 con 0x7f1304073350 2023-09-10T01:04:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.115+0000 7f12ebfff700 1 -- 172.21.15.3:0/2695718162 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8018680 con 0x7f1304073350 2023-09-10T01:04:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.115+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13041a5790 con 0x7f1304073350 2023-09-10T01:04:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.122+0000 7f131ccbcb80 1 Processor -- start 2023-09-10T01:04:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.122+0000 7f12ebfff700 1 -- 172.21.15.3:0/2695718162 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f12f80187e0 con 0x7f1304073350 2023-09-10T01:04:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.123+0000 7f131ccbcb80 1 -- start start 2023-09-10T01:04:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.123+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5591645599b0 0x559164559d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.123+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559164553b10 0x559164553f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.123+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164560740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.123+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591644a46d0 con 0x559164553b10 2023-09-10T01:04:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.123+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591644a1ff0 con 0x5591645544d0 2023-09-10T01:04:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.123+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916449d8e0 con 0x5591645599b0 2023-09-10T01:04:08.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.124+0000 7f13037fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5591645599b0 0x559164559d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.124+0000 7f13037fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5591645599b0 0x559164559d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58602/0 (socket says 172.21.15.3:58602) 2023-09-10T01:04:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.124+0000 7f13037fe700 1 -- 172.21.15.3:0/1622328613 learned_addr learned my addr 172.21.15.3:0/1622328613 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.124+0000 7f13027fc700 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164560740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f1302ffd700 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559164553b10 0x559164553f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f13037fe700 1 -- 172.21.15.3:0/1622328613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 msgr2=0x559164560740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f13037fe700 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164560740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f13037fe700 1 -- 172.21.15.3:0/1622328613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559164553b10 msgr2=0x559164553f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f12ebfff700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12ec0a09d0 0x7f12ec0a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f13037fe700 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559164553b10 0x559164553f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f13037fe700 1 -- 172.21.15.3:0/1622328613 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591642c9d80 con 0x5591645599b0 2023-09-10T01:04:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f1302ffd700 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559164553b10 0x559164553f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:04:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.125+0000 7f12ebfff700 1 -- 172.21.15.3:0/2695718162 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f12f8014070 con 0x7f1304073350 2023-09-10T01:04:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.126+0000 7f13027fc700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12ec0a09d0 0x7f12ec0a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.126+0000 7f13037fe700 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5591645599b0 0x559164559d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00b0a0 tx=0x7f12fc009c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cef47438ea32776 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.126+0000 7f12e97fa700 1 -- 172.21.15.3:0/1622328613 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00e050 con 0x5591645599b0 2023-09-10T01:04:08.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.126+0000 7f12e97fa700 1 -- 172.21.15.3:0/1622328613 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12fc009460 con 0x5591645599b0 2023-09-10T01:04:08.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.126+0000 7f12e97fa700 1 -- 172.21.15.3:0/1622328613 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0052a0 con 0x5591645599b0 2023-09-10T01:04:08.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.127+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1622328613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5591645599b0 msgr2=0x559164559d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.127+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5591645599b0 0x559164559d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00b0a0 tx=0x7f12fc009c60 comp rx=0 tx=0).stop 2023-09-10T01:04:08.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.127+0000 7f13027fc700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12ec0a09d0 0x7f12ec0a2e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12f80c9b00 tx=0x7f12f80d2530 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.128+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1622328613 shutdown_connections 2023-09-10T01:04:08.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.128+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5591645599b0 0x559164559d90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.128+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164560740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.128+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1622328613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559164553b10 0x559164553f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.128+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1622328613 >> 172.21.15.3:0/1622328613 conn(0x5591644c0030 msgr2=0x559164546560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.128+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1622328613 shutdown_connections 2023-09-10T01:04:08.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.128+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1622328613 wait complete. 2023-09-10T01:04:08.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.129+0000 7f131ccbcb80 1 -- start start 2023-09-10T01:04:08.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.129+0000 7f131ccbcb80 1 Processor -- start 2023-09-10T01:04:08.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.130+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x5591645605e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.130+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x55916455d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.130+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x55916455de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.130+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916443bb10 con 0x5591645544d0 2023-09-10T01:04:08.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.130+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916443b810 con 0x5591645599b0 2023-09-10T01:04:08.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.130+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916431f8c0 con 0x559164553b10 2023-09-10T01:04:08.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.130+0000 7f131ccbcb80 10 client.0 _ll_register_callbacks cb 0x559164385a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:08.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.131+0000 7f13037fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x55916455de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.131+0000 7f13037fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x5591645605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.131+0000 7f13037fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x5591645605e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58614/0 (socket says 172.21.15.3:58614) 2023-09-10T01:04:08.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.131+0000 7f13037fe700 1 -- 172.21.15.3:0/1666518507 learned_addr learned my addr 172.21.15.3:0/1666518507 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.131+0000 7f1302ffd700 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x55916455d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f13037fe700 1 -- 172.21.15.3:0/1666518507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 msgr2=0x55916455de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f13037fe700 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x55916455de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f13037fe700 1 -- 172.21.15.3:0/1666518507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 msgr2=0x55916455d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f13037fe700 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x55916455d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f13037fe700 1 -- 172.21.15.3:0/1666518507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130410ff10 con 0x559164553b10 2023-09-10T01:04:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f1302ffd700 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x55916455d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:04:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f13037fe700 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x5591645605e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0076c0 tx=0x7f12fc00b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cafa4044bb4264a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f12e6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f12e6ff5700 1 -- 172.21.15.3:0/1666518507 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00e070 con 0x559164553b10 2023-09-10T01:04:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5591643f5490 con 0x559164553b10 2023-09-10T01:04:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.132+0000 7f12e6ff5700 1 -- 172.21.15.3:0/1666518507 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f12fc004c60 con 0x559164553b10 2023-09-10T01:04:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.133+0000 7f12e6ff5700 1 -- 172.21.15.3:0/1666518507 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12fc004e20 con 0x559164553b10 2023-09-10T01:04:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.133+0000 7f131ccbcb80 10 client.25891 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.133+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5591642b2c00 con 0x559164553b10 2023-09-10T01:04:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.133+0000 7f12e6ff5700 1 -- 172.21.15.3:0/1666518507 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5591643f5490 con 0x559164553b10 2023-09-10T01:04:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.133+0000 7f12e6ff5700 1 -- 172.21.15.3:0/1666518507 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f12fc00b0d0 con 0x559164553b10 2023-09-10T01:04:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.134+0000 7f131ccbcb80 10 client.25891 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.134+0000 7f131ccbcb80 10 client.25891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.134+0000 7f131ccbcb80 20 client.25891 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.134+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55916456d760 0x55916456fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.134+0000 7f131ccbcb80 4 client.25891 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.134+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591644b2840 con 0x55916456d760 2023-09-10T01:04:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.134+0000 7f1302ffd700 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55916456d760 0x55916456fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.135+0000 7f1302ffd700 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55916456d760 0x55916456fb80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f0e79cce2837dc4 server_cookie=4dece43b3f458095 in_seq=0 out_seq=0 2023-09-10T01:04:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.135+0000 7f12e6ff5700 10 client.25891 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f12e6ff5700 1 -- 172.21.15.3:0/1666518507 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5591644b2840 con 0x55916456d760 2023-09-10T01:04:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f12e6ff5700 10 client.25891 handle_command_reply: tid=0 2023-09-10T01:04:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 client.25891 shutdown 2023-09-10T01:04:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 msgr2=0x5591645605e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x5591645605e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0076c0 tx=0x7f12fc00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 shutdown_connections 2023-09-10T01:04:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55916456d760 0x55916456fb80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x5591645605e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x55916455de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/1666518507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x55916455d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 >> 172.21.15.3:0/1666518507 conn(0x5591644c0030 msgr2=0x5591645457a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f12e6ff5700 0 client.25891 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 shutdown_connections 2023-09-10T01:04:08.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.136+0000 7f131ccbcb80 1 -- 172.21.15.3:0/1666518507 wait complete. 2023-09-10T01:04:08.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.137+0000 7f131ccbcb80 20 client.25891 trim_cache size 0 max 16384 2023-09-10T01:04:08.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.166+0000 7f131ccbcb80 1 Processor -- start 2023-09-10T01:04:08.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.167+0000 7f131ccbcb80 1 -- start start 2023-09-10T01:04:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164560180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164546090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645599b0 0x5591645465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591644b2840 con 0x5591645599b0 2023-09-10T01:04:08.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591643f5490 con 0x5591645544d0 2023-09-10T01:04:08.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559164547140 con 0x559164553b10 2023-09-10T01:04:08.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f13037fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164560180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.168+0000 7f1302ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164546090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164546090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55264/0 (socket says 172.21.15.3:55264) 2023-09-10T01:04:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f13037fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645599b0 0x5591645465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 -- 172.21.15.3:0/4015080415 learned_addr learned my addr 172.21.15.3:0/4015080415 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 -- 172.21.15.3:0/4015080415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 msgr2=0x559164560180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 --2- 172.21.15.3:0/4015080415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164560180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 -- 172.21.15.3:0/4015080415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645599b0 msgr2=0x5591645465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 --2- 172.21.15.3:0/4015080415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645599b0 0x5591645465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 -- 172.21.15.3:0/4015080415 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591642f01f0 con 0x5591645544d0 2023-09-10T01:04:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.169+0000 7f1302ffd700 1 --2- 172.21.15.3:0/4015080415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164546090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f12f4000c00 tx=0x7f12f4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a5fbf093e8a2dd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.170+0000 7f12e9ffb700 1 -- 172.21.15.3:0/4015080415 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f4017070 con 0x5591645544d0 2023-09-10T01:04:08.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.170+0000 7f12e9ffb700 1 -- 172.21.15.3:0/4015080415 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12f4018040 con 0x5591645544d0 2023-09-10T01:04:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.170+0000 7f12e9ffb700 1 -- 172.21.15.3:0/4015080415 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f40055d0 con 0x5591645544d0 2023-09-10T01:04:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 -- 172.21.15.3:0/4015080415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 msgr2=0x559164546090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/4015080415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164546090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f12f4000c00 tx=0x7f12f4009260 comp rx=0 tx=0).stop 2023-09-10T01:04:08.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 -- 172.21.15.3:0/4015080415 shutdown_connections 2023-09-10T01:04:08.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/4015080415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164560180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/4015080415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645599b0 0x5591645465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/4015080415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645544d0 0x559164546090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 -- 172.21.15.3:0/4015080415 >> 172.21.15.3:0/4015080415 conn(0x5591644c0030 msgr2=0x559164558850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 -- 172.21.15.3:0/4015080415 shutdown_connections 2023-09-10T01:04:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.171+0000 7f131ccbcb80 1 -- 172.21.15.3:0/4015080415 wait complete. 2023-09-10T01:04:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.172+0000 7f131ccbcb80 1 -- start start 2023-09-10T01:04:08.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.172+0000 7f131ccbcb80 1 Processor -- start 2023-09-10T01:04:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164564b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x559164563040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f131ccbcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x559164563580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559164563c00 con 0x5591645544d0 2023-09-10T01:04:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559164563d70 con 0x5591645599b0 2023-09-10T01:04:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f131ccbcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591645c15a0 con 0x559164553b10 2023-09-10T01:04:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f131ccbcb80 10 client.0 _ll_register_callbacks cb 0x7f12fc009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f13037fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164564b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f13037fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164564b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58618/0 (socket says 172.21.15.3:58618) 2023-09-10T01:04:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f13037fe700 1 -- 172.21.15.3:0/2370461339 learned_addr learned my addr 172.21.15.3:0/2370461339 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.173+0000 7f13037fe700 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x559164563580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f1302ffd700 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x559164563040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f13037fe700 1 -- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 msgr2=0x559164563580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f13037fe700 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x559164563580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f13037fe700 1 -- 172.21.15.3:0/2370461339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 msgr2=0x559164563040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f13037fe700 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x559164563040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f13037fe700 1 -- 172.21.15.3:0/2370461339 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f4004580 con 0x559164553b10 2023-09-10T01:04:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f13037fe700 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164564b70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0061e0 tx=0x7f12fc01d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d464916bdb7fb792 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.174+0000 7f12e6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.175+0000 7f12e6ff5700 1 -- 172.21.15.3:0/2370461339 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00e070 con 0x559164553b10 2023-09-10T01:04:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.175+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12fc009460 con 0x559164553b10 2023-09-10T01:04:08.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.175+0000 7f12e6ff5700 1 -- 172.21.15.3:0/2370461339 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f12fc004c60 con 0x559164553b10 2023-09-10T01:04:08.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:08 smithi003 ceph-mon[97654]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 19 op/s 2023-09-10T01:04:08.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.181+0000 7f12e6ff5700 1 -- 172.21.15.3:0/2370461339 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12fc01bae0 con 0x559164553b10 2023-09-10T01:04:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.181+0000 7f131ccbcb80 10 client.25897 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.181+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5591642c8250 con 0x559164553b10 2023-09-10T01:04:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.181+0000 7f12e6ff5700 1 -- 172.21.15.3:0/2370461339 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12fc009460 con 0x559164553b10 2023-09-10T01:04:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.181+0000 7f12e6ff5700 1 -- 172.21.15.3:0/2370461339 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f12fc00b420 con 0x559164553b10 2023-09-10T01:04:08.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.181+0000 7f131ccbcb80 10 client.25897 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.181+0000 7f131ccbcb80 10 client.25897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.182+0000 7f131ccbcb80 20 client.25897 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.182+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5591645c2740 0x5591645c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.182+0000 7f131ccbcb80 4 client.25897 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:04:08.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.182+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5591644968c0 con 0x5591645c2740 2023-09-10T01:04:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.182+0000 7f1302ffd700 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5591645c2740 0x5591645c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.183+0000 7f1302ffd700 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5591645c2740 0x5591645c4b60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=854fd60d639edb0b server_cookie=f9a2f0c48a51f880 in_seq=0 out_seq=0 2023-09-10T01:04:08.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.183+0000 7f12e6ff5700 10 client.25897 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f12e6ff5700 1 -- 172.21.15.3:0/2370461339 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5591644968c0 con 0x5591645c2740 2023-09-10T01:04:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f12e6ff5700 10 client.25897 handle_command_reply: tid=0 2023-09-10T01:04:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 client.25897 shutdown 2023-09-10T01:04:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 msgr2=0x559164564b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164564b70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0061e0 tx=0x7f12fc01d040 comp rx=0 tx=0).stop 2023-09-10T01:04:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 shutdown_connections 2023-09-10T01:04:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5591645c2740 0x5591645c4b60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559164553b10 0x559164564b70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f12e6ff5700 0 client.25897 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5591645599b0 0x559164563580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 --2- 172.21.15.3:0/2370461339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5591645544d0 0x559164563040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.190+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 >> 172.21.15.3:0/2370461339 conn(0x5591644c0030 msgr2=0x559164558850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.191+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 shutdown_connections 2023-09-10T01:04:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.191+0000 7f131ccbcb80 1 -- 172.21.15.3:0/2370461339 wait complete. 2023-09-10T01:04:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.191+0000 7f131ccbcb80 20 client.25897 trim_cache size 0 max 16384 2023-09-10T01:04:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:04:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:04:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:04:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:04:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12ec0a09d0 msgr2=0x7f12ec0a2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12ec0a09d0 0x7f12ec0a2e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12f80c9b00 tx=0x7f12f80d2530 comp rx=0 tx=0).stop 2023-09-10T01:04:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 msgr2=0x7f130419ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 0x7f130419ece0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e9a0 tx=0x7f12f800ed60 comp rx=0 tx=0).stop 2023-09-10T01:04:08.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 shutdown_connections 2023-09-10T01:04:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12ec0a09d0 0x7f12ec0a2e90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f130410cde0 0x7f1304198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1304072a30 0x7f130419e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 --2- 172.21.15.3:0/2695718162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1304073350 0x7f130419ece0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 >> 172.21.15.3:0/2695718162 conn(0x7f13040fc0b0 msgr2=0x7f1304104840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.192+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 shutdown_connections 2023-09-10T01:04:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:08.193+0000 7f13093ff700 1 -- 172.21.15.3:0/2695718162 wait complete. 2023-09-10T01:04:08.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 -- 172.21.15.3:0/3782857653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4072a30 msgr2=0x7f34a4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 --2- 172.21.15.3:0/3782857653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4072a30 0x7f34a4072e10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3494009990 tx=0x7f3494009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 -- 172.21.15.3:0/3782857653 shutdown_connections 2023-09-10T01:04:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 --2- 172.21.15.3:0/3782857653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4072a30 0x7f34a4072e10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 --2- 172.21.15.3:0/3782857653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34a410ce80 0x7f34a410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 --2- 172.21.15.3:0/3782857653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34a4073350 0x7f34a410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 -- 172.21.15.3:0/3782857653 >> 172.21.15.3:0/3782857653 conn(0x7f34a40fc170 msgr2=0x7f34a40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 -- 172.21.15.3:0/3782857653 shutdown_connections 2023-09-10T01:04:08.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.256+0000 7f34a8fff700 1 -- 172.21.15.3:0/3782857653 wait complete. 2023-09-10T01:04:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.258+0000 7f34a8fff700 1 Processor -- start 2023-09-10T01:04:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.258+0000 7f34a8fff700 1 -- start start 2023-09-10T01:04:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a8fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34a4072a30 0x7f34a419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a8fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4073350 0x7f34a419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a8fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34a410ce80 0x7f34a4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a8fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4198ed0 con 0x7f34a4072a30 2023-09-10T01:04:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a8fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4199040 con 0x7f34a410ce80 2023-09-10T01:04:08.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a8fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4199200 con 0x7f34a4073350 2023-09-10T01:04:08.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a1d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4073350 0x7f34a419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34a410ce80 0x7f34a4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a1d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4073350 0x7f34a419ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58640/0 (socket says 172.21.15.3:58640) 2023-09-10T01:04:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a1d9b700 1 -- 172.21.15.3:0/1338584650 learned_addr learned my addr 172.21.15.3:0/1338584650 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.259+0000 7f34a259c700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34a4072a30 0x7f34a419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.260+0000 7f34a259c700 1 -- 172.21.15.3:0/1338584650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4073350 msgr2=0x7f34a419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.260+0000 7f34a259c700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4073350 0x7f34a419ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.260+0000 7f34a259c700 1 -- 172.21.15.3:0/1338584650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34a410ce80 msgr2=0x7f34a4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.260+0000 7f34a259c700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34a410ce80 0x7f34a4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.260+0000 7f34a259c700 1 -- 172.21.15.3:0/1338584650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34940095f0 con 0x7f34a4072a30 2023-09-10T01:04:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.260+0000 7f34a259c700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34a4072a30 0x7f34a419e7f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f34940098e0 tx=0x7f3494004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.261+0000 7f34937fe700 1 -- 172.21.15.3:0/1338584650 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349401d070 con 0x7f34a4072a30 2023-09-10T01:04:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.261+0000 7f34937fe700 1 -- 172.21.15.3:0/1338584650 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3494005030 con 0x7f34a4072a30 2023-09-10T01:04:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.261+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a4199480 con 0x7f34a4072a30 2023-09-10T01:04:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.261+0000 7f34937fe700 1 -- 172.21.15.3:0/1338584650 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3494004ab0 con 0x7f34a4072a30 2023-09-10T01:04:08.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.262+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a41a57e0 con 0x7f34a4072a30 2023-09-10T01:04:08.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.267+0000 7f34937fe700 1 -- 172.21.15.3:0/1338584650 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f3494005410 con 0x7f34a4072a30 2023-09-10T01:04:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.269+0000 7f34bc92cb80 1 Processor -- start 2023-09-10T01:04:08.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.269+0000 7f34bc92cb80 1 -- start start 2023-09-10T01:04:08.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.269+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d83eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d848900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d84b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d797730 con 0x55f93d83f030 2023-09-10T01:04:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d795050 con 0x55f93d83e710 2023-09-10T01:04:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d790940 con 0x55f93d848e40 2023-09-10T01:04:08.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34937fe700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f348c0a08d0 0x7f348c0a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34937fe700 1 -- 172.21.15.3:0/1338584650 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f34940cd2d0 con 0x7f34a4072a30 2023-09-10T01:04:08.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34a2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d83eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34a2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d83eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55308/0 (socket says 172.21.15.3:55308) 2023-09-10T01:04:08.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34a2d9d700 1 -- 172.21.15.3:0/3124412524 learned_addr learned my addr 172.21.15.3:0/3124412524 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.270+0000 7f34a259c700 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d848900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.271+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d84b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.271+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f348c0a08d0 0x7f348c0a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.271+0000 7f34a1d9b700 1 -- 172.21.15.3:0/3124412524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 msgr2=0x55f93d83eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.271+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d83eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.271+0000 7f34a1d9b700 1 -- 172.21.15.3:0/3124412524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 msgr2=0x55f93d848900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.271+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d848900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.271+0000 7f34a1d9b700 1 -- 172.21.15.3:0/3124412524 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f93d5bcde0 con 0x55f93d848e40 2023-09-10T01:04:08.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.272+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d84b230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3498005950 tx=0x7f349800bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37d79936f7ba2b7a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.272+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f348c0a08d0 0x7f348c0a2d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f349c009550 tx=0x7f349c00b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.272+0000 7f3490ff9700 1 -- 172.21.15.3:0/3124412524 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349801b070 con 0x55f93d848e40 2023-09-10T01:04:08.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.272+0000 7f3490ff9700 1 -- 172.21.15.3:0/3124412524 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34980044f0 con 0x55f93d848e40 2023-09-10T01:04:08.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.272+0000 7f3490ff9700 1 -- 172.21.15.3:0/3124412524 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349800d820 con 0x55f93d848e40 2023-09-10T01:04:08.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 -- 172.21.15.3:0/3124412524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 msgr2=0x55f93d84b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d84b230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3498005950 tx=0x7f349800bc20 comp rx=0 tx=0).stop 2023-09-10T01:04:08.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 -- 172.21.15.3:0/3124412524 shutdown_connections 2023-09-10T01:04:08.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d84b230 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d83eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/3124412524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d848900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 -- 172.21.15.3:0/3124412524 >> 172.21.15.3:0/3124412524 conn(0x55f93d7b3090 msgr2=0x55f93d839bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.273+0000 7f34bc92cb80 1 -- 172.21.15.3:0/3124412524 shutdown_connections 2023-09-10T01:04:08.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.274+0000 7f34bc92cb80 1 -- 172.21.15.3:0/3124412524 wait complete. 2023-09-10T01:04:08.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.274+0000 7f34bc92cb80 1 -- start start 2023-09-10T01:04:08.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.275+0000 7f34bc92cb80 1 Processor -- start 2023-09-10T01:04:08.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d848860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d845b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d72eb70 con 0x55f93d83f030 2023-09-10T01:04:08.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d72e870 con 0x55f93d83e710 2023-09-10T01:04:08.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d612920 con 0x55f93d848e40 2023-09-10T01:04:08.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34bc92cb80 10 client.0 _ll_register_callbacks cb 0x55f93d678a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:08.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34a1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d845b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34a259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d848860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34a259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d848860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55320/0 (socket says 172.21.15.3:55320) 2023-09-10T01:04:08.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34a259c700 1 -- 172.21.15.3:0/1289351380 learned_addr learned my addr 172.21.15.3:0/1289351380 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.276+0000 7f34a1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d845b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42240/0 (socket says 172.21.15.3:42240) 2023-09-10T01:04:08.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f34a259c700 1 -- 172.21.15.3:0/1289351380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 msgr2=0x55f93d8461f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:04:08.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f34a259c700 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8461f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f34a259c700 1 -- 172.21.15.3:0/1289351380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 msgr2=0x55f93d845b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f34a259c700 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d845b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f34a259c700 1 -- 172.21.15.3:0/1289351380 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3498016c30 con 0x55f93d83e710 2023-09-10T01:04:08.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d845b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:08.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f34a259c700 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d848860 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f34940095c0 tx=0x7f3494074d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f501fc20d69b91fe server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.277+0000 7f3489ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:08.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f3489ffb700 1 -- 172.21.15.3:0/1289351380 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349409a210 con 0x55f93d83e710 2023-09-10T01:04:08.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f3489ffb700 1 -- 172.21.15.3:0/1289351380 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3494074e40 con 0x55f93d83e710 2023-09-10T01:04:08.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f93d6e84f0 con 0x55f93d83e710 2023-09-10T01:04:08.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f3489ffb700 1 -- 172.21.15.3:0/1289351380 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f349409e040 con 0x55f93d83e710 2023-09-10T01:04:08.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f34bc92cb80 10 client.25982 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:08.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55f93d5a5c60 con 0x55f93d83e710 2023-09-10T01:04:08.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f3489ffb700 1 -- 172.21.15.3:0/1289351380 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f93d6e84f0 con 0x55f93d83e710 2023-09-10T01:04:08.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f3489ffb700 1 -- 172.21.15.3:0/1289351380 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3494099700 con 0x55f93d83e710 2023-09-10T01:04:08.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f34bc92cb80 10 client.25982 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:08.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f34bc92cb80 10 client.25982 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:04:08.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.278+0000 7f34bc92cb80 20 client.25982 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:08.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.279+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8584f0 0x55f93d85a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.279+0000 7f34bc92cb80 4 client.25982 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:08.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.279+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f93d7a58a0 con 0x55f93d8584f0 2023-09-10T01:04:08.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.279+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8584f0 0x55f93d85a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.280+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8584f0 0x55f93d85a900 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21ec5506bdd38d79 server_cookie=9678649e9a4c9734 in_seq=0 out_seq=0 2023-09-10T01:04:08.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.280+0000 7f3489ffb700 10 client.25982 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:08.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.280+0000 7f3489ffb700 1 -- 172.21.15.3:0/1289351380 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f93d7a58a0 con 0x55f93d8584f0 2023-09-10T01:04:08.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.280+0000 7f3489ffb700 10 client.25982 handle_command_reply: tid=0 2023-09-10T01:04:08.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.280+0000 7f34bc92cb80 1 client.25982 shutdown 2023-09-10T01:04:08.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 msgr2=0x55f93d848860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d848860 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f34940095c0 tx=0x7f3494074d10 comp rx=0 tx=0).stop 2023-09-10T01:04:08.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 shutdown_connections 2023-09-10T01:04:08.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8584f0 0x55f93d85a900 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83e710 0x55f93d848860 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1289351380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d845b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f3489ffb700 0 client.25982 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:08.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 >> 172.21.15.3:0/1289351380 conn(0x55f93d7b3090 msgr2=0x55f93d8397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 shutdown_connections 2023-09-10T01:04:08.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.281+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1289351380 wait complete. 2023-09-10T01:04:08.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.282+0000 7f34bc92cb80 20 client.25982 trim_cache size 0 max 16384 2023-09-10T01:04:08.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.308+0000 7f34bc92cb80 1 Processor -- start 2023-09-10T01:04:08.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.308+0000 7f34bc92cb80 1 -- start start 2023-09-10T01:04:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83e710 0x55f93d84b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83f030 0x55f93d84f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d84fed0 con 0x55f93d83e710 2023-09-10T01:04:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d83a6e0 con 0x55f93d83f030 2023-09-10T01:04:08.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d846170 con 0x55f93d848e40 2023-09-10T01:04:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34a259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83e710 0x55f93d84b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34a259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83e710 0x55f93d84b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42252/0 (socket says 172.21.15.3:42252) 2023-09-10T01:04:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34a259c700 1 -- 172.21.15.3:0/754749245 learned_addr learned my addr 172.21.15.3:0/754749245 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.309+0000 7f34a2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83f030 0x55f93d84f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a2d9d700 1 -- 172.21.15.3:0/754749245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83f030 msgr2=0x55f93d84f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83f030 0x55f93d84f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a2d9d700 1 -- 172.21.15.3:0/754749245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83e710 msgr2=0x55f93d84b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83e710 0x55f93d84b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a2d9d700 1 -- 172.21.15.3:0/754749245 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f93d5e3250 con 0x55f93d848e40 2023-09-10T01:04:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83f030 0x55f93d84f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8540e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f349c01aa50 tx=0x7f349c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359a715006a16b68 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.310+0000 7f34917fa700 1 -- 172.21.15.3:0/754749245 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c022070 con 0x55f93d848e40 2023-09-10T01:04:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.311+0000 7f34917fa700 1 -- 172.21.15.3:0/754749245 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f349c00f040 con 0x55f93d848e40 2023-09-10T01:04:08.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.311+0000 7f34917fa700 1 -- 172.21.15.3:0/754749245 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c00e070 con 0x55f93d848e40 2023-09-10T01:04:08.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.311+0000 7f34bc92cb80 1 -- 172.21.15.3:0/754749245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 msgr2=0x55f93d8540e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.311+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8540e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f349c01aa50 tx=0x7f349c009920 comp rx=0 tx=0).stop 2023-09-10T01:04:08.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 -- 172.21.15.3:0/754749245 shutdown_connections 2023-09-10T01:04:08.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d848e40 0x55f93d8540e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83e710 0x55f93d84b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/754749245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d83f030 0x55f93d84f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 -- 172.21.15.3:0/754749245 >> 172.21.15.3:0/754749245 conn(0x55f93d7b3090 msgr2=0x55f93d838320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 -- 172.21.15.3:0/754749245 shutdown_connections 2023-09-10T01:04:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 -- 172.21.15.3:0/754749245 wait complete. 2023-09-10T01:04:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.312+0000 7f34bc92cb80 1 -- start start 2023-09-10T01:04:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.313+0000 7f34bc92cb80 1 Processor -- start 2023-09-10T01:04:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.313+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d83e710 0x55f93d84f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.313+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d84da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34bc92cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d848e40 0x55f93d84df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d84e570 con 0x55f93d83f030 2023-09-10T01:04:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d84e6b0 con 0x55f93d848e40 2023-09-10T01:04:08.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34bc92cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f93d84e870 con 0x55f93d83e710 2023-09-10T01:04:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34bc92cb80 10 client.0 _ll_register_callbacks cb 0x7f3494099d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d848e40 0x55f93d84df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d83e710 0x55f93d84f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d848e40 0x55f93d84df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55348/0 (socket says 172.21.15.3:55348) 2023-09-10T01:04:08.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a2d9d700 1 -- 172.21.15.3:0/1860168728 learned_addr learned my addr 172.21.15.3:0/1860168728 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d84da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a2d9d700 1 -- 172.21.15.3:0/1860168728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d83e710 msgr2=0x55f93d84f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d83e710 0x55f93d84f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a2d9d700 1 -- 172.21.15.3:0/1860168728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 msgr2=0x55f93d84da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.314+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d84da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.315+0000 7f34a2d9d700 1 -- 172.21.15.3:0/1860168728 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c008e10 con 0x55f93d848e40 2023-09-10T01:04:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.315+0000 7f34a1d9b700 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d84da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.315+0000 7f34a2d9d700 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d848e40 0x55f93d84df50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f349c0091a0 tx=0x7f349c003ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ca28e63561935c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.315+0000 7f3489ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.315+0000 7f3489ffb700 1 -- 172.21.15.3:0/1860168728 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c022070 con 0x55f93d848e40 2023-09-10T01:04:08.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.315+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34980044f0 con 0x55f93d848e40 2023-09-10T01:04:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.315+0000 7f3489ffb700 1 -- 172.21.15.3:0/1860168728 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f349c007910 con 0x55f93d848e40 2023-09-10T01:04:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.316+0000 7f3489ffb700 1 -- 172.21.15.3:0/1860168728 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f349c007ad0 con 0x55f93d848e40 2023-09-10T01:04:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.316+0000 7f34bc92cb80 10 client.25988 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:08.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.316+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55f93d5bb2b0 con 0x55f93d848e40 2023-09-10T01:04:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.316+0000 7f3489ffb700 1 -- 172.21.15.3:0/1860168728 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34980044f0 con 0x55f93d848e40 2023-09-10T01:04:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.316+0000 7f3489ffb700 1 -- 172.21.15.3:0/1860168728 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f349c026410 con 0x55f93d848e40 2023-09-10T01:04:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34bc92cb80 10 client.25988 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:08.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34bc92cb80 10 client.25988 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:04:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34bc92cb80 20 client.25988 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8ad880 0x55f93d8afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34bc92cb80 4 client.25988 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:04:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f93d789920 con 0x55f93d8ad880 2023-09-10T01:04:08.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34a259c700 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8ad880 0x55f93d8afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.317+0000 7f34a259c700 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8ad880 0x55f93d8afca0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6aca6136ee40f4af server_cookie=112a5c1ca36188ff in_seq=0 out_seq=0 2023-09-10T01:04:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f3489ffb700 10 client.25988 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f3489ffb700 1 -- 172.21.15.3:0/1860168728 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f93d789920 con 0x55f93d8ad880 2023-09-10T01:04:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f3489ffb700 10 client.25988 handle_command_reply: tid=0 2023-09-10T01:04:08.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 client.25988 shutdown 2023-09-10T01:04:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d848e40 msgr2=0x55f93d84df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d848e40 0x55f93d84df50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f349c0091a0 tx=0x7f349c003ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 shutdown_connections 2023-09-10T01:04:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55f93d8ad880 0x55f93d8afca0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f93d83e710 0x55f93d84f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f93d848e40 0x55f93d84df50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 --2- 172.21.15.3:0/1860168728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f93d83f030 0x55f93d84da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 >> 172.21.15.3:0/1860168728 conn(0x55f93d7b3090 msgr2=0x55f93d840f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.318+0000 7f3489ffb700 0 client.25988 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:08.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.319+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 shutdown_connections 2023-09-10T01:04:08.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.319+0000 7f34bc92cb80 1 -- 172.21.15.3:0/1860168728 wait complete. 2023-09-10T01:04:08.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.319+0000 7f34bc92cb80 20 client.25988 trim_cache size 0 max 16384 2023-09-10T01:04:08.330 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:04:08.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f348c0a08d0 msgr2=0x7f348c0a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f348c0a08d0 0x7f348c0a2d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f349c009550 tx=0x7f349c00b040 comp rx=0 tx=0).stop 2023-09-10T01:04:08.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34a4072a30 msgr2=0x7f34a419e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34a4072a30 0x7f34a419e7f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f34940098e0 tx=0x7f3494004730 comp rx=0 tx=0).stop 2023-09-10T01:04:08.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 shutdown_connections 2023-09-10T01:04:08.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f348c0a08d0 0x7f348c0a2d90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34a410ce80 0x7f34a4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34a4072a30 0x7f34a419e7f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 --2- 172.21.15.3:0/1338584650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34a4073350 0x7f34a419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 >> 172.21.15.3:0/1338584650 conn(0x7f34a40fc170 msgr2=0x7f34a410f000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 shutdown_connections 2023-09-10T01:04:08.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.320+0000 7f34a8fff700 1 -- 172.21.15.3:0/1338584650 wait complete. 2023-09-10T01:04:08.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:04:08.454 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:08 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2805622463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:04:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:08 smithi092 ceph-mon[134067]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 19 op/s 2023-09-10T01:04:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:08 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2805622463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:04:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:08 smithi136 ceph-mon[133708]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 19 op/s 2023-09-10T01:04:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:08 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2805622463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:04:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/382566088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8100610 msgr2=0x7fc3d81009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/382566088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8100610 0x7fc3d81009f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009990 tx=0x7fc3c8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/382566088 shutdown_connections 2023-09-10T01:04:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/382566088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8105730 0x7fc3d810d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/382566088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8100610 0x7fc3d81009f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/382566088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8100fc0 0x7fc3d8105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/382566088 >> 172.21.15.3:0/382566088 conn(0x7fc3d80fbe90 msgr2=0x7fc3d80fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/382566088 shutdown_connections 2023-09-10T01:04:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.697+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/382566088 wait complete. 2023-09-10T01:04:08.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.699+0000 7fc3dcf9d700 1 Processor -- start 2023-09-10T01:04:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.699+0000 7fc3dcf9d700 1 -- start start 2023-09-10T01:04:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.699+0000 7fc3dcf9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8100610 0x7fc3d806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.699+0000 7fc3dcf9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8105730 0x7fc3d806f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3dcf9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8073a10 0x7fc3d8073ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3dcf9d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8074490 con 0x7fc3d8105730 2023-09-10T01:04:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3dcf9d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8074600 con 0x7fc3d8100610 2023-09-10T01:04:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3dcf9d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d80747c0 con 0x7fc3d8073a10 2023-09-10T01:04:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8100610 0x7fc3d806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3d7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8073a10 0x7fc3d8073ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3d7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8073a10 0x7fc3d8073ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58676/0 (socket says 172.21.15.3:58676) 2023-09-10T01:04:08.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3d7fff700 1 -- 172.21.15.3:0/3448126285 learned_addr learned my addr 172.21.15.3:0/3448126285 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.700+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8105730 0x7fc3d806f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d7fff700 1 -- 172.21.15.3:0/3448126285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8100610 msgr2=0x7fc3d806efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d7fff700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8100610 0x7fc3d806efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d7fff700 1 -- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8105730 msgr2=0x7fc3d806f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d7fff700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8105730 0x7fc3d806f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d7fff700 1 -- 172.21.15.3:0/3448126285 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80095f0 con 0x7fc3d8073a10 2023-09-10T01:04:08.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d7fff700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8073a10 0x7fc3d8073ec0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000ef30 tx=0x7fc3d000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d4ff9700 1 -- 172.21.15.3:0/3448126285 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0018070 con 0x7fc3d8073a10 2023-09-10T01:04:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d4ff9700 1 -- 172.21.15.3:0/3448126285 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3d000f040 con 0x7fc3d8073a10 2023-09-10T01:04:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3d4ff9700 1 -- 172.21.15.3:0/3448126285 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d000cce0 con 0x7fc3d8073a10 2023-09-10T01:04:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.701+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d81adc20 con 0x7fc3d8073a10 2023-09-10T01:04:08.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.702+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d81ae0b0 con 0x7fc3d8073a10 2023-09-10T01:04:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.707+0000 7fc3d4ff9700 1 -- 172.21.15.3:0/3448126285 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7fc3d0004ad0 con 0x7fc3d8073a10 2023-09-10T01:04:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.708+0000 7fc3f1b5eb80 1 Processor -- start 2023-09-10T01:04:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.708+0000 7fc3f1b5eb80 1 -- start start 2023-09-10T01:04:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.709+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08741860 0x557a08741c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.709+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0874caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.709+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a0874f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.709+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a0869b730 con 0x557a0874d030 2023-09-10T01:04:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.709+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a08699050 con 0x557a08742180 2023-09-10T01:04:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.709+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a08694940 con 0x557a08741860 2023-09-10T01:04:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.709+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0874caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0874caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55370/0 (socket says 172.21.15.3:55370) 2023-09-10T01:04:08.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 -- 172.21.15.3:0/1535190464 learned_addr learned my addr 172.21.15.3:0/1535190464 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08741860 0x557a08741c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 -- 172.21.15.3:0/1535190464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08741860 msgr2=0x557a08741c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08741860 0x557a08741c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d4ff9700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3c00a08d0 0x7fc3c00a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 -- 172.21.15.3:0/1535190464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 msgr2=0x557a0874f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a0874f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 -- 172.21.15.3:0/1535190464 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a084c0de0 con 0x557a08742180 2023-09-10T01:04:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d4ff9700 1 -- 172.21.15.3:0/3448126285 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc3d00cb920 con 0x7fc3d8073a10 2023-09-10T01:04:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.710+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0874caf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8005ea0 tx=0x7fc3c8005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4098e5bc69c2e533 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.711+0000 7fc3bdffb700 1 -- 172.21.15.3:0/1535190464 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c801d070 con 0x557a08742180 2023-09-10T01:04:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.711+0000 7fc3bdffb700 1 -- 172.21.15.3:0/1535190464 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c801e040 con 0x557a08742180 2023-09-10T01:04:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.711+0000 7fc3bdffb700 1 -- 172.21.15.3:0/1535190464 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c8004920 con 0x557a08742180 2023-09-10T01:04:08.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.711+0000 7fc3d7fff700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3c00a08d0 0x7fc3c00a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1535190464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 msgr2=0x557a0874caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0874caf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8005ea0 tx=0x7fc3c8005ed0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3d7fff700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3c00a08d0 0x7fc3c00a2d90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00c8b60 tx=0x7fc3d0081f90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1535190464 shutdown_connections 2023-09-10T01:04:08.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08741860 0x557a08741c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a0874f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1535190464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0874caf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1535190464 >> 172.21.15.3:0/1535190464 conn(0x557a086b7090 msgr2=0x557a0873dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.712+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1535190464 shutdown_connections 2023-09-10T01:04:08.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.713+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1535190464 wait complete. 2023-09-10T01:04:08.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.713+0000 7fc3f1b5eb80 1 -- start start 2023-09-10T01:04:08.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.714+0000 7fc3f1b5eb80 1 Processor -- start 2023-09-10T01:04:08.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.714+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.714+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08749c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.714+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a0874a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.714+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a08632b70 con 0x557a0874d030 2023-09-10T01:04:08.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.714+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a08632870 con 0x557a08741860 2023-09-10T01:04:08.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.714+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a08516920 con 0x557a08742180 2023-09-10T01:04:08.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3f1b5eb80 10 client.0 _ll_register_callbacks cb 0x557a0857ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55376/0 (socket says 172.21.15.3:55376) 2023-09-10T01:04:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08749c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/1439355031 learned_addr learned my addr 172.21.15.3:0/1439355031 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/1439355031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 msgr2=0x557a08749c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08749c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/1439355031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 msgr2=0x557a0874a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a0874a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.715+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/1439355031 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d810db20 con 0x557a08741860 2023-09-10T01:04:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874ca80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc011820 tx=0x7fc3cc011b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa975364bf7d6ca server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/1439355031 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc016030 con 0x557a08741860 2023-09-10T01:04:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a085ec4f0 con 0x557a08741860 2023-09-10T01:04:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/1439355031 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc3cc004520 con 0x557a08741860 2023-09-10T01:04:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/1439355031 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3cc0236f0 con 0x557a08741860 2023-09-10T01:04:08.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3f1b5eb80 10 client.26000 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.716+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557a084a9c60 con 0x557a08741860 2023-09-10T01:04:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/1439355031 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557a085ec4f0 con 0x557a08741860 2023-09-10T01:04:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/1439355031 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc3cc0038b0 con 0x557a08741860 2023-09-10T01:04:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3f1b5eb80 10 client.26000 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3f1b5eb80 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3f1b5eb80 20 client.26000 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a0875ca00 0x557a0875ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3f1b5eb80 4 client.26000 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.717+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a086a98a0 con 0x557a0875ca00 2023-09-10T01:04:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.718+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a0875ca00 0x557a0875ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.718+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a0875ca00 0x557a0875ee20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b0303e79a1a941 server_cookie=428832c2ed597e90 in_seq=0 out_seq=0 2023-09-10T01:04:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.718+0000 7fc3b6ffd700 10 client.26000 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.719+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/1439355031 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557a086a98a0 con 0x557a0875ca00 2023-09-10T01:04:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.719+0000 7fc3b6ffd700 10 client.26000 handle_command_reply: tid=0 2023-09-10T01:04:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.719+0000 7fc3f1b5eb80 1 client.26000 shutdown 2023-09-10T01:04:08.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.719+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 msgr2=0x557a0874ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.719+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874ca80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc011820 tx=0x7fc3cc011b30 comp rx=0 tx=0).stop 2023-09-10T01:04:08.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 shutdown_connections 2023-09-10T01:04:08.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a0875ca00 0x557a0875ee20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a0874a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874ca80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/1439355031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08749c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3b6ffd700 0 client.26000 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 >> 172.21.15.3:0/1439355031 conn(0x557a086b7090 msgr2=0x557a0873d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 shutdown_connections 2023-09-10T01:04:08.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/1439355031 wait complete. 2023-09-10T01:04:08.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.720+0000 7fc3f1b5eb80 20 client.26000 trim_cache size 0 max 16384 2023-09-10T01:04:08.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 Processor -- start 2023-09-10T01:04:08.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 -- start start 2023-09-10T01:04:08.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a08741860 0x557a08753b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0873c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a0874d030 0x557a08758d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a0874a9b0 con 0x557a08741860 2023-09-10T01:04:08.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a08754050 con 0x557a08742180 2023-09-10T01:04:08.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.747+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a0873c9c0 con 0x557a0874d030 2023-09-10T01:04:08.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0873c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0873c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55380/0 (socket says 172.21.15.3:55380) 2023-09-10T01:04:08.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 -- 172.21.15.3:0/3045486379 learned_addr learned my addr 172.21.15.3:0/3045486379 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a08741860 0x557a08753b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a0874d030 0x557a08758d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 -- 172.21.15.3:0/3045486379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a0874d030 msgr2=0x557a08758d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a0874d030 0x557a08758d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 -- 172.21.15.3:0/3045486379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a08741860 msgr2=0x557a08753b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a08741860 0x557a08753b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 -- 172.21.15.3:0/3045486379 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a084e7250 con 0x557a08742180 2023-09-10T01:04:08.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.748+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0873c480 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009990 tx=0x7fc3c801a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9e812dbfebfda12 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.749+0000 7fc3be7fc700 1 -- 172.21.15.3:0/3045486379 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c8032070 con 0x557a08742180 2023-09-10T01:04:08.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.749+0000 7fc3be7fc700 1 -- 172.21.15.3:0/3045486379 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c8005550 con 0x557a08742180 2023-09-10T01:04:08.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.749+0000 7fc3be7fc700 1 -- 172.21.15.3:0/3045486379 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c801d040 con 0x557a08742180 2023-09-10T01:04:08.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/3045486379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 msgr2=0x557a0873c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0873c480 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009990 tx=0x7fc3c801a040 comp rx=0 tx=0).stop 2023-09-10T01:04:08.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/3045486379 shutdown_connections 2023-09-10T01:04:08.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a0874d030 0x557a08758d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a08741860 0x557a08753b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/3045486379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08742180 0x557a0873c480 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/3045486379 >> 172.21.15.3:0/3045486379 conn(0x557a086b7090 msgr2=0x557a08763760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.750+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/3045486379 shutdown_connections 2023-09-10T01:04:08.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.751+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/3045486379 wait complete. 2023-09-10T01:04:08.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.751+0000 7fc3f1b5eb80 1 -- start start 2023-09-10T01:04:08.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.751+0000 7fc3f1b5eb80 1 Processor -- start 2023-09-10T01:04:08.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.752+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.752+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08750030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.752+0000 7fc3f1b5eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a087b8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.752+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a08750970 con 0x557a0874d030 2023-09-10T01:04:08.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.752+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a087b8550 con 0x557a08741860 2023-09-10T01:04:08.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.752+0000 7fc3f1b5eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a087b8690 con 0x557a08742180 2023-09-10T01:04:08.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.752+0000 7fc3f1b5eb80 10 client.0 _ll_register_callbacks cb 0x7fc3cc009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:08.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08750030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55382/0 (socket says 172.21.15.3:55382) 2023-09-10T01:04:08.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/238903574 learned_addr learned my addr 172.21.15.3:0/238903574 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:08.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a087b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/238903574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 msgr2=0x557a08750030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08750030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/238903574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 msgr2=0x557a087b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:08.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a087b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.753+0000 7fc3d6ffd700 1 -- 172.21.15.3:0/238903574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a084c0de0 con 0x557a08741860 2023-09-10T01:04:08.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.754+0000 7fc3d6ffd700 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874faf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc011570 tx=0x7fc3cc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37152e54c2d59f5f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:08.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.754+0000 7fc3b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:08.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.754+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/238903574 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc016070 con 0x557a08741860 2023-09-10T01:04:08.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.754+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/238903574 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc3cc00f680 con 0x557a08741860 2023-09-10T01:04:08.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.754+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3c801e040 con 0x557a08741860 2023-09-10T01:04:08.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.754+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/238903574 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3cc0057d0 con 0x557a08741860 2023-09-10T01:04:08.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.755+0000 7fc3f1b5eb80 10 client.26006 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:08.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.755+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557a084bf2b0 con 0x557a08741860 2023-09-10T01:04:08.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.755+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/238903574 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c801e040 con 0x557a08741860 2023-09-10T01:04:08.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/238903574 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc3cc024050 con 0x557a08741860 2023-09-10T01:04:08.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3f1b5eb80 10 client.26006 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:08.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3f1b5eb80 10 client.26006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:08.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3f1b5eb80 20 client.26006 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:08.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a087b9c30 0x557a087bc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:08.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3f1b5eb80 4 client.26006 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:04:08.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557a0868d920 con 0x557a087b9c30 2023-09-10T01:04:08.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.756+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a087b9c30 0x557a087bc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:08.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3d77fe700 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a087b9c30 0x557a087bc050 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f63ed50d554c6eb server_cookie=7282f1c41b92ec5c in_seq=0 out_seq=0 2023-09-10T01:04:08.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3b6ffd700 10 client.26006 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3b6ffd700 1 -- 172.21.15.3:0/238903574 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557a0868d920 con 0x557a087b9c30 2023-09-10T01:04:08.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3b6ffd700 10 client.26006 handle_command_reply: tid=0 2023-09-10T01:04:08.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 client.26006 shutdown 2023-09-10T01:04:08.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 msgr2=0x557a0874faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874faf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc011570 tx=0x7fc3cc009260 comp rx=0 tx=0).stop 2023-09-10T01:04:08.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 shutdown_connections 2023-09-10T01:04:08.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557a087b9c30 0x557a087bc050 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a0874d030 0x557a087b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a08741860 0x557a0874faf0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 --2- 172.21.15.3:0/238903574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a08742180 0x557a08750030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 >> 172.21.15.3:0/238903574 conn(0x557a086b7090 msgr2=0x557a08744f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.757+0000 7fc3b6ffd700 0 client.26006 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:08.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.758+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 shutdown_connections 2023-09-10T01:04:08.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.758+0000 7fc3f1b5eb80 1 -- 172.21.15.3:0/238903574 wait complete. 2023-09-10T01:04:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.758+0000 7fc3f1b5eb80 20 client.26006 trim_cache size 0 max 16384 2023-09-10T01:04:08.782 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:04:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3c00a08d0 msgr2=0x7fc3c00a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3c00a08d0 0x7fc3c00a2d90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00c8b60 tx=0x7fc3d0081f90 comp rx=0 tx=0).stop 2023-09-10T01:04:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8073a10 msgr2=0x7fc3d8073ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8073a10 0x7fc3d8073ec0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000ef30 tx=0x7fc3d000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 shutdown_connections 2023-09-10T01:04:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3c00a08d0 0x7fc3c00a2d90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3d8073a10 0x7fc3d8073ec0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8100610 0x7fc3d806efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 --2- 172.21.15.3:0/3448126285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3d8105730 0x7fc3d806f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 >> 172.21.15.3:0/3448126285 conn(0x7fc3d80fbe90 msgr2=0x7fc3d80fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:08.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 shutdown_connections 2023-09-10T01:04:08.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:08.759+0000 7fc3dcf9d700 1 -- 172.21.15.3:0/3448126285 wait complete. 2023-09-10T01:04:09.790 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:04:09.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:04:10.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 -- 172.21.15.3:0/1964074126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8106650 msgr2=0x7fd6d8106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/1964074126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8106650 0x7fd6d8106a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009990 tx=0x7fd6c8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 -- 172.21.15.3:0/1964074126 shutdown_connections 2023-09-10T01:04:10.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/1964074126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8106650 0x7fd6d8106a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/1964074126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 0x7fd6d810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/1964074126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6d8100640 0x7fd6d8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 -- 172.21.15.3:0/1964074126 >> 172.21.15.3:0/1964074126 conn(0x7fd6d80fc170 msgr2=0x7fd6d80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 -- 172.21.15.3:0/1964074126 shutdown_connections 2023-09-10T01:04:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.141+0000 7fd6de9ff700 1 -- 172.21.15.3:0/1964074126 wait complete. 2023-09-10T01:04:10.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.143+0000 7fd6de9ff700 1 Processor -- start 2023-09-10T01:04:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.143+0000 7fd6de9ff700 1 -- start start 2023-09-10T01:04:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6de9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8100640 0x7fd6d8107520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6de9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 0x7fd6d8107a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6de9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6d8106650 0x7fd6d8107fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6de9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81a8f90 con 0x7fd6d8106650 2023-09-10T01:04:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6de9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81a9100 con 0x7fd6d8101000 2023-09-10T01:04:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6de9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81a92c0 con 0x7fd6d8100640 2023-09-10T01:04:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 0x7fd6d8107a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 0x7fd6d8107a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55392/0 (socket says 172.21.15.3:55392) 2023-09-10T01:04:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6d77fe700 1 -- 172.21.15.3:0/2652367944 learned_addr learned my addr 172.21.15.3:0/2652367944 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8100640 0x7fd6d8107520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.144+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6d8106650 0x7fd6d8107fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d77fe700 1 -- 172.21.15.3:0/2652367944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8100640 msgr2=0x7fd6d8107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8100640 0x7fd6d8107520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d77fe700 1 -- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6d8106650 msgr2=0x7fd6d8107fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6d8106650 0x7fd6d8107fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d77fe700 1 -- 172.21.15.3:0/2652367944 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009c40 con 0x7fd6d8101000 2023-09-10T01:04:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6d8106650 0x7fd6d8107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 0x7fd6d8107a60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00d950 tx=0x7fd6cc00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d57fa700 1 -- 172.21.15.3:0/2652367944 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc009940 con 0x7fd6d8101000 2023-09-10T01:04:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d57fa700 1 -- 172.21.15.3:0/2652367944 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6cc010460 con 0x7fd6d8101000 2023-09-10T01:04:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6d57fa700 1 -- 172.21.15.3:0/2652367944 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00f620 con 0x7fd6d8101000 2023-09-10T01:04:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c80095f0 con 0x7fd6d8101000 2023-09-10T01:04:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.145+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d81a9920 con 0x7fd6d8101000 2023-09-10T01:04:10.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.150+0000 7fd6d57fa700 1 -- 172.21.15.3:0/2652367944 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7fd6cc0108d0 con 0x7fd6d8101000 2023-09-10T01:04:10.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.152+0000 7fd6f237eb80 1 Processor -- start 2023-09-10T01:04:10.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.153+0000 7fd6f237eb80 1 -- start start 2023-09-10T01:04:10.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.153+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e35ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.153+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e364980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.153+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e364ec0 0x556b6e3672b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.153+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e2b37b0 con 0x556b6e35b0b0 2023-09-10T01:04:10.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.153+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e2b10d0 con 0x556b6e35a790 2023-09-10T01:04:10.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.153+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e2ac9c0 con 0x556b6e364ec0 2023-09-10T01:04:10.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e364980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e364ec0 0x556b6e3672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e364980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42326/0 (socket says 172.21.15.3:42326) 2023-09-10T01:04:10.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d7fff700 1 -- 172.21.15.3:0/2011215554 learned_addr learned my addr 172.21.15.3:0/2011215554 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e35ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d57fa700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6c00a0950 0x7fd6c00a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d77fe700 1 -- 172.21.15.3:0/2011215554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 msgr2=0x556b6e35ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e35ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d77fe700 1 -- 172.21.15.3:0/2011215554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 msgr2=0x556b6e364980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e364980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d77fe700 1 -- 172.21.15.3:0/2011215554 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b6e0d8e60 con 0x556b6e364ec0 2023-09-10T01:04:10.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.154+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e364980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:10.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.155+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6c00a0950 0x7fd6c00a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.155+0000 7fd6d57fa700 1 -- 172.21.15.3:0/2652367944 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd6cc0cc420 con 0x7fd6d8101000 2023-09-10T01:04:10.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.156+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6c00a0950 0x7fd6c00a2e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009fd0 tx=0x7fd6d0004ad0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.156+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e364ec0 0x556b6e3672b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0c9600 tx=0x7fd6cc064ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81580be3718ae8d5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.156+0000 7fd6be7fc700 1 -- 172.21.15.3:0/2011215554 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc093ea0 con 0x556b6e364ec0 2023-09-10T01:04:10.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.156+0000 7fd6be7fc700 1 -- 172.21.15.3:0/2011215554 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6cc098bf0 con 0x556b6e364ec0 2023-09-10T01:04:10.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.156+0000 7fd6be7fc700 1 -- 172.21.15.3:0/2011215554 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc099290 con 0x556b6e364ec0 2023-09-10T01:04:10.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2011215554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e364ec0 msgr2=0x556b6e3672b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e364ec0 0x556b6e3672b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0c9600 tx=0x7fd6cc064ac0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2011215554 shutdown_connections 2023-09-10T01:04:10.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e364ec0 0x556b6e3672b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e35ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2011215554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e364980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2011215554 >> 172.21.15.3:0/2011215554 conn(0x556b6e2cf110 msgr2=0x556b6e355c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2011215554 shutdown_connections 2023-09-10T01:04:10.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.157+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2011215554 wait complete. 2023-09-10T01:04:10.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.158+0000 7fd6f237eb80 1 -- start start 2023-09-10T01:04:10.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.158+0000 7fd6f237eb80 1 Processor -- start 2023-09-10T01:04:10.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.159+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e364840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.159+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35b0b0 0x556b6e361ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.159+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e362060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.159+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e24abf0 con 0x556b6e364ec0 2023-09-10T01:04:10.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.159+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e24a8f0 con 0x556b6e35b0b0 2023-09-10T01:04:10.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.159+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e12e9a0 con 0x556b6e35a790 2023-09-10T01:04:10.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.159+0000 7fd6f237eb80 10 client.0 _ll_register_callbacks cb 0x556b6e194b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:10.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6dcf9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e362060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e364840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e364840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58754/0 (socket says 172.21.15.3:58754) 2023-09-10T01:04:10.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6d7fff700 1 -- 172.21.15.3:0/2470036378 learned_addr learned my addr 172.21.15.3:0/2470036378 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6dcf9c700 1 -- 172.21.15.3:0/2470036378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 msgr2=0x556b6e364840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e364840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35b0b0 0x556b6e361ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.160+0000 7fd6dcf9c700 1 -- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35b0b0 msgr2=0x556b6e361ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.161+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35b0b0 0x556b6e361ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.161+0000 7fd6dcf9c700 1 -- 172.21.15.3:0/2470036378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d810de10 con 0x556b6e364ec0 2023-09-10T01:04:10.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.161+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35b0b0 0x556b6e361ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:04:10.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.161+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e364840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:10.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.161+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e362060 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0007fd0 tx=0x7fd6d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e13d2f4b6b554d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.161+0000 7fd6b77fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:10.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.161+0000 7fd6b77fe700 1 -- 172.21.15.3:0/2470036378 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0026070 con 0x556b6e364ec0 2023-09-10T01:04:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.162+0000 7fd6b77fe700 1 -- 172.21.15.3:0/2470036378 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd6d0016bc0 con 0x556b6e364ec0 2023-09-10T01:04:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.162+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b6e204570 con 0x556b6e364ec0 2023-09-10T01:04:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.162+0000 7fd6b77fe700 1 -- 172.21.15.3:0/2470036378 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6d000e040 con 0x556b6e364ec0 2023-09-10T01:04:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.162+0000 7fd6f237eb80 10 client.15963 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.162+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fd6d81a9920 con 0x556b6e364ec0 2023-09-10T01:04:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.162+0000 7fd6b77fe700 1 -- 172.21.15.3:0/2470036378 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556b6e204570 con 0x556b6e364ec0 2023-09-10T01:04:10.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.162+0000 7fd6b77fe700 1 -- 172.21.15.3:0/2470036378 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd6d0029020 con 0x556b6e364ec0 2023-09-10T01:04:10.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.163+0000 7fd6f237eb80 10 client.15963 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:10.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.163+0000 7fd6f237eb80 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:10.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.163+0000 7fd6f237eb80 20 client.15963 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:10.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.163+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e374950 0x556b6e376d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.163+0000 7fd6f237eb80 4 client.15963 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:10.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.163+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b6e2c1920 con 0x556b6e374950 2023-09-10T01:04:10.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.163+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e374950 0x556b6e376d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.164+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e374950 0x556b6e376d70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c73852d2e8ad46d5 server_cookie=26dc566e7bec329a in_seq=0 out_seq=0 2023-09-10T01:04:10.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.164+0000 7fd6b77fe700 10 client.15963 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.164+0000 7fd6b77fe700 1 -- 172.21.15.3:0/2470036378 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556b6e2c1920 con 0x556b6e374950 2023-09-10T01:04:10.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.164+0000 7fd6b77fe700 10 client.15963 handle_command_reply: tid=0 2023-09-10T01:04:10.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 client.15963 shutdown 2023-09-10T01:04:10.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 msgr2=0x556b6e362060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e362060 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0007fd0 tx=0x7fd6d000b040 comp rx=0 tx=0).stop 2023-09-10T01:04:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 shutdown_connections 2023-09-10T01:04:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e374950 0x556b6e376d70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e364840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e362060 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/2470036378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35b0b0 0x556b6e361ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 >> 172.21.15.3:0/2470036378 conn(0x556b6e2cf110 msgr2=0x556b6e355860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6b77fe700 0 client.15963 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.165+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 shutdown_connections 2023-09-10T01:04:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.166+0000 7fd6f237eb80 1 -- 172.21.15.3:0/2470036378 wait complete. 2023-09-10T01:04:10.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.166+0000 7fd6f237eb80 20 client.15963 trim_cache size 0 max 16384 2023-09-10T01:04:10.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:10 smithi003 ceph-mon[97654]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2023-09-10T01:04:10.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 Processor -- start 2023-09-10T01:04:10.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 -- start start 2023-09-10T01:04:10.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e364370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35b0b0 0x556b6e35ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e35f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e35ba70 con 0x556b6e364ec0 2023-09-10T01:04:10.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e352450 con 0x556b6e35a790 2023-09-10T01:04:10.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.201+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e362660 con 0x556b6e35b0b0 2023-09-10T01:04:10.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e364370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6dcf9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e35f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e364370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55410/0 (socket says 172.21.15.3:55410) 2023-09-10T01:04:10.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 -- 172.21.15.3:0/1976725070 learned_addr learned my addr 172.21.15.3:0/1976725070 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35b0b0 0x556b6e35ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 -- 172.21.15.3:0/1976725070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35b0b0 msgr2=0x556b6e35ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35b0b0 0x556b6e35ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 -- 172.21.15.3:0/1976725070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 msgr2=0x556b6e35f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e35f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.202+0000 7fd6d7fff700 1 -- 172.21.15.3:0/1976725070 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b6e37b9e0 con 0x556b6e35a790 2023-09-10T01:04:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.203+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e364370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800b2e0 tx=0x7fd6c8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=573ba603bd33693c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.203+0000 7fd6beffd700 1 -- 172.21.15.3:0/1976725070 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c801d040 con 0x556b6e35a790 2023-09-10T01:04:10.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.203+0000 7fd6beffd700 1 -- 172.21.15.3:0/1976725070 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6c801e040 con 0x556b6e35a790 2023-09-10T01:04:10.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.203+0000 7fd6beffd700 1 -- 172.21.15.3:0/1976725070 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8004500 con 0x556b6e35a790 2023-09-10T01:04:10.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.204+0000 7fd6f237eb80 1 -- 172.21.15.3:0/1976725070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 msgr2=0x556b6e364370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.204+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e364370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800b2e0 tx=0x7fd6c8005fd0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.204+0000 7fd6f237eb80 1 -- 172.21.15.3:0/1976725070 shutdown_connections 2023-09-10T01:04:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.204+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e364ec0 0x556b6e35f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.204+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e35a790 0x556b6e364370 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.204+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/1976725070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35b0b0 0x556b6e35ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.204+0000 7fd6f237eb80 1 -- 172.21.15.3:0/1976725070 >> 172.21.15.3:0/1976725070 conn(0x556b6e2cf110 msgr2=0x556b6e35a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.205+0000 7fd6f237eb80 1 -- 172.21.15.3:0/1976725070 shutdown_connections 2023-09-10T01:04:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.205+0000 7fd6f237eb80 1 -- 172.21.15.3:0/1976725070 wait complete. 2023-09-10T01:04:10.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.205+0000 7fd6f237eb80 1 -- start start 2023-09-10T01:04:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.205+0000 7fd6f237eb80 1 Processor -- start 2023-09-10T01:04:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.206+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e36a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.206+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e36aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.206+0000 7fd6f237eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e364ec0 0x556b6e3c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.206+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e3c8d10 con 0x556b6e35b0b0 2023-09-10T01:04:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.206+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e3c8e50 con 0x556b6e364ec0 2023-09-10T01:04:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.206+0000 7fd6f237eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6e3c8f90 con 0x556b6e35a790 2023-09-10T01:04:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.206+0000 7fd6f237eb80 10 client.0 _ll_register_callbacks cb 0x7fd6d0017830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e36a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e36a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58766/0 (socket says 172.21.15.3:58766) 2023-09-10T01:04:10.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 -- 172.21.15.3:0/4161621790 learned_addr learned my addr 172.21.15.3:0/4161621790 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e36aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6dcf9c700 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e364ec0 0x556b6e3c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 -- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e364ec0 msgr2=0x556b6e3c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e364ec0 0x556b6e3c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 -- 172.21.15.3:0/4161621790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 msgr2=0x556b6e36aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e36aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.207+0000 7fd6d7fff700 1 -- 172.21.15.3:0/4161621790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b6e0c1ce0 con 0x556b6e35a790 2023-09-10T01:04:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.208+0000 7fd6d7fff700 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e36a760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8000c00 tx=0x7fd6c80050c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=689b1b039baa4d0e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.208+0000 7fd6b77fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.208+0000 7fd6b77fe700 1 -- 172.21.15.3:0/4161621790 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c801d070 con 0x556b6e35a790 2023-09-10T01:04:10.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.208+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6cc098bf0 con 0x556b6e35a790 2023-09-10T01:04:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.208+0000 7fd6b77fe700 1 -- 172.21.15.3:0/4161621790 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd6c8029cf0 con 0x556b6e35a790 2023-09-10T01:04:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.209+0000 7fd6b77fe700 1 -- 172.21.15.3:0/4161621790 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6c80057d0 con 0x556b6e35a790 2023-09-10T01:04:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.209+0000 7fd6f237eb80 10 client.25948 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.209+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556b6e0ff2d0 con 0x556b6e35a790 2023-09-10T01:04:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.209+0000 7fd6b77fe700 1 -- 172.21.15.3:0/4161621790 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6cc098bf0 con 0x556b6e35a790 2023-09-10T01:04:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.209+0000 7fd6b77fe700 1 -- 172.21.15.3:0/4161621790 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd6c8004cb0 con 0x556b6e35a790 2023-09-10T01:04:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.210+0000 7fd6f237eb80 10 client.25948 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.210+0000 7fd6f237eb80 10 client.25948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.210+0000 7fd6f237eb80 20 client.25948 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.210+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e3ca270 0x556b6e3cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.210+0000 7fd6f237eb80 4 client.25948 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:04:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.210+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556b6e2a59a0 con 0x556b6e3ca270 2023-09-10T01:04:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.210+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e3ca270 0x556b6e3cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.211+0000 7fd6d77fe700 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e3ca270 0x556b6e3cc690 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96e2d3b6afea3e37 server_cookie=bbb93f148ed6872c in_seq=0 out_seq=0 2023-09-10T01:04:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.211+0000 7fd6b77fe700 10 client.25948 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.212+0000 7fd6b77fe700 1 -- 172.21.15.3:0/4161621790 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556b6e2a59a0 con 0x556b6e3ca270 2023-09-10T01:04:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.212+0000 7fd6b77fe700 10 client.25948 handle_command_reply: tid=0 2023-09-10T01:04:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.212+0000 7fd6f237eb80 1 client.25948 shutdown 2023-09-10T01:04:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.212+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 msgr2=0x556b6e36a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.212+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e36a760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8000c00 tx=0x7fd6c80050c0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 shutdown_connections 2023-09-10T01:04:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556b6e3ca270 0x556b6e3cc690 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556b6e35a790 0x556b6e36a760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b6e364ec0 0x556b6e3c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 --2- 172.21.15.3:0/4161621790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556b6e35b0b0 0x556b6e36aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 >> 172.21.15.3:0/4161621790 conn(0x556b6e2cf110 msgr2=0x556b6e360270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6b77fe700 0 client.25948 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 shutdown_connections 2023-09-10T01:04:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 1 -- 172.21.15.3:0/4161621790 wait complete. 2023-09-10T01:04:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.213+0000 7fd6f237eb80 20 client.25948 trim_cache size 0 max 16384 2023-09-10T01:04:10.223 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:04:10.223 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:04:10.223 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "a91fdbcd-0d81-4a27-a979-0c3023f3adf9", 2023-09-10T01:04:10.224 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:04:10.224 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:04:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6c00a0950 msgr2=0x7fd6c00a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6c00a0950 0x7fd6c00a2e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009fd0 tx=0x7fd6d0004ad0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 msgr2=0x7fd6d8107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 0x7fd6d8107a60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00d950 tx=0x7fd6cc00dc60 comp rx=0 tx=0).stop 2023-09-10T01:04:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 shutdown_connections 2023-09-10T01:04:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6c00a0950 0x7fd6c00a2e10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d8100640 0x7fd6d8107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6d8106650 0x7fd6d8107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 --2- 172.21.15.3:0/2652367944 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6d8101000 0x7fd6d8107a60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.214+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 >> 172.21.15.3:0/2652367944 conn(0x7fd6d80fc170 msgr2=0x7fd6d810b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.215+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 shutdown_connections 2023-09-10T01:04:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.215+0000 7fd6de9ff700 1 -- 172.21.15.3:0/2652367944 wait complete. 2023-09-10T01:04:10.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:04:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:10 smithi092 ceph-mon[134067]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2023-09-10T01:04:10.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 -- 172.21.15.3:0/1562363452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0101000 msgr2=0x7f78c010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 --2- 172.21.15.3:0/1562363452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0101000 0x7f78c010d510 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f78b8009f90 tx=0x7f78b800b330 comp rx=0 tx=0).stop 2023-09-10T01:04:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 -- 172.21.15.3:0/1562363452 shutdown_connections 2023-09-10T01:04:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 --2- 172.21.15.3:0/1562363452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0101000 0x7f78c010d510 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 --2- 172.21.15.3:0/1562363452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c0106650 0x7f78c0106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 --2- 172.21.15.3:0/1562363452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78c0100640 0x7f78c0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 -- 172.21.15.3:0/1562363452 >> 172.21.15.3:0/1562363452 conn(0x7f78c00fc170 msgr2=0x7f78c00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.592+0000 7f78c49ff700 1 -- 172.21.15.3:0/1562363452 shutdown_connections 2023-09-10T01:04:10.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.593+0000 7f78c49ff700 1 -- 172.21.15.3:0/1562363452 wait complete. 2023-09-10T01:04:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.594+0000 7f78c49ff700 1 Processor -- start 2023-09-10T01:04:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.594+0000 7f78c49ff700 1 -- start start 2023-09-10T01:04:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78c49ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78c0100640 0x7f78c010cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78c49ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c0101000 0x7f78c010d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78c49ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0106650 0x7f78c010a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78c49ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c010a7e0 con 0x7f78c0100640 2023-09-10T01:04:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78c49ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c010a950 con 0x7f78c0101000 2023-09-10T01:04:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78c49ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c010ab10 con 0x7f78c0106650 2023-09-10T01:04:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78bed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0106650 0x7f78c010a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78bed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0106650 0x7f78c010a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58776/0 (socket says 172.21.15.3:58776) 2023-09-10T01:04:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78bed9d700 1 -- 172.21.15.3:0/3344659703 learned_addr learned my addr 172.21.15.3:0/3344659703 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78be59c700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78c0100640 0x7f78c010cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.595+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c0101000 0x7f78c010d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.596+0000 7f78bed9d700 1 -- 172.21.15.3:0/3344659703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c0101000 msgr2=0x7f78c010d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.596+0000 7f78bed9d700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c0101000 0x7f78c010d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.596+0000 7f78bed9d700 1 -- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78c0100640 msgr2=0x7f78c010cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.596+0000 7f78bed9d700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78c0100640 0x7f78c010cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.596+0000 7f78bed9d700 1 -- 172.21.15.3:0/3344659703 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b40095f0 con 0x7f78c0106650 2023-09-10T01:04:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.596+0000 7f78bed9d700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0106650 0x7f78c010a010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78b8004a50 tx=0x7f78b8004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.597+0000 7f78af7fe700 1 -- 172.21.15.3:0/3344659703 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b800e040 con 0x7f78c0106650 2023-09-10T01:04:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.597+0000 7f78af7fe700 1 -- 172.21.15.3:0/3344659703 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b8004e30 con 0x7f78c0106650 2023-09-10T01:04:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.597+0000 7f78af7fe700 1 -- 172.21.15.3:0/3344659703 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b80052a0 con 0x7f78c0106650 2023-09-10T01:04:10.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.597+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b8009c40 con 0x7f78c0106650 2023-09-10T01:04:10.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.597+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c01a9840 con 0x7f78c0106650 2023-09-10T01:04:10.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.603+0000 7f78af7fe700 1 -- 172.21.15.3:0/3344659703 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f78b801d470 con 0x7f78c0106650 2023-09-10T01:04:10.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.604+0000 7f78d8318b80 1 Processor -- start 2023-09-10T01:04:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.604+0000 7f78d8318b80 1 -- start start 2023-09-10T01:04:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.605+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa683af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.605+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa684030 0x55d4fa68d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.605+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa68de40 0x55d4fa690230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.605+0000 7f78d8318b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa5dc730 con 0x55d4fa684030 2023-09-10T01:04:10.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.605+0000 7f78d8318b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa5da050 con 0x55d4fa68de40 2023-09-10T01:04:10.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.605+0000 7f78d8318b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa5d5940 con 0x55d4fa683710 2023-09-10T01:04:10.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa683af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78af7fe700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78a80a0950 0x7f78a80a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa683af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58784/0 (socket says 172.21.15.3:58784) 2023-09-10T01:04:10.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2780495074 learned_addr learned my addr 172.21.15.3:0/2780495074 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa68de40 0x55d4fa690230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78af7fe700 1 -- 172.21.15.3:0/3344659703 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f78b80cd680 con 0x7f78c0106650 2023-09-10T01:04:10.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2780495074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa68de40 msgr2=0x55d4fa690230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa68de40 0x55d4fa690230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2780495074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa684030 msgr2=0x55d4fa68d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa684030 0x55d4fa68d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2780495074 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4fa401de0 con 0x55d4fa683710 2023-09-10T01:04:10.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bed9d700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78a80a0950 0x7f78a80a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.606+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa683af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f78b40119c0 tx=0x7f78b40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d10d4994ba81a959 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.607+0000 7f78acff9700 1 -- 172.21.15.3:0/2780495074 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4011020 con 0x55d4fa683710 2023-09-10T01:04:10.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.607+0000 7f78acff9700 1 -- 172.21.15.3:0/2780495074 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b4011180 con 0x55d4fa683710 2023-09-10T01:04:10.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.607+0000 7f78acff9700 1 -- 172.21.15.3:0/2780495074 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4003d90 con 0x55d4fa683710 2023-09-10T01:04:10.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 -- 172.21.15.3:0/2780495074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 msgr2=0x55d4fa683af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78bed9d700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78a80a0950 0x7f78a80a2e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b80ca8c0 tx=0x7f78b8009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa683af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f78b40119c0 tx=0x7f78b40047c0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 -- 172.21.15.3:0/2780495074 shutdown_connections 2023-09-10T01:04:10.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa683af0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa68de40 0x55d4fa690230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2780495074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa684030 0x55d4fa68d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 -- 172.21.15.3:0/2780495074 >> 172.21.15.3:0/2780495074 conn(0x55d4fa5f8090 msgr2=0x55d4fa67ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 -- 172.21.15.3:0/2780495074 shutdown_connections 2023-09-10T01:04:10.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.608+0000 7f78d8318b80 1 -- 172.21.15.3:0/2780495074 wait complete. 2023-09-10T01:04:10.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.609+0000 7f78d8318b80 1 -- start start 2023-09-10T01:04:10.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 1 Processor -- start 2023-09-10T01:04:10.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa683710 0x55d4fa68d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa684030 0x55d4fa68aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa573b70 con 0x55d4fa68de40 2023-09-10T01:04:10.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa573870 con 0x55d4fa683710 2023-09-10T01:04:10.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa457920 con 0x55d4fa684030 2023-09-10T01:04:10.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.610+0000 7f78d8318b80 10 client.0 _ll_register_callbacks cb 0x55d4fa4bda80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:10.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa683710 0x55d4fa68d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa683710 0x55d4fa68d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55462/0 (socket says 172.21.15.3:55462) 2023-09-10T01:04:10.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 -- 172.21.15.3:0/35479878 learned_addr learned my addr 172.21.15.3:0/35479878 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 -- 172.21.15.3:0/35479878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa684030 msgr2=0x55d4fa68aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:04:10.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa684030 0x55d4fa68aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 -- 172.21.15.3:0/35479878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 msgr2=0x55d4fa68b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.611+0000 7f78bdd9b700 1 -- 172.21.15.3:0/35479878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b4019040 con 0x55d4fa683710 2023-09-10T01:04:10.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.612+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa683710 0x55d4fa68d870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009af0 tx=0x7f78b4003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ffeb55e56869b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.612+0000 7f78a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:10.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.612+0000 7f78a5ffb700 1 -- 172.21.15.3:0/35479878 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4003a10 con 0x55d4fa683710 2023-09-10T01:04:10.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.612+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4fa52d4f0 con 0x55d4fa683710 2023-09-10T01:04:10.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.612+0000 7f78a5ffb700 1 -- 172.21.15.3:0/35479878 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f78b4003b70 con 0x55d4fa683710 2023-09-10T01:04:10.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.613+0000 7f78a5ffb700 1 -- 172.21.15.3:0/35479878 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78b4003d30 con 0x55d4fa683710 2023-09-10T01:04:10.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.613+0000 7f78d8318b80 10 client.26033 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:10.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.613+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55d4fa3eac60 con 0x55d4fa683710 2023-09-10T01:04:10.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.613+0000 7f78a5ffb700 1 -- 172.21.15.3:0/35479878 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d4fa52d4f0 con 0x55d4fa683710 2023-09-10T01:04:10.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.614+0000 7f78a5ffb700 1 -- 172.21.15.3:0/35479878 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f78b402d060 con 0x55d4fa683710 2023-09-10T01:04:10.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.614+0000 7f78d8318b80 10 client.26033 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:10.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.614+0000 7f78d8318b80 10 client.26033 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:10.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.614+0000 7f78d8318b80 20 client.26033 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:10.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.614+0000 7f78d8318b80 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa69d750 0x55d4fa69fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.614+0000 7f78d8318b80 4 client.26033 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:10.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.614+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4fa5ea8a0 con 0x55d4fa69d750 2023-09-10T01:04:10.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.615+0000 7f78be59c700 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa69d750 0x55d4fa69fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.615+0000 7f78be59c700 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa69d750 0x55d4fa69fb70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2666398668fd1f11 server_cookie=6ec6c16c3a6fe717 in_seq=0 out_seq=0 2023-09-10T01:04:10.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.615+0000 7f78a5ffb700 10 client.26033 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.616+0000 7f78a5ffb700 1 -- 172.21.15.3:0/35479878 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d4fa5ea8a0 con 0x55d4fa69d750 2023-09-10T01:04:10.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.616+0000 7f78a5ffb700 10 client.26033 handle_command_reply: tid=0 2023-09-10T01:04:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.616+0000 7f78d8318b80 1 client.26033 shutdown 2023-09-10T01:04:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.616+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa683710 msgr2=0x55d4fa68d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.616+0000 7f78d8318b80 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa683710 0x55d4fa68d870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009af0 tx=0x7f78b4003970 comp rx=0 tx=0).stop 2023-09-10T01:04:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 shutdown_connections 2023-09-10T01:04:10.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa69d750 0x55d4fa69fb70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa683710 0x55d4fa68d870 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 --2- 172.21.15.3:0/35479878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa684030 0x55d4fa68aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 >> 172.21.15.3:0/35479878 conn(0x55d4fa5f8090 msgr2=0x55d4fa67e780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78a5ffb700 0 client.26033 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 shutdown_connections 2023-09-10T01:04:10.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 1 -- 172.21.15.3:0/35479878 wait complete. 2023-09-10T01:04:10.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.617+0000 7f78d8318b80 20 client.26033 trim_cache size 0 max 16384 2023-09-10T01:04:10.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.651+0000 7f78d8318b80 1 Processor -- start 2023-09-10T01:04:10.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.651+0000 7f78d8318b80 1 -- start start 2023-09-10T01:04:10.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 0x55d4fa67e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78d8318b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa67b3d0 con 0x55d4fa68de40 2023-09-10T01:04:10.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78d8318b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa685890 con 0x55d4fa684030 2023-09-10T01:04:10.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78d8318b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa694f40 con 0x55d4fa683710 2023-09-10T01:04:10.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58796/0 (socket says 172.21.15.3:58796) 2023-09-10T01:04:10.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2344144299 learned_addr learned my addr 172.21.15.3:0/2344144299 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.652+0000 7f78be59c700 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 0x55d4fa67e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2344144299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 msgr2=0x55d4fa67e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 0x55d4fa67e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2344144299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 msgr2=0x55d4fa68d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78bdd9b700 1 -- 172.21.15.3:0/2344144299 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4fa428250 con 0x55d4fa683710 2023-09-10T01:04:10.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78b4001eb0 tx=0x7f78b4003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5698cfb3ba768575 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78ad7fa700 1 -- 172.21.15.3:0/2344144299 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b400e3b0 con 0x55d4fa683710 2023-09-10T01:04:10.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78ad7fa700 1 -- 172.21.15.3:0/2344144299 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b4003ea0 con 0x55d4fa683710 2023-09-10T01:04:10.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.653+0000 7f78ad7fa700 1 -- 172.21.15.3:0/2344144299 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4010a80 con 0x55d4fa683710 2023-09-10T01:04:10.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.654+0000 7f78d8318b80 1 -- 172.21.15.3:0/2344144299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 msgr2=0x55d4fa694a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.654+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78b4001eb0 tx=0x7f78b4003970 comp rx=0 tx=0).stop 2023-09-10T01:04:10.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.654+0000 7f78d8318b80 1 -- 172.21.15.3:0/2344144299 shutdown_connections 2023-09-10T01:04:10.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.654+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694a00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.654+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa68d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.654+0000 7f78d8318b80 1 --2- 172.21.15.3:0/2344144299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 0x55d4fa67e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.654+0000 7f78d8318b80 1 -- 172.21.15.3:0/2344144299 >> 172.21.15.3:0/2344144299 conn(0x55d4fa5f8090 msgr2=0x55d4fa699150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.655+0000 7f78d8318b80 1 -- 172.21.15.3:0/2344144299 shutdown_connections 2023-09-10T01:04:10.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.655+0000 7f78d8318b80 1 -- 172.21.15.3:0/2344144299 wait complete. 2023-09-10T01:04:10.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.655+0000 7f78d8318b80 1 -- start start 2023-09-10T01:04:10.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 1 Processor -- start 2023-09-10T01:04:10.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 0x55d4fa692990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa692ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa694f70 con 0x55d4fa68de40 2023-09-10T01:04:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa693410 con 0x55d4fa684030 2023-09-10T01:04:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4fa693580 con 0x55d4fa683710 2023-09-10T01:04:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.656+0000 7f78d8318b80 10 client.0 _ll_register_callbacks cb 0x7f78b4011480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58812/0 (socket says 172.21.15.3:58812) 2023-09-10T01:04:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 -- 172.21.15.3:0/60791089 learned_addr learned my addr 172.21.15.3:0/60791089 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa692ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 -- 172.21.15.3:0/60791089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 msgr2=0x55d4fa692990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:04:10.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 0x55d4fa692990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 -- 172.21.15.3:0/60791089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 msgr2=0x55d4fa692ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:10.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa692ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.657+0000 7f78bdd9b700 1 -- 172.21.15.3:0/60791089 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4fa3eac60 con 0x55d4fa683710 2023-09-10T01:04:10.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.658+0000 7f78bdd9b700 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f78b4005a80 tx=0x7f78b4010650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6e92d0c31783a85 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.658+0000 7f78a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.658+0000 7f78a5ffb700 1 -- 172.21.15.3:0/60791089 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4040040 con 0x55d4fa683710 2023-09-10T01:04:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.658+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78b4011180 con 0x55d4fa683710 2023-09-10T01:04:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.658+0000 7f78a5ffb700 1 -- 172.21.15.3:0/60791089 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f78b4000e40 con 0x55d4fa683710 2023-09-10T01:04:10.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78a5ffb700 1 -- 172.21.15.3:0/60791089 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78b40039e0 con 0x55d4fa683710 2023-09-10T01:04:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 10 client.25963 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55d4fa4002b0 con 0x55d4fa683710 2023-09-10T01:04:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78a5ffb700 1 -- 172.21.15.3:0/60791089 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b4011180 con 0x55d4fa683710 2023-09-10T01:04:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78a5ffb700 1 -- 172.21.15.3:0/60791089 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f78b4046040 con 0x55d4fa683710 2023-09-10T01:04:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 10 client.25963 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:10.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 10 client.25963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 20 client.25963 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa6f2370 0x55d4fa6f4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 4 client.25963 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:04:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.659+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d4fa5ce920 con 0x55d4fa6f2370 2023-09-10T01:04:10.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.660+0000 7f78be59c700 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa6f2370 0x55d4fa6f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.660+0000 7f78be59c700 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa6f2370 0x55d4fa6f4760 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=882e12fd710b8d02 server_cookie=1a5d20af46a59ded in_seq=0 out_seq=0 2023-09-10T01:04:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.660+0000 7f78a5ffb700 10 client.25963 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78a5ffb700 1 -- 172.21.15.3:0/60791089 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d4fa5ce920 con 0x55d4fa6f2370 2023-09-10T01:04:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78a5ffb700 10 client.25963 handle_command_reply: tid=0 2023-09-10T01:04:10.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 client.25963 shutdown 2023-09-10T01:04:10.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 msgr2=0x55d4fa694660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f78b4005a80 tx=0x7f78b4010650 comp rx=0 tx=0).stop 2023-09-10T01:04:10.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 shutdown_connections 2023-09-10T01:04:10.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d4fa6f2370 0x55d4fa6f4760 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d4fa683710 0x55d4fa694660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d4fa68de40 0x55d4fa692ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 --2- 172.21.15.3:0/60791089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4fa684030 0x55d4fa692990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 >> 172.21.15.3:0/60791089 conn(0x55d4fa5f8090 msgr2=0x55d4fa67e670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78a5ffb700 0 client.25963 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:10.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 shutdown_connections 2023-09-10T01:04:10.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.661+0000 7f78d8318b80 1 -- 172.21.15.3:0/60791089 wait complete. 2023-09-10T01:04:10.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.662+0000 7f78d8318b80 20 client.25963 trim_cache size 0 max 16384 2023-09-10T01:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "a91fdbcd-0d81-4a27-a979-0c3023f3adf9": { 2023-09-10T01:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "a91fdbcd-0d81-4a27-a979-0c3023f3adf9", 2023-09-10T01:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:04:10.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.662+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78a80a0950 msgr2=0x7f78a80a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.662+0000 7f78c49ff700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78a80a0950 0x7f78a80a2e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b80ca8c0 tx=0x7f78b8009920 comp rx=0 tx=0).stop 2023-09-10T01:04:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.662+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0106650 msgr2=0x7f78c010a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.662+0000 7f78c49ff700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0106650 0x7f78c010a010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78b8004a50 tx=0x7f78b8004a80 comp rx=0 tx=0).stop 2023-09-10T01:04:10.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 shutdown_connections 2023-09-10T01:04:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78a80a0950 0x7f78a80a2e10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78c0106650 0x7f78c010a010 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78c0100640 0x7f78c010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 --2- 172.21.15.3:0/3344659703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c0101000 0x7f78c010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:10.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 >> 172.21.15.3:0/3344659703 conn(0x7f78c00fc170 msgr2=0x7f78c0104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:10.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 shutdown_connections 2023-09-10T01:04:10.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:10.663+0000 7f78c49ff700 1 -- 172.21.15.3:0/3344659703 wait complete. 2023-09-10T01:04:10.682 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:10 smithi136 ceph-mon[133708]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2023-09-10T01:04:10.692 INFO:tasks.cephfs.filesystem:scrub status for tag:a91fdbcd-0d81-4a27-a979-0c3023f3adf9 - {'path': '/', 'tag': 'a91fdbcd-0d81-4a27-a979-0c3023f3adf9', 'options': 'recursive,force'} 2023-09-10T01:04:11.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:11 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:04:11.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:11 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:11.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:11 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:04:11.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:11 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:11 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:04:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:11 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:11 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:04:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:11 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:11.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:11 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:04:11.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:11 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:11.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:11 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:04:11.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:11 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:12.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:12 smithi092 ceph-mon[134067]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 20 op/s 2023-09-10T01:04:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:12 smithi003 ceph-mon[97654]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 20 op/s 2023-09-10T01:04:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:12 smithi136 ceph-mon[133708]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 20 op/s 2023-09-10T01:04:14.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:14 smithi092 ceph-mon[134067]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-09-10T01:04:14.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:14 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:04:14.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:14 smithi003 ceph-mon[97654]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-09-10T01:04:14.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:14 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:04:14.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:14 smithi136 ceph-mon[133708]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-09-10T01:04:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:14 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:04:16.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:16 smithi092 ceph-mon[134067]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-09-10T01:04:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:16 smithi003 ceph-mon[97654]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-09-10T01:04:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:16 smithi136 ceph-mon[133708]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-09-10T01:04:17.210 INFO:tasks.workunit.client.0.smithi003.stderr:Updating files: 62% (7448/11967) Updating files: 63% (7540/11967) Updating files: 64% (7659/11967) Updating files: 65% (7779/11967) Updating files: 66% (7899/11967) Updating files: 67% (8018/11967) Updating files: 68% (8138/11967) Updating files: 69% (8258/11967) Updating files: 70% (8377/11967) Updating files: 71% (8497/11967) Updating files: 72% (8617/11967) Updating files: 73% (8736/11967) Updating files: 74% (8856/11967) Updating files: 75% (8976/11967) Updating files: 76% (9095/11967) Updating files: 77% (9215/11967) Updating files: 78% (9335/11967) Updating files: 79% (9454/11967) Updating files: 80% (9574/11967) Updating files: 81% (9694/11967) Updating files: 82% (9813/11967) Updating files: 83% (9933/11967) Updating files: 84% (10053/11967) Updating files: 85% (10172/11967) Updating files: 86% (10292/11967) Updating files: 87% (10412/11967) Updating files: 88% (10531/11967) Updating files: 89% (10651/11967) Updating files: 90% (10771/11967) Updating files: 91% (10890/11967) Updating files: 92% (11010/11967) Updating files: 93% (11130/11967) Updating files: 94% (11249/11967) Updating files: 95% (11369/11967) Updating files: 96% (11489/11967) Updating files: 97% (11608/11967) Updating files: 98% (11728/11967) Updating files: 98% (11834/11967) Updating files: 99% (11848/11967) Updating files: 100% (11967/11967) Updating files: 100% (11967/11967), done. 2023-09-10T01:04:17.522 INFO:tasks.workunit.client.0.smithi003.stderr:Note: switching to '9670cca5bc8e17916865dc84939edfc10b1f2c00'. 2023-09-10T01:04:17.522 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-09-10T01:04:17.522 INFO:tasks.workunit.client.0.smithi003.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-09-10T01:04:17.523 INFO:tasks.workunit.client.0.smithi003.stderr:changes and commit them, and you can discard any commits you make in this 2023-09-10T01:04:17.523 INFO:tasks.workunit.client.0.smithi003.stderr:state without impacting any branches by switching back to a branch. 2023-09-10T01:04:17.523 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-09-10T01:04:17.523 INFO:tasks.workunit.client.0.smithi003.stderr:If you want to create a new branch to retain commits you create, you may 2023-09-10T01:04:17.524 INFO:tasks.workunit.client.0.smithi003.stderr:do so (now or later) by using -c with the switch command. Example: 2023-09-10T01:04:17.524 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-09-10T01:04:17.524 INFO:tasks.workunit.client.0.smithi003.stderr: git switch -c 2023-09-10T01:04:17.524 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-09-10T01:04:17.524 INFO:tasks.workunit.client.0.smithi003.stderr:Or undo this operation with: 2023-09-10T01:04:17.524 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-09-10T01:04:17.525 INFO:tasks.workunit.client.0.smithi003.stderr: git switch - 2023-09-10T01:04:17.525 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-09-10T01:04:17.525 INFO:tasks.workunit.client.0.smithi003.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-09-10T01:04:17.525 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-09-10T01:04:17.525 INFO:tasks.workunit.client.0.smithi003.stderr:HEAD is now at 9670cca5bc8 Merge PR #48130 into wip-batrick-testing-20230909.145638 2023-09-10T01:04:17.531 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-09-10T01:04:17.586 INFO:tasks.workunit.client.0.smithi003.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-09-10T01:04:17.591 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-10T01:04:17.591 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-09-10T01:04:18.205 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-09-10T01:04:18.333 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-09-10T01:04:18.397 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-10T01:04:18.401 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-10T01:04:18.401 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-09-10T01:04:18.464 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-10T01:04:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:18 smithi092 ceph-mon[134067]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-09-10T01:04:18.471 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T01:04:18.472 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-09-10T01:04:18.515 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2023-09-10T01:04:18.517 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2023-09-10T01:04:18.517 DEBUG:teuthology.orchestra.run.smithi003:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9670cca5bc8e17916865dc84939edfc10b1f2c00 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2023-09-10T01:04:18.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:18 smithi003 ceph-mon[97654]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-09-10T01:04:18.615 INFO:tasks.workunit.client.0.smithi003.stderr:+ dbench 1 2023-09-10T01:04:18.632 INFO:tasks.workunit.client.0.smithi003.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2023-09-10T01:04:18.632 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-09-10T01:04:18.632 INFO:tasks.workunit.client.0.smithi003.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2023-09-10T01:04:18.633 INFO:tasks.workunit.client.0.smithi003.stdout:0 of 1 processes prepared for launch 0 sec 2023-09-10T01:04:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:18 smithi136 ceph-mon[133708]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-09-10T01:04:18.733 INFO:tasks.workunit.client.0.smithi003.stdout:1 of 1 processes prepared for launch 0 sec 2023-09-10T01:04:18.734 INFO:tasks.workunit.client.0.smithi003.stdout:releasing clients 2023-09-10T01:04:19.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 2993 33.51 MB/sec warmup 1 sec latency 6.327 ms 2023-09-10T01:04:20.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:20 smithi092 ceph-mon[134067]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2023-09-10T01:04:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:20 smithi003 ceph-mon[97654]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2023-09-10T01:04:20.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:20 smithi136 ceph-mon[133708]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2023-09-10T01:04:20.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 5760 22.94 MB/sec warmup 2 sec latency 6.391 ms 2023-09-10T01:04:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 -- 172.21.15.3:0/2693719785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b0100f80 msgr2=0x7fa0b010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/2693719785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b0100f80 0x7fa0b010d440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80099c0 tx=0x7fa0a8009cd0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 -- 172.21.15.3:0/2693719785 shutdown_connections 2023-09-10T01:04:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/2693719785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0b0106680 0x7fa0b0106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/2693719785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b0100f80 0x7fa0b010d440 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/2693719785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b01005c0 0x7fa0b0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 -- 172.21.15.3:0/2693719785 >> 172.21.15.3:0/2693719785 conn(0x7fa0b00fc0b0 msgr2=0x7fa0b00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.360+0000 7fa0b49ff700 1 -- 172.21.15.3:0/2693719785 shutdown_connections 2023-09-10T01:04:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.361+0000 7fa0b49ff700 1 -- 172.21.15.3:0/2693719785 wait complete. 2023-09-10T01:04:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 Processor -- start 2023-09-10T01:04:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 -- start start 2023-09-10T01:04:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b01005c0 0x7fa0b019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b0100f80 0x7fa0b019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0b0106680 0x7fa0b0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0198ca0 con 0x7fa0b01005c0 2023-09-10T01:04:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0198e10 con 0x7fa0b0100f80 2023-09-10T01:04:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0b49ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0198fd0 con 0x7fa0b0106680 2023-09-10T01:04:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b0100f80 0x7fa0b019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b01005c0 0x7fa0b019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.362+0000 7fa0add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b0100f80 0x7fa0b019eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45618/0 (socket says 172.21.15.3:45618) 2023-09-10T01:04:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0add9b700 1 -- 172.21.15.3:0/484137030 learned_addr learned my addr 172.21.15.3:0/484137030 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0b0106680 0x7fa0b0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0add9b700 1 -- 172.21.15.3:0/484137030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0b0106680 msgr2=0x7fa0b0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0add9b700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0b0106680 0x7fa0b0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0add9b700 1 -- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b01005c0 msgr2=0x7fa0b019e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0add9b700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b01005c0 0x7fa0b019e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0add9b700 1 -- 172.21.15.3:0/484137030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80095f0 con 0x7fa0b0100f80 2023-09-10T01:04:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0add9b700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b0100f80 0x7fa0b019eb00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400ef10 tx=0x7fa0a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa09f7fe700 1 -- 172.21.15.3:0/484137030 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400c9c0 con 0x7fa0b0100f80 2023-09-10T01:04:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa09f7fe700 1 -- 172.21.15.3:0/484137030 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0a400f040 con 0x7fa0b0100f80 2023-09-10T01:04:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.363+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b0199250 con 0x7fa0b0100f80 2023-09-10T01:04:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.364+0000 7fa09f7fe700 1 -- 172.21.15.3:0/484137030 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4014480 con 0x7fa0b0100f80 2023-09-10T01:04:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.364+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b01a55b0 con 0x7fa0b0100f80 2023-09-10T01:04:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.367+0000 7fa09f7fe700 1 -- 172.21.15.3:0/484137030 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7fa0a4007690 con 0x7fa0b0100f80 2023-09-10T01:04:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.368+0000 7fa0c83f7b80 1 Processor -- start 2023-09-10T01:04:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0c83f7b80 1 -- start start 2023-09-10T01:04:21.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b1ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1a6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b0ff6d0 con 0x56418b1a6300 2023-09-10T01:04:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b0fcff0 con 0x56418b1a6cc0 2023-09-10T01:04:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b0f88e0 con 0x56418b1ac1a0 2023-09-10T01:04:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.369+0000 7fa0aed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b1ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b1ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47012/0 (socket says 172.21.15.3:47012) 2023-09-10T01:04:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 -- 172.21.15.3:0/3938739635 learned_addr learned my addr 172.21.15.3:0/3938739635 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0add9b700 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa09f7fe700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0980a08d0 0x7fa0980a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 -- 172.21.15.3:0/3938739635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 msgr2=0x56418b1b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 -- 172.21.15.3:0/3938739635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 msgr2=0x56418b1a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 -- 172.21.15.3:0/3938739635 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56418af24d80 con 0x56418b1ac1a0 2023-09-10T01:04:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0980a08d0 0x7fa0980a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.370+0000 7fa09f7fe700 1 -- 172.21.15.3:0/484137030 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa0a40cc8f0 con 0x7fa0b0100f80 2023-09-10T01:04:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.371+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b1ac580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80099c0 tx=0x7fa0a8005c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c1c93a43fd14022 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.371+0000 7fa09cff9700 1 -- 172.21.15.3:0/3938739635 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8004d70 con 0x56418b1ac1a0 2023-09-10T01:04:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.371+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0980a08d0 0x7fa0980a2d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa0a802d000 tx=0x7fa0a802a000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.371+0000 7fa09cff9700 1 -- 172.21.15.3:0/3938739635 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0a801e030 con 0x56418b1ac1a0 2023-09-10T01:04:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.371+0000 7fa09cff9700 1 -- 172.21.15.3:0/3938739635 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a800f460 con 0x56418b1ac1a0 2023-09-10T01:04:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/3938739635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 msgr2=0x56418b1ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b1ac580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80099c0 tx=0x7fa0a8005c70 comp rx=0 tx=0).stop 2023-09-10T01:04:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/3938739635 shutdown_connections 2023-09-10T01:04:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b1ac580 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/3938739635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/3938739635 >> 172.21.15.3:0/3938739635 conn(0x56418b11b030 msgr2=0x56418b1a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:21.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/3938739635 shutdown_connections 2023-09-10T01:04:21.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.372+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/3938739635 wait complete. 2023-09-10T01:04:21.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 -- start start 2023-09-10T01:04:21.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 Processor -- start 2023-09-10T01:04:21.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6300 0x56418b1b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1ac1a0 0x56418b1b0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b096b10 con 0x56418b1ac1a0 2023-09-10T01:04:21.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b096810 con 0x56418b1a6300 2023-09-10T01:04:21.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418af7a8c0 con 0x56418b1a6cc0 2023-09-10T01:04:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.373+0000 7fa0c83f7b80 10 client.0 _ll_register_callbacks cb 0x56418afe0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6300 0x56418b1b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6300 0x56418b1b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45626/0 (socket says 172.21.15.3:45626) 2023-09-10T01:04:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 -- 172.21.15.3:0/2301157640 learned_addr learned my addr 172.21.15.3:0/2301157640 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0add9b700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1ac1a0 0x56418b1b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 -- 172.21.15.3:0/2301157640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 msgr2=0x56418b1b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 -- 172.21.15.3:0/2301157640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1ac1a0 msgr2=0x56418b1b0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1ac1a0 0x56418b1b0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 -- 172.21.15.3:0/2301157640 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b010dd40 con 0x56418b1a6300 2023-09-10T01:04:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0add9b700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6300 0x56418b1b2df0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0012770 tx=0x7fa0a0012a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a14f73a949299c81 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.374+0000 7fa095ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa095ffb700 1 -- 172.21.15.3:0/2301157640 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0017070 con 0x56418b1a6300 2023-09-10T01:04:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56418b050490 con 0x56418b1a6300 2023-09-10T01:04:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa095ffb700 1 -- 172.21.15.3:0/2301157640 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa0a000f480 con 0x56418b1a6300 2023-09-10T01:04:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa095ffb700 1 -- 172.21.15.3:0/2301157640 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0a0021640 con 0x56418b1a6300 2023-09-10T01:04:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 10 client.26048 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56418af0dc00 con 0x56418b1a6300 2023-09-10T01:04:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa095ffb700 1 -- 172.21.15.3:0/2301157640 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56418b050490 con 0x56418b1a6300 2023-09-10T01:04:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa095ffb700 1 -- 172.21.15.3:0/2301157640 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa0a000f640 con 0x56418b1a6300 2023-09-10T01:04:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 10 client.26048 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 10 client.26048 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 20 client.26048 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b1c0480 0x56418b1c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 4 client.26048 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.375+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56418b10d840 con 0x56418b1c0480 2023-09-10T01:04:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.376+0000 7fa0add9b700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b1c0480 0x56418b1c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.376+0000 7fa0add9b700 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b1c0480 0x56418b1c28a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10ed8c32f368f2c2 server_cookie=1ca2a2720023a9fa in_seq=0 out_seq=0 2023-09-10T01:04:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.376+0000 7fa095ffb700 10 client.26048 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.377+0000 7fa095ffb700 1 -- 172.21.15.3:0/2301157640 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56418b10d840 con 0x56418b1c0480 2023-09-10T01:04:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.377+0000 7fa095ffb700 10 client.26048 handle_command_reply: tid=0 2023-09-10T01:04:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 client.26048 shutdown 2023-09-10T01:04:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6300 msgr2=0x56418b1b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6300 0x56418b1b2df0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0012770 tx=0x7fa0a0012a80 comp rx=0 tx=0).stop 2023-09-10T01:04:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 shutdown_connections 2023-09-10T01:04:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b1c0480 0x56418b1c28a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1ac1a0 0x56418b1b0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa095ffb700 0 client.26048 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6300 0x56418b1b2df0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/2301157640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 >> 172.21.15.3:0/2301157640 conn(0x56418b11b030 msgr2=0x56418b1a07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 shutdown_connections 2023-09-10T01:04:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/2301157640 wait complete. 2023-09-10T01:04:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.378+0000 7fa0c83f7b80 20 client.26048 trim_cache size 0 max 16384 2023-09-10T01:04:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 Processor -- start 2023-09-10T01:04:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 -- start start 2023-09-10T01:04:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1ac1a0 0x56418b1ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b1ae640 con 0x56418b1a6300 2023-09-10T01:04:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b1aac80 con 0x56418b1ac1a0 2023-09-10T01:04:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.396+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b1b0c40 con 0x56418b1a6cc0 2023-09-10T01:04:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0aed9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1ac1a0 0x56418b1ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0add9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0aed9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1ac1a0 0x56418b1ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45628/0 (socket says 172.21.15.3:45628) 2023-09-10T01:04:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0add9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47024/0 (socket says 172.21.15.3:47024) 2023-09-10T01:04:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0aed9d700 1 -- 172.21.15.3:0/1108324070 learned_addr learned my addr 172.21.15.3:0/1108324070 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0ae59c700 1 -- 172.21.15.3:0/1108324070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 msgr2=0x56418b1b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0ae59c700 1 -- 172.21.15.3:0/1108324070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1ac1a0 msgr2=0x56418b1ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1ac1a0 0x56418b1ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0ae59c700 1 -- 172.21.15.3:0/1108324070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56418af4b1f0 con 0x56418b1a6300 2023-09-10T01:04:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0aed9d700 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1ac1a0 0x56418b1ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b28b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000fa70 tx=0x7fa0a0012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cde756e8e66d2e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.397+0000 7fa09d7fa700 1 -- 172.21.15.3:0/1108324070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0017070 con 0x56418b1a6300 2023-09-10T01:04:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa09d7fa700 1 -- 172.21.15.3:0/1108324070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0a0018040 con 0x56418b1a6300 2023-09-10T01:04:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa09d7fa700 1 -- 172.21.15.3:0/1108324070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0004950 con 0x56418b1a6300 2023-09-10T01:04:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1108324070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 msgr2=0x56418b1b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b28b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000fa70 tx=0x7fa0a0012dd0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1108324070 shutdown_connections 2023-09-10T01:04:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1ac1a0 0x56418b1ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b28b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1108324070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1a6cc0 0x56418b1b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.398+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1108324070 >> 172.21.15.3:0/1108324070 conn(0x56418b11b030 msgr2=0x56418b1aa150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.399+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1108324070 shutdown_connections 2023-09-10T01:04:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.399+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1108324070 wait complete. 2023-09-10T01:04:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.399+0000 7fa0c83f7b80 1 -- start start 2023-09-10T01:04:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.399+0000 7fa0c83f7b80 1 Processor -- start 2023-09-10T01:04:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0c83f7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b2132b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b213820 con 0x56418b1a6300 2023-09-10T01:04:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b213990 con 0x56418b1a6cc0 2023-09-10T01:04:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0c83f7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56418b213b50 con 0x56418b1ac1a0 2023-09-10T01:04:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0c83f7b80 10 client.0 _ll_register_callbacks cb 0x7fa0a0009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0aed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b2132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45630/0 (socket says 172.21.15.3:45630) 2023-09-10T01:04:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 -- 172.21.15.3:0/1449080590 learned_addr learned my addr 172.21.15.3:0/1449080590 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 -- 172.21.15.3:0/1449080590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 msgr2=0x56418b2132b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b2132b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 -- 172.21.15.3:0/1449080590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 msgr2=0x56418b1b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.400+0000 7fa0add9b700 1 -- 172.21.15.3:0/1449080590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56418af24d80 con 0x56418b1a6cc0 2023-09-10T01:04:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa0add9b700 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b5c40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400eee0 tx=0x7fa0a4084040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835bdbb2f9fc6d07 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa095ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa095ffb700 1 -- 172.21.15.3:0/1449080590 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4099e60 con 0x56418b1a6cc0 2023-09-10T01:04:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa095ffb700 1 -- 172.21.15.3:0/1449080590 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa0a4099760 con 0x56418b1a6cc0 2023-09-10T01:04:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0a801e030 con 0x56418b1a6cc0 2023-09-10T01:04:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa095ffb700 1 -- 172.21.15.3:0/1449080590 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0a409d590 con 0x56418b1a6cc0 2023-09-10T01:04:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa0c83f7b80 10 client.26054 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56418af0dc00 con 0x56418b1a6cc0 2023-09-10T01:04:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa095ffb700 1 -- 172.21.15.3:0/1449080590 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0a4063020 con 0x56418b1a6cc0 2023-09-10T01:04:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa095ffb700 1 -- 172.21.15.3:0/1449080590 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa0a40a1340 con 0x56418b1a6cc0 2023-09-10T01:04:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa0c83f7b80 10 client.26054 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa0c83f7b80 10 client.26054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.401+0000 7fa0c83f7b80 20 client.26054 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.402+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b215320 0x56418b217740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.402+0000 7fa0c83f7b80 4 client.26054 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:04:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.402+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56418b0f18c0 con 0x56418b215320 2023-09-10T01:04:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.402+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b215320 0x56418b217740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.402+0000 7fa0ae59c700 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b215320 0x56418b217740 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d455462703a323e server_cookie=307d321294d0a5e2 in_seq=0 out_seq=0 2023-09-10T01:04:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.402+0000 7fa095ffb700 10 client.26054 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:21.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:21 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:04:21.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:21 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:21.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:21 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa095ffb700 1 -- 172.21.15.3:0/1449080590 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56418b0f18c0 con 0x56418b215320 2023-09-10T01:04:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa095ffb700 10 client.26054 handle_command_reply: tid=0 2023-09-10T01:04:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 client.26054 shutdown 2023-09-10T01:04:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 msgr2=0x56418b1b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b5c40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400eee0 tx=0x7fa0a4084040 comp rx=0 tx=0).stop 2023-09-10T01:04:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 shutdown_connections 2023-09-10T01:04:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56418b215320 0x56418b217740 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56418b1ac1a0 0x56418b2132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56418b1a6300 0x56418b1b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 --2- 172.21.15.3:0/1449080590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56418b1a6cc0 0x56418b1b5c40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa095ffb700 0 client.26054 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 >> 172.21.15.3:0/1449080590 conn(0x56418b11b030 msgr2=0x56418b1ab090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 shutdown_connections 2023-09-10T01:04:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 1 -- 172.21.15.3:0/1449080590 wait complete. 2023-09-10T01:04:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.507+0000 7fa0c83f7b80 20 client.26054 trim_cache size 0 max 16384 2023-09-10T01:04:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:04:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:04:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:04:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:04:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0980a08d0 msgr2=0x7fa0980a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0980a08d0 0x7fa0980a2d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa0a802d000 tx=0x7fa0a802a000 comp rx=0 tx=0).stop 2023-09-10T01:04:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b0100f80 msgr2=0x7fa0b019eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b0100f80 0x7fa0b019eb00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400ef10 tx=0x7fa0a400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 shutdown_connections 2023-09-10T01:04:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa0980a08d0 0x7fa0980a2d90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0b0106680 0x7fa0b0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0b01005c0 0x7fa0b019e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 --2- 172.21.15.3:0/484137030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b0100f80 0x7fa0b019eb00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 >> 172.21.15.3:0/484137030 conn(0x7fa0b00fc0b0 msgr2=0x7fa0b010b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:21.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 shutdown_connections 2023-09-10T01:04:21.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:21.508+0000 7fa0b49ff700 1 -- 172.21.15.3:0/484137030 wait complete. 2023-09-10T01:04:21.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:21 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:04:21.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:21 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:21.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:21 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:21.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:21 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:04:21.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:21 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:21.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:21 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:21.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 7668 19.73 MB/sec warmup 3 sec latency 41.800 ms 2023-09-10T01:04:22.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:22 smithi092 ceph-mon[134067]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-09-10T01:04:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:22 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:22 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:22 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:04:22.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:22 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:04:22.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:22 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:22 smithi003 ceph-mon[97654]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-09-10T01:04:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:22 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:22 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:22 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:04:22.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:22 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:04:22.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:22 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:22 smithi136 ceph-mon[133708]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-09-10T01:04:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:22 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:22 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:22 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:04:22.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:22 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:04:22.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:22 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:04:22.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 10918 19.50 MB/sec warmup 4 sec latency 6.507 ms 2023-09-10T01:04:23.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:23 smithi003 ceph-mon[97654]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-09-10T01:04:23.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:23 smithi136 ceph-mon[133708]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-09-10T01:04:23.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:23 smithi092 ceph-mon[134067]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-09-10T01:04:23.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 13847 18.33 MB/sec warmup 5 sec latency 6.567 ms 2023-09-10T01:04:24.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 16110 17.15 MB/sec warmup 6 sec latency 13.556 ms 2023-09-10T01:04:25.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 18667 16.77 MB/sec warmup 7 sec latency 6.307 ms 2023-09-10T01:04:26.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:26 smithi003 ceph-mon[97654]: pgmap v39: 129 pgs: 129 active+clean; 7.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 616 KiB/s wr, 137 op/s 2023-09-10T01:04:26.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:26 smithi136 ceph-mon[133708]: pgmap v39: 129 pgs: 129 active+clean; 7.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 616 KiB/s wr, 137 op/s 2023-09-10T01:04:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:26 smithi092 ceph-mon[134067]: pgmap v39: 129 pgs: 129 active+clean; 7.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 616 KiB/s wr, 137 op/s 2023-09-10T01:04:26.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 21581 16.93 MB/sec warmup 8 sec latency 90.815 ms 2023-09-10T01:04:27.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 24330 16.48 MB/sec warmup 9 sec latency 7.504 ms 2023-09-10T01:04:28.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:28 smithi003 ceph-mon[97654]: pgmap v40: 129 pgs: 129 active+clean; 7.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 616 KiB/s wr, 133 op/s 2023-09-10T01:04:28.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:28 smithi136 ceph-mon[133708]: pgmap v40: 129 pgs: 129 active+clean; 7.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 616 KiB/s wr, 133 op/s 2023-09-10T01:04:28.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:28 smithi092 ceph-mon[134067]: pgmap v40: 129 pgs: 129 active+clean; 7.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 616 KiB/s wr, 133 op/s 2023-09-10T01:04:28.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 26758 16.03 MB/sec warmup 10 sec latency 7.149 ms 2023-09-10T01:04:29.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 29319 15.88 MB/sec warmup 11 sec latency 8.301 ms 2023-09-10T01:04:30.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:30 smithi003 ceph-mon[97654]: pgmap v41: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2023-09-10T01:04:30.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:30 smithi136 ceph-mon[133708]: pgmap v41: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2023-09-10T01:04:30.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:30 smithi092 ceph-mon[134067]: pgmap v41: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.2 MiB/s wr, 221 op/s 2023-09-10T01:04:30.734 INFO:tasks.workunit.client.0.smithi003.stdout: 1 32358 16.09 MB/sec warmup 12 sec latency 18.360 ms 2023-09-10T01:04:31.735 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35166 15.88 MB/sec warmup 13 sec latency 12.554 ms 2023-09-10T01:04:32.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:32 smithi003 ceph-mon[97654]: pgmap v42: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 219 op/s 2023-09-10T01:04:32.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:32 smithi136 ceph-mon[133708]: pgmap v42: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 219 op/s 2023-09-10T01:04:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:32 smithi092 ceph-mon[134067]: pgmap v42: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 219 op/s 2023-09-10T01:04:32.735 INFO:tasks.workunit.client.0.smithi003.stdout: 1 37194 15.53 MB/sec warmup 14 sec latency 126.229 ms 2023-09-10T01:04:33.735 INFO:tasks.workunit.client.0.smithi003.stdout: 1 39836 15.45 MB/sec warmup 15 sec latency 11.789 ms 2023-09-10T01:04:34.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:34 smithi003 ceph-mon[97654]: pgmap v43: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 219 op/s 2023-09-10T01:04:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:34 smithi136 ceph-mon[133708]: pgmap v43: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 219 op/s 2023-09-10T01:04:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:34 smithi092 ceph-mon[134067]: pgmap v43: 129 pgs: 129 active+clean; 13 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 219 op/s 2023-09-10T01:04:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 -- 172.21.15.3:0/11033682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d600ffe50 msgr2=0x7f6d6010c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 --2- 172.21.15.3:0/11033682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d600ffe50 0x7f6d6010c310 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009f90 tx=0x7f6d5800b330 comp rx=0 tx=0).stop 2023-09-10T01:04:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 -- 172.21.15.3:0/11033682 shutdown_connections 2023-09-10T01:04:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 --2- 172.21.15.3:0/11033682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d600ffe50 0x7f6d6010c310 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 --2- 172.21.15.3:0/11033682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d60105550 0x7f6d60105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 --2- 172.21.15.3:0/11033682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d600ff490 0x7f6d600ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 -- 172.21.15.3:0/11033682 >> 172.21.15.3:0/11033682 conn(0x7f6d600faf20 msgr2=0x7f6d600fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 -- 172.21.15.3:0/11033682 shutdown_connections 2023-09-10T01:04:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.662+0000 7f6d687ff700 1 -- 172.21.15.3:0/11033682 wait complete. 2023-09-10T01:04:34.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.664+0000 7f6d687ff700 1 Processor -- start 2023-09-10T01:04:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.664+0000 7f6d687ff700 1 -- start start 2023-09-10T01:04:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.664+0000 7f6d687ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d600ff490 0x7f6d60101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d687ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d600ffe50 0x7f6d60101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d687ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d60105550 0x7f6d601a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d687ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d60101e70 con 0x7f6d600ffe50 2023-09-10T01:04:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d687ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d601a4e10 con 0x7f6d600ff490 2023-09-10T01:04:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d687ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d601a4f50 con 0x7f6d60105550 2023-09-10T01:04:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d600ff490 0x7f6d60101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d600ff490 0x7f6d60101100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34236/0 (socket says 172.21.15.3:34236) 2023-09-10T01:04:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 -- 172.21.15.3:0/90537148 learned_addr learned my addr 172.21.15.3:0/90537148 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d600ffe50 0x7f6d60101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d60105550 0x7f6d601a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 -- 172.21.15.3:0/90537148 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d60105550 msgr2=0x7f6d601a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d60105550 0x7f6d601a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 -- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d600ffe50 msgr2=0x7f6d60101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d600ffe50 0x7f6d60101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 -- 172.21.15.3:0/90537148 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50009520 con 0x7f6d600ff490 2023-09-10T01:04:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.665+0000 7f6d6659b700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d600ff490 0x7f6d60101100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00ec10 tx=0x7f6d5c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.666+0000 7f6d577fe700 1 -- 172.21.15.3:0/90537148 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c00cd40 con 0x7f6d600ff490 2023-09-10T01:04:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.666+0000 7f6d577fe700 1 -- 172.21.15.3:0/90537148 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d5c0044e0 con 0x7f6d600ff490 2023-09-10T01:04:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.666+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c0095f0 con 0x7f6d600ff490 2023-09-10T01:04:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.666+0000 7f6d577fe700 1 -- 172.21.15.3:0/90537148 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c010690 con 0x7f6d600ff490 2023-09-10T01:04:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.666+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d58009c40 con 0x7f6d600ff490 2023-09-10T01:04:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.669+0000 7f6d577fe700 1 -- 172.21.15.3:0/90537148 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f6d5c003710 con 0x7f6d600ff490 2023-09-10T01:04:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.670+0000 7f6d7c12cb80 1 Processor -- start 2023-09-10T01:04:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d7c12cb80 1 -- start start 2023-09-10T01:04:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d577fe700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6d4c0a0950 0x7f6d4c0a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d577fe700 1 -- 172.21.15.3:0/90537148 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6d5c014070 con 0x7f6d600ff490 2023-09-10T01:04:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bbbc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbb6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6cc0 0x56050bbc2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bb0f6d0 con 0x56050bbbc1a0 2023-09-10T01:04:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bb0cff0 con 0x56050bbb6cc0 2023-09-10T01:04:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bb088e0 con 0x56050bbb6300 2023-09-10T01:04:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6d4c0a0950 0x7f6d4c0a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bbbc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bbbc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48862/0 (socket says 172.21.15.3:48862) 2023-09-10T01:04:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.671+0000 7f6d6659b700 1 -- 172.21.15.3:0/966727589 learned_addr learned my addr 172.21.15.3:0/966727589 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbb6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6cc0 0x56050bbc2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d6659b700 1 -- 172.21.15.3:0/966727589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 msgr2=0x56050bbb6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d6659b700 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbb6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d6659b700 1 -- 172.21.15.3:0/966727589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6cc0 msgr2=0x56050bbc2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d6659b700 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6cc0 0x56050bbc2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d6659b700 1 -- 172.21.15.3:0/966727589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56050b934d80 con 0x56050bbbc1a0 2023-09-10T01:04:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbb6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6d4c0a0950 0x7f6d4c0a2e10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009790 tx=0x7f6d58007e90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d6659b700 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bbbc580 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c095f10 tx=0x7f6d5c099550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb04b8d3f9f5bce6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d54ff9700 1 -- 172.21.15.3:0/966727589 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c099a10 con 0x56050bbbc1a0 2023-09-10T01:04:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d54ff9700 1 -- 172.21.15.3:0/966727589 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d5c09d840 con 0x56050bbbc1a0 2023-09-10T01:04:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.672+0000 7f6d54ff9700 1 -- 172.21.15.3:0/966727589 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c099ed0 con 0x56050bbbc1a0 2023-09-10T01:04:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/966727589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 msgr2=0x56050bbbc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bbbc580 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c095f10 tx=0x7f6d5c099550 comp rx=0 tx=0).stop 2023-09-10T01:04:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/966727589 shutdown_connections 2023-09-10T01:04:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6cc0 0x56050bbc2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bbbc580 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/966727589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbb6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/966727589 >> 172.21.15.3:0/966727589 conn(0x56050bb2b030 msgr2=0x56050bbb1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/966727589 shutdown_connections 2023-09-10T01:04:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.673+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/966727589 wait complete. 2023-09-10T01:04:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.674+0000 7f6d7c12cb80 1 -- start start 2023-09-10T01:04:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.674+0000 7f6d7c12cb80 1 Processor -- start 2023-09-10T01:04:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.674+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbc2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbc00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbbc1a0 0x56050bbc06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050baa6b10 con 0x56050bbb6cc0 2023-09-10T01:04:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050baa6810 con 0x56050bbbc1a0 2023-09-10T01:04:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050b98a8c0 con 0x56050bbb6300 2023-09-10T01:04:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d7c12cb80 10 client.0 _ll_register_callbacks cb 0x56050b9f0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbc2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbc2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50272/0 (socket says 172.21.15.3:50272) 2023-09-10T01:04:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 -- 172.21.15.3:0/3876464487 learned_addr learned my addr 172.21.15.3:0/3876464487 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbbc1a0 0x56050bbc06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbc00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 -- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbbc1a0 msgr2=0x56050bbc06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbbc1a0 0x56050bbc06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 -- 172.21.15.3:0/3876464487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 msgr2=0x56050bbc00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbc00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 -- 172.21.15.3:0/3876464487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d58009c40 con 0x56050bbb6300 2023-09-10T01:04:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbbc1a0 0x56050bbc06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d6659b700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbc2dc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00c660 tx=0x7f6d5c003a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37291a9f7a820c7c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.675+0000 7f6d49ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d49ffb700 1 -- 172.21.15.3:0/3876464487 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c099a10 con 0x56050bbb6300 2023-09-10T01:04:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56050ba60490 con 0x56050bbb6300 2023-09-10T01:04:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d49ffb700 1 -- 172.21.15.3:0/3876464487 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6d5c099c70 con 0x56050bbb6300 2023-09-10T01:04:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d49ffb700 1 -- 172.21.15.3:0/3876464487 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d5c099e30 con 0x56050bbb6300 2023-09-10T01:04:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 10 client.25993 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56050b91dc00 con 0x56050bbb6300 2023-09-10T01:04:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d49ffb700 1 -- 172.21.15.3:0/3876464487 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d5c0a5080 con 0x56050bbb6300 2023-09-10T01:04:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d49ffb700 1 -- 172.21.15.3:0/3876464487 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6d5c0ca0d0 con 0x56050bbb6300 2023-09-10T01:04:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 10 client.25993 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 10 client.25993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 20 client.25993 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bbcfff0 0x56050bbd2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 4 client.25993 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.676+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56050bb1d840 con 0x56050bbcfff0 2023-09-10T01:04:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.677+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bbcfff0 0x56050bbd2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.677+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bbcfff0 0x56050bbd2410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=560e080a0e2ccfc2 server_cookie=b85886891671c8e2 in_seq=0 out_seq=0 2023-09-10T01:04:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.677+0000 7f6d49ffb700 10 client.25993 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d49ffb700 1 -- 172.21.15.3:0/3876464487 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56050bb1d840 con 0x56050bbcfff0 2023-09-10T01:04:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d49ffb700 10 client.25993 handle_command_reply: tid=0 2023-09-10T01:04:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 client.25993 shutdown 2023-09-10T01:04:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 msgr2=0x56050bbc2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbc2dc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00c660 tx=0x7f6d5c003a80 comp rx=0 tx=0).stop 2023-09-10T01:04:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 shutdown_connections 2023-09-10T01:04:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bbcfff0 0x56050bbd2410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6300 0x56050bbc2dc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbbc1a0 0x56050bbc06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d49ffb700 0 client.25993 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/3876464487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbc00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 >> 172.21.15.3:0/3876464487 conn(0x56050bb2b030 msgr2=0x56050bbb07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 shutdown_connections 2023-09-10T01:04:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/3876464487 wait complete. 2023-09-10T01:04:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.678+0000 7f6d7c12cb80 20 client.25993 trim_cache size 0 max 16384 2023-09-10T01:04:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 Processor -- start 2023-09-10T01:04:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 -- start start 2023-09-10T01:04:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbba120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbbc1a0 0x56050bbba660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bb1d840 con 0x56050bbb6cc0 2023-09-10T01:04:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bbab8a0 con 0x56050bbb6300 2023-09-10T01:04:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bbbbc40 con 0x56050bbbc1a0 2023-09-10T01:04:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.694+0000 7f6d65d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbba120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d66d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbbc1a0 0x56050bbba660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34282/0 (socket says 172.21.15.3:34282) 2023-09-10T01:04:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d6659b700 1 -- 172.21.15.3:0/2359579193 learned_addr learned my addr 172.21.15.3:0/2359579193 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d66d9c700 1 -- 172.21.15.3:0/2359579193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 msgr2=0x56050bbc28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d66d9c700 1 -- 172.21.15.3:0/2359579193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 msgr2=0x56050bbba120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbba120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d66d9c700 1 -- 172.21.15.3:0/2359579193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56050b95b1f0 con 0x56050bbbc1a0 2023-09-10T01:04:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d6659b700 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbba120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.695+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbbc1a0 0x56050bbba660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d60103210 tx=0x7f6d58022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbb3eea155226ce9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d557fa700 1 -- 172.21.15.3:0/2359579193 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5802e070 con 0x56050bbbc1a0 2023-09-10T01:04:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d557fa700 1 -- 172.21.15.3:0/2359579193 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d58003de0 con 0x56050bbbc1a0 2023-09-10T01:04:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d557fa700 1 -- 172.21.15.3:0/2359579193 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d58005950 con 0x56050bbbc1a0 2023-09-10T01:04:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/2359579193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbbc1a0 msgr2=0x56050bbba660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbbc1a0 0x56050bbba660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d60103210 tx=0x7f6d58022000 comp rx=0 tx=0).stop 2023-09-10T01:04:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/2359579193 shutdown_connections 2023-09-10T01:04:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbbc1a0 0x56050bbba660 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/2359579193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbb6cc0 0x56050bbba120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/2359579193 >> 172.21.15.3:0/2359579193 conn(0x56050bb2b030 msgr2=0x56050bbcc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/2359579193 shutdown_connections 2023-09-10T01:04:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.696+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/2359579193 wait complete. 2023-09-10T01:04:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 -- start start 2023-09-10T01:04:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 Processor -- start 2023-09-10T01:04:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6cc0 0x56050bbc3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bc22dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bbc43a0 con 0x56050bbbc1a0 2023-09-10T01:04:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bc23310 con 0x56050bbb6300 2023-09-10T01:04:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050bc23480 con 0x56050bbb6cc0 2023-09-10T01:04:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.697+0000 7f6d7c12cb80 10 client.0 _ll_register_callbacks cb 0x7f6d5c0997e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d6659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34294/0 (socket says 172.21.15.3:34294) 2023-09-10T01:04:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d6659b700 1 -- 172.21.15.3:0/1799225276 learned_addr learned my addr 172.21.15.3:0/1799225276 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d65d9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6cc0 0x56050bbc3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d66d9c700 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bc22dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d65d9a700 1 -- 172.21.15.3:0/1799225276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 msgr2=0x56050bbc36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d65d9a700 1 -- 172.21.15.3:0/1799225276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 msgr2=0x56050bc22dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bc22dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d65d9a700 1 -- 172.21.15.3:0/1799225276 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56050b91dc00 con 0x56050bbb6cc0 2023-09-10T01:04:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d65d9a700 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6cc0 0x56050bbc3c30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x56050bbd2ce0 tx=0x7f6d50010710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb24772e618a9fce server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.698+0000 7f6d49ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d49ffb700 1 -- 172.21.15.3:0/1799225276 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50017040 con 0x56050bbb6cc0 2023-09-10T01:04:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d5c09d840 con 0x56050bbb6cc0 2023-09-10T01:04:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d49ffb700 1 -- 172.21.15.3:0/1799225276 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6d50010ce0 con 0x56050bbb6cc0 2023-09-10T01:04:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d49ffb700 1 -- 172.21.15.3:0/1799225276 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d50010ea0 con 0x56050bbb6cc0 2023-09-10T01:04:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 10 client.25999 fetch_fsmap learned FSMap version 22 2023-09-10T01:04:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f6d50009b20 con 0x56050bbb6cc0 2023-09-10T01:04:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d49ffb700 1 -- 172.21.15.3:0/1799225276 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d5c09d840 con 0x56050bbb6cc0 2023-09-10T01:04:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d49ffb700 1 -- 172.21.15.3:0/1799225276 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 22) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6d500053d0 con 0x56050bbb6cc0 2023-09-10T01:04:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 10 client.25999 fetch_fsmap finished waiting for FSMap version 22 2023-09-10T01:04:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 10 client.25999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 20 client.25999 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bc24c20 0x56050bc27010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 4 client.25999 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:04:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.699+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56050bb018c0 con 0x56050bc24c20 2023-09-10T01:04:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.700+0000 7f6d6659b700 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bc24c20 0x56050bc27010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.700+0000 7f6d6659b700 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bc24c20 0x56050bc27010 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfa9944aadea1078 server_cookie=7c56c03f5575d8a4 in_seq=0 out_seq=0 2023-09-10T01:04:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.700+0000 7f6d49ffb700 10 client.25999 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:34.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 42843 15.63 MB/sec warmup 16 sec latency 39.305 ms 2023-09-10T01:04:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d49ffb700 1 -- 172.21.15.3:0/1799225276 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56050bb018c0 con 0x56050bc24c20 2023-09-10T01:04:34.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d49ffb700 10 client.25999 handle_command_reply: tid=0 2023-09-10T01:04:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 client.25999 shutdown 2023-09-10T01:04:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6cc0 msgr2=0x56050bbc3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6cc0 0x56050bbc3c30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x56050bbd2ce0 tx=0x7f6d50010710 comp rx=0 tx=0).stop 2023-09-10T01:04:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 shutdown_connections 2023-09-10T01:04:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56050bc24c20 0x56050bc27010 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56050bbbc1a0 0x56050bc22dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56050bbb6300 0x56050bbc36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d49ffb700 0 client.25999 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 --2- 172.21.15.3:0/1799225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56050bbb6cc0 0x56050bbc3c30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 >> 172.21.15.3:0/1799225276 conn(0x56050bb2b030 msgr2=0x56050bbbb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 shutdown_connections 2023-09-10T01:04:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 1 -- 172.21.15.3:0/1799225276 wait complete. 2023-09-10T01:04:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.971+0000 7f6d7c12cb80 20 client.25999 trim_cache size 0 max 16384 2023-09-10T01:04:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:04:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:04:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:04:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:04:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6d4c0a0950 msgr2=0x7f6d4c0a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6d4c0a0950 0x7f6d4c0a2e10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009790 tx=0x7f6d58007e90 comp rx=0 tx=0).stop 2023-09-10T01:04:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d600ff490 msgr2=0x7f6d60101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d600ff490 0x7f6d60101100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00ec10 tx=0x7f6d5c00ef20 comp rx=0 tx=0).stop 2023-09-10T01:04:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 shutdown_connections 2023-09-10T01:04:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6d4c0a0950 0x7f6d4c0a2e10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d60105550 0x7f6d601a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d600ff490 0x7f6d60101100 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 --2- 172.21.15.3:0/90537148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d600ffe50 0x7f6d60101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:34.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 >> 172.21.15.3:0/90537148 conn(0x7f6d600faf20 msgr2=0x7f6d600fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:34.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 shutdown_connections 2023-09-10T01:04:34.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:34.972+0000 7f6d687ff700 1 -- 172.21.15.3:0/90537148 wait complete. 2023-09-10T01:04:35.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 44292 15.04 MB/sec warmup 17 sec latency 117.533 ms 2023-09-10T01:04:36.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:36 smithi003 ceph-mon[97654]: pgmap v44: 129 pgs: 129 active+clean; 20 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.8 MiB/s wr, 302 op/s 2023-09-10T01:04:36.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:36 smithi136 ceph-mon[133708]: pgmap v44: 129 pgs: 129 active+clean; 20 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.8 MiB/s wr, 302 op/s 2023-09-10T01:04:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:36 smithi092 ceph-mon[134067]: pgmap v44: 129 pgs: 129 active+clean; 20 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.8 MiB/s wr, 302 op/s 2023-09-10T01:04:36.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46739 14.97 MB/sec warmup 18 sec latency 41.703 ms 2023-09-10T01:04:37.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 49447 14.94 MB/sec warmup 19 sec latency 104.471 ms 2023-09-10T01:04:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:38 smithi003 ceph-mon[97654]: pgmap v45: 129 pgs: 129 active+clean; 20 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.2 MiB/s wr, 171 op/s 2023-09-10T01:04:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:38 smithi136 ceph-mon[133708]: pgmap v45: 129 pgs: 129 active+clean; 20 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.2 MiB/s wr, 171 op/s 2023-09-10T01:04:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:38 smithi092 ceph-mon[134067]: pgmap v45: 129 pgs: 129 active+clean; 20 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 1.2 MiB/s wr, 171 op/s 2023-09-10T01:04:38.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 51385 14.69 MB/sec warmup 20 sec latency 54.347 ms 2023-09-10T01:04:39.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53446 14.64 MB/sec warmup 21 sec latency 342.642 ms 2023-09-10T01:04:40.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:40 smithi003 ceph-mon[97654]: pgmap v46: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.8 MiB/s wr, 459 op/s 2023-09-10T01:04:40.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:40 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:04:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:40 smithi136 ceph-mon[133708]: pgmap v46: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.8 MiB/s wr, 459 op/s 2023-09-10T01:04:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:40 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:04:40.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:40 smithi092 ceph-mon[134067]: pgmap v46: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 1.8 MiB/s wr, 459 op/s 2023-09-10T01:04:40.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:40 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:04:40.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:04:40.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53479 13.97 MB/sec warmup 22 sec latency 879.451 ms 2023-09-10T01:04:40.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 -- 172.21.15.3:0/2926379751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 msgr2=0x7f453810a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2926379751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f453810a6a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f452c009990 tx=0x7f452c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 -- 172.21.15.3:0/2926379751 shutdown_connections 2023-09-10T01:04:40.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2926379751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45381018a0 0x7f4538101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2926379751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453810ac70 0x7f453810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2926379751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f453810a6a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 -- 172.21.15.3:0/2926379751 >> 172.21.15.3:0/2926379751 conn(0x7f4538076710 msgr2=0x7f4538076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:40.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 -- 172.21.15.3:0/2926379751 shutdown_connections 2023-09-10T01:04:40.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.912+0000 7f453d9ff700 1 -- 172.21.15.3:0/2926379751 wait complete. 2023-09-10T01:04:40.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.913+0000 7f453d9ff700 1 Processor -- start 2023-09-10T01:04:40.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f453d9ff700 1 -- start start 2023-09-10T01:04:40.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f453d9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45381018a0 0x7f45380ff4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f453d9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f45380ffa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f453d9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453810ac70 0x7f45381a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f453d9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4538100260 con 0x7f45381021c0 2023-09-10T01:04:40.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f453d9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45381a4eb0 con 0x7f453810ac70 2023-09-10T01:04:40.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f453d9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45381a4ff0 con 0x7f45381018a0 2023-09-10T01:04:40.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f45367fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f45380ffa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f4536ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45381018a0 0x7f45380ff4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f45367fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f45380ffa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48904/0 (socket says 172.21.15.3:48904) 2023-09-10T01:04:40.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f45367fc700 1 -- 172.21.15.3:0/2379767227 learned_addr learned my addr 172.21.15.3:0/2379767227 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:40.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.914+0000 7f45377fe700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453810ac70 0x7f45381a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f45367fc700 1 -- 172.21.15.3:0/2379767227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45381018a0 msgr2=0x7f45380ff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f45367fc700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45381018a0 0x7f45380ff4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f45367fc700 1 -- 172.21.15.3:0/2379767227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453810ac70 msgr2=0x7f45381a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f45367fc700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453810ac70 0x7f45381a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f45367fc700 1 -- 172.21.15.3:0/2379767227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0095f0 con 0x7f45381021c0 2023-09-10T01:04:40.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f45377fe700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453810ac70 0x7f45381a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:40.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f45367fc700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f45380ffa30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f452c005910 tx=0x7f452c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:40.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f453c9fd700 1 -- 172.21.15.3:0/2379767227 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c01d070 con 0x7f45381021c0 2023-09-10T01:04:40.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45381a51f0 con 0x7f45381021c0 2023-09-10T01:04:40.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f453c9fd700 1 -- 172.21.15.3:0/2379767227 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f452c022470 con 0x7f45381021c0 2023-09-10T01:04:40.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f453c9fd700 1 -- 172.21.15.3:0/2379767227 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c017650 con 0x7f45381021c0 2023-09-10T01:04:40.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.915+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45381a5670 con 0x7f45381021c0 2023-09-10T01:04:40.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.918+0000 7f453c9fd700 1 -- 172.21.15.3:0/2379767227 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f452c0225e0 con 0x7f45381021c0 2023-09-10T01:04:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.920+0000 7f4551379b80 1 Processor -- start 2023-09-10T01:04:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.920+0000 7f453c9fd700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45200a08d0 0x7f45200a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.920+0000 7f4551379b80 1 -- start start 2023-09-10T01:04:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.920+0000 7f453c9fd700 1 -- 172.21.15.3:0/2379767227 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f452c0cdb20 con 0x7f45381021c0 2023-09-10T01:04:40.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.920+0000 7f4551379b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309a710 0x560ea309aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.920+0000 7f45377fe700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45200a08d0 0x7f45200a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.920+0000 7f4551379b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4551379b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea30a4e40 0x560ea30a7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4551379b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea2ff3730 con 0x560ea309a710 2023-09-10T01:04:40.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4551379b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea2ff1050 con 0x560ea30a4e40 2023-09-10T01:04:40.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4551379b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea2fec940 con 0x560ea309b030 2023-09-10T01:04:40.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309a710 0x560ea309aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309a710 0x560ea309aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48918/0 (socket says 172.21.15.3:48918) 2023-09-10T01:04:40.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 -- 172.21.15.3:0/579935077 learned_addr learned my addr 172.21.15.3:0/579935077 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:40.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f45367fc700 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f45377fe700 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea30a4e40 0x560ea30a7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 -- 172.21.15.3:0/579935077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 msgr2=0x560ea30a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 -- 172.21.15.3:0/579935077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea30a4e40 msgr2=0x560ea30a7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea30a4e40 0x560ea30a7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f4536ffd700 1 -- 172.21.15.3:0/579935077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ea2e18de0 con 0x560ea309a710 2023-09-10T01:04:40.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.921+0000 7f45377fe700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45200a08d0 0x7f45200a2d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4538101600 tx=0x7f4530007b00 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:40.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.922+0000 7f4536ffd700 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309a710 0x560ea309aaf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4528009d80 tx=0x7f4528011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be1e6c62864a722 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:40.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.922+0000 7f451dffb700 1 -- 172.21.15.3:0/579935077 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4528004230 con 0x560ea309a710 2023-09-10T01:04:40.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.922+0000 7f451dffb700 1 -- 172.21.15.3:0/579935077 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4528004390 con 0x560ea309a710 2023-09-10T01:04:40.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.922+0000 7f451dffb700 1 -- 172.21.15.3:0/579935077 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45280197a0 con 0x560ea309a710 2023-09-10T01:04:40.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 -- 172.21.15.3:0/579935077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309a710 msgr2=0x560ea309aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309a710 0x560ea309aaf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4528009d80 tx=0x7f4528011720 comp rx=0 tx=0).stop 2023-09-10T01:04:40.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 -- 172.21.15.3:0/579935077 shutdown_connections 2023-09-10T01:04:40.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea30a4e40 0x560ea30a7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309a710 0x560ea309aaf0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 --2- 172.21.15.3:0/579935077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 -- 172.21.15.3:0/579935077 >> 172.21.15.3:0/579935077 conn(0x560ea300f090 msgr2=0x560ea3095bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:40.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 -- 172.21.15.3:0/579935077 shutdown_connections 2023-09-10T01:04:40.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.923+0000 7f4551379b80 1 -- 172.21.15.3:0/579935077 wait complete. 2023-09-10T01:04:40.932 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.924+0000 7f4551379b80 1 -- start start 2023-09-10T01:04:40.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.924+0000 7f4551379b80 1 Processor -- start 2023-09-10T01:04:40.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4551379b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30a4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4551379b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4551379b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea30a4e40 0x560ea30a1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4551379b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea2f8ab70 con 0x560ea30a4e40 2023-09-10T01:04:40.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4551379b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea2f8a870 con 0x560ea309a710 2023-09-10T01:04:40.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4551379b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea2e6e920 con 0x560ea309b030 2023-09-10T01:04:40.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4551379b80 10 client.0 _ll_register_callbacks cb 0x560ea2ed4a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:40.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f45367fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f4536ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f45377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea30a4e40 0x560ea30a1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f45367fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50316/0 (socket says 172.21.15.3:50316) 2023-09-10T01:04:40.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f45367fc700 1 -- 172.21.15.3:0/262708067 learned_addr learned my addr 172.21.15.3:0/262708067 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:40.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f45377fe700 1 -- 172.21.15.3:0/262708067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 msgr2=0x560ea30a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.925+0000 7f45377fe700 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f45377fe700 1 -- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 msgr2=0x560ea30a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f45377fe700 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f45377fe700 1 -- 172.21.15.3:0/262708067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45280099e0 con 0x560ea30a4e40 2023-09-10T01:04:40.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4536ffd700 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:40.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f45377fe700 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea30a4e40 0x560ea30a1f40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4530007f00 tx=0x7f4530004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3798571b06d004e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:40.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4516ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:40.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4516ffd700 1 -- 172.21.15.3:0/262708067 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453000e050 con 0x560ea30a4e40 2023-09-10T01:04:40.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ea2f444f0 con 0x560ea30a4e40 2023-09-10T01:04:40.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4516ffd700 1 -- 172.21.15.3:0/262708067 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4530024430 con 0x560ea30a4e40 2023-09-10T01:04:40.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4516ffd700 1 -- 172.21.15.3:0/262708067 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4530009340 con 0x560ea30a4e40 2023-09-10T01:04:40.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4551379b80 10 client.16032 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:40.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.926+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560ea2e01c60 con 0x560ea30a4e40 2023-09-10T01:04:40.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4516ffd700 1 -- 172.21.15.3:0/262708067 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560ea2f444f0 con 0x560ea30a4e40 2023-09-10T01:04:40.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4516ffd700 1 -- 172.21.15.3:0/262708067 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f4530014050 con 0x560ea30a4e40 2023-09-10T01:04:40.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4551379b80 10 client.16032 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:40.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4551379b80 10 client.16032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:40.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4551379b80 20 client.16032 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:40.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4551379b80 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea30b48c0 0x560ea30b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4551379b80 4 client.16032 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:40.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ea30018a0 con 0x560ea30b48c0 2023-09-10T01:04:40.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.927+0000 7f4536ffd700 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea30b48c0 0x560ea30b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.928+0000 7f4536ffd700 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea30b48c0 0x560ea30b6ce0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3830a3d221e4cf49 server_cookie=bbd6cc0cb68470a0 in_seq=0 out_seq=0 2023-09-10T01:04:40.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.928+0000 7f4516ffd700 10 client.16032 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:40.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.928+0000 7f4516ffd700 1 -- 172.21.15.3:0/262708067 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560ea30018a0 con 0x560ea30b48c0 2023-09-10T01:04:40.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.928+0000 7f4516ffd700 10 client.16032 handle_command_reply: tid=0 2023-09-10T01:04:40.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 client.16032 shutdown 2023-09-10T01:04:40.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea30a4e40 msgr2=0x560ea30a1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea30a4e40 0x560ea30a1f40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4530007f00 tx=0x7f4530004b40 comp rx=0 tx=0).stop 2023-09-10T01:04:40.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 shutdown_connections 2023-09-10T01:04:40.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea30b48c0 0x560ea30b6ce0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea30a4e40 0x560ea30a1f40 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 --2- 172.21.15.3:0/262708067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea309b030 0x560ea30a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 >> 172.21.15.3:0/262708067 conn(0x560ea300f090 msgr2=0x560ea30957e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:40.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4516ffd700 0 client.16032 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:40.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 shutdown_connections 2023-09-10T01:04:40.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 1 -- 172.21.15.3:0/262708067 wait complete. 2023-09-10T01:04:40.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.929+0000 7f4551379b80 20 client.16032 trim_cache size 0 max 16384 2023-09-10T01:04:40.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 Processor -- start 2023-09-10T01:04:40.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 -- start start 2023-09-10T01:04:40.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea309ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea30b1220 con 0x560ea309b030 2023-09-10T01:04:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea309c640 con 0x560ea309a710 2023-09-10T01:04:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4551379b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea30b1f00 con 0x560ea30a4e40 2023-09-10T01:04:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f45367fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f45367fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30a42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48934/0 (socket says 172.21.15.3:48934) 2023-09-10T01:04:40.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f4536ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.947+0000 7f45367fc700 1 -- 172.21.15.3:0/2224678358 learned_addr learned my addr 172.21.15.3:0/2224678358 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f45377fe700 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea309ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f4536ffd700 1 -- 172.21.15.3:0/2224678358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 msgr2=0x560ea309ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f4536ffd700 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea309ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f4536ffd700 1 -- 172.21.15.3:0/2224678358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 msgr2=0x560ea30a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f4536ffd700 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30a42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f4536ffd700 1 -- 172.21.15.3:0/2224678358 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ea2e3f250 con 0x560ea309a710 2023-09-10T01:04:40.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f45367fc700 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30a42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:40.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f4536ffd700 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30ab9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4528006010 tx=0x7f4528009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=755bb4cfda0a74c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:40.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f451e7fc700 1 -- 172.21.15.3:0/2224678358 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452802f070 con 0x560ea309a710 2023-09-10T01:04:40.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f451e7fc700 1 -- 172.21.15.3:0/2224678358 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4528005c60 con 0x560ea309a710 2023-09-10T01:04:40.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.948+0000 7f451e7fc700 1 -- 172.21.15.3:0/2224678358 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45280053c0 con 0x560ea309a710 2023-09-10T01:04:40.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 -- 172.21.15.3:0/2224678358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 msgr2=0x560ea30ab9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30ab9f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4528006010 tx=0x7f4528009260 comp rx=0 tx=0).stop 2023-09-10T01:04:40.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 -- 172.21.15.3:0/2224678358 shutdown_connections 2023-09-10T01:04:40.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea309ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30ab9f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 --2- 172.21.15.3:0/2224678358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30a42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 -- 172.21.15.3:0/2224678358 >> 172.21.15.3:0/2224678358 conn(0x560ea300f090 msgr2=0x560ea309ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:40.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 -- 172.21.15.3:0/2224678358 shutdown_connections 2023-09-10T01:04:40.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 -- 172.21.15.3:0/2224678358 wait complete. 2023-09-10T01:04:40.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.949+0000 7f4551379b80 1 -- start start 2023-09-10T01:04:40.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 1 Processor -- start 2023-09-10T01:04:40.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30fc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30fcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea3101760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea3101cd0 con 0x560ea309b030 2023-09-10T01:04:40.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea3101e40 con 0x560ea309a710 2023-09-10T01:04:40.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ea3102000 con 0x560ea30a4e40 2023-09-10T01:04:40.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4551379b80 10 client.0 _ll_register_callbacks cb 0x560ea2ed4a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:40.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f45367fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30fcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f45367fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30fcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48944/0 (socket says 172.21.15.3:48944) 2023-09-10T01:04:40.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f45367fc700 1 -- 172.21.15.3:0/354869391 learned_addr learned my addr 172.21.15.3:0/354869391 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:40.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f4536ffd700 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30fc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.950+0000 7f45377fe700 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea3101760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f45367fc700 1 -- 172.21.15.3:0/354869391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 msgr2=0x560ea3101760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f45367fc700 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea3101760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f45367fc700 1 -- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 msgr2=0x560ea30fc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:40.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f45367fc700 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30fc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f45367fc700 1 -- 172.21.15.3:0/354869391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45381a5670 con 0x560ea309b030 2023-09-10T01:04:40.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f45367fc700 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30fcdd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f452c0968c0 tx=0x7f452c022f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63bae9c879aadc00 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:40.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f4516ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:40.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f4516ffd700 1 -- 172.21.15.3:0/354869391 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c0a2890 con 0x560ea309b030 2023-09-10T01:04:40.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f4516ffd700 1 -- 172.21.15.3:0/354869391 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f452c09b240 con 0x560ea309b030 2023-09-10T01:04:40.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4528004390 con 0x560ea309b030 2023-09-10T01:04:40.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f4516ffd700 1 -- 172.21.15.3:0/354869391 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f452c09a4a0 con 0x560ea309b030 2023-09-10T01:04:40.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f4551379b80 10 client.16038 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:40.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.951+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560ea2e18de0 con 0x560ea309b030 2023-09-10T01:04:40.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4516ffd700 1 -- 172.21.15.3:0/354869391 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4528004390 con 0x560ea309b030 2023-09-10T01:04:40.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4516ffd700 1 -- 172.21.15.3:0/354869391 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f452c0cafe0 con 0x560ea309b030 2023-09-10T01:04:40.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4551379b80 10 client.16038 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:40.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4551379b80 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:40.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4551379b80 20 client.16038 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:40.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4551379b80 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea3109540 0x560ea310b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:40.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4551379b80 4 client.16038 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:04:40.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560ea2fe5920 con 0x560ea3109540 2023-09-10T01:04:40.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.952+0000 7f4536ffd700 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea3109540 0x560ea310b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:40.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4536ffd700 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea3109540 0x560ea310b960 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e441dddf2f96fbbb server_cookie=a647d98cd51a7fdd in_seq=0 out_seq=0 2023-09-10T01:04:40.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4516ffd700 10 client.16038 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:40.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4516ffd700 1 -- 172.21.15.3:0/354869391 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560ea2fe5920 con 0x560ea3109540 2023-09-10T01:04:40.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4516ffd700 10 client.16038 handle_command_reply: tid=0 2023-09-10T01:04:40.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 client.16038 shutdown 2023-09-10T01:04:40.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 msgr2=0x560ea30fcdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30fcdd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f452c0968c0 tx=0x7f452c022f90 comp rx=0 tx=0).stop 2023-09-10T01:04:40.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 shutdown_connections 2023-09-10T01:04:40.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560ea3109540 0x560ea310b960 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ea30a4e40 0x560ea3101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560ea309a710 0x560ea30fc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 --2- 172.21.15.3:0/354869391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ea309b030 0x560ea30fcdd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4516ffd700 0 client.16038 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:40.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 >> 172.21.15.3:0/354869391 conn(0x560ea300f090 msgr2=0x560ea3099700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:40.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 shutdown_connections 2023-09-10T01:04:40.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 1 -- 172.21.15.3:0/354869391 wait complete. 2023-09-10T01:04:40.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.953+0000 7f4551379b80 20 client.16038 trim_cache size 0 max 16384 2023-09-10T01:04:40.971 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:04:40.971 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:04:40.972 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:04:40.972 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:04:40.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45200a08d0 msgr2=0x7f45200a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45200a08d0 0x7f45200a2d90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4538101600 tx=0x7f4530007b00 comp rx=0 tx=0).stop 2023-09-10T01:04:40.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 msgr2=0x7f45380ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:40.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f45380ffa30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f452c005910 tx=0x7f452c005720 comp rx=0 tx=0).stop 2023-09-10T01:04:40.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 shutdown_connections 2023-09-10T01:04:40.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45200a08d0 0x7f45200a2d90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45381018a0 0x7f45380ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f453810ac70 0x7f45381a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 --2- 172.21.15.3:0/2379767227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45381021c0 0x7f45380ffa30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:40.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 >> 172.21.15.3:0/2379767227 conn(0x7f4538076710 msgr2=0x7f45380fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:40.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 shutdown_connections 2023-09-10T01:04:40.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:40.954+0000 7f453d9ff700 1 -- 172.21.15.3:0/2379767227 wait complete. 2023-09-10T01:04:40.977 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:04:40.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:04:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 -- 172.21.15.3:0/2255696366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 msgr2=0x7f14ac10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 --2- 172.21.15.3:0/2255696366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac10d500 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f14a400b0a0 tx=0x7f14a400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 -- 172.21.15.3:0/2255696366 shutdown_connections 2023-09-10T01:04:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 --2- 172.21.15.3:0/2255696366 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac106640 0x7f14ac106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 --2- 172.21.15.3:0/2255696366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac10d500 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 --2- 172.21.15.3:0/2255696366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100630 0x7f14ac100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 -- 172.21.15.3:0/2255696366 >> 172.21.15.3:0/2255696366 conn(0x7f14ac0fc140 msgr2=0x7f14ac0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 -- 172.21.15.3:0/2255696366 shutdown_connections 2023-09-10T01:04:41.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.221+0000 7f14b0fff700 1 -- 172.21.15.3:0/2255696366 wait complete. 2023-09-10T01:04:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.222+0000 7f14b0fff700 1 Processor -- start 2023-09-10T01:04:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.222+0000 7f14b0fff700 1 -- start start 2023-09-10T01:04:41.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.222+0000 7f14b0fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100630 0x7f14ac19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.222+0000 7f14b0fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14b0fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac106640 0x7f14ac198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14b0fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac198df0 con 0x7f14ac100ff0 2023-09-10T01:04:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14b0fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac198f60 con 0x7f14ac100630 2023-09-10T01:04:41.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14b0fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac199120 con 0x7f14ac106640 2023-09-10T01:04:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac19ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48954/0 (socket says 172.21.15.3:48954) 2023-09-10T01:04:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 -- 172.21.15.3:0/339057455 learned_addr learned my addr 172.21.15.3:0/339057455 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14aa59c700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100630 0x7f14ac19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14aad9d700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac106640 0x7f14ac198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 -- 172.21.15.3:0/339057455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac106640 msgr2=0x7f14ac198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac106640 0x7f14ac198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 -- 172.21.15.3:0/339057455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100630 msgr2=0x7f14ac19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100630 0x7f14ac19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 -- 172.21.15.3:0/339057455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4009d00 con 0x7f14ac100ff0 2023-09-10T01:04:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.223+0000 7f14a3fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac19ebf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f149400e9a0 tx=0x7f149400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.224+0000 7f14a37fe700 1 -- 172.21.15.3:0/339057455 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1494009790 con 0x7f14ac100ff0 2023-09-10T01:04:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.224+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac199400 con 0x7f14ac100ff0 2023-09-10T01:04:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.224+0000 7f14a37fe700 1 -- 172.21.15.3:0/339057455 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f149400cbd0 con 0x7f14ac100ff0 2023-09-10T01:04:41.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.224+0000 7f14a37fe700 1 -- 172.21.15.3:0/339057455 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1494018680 con 0x7f14ac100ff0 2023-09-10T01:04:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.224+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac1a56e0 con 0x7f14ac100ff0 2023-09-10T01:04:41.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.225+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac067480 con 0x7f14ac100ff0 2023-09-10T01:04:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.227+0000 7f14a37fe700 1 -- 172.21.15.3:0/339057455 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f149400cd40 con 0x7f14ac100ff0 2023-09-10T01:04:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.228+0000 7f14a37fe700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f14980a08d0 0x7f14980a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.228+0000 7f14a37fe700 1 -- 172.21.15.3:0/339057455 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1494014070 con 0x7f14ac100ff0 2023-09-10T01:04:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.228+0000 7f14aa59c700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f14980a08d0 0x7f14980a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.229+0000 7f14aa59c700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f14980a08d0 0x7f14980a2d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f149c009be0 tx=0x7f149c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.232+0000 7f14a37fe700 1 -- 172.21.15.3:0/339057455 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1494051e10 con 0x7f14ac100ff0 2023-09-10T01:04:41.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.489+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14ac110570 con 0x7f14ac100ff0 2023-09-10T01:04:41.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.490+0000 7f14a37fe700 1 -- 172.21.15.3:0/339057455 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11295 (secure 0 0 0) 0x7f1494095ad0 con 0x7f14ac100ff0 2023-09-10T01:04:41.491 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:04:41.491 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:04:38.815186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:04:41.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.492+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f14980a08d0 msgr2=0x7f14980a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.492+0000 7f14b0fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f14980a08d0 0x7f14980a2d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f149c009be0 tx=0x7f149c009260 comp rx=0 tx=0).stop 2023-09-10T01:04:41.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.492+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 msgr2=0x7f14ac19ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.492+0000 7f14b0fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac19ebf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f149400e9a0 tx=0x7f149400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 shutdown_connections 2023-09-10T01:04:41.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f14980a08d0 0x7f14980a2d90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac106640 0x7f14ac198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100630 0x7f14ac19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 --2- 172.21.15.3:0/339057455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac100ff0 0x7f14ac19ebf0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 >> 172.21.15.3:0/339057455 conn(0x7f14ac0fc140 msgr2=0x7f14ac10ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 shutdown_connections 2023-09-10T01:04:41.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.493+0000 7f14b0fff700 1 -- 172.21.15.3:0/339057455 wait complete. 2023-09-10T01:04:41.496 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2023-09-10T01:04:41.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:04:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 -- 172.21.15.3:0/2832829317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278106650 msgr2=0x7f1278106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 --2- 172.21.15.3:0/2832829317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278106650 0x7f1278106a30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1268009990 tx=0x7f1268009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 -- 172.21.15.3:0/2832829317 shutdown_connections 2023-09-10T01:04:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 --2- 172.21.15.3:0/2832829317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1278101000 0x7f127810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 --2- 172.21.15.3:0/2832829317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278106650 0x7f1278106a30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 --2- 172.21.15.3:0/2832829317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 0x7f1278100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 -- 172.21.15.3:0/2832829317 >> 172.21.15.3:0/2832829317 conn(0x7f12780fc170 msgr2=0x7f12780fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 -- 172.21.15.3:0/2832829317 shutdown_connections 2023-09-10T01:04:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.720+0000 7f127cbff700 1 -- 172.21.15.3:0/2832829317 wait complete. 2023-09-10T01:04:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 Processor -- start 2023-09-10T01:04:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 -- start start 2023-09-10T01:04:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 0x7f127819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278101000 0x7f127819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1278106650 0x7f1278198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278198d80 con 0x7f1278100640 2023-09-10T01:04:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278198ef0 con 0x7f1278101000 2023-09-10T01:04:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.721+0000 7f127cbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781990b0 con 0x7f1278106650 2023-09-10T01:04:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 0x7f127819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 0x7f127819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48978/0 (socket says 172.21.15.3:48978) 2023-09-10T01:04:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 -- 172.21.15.3:0/2709456469 learned_addr learned my addr 172.21.15.3:0/2709456469 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f1276d9d700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1278106650 0x7f1278198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f1275d9b700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278101000 0x7f127819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 -- 172.21.15.3:0/2709456469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1278106650 msgr2=0x7f1278198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1278106650 0x7f1278198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 -- 172.21.15.3:0/2709456469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278101000 msgr2=0x7f127819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278101000 0x7f127819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 -- 172.21.15.3:0/2709456469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12680095f0 con 0x7f1278100640 2023-09-10T01:04:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f1276d9d700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1278106650 0x7f1278198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f1275d9b700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278101000 0x7f127819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.722+0000 7f127659c700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 0x7f127819e6a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1268000c00 tx=0x7f1268005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.723+0000 7f12677fe700 1 -- 172.21.15.3:0/2709456469 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126801d070 con 0x7f1278100640 2023-09-10T01:04:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.723+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1278199330 con 0x7f1278100640 2023-09-10T01:04:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.723+0000 7f12677fe700 1 -- 172.21.15.3:0/2709456469 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1268022470 con 0x7f1278100640 2023-09-10T01:04:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.723+0000 7f12677fe700 1 -- 172.21.15.3:0/2709456469 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268017650 con 0x7f1278100640 2023-09-10T01:04:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.723+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12781a5690 con 0x7f1278100640 2023-09-10T01:04:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.726+0000 7f12677fe700 1 -- 172.21.15.3:0/2709456469 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f12680225e0 con 0x7f1278100640 2023-09-10T01:04:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.727+0000 7f12904a6b80 1 Processor -- start 2023-09-10T01:04:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.727+0000 7f12904a6b80 1 -- start start 2023-09-10T01:04:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12677fe700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12600a08d0 0x7f12600a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e17030 0x558e65e20900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e20e40 0x558e65e23230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12904a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65d6f730 con 0x558e65e16710 2023-09-10T01:04:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12904a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65d6d050 con 0x558e65e20e40 2023-09-10T01:04:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12904a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65d68940 con 0x558e65e17030 2023-09-10T01:04:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f12677fe700 1 -- 172.21.15.3:0/2709456469 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f12680cd920 con 0x7f1278100640 2023-09-10T01:04:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48988/0 (socket says 172.21.15.3:48988) 2023-09-10T01:04:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f127659c700 1 -- 172.21.15.3:0/2858728573 learned_addr learned my addr 172.21.15.3:0/2858728573 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f1276d9d700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12600a08d0 0x7f12600a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f1275d9b700 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e17030 0x558e65e20900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f1276d9d700 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e20e40 0x558e65e23230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f127659c700 1 -- 172.21.15.3:0/2858728573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e17030 msgr2=0x558e65e20900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f127659c700 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e17030 0x558e65e20900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f127659c700 1 -- 172.21.15.3:0/2858728573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e20e40 msgr2=0x558e65e23230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.728+0000 7f127659c700 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e20e40 0x558e65e23230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.729+0000 7f127659c700 1 -- 172.21.15.3:0/2858728573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e65b94de0 con 0x558e65e16710 2023-09-10T01:04:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.729+0000 7f127659c700 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f126809a940 tx=0x7f126809a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ac2da7ad24ab54 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.729+0000 7f1276d9d700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12600a08d0 0x7f12600a2d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1270009ec0 tx=0x7f1270009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.729+0000 7f1264ff9700 1 -- 172.21.15.3:0/2858728573 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680071c0 con 0x558e65e16710 2023-09-10T01:04:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.729+0000 7f1264ff9700 1 -- 172.21.15.3:0/2858728573 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f126809b0b0 con 0x558e65e16710 2023-09-10T01:04:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.729+0000 7f1264ff9700 1 -- 172.21.15.3:0/2858728573 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126809e7e0 con 0x558e65e16710 2023-09-10T01:04:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 -- 172.21.15.3:0/2858728573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 msgr2=0x558e65e16af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f126809a940 tx=0x7f126809a8b0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 -- 172.21.15.3:0/2858728573 shutdown_connections 2023-09-10T01:04:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e20e40 0x558e65e23230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2858728573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e17030 0x558e65e20900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 -- 172.21.15.3:0/2858728573 >> 172.21.15.3:0/2858728573 conn(0x558e65d8b090 msgr2=0x558e65e11bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 -- 172.21.15.3:0/2858728573 shutdown_connections 2023-09-10T01:04:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 -- 172.21.15.3:0/2858728573 wait complete. 2023-09-10T01:04:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.730+0000 7f12904a6b80 1 -- start start 2023-09-10T01:04:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.731+0000 7f12904a6b80 1 Processor -- start 2023-09-10T01:04:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e16710 0x558e65e21220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e17030 0x558e65e21760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e2c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f12904a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65d06b70 con 0x558e65e17030 2023-09-10T01:04:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f12904a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65d06870 con 0x558e65e16710 2023-09-10T01:04:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f12904a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65bea920 con 0x558e65e20e40 2023-09-10T01:04:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f12904a6b80 10 client.0 _ll_register_callbacks cb 0x558e65c50a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1275d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e17030 0x558e65e21760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f127659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e16710 0x558e65e21220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f127659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e16710 0x558e65e21220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34414/0 (socket says 172.21.15.3:34414) 2023-09-10T01:04:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f127659c700 1 -- 172.21.15.3:0/2438967486 learned_addr learned my addr 172.21.15.3:0/2438967486 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1276d9d700 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e2c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1275d9b700 1 -- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 msgr2=0x558e65e2c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1275d9b700 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e2c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1275d9b700 1 -- 172.21.15.3:0/2438967486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e16710 msgr2=0x558e65e21220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1275d9b700 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e16710 0x558e65e21220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1275d9b700 1 -- 172.21.15.3:0/2438967486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12781a5690 con 0x558e65e17030 2023-09-10T01:04:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.732+0000 7f1276d9d700 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e2c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f1275d9b700 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e17030 0x558e65e21760 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1278199b80 tx=0x7f126c009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76127f3218711bcf server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f125dffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f125dffb700 1 -- 172.21.15.3:0/2438967486 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c017070 con 0x558e65e17030 2023-09-10T01:04:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e65cc04f0 con 0x558e65e17030 2023-09-10T01:04:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f125dffb700 1 -- 172.21.15.3:0/2438967486 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f126c004020 con 0x558e65e17030 2023-09-10T01:04:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f125dffb700 1 -- 172.21.15.3:0/2438967486 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f126c0041e0 con 0x558e65e17030 2023-09-10T01:04:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f12904a6b80 10 client.16056 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.733+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558e65b7dc60 con 0x558e65e17030 2023-09-10T01:04:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f125dffb700 1 -- 172.21.15.3:0/2438967486 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558e65cc04f0 con 0x558e65e17030 2023-09-10T01:04:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f125dffb700 1 -- 172.21.15.3:0/2438967486 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f126c003840 con 0x558e65e17030 2023-09-10T01:04:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f12904a6b80 10 client.16056 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f12904a6b80 10 client.16056 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:04:41.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f12904a6b80 20 client.16056 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:41.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e306c0 0x558e65e32ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f12904a6b80 4 client.16056 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:41.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e65d7d8a0 con 0x558e65e306c0 2023-09-10T01:04:41.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.734+0000 7f127659c700 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e306c0 0x558e65e32ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.735+0000 7f127659c700 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e306c0 0x558e65e32ae0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7c9a82cf159eda6 server_cookie=c6a00ee9d1e92839 in_seq=0 out_seq=0 2023-09-10T01:04:41.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.735+0000 7f125dffb700 10 client.16056 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:41.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.735+0000 7f125dffb700 1 -- 172.21.15.3:0/2438967486 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558e65d7d8a0 con 0x558e65e306c0 2023-09-10T01:04:41.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.735+0000 7f125dffb700 10 client.16056 handle_command_reply: tid=0 2023-09-10T01:04:41.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 client.16056 shutdown 2023-09-10T01:04:41.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e17030 msgr2=0x558e65e21760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e17030 0x558e65e21760 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1278199b80 tx=0x7f126c009d40 comp rx=0 tx=0).stop 2023-09-10T01:04:41.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 shutdown_connections 2023-09-10T01:04:41.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e306c0 0x558e65e32ae0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e2c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e16710 0x558e65e21220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 --2- 172.21.15.3:0/2438967486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e17030 0x558e65e21760 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f125dffb700 0 client.16056 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:41.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 >> 172.21.15.3:0/2438967486 conn(0x558e65d8b090 msgr2=0x558e65e117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 shutdown_connections 2023-09-10T01:04:41.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 1 -- 172.21.15.3:0/2438967486 wait complete. 2023-09-10T01:04:41.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.736+0000 7f12904a6b80 20 client.16056 trim_cache size 0 max 16384 2023-09-10T01:04:41.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53479 13.37 MB/sec warmup 23 sec latency 1393.421 ms 2023-09-10T01:04:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 Processor -- start 2023-09-10T01:04:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 -- start start 2023-09-10T01:04:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e32ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e1f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65e1e460 con 0x558e65e16710 2023-09-10T01:04:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65e1fcc0 con 0x558e65e17030 2023-09-10T01:04:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f12904a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65e207e0 con 0x558e65e20e40 2023-09-10T01:04:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48998/0 (socket says 172.21.15.3:48998) 2023-09-10T01:04:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f127659c700 1 -- 172.21.15.3:0/4278979365 learned_addr learned my addr 172.21.15.3:0/4278979365 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.750+0000 7f1276d9d700 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e1f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f1275d9b700 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e32ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f127659c700 1 -- 172.21.15.3:0/4278979365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 msgr2=0x558e65e1f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f127659c700 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e1f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f127659c700 1 -- 172.21.15.3:0/4278979365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 msgr2=0x558e65e32ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f127659c700 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e32ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f127659c700 1 -- 172.21.15.3:0/4278979365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e65bbb250 con 0x558e65e16710 2023-09-10T01:04:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f1275d9b700 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e32ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f127659c700 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f12680252e0 tx=0x7f126801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e750ad912627e35 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f12657fa700 1 -- 172.21.15.3:0/4278979365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12680a2990 con 0x558e65e16710 2023-09-10T01:04:41.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f12657fa700 1 -- 172.21.15.3:0/4278979365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f126809e7e0 con 0x558e65e16710 2023-09-10T01:04:41.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.751+0000 7f12657fa700 1 -- 172.21.15.3:0/4278979365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126809ae70 con 0x558e65e16710 2023-09-10T01:04:41.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 -- 172.21.15.3:0/4278979365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 msgr2=0x558e65e16af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f12680252e0 tx=0x7f126801a040 comp rx=0 tx=0).stop 2023-09-10T01:04:41.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 -- 172.21.15.3:0/4278979365 shutdown_connections 2023-09-10T01:04:41.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e1f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e16af0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 --2- 172.21.15.3:0/4278979365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e32ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 -- 172.21.15.3:0/4278979365 >> 172.21.15.3:0/4278979365 conn(0x558e65d8b090 msgr2=0x558e65e37400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 -- 172.21.15.3:0/4278979365 shutdown_connections 2023-09-10T01:04:41.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 -- 172.21.15.3:0/4278979365 wait complete. 2023-09-10T01:04:41.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.752+0000 7f12904a6b80 1 -- start start 2023-09-10T01:04:41.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 1 Processor -- start 2023-09-10T01:04:41.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e1d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e1d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65e1dfd0 con 0x558e65e16710 2023-09-10T01:04:41.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65e18d00 con 0x558e65e17030 2023-09-10T01:04:41.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e65e18ec0 con 0x558e65e20e40 2023-09-10T01:04:41.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f12904a6b80 10 client.0 _ll_register_callbacks cb 0x558e65c50a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:41.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e1d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f1276d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.753+0000 7f127659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e1d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49012/0 (socket says 172.21.15.3:49012) 2023-09-10T01:04:41.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f1276d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e187c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50426/0 (socket says 172.21.15.3:50426) 2023-09-10T01:04:41.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f127659c700 1 -- 172.21.15.3:0/1528753046 learned_addr learned my addr 172.21.15.3:0/1528753046 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:41.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f1275d9b700 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e1d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f127659c700 1 -- 172.21.15.3:0/1528753046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 msgr2=0x558e65e187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f127659c700 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e187c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f127659c700 1 -- 172.21.15.3:0/1528753046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 msgr2=0x558e65e1d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:41.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f127659c700 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e1d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f127659c700 1 -- 172.21.15.3:0/1528753046 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e65b94de0 con 0x558e65e16710 2023-09-10T01:04:41.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f127659c700 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e1d380 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f12680a2460 tx=0x7f126809a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f11179c9a28d87 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f125dffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:41.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f125dffb700 1 -- 172.21.15.3:0/1528753046 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268022ae0 con 0x558e65e16710 2023-09-10T01:04:41.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f125dffb700 1 -- 172.21.15.3:0/1528753046 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f12680a2990 con 0x558e65e16710 2023-09-10T01:04:41.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.754+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f126809b0b0 con 0x558e65e16710 2023-09-10T01:04:41.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f125dffb700 1 -- 172.21.15.3:0/1528753046 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1268099ea0 con 0x558e65e16710 2023-09-10T01:04:41.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 10 client.16062 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:41.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558e65b932b0 con 0x558e65e16710 2023-09-10T01:04:41.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f125dffb700 1 -- 172.21.15.3:0/1528753046 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f126809b0b0 con 0x558e65e16710 2023-09-10T01:04:41.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f125dffb700 1 -- 172.21.15.3:0/1528753046 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f126809ac10 con 0x558e65e16710 2023-09-10T01:04:41.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 10 client.16062 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:41.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 10 client.16062 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:04:41.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 20 client.16062 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:41.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e1a0b0 0x558e65e874d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 4 client.16062 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:04:41.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558e65d61920 con 0x558e65e1a0b0 2023-09-10T01:04:41.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.755+0000 7f1275d9b700 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e1a0b0 0x558e65e874d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f1275d9b700 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e1a0b0 0x558e65e874d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89f14259d4b35368 server_cookie=276cc0938bfbd0e0 in_seq=0 out_seq=0 2023-09-10T01:04:41.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f125dffb700 10 client.16062 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:41.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f125dffb700 1 -- 172.21.15.3:0/1528753046 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558e65d61920 con 0x558e65e1a0b0 2023-09-10T01:04:41.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f125dffb700 10 client.16062 handle_command_reply: tid=0 2023-09-10T01:04:41.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 client.16062 shutdown 2023-09-10T01:04:41.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 msgr2=0x558e65e1d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e1d380 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f12680a2460 tx=0x7f126809a8b0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 shutdown_connections 2023-09-10T01:04:41.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x558e65e1a0b0 0x558e65e874d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558e65e20e40 0x558e65e187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558e65e16710 0x558e65e1d380 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 --2- 172.21.15.3:0/1528753046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558e65e17030 0x558e65e1d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f125dffb700 0 client.16062 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:04:41.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 >> 172.21.15.3:0/1528753046 conn(0x558e65d8b090 msgr2=0x558e65e1f730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.756+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 shutdown_connections 2023-09-10T01:04:41.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f12904a6b80 1 -- 172.21.15.3:0/1528753046 wait complete. 2023-09-10T01:04:41.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f12904a6b80 20 client.16062 trim_cache size 0 max 16384 2023-09-10T01:04:41.777 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:04:41.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12600a08d0 msgr2=0x7f12600a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12600a08d0 0x7f12600a2d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1270009ec0 tx=0x7f1270009920 comp rx=0 tx=0).stop 2023-09-10T01:04:41.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 msgr2=0x7f127819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 0x7f127819e6a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1268000c00 tx=0x7f1268005740 comp rx=0 tx=0).stop 2023-09-10T01:04:41.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 shutdown_connections 2023-09-10T01:04:41.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12600a08d0 0x7f12600a2d90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1278106650 0x7f1278198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1278100640 0x7f127819e6a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 --2- 172.21.15.3:0/2709456469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1278101000 0x7f127819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 >> 172.21.15.3:0/2709456469 conn(0x7f12780fc170 msgr2=0x7f127810b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 shutdown_connections 2023-09-10T01:04:41.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.757+0000 7f127cbff700 1 -- 172.21.15.3:0/2709456469 wait complete. 2023-09-10T01:04:41.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:04:41.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.986+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3598063889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4072a30 msgr2=0x7fe5f4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.986+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3598063889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4072a30 0x7fe5f4072e10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009990 tx=0x7fe5f0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.987+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3598063889 shutdown_connections 2023-09-10T01:04:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.987+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3598063889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5f410ce80 0x7fe5f410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.987+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3598063889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4072a30 0x7fe5f4072e10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.987+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3598063889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4073350 0x7fe5f410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.987+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3598063889 >> 172.21.15.3:0/3598063889 conn(0x7fe5f40fc170 msgr2=0x7fe5f40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.987+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3598063889 shutdown_connections 2023-09-10T01:04:41.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.987+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3598063889 wait complete. 2023-09-10T01:04:41.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 Processor -- start 2023-09-10T01:04:41.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 -- start start 2023-09-10T01:04:41.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4072a30 0x7fe5f419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4073350 0x7fe5f419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5f410ce80 0x7fe5f4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:41.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f4198de0 con 0x7fe5f4072a30 2023-09-10T01:04:41.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f4198f50 con 0x7fe5f4073350 2023-09-10T01:04:41.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5fbfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f4199110 con 0x7fe5f410ce80 2023-09-10T01:04:41.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.988+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4072a30 0x7fe5f419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4072a30 0x7fe5f419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49028/0 (socket says 172.21.15.3:49028) 2023-09-10T01:04:41.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3468075772 learned_addr learned my addr 172.21.15.3:0/3468075772 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:41.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5fa59c700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5f410ce80 0x7fe5f4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4073350 0x7fe5f419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3468075772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5f410ce80 msgr2=0x7fe5f4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5f410ce80 0x7fe5f4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3468075772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4073350 msgr2=0x7fe5f419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4073350 0x7fe5f419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:41.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3468075772 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec009c40 con 0x7fe5f4072a30 2023-09-10T01:04:41.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4072a30 0x7fe5f419e700 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0000c00 tx=0x7fe5f0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:41.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.989+0000 7fe5eaffd700 1 -- 172.21.15.3:0/3468075772 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f001d070 con 0x7fe5f4072a30 2023-09-10T01:04:41.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.990+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f00095f0 con 0x7fe5f4072a30 2023-09-10T01:04:41.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.990+0000 7fe5eaffd700 1 -- 172.21.15.3:0/3468075772 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5f0022470 con 0x7fe5f4072a30 2023-09-10T01:04:41.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.990+0000 7fe5eaffd700 1 -- 172.21.15.3:0/3468075772 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0017650 con 0x7fe5f4072a30 2023-09-10T01:04:41.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.990+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f41a5510 con 0x7fe5f4072a30 2023-09-10T01:04:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.993+0000 7fe5eaffd700 1 -- 172.21.15.3:0/3468075772 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7fe5f00225e0 con 0x7fe5f4072a30 2023-09-10T01:04:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.994+0000 7fe60f90bb80 1 Processor -- start 2023-09-10T01:04:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.994+0000 7fe60f90bb80 1 -- start start 2023-09-10T01:04:42.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.994+0000 7fe5eaffd700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5e00a08d0 0x7fe5e00a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.994+0000 7fe5eaffd700 1 -- 172.21.15.3:0/3468075772 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe5f00cd9b0 con 0x7fe5f4072a30 2023-09-10T01:04:42.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.994+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be304af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.994+0000 7fe5fa59c700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5e00a08d0 0x7fe5e00a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.994+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be30e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be311230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be25d730 con 0x55e5be30ee40 2023-09-10T01:04:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be25b050 con 0x55e5be304710 2023-09-10T01:04:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be256940 con 0x55e5be305030 2023-09-10T01:04:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be304af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be304af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34458/0 (socket says 172.21.15.3:34458) 2023-09-10T01:04:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3864133574 learned_addr learned my addr 172.21.15.3:0/3864133574 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be30e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5fa59c700 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be311230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3864133574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 msgr2=0x55e5be30e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be30e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3864133574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 msgr2=0x55e5be311230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be311230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.995+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3864133574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5be082de0 con 0x55e5be304710 2023-09-10T01:04:42.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.996+0000 7fe5fa59c700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5e00a08d0 0x7fe5e00a2d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe5f419f400 tx=0x7fe5ec00fdd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.996+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be304af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe5f009a9c0 tx=0x7fe5f0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f155427863c441b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.996+0000 7fe5dffff700 1 -- 172.21.15.3:0/3864133574 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f009aa30 con 0x55e5be304710 2023-09-10T01:04:42.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.996+0000 7fe5dffff700 1 -- 172.21.15.3:0/3864133574 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5f009e860 con 0x55e5be304710 2023-09-10T01:04:42.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.996+0000 7fe5dffff700 1 -- 172.21.15.3:0/3864133574 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f009a180 con 0x55e5be304710 2023-09-10T01:04:42.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.996+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3864133574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 msgr2=0x55e5be304af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.996+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be304af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe5f009a9c0 tx=0x7fe5f0009310 comp rx=0 tx=0).stop 2023-09-10T01:04:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3864133574 shutdown_connections 2023-09-10T01:04:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be311230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be304af0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3864133574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be30e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3864133574 >> 172.21.15.3:0/3864133574 conn(0x55e5be279090 msgr2=0x55e5be2ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3864133574 shutdown_connections 2023-09-10T01:04:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3864133574 wait complete. 2023-09-10T01:04:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 -- start start 2023-09-10T01:04:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.997+0000 7fe60f90bb80 1 Processor -- start 2023-09-10T01:04:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be304710 0x55e5be30f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be305030 0x55e5be30f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be31a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be1f4b70 con 0x55e5be304710 2023-09-10T01:04:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be1f4870 con 0x55e5be305030 2023-09-10T01:04:42.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be0d8920 con 0x55e5be30ee40 2023-09-10T01:04:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe60f90bb80 10 client.0 _ll_register_callbacks cb 0x55e5be13ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be304710 0x55e5be30f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be304710 0x55e5be30f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49044/0 (socket says 172.21.15.3:49044) 2023-09-10T01:04:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3978933811 learned_addr learned my addr 172.21.15.3:0/3978933811 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe5fa59c700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be31a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.998+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be305030 0x55e5be30f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3978933811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 msgr2=0x55e5be31a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be31a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be305030 msgr2=0x55e5be30f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be305030 0x55e5be30f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3978933811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f41a5510 con 0x55e5be304710 2023-09-10T01:04:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be305030 0x55e5be30f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be304710 0x55e5be30f220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009580 tx=0x7fe5f009af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba422107ae3bc0e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3978933811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00a2a10 con 0x55e5be304710 2023-09-10T01:04:42.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e5be1ae4f0 con 0x55e5be304710 2023-09-10T01:04:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3978933811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe5f009aa30 con 0x55e5be304710 2023-09-10T01:04:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3978933811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5f00a2790 con 0x55e5be304710 2023-09-10T01:04:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe60f90bb80 10 client.16074 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:41.999+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e5be06bc60 con 0x55e5be304710 2023-09-10T01:04:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3978933811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e5be1ae4f0 con 0x55e5be304710 2023-09-10T01:04:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3978933811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe5f0099ab0 con 0x55e5be304710 2023-09-10T01:04:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe60f90bb80 10 client.16074 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe60f90bb80 10 client.16074 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:04:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe60f90bb80 20 client.16074 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be31e060 0x55e5be320450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe60f90bb80 4 client.16074 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5be26b8a0 con 0x55e5be31e060 2023-09-10T01:04:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.000+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be31e060 0x55e5be320450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.001+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be31e060 0x55e5be320450 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a481d5bd7dea7736 server_cookie=79fd36c2d63962be in_seq=0 out_seq=0 2023-09-10T01:04:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.001+0000 7fe5dd7fa700 10 client.16074 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.001+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3978933811 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e5be26b8a0 con 0x55e5be31e060 2023-09-10T01:04:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.001+0000 7fe5dd7fa700 10 client.16074 handle_command_reply: tid=0 2023-09-10T01:04:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 client.16074 shutdown 2023-09-10T01:04:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be304710 msgr2=0x55e5be30f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be304710 0x55e5be30f220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009580 tx=0x7fe5f009af20 comp rx=0 tx=0).stop 2023-09-10T01:04:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 shutdown_connections 2023-09-10T01:04:42.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be31e060 0x55e5be320450 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be31a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be304710 0x55e5be30f220 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe5dd7fa700 0 client.16074 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3978933811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be305030 0x55e5be30f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 >> 172.21.15.3:0/3978933811 conn(0x55e5be279090 msgr2=0x55e5be2ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 shutdown_connections 2023-09-10T01:04:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3978933811 wait complete. 2023-09-10T01:04:42.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.002+0000 7fe60f90bb80 20 client.16074 trim_cache size 0 max 16384 2023-09-10T01:04:42.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 Processor -- start 2023-09-10T01:04:42.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 -- start start 2023-09-10T01:04:42.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be308890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be305030 0x55e5be30d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be30f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be320a80 con 0x55e5be305030 2023-09-10T01:04:42.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be325120 con 0x55e5be304710 2023-09-10T01:04:42.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be3252e0 con 0x55e5be30ee40 2023-09-10T01:04:42.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe5f959a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be305030 0x55e5be30d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe5fa59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be30f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be308890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe5f959a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be305030 0x55e5be30d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49060/0 (socket says 172.21.15.3:49060) 2023-09-10T01:04:42.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe5f959a700 1 -- 172.21.15.3:0/4169785965 learned_addr learned my addr 172.21.15.3:0/4169785965 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.016+0000 7fe5fa59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be30f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50472/0 (socket says 172.21.15.3:50472) 2023-09-10T01:04:42.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5f959a700 1 -- 172.21.15.3:0/4169785965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 msgr2=0x55e5be30f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5f959a700 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be30f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5f959a700 1 -- 172.21.15.3:0/4169785965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 msgr2=0x55e5be308890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5f959a700 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be308890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5f959a700 1 -- 172.21.15.3:0/4169785965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5be0a9250 con 0x55e5be305030 2023-09-10T01:04:42.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be308890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:42.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5f959a700 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be305030 0x55e5be30d780 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40097c0 tx=0x7fe5e4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2603fc318734e51a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5e8ff9700 1 -- 172.21.15.3:0/4169785965 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4024070 con 0x55e5be305030 2023-09-10T01:04:42.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5e8ff9700 1 -- 172.21.15.3:0/4169785965 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5e4018040 con 0x55e5be305030 2023-09-10T01:04:42.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.017+0000 7fe5e8ff9700 1 -- 172.21.15.3:0/4169785965 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4004a90 con 0x55e5be305030 2023-09-10T01:04:42.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 -- 172.21.15.3:0/4169785965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be305030 msgr2=0x55e5be30d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be305030 0x55e5be30d780 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40097c0 tx=0x7fe5e4009260 comp rx=0 tx=0).stop 2023-09-10T01:04:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 -- 172.21.15.3:0/4169785965 shutdown_connections 2023-09-10T01:04:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be30ee40 0x55e5be30f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be308890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/4169785965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be305030 0x55e5be30d780 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 -- 172.21.15.3:0/4169785965 >> 172.21.15.3:0/4169785965 conn(0x55e5be279090 msgr2=0x55e5be31a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 -- 172.21.15.3:0/4169785965 shutdown_connections 2023-09-10T01:04:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 -- 172.21.15.3:0/4169785965 wait complete. 2023-09-10T01:04:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.018+0000 7fe60f90bb80 1 -- start start 2023-09-10T01:04:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 1 Processor -- start 2023-09-10T01:04:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be36a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be36a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be310150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be310750 con 0x55e5be30ee40 2023-09-10T01:04:42.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be3108c0 con 0x55e5be304710 2023-09-10T01:04:42.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5be310a80 con 0x55e5be305030 2023-09-10T01:04:42.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe60f90bb80 10 client.0 _ll_register_callbacks cb 0x7fe5e4009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be36a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe5f9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be36a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34494/0 (socket says 172.21.15.3:34494) 2023-09-10T01:04:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe5fa59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be310150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.019+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3381823814 learned_addr learned my addr 172.21.15.3:0/3381823814 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be36a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3381823814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 msgr2=0x55e5be36a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be36a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3381823814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 msgr2=0x55e5be310150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be310150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5f9d9b700 1 -- 172.21.15.3:0/3381823814 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5be06bc60 con 0x55e5be304710 2023-09-10T01:04:42.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5f9d9b700 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be36a100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe5f009a730 tx=0x7fe5f000b280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d459bdcd4e1e708 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:42.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3381823814 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0096d70 con 0x55e5be304710 2023-09-10T01:04:42.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5f009e860 con 0x55e5be304710 2023-09-10T01:04:42.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.020+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3381823814 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe5f009aa30 con 0x55e5be304710 2023-09-10T01:04:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3381823814 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5f000fb70 con 0x55e5be304710 2023-09-10T01:04:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 10 client.26132 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fe5e4009ae0 con 0x55e5be304710 2023-09-10T01:04:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3381823814 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5f009e860 con 0x55e5be304710 2023-09-10T01:04:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3381823814 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe5f0096820 con 0x55e5be304710 2023-09-10T01:04:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 10 client.26132 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:42.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 10 client.26132 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:04:42.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 20 client.26132 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:42.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be372c10 0x55e5be375000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 4 client.26132 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:04:42.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e5be24f920 con 0x55e5be372c10 2023-09-10T01:04:42.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.021+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be372c10 0x55e5be375000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe5f959a700 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be372c10 0x55e5be375000 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf61f0c2908684ef server_cookie=dbcb9536b637b8d8 in_seq=0 out_seq=0 2023-09-10T01:04:42.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe5dd7fa700 10 client.26132 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:42.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe5dd7fa700 1 -- 172.21.15.3:0/3381823814 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e5be24f920 con 0x55e5be372c10 2023-09-10T01:04:42.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe5dd7fa700 10 client.26132 handle_command_reply: tid=0 2023-09-10T01:04:42.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 client.26132 shutdown 2023-09-10T01:04:42.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 msgr2=0x55e5be36a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be36a100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe5f009a730 tx=0x7fe5f000b280 comp rx=0 tx=0).stop 2023-09-10T01:04:42.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 shutdown_connections 2023-09-10T01:04:42.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5be30ee40 0x55e5be310150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5be304710 0x55e5be36a100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55e5be372c10 0x55e5be375000 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 --2- 172.21.15.3:0/3381823814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e5be305030 0x55e5be36a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe5dd7fa700 0 client.26132 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:04:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 >> 172.21.15.3:0/3381823814 conn(0x55e5be279090 msgr2=0x55e5be30f220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.022+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 shutdown_connections 2023-09-10T01:04:42.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe60f90bb80 1 -- 172.21.15.3:0/3381823814 wait complete. 2023-09-10T01:04:42.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe60f90bb80 20 client.26132 trim_cache size 0 max 16384 2023-09-10T01:04:42.040 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:04:42.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5e00a08d0 msgr2=0x7fe5e00a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5e00a08d0 0x7fe5e00a2d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe5f419f400 tx=0x7fe5ec00fdd0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4072a30 msgr2=0x7fe5f419e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4072a30 0x7fe5f419e700 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0000c00 tx=0x7fe5f0005740 comp rx=0 tx=0).stop 2023-09-10T01:04:42.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 shutdown_connections 2023-09-10T01:04:42.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5e00a08d0 0x7fe5e00a2d90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5f410ce80 0x7fe5f4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5f4072a30 0x7fe5f419e700 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 --2- 172.21.15.3:0/3468075772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5f4073350 0x7fe5f419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 >> 172.21.15.3:0/3468075772 conn(0x7fe5f40fc170 msgr2=0x7fe5f410efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 shutdown_connections 2023-09-10T01:04:42.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.023+0000 7fe5fbfff700 1 -- 172.21.15.3:0/3468075772 wait complete. 2023-09-10T01:04:42.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:04:42.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 -- 172.21.15.3:0/3791288030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c0729d0 msgr2=0x7f889c072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 --2- 172.21.15.3:0/3791288030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c0729d0 0x7f889c072db0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f888c009990 tx=0x7f888c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 -- 172.21.15.3:0/3791288030 shutdown_connections 2023-09-10T01:04:42.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 --2- 172.21.15.3:0/3791288030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c10cef0 0x7f889c10f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 --2- 172.21.15.3:0/3791288030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c0729d0 0x7f889c072db0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 --2- 172.21.15.3:0/3791288030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c073380 0x7f889c10c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 -- 172.21.15.3:0/3791288030 >> 172.21.15.3:0/3791288030 conn(0x7f889c0fc160 msgr2=0x7f889c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 -- 172.21.15.3:0/3791288030 shutdown_connections 2023-09-10T01:04:42.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.251+0000 7f88a27ff700 1 -- 172.21.15.3:0/3791288030 wait complete. 2023-09-10T01:04:42.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.252+0000 7f88a27ff700 1 Processor -- start 2023-09-10T01:04:42.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a27ff700 1 -- start start 2023-09-10T01:04:42.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a27ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c0729d0 0x7f889c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a27ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c073380 0x7f889c198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a27ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c10cef0 0x7f889c19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a27ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c19d940 con 0x7f889c073380 2023-09-10T01:04:42.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a27ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c19dab0 con 0x7f889c10cef0 2023-09-10T01:04:42.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a27ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c19dc70 con 0x7f889c0729d0 2023-09-10T01:04:42.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f889bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c0729d0 0x7f889c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f889bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c0729d0 0x7f889c1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50500/0 (socket says 172.21.15.3:50500) 2023-09-10T01:04:42.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f889bfff700 1 -- 172.21.15.3:0/2379933662 learned_addr learned my addr 172.21.15.3:0/2379933662 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f889b7fe700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c073380 0x7f889c198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.253+0000 7f88a0d9c700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c10cef0 0x7f889c19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f889bfff700 1 -- 172.21.15.3:0/2379933662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c10cef0 msgr2=0x7f889c19d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f889bfff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c10cef0 0x7f889c19d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f889bfff700 1 -- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c073380 msgr2=0x7f889c198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f889bfff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c073380 0x7f889c198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f889bfff700 1 -- 172.21.15.3:0/2379933662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0095f0 con 0x7f889c0729d0 2023-09-10T01:04:42.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f889bfff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c0729d0 0x7f889c1989a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f888c005910 tx=0x7f888c005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f88997fa700 1 -- 172.21.15.3:0/2379933662 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c01d070 con 0x7f889c0729d0 2023-09-10T01:04:42.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c19def0 con 0x7f889c0729d0 2023-09-10T01:04:42.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f88997fa700 1 -- 172.21.15.3:0/2379933662 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f888c022470 con 0x7f889c0729d0 2023-09-10T01:04:42.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.254+0000 7f88997fa700 1 -- 172.21.15.3:0/2379933662 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c017650 con 0x7f889c0729d0 2023-09-10T01:04:42.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.255+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c1a5800 con 0x7f889c0729d0 2023-09-10T01:04:42.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.258+0000 7f88997fa700 1 -- 172.21.15.3:0/2379933662 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f888c00fb40 con 0x7f889c0729d0 2023-09-10T01:04:42.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.259+0000 7f88b61d1b80 1 Processor -- start 2023-09-10T01:04:42.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.259+0000 7f88b61d1b80 1 -- start start 2023-09-10T01:04:42.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.259+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558ce710 0x55f2558ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.259+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.259+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558d8e40 0x55f2558db230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f255827730 con 0x55f2558d8e40 2023-09-10T01:04:42.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f255825050 con 0x55f2558cf030 2023-09-10T01:04:42.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f255820940 con 0x55f2558ce710 2023-09-10T01:04:42.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f88997fa700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f88840a08d0 0x7f88840a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f88997fa700 1 -- 172.21.15.3:0/2379933662 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f888c0cda30 con 0x7f889c0729d0 2023-09-10T01:04:42.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558d8e40 0x55f2558db230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34522/0 (socket says 172.21.15.3:34522) 2023-09-10T01:04:42.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 -- 172.21.15.3:0/2002444185 learned_addr learned my addr 172.21.15.3:0/2002444185 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f88a0d9c700 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558ce710 0x55f2558ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f88a0d9c700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f88840a08d0 0x7f88840a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 -- 172.21.15.3:0/2002444185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558ce710 msgr2=0x55f2558ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558ce710 0x55f2558ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 -- 172.21.15.3:0/2002444185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558d8e40 msgr2=0x55f2558db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558d8e40 0x55f2558db230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.260+0000 7f889bfff700 1 -- 172.21.15.3:0/2002444185 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f25564cde0 con 0x55f2558cf030 2023-09-10T01:04:42.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.261+0000 7f889b7fe700 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558d8e40 0x55f2558db230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:42.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.261+0000 7f889bfff700 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d8900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f888c006010 tx=0x7f888c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c058a5e6f6ac666 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.261+0000 7f88827fc700 1 -- 172.21.15.3:0/2002444185 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c09a590 con 0x55f2558cf030 2023-09-10T01:04:42.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.261+0000 7f88827fc700 1 -- 172.21.15.3:0/2002444185 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f888c09a3b0 con 0x55f2558cf030 2023-09-10T01:04:42.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.261+0000 7f88827fc700 1 -- 172.21.15.3:0/2002444185 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c099f40 con 0x55f2558cf030 2023-09-10T01:04:42.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 -- 172.21.15.3:0/2002444185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 msgr2=0x55f2558d8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d8900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f888c006010 tx=0x7f888c009310 comp rx=0 tx=0).stop 2023-09-10T01:04:42.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88a0d9c700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f88840a08d0 0x7f88840a2d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8894009ec0 tx=0x7f889400b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 -- 172.21.15.3:0/2002444185 shutdown_connections 2023-09-10T01:04:42.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558ce710 0x55f2558ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558d8e40 0x55f2558db230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/2002444185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d8900 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 -- 172.21.15.3:0/2002444185 >> 172.21.15.3:0/2002444185 conn(0x55f255843090 msgr2=0x55f2558c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 -- 172.21.15.3:0/2002444185 shutdown_connections 2023-09-10T01:04:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.262+0000 7f88b61d1b80 1 -- 172.21.15.3:0/2002444185 wait complete. 2023-09-10T01:04:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 -- start start 2023-09-10T01:04:42.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 Processor -- start 2023-09-10T01:04:42.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558d9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2557beb70 con 0x55f2558ce710 2023-09-10T01:04:42.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2557be870 con 0x55f2558cf030 2023-09-10T01:04:42.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2556a2920 con 0x55f2558d8e40 2023-09-10T01:04:42.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.263+0000 7f88b61d1b80 10 client.0 _ll_register_callbacks cb 0x55f255708a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:42.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558d9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558d9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49106/0 (socket says 172.21.15.3:49106) 2023-09-10T01:04:42.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 -- 172.21.15.3:0/682212094 learned_addr learned my addr 172.21.15.3:0/682212094 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889b7fe700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f88a0d9c700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 -- 172.21.15.3:0/682212094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 msgr2=0x55f2558e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558e42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 -- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 msgr2=0x55f2558d9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889bfff700 1 -- 172.21.15.3:0/682212094 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c1a5800 con 0x55f2558ce710 2023-09-10T01:04:42.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.264+0000 7f889b7fe700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:42.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f889bfff700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558d9220 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f888c006010 tx=0x7f888c0cac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a121c42adbc7f0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f887b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:42.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f887b7fe700 1 -- 172.21.15.3:0/682212094 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c09a140 con 0x55f2558ce710 2023-09-10T01:04:42.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f887b7fe700 1 -- 172.21.15.3:0/682212094 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f888c09e880 con 0x55f2558ce710 2023-09-10T01:04:42.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2557784f0 con 0x55f2558ce710 2023-09-10T01:04:42.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f887b7fe700 1 -- 172.21.15.3:0/682212094 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f888c09b150 con 0x55f2558ce710 2023-09-10T01:04:42.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f88b61d1b80 10 client.16092 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:42.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f255635c60 con 0x55f2558ce710 2023-09-10T01:04:42.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f887b7fe700 1 -- 172.21.15.3:0/682212094 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f2557784f0 con 0x55f2558ce710 2023-09-10T01:04:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.265+0000 7f887b7fe700 1 -- 172.21.15.3:0/682212094 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f888c064050 con 0x55f2558ce710 2023-09-10T01:04:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f88b61d1b80 10 client.16092 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f88b61d1b80 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f88b61d1b80 20 client.16092 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2558e8130 0x55f2558ea520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f88b61d1b80 4 client.16092 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2558358a0 con 0x55f2558e8130 2023-09-10T01:04:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f889b7fe700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2558e8130 0x55f2558ea520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f889b7fe700 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2558e8130 0x55f2558ea520 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0ac789cb396ab55 server_cookie=6d047fe9d0c36fb6 in_seq=0 out_seq=0 2023-09-10T01:04:42.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.266+0000 7f887b7fe700 10 client.16092 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:42.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f887b7fe700 1 -- 172.21.15.3:0/682212094 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f2558358a0 con 0x55f2558e8130 2023-09-10T01:04:42.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f887b7fe700 10 client.16092 handle_command_reply: tid=0 2023-09-10T01:04:42.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 client.16092 shutdown 2023-09-10T01:04:42.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 msgr2=0x55f2558d9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558d9220 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f888c006010 tx=0x7f888c0cac20 comp rx=0 tx=0).stop 2023-09-10T01:04:42.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 shutdown_connections 2023-09-10T01:04:42.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2558e8130 0x55f2558ea520 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558e42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558d9220 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/682212094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558d9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.267+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 >> 172.21.15.3:0/682212094 conn(0x55f255843090 msgr2=0x55f2558c97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.268+0000 7f887b7fe700 0 client.16092 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:42.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.268+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 shutdown_connections 2023-09-10T01:04:42.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.268+0000 7f88b61d1b80 1 -- 172.21.15.3:0/682212094 wait complete. 2023-09-10T01:04:42.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.268+0000 7f88b61d1b80 20 client.16092 trim_cache size 0 max 16384 2023-09-10T01:04:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.282+0000 7f88b61d1b80 1 Processor -- start 2023-09-10T01:04:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.282+0000 7f88b61d1b80 1 -- start start 2023-09-10T01:04:42.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.282+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558de230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.282+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558ea140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558d9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2558d8590 con 0x55f2558ce710 2023-09-10T01:04:42.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2558de770 con 0x55f2558cf030 2023-09-10T01:04:42.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2558d7e30 con 0x55f2558d8e40 2023-09-10T01:04:42.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558ea140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34554/0 (socket says 172.21.15.3:34554) 2023-09-10T01:04:42.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 -- 172.21.15.3:0/1628089955 learned_addr learned my addr 172.21.15.3:0/1628089955 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889bfff700 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558de230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f88a0d9c700 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558d9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 -- 172.21.15.3:0/1628089955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 msgr2=0x55f2558d9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558d9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 -- 172.21.15.3:0/1628089955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 msgr2=0x55f2558de230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558de230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.283+0000 7f889b7fe700 1 -- 172.21.15.3:0/1628089955 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f255673250 con 0x55f2558cf030 2023-09-10T01:04:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.284+0000 7f889b7fe700 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558ea140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f889c199fb0 tx=0x7f8890009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8cd49dcd826f02b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.284+0000 7f8882ffd700 1 -- 172.21.15.3:0/1628089955 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890016070 con 0x55f2558cf030 2023-09-10T01:04:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.284+0000 7f8882ffd700 1 -- 172.21.15.3:0/1628089955 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8890017040 con 0x55f2558cf030 2023-09-10T01:04:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.284+0000 7f8882ffd700 1 -- 172.21.15.3:0/1628089955 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890010690 con 0x55f2558cf030 2023-09-10T01:04:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 -- 172.21.15.3:0/1628089955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 msgr2=0x55f2558ea140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558ea140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f889c199fb0 tx=0x7f8890009260 comp rx=0 tx=0).stop 2023-09-10T01:04:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 -- 172.21.15.3:0/1628089955 shutdown_connections 2023-09-10T01:04:42.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558d9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f2558de230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/1628089955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2558ea140 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 -- 172.21.15.3:0/1628089955 >> 172.21.15.3:0/1628089955 conn(0x55f255843090 msgr2=0x55f2558d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 -- 172.21.15.3:0/1628089955 shutdown_connections 2023-09-10T01:04:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 -- 172.21.15.3:0/1628089955 wait complete. 2023-09-10T01:04:42.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 -- start start 2023-09-10T01:04:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.285+0000 7f88b61d1b80 1 Processor -- start 2023-09-10T01:04:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f255935290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2559357d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88b61d1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558da0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2558da7a0 con 0x55f2558ce710 2023-09-10T01:04:42.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2558da910 con 0x55f2558cf030 2023-09-10T01:04:42.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88b61d1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2558daad0 con 0x55f2558d8e40 2023-09-10T01:04:42.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88b61d1b80 10 client.0 _ll_register_callbacks cb 0x7f8894009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:42.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f889b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2559357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f889b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2559357d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34558/0 (socket says 172.21.15.3:34558) 2023-09-10T01:04:42.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f889b7fe700 1 -- 172.21.15.3:0/3695894292 learned_addr learned my addr 172.21.15.3:0/3695894292 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f889bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f255935290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.286+0000 7f88a0d9c700 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558da0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f889b7fe700 1 -- 172.21.15.3:0/3695894292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 msgr2=0x55f2558da0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f889b7fe700 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558da0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f889b7fe700 1 -- 172.21.15.3:0/3695894292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 msgr2=0x55f255935290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f889b7fe700 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f255935290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f889b7fe700 1 -- 172.21.15.3:0/3695894292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8890009eb0 con 0x55f2558cf030 2023-09-10T01:04:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f889b7fe700 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2559357d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f889c199fb0 tx=0x7f8890003d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97918c3dcad8aaf0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f887b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f887b7fe700 1 -- 172.21.15.3:0/3695894292 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890016070 con 0x55f2558cf030 2023-09-10T01:04:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f888c09a3b0 con 0x55f2558cf030 2023-09-10T01:04:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.287+0000 7f887b7fe700 1 -- 172.21.15.3:0/3695894292 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f8890003680 con 0x55f2558cf030 2023-09-10T01:04:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f887b7fe700 1 -- 172.21.15.3:0/3695894292 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8890003860 con 0x55f2558cf030 2023-09-10T01:04:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 10 client.26150 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f25564b2b0 con 0x55f2558cf030 2023-09-10T01:04:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f887b7fe700 1 -- 172.21.15.3:0/3695894292 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f888c09a3b0 con 0x55f2558cf030 2023-09-10T01:04:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f887b7fe700 1 -- 172.21.15.3:0/3695894292 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f8890004980 con 0x55f2558cf030 2023-09-10T01:04:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 10 client.26150 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 10 client.26150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 20 client.26150 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f25593d0b0 0x55f25593f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 4 client.26150 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:04:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.288+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f255819920 con 0x55f25593d0b0 2023-09-10T01:04:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f889bfff700 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f25593d0b0 0x55f25593f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f889bfff700 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f25593d0b0 0x55f25593f4d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb1e418d124ded7a server_cookie=e332a509171b0ac9 in_seq=0 out_seq=0 2023-09-10T01:04:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f887b7fe700 10 client.26150 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f887b7fe700 1 -- 172.21.15.3:0/3695894292 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f255819920 con 0x55f25593d0b0 2023-09-10T01:04:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f887b7fe700 10 client.26150 handle_command_reply: tid=0 2023-09-10T01:04:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f88b61d1b80 1 client.26150 shutdown 2023-09-10T01:04:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 msgr2=0x55f2559357d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2559357d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f889c199fb0 tx=0x7f8890003d80 comp rx=0 tx=0).stop 2023-09-10T01:04:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 shutdown_connections 2023-09-10T01:04:42.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.289+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f25593d0b0 0x55f25593f4d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2558d8e40 0x55f2558da0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2558ce710 0x55f255935290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f887b7fe700 0 client.26150 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88b61d1b80 1 --2- 172.21.15.3:0/3695894292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2558cf030 0x55f2559357d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 >> 172.21.15.3:0/3695894292 conn(0x55f255843090 msgr2=0x55f2558d2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 shutdown_connections 2023-09-10T01:04:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88b61d1b80 1 -- 172.21.15.3:0/3695894292 wait complete. 2023-09-10T01:04:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88b61d1b80 20 client.26150 trim_cache size 0 max 16384 2023-09-10T01:04:42.324 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:04:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f88840a08d0 msgr2=0x7f88840a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88a27ff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f88840a08d0 0x7f88840a2d90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8894009ec0 tx=0x7f889400b040 comp rx=0 tx=0).stop 2023-09-10T01:04:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c0729d0 msgr2=0x7f889c1989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.290+0000 7f88a27ff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c0729d0 0x7f889c1989a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f888c005910 tx=0x7f888c005720 comp rx=0 tx=0).stop 2023-09-10T01:04:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 shutdown_connections 2023-09-10T01:04:42.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f88840a08d0 0x7f88840a2d90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f889c0729d0 0x7f889c1989a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889c10cef0 0x7f889c19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 --2- 172.21.15.3:0/2379933662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c073380 0x7f889c198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:42.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 >> 172.21.15.3:0/2379933662 conn(0x7f889c0fc160 msgr2=0x7f889c10f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:42.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 shutdown_connections 2023-09-10T01:04:42.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:42.291+0000 7f88a27ff700 1 -- 172.21.15.3:0/2379933662 wait complete. 2023-09-10T01:04:42.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:42 smithi003 ceph-mon[97654]: pgmap v47: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.2 MiB/s wr, 371 op/s 2023-09-10T01:04:42.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:42 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/339057455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:04:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:42 smithi136 ceph-mon[133708]: pgmap v47: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.2 MiB/s wr, 371 op/s 2023-09-10T01:04:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:42 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/339057455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:04:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:42 smithi092 ceph-mon[134067]: pgmap v47: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.2 MiB/s wr, 371 op/s 2023-09-10T01:04:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:42 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/339057455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:04:42.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53479 12.81 MB/sec warmup 24 sec latency 2393.540 ms 2023-09-10T01:04:43.332 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:04:43.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:04:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.551+0000 7fda45bff700 1 -- 172.21.15.3:0/1654848585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40101000 msgr2=0x7fda4010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 --2- 172.21.15.3:0/1654848585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40101000 0x7fda4010d510 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fda3800b0a0 tx=0x7fda3800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 -- 172.21.15.3:0/1654848585 shutdown_connections 2023-09-10T01:04:43.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 --2- 172.21.15.3:0/1654848585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40101000 0x7fda4010d510 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 --2- 172.21.15.3:0/1654848585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40106650 0x7fda40106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 --2- 172.21.15.3:0/1654848585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda40100640 0x7fda40100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 -- 172.21.15.3:0/1654848585 >> 172.21.15.3:0/1654848585 conn(0x7fda400fc170 msgr2=0x7fda400fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 -- 172.21.15.3:0/1654848585 shutdown_connections 2023-09-10T01:04:43.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.552+0000 7fda45bff700 1 -- 172.21.15.3:0/1654848585 wait complete. 2023-09-10T01:04:43.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 Processor -- start 2023-09-10T01:04:43.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 -- start start 2023-09-10T01:04:43.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40100640 0x7fda40072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda40101000 0x7fda40072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40106650 0x7fda401a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda40073670 con 0x7fda40106650 2023-09-10T01:04:43.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda401a4d60 con 0x7fda40100640 2023-09-10T01:04:43.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.553+0000 7fda45bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda401a4ea0 con 0x7fda40101000 2023-09-10T01:04:43.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40106650 0x7fda401a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40100640 0x7fda40072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40106650 0x7fda401a4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49150/0 (socket says 172.21.15.3:49150) 2023-09-10T01:04:43.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 -- 172.21.15.3:0/507915607 learned_addr learned my addr 172.21.15.3:0/507915607 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3effd700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda40101000 0x7fda40072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 -- 172.21.15.3:0/507915607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda40101000 msgr2=0x7fda40072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda40101000 0x7fda40072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 -- 172.21.15.3:0/507915607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40100640 msgr2=0x7fda40072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40100640 0x7fda40072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.554+0000 7fda3ffff700 1 -- 172.21.15.3:0/507915607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda38009d00 con 0x7fda40106650 2023-09-10T01:04:43.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.555+0000 7fda3ffff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40106650 0x7fda401a4820 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fda38009850 tx=0x7fda380077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.555+0000 7fda3cff9700 1 -- 172.21.15.3:0/507915607 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3800e040 con 0x7fda40106650 2023-09-10T01:04:43.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.555+0000 7fda3cff9700 1 -- 172.21.15.3:0/507915607 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda380079e0 con 0x7fda40106650 2023-09-10T01:04:43.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.555+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda401a50a0 con 0x7fda40106650 2023-09-10T01:04:43.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.555+0000 7fda3cff9700 1 -- 172.21.15.3:0/507915607 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380052a0 con 0x7fda40106650 2023-09-10T01:04:43.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.555+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda401a5520 con 0x7fda40106650 2023-09-10T01:04:43.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.558+0000 7fda3cff9700 1 -- 172.21.15.3:0/507915607 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7fda3801d470 con 0x7fda40106650 2023-09-10T01:04:43.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.559+0000 7fda59462b80 1 Processor -- start 2023-09-10T01:04:43.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.559+0000 7fda3cff9700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda280a0950 0x7fda280a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.559+0000 7fda59462b80 1 -- start start 2023-09-10T01:04:43.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.559+0000 7fda3cff9700 1 -- 172.21.15.3:0/507915607 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fda380cd920 con 0x7fda40106650 2023-09-10T01:04:43.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda3effd700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda280a0950 0x7fda280a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda59462b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985942ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda59462b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda59462b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x5598594372b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda59462b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598593837b0 con 0x55985942b0b0 2023-09-10T01:04:43.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda59462b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598593810d0 con 0x55985942a790 2023-09-10T01:04:43.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda59462b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55985937c9c0 con 0x559859434ec0 2023-09-10T01:04:43.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda3f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985942ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda3f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985942ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34592/0 (socket says 172.21.15.3:34592) 2023-09-10T01:04:43.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda3ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x5598594372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda3f7fe700 1 -- 172.21.15.3:0/1848296810 learned_addr learned my addr 172.21.15.3:0/1848296810 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.560+0000 7fda3effd700 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3f7fe700 1 -- 172.21.15.3:0/1848296810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 msgr2=0x5598594372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x5598594372b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3f7fe700 1 -- 172.21.15.3:0/1848296810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 msgr2=0x559859434980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3f7fe700 1 -- 172.21.15.3:0/1848296810 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598591a8e60 con 0x55985942a790 2023-09-10T01:04:43.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3effd700 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:43.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3effd700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda280a0950 0x7fda280a2e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda400740f0 tx=0x7fda34011720 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985942ab70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fda30009d00 tx=0x7fda30011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cefa7dca02ca1f30 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda25ffb700 1 -- 172.21.15.3:0/1848296810 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30004230 con 0x55985942a790 2023-09-10T01:04:43.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda25ffb700 1 -- 172.21.15.3:0/1848296810 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda30004390 con 0x55985942a790 2023-09-10T01:04:43.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.561+0000 7fda25ffb700 1 -- 172.21.15.3:0/1848296810 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30019710 con 0x55985942a790 2023-09-10T01:04:43.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 -- 172.21.15.3:0/1848296810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 msgr2=0x55985942ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985942ab70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fda30009d00 tx=0x7fda30011720 comp rx=0 tx=0).stop 2023-09-10T01:04:43.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 -- 172.21.15.3:0/1848296810 shutdown_connections 2023-09-10T01:04:43.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x5598594372b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985942ab70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 --2- 172.21.15.3:0/1848296810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 -- 172.21.15.3:0/1848296810 >> 172.21.15.3:0/1848296810 conn(0x55985939f110 msgr2=0x559859425c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 -- 172.21.15.3:0/1848296810 shutdown_connections 2023-09-10T01:04:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.562+0000 7fda59462b80 1 -- 172.21.15.3:0/1848296810 wait complete. 2023-09-10T01:04:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 -- start start 2023-09-10T01:04:43.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 Processor -- start 2023-09-10T01:04:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x5598594348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859431b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859432190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55985931abf0 con 0x55985942b0b0 2023-09-10T01:04:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55985931a8f0 con 0x55985942a790 2023-09-10T01:04:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598591fe9a0 con 0x559859434ec0 2023-09-10T01:04:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.563+0000 7fda59462b80 10 client.0 _ll_register_callbacks cb 0x559859264b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859431b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859431b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49172/0 (socket says 172.21.15.3:49172) 2023-09-10T01:04:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 -- 172.21.15.3:0/784446034 learned_addr learned my addr 172.21.15.3:0/784446034 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3ffff700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859432190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x5598594348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 -- 172.21.15.3:0/784446034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 msgr2=0x559859432190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859432190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 -- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 msgr2=0x5598594348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x5598594348e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 -- 172.21.15.3:0/784446034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda30009960 con 0x55985942b0b0 2023-09-10T01:04:43.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3ffff700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859432190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:43.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda3effd700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859431b60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fda3401e990 tx=0x7fda340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed6719ec622eee6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.564+0000 7fda1effd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:43.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.565+0000 7fda1effd700 1 -- 172.21.15.3:0/784446034 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3400f950 con 0x55985942b0b0 2023-09-10T01:04:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.565+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5598592d4570 con 0x55985942b0b0 2023-09-10T01:04:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.565+0000 7fda1effd700 1 -- 172.21.15.3:0/784446034 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fda3400fab0 con 0x55985942b0b0 2023-09-10T01:04:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.565+0000 7fda1effd700 1 -- 172.21.15.3:0/784446034 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda3400fc70 con 0x55985942b0b0 2023-09-10T01:04:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.565+0000 7fda59462b80 10 client.16110 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:43.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.565+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559859191ce0 con 0x55985942b0b0 2023-09-10T01:04:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.565+0000 7fda1effd700 1 -- 172.21.15.3:0/784446034 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5598592d4570 con 0x55985942b0b0 2023-09-10T01:04:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda1effd700 1 -- 172.21.15.3:0/784446034 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fda3400e3b0 con 0x55985942b0b0 2023-09-10T01:04:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda59462b80 10 client.16110 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda59462b80 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:43.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda59462b80 20 client.16110 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:43.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda59462b80 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559859444b10 0x559859446f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda59462b80 4 client.16110 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:43.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559859391920 con 0x559859444b10 2023-09-10T01:04:43.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.566+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559859444b10 0x559859446f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.567+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559859444b10 0x559859446f30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c70070bb3fd3a3ab server_cookie=b5dd28463d8c7f9e in_seq=0 out_seq=0 2023-09-10T01:04:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.567+0000 7fda1effd700 10 client.16110 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.567+0000 7fda1effd700 1 -- 172.21.15.3:0/784446034 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559859391920 con 0x559859444b10 2023-09-10T01:04:43.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.567+0000 7fda1effd700 10 client.16110 handle_command_reply: tid=0 2023-09-10T01:04:43.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.567+0000 7fda59462b80 1 client.16110 shutdown 2023-09-10T01:04:43.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 msgr2=0x559859431b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859431b60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fda3401e990 tx=0x7fda340047c0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 shutdown_connections 2023-09-10T01:04:43.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559859444b10 0x559859446f30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859432190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x5598594348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda1effd700 0 client.16110 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 --2- 172.21.15.3:0/784446034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859431b60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 >> 172.21.15.3:0/784446034 conn(0x55985939f110 msgr2=0x559859425800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 shutdown_connections 2023-09-10T01:04:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 1 -- 172.21.15.3:0/784446034 wait complete. 2023-09-10T01:04:43.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.568+0000 7fda59462b80 20 client.16110 trim_cache size 0 max 16384 2023-09-10T01:04:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.585+0000 7fda59462b80 1 Processor -- start 2023-09-10T01:04:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.585+0000 7fda59462b80 1 -- start start 2023-09-10T01:04:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda59462b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985943ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda59462b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda59462b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859425770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda59462b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559859432bf0 con 0x55985942b0b0 2023-09-10T01:04:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda59462b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55985943bfe0 con 0x55985942a790 2023-09-10T01:04:43.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda59462b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559859431ee0 con 0x559859434ec0 2023-09-10T01:04:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49176/0 (socket says 172.21.15.3:49176) 2023-09-10T01:04:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 -- 172.21.15.3:0/4195168258 learned_addr learned my addr 172.21.15.3:0/4195168258 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985943ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3ffff700 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859425770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 -- 172.21.15.3:0/4195168258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 msgr2=0x559859425770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859425770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 -- 172.21.15.3:0/4195168258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 msgr2=0x55985943ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985943ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3effd700 1 -- 172.21.15.3:0/4195168258 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598591cf2d0 con 0x55985942b0b0 2023-09-10T01:04:43.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.586+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985943ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.587+0000 7fda3effd700 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434410 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fda34009410 tx=0x7fda34005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d617b21e03a9f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.587+0000 7fda267fc700 1 -- 172.21.15.3:0/4195168258 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3400f3c0 con 0x55985942b0b0 2023-09-10T01:04:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.587+0000 7fda267fc700 1 -- 172.21.15.3:0/4195168258 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda3400e8b0 con 0x55985942b0b0 2023-09-10T01:04:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.587+0000 7fda267fc700 1 -- 172.21.15.3:0/4195168258 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda34037430 con 0x55985942b0b0 2023-09-10T01:04:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 -- 172.21.15.3:0/4195168258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 msgr2=0x559859434410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434410 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fda34009410 tx=0x7fda34005dc0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 -- 172.21.15.3:0/4195168258 shutdown_connections 2023-09-10T01:04:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559859434ec0 0x559859425770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985943ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 --2- 172.21.15.3:0/4195168258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55985942b0b0 0x559859434410 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 -- 172.21.15.3:0/4195168258 >> 172.21.15.3:0/4195168258 conn(0x55985939f110 msgr2=0x55985942ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 -- 172.21.15.3:0/4195168258 shutdown_connections 2023-09-10T01:04:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 -- 172.21.15.3:0/4195168258 wait complete. 2023-09-10T01:04:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 -- start start 2023-09-10T01:04:43.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.588+0000 7fda59462b80 1 Processor -- start 2023-09-10T01:04:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda59462b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985948e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda59462b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55985942b0b0 0x55985948ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda59462b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559859434ec0 0x55985948f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda59462b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598594999f0 con 0x559859434ec0 2023-09-10T01:04:43.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda59462b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559859499b30 con 0x55985942a790 2023-09-10T01:04:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda59462b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559859499cf0 con 0x55985942b0b0 2023-09-10T01:04:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda59462b80 10 client.0 _ll_register_callbacks cb 0x7fda3401e760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda3f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985948e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda3effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55985942b0b0 0x55985948ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda3f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985948e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34616/0 (socket says 172.21.15.3:34616) 2023-09-10T01:04:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda3f7fe700 1 -- 172.21.15.3:0/2033957093 learned_addr learned my addr 172.21.15.3:0/2033957093 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.589+0000 7fda3ffff700 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559859434ec0 0x55985948f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda3f7fe700 1 -- 172.21.15.3:0/2033957093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55985942b0b0 msgr2=0x55985948ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55985942b0b0 0x55985948ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda3f7fe700 1 -- 172.21.15.3:0/2033957093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559859434ec0 msgr2=0x55985948f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559859434ec0 0x55985948f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda3f7fe700 1 -- 172.21.15.3:0/2033957093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598591a8e60 con 0x55985942a790 2023-09-10T01:04:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda3effd700 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55985942b0b0 0x55985948ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:43.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda3f7fe700 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985948e8e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fda30009c50 tx=0x7fda30003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=773f224ec6d01a66 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda1effd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda1effd700 1 -- 172.21.15.3:0/2033957093 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30003a90 con 0x55985942a790 2023-09-10T01:04:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda30004390 con 0x55985942a790 2023-09-10T01:04:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda1effd700 1 -- 172.21.15.3:0/2033957093 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fda30029070 con 0x55985942a790 2023-09-10T01:04:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda1effd700 1 -- 172.21.15.3:0/2033957093 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda300041c0 con 0x55985942a790 2023-09-10T01:04:43.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.590+0000 7fda59462b80 10 client.26168 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559859191ce0 con 0x55985942a790 2023-09-10T01:04:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda1effd700 1 -- 172.21.15.3:0/2033957093 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda30004390 con 0x55985942a790 2023-09-10T01:04:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda1effd700 1 -- 172.21.15.3:0/2033957093 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fda30011440 con 0x55985942a790 2023-09-10T01:04:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda59462b80 10 client.26168 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:43.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda59462b80 10 client.26168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda59462b80 20 client.26168 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda59462b80 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55985949b4e0 0x55985949d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda59462b80 4 client.26168 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:04:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5598593759a0 con 0x55985949b4e0 2023-09-10T01:04:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.591+0000 7fda3effd700 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55985949b4e0 0x55985949d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.592+0000 7fda3effd700 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55985949b4e0 0x55985949d900 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aebfbbdaa0a7332 server_cookie=9342581d0446b1fa in_seq=0 out_seq=0 2023-09-10T01:04:43.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.592+0000 7fda1effd700 10 client.26168 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda1effd700 1 -- 172.21.15.3:0/2033957093 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5598593759a0 con 0x55985949b4e0 2023-09-10T01:04:43.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda1effd700 10 client.26168 handle_command_reply: tid=0 2023-09-10T01:04:43.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 client.26168 shutdown 2023-09-10T01:04:43.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 msgr2=0x55985948e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985948e8e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fda30009c50 tx=0x7fda30003eb0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 shutdown_connections 2023-09-10T01:04:43.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55985949b4e0 0x55985949d900 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559859434ec0 0x55985948f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55985942a790 0x55985948e8e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 --2- 172.21.15.3:0/2033957093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55985942b0b0 0x55985948ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda1effd700 0 client.26168 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 >> 172.21.15.3:0/2033957093 conn(0x55985939f110 msgr2=0x5598594256f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 shutdown_connections 2023-09-10T01:04:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 1 -- 172.21.15.3:0/2033957093 wait complete. 2023-09-10T01:04:43.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.593+0000 7fda59462b80 20 client.26168 trim_cache size 0 max 16384 2023-09-10T01:04:43.620 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:04:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:04:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "cbaab956-c275-4609-b229-ca39ed445701", 2023-09-10T01:04:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:04:43.621 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:04:43.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda280a0950 msgr2=0x7fda280a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda280a0950 0x7fda280a2e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda400740f0 tx=0x7fda34011720 comp rx=0 tx=0).stop 2023-09-10T01:04:43.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40106650 msgr2=0x7fda401a4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40106650 0x7fda401a4820 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fda38009850 tx=0x7fda380077f0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 shutdown_connections 2023-09-10T01:04:43.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda280a0950 0x7fda280a2e10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda40106650 0x7fda401a4820 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40100640 0x7fda40072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 --2- 172.21.15.3:0/507915607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda40101000 0x7fda40072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 >> 172.21.15.3:0/507915607 conn(0x7fda400fc170 msgr2=0x7fda400fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 shutdown_connections 2023-09-10T01:04:43.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.594+0000 7fda45bff700 1 -- 172.21.15.3:0/507915607 wait complete. 2023-09-10T01:04:43.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:04:43.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53498 12.30 MB/sec warmup 25 sec latency 3364.842 ms 2023-09-10T01:04:43.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 -- 172.21.15.3:0/1285354734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3734100610 msgr2=0x7f3734100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 --2- 172.21.15.3:0/1285354734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3734100610 0x7f3734100a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f37240098f0 tx=0x7f3724009c00 comp rx=0 tx=0).stop 2023-09-10T01:04:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 -- 172.21.15.3:0/1285354734 shutdown_connections 2023-09-10T01:04:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 --2- 172.21.15.3:0/1285354734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37341066b0 0x7f3734106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 --2- 172.21.15.3:0/1285354734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3734100fd0 0x7f373410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 --2- 172.21.15.3:0/1285354734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3734100610 0x7f3734100a90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 -- 172.21.15.3:0/1285354734 >> 172.21.15.3:0/1285354734 conn(0x7f37340fc100 msgr2=0x7f37340fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 -- 172.21.15.3:0/1285354734 shutdown_connections 2023-09-10T01:04:43.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.841+0000 7f373b3ff700 1 -- 172.21.15.3:0/1285354734 wait complete. 2023-09-10T01:04:43.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.843+0000 7f373b3ff700 1 Processor -- start 2023-09-10T01:04:43.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.843+0000 7f373b3ff700 1 -- start start 2023-09-10T01:04:43.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.843+0000 7f373b3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3734100610 0x7f3734196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.843+0000 7f373b3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3734100fd0 0x7f3734196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373b3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37341066b0 0x7f373419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373b3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373419a600 con 0x7f3734100610 2023-09-10T01:04:43.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373b3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373419a770 con 0x7f37341066b0 2023-09-10T01:04:43.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373b3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373419a930 con 0x7f3734100fd0 2023-09-10T01:04:43.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373919b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3734100610 0x7f3734196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37341066b0 0x7f373419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37341066b0 0x7f373419a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34642/0 (socket says 172.21.15.3:34642) 2023-09-10T01:04:43.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 -- 172.21.15.3:0/3547997888 learned_addr learned my addr 172.21.15.3:0/3547997888 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373899a700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3734100fd0 0x7f3734196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 -- 172.21.15.3:0/3547997888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3734100fd0 msgr2=0x7f3734196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3734100fd0 0x7f3734196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 -- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3734100610 msgr2=0x7f3734196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3734100610 0x7f3734196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 -- 172.21.15.3:0/3547997888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c009d00 con 0x7f37341066b0 2023-09-10T01:04:43.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373919b700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3734100610 0x7f3734196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:43.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.844+0000 7f373999c700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37341066b0 0x7f373419a000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f372c00e3c0 tx=0x7f372c00e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.845+0000 7f372a7fc700 1 -- 172.21.15.3:0/3547997888 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c0090d0 con 0x7f37341066b0 2023-09-10T01:04:43.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.845+0000 7f372a7fc700 1 -- 172.21.15.3:0/3547997888 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f372c00f040 con 0x7f37341066b0 2023-09-10T01:04:43.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.845+0000 7f372a7fc700 1 -- 172.21.15.3:0/3547997888 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c007c20 con 0x7f37341066b0 2023-09-10T01:04:43.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.845+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3724009520 con 0x7f37341066b0 2023-09-10T01:04:43.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.845+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37341a3390 con 0x7f37341066b0 2023-09-10T01:04:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.850+0000 7f372a7fc700 1 -- 172.21.15.3:0/3547997888 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) v1 ==== 135491+0+0 (secure 0 0 0) 0x7f372c003a00 con 0x7f37341066b0 2023-09-10T01:04:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 Processor -- start 2023-09-10T01:04:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 -- start start 2023-09-10T01:04:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f372a7fc700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f37200a08d0 0x7f37200a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393f0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3933b730 con 0x557f393e6430 2023-09-10T01:04:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f39339050 con 0x557f393e5b10 2023-09-10T01:04:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f374ede4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f39334940 con 0x557f393f1260 2023-09-10T01:04:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f372a7fc700 1 -- 172.21.15.3:0/3547997888 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f372c0cc5d0 con 0x7f37341066b0 2023-09-10T01:04:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f373919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f373899a700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f37200a08d0 0x7f37200a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f373919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34652/0 (socket says 172.21.15.3:34652) 2023-09-10T01:04:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f373919b700 1 -- 172.21.15.3:0/3964696333 learned_addr learned my addr 172.21.15.3:0/3964696333 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.851+0000 7f373899a700 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393f0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373999c700 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373919b700 1 -- 172.21.15.3:0/3964696333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 msgr2=0x557f393f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373919b700 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373919b700 1 -- 172.21.15.3:0/3964696333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 msgr2=0x557f393f0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373919b700 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393f0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373919b700 1 -- 172.21.15.3:0/3964696333 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f39160de0 con 0x557f393e5b10 2023-09-10T01:04:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373919b700 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e5ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f37300115f0 tx=0x7f3730011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74365139e345e837 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f373899a700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f37200a08d0 0x7f37200a2d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3724009fd0 tx=0x7f3724019040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f371f7fe700 1 -- 172.21.15.3:0/3964696333 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730004030 con 0x557f393e5b10 2023-09-10T01:04:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f371f7fe700 1 -- 172.21.15.3:0/3964696333 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3730003680 con 0x557f393e5b10 2023-09-10T01:04:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.852+0000 7f371f7fe700 1 -- 172.21.15.3:0/3964696333 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300196f0 con 0x557f393e5b10 2023-09-10T01:04:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 -- 172.21.15.3:0/3964696333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 msgr2=0x557f393e5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e5ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f37300115f0 tx=0x7f3730011900 comp rx=0 tx=0).stop 2023-09-10T01:04:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 -- 172.21.15.3:0/3964696333 shutdown_connections 2023-09-10T01:04:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e5ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3964696333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393f0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 -- 172.21.15.3:0/3964696333 >> 172.21.15.3:0/3964696333 conn(0x557f39357090 msgr2=0x557f393dda50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 -- 172.21.15.3:0/3964696333 shutdown_connections 2023-09-10T01:04:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.853+0000 7f374ede4b80 1 -- 172.21.15.3:0/3964696333 wait complete. 2023-09-10T01:04:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.854+0000 7f374ede4b80 1 -- start start 2023-09-10T01:04:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.854+0000 7f374ede4b80 1 Processor -- start 2023-09-10T01:04:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e5b10 0x557f393edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393ee510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393f1260 0x557f393eea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f374ede4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f392d2b70 con 0x557f393e6430 2023-09-10T01:04:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f374ede4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f392d2870 con 0x557f393f1260 2023-09-10T01:04:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f374ede4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f391b6920 con 0x557f393e5b10 2023-09-10T01:04:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f374ede4b80 10 client.0 _ll_register_callbacks cb 0x557f3921ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f373919b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e5b10 0x557f393edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f373919b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e5b10 0x557f393edfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50618/0 (socket says 172.21.15.3:50618) 2023-09-10T01:04:43.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f373919b700 1 -- 172.21.15.3:0/170747410 learned_addr learned my addr 172.21.15.3:0/170747410 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f373899a700 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.855+0000 7f373999c700 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393f1260 0x557f393eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f373919b700 1 -- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393f1260 msgr2=0x557f393eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f373919b700 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393f1260 0x557f393eea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f373919b700 1 -- 172.21.15.3:0/170747410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 msgr2=0x557f393ee510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f373919b700 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393ee510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f373919b700 1 -- 172.21.15.3:0/170747410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3730009d50 con 0x557f393e5b10 2023-09-10T01:04:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f373919b700 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e5b10 0x557f393edfd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3730009770 tx=0x7f3730005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a47cca34678dc4f1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f371cff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:04:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f371cff9700 1 -- 172.21.15.3:0/170747410 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730004400 con 0x557f393e5b10 2023-09-10T01:04:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557f3928c4f0 con 0x557f393e5b10 2023-09-10T01:04:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f371cff9700 1 -- 172.21.15.3:0/170747410 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3730019450 con 0x557f393e5b10 2023-09-10T01:04:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f371cff9700 1 -- 172.21.15.3:0/170747410 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3730004560 con 0x557f393e5b10 2023-09-10T01:04:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f374ede4b80 10 client.26107 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.856+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557f39149c60 con 0x557f393e5b10 2023-09-10T01:04:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f371cff9700 1 -- 172.21.15.3:0/170747410 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557f3928c4f0 con 0x557f393e5b10 2023-09-10T01:04:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f371cff9700 1 -- 172.21.15.3:0/170747410 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3730024880 con 0x557f393e5b10 2023-09-10T01:04:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f374ede4b80 10 client.26107 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f374ede4b80 10 client.26107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f374ede4b80 20 client.26107 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f374ede4b80 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39400bf0 0x557f39403010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f374ede4b80 4 client.26107 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f393498a0 con 0x557f39400bf0 2023-09-10T01:04:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.857+0000 7f373899a700 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39400bf0 0x557f39403010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.858+0000 7f371cff9700 10 client.26107 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.858+0000 7f373899a700 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39400bf0 0x557f39403010 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19893ef01ce19ede server_cookie=c4f31b5455453cca in_seq=0 out_seq=0 2023-09-10T01:04:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.858+0000 7f371cff9700 1 -- 172.21.15.3:0/170747410 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557f393498a0 con 0x557f39400bf0 2023-09-10T01:04:43.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.858+0000 7f371cff9700 10 client.26107 handle_command_reply: tid=0 2023-09-10T01:04:43.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.858+0000 7f374ede4b80 1 client.26107 shutdown 2023-09-10T01:04:43.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e5b10 msgr2=0x557f393edfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e5b10 0x557f393edfd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3730009770 tx=0x7f3730005e00 comp rx=0 tx=0).stop 2023-09-10T01:04:43.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 shutdown_connections 2023-09-10T01:04:43.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39400bf0 0x557f39403010 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e5b10 0x557f393edfd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393f1260 0x557f393eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 --2- 172.21.15.3:0/170747410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e6430 0x557f393ee510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f371cff9700 0 client.26107 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 >> 172.21.15.3:0/170747410 conn(0x557f39357090 msgr2=0x557f393dd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 shutdown_connections 2023-09-10T01:04:43.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 1 -- 172.21.15.3:0/170747410 wait complete. 2023-09-10T01:04:43.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.859+0000 7f374ede4b80 20 client.26107 trim_cache size 0 max 16384 2023-09-10T01:04:43.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 Processor -- start 2023-09-10T01:04:43.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 -- start start 2023-09-10T01:04:43.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e6430 0x557f393ee880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393f1260 0x557f393ecc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f393d7dc0 con 0x557f393f1260 2023-09-10T01:04:43.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f393df1c0 con 0x557f393e5b10 2023-09-10T01:04:43.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f374ede4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f393fd690 con 0x557f393e6430 2023-09-10T01:04:43.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f373899a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e6430 0x557f393ee880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f373899a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e6430 0x557f393ee880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50622/0 (socket says 172.21.15.3:50622) 2023-09-10T01:04:43.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.877+0000 7f373899a700 1 -- 172.21.15.3:0/2653874896 learned_addr learned my addr 172.21.15.3:0/2653874896 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373999c700 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393f1260 0x557f393ecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373919b700 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373899a700 1 -- 172.21.15.3:0/2653874896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 msgr2=0x557f393e8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373899a700 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373899a700 1 -- 172.21.15.3:0/2653874896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393f1260 msgr2=0x557f393ecc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373899a700 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393f1260 0x557f393ecc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373899a700 1 -- 172.21.15.3:0/2653874896 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f39187250 con 0x557f393e6430 2023-09-10T01:04:43.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373999c700 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393f1260 0x557f393ecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:04:43.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f373899a700 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e6430 0x557f393ee880 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f37240098f0 tx=0x7f3724020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee0fca19e4ff7fd8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f371ffff700 1 -- 172.21.15.3:0/2653874896 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372402c030 con 0x557f393e6430 2023-09-10T01:04:43.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f371ffff700 1 -- 172.21.15.3:0/2653874896 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f372402d040 con 0x557f393e6430 2023-09-10T01:04:43.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.878+0000 7f371ffff700 1 -- 172.21.15.3:0/2653874896 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372400bb90 con 0x557f393e6430 2023-09-10T01:04:43.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 -- 172.21.15.3:0/2653874896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e6430 msgr2=0x557f393ee880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e6430 0x557f393ee880 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f37240098f0 tx=0x7f3724020000 comp rx=0 tx=0).stop 2023-09-10T01:04:43.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 -- 172.21.15.3:0/2653874896 shutdown_connections 2023-09-10T01:04:43.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393f1260 0x557f393ecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e5b10 0x557f393e8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 --2- 172.21.15.3:0/2653874896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393e6430 0x557f393ee880 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 -- 172.21.15.3:0/2653874896 >> 172.21.15.3:0/2653874896 conn(0x557f39357090 msgr2=0x557f393ec530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 -- 172.21.15.3:0/2653874896 shutdown_connections 2023-09-10T01:04:43.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 -- 172.21.15.3:0/2653874896 wait complete. 2023-09-10T01:04:43.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.879+0000 7f374ede4b80 1 -- start start 2023-09-10T01:04:43.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 1 Processor -- start 2023-09-10T01:04:43.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e5b10 0x557f393f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e6430 0x557f393f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f393f6a10 con 0x557f393e5b10 2023-09-10T01:04:43.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f393f6b50 con 0x557f393e6430 2023-09-10T01:04:43.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f393f6d10 con 0x557f393f1260 2023-09-10T01:04:43.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.880+0000 7f374ede4b80 10 client.0 _ll_register_callbacks cb 0x7f3730011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:43.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373919b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e5b10 0x557f393f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e6430 0x557f393f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373919b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e5b10 0x557f393f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49236/0 (socket says 172.21.15.3:49236) 2023-09-10T01:04:43.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e6430 0x557f393f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34690/0 (socket says 172.21.15.3:34690) 2023-09-10T01:04:43.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373919b700 1 -- 172.21.15.3:0/3271284316 learned_addr learned my addr 172.21.15.3:0/3271284316 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:43.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373999c700 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 -- 172.21.15.3:0/3271284316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 msgr2=0x557f393f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 -- 172.21.15.3:0/3271284316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e5b10 msgr2=0x557f393f7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:43.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e5b10 0x557f393f7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 -- 172.21.15.3:0/3271284316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372401f040 con 0x557f393e6430 2023-09-10T01:04:43.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373919b700 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e5b10 0x557f393f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:04:43.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f373899a700 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e6430 0x557f393f5f50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f37240098c0 tx=0x7f3724005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=868fbd5ff3773352 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:43.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f371cff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:43.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.881+0000 7f371cff9700 1 -- 172.21.15.3:0/3271284316 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372402c050 con 0x557f393e6430 2023-09-10T01:04:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3730003680 con 0x557f393e6430 2023-09-10T01:04:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f371cff9700 1 -- 172.21.15.3:0/3271284316 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3724033d90 con 0x557f393e6430 2023-09-10T01:04:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f371cff9700 1 -- 172.21.15.3:0/3271284316 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f372400b560 con 0x557f393e6430 2023-09-10T01:04:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 10 client.26186 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:43.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557f3915f2b0 con 0x557f393e6430 2023-09-10T01:04:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f371cff9700 1 -- 172.21.15.3:0/3271284316 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f372404c080 con 0x557f393e6430 2023-09-10T01:04:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f371cff9700 1 -- 172.21.15.3:0/3271284316 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3724003680 con 0x557f393e6430 2023-09-10T01:04:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 10 client.26186 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 10 client.26186 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:43.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 20 client.26186 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39455b90 0x557f39457fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 4 client.26186 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:04:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557f3932d920 con 0x557f39455b90 2023-09-10T01:04:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.882+0000 7f373919b700 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39455b90 0x557f39457fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:43.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f373919b700 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39455b90 0x557f39457fb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a0e202f429005cc server_cookie=9e1ec79f751150e in_seq=0 out_seq=0 2023-09-10T01:04:43.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f371cff9700 10 client.26186 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f371cff9700 1 -- 172.21.15.3:0/3271284316 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557f3932d920 con 0x557f39455b90 2023-09-10T01:04:43.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f371cff9700 10 client.26186 handle_command_reply: tid=0 2023-09-10T01:04:43.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 client.26186 shutdown 2023-09-10T01:04:43.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e6430 msgr2=0x557f393f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e6430 0x557f393f5f50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f37240098c0 tx=0x7f3724005bd0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 shutdown_connections 2023-09-10T01:04:43.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557f39455b90 0x557f39457fb0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557f393f1260 0x557f393f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557f393e5b10 0x557f393f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 --2- 172.21.15.3:0/3271284316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f393e6430 0x557f393f5f50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f371cff9700 0 client.26186 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 >> 172.21.15.3:0/3271284316 conn(0x557f39357090 msgr2=0x557f393dd650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 shutdown_connections 2023-09-10T01:04:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.883+0000 7f374ede4b80 1 -- 172.21.15.3:0/3271284316 wait complete. 2023-09-10T01:04:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f374ede4b80 20 client.26186 trim_cache size 0 max 16384 2023-09-10T01:04:43.908 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:04:43.908 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:04:43.909 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:04:43.909 INFO:teuthology.orchestra.run.smithi003.stdout: "cbaab956-c275-4609-b229-ca39ed445701": { 2023-09-10T01:04:43.909 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:04:43.910 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "cbaab956-c275-4609-b229-ca39ed445701", 2023-09-10T01:04:43.910 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:04:43.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:04:43.911 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:04:43.911 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:04:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f37200a08d0 msgr2=0x7f37200a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f37200a08d0 0x7f37200a2d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3724009fd0 tx=0x7f3724019040 comp rx=0 tx=0).stop 2023-09-10T01:04:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37341066b0 msgr2=0x7f373419a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37341066b0 0x7f373419a000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f372c00e3c0 tx=0x7f372c00e6d0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 shutdown_connections 2023-09-10T01:04:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f37200a08d0 0x7f37200a2d90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37341066b0 0x7f373419a000 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3734100610 0x7f3734196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 --2- 172.21.15.3:0/3547997888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3734100fd0 0x7f3734196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 >> 172.21.15.3:0/3547997888 conn(0x7f37340fc100 msgr2=0x7f37340fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 shutdown_connections 2023-09-10T01:04:43.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:04:43.884+0000 7f373b3ff700 1 -- 172.21.15.3:0/3547997888 wait complete. 2023-09-10T01:04:43.915 INFO:tasks.cephfs.filesystem:scrub status for tag:cbaab956-c275-4609-b229-ca39ed445701 - {'path': '/', 'tag': 'cbaab956-c275-4609-b229-ca39ed445701', 'options': 'recursive,force'} 2023-09-10T01:04:44.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:44 smithi003 ceph-mon[97654]: pgmap v48: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.2 MiB/s wr, 371 op/s 2023-09-10T01:04:44.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:44 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:04:44.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:44 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:44.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:44 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:04:44.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:44 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:44 smithi136 ceph-mon[133708]: pgmap v48: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.2 MiB/s wr, 371 op/s 2023-09-10T01:04:44.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:44 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:04:44.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:44 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:44.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:44 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:04:44.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:44 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:44 smithi092 ceph-mon[134067]: pgmap v48: 129 pgs: 129 active+clean; 17 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.2 MiB/s wr, 371 op/s 2023-09-10T01:04:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:44 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:04:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:44 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:44 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:04:44.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:44 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:44.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53562 11.82 MB/sec warmup 26 sec latency 906.005 ms 2023-09-10T01:04:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:45 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:04:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:45 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:04:45.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:45 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:45.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:45 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:45.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:45 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:04:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:45 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:04:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:45 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:45 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:45.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:45 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:04:45.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:45 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:04:45.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:45 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:45.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:45 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:04:45.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 54059 11.42 MB/sec warmup 27 sec latency 1439.339 ms 2023-09-10T01:04:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:46 smithi003 ceph-mon[97654]: pgmap v49: 129 pgs: 129 active+clean; 18 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 1.4 MiB/s wr, 400 op/s 2023-09-10T01:04:46.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:46 smithi003 ceph-mon[97654]: mgrmap e27: x(active, since 92s), standbys: z, y 2023-09-10T01:04:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:46 smithi136 ceph-mon[133708]: pgmap v49: 129 pgs: 129 active+clean; 18 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 1.4 MiB/s wr, 400 op/s 2023-09-10T01:04:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:46 smithi136 ceph-mon[133708]: mgrmap e27: x(active, since 92s), standbys: z, y 2023-09-10T01:04:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:46 smithi092 ceph-mon[134067]: pgmap v49: 129 pgs: 129 active+clean; 18 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 1.4 MiB/s wr, 400 op/s 2023-09-10T01:04:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:46 smithi092 ceph-mon[134067]: mgrmap e27: x(active, since 92s), standbys: z, y 2023-09-10T01:04:46.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 56992 11.66 MB/sec warmup 28 sec latency 11.124 ms 2023-09-10T01:04:47.740 INFO:tasks.workunit.client.0.smithi003.stdout: 1 57108 11.26 MB/sec warmup 29 sec latency 827.283 ms 2023-09-10T01:04:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 -- 172.21.15.3:0/1863053115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8068520 msgr2=0x7f2ca810d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/1863053115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8068520 0x7f2ca810d420 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000b0a0 tx=0x7f2ca000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 -- 172.21.15.3:0/1863053115 shutdown_connections 2023-09-10T01:04:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/1863053115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ca81066a0 0x7f2ca8106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/1863053115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8068520 0x7f2ca810d420 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/1863053115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ca8067b60 0x7f2ca8067fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 -- 172.21.15.3:0/1863053115 >> 172.21.15.3:0/1863053115 conn(0x7f2ca8074fe0 msgr2=0x7f2ca80753f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 -- 172.21.15.3:0/1863053115 shutdown_connections 2023-09-10T01:04:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.127+0000 7f2caf9ff700 1 -- 172.21.15.3:0/1863053115 wait complete. 2023-09-10T01:04:48.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.128+0000 7f2caf9ff700 1 Processor -- start 2023-09-10T01:04:48.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.128+0000 7f2caf9ff700 1 -- start start 2023-09-10T01:04:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2caf9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8067b60 0x7f2ca819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2caf9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ca8068520 0x7f2ca819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2caf9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ca81066a0 0x7f2ca81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2caf9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca8198e10 con 0x7f2ca8067b60 2023-09-10T01:04:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2caf9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca8198f80 con 0x7f2ca81066a0 2023-09-10T01:04:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2caf9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca8199140 con 0x7f2ca8068520 2023-09-10T01:04:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cad79b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8067b60 0x7f2ca819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cad79b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8067b60 0x7f2ca819e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55130/0 (socket says 172.21.15.3:55130) 2023-09-10T01:04:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cad79b700 1 -- 172.21.15.3:0/4207391473 learned_addr learned my addr 172.21.15.3:0/4207391473 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ca8068520 0x7f2ca819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cadf9c700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ca81066a0 0x7f2ca81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cad79b700 1 -- 172.21.15.3:0/4207391473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ca8068520 msgr2=0x7f2ca819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cad79b700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ca8068520 0x7f2ca819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cad79b700 1 -- 172.21.15.3:0/4207391473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ca81066a0 msgr2=0x7f2ca81987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.129+0000 7f2cad79b700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ca81066a0 0x7f2ca81987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.130+0000 7f2cad79b700 1 -- 172.21.15.3:0/4207391473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0009d00 con 0x7f2ca8067b60 2023-09-10T01:04:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.130+0000 7f2cad79b700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8067b60 0x7f2ca819e6e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400e9a0 tx=0x7f2ca400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.130+0000 7f2c9e7fc700 1 -- 172.21.15.3:0/4207391473 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4009790 con 0x7f2ca8067b60 2023-09-10T01:04:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.130+0000 7f2c9e7fc700 1 -- 172.21.15.3:0/4207391473 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ca4010430 con 0x7f2ca8067b60 2023-09-10T01:04:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.130+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca8199420 con 0x7f2ca8067b60 2023-09-10T01:04:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.130+0000 7f2c9e7fc700 1 -- 172.21.15.3:0/4207391473 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4018680 con 0x7f2ca8067b60 2023-09-10T01:04:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.130+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca81a5780 con 0x7f2ca8067b60 2023-09-10T01:04:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.136+0000 7f2cc341db80 1 Processor -- start 2023-09-10T01:04:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.138+0000 7f2cc341db80 1 -- start start 2023-09-10T01:04:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.139+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2434d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.139+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e242ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.139+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.139+0000 7f2cc341db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e237f6d0 con 0x55d0e242eb10 2023-09-10T01:04:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.139+0000 7f2cc341db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e237cff0 con 0x55d0e242f4d0 2023-09-10T01:04:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.139+0000 7f2cc341db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e23788e0 con 0x55d0e24349b0 2023-09-10T01:04:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.140+0000 7f2c9e7fc700 1 -- 172.21.15.3:0/4207391473 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f2ca40105a0 con 0x7f2ca8067b60 2023-09-10T01:04:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.140+0000 7f2cadf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2434d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.140+0000 7f2cadf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2434d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:52922/0 (socket says 172.21.15.3:52922) 2023-09-10T01:04:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.140+0000 7f2cadf9c700 1 -- 172.21.15.3:0/1240991488 learned_addr learned my addr 172.21.15.3:0/1240991488 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.140+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.141+0000 7f2cad79b700 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e242ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.141+0000 7f2cacf9a700 1 -- 172.21.15.3:0/1240991488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 msgr2=0x55d0e2434d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.141+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2434d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.141+0000 7f2cacf9a700 1 -- 172.21.15.3:0/1240991488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 msgr2=0x55d0e242ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.141+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e242ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.141+0000 7f2cacf9a700 1 -- 172.21.15.3:0/1240991488 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0e21a4d80 con 0x55d0e242f4d0 2023-09-10T01:04:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.141+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243b740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009c80 tx=0x7f2c98009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3f4005f40bb6dd3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:48.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.142+0000 7f2c937fe700 1 -- 172.21.15.3:0/1240991488 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98011cb0 con 0x55d0e242f4d0 2023-09-10T01:04:48.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.142+0000 7f2c937fe700 1 -- 172.21.15.3:0/1240991488 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c98011e10 con 0x55d0e242f4d0 2023-09-10T01:04:48.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.142+0000 7f2c937fe700 1 -- 172.21.15.3:0/1240991488 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98005280 con 0x55d0e242f4d0 2023-09-10T01:04:48.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 -- 172.21.15.3:0/1240991488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 msgr2=0x55d0e243b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243b740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009c80 tx=0x7f2c98009f90 comp rx=0 tx=0).stop 2023-09-10T01:04:48.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 -- 172.21.15.3:0/1240991488 shutdown_connections 2023-09-10T01:04:48.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2434d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243b740 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1240991488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e242ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 -- 172.21.15.3:0/1240991488 >> 172.21.15.3:0/1240991488 conn(0x55d0e239b030 msgr2=0x55d0e2421560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:48.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2cc341db80 1 -- 172.21.15.3:0/1240991488 shutdown_connections 2023-09-10T01:04:48.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.143+0000 7f2c9e7fc700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f2c940a06c0 0x7f2c940a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.144+0000 7f2c9e7fc700 1 -- 172.21.15.3:0/4207391473 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f2ca4014070 con 0x7f2ca8067b60 2023-09-10T01:04:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.144+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f2c940a06c0 0x7f2c940a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.144+0000 7f2cc341db80 1 -- 172.21.15.3:0/1240991488 wait complete. 2023-09-10T01:04:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.145+0000 7f2cc341db80 1 -- start start 2023-09-10T01:04:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.146+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f2c940a06c0 0x7f2c940a2b80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c98004c50 tx=0x7f2c98004d50 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.146+0000 7f2cc341db80 1 Processor -- start 2023-09-10T01:04:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.147+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242eb10 0x55d0e243b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.147+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242f4d0 0x55d0e2438880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.147+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2438de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.147+0000 7f2cc341db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2316b10 con 0x55d0e242f4d0 2023-09-10T01:04:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.147+0000 7f2cc341db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2316810 con 0x55d0e242eb10 2023-09-10T01:04:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.147+0000 7f2cc341db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e21fa8c0 con 0x55d0e24349b0 2023-09-10T01:04:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.147+0000 7f2cc341db80 10 client.0 _ll_register_callbacks cb 0x55d0e2260a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.148+0000 7f2cacf9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242f4d0 0x55d0e2438880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.148+0000 7f2cacf9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242f4d0 0x55d0e2438880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55146/0 (socket says 172.21.15.3:55146) 2023-09-10T01:04:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.148+0000 7f2cacf9a700 1 -- 172.21.15.3:0/1283984301 learned_addr learned my addr 172.21.15.3:0/1283984301 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.149+0000 7f2cacf9a700 1 -- 172.21.15.3:0/1283984301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 msgr2=0x55d0e2438de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:04:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.149+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2438de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.149+0000 7f2cacf9a700 1 -- 172.21.15.3:0/1283984301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242eb10 msgr2=0x55d0e243b560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:04:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.149+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242eb10 0x55d0e243b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.149+0000 7f2cacf9a700 1 -- 172.21.15.3:0/1283984301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c980098e0 con 0x55d0e242f4d0 2023-09-10T01:04:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.149+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242f4d0 0x55d0e2438880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c98019000 tx=0x7f2c98009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f82cd1e0acb695 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.150+0000 7f2c90ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:04:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.150+0000 7f2c90ff9700 1 -- 172.21.15.3:0/1283984301 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98004dc0 con 0x55d0e242f4d0 2023-09-10T01:04:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.150+0000 7f2c90ff9700 1 -- 172.21.15.3:0/1283984301 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f2c98004f20 con 0x55d0e242f4d0 2023-09-10T01:04:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.150+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0e22d0490 con 0x55d0e242f4d0 2023-09-10T01:04:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.151+0000 7f2c90ff9700 1 -- 172.21.15.3:0/1283984301 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c980050e0 con 0x55d0e242f4d0 2023-09-10T01:04:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.151+0000 7f2cc341db80 10 client.16143 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.151+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d0e218dc00 con 0x55d0e242f4d0 2023-09-10T01:04:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.152+0000 7f2c90ff9700 1 -- 172.21.15.3:0/1283984301 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d0e22d0490 con 0x55d0e242f4d0 2023-09-10T01:04:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.152+0000 7f2c90ff9700 1 -- 172.21.15.3:0/1283984301 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f2c98024a90 con 0x55d0e242f4d0 2023-09-10T01:04:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.153+0000 7f2cc341db80 10 client.16143 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.153+0000 7f2cc341db80 10 client.16143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.153+0000 7f2cc341db80 20 client.16143 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.153+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e2448af0 0x55d0e244af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.153+0000 7f2cc341db80 4 client.16143 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:04:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.153+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0e238d840 con 0x55d0e2448af0 2023-09-10T01:04:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.153+0000 7f2cad79b700 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e2448af0 0x55d0e244af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.154+0000 7f2cad79b700 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e2448af0 0x55d0e244af10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f56baf590bf65ea server_cookie=6e6ce187a2e3412e in_seq=0 out_seq=0 2023-09-10T01:04:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.154+0000 7f2c90ff9700 10 client.16143 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2c90ff9700 1 -- 172.21.15.3:0/1283984301 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d0e238d840 con 0x55d0e2448af0 2023-09-10T01:04:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2c90ff9700 10 client.16143 handle_command_reply: tid=0 2023-09-10T01:04:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 client.16143 shutdown 2023-09-10T01:04:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242f4d0 msgr2=0x55d0e2438880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242f4d0 0x55d0e2438880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c98019000 tx=0x7f2c98009f90 comp rx=0 tx=0).stop 2023-09-10T01:04:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 shutdown_connections 2023-09-10T01:04:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e2448af0 0x55d0e244af10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2438de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242eb10 0x55d0e243b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 --2- 172.21.15.3:0/1283984301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242f4d0 0x55d0e2438880 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 >> 172.21.15.3:0/1283984301 conn(0x55d0e239b030 msgr2=0x55d0e24207a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.156+0000 7f2c90ff9700 0 client.16143 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.157+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 shutdown_connections 2023-09-10T01:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.157+0000 7f2cc341db80 1 -- 172.21.15.3:0/1283984301 wait complete. 2023-09-10T01:04:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.157+0000 7f2cc341db80 20 client.16143 trim_cache size 0 max 16384 2023-09-10T01:04:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 Processor -- start 2023-09-10T01:04:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 -- start start 2023-09-10T01:04:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2434000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2420f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2435360 con 0x55d0e242eb10 2023-09-10T01:04:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2439870 con 0x55d0e242f4d0 2023-09-10T01:04:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.176+0000 7f2cc341db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2434610 con 0x55d0e24349b0 2023-09-10T01:04:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cacf9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cadf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2420f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cad79b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2434000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cacf9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40146/0 (socket says 172.21.15.3:40146) 2023-09-10T01:04:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cacf9a700 1 -- 172.21.15.3:0/2445577959 learned_addr learned my addr 172.21.15.3:0/2445577959 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cadf9c700 1 -- 172.21.15.3:0/2445577959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 msgr2=0x55d0e243fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cadf9c700 1 --2- 172.21.15.3:0/2445577959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cadf9c700 1 -- 172.21.15.3:0/2445577959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 msgr2=0x55d0e2434000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cadf9c700 1 --2- 172.21.15.3:0/2445577959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2434000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cadf9c700 1 -- 172.21.15.3:0/2445577959 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0e21cb1f0 con 0x55d0e24349b0 2023-09-10T01:04:48.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.177+0000 7f2cadf9c700 1 --2- 172.21.15.3:0/2445577959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2420f20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0015000 tx=0x7f2ca0009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20201101b1fc256 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:48.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2c93fff700 1 -- 172.21.15.3:0/2445577959 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000e040 con 0x55d0e24349b0 2023-09-10T01:04:48.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2c93fff700 1 -- 172.21.15.3:0/2445577959 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ca000bcb0 con 0x55d0e24349b0 2023-09-10T01:04:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2c93fff700 1 -- 172.21.15.3:0/2445577959 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00052a0 con 0x55d0e24349b0 2023-09-10T01:04:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 -- 172.21.15.3:0/2445577959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 msgr2=0x55d0e2420f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 --2- 172.21.15.3:0/2445577959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2420f20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0015000 tx=0x7f2ca0009c90 comp rx=0 tx=0).stop 2023-09-10T01:04:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 -- 172.21.15.3:0/2445577959 shutdown_connections 2023-09-10T01:04:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 --2- 172.21.15.3:0/2445577959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e24349b0 0x55d0e2420f20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 --2- 172.21.15.3:0/2445577959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2434000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 --2- 172.21.15.3:0/2445577959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e242f4d0 0x55d0e243fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 -- 172.21.15.3:0/2445577959 >> 172.21.15.3:0/2445577959 conn(0x55d0e239b030 msgr2=0x55d0e2425b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.178+0000 7f2cc341db80 1 -- 172.21.15.3:0/2445577959 shutdown_connections 2023-09-10T01:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.179+0000 7f2cc341db80 1 -- 172.21.15.3:0/2445577959 wait complete. 2023-09-10T01:04:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.179+0000 7f2cc341db80 1 -- start start 2023-09-10T01:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.179+0000 7f2cc341db80 1 Processor -- start 2023-09-10T01:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.179+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2490ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e242f4d0 0x55d0e2491000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cc341db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e24349b0 0x55d0e2495990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cc341db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2495ed0 con 0x55d0e242eb10 2023-09-10T01:04:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cc341db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2496040 con 0x55d0e24349b0 2023-09-10T01:04:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cc341db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e2496200 con 0x55d0e242f4d0 2023-09-10T01:04:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cc341db80 10 client.0 _ll_register_callbacks cb 0x7f2c98004030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:04:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e24349b0 0x55d0e2495990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e24349b0 0x55d0e2495990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40154/0 (socket says 172.21.15.3:40154) 2023-09-10T01:04:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 -- 172.21.15.3:0/851371609 learned_addr learned my addr 172.21.15.3:0/851371609 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:04:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cad79b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2490ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e242f4d0 0x55d0e2491000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 -- 172.21.15.3:0/851371609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e242f4d0 msgr2=0x55d0e2491000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e242f4d0 0x55d0e2491000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 -- 172.21.15.3:0/851371609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 msgr2=0x55d0e2490ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:04:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2490ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 -- 172.21.15.3:0/851371609 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca810dce0 con 0x55d0e24349b0 2023-09-10T01:04:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cacf9a700 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e242f4d0 0x55d0e2491000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2cadf9c700 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e24349b0 0x55d0e2495990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009590 tx=0x7f2ca0007710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4c6da58033bffe server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.180+0000 7f2c90ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2c90ff9700 1 -- 172.21.15.3:0/851371609 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000e040 con 0x55d0e24349b0 2023-09-10T01:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c98011e10 con 0x55d0e24349b0 2023-09-10T01:04:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2c90ff9700 1 -- 172.21.15.3:0/851371609 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f2ca000be20 con 0x55d0e24349b0 2023-09-10T01:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2c90ff9700 1 -- 172.21.15.3:0/851371609 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ca00050c0 con 0x55d0e24349b0 2023-09-10T01:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2cc341db80 10 client.26201 fetch_fsmap learned FSMap version 23 2023-09-10T01:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d0e21a3250 con 0x55d0e24349b0 2023-09-10T01:04:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2c90ff9700 1 -- 172.21.15.3:0/851371609 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c98011e10 con 0x55d0e24349b0 2023-09-10T01:04:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.181+0000 7f2c90ff9700 1 -- 172.21.15.3:0/851371609 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f2ca000b0d0 con 0x55d0e24349b0 2023-09-10T01:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cc341db80 10 client.26201 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cc341db80 10 client.26201 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cc341db80 20 client.26201 populate_metadata read hostname 'smithi003' 2023-09-10T01:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cc341db80 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e249d620 0x55d0e249fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:04:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cc341db80 4 client.26201 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:04:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d0e23718c0 con 0x55d0e249d620 2023-09-10T01:04:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cad79b700 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e249d620 0x55d0e249fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:04:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2cad79b700 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e249d620 0x55d0e249fa40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a510444509a5c173 server_cookie=4eea03e78fa5eef6 in_seq=0 out_seq=0 2023-09-10T01:04:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.182+0000 7f2c90ff9700 10 client.26201 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.302+0000 7f2c90ff9700 1 -- 172.21.15.3:0/851371609 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d0e23718c0 con 0x55d0e249d620 2023-09-10T01:04:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.302+0000 7f2c90ff9700 10 client.26201 handle_command_reply: tid=0 2023-09-10T01:04:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.302+0000 7f2cc341db80 1 client.26201 shutdown 2023-09-10T01:04:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e24349b0 msgr2=0x55d0e2495990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e24349b0 0x55d0e2495990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0009590 tx=0x7f2ca0007710 comp rx=0 tx=0).stop 2023-09-10T01:04:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 shutdown_connections 2023-09-10T01:04:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d0e249d620 0x55d0e249fa40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0e24349b0 0x55d0e2495990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d0e242eb10 0x55d0e2490ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2c90ff9700 0 client.26201 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:04:48.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 --2- 172.21.15.3:0/851371609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d0e242f4d0 0x55d0e2491000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 >> 172.21.15.3:0/851371609 conn(0x55d0e239b030 msgr2=0x55d0e2439ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:48.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 shutdown_connections 2023-09-10T01:04:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 1 -- 172.21.15.3:0/851371609 wait complete. 2023-09-10T01:04:48.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.303+0000 7f2cc341db80 20 client.26201 trim_cache size 0 max 16384 2023-09-10T01:04:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:04:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:04:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:04:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:04:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f2c940a06c0 msgr2=0x7f2c940a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f2c940a06c0 0x7f2c940a2b80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2c98004c50 tx=0x7f2c98004d50 comp rx=0 tx=0).stop 2023-09-10T01:04:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8067b60 msgr2=0x7f2ca819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:04:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8067b60 0x7f2ca819e6e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400e9a0 tx=0x7f2ca400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 shutdown_connections 2023-09-10T01:04:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f2c940a06c0 0x7f2c940a2b80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ca81066a0 0x7f2ca81987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ca8067b60 0x7f2ca819e6e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 --2- 172.21.15.3:0/4207391473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ca8068520 0x7f2ca819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:04:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.304+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 >> 172.21.15.3:0/4207391473 conn(0x7f2ca8074fe0 msgr2=0x7f2ca810b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:04:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.305+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 shutdown_connections 2023-09-10T01:04:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:04:48.305+0000 7f2caf9ff700 1 -- 172.21.15.3:0/4207391473 wait complete. 2023-09-10T01:04:48.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:48 smithi003 ceph-mon[97654]: pgmap v50: 129 pgs: 129 active+clean; 18 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 745 KiB/s wr, 316 op/s 2023-09-10T01:04:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:48 smithi136 ceph-mon[133708]: pgmap v50: 129 pgs: 129 active+clean; 18 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 745 KiB/s wr, 316 op/s 2023-09-10T01:04:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:48 smithi092 ceph-mon[134067]: pgmap v50: 129 pgs: 129 active+clean; 18 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 745 KiB/s wr, 316 op/s 2023-09-10T01:04:48.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 57108 10.89 MB/sec warmup 30 sec latency 1827.403 ms 2023-09-10T01:04:49.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:49 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:04:49.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:49 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:04:49.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:49 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:04:49.739 INFO:tasks.workunit.client.0.smithi003.stdout: 1 58038 10.61 MB/sec warmup 31 sec latency 1930.969 ms 2023-09-10T01:04:50.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:50 smithi003 ceph-mon[97654]: pgmap v51: 129 pgs: 129 active+clean; 20 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 923 KiB/s wr, 335 op/s 2023-09-10T01:04:50.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:50 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:04:50.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:50 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:04:50.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:50 smithi136 ceph-mon[133708]: pgmap v51: 129 pgs: 129 active+clean; 20 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 923 KiB/s wr, 335 op/s 2023-09-10T01:04:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:50 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:04:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:50 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:04:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:50 smithi092 ceph-mon[134067]: pgmap v51: 129 pgs: 129 active+clean; 20 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 923 KiB/s wr, 335 op/s 2023-09-10T01:04:50.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:50 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:04:50.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:50 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:04:50.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 60579 10.80 MB/sec warmup 32 sec latency 17.819 ms 2023-09-10T01:04:51.740 INFO:tasks.workunit.client.0.smithi003.stdout: 1 61938 10.64 MB/sec warmup 33 sec latency 13.492 ms 2023-09-10T01:04:52.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:52 smithi003 ceph-mon[97654]: pgmap v52: 129 pgs: 129 active+clean; 20 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 466 KiB/s wr, 84 op/s 2023-09-10T01:04:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:52 smithi136 ceph-mon[133708]: pgmap v52: 129 pgs: 129 active+clean; 20 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 466 KiB/s wr, 84 op/s 2023-09-10T01:04:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:52 smithi092 ceph-mon[134067]: pgmap v52: 129 pgs: 129 active+clean; 20 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 466 KiB/s wr, 84 op/s 2023-09-10T01:04:52.740 INFO:tasks.workunit.client.0.smithi003.stdout: 1 64269 10.74 MB/sec warmup 34 sec latency 17.485 ms 2023-09-10T01:04:53.740 INFO:tasks.workunit.client.0.smithi003.stdout: 1 66199 10.63 MB/sec warmup 35 sec latency 24.952 ms 2023-09-10T01:04:54.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:54 smithi136 ceph-mon[133708]: pgmap v53: 129 pgs: 129 active+clean; 20 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 466 KiB/s wr, 84 op/s 2023-09-10T01:04:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:54 smithi092 ceph-mon[134067]: pgmap v53: 129 pgs: 129 active+clean; 20 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 466 KiB/s wr, 84 op/s 2023-09-10T01:04:54.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:54 smithi003 ceph-mon[97654]: pgmap v53: 129 pgs: 129 active+clean; 20 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 466 KiB/s wr, 84 op/s 2023-09-10T01:04:54.740 INFO:tasks.workunit.client.0.smithi003.stdout: 1 67832 10.67 MB/sec warmup 36 sec latency 120.944 ms 2023-09-10T01:04:55.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68556 10.44 MB/sec warmup 37 sec latency 225.006 ms 2023-09-10T01:04:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:56 smithi136 ceph-mon[133708]: pgmap v54: 129 pgs: 129 active+clean; 34 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 3.5 MiB/s wr, 273 op/s 2023-09-10T01:04:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:56 smithi092 ceph-mon[134067]: pgmap v54: 129 pgs: 129 active+clean; 34 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 3.5 MiB/s wr, 273 op/s 2023-09-10T01:04:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:56 smithi003 ceph-mon[97654]: pgmap v54: 129 pgs: 129 active+clean; 34 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 3.5 MiB/s wr, 273 op/s 2023-09-10T01:04:56.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71186 10.61 MB/sec warmup 38 sec latency 15.560 ms 2023-09-10T01:04:57.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 72469 10.47 MB/sec warmup 39 sec latency 12.282 ms 2023-09-10T01:04:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:04:58 smithi136 ceph-mon[133708]: pgmap v55: 129 pgs: 129 active+clean; 34 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 3.3 MiB/s wr, 244 op/s 2023-09-10T01:04:58.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:04:58 smithi092 ceph-mon[134067]: pgmap v55: 129 pgs: 129 active+clean; 34 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 3.3 MiB/s wr, 244 op/s 2023-09-10T01:04:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:04:58 smithi003 ceph-mon[97654]: pgmap v55: 129 pgs: 129 active+clean; 34 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 3.3 MiB/s wr, 244 op/s 2023-09-10T01:04:58.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 74911 10.56 MB/sec warmup 40 sec latency 10.978 ms 2023-09-10T01:04:59.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 76369 10.44 MB/sec warmup 41 sec latency 21.571 ms 2023-09-10T01:05:00.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:00 smithi136 ceph-mon[133708]: pgmap v56: 129 pgs: 129 active+clean; 33 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 337 op/s 2023-09-10T01:05:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:00 smithi092 ceph-mon[134067]: pgmap v56: 129 pgs: 129 active+clean; 33 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 337 op/s 2023-09-10T01:05:00.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:00 smithi003 ceph-mon[97654]: pgmap v56: 129 pgs: 129 active+clean; 33 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 337 op/s 2023-09-10T01:05:00.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78539 10.52 MB/sec warmup 42 sec latency 10.538 ms 2023-09-10T01:05:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 -- 172.21.15.3:0/2775394219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238106670 msgr2=0x7f9238106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 --2- 172.21.15.3:0/2775394219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238106670 0x7f9238106a50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f92280099c0 tx=0x7f9228009cd0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 -- 172.21.15.3:0/2775394219 shutdown_connections 2023-09-10T01:05:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 --2- 172.21.15.3:0/2775394219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9238100f70 0x7f923810d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 --2- 172.21.15.3:0/2775394219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238106670 0x7f9238106a50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 --2- 172.21.15.3:0/2775394219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92381005b0 0x7f9238100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 -- 172.21.15.3:0/2775394219 >> 172.21.15.3:0/2775394219 conn(0x7f92380fc0c0 msgr2=0x7f92380fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 -- 172.21.15.3:0/2775394219 shutdown_connections 2023-09-10T01:05:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.466+0000 7f923d5ff700 1 -- 172.21.15.3:0/2775394219 wait complete. 2023-09-10T01:05:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.467+0000 7f923d5ff700 1 Processor -- start 2023-09-10T01:05:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.467+0000 7f923d5ff700 1 -- start start 2023-09-10T01:05:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f923d5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92381005b0 0x7f923819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f923d5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238100f70 0x7f923819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f923d5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9238106670 0x7f92381986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f923d5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238198d30 con 0x7f92381005b0 2023-09-10T01:05:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f923d5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238198ea0 con 0x7f9238100f70 2023-09-10T01:05:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f923d5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238199060 con 0x7f9238106670 2023-09-10T01:05:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f92367fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238100f70 0x7f923819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f92367fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238100f70 0x7f923819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:37212/0 (socket says 172.21.15.3:37212) 2023-09-10T01:05:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f92367fc700 1 -- 172.21.15.3:0/44850269 learned_addr learned my addr 172.21.15.3:0/44850269 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f92377fe700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9238106670 0x7f92381986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.468+0000 7f9236ffd700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92381005b0 0x7f923819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f92367fc700 1 -- 172.21.15.3:0/44850269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9238106670 msgr2=0x7f92381986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f92367fc700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9238106670 0x7f92381986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f92367fc700 1 -- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92381005b0 msgr2=0x7f923819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f92367fc700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92381005b0 0x7f923819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f92367fc700 1 -- 172.21.15.3:0/44850269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92280095f0 con 0x7f9238100f70 2023-09-10T01:05:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f92367fc700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238100f70 0x7f923819eb90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f922c00e9a0 tx=0x7f922c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f921ffff700 1 -- 172.21.15.3:0/44850269 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c009790 con 0x7f9238100f70 2023-09-10T01:05:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9238199340 con 0x7f9238100f70 2023-09-10T01:05:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f921ffff700 1 -- 172.21.15.3:0/44850269 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f922c010430 con 0x7f9238100f70 2023-09-10T01:05:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f921ffff700 1 -- 172.21.15.3:0/44850269 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c018680 con 0x7f9238100f70 2023-09-10T01:05:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.469+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92381a5640 con 0x7f9238100f70 2023-09-10T01:05:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.473+0000 7f921ffff700 1 -- 172.21.15.3:0/44850269 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f922c0105a0 con 0x7f9238100f70 2023-09-10T01:05:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.474+0000 7f9250ed1b80 1 Processor -- start 2023-09-10T01:05:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.474+0000 7f9250ed1b80 1 -- start start 2023-09-10T01:05:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e2701a0 0x55a42e270580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e26a300 0x55a42e26a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26acc0 0x55a42e276f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f921ffff700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92200a07b0 0x7f92200a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e1c36d0 con 0x55a42e26acc0 2023-09-10T01:05:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e1c0ff0 con 0x55a42e2701a0 2023-09-10T01:05:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e1bc8e0 con 0x55a42e26a300 2023-09-10T01:05:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f921ffff700 1 -- 172.21.15.3:0/44850269 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f922c014070 con 0x7f9238100f70 2023-09-10T01:05:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f92377fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e2701a0 0x55a42e270580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e26a300 0x55a42e26a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f92367fc700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92200a07b0 0x7f92200a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f92377fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e2701a0 0x55a42e270580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:37224/0 (socket says 172.21.15.3:37224) 2023-09-10T01:05:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f92377fe700 1 -- 172.21.15.3:0/3279567382 learned_addr learned my addr 172.21.15.3:0/3279567382 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.475+0000 7f92377fe700 1 -- 172.21.15.3:0/3279567382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e26a300 msgr2=0x55a42e26a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f92377fe700 1 --2- 172.21.15.3:0/3279567382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e26a300 0x55a42e26a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f92377fe700 1 -- 172.21.15.3:0/3279567382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26acc0 msgr2=0x55a42e276f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f92377fe700 1 --2- 172.21.15.3:0/3279567382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26acc0 0x55a42e276f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f92377fe700 1 -- 172.21.15.3:0/3279567382 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a42dfe8d80 con 0x55a42e2701a0 2023-09-10T01:05:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f92377fe700 1 --2- 172.21.15.3:0/3279567382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e2701a0 0x55a42e270580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9230009520 tx=0x7f92300087f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f8b687274c137cb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f92367fc700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92200a07b0 0x7f92200a2c70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f922c096070 tx=0x7f922c011f90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f921d7fa700 1 -- 172.21.15.3:0/3279567382 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923000e070 con 0x55a42e2701a0 2023-09-10T01:05:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f921d7fa700 1 -- 172.21.15.3:0/3279567382 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f923000f040 con 0x55a42e2701a0 2023-09-10T01:05:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.476+0000 7f921d7fa700 1 -- 172.21.15.3:0/3279567382 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230016800 con 0x55a42e2701a0 2023-09-10T01:05:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3279567382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e2701a0 msgr2=0x55a42e270580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3279567382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e2701a0 0x55a42e270580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9230009520 tx=0x7f92300087f0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3279567382 shutdown_connections 2023-09-10T01:05:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3279567382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26acc0 0x55a42e276f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3279567382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e2701a0 0x55a42e270580 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3279567382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e26a300 0x55a42e26a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3279567382 >> 172.21.15.3:0/3279567382 conn(0x55a42e1df030 msgr2=0x55a42e265560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:01.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3279567382 shutdown_connections 2023-09-10T01:05:01.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.477+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3279567382 wait complete. 2023-09-10T01:05:01.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.478+0000 7f9250ed1b80 1 -- start start 2023-09-10T01:05:01.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.478+0000 7f9250ed1b80 1 Processor -- start 2023-09-10T01:05:01.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e276d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2740a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2745e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e15ab10 con 0x55a42e26a300 2023-09-10T01:05:01.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e15a810 con 0x55a42e26acc0 2023-09-10T01:05:01.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e03e8c0 con 0x55a42e2701a0 2023-09-10T01:05:01.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9250ed1b80 10 client.0 _ll_register_callbacks cb 0x55a42e0a4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:01.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2740a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2740a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:37234/0 (socket says 172.21.15.3:37234) 2023-09-10T01:05:01.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 -- 172.21.15.3:0/149498471 learned_addr learned my addr 172.21.15.3:0/149498471 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:01.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f92377fe700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e276d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f92377fe700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 -- 172.21.15.3:0/149498471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 msgr2=0x55a42e2745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 -- 172.21.15.3:0/149498471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 msgr2=0x55a42e276d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e276d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f9236ffd700 1 -- 172.21.15.3:0/149498471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92300091d0 con 0x55a42e26acc0 2023-09-10T01:05:01.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.479+0000 7f92377fe700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:01.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f9236ffd700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2740a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92280058e0 tx=0x7f92280050f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13ca7063d007fedc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:01.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f921aff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:01.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f921aff5700 1 -- 172.21.15.3:0/149498471 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92280056e0 con 0x55a42e26acc0 2023-09-10T01:05:01.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a42e114490 con 0x55a42e26acc0 2023-09-10T01:05:01.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f921aff5700 1 -- 172.21.15.3:0/149498471 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f922800b790 con 0x55a42e26acc0 2023-09-10T01:05:01.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f921aff5700 1 -- 172.21.15.3:0/149498471 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f922800b950 con 0x55a42e26acc0 2023-09-10T01:05:01.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f9250ed1b80 10 client.26213 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:01.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a42dfd1c00 con 0x55a42e26acc0 2023-09-10T01:05:01.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.480+0000 7f921aff5700 1 -- 172.21.15.3:0/149498471 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a42e114490 con 0x55a42e26acc0 2023-09-10T01:05:01.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f921aff5700 1 -- 172.21.15.3:0/149498471 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9228024c80 con 0x55a42e26acc0 2023-09-10T01:05:01.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f9250ed1b80 10 client.26213 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:01.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f9250ed1b80 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:01.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f9250ed1b80 20 client.26213 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:01.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e284360 0x55a42e286780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f9250ed1b80 4 client.26213 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:01.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a42e1d1840 con 0x55a42e284360 2023-09-10T01:05:01.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.481+0000 7f92377fe700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e284360 0x55a42e286780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.482+0000 7f92377fe700 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e284360 0x55a42e286780 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c42531dfbb5b57e server_cookie=b3e4b76289f72fb0 in_seq=0 out_seq=0 2023-09-10T01:05:01.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.482+0000 7f921aff5700 10 client.26213 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:01.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.482+0000 7f921aff5700 1 -- 172.21.15.3:0/149498471 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a42e1d1840 con 0x55a42e284360 2023-09-10T01:05:01.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.482+0000 7f921aff5700 10 client.26213 handle_command_reply: tid=0 2023-09-10T01:05:01.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.482+0000 7f9250ed1b80 1 client.26213 shutdown 2023-09-10T01:05:01.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 msgr2=0x55a42e2740a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2740a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92280058e0 tx=0x7f92280050f0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 shutdown_connections 2023-09-10T01:05:01.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e284360 0x55a42e286780 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e276d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/149498471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2740a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 >> 172.21.15.3:0/149498471 conn(0x55a42e1df030 msgr2=0x55a42e2647a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f921aff5700 0 client.26213 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 shutdown_connections 2023-09-10T01:05:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 1 -- 172.21.15.3:0/149498471 wait complete. 2023-09-10T01:05:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.483+0000 7f9250ed1b80 20 client.26213 trim_cache size 0 max 16384 2023-09-10T01:05:01.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 Processor -- start 2023-09-10T01:05:01.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 -- start start 2023-09-10T01:05:01.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e27b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e264f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e265460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e1d1840 con 0x55a42e26a300 2023-09-10T01:05:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e114490 con 0x55a42e26acc0 2023-09-10T01:05:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.499+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e266d10 con 0x55a42e2701a0 2023-09-10T01:05:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e264f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f92377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e27b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e264f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:37250/0 (socket says 172.21.15.3:37250) 2023-09-10T01:05:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 -- 172.21.15.3:0/3895509351 learned_addr learned my addr 172.21.15.3:0/3895509351 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f92377fe700 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e265460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 -- 172.21.15.3:0/3895509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 msgr2=0x55a42e265460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e265460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 -- 172.21.15.3:0/3895509351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 msgr2=0x55a42e27b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e27b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 -- 172.21.15.3:0/3895509351 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a42e28b370 con 0x55a42e26acc0 2023-09-10T01:05:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f92377fe700 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e265460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:05:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.500+0000 7f9236ffd700 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e264f20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f922800eb20 tx=0x7f922800f4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b24f97244561018 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f921dffb700 1 -- 172.21.15.3:0/3895509351 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922803a070 con 0x55a42e26acc0 2023-09-10T01:05:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f921dffb700 1 -- 172.21.15.3:0/3895509351 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f922801e040 con 0x55a42e26acc0 2023-09-10T01:05:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f921dffb700 1 -- 172.21.15.3:0/3895509351 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922800ee00 con 0x55a42e26acc0 2023-09-10T01:05:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3895509351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 msgr2=0x55a42e264f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e264f20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f922800eb20 tx=0x7f922800f4e0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3895509351 shutdown_connections 2023-09-10T01:05:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e265460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e27b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/3895509351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e264f20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3895509351 >> 172.21.15.3:0/3895509351 conn(0x55a42e1df030 msgr2=0x55a42e269b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.501+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3895509351 shutdown_connections 2023-09-10T01:05:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.502+0000 7f9250ed1b80 1 -- 172.21.15.3:0/3895509351 wait complete. 2023-09-10T01:05:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.502+0000 7f9250ed1b80 1 -- start start 2023-09-10T01:05:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.502+0000 7f9250ed1b80 1 Processor -- start 2023-09-10T01:05:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e2cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9250ed1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2d7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e2d76a0 con 0x55a42e26a300 2023-09-10T01:05:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e2d77e0 con 0x55a42e26acc0 2023-09-10T01:05:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9250ed1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42e2d79a0 con 0x55a42e2701a0 2023-09-10T01:05:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9250ed1b80 10 client.0 _ll_register_callbacks cb 0x7f9228003ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f92377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e2cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f92377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e2cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43214/0 (socket says 172.21.15.3:43214) 2023-09-10T01:05:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f92377fe700 1 -- 172.21.15.3:0/544813368 learned_addr learned my addr 172.21.15.3:0/544813368 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.503+0000 7f92377fe700 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2d7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f9236ffd700 1 -- 172.21.15.3:0/544813368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 msgr2=0x55a42e2d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f9236ffd700 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2d7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f9236ffd700 1 -- 172.21.15.3:0/544813368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 msgr2=0x55a42e2cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f9236ffd700 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e2cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f9236ffd700 1 -- 172.21.15.3:0/544813368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a42dfd1c00 con 0x55a42e26acc0 2023-09-10T01:05:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f9236ffd700 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2cc810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f922800f7f0 tx=0x7f9228009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f25a876862eec9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f921aff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f921aff5700 1 -- 172.21.15.3:0/544813368 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922803a070 con 0x55a42e26acc0 2023-09-10T01:05:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f921aff5700 1 -- 172.21.15.3:0/544813368 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f922800ed60 con 0x55a42e26acc0 2023-09-10T01:05:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.504+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f923000f040 con 0x55a42e26acc0 2023-09-10T01:05:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f921aff5700 1 -- 172.21.15.3:0/544813368 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f922800e3e0 con 0x55a42e26acc0 2023-09-10T01:05:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 10 client.26219 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a42e00f1f0 con 0x55a42e26acc0 2023-09-10T01:05:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f921aff5700 1 -- 172.21.15.3:0/544813368 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f923000f040 con 0x55a42e26acc0 2023-09-10T01:05:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f921aff5700 1 -- 172.21.15.3:0/544813368 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f922800b260 con 0x55a42e26acc0 2023-09-10T01:05:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 10 client.26219 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 10 client.26219 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 20 client.26219 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e2d8c70 0x55a42e2db090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 4 client.26219 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:05:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a42e1b58c0 con 0x55a42e2d8c70 2023-09-10T01:05:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.505+0000 7f92377fe700 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e2d8c70 0x55a42e2db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.506+0000 7f92377fe700 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e2d8c70 0x55a42e2db090 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63ccafcc1721e395 server_cookie=cdb5486bba795511 in_seq=0 out_seq=0 2023-09-10T01:05:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.506+0000 7f921aff5700 10 client.26219 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:01.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f921aff5700 1 -- 172.21.15.3:0/544813368 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a42e1b58c0 con 0x55a42e2d8c70 2023-09-10T01:05:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f921aff5700 10 client.26219 handle_command_reply: tid=0 2023-09-10T01:05:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 client.26219 shutdown 2023-09-10T01:05:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 msgr2=0x55a42e2cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2cc810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f922800f7f0 tx=0x7f9228009550 comp rx=0 tx=0).stop 2023-09-10T01:05:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 shutdown_connections 2023-09-10T01:05:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a42e2d8c70 0x55a42e2db090 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a42e2701a0 0x55a42e2d7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f921aff5700 0 client.26219 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42e26a300 0x55a42e2cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 --2- 172.21.15.3:0/544813368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a42e26acc0 0x55a42e2cc810 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.606+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 >> 172.21.15.3:0/544813368 conn(0x55a42e1df030 msgr2=0x55a42e269b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.607+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 shutdown_connections 2023-09-10T01:05:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.607+0000 7f9250ed1b80 1 -- 172.21.15.3:0/544813368 wait complete. 2023-09-10T01:05:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.607+0000 7f9250ed1b80 20 client.26219 trim_cache size 0 max 16384 2023-09-10T01:05:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:05:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:05:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:05:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:05:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92200a07b0 msgr2=0x7f92200a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92200a07b0 0x7f92200a2c70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f922c096070 tx=0x7f922c011f90 comp rx=0 tx=0).stop 2023-09-10T01:05:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238100f70 msgr2=0x7f923819eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238100f70 0x7f923819eb90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f922c00e9a0 tx=0x7f922c00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 shutdown_connections 2023-09-10T01:05:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92200a07b0 0x7f92200a2c70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9238106670 0x7f92381986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92381005b0 0x7f923819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 --2- 172.21.15.3:0/44850269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9238100f70 0x7f923819eb90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 >> 172.21.15.3:0/44850269 conn(0x7f92380fc0c0 msgr2=0x7f9238072830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 shutdown_connections 2023-09-10T01:05:01.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:01.608+0000 7f923d5ff700 1 -- 172.21.15.3:0/44850269 wait complete. 2023-09-10T01:05:01.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 79119 10.31 MB/sec warmup 43 sec latency 108.094 ms 2023-09-10T01:05:02.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:02 smithi136 ceph-mon[133708]: pgmap v57: 129 pgs: 129 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.5 MiB/s wr, 460 op/s 2023-09-10T01:05:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:02 smithi092 ceph-mon[134067]: pgmap v57: 129 pgs: 129 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.5 MiB/s wr, 460 op/s 2023-09-10T01:05:02.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:02 smithi003 ceph-mon[97654]: pgmap v57: 129 pgs: 129 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.5 MiB/s wr, 460 op/s 2023-09-10T01:05:02.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 81575 10.43 MB/sec warmup 44 sec latency 57.102 ms 2023-09-10T01:05:03.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82673 10.28 MB/sec warmup 45 sec latency 15.696 ms 2023-09-10T01:05:04.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:04 smithi136 ceph-mon[133708]: pgmap v58: 129 pgs: 129 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 422 op/s 2023-09-10T01:05:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:04 smithi092 ceph-mon[134067]: pgmap v58: 129 pgs: 129 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 422 op/s 2023-09-10T01:05:04.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:04 smithi003 ceph-mon[97654]: pgmap v58: 129 pgs: 129 active+clean; 33 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 422 op/s 2023-09-10T01:05:04.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 85318 10.40 MB/sec warmup 46 sec latency 10.061 ms 2023-09-10T01:05:05.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 86394 10.25 MB/sec warmup 47 sec latency 37.471 ms 2023-09-10T01:05:06.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:06 smithi136 ceph-mon[133708]: pgmap v59: 129 pgs: 129 active+clean; 33 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 588 op/s 2023-09-10T01:05:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:06 smithi092 ceph-mon[134067]: pgmap v59: 129 pgs: 129 active+clean; 33 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 588 op/s 2023-09-10T01:05:06.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:06 smithi003 ceph-mon[97654]: pgmap v59: 129 pgs: 129 active+clean; 33 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 588 op/s 2023-09-10T01:05:06.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 88958 10.39 MB/sec warmup 48 sec latency 35.907 ms 2023-09-10T01:05:07.741 INFO:tasks.workunit.client.0.smithi003.stdout: 1 90179 10.29 MB/sec warmup 49 sec latency 13.147 ms 2023-09-10T01:05:08.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:08 smithi136 ceph-mon[133708]: pgmap v60: 129 pgs: 129 active+clean; 33 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 400 op/s 2023-09-10T01:05:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:08 smithi092 ceph-mon[134067]: pgmap v60: 129 pgs: 129 active+clean; 33 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 400 op/s 2023-09-10T01:05:08.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:08 smithi003 ceph-mon[97654]: pgmap v60: 129 pgs: 129 active+clean; 33 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 400 op/s 2023-09-10T01:05:08.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92589 10.36 MB/sec warmup 50 sec latency 14.827 ms 2023-09-10T01:05:09.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 93901 10.26 MB/sec warmup 51 sec latency 90.501 ms 2023-09-10T01:05:10.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:10 smithi136 ceph-mon[133708]: pgmap v61: 129 pgs: 129 active+clean; 34 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2023-09-10T01:05:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:10 smithi092 ceph-mon[134067]: pgmap v61: 129 pgs: 129 active+clean; 34 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2023-09-10T01:05:10.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:10 smithi003 ceph-mon[97654]: pgmap v61: 129 pgs: 129 active+clean; 34 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 411 op/s 2023-09-10T01:05:10.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96155 10.33 MB/sec warmup 52 sec latency 29.685 ms 2023-09-10T01:05:11.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 97676 10.24 MB/sec warmup 53 sec latency 13.035 ms 2023-09-10T01:05:12.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:12 smithi136 ceph-mon[133708]: pgmap v62: 129 pgs: 129 active+clean; 39 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 401 op/s 2023-09-10T01:05:12.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:12 smithi092 ceph-mon[134067]: pgmap v62: 129 pgs: 129 active+clean; 39 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 401 op/s 2023-09-10T01:05:12.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:12 smithi003 ceph-mon[97654]: pgmap v62: 129 pgs: 129 active+clean; 39 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.2 MiB/s wr, 401 op/s 2023-09-10T01:05:12.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 99804 10.30 MB/sec warmup 54 sec latency 10.761 ms 2023-09-10T01:05:13.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 101912 10.26 MB/sec warmup 55 sec latency 23.942 ms 2023-09-10T01:05:13.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:05:14.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 -- 172.21.15.3:0/1469070015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581021c0 msgr2=0x7fe85810a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1469070015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581021c0 0x7fe85810a6a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0098f0 tx=0x7fe84c009c00 comp rx=0 tx=0).stop 2023-09-10T01:05:14.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 -- 172.21.15.3:0/1469070015 shutdown_connections 2023-09-10T01:05:14.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1469070015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8581018a0 0x7fe858101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1469070015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe85810ac70 0x7fe85810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1469070015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581021c0 0x7fe85810a6a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 -- 172.21.15.3:0/1469070015 >> 172.21.15.3:0/1469070015 conn(0x7fe858076710 msgr2=0x7fe858076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 -- 172.21.15.3:0/1469070015 shutdown_connections 2023-09-10T01:05:14.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.137+0000 7fe85cbff700 1 -- 172.21.15.3:0/1469070015 wait complete. 2023-09-10T01:05:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.138+0000 7fe85cbff700 1 Processor -- start 2023-09-10T01:05:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.138+0000 7fe85cbff700 1 -- start start 2023-09-10T01:05:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85cbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581018a0 0x7fe85819e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85cbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8581021c0 0x7fe85819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85cbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe85810ac70 0x7fe858198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85cbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe858198eb0 con 0x7fe85810ac70 2023-09-10T01:05:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85cbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe858199020 con 0x7fe8581018a0 2023-09-10T01:05:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85cbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8581991e0 con 0x7fe8581021c0 2023-09-10T01:05:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581018a0 0x7fe85819e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581018a0 0x7fe85819e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52470/0 (socket says 172.21.15.3:52470) 2023-09-10T01:05:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85659c700 1 -- 172.21.15.3:0/1453969845 learned_addr learned my addr 172.21.15.3:0/1453969845 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe855d9b700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8581021c0 0x7fe85819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe856d9d700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe85810ac70 0x7fe858198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85659c700 1 -- 172.21.15.3:0/1453969845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8581021c0 msgr2=0x7fe85819ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85659c700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8581021c0 0x7fe85819ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.139+0000 7fe85659c700 1 -- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe85810ac70 msgr2=0x7fe858198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe85659c700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe85810ac70 0x7fe858198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe85659c700 1 -- 172.21.15.3:0/1453969845 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c009520 con 0x7fe8581018a0 2023-09-10T01:05:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe85659c700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581018a0 0x7fe85819e7d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe84800e9c0 tx=0x7fe84800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe8477fe700 1 -- 172.21.15.3:0/1453969845 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848004a30 con 0x7fe8581018a0 2023-09-10T01:05:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe8477fe700 1 -- 172.21.15.3:0/1453969845 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe84800cc80 con 0x7fe8581018a0 2023-09-10T01:05:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8581994c0 con 0x7fe8581018a0 2023-09-10T01:05:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe8477fe700 1 -- 172.21.15.3:0/1453969845 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8480054e0 con 0x7fe8581018a0 2023-09-10T01:05:14.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.140+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8581a57c0 con 0x7fe8581018a0 2023-09-10T01:05:14.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.143+0000 7fe8477fe700 1 -- 172.21.15.3:0/1453969845 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe848004b90 con 0x7fe8581018a0 2023-09-10T01:05:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 Processor -- start 2023-09-10T01:05:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 -- start start 2023-09-10T01:05:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe8477fe700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe8400a0830 0x7fe8400a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe8477fe700 1 -- 172.21.15.3:0/1453969845 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe848014070 con 0x7fe8581018a0 2023-09-10T01:05:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b2710 0x561db81b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b3030 0x561db81bc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81bce40 0x561db81bf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db810b730 con 0x561db81b2710 2023-09-10T01:05:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db8109050 con 0x561db81b3030 2023-09-10T01:05:14.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe87055ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db8104940 con 0x561db81bce40 2023-09-10T01:05:14.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe855d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b3030 0x561db81bc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe856d9d700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe8400a0830 0x7fe8400a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.145+0000 7fe85659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b2710 0x561db81b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b3030 0x561db81bc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52486/0 (socket says 172.21.15.3:52486) 2023-09-10T01:05:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 -- 172.21.15.3:0/1029474406 learned_addr learned my addr 172.21.15.3:0/1029474406 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe856d9d700 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81bce40 0x561db81bf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 -- 172.21.15.3:0/1029474406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81bce40 msgr2=0x561db81bf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81bce40 0x561db81bf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 -- 172.21.15.3:0/1029474406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b2710 msgr2=0x561db81b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b2710 0x561db81b2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 -- 172.21.15.3:0/1029474406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561db7f30de0 con 0x561db81b3030 2023-09-10T01:05:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe855d9b700 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b3030 0x561db81bc900 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe84c000c00 tx=0x7fe84c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=944362b6d7009186 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe856d9d700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe8400a0830 0x7fe8400a2cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe850009ec0 tx=0x7fe850009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe844ff9700 1 -- 172.21.15.3:0/1029474406 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c00f720 con 0x561db81b3030 2023-09-10T01:05:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe844ff9700 1 -- 172.21.15.3:0/1029474406 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe84c01d040 con 0x561db81b3030 2023-09-10T01:05:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.146+0000 7fe844ff9700 1 -- 172.21.15.3:0/1029474406 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c005560 con 0x561db81b3030 2023-09-10T01:05:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 -- 172.21.15.3:0/1029474406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b3030 msgr2=0x561db81bc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b3030 0x561db81bc900 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe84c000c00 tx=0x7fe84c005dc0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 -- 172.21.15.3:0/1029474406 shutdown_connections 2023-09-10T01:05:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81bce40 0x561db81bf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b2710 0x561db81b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1029474406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b3030 0x561db81bc900 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 -- 172.21.15.3:0/1029474406 >> 172.21.15.3:0/1029474406 conn(0x561db8127090 msgr2=0x561db81adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 -- 172.21.15.3:0/1029474406 shutdown_connections 2023-09-10T01:05:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.147+0000 7fe87055ab80 1 -- 172.21.15.3:0/1029474406 wait complete. 2023-09-10T01:05:14.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.148+0000 7fe87055ab80 1 -- start start 2023-09-10T01:05:14.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.148+0000 7fe87055ab80 1 Processor -- start 2023-09-10T01:05:14.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.148+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81bc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81b9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe87055ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db80a2b70 con 0x561db81b3030 2023-09-10T01:05:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe87055ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db80a2870 con 0x561db81bce40 2023-09-10T01:05:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe87055ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db7f86920 con 0x561db81b2710 2023-09-10T01:05:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe87055ab80 10 client.0 _ll_register_callbacks cb 0x561db7feca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81bc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe856d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81bc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54666/0 (socket says 172.21.15.3:54666) 2023-09-10T01:05:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 -- 172.21.15.3:0/4231304030 learned_addr learned my addr 172.21.15.3:0/4231304030 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe856d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81ba220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52496/0 (socket says 172.21.15.3:52496) 2023-09-10T01:05:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe855d9b700 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81b9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 -- 172.21.15.3:0/4231304030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 msgr2=0x561db81ba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81ba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 -- 172.21.15.3:0/4231304030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 msgr2=0x561db81b9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81b9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 -- 172.21.15.3:0/4231304030 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85810ddb0 con 0x561db81b2710 2023-09-10T01:05:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe85659c700 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81bc8b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe84800cfd0 tx=0x7fe848098e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1960730eb0c24b05 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.149+0000 7fe83dffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe83dffb700 1 -- 172.21.15.3:0/4231304030 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8480caea0 con 0x561db81b2710 2023-09-10T01:05:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe83dffb700 1 -- 172.21.15.3:0/4231304030 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe8480994a0 con 0x561db81b2710 2023-09-10T01:05:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561db805c4f0 con 0x561db81b2710 2023-09-10T01:05:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe83dffb700 1 -- 172.21.15.3:0/4231304030 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe84809d2d0 con 0x561db81b2710 2023-09-10T01:05:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 10 client.26152 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:14.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561db7f19c60 con 0x561db81b2710 2023-09-10T01:05:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe83dffb700 1 -- 172.21.15.3:0/4231304030 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe84807e080 con 0x561db81b2710 2023-09-10T01:05:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe83dffb700 1 -- 172.21.15.3:0/4231304030 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe848099700 con 0x561db81b2710 2023-09-10T01:05:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 10 client.26152 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 10 client.26152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:14.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 20 client.26152 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db81cc9c0 0x561db81cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 4 client.26152 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.150+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561db81198a0 con 0x561db81cc9c0 2023-09-10T01:05:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.151+0000 7fe855d9b700 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db81cc9c0 0x561db81cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.151+0000 7fe855d9b700 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db81cc9c0 0x561db81cede0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55cf960007c4ba9e server_cookie=ff30a2a0274afbcb in_seq=0 out_seq=0 2023-09-10T01:05:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.151+0000 7fe83dffb700 10 client.26152 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe83dffb700 1 -- 172.21.15.3:0/4231304030 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561db81198a0 con 0x561db81cc9c0 2023-09-10T01:05:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe83dffb700 10 client.26152 handle_command_reply: tid=0 2023-09-10T01:05:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 client.26152 shutdown 2023-09-10T01:05:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 msgr2=0x561db81bc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81bc8b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe84800cfd0 tx=0x7fe848098e70 comp rx=0 tx=0).stop 2023-09-10T01:05:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 shutdown_connections 2023-09-10T01:05:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db81cc9c0 0x561db81cede0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81bc8b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81ba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe83dffb700 0 client.26152 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 --2- 172.21.15.3:0/4231304030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81b9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 >> 172.21.15.3:0/4231304030 conn(0x561db8127090 msgr2=0x561db81ad780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 shutdown_connections 2023-09-10T01:05:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 1 -- 172.21.15.3:0/4231304030 wait complete. 2023-09-10T01:05:14.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.152+0000 7fe87055ab80 20 client.26152 trim_cache size 0 max 16384 2023-09-10T01:05:14.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.170+0000 7fe87055ab80 1 Processor -- start 2023-09-10T01:05:14.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe87055ab80 1 -- start start 2023-09-10T01:05:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81c3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81adc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe87055ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db81c3f10 con 0x561db81b3030 2023-09-10T01:05:14.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe87055ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db81a5400 con 0x561db81bce40 2023-09-10T01:05:14.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe87055ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db81c9fe0 con 0x561db81b2710 2023-09-10T01:05:14.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe85659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe85659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81c3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54680/0 (socket says 172.21.15.3:54680) 2023-09-10T01:05:14.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe85659c700 1 -- 172.21.15.3:0/1345348203 learned_addr learned my addr 172.21.15.3:0/1345348203 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe855d9b700 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.171+0000 7fe856d9d700 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe85659c700 1 -- 172.21.15.3:0/1345348203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 msgr2=0x561db81adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe85659c700 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe85659c700 1 -- 172.21.15.3:0/1345348203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 msgr2=0x561db81ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe85659c700 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe85659c700 1 -- 172.21.15.3:0/1345348203 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561db7f57250 con 0x561db81b2710 2023-09-10T01:05:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe855d9b700 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:14.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe85659c700 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81c3990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe84800f010 tx=0x7fe848097a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f408c2b90636a6e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.172+0000 7fe8457fa700 1 -- 172.21.15.3:0/1345348203 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84800cdf0 con 0x561db81b2710 2023-09-10T01:05:14.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe8457fa700 1 -- 172.21.15.3:0/1345348203 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8480c9e90 con 0x561db81b2710 2023-09-10T01:05:14.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe8457fa700 1 -- 172.21.15.3:0/1345348203 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8480a1510 con 0x561db81b2710 2023-09-10T01:05:14.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 -- 172.21.15.3:0/1345348203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 msgr2=0x561db81c3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81c3990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe84800f010 tx=0x7fe848097a50 comp rx=0 tx=0).stop 2023-09-10T01:05:14.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 -- 172.21.15.3:0/1345348203 shutdown_connections 2023-09-10T01:05:14.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b2710 0x561db81c3990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81bce40 0x561db81adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 --2- 172.21.15.3:0/1345348203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81b3030 0x561db81ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 -- 172.21.15.3:0/1345348203 >> 172.21.15.3:0/1345348203 conn(0x561db8127090 msgr2=0x561db81c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 -- 172.21.15.3:0/1345348203 shutdown_connections 2023-09-10T01:05:14.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.173+0000 7fe87055ab80 1 -- 172.21.15.3:0/1345348203 wait complete. 2023-09-10T01:05:14.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 -- start start 2023-09-10T01:05:14.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 Processor -- start 2023-09-10T01:05:14.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b2710 0x561db81c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b3030 0x561db81c1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81bce40 0x561db81c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db81c3f60 con 0x561db81bce40 2023-09-10T01:05:14.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db81c2400 con 0x561db81b2710 2023-09-10T01:05:14.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.174+0000 7fe87055ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561db81c2570 con 0x561db81b3030 2023-09-10T01:05:14.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe87055ab80 10 client.0 _ll_register_callbacks cb 0x7fe8480990b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:14.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b2710 0x561db81c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe855d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b3030 0x561db81c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b2710 0x561db81c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52520/0 (socket says 172.21.15.3:52520) 2023-09-10T01:05:14.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 -- 172.21.15.3:0/3441806136 learned_addr learned my addr 172.21.15.3:0/3441806136 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 -- 172.21.15.3:0/3441806136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b3030 msgr2=0x561db81c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b3030 0x561db81c1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 -- 172.21.15.3:0/3441806136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81bce40 msgr2=0x561db81c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81bce40 0x561db81c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 -- 172.21.15.3:0/3441806136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561db7f30de0 con 0x561db81b2710 2023-09-10T01:05:14.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe855d9b700 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b3030 0x561db81c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:14.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe85659c700 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b2710 0x561db81c3690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe8480a0fe0 tx=0x7fe848004ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d2e970746e1b15 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.175+0000 7fe83dffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:14.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe83dffb700 1 -- 172.21.15.3:0/3441806136 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8480994a0 con 0x561db81b2710 2023-09-10T01:05:14.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe83dffb700 1 -- 172.21.15.3:0/3441806136 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe8480a1510 con 0x561db81b2710 2023-09-10T01:05:14.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe84c01d040 con 0x561db81b2710 2023-09-10T01:05:14.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe83dffb700 1 -- 172.21.15.3:0/3441806136 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe84809d2d0 con 0x561db81b2710 2023-09-10T01:05:14.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe87055ab80 10 client.26237 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:14.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561db7f19c60 con 0x561db81b2710 2023-09-10T01:05:14.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe83dffb700 1 -- 172.21.15.3:0/3441806136 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe84c01d040 con 0x561db81b2710 2023-09-10T01:05:14.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.176+0000 7fe83dffb700 1 -- 172.21.15.3:0/3441806136 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe848098400 con 0x561db81b2710 2023-09-10T01:05:14.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe87055ab80 10 client.26237 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:14.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe87055ab80 10 client.26237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:14.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe87055ab80 20 client.26237 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:14.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe87055ab80 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db8221640 0x561db8223a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe87055ab80 4 client.26237 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:05:14.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561db80fd920 con 0x561db8221640 2023-09-10T01:05:14.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe855d9b700 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db8221640 0x561db8223a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe855d9b700 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db8221640 0x561db8223a30 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c0f2114da4dad81 server_cookie=4c828d466c093ef5 in_seq=0 out_seq=0 2023-09-10T01:05:14.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe83dffb700 10 client.26237 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe83dffb700 1 -- 172.21.15.3:0/3441806136 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561db80fd920 con 0x561db8221640 2023-09-10T01:05:14.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe83dffb700 10 client.26237 handle_command_reply: tid=0 2023-09-10T01:05:14.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.177+0000 7fe87055ab80 1 client.26237 shutdown 2023-09-10T01:05:14.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b2710 msgr2=0x561db81c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b2710 0x561db81c3690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe8480a0fe0 tx=0x7fe848004ea0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 shutdown_connections 2023-09-10T01:05:14.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561db8221640 0x561db8223a30 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561db81bce40 0x561db81c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561db81b2710 0x561db81c3690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe83dffb700 0 client.26237 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 --2- 172.21.15.3:0/3441806136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561db81b3030 0x561db81c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 >> 172.21.15.3:0/3441806136 conn(0x561db8127090 msgr2=0x561db81ad670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 shutdown_connections 2023-09-10T01:05:14.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 1 -- 172.21.15.3:0/3441806136 wait complete. 2023-09-10T01:05:14.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.178+0000 7fe87055ab80 20 client.26237 trim_cache size 0 max 16384 2023-09-10T01:05:14.193 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:05:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:05:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:05:14.193 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:05:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe8400a0830 msgr2=0x7fe8400a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe8400a0830 0x7fe8400a2cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe850009ec0 tx=0x7fe850009920 comp rx=0 tx=0).stop 2023-09-10T01:05:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581018a0 msgr2=0x7fe85819e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581018a0 0x7fe85819e7d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe84800e9c0 tx=0x7fe84800ecd0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 shutdown_connections 2023-09-10T01:05:14.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe8400a0830 0x7fe8400a2cf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe85810ac70 0x7fe858198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8581018a0 0x7fe85819e7d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 --2- 172.21.15.3:0/1453969845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8581021c0 0x7fe85819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 >> 172.21.15.3:0/1453969845 conn(0x7fe858076710 msgr2=0x7fe8580fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 shutdown_connections 2023-09-10T01:05:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.179+0000 7fe85cbff700 1 -- 172.21.15.3:0/1453969845 wait complete. 2023-09-10T01:05:14.198 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:05:14.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:05:14.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:14 smithi003 ceph-mon[97654]: pgmap v63: 129 pgs: 129 active+clean; 39 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 260 op/s 2023-09-10T01:05:14.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:14 smithi136 ceph-mon[133708]: pgmap v63: 129 pgs: 129 active+clean; 39 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 260 op/s 2023-09-10T01:05:14.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 -- 172.21.15.3:0/2583227485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580732c0 msgr2=0x7f12581110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 --2- 172.21.15.3:0/2583227485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580732c0 0x7f12581110f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f12480098f0 tx=0x7f1248009c00 comp rx=0 tx=0).stop 2023-09-10T01:05:14.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 -- 172.21.15.3:0/2583227485 shutdown_connections 2023-09-10T01:05:14.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 --2- 172.21.15.3:0/2583227485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1258111630 0x7f1258113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 --2- 172.21.15.3:0/2583227485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12580729a0 0x7f1258072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 --2- 172.21.15.3:0/2583227485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580732c0 0x7f12581110f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 -- 172.21.15.3:0/2583227485 >> 172.21.15.3:0/2583227485 conn(0x7f12580fc110 msgr2=0x7f12580fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 -- 172.21.15.3:0/2583227485 shutdown_connections 2023-09-10T01:05:14.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.431+0000 7f125fbff700 1 -- 172.21.15.3:0/2583227485 wait complete. 2023-09-10T01:05:14.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.432+0000 7f125fbff700 1 Processor -- start 2023-09-10T01:05:14.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.432+0000 7f125fbff700 1 -- start start 2023-09-10T01:05:14.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125fbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580729a0 0x7f125819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125fbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12580732c0 0x7f125819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125fbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1258111630 0x7f12581a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125fbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12581a1d80 con 0x7f1258111630 2023-09-10T01:05:14.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125fbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12581a1ef0 con 0x7f12580732c0 2023-09-10T01:05:14.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125fbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12581a20b0 con 0x7f12580729a0 2023-09-10T01:05:14.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125d99b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580729a0 0x7f125819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125e19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1258111630 0x7f12581a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125d99b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580729a0 0x7f125819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54712/0 (socket says 172.21.15.3:54712) 2023-09-10T01:05:14.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125d99b700 1 -- 172.21.15.3:0/3854371988 learned_addr learned my addr 172.21.15.3:0/3854371988 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125d19a700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12580732c0 0x7f125819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125e19c700 1 -- 172.21.15.3:0/3854371988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580729a0 msgr2=0x7f125819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125e19c700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580729a0 0x7f125819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125e19c700 1 -- 172.21.15.3:0/3854371988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12580732c0 msgr2=0x7f125819d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125e19c700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12580732c0 0x7f125819d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.433+0000 7f125e19c700 1 -- 172.21.15.3:0/3854371988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1248009520 con 0x7f1258111630 2023-09-10T01:05:14.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.434+0000 7f125e19c700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1258111630 0x7f12581a1740 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f125000ef90 tx=0x7f125000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.434+0000 7f124effd700 1 -- 172.21.15.3:0/3854371988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250018070 con 0x7f1258111630 2023-09-10T01:05:14.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.434+0000 7f124effd700 1 -- 172.21.15.3:0/3854371988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f125000f040 con 0x7f1258111630 2023-09-10T01:05:14.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.434+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12581a2330 con 0x7f1258111630 2023-09-10T01:05:14.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.434+0000 7f124effd700 1 -- 172.21.15.3:0/3854371988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125000cc30 con 0x7f1258111630 2023-09-10T01:05:14.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.434+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12581a9bd0 con 0x7f1258111630 2023-09-10T01:05:14.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.435+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1258103b10 con 0x7f1258111630 2023-09-10T01:05:14.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.438+0000 7f124effd700 1 -- 172.21.15.3:0/3854371988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f1250003a00 con 0x7f1258111630 2023-09-10T01:05:14.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.440+0000 7f124effd700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12440a07b0 0x7f12440a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.440+0000 7f124effd700 1 -- 172.21.15.3:0/3854371988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f12500cbe50 con 0x7f1258111630 2023-09-10T01:05:14.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.440+0000 7f125d99b700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12440a07b0 0x7f12440a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.441+0000 7f125d99b700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12440a07b0 0x7f12440a2c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1254005950 tx=0x7f1254009310 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.445+0000 7f124effd700 1 -- 172.21.15.3:0/3854371988 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f12500952e0 con 0x7f1258111630 2023-09-10T01:05:14.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:14 smithi092 ceph-mon[134067]: pgmap v63: 129 pgs: 129 active+clean; 39 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 260 op/s 2023-09-10T01:05:14.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.729+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1258116b90 con 0x7f1258111630 2023-09-10T01:05:14.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.730+0000 7f124effd700 1 -- 172.21.15.3:0/3854371988 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11295 (secure 0 0 0) 0x7f1250098900 con 0x7f1258111630 2023-09-10T01:05:14.731 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:05:14.731 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:04:38.815186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:05:14.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.732+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12440a07b0 msgr2=0x7f12440a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.732+0000 7f125fbff700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12440a07b0 0x7f12440a2c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1254005950 tx=0x7f1254009310 comp rx=0 tx=0).stop 2023-09-10T01:05:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1258111630 msgr2=0x7f12581a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1258111630 0x7f12581a1740 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f125000ef90 tx=0x7f125000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 shutdown_connections 2023-09-10T01:05:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12440a07b0 0x7f12440a2c70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580729a0 0x7f125819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1258111630 0x7f12581a1740 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 --2- 172.21.15.3:0/3854371988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12580732c0 0x7f125819d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 >> 172.21.15.3:0/3854371988 conn(0x7f12580fc110 msgr2=0x7f12580fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 shutdown_connections 2023-09-10T01:05:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:14.733+0000 7f125fbff700 1 -- 172.21.15.3:0/3854371988 wait complete. 2023-09-10T01:05:14.736 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2023-09-10T01:05:14.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 103432 10.28 MB/sec warmup 56 sec latency 20.541 ms 2023-09-10T01:05:14.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:05:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 -- 172.21.15.3:0/294670136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf40729a0 msgr2=0x7efbf4072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 --2- 172.21.15.3:0/294670136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf40729a0 0x7efbf4072d80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efbe4009990 tx=0x7efbe4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 -- 172.21.15.3:0/294670136 shutdown_connections 2023-09-10T01:05:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 --2- 172.21.15.3:0/294670136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf41115b0 0x7efbf41139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 --2- 172.21.15.3:0/294670136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf40729a0 0x7efbf4072d80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 --2- 172.21.15.3:0/294670136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 0x7efbf4111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 -- 172.21.15.3:0/294670136 >> 172.21.15.3:0/294670136 conn(0x7efbf40fc0b0 msgr2=0x7efbf40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 -- 172.21.15.3:0/294670136 shutdown_connections 2023-09-10T01:05:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.766+0000 7efbf8bff700 1 -- 172.21.15.3:0/294670136 wait complete. 2023-09-10T01:05:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.767+0000 7efbf8bff700 1 Processor -- start 2023-09-10T01:05:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.767+0000 7efbf8bff700 1 -- start start 2023-09-10T01:05:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf8bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf40729a0 0x7efbf419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf8bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 0x7efbf419d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf8bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf41115b0 0x7efbf41a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf8bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf41a1d00 con 0x7efbf41115b0 2023-09-10T01:05:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf8bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf41a1e70 con 0x7efbf40732c0 2023-09-10T01:05:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf8bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf41a2030 con 0x7efbf40729a0 2023-09-10T01:05:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 0x7efbf419d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf40729a0 0x7efbf419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 0x7efbf419d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57360/0 (socket says 172.21.15.3:57360) 2023-09-10T01:05:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 -- 172.21.15.3:0/2787281179 learned_addr learned my addr 172.21.15.3:0/2787281179 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf41115b0 0x7efbf41a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 -- 172.21.15.3:0/2787281179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf40729a0 msgr2=0x7efbf419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf40729a0 0x7efbf419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 -- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf41115b0 msgr2=0x7efbf41a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf41115b0 0x7efbf41a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf1d9b700 1 -- 172.21.15.3:0/2787281179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe40095f0 con 0x7efbf40732c0 2023-09-10T01:05:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.768+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf41115b0 0x7efbf41a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.769+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 0x7efbf419d260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efbe800e9a0 tx=0x7efbe800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.769+0000 7efbe37fe700 1 -- 172.21.15.3:0/2787281179 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8009790 con 0x7efbf40732c0 2023-09-10T01:05:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.769+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf41a2310 con 0x7efbf40732c0 2023-09-10T01:05:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.769+0000 7efbe37fe700 1 -- 172.21.15.3:0/2787281179 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbe800cbd0 con 0x7efbf40732c0 2023-09-10T01:05:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.769+0000 7efbe37fe700 1 -- 172.21.15.3:0/2787281179 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8018680 con 0x7efbf40732c0 2023-09-10T01:05:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.769+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf41a9b50 con 0x7efbf40732c0 2023-09-10T01:05:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.774+0000 7efbe37fe700 1 -- 172.21.15.3:0/2787281179 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7efbe800cd40 con 0x7efbf40732c0 2023-09-10T01:05:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.775+0000 7efc0c55fb80 1 Processor -- start 2023-09-10T01:05:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.775+0000 7efc0c55fb80 1 -- start start 2023-09-10T01:05:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a81a0 0x55a1687a8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2300 0x55a1687a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1686fb6d0 con 0x55a1687a2cc0 2023-09-10T01:05:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1686f8ff0 con 0x55a1687a2300 2023-09-10T01:05:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1686f48e0 con 0x55a1687a81a0 2023-09-10T01:05:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a81a0 0x55a1687a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2300 0x55a1687a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a81a0 0x55a1687a8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57150/0 (socket says 172.21.15.3:57150) 2023-09-10T01:05:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1452346688 learned_addr learned my addr 172.21.15.3:0/1452346688 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1452346688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2300 msgr2=0x55a1687a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2300 0x55a1687a2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1452346688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 msgr2=0x55a1687aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.776+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1452346688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a168520d80 con 0x55a1687a81a0 2023-09-10T01:05:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.777+0000 7efbe37fe700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efbdc0a07b0 0x7efbdc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.777+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a81a0 0x55a1687a8580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efbec0093f0 tx=0x7efbec009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=575f2daa75e2d301 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.777+0000 7efbe0ff9700 1 -- 172.21.15.3:0/1452346688 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec00e030 con 0x55a1687a81a0 2023-09-10T01:05:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.777+0000 7efbe37fe700 1 -- 172.21.15.3:0/2787281179 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efbe8014070 con 0x7efbf40732c0 2023-09-10T01:05:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.777+0000 7efbe0ff9700 1 -- 172.21.15.3:0/1452346688 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbec00f040 con 0x55a1687a81a0 2023-09-10T01:05:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.777+0000 7efbe0ff9700 1 -- 172.21.15.3:0/1452346688 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec014a40 con 0x55a1687a81a0 2023-09-10T01:05:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.777+0000 7efbf259c700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efbdc0a07b0 0x7efbdc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1452346688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a81a0 msgr2=0x55a1687a8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a81a0 0x55a1687a8580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efbec0093f0 tx=0x7efbec009920 comp rx=0 tx=0).stop 2023-09-10T01:05:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efbf259c700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efbdc0a07b0 0x7efbdc0a2c70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efbe400b6f0 tx=0x7efbe4005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1452346688 shutdown_connections 2023-09-10T01:05:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a81a0 0x55a1687a8580 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1452346688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2300 0x55a1687a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1452346688 >> 172.21.15.3:0/1452346688 conn(0x55a168717030 msgr2=0x55a16879d560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1452346688 shutdown_connections 2023-09-10T01:05:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.778+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1452346688 wait complete. 2023-09-10T01:05:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.779+0000 7efc0c55fb80 1 -- start start 2023-09-10T01:05:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.779+0000 7efc0c55fb80 1 Processor -- start 2023-09-10T01:05:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687aed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687ac010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a81a0 0x55a1687ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a168692b10 con 0x55a1687a2cc0 2023-09-10T01:05:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a168692810 con 0x55a1687a81a0 2023-09-10T01:05:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1685768c0 con 0x55a1687a2300 2023-09-10T01:05:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efc0c55fb80 10 client.0 _ll_register_callbacks cb 0x55a1685dca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efbf259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efbf259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687aed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57154/0 (socket says 172.21.15.3:57154) 2023-09-10T01:05:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efbf259c700 1 -- 172.21.15.3:0/1318380494 learned_addr learned my addr 172.21.15.3:0/1318380494 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.780+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a81a0 0x55a1687ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbf1d9b700 1 -- 172.21.15.3:0/1318380494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 msgr2=0x55a1687aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbf1d9b700 1 -- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a81a0 msgr2=0x55a1687ac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a81a0 0x55a1687ac5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbf1d9b700 1 -- 172.21.15.3:0/1318380494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbec0090d0 con 0x55a1687a2cc0 2023-09-10T01:05:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbf259c700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687ac010 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efbe80c99c0 tx=0x7efbe80c9f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba424301bf7b5f0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbd9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:05:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbd9ffb700 1 -- 172.21.15.3:0/1318380494 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe80cc430 con 0x55a1687a2cc0 2023-09-10T01:05:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a16864c490 con 0x55a1687a2cc0 2023-09-10T01:05:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbd9ffb700 1 -- 172.21.15.3:0/1318380494 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efbe80cc070 con 0x55a1687a2cc0 2023-09-10T01:05:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efbd9ffb700 1 -- 172.21.15.3:0/1318380494 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efbe8099260 con 0x55a1687a2cc0 2023-09-10T01:05:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efc0c55fb80 10 client.16197 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.781+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a168509c00 con 0x55a1687a2cc0 2023-09-10T01:05:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efbd9ffb700 1 -- 172.21.15.3:0/1318380494 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a16864c490 con 0x55a1687a2cc0 2023-09-10T01:05:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efbd9ffb700 1 -- 172.21.15.3:0/1318380494 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7efbe809d150 con 0x55a1687a2cc0 2023-09-10T01:05:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efc0c55fb80 10 client.16197 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efc0c55fb80 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efc0c55fb80 20 client.16197 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687bc3f0 0x55a1687be810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efc0c55fb80 4 client.16197 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a168709840 con 0x55a1687bc3f0 2023-09-10T01:05:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.782+0000 7efbf259c700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687bc3f0 0x55a1687be810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.783+0000 7efbf259c700 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687bc3f0 0x55a1687be810 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a34c0f272158cf server_cookie=3a39ac60d7e23610 in_seq=0 out_seq=0 2023-09-10T01:05:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.783+0000 7efbd9ffb700 10 client.16197 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.784+0000 7efbd9ffb700 1 -- 172.21.15.3:0/1318380494 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a168709840 con 0x55a1687bc3f0 2023-09-10T01:05:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.784+0000 7efbd9ffb700 10 client.16197 handle_command_reply: tid=0 2023-09-10T01:05:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.784+0000 7efc0c55fb80 1 client.16197 shutdown 2023-09-10T01:05:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.784+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 msgr2=0x55a1687ac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.784+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687ac010 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efbe80c99c0 tx=0x7efbe80c9f20 comp rx=0 tx=0).stop 2023-09-10T01:05:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 shutdown_connections 2023-09-10T01:05:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687bc3f0 0x55a1687be810 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a81a0 0x55a1687ac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1318380494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687ac010 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 >> 172.21.15.3:0/1318380494 conn(0x55a168717030 msgr2=0x55a16879c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efbd9ffb700 0 client.16197 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 shutdown_connections 2023-09-10T01:05:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1318380494 wait complete. 2023-09-10T01:05:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.785+0000 7efc0c55fb80 20 client.16197 trim_cache size 0 max 16384 2023-09-10T01:05:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.801+0000 7efc0c55fb80 1 Processor -- start 2023-09-10T01:05:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efc0c55fb80 1 -- start start 2023-09-10T01:05:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a168717030 0x55a1687a7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2300 0x55a1687ae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2cc0 0x55a16879cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a16879a000 con 0x55a1687a2300 2023-09-10T01:05:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1687a8910 con 0x55a1687a2cc0 2023-09-10T01:05:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1687aa570 con 0x55a168717030 2023-09-10T01:05:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efbf2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2cc0 0x55a16879cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efbf1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2300 0x55a1687ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efbf2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2cc0 0x55a16879cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57394/0 (socket says 172.21.15.3:57394) 2023-09-10T01:05:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1359053339 learned_addr learned my addr 172.21.15.3:0/1359053339 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.802+0000 7efbf259c700 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a168717030 0x55a1687a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1359053339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a168717030 msgr2=0x55a1687a7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a168717030 0x55a1687a7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1359053339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2300 msgr2=0x55a1687ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2300 0x55a1687ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbf2d9d700 1 -- 172.21.15.3:0/1359053339 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1685471f0 con 0x55a1687a2cc0 2023-09-10T01:05:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbf259c700 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a168717030 0x55a1687a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2cc0 0x55a16879cf20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efbec007ac0 tx=0x7efbec0144d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=606311460e7b7b59 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbe17fa700 1 -- 172.21.15.3:0/1359053339 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec00e040 con 0x55a1687a2cc0 2023-09-10T01:05:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbe17fa700 1 -- 172.21.15.3:0/1359053339 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbec004cf0 con 0x55a1687a2cc0 2023-09-10T01:05:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.803+0000 7efbe17fa700 1 -- 172.21.15.3:0/1359053339 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbec0157b0 con 0x55a1687a2cc0 2023-09-10T01:05:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1359053339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2cc0 msgr2=0x55a16879cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2cc0 0x55a16879cf20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efbec007ac0 tx=0x7efbec0144d0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1359053339 shutdown_connections 2023-09-10T01:05:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a168717030 0x55a1687a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a1687a2cc0 0x55a16879cf20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/1359053339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2300 0x55a1687ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1359053339 >> 172.21.15.3:0/1359053339 conn(0x55a1687a67d0 msgr2=0x55a1687a1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1359053339 shutdown_connections 2023-09-10T01:05:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.804+0000 7efc0c55fb80 1 -- 172.21.15.3:0/1359053339 wait complete. 2023-09-10T01:05:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 -- start start 2023-09-10T01:05:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 Processor -- start 2023-09-10T01:05:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a168717030 0x55a1687b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687b1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1687b1bf0 con 0x55a1687a2cc0 2023-09-10T01:05:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1687b1d60 con 0x55a168717030 2023-09-10T01:05:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.805+0000 7efc0c55fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1687b1f20 con 0x55a1687a2300 2023-09-10T01:05:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efc0c55fb80 10 client.0 _ll_register_callbacks cb 0x7efbe8010aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a168717030 0x55a1687b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a168717030 0x55a1687b2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57404/0 (socket says 172.21.15.3:57404) 2023-09-10T01:05:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 -- 172.21.15.3:0/3926705536 learned_addr learned my addr 172.21.15.3:0/3926705536 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 -- 172.21.15.3:0/3926705536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 msgr2=0x55a1687b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 -- 172.21.15.3:0/3926705536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 msgr2=0x55a1687b1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687b1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 -- 172.21.15.3:0/3926705536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a168520d80 con 0x55a168717030 2023-09-10T01:05:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf2d9d700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbf259c700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a168717030 0x55a1687b2ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55a1687bf120 tx=0x7efbe4009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8ab61f434ad8adb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbd9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.806+0000 7efbd9ffb700 1 -- 172.21.15.3:0/3926705536 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe40342d0 con 0x55a168717030 2023-09-10T01:05:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efbec00f040 con 0x55a168717030 2023-09-10T01:05:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efbd9ffb700 1 -- 172.21.15.3:0/3926705536 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efbe4034430 con 0x55a168717030 2023-09-10T01:05:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efbd9ffb700 1 -- 172.21.15.3:0/3926705536 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efbe4004500 con 0x55a168717030 2023-09-10T01:05:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 10 client.26261 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a16851f250 con 0x55a168717030 2023-09-10T01:05:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efbd9ffb700 1 -- 172.21.15.3:0/3926705536 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbec00f040 con 0x55a168717030 2023-09-10T01:05:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efbd9ffb700 1 -- 172.21.15.3:0/3926705536 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7efbe400f730 con 0x55a168717030 2023-09-10T01:05:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 10 client.26261 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 10 client.26261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 20 client.26261 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687a81a0 0x55a1688133d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 4 client.26261 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:05:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.807+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a1686ed8c0 con 0x55a1687a81a0 2023-09-10T01:05:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.808+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687a81a0 0x55a1688133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.808+0000 7efbf1d9b700 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687a81a0 0x55a1688133d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cc2a9dc2d31b0dd server_cookie=80c1c56c600ba4d5 in_seq=0 out_seq=0 2023-09-10T01:05:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.808+0000 7efbd9ffb700 10 client.26261 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.903+0000 7efbd9ffb700 1 -- 172.21.15.3:0/3926705536 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a1686ed8c0 con 0x55a1687a81a0 2023-09-10T01:05:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.903+0000 7efbd9ffb700 10 client.26261 handle_command_reply: tid=0 2023-09-10T01:05:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.903+0000 7efc0c55fb80 1 client.26261 shutdown 2023-09-10T01:05:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.903+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a168717030 msgr2=0x55a1687b2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.903+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a168717030 0x55a1687b2ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x55a1687bf120 tx=0x7efbe4009310 comp rx=0 tx=0).stop 2023-09-10T01:05:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.906+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 shutdown_connections 2023-09-10T01:05:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.906+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a1687a81a0 0x55a1688133d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.906+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a1687a2cc0 0x55a1687b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.906+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a168717030 0x55a1687b2ef0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.906+0000 7efc0c55fb80 1 --2- 172.21.15.3:0/3926705536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a1687a2300 0x55a1687b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.906+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 >> 172.21.15.3:0/3926705536 conn(0x55a1687a67d0 msgr2=0x55a1687b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.906+0000 7efbd9ffb700 0 client.26261 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.907+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 shutdown_connections 2023-09-10T01:05:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.907+0000 7efc0c55fb80 1 -- 172.21.15.3:0/3926705536 wait complete. 2023-09-10T01:05:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.907+0000 7efc0c55fb80 20 client.26261 trim_cache size 0 max 16384 2023-09-10T01:05:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:05:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:05:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:05:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:05:14.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efbdc0a07b0 msgr2=0x7efbdc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efbdc0a07b0 0x7efbdc0a2c70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efbe400b6f0 tx=0x7efbe4005fd0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 msgr2=0x7efbf419d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:14.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 0x7efbf419d260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7efbe800e9a0 tx=0x7efbe800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 shutdown_connections 2023-09-10T01:05:14.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efbdc0a07b0 0x7efbdc0a2c70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf40729a0 0x7efbf419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efbf41115b0 0x7efbf41a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 --2- 172.21.15.3:0/2787281179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efbf40732c0 0x7efbf419d260 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:14.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 >> 172.21.15.3:0/2787281179 conn(0x7efbf40fc0b0 msgr2=0x7efbf40fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:14.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 shutdown_connections 2023-09-10T01:05:14.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:14.908+0000 7efbf8bff700 1 -- 172.21.15.3:0/2787281179 wait complete. 2023-09-10T01:05:15.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.016+0000 7feb727ff700 1 -- 172.21.15.3:0/849131482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c10acd0 msgr2=0x7feb6c10d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.016+0000 7feb727ff700 1 --2- 172.21.15.3:0/849131482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c10acd0 0x7feb6c10d140 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7feb6400b0a0 tx=0x7feb6400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.017+0000 7feb727ff700 1 -- 172.21.15.3:0/849131482 shutdown_connections 2023-09-10T01:05:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.017+0000 7feb727ff700 1 --2- 172.21.15.3:0/849131482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c101780 0x7feb6c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.017+0000 7feb727ff700 1 --2- 172.21.15.3:0/849131482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c10acd0 0x7feb6c10d140 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.017+0000 7feb727ff700 1 --2- 172.21.15.3:0/849131482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb6c102130 0x7feb6c10a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.017+0000 7feb727ff700 1 -- 172.21.15.3:0/849131482 >> 172.21.15.3:0/849131482 conn(0x7feb6c0fb010 msgr2=0x7feb6c0fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.017+0000 7feb727ff700 1 -- 172.21.15.3:0/849131482 shutdown_connections 2023-09-10T01:05:15.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.017+0000 7feb727ff700 1 -- 172.21.15.3:0/849131482 wait complete. 2023-09-10T01:05:15.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 Processor -- start 2023-09-10T01:05:15.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 -- start start 2023-09-10T01:05:15.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c101780 0x7feb6c06f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c102130 0x7feb6c06fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb6c10acd0 0x7feb6c06a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c06b030 con 0x7feb6c10acd0 2023-09-10T01:05:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c06b1a0 con 0x7feb6c101780 2023-09-10T01:05:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.019+0000 7feb727ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c06b360 con 0x7feb6c102130 2023-09-10T01:05:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c101780 0x7feb6c06f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c101780 0x7feb6c06f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57428/0 (socket says 172.21.15.3:57428) 2023-09-10T01:05:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 -- 172.21.15.3:0/1141055596 learned_addr learned my addr 172.21.15.3:0/1141055596 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb70d9c700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb6c10acd0 0x7feb6c06a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 -- 172.21.15.3:0/1141055596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c102130 msgr2=0x7feb6c06fd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:05:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c102130 0x7feb6c06fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 -- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb6c10acd0 msgr2=0x7feb6c06a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb6c10acd0 0x7feb6c06a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.020+0000 7feb6bfff700 1 -- 172.21.15.3:0/1141055596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb64009d00 con 0x7feb6c101780 2023-09-10T01:05:15.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.021+0000 7feb70d9c700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb6c10acd0 0x7feb6c06a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:15.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.021+0000 7feb6bfff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c101780 0x7feb6c06f800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00e9a0 tx=0x7feb5c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.021+0000 7feb697fa700 1 -- 172.21.15.3:0/1141055596 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c009790 con 0x7feb6c101780 2023-09-10T01:05:15.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.021+0000 7feb697fa700 1 -- 172.21.15.3:0/1141055596 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb5c00cbd0 con 0x7feb6c101780 2023-09-10T01:05:15.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.021+0000 7feb697fa700 1 -- 172.21.15.3:0/1141055596 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c018680 con 0x7feb6c101780 2023-09-10T01:05:15.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.021+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6c06b640 con 0x7feb6c101780 2023-09-10T01:05:15.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.022+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6c06e740 con 0x7feb6c101780 2023-09-10T01:05:15.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.027+0000 7feb8612eb80 1 Processor -- start 2023-09-10T01:05:15.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.027+0000 7feb697fa700 1 -- 172.21.15.3:0/1141055596 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7feb5c00cd40 con 0x7feb6c101780 2023-09-10T01:05:15.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.027+0000 7feb8612eb80 1 -- start start 2023-09-10T01:05:15.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.028+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ecb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.028+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ecc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.028+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6ee08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.028+0000 7feb8612eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6e24730 con 0x5568f6ede4e0 2023-09-10T01:05:15.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.028+0000 7feb8612eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6e22050 con 0x5568f6ecbc50 2023-09-10T01:05:15.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.028+0000 7feb8612eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6e1d940 con 0x5568f6ecd360 2023-09-10T01:05:15.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.028+0000 7feb6bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ecc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ecc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57444/0 (socket says 172.21.15.3:57444) 2023-09-10T01:05:15.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 -- 172.21.15.3:0/1193599514 learned_addr learned my addr 172.21.15.3:0/1193599514 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb70d9c700 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ecb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 -- 172.21.15.3:0/1193599514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 msgr2=0x5568f6ecb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ecb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 -- 172.21.15.3:0/1193599514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 msgr2=0x5568f6ee08d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:05:15.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6ee08d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 -- 172.21.15.3:0/1193599514 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568f6c49de0 con 0x5568f6ecbc50 2023-09-10T01:05:15.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb70d9c700 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ecb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:05:15.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.029+0000 7feb6bfff700 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ecc0d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb5c0c98f0 tx=0x7feb5c0d26f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3cfc089a996b977 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.030+0000 7feb697fa700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb540a0830 0x7feb540a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.030+0000 7feb697fa700 1 -- 172.21.15.3:0/1141055596 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7feb5c014070 con 0x7feb6c101780 2023-09-10T01:05:15.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.030+0000 7feb527fc700 1 -- 172.21.15.3:0/1193599514 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c0ccc80 con 0x5568f6ecbc50 2023-09-10T01:05:15.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.030+0000 7feb527fc700 1 -- 172.21.15.3:0/1193599514 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb5c0ccde0 con 0x5568f6ecbc50 2023-09-10T01:05:15.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.030+0000 7feb527fc700 1 -- 172.21.15.3:0/1193599514 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c0d8970 con 0x5568f6ecbc50 2023-09-10T01:05:15.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.031+0000 7feb70d9c700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb540a0830 0x7feb540a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.031+0000 7feb8612eb80 1 -- 172.21.15.3:0/1193599514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 msgr2=0x5568f6ecc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.031+0000 7feb8612eb80 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ecc0d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb5c0c98f0 tx=0x7feb5c0d26f0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb8612eb80 1 -- 172.21.15.3:0/1193599514 shutdown_connections 2023-09-10T01:05:15.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb8612eb80 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ecb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb8612eb80 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6ee08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb8612eb80 1 --2- 172.21.15.3:0/1193599514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ecc0d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb8612eb80 1 -- 172.21.15.3:0/1193599514 >> 172.21.15.3:0/1193599514 conn(0x5568f6e40090 msgr2=0x5568f6ec6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb8612eb80 1 -- 172.21.15.3:0/1193599514 shutdown_connections 2023-09-10T01:05:15.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb70d9c700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb540a0830 0x7feb540a2cf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb64009850 tx=0x7feb64009780 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.032+0000 7feb8612eb80 1 -- 172.21.15.3:0/1193599514 wait complete. 2023-09-10T01:05:15.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.033+0000 7feb8612eb80 1 -- start start 2023-09-10T01:05:15.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.033+0000 7feb8612eb80 1 Processor -- start 2023-09-10T01:05:15.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ed1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ece630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ede4e0 0x5568f6ececb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb8612eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6dbbb70 con 0x5568f6ecbc50 2023-09-10T01:05:15.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb8612eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6dbb870 con 0x5568f6ede4e0 2023-09-10T01:05:15.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb8612eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6c9f920 con 0x5568f6ecd360 2023-09-10T01:05:15.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb8612eb80 10 client.0 _ll_register_callbacks cb 0x5568f6d05a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:15.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb70d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ede4e0 0x5568f6ececb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb6b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ece630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb70d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ede4e0 0x5568f6ececb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57460/0 (socket says 172.21.15.3:57460) 2023-09-10T01:05:15.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.035+0000 7feb70d9c700 1 -- 172.21.15.3:0/3799608464 learned_addr learned my addr 172.21.15.3:0/3799608464 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb6bfff700 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ed1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb6b7fe700 1 -- 172.21.15.3:0/3799608464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ede4e0 msgr2=0x5568f6ececb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb6b7fe700 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ede4e0 0x5568f6ececb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb6b7fe700 1 -- 172.21.15.3:0/3799608464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 msgr2=0x5568f6ed1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb6b7fe700 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ed1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb6b7fe700 1 -- 172.21.15.3:0/3799608464 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6c06e740 con 0x5568f6ecd360 2023-09-10T01:05:15.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb6b7fe700 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ece630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feb6c06c1e0 tx=0x7feb6000e630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=288cc861433c3bee server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.036+0000 7feb4b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:15.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.037+0000 7feb4b7fe700 1 -- 172.21.15.3:0/3799608464 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60005b20 con 0x5568f6ecd360 2023-09-10T01:05:15.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.037+0000 7feb4b7fe700 1 -- 172.21.15.3:0/3799608464 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7feb60014070 con 0x5568f6ecd360 2023-09-10T01:05:15.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.037+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568f6d754f0 con 0x5568f6ecd360 2023-09-10T01:05:15.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.037+0000 7feb4b7fe700 1 -- 172.21.15.3:0/3799608464 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb60004070 con 0x5568f6ecd360 2023-09-10T01:05:15.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.037+0000 7feb8612eb80 10 client.26188 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:15.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.037+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5568f6c32c60 con 0x5568f6ecd360 2023-09-10T01:05:15.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb4b7fe700 1 -- 172.21.15.3:0/3799608464 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5568f6d754f0 con 0x5568f6ecd360 2023-09-10T01:05:15.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb4b7fe700 1 -- 172.21.15.3:0/3799608464 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7feb60005310 con 0x5568f6ecd360 2023-09-10T01:05:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb8612eb80 10 client.26188 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb8612eb80 10 client.26188 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:05:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb8612eb80 20 client.26188 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb8612eb80 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6eee0d0 0x5568f6ef04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb8612eb80 4 client.26188 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:15.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568f6e328a0 con 0x5568f6eee0d0 2023-09-10T01:05:15.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.038+0000 7feb6bfff700 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6eee0d0 0x5568f6ef04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.039+0000 7feb6bfff700 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6eee0d0 0x5568f6ef04f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4cae39ba51e874aa server_cookie=912ca2c9ce36ccd9 in_seq=0 out_seq=0 2023-09-10T01:05:15.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.039+0000 7feb4b7fe700 10 client.26188 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:15.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.040+0000 7feb4b7fe700 1 -- 172.21.15.3:0/3799608464 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5568f6e328a0 con 0x5568f6eee0d0 2023-09-10T01:05:15.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.040+0000 7feb4b7fe700 10 client.26188 handle_command_reply: tid=0 2023-09-10T01:05:15.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.040+0000 7feb8612eb80 1 client.26188 shutdown 2023-09-10T01:05:15.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.040+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 msgr2=0x5568f6ece630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.040+0000 7feb8612eb80 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ece630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feb6c06c1e0 tx=0x7feb6000e630 comp rx=0 tx=0).stop 2023-09-10T01:05:15.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 shutdown_connections 2023-09-10T01:05:15.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6eee0d0 0x5568f6ef04f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ede4e0 0x5568f6ececb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ed1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 --2- 172.21.15.3:0/3799608464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ece630 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 >> 172.21.15.3:0/3799608464 conn(0x5568f6e40090 msgr2=0x5568f6ec6780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb4b7fe700 0 client.26188 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:15.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 shutdown_connections 2023-09-10T01:05:15.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.041+0000 7feb8612eb80 1 -- 172.21.15.3:0/3799608464 wait complete. 2023-09-10T01:05:15.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.042+0000 7feb8612eb80 20 client.26188 trim_cache size 0 max 16384 2023-09-10T01:05:15.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.057+0000 7feb8612eb80 1 Processor -- start 2023-09-10T01:05:15.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.057+0000 7feb8612eb80 1 -- start start 2023-09-10T01:05:15.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ee5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ed0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6edda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb8612eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6ecf590 con 0x5568f6ede4e0 2023-09-10T01:05:15.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb8612eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6ec8230 con 0x5568f6ecbc50 2023-09-10T01:05:15.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb8612eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6ed1950 con 0x5568f6ecd360 2023-09-10T01:05:15.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb70d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6edda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb70d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6edda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53470/0 (socket says 172.21.15.3:53470) 2023-09-10T01:05:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb70d9c700 1 -- 172.21.15.3:0/690663984 learned_addr learned my addr 172.21.15.3:0/690663984 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.058+0000 7feb6b7fe700 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ed0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb70d9c700 1 -- 172.21.15.3:0/690663984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 msgr2=0x5568f6ed0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb70d9c700 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ed0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb6bfff700 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ee5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb70d9c700 1 -- 172.21.15.3:0/690663984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 msgr2=0x5568f6ee5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb70d9c700 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ee5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb70d9c700 1 -- 172.21.15.3:0/690663984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568f6c70250 con 0x5568f6ede4e0 2023-09-10T01:05:15.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb6bfff700 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ee5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:05:15.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb70d9c700 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6edda20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7feb6400bde0 tx=0x7feb6401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13da3093e3ee1ccc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb52ffd700 1 -- 172.21.15.3:0/690663984 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6400e040 con 0x5568f6ede4e0 2023-09-10T01:05:15.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.059+0000 7feb52ffd700 1 -- 172.21.15.3:0/690663984 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6400be10 con 0x5568f6ede4e0 2023-09-10T01:05:15.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb52ffd700 1 -- 172.21.15.3:0/690663984 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb640052a0 con 0x5568f6ede4e0 2023-09-10T01:05:15.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb8612eb80 1 -- 172.21.15.3:0/690663984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 msgr2=0x5568f6edda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb8612eb80 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6edda20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7feb6400bde0 tx=0x7feb6401c000 comp rx=0 tx=0).stop 2023-09-10T01:05:15.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb8612eb80 1 -- 172.21.15.3:0/690663984 shutdown_connections 2023-09-10T01:05:15.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb8612eb80 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ede4e0 0x5568f6edda20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb8612eb80 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecbc50 0x5568f6ee5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb8612eb80 1 --2- 172.21.15.3:0/690663984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ecd360 0x5568f6ed0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.060+0000 7feb8612eb80 1 -- 172.21.15.3:0/690663984 >> 172.21.15.3:0/690663984 conn(0x5568f6e40090 msgr2=0x5568f6ee9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.061+0000 7feb8612eb80 1 -- 172.21.15.3:0/690663984 shutdown_connections 2023-09-10T01:05:15.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.061+0000 7feb8612eb80 1 -- 172.21.15.3:0/690663984 wait complete. 2023-09-10T01:05:15.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.061+0000 7feb8612eb80 1 -- start start 2023-09-10T01:05:15.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.061+0000 7feb8612eb80 1 Processor -- start 2023-09-10T01:05:15.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ee53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecd360 0x5568f6ee3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb8612eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ede4e0 0x5568f6ee3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb8612eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6ee46b0 con 0x5568f6ecbc50 2023-09-10T01:05:15.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb8612eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6f41f30 con 0x5568f6ecd360 2023-09-10T01:05:15.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb8612eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f6f42070 con 0x5568f6ede4e0 2023-09-10T01:05:15.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb8612eb80 10 client.0 _ll_register_callbacks cb 0x7feb60009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:15.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb6b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecd360 0x5568f6ee3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb6bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ee53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb70d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ede4e0 0x5568f6ee3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb6b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecd360 0x5568f6ee3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57474/0 (socket says 172.21.15.3:57474) 2023-09-10T01:05:15.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb6b7fe700 1 -- 172.21.15.3:0/769609552 learned_addr learned my addr 172.21.15.3:0/769609552 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb70d9c700 1 -- 172.21.15.3:0/769609552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecd360 msgr2=0x5568f6ee3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb70d9c700 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecd360 0x5568f6ee3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb70d9c700 1 -- 172.21.15.3:0/769609552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 msgr2=0x5568f6ee53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb70d9c700 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ee53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb70d9c700 1 -- 172.21.15.3:0/769609552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6c10de00 con 0x5568f6ede4e0 2023-09-10T01:05:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb6bfff700 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ee53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.062+0000 7feb6b7fe700 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecd360 0x5568f6ee3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb70d9c700 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ede4e0 0x5568f6ee3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feb640077c0 tx=0x7feb6400b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a833ca15211752 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb4b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:15.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb4b7fe700 1 -- 172.21.15.3:0/769609552 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6400e050 con 0x5568f6ede4e0 2023-09-10T01:05:15.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb5c0ccde0 con 0x5568f6ede4e0 2023-09-10T01:05:15.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb4b7fe700 1 -- 172.21.15.3:0/769609552 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7feb64024040 con 0x5568f6ede4e0 2023-09-10T01:05:15.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb4b7fe700 1 -- 172.21.15.3:0/769609552 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb6401cd60 con 0x5568f6ede4e0 2023-09-10T01:05:15.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb8612eb80 10 client.26194 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:15.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5568f6c482b0 con 0x5568f6ede4e0 2023-09-10T01:05:15.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.063+0000 7feb4b7fe700 1 -- 172.21.15.3:0/769609552 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb5c0ccde0 con 0x5568f6ede4e0 2023-09-10T01:05:15.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb4b7fe700 1 -- 172.21.15.3:0/769609552 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7feb6402a410 con 0x5568f6ede4e0 2023-09-10T01:05:15.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb8612eb80 10 client.26194 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:15.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb8612eb80 10 client.26194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:05:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb8612eb80 20 client.26194 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb8612eb80 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6f43400 0x5568f6f45820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb8612eb80 4 client.26194 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:05:15.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5568f6e16920 con 0x5568f6f43400 2023-09-10T01:05:15.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.064+0000 7feb6bfff700 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6f43400 0x5568f6f45820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb4b7fe700 10 client.26194 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:15.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb6bfff700 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6f43400 0x5568f6f45820 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a191adfcc0273368 server_cookie=2ad05e2600695833 in_seq=0 out_seq=0 2023-09-10T01:05:15.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb4b7fe700 1 -- 172.21.15.3:0/769609552 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5568f6e16920 con 0x5568f6f43400 2023-09-10T01:05:15.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb4b7fe700 10 client.26194 handle_command_reply: tid=0 2023-09-10T01:05:15.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 client.26194 shutdown 2023-09-10T01:05:15.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ede4e0 msgr2=0x5568f6ee3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ede4e0 0x5568f6ee3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feb640077c0 tx=0x7feb6400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 shutdown_connections 2023-09-10T01:05:15.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5568f6f43400 0x5568f6f45820 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5568f6ede4e0 0x5568f6ee3f70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5568f6ecbc50 0x5568f6ee53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb4b7fe700 0 client.26194 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 --2- 172.21.15.3:0/769609552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568f6ecd360 0x5568f6ee3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.065+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 >> 172.21.15.3:0/769609552 conn(0x5568f6e40090 msgr2=0x5568f6edda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.066+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 shutdown_connections 2023-09-10T01:05:15.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.066+0000 7feb8612eb80 1 -- 172.21.15.3:0/769609552 wait complete. 2023-09-10T01:05:15.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.066+0000 7feb8612eb80 20 client.26194 trim_cache size 0 max 16384 2023-09-10T01:05:15.080 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:05:15.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb540a0830 msgr2=0x7feb540a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb540a0830 0x7feb540a2cf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feb64009850 tx=0x7feb64009780 comp rx=0 tx=0).stop 2023-09-10T01:05:15.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c101780 msgr2=0x7feb6c06f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c101780 0x7feb6c06f800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00e9a0 tx=0x7feb5c00ed60 comp rx=0 tx=0).stop 2023-09-10T01:05:15.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 shutdown_connections 2023-09-10T01:05:15.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb540a0830 0x7feb540a2cf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb6c10acd0 0x7feb6c06a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb6c101780 0x7feb6c06f800 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 --2- 172.21.15.3:0/1141055596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c102130 0x7feb6c06fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 >> 172.21.15.3:0/1141055596 conn(0x7feb6c0fb010 msgr2=0x7feb6c1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 shutdown_connections 2023-09-10T01:05:15.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.067+0000 7feb727ff700 1 -- 172.21.15.3:0/1141055596 wait complete. 2023-09-10T01:05:15.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:05:15.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 -- 172.21.15.3:0/1541835483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 msgr2=0x7f266c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 --2- 172.21.15.3:0/1541835483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 0x7f266c10d510 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2664009f90 tx=0x7f266400b330 comp rx=0 tx=0).stop 2023-09-10T01:05:15.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 -- 172.21.15.3:0/1541835483 shutdown_connections 2023-09-10T01:05:15.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 --2- 172.21.15.3:0/1541835483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 0x7f266c10d510 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 --2- 172.21.15.3:0/1541835483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f266c106650 0x7f266c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 --2- 172.21.15.3:0/1541835483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c100640 0x7f266c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 -- 172.21.15.3:0/1541835483 >> 172.21.15.3:0/1541835483 conn(0x7f266c0fc170 msgr2=0x7f266c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 -- 172.21.15.3:0/1541835483 shutdown_connections 2023-09-10T01:05:15.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.378+0000 7f26725ff700 1 -- 172.21.15.3:0/1541835483 wait complete. 2023-09-10T01:05:15.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.379+0000 7f26725ff700 1 Processor -- start 2023-09-10T01:05:15.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.379+0000 7f26725ff700 1 -- start start 2023-09-10T01:05:15.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f26725ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f266c100640 0x7f266c196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f26725ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 0x7f266c196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f26725ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c106650 0x7f266c19b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f26725ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c19b640 con 0x7f266c100640 2023-09-10T01:05:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f26725ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c19b7b0 con 0x7f266c101000 2023-09-10T01:05:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f26725ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c19b970 con 0x7f266c106650 2023-09-10T01:05:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c106650 0x7f266c19b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f266bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f266c100640 0x7f266c196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c106650 0x7f266c19b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57230/0 (socket says 172.21.15.3:57230) 2023-09-10T01:05:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 -- 172.21.15.3:0/698952597 learned_addr learned my addr 172.21.15.3:0/698952597 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f266b7fe700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 0x7f266c196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 -- 172.21.15.3:0/698952597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 msgr2=0x7f266c196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 0x7f266c196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 -- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f266c100640 msgr2=0x7f266c196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f266c100640 0x7f266c196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.380+0000 7f2670b9c700 1 -- 172.21.15.3:0/698952597 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2664009c40 con 0x7f266c106650 2023-09-10T01:05:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.381+0000 7f2670b9c700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c106650 0x7f266c19b000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2664004a50 tx=0x7f2664004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.381+0000 7f26697fa700 1 -- 172.21.15.3:0/698952597 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266400e040 con 0x7f266c106650 2023-09-10T01:05:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.381+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c19bbf0 con 0x7f266c106650 2023-09-10T01:05:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.381+0000 7f26697fa700 1 -- 172.21.15.3:0/698952597 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f266400bbf0 con 0x7f266c106650 2023-09-10T01:05:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.381+0000 7f26697fa700 1 -- 172.21.15.3:0/698952597 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26640052a0 con 0x7f266c106650 2023-09-10T01:05:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.381+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c1a3490 con 0x7f266c106650 2023-09-10T01:05:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.384+0000 7f26697fa700 1 -- 172.21.15.3:0/698952597 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f266401d470 con 0x7f266c106650 2023-09-10T01:05:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.386+0000 7f26697fa700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f26540a0830 0x7f26540a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.386+0000 7f2685fb9b80 1 Processor -- start 2023-09-10T01:05:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.386+0000 7f26697fa700 1 -- 172.21.15.3:0/698952597 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f26640cdac0 con 0x7f266c106650 2023-09-10T01:05:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.386+0000 7f266bfff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f26540a0830 0x7f26540a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.386+0000 7f2685fb9b80 1 -- start start 2023-09-10T01:05:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622bebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622bee640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622b36730 con 0x559622bec250 2023-09-10T01:05:15.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622b34050 con 0x559622bdd710 2023-09-10T01:05:15.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622b2f940 con 0x559622bde030 2023-09-10T01:05:15.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622bee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622bee640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53506/0 (socket says 172.21.15.3:53506) 2023-09-10T01:05:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 -- 172.21.15.3:0/473266054 learned_addr learned my addr 172.21.15.3:0/473266054 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f266bfff700 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f266bfff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f26540a0830 0x7f26540a2cf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f265c005fd0 tx=0x7f265c009310 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f266b7fe700 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622bebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 -- 172.21.15.3:0/473266054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 msgr2=0x559622bebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622bebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 -- 172.21.15.3:0/473266054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 msgr2=0x559622bddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bddaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.387+0000 7f2670b9c700 1 -- 172.21.15.3:0/473266054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55962295bde0 con 0x559622bec250 2023-09-10T01:05:15.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.388+0000 7f2670b9c700 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622bee640 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2664097110 tx=0x7f266409a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee20478359f0bce8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.388+0000 7f26527fc700 1 -- 172.21.15.3:0/473266054 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266409b310 con 0x559622bec250 2023-09-10T01:05:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.388+0000 7f26527fc700 1 -- 172.21.15.3:0/473266054 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f266400bd60 con 0x559622bec250 2023-09-10T01:05:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.388+0000 7f26527fc700 1 -- 172.21.15.3:0/473266054 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26640ba0e0 con 0x559622bec250 2023-09-10T01:05:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 -- 172.21.15.3:0/473266054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 msgr2=0x559622bee640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622bee640 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2664097110 tx=0x7f266409a7d0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 -- 172.21.15.3:0/473266054 shutdown_connections 2023-09-10T01:05:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622bee640 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/473266054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622bebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 -- 172.21.15.3:0/473266054 >> 172.21.15.3:0/473266054 conn(0x559622b52090 msgr2=0x559622bd8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 -- 172.21.15.3:0/473266054 shutdown_connections 2023-09-10T01:05:15.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.389+0000 7f2685fb9b80 1 -- 172.21.15.3:0/473266054 wait complete. 2023-09-10T01:05:15.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.390+0000 7f2685fb9b80 1 -- start start 2023-09-10T01:05:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.390+0000 7f2685fb9b80 1 Processor -- start 2023-09-10T01:05:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622be8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bec250 0x559622be9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622acdb70 con 0x559622bde030 2023-09-10T01:05:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622acd870 con 0x559622bdd710 2023-09-10T01:05:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596229b1920 con 0x559622bec250 2023-09-10T01:05:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2685fb9b80 10 client.0 _ll_register_callbacks cb 0x559622a17a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bebce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57510/0 (socket says 172.21.15.3:57510) 2023-09-10T01:05:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 -- 172.21.15.3:0/1418603596 learned_addr learned my addr 172.21.15.3:0/1418603596 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266b7fe700 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622be8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f2670b9c700 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bec250 0x559622be9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 -- 172.21.15.3:0/1418603596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bec250 msgr2=0x559622be9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bec250 0x559622be9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 -- 172.21.15.3:0/1418603596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 msgr2=0x559622be8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622be8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.391+0000 7f266bfff700 1 -- 172.21.15.3:0/1418603596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c1a3490 con 0x559622bdd710 2023-09-10T01:05:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f266bfff700 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bebce0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f265c004460 tx=0x7f265c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0298a660dda3a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f264b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f264b7fe700 1 -- 172.21.15.3:0/1418603596 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c022070 con 0x559622bdd710 2023-09-10T01:05:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f264b7fe700 1 -- 172.21.15.3:0/1418603596 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f265c003d70 con 0x559622bdd710 2023-09-10T01:05:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559622a874f0 con 0x559622bdd710 2023-09-10T01:05:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f264b7fe700 1 -- 172.21.15.3:0/1418603596 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f265c027420 con 0x559622bdd710 2023-09-10T01:05:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f2685fb9b80 10 client.26288 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559622944c60 con 0x559622bdd710 2023-09-10T01:05:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.392+0000 7f264b7fe700 1 -- 172.21.15.3:0/1418603596 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559622a874f0 con 0x559622bdd710 2023-09-10T01:05:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f264b7fe700 1 -- 172.21.15.3:0/1418603596 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f265c01d670 con 0x559622bdd710 2023-09-10T01:05:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f2685fb9b80 10 client.26288 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f2685fb9b80 10 client.26288 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:05:15.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f2685fb9b80 20 client.26288 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:15.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622bfbc70 0x559622bfe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f2685fb9b80 4 client.26288 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:15.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559622b448a0 con 0x559622bfbc70 2023-09-10T01:05:15.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.393+0000 7f266b7fe700 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622bfbc70 0x559622bfe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.394+0000 7f266b7fe700 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622bfbc70 0x559622bfe090 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=335d91ebbab8aa35 server_cookie=487ef765ed4e0672 in_seq=0 out_seq=0 2023-09-10T01:05:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.394+0000 7f264b7fe700 10 client.26288 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.394+0000 7f264b7fe700 1 -- 172.21.15.3:0/1418603596 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559622b448a0 con 0x559622bfbc70 2023-09-10T01:05:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.394+0000 7f264b7fe700 10 client.26288 handle_command_reply: tid=0 2023-09-10T01:05:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.394+0000 7f2685fb9b80 1 client.26288 shutdown 2023-09-10T01:05:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 msgr2=0x559622bebce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bebce0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f265c004460 tx=0x7f265c0047c0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 shutdown_connections 2023-09-10T01:05:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622bfbc70 0x559622bfe090 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bec250 0x559622be9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bdd710 0x559622bebce0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/1418603596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622be8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 >> 172.21.15.3:0/1418603596 conn(0x559622b52090 msgr2=0x559622bd8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f264b7fe700 0 client.26288 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 shutdown_connections 2023-09-10T01:05:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 1 -- 172.21.15.3:0/1418603596 wait complete. 2023-09-10T01:05:15.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.395+0000 7f2685fb9b80 20 client.26288 trim_cache size 0 max 16384 2023-09-10T01:05:15.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:15 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3854371988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:05:15.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 Processor -- start 2023-09-10T01:05:15.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 -- start start 2023-09-10T01:05:15.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622beb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622be42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bebd50 0x559622be87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622bddd40 con 0x559622bec250 2023-09-10T01:05:15.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622be9c30 con 0x559622bebd50 2023-09-10T01:05:15.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.411+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622bde7e0 con 0x559622bde030 2023-09-10T01:05:15.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622beb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bebd50 0x559622be87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622be42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622beb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57260/0 (socket says 172.21.15.3:57260) 2023-09-10T01:05:15.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 -- 172.21.15.3:0/2068991789 learned_addr learned my addr 172.21.15.3:0/2068991789 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bebd50 0x559622be87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57516/0 (socket says 172.21.15.3:57516) 2023-09-10T01:05:15.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 -- 172.21.15.3:0/2068991789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bebd50 msgr2=0x559622be87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 --2- 172.21.15.3:0/2068991789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bebd50 0x559622be87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 -- 172.21.15.3:0/2068991789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 msgr2=0x559622be42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 --2- 172.21.15.3:0/2068991789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622be42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 -- 172.21.15.3:0/2068991789 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559622982250 con 0x559622bde030 2023-09-10T01:05:15.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.412+0000 7f266bfff700 1 --2- 172.21.15.3:0/2068991789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622beb810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f265c012fd0 tx=0x7f265c009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4300c9ebc58f2be6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2652ffd700 1 -- 172.21.15.3:0/2068991789 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c022040 con 0x559622bde030 2023-09-10T01:05:15.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2652ffd700 1 -- 172.21.15.3:0/2068991789 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f265c003c40 con 0x559622bde030 2023-09-10T01:05:15.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2652ffd700 1 -- 172.21.15.3:0/2068991789 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c004dc0 con 0x559622bde030 2023-09-10T01:05:15.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 -- 172.21.15.3:0/2068991789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 msgr2=0x559622beb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/2068991789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622beb810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f265c012fd0 tx=0x7f265c009400 comp rx=0 tx=0).stop 2023-09-10T01:05:15.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 -- 172.21.15.3:0/2068991789 shutdown_connections 2023-09-10T01:05:15.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/2068991789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bde030 0x559622beb810 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/2068991789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bebd50 0x559622be87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/2068991789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bec250 0x559622be42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 -- 172.21.15.3:0/2068991789 >> 172.21.15.3:0/2068991789 conn(0x559622b52090 msgr2=0x559622bf7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.413+0000 7f2685fb9b80 1 -- 172.21.15.3:0/2068991789 shutdown_connections 2023-09-10T01:05:15.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.414+0000 7f2685fb9b80 1 -- 172.21.15.3:0/2068991789 wait complete. 2023-09-10T01:05:15.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.414+0000 7f2685fb9b80 1 -- start start 2023-09-10T01:05:15.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.414+0000 7f2685fb9b80 1 Processor -- start 2023-09-10T01:05:15.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622bf29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bebd50 0x559622bf0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2685fb9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bec250 0x559622bf1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622bf17a0 con 0x559622bde030 2023-09-10T01:05:15.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622bf1910 con 0x559622bec250 2023-09-10T01:05:15.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2685fb9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559622bf1ad0 con 0x559622bebd50 2023-09-10T01:05:15.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2685fb9b80 10 client.0 _ll_register_callbacks cb 0x7f265c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:15.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f266bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622bf29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bec250 0x559622bf1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bec250 0x559622bf1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57528/0 (socket says 172.21.15.3:57528) 2023-09-10T01:05:15.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 -- 172.21.15.3:0/170525757 learned_addr learned my addr 172.21.15.3:0/170525757 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f266b7fe700 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bebd50 0x559622bf0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 -- 172.21.15.3:0/170525757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bebd50 msgr2=0x559622bf0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bebd50 0x559622bf0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 -- 172.21.15.3:0/170525757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 msgr2=0x559622bf29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622bf29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f2670b9c700 1 -- 172.21.15.3:0/170525757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559622944c60 con 0x559622bec250 2023-09-10T01:05:15.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.415+0000 7f266bfff700 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622bf29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:15.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f2670b9c700 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bec250 0x559622bf1260 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f266402dff0 tx=0x7f266400bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c29f1f71af2fd2e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f264b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:15.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f264b7fe700 1 -- 172.21.15.3:0/170525757 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266409ea40 con 0x559622bec250 2023-09-10T01:05:15.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f266400bd60 con 0x559622bec250 2023-09-10T01:05:15.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f264b7fe700 1 -- 172.21.15.3:0/170525757 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f26640a2bf0 con 0x559622bec250 2023-09-10T01:05:15.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f264b7fe700 1 -- 172.21.15.3:0/170525757 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f266409ac10 con 0x559622bec250 2023-09-10T01:05:15.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f2685fb9b80 10 client.26294 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:15.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55962295a2b0 con 0x559622bec250 2023-09-10T01:05:15.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.416+0000 7f264b7fe700 1 -- 172.21.15.3:0/170525757 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f266400bd60 con 0x559622bec250 2023-09-10T01:05:15.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f264b7fe700 1 -- 172.21.15.3:0/170525757 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f26640a2760 con 0x559622bec250 2023-09-10T01:05:15.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f2685fb9b80 10 client.26294 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:15.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f2685fb9b80 10 client.26294 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:05:15.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f2685fb9b80 20 client.26294 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:15.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622c50ab0 0x559622c52ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f2685fb9b80 4 client.26294 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:05:15.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559622b28920 con 0x559622c50ab0 2023-09-10T01:05:15.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f266bfff700 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622c50ab0 0x559622c52ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f266bfff700 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622c50ab0 0x559622c52ed0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6bab096aa543eef8 server_cookie=93f8d81d1c87b2b5 in_seq=0 out_seq=0 2023-09-10T01:05:15.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.417+0000 7f264b7fe700 10 client.26294 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:15.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f264b7fe700 1 -- 172.21.15.3:0/170525757 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559622b28920 con 0x559622c50ab0 2023-09-10T01:05:15.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f264b7fe700 10 client.26294 handle_command_reply: tid=0 2023-09-10T01:05:15.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 client.26294 shutdown 2023-09-10T01:05:15.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bec250 msgr2=0x559622bf1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bec250 0x559622bf1260 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f266402dff0 tx=0x7f266400bed0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 shutdown_connections 2023-09-10T01:05:15.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559622bec250 0x559622bf1260 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559622bde030 0x559622bf29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x559622c50ab0 0x559622c52ed0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 --2- 172.21.15.3:0/170525757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559622bebd50 0x559622bf0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f264b7fe700 0 client.26294 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:15.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 >> 172.21.15.3:0/170525757 conn(0x559622b52090 msgr2=0x559622bf7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 shutdown_connections 2023-09-10T01:05:15.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 1 -- 172.21.15.3:0/170525757 wait complete. 2023-09-10T01:05:15.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.418+0000 7f2685fb9b80 20 client.26294 trim_cache size 0 max 16384 2023-09-10T01:05:15.433 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:05:15.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f26540a0830 msgr2=0x7f26540a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f26540a0830 0x7f26540a2cf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f265c005fd0 tx=0x7f265c009310 comp rx=0 tx=0).stop 2023-09-10T01:05:15.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c106650 msgr2=0x7f266c19b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c106650 0x7f266c19b000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2664004a50 tx=0x7f2664004a80 comp rx=0 tx=0).stop 2023-09-10T01:05:15.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 shutdown_connections 2023-09-10T01:05:15.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f26540a0830 0x7f26540a2cf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f266c106650 0x7f266c19b000 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f266c100640 0x7f266c196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 --2- 172.21.15.3:0/698952597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c101000 0x7f266c196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 >> 172.21.15.3:0/698952597 conn(0x7f266c0fc170 msgr2=0x7f266c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 shutdown_connections 2023-09-10T01:05:15.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.419+0000 7f26725ff700 1 -- 172.21.15.3:0/698952597 wait complete. 2023-09-10T01:05:15.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:15 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3854371988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:05:15.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:05:15.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:15 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3854371988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:05:15.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 -- 172.21.15.3:0/1683857302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda4072a30 msgr2=0x7efda4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 --2- 172.21.15.3:0/1683857302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda4072a30 0x7efda4072e10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd94009990 tx=0x7efd94009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 -- 172.21.15.3:0/1683857302 shutdown_connections 2023-09-10T01:05:15.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 --2- 172.21.15.3:0/1683857302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efda410ce80 0x7efda410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 --2- 172.21.15.3:0/1683857302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda4072a30 0x7efda4072e10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 --2- 172.21.15.3:0/1683857302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efda4073350 0x7efda410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 -- 172.21.15.3:0/1683857302 >> 172.21.15.3:0/1683857302 conn(0x7efda40fc170 msgr2=0x7efda40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 -- 172.21.15.3:0/1683857302 shutdown_connections 2023-09-10T01:05:15.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.656+0000 7efda8fff700 1 -- 172.21.15.3:0/1683857302 wait complete. 2023-09-10T01:05:15.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.657+0000 7efda8fff700 1 Processor -- start 2023-09-10T01:05:15.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.658+0000 7efda8fff700 1 -- start start 2023-09-10T01:05:15.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.658+0000 7efda8fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efda4072a30 0x7efda41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.658+0000 7efda8fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efda4073350 0x7efda4198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.658+0000 7efda8fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda410ce80 0x7efda419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.658+0000 7efda8fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda419d940 con 0x7efda4072a30 2023-09-10T01:05:15.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.658+0000 7efda8fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda419dab0 con 0x7efda410ce80 2023-09-10T01:05:15.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda8fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda419dc70 con 0x7efda4073350 2023-09-10T01:05:15.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efda4072a30 0x7efda41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda410ce80 0x7efda419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efda4073350 0x7efda4198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efda4072a30 0x7efda41989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53548/0 (socket says 172.21.15.3:53548) 2023-09-10T01:05:15.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda410ce80 0x7efda419d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57544/0 (socket says 172.21.15.3:57544) 2023-09-10T01:05:15.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda259c700 1 -- 172.21.15.3:0/4169024694 learned_addr learned my addr 172.21.15.3:0/4169024694 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 -- 172.21.15.3:0/4169024694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efda4073350 msgr2=0x7efda4198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efda4073350 0x7efda4198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 -- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efda4072a30 msgr2=0x7efda41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efda4072a30 0x7efda41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 -- 172.21.15.3:0/4169024694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c009520 con 0x7efda410ce80 2023-09-10T01:05:15.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.659+0000 7efda2d9d700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda410ce80 0x7efda419d370 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd9c007f10 tx=0x7efd9c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.660+0000 7efd9b7fe700 1 -- 172.21.15.3:0/4169024694 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c009ea0 con 0x7efda410ce80 2023-09-10T01:05:15.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.660+0000 7efd9b7fe700 1 -- 172.21.15.3:0/4169024694 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd9c00f040 con 0x7efda410ce80 2023-09-10T01:05:15.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.660+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd940095f0 con 0x7efda410ce80 2023-09-10T01:05:15.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.660+0000 7efd9b7fe700 1 -- 172.21.15.3:0/4169024694 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c004350 con 0x7efda410ce80 2023-09-10T01:05:15.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.660+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda41a5620 con 0x7efda410ce80 2023-09-10T01:05:15.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.663+0000 7efd9b7fe700 1 -- 172.21.15.3:0/4169024694 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7efd9c00db20 con 0x7efda410ce80 2023-09-10T01:05:15.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.664+0000 7efdbc846b80 1 Processor -- start 2023-09-10T01:05:15.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.664+0000 7efdbc846b80 1 -- start start 2023-09-10T01:05:15.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c39af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c4c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efd9b7fe700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efd900a07b0 0x7efd900a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c4ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efdbc846b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b92730 con 0x560b57c3a030 2023-09-10T01:05:15.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efdbc846b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b90050 con 0x560b57c4c650 2023-09-10T01:05:15.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efdbc846b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b8b940 con 0x560b57c39710 2023-09-10T01:05:15.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efd9b7fe700 1 -- 172.21.15.3:0/4169024694 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efd9c0cbc20 con 0x7efda410ce80 2023-09-10T01:05:15.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c39af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c4ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c39af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57284/0 (socket says 172.21.15.3:57284) 2023-09-10T01:05:15.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efd9bfff700 1 -- 172.21.15.3:0/4257206102 learned_addr learned my addr 172.21.15.3:0/4257206102 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efda259c700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efd900a07b0 0x7efd900a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.665+0000 7efd9bfff700 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c4c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efda2d9d700 1 -- 172.21.15.3:0/4257206102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 msgr2=0x560b57c39af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efda2d9d700 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c39af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efda2d9d700 1 -- 172.21.15.3:0/4257206102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 msgr2=0x560b57c4c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efda2d9d700 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c4c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efda2d9d700 1 -- 172.21.15.3:0/4257206102 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b579b7de0 con 0x560b57c4c650 2023-09-10T01:05:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efda2d9d700 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c4ea40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0c8f80 tx=0x7efd9c098ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23c1d51a06148f7e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efd98ff9700 1 -- 172.21.15.3:0/4257206102 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c098640 con 0x560b57c4c650 2023-09-10T01:05:15.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efda259c700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efd900a07b0 0x7efd900a2c70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd94006010 tx=0x7efd9400b260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efd98ff9700 1 -- 172.21.15.3:0/4257206102 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd9c01ec80 con 0x560b57c4c650 2023-09-10T01:05:15.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.666+0000 7efd98ff9700 1 -- 172.21.15.3:0/4257206102 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c09cb10 con 0x560b57c4c650 2023-09-10T01:05:15.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 -- 172.21.15.3:0/4257206102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 msgr2=0x560b57c4ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c4ea40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0c8f80 tx=0x7efd9c098ab0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 -- 172.21.15.3:0/4257206102 shutdown_connections 2023-09-10T01:05:15.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c39af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c4ea40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 --2- 172.21.15.3:0/4257206102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c4c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 -- 172.21.15.3:0/4257206102 >> 172.21.15.3:0/4257206102 conn(0x560b57bae090 msgr2=0x560b57c34bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 -- 172.21.15.3:0/4257206102 shutdown_connections 2023-09-10T01:05:15.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.667+0000 7efdbc846b80 1 -- 172.21.15.3:0/4257206102 wait complete. 2023-09-10T01:05:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 -- start start 2023-09-10T01:05:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 Processor -- start 2023-09-10T01:05:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c49330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c49870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c49db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b29b70 con 0x560b57c3a030 2023-09-10T01:05:15.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b29870 con 0x560b57c4c650 2023-09-10T01:05:15.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.668+0000 7efdbc846b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57a0d920 con 0x560b57c39710 2023-09-10T01:05:15.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efdbc846b80 10 client.0 _ll_register_callbacks cb 0x560b57a73a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c49870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c49db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c49870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53576/0 (socket says 172.21.15.3:53576) 2023-09-10T01:05:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c49db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57556/0 (socket says 172.21.15.3:57556) 2023-09-10T01:05:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efd9bfff700 1 -- 172.21.15.3:0/2331212746 learned_addr learned my addr 172.21.15.3:0/2331212746 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda259c700 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c49330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 -- 172.21.15.3:0/2331212746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 msgr2=0x560b57c49330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c49330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 -- 172.21.15.3:0/2331212746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 msgr2=0x560b57c49870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c49870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 -- 172.21.15.3:0/2331212746 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda41a5620 con 0x560b57c4c650 2023-09-10T01:05:15.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efd9bfff700 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c49870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:15.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efda2d9d700 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c49db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0c8fb0 tx=0x7efd9c0183b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=395dcbe89475a8d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efd89ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:15.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.669+0000 7efd89ffb700 1 -- 172.21.15.3:0/2331212746 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c098640 con 0x560b57c4c650 2023-09-10T01:05:15.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b57ae34f0 con 0x560b57c4c650 2023-09-10T01:05:15.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efd89ffb700 1 -- 172.21.15.3:0/2331212746 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efd9c0993e0 con 0x560b57c4c650 2023-09-10T01:05:15.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efd89ffb700 1 -- 172.21.15.3:0/2331212746 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd9c0b6950 con 0x560b57c4c650 2023-09-10T01:05:15.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 10 client.26306 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:15.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560b579a0c60 con 0x560b57c4c650 2023-09-10T01:05:15.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efd89ffb700 1 -- 172.21.15.3:0/2331212746 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560b57ae34f0 con 0x560b57c4c650 2023-09-10T01:05:15.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efd89ffb700 1 -- 172.21.15.3:0/2331212746 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7efd9c0b81f0 con 0x560b57c4c650 2023-09-10T01:05:15.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 10 client.26306 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:15.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 10 client.26306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:15.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 20 client.26306 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:15.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57c5bdf0 0x560b57c5e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 4 client.26306 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b57ba08a0 con 0x560b57c5bdf0 2023-09-10T01:05:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.670+0000 7efda259c700 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57c5bdf0 0x560b57c5e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.671+0000 7efda259c700 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57c5bdf0 0x560b57c5e1e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15bf9d5756e799c1 server_cookie=e8e5dc777246a213 in_seq=0 out_seq=0 2023-09-10T01:05:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.671+0000 7efd89ffb700 10 client.26306 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efd89ffb700 1 -- 172.21.15.3:0/2331212746 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560b57ba08a0 con 0x560b57c5bdf0 2023-09-10T01:05:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efd89ffb700 10 client.26306 handle_command_reply: tid=0 2023-09-10T01:05:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 client.26306 shutdown 2023-09-10T01:05:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 msgr2=0x560b57c49db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c49db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0c8fb0 tx=0x7efd9c0183b0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 shutdown_connections 2023-09-10T01:05:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57c5bdf0 0x560b57c5e1e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c49330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c49db0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2331212746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c49870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 >> 172.21.15.3:0/2331212746 conn(0x560b57bae090 msgr2=0x560b57c34780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efd89ffb700 0 client.26306 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:15.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 shutdown_connections 2023-09-10T01:05:15.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.672+0000 7efdbc846b80 1 -- 172.21.15.3:0/2331212746 wait complete. 2023-09-10T01:05:15.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.673+0000 7efdbc846b80 20 client.26306 trim_cache size 0 max 16384 2023-09-10T01:05:15.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 Processor -- start 2023-09-10T01:05:15.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 -- start start 2023-09-10T01:05:15.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c39710 0x560b57c485c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c3a030 0x560b57c49be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c346f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c4eee0 con 0x560b57c39710 2023-09-10T01:05:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c3b6c0 con 0x560b57c4c650 2023-09-10T01:05:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efdbc846b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c2c400 con 0x560b57c3a030 2023-09-10T01:05:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c3a030 0x560b57c49be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efda259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c39710 0x560b57c485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c346f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c3a030 0x560b57c49be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57312/0 (socket says 172.21.15.3:57312) 2023-09-10T01:05:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.687+0000 7efd9bfff700 1 -- 172.21.15.3:0/2249952165 learned_addr learned my addr 172.21.15.3:0/2249952165 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd9bfff700 1 -- 172.21.15.3:0/2249952165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 msgr2=0x560b57c346f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd9bfff700 1 --2- 172.21.15.3:0/2249952165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c346f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd9bfff700 1 -- 172.21.15.3:0/2249952165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c39710 msgr2=0x560b57c485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd9bfff700 1 --2- 172.21.15.3:0/2249952165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c39710 0x560b57c485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd9bfff700 1 -- 172.21.15.3:0/2249952165 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b579de250 con 0x560b57c3a030 2023-09-10T01:05:15.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd9bfff700 1 --2- 172.21.15.3:0/2249952165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c3a030 0x560b57c49be0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009310 tx=0x7efd8c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7c43003a34b08e9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd997fa700 1 -- 172.21.15.3:0/2249952165 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c016030 con 0x560b57c3a030 2023-09-10T01:05:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd997fa700 1 -- 172.21.15.3:0/2249952165 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd8c003680 con 0x560b57c3a030 2023-09-10T01:05:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.688+0000 7efd997fa700 1 -- 172.21.15.3:0/2249952165 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c01f6d0 con 0x560b57c3a030 2023-09-10T01:05:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 -- 172.21.15.3:0/2249952165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c3a030 msgr2=0x560b57c49be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2249952165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c3a030 0x560b57c49be0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009310 tx=0x7efd8c0047c0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 -- 172.21.15.3:0/2249952165 shutdown_connections 2023-09-10T01:05:15.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2249952165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c346f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2249952165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c39710 0x560b57c485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2249952165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c3a030 0x560b57c49be0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 -- 172.21.15.3:0/2249952165 >> 172.21.15.3:0/2249952165 conn(0x560b57bae090 msgr2=0x560b57c44610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 -- 172.21.15.3:0/2249952165 shutdown_connections 2023-09-10T01:05:15.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.689+0000 7efdbc846b80 1 -- 172.21.15.3:0/2249952165 wait complete. 2023-09-10T01:05:15.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 -- start start 2023-09-10T01:05:15.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 Processor -- start 2023-09-10T01:05:15.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c52f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c51290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c517d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c51d10 con 0x560b57c3a030 2023-09-10T01:05:15.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c51e80 con 0x560b57c4c650 2023-09-10T01:05:15.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.690+0000 7efdbc846b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c52040 con 0x560b57c39710 2023-09-10T01:05:15.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efdbc846b80 10 client.0 _ll_register_callbacks cb 0x7efd8c011500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:15.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c51290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c52f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57570/0 (socket says 172.21.15.3:57570) 2023-09-10T01:05:15.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda2d9d700 1 -- 172.21.15.3:0/2725157421 learned_addr learned my addr 172.21.15.3:0/2725157421 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:15.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda259c700 1 -- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 msgr2=0x560b57c517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda259c700 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda259c700 1 -- 172.21.15.3:0/2725157421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 msgr2=0x560b57c51290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda259c700 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c51290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda259c700 1 -- 172.21.15.3:0/2725157421 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8c019040 con 0x560b57c39710 2023-09-10T01:05:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda2d9d700 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:15.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efda259c700 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c52f60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x560b57c5eac0 tx=0x7efd94009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8576584c239d5d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:15.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.691+0000 7efd89ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:15.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efd89ffb700 1 -- 172.21.15.3:0/2725157421 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9402b040 con 0x560b57c39710 2023-09-10T01:05:15.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd9c01ec80 con 0x560b57c39710 2023-09-10T01:05:15.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efd89ffb700 1 -- 172.21.15.3:0/2725157421 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efd94033250 con 0x560b57c39710 2023-09-10T01:05:15.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efd89ffb700 1 -- 172.21.15.3:0/2725157421 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd94033410 con 0x560b57c39710 2023-09-10T01:05:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 10 client.26230 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560b579b62b0 con 0x560b57c39710 2023-09-10T01:05:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efd89ffb700 1 -- 172.21.15.3:0/2725157421 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd9c01ec80 con 0x560b57c39710 2023-09-10T01:05:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efd89ffb700 1 -- 172.21.15.3:0/2725157421 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7efd94032420 con 0x560b57c39710 2023-09-10T01:05:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 10 client.26230 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:15.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 10 client.26230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:15.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 20 client.26230 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:15.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57cb0fd0 0x560b57cb33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:15.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 4 client.26230 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:05:15.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.692+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560b57b84920 con 0x560b57cb0fd0 2023-09-10T01:05:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.693+0000 7efd9bfff700 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57cb0fd0 0x560b57cb33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.693+0000 7efd9bfff700 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57cb0fd0 0x560b57cb33c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c1d60c7a905b6e6 server_cookie=48517a7299374d60 in_seq=0 out_seq=0 2023-09-10T01:05:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.693+0000 7efd89ffb700 10 client.26230 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efd89ffb700 1 -- 172.21.15.3:0/2725157421 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560b57b84920 con 0x560b57cb0fd0 2023-09-10T01:05:15.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efd89ffb700 10 client.26230 handle_command_reply: tid=0 2023-09-10T01:05:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 client.26230 shutdown 2023-09-10T01:05:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 msgr2=0x560b57c52f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c52f60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x560b57c5eac0 tx=0x7efd94009310 comp rx=0 tx=0).stop 2023-09-10T01:05:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 shutdown_connections 2023-09-10T01:05:15.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560b57cb0fd0 0x560b57cb33c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560b57c39710 0x560b57c52f60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560b57c4c650 0x560b57c517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efd89ffb700 0 client.26230 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 --2- 172.21.15.3:0/2725157421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560b57c3a030 0x560b57c51290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 >> 172.21.15.3:0/2725157421 conn(0x560b57bae090 msgr2=0x560b57c34670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 shutdown_connections 2023-09-10T01:05:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 1 -- 172.21.15.3:0/2725157421 wait complete. 2023-09-10T01:05:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.695+0000 7efdbc846b80 20 client.26230 trim_cache size 0 max 16384 2023-09-10T01:05:15.710 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:05:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efd900a07b0 msgr2=0x7efd900a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efd900a07b0 0x7efd900a2c70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efd94006010 tx=0x7efd9400b260 comp rx=0 tx=0).stop 2023-09-10T01:05:15.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda410ce80 msgr2=0x7efda419d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda410ce80 0x7efda419d370 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd9c007f10 tx=0x7efd9c00d3b0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 shutdown_connections 2023-09-10T01:05:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efd900a07b0 0x7efd900a2c70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda410ce80 0x7efda419d370 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efda4072a30 0x7efda41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 --2- 172.21.15.3:0/4169024694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efda4073350 0x7efda4198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 >> 172.21.15.3:0/4169024694 conn(0x7efda40fc170 msgr2=0x7efda410f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 shutdown_connections 2023-09-10T01:05:15.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:15.696+0000 7efda8fff700 1 -- 172.21.15.3:0/4169024694 wait complete. 2023-09-10T01:05:15.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 105352 10.22 MB/sec warmup 57 sec latency 34.325 ms 2023-09-10T01:05:16.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:16 smithi136 ceph-mon[133708]: pgmap v64: 129 pgs: 129 active+clean; 44 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 348 op/s 2023-09-10T01:05:16.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:16 smithi092 ceph-mon[134067]: pgmap v64: 129 pgs: 129 active+clean; 44 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 348 op/s 2023-09-10T01:05:16.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:16 smithi003 ceph-mon[97654]: pgmap v64: 129 pgs: 129 active+clean; 44 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 348 op/s 2023-09-10T01:05:16.716 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:05:16.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:05:16.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107053 10.25 MB/sec warmup 58 sec latency 15.032 ms 2023-09-10T01:05:16.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 -- 172.21.15.3:0/2860120249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 msgr2=0x7f23f8072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:16.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/2860120249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f8072e10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009990 tx=0x7f23e8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 -- 172.21.15.3:0/2860120249 shutdown_connections 2023-09-10T01:05:16.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/2860120249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23f810ce50 0x7f23f810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/2860120249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f8072e10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/2860120249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f8073350 0x7f23f810c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 -- 172.21.15.3:0/2860120249 >> 172.21.15.3:0/2860120249 conn(0x7f23f80fc100 msgr2=0x7f23f80fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 -- 172.21.15.3:0/2860120249 shutdown_connections 2023-09-10T01:05:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.942+0000 7f23fe7ff700 1 -- 172.21.15.3:0/2860120249 wait complete. 2023-09-10T01:05:16.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.943+0000 7f23fe7ff700 1 Processor -- start 2023-09-10T01:05:16.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.943+0000 7f23fe7ff700 1 -- start start 2023-09-10T01:05:16.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fe7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fe7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f8073350 0x7f23f819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fe7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23f810ce50 0x7f23f8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fe7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8198e00 con 0x7f23f810ce50 2023-09-10T01:05:16.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fe7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8198f70 con 0x7f23f8072a30 2023-09-10T01:05:16.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fe7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8199130 con 0x7f23f8073350 2023-09-10T01:05:16.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f819e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57580/0 (socket says 172.21.15.3:57580) 2023-09-10T01:05:16.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 -- 172.21.15.3:0/433920014 learned_addr learned my addr 172.21.15.3:0/433920014 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:16.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f77fe700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f8073350 0x7f23f819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23f810ce50 0x7f23f8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 -- 172.21.15.3:0/433920014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f8073350 msgr2=0x7f23f819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:16.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f8073350 0x7f23f819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 -- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23f810ce50 msgr2=0x7f23f8198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:16.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23f810ce50 0x7f23f8198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23f7fff700 1 -- 172.21.15.3:0/433920014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80095f0 con 0x7f23f8072a30 2023-09-10T01:05:16.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.944+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23f810ce50 0x7f23f8198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:16.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.945+0000 7f23f7fff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f819e720 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f23e8000c00 tx=0x7f23e8004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:16.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.945+0000 7f23f57fa700 1 -- 172.21.15.3:0/433920014 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e801d070 con 0x7f23f8072a30 2023-09-10T01:05:16.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.945+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f81993b0 con 0x7f23f8072a30 2023-09-10T01:05:16.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.945+0000 7f23f57fa700 1 -- 172.21.15.3:0/433920014 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e8021410 con 0x7f23f8072a30 2023-09-10T01:05:16.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.945+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f81a5710 con 0x7f23f8072a30 2023-09-10T01:05:16.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.945+0000 7f23f57fa700 1 -- 172.21.15.3:0/433920014 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8017650 con 0x7f23f8072a30 2023-09-10T01:05:16.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.950+0000 7f23f57fa700 1 -- 172.21.15.3:0/433920014 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f23e8021580 con 0x7f23f8072a30 2023-09-10T01:05:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.951+0000 7f2412086b80 1 Processor -- start 2023-09-10T01:05:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.951+0000 7f2412086b80 1 -- start start 2023-09-10T01:05:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f2412086b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa21ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f2412086b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa224980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f2412086b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa224ec0 0x5633fa2272b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f23f57fa700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f23e00a0830 0x7f23e00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f2412086b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa1737b0 con 0x5633fa224ec0 2023-09-10T01:05:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f2412086b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa1710d0 con 0x5633fa21a790 2023-09-10T01:05:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f2412086b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa16c9c0 con 0x5633fa21b0b0 2023-09-10T01:05:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f23f57fa700 1 -- 172.21.15.3:0/433920014 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f23e80cd6a0 con 0x7f23f8072a30 2023-09-10T01:05:16.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f23fcd9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa21ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f23fcd9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa21ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57586/0 (socket says 172.21.15.3:57586) 2023-09-10T01:05:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f23fcd9c700 1 -- 172.21.15.3:0/3120344559 learned_addr learned my addr 172.21.15.3:0/3120344559 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa224980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.952+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa224ec0 0x5633fa2272b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23f7fff700 1 -- 172.21.15.3:0/3120344559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 msgr2=0x5633fa21ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa21ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23f7fff700 1 -- 172.21.15.3:0/3120344559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa224ec0 msgr2=0x5633fa2272b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa224ec0 0x5633fa2272b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23f7fff700 1 -- 172.21.15.3:0/3120344559 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633f9f98e60 con 0x5633fa21b0b0 2023-09-10T01:05:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23f77fe700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f23e00a0830 0x7f23e00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa224ec0 0x5633fa2272b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:16.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa224980 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23e80caa60 tx=0x7f23e8096c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a156374a41398c12 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:16.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23de7fc700 1 -- 172.21.15.3:0/3120344559 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e809a1b0 con 0x5633fa21b0b0 2023-09-10T01:05:16.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23de7fc700 1 -- 172.21.15.3:0/3120344559 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e809af50 con 0x5633fa21b0b0 2023-09-10T01:05:16.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.953+0000 7f23de7fc700 1 -- 172.21.15.3:0/3120344559 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8098c80 con 0x5633fa21b0b0 2023-09-10T01:05:16.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f2412086b80 1 -- 172.21.15.3:0/3120344559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 msgr2=0x5633fa224980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:16.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f23f77fe700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f23e00a0830 0x7f23e00a2cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f23f8199c00 tx=0x7f23ec009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:16.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f2412086b80 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa224980 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23e80caa60 tx=0x7f23e8096c10 comp rx=0 tx=0).stop 2023-09-10T01:05:16.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f2412086b80 1 -- 172.21.15.3:0/3120344559 shutdown_connections 2023-09-10T01:05:16.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f2412086b80 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa224ec0 0x5633fa2272b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f2412086b80 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa21ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f2412086b80 1 --2- 172.21.15.3:0/3120344559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa224980 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.954+0000 7f2412086b80 1 -- 172.21.15.3:0/3120344559 >> 172.21.15.3:0/3120344559 conn(0x5633fa18f110 msgr2=0x5633fa215c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:16.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.955+0000 7f2412086b80 1 -- 172.21.15.3:0/3120344559 shutdown_connections 2023-09-10T01:05:16.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.955+0000 7f2412086b80 1 -- 172.21.15.3:0/3120344559 wait complete. 2023-09-10T01:05:16.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.956+0000 7f2412086b80 1 -- start start 2023-09-10T01:05:16.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.956+0000 7f2412086b80 1 Processor -- start 2023-09-10T01:05:16.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f2412086b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa2252a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f2412086b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa2257e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f2412086b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa224ec0 0x5633fa230320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f2412086b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa10abf0 con 0x5633fa21a790 2023-09-10T01:05:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f2412086b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa10a8f0 con 0x5633fa224ec0 2023-09-10T01:05:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f2412086b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f9fee9a0 con 0x5633fa21b0b0 2023-09-10T01:05:16.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f2412086b80 10 client.0 _ll_register_callbacks cb 0x5633fa054b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa2257e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23fcd9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa224ec0 0x5633fa230320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23f7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa2252a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa2257e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57338/0 (socket says 172.21.15.3:57338) 2023-09-10T01:05:16.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23fcd9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa224ec0 0x5633fa230320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57592/0 (socket says 172.21.15.3:57592) 2023-09-10T01:05:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23f77fe700 1 -- 172.21.15.3:0/2599283217 learned_addr learned my addr 172.21.15.3:0/2599283217 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23fcd9c700 1 -- 172.21.15.3:0/2599283217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 msgr2=0x5633fa2257e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa2257e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23fcd9c700 1 -- 172.21.15.3:0/2599283217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 msgr2=0x5633fa2252a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa2252a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.957+0000 7f23fcd9c700 1 -- 172.21.15.3:0/2599283217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f81a5710 con 0x5633fa224ec0 2023-09-10T01:05:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa224ec0 0x5633fa230320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f23f0012c30 tx=0x7f23f00124b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acca6d470871a9f4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f23d77fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f23d77fe700 1 -- 172.21.15.3:0/2599283217 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f000e040 con 0x5633fa224ec0 2023-09-10T01:05:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633fa0c4570 con 0x5633fa224ec0 2023-09-10T01:05:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f23d77fe700 1 -- 172.21.15.3:0/2599283217 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f23f0003a00 con 0x5633fa224ec0 2023-09-10T01:05:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f23d77fe700 1 -- 172.21.15.3:0/2599283217 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23f0003bc0 con 0x5633fa224ec0 2023-09-10T01:05:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f2412086b80 10 client.26324 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5633f9f81ce0 con 0x5633fa224ec0 2023-09-10T01:05:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.958+0000 7f23d77fe700 1 -- 172.21.15.3:0/2599283217 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5633fa0c4570 con 0x5633fa224ec0 2023-09-10T01:05:16.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f23d77fe700 1 -- 172.21.15.3:0/2599283217 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f23f0004350 con 0x5633fa224ec0 2023-09-10T01:05:16.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f2412086b80 10 client.26324 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:16.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f2412086b80 10 client.26324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:16.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f2412086b80 20 client.26324 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:16.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f2412086b80 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa234690 0x5633fa236ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f2412086b80 4 client.26324 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:16.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5633fa181920 con 0x5633fa234690 2023-09-10T01:05:16.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f23f7fff700 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa234690 0x5633fa236ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f23f7fff700 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa234690 0x5633fa236ab0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5c93b5f058f8e1e server_cookie=bd56959a09ca20 in_seq=0 out_seq=0 2023-09-10T01:05:16.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.959+0000 7f23d77fe700 10 client.26324 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:16.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.960+0000 7f23d77fe700 1 -- 172.21.15.3:0/2599283217 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5633fa181920 con 0x5633fa234690 2023-09-10T01:05:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.960+0000 7f23d77fe700 10 client.26324 handle_command_reply: tid=0 2023-09-10T01:05:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.960+0000 7f2412086b80 1 client.26324 shutdown 2023-09-10T01:05:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.960+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa224ec0 msgr2=0x5633fa230320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.960+0000 7f2412086b80 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa224ec0 0x5633fa230320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f23f0012c30 tx=0x7f23f00124b0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.960+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 shutdown_connections 2023-09-10T01:05:16.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.960+0000 7f2412086b80 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa234690 0x5633fa236ab0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f2412086b80 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa224ec0 0x5633fa230320 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f2412086b80 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa2252a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f2412086b80 1 --2- 172.21.15.3:0/2599283217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa21b0b0 0x5633fa2257e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 >> 172.21.15.3:0/2599283217 conn(0x5633fa18f110 msgr2=0x5633fa215800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:16.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f23d77fe700 0 client.26324 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:16.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 shutdown_connections 2023-09-10T01:05:16.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f2412086b80 1 -- 172.21.15.3:0/2599283217 wait complete. 2023-09-10T01:05:16.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.961+0000 7f2412086b80 20 client.26324 trim_cache size 0 max 16384 2023-09-10T01:05:16.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 Processor -- start 2023-09-10T01:05:16.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 -- start start 2023-09-10T01:05:16.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa22a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21b0b0 0x5633fa21f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa21f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa22a820 con 0x5633fa21b0b0 2023-09-10T01:05:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa21b7f0 con 0x5633fa21a790 2023-09-10T01:05:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.980+0000 7f2412086b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa224b80 con 0x5633fa224ec0 2023-09-10T01:05:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa22a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa22a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57594/0 (socket says 172.21.15.3:57594) 2023-09-10T01:05:16.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 -- 172.21.15.3:0/313998251 learned_addr learned my addr 172.21.15.3:0/313998251 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:16.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f77fe700 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21b0b0 0x5633fa21f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa21f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 -- 172.21.15.3:0/313998251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 msgr2=0x5633fa21f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa21f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 -- 172.21.15.3:0/313998251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21b0b0 msgr2=0x5633fa21f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21b0b0 0x5633fa21f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 -- 172.21.15.3:0/313998251 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633f9fbf2d0 con 0x5633fa21a790 2023-09-10T01:05:16.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23f7fff700 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa22a2e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f23e809aa10 tx=0x7f23e8005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6e3560d330894 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:16.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23deffd700 1 -- 172.21.15.3:0/313998251 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e80b6070 con 0x5633fa21a790 2023-09-10T01:05:16.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23deffd700 1 -- 172.21.15.3:0/313998251 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e809e700 con 0x5633fa21a790 2023-09-10T01:05:16.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.981+0000 7f23deffd700 1 -- 172.21.15.3:0/313998251 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e80cd2f0 con 0x5633fa21a790 2023-09-10T01:05:16.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 -- 172.21.15.3:0/313998251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 msgr2=0x5633fa22a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:16.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa22a2e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f23e809aa10 tx=0x7f23e8005f10 comp rx=0 tx=0).stop 2023-09-10T01:05:16.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 -- 172.21.15.3:0/313998251 shutdown_connections 2023-09-10T01:05:16.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa21f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21a790 0x5633fa22a2e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 --2- 172.21.15.3:0/313998251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21b0b0 0x5633fa21f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 -- 172.21.15.3:0/313998251 >> 172.21.15.3:0/313998251 conn(0x5633fa18f110 msgr2=0x5633fa2252a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:16.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 -- 172.21.15.3:0/313998251 shutdown_connections 2023-09-10T01:05:16.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.982+0000 7f2412086b80 1 -- 172.21.15.3:0/313998251 wait complete. 2023-09-10T01:05:16.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.983+0000 7f2412086b80 1 -- start start 2023-09-10T01:05:16.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.983+0000 7f2412086b80 1 Processor -- start 2023-09-10T01:05:16.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.983+0000 7f2412086b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa228a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.983+0000 7f2412086b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21b0b0 0x5633fa228fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f2412086b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa288490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f2412086b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa229890 con 0x5633fa21a790 2023-09-10T01:05:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f2412086b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa2889d0 con 0x5633fa21b0b0 2023-09-10T01:05:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f2412086b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fa288b10 con 0x5633fa224ec0 2023-09-10T01:05:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f2412086b80 10 client.0 _ll_register_callbacks cb 0x5633fa054b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21b0b0 0x5633fa228fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21b0b0 0x5633fa228fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57600/0 (socket says 172.21.15.3:57600) 2023-09-10T01:05:16.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 -- 172.21.15.3:0/3849713900 learned_addr learned my addr 172.21.15.3:0/3849713900 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:16.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23fcd9c700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa288490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa228a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 -- 172.21.15.3:0/3849713900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 msgr2=0x5633fa288490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa288490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 -- 172.21.15.3:0/3849713900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 msgr2=0x5633fa228a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa228a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 -- 172.21.15.3:0/3849713900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633f9f98e60 con 0x5633fa21b0b0 2023-09-10T01:05:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa228a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23f77fe700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21b0b0 0x5633fa228fd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f23ec003900 tx=0x7f23ec0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff5bd8565044a493 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.984+0000 7f23d77fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f23d77fe700 1 -- 172.21.15.3:0/3849713900 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec01d610 con 0x5633fa21b0b0 2023-09-10T01:05:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23e809af50 con 0x5633fa21b0b0 2023-09-10T01:05:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f23d77fe700 1 -- 172.21.15.3:0/3849713900 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f23ec01d770 con 0x5633fa21b0b0 2023-09-10T01:05:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f23d77fe700 1 -- 172.21.15.3:0/3849713900 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23ec01d930 con 0x5633fa21b0b0 2023-09-10T01:05:16.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 10 client.26330 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:16.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5633f9f97330 con 0x5633fa21b0b0 2023-09-10T01:05:16.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f23d77fe700 1 -- 172.21.15.3:0/3849713900 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e809af50 con 0x5633fa21b0b0 2023-09-10T01:05:16.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f23d77fe700 1 -- 172.21.15.3:0/3849713900 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f23ec027bd0 con 0x5633fa21b0b0 2023-09-10T01:05:16.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 10 client.26330 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 10 client.26330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 20 client.26330 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa28a420 0x5633fa28c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 4 client.26330 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:05:16.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.985+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5633fa1659a0 con 0x5633fa28a420 2023-09-10T01:05:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.986+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa28a420 0x5633fa28c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.986+0000 7f23f7fff700 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa28a420 0x5633fa28c840 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3df72acfe45b92fc server_cookie=7c743dc475373034 in_seq=0 out_seq=0 2023-09-10T01:05:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.986+0000 7f23d77fe700 10 client.26330 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f23d77fe700 1 -- 172.21.15.3:0/3849713900 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5633fa1659a0 con 0x5633fa28a420 2023-09-10T01:05:16.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f23d77fe700 10 client.26330 handle_command_reply: tid=0 2023-09-10T01:05:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 client.26330 shutdown 2023-09-10T01:05:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21b0b0 msgr2=0x5633fa228fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21b0b0 0x5633fa228fd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f23ec003900 tx=0x7f23ec0047c0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 shutdown_connections 2023-09-10T01:05:17.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633fa28a420 0x5633fa28c840 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633fa224ec0 0x5633fa288490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633fa21a790 0x5633fa228a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 --2- 172.21.15.3:0/3849713900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633fa21b0b0 0x5633fa228fd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 >> 172.21.15.3:0/3849713900 conn(0x5633fa18f110 msgr2=0x5633fa2252a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f23d77fe700 0 client.26330 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 shutdown_connections 2023-09-10T01:05:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.987+0000 7f2412086b80 1 -- 172.21.15.3:0/3849713900 wait complete. 2023-09-10T01:05:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.988+0000 7f2412086b80 20 client.26330 trim_cache size 0 max 16384 2023-09-10T01:05:17.002 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:05:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:05:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "672e92da-c40f-4d5d-bcaf-b216c32b5c3b", 2023-09-10T01:05:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:05:17.003 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:05:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.988+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f23e00a0830 msgr2=0x7f23e00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.988+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f23e00a0830 0x7f23e00a2cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f23f8199c00 tx=0x7f23ec009260 comp rx=0 tx=0).stop 2023-09-10T01:05:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.988+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 msgr2=0x7f23f819e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.988+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f819e720 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f23e8000c00 tx=0x7f23e8004790 comp rx=0 tx=0).stop 2023-09-10T01:05:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 shutdown_connections 2023-09-10T01:05:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f23e00a0830 0x7f23e00a2cf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23f810ce50 0x7f23f8198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23f8072a30 0x7f23f819e720 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 --2- 172.21.15.3:0/433920014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f8073350 0x7f23f819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 >> 172.21.15.3:0/433920014 conn(0x7f23f80fc100 msgr2=0x7f23f810f090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 shutdown_connections 2023-09-10T01:05:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:16.989+0000 7f23fe7ff700 1 -- 172.21.15.3:0/433920014 wait complete. 2023-09-10T01:05:17.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:05:17.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 -- 172.21.15.3:0/3469334520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130100640 msgr2=0x7fe130100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 --2- 172.21.15.3:0/3469334520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130100640 0x7fe130100ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe124009990 tx=0x7fe124009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 -- 172.21.15.3:0/3469334520 shutdown_connections 2023-09-10T01:05:17.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 --2- 172.21.15.3:0/3469334520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe130101000 0x7fe13010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 --2- 172.21.15.3:0/3469334520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe130106650 0x7fe130106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 --2- 172.21.15.3:0/3469334520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130100640 0x7fe130100ac0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 -- 172.21.15.3:0/3469334520 >> 172.21.15.3:0/3469334520 conn(0x7fe1300fc170 msgr2=0x7fe1300fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 -- 172.21.15.3:0/3469334520 shutdown_connections 2023-09-10T01:05:17.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.266+0000 7fe1365ff700 1 -- 172.21.15.3:0/3469334520 wait complete. 2023-09-10T01:05:17.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.267+0000 7fe1365ff700 1 Processor -- start 2023-09-10T01:05:17.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe1365ff700 1 -- start start 2023-09-10T01:05:17.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe1365ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe130100640 0x7fe13019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe1365ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130101000 0x7fe13019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe1365ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe130106650 0x7fe130198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe1365ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe130198db0 con 0x7fe130100640 2023-09-10T01:05:17.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe1365ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe130198f20 con 0x7fe130101000 2023-09-10T01:05:17.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe1365ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301990e0 con 0x7fe130106650 2023-09-10T01:05:17.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe12f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130101000 0x7fe13019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe12f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130101000 0x7fe13019ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57622/0 (socket says 172.21.15.3:57622) 2023-09-10T01:05:17.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe12f7fe700 1 -- 172.21.15.3:0/1020682666 learned_addr learned my addr 172.21.15.3:0/1020682666 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:17.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe134b9c700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe130106650 0x7fe130198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.268+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe130100640 0x7fe13019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12f7fe700 1 -- 172.21.15.3:0/1020682666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe130106650 msgr2=0x7fe130198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe130106650 0x7fe130198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12f7fe700 1 -- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe130100640 msgr2=0x7fe13019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe130100640 0x7fe13019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12f7fe700 1 -- 172.21.15.3:0/1020682666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1240095f0 con 0x7fe130101000 2023-09-10T01:05:17.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130101000 0x7fe13019ec10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe124005910 tx=0x7fe124005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:17.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12d7fa700 1 -- 172.21.15.3:0/1020682666 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12401d070 con 0x7fe130101000 2023-09-10T01:05:17.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe130199360 con 0x7fe130101000 2023-09-10T01:05:17.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12d7fa700 1 -- 172.21.15.3:0/1020682666 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe124022470 con 0x7fe130101000 2023-09-10T01:05:17.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.269+0000 7fe12d7fa700 1 -- 172.21.15.3:0/1020682666 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124017650 con 0x7fe130101000 2023-09-10T01:05:17.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.270+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1301a56c0 con 0x7fe130101000 2023-09-10T01:05:17.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.273+0000 7fe12d7fa700 1 -- 172.21.15.3:0/1020682666 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe1240225e0 con 0x7fe130101000 2023-09-10T01:05:17.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.274+0000 7fe149ff4b80 1 Processor -- start 2023-09-10T01:05:17.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe149ff4b80 1 -- start start 2023-09-10T01:05:17.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d45af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d52230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016c9e730 con 0x557016d4fe40 2023-09-10T01:05:17.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016c9c050 con 0x557016d45710 2023-09-10T01:05:17.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016c97940 con 0x557016d46030 2023-09-10T01:05:17.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe134b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d45af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe134b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d45af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57634/0 (socket says 172.21.15.3:57634) 2023-09-10T01:05:17.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe134b9c700 1 -- 172.21.15.3:0/683077070 learned_addr learned my addr 172.21.15.3:0/683077070 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:17.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe12ffff700 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.275+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d52230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe134b9c700 1 -- 172.21.15.3:0/683077070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 msgr2=0x557016d4f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe134b9c700 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe134b9c700 1 -- 172.21.15.3:0/683077070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 msgr2=0x557016d52230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe134b9c700 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d52230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe134b9c700 1 -- 172.21.15.3:0/683077070 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557016ac3de0 con 0x557016d45710 2023-09-10T01:05:17.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d52230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:17.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe12d7fa700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1180a07b0 0x7fe1180a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe134b9c700 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d45af0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe128011600 tx=0x7fe128009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=350c85223dbfdd30 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:17.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe12d7fa700 1 -- 172.21.15.3:0/1020682666 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe1240cd740 con 0x7fe130101000 2023-09-10T01:05:17.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1180a07b0 0x7fe1180a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe1167fc700 1 -- 172.21.15.3:0/683077070 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12800e040 con 0x557016d45710 2023-09-10T01:05:17.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe1167fc700 1 -- 172.21.15.3:0/683077070 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe12800f040 con 0x557016d45710 2023-09-10T01:05:17.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.276+0000 7fe1167fc700 1 -- 172.21.15.3:0/683077070 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128004760 con 0x557016d45710 2023-09-10T01:05:17.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 -- 172.21.15.3:0/683077070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 msgr2=0x557016d45af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1180a07b0 0x7fe1180a2c70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x557016d3b280 tx=0x7fe120009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:17.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d45af0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe128011600 tx=0x7fe128009920 comp rx=0 tx=0).stop 2023-09-10T01:05:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 -- 172.21.15.3:0/683077070 shutdown_connections 2023-09-10T01:05:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d52230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d45af0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/683077070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 -- 172.21.15.3:0/683077070 >> 172.21.15.3:0/683077070 conn(0x557016cba090 msgr2=0x557016d40bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 -- 172.21.15.3:0/683077070 shutdown_connections 2023-09-10T01:05:17.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.277+0000 7fe149ff4b80 1 -- 172.21.15.3:0/683077070 wait complete. 2023-09-10T01:05:17.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.278+0000 7fe149ff4b80 1 -- start start 2023-09-10T01:05:17.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.278+0000 7fe149ff4b80 1 Processor -- start 2023-09-10T01:05:17.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d4cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016c35b70 con 0x557016d4fe40 2023-09-10T01:05:17.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016c35870 con 0x557016d45710 2023-09-10T01:05:17.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016b19920 con 0x557016d46030 2023-09-10T01:05:17.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe149ff4b80 10 client.0 _ll_register_callbacks cb 0x557016b7fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:17.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe12f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe12f7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57394/0 (socket says 172.21.15.3:57394) 2023-09-10T01:05:17.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe12f7fe700 1 -- 172.21.15.3:0/2286549558 learned_addr learned my addr 172.21.15.3:0/2286549558 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:17.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe134b9c700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d4cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.279+0000 7fe12ffff700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe12f7fe700 1 -- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 msgr2=0x557016d4f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe12f7fe700 1 -- 172.21.15.3:0/2286549558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 msgr2=0x557016d4cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d4cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe12f7fe700 1 -- 172.21.15.3:0/2286549558 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1280112b0 con 0x557016d46030 2023-09-10T01:05:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe134b9c700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d4cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4c9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe124006010 tx=0x7fe124009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1df4b12bdf651b19 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:17.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe10f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe10f7fe700 1 -- 172.21.15.3:0/2286549558 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1240cd590 con 0x557016d46030 2023-09-10T01:05:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557016bef4f0 con 0x557016d46030 2023-09-10T01:05:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe10f7fe700 1 -- 172.21.15.3:0/2286549558 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe12409b110 con 0x557016d46030 2023-09-10T01:05:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe10f7fe700 1 -- 172.21.15.3:0/2286549558 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe12409a7e0 con 0x557016d46030 2023-09-10T01:05:17.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe149ff4b80 10 client.26260 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.280+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557016aacc60 con 0x557016d46030 2023-09-10T01:05:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe10f7fe700 1 -- 172.21.15.3:0/2286549558 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557016bef4f0 con 0x557016d46030 2023-09-10T01:05:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe10f7fe700 1 -- 172.21.15.3:0/2286549558 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe124099f00 con 0x557016d46030 2023-09-10T01:05:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe149ff4b80 10 client.26260 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:17.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe149ff4b80 10 client.26260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:17.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe149ff4b80 20 client.26260 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:17.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016d5f550 0x557016d61970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe149ff4b80 4 client.26260 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:17.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557016cac8a0 con 0x557016d5f550 2023-09-10T01:05:17.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.281+0000 7fe12ffff700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016d5f550 0x557016d61970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.282+0000 7fe12ffff700 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016d5f550 0x557016d61970 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b48cc098b740bd9e server_cookie=bf54d63eb82afca9 in_seq=0 out_seq=0 2023-09-10T01:05:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.282+0000 7fe10f7fe700 10 client.26260 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.282+0000 7fe10f7fe700 1 -- 172.21.15.3:0/2286549558 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557016cac8a0 con 0x557016d5f550 2023-09-10T01:05:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.282+0000 7fe10f7fe700 10 client.26260 handle_command_reply: tid=0 2023-09-10T01:05:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.282+0000 7fe149ff4b80 1 client.26260 shutdown 2023-09-10T01:05:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 msgr2=0x557016d4c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4c9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe124006010 tx=0x7fe124009310 comp rx=0 tx=0).stop 2023-09-10T01:05:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 shutdown_connections 2023-09-10T01:05:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016d5f550 0x557016d61970 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d4cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/2286549558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d4c9c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe10f7fe700 0 client.26260 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:17.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 >> 172.21.15.3:0/2286549558 conn(0x557016cba090 msgr2=0x557016d407e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 shutdown_connections 2023-09-10T01:05:17.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 1 -- 172.21.15.3:0/2286549558 wait complete. 2023-09-10T01:05:17.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.283+0000 7fe149ff4b80 20 client.26260 trim_cache size 0 max 16384 2023-09-10T01:05:17.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.302+0000 7fe149ff4b80 1 Processor -- start 2023-09-10T01:05:17.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.302+0000 7fe149ff4b80 1 -- start start 2023-09-10T01:05:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d3f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d44f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016d46700 con 0x557016d4fe40 2023-09-10T01:05:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016d4d940 con 0x557016d45710 2023-09-10T01:05:17.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016d4c7a0 con 0x557016d46030 2023-09-10T01:05:17.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57646/0 (socket says 172.21.15.3:57646) 2023-09-10T01:05:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 -- 172.21.15.3:0/1338687381 learned_addr learned my addr 172.21.15.3:0/1338687381 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe134b9c700 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d44f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d3f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 -- 172.21.15.3:0/1338687381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 msgr2=0x557016d3f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d3f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 -- 172.21.15.3:0/1338687381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 msgr2=0x557016d44f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d44f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.303+0000 7fe12ffff700 1 -- 172.21.15.3:0/1338687381 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557016aea250 con 0x557016d45710 2023-09-10T01:05:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.304+0000 7fe12ffff700 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4c260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x557016d62280 tx=0x7fe12001b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b6f44145e029668 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.304+0000 7fe116ffd700 1 -- 172.21.15.3:0/1338687381 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120024040 con 0x557016d45710 2023-09-10T01:05:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.304+0000 7fe116ffd700 1 -- 172.21.15.3:0/1338687381 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1200293a0 con 0x557016d45710 2023-09-10T01:05:17.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.304+0000 7fe116ffd700 1 -- 172.21.15.3:0/1338687381 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120017070 con 0x557016d45710 2023-09-10T01:05:17.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 -- 172.21.15.3:0/1338687381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 msgr2=0x557016d4c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4c260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x557016d62280 tx=0x7fe12001b000 comp rx=0 tx=0).stop 2023-09-10T01:05:17.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 -- 172.21.15.3:0/1338687381 shutdown_connections 2023-09-10T01:05:17.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d4fe40 0x557016d44f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d45710 0x557016d4c260 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/1338687381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d46030 0x557016d3f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 -- 172.21.15.3:0/1338687381 >> 172.21.15.3:0/1338687381 conn(0x557016cba090 msgr2=0x557016d662d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 -- 172.21.15.3:0/1338687381 shutdown_connections 2023-09-10T01:05:17.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 -- 172.21.15.3:0/1338687381 wait complete. 2023-09-10T01:05:17.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.305+0000 7fe149ff4b80 1 -- start start 2023-09-10T01:05:17.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 1 Processor -- start 2023-09-10T01:05:17.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d45710 0x557016d56be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d46030 0x557016d54f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d4fe40 0x557016d55650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016d55d60 con 0x557016d45710 2023-09-10T01:05:17.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016db3390 con 0x557016d46030 2023-09-10T01:05:17.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557016db34d0 con 0x557016d4fe40 2023-09-10T01:05:17.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.306+0000 7fe149ff4b80 10 client.0 _ll_register_callbacks cb 0x557016b7fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:17.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe12f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d46030 0x557016d54f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe12ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d45710 0x557016d56be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe12f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d46030 0x557016d54f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57654/0 (socket says 172.21.15.3:57654) 2023-09-10T01:05:17.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe12f7fe700 1 -- 172.21.15.3:0/559324347 learned_addr learned my addr 172.21.15.3:0/559324347 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:17.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe134b9c700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d4fe40 0x557016d55650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe134b9c700 1 -- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d46030 msgr2=0x557016d54f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe134b9c700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d46030 0x557016d54f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe134b9c700 1 -- 172.21.15.3:0/559324347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d45710 msgr2=0x557016d56be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:17.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe134b9c700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d45710 0x557016d56be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe134b9c700 1 -- 172.21.15.3:0/559324347 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe120004160 con 0x557016d4fe40 2023-09-10T01:05:17.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe12f7fe700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d46030 0x557016d54f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:17.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe12ffff700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d45710 0x557016d56be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:17.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe134b9c700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d4fe40 0x557016d55650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe12800b010 tx=0x7fe128009990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf61496a9ab3cb6e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:17.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.307+0000 7fe10f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:17.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe10f7fe700 1 -- 172.21.15.3:0/559324347 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12800e040 con 0x557016d4fe40 2023-09-10T01:05:17.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe10f7fe700 1 -- 172.21.15.3:0/559324347 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe128007710 con 0x557016d4fe40 2023-09-10T01:05:17.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe12800f040 con 0x557016d4fe40 2023-09-10T01:05:17.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe10f7fe700 1 -- 172.21.15.3:0/559324347 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1280078f0 con 0x557016d4fe40 2023-09-10T01:05:17.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 10 client.26266 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:17.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557016ac22b0 con 0x557016d4fe40 2023-09-10T01:05:17.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe10f7fe700 1 -- 172.21.15.3:0/559324347 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe12800f040 con 0x557016d4fe40 2023-09-10T01:05:17.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe10f7fe700 1 -- 172.21.15.3:0/559324347 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe1280056e0 con 0x557016d4fe40 2023-09-10T01:05:17.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 10 client.26266 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:17.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 10 client.26266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:17.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 20 client.26266 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:17.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016db4bb0 0x557016db6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:17.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 4 client.26266 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:05:17.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.308+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557016c90920 con 0x557016db4bb0 2023-09-10T01:05:17.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.309+0000 7fe12ffff700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016db4bb0 0x557016db6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:17.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.309+0000 7fe12ffff700 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016db4bb0 0x557016db6fd0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c74ac2dcdd6a46 server_cookie=8301159dec6229f2 in_seq=0 out_seq=0 2023-09-10T01:05:17.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.309+0000 7fe10f7fe700 10 client.26266 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:17.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe10f7fe700 1 -- 172.21.15.3:0/559324347 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557016c90920 con 0x557016db4bb0 2023-09-10T01:05:17.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe10f7fe700 10 client.26266 handle_command_reply: tid=0 2023-09-10T01:05:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 client.26266 shutdown 2023-09-10T01:05:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d4fe40 msgr2=0x557016d55650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d4fe40 0x557016d55650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe12800b010 tx=0x7fe128009990 comp rx=0 tx=0).stop 2023-09-10T01:05:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 shutdown_connections 2023-09-10T01:05:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557016db4bb0 0x557016db6fd0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557016d4fe40 0x557016d55650 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557016d45710 0x557016d56be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe10f7fe700 0 client.26266 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:17.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 --2- 172.21.15.3:0/559324347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557016d46030 0x557016d54f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 >> 172.21.15.3:0/559324347 conn(0x557016cba090 msgr2=0x557016d44f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 shutdown_connections 2023-09-10T01:05:17.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.310+0000 7fe149ff4b80 1 -- 172.21.15.3:0/559324347 wait complete. 2023-09-10T01:05:17.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe149ff4b80 20 client.26266 trim_cache size 0 max 16384 2023-09-10T01:05:17.325 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:05:17.325 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:05:17.326 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:05:17.326 INFO:teuthology.orchestra.run.smithi003.stdout: "672e92da-c40f-4d5d-bcaf-b216c32b5c3b": { 2023-09-10T01:05:17.326 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:05:17.326 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "672e92da-c40f-4d5d-bcaf-b216c32b5c3b", 2023-09-10T01:05:17.327 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:05:17.327 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:05:17.327 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:05:17.327 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:05:17.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1180a07b0 msgr2=0x7fe1180a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1180a07b0 0x7fe1180a2c70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x557016d3b280 tx=0x7fe120009260 comp rx=0 tx=0).stop 2023-09-10T01:05:17.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130101000 msgr2=0x7fe13019ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:17.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130101000 0x7fe13019ec10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe124005910 tx=0x7fe124005720 comp rx=0 tx=0).stop 2023-09-10T01:05:17.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 shutdown_connections 2023-09-10T01:05:17.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1180a07b0 0x7fe1180a2c70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe130106650 0x7fe130198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe130100640 0x7fe13019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 --2- 172.21.15.3:0/1020682666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe130101000 0x7fe13019ec10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:17.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.311+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 >> 172.21.15.3:0/1020682666 conn(0x7fe1300fc170 msgr2=0x7fe1300fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:17.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.312+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 shutdown_connections 2023-09-10T01:05:17.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:17.312+0000 7fe1365ff700 1 -- 172.21.15.3:0/1020682666 wait complete. 2023-09-10T01:05:17.334 INFO:tasks.cephfs.filesystem:scrub status for tag:672e92da-c40f-4d5d-bcaf-b216c32b5c3b - {'path': '/', 'tag': '672e92da-c40f-4d5d-bcaf-b216c32b5c3b', 'options': 'recursive,force'} 2023-09-10T01:05:17.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 109475 10.29 MB/sec warmup 59 sec latency 14.719 ms 2023-09-10T01:05:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: pgmap v65: 129 pgs: 129 active+clean; 44 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2023-09-10T01:05:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:05:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:18.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:18.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:18.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:18.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:18 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: pgmap v65: 129 pgs: 129 active+clean; 44 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2023-09-10T01:05:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:05:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:18.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:18.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:18 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: pgmap v65: 129 pgs: 129 active+clean; 44 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 181 op/s 2023-09-10T01:05:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:05:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:18.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:18.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:18.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:18.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:18 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:18.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 110914 10.24 MB/sec warmup 60 sec latency 17.691 ms 2023-09-10T01:05:19.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 113659 10.35 MB/sec warmup 61 sec latency 9.633 ms 2023-09-10T01:05:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:20 smithi136 ceph-mon[133708]: pgmap v66: 129 pgs: 129 active+clean; 41 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 239 op/s 2023-09-10T01:05:20.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:20 smithi092 ceph-mon[134067]: pgmap v66: 129 pgs: 129 active+clean; 41 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 239 op/s 2023-09-10T01:05:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:20 smithi003 ceph-mon[97654]: pgmap v66: 129 pgs: 129 active+clean; 41 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 239 op/s 2023-09-10T01:05:20.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 114669 10.24 MB/sec warmup 62 sec latency 22.061 ms 2023-09-10T01:05:21.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 117233 10.33 MB/sec warmup 63 sec latency 14.380 ms 2023-09-10T01:05:22.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:22 smithi136 ceph-mon[133708]: pgmap v67: 129 pgs: 129 active+clean; 41 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 320 op/s 2023-09-10T01:05:22.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:22 smithi092 ceph-mon[134067]: pgmap v67: 129 pgs: 129 active+clean; 41 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 320 op/s 2023-09-10T01:05:22.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:22 smithi003 ceph-mon[97654]: pgmap v67: 129 pgs: 129 active+clean; 41 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 320 op/s 2023-09-10T01:05:22.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 118269 10.22 MB/sec warmup 64 sec latency 22.717 ms 2023-09-10T01:05:23.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:23 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:05:23.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:23 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:23.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:23 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:23.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:23 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:05:23.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:23 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:23.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:23 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:23.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:23 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:05:23.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:23 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:23.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:23 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:23.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 120858 10.32 MB/sec warmup 65 sec latency 15.770 ms 2023-09-10T01:05:24.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:24 smithi136 ceph-mon[133708]: pgmap v68: 129 pgs: 129 active+clean; 41 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 237 op/s 2023-09-10T01:05:24.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:24 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:05:24.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:24 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:05:24.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:24 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:24.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:24 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:05:24.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:24 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:05:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:24 smithi092 ceph-mon[134067]: pgmap v68: 129 pgs: 129 active+clean; 41 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 237 op/s 2023-09-10T01:05:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:24 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:05:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:24 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:05:24.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:24 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:24.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:24 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:05:24.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:24 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:05:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:24 smithi003 ceph-mon[97654]: pgmap v68: 129 pgs: 129 active+clean; 41 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 237 op/s 2023-09-10T01:05:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:24 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:05:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:24 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:05:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:24 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:05:24.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:24 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:05:24.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:24 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:05:24.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 122139 10.25 MB/sec warmup 66 sec latency 10.162 ms 2023-09-10T01:05:25.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:25 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:05:25.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:25 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:05:25.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:25 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:05:25.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124560 10.30 MB/sec warmup 67 sec latency 10.534 ms 2023-09-10T01:05:26.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:26 smithi136 ceph-mon[133708]: pgmap v69: 129 pgs: 129 active+clean; 47 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 337 op/s 2023-09-10T01:05:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:26 smithi092 ceph-mon[134067]: pgmap v69: 129 pgs: 129 active+clean; 47 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 337 op/s 2023-09-10T01:05:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:26 smithi003 ceph-mon[97654]: pgmap v69: 129 pgs: 129 active+clean; 47 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 337 op/s 2023-09-10T01:05:26.742 INFO:tasks.workunit.client.0.smithi003.stdout: 1 126047 10.23 MB/sec warmup 68 sec latency 47.204 ms 2023-09-10T01:05:27.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128055 10.28 MB/sec warmup 69 sec latency 93.757 ms 2023-09-10T01:05:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 -- 172.21.15.3:0/3625889615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c1115b0 msgr2=0x7f054c1139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 --2- 172.21.15.3:0/3625889615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c1115b0 0x7f054c1139a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f054400b0a0 tx=0x7f054400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 -- 172.21.15.3:0/3625889615 shutdown_connections 2023-09-10T01:05:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 --2- 172.21.15.3:0/3625889615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c1115b0 0x7f054c1139a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 --2- 172.21.15.3:0/3625889615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 0x7f054c072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 --2- 172.21.15.3:0/3625889615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f054c0732c0 0x7f054c111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 -- 172.21.15.3:0/3625889615 >> 172.21.15.3:0/3625889615 conn(0x7f054c0fc0b0 msgr2=0x7f054c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 -- 172.21.15.3:0/3625889615 shutdown_connections 2023-09-10T01:05:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.063+0000 7f0551fff700 1 -- 172.21.15.3:0/3625889615 wait complete. 2023-09-10T01:05:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.064+0000 7f0551fff700 1 Processor -- start 2023-09-10T01:05:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.065+0000 7f0551fff700 1 -- start start 2023-09-10T01:05:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.065+0000 7f0551fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 0x7f054c19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.065+0000 7f0551fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c0732c0 0x7f054c19d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.065+0000 7f0551fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f054c1115b0 0x7f054c1a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.065+0000 7f0551fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1a1e80 con 0x7f054c0729a0 2023-09-10T01:05:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.065+0000 7f0551fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1a1ff0 con 0x7f054c1115b0 2023-09-10T01:05:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.065+0000 7f0551fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1a21b0 con 0x7f054c0732c0 2023-09-10T01:05:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 0x7f054c19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c0732c0 0x7f054c19d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c0732c0 0x7f054c19d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54308/0 (socket says 172.21.15.3:54308) 2023-09-10T01:05:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 -- 172.21.15.3:0/1040636738 learned_addr learned my addr 172.21.15.3:0/1040636738 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 0x7f054c19cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38594/0 (socket says 172.21.15.3:38594) 2023-09-10T01:05:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054bfff700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f054c1115b0 0x7f054c1a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 -- 172.21.15.3:0/1040636738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f054c1115b0 msgr2=0x7f054c1a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f054c1115b0 0x7f054c1a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 -- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 msgr2=0x7f054c19cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 0x7f054c19cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054affd700 1 -- 172.21.15.3:0/1040636738 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0544009d00 con 0x7f054c0732c0 2023-09-10T01:05:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.066+0000 7f054b7fe700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 0x7f054c19cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.067+0000 7f054affd700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c0732c0 0x7f054c19d3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f054000e9a0 tx=0x7f054000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.067+0000 7f0548ff9700 1 -- 172.21.15.3:0/1040636738 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540009790 con 0x7f054c0732c0 2023-09-10T01:05:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.067+0000 7f0548ff9700 1 -- 172.21.15.3:0/1040636738 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054000cbd0 con 0x7f054c0732c0 2023-09-10T01:05:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.067+0000 7f0548ff9700 1 -- 172.21.15.3:0/1040636738 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540018680 con 0x7f054c0732c0 2023-09-10T01:05:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.067+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c1a2490 con 0x7f054c0732c0 2023-09-10T01:05:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.068+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c1a9cd0 con 0x7f054c0732c0 2023-09-10T01:05:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.071+0000 7f0548ff9700 1 -- 172.21.15.3:0/1040636738 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f054000cd40 con 0x7f054c0732c0 2023-09-10T01:05:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.073+0000 7f0548ff9700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05340a0830 0x7f05340a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.073+0000 7f0565908b80 1 Processor -- start 2023-09-10T01:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.073+0000 7f0548ff9700 1 -- 172.21.15.3:0/1040636738 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0540014070 con 0x7f054c0732c0 2023-09-10T01:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.073+0000 7f054b7fe700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05340a0830 0x7f05340a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.073+0000 7f0565908b80 1 -- start start 2023-09-10T01:05:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f0565908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x563810384580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f0565908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381037e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f0565908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f0565908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638102d76d0 con 0x56381037ecc0 2023-09-10T01:05:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f0565908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638102d4ff0 con 0x5638103841a0 2023-09-10T01:05:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f0565908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638102d08e0 con 0x56381037e300 2023-09-10T01:05:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x563810384580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x563810384580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44384/0 (socket says 172.21.15.3:44384) 2023-09-10T01:05:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f054b7fe700 1 -- 172.21.15.3:0/3472829143 learned_addr learned my addr 172.21.15.3:0/3472829143 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f054bfff700 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.074+0000 7f054affd700 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381037e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f054b7fe700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05340a0830 0x7f05340a2cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f053c011490 tx=0x7f053c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f054b7fe700 1 -- 172.21.15.3:0/3472829143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 msgr2=0x56381037e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f054b7fe700 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381037e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f054b7fe700 1 -- 172.21.15.3:0/3472829143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 msgr2=0x56381038af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f054b7fe700 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f054b7fe700 1 -- 172.21.15.3:0/3472829143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638100fcd80 con 0x5638103841a0 2023-09-10T01:05:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f054b7fe700 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x563810384580 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f053c0043d0 tx=0x7f053c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de2266975074a2a3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f0531ffb700 1 -- 172.21.15.3:0/3472829143 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c003da0 con 0x5638103841a0 2023-09-10T01:05:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f0531ffb700 1 -- 172.21.15.3:0/3472829143 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f053c0268c0 con 0x5638103841a0 2023-09-10T01:05:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.075+0000 7f0531ffb700 1 -- 172.21.15.3:0/3472829143 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c01e8b0 con 0x5638103841a0 2023-09-10T01:05:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 -- 172.21.15.3:0/3472829143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 msgr2=0x563810384580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x563810384580 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f053c0043d0 tx=0x7f053c014000 comp rx=0 tx=0).stop 2023-09-10T01:05:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 -- 172.21.15.3:0/3472829143 shutdown_connections 2023-09-10T01:05:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x563810384580 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 --2- 172.21.15.3:0/3472829143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381037e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 -- 172.21.15.3:0/3472829143 >> 172.21.15.3:0/3472829143 conn(0x5638102f3030 msgr2=0x563810379560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 -- 172.21.15.3:0/3472829143 shutdown_connections 2023-09-10T01:05:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.076+0000 7f0565908b80 1 -- 172.21.15.3:0/3472829143 wait complete. 2023-09-10T01:05:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.077+0000 7f0565908b80 1 -- start start 2023-09-10T01:05:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.077+0000 7f0565908b80 1 Processor -- start 2023-09-10T01:05:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f0565908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f0565908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810388010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f0565908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x5638103885b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f0565908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56381026eb10 con 0x56381037e300 2023-09-10T01:05:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f0565908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56381026e810 con 0x56381037ecc0 2023-09-10T01:05:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f0565908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638101528c0 con 0x5638103841a0 2023-09-10T01:05:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f0565908b80 10 client.0 _ll_register_callbacks cb 0x5638101b8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f054affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810388010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f054affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810388010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44398/0 (socket says 172.21.15.3:44398) 2023-09-10T01:05:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f054affd700 1 -- 172.21.15.3:0/2983954779 learned_addr learned my addr 172.21.15.3:0/2983954779 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f054bfff700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x5638103885b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.078+0000 7f054b7fe700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f054affd700 1 -- 172.21.15.3:0/2983954779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 msgr2=0x5638103885b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f054affd700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x5638103885b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f054affd700 1 -- 172.21.15.3:0/2983954779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 msgr2=0x56381038ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f054affd700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f054affd700 1 -- 172.21.15.3:0/2983954779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c017070 con 0x56381037ecc0 2023-09-10T01:05:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f054bfff700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x5638103885b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f054affd700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810388010 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f054001e000 tx=0x7f0540099e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10655f869d2fc2d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f052affd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f052affd700 1 -- 172.21.15.3:0/2983954779 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400a1640 con 0x56381037ecc0 2023-09-10T01:05:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f052affd700 1 -- 172.21.15.3:0/2983954779 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f05400cc150 con 0x56381037ecc0 2023-09-10T01:05:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563810228490 con 0x56381037ecc0 2023-09-10T01:05:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f052affd700 1 -- 172.21.15.3:0/2983954779 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05400cc310 con 0x56381037ecc0 2023-09-10T01:05:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f0565908b80 10 client.26360 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5638100e5c00 con 0x56381037ecc0 2023-09-10T01:05:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.079+0000 7f052affd700 1 -- 172.21.15.3:0/2983954779 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563810228490 con 0x56381037ecc0 2023-09-10T01:05:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f052affd700 1 -- 172.21.15.3:0/2983954779 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f05400b1bd0 con 0x56381037ecc0 2023-09-10T01:05:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f0565908b80 10 client.26360 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f0565908b80 10 client.26360 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f0565908b80 20 client.26360 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f0565908b80 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103a9400 0x5638103ab820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f0565908b80 4 client.26360 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638102e5840 con 0x5638103a9400 2023-09-10T01:05:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f054b7fe700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103a9400 0x5638103ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f052affd700 10 client.26360 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.080+0000 7f054b7fe700 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103a9400 0x5638103ab820 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c8fdafebea6de32 server_cookie=3f874e251a1a8bce in_seq=0 out_seq=0 2023-09-10T01:05:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f052affd700 1 -- 172.21.15.3:0/2983954779 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5638102e5840 con 0x5638103a9400 2023-09-10T01:05:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f052affd700 10 client.26360 handle_command_reply: tid=0 2023-09-10T01:05:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 client.26360 shutdown 2023-09-10T01:05:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 msgr2=0x563810388010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810388010 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f054001e000 tx=0x7f0540099e10 comp rx=0 tx=0).stop 2023-09-10T01:05:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 shutdown_connections 2023-09-10T01:05:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103a9400 0x5638103ab820 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x5638103885b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f052affd700 0 client.26360 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 --2- 172.21.15.3:0/2983954779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810388010 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 >> 172.21.15.3:0/2983954779 conn(0x5638102f3030 msgr2=0x5638103787a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 shutdown_connections 2023-09-10T01:05:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.082+0000 7f0565908b80 1 -- 172.21.15.3:0/2983954779 wait complete. 2023-09-10T01:05:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.083+0000 7f0565908b80 20 client.26360 trim_cache size 0 max 16384 2023-09-10T01:05:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 Processor -- start 2023-09-10T01:05:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 -- start start 2023-09-10T01:05:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810383000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x563810383540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563810375fc0 con 0x56381037e300 2023-09-10T01:05:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638103a69f0 con 0x56381037ecc0 2023-09-10T01:05:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.098+0000 7f0565908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563810388bb0 con 0x5638103841a0 2023-09-10T01:05:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810383000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x563810383540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810383000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44408/0 (socket says 172.21.15.3:44408) 2023-09-10T01:05:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054affd700 1 -- 172.21.15.3:0/2532490714 learned_addr learned my addr 172.21.15.3:0/2532490714 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38624/0 (socket says 172.21.15.3:38624) 2023-09-10T01:05:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054bfff700 1 -- 172.21.15.3:0/2532490714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 msgr2=0x563810383000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054bfff700 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810383000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054bfff700 1 -- 172.21.15.3:0/2532490714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 msgr2=0x56381038a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054bfff700 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054bfff700 1 -- 172.21.15.3:0/2532490714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638101231f0 con 0x5638103841a0 2023-09-10T01:05:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054b7fe700 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054affd700 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810383000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.099+0000 7f054bfff700 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x563810383540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0544009fd0 tx=0x7f054400bc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c82866aa88511d5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f05327fc700 1 -- 172.21.15.3:0/2532490714 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054400e040 con 0x5638103841a0 2023-09-10T01:05:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f05327fc700 1 -- 172.21.15.3:0/2532490714 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0544021070 con 0x5638103841a0 2023-09-10T01:05:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f05327fc700 1 -- 172.21.15.3:0/2532490714 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544005370 con 0x5638103841a0 2023-09-10T01:05:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 -- 172.21.15.3:0/2532490714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 msgr2=0x563810383540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x563810383540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0544009fd0 tx=0x7f054400bc00 comp rx=0 tx=0).stop 2023-09-10T01:05:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 -- 172.21.15.3:0/2532490714 shutdown_connections 2023-09-10T01:05:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638103841a0 0x563810383540 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037e300 0x56381038a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 --2- 172.21.15.3:0/2532490714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56381037ecc0 0x563810383000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 -- 172.21.15.3:0/2532490714 >> 172.21.15.3:0/2532490714 conn(0x5638102f3030 msgr2=0x563810387fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 -- 172.21.15.3:0/2532490714 shutdown_connections 2023-09-10T01:05:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.100+0000 7f0565908b80 1 -- 172.21.15.3:0/2532490714 wait complete. 2023-09-10T01:05:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.101+0000 7f0565908b80 1 -- start start 2023-09-10T01:05:28.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.101+0000 7f0565908b80 1 Processor -- start 2023-09-10T01:05:28.110 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f0565908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381038d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.110 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f0565908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.110 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f0565908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x5638103fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.110 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f0565908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56381038e340 con 0x56381037ecc0 2023-09-10T01:05:28.111 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f0565908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638103fc730 con 0x5638103841a0 2023-09-10T01:05:28.111 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f0565908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638103fc8a0 con 0x56381037e300 2023-09-10T01:05:28.111 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f0565908b80 10 client.0 _ll_register_callbacks cb 0x7f0540099200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:28.111 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381038d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.111 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x5638103fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38636/0 (socket says 172.21.15.3:38636) 2023-09-10T01:05:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381038d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54346/0 (socket says 172.21.15.3:54346) 2023-09-10T01:05:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x5638103fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44416/0 (socket says 172.21.15.3:44416) 2023-09-10T01:05:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054affd700 1 -- 172.21.15.3:0/3379752536 learned_addr learned my addr 172.21.15.3:0/3379752536 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054b7fe700 1 -- 172.21.15.3:0/3379752536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 msgr2=0x5638103fc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054b7fe700 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x5638103fc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054b7fe700 1 -- 172.21.15.3:0/3379752536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 msgr2=0x56381038da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:28.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054b7fe700 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.102+0000 7f054b7fe700 1 -- 172.21.15.3:0/3379752536 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638100fcd80 con 0x56381037e300 2023-09-10T01:05:28.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f054b7fe700 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381038d510 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f053c009750 tx=0x7f053c0092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=335705235ec95eb9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f052affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f052affd700 1 -- 172.21.15.3:0/3379752536 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c02a040 con 0x56381037e300 2023-09-10T01:05:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f053c0268c0 con 0x56381037e300 2023-09-10T01:05:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f052affd700 1 -- 172.21.15.3:0/3379752536 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f053c026610 con 0x56381037e300 2023-09-10T01:05:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f052affd700 1 -- 172.21.15.3:0/3379752536 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f053c026a30 con 0x56381037e300 2023-09-10T01:05:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f0565908b80 10 client.26284 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5638100e5c00 con 0x56381037e300 2023-09-10T01:05:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f052affd700 1 -- 172.21.15.3:0/3379752536 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f053c0268c0 con 0x56381037e300 2023-09-10T01:05:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.103+0000 7f052affd700 1 -- 172.21.15.3:0/3379752536 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f053c024020 con 0x56381037e300 2023-09-10T01:05:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f0565908b80 10 client.26284 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f0565908b80 10 client.26284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f0565908b80 20 client.26284 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f0565908b80 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103fdff0 0x563810400410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f0565908b80 4 client.26284 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:05:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5638102c98c0 con 0x5638103fdff0 2023-09-10T01:05:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f054affd700 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103fdff0 0x563810400410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f054affd700 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103fdff0 0x563810400410 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fcc93b020b6c743 server_cookie=8cca436d32faa077 in_seq=0 out_seq=0 2023-09-10T01:05:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.104+0000 7f052affd700 10 client.26284 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f052affd700 1 -- 172.21.15.3:0/3379752536 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5638102c98c0 con 0x5638103fdff0 2023-09-10T01:05:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f052affd700 10 client.26284 handle_command_reply: tid=0 2023-09-10T01:05:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 client.26284 shutdown 2023-09-10T01:05:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 msgr2=0x56381038d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381038d510 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f053c009750 tx=0x7f053c0092d0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 shutdown_connections 2023-09-10T01:05:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638103fdff0 0x563810400410 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56381037e300 0x56381038d510 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638103841a0 0x5638103fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f052affd700 0 client.26284 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 --2- 172.21.15.3:0/3379752536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56381037ecc0 0x56381038da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 >> 172.21.15.3:0/3379752536 conn(0x5638102f3030 msgr2=0x563810383000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 shutdown_connections 2023-09-10T01:05:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 1 -- 172.21.15.3:0/3379752536 wait complete. 2023-09-10T01:05:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.198+0000 7f0565908b80 20 client.26284 trim_cache size 0 max 16384 2023-09-10T01:05:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:05:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:05:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:05:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:05:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.199+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05340a0830 msgr2=0x7f05340a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.199+0000 7f0551fff700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05340a0830 0x7f05340a2cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f053c011490 tx=0x7f053c009260 comp rx=0 tx=0).stop 2023-09-10T01:05:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.199+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c0732c0 msgr2=0x7f054c19d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.199+0000 7f0551fff700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c0732c0 0x7f054c19d3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f054000e9a0 tx=0x7f054000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 shutdown_connections 2023-09-10T01:05:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05340a0830 0x7f05340a2cf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f054c1115b0 0x7f054c1a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f054c0729a0 0x7f054c19cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 --2- 172.21.15.3:0/1040636738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f054c0732c0 0x7f054c19d3e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 >> 172.21.15.3:0/1040636738 conn(0x7f054c0fc0b0 msgr2=0x7f054c0fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 shutdown_connections 2023-09-10T01:05:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:28.200+0000 7f0551fff700 1 -- 172.21.15.3:0/1040636738 wait complete. 2023-09-10T01:05:28.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:28 smithi136 ceph-mon[133708]: pgmap v70: 129 pgs: 129 active+clean; 47 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 249 op/s 2023-09-10T01:05:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:28 smithi092 ceph-mon[134067]: pgmap v70: 129 pgs: 129 active+clean; 47 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 249 op/s 2023-09-10T01:05:28.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:28 smithi003 ceph-mon[97654]: pgmap v70: 129 pgs: 129 active+clean; 47 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.3 MiB/s wr, 249 op/s 2023-09-10T01:05:28.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 129312 10.21 MB/sec warmup 70 sec latency 21.892 ms 2023-09-10T01:05:29.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 131637 10.26 MB/sec warmup 71 sec latency 13.952 ms 2023-09-10T01:05:30.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:30 smithi136 ceph-mon[133708]: pgmap v71: 129 pgs: 129 active+clean; 47 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 313 op/s 2023-09-10T01:05:30.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:30 smithi092 ceph-mon[134067]: pgmap v71: 129 pgs: 129 active+clean; 47 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 313 op/s 2023-09-10T01:05:30.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:30 smithi003 ceph-mon[97654]: pgmap v71: 129 pgs: 129 active+clean; 47 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 313 op/s 2023-09-10T01:05:30.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 133137 10.19 MB/sec warmup 72 sec latency 11.400 ms 2023-09-10T01:05:31.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 135190 10.24 MB/sec warmup 73 sec latency 54.823 ms 2023-09-10T01:05:32.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:32 smithi136 ceph-mon[133708]: pgmap v72: 129 pgs: 129 active+clean; 47 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 354 op/s 2023-09-10T01:05:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:32 smithi092 ceph-mon[134067]: pgmap v72: 129 pgs: 129 active+clean; 47 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 354 op/s 2023-09-10T01:05:32.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:32 smithi003 ceph-mon[97654]: pgmap v72: 129 pgs: 129 active+clean; 47 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 354 op/s 2023-09-10T01:05:32.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 136714 10.18 MB/sec warmup 74 sec latency 20.669 ms 2023-09-10T01:05:33.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 138808 10.22 MB/sec warmup 75 sec latency 10.887 ms 2023-09-10T01:05:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:34 smithi136 ceph-mon[133708]: pgmap v73: 129 pgs: 129 active+clean; 47 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 262 op/s 2023-09-10T01:05:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:34 smithi092 ceph-mon[134067]: pgmap v73: 129 pgs: 129 active+clean; 47 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 262 op/s 2023-09-10T01:05:34.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:34 smithi003 ceph-mon[97654]: pgmap v73: 129 pgs: 129 active+clean; 47 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 262 op/s 2023-09-10T01:05:34.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 140873 10.19 MB/sec warmup 76 sec latency 10.390 ms 2023-09-10T01:05:35.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142634 10.21 MB/sec warmup 77 sec latency 10.749 ms 2023-09-10T01:05:36.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:36 smithi136 ceph-mon[133708]: pgmap v74: 129 pgs: 129 active+clean; 51 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 386 op/s 2023-09-10T01:05:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:36 smithi092 ceph-mon[134067]: pgmap v74: 129 pgs: 129 active+clean; 51 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 386 op/s 2023-09-10T01:05:36.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:36 smithi003 ceph-mon[97654]: pgmap v74: 129 pgs: 129 active+clean; 51 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.2 MiB/s wr, 386 op/s 2023-09-10T01:05:36.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 144486 10.18 MB/sec warmup 78 sec latency 60.519 ms 2023-09-10T01:05:37.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 146170 10.19 MB/sec warmup 79 sec latency 17.284 ms 2023-09-10T01:05:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:38 smithi136 ceph-mon[133708]: pgmap v75: 129 pgs: 129 active+clean; 51 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 287 op/s 2023-09-10T01:05:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:38 smithi092 ceph-mon[134067]: pgmap v75: 129 pgs: 129 active+clean; 51 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 287 op/s 2023-09-10T01:05:38.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:38 smithi003 ceph-mon[97654]: pgmap v75: 129 pgs: 129 active+clean; 51 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 287 op/s 2023-09-10T01:05:38.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 148669 10.23 MB/sec warmup 80 sec latency 13.313 ms 2023-09-10T01:05:39.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 150059 10.20 MB/sec warmup 81 sec latency 13.593 ms 2023-09-10T01:05:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:40 smithi136 ceph-mon[133708]: pgmap v76: 129 pgs: 129 active+clean; 55 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 371 op/s 2023-09-10T01:05:40.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:40 smithi092 ceph-mon[134067]: pgmap v76: 129 pgs: 129 active+clean; 55 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 371 op/s 2023-09-10T01:05:40.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:40 smithi003 ceph-mon[97654]: pgmap v76: 129 pgs: 129 active+clean; 55 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 371 op/s 2023-09-10T01:05:40.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 152680 10.27 MB/sec warmup 82 sec latency 43.944 ms 2023-09-10T01:05:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.369+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/1210514325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4106680 msgr2=0x7ff6f4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.369+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/1210514325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4106680 0x7ff6f4106a60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40099c0 tx=0x7ff6e4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.370+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/1210514325 shutdown_connections 2023-09-10T01:05:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.370+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/1210514325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f4100f80 0x7ff6f410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.370+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/1210514325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4106680 0x7ff6f4106a60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.370+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/1210514325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 0x7ff6f4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.370+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/1210514325 >> 172.21.15.3:0/1210514325 conn(0x7ff6f40fc0b0 msgr2=0x7ff6f40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.370+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/1210514325 shutdown_connections 2023-09-10T01:05:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.370+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/1210514325 wait complete. 2023-09-10T01:05:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.371+0000 7ff6fa1ff700 1 Processor -- start 2023-09-10T01:05:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.371+0000 7ff6fa1ff700 1 -- start start 2023-09-10T01:05:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6fa1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 0x7ff6f419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6fa1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4100f80 0x7ff6f419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6fa1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f4106680 0x7ff6f41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6fa1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4198d90 con 0x7ff6f4106680 2023-09-10T01:05:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6fa1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4198f00 con 0x7ff6f4100f80 2023-09-10T01:05:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6fa1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41990c0 con 0x7ff6f41005c0 2023-09-10T01:05:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 0x7ff6f419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 0x7ff6f419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47618/0 (socket says 172.21.15.3:47618) 2023-09-10T01:05:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6f37fe700 1 -- 172.21.15.3:0/2114957120 learned_addr learned my addr 172.21.15.3:0/2114957120 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4100f80 0x7ff6f419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.372+0000 7ff6f3fff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f4106680 0x7ff6f41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f37fe700 1 -- 172.21.15.3:0/2114957120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4100f80 msgr2=0x7ff6f419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4100f80 0x7ff6f419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f37fe700 1 -- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f4106680 msgr2=0x7ff6f41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f4106680 0x7ff6f41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f37fe700 1 -- 172.21.15.3:0/2114957120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec009c40 con 0x7ff6f41005c0 2023-09-10T01:05:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 0x7ff6f419e650 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400f690 tx=0x7ff6e400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f0ff9700 1 -- 172.21.15.3:0/2114957120 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e401c070 con 0x7ff6f41005c0 2023-09-10T01:05:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f0ff9700 1 -- 172.21.15.3:0/2114957120 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6e400fe20 con 0x7ff6f41005c0 2023-09-10T01:05:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6f0ff9700 1 -- 172.21.15.3:0/2114957120 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4022d30 con 0x7ff6f41005c0 2023-09-10T01:05:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e40095f0 con 0x7ff6f41005c0 2023-09-10T01:05:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.373+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f41a54a0 con 0x7ff6f41005c0 2023-09-10T01:05:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.377+0000 7ff6f0ff9700 1 -- 172.21.15.3:0/2114957120 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ff6e4020070 con 0x7ff6f41005c0 2023-09-10T01:05:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.378+0000 7ff70da62b80 1 Processor -- start 2023-09-10T01:05:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff70da62b80 1 -- start start 2023-09-10T01:05:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff6f0ff9700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff6dc0a0830 0x7ff6dc0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47ba31a0 0x555a47ba3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47b9d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9dcc0 0x555a47ba9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff70da62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47af66d0 con 0x555a47b9d300 2023-09-10T01:05:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff70da62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47af3ff0 con 0x555a47ba31a0 2023-09-10T01:05:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff70da62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47aef8e0 con 0x555a47b9dcc0 2023-09-10T01:05:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff6f0ff9700 1 -- 172.21.15.3:0/2114957120 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff6e40cd220 con 0x7ff6f41005c0 2023-09-10T01:05:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.379+0000 7ff6f3fff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff6dc0a0830 0x7ff6dc0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9dcc0 0x555a47ba9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47b9d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47ba31a0 0x555a47ba3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9dcc0 0x555a47ba9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47632/0 (socket says 172.21.15.3:47632) 2023-09-10T01:05:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47ba31a0 0x555a47ba3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44954/0 (socket says 172.21.15.3:44954) 2023-09-10T01:05:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f3fff700 1 -- 172.21.15.3:0/1517746251 learned_addr learned my addr 172.21.15.3:0/1517746251 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 -- 172.21.15.3:0/1517746251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9dcc0 msgr2=0x555a47ba9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/1517746251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9dcc0 0x555a47ba9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 -- 172.21.15.3:0/1517746251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 msgr2=0x555a47b9d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/1517746251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47b9d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 -- 172.21.15.3:0/1517746251 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a4791bd80 con 0x555a47ba31a0 2023-09-10T01:05:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/1517746251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47ba31a0 0x555a47ba3580 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40ca5e0 tx=0x7ff6e4022750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eaa8014dcd58f3e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6f3fff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff6dc0a0830 0x7ff6dc0a2cf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec007d90 tx=0x7ff6ec009480 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6d9ffb700 1 -- 172.21.15.3:0/1517746251 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e409a4f0 con 0x555a47ba31a0 2023-09-10T01:05:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.380+0000 7ff6d9ffb700 1 -- 172.21.15.3:0/1517746251 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6e409e320 con 0x555a47ba31a0 2023-09-10T01:05:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff6d9ffb700 1 -- 172.21.15.3:0/1517746251 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e409a750 con 0x555a47ba31a0 2023-09-10T01:05:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 -- 172.21.15.3:0/1517746251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47ba31a0 msgr2=0x555a47ba3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1517746251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47ba31a0 0x555a47ba3580 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40ca5e0 tx=0x7ff6e4022750 comp rx=0 tx=0).stop 2023-09-10T01:05:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 -- 172.21.15.3:0/1517746251 shutdown_connections 2023-09-10T01:05:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1517746251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9dcc0 0x555a47ba9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1517746251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47ba31a0 0x555a47ba3580 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1517746251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47b9d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 -- 172.21.15.3:0/1517746251 >> 172.21.15.3:0/1517746251 conn(0x555a47b12030 msgr2=0x555a47b98560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 -- 172.21.15.3:0/1517746251 shutdown_connections 2023-09-10T01:05:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.381+0000 7ff70da62b80 1 -- 172.21.15.3:0/1517746251 wait complete. 2023-09-10T01:05:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.382+0000 7ff70da62b80 1 -- start start 2023-09-10T01:05:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.382+0000 7ff70da62b80 1 Processor -- start 2023-09-10T01:05:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bacb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47ba6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff70da62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47a8db10 con 0x555a47b9d300 2023-09-10T01:05:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff70da62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47a8d810 con 0x555a47b9dcc0 2023-09-10T01:05:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff70da62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a479718c0 con 0x555a47ba31a0 2023-09-10T01:05:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff70da62b80 10 client.0 _ll_register_callbacks cb 0x555a479d7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.383+0000 7ff6f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44966/0 (socket says 172.21.15.3:44966) 2023-09-10T01:05:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 -- 172.21.15.3:0/1423951191 learned_addr learned my addr 172.21.15.3:0/1423951191 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bacb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f3fff700 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47ba6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 -- 172.21.15.3:0/1423951191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 msgr2=0x555a47ba6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47ba6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 -- 172.21.15.3:0/1423951191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 msgr2=0x555a47bacb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bacb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 -- 172.21.15.3:0/1423951191 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f41a54a0 con 0x555a47b9dcc0 2023-09-10T01:05:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bad0a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8012840 tx=0x7ff6e8012b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6330a5c29319d5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/1423951191 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8017070 con 0x555a47b9dcc0 2023-09-10T01:05:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/1423951191 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff6e8004830 con 0x555a47b9dcc0 2023-09-10T01:05:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.384+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a47a47490 con 0x555a47b9dcc0 2023-09-10T01:05:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/1423951191 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6e8004a30 con 0x555a47b9dcc0 2023-09-10T01:05:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 10 client.26378 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555a47904c00 con 0x555a47b9dcc0 2023-09-10T01:05:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/1423951191 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555a47a47490 con 0x555a47b9dcc0 2023-09-10T01:05:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/1423951191 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff6e801c6d0 con 0x555a47b9dcc0 2023-09-10T01:05:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 10 client.26378 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 10 client.26378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 20 client.26378 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47bb70c0 0x555a47bb94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 4 client.26378 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a47b04840 con 0x555a47bb70c0 2023-09-10T01:05:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.385+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47bb70c0 0x555a47bb94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.386+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47bb70c0 0x555a47bb94e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bae9fcc1c409722 server_cookie=4c8ffa39ceab57e6 in_seq=0 out_seq=0 2023-09-10T01:05:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.386+0000 7ff6d2ffd700 10 client.26378 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/1423951191 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555a47b04840 con 0x555a47bb70c0 2023-09-10T01:05:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff6d2ffd700 10 client.26378 handle_command_reply: tid=0 2023-09-10T01:05:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 client.26378 shutdown 2023-09-10T01:05:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 msgr2=0x555a47bad0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bad0a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8012840 tx=0x7ff6e8012b50 comp rx=0 tx=0).stop 2023-09-10T01:05:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 shutdown_connections 2023-09-10T01:05:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47bb70c0 0x555a47bb94e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47ba6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bacb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff6d2ffd700 0 client.26378 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 --2- 172.21.15.3:0/1423951191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bad0a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 >> 172.21.15.3:0/1423951191 conn(0x555a47b12030 msgr2=0x555a47b977a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 shutdown_connections 2023-09-10T01:05:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.387+0000 7ff70da62b80 1 -- 172.21.15.3:0/1423951191 wait complete. 2023-09-10T01:05:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.388+0000 7ff70da62b80 20 client.26378 trim_cache size 0 max 16384 2023-09-10T01:05:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.406+0000 7ff70da62b80 1 Processor -- start 2023-09-10T01:05:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.406+0000 7ff70da62b80 1 -- start start 2023-09-10T01:05:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.406+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9d300 0x555a47ba76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47b97f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47ba31a0 0x555a47b98460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff70da62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47ba1a40 con 0x555a47ba31a0 2023-09-10T01:05:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff70da62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47bb46a0 con 0x555a47b9dcc0 2023-09-10T01:05:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff70da62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47ba5310 con 0x555a47b9d300 2023-09-10T01:05:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47b97f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9d300 0x555a47ba76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47b97f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44974/0 (socket says 172.21.15.3:44974) 2023-09-10T01:05:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f2ffd700 1 -- 172.21.15.3:0/4052477282 learned_addr learned my addr 172.21.15.3:0/4052477282 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9d300 0x555a47ba76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47648/0 (socket says 172.21.15.3:47648) 2023-09-10T01:05:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f3fff700 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47ba31a0 0x555a47b98460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f37fe700 1 -- 172.21.15.3:0/4052477282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 msgr2=0x555a47b97f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47b97f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f37fe700 1 -- 172.21.15.3:0/4052477282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47ba31a0 msgr2=0x555a47b98460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47ba31a0 0x555a47b98460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.407+0000 7ff6f37fe700 1 -- 172.21.15.3:0/4052477282 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a479421f0 con 0x555a47b9d300 2023-09-10T01:05:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.408+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9d300 0x555a47ba76e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4098840 tx=0x7ff6e40af040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff3865ce2269f11f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.408+0000 7ff6da7fc700 1 -- 172.21.15.3:0/4052477282 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e407d070 con 0x555a47b9d300 2023-09-10T01:05:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.408+0000 7ff6da7fc700 1 -- 172.21.15.3:0/4052477282 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6e409a4f0 con 0x555a47b9d300 2023-09-10T01:05:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.408+0000 7ff6da7fc700 1 -- 172.21.15.3:0/4052477282 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e40999e0 con 0x555a47b9d300 2023-09-10T01:05:41.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 -- 172.21.15.3:0/4052477282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9d300 msgr2=0x555a47ba76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9d300 0x555a47ba76e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4098840 tx=0x7ff6e40af040 comp rx=0 tx=0).stop 2023-09-10T01:05:41.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 -- 172.21.15.3:0/4052477282 shutdown_connections 2023-09-10T01:05:41.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47b9d300 0x555a47ba76e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47ba31a0 0x555a47b98460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 --2- 172.21.15.3:0/4052477282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47b97f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 -- 172.21.15.3:0/4052477282 >> 172.21.15.3:0/4052477282 conn(0x555a47b12030 msgr2=0x555a47bbde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:41.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 -- 172.21.15.3:0/4052477282 shutdown_connections 2023-09-10T01:05:41.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 -- 172.21.15.3:0/4052477282 wait complete. 2023-09-10T01:05:41.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.409+0000 7ff70da62b80 1 -- start start 2023-09-10T01:05:41.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 1 Processor -- start 2023-09-10T01:05:41.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bb0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bb1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47bb17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47c09fb0 con 0x555a47b9d300 2023-09-10T01:05:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47c0a120 con 0x555a47b9dcc0 2023-09-10T01:05:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47c1b2f0 con 0x555a47ba31a0 2023-09-10T01:05:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.410+0000 7ff70da62b80 10 client.0 _ll_register_callbacks cb 0x555a479d7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bb0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47bb17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bb0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41596/0 (socket says 172.21.15.3:41596) 2023-09-10T01:05:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 -- 172.21.15.3:0/186501569 learned_addr learned my addr 172.21.15.3:0/186501569 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bb1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 -- 172.21.15.3:0/186501569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 msgr2=0x555a47bb17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47bb17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 -- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 msgr2=0x555a47bb1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bb1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f37fe700 1 -- 172.21.15.3:0/186501569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a4791bd80 con 0x555a47b9d300 2023-09-10T01:05:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f3fff700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47bb17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.411+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bb1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff6f37fe700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bb0d50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40a1fa0 tx=0x7ff6e4005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a016c4ebee1e40 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff6d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:05:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/186501569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e407d030 con 0x555a47b9d300 2023-09-10T01:05:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/186501569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff6e407e070 con 0x555a47b9d300 2023-09-10T01:05:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6e409e320 con 0x555a47b9d300 2023-09-10T01:05:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/186501569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6e409e0b0 con 0x555a47b9d300 2023-09-10T01:05:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 10 client.16326 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555a4791a250 con 0x555a47b9d300 2023-09-10T01:05:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/186501569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6e409e320 con 0x555a47b9d300 2023-09-10T01:05:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/186501569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff6e4099a70 con 0x555a47b9d300 2023-09-10T01:05:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 10 client.16326 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 10 client.16326 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 20 client.16326 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47c1c5b0 0x555a47c1e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.412+0000 7ff70da62b80 4 client.16326 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:05:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.413+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555a47ae88c0 con 0x555a47c1c5b0 2023-09-10T01:05:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.413+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47c1c5b0 0x555a47c1e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:41.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.413+0000 7ff6f2ffd700 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47c1c5b0 0x555a47c1e9a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d33109d62438c2fa server_cookie=fce945bcfa21b531 in_seq=0 out_seq=0 2023-09-10T01:05:41.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.413+0000 7ff6d2ffd700 10 client.16326 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.589+0000 7ff6d2ffd700 1 -- 172.21.15.3:0/186501569 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555a47ae88c0 con 0x555a47c1c5b0 2023-09-10T01:05:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.589+0000 7ff6d2ffd700 10 client.16326 handle_command_reply: tid=0 2023-09-10T01:05:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.589+0000 7ff70da62b80 1 client.16326 shutdown 2023-09-10T01:05:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.589+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 msgr2=0x555a47bb0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.589+0000 7ff70da62b80 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bb0d50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40a1fa0 tx=0x7ff6e4005a90 comp rx=0 tx=0).stop 2023-09-10T01:05:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 shutdown_connections 2023-09-10T01:05:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555a47c1c5b0 0x555a47c1e9a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555a47ba31a0 0x555a47bb17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a47b9d300 0x555a47bb0d50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff6d2ffd700 0 client.16326 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 --2- 172.21.15.3:0/186501569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a47b9dcc0 0x555a47bb1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 >> 172.21.15.3:0/186501569 conn(0x555a47b12030 msgr2=0x555a47ba76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 shutdown_connections 2023-09-10T01:05:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 1 -- 172.21.15.3:0/186501569 wait complete. 2023-09-10T01:05:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.590+0000 7ff70da62b80 20 client.16326 trim_cache size 0 max 16384 2023-09-10T01:05:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:05:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:05:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:05:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:05:41.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff6dc0a0830 msgr2=0x7ff6dc0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff6dc0a0830 0x7ff6dc0a2cf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec007d90 tx=0x7ff6ec009480 comp rx=0 tx=0).stop 2023-09-10T01:05:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 msgr2=0x7ff6f419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 0x7ff6f419e650 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400f690 tx=0x7ff6e400bfd0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 shutdown_connections 2023-09-10T01:05:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff6dc0a0830 0x7ff6dc0a2cf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f41005c0 0x7ff6f419e650 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6f4106680 0x7ff6f41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 --2- 172.21.15.3:0/2114957120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f4100f80 0x7ff6f419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 >> 172.21.15.3:0/2114957120 conn(0x7ff6f40fc0b0 msgr2=0x7ff6f4072830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 shutdown_connections 2023-09-10T01:05:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:41.591+0000 7ff6fa1ff700 1 -- 172.21.15.3:0/2114957120 wait complete. 2023-09-10T01:05:41.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 153434 10.17 MB/sec warmup 83 sec latency 62.649 ms 2023-09-10T01:05:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:42 smithi136 ceph-mon[133708]: pgmap v77: 129 pgs: 129 active+clean; 55 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 336 op/s 2023-09-10T01:05:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:42 smithi092 ceph-mon[134067]: pgmap v77: 129 pgs: 129 active+clean; 55 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 336 op/s 2023-09-10T01:05:42.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:42 smithi003 ceph-mon[97654]: pgmap v77: 129 pgs: 129 active+clean; 55 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 336 op/s 2023-09-10T01:05:42.743 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156099 10.25 MB/sec warmup 84 sec latency 22.257 ms 2023-09-10T01:05:43.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 157016 10.15 MB/sec warmup 85 sec latency 73.569 ms 2023-09-10T01:05:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:44 smithi136 ceph-mon[133708]: pgmap v78: 129 pgs: 129 active+clean; 55 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 238 op/s 2023-09-10T01:05:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:44 smithi092 ceph-mon[134067]: pgmap v78: 129 pgs: 129 active+clean; 55 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 238 op/s 2023-09-10T01:05:44.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:44 smithi003 ceph-mon[97654]: pgmap v78: 129 pgs: 129 active+clean; 55 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 238 op/s 2023-09-10T01:05:44.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 159635 10.23 MB/sec warmup 86 sec latency 77.546 ms 2023-09-10T01:05:45.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 160747 10.15 MB/sec warmup 87 sec latency 43.040 ms 2023-09-10T01:05:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:46 smithi136 ceph-mon[133708]: pgmap v79: 129 pgs: 129 active+clean; 51 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 403 op/s 2023-09-10T01:05:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:46 smithi092 ceph-mon[134067]: pgmap v79: 129 pgs: 129 active+clean; 51 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 403 op/s 2023-09-10T01:05:46.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:46 smithi003 ceph-mon[97654]: pgmap v79: 129 pgs: 129 active+clean; 51 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 403 op/s 2023-09-10T01:05:46.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 163182 10.21 MB/sec warmup 88 sec latency 40.111 ms 2023-09-10T01:05:47.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:05:47.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 -- 172.21.15.3:0/1759906211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 msgr2=0x7f1e10100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 --2- 172.21.15.3:0/1759906211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10100ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009990 tx=0x7f1e00009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 -- 172.21.15.3:0/1759906211 shutdown_connections 2023-09-10T01:05:47.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 --2- 172.21.15.3:0/1759906211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e10101000 0x7f1e1010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 --2- 172.21.15.3:0/1759906211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10106650 0x7f1e10106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 --2- 172.21.15.3:0/1759906211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10100ac0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 -- 172.21.15.3:0/1759906211 >> 172.21.15.3:0/1759906211 conn(0x7f1e100fc170 msgr2=0x7f1e100fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:47.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 -- 172.21.15.3:0/1759906211 shutdown_connections 2023-09-10T01:05:47.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.552+0000 7f1e177ff700 1 -- 172.21.15.3:0/1759906211 wait complete. 2023-09-10T01:05:47.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.553+0000 7f1e177ff700 1 Processor -- start 2023-09-10T01:05:47.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.553+0000 7f1e177ff700 1 -- start start 2023-09-10T01:05:47.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e177ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e177ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10101000 0x7f1e10072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e177ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e10106650 0x7f1e101a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e177ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e10073670 con 0x7f1e10106650 2023-09-10T01:05:47.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e177ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101a4e50 con 0x7f1e10101000 2023-09-10T01:05:47.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e177ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101a4f90 con 0x7f1e10100640 2023-09-10T01:05:47.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e1559b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10101000 0x7f1e10072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e1559b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40260/0 (socket says 172.21.15.3:40260) 2023-09-10T01:05:47.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e15d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e10106650 0x7f1e101a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10101000 0x7f1e10072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53034/0 (socket says 172.21.15.3:53034) 2023-09-10T01:05:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.554+0000 7f1e1559b700 1 -- 172.21.15.3:0/2328912709 learned_addr learned my addr 172.21.15.3:0/2328912709 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e14d9a700 1 -- 172.21.15.3:0/2328912709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 msgr2=0x7f1e10072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e14d9a700 1 -- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e10106650 msgr2=0x7f1e101a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e10106650 0x7f1e101a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e14d9a700 1 -- 172.21.15.3:0/2328912709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e000095f0 con 0x7f1e10101000 2023-09-10T01:05:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e10106650 0x7f1e101a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e1559b700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10101000 0x7f1e10072e40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009990 tx=0x7f1e00009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e067fc700 1 -- 172.21.15.3:0/2328912709 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0001d070 con 0x7f1e10101000 2023-09-10T01:05:47.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e101a5190 con 0x7f1e10101000 2023-09-10T01:05:47.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e067fc700 1 -- 172.21.15.3:0/2328912709 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e00021410 con 0x7f1e10101000 2023-09-10T01:05:47.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e067fc700 1 -- 172.21.15.3:0/2328912709 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00017650 con 0x7f1e10101000 2023-09-10T01:05:47.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.555+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e101a56c0 con 0x7f1e10101000 2023-09-10T01:05:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.559+0000 7f1e067fc700 1 -- 172.21.15.3:0/2328912709 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f1e00021810 con 0x7f1e10101000 2023-09-10T01:05:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 Processor -- start 2023-09-10T01:05:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 -- start start 2023-09-10T01:05:47.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d521af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e067fc700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dfc0a07b0 0x7f1dfc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d52be40 0x56490d52e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d47a730 con 0x56490d521710 2023-09-10T01:05:47.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d478050 con 0x56490d52be40 2023-09-10T01:05:47.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.561+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d473940 con 0x56490d522030 2023-09-10T01:05:47.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e067fc700 1 -- 172.21.15.3:0/2328912709 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1e000cd600 con 0x7f1e10101000 2023-09-10T01:05:47.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e1559b700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dfc0a07b0 0x7f1dfc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40268/0 (socket says 172.21.15.3:40268) 2023-09-10T01:05:47.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 -- 172.21.15.3:0/3787509180 learned_addr learned my addr 172.21.15.3:0/3787509180 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:47.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d521af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d52be40 0x56490d52e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 -- 172.21.15.3:0/3787509180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d52be40 msgr2=0x56490d52e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d52be40 0x56490d52e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 -- 172.21.15.3:0/3787509180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 msgr2=0x56490d521af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d521af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 -- 172.21.15.3:0/3787509180 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56490d29fde0 con 0x56490d522030 2023-09-10T01:05:47.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d521af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:47.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52b900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1e000a2200 tx=0x7f1e00096b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=943c659ec178208f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.562+0000 7f1e1559b700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dfc0a07b0 0x7f1dfc0a2c70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c011550 tx=0x7f1e0c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1dfb7fe700 1 -- 172.21.15.3:0/3787509180 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0009a0b0 con 0x56490d522030 2023-09-10T01:05:47.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1dfb7fe700 1 -- 172.21.15.3:0/3787509180 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e0009ae50 con 0x56490d522030 2023-09-10T01:05:47.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1dfb7fe700 1 -- 172.21.15.3:0/3787509180 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e000a2730 con 0x56490d522030 2023-09-10T01:05:47.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3787509180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 msgr2=0x56490d52b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52b900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1e000a2200 tx=0x7f1e00096b10 comp rx=0 tx=0).stop 2023-09-10T01:05:47.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3787509180 shutdown_connections 2023-09-10T01:05:47.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d52be40 0x56490d52e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d521af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3787509180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52b900 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3787509180 >> 172.21.15.3:0/3787509180 conn(0x56490d496090 msgr2=0x56490d51cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:47.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.563+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3787509180 shutdown_connections 2023-09-10T01:05:47.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.564+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3787509180 wait complete. 2023-09-10T01:05:47.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.564+0000 7f1e2b13bb80 1 -- start start 2023-09-10T01:05:47.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.564+0000 7f1e2b13bb80 1 Processor -- start 2023-09-10T01:05:47.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d521710 0x56490d52c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d52be40 0x56490d5372a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d411b70 con 0x56490d52be40 2023-09-10T01:05:47.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d411870 con 0x56490d521710 2023-09-10T01:05:47.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d2f5920 con 0x56490d522030 2023-09-10T01:05:47.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e2b13bb80 10 client.0 _ll_register_callbacks cb 0x56490d35ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:47.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e1559b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d521710 0x56490d52c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e1559b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d521710 0x56490d52c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53066/0 (socket says 172.21.15.3:53066) 2023-09-10T01:05:47.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e1559b700 1 -- 172.21.15.3:0/3806328246 learned_addr learned my addr 172.21.15.3:0/3806328246 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:47.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e15d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d52be40 0x56490d5372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e14d9a700 1 -- 172.21.15.3:0/3806328246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d521710 msgr2=0x56490d52c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.565+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d521710 0x56490d52c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1e14d9a700 1 -- 172.21.15.3:0/3806328246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d52be40 msgr2=0x56490d5372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d52be40 0x56490d5372a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1e14d9a700 1 -- 172.21.15.3:0/3806328246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e101a56c0 con 0x56490d522030 2023-09-10T01:05:47.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52c760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1e00005b00 tx=0x7f1e000a7140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85ffee5eb2f797cf server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1df8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:47.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1df8ff9700 1 -- 172.21.15.3:0/3806328246 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0009e370 con 0x56490d522030 2023-09-10T01:05:47.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56490d3cb4f0 con 0x56490d522030 2023-09-10T01:05:47.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1df8ff9700 1 -- 172.21.15.3:0/3806328246 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1e000a2730 con 0x56490d522030 2023-09-10T01:05:47.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1df8ff9700 1 -- 172.21.15.3:0/3806328246 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e0009e580 con 0x56490d522030 2023-09-10T01:05:47.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1e2b13bb80 10 client.26314 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:47.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56490d288c60 con 0x56490d522030 2023-09-10T01:05:47.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.566+0000 7f1df8ff9700 1 -- 172.21.15.3:0/3806328246 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56490d3cb4f0 con 0x56490d522030 2023-09-10T01:05:47.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1df8ff9700 1 -- 172.21.15.3:0/3806328246 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f1e000a9890 con 0x56490d522030 2023-09-10T01:05:47.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e2b13bb80 10 client.26314 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:47.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e2b13bb80 10 client.26314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:47.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e2b13bb80 20 client.26314 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:47.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d53b190 0x56490d53d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e2b13bb80 4 client.26314 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:47.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56490d4888a0 con 0x56490d53b190 2023-09-10T01:05:47.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e1559b700 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d53b190 0x56490d53d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1e1559b700 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d53b190 0x56490d53d5b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0aad0b53a3059d9 server_cookie=29be2f3863243df3 in_seq=0 out_seq=0 2023-09-10T01:05:47.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.567+0000 7f1df8ff9700 10 client.26314 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:47.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1df8ff9700 1 -- 172.21.15.3:0/3806328246 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56490d4888a0 con 0x56490d53b190 2023-09-10T01:05:47.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1df8ff9700 10 client.26314 handle_command_reply: tid=0 2023-09-10T01:05:47.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 client.26314 shutdown 2023-09-10T01:05:47.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 msgr2=0x56490d52c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52c760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1e00005b00 tx=0x7f1e000a7140 comp rx=0 tx=0).stop 2023-09-10T01:05:47.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 shutdown_connections 2023-09-10T01:05:47.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d53b190 0x56490d53d5b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d52be40 0x56490d5372a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d521710 0x56490d52c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/3806328246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d522030 0x56490d52c760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1df8ff9700 0 client.26314 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:47.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 >> 172.21.15.3:0/3806328246 conn(0x56490d496090 msgr2=0x56490d51c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:47.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 shutdown_connections 2023-09-10T01:05:47.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.569+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/3806328246 wait complete. 2023-09-10T01:05:47.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.570+0000 7f1e2b13bb80 20 client.26314 trim_cache size 0 max 16384 2023-09-10T01:05:47.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.588+0000 7f1e2b13bb80 1 Processor -- start 2023-09-10T01:05:47.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.588+0000 7f1e2b13bb80 1 -- start start 2023-09-10T01:05:47.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d526230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d526770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d5369a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d52b5a0 con 0x56490d521710 2023-09-10T01:05:47.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d522470 con 0x56490d522030 2023-09-10T01:05:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d529480 con 0x56490d52be40 2023-09-10T01:05:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e15d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d5369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e1559b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d526230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d526770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e15d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d5369a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40280/0 (socket says 172.21.15.3:40280) 2023-09-10T01:05:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d526770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53080/0 (socket says 172.21.15.3:53080) 2023-09-10T01:05:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e15d9c700 1 -- 172.21.15.3:0/783414627 learned_addr learned my addr 172.21.15.3:0/783414627 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e1559b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d526230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57852/0 (socket says 172.21.15.3:57852) 2023-09-10T01:05:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e15d9c700 1 -- 172.21.15.3:0/783414627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 msgr2=0x56490d526770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/783414627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d526770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e15d9c700 1 -- 172.21.15.3:0/783414627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 msgr2=0x56490d526230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.589+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/783414627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d526230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.590+0000 7f1e15d9c700 1 -- 172.21.15.3:0/783414627 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56490d2c6250 con 0x56490d52be40 2023-09-10T01:05:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.590+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/783414627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d5369a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1e080075e0 tx=0x7f1e08009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8af63dba2d042d2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.590+0000 7f1dfbfff700 1 -- 172.21.15.3:0/783414627 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e08004db0 con 0x56490d52be40 2023-09-10T01:05:47.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.590+0000 7f1dfbfff700 1 -- 172.21.15.3:0/783414627 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e0800f040 con 0x56490d52be40 2023-09-10T01:05:47.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.590+0000 7f1dfbfff700 1 -- 172.21.15.3:0/783414627 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0801a900 con 0x56490d52be40 2023-09-10T01:05:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/783414627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 msgr2=0x56490d5369a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/783414627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d5369a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1e080075e0 tx=0x7f1e08009920 comp rx=0 tx=0).stop 2023-09-10T01:05:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/783414627 shutdown_connections 2023-09-10T01:05:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/783414627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d5369a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/783414627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d526230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/783414627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d526770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/783414627 >> 172.21.15.3:0/783414627 conn(0x56490d496090 msgr2=0x56490d52c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/783414627 shutdown_connections 2023-09-10T01:05:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/783414627 wait complete. 2023-09-10T01:05:47.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.591+0000 7f1e2b13bb80 1 -- start start 2023-09-10T01:05:47.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 1 Processor -- start 2023-09-10T01:05:47.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d5313e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d52f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d52fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d530510 con 0x56490d521710 2023-09-10T01:05:47.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d58efd0 con 0x56490d522030 2023-09-10T01:05:47.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490d58f110 con 0x56490d52be40 2023-09-10T01:05:47.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e2b13bb80 10 client.0 _ll_register_callbacks cb 0x7f1e00083f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:47.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d52f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e14d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d52f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53084/0 (socket says 172.21.15.3:53084) 2023-09-10T01:05:47.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e14d9a700 1 -- 172.21.15.3:0/1816136237 learned_addr learned my addr 172.21.15.3:0/1816136237 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:47.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.592+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d52fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e1559b700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d5313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e15d9c700 1 -- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 msgr2=0x56490d52f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d52f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e15d9c700 1 -- 172.21.15.3:0/1816136237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 msgr2=0x56490d5313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:47.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d5313e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e15d9c700 1 -- 172.21.15.3:0/1816136237 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e080092f0 con 0x56490d52be40 2023-09-10T01:05:47.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e14d9a700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d52f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:47.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e1559b700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d5313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:47.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1e15d9c700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d52fe00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009010 tx=0x7f1e0801a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbd0be46dbb8c77d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1df8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:47.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.593+0000 7f1df8ff9700 1 -- 172.21.15.3:0/1816136237 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0801a990 con 0x56490d52be40 2023-09-10T01:05:47.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56490d52bb80 con 0x56490d52be40 2023-09-10T01:05:47.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1df8ff9700 1 -- 172.21.15.3:0/1816136237 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f1e0801aaf0 con 0x56490d52be40 2023-09-10T01:05:47.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1df8ff9700 1 -- 172.21.15.3:0/1816136237 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e0801acd0 con 0x56490d52be40 2023-09-10T01:05:47.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 10 client.26320 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:47.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56490d29e2b0 con 0x56490d52be40 2023-09-10T01:05:47.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1df8ff9700 1 -- 172.21.15.3:0/1816136237 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56490d52bb80 con 0x56490d52be40 2023-09-10T01:05:47.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1df8ff9700 1 -- 172.21.15.3:0/1816136237 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f1e08020070 con 0x56490d52be40 2023-09-10T01:05:47.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 10 client.26320 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:47.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 10 client.26320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:47.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 20 client.26320 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:47.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d590800 0x56490d592c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 4 client.26320 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:05:47.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.594+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f1e0009ae50 con 0x56490d590800 2023-09-10T01:05:47.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.595+0000 7f1e1559b700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d590800 0x56490d592c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.595+0000 7f1df8ff9700 10 client.26320 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:47.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.595+0000 7f1e1559b700 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d590800 0x56490d592c20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2353fc3f305bcebf server_cookie=c94d56fcc3e0f49 in_seq=0 out_seq=0 2023-09-10T01:05:47.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.595+0000 7f1df8ff9700 1 -- 172.21.15.3:0/1816136237 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f1e0009ae50 con 0x56490d590800 2023-09-10T01:05:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.595+0000 7f1df8ff9700 10 client.26320 handle_command_reply: tid=0 2023-09-10T01:05:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.595+0000 7f1e2b13bb80 1 client.26320 shutdown 2023-09-10T01:05:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 msgr2=0x56490d52fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d52fe00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009010 tx=0x7f1e0801a440 comp rx=0 tx=0).stop 2023-09-10T01:05:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 shutdown_connections 2023-09-10T01:05:47.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56490d590800 0x56490d592c20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56490d52be40 0x56490d52fe00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56490d521710 0x56490d5313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 --2- 172.21.15.3:0/1816136237 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56490d522030 0x56490d52f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 >> 172.21.15.3:0/1816136237 conn(0x56490d496090 msgr2=0x56490d52c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:47.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1df8ff9700 0 client.26320 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:47.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 shutdown_connections 2023-09-10T01:05:47.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 1 -- 172.21.15.3:0/1816136237 wait complete. 2023-09-10T01:05:47.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.596+0000 7f1e2b13bb80 20 client.26320 trim_cache size 0 max 16384 2023-09-10T01:05:47.611 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:05:47.612 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:05:47.612 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:05:47.612 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:05:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dfc0a07b0 msgr2=0x7f1dfc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dfc0a07b0 0x7f1dfc0a2c70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c011550 tx=0x7f1e0c009260 comp rx=0 tx=0).stop 2023-09-10T01:05:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10101000 msgr2=0x7f1e10072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10101000 0x7f1e10072e40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009990 tx=0x7f1e00009580 comp rx=0 tx=0).stop 2023-09-10T01:05:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 shutdown_connections 2023-09-10T01:05:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1dfc0a07b0 0x7f1dfc0a2c70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e10100640 0x7f1e10072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e10106650 0x7f1e101a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 --2- 172.21.15.3:0/2328912709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e10101000 0x7f1e10072e40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 >> 172.21.15.3:0/2328912709 conn(0x7f1e100fc170 msgr2=0x7f1e100fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 shutdown_connections 2023-09-10T01:05:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.597+0000 7f1e177ff700 1 -- 172.21.15.3:0/2328912709 wait complete. 2023-09-10T01:05:47.616 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:05:47.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:05:47.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 164255 10.14 MB/sec warmup 89 sec latency 22.021 ms 2023-09-10T01:05:47.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.840+0000 7fdd68bff700 1 -- 172.21.15.3:0/765589675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 msgr2=0x7fdd64111100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.840+0000 7fdd68bff700 1 --2- 172.21.15.3:0/765589675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd64111100 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009990 tx=0x7fdd58009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.841+0000 7fdd68bff700 1 -- 172.21.15.3:0/765589675 shutdown_connections 2023-09-10T01:05:47.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.841+0000 7fdd68bff700 1 --2- 172.21.15.3:0/765589675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd64111640 0x7fdd64113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.841+0000 7fdd68bff700 1 --2- 172.21.15.3:0/765589675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd640729a0 0x7fdd64072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.841+0000 7fdd68bff700 1 --2- 172.21.15.3:0/765589675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd64111100 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.841+0000 7fdd68bff700 1 -- 172.21.15.3:0/765589675 >> 172.21.15.3:0/765589675 conn(0x7fdd640fc140 msgr2=0x7fdd640fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:47.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.841+0000 7fdd68bff700 1 -- 172.21.15.3:0/765589675 shutdown_connections 2023-09-10T01:05:47.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.841+0000 7fdd68bff700 1 -- 172.21.15.3:0/765589675 wait complete. 2023-09-10T01:05:47.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.842+0000 7fdd68bff700 1 Processor -- start 2023-09-10T01:05:47.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.842+0000 7fdd68bff700 1 -- start start 2023-09-10T01:05:47.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.842+0000 7fdd68bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd640729a0 0x7fdd6419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.842+0000 7fdd68bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd6419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.842+0000 7fdd68bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd64111640 0x7fdd641a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd68bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd641a1dd0 con 0x7fdd640729a0 2023-09-10T01:05:47.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd68bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd641a1f40 con 0x7fdd64111640 2023-09-10T01:05:47.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd68bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd641a2100 con 0x7fdd640732c0 2023-09-10T01:05:47.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd6419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd62d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd64111640 0x7fdd641a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd6259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd640729a0 0x7fdd6419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd6419d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40310/0 (socket says 172.21.15.3:40310) 2023-09-10T01:05:47.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 -- 172.21.15.3:0/3312984800 learned_addr learned my addr 172.21.15.3:0/3312984800 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:47.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd62d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd64111640 0x7fdd641a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53088/0 (socket says 172.21.15.3:53088) 2023-09-10T01:05:47.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 -- 172.21.15.3:0/3312984800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd64111640 msgr2=0x7fdd641a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd64111640 0x7fdd641a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 -- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd640729a0 msgr2=0x7fdd6419cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:47.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd640729a0 0x7fdd6419cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:47.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 -- 172.21.15.3:0/3312984800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580095f0 con 0x7fdd640732c0 2023-09-10T01:05:47.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.843+0000 7fdd61d9b700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd6419d330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdd58005910 tx=0x7fdd58005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.844+0000 7fdd537fe700 1 -- 172.21.15.3:0/3312984800 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5801d070 con 0x7fdd640732c0 2023-09-10T01:05:47.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.844+0000 7fdd537fe700 1 -- 172.21.15.3:0/3312984800 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd58022470 con 0x7fdd640732c0 2023-09-10T01:05:47.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.844+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd641a2380 con 0x7fdd640732c0 2023-09-10T01:05:47.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.844+0000 7fdd537fe700 1 -- 172.21.15.3:0/3312984800 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58017650 con 0x7fdd640732c0 2023-09-10T01:05:47.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.844+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd641a9c20 con 0x7fdd640732c0 2023-09-10T01:05:47.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.845+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd64103b40 con 0x7fdd640732c0 2023-09-10T01:05:47.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.847+0000 7fdd537fe700 1 -- 172.21.15.3:0/3312984800 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fdd580225e0 con 0x7fdd640732c0 2023-09-10T01:05:47.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.848+0000 7fdd537fe700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd4c0a0830 0x7fdd4c0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:47.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.848+0000 7fdd537fe700 1 -- 172.21.15.3:0/3312984800 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdd580cdf80 con 0x7fdd640732c0 2023-09-10T01:05:47.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.849+0000 7fdd6259c700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd4c0a0830 0x7fdd4c0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.849+0000 7fdd6259c700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd4c0a0830 0x7fdd4c0a2cf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009c60 tx=0x7fdd54009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:47.850+0000 7fdd537fe700 1 -- 172.21.15.3:0/3312984800 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fdd58097410 con 0x7fdd640732c0 2023-09-10T01:05:48.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.126+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd6419e210 con 0x7fdd640732c0 2023-09-10T01:05:48.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.127+0000 7fdd537fe700 1 -- 172.21.15.3:0/3312984800 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11295 (secure 0 0 0) 0x7fdd5809aa30 con 0x7fdd640732c0 2023-09-10T01:05:48.128 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:05:48.128 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:04:38.815186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:05:48.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd4c0a0830 msgr2=0x7fdd4c0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd4c0a0830 0x7fdd4c0a2cf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009c60 tx=0x7fdd54009260 comp rx=0 tx=0).stop 2023-09-10T01:05:48.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 msgr2=0x7fdd6419d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd6419d330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdd58005910 tx=0x7fdd58005720 comp rx=0 tx=0).stop 2023-09-10T01:05:48.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 shutdown_connections 2023-09-10T01:05:48.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd4c0a0830 0x7fdd4c0a2cf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd64111640 0x7fdd641a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd640729a0 0x7fdd6419cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 --2- 172.21.15.3:0/3312984800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd640732c0 0x7fdd6419d330 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 >> 172.21.15.3:0/3312984800 conn(0x7fdd640fc140 msgr2=0x7fdd640fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 shutdown_connections 2023-09-10T01:05:48.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.130+0000 7fdd68bff700 1 -- 172.21.15.3:0/3312984800 wait complete. 2023-09-10T01:05:48.134 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2023-09-10T01:05:48.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:05:48.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.388+0000 7f6c815ff700 1 -- 172.21.15.3:0/1943018322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 msgr2=0x7f6c7c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.388+0000 7f6c815ff700 1 --2- 172.21.15.3:0/1943018322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c10d510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400b0a0 tx=0x7f6c7400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.389+0000 7f6c815ff700 1 -- 172.21.15.3:0/1943018322 shutdown_connections 2023-09-10T01:05:48.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.389+0000 7f6c815ff700 1 --2- 172.21.15.3:0/1943018322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c10d510 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.389+0000 7f6c815ff700 1 --2- 172.21.15.3:0/1943018322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c106650 0x7f6c7c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.389+0000 7f6c815ff700 1 --2- 172.21.15.3:0/1943018322 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c100640 0x7f6c7c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.389+0000 7f6c815ff700 1 -- 172.21.15.3:0/1943018322 >> 172.21.15.3:0/1943018322 conn(0x7f6c7c0fc170 msgr2=0x7f6c7c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.389+0000 7f6c815ff700 1 -- 172.21.15.3:0/1943018322 shutdown_connections 2023-09-10T01:05:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.389+0000 7f6c815ff700 1 -- 172.21.15.3:0/1943018322 wait complete. 2023-09-10T01:05:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 Processor -- start 2023-09-10T01:05:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 -- start start 2023-09-10T01:05:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100640 0x7f6c7c072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c106650 0x7f6c7c1a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c073670 con 0x7f6c7c100640 2023-09-10T01:05:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c1a4d40 con 0x7f6c7c101000 2023-09-10T01:05:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.390+0000 7f6c815ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c1a4e80 con 0x7f6c7c106650 2023-09-10T01:05:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53116/0 (socket says 172.21.15.3:53116) 2023-09-10T01:05:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100640 0x7f6c7c072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/2059107943 learned_addr learned my addr 172.21.15.3:0/2059107943 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c106650 0x7f6c7c1a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7b7fe700 1 -- 172.21.15.3:0/2059107943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 msgr2=0x7f6c7c072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7b7fe700 1 -- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100640 msgr2=0x7f6c7c072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100640 0x7f6c7c072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7b7fe700 1 -- 172.21.15.3:0/2059107943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74009d00 con 0x7f6c7c106650 2023-09-10T01:05:48.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:48.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.391+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c106650 0x7f6c7c1a4800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009fd0 tx=0x7f6c740075c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.392+0000 7f6c63fff700 1 -- 172.21.15.3:0/2059107943 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7400e040 con 0x7f6c7c106650 2023-09-10T01:05:48.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.392+0000 7f6c63fff700 1 -- 172.21.15.3:0/2059107943 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c74004c50 con 0x7f6c7c106650 2023-09-10T01:05:48.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.392+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c1a5080 con 0x7f6c7c106650 2023-09-10T01:05:48.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.392+0000 7f6c63fff700 1 -- 172.21.15.3:0/2059107943 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7401dea0 con 0x7f6c7c106650 2023-09-10T01:05:48.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.392+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c1a5550 con 0x7f6c7c106650 2023-09-10T01:05:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.395+0000 7f6c63fff700 1 -- 172.21.15.3:0/2059107943 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6c7401d470 con 0x7f6c7c106650 2023-09-10T01:05:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.396+0000 7f6c63fff700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6c640a0830 0x7f6c640a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 Processor -- start 2023-09-10T01:05:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c63fff700 1 -- 172.21.15.3:0/2059107943 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6c740cd940 con 0x7f6c7c106650 2023-09-10T01:05:48.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c7affd700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6c640a0830 0x7f6c640a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 -- start start 2023-09-10T01:05:48.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d67af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d71900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d74230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3cc0730 con 0x5629e3d68030 2023-09-10T01:05:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3cbe050 con 0x5629e3d67710 2023-09-10T01:05:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.397+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3cb9940 con 0x5629e3d71e40 2023-09-10T01:05:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d67af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d67af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53120/0 (socket says 172.21.15.3:53120) 2023-09-10T01:05:48.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 -- 172.21.15.3:0/519086784 learned_addr learned my addr 172.21.15.3:0/519086784 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d71900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6c640a0830 0x7f6c640a2cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c005950 tx=0x7f6c6c00a3b0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d74230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 -- 172.21.15.3:0/519086784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 msgr2=0x5629e3d74230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d74230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 -- 172.21.15.3:0/519086784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 msgr2=0x5629e3d71900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d71900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 -- 172.21.15.3:0/519086784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629e3ae5de0 con 0x5629e3d67710 2023-09-10T01:05:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.398+0000 7f6c7affd700 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d67af0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00fdf0 tx=0x7f6c6c012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9f9ad0c85e2f00 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c617fa700 1 -- 172.21.15.3:0/519086784 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c010ac0 con 0x5629e3d67710 2023-09-10T01:05:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c617fa700 1 -- 172.21.15.3:0/519086784 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c6c020070 con 0x5629e3d67710 2023-09-10T01:05:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c617fa700 1 -- 172.21.15.3:0/519086784 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c01b420 con 0x5629e3d67710 2023-09-10T01:05:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/519086784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 msgr2=0x5629e3d67af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d67af0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00fdf0 tx=0x7f6c6c012000 comp rx=0 tx=0).stop 2023-09-10T01:05:48.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/519086784 shutdown_connections 2023-09-10T01:05:48.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d74230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d67af0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/519086784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d71900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.399+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/519086784 >> 172.21.15.3:0/519086784 conn(0x5629e3cdc090 msgr2=0x5629e3d62bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.400+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/519086784 shutdown_connections 2023-09-10T01:05:48.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.400+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/519086784 wait complete. 2023-09-10T01:05:48.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.400+0000 7f6c94fc1b80 1 -- start start 2023-09-10T01:05:48.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 1 Processor -- start 2023-09-10T01:05:48.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d67710 0x5629e3d717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d68030 0x5629e3d6e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d6ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3c57b70 con 0x5629e3d67710 2023-09-10T01:05:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3c57870 con 0x5629e3d68030 2023-09-10T01:05:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3b3b920 con 0x5629e3d71e40 2023-09-10T01:05:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c94fc1b80 10 client.0 _ll_register_callbacks cb 0x5629e3ba1a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d68030 0x5629e3d6e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d68030 0x5629e3d6e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53124/0 (socket says 172.21.15.3:53124) 2023-09-10T01:05:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/3352597848 learned_addr learned my addr 172.21.15.3:0/3352597848 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c7affd700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d67710 0x5629e3d717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.401+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d6ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 msgr2=0x5629e3d6ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d6ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/3352597848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d67710 msgr2=0x5629e3d717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d67710 0x5629e3d717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/3352597848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c00fad0 con 0x5629e3d68030 2023-09-10T01:05:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d6ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7affd700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d67710 0x5629e3d717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d68030 0x5629e3d6e9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6c70012c70 tx=0x7f6c70012f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=669893bcbbb5db78 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c5eff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3352597848 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70017070 con 0x5629e3d68030 2023-09-10T01:05:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5629e3c114f0 con 0x5629e3d68030 2023-09-10T01:05:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.402+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3352597848 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6c70004020 con 0x5629e3d68030 2023-09-10T01:05:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3352597848 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c700122e0 con 0x5629e3d68030 2023-09-10T01:05:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 10 client.26420 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:48.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5629e3acec60 con 0x5629e3d68030 2023-09-10T01:05:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3352597848 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5629e3c114f0 con 0x5629e3d68030 2023-09-10T01:05:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3352597848 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f6c700039c0 con 0x5629e3d68030 2023-09-10T01:05:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 10 client.26420 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 10 client.26420 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:05:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 20 client.26420 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:48.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3d819f0 0x5629e3d83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 4 client.26420 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:48.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629e3cce8a0 con 0x5629e3d819f0 2023-09-10T01:05:48.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.403+0000 7f6c7affd700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3d819f0 0x5629e3d83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.404+0000 7f6c7affd700 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3d819f0 0x5629e3d83e10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c31905afb327cbd7 server_cookie=d81cb95cab2b9a0d in_seq=0 out_seq=0 2023-09-10T01:05:48.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.404+0000 7f6c5eff5700 10 client.26420 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:48.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3352597848 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5629e3cce8a0 con 0x5629e3d819f0 2023-09-10T01:05:48.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c5eff5700 10 client.26420 handle_command_reply: tid=0 2023-09-10T01:05:48.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 client.26420 shutdown 2023-09-10T01:05:48.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d68030 msgr2=0x5629e3d6e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d68030 0x5629e3d6e9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6c70012c70 tx=0x7f6c70012f80 comp rx=0 tx=0).stop 2023-09-10T01:05:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 shutdown_connections 2023-09-10T01:05:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3d819f0 0x5629e3d83e10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d6ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d67710 0x5629e3d717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c5eff5700 0 client.26420 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3352597848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d68030 0x5629e3d6e9c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 >> 172.21.15.3:0/3352597848 conn(0x5629e3cdc090 msgr2=0x5629e3d627e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 shutdown_connections 2023-09-10T01:05:48.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3352597848 wait complete. 2023-09-10T01:05:48.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.405+0000 7f6c94fc1b80 20 client.26420 trim_cache size 0 max 16384 2023-09-10T01:05:48.422 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:48 smithi136 ceph-mon[133708]: pgmap v80: 129 pgs: 129 active+clean; 51 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 279 op/s 2023-09-10T01:05:48.423 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:48 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3312984800' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:05:48.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 Processor -- start 2023-09-10T01:05:48.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 -- start start 2023-09-10T01:05:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d74230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d69cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3d689f0 con 0x5629e3d68030 2023-09-10T01:05:48.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3d5f3d0 con 0x5629e3d67710 2023-09-10T01:05:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3d6f540 con 0x5629e3d71e40 2023-09-10T01:05:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d74230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.419+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d74230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53130/0 (socket says 172.21.15.3:53130) 2023-09-10T01:05:48.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7affd700 1 -- 172.21.15.3:0/74392939 learned_addr learned my addr 172.21.15.3:0/74392939 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d69cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/74392939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 msgr2=0x5629e3d69cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d69cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/74392939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 msgr2=0x5629e3d74230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d74230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/74392939 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629e3b0c250 con 0x5629e3d68030 2023-09-10T01:05:48.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7affd700 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d74230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:48.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7b7fe700 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d69cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:48.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d712e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6c70012c40 tx=0x7f6c7000f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f4f20890aaa4e8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c61ffb700 1 -- 172.21.15.3:0/74392939 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70017070 con 0x5629e3d68030 2023-09-10T01:05:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c61ffb700 1 -- 172.21.15.3:0/74392939 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c70018040 con 0x5629e3d68030 2023-09-10T01:05:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.420+0000 7f6c61ffb700 1 -- 172.21.15.3:0/74392939 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70005330 con 0x5629e3d68030 2023-09-10T01:05:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/74392939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 msgr2=0x5629e3d712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d712e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6c70012c40 tx=0x7f6c7000f6a0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/74392939 shutdown_connections 2023-09-10T01:05:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d71e40 0x5629e3d69cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3d74230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/74392939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d68030 0x5629e3d712e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/74392939 >> 172.21.15.3:0/74392939 conn(0x5629e3cdc090 msgr2=0x5629e3d61320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/74392939 shutdown_connections 2023-09-10T01:05:48.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.421+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/74392939 wait complete. 2023-09-10T01:05:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 -- start start 2023-09-10T01:05:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 Processor -- start 2023-09-10T01:05:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3dce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d68030 0x5629e3dceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d71e40 0x5629e3d76a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3d76f80 con 0x5629e3d71e40 2023-09-10T01:05:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3d770f0 con 0x5629e3d67710 2023-09-10T01:05:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629e3d772b0 con 0x5629e3d68030 2023-09-10T01:05:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.422+0000 7f6c94fc1b80 10 client.0 _ll_register_callbacks cb 0x7f6c70009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:48.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3dce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d71e40 0x5629e3d76a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7a7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d68030 0x5629e3dceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3dce9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53142/0 (socket says 172.21.15.3:53142) 2023-09-10T01:05:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7affd700 1 -- 172.21.15.3:0/3906833822 learned_addr learned my addr 172.21.15.3:0/3906833822 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/3906833822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 msgr2=0x5629e3dce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3dce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/3906833822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d71e40 msgr2=0x5629e3d76a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d71e40 0x5629e3d76a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7a7fc700 1 -- 172.21.15.3:0/3906833822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c029040 con 0x5629e3d68030 2023-09-10T01:05:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c7a7fc700 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d68030 0x5629e3dceee0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5629e3d6f9f0 tx=0x7f6c700123e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d71812bdfe90120b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c5eff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.423+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3906833822 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70017070 con 0x5629e3d68030 2023-09-10T01:05:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3906833822 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6c70003a90 con 0x5629e3d68030 2023-09-10T01:05:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c6c020070 con 0x5629e3d68030 2023-09-10T01:05:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3906833822 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c7000fb80 con 0x5629e3d68030 2023-09-10T01:05:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 10 client.26344 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f6c7c1a5550 con 0x5629e3d68030 2023-09-10T01:05:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3906833822 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c6c020070 con 0x5629e3d68030 2023-09-10T01:05:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3906833822 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f6c7001c7d0 con 0x5629e3d68030 2023-09-10T01:05:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 10 client.26344 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:48.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 10 client.26344 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:05:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 20 client.26344 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3dd64d0 0x5629e3dd88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 4 client.26344 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:05:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5629e3cb2920 con 0x5629e3dd64d0 2023-09-10T01:05:48.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.424+0000 7f6c7affd700 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3dd64d0 0x5629e3dd88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c7affd700 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3dd64d0 0x5629e3dd88f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b33e1bae982110fd server_cookie=e540a3fbd89cd67d in_seq=0 out_seq=0 2023-09-10T01:05:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c5eff5700 10 client.26344 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c5eff5700 1 -- 172.21.15.3:0/3906833822 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5629e3cb2920 con 0x5629e3dd64d0 2023-09-10T01:05:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c5eff5700 10 client.26344 handle_command_reply: tid=0 2023-09-10T01:05:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 client.26344 shutdown 2023-09-10T01:05:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d68030 msgr2=0x5629e3dceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d68030 0x5629e3dceee0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5629e3d6f9f0 tx=0x7f6c700123e0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 shutdown_connections 2023-09-10T01:05:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5629e3dd64d0 0x5629e3dd88f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629e3d71e40 0x5629e3d76a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629e3d67710 0x5629e3dce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 --2- 172.21.15.3:0/3906833822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629e3d68030 0x5629e3dceee0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c5eff5700 0 client.26344 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:05:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.425+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 >> 172.21.15.3:0/3906833822 conn(0x5629e3cdc090 msgr2=0x5629e3d69cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 shutdown_connections 2023-09-10T01:05:48.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c94fc1b80 1 -- 172.21.15.3:0/3906833822 wait complete. 2023-09-10T01:05:48.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c94fc1b80 20 client.26344 trim_cache size 0 max 16384 2023-09-10T01:05:48.444 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:05:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6c640a0830 msgr2=0x7f6c640a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6c640a0830 0x7f6c640a2cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c005950 tx=0x7f6c6c00a3b0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c106650 msgr2=0x7f6c7c1a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c106650 0x7f6c7c1a4800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009fd0 tx=0x7f6c740075c0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 shutdown_connections 2023-09-10T01:05:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6c640a0830 0x7f6c640a2cf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c7c106650 0x7f6c7c1a4800 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c7c100640 0x7f6c7c072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 --2- 172.21.15.3:0/2059107943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c7c101000 0x7f6c7c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.426+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 >> 172.21.15.3:0/2059107943 conn(0x7f6c7c0fc170 msgr2=0x7f6c7c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.427+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 shutdown_connections 2023-09-10T01:05:48.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.427+0000 7f6c815ff700 1 -- 172.21.15.3:0/2059107943 wait complete. 2023-09-10T01:05:48.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:48 smithi003 ceph-mon[97654]: pgmap v80: 129 pgs: 129 active+clean; 51 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 279 op/s 2023-09-10T01:05:48.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:48 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3312984800' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:05:48.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:05:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:48 smithi092 ceph-mon[134067]: pgmap v80: 129 pgs: 129 active+clean; 51 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 279 op/s 2023-09-10T01:05:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:48 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3312984800' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:05:48.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.668+0000 7f036a7ff700 1 -- 172.21.15.3:0/513511064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364106650 msgr2=0x7f0364106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.668+0000 7f036a7ff700 1 --2- 172.21.15.3:0/513511064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364106650 0x7f0364106a30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0354009990 tx=0x7f0354009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.669+0000 7f036a7ff700 1 -- 172.21.15.3:0/513511064 shutdown_connections 2023-09-10T01:05:48.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.669+0000 7f036a7ff700 1 --2- 172.21.15.3:0/513511064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0364101000 0x7f036410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.669+0000 7f036a7ff700 1 --2- 172.21.15.3:0/513511064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364106650 0x7f0364106a30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.669+0000 7f036a7ff700 1 --2- 172.21.15.3:0/513511064 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364100640 0x7f0364100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.669+0000 7f036a7ff700 1 -- 172.21.15.3:0/513511064 >> 172.21.15.3:0/513511064 conn(0x7f03640fc170 msgr2=0x7f03640fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.669+0000 7f036a7ff700 1 -- 172.21.15.3:0/513511064 shutdown_connections 2023-09-10T01:05:48.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.669+0000 7f036a7ff700 1 -- 172.21.15.3:0/513511064 wait complete. 2023-09-10T01:05:48.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 Processor -- start 2023-09-10T01:05:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 -- start start 2023-09-10T01:05:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364100640 0x7f0364196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364101000 0x7f0364196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0364106650 0x7f036419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036419a610 con 0x7f0364106650 2023-09-10T01:05:48.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036419a780 con 0x7f0364100640 2023-09-10T01:05:48.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f036a7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036419a940 con 0x7f0364101000 2023-09-10T01:05:48.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f0363fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364100640 0x7f0364196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f03637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364101000 0x7f0364196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f0363fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364100640 0x7f0364196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53170/0 (socket says 172.21.15.3:53170) 2023-09-10T01:05:48.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f03637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364101000 0x7f0364196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40394/0 (socket says 172.21.15.3:40394) 2023-09-10T01:05:48.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.670+0000 7f0363fff700 1 -- 172.21.15.3:0/1449974678 learned_addr learned my addr 172.21.15.3:0/1449974678 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f0368d9c700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0364106650 0x7f036419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03637fe700 1 -- 172.21.15.3:0/1449974678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364100640 msgr2=0x7f0364196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03637fe700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364100640 0x7f0364196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03637fe700 1 -- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0364106650 msgr2=0x7f036419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03637fe700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0364106650 0x7f036419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03637fe700 1 -- 172.21.15.3:0/1449974678 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0358009520 con 0x7f0364101000 2023-09-10T01:05:48.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03637fe700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364101000 0x7f0364196c60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f035800e9f0 tx=0x7f035800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03617fa700 1 -- 172.21.15.3:0/1449974678 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035800cb70 con 0x7f0364101000 2023-09-10T01:05:48.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03617fa700 1 -- 172.21.15.3:0/1449974678 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f035800ee80 con 0x7f0364101000 2023-09-10T01:05:48.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f03617fa700 1 -- 172.21.15.3:0/1449974678 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358003d30 con 0x7f0364101000 2023-09-10T01:05:48.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.671+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c009c40 con 0x7f0364101000 2023-09-10T01:05:48.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.672+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03540095f0 con 0x7f0364101000 2023-09-10T01:05:48.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.677+0000 7f03617fa700 1 -- 172.21.15.3:0/1449974678 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f035800ccd0 con 0x7f0364101000 2023-09-10T01:05:48.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 Processor -- start 2023-09-10T01:05:48.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 -- start start 2023-09-10T01:05:48.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4e4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4ee900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4eee40 0x564dad4f1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad43d730 con 0x564dad4eee40 2023-09-10T01:05:48.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad43b050 con 0x564dad4e5030 2023-09-10T01:05:48.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad436940 con 0x564dad4e4710 2023-09-10T01:05:48.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f03617fa700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f034c0a0830 0x7f034c0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f0363fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f0363fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4ee900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53178/0 (socket says 172.21.15.3:53178) 2023-09-10T01:05:48.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f0363fff700 1 -- 172.21.15.3:0/3846152786 learned_addr learned my addr 172.21.15.3:0/3846152786 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f03617fa700 1 -- 172.21.15.3:0/1449974678 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0358014070 con 0x7f0364101000 2023-09-10T01:05:48.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.678+0000 7f0368d9c700 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4e4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f03637fe700 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4eee40 0x564dad4f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0368d9c700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f034c0a0830 0x7f034c0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0363fff700 1 -- 172.21.15.3:0/3846152786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 msgr2=0x564dad4e4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0363fff700 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4e4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0363fff700 1 -- 172.21.15.3:0/3846152786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4eee40 msgr2=0x564dad4f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0363fff700 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4eee40 0x564dad4f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0363fff700 1 -- 172.21.15.3:0/3846152786 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564dad262de0 con 0x564dad4e5030 2023-09-10T01:05:48.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f03637fe700 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4eee40 0x564dad4f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:48.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0363fff700 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4ee900 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0354005f30 tx=0x7f0354005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e39babe2771c85 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f0368d9c700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f034c0a0830 0x7f034c0a2cf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f035c009550 tx=0x7f035c00b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.679+0000 7f034a7fc700 1 -- 172.21.15.3:0/3846152786 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354026070 con 0x564dad4e5030 2023-09-10T01:05:48.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f034a7fc700 1 -- 172.21.15.3:0/3846152786 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f035401e040 con 0x564dad4e5030 2023-09-10T01:05:48.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f034a7fc700 1 -- 172.21.15.3:0/3846152786 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035401d070 con 0x564dad4e5030 2023-09-10T01:05:48.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 -- 172.21.15.3:0/3846152786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 msgr2=0x564dad4ee900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4ee900 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0354005f30 tx=0x7f0354005c00 comp rx=0 tx=0).stop 2023-09-10T01:05:48.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 -- 172.21.15.3:0/3846152786 shutdown_connections 2023-09-10T01:05:48.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4e4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4eee40 0x564dad4f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/3846152786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4ee900 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 -- 172.21.15.3:0/3846152786 >> 172.21.15.3:0/3846152786 conn(0x564dad459090 msgr2=0x564dad4dfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 -- 172.21.15.3:0/3846152786 shutdown_connections 2023-09-10T01:05:48.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.680+0000 7f037e1b0b80 1 -- 172.21.15.3:0/3846152786 wait complete. 2023-09-10T01:05:48.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.681+0000 7f037e1b0b80 1 -- start start 2023-09-10T01:05:48.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.681+0000 7f037e1b0b80 1 Processor -- start 2023-09-10T01:05:48.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4ee740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4eee40 0x564dad4ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad3d4b70 con 0x564dad4e4710 2023-09-10T01:05:48.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad3d4870 con 0x564dad4e5030 2023-09-10T01:05:48.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad2b8920 con 0x564dad4eee40 2023-09-10T01:05:48.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f037e1b0b80 10 client.0 _ll_register_callbacks cb 0x564dad31ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:48.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f0368d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4eee40 0x564dad4ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f0363fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4eb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53184/0 (socket says 172.21.15.3:53184) 2023-09-10T01:05:48.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 -- 172.21.15.3:0/116894832 learned_addr learned my addr 172.21.15.3:0/116894832 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f0368d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4eee40 0x564dad4ebf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40404/0 (socket says 172.21.15.3:40404) 2023-09-10T01:05:48.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 -- 172.21.15.3:0/116894832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4eee40 msgr2=0x564dad4ebf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4eee40 0x564dad4ebf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 -- 172.21.15.3:0/116894832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 msgr2=0x564dad4ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4ee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f03637fe700 1 -- 172.21.15.3:0/116894832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036410de10 con 0x564dad4e5030 2023-09-10T01:05:48.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.682+0000 7f0368d9c700 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4eee40 0x564dad4ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:48.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f03637fe700 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4eb9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f03580995d0 tx=0x7f0358099200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f0a49bb1cb558ba server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f03437fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:48.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f03437fe700 1 -- 172.21.15.3:0/116894832 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358099640 con 0x564dad4e5030 2023-09-10T01:05:48.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564dad38e4f0 con 0x564dad4e5030 2023-09-10T01:05:48.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f03437fe700 1 -- 172.21.15.3:0/116894832 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0358082e40 con 0x564dad4e5030 2023-09-10T01:05:48.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f03437fe700 1 -- 172.21.15.3:0/116894832 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035809d470 con 0x564dad4e5030 2023-09-10T01:05:48.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f037e1b0b80 10 client.26438 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:48.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564dad24bc60 con 0x564dad4e5030 2023-09-10T01:05:48.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.683+0000 7f03437fe700 1 -- 172.21.15.3:0/116894832 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564dad38e4f0 con 0x564dad4e5030 2023-09-10T01:05:48.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f03437fe700 1 -- 172.21.15.3:0/116894832 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f03580cc150 con 0x564dad4e5030 2023-09-10T01:05:48.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f037e1b0b80 10 client.26438 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:48.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f037e1b0b80 10 client.26438 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:05:48.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f037e1b0b80 20 client.26438 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:48.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad4fe700 0x564dad500b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f037e1b0b80 4 client.26438 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:48.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564dad44b8a0 con 0x564dad4fe700 2023-09-10T01:05:48.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.684+0000 7f0363fff700 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad4fe700 0x564dad500b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.685+0000 7f0363fff700 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad4fe700 0x564dad500b20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a544251fa4e716 server_cookie=95eb2834ab8f23a5 in_seq=0 out_seq=0 2023-09-10T01:05:48.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.685+0000 7f03437fe700 10 client.26438 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:48.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.685+0000 7f03437fe700 1 -- 172.21.15.3:0/116894832 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564dad44b8a0 con 0x564dad4fe700 2023-09-10T01:05:48.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.685+0000 7f03437fe700 10 client.26438 handle_command_reply: tid=0 2023-09-10T01:05:48.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 client.26438 shutdown 2023-09-10T01:05:48.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 msgr2=0x564dad4eb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4eb9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f03580995d0 tx=0x7f0358099200 comp rx=0 tx=0).stop 2023-09-10T01:05:48.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 shutdown_connections 2023-09-10T01:05:48.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad4fe700 0x564dad500b20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4eee40 0x564dad4ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/116894832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4e5030 0x564dad4eb9c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f03437fe700 0 client.26438 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:48.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 >> 172.21.15.3:0/116894832 conn(0x564dad459090 msgr2=0x564dad4df7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 shutdown_connections 2023-09-10T01:05:48.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 1 -- 172.21.15.3:0/116894832 wait complete. 2023-09-10T01:05:48.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.686+0000 7f037e1b0b80 20 client.26438 trim_cache size 0 max 16384 2023-09-10T01:05:48.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 Processor -- start 2023-09-10T01:05:48.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 -- start start 2023-09-10T01:05:48.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4ee270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e5030 0x564dad4df6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad4dfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad4e0170 con 0x564dad4e5030 2023-09-10T01:05:48.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad4d9d70 con 0x564dad4eee40 2023-09-10T01:05:48.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad4e7630 con 0x564dad4e4710 2023-09-10T01:05:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f0363fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f0363fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4ee270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40416/0 (socket says 172.21.15.3:40416) 2023-09-10T01:05:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f03637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e5030 0x564dad4df6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f0363fff700 1 -- 172.21.15.3:0/1374907205 learned_addr learned my addr 172.21.15.3:0/1374907205 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.701+0000 7f0368d9c700 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad4dfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f0363fff700 1 -- 172.21.15.3:0/1374907205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 msgr2=0x564dad4dfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f0363fff700 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad4dfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f0363fff700 1 -- 172.21.15.3:0/1374907205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e5030 msgr2=0x564dad4df6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f0363fff700 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e5030 0x564dad4df6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f0363fff700 1 -- 172.21.15.3:0/1374907205 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564dad289250 con 0x564dad4e4710 2023-09-10T01:05:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f0363fff700 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4ee270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f03540098e0 tx=0x7f0354005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d2852e470153ff server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f034affd700 1 -- 172.21.15.3:0/1374907205 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354026070 con 0x564dad4e4710 2023-09-10T01:05:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f034affd700 1 -- 172.21.15.3:0/1374907205 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0354027040 con 0x564dad4e4710 2023-09-10T01:05:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.702+0000 7f034affd700 1 -- 172.21.15.3:0/1374907205 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035401d040 con 0x564dad4e4710 2023-09-10T01:05:48.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 -- 172.21.15.3:0/1374907205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 msgr2=0x564dad4ee270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4ee270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f03540098e0 tx=0x7f0354005720 comp rx=0 tx=0).stop 2023-09-10T01:05:48.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 -- 172.21.15.3:0/1374907205 shutdown_connections 2023-09-10T01:05:48.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e4710 0x564dad4ee270 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad4dfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/1374907205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e5030 0x564dad4df6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 -- 172.21.15.3:0/1374907205 >> 172.21.15.3:0/1374907205 conn(0x564dad459090 msgr2=0x564dad4eb980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 -- 172.21.15.3:0/1374907205 shutdown_connections 2023-09-10T01:05:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.703+0000 7f037e1b0b80 1 -- 172.21.15.3:0/1374907205 wait complete. 2023-09-10T01:05:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 -- start start 2023-09-10T01:05:48.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 Processor -- start 2023-09-10T01:05:48.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e5030 0x564dad4f4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad551530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad551a70 con 0x564dad4e4710 2023-09-10T01:05:48.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad551bb0 con 0x564dad4eee40 2023-09-10T01:05:48.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dad551cf0 con 0x564dad4e5030 2023-09-10T01:05:48.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f037e1b0b80 10 client.0 _ll_register_callbacks cb 0x7f03580959a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:48.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f03637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e5030 0x564dad4f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f0368d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad551530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.704+0000 7f0363fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e5030 0x564dad4f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40418/0 (socket says 172.21.15.3:40418) 2023-09-10T01:05:48.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f0368d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad551530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53202/0 (socket says 172.21.15.3:53202) 2023-09-10T01:05:48.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 -- 172.21.15.3:0/2386041069 learned_addr learned my addr 172.21.15.3:0/2386041069 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 -- 172.21.15.3:0/2386041069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 msgr2=0x564dad551530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad551530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 -- 172.21.15.3:0/2386041069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 msgr2=0x564dad4f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 -- 172.21.15.3:0/2386041069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564dad262de0 con 0x564dad4e5030 2023-09-10T01:05:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03637fe700 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e5030 0x564dad4f4110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03580c9ee0 tx=0x7f03580036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b71e46dd6d20841 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03437fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:48.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03437fe700 1 -- 172.21.15.3:0/2386041069 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358098fa0 con 0x564dad4e5030 2023-09-10T01:05:48.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f035401e040 con 0x564dad4e5030 2023-09-10T01:05:48.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03437fe700 1 -- 172.21.15.3:0/2386041069 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0358082e40 con 0x564dad4e5030 2023-09-10T01:05:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.705+0000 7f03437fe700 1 -- 172.21.15.3:0/2386041069 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035809d470 con 0x564dad4e5030 2023-09-10T01:05:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 10 client.26362 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564dad2612b0 con 0x564dad4e5030 2023-09-10T01:05:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f03437fe700 1 -- 172.21.15.3:0/2386041069 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f035401e040 con 0x564dad4e5030 2023-09-10T01:05:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f03437fe700 1 -- 172.21.15.3:0/2386041069 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f0358098c50 con 0x564dad4e5030 2023-09-10T01:05:48.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 10 client.26362 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 10 client.26362 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:05:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 20 client.26362 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad552e50 0x564dad555240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 4 client.26362 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:05:48.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564dad42f920 con 0x564dad552e50 2023-09-10T01:05:48.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.706+0000 7f0363fff700 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad552e50 0x564dad555240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f0363fff700 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad552e50 0x564dad555240 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef2481694eca2c6f server_cookie=b5060cafec4081c0 in_seq=0 out_seq=0 2023-09-10T01:05:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f03437fe700 10 client.26362 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f03437fe700 1 -- 172.21.15.3:0/2386041069 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564dad42f920 con 0x564dad552e50 2023-09-10T01:05:48.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f03437fe700 10 client.26362 handle_command_reply: tid=0 2023-09-10T01:05:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 client.26362 shutdown 2023-09-10T01:05:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e5030 msgr2=0x564dad4f4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e5030 0x564dad4f4110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03580c9ee0 tx=0x7f03580036f0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 shutdown_connections 2023-09-10T01:05:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564dad4eee40 0x564dad551530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dad4e4710 0x564dad4f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x564dad552e50 0x564dad555240 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 --2- 172.21.15.3:0/2386041069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564dad4e5030 0x564dad4f4110 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 >> 172.21.15.3:0/2386041069 conn(0x564dad459090 msgr2=0x564dad4df670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f03437fe700 0 client.26362 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:05:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 shutdown_connections 2023-09-10T01:05:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.707+0000 7f037e1b0b80 1 -- 172.21.15.3:0/2386041069 wait complete. 2023-09-10T01:05:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f037e1b0b80 20 client.26362 trim_cache size 0 max 16384 2023-09-10T01:05:48.725 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:05:48.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f034c0a0830 msgr2=0x7f034c0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f034c0a0830 0x7f034c0a2cf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f035c009550 tx=0x7f035c00b040 comp rx=0 tx=0).stop 2023-09-10T01:05:48.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364101000 msgr2=0x7f0364196c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364101000 0x7f0364196c60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f035800e9f0 tx=0x7f035800ed00 comp rx=0 tx=0).stop 2023-09-10T01:05:48.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 shutdown_connections 2023-09-10T01:05:48.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f034c0a0830 0x7f034c0a2cf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0364106650 0x7f036419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0364100640 0x7f0364196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 --2- 172.21.15.3:0/1449974678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0364101000 0x7f0364196c60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.708+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 >> 172.21.15.3:0/1449974678 conn(0x7f03640fc170 msgr2=0x7f03640fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.709+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 shutdown_connections 2023-09-10T01:05:48.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.709+0000 7f036a7ff700 1 -- 172.21.15.3:0/1449974678 wait complete. 2023-09-10T01:05:48.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:05:48.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 166735 10.20 MB/sec warmup 90 sec latency 28.528 ms 2023-09-10T01:05:48.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 -- 172.21.15.3:0/2454057013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868100640 msgr2=0x7f2868100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 --2- 172.21.15.3:0/2454057013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868100640 0x7f2868100ac0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f285c009990 tx=0x7f285c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 -- 172.21.15.3:0/2454057013 shutdown_connections 2023-09-10T01:05:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 --2- 172.21.15.3:0/2454057013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2868106650 0x7f2868106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 --2- 172.21.15.3:0/2454057013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868101000 0x7f286810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 --2- 172.21.15.3:0/2454057013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868100640 0x7f2868100ac0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 -- 172.21.15.3:0/2454057013 >> 172.21.15.3:0/2454057013 conn(0x7f28680fc170 msgr2=0x7f28680fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 -- 172.21.15.3:0/2454057013 shutdown_connections 2023-09-10T01:05:48.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.943+0000 7f286d9ff700 1 -- 172.21.15.3:0/2454057013 wait complete. 2023-09-10T01:05:48.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.944+0000 7f286d9ff700 1 Processor -- start 2023-09-10T01:05:48.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.945+0000 7f286d9ff700 1 -- start start 2023-09-10T01:05:48.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.945+0000 7f286d9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868100640 0x7f2868072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.945+0000 7f286d9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868101000 0x7f2868072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.945+0000 7f286d9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2868106650 0x7f28681a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.945+0000 7f286d9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2868073670 con 0x7f2868101000 2023-09-10T01:05:48.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.945+0000 7f286d9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28681a4e20 con 0x7f2868100640 2023-09-10T01:05:48.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.945+0000 7f286d9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28681a4f60 con 0x7f2868106650 2023-09-10T01:05:48.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868100640 0x7f2868072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868100640 0x7f2868072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53210/0 (socket says 172.21.15.3:53210) 2023-09-10T01:05:48.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 -- 172.21.15.3:0/1034303726 learned_addr learned my addr 172.21.15.3:0/1034303726 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f28667fc700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868101000 0x7f2868072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f28677fe700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2868106650 0x7f28681a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 -- 172.21.15.3:0/1034303726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2868106650 msgr2=0x7f28681a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2868106650 0x7f28681a48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 -- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868101000 msgr2=0x7f2868072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868101000 0x7f2868072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 -- 172.21.15.3:0/1034303726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c0095f0 con 0x7f2868100640 2023-09-10T01:05:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868100640 0x7f2868072900 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f285800e9a0 tx=0x7f285800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.946+0000 7f286c9fd700 1 -- 172.21.15.3:0/1034303726 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858009790 con 0x7f2868100640 2023-09-10T01:05:48.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.947+0000 7f286c9fd700 1 -- 172.21.15.3:0/1034303726 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2858010430 con 0x7f2868100640 2023-09-10T01:05:48.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.947+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28681a51c0 con 0x7f2868100640 2023-09-10T01:05:48.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.947+0000 7f286c9fd700 1 -- 172.21.15.3:0/1034303726 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858018680 con 0x7f2868100640 2023-09-10T01:05:48.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.947+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28681a56a0 con 0x7f2868100640 2023-09-10T01:05:48.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.950+0000 7f286c9fd700 1 -- 172.21.15.3:0/1034303726 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f28580105a0 con 0x7f2868100640 2023-09-10T01:05:48.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.951+0000 7f2881313b80 1 Processor -- start 2023-09-10T01:05:48.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.951+0000 7f2881313b80 1 -- start start 2023-09-10T01:05:48.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2881313b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca352af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2881313b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca35c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2881313b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca35ce40 0x55c9ca35f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2881313b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca2ab730 con 0x55c9ca352710 2023-09-10T01:05:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2881313b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca2a9050 con 0x55c9ca353030 2023-09-10T01:05:48.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2881313b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca2a4940 con 0x55c9ca35ce40 2023-09-10T01:05:48.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca35c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f28677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca352af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f286c9fd700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f28500a07b0 0x7f28500a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f28677fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca352af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58024/0 (socket says 172.21.15.3:58024) 2023-09-10T01:05:48.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f28677fe700 1 -- 172.21.15.3:0/1107392517 learned_addr learned my addr 172.21.15.3:0/1107392517 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f28667fc700 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca35ce40 0x55c9ca35f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f286c9fd700 1 -- 172.21.15.3:0/1034303726 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f2858014070 con 0x7f2868100640 2023-09-10T01:05:48.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f28677fe700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f28500a07b0 0x7f28500a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2866ffd700 1 -- 172.21.15.3:0/1107392517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca35ce40 msgr2=0x55c9ca35f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca35ce40 0x55c9ca35f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2866ffd700 1 -- 172.21.15.3:0/1107392517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 msgr2=0x55c9ca352af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca352af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.952+0000 7f2866ffd700 1 -- 172.21.15.3:0/1107392517 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9ca0d0de0 con 0x55c9ca353030 2023-09-10T01:05:48.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.953+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca35c900 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f28580c9800 tx=0x7f2858098ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d3538d18d77fd72 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.953+0000 7f284dffb700 1 -- 172.21.15.3:0/1107392517 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858099560 con 0x55c9ca353030 2023-09-10T01:05:48.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.953+0000 7f284dffb700 1 -- 172.21.15.3:0/1107392517 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f285809d390 con 0x55c9ca353030 2023-09-10T01:05:48.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.953+0000 7f284dffb700 1 -- 172.21.15.3:0/1107392517 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28580840d0 con 0x55c9ca353030 2023-09-10T01:05:48.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 -- 172.21.15.3:0/1107392517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 msgr2=0x55c9ca35c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f28677fe700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f28500a07b0 0x7f28500a2c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28600060b0 tx=0x7f2860004b40 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca35c900 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f28580c9800 tx=0x7f2858098ec0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 -- 172.21.15.3:0/1107392517 shutdown_connections 2023-09-10T01:05:48.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca35ce40 0x55c9ca35f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca352af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 --2- 172.21.15.3:0/1107392517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca35c900 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 -- 172.21.15.3:0/1107392517 >> 172.21.15.3:0/1107392517 conn(0x55c9ca2c7090 msgr2=0x55c9ca34dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 -- 172.21.15.3:0/1107392517 shutdown_connections 2023-09-10T01:05:48.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.954+0000 7f2881313b80 1 -- 172.21.15.3:0/1107392517 wait complete. 2023-09-10T01:05:48.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 -- start start 2023-09-10T01:05:48.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 Processor -- start 2023-09-10T01:05:48.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca35c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca353030 0x55c9ca359b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca35a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca242b70 con 0x55c9ca352710 2023-09-10T01:05:48.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca242870 con 0x55c9ca35ce40 2023-09-10T01:05:48.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.955+0000 7f2881313b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca126920 con 0x55c9ca353030 2023-09-10T01:05:48.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f2881313b80 10 client.0 _ll_register_callbacks cb 0x55c9ca18ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:48.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca353030 0x55c9ca359b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28677fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca35a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca35c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca353030 0x55c9ca359b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40476/0 (socket says 172.21.15.3:40476) 2023-09-10T01:05:48.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 -- 172.21.15.3:0/3337742175 learned_addr learned my addr 172.21.15.3:0/3337742175 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 -- 172.21.15.3:0/3337742175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 msgr2=0x55c9ca35a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca35a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 -- 172.21.15.3:0/3337742175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 msgr2=0x55c9ca35c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca35c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 -- 172.21.15.3:0/3337742175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28681a56a0 con 0x55c9ca353030 2023-09-10T01:05:48.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f28667fc700 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca353030 0x55c9ca359b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f285c015000 tx=0x7f285c00bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efa8821e814f0048 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.956+0000 7f2846ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:48.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.957+0000 7f2846ffd700 1 -- 172.21.15.3:0/3337742175 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c01d070 con 0x55c9ca353030 2023-09-10T01:05:48.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.957+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c9ca1fc4f0 con 0x55c9ca353030 2023-09-10T01:05:48.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.957+0000 7f2846ffd700 1 -- 172.21.15.3:0/3337742175 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f285c004500 con 0x55c9ca353030 2023-09-10T01:05:48.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.957+0000 7f2846ffd700 1 -- 172.21.15.3:0/3337742175 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f285c02d7f0 con 0x55c9ca353030 2023-09-10T01:05:48.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.957+0000 7f2881313b80 10 client.26374 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:48.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.957+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c9ca0b9c60 con 0x55c9ca353030 2023-09-10T01:05:48.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.957+0000 7f2846ffd700 1 -- 172.21.15.3:0/3337742175 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c9ca1fc4f0 con 0x55c9ca353030 2023-09-10T01:05:48.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2846ffd700 1 -- 172.21.15.3:0/3337742175 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f285c004750 con 0x55c9ca353030 2023-09-10T01:05:48.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2881313b80 10 client.26374 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:48.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2881313b80 10 client.26374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:48.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2881313b80 20 client.26374 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:48.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2881313b80 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca36c9b0 0x55c9ca36edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2881313b80 4 client.26374 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:48.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c9ca2b98a0 con 0x55c9ca36c9b0 2023-09-10T01:05:48.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2866ffd700 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca36c9b0 0x55c9ca36edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2866ffd700 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca36c9b0 0x55c9ca36edd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23e2fcaf86eb8157 server_cookie=694b674385a51843 in_seq=0 out_seq=0 2023-09-10T01:05:48.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.958+0000 7f2846ffd700 10 client.26374 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:48.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2846ffd700 1 -- 172.21.15.3:0/3337742175 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c9ca2b98a0 con 0x55c9ca36c9b0 2023-09-10T01:05:48.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2846ffd700 10 client.26374 handle_command_reply: tid=0 2023-09-10T01:05:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 client.26374 shutdown 2023-09-10T01:05:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca353030 msgr2=0x55c9ca359b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca353030 0x55c9ca359b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f285c015000 tx=0x7f285c00bf40 comp rx=0 tx=0).stop 2023-09-10T01:05:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 shutdown_connections 2023-09-10T01:05:48.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca36c9b0 0x55c9ca36edd0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca35a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca352710 0x55c9ca35c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 --2- 172.21.15.3:0/3337742175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca353030 0x55c9ca359b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2846ffd700 0 client.26374 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:48.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 >> 172.21.15.3:0/3337742175 conn(0x55c9ca2c7090 msgr2=0x55c9ca34d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 shutdown_connections 2023-09-10T01:05:48.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.960+0000 7f2881313b80 1 -- 172.21.15.3:0/3337742175 wait complete. 2023-09-10T01:05:48.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.961+0000 7f2881313b80 20 client.26374 trim_cache size 0 max 16384 2023-09-10T01:05:48.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.974+0000 7f2881313b80 1 Processor -- start 2023-09-10T01:05:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2881313b80 1 -- start start 2023-09-10T01:05:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2881313b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca359ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2881313b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca368ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2881313b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca35ce40 0x55c9ca35c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2881313b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca354890 con 0x55c9ca35ce40 2023-09-10T01:05:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2881313b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca35a840 con 0x55c9ca353030 2023-09-10T01:05:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2881313b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca35f5f0 con 0x55c9ca352710 2023-09-10T01:05:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f28667fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca368ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca359ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f28667fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca368ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53238/0 (socket says 172.21.15.3:53238) 2023-09-10T01:05:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f28667fc700 1 -- 172.21.15.3:0/3376341750 learned_addr learned my addr 172.21.15.3:0/3376341750 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f28677fe700 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca35ce40 0x55c9ca35c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2866ffd700 1 -- 172.21.15.3:0/3376341750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 msgr2=0x55c9ca368ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2866ffd700 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca368ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2866ffd700 1 -- 172.21.15.3:0/3376341750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca35ce40 msgr2=0x55c9ca35c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2866ffd700 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca35ce40 0x55c9ca35c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f2866ffd700 1 -- 172.21.15.3:0/3376341750 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9ca0f7250 con 0x55c9ca352710 2023-09-10T01:05:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f28667fc700 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca368ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:48.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.975+0000 7f28677fe700 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca35ce40 0x55c9ca35c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.976+0000 7f2866ffd700 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca359ac0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55c9ca353560 tx=0x7f2858099da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6555c6082f230680 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.976+0000 7f284e7fc700 1 -- 172.21.15.3:0/3376341750 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28580995e0 con 0x55c9ca352710 2023-09-10T01:05:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.976+0000 7f284e7fc700 1 -- 172.21.15.3:0/3376341750 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2858010880 con 0x55c9ca352710 2023-09-10T01:05:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.976+0000 7f284e7fc700 1 -- 172.21.15.3:0/3376341750 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858099a20 con 0x55c9ca352710 2023-09-10T01:05:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 -- 172.21.15.3:0/3376341750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 msgr2=0x55c9ca359ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca359ac0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55c9ca353560 tx=0x7f2858099da0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 -- 172.21.15.3:0/3376341750 shutdown_connections 2023-09-10T01:05:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca359ac0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca35ce40 0x55c9ca35c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 --2- 172.21.15.3:0/3376341750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca353030 0x55c9ca368ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 -- 172.21.15.3:0/3376341750 >> 172.21.15.3:0/3376341750 conn(0x55c9ca2c7090 msgr2=0x55c9ca373730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 -- 172.21.15.3:0/3376341750 shutdown_connections 2023-09-10T01:05:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 -- 172.21.15.3:0/3376341750 wait complete. 2023-09-10T01:05:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.977+0000 7f2881313b80 1 -- start start 2023-09-10T01:05:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 1 Processor -- start 2023-09-10T01:05:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca361670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca353030 0x55c9ca361bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca3bf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca3bfd20 con 0x55c9ca353030 2023-09-10T01:05:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca3bfe60 con 0x55c9ca35ce40 2023-09-10T01:05:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9ca3bffa0 con 0x55c9ca352710 2023-09-10T01:05:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2881313b80 10 client.0 _ll_register_callbacks cb 0x7f285c009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca361670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2866ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca361670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40496/0 (socket says 172.21.15.3:40496) 2023-09-10T01:05:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f2866ffd700 1 -- 172.21.15.3:0/1154559772 learned_addr learned my addr 172.21.15.3:0/1154559772 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f28667fc700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca353030 0x55c9ca361bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.978+0000 7f28677fe700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca3bf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2866ffd700 1 -- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 msgr2=0x55c9ca3bf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca3bf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2866ffd700 1 -- 172.21.15.3:0/1154559772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca353030 msgr2=0x55c9ca361bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca353030 0x55c9ca361bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2866ffd700 1 -- 172.21.15.3:0/1154559772 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9ca0b9c60 con 0x55c9ca352710 2023-09-10T01:05:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f28677fe700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca3bf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f28667fc700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca353030 0x55c9ca361bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2866ffd700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca361670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2858009230 tx=0x7f28580cc520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d616d9b994084836 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2846ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2846ffd700 1 -- 172.21.15.3:0/1154559772 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858098a50 con 0x55c9ca352710 2023-09-10T01:05:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f285809d390 con 0x55c9ca352710 2023-09-10T01:05:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2846ffd700 1 -- 172.21.15.3:0/1154559772 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f2858098bb0 con 0x55c9ca352710 2023-09-10T01:05:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2846ffd700 1 -- 172.21.15.3:0/1154559772 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28580651a0 con 0x55c9ca352710 2023-09-10T01:05:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2881313b80 10 client.26380 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c9ca0d0de0 con 0x55c9ca352710 2023-09-10T01:05:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.979+0000 7f2846ffd700 1 -- 172.21.15.3:0/1154559772 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f285809d390 con 0x55c9ca352710 2023-09-10T01:05:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2846ffd700 1 -- 172.21.15.3:0/1154559772 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f28580a1140 con 0x55c9ca352710 2023-09-10T01:05:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2881313b80 10 client.26380 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2881313b80 10 client.26380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2881313b80 20 client.26380 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2881313b80 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca3c1230 0x55c9ca3c3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2881313b80 4 client.26380 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:05:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c9ca29d920 con 0x55c9ca3c1230 2023-09-10T01:05:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f28667fc700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca3c1230 0x55c9ca3c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f28667fc700 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca3c1230 0x55c9ca3c3650 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aaa7152ed44a52f server_cookie=b39c703678a718ae in_seq=0 out_seq=0 2023-09-10T01:05:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.980+0000 7f2846ffd700 10 client.26380 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2846ffd700 1 -- 172.21.15.3:0/1154559772 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c9ca29d920 con 0x55c9ca3c1230 2023-09-10T01:05:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2846ffd700 10 client.26380 handle_command_reply: tid=0 2023-09-10T01:05:48.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 client.26380 shutdown 2023-09-10T01:05:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 msgr2=0x55c9ca361670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca361670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2858009230 tx=0x7f28580cc520 comp rx=0 tx=0).stop 2023-09-10T01:05:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 shutdown_connections 2023-09-10T01:05:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c9ca3c1230 0x55c9ca3c3650 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c9ca352710 0x55c9ca361670 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9ca35ce40 0x55c9ca3bf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2846ffd700 0 client.26380 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:49.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 --2- 172.21.15.3:0/1154559772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9ca353030 0x55c9ca361bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 >> 172.21.15.3:0/1154559772 conn(0x55c9ca2c7090 msgr2=0x55c9ca368ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:49.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 shutdown_connections 2023-09-10T01:05:49.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 1 -- 172.21.15.3:0/1154559772 wait complete. 2023-09-10T01:05:49.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.981+0000 7f2881313b80 20 client.26380 trim_cache size 0 max 16384 2023-09-10T01:05:49.001 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:05:49.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f28500a07b0 msgr2=0x7f28500a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:49.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f28500a07b0 0x7f28500a2c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28600060b0 tx=0x7f2860004b40 comp rx=0 tx=0).stop 2023-09-10T01:05:49.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868100640 msgr2=0x7f2868072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:49.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868100640 0x7f2868072900 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f285800e9a0 tx=0x7f285800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 shutdown_connections 2023-09-10T01:05:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f28500a07b0 0x7f28500a2c70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2868106650 0x7f28681a48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2868100640 0x7f2868072900 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 --2- 172.21.15.3:0/1034303726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2868101000 0x7f2868072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 >> 172.21.15.3:0/1034303726 conn(0x7f28680fc170 msgr2=0x7f28680fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:49.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 shutdown_connections 2023-09-10T01:05:49.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:48.982+0000 7f286d9ff700 1 -- 172.21.15.3:0/1034303726 wait complete. 2023-09-10T01:05:49.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 167711 10.12 MB/sec warmup 91 sec latency 50.710 ms 2023-09-10T01:05:50.006 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:05:50.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:05:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/161632024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 msgr2=0x7fb7e8106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/161632024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8106a30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009990 tx=0x7fb7d8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/161632024 shutdown_connections 2023-09-10T01:05:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/161632024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7e8101000 0x7fb7e810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/161632024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8106a30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/161632024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 0x7fb7e8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/161632024 >> 172.21.15.3:0/161632024 conn(0x7fb7e80fc170 msgr2=0x7fb7e80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/161632024 shutdown_connections 2023-09-10T01:05:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.224+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/161632024 wait complete. 2023-09-10T01:05:50.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 Processor -- start 2023-09-10T01:05:50.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 -- start start 2023-09-10T01:05:50.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 0x7fb7e8196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7e8101000 0x7fb7e8196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e819a5f0 con 0x7fb7e8101000 2023-09-10T01:05:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e819a760 con 0x7fb7e8106650 2023-09-10T01:05:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.225+0000 7fb7ee3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e819a920 con 0x7fb7e8100640 2023-09-10T01:05:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7e7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 0x7fb7e8196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53266/0 (socket says 172.21.15.3:53266) 2023-09-10T01:05:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7e8101000 0x7fb7e8196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 -- 172.21.15.3:0/33290565 learned_addr learned my addr 172.21.15.3:0/33290565 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7e7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 0x7fb7e8196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40512/0 (socket says 172.21.15.3:40512) 2023-09-10T01:05:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 -- 172.21.15.3:0/33290565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 msgr2=0x7fb7e8196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 0x7fb7e8196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 -- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7e8101000 msgr2=0x7fb7e8196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7e8101000 0x7fb7e8196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 -- 172.21.15.3:0/33290565 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d80095f0 con 0x7fb7e8106650 2023-09-10T01:05:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 0x7fb7e8196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.226+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8199ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000ef30 tx=0x7fb7e000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.227+0000 7fb7e57fa700 1 -- 172.21.15.3:0/33290565 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0018070 con 0x7fb7e8106650 2023-09-10T01:05:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.227+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e819aba0 con 0x7fb7e8106650 2023-09-10T01:05:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.227+0000 7fb7e57fa700 1 -- 172.21.15.3:0/33290565 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7e000f040 con 0x7fb7e8106650 2023-09-10T01:05:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.227+0000 7fb7e57fa700 1 -- 172.21.15.3:0/33290565 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e000cd00 con 0x7fb7e8106650 2023-09-10T01:05:50.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.227+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e81a3560 con 0x7fb7e8106650 2023-09-10T01:05:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.230+0000 7fb7e57fa700 1 -- 172.21.15.3:0/33290565 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fb7e0003a00 con 0x7fb7e8106650 2023-09-10T01:05:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.231+0000 7fb801d19b80 1 Processor -- start 2023-09-10T01:05:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.231+0000 7fb801d19b80 1 -- start start 2023-09-10T01:05:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dfea3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfead980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7e57fa700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb7d00a0830 0x7fb7d00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb801d19b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfdfc7b0 con 0x5590dfea3790 2023-09-10T01:05:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb801d19b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfdfa0d0 con 0x5590dfea40b0 2023-09-10T01:05:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb801d19b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfdf59c0 con 0x5590dfeadec0 2023-09-10T01:05:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7e57fa700 1 -- 172.21.15.3:0/33290565 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fb7e00cbd80 con 0x7fb7e8106650 2023-09-10T01:05:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dfea3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dfea3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58078/0 (socket says 172.21.15.3:58078) 2023-09-10T01:05:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7e77fe700 1 -- 172.21.15.3:0/3662453138 learned_addr learned my addr 172.21.15.3:0/3662453138 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb7d00a0830 0x7fb7d00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.232+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfead980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ec99c700 1 -- 172.21.15.3:0/3662453138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 msgr2=0x5590dfead980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfead980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ec99c700 1 -- 172.21.15.3:0/3662453138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 msgr2=0x5590dfea3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dfea3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ec99c700 1 -- 172.21.15.3:0/3662453138 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590dfc21e60 con 0x5590dfeadec0 2023-09-10T01:05:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb02b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00c90e0 tx=0x7fb7e0098a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74420507e3aa4bcf server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ce7fc700 1 -- 172.21.15.3:0/3662453138 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0004ad0 con 0x5590dfeadec0 2023-09-10T01:05:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ce7fc700 1 -- 172.21.15.3:0/3662453138 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7e001b320 con 0x5590dfeadec0 2023-09-10T01:05:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.233+0000 7fb7ce7fc700 1 -- 172.21.15.3:0/3662453138 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e009cd00 con 0x5590dfeadec0 2023-09-10T01:05:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 -- 172.21.15.3:0/3662453138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 msgr2=0x5590dfeb02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb02b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00c90e0 tx=0x7fb7e0098a10 comp rx=0 tx=0).stop 2023-09-10T01:05:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb7d00a0830 0x7fb7d00a2cf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800b2e0 tx=0x7fb7d8005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 -- 172.21.15.3:0/3662453138 shutdown_connections 2023-09-10T01:05:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb02b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dfea3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3662453138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfead980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 -- 172.21.15.3:0/3662453138 >> 172.21.15.3:0/3662453138 conn(0x5590dfe18110 msgr2=0x5590dfe9ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 -- 172.21.15.3:0/3662453138 shutdown_connections 2023-09-10T01:05:50.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.234+0000 7fb801d19b80 1 -- 172.21.15.3:0/3662453138 wait complete. 2023-09-10T01:05:50.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 -- start start 2023-09-10T01:05:50.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 Processor -- start 2023-09-10T01:05:50.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfeaaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfeab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfd93bf0 con 0x5590dfeadec0 2023-09-10T01:05:50.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfd938f0 con 0x5590dfea40b0 2023-09-10T01:05:50.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfc779a0 con 0x5590dfea3790 2023-09-10T01:05:50.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.235+0000 7fb801d19b80 10 client.0 _ll_register_callbacks cb 0x5590dfcddb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:50.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfeab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfeab0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53294/0 (socket says 172.21.15.3:53294) 2023-09-10T01:05:50.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 -- 172.21.15.3:0/109611755 learned_addr learned my addr 172.21.15.3:0/109611755 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfeaaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 -- 172.21.15.3:0/109611755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 msgr2=0x5590dfeaaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfeaaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 -- 172.21.15.3:0/109611755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 msgr2=0x5590dfeab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 -- 172.21.15.3:0/109611755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e81a3560 con 0x5590dfea40b0 2023-09-10T01:05:50.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfeab0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc013910 tx=0x7fb7dc013c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9672252566035334 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:50.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.236+0000 7fb7c77fe700 1 -- 172.21.15.3:0/109611755 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc004030 con 0x5590dfea40b0 2023-09-10T01:05:50.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590dfd4d570 con 0x5590dfea40b0 2023-09-10T01:05:50.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb7c77fe700 1 -- 172.21.15.3:0/109611755 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fb7dc004190 con 0x5590dfea40b0 2023-09-10T01:05:50.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb7c77fe700 1 -- 172.21.15.3:0/109611755 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7dc004350 con 0x5590dfea40b0 2023-09-10T01:05:50.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb801d19b80 10 client.26474 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:50.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5590dfc0ace0 con 0x5590dfea40b0 2023-09-10T01:05:50.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb7c77fe700 1 -- 172.21.15.3:0/109611755 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5590dfd4d570 con 0x5590dfea40b0 2023-09-10T01:05:50.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb7c77fe700 1 -- 172.21.15.3:0/109611755 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fb7dc012b40 con 0x5590dfea40b0 2023-09-10T01:05:50.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb801d19b80 10 client.26474 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:50.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.237+0000 7fb801d19b80 10 client.26474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:50.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.238+0000 7fb801d19b80 20 client.26474 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:50.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.238+0000 7fb801d19b80 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dfebdaf0 0x5590dfebff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.238+0000 7fb801d19b80 4 client.26474 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:50.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.238+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590dfe0a920 con 0x5590dfebdaf0 2023-09-10T01:05:50.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.238+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dfebdaf0 0x5590dfebff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.238+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dfebdaf0 0x5590dfebff10 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ab240730e822956 server_cookie=4c8afff441292da4 in_seq=0 out_seq=0 2023-09-10T01:05:50.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.238+0000 7fb7c77fe700 10 client.26474 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb7c77fe700 1 -- 172.21.15.3:0/109611755 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5590dfe0a920 con 0x5590dfebdaf0 2023-09-10T01:05:50.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb7c77fe700 10 client.26474 handle_command_reply: tid=0 2023-09-10T01:05:50.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 client.26474 shutdown 2023-09-10T01:05:50.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 msgr2=0x5590dfeab0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfeab0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc013910 tx=0x7fb7dc013c20 comp rx=0 tx=0).stop 2023-09-10T01:05:50.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 shutdown_connections 2023-09-10T01:05:50.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dfebdaf0 0x5590dfebff10 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfeaaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb7c77fe700 0 client.26474 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 --2- 172.21.15.3:0/109611755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfeab0e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.239+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 >> 172.21.15.3:0/109611755 conn(0x5590dfe18110 msgr2=0x5590dfe9e860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.240+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 shutdown_connections 2023-09-10T01:05:50.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.240+0000 7fb801d19b80 1 -- 172.21.15.3:0/109611755 wait complete. 2023-09-10T01:05:50.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.240+0000 7fb801d19b80 20 client.26474 trim_cache size 0 max 16384 2023-09-10T01:05:50.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.258+0000 7fb801d19b80 1 Processor -- start 2023-09-10T01:05:50.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.258+0000 7fb801d19b80 1 -- start start 2023-09-10T01:05:50.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfea7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfea8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeaab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb801d19b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfeb0750 con 0x5590dfeadec0 2023-09-10T01:05:50.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb801d19b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfea59b0 con 0x5590dfea40b0 2023-09-10T01:05:50.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb801d19b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfe9f760 con 0x5590dfea3790 2023-09-10T01:05:50.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb7ec99c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeaab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfea8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb7e7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfea7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfea8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53310/0 (socket says 172.21.15.3:53310) 2023-09-10T01:05:50.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.259+0000 7fb7e77fe700 1 -- 172.21.15.3:0/3549125066 learned_addr learned my addr 172.21.15.3:0/3549125066 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7e77fe700 1 -- 172.21.15.3:0/3549125066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 msgr2=0x5590dfea7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfea7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7e77fe700 1 -- 172.21.15.3:0/3549125066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 msgr2=0x5590dfeaab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeaab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7e77fe700 1 -- 172.21.15.3:0/3549125066 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590dfc482d0 con 0x5590dfea40b0 2023-09-10T01:05:50.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfea7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:50.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeaab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:50.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfea8350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009340 tx=0x7fb7dc012fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b8f8a50cb612c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7ceffd700 1 -- 172.21.15.3:0/3549125066 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc005240 con 0x5590dfea40b0 2023-09-10T01:05:50.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7ceffd700 1 -- 172.21.15.3:0/3549125066 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7dc012ca0 con 0x5590dfea40b0 2023-09-10T01:05:50.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.260+0000 7fb7ceffd700 1 -- 172.21.15.3:0/3549125066 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc003880 con 0x5590dfea40b0 2023-09-10T01:05:50.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 -- 172.21.15.3:0/3549125066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 msgr2=0x5590dfea8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfea8350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009340 tx=0x7fb7dc012fd0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 -- 172.21.15.3:0/3549125066 shutdown_connections 2023-09-10T01:05:50.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfea3790 0x5590dfea7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfeadec0 0x5590dfeaab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 --2- 172.21.15.3:0/3549125066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dfea8350 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 -- 172.21.15.3:0/3549125066 >> 172.21.15.3:0/3549125066 conn(0x5590dfe18110 msgr2=0x5590dfea5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 -- 172.21.15.3:0/3549125066 shutdown_connections 2023-09-10T01:05:50.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.261+0000 7fb801d19b80 1 -- 172.21.15.3:0/3549125066 wait complete. 2023-09-10T01:05:50.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.262+0000 7fb801d19b80 1 -- start start 2023-09-10T01:05:50.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.262+0000 7fb801d19b80 1 Processor -- start 2023-09-10T01:05:50.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.262+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dff0ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.262+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dff0b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.262+0000 7fb801d19b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.262+0000 7fb801d19b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfeb4030 con 0x5590dfea3790 2023-09-10T01:05:50.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb801d19b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfeb41a0 con 0x5590dfea40b0 2023-09-10T01:05:50.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb801d19b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590dfeb4360 con 0x5590dfeadec0 2023-09-10T01:05:50.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb801d19b80 10 client.0 _ll_register_callbacks cb 0x7fb7dc013350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:50.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dff0b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dff0b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53316/0 (socket says 172.21.15.3:53316) 2023-09-10T01:05:50.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 -- 172.21.15.3:0/1537944716 learned_addr learned my addr 172.21.15.3:0/1537944716 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dff0ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 -- 172.21.15.3:0/1537944716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 msgr2=0x5590dfeb3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 -- 172.21.15.3:0/1537944716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 msgr2=0x5590dff0ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dff0ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 -- 172.21.15.3:0/1537944716 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e810de10 con 0x5590dfea40b0 2023-09-10T01:05:50.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7ec99c700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:50.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7e77fe700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dff0b220 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc013900 tx=0x7fb7dc009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949cd33f3570f2d7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.263+0000 7fb7c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:50.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb7c77fe700 1 -- 172.21.15.3:0/1537944716 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0129f0 con 0x5590dfea40b0 2023-09-10T01:05:50.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590dfdee9a0 con 0x5590dfea40b0 2023-09-10T01:05:50.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb7c77fe700 1 -- 172.21.15.3:0/1537944716 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fb7dc012b50 con 0x5590dfea40b0 2023-09-10T01:05:50.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb7c77fe700 1 -- 172.21.15.3:0/1537944716 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7dc003680 con 0x5590dfea40b0 2023-09-10T01:05:50.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 10 client.26480 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:50.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5590dfc21e60 con 0x5590dfea40b0 2023-09-10T01:05:50.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb7c77fe700 1 -- 172.21.15.3:0/1537944716 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7dc041020 con 0x5590dfea40b0 2023-09-10T01:05:50.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb7c77fe700 1 -- 172.21.15.3:0/1537944716 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fb7dc0387e0 con 0x5590dfea40b0 2023-09-10T01:05:50.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 10 client.26480 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:50.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 10 client.26480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:50.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 20 client.26480 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:50.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dff139d0 0x5590dff15df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 4 client.26480 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:05:50.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.264+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5590dfdaf130 con 0x5590dff139d0 2023-09-10T01:05:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.265+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dff139d0 0x5590dff15df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.265+0000 7fb7e7fff700 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dff139d0 0x5590dff15df0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a2107b834bc658 server_cookie=f87c558df3bc3611 in_seq=0 out_seq=0 2023-09-10T01:05:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.265+0000 7fb7c77fe700 10 client.26480 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.266+0000 7fb7c77fe700 1 -- 172.21.15.3:0/1537944716 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5590dfdaf130 con 0x5590dff139d0 2023-09-10T01:05:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.266+0000 7fb7c77fe700 10 client.26480 handle_command_reply: tid=0 2023-09-10T01:05:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.266+0000 7fb801d19b80 1 client.26480 shutdown 2023-09-10T01:05:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.266+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 msgr2=0x5590dff0b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.266+0000 7fb801d19b80 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dff0b220 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc013900 tx=0x7fb7dc009750 comp rx=0 tx=0).stop 2023-09-10T01:05:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 shutdown_connections 2023-09-10T01:05:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5590dff139d0 0x5590dff15df0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5590dfeadec0 0x5590dfeb3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590dfea3790 0x5590dff0ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 --2- 172.21.15.3:0/1537944716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590dfea40b0 0x5590dff0b220 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 >> 172.21.15.3:0/1537944716 conn(0x5590dfe18110 msgr2=0x5590dfea2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb7c77fe700 0 client.26480 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 shutdown_connections 2023-09-10T01:05:50.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 1 -- 172.21.15.3:0/1537944716 wait complete. 2023-09-10T01:05:50.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.267+0000 7fb801d19b80 20 client.26480 trim_cache size 0 max 16384 2023-09-10T01:05:50.283 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:05:50.283 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:05:50.283 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "85d4b707-7c1d-4cf2-b657-083f67a5e7ea", 2023-09-10T01:05:50.283 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:05:50.284 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:05:50.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb7d00a0830 msgr2=0x7fb7d00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb7d00a0830 0x7fb7d00a2cf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800b2e0 tx=0x7fb7d8005fd0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 msgr2=0x7fb7e8199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8199ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000ef30 tx=0x7fb7e000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 shutdown_connections 2023-09-10T01:05:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb7d00a0830 0x7fb7d00a2cf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7e8100640 0x7fb7e8196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e8106650 0x7fb7e8199ff0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 --2- 172.21.15.3:0/33290565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7e8101000 0x7fb7e8196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 >> 172.21.15.3:0/33290565 conn(0x7fb7e80fc170 msgr2=0x7fb7e80fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 shutdown_connections 2023-09-10T01:05:50.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.268+0000 7fb7ee3ff700 1 -- 172.21.15.3:0/33290565 wait complete. 2023-09-10T01:05:50.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:05:50.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:50 smithi003 ceph-mon[97654]: pgmap v81: 129 pgs: 129 active+clean; 54 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 366 op/s 2023-09-10T01:05:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:50 smithi136 ceph-mon[133708]: pgmap v81: 129 pgs: 129 active+clean; 54 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 366 op/s 2023-09-10T01:05:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:50 smithi092 ceph-mon[134067]: pgmap v81: 129 pgs: 129 active+clean; 54 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 366 op/s 2023-09-10T01:05:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 -- 172.21.15.3:0/3560385608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48106650 msgr2=0x7f3c48106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/3560385608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48106650 0x7f3c48106a30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009990 tx=0x7f3c44009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 -- 172.21.15.3:0/3560385608 shutdown_connections 2023-09-10T01:05:50.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/3560385608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c48101000 0x7f3c4810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/3560385608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48106650 0x7f3c48106a30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/3560385608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c48100640 0x7f3c48100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 -- 172.21.15.3:0/3560385608 >> 172.21.15.3:0/3560385608 conn(0x7f3c480fc170 msgr2=0x7f3c480fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 -- 172.21.15.3:0/3560385608 shutdown_connections 2023-09-10T01:05:50.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.506+0000 7f3c4fdff700 1 -- 172.21.15.3:0/3560385608 wait complete. 2023-09-10T01:05:50.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.507+0000 7f3c4fdff700 1 Processor -- start 2023-09-10T01:05:50.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.507+0000 7f3c4fdff700 1 -- start start 2023-09-10T01:05:50.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4fdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48100640 0x7f3c481966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4fdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c48101000 0x7f3c48196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4fdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c48106650 0x7f3c48199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4fdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4819a5e0 con 0x7f3c48101000 2023-09-10T01:05:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4fdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4819a750 con 0x7f3c48100640 2023-09-10T01:05:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4fdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4819a910 con 0x7f3c48106650 2023-09-10T01:05:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48100640 0x7f3c481966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48100640 0x7f3c481966f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53334/0 (socket says 172.21.15.3:53334) 2023-09-10T01:05:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 -- 172.21.15.3:0/2967208471 learned_addr learned my addr 172.21.15.3:0/2967208471 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c48101000 0x7f3c48196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4e39c700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c48106650 0x7f3c48199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 -- 172.21.15.3:0/2967208471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c48106650 msgr2=0x7f3c48199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c48106650 0x7f3c48199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 -- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c48101000 msgr2=0x7f3c48196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c48101000 0x7f3c48196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.508+0000 7f3c4db9b700 1 -- 172.21.15.3:0/2967208471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440095f0 con 0x7f3c48100640 2023-09-10T01:05:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.509+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48100640 0x7f3c481966f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3c44005910 tx=0x7f3c44004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.509+0000 7f3c3effd700 1 -- 172.21.15.3:0/2967208471 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4401d070 con 0x7f3c48100640 2023-09-10T01:05:50.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.509+0000 7f3c3effd700 1 -- 172.21.15.3:0/2967208471 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c44022470 con 0x7f3c48100640 2023-09-10T01:05:50.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.509+0000 7f3c3effd700 1 -- 172.21.15.3:0/2967208471 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44017650 con 0x7f3c48100640 2023-09-10T01:05:50.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.509+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4819ab90 con 0x7f3c48100640 2023-09-10T01:05:50.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.509+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4819ae40 con 0x7f3c48100640 2023-09-10T01:05:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.512+0000 7f3c3effd700 1 -- 172.21.15.3:0/2967208471 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3c440225e0 con 0x7f3c48100640 2023-09-10T01:05:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.514+0000 7f3c3effd700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c340a07b0 0x7f3c340a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.514+0000 7f3c3effd700 1 -- 172.21.15.3:0/2967208471 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3c440cd5e0 con 0x7f3c48100640 2023-09-10T01:05:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.514+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c340a07b0 0x7f3c340a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.514+0000 7f3c637afb80 1 Processor -- start 2023-09-10T01:05:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.514+0000 7f3c637afb80 1 -- start start 2023-09-10T01:05:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e8b10 0x55a0140e8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e9430 0x55a0140f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c637afb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a01403e730 con 0x55a0140f4260 2023-09-10T01:05:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c637afb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a01403c050 con 0x55a0140e9430 2023-09-10T01:05:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c637afb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a014037940 con 0x55a0140e8b10 2023-09-10T01:05:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c340a07b0 0x7f3c340a2c70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3c48197d00 tx=0x7f3c38009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4db9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e8b10 0x55a0140e8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4d39a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e9430 0x55a0140f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4e39c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4d39a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e9430 0x55a0140f3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53350/0 (socket says 172.21.15.3:53350) 2023-09-10T01:05:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4db9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e8b10 0x55a0140e8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40570/0 (socket says 172.21.15.3:40570) 2023-09-10T01:05:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4d39a700 1 -- 172.21.15.3:0/1024129982 learned_addr learned my addr 172.21.15.3:0/1024129982 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.515+0000 7f3c4e39c700 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58142/0 (socket says 172.21.15.3:58142) 2023-09-10T01:05:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c4db9b700 1 -- 172.21.15.3:0/1024129982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e9430 msgr2=0x55a0140f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e9430 0x55a0140f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c4db9b700 1 -- 172.21.15.3:0/1024129982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 msgr2=0x55a0140f6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c4db9b700 1 -- 172.21.15.3:0/1024129982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a013e63de0 con 0x55a0140e8b10 2023-09-10T01:05:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e8b10 0x55a0140e8ef0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c440ca940 tx=0x7f3c4400b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=465dbae7cdbdcc31 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c33fff700 1 -- 172.21.15.3:0/1024129982 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440cd050 con 0x55a0140e8b10 2023-09-10T01:05:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c33fff700 1 -- 172.21.15.3:0/1024129982 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c4409a090 con 0x55a0140e8b10 2023-09-10T01:05:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.516+0000 7f3c33fff700 1 -- 172.21.15.3:0/1024129982 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440cd1b0 con 0x55a0140e8b10 2023-09-10T01:05:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 -- 172.21.15.3:0/1024129982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e8b10 msgr2=0x55a0140e8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e8b10 0x55a0140e8ef0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c440ca940 tx=0x7f3c4400b6b0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 -- 172.21.15.3:0/1024129982 shutdown_connections 2023-09-10T01:05:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e8b10 0x55a0140e8ef0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1024129982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e9430 0x55a0140f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 -- 172.21.15.3:0/1024129982 >> 172.21.15.3:0/1024129982 conn(0x55a01405a090 msgr2=0x55a0140e0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 -- 172.21.15.3:0/1024129982 shutdown_connections 2023-09-10T01:05:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 -- 172.21.15.3:0/1024129982 wait complete. 2023-09-10T01:05:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.517+0000 7f3c637afb80 1 -- start start 2023-09-10T01:05:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 1 Processor -- start 2023-09-10T01:05:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e8b10 0x55a0140f4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a013fd5b70 con 0x55a0140f4260 2023-09-10T01:05:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a013fd5870 con 0x55a0140e8b10 2023-09-10T01:05:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a013eb9920 con 0x55a0140e9430 2023-09-10T01:05:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.518+0000 7f3c637afb80 10 client.0 _ll_register_callbacks cb 0x55a013f1fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e8b10 0x55a0140f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e8b10 0x55a0140f4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53352/0 (socket says 172.21.15.3:53352) 2023-09-10T01:05:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 -- 172.21.15.3:0/3189167090 learned_addr learned my addr 172.21.15.3:0/3189167090 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4e39c700 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 -- 172.21.15.3:0/3189167090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 msgr2=0x55a0140f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140f4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 -- 172.21.15.3:0/3189167090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 msgr2=0x55a0140f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 -- 172.21.15.3:0/3189167090 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4819ae40 con 0x55a0140e8b10 2023-09-10T01:05:50.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e8b10 0x55a0140f4640 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c44096c30 tx=0x7f3c44022b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c54622f36a2bce15 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c317fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:50.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c317fa700 1 -- 172.21.15.3:0/3189167090 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4409e560 con 0x55a0140e8b10 2023-09-10T01:05:50.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c317fa700 1 -- 172.21.15.3:0/3189167090 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3c440a2710 con 0x55a0140e8b10 2023-09-10T01:05:50.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.519+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a013f8f4f0 con 0x55a0140e8b10 2023-09-10T01:05:50.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c317fa700 1 -- 172.21.15.3:0/3189167090 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c4409ae30 con 0x55a0140e8b10 2023-09-10T01:05:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 10 client.26492 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a013e4cc60 con 0x55a0140e8b10 2023-09-10T01:05:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c317fa700 1 -- 172.21.15.3:0/3189167090 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a013f8f4f0 con 0x55a0140e8b10 2023-09-10T01:05:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c317fa700 1 -- 172.21.15.3:0/3189167090 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3c440a0310 con 0x55a0140e8b10 2023-09-10T01:05:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 10 client.26492 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 10 client.26492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 20 client.26492 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a0141035a0 0x55a0141059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 4 client.26492 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.520+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a01404c8a0 con 0x55a0141035a0 2023-09-10T01:05:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.521+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a0141035a0 0x55a0141059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.521+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a0141035a0 0x55a0141059c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a5e99c088075670 server_cookie=8cf2239b2638f7ec in_seq=0 out_seq=0 2023-09-10T01:05:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.521+0000 7f3c317fa700 10 client.26492 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.522+0000 7f3c317fa700 1 -- 172.21.15.3:0/3189167090 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a01404c8a0 con 0x55a0141035a0 2023-09-10T01:05:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.522+0000 7f3c317fa700 10 client.26492 handle_command_reply: tid=0 2023-09-10T01:05:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.522+0000 7f3c637afb80 1 client.26492 shutdown 2023-09-10T01:05:50.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.522+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e8b10 msgr2=0x55a0140f4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.522+0000 7f3c637afb80 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e8b10 0x55a0140f4640 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c44096c30 tx=0x7f3c44022b30 comp rx=0 tx=0).stop 2023-09-10T01:05:50.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 shutdown_connections 2023-09-10T01:05:50.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a0141035a0 0x55a0141059c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140f4260 0x55a0140f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140e8b10 0x55a0140f4640 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 --2- 172.21.15.3:0/3189167090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 >> 172.21.15.3:0/3189167090 conn(0x55a01405a090 msgr2=0x55a0140e06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c317fa700 0 client.26492 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 shutdown_connections 2023-09-10T01:05:50.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 1 -- 172.21.15.3:0/3189167090 wait complete. 2023-09-10T01:05:50.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.523+0000 7f3c637afb80 20 client.26492 trim_cache size 0 max 16384 2023-09-10T01:05:50.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.541+0000 7f3c637afb80 1 Processor -- start 2023-09-10T01:05:50.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.541+0000 7f3c637afb80 1 -- start start 2023-09-10T01:05:50.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a0140f8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140fedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a0140ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c637afb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0140ee200 con 0x55a0140e8b10 2023-09-10T01:05:50.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c637afb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0140ed9b0 con 0x55a0140f4260 2023-09-10T01:05:50.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c637afb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0140f3940 con 0x55a0140e9430 2023-09-10T01:05:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a0140ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4db9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a0140f8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4d39a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140fedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a0140ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53368/0 (socket says 172.21.15.3:53368) 2023-09-10T01:05:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 -- 172.21.15.3:0/513606583 learned_addr learned my addr 172.21.15.3:0/513606583 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 -- 172.21.15.3:0/513606583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 msgr2=0x55a0140fedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 --2- 172.21.15.3:0/513606583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140fedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 -- 172.21.15.3:0/513606583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 msgr2=0x55a0140f8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 --2- 172.21.15.3:0/513606583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a0140f8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.542+0000 7f3c4e39c700 1 -- 172.21.15.3:0/513606583 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a013e8a250 con 0x55a0140f4260 2023-09-10T01:05:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.543+0000 7f3c4e39c700 1 --2- 172.21.15.3:0/513606583 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a0140ee6c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009d10 tx=0x7f3c4000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=347fcc2112669db0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.543+0000 7f3c3cbff700 1 -- 172.21.15.3:0/513606583 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4000e030 con 0x55a0140f4260 2023-09-10T01:05:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.543+0000 7f3c3cbff700 1 -- 172.21.15.3:0/513606583 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c4000f040 con 0x55a0140f4260 2023-09-10T01:05:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.543+0000 7f3c3cbff700 1 -- 172.21.15.3:0/513606583 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40016a50 con 0x55a0140f4260 2023-09-10T01:05:50.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 -- 172.21.15.3:0/513606583 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 msgr2=0x55a0140ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 --2- 172.21.15.3:0/513606583 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a0140ee6c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009d10 tx=0x7f3c4000b040 comp rx=0 tx=0).stop 2023-09-10T01:05:50.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 -- 172.21.15.3:0/513606583 shutdown_connections 2023-09-10T01:05:50.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 --2- 172.21.15.3:0/513606583 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a0140ee6c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 --2- 172.21.15.3:0/513606583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a0140f8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 --2- 172.21.15.3:0/513606583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a0140fedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 -- 172.21.15.3:0/513606583 >> 172.21.15.3:0/513606583 conn(0x55a01405a090 msgr2=0x55a0140df320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 -- 172.21.15.3:0/513606583 shutdown_connections 2023-09-10T01:05:50.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 -- 172.21.15.3:0/513606583 wait complete. 2023-09-10T01:05:50.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 -- start start 2023-09-10T01:05:50.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.544+0000 7f3c637afb80 1 Processor -- start 2023-09-10T01:05:50.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a01414b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a01414bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c637afb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a01414f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c637afb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a01414fc10 con 0x55a0140e8b10 2023-09-10T01:05:50.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c637afb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a01414fd80 con 0x55a0140f4260 2023-09-10T01:05:50.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c637afb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a01414ff40 con 0x55a0140e9430 2023-09-10T01:05:50.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c637afb80 10 client.0 _ll_register_callbacks cb 0x7f3c40009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:50.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c4d39a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a01414bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c4e39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a01414f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c4d39a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a01414bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40604/0 (socket says 172.21.15.3:40604) 2023-09-10T01:05:50.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c4d39a700 1 -- 172.21.15.3:0/1731593140 learned_addr learned my addr 172.21.15.3:0/1731593140 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:50.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.545+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a01414b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c4d39a700 1 -- 172.21.15.3:0/1731593140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 msgr2=0x55a01414f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a01414f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c4d39a700 1 -- 172.21.15.3:0/1731593140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 msgr2=0x55a01414b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:50.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a01414b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c4d39a700 1 -- 172.21.15.3:0/1731593140 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c400092a0 con 0x55a0140e9430 2023-09-10T01:05:50.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c4d39a700 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a01414bab0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3c380094f0 tx=0x7f3c380047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda8e540402515e1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:50.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c317fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:05:50.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c317fa700 1 -- 172.21.15.3:0/1731593140 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c38023040 con 0x55a0140e9430 2023-09-10T01:05:50.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c4409a090 con 0x55a0140e9430 2023-09-10T01:05:50.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c317fa700 1 -- 172.21.15.3:0/1731593140 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3c38028900 con 0x55a0140e9430 2023-09-10T01:05:50.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c317fa700 1 -- 172.21.15.3:0/1731593140 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c38028b00 con 0x55a0140e9430 2023-09-10T01:05:50.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c637afb80 10 client.26413 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:50.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.546+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a013e622b0 con 0x55a0140e9430 2023-09-10T01:05:50.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c317fa700 1 -- 172.21.15.3:0/1731593140 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c4409a090 con 0x55a0140e9430 2023-09-10T01:05:50.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c317fa700 1 -- 172.21.15.3:0/1731593140 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3c3802e030 con 0x55a0140e9430 2023-09-10T01:05:50.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c637afb80 10 client.26413 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:50.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c637afb80 10 client.26413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:50.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c637afb80 20 client.26413 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:50.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a014158a40 0x55a01415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:50.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c637afb80 4 client.26413 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:05:50.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a014030920 con 0x55a014158a40 2023-09-10T01:05:50.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a014158a40 0x55a01415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:50.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c4db9b700 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a014158a40 0x55a01415ae60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49309a62870e59a5 server_cookie=79fc0a91bc458993 in_seq=0 out_seq=0 2023-09-10T01:05:50.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.547+0000 7f3c317fa700 10 client.26413 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c317fa700 1 -- 172.21.15.3:0/1731593140 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a014030920 con 0x55a014158a40 2023-09-10T01:05:50.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c317fa700 10 client.26413 handle_command_reply: tid=0 2023-09-10T01:05:50.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 client.26413 shutdown 2023-09-10T01:05:50.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 msgr2=0x55a01414bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a01414bab0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3c380094f0 tx=0x7f3c380047c0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 shutdown_connections 2023-09-10T01:05:50.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a014158a40 0x55a01415ae60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0140f4260 0x55a01414f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a0140e8b10 0x55a01414b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c317fa700 0 client.26413 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:50.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 --2- 172.21.15.3:0/1731593140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a0140e9430 0x55a01414bab0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 >> 172.21.15.3:0/1731593140 conn(0x55a01405a090 msgr2=0x55a0140ee6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 shutdown_connections 2023-09-10T01:05:50.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.548+0000 7f3c637afb80 1 -- 172.21.15.3:0/1731593140 wait complete. 2023-09-10T01:05:50.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c637afb80 20 client.26413 trim_cache size 0 max 16384 2023-09-10T01:05:50.567 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:05:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:05:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:05:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "85d4b707-7c1d-4cf2-b657-083f67a5e7ea": { 2023-09-10T01:05:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:05:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "85d4b707-7c1d-4cf2-b657-083f67a5e7ea", 2023-09-10T01:05:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:05:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:05:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:05:50.569 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:05:50.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c340a07b0 msgr2=0x7f3c340a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c340a07b0 0x7f3c340a2c70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3c48197d00 tx=0x7f3c38009260 comp rx=0 tx=0).stop 2023-09-10T01:05:50.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48100640 msgr2=0x7f3c481966f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:50.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48100640 0x7f3c481966f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3c44005910 tx=0x7f3c44004710 comp rx=0 tx=0).stop 2023-09-10T01:05:50.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 shutdown_connections 2023-09-10T01:05:50.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c340a07b0 0x7f3c340a2c70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c48106650 0x7f3c48199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c48100640 0x7f3c481966f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 --2- 172.21.15.3:0/2967208471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c48101000 0x7f3c48196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:50.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 >> 172.21.15.3:0/2967208471 conn(0x7f3c480fc170 msgr2=0x7f3c480fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:50.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 shutdown_connections 2023-09-10T01:05:50.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:05:50.549+0000 7f3c4fdff700 1 -- 172.21.15.3:0/2967208471 wait complete. 2023-09-10T01:05:50.572 INFO:tasks.cephfs.filesystem:scrub status for tag:85d4b707-7c1d-4cf2-b657-083f67a5e7ea - {'path': '/', 'tag': '85d4b707-7c1d-4cf2-b657-083f67a5e7ea', 'options': 'recursive,force'} 2023-09-10T01:05:50.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 170392 10.19 MB/sec warmup 92 sec latency 15.503 ms 2023-09-10T01:05:51.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:05:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:51.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:51.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:05:51.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:05:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:05:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:05:51.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:51.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:05:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:05:51.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 171408 10.12 MB/sec warmup 93 sec latency 29.984 ms 2023-09-10T01:05:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:52 smithi092 ceph-mon[134067]: pgmap v82: 129 pgs: 129 active+clean; 55 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 312 op/s 2023-09-10T01:05:52.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:52 smithi003 ceph-mon[97654]: pgmap v82: 129 pgs: 129 active+clean; 55 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 312 op/s 2023-09-10T01:05:52.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:52 smithi136 ceph-mon[133708]: pgmap v82: 129 pgs: 129 active+clean; 55 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 312 op/s 2023-09-10T01:05:52.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 173952 10.18 MB/sec warmup 94 sec latency 21.962 ms 2023-09-10T01:05:53.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 174881 10.10 MB/sec warmup 95 sec latency 170.547 ms 2023-09-10T01:05:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:54 smithi092 ceph-mon[134067]: pgmap v83: 129 pgs: 129 active+clean; 55 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:05:54.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:54 smithi003 ceph-mon[97654]: pgmap v83: 129 pgs: 129 active+clean; 55 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:05:54.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:54 smithi136 ceph-mon[133708]: pgmap v83: 129 pgs: 129 active+clean; 55 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:05:54.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 177427 10.16 MB/sec warmup 96 sec latency 32.312 ms 2023-09-10T01:05:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 -- 172.21.15.3:0/1076469177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4106680 msgr2=0x7f5dd4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/1076469177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4106680 0x7f5dd4106a60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009990 tx=0x7f5dd0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 -- 172.21.15.3:0/1076469177 shutdown_connections 2023-09-10T01:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/1076469177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4106680 0x7f5dd4106a60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/1076469177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd4100f80 0x7f5dd410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/1076469177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dd41005c0 0x7f5dd4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 -- 172.21.15.3:0/1076469177 >> 172.21.15.3:0/1076469177 conn(0x7f5dd40fc0b0 msgr2=0x7f5dd40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 -- 172.21.15.3:0/1076469177 shutdown_connections 2023-09-10T01:05:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.750+0000 7f5ddbdff700 1 -- 172.21.15.3:0/1076469177 wait complete. 2023-09-10T01:05:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.751+0000 7f5ddbdff700 1 Processor -- start 2023-09-10T01:05:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.752+0000 7f5ddbdff700 1 -- start start 2023-09-10T01:05:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.752+0000 7f5ddbdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd41005c0 0x7f5dd4072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.752+0000 7f5ddbdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4100f80 0x7f5dd4072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.752+0000 7f5ddbdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dd4106680 0x7f5dd41a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.752+0000 7f5ddbdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd4073670 con 0x7f5dd41005c0 2023-09-10T01:05:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.752+0000 7f5ddbdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd41a4d50 con 0x7f5dd4106680 2023-09-10T01:05:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.752+0000 7f5ddbdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd41a4e90 con 0x7f5dd4100f80 2023-09-10T01:05:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd41005c0 0x7f5dd4072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4100f80 0x7f5dd4072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd41005c0 0x7f5dd4072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52262/0 (socket says 172.21.15.3:52262) 2023-09-10T01:05:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4100f80 0x7f5dd4072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38730/0 (socket says 172.21.15.3:38730) 2023-09-10T01:05:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/901612908 learned_addr learned my addr 172.21.15.3:0/901612908 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dda39c700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dd4106680 0x7f5dd41a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 -- 172.21.15.3:0/901612908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dd4106680 msgr2=0x7f5dd41a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dd4106680 0x7f5dd41a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 -- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd41005c0 msgr2=0x7f5dd4072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd41005c0 0x7f5dd4072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 -- 172.21.15.3:0/901612908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc009c40 con 0x7f5dd4100f80 2023-09-10T01:05:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd41005c0 0x7f5dd4072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:54.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dd939a700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4100f80 0x7f5dd4072e40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400e9f0 tx=0x7f5dc400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dcaffd700 1 -- 172.21.15.3:0/901612908 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40096b0 con 0x7f5dd4100f80 2023-09-10T01:05:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd00095f0 con 0x7f5dd4100f80 2023-09-10T01:05:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dcaffd700 1 -- 172.21.15.3:0/901612908 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5dc4004cf0 con 0x7f5dd4100f80 2023-09-10T01:05:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.753+0000 7f5dcaffd700 1 -- 172.21.15.3:0/901612908 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc400cdb0 con 0x7f5dd4100f80 2023-09-10T01:05:54.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.754+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd41a5440 con 0x7f5dd4100f80 2023-09-10T01:05:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.758+0000 7f5dcaffd700 1 -- 172.21.15.3:0/901612908 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f5dc4004320 con 0x7f5dd4100f80 2023-09-10T01:05:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 Processor -- start 2023-09-10T01:05:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 -- start start 2023-09-10T01:05:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5dcaffd700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5dc00a0830 0x7f5dc00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c76580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c70780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5dcaffd700 1 -- 172.21.15.3:0/901612908 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f5dc4014070 con 0x7f5dd4100f80 2023-09-10T01:05:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4bc96d0 con 0x55acb4c70cc0 2023-09-10T01:05:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4bc6ff0 con 0x55acb4c761a0 2023-09-10T01:05:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.760+0000 7f5def81fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4bc28e0 con 0x55acb4c70300 2023-09-10T01:05:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c70780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c76580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c70780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38746/0 (socket says 172.21.15.3:38746) 2023-09-10T01:05:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 -- 172.21.15.3:0/1830083838 learned_addr learned my addr 172.21.15.3:0/1830083838 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dda39c700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5dc00a0830 0x7f5dc00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dda39c700 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 -- 172.21.15.3:0/1830083838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 msgr2=0x55acb4c76580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c76580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 -- 172.21.15.3:0/1830083838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 msgr2=0x55acb4c7cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 -- 172.21.15.3:0/1830083838 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acb49eed80 con 0x55acb4c70300 2023-09-10T01:05:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c76580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dd939a700 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c70780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4095a10 tx=0x7f5dc4098ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d885904aeff09d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dbffff700 1 -- 172.21.15.3:0/1830083838 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc409d120 con 0x55acb4c70300 2023-09-10T01:05:54.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dda39c700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5dc00a0830 0x7f5dc00a2cf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4074a10 tx=0x7f5dcc009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:54.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dbffff700 1 -- 172.21.15.3:0/1830083838 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5dc4099510 con 0x55acb4c70300 2023-09-10T01:05:54.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.761+0000 7f5dbffff700 1 -- 172.21.15.3:0/1830083838 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc401e300 con 0x55acb4c70300 2023-09-10T01:05:54.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 -- 172.21.15.3:0/1830083838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 msgr2=0x55acb4c70780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c70780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4095a10 tx=0x7f5dc4098ef0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 -- 172.21.15.3:0/1830083838 shutdown_connections 2023-09-10T01:05:54.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c76580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 --2- 172.21.15.3:0/1830083838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c70780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 -- 172.21.15.3:0/1830083838 >> 172.21.15.3:0/1830083838 conn(0x55acb4be5030 msgr2=0x55acb4c6b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:54.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 -- 172.21.15.3:0/1830083838 shutdown_connections 2023-09-10T01:05:54.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.762+0000 7f5def81fb80 1 -- 172.21.15.3:0/1830083838 wait complete. 2023-09-10T01:05:54.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.763+0000 7f5def81fb80 1 -- start start 2023-09-10T01:05:54.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.763+0000 7f5def81fb80 1 Processor -- start 2023-09-10T01:05:54.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.763+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.763+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.763+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c761a0 0x55acb4c7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.763+0000 7f5def81fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4b60b10 con 0x55acb4c70cc0 2023-09-10T01:05:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.763+0000 7f5def81fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4b60810 con 0x55acb4c70300 2023-09-10T01:05:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5def81fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4a448c0 con 0x55acb4c761a0 2023-09-10T01:05:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5def81fb80 10 client.0 _ll_register_callbacks cb 0x55acb4aaaa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:54.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:46876/0 (socket says 172.21.15.3:46876) 2023-09-10T01:05:54.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3022188992 learned_addr learned my addr 172.21.15.3:0/3022188992 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:54.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dda39c700 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c761a0 0x55acb4c7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd939a700 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3022188992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c761a0 msgr2=0x55acb4c7a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c761a0 0x55acb4c7a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3022188992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 msgr2=0x55acb4c7a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3022188992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd410dd40 con 0x55acb4c70300 2023-09-10T01:05:54.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7ceb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00052a0 tx=0x7f5dd0005380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5805eb89f93f14c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:54.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.764+0000 7f5dbd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:54.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3022188992 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd001d070 con 0x55acb4c70300 2023-09-10T01:05:54.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acb4b1a490 con 0x55acb4c70300 2023-09-10T01:05:54.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3022188992 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f5dd0004910 con 0x55acb4c70300 2023-09-10T01:05:54.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3022188992 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dd0004ad0 con 0x55acb4c70300 2023-09-10T01:05:54.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 10 client.26510 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:54.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f5dd41a5440 con 0x55acb4c70300 2023-09-10T01:05:54.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3022188992 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55acb4b1a490 con 0x55acb4c70300 2023-09-10T01:05:54.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3022188992 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5dd0024450 con 0x55acb4c70300 2023-09-10T01:05:54.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 10 client.26510 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:54.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 10 client.26510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:54.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 20 client.26510 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:54.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4c8a590 0x55acb4c8c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 4 client.26510 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:05:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.765+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acb4bd7840 con 0x55acb4c8a590 2023-09-10T01:05:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.766+0000 7f5dd939a700 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4c8a590 0x55acb4c8c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.766+0000 7f5dd939a700 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4c8a590 0x55acb4c8c9b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9baf438c303cd75a server_cookie=9a1450dc6f429ed4 in_seq=0 out_seq=0 2023-09-10T01:05:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.766+0000 7f5dbd7fa700 10 client.26510 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3022188992 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55acb4bd7840 con 0x55acb4c8a590 2023-09-10T01:05:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5dbd7fa700 10 client.26510 handle_command_reply: tid=0 2023-09-10T01:05:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 client.26510 shutdown 2023-09-10T01:05:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 msgr2=0x55acb4c7ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7ceb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00052a0 tx=0x7f5dd0005380 comp rx=0 tx=0).stop 2023-09-10T01:05:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 shutdown_connections 2023-09-10T01:05:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4c8a590 0x55acb4c8c9b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c761a0 0x55acb4c7a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7ceb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5dbd7fa700 0 client.26510 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3022188992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 >> 172.21.15.3:0/3022188992 conn(0x55acb4be5030 msgr2=0x55acb4c6a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 shutdown_connections 2023-09-10T01:05:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 1 -- 172.21.15.3:0/3022188992 wait complete. 2023-09-10T01:05:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.767+0000 7f5def81fb80 20 client.26510 trim_cache size 0 max 16384 2023-09-10T01:05:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.783+0000 7f5def81fb80 1 Processor -- start 2023-09-10T01:05:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5def81fb80 1 -- start start 2023-09-10T01:05:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c6af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c741f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5def81fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4c71430 con 0x55acb4c70cc0 2023-09-10T01:05:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5def81fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4c67fc0 con 0x55acb4c761a0 2023-09-10T01:05:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5def81fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4c7b710 con 0x55acb4c70300 2023-09-10T01:05:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c6af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5dda39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c741f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5dd939a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c6af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38756/0 (socket says 172.21.15.3:38756) 2023-09-10T01:05:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5dda39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c741f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:46884/0 (socket says 172.21.15.3:46884) 2023-09-10T01:05:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.784+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/979184236 learned_addr learned my addr 172.21.15.3:0/979184236 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/979184236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 msgr2=0x55acb4c741f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c741f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/979184236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 msgr2=0x55acb4c7ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/979184236 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acb49d7c00 con 0x55acb4c70300 2023-09-10T01:05:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dda39c700 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c741f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:05:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c6af20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009350 tx=0x7f5dd0009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7c33f9c080000c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dc8bff700 1 -- 172.21.15.3:0/979184236 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd001d040 con 0x55acb4c70300 2023-09-10T01:05:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dc8bff700 1 -- 172.21.15.3:0/979184236 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5dd001e040 con 0x55acb4c70300 2023-09-10T01:05:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.785+0000 7f5dc8bff700 1 -- 172.21.15.3:0/979184236 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd000f6c0 con 0x55acb4c70300 2023-09-10T01:05:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 -- 172.21.15.3:0/979184236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 msgr2=0x55acb4c6af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c6af20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009350 tx=0x7f5dd0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 -- 172.21.15.3:0/979184236 shutdown_connections 2023-09-10T01:05:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70300 0x55acb4c6af20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c761a0 0x55acb4c741f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 --2- 172.21.15.3:0/979184236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c70cc0 0x55acb4c7ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 -- 172.21.15.3:0/979184236 >> 172.21.15.3:0/979184236 conn(0x55acb4be5030 msgr2=0x55acb4c75170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 -- 172.21.15.3:0/979184236 shutdown_connections 2023-09-10T01:05:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.786+0000 7f5def81fb80 1 -- 172.21.15.3:0/979184236 wait complete. 2023-09-10T01:05:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 -- start start 2023-09-10T01:05:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 Processor -- start 2023-09-10T01:05:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70cc0 0x55acb4c7fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c761a0 0x55acb4c7ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4cdd3e0 con 0x55acb4c761a0 2023-09-10T01:05:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4cdd520 con 0x55acb4c70300 2023-09-10T01:05:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acb4cdd660 con 0x55acb4c70cc0 2023-09-10T01:05:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5def81fb80 10 client.0 _ll_register_callbacks cb 0x7f5dd000b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:05:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5dd9b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:46896/0 (socket says 172.21.15.3:46896) 2023-09-10T01:05:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3334189437 learned_addr learned my addr 172.21.15.3:0/3334189437 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:05:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.787+0000 7f5dd939a700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70cc0 0x55acb4c7fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dda39c700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c761a0 0x55acb4c7ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3334189437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70cc0 msgr2=0x55acb4c7fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70cc0 0x55acb4c7fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3334189437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c761a0 msgr2=0x55acb4c7ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:05:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c761a0 0x55acb4c7ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dd9b9b700 1 -- 172.21.15.3:0/3334189437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc0186c0 con 0x55acb4c70300 2023-09-10T01:05:54.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dda39c700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c761a0 0x55acb4c7ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:05:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dd9b9b700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7f510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00247e0 tx=0x7f5dd000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab4822256c918c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:05:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dbd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:05:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3334189437 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd001d070 con 0x55acb4c70300 2023-09-10T01:05:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3334189437 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f5dd0000e40 con 0x55acb4c70300 2023-09-10T01:05:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.788+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5dc4099510 con 0x55acb4c70300 2023-09-10T01:05:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3334189437 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dd0004e70 con 0x55acb4c70300 2023-09-10T01:05:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 10 client.26516 fetch_fsmap learned FSMap version 23 2023-09-10T01:05:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f5dd41a5440 con 0x55acb4c70300 2023-09-10T01:05:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3334189437 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5dc4099510 con 0x55acb4c70300 2023-09-10T01:05:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3334189437 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5dd000fab0 con 0x55acb4c70300 2023-09-10T01:05:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 10 client.26516 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:05:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 10 client.26516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:05:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 20 client.26516 populate_metadata read hostname 'smithi003' 2023-09-10T01:05:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4cde890 0x55acb4ce0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:05:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 4 client.26516 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:05:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55acb4bbb8c0 con 0x55acb4cde890 2023-09-10T01:05:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.789+0000 7f5dd939a700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4cde890 0x55acb4ce0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:05:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.790+0000 7f5dd939a700 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4cde890 0x55acb4ce0c80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97b7a846d63560b1 server_cookie=461b63a2c7e842fb in_seq=0 out_seq=0 2023-09-10T01:05:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.790+0000 7f5dbd7fa700 10 client.26516 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:54.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.885+0000 7f5dbd7fa700 1 -- 172.21.15.3:0/3334189437 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55acb4bbb8c0 con 0x55acb4cde890 2023-09-10T01:05:54.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.885+0000 7f5dbd7fa700 10 client.26516 handle_command_reply: tid=0 2023-09-10T01:05:54.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.885+0000 7f5def81fb80 1 client.26516 shutdown 2023-09-10T01:05:54.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 msgr2=0x55acb4c7f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7f510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00247e0 tx=0x7f5dd000bf10 comp rx=0 tx=0).stop 2023-09-10T01:05:54.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 shutdown_connections 2023-09-10T01:05:54.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acb4cde890 0x55acb4ce0c80 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acb4c761a0 0x55acb4c7ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acb4c70300 0x55acb4c7f510 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5dbd7fa700 0 client.26516 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:05:54.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 --2- 172.21.15.3:0/3334189437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acb4c70cc0 0x55acb4c7fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 >> 172.21.15.3:0/3334189437 conn(0x55acb4be5030 msgr2=0x55acb4c75170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:54.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 shutdown_connections 2023-09-10T01:05:54.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 1 -- 172.21.15.3:0/3334189437 wait complete. 2023-09-10T01:05:54.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.886+0000 7f5def81fb80 20 client.26516 trim_cache size 0 max 16384 2023-09-10T01:05:54.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:05:54.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:05:54.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:05:54.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:05:54.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5dc00a0830 msgr2=0x7f5dc00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5dc00a0830 0x7f5dc00a2cf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4074a10 tx=0x7f5dcc009920 comp rx=0 tx=0).stop 2023-09-10T01:05:54.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4100f80 msgr2=0x7f5dd4072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:05:54.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4100f80 0x7f5dd4072e40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400e9f0 tx=0x7f5dc400ed00 comp rx=0 tx=0).stop 2023-09-10T01:05:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 shutdown_connections 2023-09-10T01:05:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5dc00a0830 0x7f5dc00a2cf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dd4106680 0x7f5dd41a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dd41005c0 0x7f5dd4072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 --2- 172.21.15.3:0/901612908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dd4100f80 0x7f5dd4072e40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:05:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 >> 172.21.15.3:0/901612908 conn(0x7f5dd40fc0b0 msgr2=0x7f5dd40fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:05:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 shutdown_connections 2023-09-10T01:05:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:05:54.887+0000 7f5ddbdff700 1 -- 172.21.15.3:0/901612908 wait complete. 2023-09-10T01:05:55.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 178501 10.09 MB/sec warmup 97 sec latency 80.820 ms 2023-09-10T01:05:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:56 smithi092 ceph-mon[134067]: pgmap v84: 129 pgs: 129 active+clean; 59 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 404 op/s 2023-09-10T01:05:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:56 smithi003 ceph-mon[97654]: pgmap v84: 129 pgs: 129 active+clean; 59 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 404 op/s 2023-09-10T01:05:56.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:56 smithi136 ceph-mon[133708]: pgmap v84: 129 pgs: 129 active+clean; 59 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 404 op/s 2023-09-10T01:05:56.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181028 10.15 MB/sec warmup 98 sec latency 45.333 ms 2023-09-10T01:05:57.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 182119 10.08 MB/sec warmup 99 sec latency 24.817 ms 2023-09-10T01:05:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:58 smithi092 ceph-mon[134067]: pgmap v85: 129 pgs: 129 active+clean; 59 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 238 op/s 2023-09-10T01:05:58.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:58 smithi003 ceph-mon[97654]: pgmap v85: 129 pgs: 129 active+clean; 59 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 238 op/s 2023-09-10T01:05:58.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:58 smithi136 ceph-mon[133708]: pgmap v85: 129 pgs: 129 active+clean; 59 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 238 op/s 2023-09-10T01:05:58.744 INFO:tasks.workunit.client.0.smithi003.stdout: 1 184704 10.15 MB/sec warmup 100 sec latency 15.434 ms 2023-09-10T01:05:59.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:59 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:05:59.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:05:59 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:05:59.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:59 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:05:59.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:05:59 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:05:59.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:59 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:05:59.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:05:59 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:05:59.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 186027 10.10 MB/sec warmup 101 sec latency 11.565 ms 2023-09-10T01:06:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:00 smithi092 ceph-mon[134067]: pgmap v86: 129 pgs: 129 active+clean; 58 MiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2023-09-10T01:06:00.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:00 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:06:00.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:00 smithi003 ceph-mon[97654]: pgmap v86: 129 pgs: 129 active+clean; 58 MiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2023-09-10T01:06:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:00 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:06:00.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:00 smithi136 ceph-mon[133708]: pgmap v86: 129 pgs: 129 active+clean; 58 MiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2023-09-10T01:06:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:00 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:06:00.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 187710 10.09 MB/sec warmup 102 sec latency 63.952 ms 2023-09-10T01:06:01.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 188457 10.04 MB/sec warmup 103 sec latency 60.472 ms 2023-09-10T01:06:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:02 smithi092 ceph-mon[134067]: pgmap v87: 129 pgs: 129 active+clean; 59 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 321 op/s 2023-09-10T01:06:02.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:02 smithi003 ceph-mon[97654]: pgmap v87: 129 pgs: 129 active+clean; 59 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 321 op/s 2023-09-10T01:06:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:02 smithi136 ceph-mon[133708]: pgmap v87: 129 pgs: 129 active+clean; 59 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 321 op/s 2023-09-10T01:06:02.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 190366 10.01 MB/sec warmup 104 sec latency 26.453 ms 2023-09-10T01:06:03.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 192025 10.03 MB/sec warmup 105 sec latency 16.300 ms 2023-09-10T01:06:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:04 smithi092 ceph-mon[134067]: pgmap v88: 129 pgs: 129 active+clean; 59 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2023-09-10T01:06:04.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:04 smithi003 ceph-mon[97654]: pgmap v88: 129 pgs: 129 active+clean; 59 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2023-09-10T01:06:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:04 smithi136 ceph-mon[133708]: pgmap v88: 129 pgs: 129 active+clean; 59 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2023-09-10T01:06:04.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 194325 10.04 MB/sec warmup 106 sec latency 12.964 ms 2023-09-10T01:06:05.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195807 10.03 MB/sec warmup 107 sec latency 36.427 ms 2023-09-10T01:06:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:06 smithi092 ceph-mon[134067]: pgmap v89: 129 pgs: 129 active+clean; 62 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 426 op/s 2023-09-10T01:06:06.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:06 smithi003 ceph-mon[97654]: pgmap v89: 129 pgs: 129 active+clean; 62 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 426 op/s 2023-09-10T01:06:06.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:06 smithi136 ceph-mon[133708]: pgmap v89: 129 pgs: 129 active+clean; 62 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 426 op/s 2023-09-10T01:06:06.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 198268 10.05 MB/sec warmup 108 sec latency 56.419 ms 2023-09-10T01:06:07.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 199446 10.02 MB/sec warmup 109 sec latency 25.060 ms 2023-09-10T01:06:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 -- 172.21.15.3:0/1555531288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed0010ce40 msgr2=0x7fed0010f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 --2- 172.21.15.3:0/1555531288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed0010ce40 0x7fed0010f270 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fecf800b0a0 tx=0x7fecf800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 -- 172.21.15.3:0/1555531288 shutdown_connections 2023-09-10T01:06:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 --2- 172.21.15.3:0/1555531288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed0010ce40 0x7fed0010f270 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 --2- 172.21.15.3:0/1555531288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000729d0 0x7fed00072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 --2- 172.21.15.3:0/1555531288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed000732f0 0x7fed0010c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 -- 172.21.15.3:0/1555531288 >> 172.21.15.3:0/1555531288 conn(0x7fed000fc0b0 msgr2=0x7fed000fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 -- 172.21.15.3:0/1555531288 shutdown_connections 2023-09-10T01:06:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.044+0000 7fed069ff700 1 -- 172.21.15.3:0/1555531288 wait complete. 2023-09-10T01:06:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 Processor -- start 2023-09-10T01:06:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 -- start start 2023-09-10T01:06:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed000729d0 0x7fed0019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000732f0 0x7fed0019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed0010ce40 0x7fed00198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00198ed0 con 0x7fed000732f0 2023-09-10T01:06:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00199040 con 0x7fed000729d0 2023-09-10T01:06:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.045+0000 7fed069ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00199200 con 0x7fed0010ce40 2023-09-10T01:06:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed000729d0 0x7fed0019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fed04f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed0010ce40 0x7fed00198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed000729d0 0x7fed0019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55216/0 (socket says 172.21.15.3:55216) 2023-09-10T01:06:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 -- 172.21.15.3:0/3837293559 learned_addr learned my addr 172.21.15.3:0/3837293559 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecff7fe700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000732f0 0x7fed0019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 -- 172.21.15.3:0/3837293559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed0010ce40 msgr2=0x7fed00198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed0010ce40 0x7fed00198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 -- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000732f0 msgr2=0x7fed0019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000732f0 0x7fed0019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 -- 172.21.15.3:0/3837293559 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8009d00 con 0x7fed000729d0 2023-09-10T01:06:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fed04f9c700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed0010ce40 0x7fed00198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfffff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed000729d0 0x7fed0019e7f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e9a0 tx=0x7fecf000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.046+0000 7fecfd7fa700 1 -- 172.21.15.3:0/3837293559 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0009790 con 0x7fed000729d0 2023-09-10T01:06:08.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.047+0000 7fecfd7fa700 1 -- 172.21.15.3:0/3837293559 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecf0010430 con 0x7fed000729d0 2023-09-10T01:06:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.047+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed001994e0 con 0x7fed000729d0 2023-09-10T01:06:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.047+0000 7fecfd7fa700 1 -- 172.21.15.3:0/3837293559 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0018680 con 0x7fed000729d0 2023-09-10T01:06:08.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.047+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed001a57e0 con 0x7fed000729d0 2023-09-10T01:06:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.051+0000 7fecfd7fa700 1 -- 172.21.15.3:0/3837293559 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fecf00105a0 con 0x7fed000729d0 2023-09-10T01:06:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.052+0000 7fed1a3fcb80 1 Processor -- start 2023-09-10T01:06:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed1a3fcb80 1 -- start start 2023-09-10T01:06:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc93580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8d300 0x56136fc8d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8dcc0 0x56136fc99f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fbe66d0 con 0x56136fc8dcc0 2023-09-10T01:06:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fbe3ff0 con 0x56136fc8d300 2023-09-10T01:06:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fbdf8e0 con 0x56136fc931a0 2023-09-10T01:06:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed04f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc93580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fecfffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8d300 0x56136fc8d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed04f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc93580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45794/0 (socket says 172.21.15.3:45794) 2023-09-10T01:06:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fed04f9c700 1 -- 172.21.15.3:0/3332970799 learned_addr learned my addr 172.21.15.3:0/3332970799 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.053+0000 7fecff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8dcc0 0x56136fc99f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fed04f9c700 1 -- 172.21.15.3:0/3332970799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8d300 msgr2=0x56136fc8d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fed04f9c700 1 --2- 172.21.15.3:0/3332970799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8d300 0x56136fc8d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fed04f9c700 1 -- 172.21.15.3:0/3332970799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8dcc0 msgr2=0x56136fc99f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fed04f9c700 1 --2- 172.21.15.3:0/3332970799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8dcc0 0x56136fc99f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fed04f9c700 1 -- 172.21.15.3:0/3332970799 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56136fa0bd80 con 0x56136fc931a0 2023-09-10T01:06:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fecfd7fa700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fece80a07b0 0x7fece80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fed04f9c700 1 --2- 172.21.15.3:0/3332970799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc93580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fecf800b0a0 tx=0x7fecf8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baa4d2a84b73ead9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fecfd7fa700 1 -- 172.21.15.3:0/3837293559 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fecf0014070 con 0x7fed000729d0 2023-09-10T01:06:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fecfffff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fece80a07b0 0x7fece80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fece67fc700 1 -- 172.21.15.3:0/3332970799 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800e040 con 0x56136fc931a0 2023-09-10T01:06:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fece67fc700 1 -- 172.21.15.3:0/3332970799 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecf800bcb0 con 0x56136fc931a0 2023-09-10T01:06:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.054+0000 7fece67fc700 1 -- 172.21.15.3:0/3332970799 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8005040 con 0x56136fc931a0 2023-09-10T01:06:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/3332970799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 msgr2=0x56136fc93580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/3332970799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc93580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fecf800b0a0 tx=0x7fecf8009c00 comp rx=0 tx=0).stop 2023-09-10T01:06:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fecfffff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fece80a07b0 0x7fece80a2c70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fecf00c9950 tx=0x7fecf00cfe20 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/3332970799 shutdown_connections 2023-09-10T01:06:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/3332970799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc93580 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/3332970799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8dcc0 0x56136fc99f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/3332970799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8d300 0x56136fc8d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/3332970799 >> 172.21.15.3:0/3332970799 conn(0x56136fc02030 msgr2=0x56136fc88560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.055+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/3332970799 shutdown_connections 2023-09-10T01:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.056+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/3332970799 wait complete. 2023-09-10T01:06:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.056+0000 7fed1a3fcb80 1 -- start start 2023-09-10T01:06:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.056+0000 7fed1a3fcb80 1 Processor -- start 2023-09-10T01:06:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc99dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc96fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc97550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fb7db10 con 0x56136fc8d300 2023-09-10T01:06:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fb7d810 con 0x56136fc8dcc0 2023-09-10T01:06:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fa618c0 con 0x56136fc931a0 2023-09-10T01:06:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed1a3fcb80 10 client.0 _ll_register_callbacks cb 0x56136fac7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc96fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed04f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc99dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc96fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55236/0 (socket says 172.21.15.3:55236) 2023-09-10T01:06:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 -- 172.21.15.3:0/2482051035 learned_addr learned my addr 172.21.15.3:0/2482051035 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fed04f9c700 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc97550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 -- 172.21.15.3:0/2482051035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 msgr2=0x56136fc97550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc97550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 -- 172.21.15.3:0/2482051035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 msgr2=0x56136fc99dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc99dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.057+0000 7fecff7fe700 1 -- 172.21.15.3:0/2482051035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0010ff80 con 0x56136fc8dcc0 2023-09-10T01:06:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fecff7fe700 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc96fd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fecf4012700 tx=0x7fecf4012a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8afb8f12e6f06e99 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fecdf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2482051035 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4017070 con 0x56136fc8dcc0 2023-09-10T01:06:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56136fb37490 con 0x56136fc8dcc0 2023-09-10T01:06:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2482051035 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fecf4010430 con 0x56136fc8dcc0 2023-09-10T01:06:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2482051035 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecf40105f0 con 0x56136fc8dcc0 2023-09-10T01:06:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fed1a3fcb80 10 client.26528 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56136f9f4c00 con 0x56136fc8dcc0 2023-09-10T01:06:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.058+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2482051035 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56136fb37490 con 0x56136fc8dcc0 2023-09-10T01:06:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2482051035 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fecf401c860 con 0x56136fc8dcc0 2023-09-10T01:06:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed1a3fcb80 10 client.26528 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed1a3fcb80 10 client.26528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed1a3fcb80 20 client.26528 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fca73e0 0x56136fca9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed1a3fcb80 4 client.26528 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56136fbf4840 con 0x56136fca73e0 2023-09-10T01:06:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed04f9c700 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fca73e0 0x56136fca9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fed04f9c700 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fca73e0 0x56136fca9800 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8757a5fff94a525f server_cookie=73baa0fcf455bf16 in_seq=0 out_seq=0 2023-09-10T01:06:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.059+0000 7fecdf7fe700 10 client.26528 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2482051035 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56136fbf4840 con 0x56136fca73e0 2023-09-10T01:06:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fecdf7fe700 10 client.26528 handle_command_reply: tid=0 2023-09-10T01:06:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 client.26528 shutdown 2023-09-10T01:06:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 msgr2=0x56136fc96fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc96fd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fecf4012700 tx=0x7fecf4012a10 comp rx=0 tx=0).stop 2023-09-10T01:06:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 shutdown_connections 2023-09-10T01:06:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fca73e0 0x56136fca9800 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc97550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc99dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2482051035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc96fd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fecdf7fe700 0 client.26528 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.060+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 >> 172.21.15.3:0/2482051035 conn(0x56136fc02030 msgr2=0x56136fc877a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.061+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 shutdown_connections 2023-09-10T01:06:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.061+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2482051035 wait complete. 2023-09-10T01:06:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.061+0000 7fed1a3fcb80 20 client.26528 trim_cache size 0 max 16384 2023-09-10T01:06:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.079+0000 7fed1a3fcb80 1 Processor -- start 2023-09-10T01:06:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed1a3fcb80 1 -- start start 2023-09-10T01:06:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc96280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc8ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fc9e900 con 0x56136fc8d300 2023-09-10T01:06:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fc97880 con 0x56136fc8dcc0 2023-09-10T01:06:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fc91bb0 con 0x56136fc931a0 2023-09-10T01:06:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed04f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc96280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fecff7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed04f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc96280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42860/0 (socket says 172.21.15.3:42860) 2023-09-10T01:06:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed04f9c700 1 -- 172.21.15.3:0/1442273350 learned_addr learned my addr 172.21.15.3:0/1442273350 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.080+0000 7fed04f9c700 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc8ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fed04f9c700 1 -- 172.21.15.3:0/1442273350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 msgr2=0x56136fc8ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fed04f9c700 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc8ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fed04f9c700 1 -- 172.21.15.3:0/1442273350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 msgr2=0x56136fc9e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fed04f9c700 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fed04f9c700 1 -- 172.21.15.3:0/1442273350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56136fa321f0 con 0x56136fc8d300 2023-09-10T01:06:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fecff7fe700 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fed04f9c700 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc96280 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fecf80062a0 tx=0x7fecf8007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ad9d4c93b1573d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fece6ffd700 1 -- 172.21.15.3:0/1442273350 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800e040 con 0x56136fc8d300 2023-09-10T01:06:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fece6ffd700 1 -- 172.21.15.3:0/1442273350 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecf800f040 con 0x56136fc8d300 2023-09-10T01:06:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.081+0000 7fece6ffd700 1 -- 172.21.15.3:0/1442273350 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8034930 con 0x56136fc8d300 2023-09-10T01:06:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/1442273350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 msgr2=0x56136fc96280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc96280 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fecf80062a0 tx=0x7fecf8007690 comp rx=0 tx=0).stop 2023-09-10T01:06:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/1442273350 shutdown_connections 2023-09-10T01:06:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc931a0 0x56136fc8ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc8d300 0x56136fc96280 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/1442273350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/1442273350 >> 172.21.15.3:0/1442273350 conn(0x56136fc02030 msgr2=0x56136fc8c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/1442273350 shutdown_connections 2023-09-10T01:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.082+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/1442273350 wait complete. 2023-09-10T01:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.083+0000 7fed1a3fcb80 1 -- start start 2023-09-10T01:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.083+0000 7fed1a3fcb80 1 Processor -- start 2023-09-10T01:06:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc8d300 0x56136fc9c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed1a3fcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc931a0 0x56136fc9ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fcfa3a0 con 0x56136fc931a0 2023-09-10T01:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fcfa510 con 0x56136fc8dcc0 2023-09-10T01:06:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed1a3fcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136fcfa6d0 con 0x56136fc8d300 2023-09-10T01:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed1a3fcb80 10 client.0 _ll_register_callbacks cb 0x7fecf4009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed04f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc8d300 0x56136fc9c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55256/0 (socket says 172.21.15.3:55256) 2023-09-10T01:06:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 -- 172.21.15.3:0/2029419852 learned_addr learned my addr 172.21.15.3:0/2029419852 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fed04f9c700 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc931a0 0x56136fc9ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 -- 172.21.15.3:0/2029419852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc8d300 msgr2=0x56136fc9c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc8d300 0x56136fc9c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 -- 172.21.15.3:0/2029419852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc931a0 msgr2=0x56136fc9ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc931a0 0x56136fc9ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.084+0000 7fecff7fe700 1 -- 172.21.15.3:0/2029419852 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56136f9f4c00 con 0x56136fc8dcc0 2023-09-10T01:06:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fecff7fe700 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9c8f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fecf4000c00 tx=0x7fecf401cac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cee025637ff01df4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fecdf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2029419852 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4017040 con 0x56136fc8dcc0 2023-09-10T01:06:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecf800bcb0 con 0x56136fc8dcc0 2023-09-10T01:06:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2029419852 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fecf40054d0 con 0x56136fc8dcc0 2023-09-10T01:06:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2029419852 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecf40056b0 con 0x56136fc8dcc0 2023-09-10T01:06:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fed1a3fcb80 10 client.26534 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.085+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fecf802f040 con 0x56136fc8dcc0 2023-09-10T01:06:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2029419852 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecf800bcb0 con 0x56136fc8dcc0 2023-09-10T01:06:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2029419852 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fecf402a5d0 con 0x56136fc8dcc0 2023-09-10T01:06:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed1a3fcb80 10 client.26534 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed1a3fcb80 10 client.26534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed1a3fcb80 20 client.26534 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fcfbe10 0x56136fcfe230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed1a3fcb80 4 client.26534 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:06:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56136fbd88c0 con 0x56136fcfbe10 2023-09-10T01:06:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed04f9c700 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fcfbe10 0x56136fcfe230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fed04f9c700 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fcfbe10 0x56136fcfe230 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa75a3117875d399 server_cookie=7b74ac6963f599e0 in_seq=0 out_seq=0 2023-09-10T01:06:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.086+0000 7fecdf7fe700 10 client.26534 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.190+0000 7fecdf7fe700 1 -- 172.21.15.3:0/2029419852 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56136fbd88c0 con 0x56136fcfbe10 2023-09-10T01:06:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.190+0000 7fecdf7fe700 10 client.26534 handle_command_reply: tid=0 2023-09-10T01:06:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 client.26534 shutdown 2023-09-10T01:06:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 msgr2=0x56136fc9c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9c8f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fecf4000c00 tx=0x7fecf401cac0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 shutdown_connections 2023-09-10T01:06:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56136fcfbe10 0x56136fcfe230 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56136fc8d300 0x56136fc9c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56136fc931a0 0x56136fc9ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 --2- 172.21.15.3:0/2029419852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56136fc8dcc0 0x56136fc9c8f0 secure :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fecf4000c00 tx=0x7fecf401cac0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 >> 172.21.15.3:0/2029419852 conn(0x56136fc02030 msgr2=0x56136fc8cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.191+0000 7fecdf7fe700 0 client.26534 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.198+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 shutdown_connections 2023-09-10T01:06:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.198+0000 7fed1a3fcb80 1 -- 172.21.15.3:0/2029419852 wait complete. 2023-09-10T01:06:08.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:06:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:06:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:06:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:06:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.199+0000 7fed1a3fcb80 20 client.26534 trim_cache size 0 max 16384 2023-09-10T01:06:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.205+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fece80a07b0 msgr2=0x7fece80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.205+0000 7fed069ff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fece80a07b0 0x7fece80a2c70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fecf00c9950 tx=0x7fecf00cfe20 comp rx=0 tx=0).stop 2023-09-10T01:06:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.205+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed000729d0 msgr2=0x7fed0019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.205+0000 7fed069ff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed000729d0 0x7fed0019e7f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e9a0 tx=0x7fecf000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.207+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 shutdown_connections 2023-09-10T01:06:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.207+0000 7fed069ff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fece80a07b0 0x7fece80a2c70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.207+0000 7fed069ff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed0010ce40 0x7fed00198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.207+0000 7fed069ff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed000729d0 0x7fed0019e7f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.207+0000 7fed069ff700 1 --2- 172.21.15.3:0/3837293559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed000732f0 0x7fed0019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.207+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 >> 172.21.15.3:0/3837293559 conn(0x7fed000fc0b0 msgr2=0x7fed001048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.211+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 shutdown_connections 2023-09-10T01:06:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:08.212+0000 7fed069ff700 1 -- 172.21.15.3:0/3837293559 wait complete. 2023-09-10T01:06:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:08 smithi092 ceph-mon[134067]: pgmap v90: 129 pgs: 129 active+clean; 62 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 305 op/s 2023-09-10T01:06:08.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:08 smithi003 ceph-mon[97654]: pgmap v90: 129 pgs: 129 active+clean; 62 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 305 op/s 2023-09-10T01:06:08.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:08 smithi136 ceph-mon[133708]: pgmap v90: 129 pgs: 129 active+clean; 62 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 305 op/s 2023-09-10T01:06:08.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 201848 10.05 MB/sec warmup 110 sec latency 98.592 ms 2023-09-10T01:06:09.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 203235 10.03 MB/sec warmup 111 sec latency 20.728 ms 2023-09-10T01:06:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:10 smithi092 ceph-mon[134067]: pgmap v91: 129 pgs: 129 active+clean; 56 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 413 op/s 2023-09-10T01:06:10.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:10 smithi003 ceph-mon[97654]: pgmap v91: 129 pgs: 129 active+clean; 56 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 413 op/s 2023-09-10T01:06:10.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:10 smithi136 ceph-mon[133708]: pgmap v91: 129 pgs: 129 active+clean; 56 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 413 op/s 2023-09-10T01:06:10.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 205848 10.08 MB/sec warmup 112 sec latency 60.188 ms 2023-09-10T01:06:11.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 206950 10.02 MB/sec warmup 113 sec latency 56.001 ms 2023-09-10T01:06:12.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:12 smithi092 ceph-mon[134067]: pgmap v92: 129 pgs: 129 active+clean; 59 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2023-09-10T01:06:12.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:12 smithi003 ceph-mon[97654]: pgmap v92: 129 pgs: 129 active+clean; 59 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2023-09-10T01:06:12.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:12 smithi136 ceph-mon[133708]: pgmap v92: 129 pgs: 129 active+clean; 59 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2023-09-10T01:06:12.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 209363 10.07 MB/sec warmup 114 sec latency 57.441 ms 2023-09-10T01:06:13.745 INFO:tasks.workunit.client.0.smithi003.stdout: 1 210655 10.03 MB/sec warmup 115 sec latency 12.128 ms 2023-09-10T01:06:14.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:14 smithi092 ceph-mon[134067]: pgmap v93: 129 pgs: 129 active+clean; 59 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 303 op/s 2023-09-10T01:06:14.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:14 smithi003 ceph-mon[97654]: pgmap v93: 129 pgs: 129 active+clean; 59 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 303 op/s 2023-09-10T01:06:14.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:14 smithi136 ceph-mon[133708]: pgmap v93: 129 pgs: 129 active+clean; 59 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 303 op/s 2023-09-10T01:06:14.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213087 10.07 MB/sec warmup 116 sec latency 18.731 ms 2023-09-10T01:06:15.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 214429 10.03 MB/sec warmup 117 sec latency 13.635 ms 2023-09-10T01:06:16.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:16 smithi092 ceph-mon[134067]: pgmap v94: 129 pgs: 129 active+clean; 65 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 417 op/s 2023-09-10T01:06:16.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:16 smithi003 ceph-mon[97654]: pgmap v94: 129 pgs: 129 active+clean; 65 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 417 op/s 2023-09-10T01:06:16.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:16 smithi136 ceph-mon[133708]: pgmap v94: 129 pgs: 129 active+clean; 65 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 417 op/s 2023-09-10T01:06:16.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 216791 10.06 MB/sec warmup 118 sec latency 32.301 ms 2023-09-10T01:06:17.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 217910 10.02 MB/sec warmup 119 sec latency 96.076 ms 2023-09-10T01:06:18.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:18 smithi003 ceph-mon[97654]: pgmap v95: 129 pgs: 129 active+clean; 65 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2023-09-10T01:06:18.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:18 smithi136 ceph-mon[133708]: pgmap v95: 129 pgs: 129 active+clean; 65 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2023-09-10T01:06:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:18 smithi092 ceph-mon[134067]: pgmap v95: 129 pgs: 129 active+clean; 65 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2023-09-10T01:06:19.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 221775 5.69 MB/sec execute 1 sec latency 14.012 ms 2023-09-10T01:06:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:20 smithi003 ceph-mon[97654]: pgmap v96: 129 pgs: 129 active+clean; 69 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 375 op/s 2023-09-10T01:06:20.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:06:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:20 smithi136 ceph-mon[133708]: pgmap v96: 129 pgs: 129 active+clean; 69 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 375 op/s 2023-09-10T01:06:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:20 smithi092 ceph-mon[134067]: pgmap v96: 129 pgs: 129 active+clean; 69 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 375 op/s 2023-09-10T01:06:20.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 223822 9.56 MB/sec execute 2 sec latency 134.823 ms 2023-09-10T01:06:20.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 -- 172.21.15.3:0/165918045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0106650 msgr2=0x7fa4c0106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/165918045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0106650 0x7fa4c0106a30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009990 tx=0x7fa4b0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 -- 172.21.15.3:0/165918045 shutdown_connections 2023-09-10T01:06:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/165918045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0106650 0x7fa4c0106a30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/165918045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0101000 0x7fa4c010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/165918045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4c0100640 0x7fa4c0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 -- 172.21.15.3:0/165918045 >> 172.21.15.3:0/165918045 conn(0x7fa4c00fc170 msgr2=0x7fa4c00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.793+0000 7fa4c57ff700 1 -- 172.21.15.3:0/165918045 shutdown_connections 2023-09-10T01:06:20.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.794+0000 7fa4c57ff700 1 -- 172.21.15.3:0/165918045 wait complete. 2023-09-10T01:06:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.795+0000 7fa4c57ff700 1 Processor -- start 2023-09-10T01:06:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.795+0000 7fa4c57ff700 1 -- start start 2023-09-10T01:06:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4c57ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0100640 0x7fa4c019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4c57ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0101000 0x7fa4c019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4c57ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4c0106650 0x7fa4c0198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4c57ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c0198dd0 con 0x7fa4c0106650 2023-09-10T01:06:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4c57ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c0198f40 con 0x7fa4c0100640 2023-09-10T01:06:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4c57ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c0199100 con 0x7fa4c0101000 2023-09-10T01:06:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4be7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0101000 0x7fa4c019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0100640 0x7fa4c019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0100640 0x7fa4c019e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53696/0 (socket says 172.21.15.3:53696) 2023-09-10T01:06:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4be7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0101000 0x7fa4c019ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38780/0 (socket says 172.21.15.3:38780) 2023-09-10T01:06:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 -- 172.21.15.3:0/679814061 learned_addr learned my addr 172.21.15.3:0/679814061 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4bf7fe700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4c0106650 0x7fa4c0198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 -- 172.21.15.3:0/679814061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0101000 msgr2=0x7fa4c019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0101000 0x7fa4c019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 -- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4c0106650 msgr2=0x7fa4c0198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4c0106650 0x7fa4c0198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4beffd700 1 -- 172.21.15.3:0/679814061 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b4009520 con 0x7fa4c0100640 2023-09-10T01:06:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.796+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0101000 0x7fa4c019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.797+0000 7fa4beffd700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0100640 0x7fa4c019e6f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0005910 tx=0x7fa4b0004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.797+0000 7fa4a7fff700 1 -- 172.21.15.3:0/679814061 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b001d070 con 0x7fa4c0100640 2023-09-10T01:06:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.797+0000 7fa4a7fff700 1 -- 172.21.15.3:0/679814061 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4b0022470 con 0x7fa4c0100640 2023-09-10T01:06:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.797+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b00095f0 con 0x7fa4c0100640 2023-09-10T01:06:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.797+0000 7fa4a7fff700 1 -- 172.21.15.3:0/679814061 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0017650 con 0x7fa4c0100640 2023-09-10T01:06:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.797+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c01a5500 con 0x7fa4c0100640 2023-09-10T01:06:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.801+0000 7fa4a7fff700 1 -- 172.21.15.3:0/679814061 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa4b00225e0 con 0x7fa4c0100640 2023-09-10T01:06:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.802+0000 7fa4a7fff700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a80a0830 0x7fa4a80a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.802+0000 7fa4a7fff700 1 -- 172.21.15.3:0/679814061 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa4b00cd4c0 con 0x7fa4c0100640 2023-09-10T01:06:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.802+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a80a0830 0x7fa4a80a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.803+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a80a0830 0x7fa4a80a2cf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009fd0 tx=0x7fa4b4009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.803+0000 7fa4d9080b80 1 Processor -- start 2023-09-10T01:06:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.803+0000 7fa4d9080b80 1 -- start start 2023-09-10T01:06:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be74af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be7e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be81230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384bdcd730 con 0x56384be75030 2023-09-10T01:06:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384bdcb050 con 0x56384be7ee40 2023-09-10T01:06:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384bdc6940 con 0x56384be74710 2023-09-10T01:06:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be74af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be74af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38796/0 (socket says 172.21.15.3:38796) 2023-09-10T01:06:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 -- 172.21.15.3:0/2053172047 learned_addr learned my addr 172.21.15.3:0/2053172047 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4bf7fe700 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be81230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be7e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 -- 172.21.15.3:0/2053172047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 msgr2=0x56384be81230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be81230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 -- 172.21.15.3:0/2053172047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 msgr2=0x56384be7e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be7e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.804+0000 7fa4beffd700 1 -- 172.21.15.3:0/2053172047 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56384bbf2de0 con 0x56384be74710 2023-09-10T01:06:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.805+0000 7fa4beffd700 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be74af0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0006010 tx=0x7fa4b009a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3517275eb1d7b838 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.805+0000 7fa4a57fa700 1 -- 172.21.15.3:0/2053172047 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b009ad10 con 0x56384be74710 2023-09-10T01:06:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.805+0000 7fa4a57fa700 1 -- 172.21.15.3:0/2053172047 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4b009a610 con 0x56384be74710 2023-09-10T01:06:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.805+0000 7fa4a57fa700 1 -- 172.21.15.3:0/2053172047 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b009a870 con 0x56384be74710 2023-09-10T01:06:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2053172047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 msgr2=0x56384be74af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be74af0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0006010 tx=0x7fa4b009a4b0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2053172047 shutdown_connections 2023-09-10T01:06:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be74af0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be81230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2053172047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be7e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2053172047 >> 172.21.15.3:0/2053172047 conn(0x56384bde9090 msgr2=0x56384be6fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2053172047 shutdown_connections 2023-09-10T01:06:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2053172047 wait complete. 2023-09-10T01:06:20.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.806+0000 7fa4d9080b80 1 -- start start 2023-09-10T01:06:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.807+0000 7fa4d9080b80 1 Processor -- start 2023-09-10T01:06:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be7f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be75030 0x56384be7f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be8a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384bd64b70 con 0x56384be7ee40 2023-09-10T01:06:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384bd64870 con 0x56384be75030 2023-09-10T01:06:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384bc48920 con 0x56384be74710 2023-09-10T01:06:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4d9080b80 10 client.0 _ll_register_callbacks cb 0x56384bcaea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4be7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be75030 0x56384be7f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4bf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be8a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4be7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be75030 0x56384be7f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53708/0 (socket says 172.21.15.3:53708) 2023-09-10T01:06:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4be7fc700 1 -- 172.21.15.3:0/1206110512 learned_addr learned my addr 172.21.15.3:0/1206110512 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.808+0000 7fa4beffd700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be7f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4be7fc700 1 -- 172.21.15.3:0/1206110512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 msgr2=0x56384be7f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be7f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4be7fc700 1 -- 172.21.15.3:0/1206110512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 msgr2=0x56384be8a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be8a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4be7fc700 1 -- 172.21.15.3:0/1206110512 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c01a5500 con 0x56384be75030 2023-09-10T01:06:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4beffd700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be7f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4bf7fe700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be8a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be75030 0x56384be7f760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400eae0 tx=0x7fa4b40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaef463dda1ec473 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4a2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/1206110512 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b401eaa0 con 0x56384be75030 2023-09-10T01:06:20.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/1206110512 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa4b401ec00 con 0x56384be75030 2023-09-10T01:06:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56384bd1e4f0 con 0x56384be75030 2023-09-10T01:06:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/1206110512 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4b401edc0 con 0x56384be75030 2023-09-10T01:06:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4d9080b80 10 client.26546 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.809+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56384bbdbc60 con 0x56384be75030 2023-09-10T01:06:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/1206110512 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56384bd1e4f0 con 0x56384be75030 2023-09-10T01:06:20.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/1206110512 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa4b40263e0 con 0x56384be75030 2023-09-10T01:06:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4d9080b80 10 client.26546 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4d9080b80 10 client.26546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4d9080b80 20 client.26546 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384be8e780 0x56384be90ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4d9080b80 4 client.26546 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56384bddb8a0 con 0x56384be8e780 2023-09-10T01:06:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.810+0000 7fa4beffd700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384be8e780 0x56384be90ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.811+0000 7fa4a2ff5700 10 client.26546 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.811+0000 7fa4beffd700 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384be8e780 0x56384be90ba0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e97e6b6a2f860c48 server_cookie=a83b605d037c2ba3 in_seq=0 out_seq=0 2023-09-10T01:06:20.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.811+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/1206110512 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56384bddb8a0 con 0x56384be8e780 2023-09-10T01:06:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.811+0000 7fa4a2ff5700 10 client.26546 handle_command_reply: tid=0 2023-09-10T01:06:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.811+0000 7fa4d9080b80 1 client.26546 shutdown 2023-09-10T01:06:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.811+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be75030 msgr2=0x56384be7f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.811+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be75030 0x56384be7f760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400eae0 tx=0x7fa4b40047c0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 shutdown_connections 2023-09-10T01:06:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384be8e780 0x56384be90ba0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be7f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be8a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4a2ff5700 0 client.26546 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1206110512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be75030 0x56384be7f760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 >> 172.21.15.3:0/1206110512 conn(0x56384bde9090 msgr2=0x56384be6f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 shutdown_connections 2023-09-10T01:06:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1206110512 wait complete. 2023-09-10T01:06:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.812+0000 7fa4d9080b80 20 client.26546 trim_cache size 0 max 16384 2023-09-10T01:06:20.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 Processor -- start 2023-09-10T01:06:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 -- start start 2023-09-10T01:06:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be74710 0x56384be84230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be75030 0x56384be7d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be899a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384be75670 con 0x56384be7ee40 2023-09-10T01:06:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384be8bc60 con 0x56384be74710 2023-09-10T01:06:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384be7e5a0 con 0x56384be75030 2023-09-10T01:06:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be74710 0x56384be84230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4be7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be75030 0x56384be7d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be74710 0x56384be84230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53712/0 (socket says 172.21.15.3:53712) 2023-09-10T01:06:20.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4beffd700 1 -- 172.21.15.3:0/1050000444 learned_addr learned my addr 172.21.15.3:0/1050000444 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.832+0000 7fa4bf7fe700 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be899a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4beffd700 1 -- 172.21.15.3:0/1050000444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be75030 msgr2=0x56384be7d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4beffd700 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be75030 0x56384be7d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4beffd700 1 -- 172.21.15.3:0/1050000444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 msgr2=0x56384be899a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4beffd700 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be899a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4beffd700 1 -- 172.21.15.3:0/1050000444 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56384bc19250 con 0x56384be74710 2023-09-10T01:06:20.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4beffd700 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be74710 0x56384be84230 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0097080 tx=0x7fa4b0085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbfe0a9c56dff4c3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4a5ffb700 1 -- 172.21.15.3:0/1050000444 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00adea0 con 0x56384be74710 2023-09-10T01:06:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4a5ffb700 1 -- 172.21.15.3:0/1050000444 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4b00a2670 con 0x56384be74710 2023-09-10T01:06:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.833+0000 7fa4a5ffb700 1 -- 172.21.15.3:0/1050000444 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00cd0e0 con 0x56384be74710 2023-09-10T01:06:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1050000444 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be74710 msgr2=0x56384be84230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be74710 0x56384be84230 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0097080 tx=0x7fa4b0085040 comp rx=0 tx=0).stop 2023-09-10T01:06:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1050000444 shutdown_connections 2023-09-10T01:06:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be7ee40 0x56384be899a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be74710 0x56384be84230 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/1050000444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be75030 0x56384be7d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1050000444 >> 172.21.15.3:0/1050000444 conn(0x56384bde9090 msgr2=0x56384be79270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1050000444 shutdown_connections 2023-09-10T01:06:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.834+0000 7fa4d9080b80 1 -- 172.21.15.3:0/1050000444 wait complete. 2023-09-10T01:06:20.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.835+0000 7fa4d9080b80 1 -- start start 2023-09-10T01:06:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.835+0000 7fa4d9080b80 1 Processor -- start 2023-09-10T01:06:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be83fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4d9080b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be80830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384be80d70 con 0x56384be75030 2023-09-10T01:06:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384be80ee0 con 0x56384be7ee40 2023-09-10T01:06:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4d9080b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56384be810a0 con 0x56384be74710 2023-09-10T01:06:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4d9080b80 10 client.0 _ll_register_callbacks cb 0x7fa4b400e8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:20.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be83fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be83fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38824/0 (socket says 172.21.15.3:38824) 2023-09-10T01:06:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 -- 172.21.15.3:0/2180815469 learned_addr learned my addr 172.21.15.3:0/2180815469 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be802f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4bf7fe700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be80830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 -- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 msgr2=0x56384be80830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be80830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 -- 172.21.15.3:0/2180815469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 msgr2=0x56384be802f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:20.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be802f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4beffd700 1 -- 172.21.15.3:0/2180815469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56384bbf2de0 con 0x56384be74710 2023-09-10T01:06:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.836+0000 7fa4bf7fe700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be80830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4beffd700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be83fe0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa4b009a4f0 tx=0x7fa4b0096290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa719dece8aaa988 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:20.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4a2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/2180815469 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00adea0 con 0x56384be74710 2023-09-10T01:06:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/2180815469 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa4b009e130 con 0x56384be74710 2023-09-10T01:06:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4b009a610 con 0x56384be74710 2023-09-10T01:06:20.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/2180815469 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4b00cd0e0 con 0x56384be74710 2023-09-10T01:06:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4d9080b80 10 client.26467 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56384bbf12b0 con 0x56384be74710 2023-09-10T01:06:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/2180815469 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4b009a610 con 0x56384be74710 2023-09-10T01:06:20.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.837+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/2180815469 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa4b009a870 con 0x56384be74710 2023-09-10T01:06:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4d9080b80 10 client.26467 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4d9080b80 10 client.26467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4d9080b80 20 client.26467 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384bee3490 0x56384bee5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4d9080b80 4 client.26467 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:06:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56384bdbf920 con 0x56384bee3490 2023-09-10T01:06:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384bee3490 0x56384bee5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4be7fc700 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384bee3490 0x56384bee5880 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20e39645f7b06aa server_cookie=38ace930dd534c7 in_seq=0 out_seq=0 2023-09-10T01:06:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4a2ff5700 10 client.26467 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4a2ff5700 1 -- 172.21.15.3:0/2180815469 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56384bdbf920 con 0x56384bee3490 2023-09-10T01:06:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4a2ff5700 10 client.26467 handle_command_reply: tid=0 2023-09-10T01:06:20.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.838+0000 7fa4d9080b80 1 client.26467 shutdown 2023-09-10T01:06:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 msgr2=0x56384be83fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be83fe0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa4b009a4f0 tx=0x7fa4b0096290 comp rx=0 tx=0).stop 2023-09-10T01:06:20.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 shutdown_connections 2023-09-10T01:06:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56384bee3490 0x56384bee5880 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56384be74710 0x56384be83fe0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56384be7ee40 0x56384be80830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 --2- 172.21.15.3:0/2180815469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56384be75030 0x56384be802f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4a2ff5700 0 client.26467 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 >> 172.21.15.3:0/2180815469 conn(0x56384bde9090 msgr2=0x56384be73700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 shutdown_connections 2023-09-10T01:06:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 1 -- 172.21.15.3:0/2180815469 wait complete. 2023-09-10T01:06:20.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.839+0000 7fa4d9080b80 20 client.26467 trim_cache size 0 max 16384 2023-09-10T01:06:20.857 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:06:20.857 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:06:20.857 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:06:20.857 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:06:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a80a0830 msgr2=0x7fa4a80a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a80a0830 0x7fa4a80a2cf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009fd0 tx=0x7fa4b4009260 comp rx=0 tx=0).stop 2023-09-10T01:06:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0100640 msgr2=0x7fa4c019e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0100640 0x7fa4c019e6f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0005910 tx=0x7fa4b0004710 comp rx=0 tx=0).stop 2023-09-10T01:06:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 shutdown_connections 2023-09-10T01:06:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a80a0830 0x7fa4a80a2cf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4c0106650 0x7fa4c0198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4c0100640 0x7fa4c019e6f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 --2- 172.21.15.3:0/679814061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4c0101000 0x7fa4c019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 >> 172.21.15.3:0/679814061 conn(0x7fa4c00fc170 msgr2=0x7fa4c010b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 shutdown_connections 2023-09-10T01:06:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:20.840+0000 7fa4c57ff700 1 -- 172.21.15.3:0/679814061 wait complete. 2023-09-10T01:06:20.861 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:06:20.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:06:21.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.074+0000 7fd9b35ff700 1 -- 172.21.15.3:0/2108353125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac106640 msgr2=0x7fd9ac106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.074+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/2108353125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac106640 0x7fd9ac106a20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009990 tx=0x7fd9a8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.075+0000 7fd9b35ff700 1 -- 172.21.15.3:0/2108353125 shutdown_connections 2023-09-10T01:06:21.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.075+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/2108353125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac106640 0x7fd9ac106a20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.075+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/2108353125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9ac100ff0 0x7fd9ac10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.075+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/2108353125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9ac100630 0x7fd9ac100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.075+0000 7fd9b35ff700 1 -- 172.21.15.3:0/2108353125 >> 172.21.15.3:0/2108353125 conn(0x7fd9ac0fc140 msgr2=0x7fd9ac0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.075+0000 7fd9b35ff700 1 -- 172.21.15.3:0/2108353125 shutdown_connections 2023-09-10T01:06:21.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.075+0000 7fd9b35ff700 1 -- 172.21.15.3:0/2108353125 wait complete. 2023-09-10T01:06:21.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.076+0000 7fd9b35ff700 1 Processor -- start 2023-09-10T01:06:21.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.076+0000 7fd9b35ff700 1 -- start start 2023-09-10T01:06:21.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.076+0000 7fd9b35ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac100630 0x7fd9ac19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.076+0000 7fd9b35ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9ac100ff0 0x7fd9ac19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.076+0000 7fd9b35ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9ac106640 0x7fd9ac196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b35ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac196c00 con 0x7fd9ac106640 2023-09-10T01:06:21.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b35ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac196d70 con 0x7fd9ac100ff0 2023-09-10T01:06:21.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b35ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac196f30 con 0x7fd9ac100630 2023-09-10T01:06:21.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac100630 0x7fd9ac19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b0b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9ac100ff0 0x7fd9ac19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac100630 0x7fd9ac19c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38856/0 (socket says 172.21.15.3:38856) 2023-09-10T01:06:21.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b0b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9ac100ff0 0x7fd9ac19ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53734/0 (socket says 172.21.15.3:53734) 2023-09-10T01:06:21.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 -- 172.21.15.3:0/58611088 learned_addr learned my addr 172.21.15.3:0/58611088 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b1b9c700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9ac106640 0x7fd9ac196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 -- 172.21.15.3:0/58611088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9ac100ff0 msgr2=0x7fd9ac19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9ac100ff0 0x7fd9ac19ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 -- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9ac106640 msgr2=0x7fd9ac196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9ac106640 0x7fd9ac196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 -- 172.21.15.3:0/58611088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a80095f0 con 0x7fd9ac100630 2023-09-10T01:06:21.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.077+0000 7fd9b139b700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac100630 0x7fd9ac19c520 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8005910 tx=0x7fd9a8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.078+0000 7fd9a27fc700 1 -- 172.21.15.3:0/58611088 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a801d070 con 0x7fd9ac100630 2023-09-10T01:06:21.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.078+0000 7fd9a27fc700 1 -- 172.21.15.3:0/58611088 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9a8022470 con 0x7fd9ac100630 2023-09-10T01:06:21.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.078+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac1971b0 con 0x7fd9ac100630 2023-09-10T01:06:21.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.078+0000 7fd9a27fc700 1 -- 172.21.15.3:0/58611088 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8017650 con 0x7fd9ac100630 2023-09-10T01:06:21.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.078+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac1a3510 con 0x7fd9ac100630 2023-09-10T01:06:21.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.079+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac067480 con 0x7fd9ac100630 2023-09-10T01:06:21.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.081+0000 7fd9a27fc700 1 -- 172.21.15.3:0/58611088 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd9a800fad0 con 0x7fd9ac100630 2023-09-10T01:06:21.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.083+0000 7fd9a27fc700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9980a07b0 0x7fd9980a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.083+0000 7fd9a27fc700 1 -- 172.21.15.3:0/58611088 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd9a80cdda0 con 0x7fd9ac100630 2023-09-10T01:06:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.083+0000 7fd9b0b9a700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9980a07b0 0x7fd9980a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.084+0000 7fd9b0b9a700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9980a07b0 0x7fd9980a2c70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0feb90 tx=0x7fd99c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.084+0000 7fd9a27fc700 1 -- 172.21.15.3:0/58611088 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd9a8097230 con 0x7fd9ac100630 2023-09-10T01:06:21.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.385+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9ac197aa0 con 0x7fd9ac100630 2023-09-10T01:06:21.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.386+0000 7fd9a27fc700 1 -- 172.21.15.3:0/58611088 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11295 (secure 0 0 0) 0x7fd9a809a850 con 0x7fd9ac100630 2023-09-10T01:06:21.387 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:06:21.387 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:04:38.815186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:06:21.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.388+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9980a07b0 msgr2=0x7fd9980a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.388+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9980a07b0 0x7fd9980a2c70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0feb90 tx=0x7fd99c009260 comp rx=0 tx=0).stop 2023-09-10T01:06:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.388+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac100630 msgr2=0x7fd9ac19c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.388+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac100630 0x7fd9ac19c520 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8005910 tx=0x7fd9a8004730 comp rx=0 tx=0).stop 2023-09-10T01:06:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 shutdown_connections 2023-09-10T01:06:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9980a07b0 0x7fd9980a2c70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9ac100630 0x7fd9ac19c520 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9ac106640 0x7fd9ac196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 --2- 172.21.15.3:0/58611088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9ac100ff0 0x7fd9ac19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 >> 172.21.15.3:0/58611088 conn(0x7fd9ac0fc140 msgr2=0x7fd9ac0fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 shutdown_connections 2023-09-10T01:06:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.389+0000 7fd9b35ff700 1 -- 172.21.15.3:0/58611088 wait complete. 2023-09-10T01:06:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 23 2023-09-10T01:06:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 -- 172.21.15.3:0/1961329861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 msgr2=0x7f9cf4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/1961329861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf4106a60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00099c0 tx=0x7f9cf0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 -- 172.21.15.3:0/1961329861 shutdown_connections 2023-09-10T01:06:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/1961329861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cf4100f80 0x7f9cf410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/1961329861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf4106a60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/1961329861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cf41005c0 0x7f9cf4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 -- 172.21.15.3:0/1961329861 >> 172.21.15.3:0/1961329861 conn(0x7f9cf40fc0b0 msgr2=0x7f9cf40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 -- 172.21.15.3:0/1961329861 shutdown_connections 2023-09-10T01:06:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.403+0000 7f9cfbbff700 1 -- 172.21.15.3:0/1961329861 wait complete. 2023-09-10T01:06:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 Processor -- start 2023-09-10T01:06:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 -- start start 2023-09-10T01:06:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cf41005c0 0x7f9cf4196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cf4100f80 0x7f9cf4196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf419afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf419b610 con 0x7f9cf4100f80 2023-09-10T01:06:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf419b780 con 0x7f9cf4106680 2023-09-10T01:06:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfbbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf419b940 con 0x7f9cf41005c0 2023-09-10T01:06:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cfa19c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf419afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.405+0000 7f9cf999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cf41005c0 0x7f9cf4196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf419afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53752/0 (socket says 172.21.15.3:53752) 2023-09-10T01:06:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cf999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cf41005c0 0x7f9cf4196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38888/0 (socket says 172.21.15.3:38888) 2023-09-10T01:06:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 -- 172.21.15.3:0/4285500155 learned_addr learned my addr 172.21.15.3:0/4285500155 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cf919a700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cf4100f80 0x7f9cf4196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 -- 172.21.15.3:0/4285500155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cf41005c0 msgr2=0x7f9cf4196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cf41005c0 0x7f9cf4196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 -- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cf4100f80 msgr2=0x7f9cf4196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cf4100f80 0x7f9cf4196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 -- 172.21.15.3:0/4285500155 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec009c40 con 0x7f9cf4106680 2023-09-10T01:06:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cf919a700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cf4100f80 0x7f9cf4196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:21.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf419afd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00c370 tx=0x7f9cec00c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9ceaffd700 1 -- 172.21.15.3:0/4285500155 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec017070 con 0x7f9cf4106680 2023-09-10T01:06:21.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.406+0000 7f9ceaffd700 1 -- 172.21.15.3:0/4285500155 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cec00f040 con 0x7f9cf4106680 2023-09-10T01:06:21.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.407+0000 7f9ceaffd700 1 -- 172.21.15.3:0/4285500155 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec00e050 con 0x7f9cf4106680 2023-09-10T01:06:21.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.407+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf00095f0 con 0x7f9cf4106680 2023-09-10T01:06:21.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.407+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf4072830 con 0x7f9cf4106680 2023-09-10T01:06:21.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:06:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.410+0000 7f9ceaffd700 1 -- 172.21.15.3:0/4285500155 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9cec004520 con 0x7f9cf4106680 2023-09-10T01:06:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.411+0000 7f9d0f60db80 1 Processor -- start 2023-09-10T01:06:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9d0f60db80 1 -- start start 2023-09-10T01:06:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9ceaffd700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ce00a0830 0x7f9ce00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9ceaffd700 1 -- 172.21.15.3:0/4285500155 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9cec0cbcf0 con 0x7f9cf4106680 2023-09-10T01:06:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9cf919a700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ce00a0830 0x7f9ce00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684300 0x55b9ba684780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684cc0 0x55b9ba690f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba5dd6d0 con 0x55b9ba68a1a0 2023-09-10T01:06:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba5daff0 con 0x55b9ba684300 2023-09-10T01:06:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba5d68e0 con 0x55b9ba684cc0 2023-09-10T01:06:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9cf919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684300 0x55b9ba684780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9cf999b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9cf919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684300 0x55b9ba684780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53766/0 (socket says 172.21.15.3:53766) 2023-09-10T01:06:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.412+0000 7f9cf919a700 1 -- 172.21.15.3:0/3375082813 learned_addr learned my addr 172.21.15.3:0/3375082813 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684cc0 0x55b9ba690f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf919a700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ce00a0830 0x7f9ce00a2cf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40135f0 tx=0x7f9ce40139b0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf999b700 1 -- 172.21.15.3:0/3375082813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684cc0 msgr2=0x55b9ba690f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf999b700 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684cc0 0x55b9ba690f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf999b700 1 -- 172.21.15.3:0/3375082813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684300 msgr2=0x55b9ba684780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf999b700 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684300 0x55b9ba684780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf999b700 1 -- 172.21.15.3:0/3375082813 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9ba402d80 con 0x55b9ba68a1a0 2023-09-10T01:06:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684cc0 0x55b9ba690f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf919a700 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684300 0x55b9ba684780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cf999b700 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68a580 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00098e0 tx=0x7f9cf0005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e72128aac77e4a8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.413+0000 7f9cdffff700 1 -- 172.21.15.3:0/3375082813 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf001bbc0 con 0x55b9ba68a1a0 2023-09-10T01:06:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.414+0000 7f9cdffff700 1 -- 172.21.15.3:0/3375082813 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cf0022040 con 0x55b9ba68a1a0 2023-09-10T01:06:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.414+0000 7f9cdffff700 1 -- 172.21.15.3:0/3375082813 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0003ac0 con 0x55b9ba68a1a0 2023-09-10T01:06:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.414+0000 7f9d0f60db80 1 -- 172.21.15.3:0/3375082813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 msgr2=0x55b9ba68a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.414+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68a580 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00098e0 tx=0x7f9cf0005c70 comp rx=0 tx=0).stop 2023-09-10T01:06:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 -- 172.21.15.3:0/3375082813 shutdown_connections 2023-09-10T01:06:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684cc0 0x55b9ba690f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68a580 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/3375082813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684300 0x55b9ba684780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 -- 172.21.15.3:0/3375082813 >> 172.21.15.3:0/3375082813 conn(0x55b9ba5f9030 msgr2=0x55b9ba67f560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 -- 172.21.15.3:0/3375082813 shutdown_connections 2023-09-10T01:06:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 -- 172.21.15.3:0/3375082813 wait complete. 2023-09-10T01:06:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 -- start start 2023-09-10T01:06:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.415+0000 7f9d0f60db80 1 Processor -- start 2023-09-10T01:06:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba68e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba574b10 con 0x55b9ba68a1a0 2023-09-10T01:06:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba574810 con 0x55b9ba684cc0 2023-09-10T01:06:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba4588c0 con 0x55b9ba684300 2023-09-10T01:06:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9d0f60db80 10 client.0 _ll_register_callbacks cb 0x55b9ba4bea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9cf999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9cf999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38912/0 (socket says 172.21.15.3:38912) 2023-09-10T01:06:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.416+0000 7f9cf999b700 1 -- 172.21.15.3:0/2854784857 learned_addr learned my addr 172.21.15.3:0/2854784857 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cf999b700 1 -- 172.21.15.3:0/2854784857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 msgr2=0x55b9ba68e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cf999b700 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba68e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cf999b700 1 -- 172.21.15.3:0/2854784857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 msgr2=0x55b9ba68e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cf999b700 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cf999b700 1 -- 172.21.15.3:0/2854784857 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce400f3c0 con 0x55b9ba684300 2023-09-10T01:06:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:06:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cf999b700 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0029040 tx=0x7f9cf0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1400cc8e33a05d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.417+0000 7f9cdd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.418+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/2854784857 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0005480 con 0x55b9ba684300 2023-09-10T01:06:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.418+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9ba52e490 con 0x55b9ba684300 2023-09-10T01:06:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.418+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/2854784857 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9cf000b890 con 0x55b9ba684300 2023-09-10T01:06:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.418+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/2854784857 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cf00055e0 con 0x55b9ba684300 2023-09-10T01:06:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.418+0000 7f9d0f60db80 10 client.26485 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.418+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f9cf410dd40 con 0x55b9ba684300 2023-09-10T01:06:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/2854784857 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b9ba52e490 con 0x55b9ba684300 2023-09-10T01:06:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/2854784857 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9cf000f4f0 con 0x55b9ba684300 2023-09-10T01:06:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9d0f60db80 10 client.26485 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9d0f60db80 10 client.26485 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9d0f60db80 20 client.26485 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6a6600 0x55b9ba6a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9d0f60db80 4 client.26485 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9ba5eb840 con 0x55b9ba6a6600 2023-09-10T01:06:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.419+0000 7f9cf919a700 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6a6600 0x55b9ba6a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.420+0000 7f9cdd7fa700 10 client.26485 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.420+0000 7f9cf919a700 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6a6600 0x55b9ba6a8aa0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2845d14cde495238 server_cookie=c191e52b9c74d81e in_seq=0 out_seq=0 2023-09-10T01:06:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/2854784857 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b9ba5eb840 con 0x55b9ba6a6600 2023-09-10T01:06:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9cdd7fa700 10 client.26485 handle_command_reply: tid=0 2023-09-10T01:06:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 client.26485 shutdown 2023-09-10T01:06:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 msgr2=0x55b9ba690de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0029040 tx=0x7f9cf0009310 comp rx=0 tx=0).stop 2023-09-10T01:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 shutdown_connections 2023-09-10T01:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6a6600 0x55b9ba6a8aa0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/2854784857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba68e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 >> 172.21.15.3:0/2854784857 conn(0x55b9ba5f9030 msgr2=0x55b9ba67e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.421+0000 7f9cdd7fa700 0 client.26485 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.422+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 shutdown_connections 2023-09-10T01:06:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.422+0000 7f9d0f60db80 1 -- 172.21.15.3:0/2854784857 wait complete. 2023-09-10T01:06:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.422+0000 7f9d0f60db80 20 client.26485 trim_cache size 0 max 16384 2023-09-10T01:06:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.446+0000 7f9d0f60db80 1 Processor -- start 2023-09-10T01:06:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.446+0000 7f9d0f60db80 1 -- start start 2023-09-10T01:06:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba684cc0 0x55b9ba68d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba68f020 con 0x55b9ba684cc0 2023-09-10T01:06:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba680270 con 0x55b9ba68a1a0 2023-09-10T01:06:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba687db0 con 0x55b9ba684300 2023-09-10T01:06:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cf999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cfa19c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cfa19c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53786/0 (socket says 172.21.15.3:53786) 2023-09-10T01:06:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cf999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38928/0 (socket says 172.21.15.3:38928) 2023-09-10T01:06:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cfa19c700 1 -- 172.21.15.3:0/1392783968 learned_addr learned my addr 172.21.15.3:0/1392783968 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cf919a700 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba684cc0 0x55b9ba68d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cf999b700 1 -- 172.21.15.3:0/1392783968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba68a1a0 msgr2=0x55b9ba68dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cf999b700 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cf999b700 1 -- 172.21.15.3:0/1392783968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba684cc0 msgr2=0x55b9ba68d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.447+0000 7f9cf999b700 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba684cc0 0x55b9ba68d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.448+0000 7f9cf999b700 1 -- 172.21.15.3:0/1392783968 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9ba3ebc00 con 0x55b9ba684300 2023-09-10T01:06:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.448+0000 7f9cf919a700 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba684cc0 0x55b9ba68d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.448+0000 7f9cf999b700 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690980 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00098e0 tx=0x7f9cf0009cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e78f9144e6da0c9b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.448+0000 7f9ce8bff700 1 -- 172.21.15.3:0/1392783968 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf000f660 con 0x55b9ba684300 2023-09-10T01:06:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.448+0000 7f9ce8bff700 1 -- 172.21.15.3:0/1392783968 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cf000fc60 con 0x55b9ba684300 2023-09-10T01:06:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.448+0000 7f9ce8bff700 1 -- 172.21.15.3:0/1392783968 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0003b00 con 0x55b9ba684300 2023-09-10T01:06:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1392783968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 msgr2=0x55b9ba690980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690980 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00098e0 tx=0x7f9cf0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1392783968 shutdown_connections 2023-09-10T01:06:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba690980 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba68a1a0 0x55b9ba68dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1392783968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba684cc0 0x55b9ba68d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1392783968 >> 172.21.15.3:0/1392783968 conn(0x55b9ba5f9030 msgr2=0x55b9ba67f090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1392783968 shutdown_connections 2023-09-10T01:06:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.449+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1392783968 wait complete. 2023-09-10T01:06:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.450+0000 7f9d0f60db80 1 -- start start 2023-09-10T01:06:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.450+0000 7f9d0f60db80 1 Processor -- start 2023-09-10T01:06:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.450+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba694e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.450+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba695380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.450+0000 7f9d0f60db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba6933d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba693ac0 con 0x55b9ba68a1a0 2023-09-10T01:06:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba693c30 con 0x55b9ba684cc0 2023-09-10T01:06:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9d0f60db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ba693df0 con 0x55b9ba684300 2023-09-10T01:06:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9d0f60db80 10 client.0 _ll_register_callbacks cb 0x7f9ce400b9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba695380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba695380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53790/0 (socket says 172.21.15.3:53790) 2023-09-10T01:06:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 -- 172.21.15.3:0/1191073933 learned_addr learned my addr 172.21.15.3:0/1191073933 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf999b700 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba694e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cfa19c700 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba6933d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 -- 172.21.15.3:0/1191073933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 msgr2=0x55b9ba694e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba694e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 -- 172.21.15.3:0/1191073933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 msgr2=0x55b9ba6933d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba6933d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.451+0000 7f9cf919a700 1 -- 172.21.15.3:0/1191073933 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf410dd40 con 0x55b9ba684cc0 2023-09-10T01:06:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9cf919a700 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba695380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55b9ba67f060 tx=0x7f9ce40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5569f16de72c7202 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9cdd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/1191073933 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce40284f0 con 0x55b9ba684cc0 2023-09-10T01:06:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9ba5cf8c0 con 0x55b9ba684cc0 2023-09-10T01:06:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/1191073933 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9ce4025040 con 0x55b9ba684cc0 2023-09-10T01:06:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/1191073933 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ce4028880 con 0x55b9ba684cc0 2023-09-10T01:06:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9d0f60db80 10 client.26573 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b9ba4291f0 con 0x55b9ba684cc0 2023-09-10T01:06:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.452+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/1191073933 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b9ba5cf8c0 con 0x55b9ba684cc0 2023-09-10T01:06:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/1191073933 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9ce400fbd0 con 0x55b9ba684cc0 2023-09-10T01:06:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9d0f60db80 10 client.26573 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9d0f60db80 10 client.26573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9d0f60db80 20 client.26573 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6fb6d0 0x55b9ba6fdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9d0f60db80 4 client.26573 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:06:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b9ba590050 con 0x55b9ba6fb6d0 2023-09-10T01:06:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9cf999b700 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6fb6d0 0x55b9ba6fdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9cdd7fa700 10 client.26573 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.453+0000 7f9cf999b700 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6fb6d0 0x55b9ba6fdaf0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321e0e35ddf67451 server_cookie=32968bfaa9265d19 in_seq=0 out_seq=0 2023-09-10T01:06:21.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9cdd7fa700 1 -- 172.21.15.3:0/1191073933 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b9ba590050 con 0x55b9ba6fb6d0 2023-09-10T01:06:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9cdd7fa700 10 client.26573 handle_command_reply: tid=0 2023-09-10T01:06:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 client.26573 shutdown 2023-09-10T01:06:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 msgr2=0x55b9ba695380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba695380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55b9ba67f060 tx=0x7f9ce40047c0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 shutdown_connections 2023-09-10T01:06:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b9ba6fb6d0 0x55b9ba6fdaf0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b9ba684300 0x55b9ba694e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b9ba68a1a0 0x55b9ba6933d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 --2- 172.21.15.3:0/1191073933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b9ba684cc0 0x55b9ba695380 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9cdd7fa700 0 client.26573 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.572+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 >> 172.21.15.3:0/1191073933 conn(0x55b9ba5f9030 msgr2=0x55b9ba6832f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.573+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 shutdown_connections 2023-09-10T01:06:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.573+0000 7f9d0f60db80 1 -- 172.21.15.3:0/1191073933 wait complete. 2023-09-10T01:06:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.573+0000 7f9d0f60db80 20 client.26573 trim_cache size 0 max 16384 2023-09-10T01:06:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:06:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:06:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:06:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:06:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ce00a0830 msgr2=0x7f9ce00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ce00a0830 0x7f9ce00a2cf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40135f0 tx=0x7f9ce40139b0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 msgr2=0x7f9cf419afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf419afd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00c370 tx=0x7f9cec00c680 comp rx=0 tx=0).stop 2023-09-10T01:06:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 shutdown_connections 2023-09-10T01:06:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ce00a0830 0x7f9ce00a2cf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cf41005c0 0x7f9cf4196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cf4106680 0x7f9cf419afd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 --2- 172.21.15.3:0/4285500155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cf4100f80 0x7f9cf4196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 >> 172.21.15.3:0/4285500155 conn(0x7f9cf40fc0b0 msgr2=0x7f9cf40fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 shutdown_connections 2023-09-10T01:06:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:21.574+0000 7f9cfbbff700 1 -- 172.21.15.3:0/4285500155 wait complete. 2023-09-10T01:06:21.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 -- 172.21.15.3:0/152143624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d81005f0 msgr2=0x7f35d81009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/152143624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d81005f0 0x7f35d81009d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009990 tx=0x7f35c8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 -- 172.21.15.3:0/152143624 shutdown_connections 2023-09-10T01:06:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/152143624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 0x7f35d810d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/152143624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d81005f0 0x7f35d81009d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/152143624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d8100fa0 0x7f35d8104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 -- 172.21.15.3:0/152143624 >> 172.21.15.3:0/152143624 conn(0x7f35d80fbe50 msgr2=0x7f35d80fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 -- 172.21.15.3:0/152143624 shutdown_connections 2023-09-10T01:06:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.701+0000 7f35dc9fe700 1 -- 172.21.15.3:0/152143624 wait complete. 2023-09-10T01:06:21.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.702+0000 7f35dc9fe700 1 Processor -- start 2023-09-10T01:06:21.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35dc9fe700 1 -- start start 2023-09-10T01:06:21.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35dc9fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d81005f0 0x7f35d8072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35dc9fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 0x7f35d8072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35dc9fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d8073380 0x7f35d81a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35dc9fe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81a4d50 con 0x7f35d81005f0 2023-09-10T01:06:21.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35dc9fe700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81a4ec0 con 0x7f35d8073380 2023-09-10T01:06:21.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35dc9fe700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81a5080 con 0x7f35d8105710 2023-09-10T01:06:21.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35d659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 0x7f35d8072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35d659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 0x7f35d8072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38952/0 (socket says 172.21.15.3:38952) 2023-09-10T01:06:21.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35d659c700 1 -- 172.21.15.3:0/2741981580 learned_addr learned my addr 172.21.15.3:0/2741981580 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.703+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d81005f0 0x7f35d8072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35d759e700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d8073380 0x7f35d81a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35d659c700 1 -- 172.21.15.3:0/2741981580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d8073380 msgr2=0x7f35d81a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35d659c700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d8073380 0x7f35d81a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35d659c700 1 -- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d81005f0 msgr2=0x7f35d8072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35d659c700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d81005f0 0x7f35d8072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35d659c700 1 -- 172.21.15.3:0/2741981580 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0009c40 con 0x7f35d8105710 2023-09-10T01:06:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35d659c700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 0x7f35d8072e40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00e970 tx=0x7f35cc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35bffff700 1 -- 172.21.15.3:0/2741981580 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc009790 con 0x7f35d8105710 2023-09-10T01:06:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35bffff700 1 -- 172.21.15.3:0/2741981580 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35cc00c7c0 con 0x7f35d8105710 2023-09-10T01:06:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.704+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c80095f0 con 0x7f35d8105710 2023-09-10T01:06:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.705+0000 7f35bffff700 1 -- 172.21.15.3:0/2741981580 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc0054b0 con 0x7f35d8105710 2023-09-10T01:06:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.705+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d81a56e0 con 0x7f35d8105710 2023-09-10T01:06:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.708+0000 7f35bffff700 1 -- 172.21.15.3:0/2741981580 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f35cc00c930 con 0x7f35d8105710 2023-09-10T01:06:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.710+0000 7f35f0c09b80 1 Processor -- start 2023-09-10T01:06:21.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.710+0000 7f35f0c09b80 1 -- start start 2023-09-10T01:06:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.710+0000 7f35bffff700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f35c00a07b0 0x7f35c00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.710+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c2fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c30180 0x556d41c3aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35bffff700 1 -- 172.21.15.3:0/2741981580 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f35cc014070 con 0x7f35d8105710 2023-09-10T01:06:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c3d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41b89730 con 0x556d41c2f860 2023-09-10T01:06:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41b87050 con 0x556d41c3b030 2023-09-10T01:06:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41b82940 con 0x556d41c30180 2023-09-10T01:06:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d759e700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f35c00a07b0 0x7f35c00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c30180 0x556d41c3aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c30180 0x556d41c3aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38954/0 (socket says 172.21.15.3:38954) 2023-09-10T01:06:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 -- 172.21.15.3:0/1243240267 learned_addr learned my addr 172.21.15.3:0/1243240267 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d759e700 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c3d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c2fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 -- 172.21.15.3:0/1243240267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 msgr2=0x556d41c3d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c3d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 -- 172.21.15.3:0/1243240267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 msgr2=0x556d41c2fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c2fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 -- 172.21.15.3:0/1243240267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d419aede0 con 0x556d41c30180 2023-09-10T01:06:21.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d759e700 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c3d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.711+0000 7f35d659c700 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c30180 0x556d41c3aaf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0a1130 tx=0x7f35cc099cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f83d89d73f794c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.712+0000 7f35d759e700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f35c00a07b0 0x7f35c00a2c70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009fd0 tx=0x7f35d0009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.712+0000 7f35bd7fa700 1 -- 172.21.15.3:0/1243240267 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc0a1660 con 0x556d41c30180 2023-09-10T01:06:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.712+0000 7f35bd7fa700 1 -- 172.21.15.3:0/1243240267 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35cc09d170 con 0x556d41c30180 2023-09-10T01:06:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.712+0000 7f35bd7fa700 1 -- 172.21.15.3:0/1243240267 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc005710 con 0x556d41c30180 2023-09-10T01:06:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.712+0000 7f35f0c09b80 1 -- 172.21.15.3:0/1243240267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c30180 msgr2=0x556d41c3aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.712+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c30180 0x556d41c3aaf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0a1130 tx=0x7f35cc099cf0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 -- 172.21.15.3:0/1243240267 shutdown_connections 2023-09-10T01:06:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c3d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c2fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/1243240267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c30180 0x556d41c3aaf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 -- 172.21.15.3:0/1243240267 >> 172.21.15.3:0/1243240267 conn(0x556d41ba5090 msgr2=0x556d41c2bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 -- 172.21.15.3:0/1243240267 shutdown_connections 2023-09-10T01:06:21.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 -- 172.21.15.3:0/1243240267 wait complete. 2023-09-10T01:06:21.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 -- start start 2023-09-10T01:06:21.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.713+0000 7f35f0c09b80 1 Processor -- start 2023-09-10T01:06:21.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c37d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c30180 0x556d41c382b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c387f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41b20b70 con 0x556d41c2f860 2023-09-10T01:06:21.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41b20870 con 0x556d41c30180 2023-09-10T01:06:21.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41a04920 con 0x556d41c3b030 2023-09-10T01:06:21.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35f0c09b80 10 client.0 _ll_register_callbacks cb 0x556d41a6aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:21.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35d759e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35d759e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c387f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38960/0 (socket says 172.21.15.3:38960) 2023-09-10T01:06:21.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35d6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c37d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35d759e700 1 -- 172.21.15.3:0/527332154 learned_addr learned my addr 172.21.15.3:0/527332154 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.714+0000 7f35d659c700 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c30180 0x556d41c382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35d759e700 1 -- 172.21.15.3:0/527332154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c30180 msgr2=0x556d41c382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35d759e700 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c30180 0x556d41c382b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35d759e700 1 -- 172.21.15.3:0/527332154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 msgr2=0x556d41c37d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35d759e700 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c37d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35d759e700 1 -- 172.21.15.3:0/527332154 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d81a56e0 con 0x556d41c3b030 2023-09-10T01:06:21.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c37d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:21.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35d759e700 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c387f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f35d00112e0 tx=0x7f35d000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88d6d543fd3dc990 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35baff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:21.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35baff5700 1 -- 172.21.15.3:0/527332154 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0021070 con 0x556d41c3b030 2023-09-10T01:06:21.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d41ada4f0 con 0x556d41c3b030 2023-09-10T01:06:21.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.715+0000 7f35baff5700 1 -- 172.21.15.3:0/527332154 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f35d0003e10 con 0x556d41c3b030 2023-09-10T01:06:21.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35baff5700 1 -- 172.21.15.3:0/527332154 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35d000e040 con 0x556d41c3b030 2023-09-10T01:06:21.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 10 client.26503 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:21.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556d41997c60 con 0x556d41c3b030 2023-09-10T01:06:21.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35baff5700 1 -- 172.21.15.3:0/527332154 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556d41ada4f0 con 0x556d41c3b030 2023-09-10T01:06:21.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35baff5700 1 -- 172.21.15.3:0/527332154 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f35d00078d0 con 0x556d41c3b030 2023-09-10T01:06:21.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 10 client.26503 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:21.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 10 client.26503 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:06:21.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 20 client.26503 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:21.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41c4ab90 0x556d41c4cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 4 client.26503 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:21.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.716+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d41b978a0 con 0x556d41c4ab90 2023-09-10T01:06:21.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.717+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41c4ab90 0x556d41c4cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.717+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41c4ab90 0x556d41c4cfb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b5acf99f87bf8e26 server_cookie=e50eae48b9723e3 in_seq=0 out_seq=0 2023-09-10T01:06:21.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.717+0000 7f35baff5700 10 client.26503 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:21.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35baff5700 1 -- 172.21.15.3:0/527332154 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556d41b978a0 con 0x556d41c4ab90 2023-09-10T01:06:21.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35baff5700 10 client.26503 handle_command_reply: tid=0 2023-09-10T01:06:21.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 client.26503 shutdown 2023-09-10T01:06:21.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 msgr2=0x556d41c387f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c387f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f35d00112e0 tx=0x7f35d000b040 comp rx=0 tx=0).stop 2023-09-10T01:06:21.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 shutdown_connections 2023-09-10T01:06:21.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41c4ab90 0x556d41c4cfb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c387f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c2f860 0x556d41c37d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/527332154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c30180 0x556d41c382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 >> 172.21.15.3:0/527332154 conn(0x556d41ba5090 msgr2=0x556d41c2b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35baff5700 0 client.26503 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:21.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 shutdown_connections 2023-09-10T01:06:21.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.718+0000 7f35f0c09b80 1 -- 172.21.15.3:0/527332154 wait complete. 2023-09-10T01:06:21.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.719+0000 7f35f0c09b80 20 client.26503 trim_cache size 0 max 16384 2023-09-10T01:06:21.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.733+0000 7f35f0c09b80 1 Processor -- start 2023-09-10T01:06:21.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.733+0000 7f35f0c09b80 1 -- start start 2023-09-10T01:06:21.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c2f860 0x556d41c32870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c38640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c35060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41c30a80 con 0x556d41c30180 2023-09-10T01:06:21.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41c307c0 con 0x556d41c2f860 2023-09-10T01:06:21.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41c46950 con 0x556d41c3b030 2023-09-10T01:06:21.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c35060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c2f860 0x556d41c32870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c35060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38970/0 (socket says 172.21.15.3:38970) 2023-09-10T01:06:21.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c2f860 0x556d41c32870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53828/0 (socket says 172.21.15.3:53828) 2023-09-10T01:06:21.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c38640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 -- 172.21.15.3:0/3469818690 learned_addr learned my addr 172.21.15.3:0/3469818690 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 -- 172.21.15.3:0/3469818690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c2f860 msgr2=0x556d41c32870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c2f860 0x556d41c32870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 -- 172.21.15.3:0/3469818690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 msgr2=0x556d41c38640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c38640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.734+0000 7f35d759e700 1 -- 172.21.15.3:0/3469818690 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d419d5250 con 0x556d41c3b030 2023-09-10T01:06:21.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.735+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c2f860 0x556d41c32870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:21.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.735+0000 7f35d759e700 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c35060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f35d0007c50 tx=0x7f35d00115a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c918d573566aeb01 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.735+0000 7f35bdffb700 1 -- 172.21.15.3:0/3469818690 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0021070 con 0x556d41c3b030 2023-09-10T01:06:21.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.735+0000 7f35bdffb700 1 -- 172.21.15.3:0/3469818690 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35d0022040 con 0x556d41c3b030 2023-09-10T01:06:21.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.735+0000 7f35bdffb700 1 -- 172.21.15.3:0/3469818690 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d000e070 con 0x556d41c3b030 2023-09-10T01:06:21.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 -- 172.21.15.3:0/3469818690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 msgr2=0x556d41c35060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c35060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f35d0007c50 tx=0x7f35d00115a0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 -- 172.21.15.3:0/3469818690 shutdown_connections 2023-09-10T01:06:21.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c3b030 0x556d41c35060 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c2f860 0x556d41c32870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/3469818690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c38640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 -- 172.21.15.3:0/3469818690 >> 172.21.15.3:0/3469818690 conn(0x556d41ba5090 msgr2=0x556d41c36bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 -- 172.21.15.3:0/3469818690 shutdown_connections 2023-09-10T01:06:21.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 -- 172.21.15.3:0/3469818690 wait complete. 2023-09-10T01:06:21.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.736+0000 7f35f0c09b80 1 -- start start 2023-09-10T01:06:21.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 1 Processor -- start 2023-09-10T01:06:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c2f860 0x556d41c40110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c40650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c40b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41c42160 con 0x556d41c30180 2023-09-10T01:06:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41c9e9b0 con 0x556d41c3b030 2023-09-10T01:06:21.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d41c9eb20 con 0x556d41c2f860 2023-09-10T01:06:21.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35f0c09b80 10 client.0 _ll_register_callbacks cb 0x7f35d0011eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:21.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35d759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c40b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35d659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c40650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.737+0000 7f35d6d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c2f860 0x556d41c40110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c40650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50580/0 (socket says 172.21.15.3:50580) 2023-09-10T01:06:21.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 -- 172.21.15.3:0/4137257957 learned_addr learned my addr 172.21.15.3:0/4137257957 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:21.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 -- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c2f860 msgr2=0x556d41c40110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c2f860 0x556d41c40110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 -- 172.21.15.3:0/4137257957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 msgr2=0x556d41c40b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:21.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c40b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 -- 172.21.15.3:0/4137257957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d810db00 con 0x556d41c30180 2023-09-10T01:06:21.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d759e700 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c40b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:21.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c2f860 0x556d41c40110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:21.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35d659c700 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c40650 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35cc095af0 tx=0x7f35cc0994f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699925ab648d5aa3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:21.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35baff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:06:21.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35baff5700 1 -- 172.21.15.3:0/4137257957 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc0a1660 con 0x556d41c30180 2023-09-10T01:06:21.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.738+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35cc09d170 con 0x556d41c30180 2023-09-10T01:06:21.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35baff5700 1 -- 172.21.15.3:0/4137257957 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f35cc005630 con 0x556d41c30180 2023-09-10T01:06:21.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35baff5700 1 -- 172.21.15.3:0/4137257957 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35cc0cc170 con 0x556d41c30180 2023-09-10T01:06:21.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 10 client.16530 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:21.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556d41997c60 con 0x556d41c30180 2023-09-10T01:06:21.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35baff5700 1 -- 172.21.15.3:0/4137257957 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35cc09d170 con 0x556d41c30180 2023-09-10T01:06:21.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35baff5700 1 -- 172.21.15.3:0/4137257957 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f35cc0840d0 con 0x556d41c30180 2023-09-10T01:06:21.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 10 client.16530 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:21.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 10 client.16530 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:06:21.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 20 client.16530 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:21.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41ca0100 0x556d41ca2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:21.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 4 client.16530 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:06:21.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.739+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d41b7b920 con 0x556d41ca0100 2023-09-10T01:06:21.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.740+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41ca0100 0x556d41ca2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:21.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.740+0000 7f35d6d9d700 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41ca0100 0x556d41ca2520 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e82dd2308fee0c7 server_cookie=ae76878e95a3dad6 in_seq=0 out_seq=0 2023-09-10T01:06:21.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.740+0000 7f35baff5700 10 client.16530 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:21.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35baff5700 1 -- 172.21.15.3:0/4137257957 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d41b7b920 con 0x556d41ca0100 2023-09-10T01:06:21.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35baff5700 10 client.16530 handle_command_reply: tid=0 2023-09-10T01:06:21.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 client.16530 shutdown 2023-09-10T01:06:21.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 msgr2=0x556d41c40650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c40650 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35cc095af0 tx=0x7f35cc0994f0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 shutdown_connections 2023-09-10T01:06:21.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556d41ca0100 0x556d41ca2520 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556d41c2f860 0x556d41c40110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d41c3b030 0x556d41c40b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 --2- 172.21.15.3:0/4137257957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556d41c30180 0x556d41c40650 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 >> 172.21.15.3:0/4137257957 conn(0x556d41ba5090 msgr2=0x556d41c38340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35baff5700 0 client.16530 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:21.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 shutdown_connections 2023-09-10T01:06:21.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 1 -- 172.21.15.3:0/4137257957 wait complete. 2023-09-10T01:06:21.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.741+0000 7f35f0c09b80 20 client.16530 trim_cache size 0 max 16384 2023-09-10T01:06:21.757 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:06:21.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f35c00a07b0 msgr2=0x7f35c00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f35c00a07b0 0x7f35c00a2c70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009fd0 tx=0x7f35d0009920 comp rx=0 tx=0).stop 2023-09-10T01:06:21.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 msgr2=0x7f35d8072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:21.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 0x7f35d8072e40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00e970 tx=0x7f35cc00ec80 comp rx=0 tx=0).stop 2023-09-10T01:06:21.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 shutdown_connections 2023-09-10T01:06:21.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f35c00a07b0 0x7f35c00a2c70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35d8073380 0x7f35d81a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35d81005f0 0x7f35d8072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 --2- 172.21.15.3:0/2741981580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8105710 0x7f35d8072e40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:21.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 >> 172.21.15.3:0/2741981580 conn(0x7f35d80fbe50 msgr2=0x7f35d8104020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:21.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 shutdown_connections 2023-09-10T01:06:21.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:21.742+0000 7f35dc9fe700 1 -- 172.21.15.3:0/2741981580 wait complete. 2023-09-10T01:06:21.761 INFO:tasks.workunit.client.0.smithi003.stdout: 1 224701 7.16 MB/sec execute 3 sec latency 155.267 ms 2023-09-10T01:06:21.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:06:22.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 -- 172.21.15.3:0/2083083888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd480732c0 msgr2=0x7fdd481110e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 --2- 172.21.15.3:0/2083083888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd480732c0 0x7fdd481110e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009990 tx=0x7fdd38009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 -- 172.21.15.3:0/2083083888 shutdown_connections 2023-09-10T01:06:22.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 --2- 172.21.15.3:0/2083083888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd48111620 0x7fdd48113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 --2- 172.21.15.3:0/2083083888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd480729a0 0x7fdd48072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 --2- 172.21.15.3:0/2083083888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd480732c0 0x7fdd481110e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 -- 172.21.15.3:0/2083083888 >> 172.21.15.3:0/2083083888 conn(0x7fdd480fc100 msgr2=0x7fdd480fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.020+0000 7fdd507ff700 1 -- 172.21.15.3:0/2083083888 shutdown_connections 2023-09-10T01:06:22.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.021+0000 7fdd507ff700 1 -- 172.21.15.3:0/2083083888 wait complete. 2023-09-10T01:06:22.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.021+0000 7fdd507ff700 1 Processor -- start 2023-09-10T01:06:22.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd507ff700 1 -- start start 2023-09-10T01:06:22.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd507ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd480729a0 0x7fdd4819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd507ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd480732c0 0x7fdd4819d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd507ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd48111620 0x7fdd481a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd507ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd481a1db0 con 0x7fdd480729a0 2023-09-10T01:06:22.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd507ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd481a1f20 con 0x7fdd480732c0 2023-09-10T01:06:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd507ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd481a20e0 con 0x7fdd48111620 2023-09-10T01:06:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd4dd9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd480732c0 0x7fdd4819d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd4dd9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd480732c0 0x7fdd4819d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53866/0 (socket says 172.21.15.3:53866) 2023-09-10T01:06:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd4e59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd480729a0 0x7fdd4819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd4ed9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd48111620 0x7fdd481a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.022+0000 7fdd4dd9a700 1 -- 172.21.15.3:0/3670605339 learned_addr learned my addr 172.21.15.3:0/3670605339 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3670605339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd480732c0 msgr2=0x7fdd4819d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd480732c0 0x7fdd4819d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd480729a0 msgr2=0x7fdd4819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd480729a0 0x7fdd4819cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3670605339 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd380095f0 con 0x7fdd48111620 2023-09-10T01:06:22.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd480732c0 0x7fdd4819d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:22.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd48111620 0x7fdd481a1770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000ef90 tx=0x7fdd4000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd3f7fe700 1 -- 172.21.15.3:0/3670605339 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40018070 con 0x7fdd48111620 2023-09-10T01:06:22.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd481a2360 con 0x7fdd48111620 2023-09-10T01:06:22.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd3f7fe700 1 -- 172.21.15.3:0/3670605339 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd4000f040 con 0x7fdd48111620 2023-09-10T01:06:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd3f7fe700 1 -- 172.21.15.3:0/3670605339 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000cc30 con 0x7fdd48111620 2023-09-10T01:06:22.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.023+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd481a9c00 con 0x7fdd48111620 2023-09-10T01:06:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.027+0000 7fdd3f7fe700 1 -- 172.21.15.3:0/3670605339 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fdd40003a00 con 0x7fdd48111620 2023-09-10T01:06:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.028+0000 7fdd640dab80 1 Processor -- start 2023-09-10T01:06:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.028+0000 7fdd640dab80 1 -- start start 2023-09-10T01:06:22.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd3f7fe700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd340a07b0 0x7fdd340a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bd030 0x55a5a24c6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd640dab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a2415730 con 0x55a5a24bd030 2023-09-10T01:06:22.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd640dab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a2413050 con 0x55a5a24c6e40 2023-09-10T01:06:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd640dab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a240e940 con 0x55a5a24bc710 2023-09-10T01:06:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd3f7fe700 1 -- 172.21.15.3:0/3670605339 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdd400cbe50 con 0x7fdd48111620 2023-09-10T01:06:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4e59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4e59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24bcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39004/0 (socket says 172.21.15.3:39004) 2023-09-10T01:06:22.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4e59b700 1 -- 172.21.15.3:0/3656666598 learned_addr learned my addr 172.21.15.3:0/3656666598 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd340a07b0 0x7fdd340a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bd030 0x55a5a24c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3656666598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 msgr2=0x55a5a24bcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24bcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.029+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3656666598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bd030 msgr2=0x55a5a24c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.030+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bd030 0x55a5a24c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.030+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3656666598 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5a223ade0 con 0x55a5a24c6e40 2023-09-10T01:06:22.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.030+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bd030 0x55a5a24c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:06:22.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.030+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c9230 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdd400c91b0 tx=0x7fdd40098900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f022688305c4e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.030+0000 7fdd3cff9700 1 -- 172.21.15.3:0/3656666598 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd400996a0 con 0x55a5a24c6e40 2023-09-10T01:06:22.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.030+0000 7fdd3cff9700 1 -- 172.21.15.3:0/3656666598 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd40098fa0 con 0x55a5a24c6e40 2023-09-10T01:06:22.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.030+0000 7fdd3cff9700 1 -- 172.21.15.3:0/3656666598 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40097340 con 0x55a5a24c6e40 2023-09-10T01:06:22.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 -- 172.21.15.3:0/3656666598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 msgr2=0x55a5a24c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd340a07b0 0x7fdd340a2c70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800b6f0 tx=0x7fdd3801a040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c9230 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdd400c91b0 tx=0x7fdd40098900 comp rx=0 tx=0).stop 2023-09-10T01:06:22.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 -- 172.21.15.3:0/3656666598 shutdown_connections 2023-09-10T01:06:22.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24bcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c9230 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3656666598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bd030 0x55a5a24c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 -- 172.21.15.3:0/3656666598 >> 172.21.15.3:0/3656666598 conn(0x55a5a2431090 msgr2=0x55a5a24b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 -- 172.21.15.3:0/3656666598 shutdown_connections 2023-09-10T01:06:22.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 -- 172.21.15.3:0/3656666598 wait complete. 2023-09-10T01:06:22.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.031+0000 7fdd640dab80 1 -- start start 2023-09-10T01:06:22.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 1 Processor -- start 2023-09-10T01:06:22.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bd030 0x55a5a24c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a23acb70 con 0x55a5a24bc710 2023-09-10T01:06:22.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a23ac870 con 0x55a5a24c6e40 2023-09-10T01:06:22.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a2290920 con 0x55a5a24bd030 2023-09-10T01:06:22.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd640dab80 10 client.0 _ll_register_callbacks cb 0x55a5a22f6a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:22.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.032+0000 7fdd4e59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53884/0 (socket says 172.21.15.3:53884) 2023-09-10T01:06:22.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3596584316 learned_addr learned my addr 172.21.15.3:0/3596584316 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4e59b700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50608/0 (socket says 172.21.15.3:50608) 2023-09-10T01:06:22.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bd030 0x55a5a24c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3596584316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bd030 msgr2=0x55a5a24c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bd030 0x55a5a24c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3596584316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 msgr2=0x55a5a24c68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24c68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3596584316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd481a9c00 con 0x55a5a24c6e40 2023-09-10T01:06:22.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bd030 0x55a5a24c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:22.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c4260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000c6f0 tx=0x7fdd4001b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78640e020018d650 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd31ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:22.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd31ffb700 1 -- 172.21.15.3:0/3596584316 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd400a0f80 con 0x55a5a24c6e40 2023-09-10T01:06:22.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5a23664f0 con 0x55a5a24c6e40 2023-09-10T01:06:22.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.033+0000 7fdd31ffb700 1 -- 172.21.15.3:0/3596584316 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdd400b1120 con 0x55a5a24c6e40 2023-09-10T01:06:22.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd31ffb700 1 -- 172.21.15.3:0/3596584316 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd400b12e0 con 0x55a5a24c6e40 2023-09-10T01:06:22.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 10 client.26603 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:22.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a5a2223c60 con 0x55a5a24c6e40 2023-09-10T01:06:22.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd31ffb700 1 -- 172.21.15.3:0/3596584316 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a5a23664f0 con 0x55a5a24c6e40 2023-09-10T01:06:22.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd31ffb700 1 -- 172.21.15.3:0/3596584316 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fdd40093bb0 con 0x55a5a24c6e40 2023-09-10T01:06:22.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 10 client.26603 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:22.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 10 client.26603 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:06:22.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 20 client.26603 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:22.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a24d6500 0x55a5a24d88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 4 client.26603 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:22.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5a24238a0 con 0x55a5a24d6500 2023-09-10T01:06:22.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.034+0000 7fdd4e59b700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a24d6500 0x55a5a24d88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.035+0000 7fdd4e59b700 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a24d6500 0x55a5a24d88f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b4ceb22c091cf5 server_cookie=5f0ca5fcf299da3a in_seq=0 out_seq=0 2023-09-10T01:06:22.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.035+0000 7fdd31ffb700 10 client.26603 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:22.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd31ffb700 1 -- 172.21.15.3:0/3596584316 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a5a24238a0 con 0x55a5a24d6500 2023-09-10T01:06:22.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd31ffb700 10 client.26603 handle_command_reply: tid=0 2023-09-10T01:06:22.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 client.26603 shutdown 2023-09-10T01:06:22.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 msgr2=0x55a5a24c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c4260 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000c6f0 tx=0x7fdd4001b460 comp rx=0 tx=0).stop 2023-09-10T01:06:22.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 shutdown_connections 2023-09-10T01:06:22.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24c6e40 0x55a5a24c4260 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24c68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a24d6500 0x55a5a24d88f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3596584316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bd030 0x55a5a24c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 >> 172.21.15.3:0/3596584316 conn(0x55a5a2431090 msgr2=0x55a5a24b7780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd31ffb700 0 client.26603 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:22.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 shutdown_connections 2023-09-10T01:06:22.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.036+0000 7fdd640dab80 1 -- 172.21.15.3:0/3596584316 wait complete. 2023-09-10T01:06:22.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.037+0000 7fdd640dab80 20 client.26603 trim_cache size 0 max 16384 2023-09-10T01:06:22.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.051+0000 7fdd640dab80 1 Processor -- start 2023-09-10T01:06:22.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd640dab80 1 -- start start 2023-09-10T01:06:22.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24d2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24bee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24c6e40 0x55a5a24bf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd640dab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a24bd9c0 con 0x55a5a24bc710 2023-09-10T01:06:22.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd640dab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a24d8f50 con 0x55a5a24bd030 2023-09-10T01:06:22.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd640dab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a24dd5f0 con 0x55a5a24c6e40 2023-09-10T01:06:22.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd4dd9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24bee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd4ed9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24c6e40 0x55a5a24bf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd4e59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24d2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd4dd9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24bee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53892/0 (socket says 172.21.15.3:53892) 2023-09-10T01:06:22.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.052+0000 7fdd4dd9a700 1 -- 172.21.15.3:0/3643238057 learned_addr learned my addr 172.21.15.3:0/3643238057 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3643238057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 msgr2=0x55a5a24bee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3643238057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24bee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3643238057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 msgr2=0x55a5a24d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3643238057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24d2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/3643238057 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5a2261250 con 0x55a5a24c6e40 2023-09-10T01:06:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/3643238057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24c6e40 0x55a5a24bf380 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd40014f30 tx=0x7fdd400952e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e921e0b3b9d033f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd3d7fa700 1 -- 172.21.15.3:0/3643238057 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd400a0f80 con 0x55a5a24c6e40 2023-09-10T01:06:22.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd3d7fa700 1 -- 172.21.15.3:0/3643238057 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd400a0af0 con 0x55a5a24c6e40 2023-09-10T01:06:22.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.053+0000 7fdd3d7fa700 1 -- 172.21.15.3:0/3643238057 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40093cb0 con 0x55a5a24c6e40 2023-09-10T01:06:22.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 -- 172.21.15.3:0/3643238057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24c6e40 msgr2=0x55a5a24bf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3643238057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24c6e40 0x55a5a24bf380 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd40014f30 tx=0x7fdd400952e0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 -- 172.21.15.3:0/3643238057 shutdown_connections 2023-09-10T01:06:22.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3643238057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24c6e40 0x55a5a24bf380 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3643238057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24bc710 0x55a5a24d2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 --2- 172.21.15.3:0/3643238057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24bee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 -- 172.21.15.3:0/3643238057 >> 172.21.15.3:0/3643238057 conn(0x55a5a2431090 msgr2=0x55a5a24d2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 -- 172.21.15.3:0/3643238057 shutdown_connections 2023-09-10T01:06:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 -- 172.21.15.3:0/3643238057 wait complete. 2023-09-10T01:06:22.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.054+0000 7fdd640dab80 1 -- start start 2023-09-10T01:06:22.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 1 Processor -- start 2023-09-10T01:06:22.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24c6e40 0x55a5a24cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a24cbfa0 con 0x55a5a24c6e40 2023-09-10T01:06:22.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a24cc110 con 0x55a5a24bd030 2023-09-10T01:06:22.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5a24cc2d0 con 0x55a5a24bc710 2023-09-10T01:06:22.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd640dab80 10 client.0 _ll_register_callbacks cb 0x7fdd44009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:22.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd4ed9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24c6e40 0x55a5a24cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd4e59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd4ed9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24c6e40 0x55a5a24cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50616/0 (socket says 172.21.15.3:50616) 2023-09-10T01:06:22.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/1868891757 learned_addr learned my addr 172.21.15.3:0/1868891757 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.055+0000 7fdd4dd9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/1868891757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 msgr2=0x55a5a24cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 msgr2=0x55a5a24cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd4ed9c700 1 -- 172.21.15.3:0/1868891757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5a2223c60 con 0x55a5a24c6e40 2023-09-10T01:06:22.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd4dd9a700 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:22.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd4ed9c700 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24c6e40 0x55a5a24cb9d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009510 tx=0x7fdd40098d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e382660f7e128e0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd31ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:06:22.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd31ffb700 1 -- 172.21.15.3:0/1868891757 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd400b1320 con 0x55a5a24c6e40 2023-09-10T01:06:22.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd31ffb700 1 -- 172.21.15.3:0/1868891757 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdd400a0f80 con 0x55a5a24c6e40 2023-09-10T01:06:22.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd40098fa0 con 0x55a5a24c6e40 2023-09-10T01:06:22.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd31ffb700 1 -- 172.21.15.3:0/1868891757 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd40094f00 con 0x55a5a24c6e40 2023-09-10T01:06:22.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.056+0000 7fdd640dab80 10 client.16545 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:22.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a5a22392b0 con 0x55a5a24c6e40 2023-09-10T01:06:22.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd31ffb700 1 -- 172.21.15.3:0/1868891757 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd40098fa0 con 0x55a5a24c6e40 2023-09-10T01:06:22.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd31ffb700 1 -- 172.21.15.3:0/1868891757 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fdd40097f00 con 0x55a5a24c6e40 2023-09-10T01:06:22.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd640dab80 10 client.16545 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:22.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd640dab80 10 client.16545 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:06:22.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd640dab80 20 client.16545 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:22.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd640dab80 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a252b000 0x55a5a252d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd640dab80 4 client.16545 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:06:22.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a5a2407920 con 0x55a5a252b000 2023-09-10T01:06:22.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.057+0000 7fdd4e59b700 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a252b000 0x55a5a252d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd31ffb700 10 client.16545 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:22.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd4e59b700 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a252b000 0x55a5a252d3f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e94b06e6d45f9a6 server_cookie=625aea1374110f9f in_seq=0 out_seq=0 2023-09-10T01:06:22.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd31ffb700 1 -- 172.21.15.3:0/1868891757 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a5a2407920 con 0x55a5a252b000 2023-09-10T01:06:22.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd31ffb700 10 client.16545 handle_command_reply: tid=0 2023-09-10T01:06:22.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 client.16545 shutdown 2023-09-10T01:06:22.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24c6e40 msgr2=0x55a5a24cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24c6e40 0x55a5a24cb9d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009510 tx=0x7fdd40098d70 comp rx=0 tx=0).stop 2023-09-10T01:06:22.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 shutdown_connections 2023-09-10T01:06:22.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55a5a252b000 0x55a5a252d3f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a5a24bc710 0x55a5a24cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5a24c6e40 0x55a5a24cb9d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 --2- 172.21.15.3:0/1868891757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a24bd030 0x55a5a24cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd31ffb700 0 client.16545 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:22.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 >> 172.21.15.3:0/1868891757 conn(0x55a5a2431090 msgr2=0x55a5a24d2140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 shutdown_connections 2023-09-10T01:06:22.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.058+0000 7fdd640dab80 1 -- 172.21.15.3:0/1868891757 wait complete. 2023-09-10T01:06:22.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd640dab80 20 client.16545 trim_cache size 0 max 16384 2023-09-10T01:06:22.077 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:06:22.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd340a07b0 msgr2=0x7fdd340a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd340a07b0 0x7fdd340a2c70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800b6f0 tx=0x7fdd3801a040 comp rx=0 tx=0).stop 2023-09-10T01:06:22.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd48111620 msgr2=0x7fdd481a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd48111620 0x7fdd481a1770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000ef90 tx=0x7fdd4000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 shutdown_connections 2023-09-10T01:06:22.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdd340a07b0 0x7fdd340a2c70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd48111620 0x7fdd481a1770 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd480729a0 0x7fdd4819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 --2- 172.21.15.3:0/3670605339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd480732c0 0x7fdd4819d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 >> 172.21.15.3:0/3670605339 conn(0x7fdd480fc100 msgr2=0x7fdd480fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.059+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 shutdown_connections 2023-09-10T01:06:22.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.060+0000 7fdd507ff700 1 -- 172.21.15.3:0/3670605339 wait complete. 2023-09-10T01:06:22.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:06:22.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 -- 172.21.15.3:0/2529974741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3141066b0 msgr2=0x7fe314106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 --2- 172.21.15.3:0/2529974741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3141066b0 0x7fe314106a90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe304009990 tx=0x7fe304009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 -- 172.21.15.3:0/2529974741 shutdown_connections 2023-09-10T01:06:22.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 --2- 172.21.15.3:0/2529974741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe314100fd0 0x7fe31410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 --2- 172.21.15.3:0/2529974741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3141066b0 0x7fe314106a90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 --2- 172.21.15.3:0/2529974741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe314100610 0x7fe314100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 -- 172.21.15.3:0/2529974741 >> 172.21.15.3:0/2529974741 conn(0x7fe3140fc100 msgr2=0x7fe3140fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.376+0000 7fe318dff700 1 -- 172.21.15.3:0/2529974741 shutdown_connections 2023-09-10T01:06:22.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.377+0000 7fe318dff700 1 -- 172.21.15.3:0/2529974741 wait complete. 2023-09-10T01:06:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 Processor -- start 2023-09-10T01:06:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 -- start start 2023-09-10T01:06:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe314100610 0x7fe3141989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe314100fd0 0x7fe314198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3141066b0 0x7fe31419d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419d950 con 0x7fe3141066b0 2023-09-10T01:06:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419dac0 con 0x7fe314100610 2023-09-10T01:06:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe318dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419dc80 con 0x7fe314100fd0 2023-09-10T01:06:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe31259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe314100610 0x7fe3141989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe31259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe314100610 0x7fe3141989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53918/0 (socket says 172.21.15.3:53918) 2023-09-10T01:06:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.378+0000 7fe31259c700 1 -- 172.21.15.3:0/14908262 learned_addr learned my addr 172.21.15.3:0/14908262 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe311d9b700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe314100fd0 0x7fe314198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe312d9d700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3141066b0 0x7fe31419d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe311d9b700 1 -- 172.21.15.3:0/14908262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe314100610 msgr2=0x7fe3141989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe311d9b700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe314100610 0x7fe3141989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe311d9b700 1 -- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3141066b0 msgr2=0x7fe31419d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe311d9b700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3141066b0 0x7fe31419d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe311d9b700 1 -- 172.21.15.3:0/14908262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe308009520 con 0x7fe314100fd0 2023-09-10T01:06:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe31259c700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe314100610 0x7fe3141989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:22.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe311d9b700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe314100fd0 0x7fe314198ef0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9f0 tx=0x7fe30800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe3037fe700 1 -- 172.21.15.3:0/14908262 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30800cb70 con 0x7fe314100fd0 2023-09-10T01:06:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c009c40 con 0x7fe314100fd0 2023-09-10T01:06:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe3037fe700 1 -- 172.21.15.3:0/14908262 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe30800ee80 con 0x7fe314100fd0 2023-09-10T01:06:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.379+0000 7fe3037fe700 1 -- 172.21.15.3:0/14908262 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308003d30 con 0x7fe314100fd0 2023-09-10T01:06:22.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.380+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3040095f0 con 0x7fe314100fd0 2023-09-10T01:06:22.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.383+0000 7fe3037fe700 1 -- 172.21.15.3:0/14908262 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe30800ccd0 con 0x7fe314100fd0 2023-09-10T01:06:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.384+0000 7fe32c631b80 1 Processor -- start 2023-09-10T01:06:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.384+0000 7fe32c631b80 1 -- start start 2023-09-10T01:06:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.384+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e68af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.384+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e72900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e75230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe32c631b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798dc1730 con 0x558798e69030 2023-09-10T01:06:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe32c631b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798dbf050 con 0x558798e72e40 2023-09-10T01:06:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe32c631b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798dba940 con 0x558798e68710 2023-09-10T01:06:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe3037fe700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2fc0a07b0 0x7fe2fc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e75230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e75230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53920/0 (socket says 172.21.15.3:53920) 2023-09-10T01:06:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 -- 172.21.15.3:0/211087469 learned_addr learned my addr 172.21.15.3:0/211087469 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe31259c700 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e72900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe312d9d700 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e68af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe3037fe700 1 -- 172.21.15.3:0/14908262 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe308014070 con 0x7fe314100fd0 2023-09-10T01:06:22.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 -- 172.21.15.3:0/211087469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 msgr2=0x558798e68af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e68af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe312d9d700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2fc0a07b0 0x7fe2fc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 -- 172.21.15.3:0/211087469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 msgr2=0x558798e72900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e72900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.385+0000 7fe311d9b700 1 -- 172.21.15.3:0/211087469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558798be6de0 con 0x558798e72e40 2023-09-10T01:06:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.386+0000 7fe311d9b700 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e75230 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe3080c98d0 tx=0x7fe308095400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc28f06396dc1b3a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.386+0000 7fe300ff9700 1 -- 172.21.15.3:0/211087469 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308099dc0 con 0x558798e72e40 2023-09-10T01:06:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.386+0000 7fe300ff9700 1 -- 172.21.15.3:0/211087469 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3080996c0 con 0x558798e72e40 2023-09-10T01:06:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.386+0000 7fe300ff9700 1 -- 172.21.15.3:0/211087469 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308099920 con 0x558798e72e40 2023-09-10T01:06:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.386+0000 7fe32c631b80 1 -- 172.21.15.3:0/211087469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 msgr2=0x558798e75230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.386+0000 7fe312d9d700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2fc0a07b0 0x7fe2fc0a2c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009550 tx=0x7fe30c00b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.386+0000 7fe32c631b80 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e75230 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe3080c98d0 tx=0x7fe308095400 comp rx=0 tx=0).stop 2023-09-10T01:06:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 -- 172.21.15.3:0/211087469 shutdown_connections 2023-09-10T01:06:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e68af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e75230 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 --2- 172.21.15.3:0/211087469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e72900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 -- 172.21.15.3:0/211087469 >> 172.21.15.3:0/211087469 conn(0x558798ddd090 msgr2=0x558798e63bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 -- 172.21.15.3:0/211087469 shutdown_connections 2023-09-10T01:06:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 -- 172.21.15.3:0/211087469 wait complete. 2023-09-10T01:06:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.387+0000 7fe32c631b80 1 -- start start 2023-09-10T01:06:22.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 1 Processor -- start 2023-09-10T01:06:22.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e6fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e70220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798d58b70 con 0x558798e69030 2023-09-10T01:06:22.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798d58870 con 0x558798e72e40 2023-09-10T01:06:22.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798c3c920 con 0x558798e68710 2023-09-10T01:06:22.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe32c631b80 10 client.0 _ll_register_callbacks cb 0x558798ca2a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:22.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe312d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e70220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe312d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e70220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53932/0 (socket says 172.21.15.3:53932) 2023-09-10T01:06:22.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe312d9d700 1 -- 172.21.15.3:0/762812461 learned_addr learned my addr 172.21.15.3:0/762812461 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.388+0000 7fe311d9b700 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e6fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe31259c700 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe312d9d700 1 -- 172.21.15.3:0/762812461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 msgr2=0x558798e728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe312d9d700 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe312d9d700 1 -- 172.21.15.3:0/762812461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 msgr2=0x558798e6fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe312d9d700 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e6fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe312d9d700 1 -- 172.21.15.3:0/762812461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3040095f0 con 0x558798e72e40 2023-09-10T01:06:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe312d9d700 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e70220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe30c018ed0 tx=0x7fe30c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a5523ea4e1210a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.389+0000 7fe2f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/762812461 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c00e070 con 0x558798e72e40 2023-09-10T01:06:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558798d124f0 con 0x558798e72e40 2023-09-10T01:06:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/762812461 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe30c01f050 con 0x558798e72e40 2023-09-10T01:06:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/762812461 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe30c01adb0 con 0x558798e72e40 2023-09-10T01:06:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe32c631b80 10 client.26621 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558798bcfc60 con 0x558798e72e40 2023-09-10T01:06:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/762812461 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558798d124f0 con 0x558798e72e40 2023-09-10T01:06:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/762812461 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe30c0193d0 con 0x558798e72e40 2023-09-10T01:06:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe32c631b80 10 client.26621 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe32c631b80 10 client.26621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe32c631b80 20 client.26621 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.390+0000 7fe32c631b80 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798e82a30 0x558798e84e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.391+0000 7fe32c631b80 4 client.26621 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.391+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558798dcf8a0 con 0x558798e82a30 2023-09-10T01:06:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.391+0000 7fe31259c700 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798e82a30 0x558798e84e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.391+0000 7fe2f9ffb700 10 client.26621 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:22.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.391+0000 7fe31259c700 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798e82a30 0x558798e84e50 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217d031ca6bc69b server_cookie=829afeb891285c58 in_seq=0 out_seq=0 2023-09-10T01:06:22.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/762812461 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558798dcf8a0 con 0x558798e82a30 2023-09-10T01:06:22.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe2f9ffb700 10 client.26621 handle_command_reply: tid=0 2023-09-10T01:06:22.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 client.26621 shutdown 2023-09-10T01:06:22.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 msgr2=0x558798e70220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e70220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe30c018ed0 tx=0x7fe30c004ad0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 shutdown_connections 2023-09-10T01:06:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798e82a30 0x558798e84e50 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e68710 0x558798e728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e72e40 0x558798e70220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 --2- 172.21.15.3:0/762812461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e6fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe2f9ffb700 0 client.26621 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:22.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 >> 172.21.15.3:0/762812461 conn(0x558798ddd090 msgr2=0x558798e63780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 shutdown_connections 2023-09-10T01:06:22.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.392+0000 7fe32c631b80 1 -- 172.21.15.3:0/762812461 wait complete. 2023-09-10T01:06:22.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.393+0000 7fe32c631b80 20 client.26621 trim_cache size 0 max 16384 2023-09-10T01:06:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.406+0000 7fe32c631b80 1 Processor -- start 2023-09-10T01:06:22.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe32c631b80 1 -- start start 2023-09-10T01:06:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e68710 0x558798e799f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e69030 0x558798e62320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e7e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe32c631b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798e603d0 con 0x558798e68710 2023-09-10T01:06:22.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe32c631b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798e651e0 con 0x558798e69030 2023-09-10T01:06:22.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe32c631b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798e79f30 con 0x558798e72e40 2023-09-10T01:06:22.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe311d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e69030 0x558798e62320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe311d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e69030 0x558798e62320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53942/0 (socket says 172.21.15.3:53942) 2023-09-10T01:06:22.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe311d9b700 1 -- 172.21.15.3:0/1171284898 learned_addr learned my addr 172.21.15.3:0/1171284898 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe31259c700 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e68710 0x558798e799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.407+0000 7fe312d9d700 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e7e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe311d9b700 1 -- 172.21.15.3:0/1171284898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 msgr2=0x558798e7e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe311d9b700 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e7e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe311d9b700 1 -- 172.21.15.3:0/1171284898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e68710 msgr2=0x558798e799f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe311d9b700 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e68710 0x558798e799f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe311d9b700 1 -- 172.21.15.3:0/1171284898 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558798c0d250 con 0x558798e69030 2023-09-10T01:06:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe312d9d700 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e7e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe311d9b700 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e69030 0x558798e62320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe308096130 tx=0x7fe308099280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3212fb042ff051d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe3017fa700 1 -- 172.21.15.3:0/1171284898 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3080a1730 con 0x558798e69030 2023-09-10T01:06:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe3017fa700 1 -- 172.21.15.3:0/1171284898 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe30809d4f0 con 0x558798e69030 2023-09-10T01:06:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.408+0000 7fe3017fa700 1 -- 172.21.15.3:0/1171284898 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30809d2d0 con 0x558798e69030 2023-09-10T01:06:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 -- 172.21.15.3:0/1171284898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e69030 msgr2=0x558798e62320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e69030 0x558798e62320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe308096130 tx=0x7fe308099280 comp rx=0 tx=0).stop 2023-09-10T01:06:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 -- 172.21.15.3:0/1171284898 shutdown_connections 2023-09-10T01:06:22.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e7e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e68710 0x558798e799f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 --2- 172.21.15.3:0/1171284898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e69030 0x558798e62320 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 -- 172.21.15.3:0/1171284898 >> 172.21.15.3:0/1171284898 conn(0x558798ddd090 msgr2=0x558798e897b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 -- 172.21.15.3:0/1171284898 shutdown_connections 2023-09-10T01:06:22.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 -- 172.21.15.3:0/1171284898 wait complete. 2023-09-10T01:06:22.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.409+0000 7fe32c631b80 1 -- start start 2023-09-10T01:06:22.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.410+0000 7fe32c631b80 1 Processor -- start 2023-09-10T01:06:22.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.410+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e68710 0x558798e795e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e77910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe32c631b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e77e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe32c631b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798e78390 con 0x558798e69030 2023-09-10T01:06:22.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe32c631b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798e78500 con 0x558798e68710 2023-09-10T01:06:22.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe32c631b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558798e786c0 con 0x558798e72e40 2023-09-10T01:06:22.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe32c631b80 10 client.0 _ll_register_callbacks cb 0x7fe30c008f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e68710 0x558798e795e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e68710 0x558798e795e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53950/0 (socket says 172.21.15.3:53950) 2023-09-10T01:06:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 -- 172.21.15.3:0/4155421086 learned_addr learned my addr 172.21.15.3:0/4155421086 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe312d9d700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e77e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe311d9b700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e77910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 -- 172.21.15.3:0/4155421086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 msgr2=0x558798e77e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e77e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 -- 172.21.15.3:0/4155421086 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 msgr2=0x558798e77910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:22.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e77910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe31259c700 1 -- 172.21.15.3:0/4155421086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558798be6de0 con 0x558798e68710 2023-09-10T01:06:22.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.411+0000 7fe312d9d700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e77e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:22.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe31259c700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e68710 0x558798e795e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe304005b40 tx=0x7fe304005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29909d5b0453bb3d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:22.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe2f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:22.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/4155421086 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30401d070 con 0x558798e68710 2023-09-10T01:06:22.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/4155421086 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe30400f6f0 con 0x558798e68710 2023-09-10T01:06:22.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3080996c0 con 0x558798e68710 2023-09-10T01:06:22.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/4155421086 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe30402b070 con 0x558798e68710 2023-09-10T01:06:22.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe32c631b80 10 client.26627 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:22.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558798be52b0 con 0x558798e68710 2023-09-10T01:06:22.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.412+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/4155421086 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3080996c0 con 0x558798e68710 2023-09-10T01:06:22.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/4155421086 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fe3040259e0 con 0x558798e68710 2023-09-10T01:06:22.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe32c631b80 10 client.26627 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:22.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe32c631b80 10 client.26627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:22.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe32c631b80 20 client.26627 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:22.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe32c631b80 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798ed7d70 0x558798eda190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:22.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe32c631b80 4 client.26627 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:06:22.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558798db3920 con 0x558798ed7d70 2023-09-10T01:06:22.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe311d9b700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798ed7d70 0x558798eda190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:22.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe311d9b700 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798ed7d70 0x558798eda190 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=751f96780522a712 server_cookie=539be65cf1f38a3e in_seq=0 out_seq=0 2023-09-10T01:06:22.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.413+0000 7fe2f9ffb700 10 client.26627 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:22.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe2f9ffb700 1 -- 172.21.15.3:0/4155421086 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558798db3920 con 0x558798ed7d70 2023-09-10T01:06:22.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe2f9ffb700 10 client.26627 handle_command_reply: tid=0 2023-09-10T01:06:22.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 client.26627 shutdown 2023-09-10T01:06:22.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e68710 msgr2=0x558798e795e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e68710 0x558798e795e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe304005b40 tx=0x7fe304005fb0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 shutdown_connections 2023-09-10T01:06:22.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558798ed7d70 0x558798eda190 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558798e72e40 0x558798e77e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558798e68710 0x558798e795e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 --2- 172.21.15.3:0/4155421086 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558798e69030 0x558798e77910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 >> 172.21.15.3:0/4155421086 conn(0x558798ddd090 msgr2=0x558798e62320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe2f9ffb700 0 client.26627 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:22.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 shutdown_connections 2023-09-10T01:06:22.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 1 -- 172.21.15.3:0/4155421086 wait complete. 2023-09-10T01:06:22.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.414+0000 7fe32c631b80 20 client.26627 trim_cache size 0 max 16384 2023-09-10T01:06:22.433 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:06:22.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2fc0a07b0 msgr2=0x7fe2fc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2fc0a07b0 0x7fe2fc0a2c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009550 tx=0x7fe30c00b040 comp rx=0 tx=0).stop 2023-09-10T01:06:22.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe314100fd0 msgr2=0x7fe314198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:22.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe314100fd0 0x7fe314198ef0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9f0 tx=0x7fe30800ed00 comp rx=0 tx=0).stop 2023-09-10T01:06:22.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 shutdown_connections 2023-09-10T01:06:22.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2fc0a07b0 0x7fe2fc0a2c70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3141066b0 0x7fe31419d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe314100610 0x7fe3141989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 --2- 172.21.15.3:0/14908262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe314100fd0 0x7fe314198ef0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:22.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 >> 172.21.15.3:0/14908262 conn(0x7fe3140fc100 msgr2=0x7fe314072830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:22.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 shutdown_connections 2023-09-10T01:06:22.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:22.415+0000 7fe318dff700 1 -- 172.21.15.3:0/14908262 wait complete. 2023-09-10T01:06:22.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:22 smithi003 ceph-mon[97654]: pgmap v97: 129 pgs: 129 active+clean; 70 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 294 op/s 2023-09-10T01:06:22.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:22 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/58611088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:06:22.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:22 smithi136 ceph-mon[133708]: pgmap v97: 129 pgs: 129 active+clean; 70 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 294 op/s 2023-09-10T01:06:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:22 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/58611088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:06:22.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:22 smithi092 ceph-mon[134067]: pgmap v97: 129 pgs: 129 active+clean; 70 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 294 op/s 2023-09-10T01:06:22.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:22 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/58611088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:06:22.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 227040 9.21 MB/sec execute 4 sec latency 36.925 ms 2023-09-10T01:06:23.438 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:06:23.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:06:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 -- 172.21.15.3:0/2455514459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 msgr2=0x7ffa88101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 --2- 172.21.15.3:0/2455514459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 0x7ffa88101b60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009990 tx=0x7ffa7c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 -- 172.21.15.3:0/2455514459 shutdown_connections 2023-09-10T01:06:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 --2- 172.21.15.3:0/2455514459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 0x7ffa88101b60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 --2- 172.21.15.3:0/2455514459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa8810acd0 0x7ffa8810d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 --2- 172.21.15.3:0/2455514459 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa88102130 0x7ffa8810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 -- 172.21.15.3:0/2455514459 >> 172.21.15.3:0/2455514459 conn(0x7ffa880fb010 msgr2=0x7ffa880fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 -- 172.21.15.3:0/2455514459 shutdown_connections 2023-09-10T01:06:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.685+0000 7ffa8edff700 1 -- 172.21.15.3:0/2455514459 wait complete. 2023-09-10T01:06:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.687+0000 7ffa8edff700 1 Processor -- start 2023-09-10T01:06:23.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.687+0000 7ffa8edff700 1 -- start start 2023-09-10T01:06:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.687+0000 7ffa8edff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 0x7ffa8819c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.687+0000 7ffa8edff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa88102130 0x7ffa8819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8edff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa8810acd0 0x7ffa881965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8edff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88196c30 con 0x7ffa88102130 2023-09-10T01:06:23.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8edff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88196da0 con 0x7ffa8810acd0 2023-09-10T01:06:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8edff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88196f60 con 0x7ffa88101780 2023-09-10T01:06:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8cb9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 0x7ffa8819c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8cb9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 0x7ffa8819c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39100/0 (socket says 172.21.15.3:39100) 2023-09-10T01:06:23.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/619175250 learned_addr learned my addr 172.21.15.3:0/619175250 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa8d39c700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa8810acd0 0x7ffa881965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.688+0000 7ffa87fff700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa88102130 0x7ffa8819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8d39c700 1 -- 172.21.15.3:0/619175250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 msgr2=0x7ffa8819c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8d39c700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 0x7ffa8819c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8d39c700 1 -- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa88102130 msgr2=0x7ffa8819ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8d39c700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa88102130 0x7ffa8819ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8d39c700 1 -- 172.21.15.3:0/619175250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c0095f0 con 0x7ffa8810acd0 2023-09-10T01:06:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8d39c700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa8810acd0 0x7ffa881965c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000efc0 tx=0x7ffa8000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa85ffb700 1 -- 172.21.15.3:0/619175250 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80018070 con 0x7ffa8810acd0 2023-09-10T01:06:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa85ffb700 1 -- 172.21.15.3:0/619175250 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa8000f040 con 0x7ffa8810acd0 2023-09-10T01:06:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa85ffb700 1 -- 172.21.15.3:0/619175250 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000cdf0 con 0x7ffa8810acd0 2023-09-10T01:06:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa881971e0 con 0x7ffa8810acd0 2023-09-10T01:06:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.689+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa881a3540 con 0x7ffa8810acd0 2023-09-10T01:06:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.694+0000 7ffa85ffb700 1 -- 172.21.15.3:0/619175250 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ffa80003a00 con 0x7ffa8810acd0 2023-09-10T01:06:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.695+0000 7ffaa2627b80 1 Processor -- start 2023-09-10T01:06:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.695+0000 7ffaa2627b80 1 -- start start 2023-09-10T01:06:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4732b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47330b0 0x558dd4740d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47412d0 0x558dd47436c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd468b7b0 con 0x558dd4732790 2023-09-10T01:06:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd46890d0 con 0x558dd47412d0 2023-09-10T01:06:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd46849c0 con 0x558dd47330b0 2023-09-10T01:06:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffa85ffb700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffa780a07b0 0x7ffa780a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffa8cb9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47330b0 0x558dd4740d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffa8cb9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47330b0 0x558dd4740d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39112/0 (socket says 172.21.15.3:39112) 2023-09-10T01:06:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/1973720181 learned_addr learned my addr 172.21.15.3:0/1973720181 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.696+0000 7ffa85ffb700 1 -- 172.21.15.3:0/619175250 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ffa800cbff0 con 0x7ffa8810acd0 2023-09-10T01:06:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa87fff700 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4732b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa87fff700 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47412d0 0x558dd47436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/1973720181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47412d0 msgr2=0x558dd47436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47412d0 0x558dd47436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa8d39c700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffa780a07b0 0x7ffa780a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/1973720181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 msgr2=0x558dd4732b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4732b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/1973720181 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dd44b0e60 con 0x558dd47330b0 2023-09-10T01:06:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.697+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47330b0 0x558dd4740d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b640 tx=0x7ffa7c00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63cdd674ae3f49cb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.698+0000 7ffa72ffd700 1 -- 172.21.15.3:0/1973720181 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c01d030 con 0x558dd47330b0 2023-09-10T01:06:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.698+0000 7ffa72ffd700 1 -- 172.21.15.3:0/1973720181 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa7c01e040 con 0x558dd47330b0 2023-09-10T01:06:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.698+0000 7ffa72ffd700 1 -- 172.21.15.3:0/1973720181 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c0049a0 con 0x558dd47330b0 2023-09-10T01:06:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.698+0000 7ffaa2627b80 1 -- 172.21.15.3:0/1973720181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47330b0 msgr2=0x558dd4740d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.698+0000 7ffa8d39c700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffa780a07b0 0x7ffa780a2c70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa800a0c50 tx=0x7ffa8000cf70 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.698+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47330b0 0x558dd4740d90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b640 tx=0x7ffa7c00b670 comp rx=0 tx=0).stop 2023-09-10T01:06:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.699+0000 7ffaa2627b80 1 -- 172.21.15.3:0/1973720181 shutdown_connections 2023-09-10T01:06:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.699+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47412d0 0x558dd47436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.699+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4732b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.699+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/1973720181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47330b0 0x558dd4740d90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.699+0000 7ffaa2627b80 1 -- 172.21.15.3:0/1973720181 >> 172.21.15.3:0/1973720181 conn(0x558dd46a7110 msgr2=0x558dd472dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.700+0000 7ffaa2627b80 1 -- 172.21.15.3:0/1973720181 shutdown_connections 2023-09-10T01:06:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.700+0000 7ffaa2627b80 1 -- 172.21.15.3:0/1973720181 wait complete. 2023-09-10T01:06:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.700+0000 7ffaa2627b80 1 -- start start 2023-09-10T01:06:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.700+0000 7ffaa2627b80 1 Processor -- start 2023-09-10T01:06:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd4732790 0x558dd4740d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd473df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47412d0 0x558dd473e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd4622bf0 con 0x558dd47412d0 2023-09-10T01:06:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd46228f0 con 0x558dd47330b0 2023-09-10T01:06:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd45069a0 con 0x558dd4732790 2023-09-10T01:06:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffaa2627b80 10 client.0 _ll_register_callbacks cb 0x558dd456cb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffa87fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd4732790 0x558dd4740d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffa87fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd4732790 0x558dd4740d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39124/0 (socket says 172.21.15.3:39124) 2023-09-10T01:06:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffa87fff700 1 -- 172.21.15.3:0/3814217989 learned_addr learned my addr 172.21.15.3:0/3814217989 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd473df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.701+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47412d0 0x558dd473e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/3814217989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd4732790 msgr2=0x558dd4740d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd4732790 0x558dd4740d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/3814217989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47412d0 msgr2=0x558dd473e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47412d0 0x558dd473e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/3814217989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8810de00 con 0x558dd47330b0 2023-09-10T01:06:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd4732790 0x558dd4740d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd473df20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00bb30 tx=0x7ffa7c00bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2154217bafadb55 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa67fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa67fff700 1 -- 172.21.15.3:0/3814217989 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c031070 con 0x558dd47330b0 2023-09-10T01:06:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffa67fff700 1 -- 172.21.15.3:0/3814217989 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ffa7c005980 con 0x558dd47330b0 2023-09-10T01:06:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.702+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558dd45dc570 con 0x558dd47330b0 2023-09-10T01:06:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffa67fff700 1 -- 172.21.15.3:0/3814217989 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa7c01d070 con 0x558dd47330b0 2023-09-10T01:06:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 10 client.26639 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558dd4499ce0 con 0x558dd47330b0 2023-09-10T01:06:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffa67fff700 1 -- 172.21.15.3:0/3814217989 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558dd45dc570 con 0x558dd47330b0 2023-09-10T01:06:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffa67fff700 1 -- 172.21.15.3:0/3814217989 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ffa7c034020 con 0x558dd47330b0 2023-09-10T01:06:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 10 client.26639 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 10 client.26639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 20 client.26639 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:23.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47508e0 0x558dd4752d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 4 client.26639 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.703+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558dd4699920 con 0x558dd47508e0 2023-09-10T01:06:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.704+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47508e0 0x558dd4752d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.704+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47508e0 0x558dd4752d00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86c17daf77417900 server_cookie=6b040115fbb84860 in_seq=0 out_seq=0 2023-09-10T01:06:23.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.704+0000 7ffa67fff700 10 client.26639 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffa67fff700 1 -- 172.21.15.3:0/3814217989 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558dd4699920 con 0x558dd47508e0 2023-09-10T01:06:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffa67fff700 10 client.26639 handle_command_reply: tid=0 2023-09-10T01:06:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 client.26639 shutdown 2023-09-10T01:06:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 msgr2=0x558dd473df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd473df20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00bb30 tx=0x7ffa7c00bf10 comp rx=0 tx=0).stop 2023-09-10T01:06:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 shutdown_connections 2023-09-10T01:06:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47508e0 0x558dd4752d00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd4732790 0x558dd4740d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47412d0 0x558dd473e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3814217989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd473df20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 >> 172.21.15.3:0/3814217989 conn(0x558dd46a7110 msgr2=0x558dd472d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffa67fff700 0 client.26639 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 shutdown_connections 2023-09-10T01:06:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.705+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3814217989 wait complete. 2023-09-10T01:06:23.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.706+0000 7ffaa2627b80 20 client.26639 trim_cache size 0 max 16384 2023-09-10T01:06:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.726+0000 7ffaa2627b80 1 Processor -- start 2023-09-10T01:06:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.726+0000 7ffaa2627b80 1 -- start start 2023-09-10T01:06:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd4732790 0x558dd4739370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47330b0 0x558dd4740840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd473b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd4734290 con 0x558dd47330b0 2023-09-10T01:06:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd4743a80 con 0x558dd4732790 2023-09-10T01:06:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd4757660 con 0x558dd47412d0 2023-09-10T01:06:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffa87fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd4732790 0x558dd4739370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffa8cb9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47330b0 0x558dd4740840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffa87fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd4732790 0x558dd4739370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53988/0 (socket says 172.21.15.3:53988) 2023-09-10T01:06:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffa87fff700 1 -- 172.21.15.3:0/3208604287 learned_addr learned my addr 172.21.15.3:0/3208604287 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd473b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.727+0000 7ffa87fff700 1 -- 172.21.15.3:0/3208604287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 msgr2=0x558dd473b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd473b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa87fff700 1 -- 172.21.15.3:0/3208604287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47330b0 msgr2=0x558dd4740840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47330b0 0x558dd4740840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa87fff700 1 -- 172.21.15.3:0/3208604287 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dd4757930 con 0x558dd4732790 2023-09-10T01:06:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47330b0 0x558dd4740840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa87fff700 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd4732790 0x558dd4739370 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x558dd473efb0 tx=0x7ffa74014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d92ac85d2ae932c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa737fe700 1 -- 172.21.15.3:0/3208604287 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74012220 con 0x558dd4732790 2023-09-10T01:06:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa737fe700 1 -- 172.21.15.3:0/3208604287 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa7402b070 con 0x558dd4732790 2023-09-10T01:06:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.728+0000 7ffa737fe700 1 -- 172.21.15.3:0/3208604287 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74003680 con 0x558dd4732790 2023-09-10T01:06:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3208604287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd4732790 msgr2=0x558dd4739370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd4732790 0x558dd4739370 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x558dd473efb0 tx=0x7ffa74014040 comp rx=0 tx=0).stop 2023-09-10T01:06:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3208604287 shutdown_connections 2023-09-10T01:06:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd473b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd4732790 0x558dd4739370 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/3208604287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd47330b0 0x558dd4740840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3208604287 >> 172.21.15.3:0/3208604287 conn(0x558dd46a7110 msgr2=0x558dd472c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3208604287 shutdown_connections 2023-09-10T01:06:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 -- 172.21.15.3:0/3208604287 wait complete. 2023-09-10T01:06:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.729+0000 7ffaa2627b80 1 -- start start 2023-09-10T01:06:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.730+0000 7ffaa2627b80 1 Processor -- start 2023-09-10T01:06:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.730+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4747060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.730+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd47475a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffaa2627b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd479d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd479dd30 con 0x558dd4732790 2023-09-10T01:06:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd479dea0 con 0x558dd47330b0 2023-09-10T01:06:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffaa2627b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd479e060 con 0x558dd47412d0 2023-09-10T01:06:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffaa2627b80 10 client.0 _ll_register_callbacks cb 0x7ffa74009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa87fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4747060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa87fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4747060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50692/0 (socket says 172.21.15.3:50692) 2023-09-10T01:06:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa87fff700 1 -- 172.21.15.3:0/2212245935 learned_addr learned my addr 172.21.15.3:0/2212245935 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd47475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa87fff700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd479d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/2212245935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 msgr2=0x558dd479d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd479d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/2212245935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 msgr2=0x558dd4747060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4747060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa8cb9b700 1 -- 172.21.15.3:0/2212245935 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558dd4499ce0 con 0x558dd47330b0 2023-09-10T01:06:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.731+0000 7ffa87fff700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4747060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffa8cb9b700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd47475a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009ce0 tx=0x7ffa7c0053d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bce70d84d6bd6ab server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffa67fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffa67fff700 1 -- 172.21.15.3:0/2212245935 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c031070 con 0x558dd47330b0 2023-09-10T01:06:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa7c01e040 con 0x558dd47330b0 2023-09-10T01:06:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffa67fff700 1 -- 172.21.15.3:0/2212245935 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ffa7c000e40 con 0x558dd47330b0 2023-09-10T01:06:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffa67fff700 1 -- 172.21.15.3:0/2212245935 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa7c01d040 con 0x558dd47330b0 2023-09-10T01:06:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffaa2627b80 10 client.26645 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:23.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.732+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7ffa74025510 con 0x558dd47330b0 2023-09-10T01:06:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffa67fff700 1 -- 172.21.15.3:0/2212245935 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa7c01e040 con 0x558dd47330b0 2023-09-10T01:06:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffa67fff700 1 -- 172.21.15.3:0/2212245935 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ffa7c00b850 con 0x558dd47330b0 2023-09-10T01:06:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffaa2627b80 10 client.26645 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffaa2627b80 10 client.26645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:23.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffaa2627b80 20 client.26645 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47a66f0 0x558dd47a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffaa2627b80 4 client.26645 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:06:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558dd467d9a0 con 0x558dd47a66f0 2023-09-10T01:06:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.733+0000 7ffa87fff700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47a66f0 0x558dd47a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.734+0000 7ffa87fff700 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47a66f0 0x558dd47a8b10 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e667402e6fb4176a server_cookie=f08174cc44343173 in_seq=0 out_seq=0 2023-09-10T01:06:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.734+0000 7ffa67fff700 10 client.26645 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffa67fff700 1 -- 172.21.15.3:0/2212245935 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558dd467d9a0 con 0x558dd47a66f0 2023-09-10T01:06:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffa67fff700 10 client.26645 handle_command_reply: tid=0 2023-09-10T01:06:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 client.26645 shutdown 2023-09-10T01:06:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 msgr2=0x558dd47475a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd47475a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009ce0 tx=0x7ffa7c0053d0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 shutdown_connections 2023-09-10T01:06:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dd47a66f0 0x558dd47a8b10 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dd47412d0 0x558dd479d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dd4732790 0x558dd4747060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 --2- 172.21.15.3:0/2212245935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dd47330b0 0x558dd47475a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 >> 172.21.15.3:0/2212245935 conn(0x558dd46a7110 msgr2=0x558dd4731780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.735+0000 7ffa67fff700 0 client.26645 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 shutdown_connections 2023-09-10T01:06:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffaa2627b80 1 -- 172.21.15.3:0/2212245935 wait complete. 2023-09-10T01:06:23.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffaa2627b80 20 client.26645 trim_cache size 0 max 16384 2023-09-10T01:06:23.750 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:06:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:06:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "4ea45e60-c7cf-402f-bb16-16e2904622fe", 2023-09-10T01:06:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:06:23.751 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:06:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffa780a07b0 msgr2=0x7ffa780a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffa8edff700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffa780a07b0 0x7ffa780a2c70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa800a0c50 tx=0x7ffa8000cf70 comp rx=0 tx=0).stop 2023-09-10T01:06:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa8810acd0 msgr2=0x7ffa881965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffa8edff700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa8810acd0 0x7ffa881965c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000efc0 tx=0x7ffa8000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 shutdown_connections 2023-09-10T01:06:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffa8edff700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffa780a07b0 0x7ffa780a2c70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.736+0000 7ffa8edff700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa88101780 0x7ffa8819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.737+0000 7ffa8edff700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa8810acd0 0x7ffa881965c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.737+0000 7ffa8edff700 1 --2- 172.21.15.3:0/619175250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffa88102130 0x7ffa8819ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.737+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 >> 172.21.15.3:0/619175250 conn(0x7ffa880fb010 msgr2=0x7ffa8810cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.737+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 shutdown_connections 2023-09-10T01:06:23.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.737+0000 7ffa8edff700 1 -- 172.21.15.3:0/619175250 wait complete. 2023-09-10T01:06:23.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 228127 8.10 MB/sec execute 5 sec latency 14.376 ms 2023-09-10T01:06:23.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:06:23.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 -- 172.21.15.3:0/3854837861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 msgr2=0x7ffb4010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3854837861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4010d510 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800b0a0 tx=0x7ffb3800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 -- 172.21.15.3:0/3854837861 shutdown_connections 2023-09-10T01:06:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3854837861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4010d510 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3854837861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb40106650 0x7ffb40106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3854837861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb40100640 0x7ffb40100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 -- 172.21.15.3:0/3854837861 >> 172.21.15.3:0/3854837861 conn(0x7ffb400fc170 msgr2=0x7ffb400fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 -- 172.21.15.3:0/3854837861 shutdown_connections 2023-09-10T01:06:23.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.990+0000 7ffb459ff700 1 -- 172.21.15.3:0/3854837861 wait complete. 2023-09-10T01:06:23.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 Processor -- start 2023-09-10T01:06:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 -- start start 2023-09-10T01:06:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb40100640 0x7ffb4019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb40106650 0x7ffb401986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:23.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40198d20 con 0x7ffb40106650 2023-09-10T01:06:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40198e90 con 0x7ffb40101000 2023-09-10T01:06:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb459ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40199050 con 0x7ffb40100640 2023-09-10T01:06:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb3effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb40100640 0x7ffb4019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4019eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54020/0 (socket says 172.21.15.3:54020) 2023-09-10T01:06:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb3effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb40100640 0x7ffb4019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39162/0 (socket says 172.21.15.3:39162) 2023-09-10T01:06:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3658671640 learned_addr learned my addr 172.21.15.3:0/3658671640 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.992+0000 7ffb3f7fe700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb40106650 0x7ffb401986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3658671640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb40100640 msgr2=0x7ffb4019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb40100640 0x7ffb4019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb40106650 msgr2=0x7ffb401986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:23.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb40106650 0x7ffb401986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:23.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3658671640 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb38009d00 con 0x7ffb40101000 2023-09-10T01:06:23.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4019eb80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400e9a0 tx=0x7ffb3400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:23.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb449fd700 1 -- 172.21.15.3:0/3658671640 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34009790 con 0x7ffb40101000 2023-09-10T01:06:24.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb449fd700 1 -- 172.21.15.3:0/3658671640 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb34010430 con 0x7ffb40101000 2023-09-10T01:06:24.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb449fd700 1 -- 172.21.15.3:0/3658671640 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34018680 con 0x7ffb40101000 2023-09-10T01:06:24.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb40199330 con 0x7ffb40101000 2023-09-10T01:06:24.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.993+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb401a5630 con 0x7ffb40101000 2023-09-10T01:06:24.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.997+0000 7ffb449fd700 1 -- 172.21.15.3:0/3658671640 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ffb340105a0 con 0x7ffb40101000 2023-09-10T01:06:24.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.998+0000 7ffb59342b80 1 Processor -- start 2023-09-10T01:06:24.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.998+0000 7ffb59342b80 1 -- start start 2023-09-10T01:06:24.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ac9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb59342b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625a22730 con 0x55d625ac9710 2023-09-10T01:06:24.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb59342b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625a20050 con 0x55d625ad3e40 2023-09-10T01:06:24.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb59342b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625a1b940 con 0x55d625aca030 2023-09-10T01:06:24.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb449fd700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffb280a07b0 0x7ffb280a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54030/0 (socket says 172.21.15.3:54030) 2023-09-10T01:06:24.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/1936086156 learned_addr learned my addr 172.21.15.3:0/1936086156 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:24.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3effd700 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb449fd700 1 -- 172.21.15.3:0/3658671640 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ffb34014070 con 0x7ffb40101000 2023-09-10T01:06:24.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3f7fe700 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ac9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3f7fe700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffb280a07b0 0x7ffb280a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/1936086156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 msgr2=0x55d625ad3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/1936086156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 msgr2=0x55d625ac9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:23.999+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ac9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.000+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/1936086156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d625847de0 con 0x55d625ad3e40 2023-09-10T01:06:24.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.000+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad6230 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffb34099590 tx=0x7ffb3406c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff336609921674cd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:24.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.000+0000 7ffb25ffb700 1 -- 172.21.15.3:0/1936086156 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34065340 con 0x55d625ad3e40 2023-09-10T01:06:24.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.000+0000 7ffb25ffb700 1 -- 172.21.15.3:0/1936086156 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb34099d00 con 0x55d625ad3e40 2023-09-10T01:06:24.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.000+0000 7ffb25ffb700 1 -- 172.21.15.3:0/1936086156 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34097a30 con 0x55d625ad3e40 2023-09-10T01:06:24.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 -- 172.21.15.3:0/1936086156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 msgr2=0x55d625ad6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:24.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad6230 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffb34099590 tx=0x7ffb3406c720 comp rx=0 tx=0).stop 2023-09-10T01:06:24.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb3f7fe700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffb280a07b0 0x7ffb280a2c70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800b780 tx=0x7ffb38006230 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:24.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 -- 172.21.15.3:0/1936086156 shutdown_connections 2023-09-10T01:06:24.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad6230 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ac9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 --2- 172.21.15.3:0/1936086156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 -- 172.21.15.3:0/1936086156 >> 172.21.15.3:0/1936086156 conn(0x55d625a3e090 msgr2=0x55d625ac4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:24.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 -- 172.21.15.3:0/1936086156 shutdown_connections 2023-09-10T01:06:24.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.001+0000 7ffb59342b80 1 -- 172.21.15.3:0/1936086156 wait complete. 2023-09-10T01:06:24.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.002+0000 7ffb59342b80 1 -- start start 2023-09-10T01:06:24.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.002+0000 7ffb59342b80 1 Processor -- start 2023-09-10T01:06:24.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ac9710 0x55d625ad3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ad3e40 0x55d625ad1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb59342b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6259b9b70 con 0x55d625aca030 2023-09-10T01:06:24.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb59342b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6259b9870 con 0x55d625ac9710 2023-09-10T01:06:24.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb59342b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d62589d920 con 0x55d625ad3e40 2023-09-10T01:06:24.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb59342b80 10 client.0 _ll_register_callbacks cb 0x55d625903a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:24.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb3effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ac9710 0x55d625ad3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb3effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ac9710 0x55d625ad3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54032/0 (socket says 172.21.15.3:54032) 2023-09-10T01:06:24.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb3effd700 1 -- 172.21.15.3:0/837657822 learned_addr learned my addr 172.21.15.3:0/837657822 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:24.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb3f7fe700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ad3e40 0x55d625ad1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb3effd700 1 -- 172.21.15.3:0/837657822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ad3e40 msgr2=0x55d625ad1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.003+0000 7ffb3effd700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ad3e40 0x55d625ad1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb3effd700 1 -- 172.21.15.3:0/837657822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 msgr2=0x55d625ad0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb3effd700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb3effd700 1 -- 172.21.15.3:0/837657822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb401a5630 con 0x55d625ac9710 2023-09-10T01:06:24.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb3f7fe700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ad3e40 0x55d625ad1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:24.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb3effd700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ac9710 0x55d625ad3800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffb30007e90 tx=0x7ffb30011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6522a248f50ff416 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:24.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb1effd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:24.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb1effd700 1 -- 172.21.15.3:0/837657822 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30016070 con 0x55d625ac9710 2023-09-10T01:06:24.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb1effd700 1 -- 172.21.15.3:0/837657822 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ffb30010700 con 0x55d625ac9710 2023-09-10T01:06:24.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d6259734f0 con 0x55d625ac9710 2023-09-10T01:06:24.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb1effd700 1 -- 172.21.15.3:0/837657822 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb30010900 con 0x55d625ac9710 2023-09-10T01:06:24.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb59342b80 10 client.26657 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:24.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.004+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d625830c60 con 0x55d625ac9710 2023-09-10T01:06:24.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb1effd700 1 -- 172.21.15.3:0/837657822 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d6259734f0 con 0x55d625ac9710 2023-09-10T01:06:24.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb1effd700 1 -- 172.21.15.3:0/837657822 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ffb3001ba50 con 0x55d625ac9710 2023-09-10T01:06:24.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb59342b80 10 client.26657 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:24.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb59342b80 10 client.26657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:24.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb59342b80 20 client.26657 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:24.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb59342b80 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625ae3960 0x55d625ae5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb59342b80 4 client.26657 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:24.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d625a308a0 con 0x55d625ae3960 2023-09-10T01:06:24.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.005+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625ae3960 0x55d625ae5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.006+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625ae3960 0x55d625ae5d80 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=334ed809ad58072c server_cookie=9011537702ba64b8 in_seq=0 out_seq=0 2023-09-10T01:06:24.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.006+0000 7ffb1effd700 10 client.26657 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:24.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.006+0000 7ffb1effd700 1 -- 172.21.15.3:0/837657822 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d625a308a0 con 0x55d625ae3960 2023-09-10T01:06:24.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.006+0000 7ffb1effd700 10 client.26657 handle_command_reply: tid=0 2023-09-10T01:06:24.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.006+0000 7ffb59342b80 1 client.26657 shutdown 2023-09-10T01:06:24.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ac9710 msgr2=0x55d625ad3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:24.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ac9710 0x55d625ad3800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffb30007e90 tx=0x7ffb30011720 comp rx=0 tx=0).stop 2023-09-10T01:06:24.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 shutdown_connections 2023-09-10T01:06:24.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625ae3960 0x55d625ae5d80 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ad3e40 0x55d625ad1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ac9710 0x55d625ad3800 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 --2- 172.21.15.3:0/837657822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb1effd700 0 client.26657 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:24.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 >> 172.21.15.3:0/837657822 conn(0x55d625a3e090 msgr2=0x55d625ac47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:24.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 shutdown_connections 2023-09-10T01:06:24.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 1 -- 172.21.15.3:0/837657822 wait complete. 2023-09-10T01:06:24.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.007+0000 7ffb59342b80 20 client.26657 trim_cache size 0 max 16384 2023-09-10T01:06:24.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 Processor -- start 2023-09-10T01:06:24.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 -- start start 2023-09-10T01:06:24.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ad3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625acef70 con 0x55d625ac9710 2023-09-10T01:06:24.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625ac39a0 con 0x55d625ad3e40 2023-09-10T01:06:24.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.029+0000 7ffb59342b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625ad0d30 con 0x55d625aca030 2023-09-10T01:06:24.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39204/0 (socket says 172.21.15.3:39204) 2023-09-10T01:06:24.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3560742885 learned_addr learned my addr 172.21.15.3:0/3560742885 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:24.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ad3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3560742885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 msgr2=0x55d625ad07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3560742885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 msgr2=0x55d625ad3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ad3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3560742885 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d62586e250 con 0x55d625aca030 2023-09-10T01:06:24.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3effd700 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ad3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad02b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffb340a1140 tx=0x7ffb34010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb0cb7d393c23149 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb267fc700 1 -- 172.21.15.3:0/3560742885 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34099600 con 0x55d625aca030 2023-09-10T01:06:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb267fc700 1 -- 172.21.15.3:0/3560742885 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb340979a0 con 0x55d625aca030 2023-09-10T01:06:24.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.030+0000 7ffb267fc700 1 -- 172.21.15.3:0/3560742885 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3409d020 con 0x55d625aca030 2023-09-10T01:06:24.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 -- 172.21.15.3:0/3560742885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 msgr2=0x55d625ad02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:24.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad02b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffb340a1140 tx=0x7ffb34010c80 comp rx=0 tx=0).stop 2023-09-10T01:06:24.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 -- 172.21.15.3:0/3560742885 shutdown_connections 2023-09-10T01:06:24.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625ac9710 0x55d625ad3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3560742885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625aca030 0x55d625ad02b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 -- 172.21.15.3:0/3560742885 >> 172.21.15.3:0/3560742885 conn(0x55d625a3e090 msgr2=0x55d625acbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:24.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 -- 172.21.15.3:0/3560742885 shutdown_connections 2023-09-10T01:06:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.031+0000 7ffb59342b80 1 -- 172.21.15.3:0/3560742885 wait complete. 2023-09-10T01:06:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.032+0000 7ffb59342b80 1 -- start start 2023-09-10T01:06:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.032+0000 7ffb59342b80 1 Processor -- start 2023-09-10T01:06:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ac9710 0x55d625ada9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb59342b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb59342b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625ad98f0 con 0x55d625aca030 2023-09-10T01:06:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb59342b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625ad9a60 con 0x55d625ad3e40 2023-09-10T01:06:24.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb59342b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d625b37780 con 0x55d625ac9710 2023-09-10T01:06:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb59342b80 10 client.0 _ll_register_callbacks cb 0x7ffb30007960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50754/0 (socket says 172.21.15.3:50754) 2023-09-10T01:06:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.033+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3880144925 learned_addr learned my addr 172.21.15.3:0/3880144925 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:24.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3f7fe700 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3effd700 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ac9710 0x55d625ada9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3880144925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ac9710 msgr2=0x55d625ada9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ac9710 0x55d625ada9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 msgr2=0x55d625ad9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3e7fc700 1 -- 172.21.15.3:0/3880144925 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d625830c60 con 0x55d625aca030 2023-09-10T01:06:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb3e7fc700 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad8ce0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffb34099300 tx=0x7ffb34097b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88ca56ad1516304 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb1effd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:06:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb1effd700 1 -- 172.21.15.3:0/3880144925 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb340840f0 con 0x55d625aca030 2023-09-10T01:06:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb1effd700 1 -- 172.21.15.3:0/3880144925 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ffb34084250 con 0x55d625aca030 2023-09-10T01:06:24.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.034+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb34099d00 con 0x55d625aca030 2023-09-10T01:06:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb1effd700 1 -- 172.21.15.3:0/3880144925 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb3409d050 con 0x55d625aca030 2023-09-10T01:06:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 10 client.16599 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d625847de0 con 0x55d625aca030 2023-09-10T01:06:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb1effd700 1 -- 172.21.15.3:0/3880144925 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb34099d00 con 0x55d625aca030 2023-09-10T01:06:24.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb1effd700 1 -- 172.21.15.3:0/3880144925 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ffb34098af0 con 0x55d625aca030 2023-09-10T01:06:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 10 client.16599 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 20 client.16599 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625b389e0 0x55d625b3ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 4 client.16599 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:06:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d625a14920 con 0x55d625b389e0 2023-09-10T01:06:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.035+0000 7ffb3effd700 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625b389e0 0x55d625b3ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.036+0000 7ffb3effd700 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625b389e0 0x55d625b3ae00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f436b4e8966d38f server_cookie=4c334ac1c615f531 in_seq=0 out_seq=0 2023-09-10T01:06:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.036+0000 7ffb1effd700 10 client.16599 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:24.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.036+0000 7ffb1effd700 1 -- 172.21.15.3:0/3880144925 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d625a14920 con 0x55d625b389e0 2023-09-10T01:06:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.036+0000 7ffb1effd700 10 client.16599 handle_command_reply: tid=0 2023-09-10T01:06:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.036+0000 7ffb59342b80 1 client.16599 shutdown 2023-09-10T01:06:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.036+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 msgr2=0x55d625ad8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.036+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad8ce0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffb34099300 tx=0x7ffb34097b10 comp rx=0 tx=0).stop 2023-09-10T01:06:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 shutdown_connections 2023-09-10T01:06:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d625b389e0 0x55d625b3ae00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d625ac9710 0x55d625ada9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d625ad3e40 0x55d625ad9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 --2- 172.21.15.3:0/3880144925 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d625aca030 0x55d625ad8ce0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 >> 172.21.15.3:0/3880144925 conn(0x55d625a3e090 msgr2=0x55d625ad0280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb1effd700 0 client.16599 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 shutdown_connections 2023-09-10T01:06:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 1 -- 172.21.15.3:0/3880144925 wait complete. 2023-09-10T01:06:24.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.037+0000 7ffb59342b80 20 client.16599 trim_cache size 0 max 16384 2023-09-10T01:06:24.052 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:06:24.052 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:06:24.052 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:06:24.052 INFO:teuthology.orchestra.run.smithi003.stdout: "4ea45e60-c7cf-402f-bb16-16e2904622fe": { 2023-09-10T01:06:24.052 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:06:24.053 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "4ea45e60-c7cf-402f-bb16-16e2904622fe", 2023-09-10T01:06:24.053 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:06:24.053 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:06:24.053 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:06:24.053 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:06:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffb280a07b0 msgr2=0x7ffb280a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffb280a07b0 0x7ffb280a2c70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800b780 tx=0x7ffb38006230 comp rx=0 tx=0).stop 2023-09-10T01:06:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 msgr2=0x7ffb4019eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4019eb80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400e9a0 tx=0x7ffb3400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 shutdown_connections 2023-09-10T01:06:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffb280a07b0 0x7ffb280a2c70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb40100640 0x7ffb4019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb40106650 0x7ffb401986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 --2- 172.21.15.3:0/3658671640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb40101000 0x7ffb4019eb80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 >> 172.21.15.3:0/3658671640 conn(0x7ffb400fc170 msgr2=0x7ffb4010ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 shutdown_connections 2023-09-10T01:06:24.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:24.038+0000 7ffb459ff700 1 -- 172.21.15.3:0/3658671640 wait complete. 2023-09-10T01:06:24.057 INFO:tasks.cephfs.filesystem:scrub status for tag:4ea45e60-c7cf-402f-bb16-16e2904622fe - {'path': '/', 'tag': '4ea45e60-c7cf-402f-bb16-16e2904622fe', 'options': 'recursive,force'} 2023-09-10T01:06:24.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: pgmap v98: 129 pgs: 129 active+clean; 70 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:06:24.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:06:24.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:06:24.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:06:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:06:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:06:24.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: pgmap v98: 129 pgs: 129 active+clean; 70 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:06:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:06:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:06:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:06:24.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:06:24.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:06:24.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: pgmap v98: 129 pgs: 129 active+clean; 70 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:06:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:06:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:06:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:06:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:06:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:06:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:24.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 230430 8.84 MB/sec execute 6 sec latency 21.889 ms 2023-09-10T01:06:25.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:25 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:06:25.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:25 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:25.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:25 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:06:25.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:25 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:06:25.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:25 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:06:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:25 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:06:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:25 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:25 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:06:25.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:25 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:06:25.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:25 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:06:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:25 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:06:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:25 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:25 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:06:25.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:25 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:06:25.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:25 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:06:25.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 231597 8.37 MB/sec execute 7 sec latency 20.644 ms 2023-09-10T01:06:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:26 smithi003 ceph-mon[97654]: pgmap v99: 129 pgs: 129 active+clean; 67 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 409 op/s 2023-09-10T01:06:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:26 smithi136 ceph-mon[133708]: pgmap v99: 129 pgs: 129 active+clean; 67 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 409 op/s 2023-09-10T01:06:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:26 smithi092 ceph-mon[134067]: pgmap v99: 129 pgs: 129 active+clean; 67 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 409 op/s 2023-09-10T01:06:26.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 234221 9.41 MB/sec execute 8 sec latency 10.913 ms 2023-09-10T01:06:27.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:27 smithi003 ceph-mon[97654]: pgmap v100: 129 pgs: 129 active+clean; 67 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 295 op/s 2023-09-10T01:06:27.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:27 smithi136 ceph-mon[133708]: pgmap v100: 129 pgs: 129 active+clean; 67 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 295 op/s 2023-09-10T01:06:27.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:27 smithi092 ceph-mon[134067]: pgmap v100: 129 pgs: 129 active+clean; 67 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 295 op/s 2023-09-10T01:06:27.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 235056 8.61 MB/sec execute 9 sec latency 45.260 ms 2023-09-10T01:06:28.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 237504 9.13 MB/sec execute 10 sec latency 18.523 ms 2023-09-10T01:06:29.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:29 smithi136 ceph-mon[133708]: pgmap v101: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 406 op/s 2023-09-10T01:06:29.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:29 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:06:29.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:29 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:06:29.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:29 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:06:29.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:29 smithi092 ceph-mon[134067]: pgmap v101: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 406 op/s 2023-09-10T01:06:29.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:29 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:06:29.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:29 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:06:29.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:29 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:06:29.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 238734 8.89 MB/sec execute 11 sec latency 12.423 ms 2023-09-10T01:06:29.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:29 smithi003 ceph-mon[97654]: pgmap v101: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 406 op/s 2023-09-10T01:06:29.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:29 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:06:29.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:29 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:06:29.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:29 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:06:30.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241328 9.47 MB/sec execute 12 sec latency 10.147 ms 2023-09-10T01:06:31.746 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242417 9.01 MB/sec execute 13 sec latency 13.456 ms 2023-09-10T01:06:32.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:32 smithi003 ceph-mon[97654]: pgmap v102: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 316 op/s 2023-09-10T01:06:32.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:32 smithi136 ceph-mon[133708]: pgmap v102: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 316 op/s 2023-09-10T01:06:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:32 smithi092 ceph-mon[134067]: pgmap v102: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 316 op/s 2023-09-10T01:06:32.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 244872 9.48 MB/sec execute 14 sec latency 33.400 ms 2023-09-10T01:06:33.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 246019 9.09 MB/sec execute 15 sec latency 14.323 ms 2023-09-10T01:06:34.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:34 smithi003 ceph-mon[97654]: pgmap v103: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 289 op/s 2023-09-10T01:06:34.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:34 smithi136 ceph-mon[133708]: pgmap v103: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 289 op/s 2023-09-10T01:06:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:34 smithi092 ceph-mon[134067]: pgmap v103: 129 pgs: 129 active+clean; 71 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 289 op/s 2023-09-10T01:06:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.725+0000 7f19027ff700 1 -- 172.21.15.3:0/830677948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc106680 msgr2=0x7f18fc106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.725+0000 7f19027ff700 1 --2- 172.21.15.3:0/830677948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc106680 0x7f18fc106a60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0099c0 tx=0x7f18ec009cd0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.725+0000 7f19027ff700 1 -- 172.21.15.3:0/830677948 shutdown_connections 2023-09-10T01:06:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.726+0000 7f19027ff700 1 --2- 172.21.15.3:0/830677948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18fc100f80 0x7f18fc10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.726+0000 7f19027ff700 1 --2- 172.21.15.3:0/830677948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc106680 0x7f18fc106a60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.726+0000 7f19027ff700 1 --2- 172.21.15.3:0/830677948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 0x7f18fc100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.726+0000 7f19027ff700 1 -- 172.21.15.3:0/830677948 >> 172.21.15.3:0/830677948 conn(0x7f18fc0fc0b0 msgr2=0x7f18fc0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.726+0000 7f19027ff700 1 -- 172.21.15.3:0/830677948 shutdown_connections 2023-09-10T01:06:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.726+0000 7f19027ff700 1 -- 172.21.15.3:0/830677948 wait complete. 2023-09-10T01:06:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.727+0000 7f19027ff700 1 Processor -- start 2023-09-10T01:06:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.727+0000 7f19027ff700 1 -- start start 2023-09-10T01:06:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f19027ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 0x7f18fc196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f19027ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc100f80 0x7f18fc196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f19027ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18fc106680 0x7f18fc19afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f19027ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc19b5f0 con 0x7f18fc106680 2023-09-10T01:06:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f19027ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc19b760 con 0x7f18fc100f80 2023-09-10T01:06:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f19027ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc19b920 con 0x7f18fc1005c0 2023-09-10T01:06:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 0x7f18fc196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 0x7f18fc196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:32922/0 (socket says 172.21.15.3:32922) 2023-09-10T01:06:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f1900d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18fc106680 0x7f18fc19afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f18fbfff700 1 -- 172.21.15.3:0/2206836692 learned_addr learned my addr 172.21.15.3:0/2206836692 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.728+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc100f80 0x7f18fc196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18fbfff700 1 -- 172.21.15.3:0/2206836692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc100f80 msgr2=0x7f18fc196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc100f80 0x7f18fc196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18fbfff700 1 -- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18fc106680 msgr2=0x7f18fc19afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18fc106680 0x7f18fc19afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18fbfff700 1 -- 172.21.15.3:0/2206836692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec0095f0 con 0x7f18fc1005c0 2023-09-10T01:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc100f80 0x7f18fc196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 0x7f18fc196610 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00b580 tx=0x7f18ec00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18f97fa700 1 -- 172.21.15.3:0/2206836692 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec01d070 con 0x7f18fc1005c0 2023-09-10T01:06:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18f97fa700 1 -- 172.21.15.3:0/2206836692 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18ec005050 con 0x7f18fc1005c0 2023-09-10T01:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc19bba0 con 0x7f18fc1005c0 2023-09-10T01:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.729+0000 7f18f97fa700 1 -- 172.21.15.3:0/2206836692 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec004a40 con 0x7f18fc1005c0 2023-09-10T01:06:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.730+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc1a3440 con 0x7f18fc1005c0 2023-09-10T01:06:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.733+0000 7f18f97fa700 1 -- 172.21.15.3:0/2206836692 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f18ec005430 con 0x7f18fc1005c0 2023-09-10T01:06:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.734+0000 7f1916202b80 1 Processor -- start 2023-09-10T01:06:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.734+0000 7f1916202b80 1 -- start start 2023-09-10T01:06:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.734+0000 7f18f97fa700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18e40a0830 0x7f18e40a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.734+0000 7f18f97fa700 1 -- 172.21.15.3:0/2206836692 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f18ec0cd140 con 0x7f18fc1005c0 2023-09-10T01:06:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1916202b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5b25b0 0x559b4a5b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1916202b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5acb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1916202b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5b9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1916202b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a5016d0 con 0x559b4a5ac710 2023-09-10T01:06:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1900d9c700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18e40a0830 0x7f18e40a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1916202b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a4feff0 con 0x559b4a5b25b0 2023-09-10T01:06:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1916202b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a4fa8e0 con 0x559b4a5ad0d0 2023-09-10T01:06:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5b25b0 0x559b4a5b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5b25b0 0x559b4a5b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:41340/0 (socket says 172.21.15.3:41340) 2023-09-10T01:06:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 -- 172.21.15.3:0/3781441294 learned_addr learned my addr 172.21.15.3:0/3781441294 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1900d9c700 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5b9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 -- 172.21.15.3:0/3781441294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 msgr2=0x559b4a5b9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5b9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 -- 172.21.15.3:0/3781441294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 msgr2=0x559b4a5acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5acb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 -- 172.21.15.3:0/3781441294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b4a326d80 con 0x559b4a5b25b0 2023-09-10T01:06:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f1900d9c700 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5b9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.735+0000 7f18fbfff700 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5b25b0 0x559b4a5b2990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f18ec096d00 tx=0x7f18ec099f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55f6155e4612e797 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:34.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1900d9c700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18e40a0830 0x7f18e40a2cf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009f50 tx=0x7f18f4003aa0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f18e27fc700 1 -- 172.21.15.3:0/3781441294 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec094ea0 con 0x559b4a5b25b0 2023-09-10T01:06:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f18e27fc700 1 -- 172.21.15.3:0/3781441294 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18ec0a2270 con 0x559b4a5b25b0 2023-09-10T01:06:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f18e27fc700 1 -- 172.21.15.3:0/3781441294 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec004c20 con 0x559b4a5b25b0 2023-09-10T01:06:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1916202b80 1 -- 172.21.15.3:0/3781441294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5b25b0 msgr2=0x559b4a5b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1916202b80 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5b25b0 0x559b4a5b2990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f18ec096d00 tx=0x7f18ec099f00 comp rx=0 tx=0).stop 2023-09-10T01:06:34.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1916202b80 1 -- 172.21.15.3:0/3781441294 shutdown_connections 2023-09-10T01:06:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1916202b80 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5b9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1916202b80 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5b25b0 0x559b4a5b2990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1916202b80 1 --2- 172.21.15.3:0/3781441294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5acb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.736+0000 7f1916202b80 1 -- 172.21.15.3:0/3781441294 >> 172.21.15.3:0/3781441294 conn(0x559b4a51d030 msgr2=0x559b4a5a3560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:34.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.737+0000 7f1916202b80 1 -- 172.21.15.3:0/3781441294 shutdown_connections 2023-09-10T01:06:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.737+0000 7f1916202b80 1 -- 172.21.15.3:0/3781441294 wait complete. 2023-09-10T01:06:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.737+0000 7f1916202b80 1 -- start start 2023-09-10T01:06:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.737+0000 7f1916202b80 1 Processor -- start 2023-09-10T01:06:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1916202b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ac710 0x559b4a5bbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1916202b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5bc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1916202b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1916202b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a498b10 con 0x559b4a5b25b0 2023-09-10T01:06:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1916202b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a498810 con 0x559b4a5ac710 2023-09-10T01:06:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1916202b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a37c8c0 con 0x559b4a5ad0d0 2023-09-10T01:06:34.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1916202b80 10 client.0 _ll_register_callbacks cb 0x559b4a3e2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ac710 0x559b4a5bbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fb7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5bc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ac710 0x559b4a5bbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:41342/0 (socket says 172.21.15.3:41342) 2023-09-10T01:06:34.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 -- 172.21.15.3:0/2678362022 learned_addr learned my addr 172.21.15.3:0/2678362022 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f1900d9c700 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 -- 172.21.15.3:0/2678362022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 msgr2=0x559b4a5bc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5bc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 -- 172.21.15.3:0/2678362022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 msgr2=0x559b4a5b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.738+0000 7f18fbfff700 1 -- 172.21.15.3:0/2678362022 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc1a3440 con 0x559b4a5ac710 2023-09-10T01:06:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ac710 0x559b4a5bbfe0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009fd0 tx=0x7f18ec0ca440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9104de10e90d4436 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f18db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f18db7fe700 1 -- 172.21.15.3:0/2678362022 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec094ea0 con 0x559b4a5ac710 2023-09-10T01:06:34.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f18db7fe700 1 -- 172.21.15.3:0/2678362022 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f18ec004c00 con 0x559b4a5ac710 2023-09-10T01:06:34.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b4a452490 con 0x559b4a5ac710 2023-09-10T01:06:34.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f18db7fe700 1 -- 172.21.15.3:0/2678362022 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18ec004dc0 con 0x559b4a5ac710 2023-09-10T01:06:34.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f1916202b80 10 client.26675 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:34.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559b4a30fc00 con 0x559b4a5ac710 2023-09-10T01:06:34.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f18db7fe700 1 -- 172.21.15.3:0/2678362022 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b4a452490 con 0x559b4a5ac710 2023-09-10T01:06:34.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.739+0000 7f18db7fe700 1 -- 172.21.15.3:0/2678362022 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f18ec099780 con 0x559b4a5ac710 2023-09-10T01:06:34.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f1916202b80 10 client.26675 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:34.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f1916202b80 10 client.26675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:34.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f1916202b80 20 client.26675 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:34.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f1916202b80 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a5c5f60 0x559b4a5c8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f1916202b80 4 client.26675 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b4a50f840 con 0x559b4a5c5f60 2023-09-10T01:06:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a5c5f60 0x559b4a5c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a5c5f60 0x559b4a5c8350 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82f884a287b7d05c server_cookie=890a0a014007bf66 in_seq=0 out_seq=0 2023-09-10T01:06:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.740+0000 7f18db7fe700 10 client.26675 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:34.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f18db7fe700 1 -- 172.21.15.3:0/2678362022 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559b4a50f840 con 0x559b4a5c5f60 2023-09-10T01:06:34.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f18db7fe700 10 client.26675 handle_command_reply: tid=0 2023-09-10T01:06:34.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 client.26675 shutdown 2023-09-10T01:06:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ac710 msgr2=0x559b4a5bbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ac710 0x559b4a5bbfe0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009fd0 tx=0x7f18ec0ca440 comp rx=0 tx=0).stop 2023-09-10T01:06:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 shutdown_connections 2023-09-10T01:06:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a5c5f60 0x559b4a5c8350 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f18db7fe700 0 client.26675 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ac710 0x559b4a5bbfe0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 --2- 172.21.15.3:0/2678362022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ad0d0 0x559b4a5bc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 >> 172.21.15.3:0/2678362022 conn(0x559b4a51d030 msgr2=0x559b4a5a27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.741+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 shutdown_connections 2023-09-10T01:06:34.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.742+0000 7f1916202b80 1 -- 172.21.15.3:0/2678362022 wait complete. 2023-09-10T01:06:34.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.742+0000 7f1916202b80 20 client.26675 trim_cache size 0 max 16384 2023-09-10T01:06:34.761 INFO:tasks.workunit.client.0.smithi003.stdout: 1 248613 9.56 MB/sec execute 16 sec latency 14.042 ms 2023-09-10T01:06:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.757+0000 7f1916202b80 1 Processor -- start 2023-09-10T01:06:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.757+0000 7f1916202b80 1 -- start start 2023-09-10T01:06:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1916202b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1916202b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1916202b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5b25b0 0x559b4a5b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1916202b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a5b4270 con 0x559b4a5ac710 2023-09-10T01:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1916202b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a5b2fe0 con 0x559b4a5ad0d0 2023-09-10T01:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1916202b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a5a4b80 con 0x559b4a5b25b0 2023-09-10T01:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34416/0 (socket says 172.21.15.3:34416) 2023-09-10T01:06:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 -- 172.21.15.3:0/2693311372 learned_addr learned my addr 172.21.15.3:0/2693311372 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fb7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1900d9c700 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5b25b0 0x559b4a5b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 -- 172.21.15.3:0/2693311372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5b25b0 msgr2=0x559b4a5b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5b25b0 0x559b4a5b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 -- 172.21.15.3:0/2693311372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 msgr2=0x559b4a5aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f18fbfff700 1 -- 172.21.15.3:0/2693311372 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b4a34d1f0 con 0x559b4a5ac710 2023-09-10T01:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.758+0000 7f1900d9c700 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5b25b0 0x559b4a5b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f18fbfff700 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5b61e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f18ec09a220 tx=0x7f18ec096680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8326c9adc26c66b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f18e2ffd700 1 -- 172.21.15.3:0/2693311372 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec094ea0 con 0x559b4a5ac710 2023-09-10T01:06:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f18e2ffd700 1 -- 172.21.15.3:0/2693311372 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18ec0b8180 con 0x559b4a5ac710 2023-09-10T01:06:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f18e2ffd700 1 -- 172.21.15.3:0/2693311372 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec09a750 con 0x559b4a5ac710 2023-09-10T01:06:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f1916202b80 1 -- 172.21.15.3:0/2693311372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 msgr2=0x559b4a5b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f1916202b80 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5b61e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f18ec09a220 tx=0x7f18ec096680 comp rx=0 tx=0).stop 2023-09-10T01:06:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f1916202b80 1 -- 172.21.15.3:0/2693311372 shutdown_connections 2023-09-10T01:06:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f1916202b80 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5b25b0 0x559b4a5b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f1916202b80 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5ac710 0x559b4a5b61e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.759+0000 7f1916202b80 1 --2- 172.21.15.3:0/2693311372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.760+0000 7f1916202b80 1 -- 172.21.15.3:0/2693311372 >> 172.21.15.3:0/2693311372 conn(0x559b4a51d030 msgr2=0x559b4a5b5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.760+0000 7f1916202b80 1 -- 172.21.15.3:0/2693311372 shutdown_connections 2023-09-10T01:06:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.760+0000 7f1916202b80 1 -- 172.21.15.3:0/2693311372 wait complete. 2023-09-10T01:06:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.760+0000 7f1916202b80 1 -- start start 2023-09-10T01:06:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.760+0000 7f1916202b80 1 Processor -- start 2023-09-10T01:06:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1916202b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ac710 0x559b4a5b9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1916202b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5ba090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1916202b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5ba5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1916202b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a618d80 con 0x559b4a5b25b0 2023-09-10T01:06:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1916202b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a618ec0 con 0x559b4a5ad0d0 2023-09-10T01:06:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1916202b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4a619080 con 0x559b4a5ac710 2023-09-10T01:06:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1916202b80 10 client.0 _ll_register_callbacks cb 0x7f18ec099bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ac710 0x559b4a5b9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ac710 0x559b4a5b9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:32956/0 (socket says 172.21.15.3:32956) 2023-09-10T01:06:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 -- 172.21.15.3:0/3746910212 learned_addr learned my addr 172.21.15.3:0/3746910212 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5ba090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f1900d9c700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5ba5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 -- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 msgr2=0x559b4a5ba090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5ba090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 -- 172.21.15.3:0/3746910212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 msgr2=0x559b4a5ba5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5ba5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fbfff700 1 -- 172.21.15.3:0/3746910212 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ec07a070 con 0x559b4a5ac710 2023-09-10T01:06:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.761+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5ba090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f18fbfff700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ac710 0x559b4a5b9b50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0985e0 tx=0x7f18ec005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37f113c208ccb8c2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f18db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f18db7fe700 1 -- 172.21.15.3:0/3746910212 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec094ea0 con 0x559b4a5ac710 2023-09-10T01:06:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18ec0a2270 con 0x559b4a5ac710 2023-09-10T01:06:34.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f18db7fe700 1 -- 172.21.15.3:0/3746910212 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f18ec004e10 con 0x559b4a5ac710 2023-09-10T01:06:34.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f18db7fe700 1 -- 172.21.15.3:0/3746910212 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18ec0960d0 con 0x559b4a5ac710 2023-09-10T01:06:34.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f1916202b80 10 client.26599 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559b4a325250 con 0x559b4a5ac710 2023-09-10T01:06:34.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f18db7fe700 1 -- 172.21.15.3:0/3746910212 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18ec0a2270 con 0x559b4a5ac710 2023-09-10T01:06:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f18db7fe700 1 -- 172.21.15.3:0/3746910212 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f18ec099280 con 0x559b4a5ac710 2023-09-10T01:06:34.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f1916202b80 10 client.26599 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f1916202b80 10 client.26599 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f1916202b80 20 client.26599 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:34.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.762+0000 7f1916202b80 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a61a2a0 0x559b4a61c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.763+0000 7f1916202b80 4 client.26599 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:06:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.763+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559b4a4f38c0 con 0x559b4a61a2a0 2023-09-10T01:06:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.763+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a61a2a0 0x559b4a61c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:34.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.763+0000 7f18fb7fe700 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a61a2a0 0x559b4a61c690 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0cf942bd4d61b46 server_cookie=290130bb7648b60f in_seq=0 out_seq=0 2023-09-10T01:06:34.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.763+0000 7f18db7fe700 10 client.26599 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f18db7fe700 1 -- 172.21.15.3:0/3746910212 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559b4a4f38c0 con 0x559b4a61a2a0 2023-09-10T01:06:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f18db7fe700 10 client.26599 handle_command_reply: tid=0 2023-09-10T01:06:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 client.26599 shutdown 2023-09-10T01:06:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ac710 msgr2=0x559b4a5b9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ac710 0x559b4a5b9b50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0985e0 tx=0x7f18ec005c00 comp rx=0 tx=0).stop 2023-09-10T01:06:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 shutdown_connections 2023-09-10T01:06:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b4a61a2a0 0x559b4a61c690 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b4a5ac710 0x559b4a5b9b50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f18db7fe700 0 client.26599 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:34.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b4a5b25b0 0x559b4a5ba5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 --2- 172.21.15.3:0/3746910212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b4a5ad0d0 0x559b4a5ba090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 >> 172.21.15.3:0/3746910212 conn(0x559b4a51d030 msgr2=0x559b4a5a72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:34.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 shutdown_connections 2023-09-10T01:06:34.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.866+0000 7f1916202b80 1 -- 172.21.15.3:0/3746910212 wait complete. 2023-09-10T01:06:34.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.867+0000 7f1916202b80 20 client.26599 trim_cache size 0 max 16384 2023-09-10T01:06:34.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:06:34.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:06:34.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:06:34.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:06:34.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.867+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18e40a0830 msgr2=0x7f18e40a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.867+0000 7f19027ff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18e40a0830 0x7f18e40a2cf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009f50 tx=0x7f18f4003aa0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.867+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 msgr2=0x7f18fc196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:34.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.867+0000 7f19027ff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 0x7f18fc196610 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00b580 tx=0x7f18ec00f690 comp rx=0 tx=0).stop 2023-09-10T01:06:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 shutdown_connections 2023-09-10T01:06:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18e40a0830 0x7f18e40a2cf0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18fc1005c0 0x7f18fc196610 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18fc106680 0x7f18fc19afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 --2- 172.21.15.3:0/2206836692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18fc100f80 0x7f18fc196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 >> 172.21.15.3:0/2206836692 conn(0x7f18fc0fc0b0 msgr2=0x7f18fc0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 shutdown_connections 2023-09-10T01:06:34.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:34.868+0000 7f19027ff700 1 -- 172.21.15.3:0/2206836692 wait complete. 2023-09-10T01:06:35.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249822 9.32 MB/sec execute 17 sec latency 44.686 ms 2023-09-10T01:06:36.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:36 smithi003 ceph-mon[97654]: pgmap v104: 129 pgs: 129 active+clean; 76 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 392 op/s 2023-09-10T01:06:36.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:36 smithi136 ceph-mon[133708]: pgmap v104: 129 pgs: 129 active+clean; 76 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 392 op/s 2023-09-10T01:06:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:36 smithi092 ceph-mon[134067]: pgmap v104: 129 pgs: 129 active+clean; 76 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 392 op/s 2023-09-10T01:06:36.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 252174 9.56 MB/sec execute 18 sec latency 11.206 ms 2023-09-10T01:06:37.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 253385 9.34 MB/sec execute 19 sec latency 46.468 ms 2023-09-10T01:06:38.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:38 smithi003 ceph-mon[97654]: pgmap v105: 129 pgs: 129 active+clean; 76 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 217 op/s 2023-09-10T01:06:38.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:38 smithi136 ceph-mon[133708]: pgmap v105: 129 pgs: 129 active+clean; 76 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 217 op/s 2023-09-10T01:06:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:38 smithi092 ceph-mon[134067]: pgmap v105: 129 pgs: 129 active+clean; 76 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 217 op/s 2023-09-10T01:06:38.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 255728 9.56 MB/sec execute 20 sec latency 28.057 ms 2023-09-10T01:06:39.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 257113 9.37 MB/sec execute 21 sec latency 12.581 ms 2023-09-10T01:06:40.309 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:40 smithi003 ceph-mon[97654]: pgmap v106: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 379 op/s 2023-09-10T01:06:40.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:40 smithi136 ceph-mon[133708]: pgmap v106: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 379 op/s 2023-09-10T01:06:40.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:40 smithi092 ceph-mon[134067]: pgmap v106: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 379 op/s 2023-09-10T01:06:40.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 259354 9.57 MB/sec execute 22 sec latency 10.579 ms 2023-09-10T01:06:41.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 260888 9.40 MB/sec execute 23 sec latency 17.904 ms 2023-09-10T01:06:42.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:42 smithi003 ceph-mon[97654]: pgmap v107: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:06:42.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:42 smithi136 ceph-mon[133708]: pgmap v107: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:06:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:42 smithi092 ceph-mon[134067]: pgmap v107: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:06:42.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 262907 9.57 MB/sec execute 24 sec latency 58.825 ms 2023-09-10T01:06:43.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 264205 9.40 MB/sec execute 25 sec latency 61.071 ms 2023-09-10T01:06:44.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:44 smithi136 ceph-mon[133708]: pgmap v108: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 264 op/s 2023-09-10T01:06:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:44 smithi092 ceph-mon[134067]: pgmap v108: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 264 op/s 2023-09-10T01:06:44.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:44 smithi003 ceph-mon[97654]: pgmap v108: 129 pgs: 129 active+clean; 81 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 264 op/s 2023-09-10T01:06:44.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 266455 9.57 MB/sec execute 26 sec latency 13.319 ms 2023-09-10T01:06:45.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 268469 9.48 MB/sec execute 27 sec latency 14.875 ms 2023-09-10T01:06:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:46 smithi136 ceph-mon[133708]: pgmap v109: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 373 op/s 2023-09-10T01:06:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:46 smithi092 ceph-mon[134067]: pgmap v109: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 373 op/s 2023-09-10T01:06:46.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:46 smithi003 ceph-mon[97654]: pgmap v109: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 373 op/s 2023-09-10T01:06:46.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 270178 9.57 MB/sec execute 28 sec latency 11.562 ms 2023-09-10T01:06:47.747 INFO:tasks.workunit.client.0.smithi003.stdout: 1 272093 9.51 MB/sec execute 29 sec latency 31.875 ms 2023-09-10T01:06:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.021+0000 7ff5621ff700 1 -- 172.21.15.3:0/2328656321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c072a30 msgr2=0x7ff55c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.021+0000 7ff5621ff700 1 --2- 172.21.15.3:0/2328656321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c072a30 0x7ff55c072e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009990 tx=0x7ff54c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.022+0000 7ff5621ff700 1 -- 172.21.15.3:0/2328656321 shutdown_connections 2023-09-10T01:06:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.022+0000 7ff5621ff700 1 --2- 172.21.15.3:0/2328656321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c10cde0 0x7ff55c10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.022+0000 7ff5621ff700 1 --2- 172.21.15.3:0/2328656321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c072a30 0x7ff55c072e10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.022+0000 7ff5621ff700 1 --2- 172.21.15.3:0/2328656321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c073350 0x7ff55c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.022+0000 7ff5621ff700 1 -- 172.21.15.3:0/2328656321 >> 172.21.15.3:0/2328656321 conn(0x7ff55c0fc0b0 msgr2=0x7ff55c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.022+0000 7ff5621ff700 1 -- 172.21.15.3:0/2328656321 shutdown_connections 2023-09-10T01:06:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.022+0000 7ff5621ff700 1 -- 172.21.15.3:0/2328656321 wait complete. 2023-09-10T01:06:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.023+0000 7ff5621ff700 1 Processor -- start 2023-09-10T01:06:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.023+0000 7ff5621ff700 1 -- start start 2023-09-10T01:06:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.023+0000 7ff5621ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c072a30 0x7ff55c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.023+0000 7ff5621ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c073350 0x7ff55c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff5621ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c10cde0 0x7ff55c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff5621ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c198db0 con 0x7ff55c072a30 2023-09-10T01:06:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff5621ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c198f20 con 0x7ff55c073350 2023-09-10T01:06:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff5621ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c1990e0 con 0x7ff55c10cde0 2023-09-10T01:06:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c073350 0x7ff55c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c072a30 0x7ff55c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c073350 0x7ff55c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36054/0 (socket says 172.21.15.3:36054) 2023-09-10T01:06:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 -- 172.21.15.3:0/856947285 learned_addr learned my addr 172.21.15.3:0/856947285 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55bfff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c10cde0 0x7ff55c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 -- 172.21.15.3:0/856947285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c10cde0 msgr2=0x7ff55c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c10cde0 0x7ff55c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 -- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c072a30 msgr2=0x7ff55c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c072a30 0x7ff55c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 -- 172.21.15.3:0/856947285 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0095f0 con 0x7ff55c073350 2023-09-10T01:06:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.024+0000 7ff55affd700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c073350 0x7ff55c19ec10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff55000e9a0 tx=0x7ff55000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.025+0000 7ff558ff9700 1 -- 172.21.15.3:0/856947285 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550009790 con 0x7ff55c073350 2023-09-10T01:06:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.025+0000 7ff558ff9700 1 -- 172.21.15.3:0/856947285 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff550010430 con 0x7ff55c073350 2023-09-10T01:06:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.025+0000 7ff558ff9700 1 -- 172.21.15.3:0/856947285 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550018680 con 0x7ff55c073350 2023-09-10T01:06:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.025+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c1993c0 con 0x7ff55c073350 2023-09-10T01:06:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.025+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c1a56c0 con 0x7ff55c073350 2023-09-10T01:06:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.028+0000 7ff558ff9700 1 -- 172.21.15.3:0/856947285 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ff5500105a0 con 0x7ff55c073350 2023-09-10T01:06:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.030+0000 7ff575bb4b80 1 Processor -- start 2023-09-10T01:06:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.030+0000 7ff575bb4b80 1 -- start start 2023-09-10T01:06:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.030+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x55577957c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576300 0x555779576780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779582f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557794cf6d0 con 0x555779576cc0 2023-09-10T01:06:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557794ccff0 con 0x555779576300 2023-09-10T01:06:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557794c88e0 con 0x55577957c1a0 2023-09-10T01:06:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff558ff9700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff5440a0830 0x7ff5440a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576300 0x555779576780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576300 0x555779576780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36068/0 (socket says 172.21.15.3:36068) 2023-09-10T01:06:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 -- 172.21.15.3:0/4191704569 learned_addr learned my addr 172.21.15.3:0/4191704569 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff558ff9700 1 -- 172.21.15.3:0/856947285 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff550014070 con 0x7ff55c073350 2023-09-10T01:06:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55bfff700 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x55577957c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55affd700 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779582f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55bfff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff5440a0830 0x7ff5440a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 -- 172.21.15.3:0/4191704569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 msgr2=0x55577957c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x55577957c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 -- 172.21.15.3:0/4191704569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 msgr2=0x555779582f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779582f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.031+0000 7ff55b7fe700 1 -- 172.21.15.3:0/4191704569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557792f4d80 con 0x555779576300 2023-09-10T01:06:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.032+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576300 0x555779576780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff54c005ea0 tx=0x7ff54c005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d8b2cf942739e9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.032+0000 7ff55bfff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff5440a0830 0x7ff5440a2cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff554009ec0 tx=0x7ff55400b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.032+0000 7ff541ffb700 1 -- 172.21.15.3:0/4191704569 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c01d070 con 0x555779576300 2023-09-10T01:06:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.032+0000 7ff541ffb700 1 -- 172.21.15.3:0/4191704569 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff54c01e040 con 0x555779576300 2023-09-10T01:06:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.032+0000 7ff541ffb700 1 -- 172.21.15.3:0/4191704569 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c0049c0 con 0x555779576300 2023-09-10T01:06:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 -- 172.21.15.3:0/4191704569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576300 msgr2=0x555779576780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576300 0x555779576780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff54c005ea0 tx=0x7ff54c005ed0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 -- 172.21.15.3:0/4191704569 shutdown_connections 2023-09-10T01:06:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x55577957c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779582f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/4191704569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576300 0x555779576780 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 -- 172.21.15.3:0/4191704569 >> 172.21.15.3:0/4191704569 conn(0x5557794eb030 msgr2=0x555779571560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 -- 172.21.15.3:0/4191704569 shutdown_connections 2023-09-10T01:06:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 -- 172.21.15.3:0/4191704569 wait complete. 2023-09-10T01:06:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.033+0000 7ff575bb4b80 1 -- start start 2023-09-10T01:06:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 1 Processor -- start 2023-09-10T01:06:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x555779582ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779580000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x555779580540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555779466b10 con 0x555779576cc0 2023-09-10T01:06:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555779466810 con 0x55577957c1a0 2023-09-10T01:06:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577934a8c0 con 0x555779576300 2023-09-10T01:06:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.034+0000 7ff575bb4b80 10 client.0 _ll_register_callbacks cb 0x5557793b0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x555779582ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x555779582ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:35578/0 (socket says 172.21.15.3:35578) 2023-09-10T01:06:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2445601462 learned_addr learned my addr 172.21.15.3:0/2445601462 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55affd700 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779580000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55bfff700 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x555779580540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 msgr2=0x555779580540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x555779580540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2445601462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 msgr2=0x555779580000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779580000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2445601462 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c10ff10 con 0x555779576300 2023-09-10T01:06:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x555779582ce0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff54c000c00 tx=0x7ff54c00fb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cb1049c408984f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.035+0000 7ff53affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff53affd700 1 -- 172.21.15.3:0/2445601462 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c01d040 con 0x555779576300 2023-09-10T01:06:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555779420490 con 0x555779576300 2023-09-10T01:06:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff53affd700 1 -- 172.21.15.3:0/2445601462 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff54c005250 con 0x555779576300 2023-09-10T01:06:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff53affd700 1 -- 172.21.15.3:0/2445601462 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54c005680 con 0x555779576300 2023-09-10T01:06:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 10 client.26611 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5557792ddc00 con 0x555779576300 2023-09-10T01:06:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff53affd700 1 -- 172.21.15.3:0/2445601462 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555779420490 con 0x555779576300 2023-09-10T01:06:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff53affd700 1 -- 172.21.15.3:0/2445601462 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff54c004cc0 con 0x555779576300 2023-09-10T01:06:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 10 client.26611 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 10 client.26611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 20 client.26611 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55577958ffb0 0x5557795923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 4 client.26611 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.036+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557794dd840 con 0x55577958ffb0 2023-09-10T01:06:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.037+0000 7ff55affd700 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55577958ffb0 0x5557795923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.037+0000 7ff55affd700 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55577958ffb0 0x5557795923d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fea9e2fa8c45e4ea server_cookie=f47c3161392ef79a in_seq=0 out_seq=0 2023-09-10T01:06:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.037+0000 7ff53affd700 10 client.26611 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff53affd700 1 -- 172.21.15.3:0/2445601462 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5557794dd840 con 0x55577958ffb0 2023-09-10T01:06:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff53affd700 10 client.26611 handle_command_reply: tid=0 2023-09-10T01:06:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 client.26611 shutdown 2023-09-10T01:06:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 msgr2=0x555779582ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x555779582ce0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff54c000c00 tx=0x7ff54c00fb10 comp rx=0 tx=0).stop 2023-09-10T01:06:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 shutdown_connections 2023-09-10T01:06:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55577958ffb0 0x5557795923d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x555779582ce0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x555779580540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2445601462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779580000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff53affd700 0 client.26611 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 >> 172.21.15.3:0/2445601462 conn(0x5557794eb030 msgr2=0x5557795707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 shutdown_connections 2023-09-10T01:06:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2445601462 wait complete. 2023-09-10T01:06:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.038+0000 7ff575bb4b80 20 client.26611 trim_cache size 0 max 16384 2023-09-10T01:06:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 Processor -- start 2023-09-10T01:06:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 -- start start 2023-09-10T01:06:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576300 0x55577957b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576cc0 0x555779587530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x555779575a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557794dd840 con 0x555779576300 2023-09-10T01:06:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555779420490 con 0x555779576cc0 2023-09-10T01:06:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577956b8a0 con 0x55577957c1a0 2023-09-10T01:06:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.055+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576300 0x55577957b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x555779575a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576300 0x55577957b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43948/0 (socket says 172.21.15.3:43948) 2023-09-10T01:06:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55b7fe700 1 -- 172.21.15.3:0/212817975 learned_addr learned my addr 172.21.15.3:0/212817975 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55affd700 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576cc0 0x555779587530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55bfff700 1 -- 172.21.15.3:0/212817975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576cc0 msgr2=0x555779587530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55bfff700 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576cc0 0x555779587530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55bfff700 1 -- 172.21.15.3:0/212817975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576300 msgr2=0x55577957b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55bfff700 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576300 0x55577957b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55bfff700 1 -- 172.21.15.3:0/212817975 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55577931b1f0 con 0x55577957c1a0 2023-09-10T01:06:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576300 0x55577957b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55affd700 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576cc0 0x555779587530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff55bfff700 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x555779575a00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff5540089f0 tx=0x7ff554010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d33d52ddf124f757 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff5427fc700 1 -- 172.21.15.3:0/212817975 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554022050 con 0x55577957c1a0 2023-09-10T01:06:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.056+0000 7ff5427fc700 1 -- 172.21.15.3:0/212817975 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff554023040 con 0x55577957c1a0 2023-09-10T01:06:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff5427fc700 1 -- 172.21.15.3:0/212817975 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55400e070 con 0x55577957c1a0 2023-09-10T01:06:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 -- 172.21.15.3:0/212817975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 msgr2=0x555779575a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x555779575a00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff5540089f0 tx=0x7ff554010000 comp rx=0 tx=0).stop 2023-09-10T01:06:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 -- 172.21.15.3:0/212817975 shutdown_connections 2023-09-10T01:06:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55577957c1a0 0x555779575a00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576300 0x55577957b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/212817975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555779576cc0 0x555779587530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 -- 172.21.15.3:0/212817975 >> 172.21.15.3:0/212817975 conn(0x5557794eb030 msgr2=0x5557795752f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:48.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 -- 172.21.15.3:0/212817975 shutdown_connections 2023-09-10T01:06:48.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.057+0000 7ff575bb4b80 1 -- 172.21.15.3:0/212817975 wait complete. 2023-09-10T01:06:48.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.058+0000 7ff575bb4b80 1 -- start start 2023-09-10T01:06:48.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.058+0000 7ff575bb4b80 1 Processor -- start 2023-09-10T01:06:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x5557795852f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779585830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff575bb4b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x5557795e2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557795e3300 con 0x555779576cc0 2023-09-10T01:06:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557795e3440 con 0x55577957c1a0 2023-09-10T01:06:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff575bb4b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557795e35b0 con 0x555779576300 2023-09-10T01:06:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff575bb4b80 10 client.0 _ll_register_callbacks cb 0x7ff550010b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x5557795852f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x5557795852f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:35600/0 (socket says 172.21.15.3:35600) 2023-09-10T01:06:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2174791713 learned_addr learned my addr 172.21.15.3:0/2174791713 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55affd700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779585830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55bfff700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x5557795e2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 msgr2=0x5557795e2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x5557795e2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2174791713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 msgr2=0x555779585830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779585830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55b7fe700 1 -- 172.21.15.3:0/2174791713 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557792ddc00 con 0x555779576300 2023-09-10T01:06:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.059+0000 7ff55affd700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779585830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff55b7fe700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x5557795852f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00f8b0 tx=0x7ff54c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50fc230c6ed2e0c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff53affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff53affd700 1 -- 172.21.15.3:0/2174791713 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c022db0 con 0x555779576300 2023-09-10T01:06:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff54c01e040 con 0x555779576300 2023-09-10T01:06:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff53affd700 1 -- 172.21.15.3:0/2174791713 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff54c00e3e0 con 0x555779576300 2023-09-10T01:06:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff53affd700 1 -- 172.21.15.3:0/2174791713 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54c0227e0 con 0x555779576300 2023-09-10T01:06:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff575bb4b80 10 client.26617 fetch_fsmap learned FSMap version 23 2023-09-10T01:06:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7ff55c1a56c0 con 0x555779576300 2023-09-10T01:06:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.060+0000 7ff53affd700 1 -- 172.21.15.3:0/2174791713 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff54c01e040 con 0x555779576300 2023-09-10T01:06:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff53affd700 1 -- 172.21.15.3:0/2174791713 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff54c0050d0 con 0x555779576300 2023-09-10T01:06:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff575bb4b80 10 client.26617 fetch_fsmap finished waiting for FSMap version 23 2023-09-10T01:06:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff575bb4b80 10 client.26617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff575bb4b80 20 client.26617 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5557795e4940 0x5557795e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff575bb4b80 4 client.26617 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:06:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5557794c18c0 con 0x5557795e4940 2023-09-10T01:06:48.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff55affd700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5557795e4940 0x5557795e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:48.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff53affd700 10 client.26617 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:48.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.061+0000 7ff55affd700 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5557795e4940 0x5557795e6d60 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=689af9b08a933210 server_cookie=d7d40d0ae5fbd40e in_seq=0 out_seq=0 2023-09-10T01:06:48.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff53affd700 1 -- 172.21.15.3:0/2174791713 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5557794c18c0 con 0x5557795e4940 2023-09-10T01:06:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff53affd700 10 client.26617 handle_command_reply: tid=0 2023-09-10T01:06:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 client.26617 shutdown 2023-09-10T01:06:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 msgr2=0x5557795852f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x5557795852f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00f8b0 tx=0x7ff54c00ed70 comp rx=0 tx=0).stop 2023-09-10T01:06:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 shutdown_connections 2023-09-10T01:06:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5557795e4940 0x5557795e6d60 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555779576300 0x5557795852f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55577957c1a0 0x5557795e2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 --2- 172.21.15.3:0/2174791713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555779576cc0 0x555779585830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff53affd700 0 client.26617 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.183+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 >> 172.21.15.3:0/2174791713 conn(0x5557794eb030 msgr2=0x555779575b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.184+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 shutdown_connections 2023-09-10T01:06:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.184+0000 7ff575bb4b80 1 -- 172.21.15.3:0/2174791713 wait complete. 2023-09-10T01:06:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.184+0000 7ff575bb4b80 20 client.26617 trim_cache size 0 max 16384 2023-09-10T01:06:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:06:48.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:06:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:06:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:06:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff5440a0830 msgr2=0x7ff5440a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff5440a0830 0x7ff5440a2cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff554009ec0 tx=0x7ff55400b040 comp rx=0 tx=0).stop 2023-09-10T01:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c073350 msgr2=0x7ff55c19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c073350 0x7ff55c19ec10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff55000e9a0 tx=0x7ff55000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 shutdown_connections 2023-09-10T01:06:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff5440a0830 0x7ff5440a2cf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c10cde0 0x7ff55c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c072a30 0x7ff55c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 --2- 172.21.15.3:0/856947285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff55c073350 0x7ff55c19ec10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 >> 172.21.15.3:0/856947285 conn(0x7ff55c0fc0b0 msgr2=0x7ff55c10ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 shutdown_connections 2023-09-10T01:06:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:06:48.185+0000 7ff5621ff700 1 -- 172.21.15.3:0/856947285 wait complete. 2023-09-10T01:06:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:48 smithi136 ceph-mon[133708]: pgmap v110: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 270 op/s 2023-09-10T01:06:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:48 smithi092 ceph-mon[134067]: pgmap v110: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 270 op/s 2023-09-10T01:06:48.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:48 smithi003 ceph-mon[97654]: pgmap v110: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 270 op/s 2023-09-10T01:06:48.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 273585 9.57 MB/sec execute 30 sec latency 66.515 ms 2023-09-10T01:06:49.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 275642 9.51 MB/sec execute 31 sec latency 28.873 ms 2023-09-10T01:06:50.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:50 smithi136 ceph-mon[133708]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 447 op/s 2023-09-10T01:06:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:50 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:06:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:50 smithi092 ceph-mon[134067]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 447 op/s 2023-09-10T01:06:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:50 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:06:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:50 smithi003 ceph-mon[97654]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 447 op/s 2023-09-10T01:06:50.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:50 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:06:50.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 277085 9.57 MB/sec execute 32 sec latency 184.833 ms 2023-09-10T01:06:51.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 279123 9.50 MB/sec execute 33 sec latency 16.807 ms 2023-09-10T01:06:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:52 smithi136 ceph-mon[133708]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 286 op/s 2023-09-10T01:06:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:52 smithi092 ceph-mon[134067]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 286 op/s 2023-09-10T01:06:52.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:52 smithi003 ceph-mon[97654]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 286 op/s 2023-09-10T01:06:52.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 280834 9.57 MB/sec execute 34 sec latency 11.818 ms 2023-09-10T01:06:53.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 282695 9.50 MB/sec execute 35 sec latency 52.900 ms 2023-09-10T01:06:54.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:06:54.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.273+0000 7fc4c23ff700 1 -- 172.21.15.3:0/881618658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc072a30 msgr2=0x7fc4bc072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.273+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/881618658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc072a30 0x7fc4bc072e10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009990 tx=0x7fc4ac009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.273+0000 7fc4c23ff700 1 -- 172.21.15.3:0/881618658 shutdown_connections 2023-09-10T01:06:54.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.273+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/881618658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4bc10ce80 0x7fc4bc10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.273+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/881618658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc072a30 0x7fc4bc072e10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.273+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/881618658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4bc073350 0x7fc4bc10c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.273+0000 7fc4c23ff700 1 -- 172.21.15.3:0/881618658 >> 172.21.15.3:0/881618658 conn(0x7fc4bc0fc170 msgr2=0x7fc4bc0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.274+0000 7fc4c23ff700 1 -- 172.21.15.3:0/881618658 shutdown_connections 2023-09-10T01:06:54.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.274+0000 7fc4c23ff700 1 -- 172.21.15.3:0/881618658 wait complete. 2023-09-10T01:06:54.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.275+0000 7fc4c23ff700 1 Processor -- start 2023-09-10T01:06:54.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.275+0000 7fc4c23ff700 1 -- start start 2023-09-10T01:06:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.275+0000 7fc4c23ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4bc072a30 0x7fc4bc198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.275+0000 7fc4c23ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc073350 0x7fc4bc198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.275+0000 7fc4c23ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4bc10ce80 0x7fc4bc19d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c23ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc19d920 con 0x7fc4bc10ce80 2023-09-10T01:06:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c23ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc19da90 con 0x7fc4bc073350 2023-09-10T01:06:54.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c23ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc19dc50 con 0x7fc4bc072a30 2023-09-10T01:06:54.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc073350 0x7fc4bc198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c099c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4bc10ce80 0x7fc4bc19d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4bbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4bc072a30 0x7fc4bc198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc073350 0x7fc4bc198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36138/0 (socket says 172.21.15.3:36138) 2023-09-10T01:06:54.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4bb7fe700 1 -- 172.21.15.3:0/806675372 learned_addr learned my addr 172.21.15.3:0/806675372 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:54.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c099c700 1 -- 172.21.15.3:0/806675372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4bc072a30 msgr2=0x7fc4bc198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c099c700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4bc072a30 0x7fc4bc198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c099c700 1 -- 172.21.15.3:0/806675372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc073350 msgr2=0x7fc4bc198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c099c700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc073350 0x7fc4bc198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c099c700 1 -- 172.21.15.3:0/806675372 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0095f0 con 0x7fc4bc10ce80 2023-09-10T01:06:54.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4bc072a30 0x7fc4bc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:54.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.276+0000 7fc4c099c700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4bc10ce80 0x7fc4bc19d350 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400efc0 tx=0x7fc4b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.277+0000 7fc4b97fa700 1 -- 172.21.15.3:0/806675372 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4018070 con 0x7fc4bc10ce80 2023-09-10T01:06:54.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.277+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc19ded0 con 0x7fc4bc10ce80 2023-09-10T01:06:54.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.277+0000 7fc4b97fa700 1 -- 172.21.15.3:0/806675372 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4b400f040 con 0x7fc4bc10ce80 2023-09-10T01:06:54.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.277+0000 7fc4b97fa700 1 -- 172.21.15.3:0/806675372 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400cd40 con 0x7fc4bc10ce80 2023-09-10T01:06:54.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.277+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc1a57e0 con 0x7fc4bc10ce80 2023-09-10T01:06:54.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.280+0000 7fc4b97fa700 1 -- 172.21.15.3:0/806675372 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fc4b4003a00 con 0x7fc4bc10ce80 2023-09-10T01:06:54.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.281+0000 7fc4d5dabb80 1 Processor -- start 2023-09-10T01:06:54.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4d5dabb80 1 -- start start 2023-09-10T01:06:54.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda37710 0x560fbda37af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4b97fa700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc4a40a07b0 0x7fc4a40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda38030 0x560fbda41900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda44230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbd990730 con 0x560fbda38030 2023-09-10T01:06:54.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbd98e050 con 0x560fbda41e40 2023-09-10T01:06:54.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbd989940 con 0x560fbda37710 2023-09-10T01:06:54.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4b97fa700 1 -- 172.21.15.3:0/806675372 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc4b40cbf00 con 0x7fc4bc10ce80 2023-09-10T01:06:54.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda37710 0x560fbda37af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4c099c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda44230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda37710 0x560fbda37af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:35624/0 (socket says 172.21.15.3:35624) 2023-09-10T01:06:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4c099c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda44230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36152/0 (socket says 172.21.15.3:36152) 2023-09-10T01:06:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.282+0000 7fc4bb7fe700 1 -- 172.21.15.3:0/2728705969 learned_addr learned my addr 172.21.15.3:0/2728705969 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4bb7fe700 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda38030 0x560fbda41900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc4a40a07b0 0x7fc4a40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4c099c700 1 -- 172.21.15.3:0/2728705969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda37710 msgr2=0x560fbda37af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4c099c700 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda37710 0x560fbda37af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4c099c700 1 -- 172.21.15.3:0/2728705969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda38030 msgr2=0x560fbda41900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4c099c700 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda38030 0x560fbda41900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4c099c700 1 -- 172.21.15.3:0/2728705969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560fbd7b5de0 con 0x560fbda41e40 2023-09-10T01:06:54.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4c099c700 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda44230 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40c9260 tx=0x7fc4b4098f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c136485fa0bd20c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4a27fc700 1 -- 172.21.15.3:0/2728705969 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b401b320 con 0x560fbda41e40 2023-09-10T01:06:54.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc4a40a07b0 0x7fc4a40a2c70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00b2e0 tx=0x7fc4ac005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4a27fc700 1 -- 172.21.15.3:0/2728705969 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4b4099750 con 0x560fbda41e40 2023-09-10T01:06:54.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.283+0000 7fc4a27fc700 1 -- 172.21.15.3:0/2728705969 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40a1030 con 0x560fbda41e40 2023-09-10T01:06:54.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2728705969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 msgr2=0x560fbda44230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda44230 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40c9260 tx=0x7fc4b4098f10 comp rx=0 tx=0).stop 2023-09-10T01:06:54.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2728705969 shutdown_connections 2023-09-10T01:06:54.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda37710 0x560fbda37af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda44230 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2728705969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda38030 0x560fbda41900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2728705969 >> 172.21.15.3:0/2728705969 conn(0x560fbd9ac090 msgr2=0x560fbda32bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2728705969 shutdown_connections 2023-09-10T01:06:54.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.284+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2728705969 wait complete. 2023-09-10T01:06:54.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.285+0000 7fc4d5dabb80 1 -- start start 2023-09-10T01:06:54.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.285+0000 7fc4d5dabb80 1 Processor -- start 2023-09-10T01:06:54.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda418e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda38030 0x560fbda3eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda41e40 0x560fbda3f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbd927b70 con 0x560fbda37710 2023-09-10T01:06:54.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbd927870 con 0x560fbda38030 2023-09-10T01:06:54.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbd80b920 con 0x560fbda41e40 2023-09-10T01:06:54.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4d5dabb80 10 client.0 _ll_register_callbacks cb 0x560fbd871a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:54.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda38030 0x560fbda3eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4c099c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda41e40 0x560fbda3f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4bbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4c099c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda41e40 0x560fbda3f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:35636/0 (socket says 172.21.15.3:35636) 2023-09-10T01:06:54.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda38030 0x560fbda3eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36154/0 (socket says 172.21.15.3:36154) 2023-09-10T01:06:54.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.286+0000 7fc4c099c700 1 -- 172.21.15.3:0/2846923655 learned_addr learned my addr 172.21.15.3:0/2846923655 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:54.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4c099c700 1 -- 172.21.15.3:0/2846923655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda38030 msgr2=0x560fbda3eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4c099c700 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda38030 0x560fbda3eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4c099c700 1 -- 172.21.15.3:0/2846923655 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 msgr2=0x560fbda418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4c099c700 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4c099c700 1 -- 172.21.15.3:0/2846923655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc1a57e0 con 0x560fbda41e40 2023-09-10T01:06:54.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4c099c700 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda41e40 0x560fbda3f2a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40a0b00 tx=0x7fc4b40092e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86b75c1fc5271555 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc49b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:54.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc49b7fe700 1 -- 172.21.15.3:0/2846923655 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4010220 con 0x560fbda41e40 2023-09-10T01:06:54.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560fbd8e14f0 con 0x560fbda41e40 2023-09-10T01:06:54.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc49b7fe700 1 -- 172.21.15.3:0/2846923655 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc4b40a1030 con 0x560fbda41e40 2023-09-10T01:06:54.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc49b7fe700 1 -- 172.21.15.3:0/2846923655 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4b4099050 con 0x560fbda41e40 2023-09-10T01:06:54.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4d5dabb80 10 client.26629 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:54.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.287+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560fbd79ec60 con 0x560fbda41e40 2023-09-10T01:06:54.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc49b7fe700 1 -- 172.21.15.3:0/2846923655 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560fbd8e14f0 con 0x560fbda41e40 2023-09-10T01:06:54.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc49b7fe700 1 -- 172.21.15.3:0/2846923655 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc4b4098540 con 0x560fbda41e40 2023-09-10T01:06:54.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc4d5dabb80 10 client.26629 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:54.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc4d5dabb80 10 client.26629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:54.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc4d5dabb80 20 client.26629 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:54.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbda51550 0x560fbda53970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc4d5dabb80 4 client.26629 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:54.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560fbd99e8a0 con 0x560fbda51550 2023-09-10T01:06:54.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.288+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbda51550 0x560fbda53970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.289+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbda51550 0x560fbda53970 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f791099cc7f6584 server_cookie=250a90f99d3e3dea in_seq=0 out_seq=0 2023-09-10T01:06:54.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.289+0000 7fc49b7fe700 10 client.26629 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:54.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc49b7fe700 1 -- 172.21.15.3:0/2846923655 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560fbd99e8a0 con 0x560fbda51550 2023-09-10T01:06:54.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc49b7fe700 10 client.26629 handle_command_reply: tid=0 2023-09-10T01:06:54.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 client.26629 shutdown 2023-09-10T01:06:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda41e40 msgr2=0x560fbda3f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda41e40 0x560fbda3f2a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40a0b00 tx=0x7fc4b40092e0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 shutdown_connections 2023-09-10T01:06:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbda51550 0x560fbda53970 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda41e40 0x560fbda3f2a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/2846923655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda38030 0x560fbda3eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc49b7fe700 0 client.26629 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 >> 172.21.15.3:0/2846923655 conn(0x560fbd9ac090 msgr2=0x560fbda32780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 shutdown_connections 2023-09-10T01:06:54.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/2846923655 wait complete. 2023-09-10T01:06:54.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.290+0000 7fc4d5dabb80 20 client.26629 trim_cache size 0 max 16384 2023-09-10T01:06:54.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.308+0000 7fc4d5dabb80 1 Processor -- start 2023-09-10T01:06:54.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4d5dabb80 1 -- start start 2023-09-10T01:06:54.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda39e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda3a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda3bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbda389c0 con 0x560fbda37710 2023-09-10T01:06:54.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbda340b0 con 0x560fbda41e40 2023-09-10T01:06:54.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbda2a400 con 0x560fbda38030 2023-09-10T01:06:54.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda39e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda39e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43974/0 (socket says 172.21.15.3:43974) 2023-09-10T01:06:54.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3245387806 learned_addr learned my addr 172.21.15.3:0/3245387806 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:54.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4c099c700 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda3bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bb7fe700 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda3a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3245387806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 msgr2=0x560fbda3a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda3a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3245387806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 msgr2=0x560fbda3bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda3bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.309+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3245387806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560fbd7dc250 con 0x560fbda37710 2023-09-10T01:06:54.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.310+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda39e40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009990 tx=0x7fc4ac020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e8e74419ad7ce server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.310+0000 7fc4a2ffd700 1 -- 172.21.15.3:0/3245387806 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac02b040 con 0x560fbda37710 2023-09-10T01:06:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.310+0000 7fc4a2ffd700 1 -- 172.21.15.3:0/3245387806 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4ac02c040 con 0x560fbda37710 2023-09-10T01:06:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.310+0000 7fc4a2ffd700 1 -- 172.21.15.3:0/3245387806 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac00f580 con 0x560fbda37710 2023-09-10T01:06:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3245387806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 msgr2=0x560fbda39e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda39e40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009990 tx=0x7fc4ac020000 comp rx=0 tx=0).stop 2023-09-10T01:06:54.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3245387806 shutdown_connections 2023-09-10T01:06:54.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda41e40 0x560fbda3bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda37710 0x560fbda39e40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3245387806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda3a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3245387806 >> 172.21.15.3:0/3245387806 conn(0x560fbd9ac090 msgr2=0x560fbda4dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3245387806 shutdown_connections 2023-09-10T01:06:54.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3245387806 wait complete. 2023-09-10T01:06:54.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.311+0000 7fc4d5dabb80 1 -- start start 2023-09-10T01:06:54.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.312+0000 7fc4d5dabb80 1 Processor -- start 2023-09-10T01:06:54.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.312+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda37710 0x560fbda48a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.312+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda46d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.312+0000 7fc4d5dabb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda41e40 0x560fbda47350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.312+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbda47a60 con 0x560fbda41e40 2023-09-10T01:06:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.312+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbda47bd0 con 0x560fbda37710 2023-09-10T01:06:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.312+0000 7fc4d5dabb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560fbdaa5350 con 0x560fbda38030 2023-09-10T01:06:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4d5dabb80 10 client.0 _ll_register_callbacks cb 0x7fc4b0009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda37710 0x560fbda48a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bb7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda46d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda37710 0x560fbda48a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36170/0 (socket says 172.21.15.3:36170) 2023-09-10T01:06:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3293479762 learned_addr learned my addr 172.21.15.3:0/3293479762 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4c099c700 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda41e40 0x560fbda47350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3293479762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 msgr2=0x560fbda46d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda46d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3293479762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda41e40 msgr2=0x560fbda47350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda41e40 0x560fbda47350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 -- 172.21.15.3:0/3293479762 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc10ffb0 con 0x560fbda37710 2023-09-10T01:06:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4c099c700 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda41e40 0x560fbda47350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc4bbfff700 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda37710 0x560fbda48a60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac000c00 tx=0x7fc4ac00b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a1b57e089e1b203 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc49b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.313+0000 7fc49b7fe700 1 -- 172.21.15.3:0/3293479762 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac029070 con 0x560fbda37710 2023-09-10T01:06:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4b4099750 con 0x560fbda37710 2023-09-10T01:06:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc49b7fe700 1 -- 172.21.15.3:0/3293479762 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc4ac004a30 con 0x560fbda37710 2023-09-10T01:06:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc49b7fe700 1 -- 172.21.15.3:0/3293479762 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4ac005710 con 0x560fbda37710 2023-09-10T01:06:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 10 client.26717 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560fbd7b42b0 con 0x560fbda37710 2023-09-10T01:06:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc49b7fe700 1 -- 172.21.15.3:0/3293479762 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4b4099750 con 0x560fbda37710 2023-09-10T01:06:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc49b7fe700 1 -- 172.21.15.3:0/3293479762 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc4ac00fb20 con 0x560fbda37710 2023-09-10T01:06:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 10 client.26717 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 10 client.26717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 20 client.26717 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbdaa66f0 0x560fbdaa8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 4 client.26717 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:06:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.314+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560fbd982920 con 0x560fbdaa66f0 2023-09-10T01:06:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.315+0000 7fc4bb7fe700 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbdaa66f0 0x560fbdaa8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.315+0000 7fc4bb7fe700 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbdaa66f0 0x560fbdaa8b10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72be67889a09157 server_cookie=ae6bc62d919351c1 in_seq=0 out_seq=0 2023-09-10T01:06:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.315+0000 7fc49b7fe700 10 client.26717 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc49b7fe700 1 -- 172.21.15.3:0/3293479762 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560fbd982920 con 0x560fbdaa66f0 2023-09-10T01:06:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc49b7fe700 10 client.26717 handle_command_reply: tid=0 2023-09-10T01:06:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 client.26717 shutdown 2023-09-10T01:06:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda37710 msgr2=0x560fbda48a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda37710 0x560fbda48a60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac000c00 tx=0x7fc4ac00b410 comp rx=0 tx=0).stop 2023-09-10T01:06:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 shutdown_connections 2023-09-10T01:06:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560fbdaa66f0 0x560fbdaa8b10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560fbda41e40 0x560fbda47350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fbda37710 0x560fbda48a60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 --2- 172.21.15.3:0/3293479762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560fbda38030 0x560fbda46d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc49b7fe700 0 client.26717 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 >> 172.21.15.3:0/3293479762 conn(0x560fbd9ac090 msgr2=0x560fbda39e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 shutdown_connections 2023-09-10T01:06:54.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.316+0000 7fc4d5dabb80 1 -- 172.21.15.3:0/3293479762 wait complete. 2023-09-10T01:06:54.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4d5dabb80 20 client.26717 trim_cache size 0 max 16384 2023-09-10T01:06:54.331 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:06:54.331 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:06:54.331 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:06:54.331 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:06:54.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc4a40a07b0 msgr2=0x7fc4a40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc4a40a07b0 0x7fc4a40a2c70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00b2e0 tx=0x7fc4ac005fb0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4bc10ce80 msgr2=0x7fc4bc19d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4bc10ce80 0x7fc4bc19d350 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400efc0 tx=0x7fc4b400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 shutdown_connections 2023-09-10T01:06:54.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc4a40a07b0 0x7fc4a40a2c70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4bc072a30 0x7fc4bc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4bc10ce80 0x7fc4bc19d350 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 --2- 172.21.15.3:0/806675372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4bc073350 0x7fc4bc198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 >> 172.21.15.3:0/806675372 conn(0x7fc4bc0fc170 msgr2=0x7fc4bc10efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 shutdown_connections 2023-09-10T01:06:54.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.317+0000 7fc4c23ff700 1 -- 172.21.15.3:0/806675372 wait complete. 2023-09-10T01:06:54.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:54 smithi003 ceph-mon[97654]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 286 op/s 2023-09-10T01:06:54.336 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:06:54.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:06:54.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:54 smithi136 ceph-mon[133708]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 286 op/s 2023-09-10T01:06:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:54 smithi092 ceph-mon[134067]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 286 op/s 2023-09-10T01:06:54.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 -- 172.21.15.3:0/167322706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 msgr2=0x7fd39c10c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/167322706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 0x7fd39c10c990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009990 tx=0x7fd38c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 -- 172.21.15.3:0/167322706 shutdown_connections 2023-09-10T01:06:54.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/167322706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c10ced0 0x7fd39c10f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/167322706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd39c0729d0 0x7fd39c072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/167322706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 0x7fd39c10c990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 -- 172.21.15.3:0/167322706 >> 172.21.15.3:0/167322706 conn(0x7fd39c0fc140 msgr2=0x7fd39c0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 -- 172.21.15.3:0/167322706 shutdown_connections 2023-09-10T01:06:54.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.548+0000 7fd3a45ff700 1 -- 172.21.15.3:0/167322706 wait complete. 2023-09-10T01:06:54.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.549+0000 7fd3a45ff700 1 Processor -- start 2023-09-10T01:06:54.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.549+0000 7fd3a45ff700 1 -- start start 2023-09-10T01:06:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a45ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c0729d0 0x7fd39c19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a45ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 0x7fd39c19ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a45ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd39c10ced0 0x7fd39c198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a45ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c198ec0 con 0x7fd39c10ced0 2023-09-10T01:06:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a45ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c199030 con 0x7fd39c0729d0 2023-09-10T01:06:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a45ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c1991f0 con 0x7fd39c0732f0 2023-09-10T01:06:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c0729d0 0x7fd39c19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a1b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 0x7fd39c19ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c0729d0 0x7fd39c19e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36192/0 (socket says 172.21.15.3:36192) 2023-09-10T01:06:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 -- 172.21.15.3:0/858238003 learned_addr learned my addr 172.21.15.3:0/858238003 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a2b9c700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd39c10ced0 0x7fd39c198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 -- 172.21.15.3:0/858238003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 msgr2=0x7fd39c19ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 0x7fd39c19ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 -- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd39c10ced0 msgr2=0x7fd39c198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd39c10ced0 0x7fd39c198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a239b700 1 -- 172.21.15.3:0/858238003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c0095f0 con 0x7fd39c0729d0 2023-09-10T01:06:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a1b9a700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 0x7fd39c19ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.550+0000 7fd3a2b9c700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd39c10ced0 0x7fd39c198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:54.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.551+0000 7fd3a239b700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c0729d0 0x7fd39c19e7e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd39800e9a0 tx=0x7fd39800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.551+0000 7fd3937fe700 1 -- 172.21.15.3:0/858238003 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398009790 con 0x7fd39c0729d0 2023-09-10T01:06:54.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.551+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39c1994d0 con 0x7fd39c0729d0 2023-09-10T01:06:54.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.551+0000 7fd3937fe700 1 -- 172.21.15.3:0/858238003 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd398010430 con 0x7fd39c0729d0 2023-09-10T01:06:54.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.551+0000 7fd3937fe700 1 -- 172.21.15.3:0/858238003 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398018680 con 0x7fd39c0729d0 2023-09-10T01:06:54.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.551+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39c1a57d0 con 0x7fd39c0729d0 2023-09-10T01:06:54.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.552+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39c067480 con 0x7fd39c0729d0 2023-09-10T01:06:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.554+0000 7fd3937fe700 1 -- 172.21.15.3:0/858238003 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd3980105a0 con 0x7fd39c0729d0 2023-09-10T01:06:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.555+0000 7fd3937fe700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3880a07b0 0x7fd3880a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:54.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.556+0000 7fd3937fe700 1 -- 172.21.15.3:0/858238003 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd398014070 con 0x7fd39c0729d0 2023-09-10T01:06:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.556+0000 7fd3a1b9a700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3880a07b0 0x7fd3880a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.556+0000 7fd3a1b9a700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3880a07b0 0x7fd3880a2c70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd39c10e3d0 tx=0x7fd38c005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:54.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.557+0000 7fd3937fe700 1 -- 172.21.15.3:0/858238003 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd398095c60 con 0x7fd39c0729d0 2023-09-10T01:06:54.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 284245 9.57 MB/sec execute 36 sec latency 31.757 ms 2023-09-10T01:06:54.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.823+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd39c199d60 con 0x7fd39c0729d0 2023-09-10T01:06:54.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.824+0000 7fd3937fe700 1 -- 172.21.15.3:0/858238003 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11296 (secure 0 0 0) 0x7fd398099280 con 0x7fd39c0729d0 2023-09-10T01:06:54.825 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:06:54.826 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:06:48.069945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:06:54.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3880a07b0 msgr2=0x7fd3880a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3880a07b0 0x7fd3880a2c70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd39c10e3d0 tx=0x7fd38c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c0729d0 msgr2=0x7fd39c19e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:54.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c0729d0 0x7fd39c19e7e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd39800e9a0 tx=0x7fd39800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 shutdown_connections 2023-09-10T01:06:54.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3880a07b0 0x7fd3880a2c70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd39c10ced0 0x7fd39c198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd39c0729d0 0x7fd39c19e7e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 --2- 172.21.15.3:0/858238003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd39c0732f0 0x7fd39c19ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:54.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 >> 172.21.15.3:0/858238003 conn(0x7fd39c0fc140 msgr2=0x7fd39c10f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:54.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 shutdown_connections 2023-09-10T01:06:54.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:54.827+0000 7fd3a45ff700 1 -- 172.21.15.3:0/858238003 wait complete. 2023-09-10T01:06:54.831 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2023-09-10T01:06:54.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:06:55.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.062+0000 7fe1829ff700 1 -- 172.21.15.3:0/2809968435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c0732c0 msgr2=0x7fe17c111120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.062+0000 7fe1829ff700 1 --2- 172.21.15.3:0/2809968435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c0732c0 0x7fe17c111120 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe1700098f0 tx=0x7fe170009c00 comp rx=0 tx=0).stop 2023-09-10T01:06:55.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.063+0000 7fe1829ff700 1 -- 172.21.15.3:0/2809968435 shutdown_connections 2023-09-10T01:06:55.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.063+0000 7fe1829ff700 1 --2- 172.21.15.3:0/2809968435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe17c0729a0 0x7fe17c072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.063+0000 7fe1829ff700 1 --2- 172.21.15.3:0/2809968435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe17c111660 0x7fe17c113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.063+0000 7fe1829ff700 1 --2- 172.21.15.3:0/2809968435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c0732c0 0x7fe17c111120 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.063+0000 7fe1829ff700 1 -- 172.21.15.3:0/2809968435 >> 172.21.15.3:0/2809968435 conn(0x7fe17c0fc160 msgr2=0x7fe17c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.063+0000 7fe1829ff700 1 -- 172.21.15.3:0/2809968435 shutdown_connections 2023-09-10T01:06:55.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.063+0000 7fe1829ff700 1 -- 172.21.15.3:0/2809968435 wait complete. 2023-09-10T01:06:55.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 Processor -- start 2023-09-10T01:06:55.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 -- start start 2023-09-10T01:06:55.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe17c0729a0 0x7fe17c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe17c0732c0 0x7fe17c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c111660 0x7fe17c1a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1a1e90 con 0x7fe17c0729a0 2023-09-10T01:06:55.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1a2000 con 0x7fe17c111660 2023-09-10T01:06:55.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe1829ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1a21c0 con 0x7fe17c0732c0 2023-09-10T01:06:55.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe17c0729a0 0x7fe17c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe17b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe17c0732c0 0x7fe17c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe180f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c111660 0x7fe17c1a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe180f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c111660 0x7fe17c1a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43296/0 (socket says 172.21.15.3:43296) 2023-09-10T01:06:55.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.064+0000 7fe180f9c700 1 -- 172.21.15.3:0/279162271 learned_addr learned my addr 172.21.15.3:0/279162271 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe180f9c700 1 -- 172.21.15.3:0/279162271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe17c0732c0 msgr2=0x7fe17c19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe180f9c700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe17c0732c0 0x7fe17c19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe180f9c700 1 -- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe17c0729a0 msgr2=0x7fe17c19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe180f9c700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe17c0729a0 0x7fe17c19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe180f9c700 1 -- 172.21.15.3:0/279162271 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c0095f0 con 0x7fe17c111660 2023-09-10T01:06:55.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe180f9c700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c111660 0x7fe17c1a1850 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe17400ef30 tx=0x7fe17400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe1797fa700 1 -- 172.21.15.3:0/279162271 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174018070 con 0x7fe17c111660 2023-09-10T01:06:55.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe1797fa700 1 -- 172.21.15.3:0/279162271 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe17400f040 con 0x7fe17c111660 2023-09-10T01:06:55.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe170009520 con 0x7fe17c111660 2023-09-10T01:06:55.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe1797fa700 1 -- 172.21.15.3:0/279162271 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17400cd00 con 0x7fe17c111660 2023-09-10T01:06:55.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.065+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c1a9b00 con 0x7fe17c111660 2023-09-10T01:06:55.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.069+0000 7fe1797fa700 1 -- 172.21.15.3:0/279162271 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe174003a00 con 0x7fe17c111660 2023-09-10T01:06:55.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.070+0000 7fe19635fb80 1 Processor -- start 2023-09-10T01:06:55.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.070+0000 7fe1797fa700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1640a0830 0x7fe1640a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.070+0000 7fe19635fb80 1 -- start start 2023-09-10T01:06:55.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.070+0000 7fe1797fa700 1 -- 172.21.15.3:0/279162271 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe1740cbd80 con 0x7fe17c111660 2023-09-10T01:06:55.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe17b7fe700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1640a0830 0x7fe1640a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d4710 0x5569386d4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d5030 0x5569386de900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386e1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe19635fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55693862d730 con 0x5569386dee40 2023-09-10T01:06:55.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe19635fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55693862b050 con 0x5569386d5030 2023-09-10T01:06:55.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe19635fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556938626940 con 0x5569386d4710 2023-09-10T01:06:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d4710 0x5569386d4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe17b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d5030 0x5569386de900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d4710 0x5569386d4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40102/0 (socket says 172.21.15.3:40102) 2023-09-10T01:06:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe17bfff700 1 -- 172.21.15.3:0/2233540082 learned_addr learned my addr 172.21.15.3:0/2233540082 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.071+0000 7fe180f9c700 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe17bfff700 1 -- 172.21.15.3:0/2233540082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d5030 msgr2=0x5569386de900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe17bfff700 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d5030 0x5569386de900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe17bfff700 1 -- 172.21.15.3:0/2233540082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 msgr2=0x5569386e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe17bfff700 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386e1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe17b7fe700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1640a0830 0x7fe1640a2cf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe170009fd0 tx=0x7fe170019040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe17bfff700 1 -- 172.21.15.3:0/2233540082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556938452de0 con 0x5569386d4710 2023-09-10T01:06:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe17bfff700 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d4710 0x5569386d4af0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009fc0 tx=0x7fe16c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff3ea692575cf5cd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe1627fc700 1 -- 172.21.15.3:0/2233540082 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c00e670 con 0x5569386d4710 2023-09-10T01:06:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe1627fc700 1 -- 172.21.15.3:0/2233540082 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe16c00ec90 con 0x5569386d4710 2023-09-10T01:06:55.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.072+0000 7fe1627fc700 1 -- 172.21.15.3:0/2233540082 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c019cc0 con 0x5569386d4710 2023-09-10T01:06:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 -- 172.21.15.3:0/2233540082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d4710 msgr2=0x5569386d4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d4710 0x5569386d4af0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009fc0 tx=0x7fe16c011720 comp rx=0 tx=0).stop 2023-09-10T01:06:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 -- 172.21.15.3:0/2233540082 shutdown_connections 2023-09-10T01:06:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d4710 0x5569386d4af0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 --2- 172.21.15.3:0/2233540082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d5030 0x5569386de900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 -- 172.21.15.3:0/2233540082 >> 172.21.15.3:0/2233540082 conn(0x556938649090 msgr2=0x5569386cfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 -- 172.21.15.3:0/2233540082 shutdown_connections 2023-09-10T01:06:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.073+0000 7fe19635fb80 1 -- 172.21.15.3:0/2233540082 wait complete. 2023-09-10T01:06:55.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 -- start start 2023-09-10T01:06:55.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 Processor -- start 2023-09-10T01:06:55.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386de840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386dbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386dc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569385c4b70 con 0x5569386dee40 2023-09-10T01:06:55.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569385c4870 con 0x5569386d4710 2023-09-10T01:06:55.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.074+0000 7fe19635fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569384a8920 con 0x5569386d5030 2023-09-10T01:06:55.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe19635fb80 10 client.0 _ll_register_callbacks cb 0x55693850ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:55.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386de840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386dbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386de840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43304/0 (socket says 172.21.15.3:43304) 2023-09-10T01:06:55.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 -- 172.21.15.3:0/12656588 learned_addr learned my addr 172.21.15.3:0/12656588 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe180f9c700 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386dc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 -- 172.21.15.3:0/12656588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 msgr2=0x5569386dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386dbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 -- 172.21.15.3:0/12656588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 msgr2=0x5569386dc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386dc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 -- 172.21.15.3:0/12656588 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c009c20 con 0x5569386d4710 2023-09-10T01:06:55.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe17bfff700 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386de840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe16c01c040 tx=0x7fe16c01aae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c242f660eb71f125 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.075+0000 7fe15b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:55.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.076+0000 7fe15b7fe700 1 -- 172.21.15.3:0/12656588 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c004030 con 0x5569386d4710 2023-09-10T01:06:55.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.076+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55693857e4f0 con 0x5569386d4710 2023-09-10T01:06:55.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.076+0000 7fe15b7fe700 1 -- 172.21.15.3:0/12656588 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe16c004190 con 0x5569386d4710 2023-09-10T01:06:55.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.079+0000 7fe15b7fe700 1 -- 172.21.15.3:0/12656588 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe16c004350 con 0x5569386d4710 2023-09-10T01:06:55.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.079+0000 7fe19635fb80 10 client.26735 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:55.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.079+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55693843bc60 con 0x5569386d4710 2023-09-10T01:06:55.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe15b7fe700 1 -- 172.21.15.3:0/12656588 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe16c032030 con 0x5569386d4710 2023-09-10T01:06:55.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe15b7fe700 1 -- 172.21.15.3:0/12656588 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe16c02c060 con 0x5569386d4710 2023-09-10T01:06:55.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe19635fb80 10 client.26735 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:55.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe19635fb80 10 client.26735 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:06:55.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe19635fb80 20 client.26735 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:55.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe19635fb80 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5569386ee800 0x5569386f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe19635fb80 4 client.26735 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:55.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55693863b8a0 con 0x5569386ee800 2023-09-10T01:06:55.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.080+0000 7fe17b7fe700 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5569386ee800 0x5569386f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.081+0000 7fe17b7fe700 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5569386ee800 0x5569386f0c20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b2e81b2dfd032a00 server_cookie=e31c17e3065bcbb6 in_seq=0 out_seq=0 2023-09-10T01:06:55.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.081+0000 7fe15b7fe700 10 client.26735 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:55.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe15b7fe700 1 -- 172.21.15.3:0/12656588 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55693863b8a0 con 0x5569386ee800 2023-09-10T01:06:55.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe15b7fe700 10 client.26735 handle_command_reply: tid=0 2023-09-10T01:06:55.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 client.26735 shutdown 2023-09-10T01:06:55.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 msgr2=0x5569386de840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386de840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe16c01c040 tx=0x7fe16c01aae0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 shutdown_connections 2023-09-10T01:06:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5569386ee800 0x5569386f0c20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386dee40 0x5569386dc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386de840 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 --2- 172.21.15.3:0/12656588 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386dbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 >> 172.21.15.3:0/12656588 conn(0x556938649090 msgr2=0x5569386cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.082+0000 7fe15b7fe700 0 client.26735 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.083+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 shutdown_connections 2023-09-10T01:06:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.083+0000 7fe19635fb80 1 -- 172.21.15.3:0/12656588 wait complete. 2023-09-10T01:06:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.083+0000 7fe19635fb80 20 client.26735 trim_cache size 0 max 16384 2023-09-10T01:06:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.099+0000 7fe19635fb80 1 Processor -- start 2023-09-10T01:06:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.099+0000 7fe19635fb80 1 -- start start 2023-09-10T01:06:55.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386e1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d5030 0x5569386de370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386dee40 0x5569386d6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe19635fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569386d59f0 con 0x5569386d5030 2023-09-10T01:06:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe19635fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569386de8b0 con 0x5569386d4710 2023-09-10T01:06:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe19635fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569386dea70 con 0x5569386dee40 2023-09-10T01:06:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe180f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386dee40 0x5569386d6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386e1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43306/0 (socket says 172.21.15.3:43306) 2023-09-10T01:06:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe17b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d5030 0x5569386de370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe17bfff700 1 -- 172.21.15.3:0/3120684364 learned_addr learned my addr 172.21.15.3:0/3120684364 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe180f9c700 1 -- 172.21.15.3:0/3120684364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 msgr2=0x5569386e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe180f9c700 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386e1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe180f9c700 1 -- 172.21.15.3:0/3120684364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d5030 msgr2=0x5569386de370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe180f9c700 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d5030 0x5569386de370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe180f9c700 1 -- 172.21.15.3:0/3120684364 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556938479250 con 0x5569386dee40 2023-09-10T01:06:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe17bfff700 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.100+0000 7fe17b7fe700 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d5030 0x5569386de370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.101+0000 7fe180f9c700 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386dee40 0x5569386d6da0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe1740c90e0 tx=0x7fe174098a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dd6b2debd999ac9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.101+0000 7fe162ffd700 1 -- 172.21.15.3:0/3120684364 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174004ad0 con 0x5569386dee40 2023-09-10T01:06:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.101+0000 7fe162ffd700 1 -- 172.21.15.3:0/3120684364 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe17401b320 con 0x5569386dee40 2023-09-10T01:06:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.101+0000 7fe162ffd700 1 -- 172.21.15.3:0/3120684364 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1740a0f40 con 0x5569386dee40 2023-09-10T01:06:55.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.101+0000 7fe19635fb80 1 -- 172.21.15.3:0/3120684364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386dee40 msgr2=0x5569386d6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.101+0000 7fe19635fb80 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386dee40 0x5569386d6da0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe1740c90e0 tx=0x7fe174098a10 comp rx=0 tx=0).stop 2023-09-10T01:06:55.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 -- 172.21.15.3:0/3120684364 shutdown_connections 2023-09-10T01:06:55.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386dee40 0x5569386d6da0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386d4710 0x5569386e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 --2- 172.21.15.3:0/3120684364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d5030 0x5569386de370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 -- 172.21.15.3:0/3120684364 >> 172.21.15.3:0/3120684364 conn(0x556938649090 msgr2=0x5569386eaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 -- 172.21.15.3:0/3120684364 shutdown_connections 2023-09-10T01:06:55.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 -- 172.21.15.3:0/3120684364 wait complete. 2023-09-10T01:06:55.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 -- start start 2023-09-10T01:06:55.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.102+0000 7fe19635fb80 1 Processor -- start 2023-09-10T01:06:55.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d4710 0x5569386e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386e3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe19635fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386dee40 0x5569386e44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe19635fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569386e4be0 con 0x5569386d4710 2023-09-10T01:06:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe19635fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569386e4d50 con 0x5569386dee40 2023-09-10T01:06:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe19635fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556938742700 con 0x5569386d5030 2023-09-10T01:06:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe19635fb80 10 client.0 _ll_register_callbacks cb 0x7fe174009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d4710 0x5569386e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386e3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d4710 0x5569386e5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33114/0 (socket says 172.21.15.3:33114) 2023-09-10T01:06:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 -- 172.21.15.3:0/514402212 learned_addr learned my addr 172.21.15.3:0/514402212 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe180f9c700 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386dee40 0x5569386e44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 -- 172.21.15.3:0/514402212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 msgr2=0x5569386e3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386e3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 -- 172.21.15.3:0/514402212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386dee40 msgr2=0x5569386e44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386dee40 0x5569386e44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.103+0000 7fe17bfff700 1 -- 172.21.15.3:0/514402212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55693843bc60 con 0x5569386d4710 2023-09-10T01:06:55.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe17bfff700 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d4710 0x5569386e5b70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009860 tx=0x7fe16c019800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3e175739e6f2a1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe15b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:06:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe15b7fe700 1 -- 172.21.15.3:0/514402212 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0045d0 con 0x5569386d4710 2023-09-10T01:06:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe16c00ec90 con 0x5569386d4710 2023-09-10T01:06:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe15b7fe700 1 -- 172.21.15.3:0/514402212 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe16c000e40 con 0x5569386d4710 2023-09-10T01:06:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe15b7fe700 1 -- 172.21.15.3:0/514402212 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe16c00e470 con 0x5569386d4710 2023-09-10T01:06:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe19635fb80 10 client.16677 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:55.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.104+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7fe17c114450 con 0x5569386d4710 2023-09-10T01:06:55.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe15b7fe700 1 -- 172.21.15.3:0/514402212 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe16c00ec90 con 0x5569386d4710 2023-09-10T01:06:55.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe15b7fe700 1 -- 172.21.15.3:0/514402212 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe16c02c040 con 0x5569386d4710 2023-09-10T01:06:55.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe19635fb80 10 client.16677 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:55.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe19635fb80 10 client.16677 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:06:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe19635fb80 20 client.16677 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe19635fb80 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556938743a20 0x556938745e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe19635fb80 4 client.16677 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:06:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55693861f920 con 0x556938743a20 2023-09-10T01:06:55.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.105+0000 7fe17b7fe700 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556938743a20 0x556938745e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe17b7fe700 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556938743a20 0x556938745e10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36de454519b31086 server_cookie=55aef0da49d2cd26 in_seq=0 out_seq=0 2023-09-10T01:06:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe15b7fe700 10 client.16677 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe15b7fe700 1 -- 172.21.15.3:0/514402212 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55693861f920 con 0x556938743a20 2023-09-10T01:06:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe15b7fe700 10 client.16677 handle_command_reply: tid=0 2023-09-10T01:06:55.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 client.16677 shutdown 2023-09-10T01:06:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d4710 msgr2=0x5569386e5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d4710 0x5569386e5b70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009860 tx=0x7fe16c019800 comp rx=0 tx=0).stop 2023-09-10T01:06:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 shutdown_connections 2023-09-10T01:06:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x556938743a20 0x556938745e10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5569386dee40 0x5569386e44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5569386d4710 0x5569386e5b70 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 --2- 172.21.15.3:0/514402212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5569386d5030 0x5569386e3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 >> 172.21.15.3:0/514402212 conn(0x556938649090 msgr2=0x5569386d6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe15b7fe700 0 client.16677 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:06:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 shutdown_connections 2023-09-10T01:06:55.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 1 -- 172.21.15.3:0/514402212 wait complete. 2023-09-10T01:06:55.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.106+0000 7fe19635fb80 20 client.16677 trim_cache size 0 max 16384 2023-09-10T01:06:55.121 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:06:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1640a0830 msgr2=0x7fe1640a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1640a0830 0x7fe1640a2cf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe170009fd0 tx=0x7fe170019040 comp rx=0 tx=0).stop 2023-09-10T01:06:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c111660 msgr2=0x7fe17c1a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c111660 0x7fe17c1a1850 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe17400ef30 tx=0x7fe17400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 shutdown_connections 2023-09-10T01:06:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe1640a0830 0x7fe1640a2cf0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe17c111660 0x7fe17c1a1850 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe17c0729a0 0x7fe17c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 --2- 172.21.15.3:0/279162271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe17c0732c0 0x7fe17c19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 >> 172.21.15.3:0/279162271 conn(0x7fe17c0fc160 msgr2=0x7fe17c0fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 shutdown_connections 2023-09-10T01:06:55.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.107+0000 7fe1829ff700 1 -- 172.21.15.3:0/279162271 wait complete. 2023-09-10T01:06:55.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/858238003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:06:55.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:06:55.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.350+0000 7eff9f5ff700 1 -- 172.21.15.3:0/2447130901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810edd0 msgr2=0x7eff9810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.350+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/2447130901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810edd0 0x7eff9810ca80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7eff9000b0a0 tx=0x7eff9000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.350+0000 7eff9f5ff700 1 -- 172.21.15.3:0/2447130901 shutdown_connections 2023-09-10T01:06:55.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.350+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/2447130901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810edd0 0x7eff9810ca80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.350+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/2447130901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 0x7eff98107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.350+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/2447130901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff9810f1b0 0x7eff9810cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.350+0000 7eff9f5ff700 1 -- 172.21.15.3:0/2447130901 >> 172.21.15.3:0/2447130901 conn(0x7eff98071dd0 msgr2=0x7eff9806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.352+0000 7eff9f5ff700 1 -- 172.21.15.3:0/2447130901 shutdown_connections 2023-09-10T01:06:55.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.352+0000 7eff9f5ff700 1 -- 172.21.15.3:0/2447130901 wait complete. 2023-09-10T01:06:55.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.354+0000 7eff9f5ff700 1 Processor -- start 2023-09-10T01:06:55.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.354+0000 7eff9f5ff700 1 -- start start 2023-09-10T01:06:55.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.355+0000 7eff9f5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 0x7eff981ab530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9f5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff9810edd0 0x7eff981aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9f5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810f1b0 0x7eff981a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 0x7eff981ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9f5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981a5c10 con 0x7eff98107600 2023-09-10T01:06:55.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9f5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981a5d80 con 0x7eff9810edd0 2023-09-10T01:06:55.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9f5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff981a5f40 con 0x7eff9810f1b0 2023-09-10T01:06:55.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 0x7eff981ab530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33120/0 (socket says 172.21.15.3:33120) 2023-09-10T01:06:55.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9d39b700 1 -- 172.21.15.3:0/3466660743 learned_addr learned my addr 172.21.15.3:0/3466660743 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9db9c700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810f1b0 0x7eff981a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff9810edd0 0x7eff981aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9d39b700 1 -- 172.21.15.3:0/3466660743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810f1b0 msgr2=0x7eff981a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9d39b700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810f1b0 0x7eff981a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.356+0000 7eff9d39b700 1 -- 172.21.15.3:0/3466660743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff9810edd0 msgr2=0x7eff981aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff9d39b700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff9810edd0 0x7eff981aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff9d39b700 1 -- 172.21.15.3:0/3466660743 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff90009d00 con 0x7eff98107600 2023-09-10T01:06:55.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff9810edd0 0x7eff981aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:06:55.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff9db9c700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810f1b0 0x7eff981a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:06:55.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff9d39b700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 0x7eff981ab530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7eff900062a0 tx=0x7eff90007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff8e7fc700 1 -- 172.21.15.3:0/3466660743 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9000e040 con 0x7eff98107600 2023-09-10T01:06:55.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff981a61c0 con 0x7eff98107600 2023-09-10T01:06:55.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff8e7fc700 1 -- 172.21.15.3:0/3466660743 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff90012cf0 con 0x7eff98107600 2023-09-10T01:06:55.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff8e7fc700 1 -- 172.21.15.3:0/3466660743 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90005690 con 0x7eff98107600 2023-09-10T01:06:55.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.357+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff981b2520 con 0x7eff98107600 2023-09-10T01:06:55.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.361+0000 7eff8e7fc700 1 -- 172.21.15.3:0/3466660743 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7eff90012830 con 0x7eff98107600 2023-09-10T01:06:55.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.362+0000 7effb2e9bb80 1 Processor -- start 2023-09-10T01:06:55.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.362+0000 7eff8e7fc700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7eff840a0830 0x7eff840a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.362+0000 7effb2e9bb80 1 -- start start 2023-09-10T01:06:55.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.362+0000 7eff8e7fc700 1 -- 172.21.15.3:0/3466660743 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7eff900cddc0 con 0x7eff98107600 2023-09-10T01:06:55.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9db9c700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7eff840a0830 0x7eff840a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd82710 0x5581cdd82af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd83030 0x5581cdd8c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd8f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdcdb730 con 0x5581cdd8ce40 2023-09-10T01:06:55.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdcd9050 con 0x5581cdd82710 2023-09-10T01:06:55.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdcd4940 con 0x5581cdd83030 2023-09-10T01:06:55.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd82710 0x5581cdd82af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd82710 0x5581cdd82af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43348/0 (socket says 172.21.15.3:43348) 2023-09-10T01:06:55.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 -- 172.21.15.3:0/935148907 learned_addr learned my addr 172.21.15.3:0/935148907 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9db9c700 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd8f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd83030 0x5581cdd8c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 -- 172.21.15.3:0/935148907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd83030 msgr2=0x5581cdd8c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd83030 0x5581cdd8c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 -- 172.21.15.3:0/935148907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 msgr2=0x5581cdd8f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd8f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.363+0000 7eff9d39b700 1 -- 172.21.15.3:0/935148907 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581cdb00de0 con 0x5581cdd82710 2023-09-10T01:06:55.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.364+0000 7eff9d39b700 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd82710 0x5581cdd82af0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7eff90097a10 tx=0x7eff90005160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43bf441ef526da0d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.364+0000 7eff9db9c700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7eff840a0830 0x7eff840a2cf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7eff981ac230 tx=0x7eff88013560 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.364+0000 7eff837fe700 1 -- 172.21.15.3:0/935148907 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9009b6a0 con 0x5581cdd82710 2023-09-10T01:06:55.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.364+0000 7eff837fe700 1 -- 172.21.15.3:0/935148907 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff9009afa0 con 0x5581cdd82710 2023-09-10T01:06:55.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.364+0000 7eff837fe700 1 -- 172.21.15.3:0/935148907 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900976b0 con 0x5581cdd82710 2023-09-10T01:06:55.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 -- 172.21.15.3:0/935148907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd82710 msgr2=0x5581cdd82af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd82710 0x5581cdd82af0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7eff90097a10 tx=0x7eff90005160 comp rx=0 tx=0).stop 2023-09-10T01:06:55.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 -- 172.21.15.3:0/935148907 shutdown_connections 2023-09-10T01:06:55.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd8f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd82710 0x5581cdd82af0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/935148907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd83030 0x5581cdd8c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 -- 172.21.15.3:0/935148907 >> 172.21.15.3:0/935148907 conn(0x5581cdcf7090 msgr2=0x5581cdd7dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 -- 172.21.15.3:0/935148907 shutdown_connections 2023-09-10T01:06:55.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 -- 172.21.15.3:0/935148907 wait complete. 2023-09-10T01:06:55.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 -- start start 2023-09-10T01:06:55.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.365+0000 7effb2e9bb80 1 Processor -- start 2023-09-10T01:06:55.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd82710 0x5581cdd8d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd8d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd8ce40 0x5581cdd982a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdc72b70 con 0x5581cdd82710 2023-09-10T01:06:55.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdc72870 con 0x5581cdd83030 2023-09-10T01:06:55.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdb56920 con 0x5581cdd8ce40 2023-09-10T01:06:55.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7effb2e9bb80 10 client.0 _ll_register_callbacks cb 0x5581cdbbca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:55.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd82710 0x5581cdd8d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7eff9cb9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd8d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd82710 0x5581cdd8d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33134/0 (socket says 172.21.15.3:33134) 2023-09-10T01:06:55.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7eff9d39b700 1 -- 172.21.15.3:0/1803861214 learned_addr learned my addr 172.21.15.3:0/1803861214 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.366+0000 7eff9db9c700 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd8ce40 0x5581cdd982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff9d39b700 1 -- 172.21.15.3:0/1803861214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd8ce40 msgr2=0x5581cdd982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff9d39b700 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd8ce40 0x5581cdd982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff9d39b700 1 -- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 msgr2=0x5581cdd8d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff9d39b700 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd8d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff9d39b700 1 -- 172.21.15.3:0/1803861214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff900ba040 con 0x5581cdd82710 2023-09-10T01:06:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff9d39b700 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd82710 0x5581cdd8d220 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7eff900a2a50 tx=0x7eff90003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510ccb6f79ce3677 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff80ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:06:55.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff80ff9700 1 -- 172.21.15.3:0/1803861214 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9009b6a0 con 0x5581cdd82710 2023-09-10T01:06:55.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff80ff9700 1 -- 172.21.15.3:0/1803861214 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7eff900a2f80 con 0x5581cdd82710 2023-09-10T01:06:55.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581cdc2c4f0 con 0x5581cdd82710 2023-09-10T01:06:55.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7eff80ff9700 1 -- 172.21.15.3:0/1803861214 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff9009edd0 con 0x5581cdd82710 2023-09-10T01:06:55.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7effb2e9bb80 10 client.16686 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:55.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.367+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5581cdae9c60 con 0x5581cdd82710 2023-09-10T01:06:55.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7eff80ff9700 1 -- 172.21.15.3:0/1803861214 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5581cdc2c4f0 con 0x5581cdd82710 2023-09-10T01:06:55.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7eff80ff9700 1 -- 172.21.15.3:0/1803861214 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7eff9009b200 con 0x5581cdd82710 2023-09-10T01:06:55.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7effb2e9bb80 10 client.16686 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:55.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7effb2e9bb80 10 client.16686 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:06:55.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7effb2e9bb80 20 client.16686 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:55.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cdd9c0a0 0x5581cdd9e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7effb2e9bb80 4 client.16686 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:55.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.368+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581cdce98a0 con 0x5581cdd9c0a0 2023-09-10T01:06:55.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.369+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cdd9c0a0 0x5581cdd9e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.369+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cdd9c0a0 0x5581cdd9e490 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a2656104d2397f server_cookie=2406b094d64bb0ad in_seq=0 out_seq=0 2023-09-10T01:06:55.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.369+0000 7eff80ff9700 10 client.16686 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:55.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7eff80ff9700 1 -- 172.21.15.3:0/1803861214 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5581cdce98a0 con 0x5581cdd9c0a0 2023-09-10T01:06:55.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7eff80ff9700 10 client.16686 handle_command_reply: tid=0 2023-09-10T01:06:55.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7effb2e9bb80 1 client.16686 shutdown 2023-09-10T01:06:55.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd82710 msgr2=0x5581cdd8d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd82710 0x5581cdd8d220 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7eff900a2a50 tx=0x7eff90003c90 comp rx=0 tx=0).stop 2023-09-10T01:06:55.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 shutdown_connections 2023-09-10T01:06:55.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cdd9c0a0 0x5581cdd9e490 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd8ce40 0x5581cdd982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.370+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd82710 0x5581cdd8d220 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.371+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1803861214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.371+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 >> 172.21.15.3:0/1803861214 conn(0x5581cdcf7090 msgr2=0x5581cdd7d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.371+0000 7eff80ff9700 0 client.16686 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:55.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.371+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 shutdown_connections 2023-09-10T01:06:55.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.371+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1803861214 wait complete. 2023-09-10T01:06:55.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.371+0000 7effb2e9bb80 20 client.16686 trim_cache size 0 max 16384 2023-09-10T01:06:55.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.385+0000 7effb2e9bb80 1 Processor -- start 2023-09-10T01:06:55.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7effb2e9bb80 1 -- start start 2023-09-10T01:06:55.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd86890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd87260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd877a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdd7f200 con 0x5581cdd8ce40 2023-09-10T01:06:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdd8dad0 con 0x5581cdd83030 2023-09-10T01:06:55.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cdd84280 con 0x5581cdd82710 2023-09-10T01:06:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd86890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd86890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40178/0 (socket says 172.21.15.3:40178) 2023-09-10T01:06:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7eff9d39b700 1 -- 172.21.15.3:0/1047832950 learned_addr learned my addr 172.21.15.3:0/1047832950 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd87260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.386+0000 7eff9db9c700 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd877a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff9d39b700 1 -- 172.21.15.3:0/1047832950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 msgr2=0x5581cdd87260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff9d39b700 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd87260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff9d39b700 1 -- 172.21.15.3:0/1047832950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 msgr2=0x5581cdd877a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff9d39b700 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd877a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff9d39b700 1 -- 172.21.15.3:0/1047832950 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581cdda2fa0 con 0x5581cdd82710 2023-09-10T01:06:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff9db9c700 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd877a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff9d39b700 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd86890 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7eff900992f0 tx=0x7eff90099550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77a5bb4b64c1894c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff83fff700 1 -- 172.21.15.3:0/1047832950 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9009b6a0 con 0x5581cdd82710 2023-09-10T01:06:55.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff83fff700 1 -- 172.21.15.3:0/1047832950 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff9009a900 con 0x5581cdd82710 2023-09-10T01:06:55.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.387+0000 7eff83fff700 1 -- 172.21.15.3:0/1047832950 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900057f0 con 0x5581cdd82710 2023-09-10T01:06:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1047832950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 msgr2=0x5581cdd86890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd86890 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7eff900992f0 tx=0x7eff90099550 comp rx=0 tx=0).stop 2023-09-10T01:06:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1047832950 shutdown_connections 2023-09-10T01:06:55.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd86890 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd8ce40 0x5581cdd877a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/1047832950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd83030 0x5581cdd87260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1047832950 >> 172.21.15.3:0/1047832950 conn(0x5581cdcf7090 msgr2=0x5581cdd988a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1047832950 shutdown_connections 2023-09-10T01:06:55.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.388+0000 7effb2e9bb80 1 -- 172.21.15.3:0/1047832950 wait complete. 2023-09-10T01:06:55.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.389+0000 7effb2e9bb80 1 -- start start 2023-09-10T01:06:55.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.389+0000 7effb2e9bb80 1 Processor -- start 2023-09-10T01:06:55.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd8e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd83030 0x5581cdd8e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7effb2e9bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd8ce40 0x5581cdd8eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cddeeea0 con 0x5581cdd83030 2023-09-10T01:06:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cddef010 con 0x5581cdd8ce40 2023-09-10T01:06:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7effb2e9bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581cddef1d0 con 0x5581cdd82710 2023-09-10T01:06:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7effb2e9bb80 10 client.0 _ll_register_callbacks cb 0x7eff9009ae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:55.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd8e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9db9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd8ce40 0x5581cdd8eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd8e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40192/0 (socket says 172.21.15.3:40192) 2023-09-10T01:06:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 -- 172.21.15.3:0/2700869158 learned_addr learned my addr 172.21.15.3:0/2700869158 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd83030 0x5581cdd8e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 -- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd8ce40 msgr2=0x5581cdd8eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd8ce40 0x5581cdd8eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 -- 172.21.15.3:0/2700869158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd83030 msgr2=0x5581cdd8e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd83030 0x5581cdd8e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9d39b700 1 -- 172.21.15.3:0/2700869158 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff900ba040 con 0x5581cdd82710 2023-09-10T01:06:55.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.390+0000 7eff9db9c700 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd8ce40 0x5581cdd8eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:55.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7eff9d39b700 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd8e0f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7eff90097a10 tx=0x7eff90005160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21016a2e43bee073 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7eff80ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:55.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7eff80ff9700 1 -- 172.21.15.3:0/2700869158 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9009edd0 con 0x5581cdd82710 2023-09-10T01:06:55.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7eff80ff9700 1 -- 172.21.15.3:0/2700869158 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7eff9009b200 con 0x5581cdd82710 2023-09-10T01:06:55.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff9009afa0 con 0x5581cdd82710 2023-09-10T01:06:55.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7eff80ff9700 1 -- 172.21.15.3:0/2700869158 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff9009a530 con 0x5581cdd82710 2023-09-10T01:06:55.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7effb2e9bb80 10 client.26674 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:55.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5581cdb27250 con 0x5581cdd82710 2023-09-10T01:06:55.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.391+0000 7eff80ff9700 1 -- 172.21.15.3:0/2700869158 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff9009afa0 con 0x5581cdd82710 2023-09-10T01:06:55.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7eff80ff9700 1 -- 172.21.15.3:0/2700869158 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7eff90019cb0 con 0x5581cdd82710 2023-09-10T01:06:55.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7effb2e9bb80 10 client.26674 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:55.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7effb2e9bb80 10 client.26674 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:06:55.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7effb2e9bb80 20 client.26674 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:55.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cddf03e0 0x5581cddf27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7effb2e9bb80 4 client.26674 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:06:55.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5581cdccd920 con 0x5581cddf03e0 2023-09-10T01:06:55.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cddf03e0 0x5581cddf27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7eff80ff9700 10 client.26674 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:55.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.392+0000 7eff9cb9a700 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cddf03e0 0x5581cddf27d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e2ba5e0ee80663a server_cookie=ee675758cb959168 in_seq=0 out_seq=0 2023-09-10T01:06:55.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7eff80ff9700 1 -- 172.21.15.3:0/2700869158 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5581cdccd920 con 0x5581cddf03e0 2023-09-10T01:06:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7eff80ff9700 10 client.26674 handle_command_reply: tid=0 2023-09-10T01:06:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 client.26674 shutdown 2023-09-10T01:06:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 msgr2=0x5581cdd8e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd8e0f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7eff90097a10 tx=0x7eff90005160 comp rx=0 tx=0).stop 2023-09-10T01:06:55.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 shutdown_connections 2023-09-10T01:06:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5581cddf03e0 0x5581cddf27d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5581cdd82710 0x5581cdd8e0f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5581cdd8ce40 0x5581cdd8eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 --2- 172.21.15.3:0/2700869158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581cdd83030 0x5581cdd8e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7eff80ff9700 0 client.26674 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:06:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 >> 172.21.15.3:0/2700869158 conn(0x5581cdcf7090 msgr2=0x5581cdd87260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 shutdown_connections 2023-09-10T01:06:55.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 1 -- 172.21.15.3:0/2700869158 wait complete. 2023-09-10T01:06:55.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.393+0000 7effb2e9bb80 20 client.26674 trim_cache size 0 max 16384 2023-09-10T01:06:55.409 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:06:55.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7eff840a0830 msgr2=0x7eff840a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7eff840a0830 0x7eff840a2cf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7eff981ac230 tx=0x7eff88013560 comp rx=0 tx=0).stop 2023-09-10T01:06:55.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 msgr2=0x7eff981ab530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 0x7eff981ab530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7eff900062a0 tx=0x7eff90007be0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 shutdown_connections 2023-09-10T01:06:55.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7eff840a0830 0x7eff840a2cf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9810f1b0 0x7eff981a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff98107600 0x7eff981ab530 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 --2- 172.21.15.3:0/3466660743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff9810edd0 0x7eff981aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 >> 172.21.15.3:0/3466660743 conn(0x7eff98071dd0 msgr2=0x7eff98109500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 shutdown_connections 2023-09-10T01:06:55.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.394+0000 7eff9f5ff700 1 -- 172.21.15.3:0/3466660743 wait complete. 2023-09-10T01:06:55.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/858238003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:06:55.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:06:55.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/858238003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:06:55.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.657+0000 7f36385ff700 1 -- 172.21.15.3:0/3134724416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630100640 msgr2=0x7f3630100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.657+0000 7f36385ff700 1 --2- 172.21.15.3:0/3134724416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630100640 0x7f3630100ac0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3620009990 tx=0x7f3620009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.657+0000 7f36385ff700 1 -- 172.21.15.3:0/3134724416 shutdown_connections 2023-09-10T01:06:55.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.657+0000 7f36385ff700 1 --2- 172.21.15.3:0/3134724416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3630106650 0x7f3630106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.657+0000 7f36385ff700 1 --2- 172.21.15.3:0/3134724416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3630101000 0x7f363010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.657+0000 7f36385ff700 1 --2- 172.21.15.3:0/3134724416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630100640 0x7f3630100ac0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.657+0000 7f36385ff700 1 -- 172.21.15.3:0/3134724416 >> 172.21.15.3:0/3134724416 conn(0x7f36300fc170 msgr2=0x7f36300fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.658+0000 7f36385ff700 1 -- 172.21.15.3:0/3134724416 shutdown_connections 2023-09-10T01:06:55.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.658+0000 7f36385ff700 1 -- 172.21.15.3:0/3134724416 wait complete. 2023-09-10T01:06:55.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.659+0000 7f36385ff700 1 Processor -- start 2023-09-10T01:06:55.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.659+0000 7f36385ff700 1 -- start start 2023-09-10T01:06:55.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.659+0000 7f36385ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3630100640 0x7f363019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.659+0000 7f36385ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630101000 0x7f363019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.659+0000 7f36385ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3630106650 0x7f3630198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f36385ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3630198dc0 con 0x7f3630106650 2023-09-10T01:06:55.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f36385ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3630198f30 con 0x7f3630101000 2023-09-10T01:06:55.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f36385ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36301990f0 con 0x7f3630100640 2023-09-10T01:06:55.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f363639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3630100640 0x7f363019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f363639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3630100640 0x7f363019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40198/0 (socket says 172.21.15.3:40198) 2023-09-10T01:06:55.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f363639b700 1 -- 172.21.15.3:0/926477819 learned_addr learned my addr 172.21.15.3:0/926477819 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3635b9a700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630101000 0x7f363019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3636b9c700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3630106650 0x7f3630198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3635b9a700 1 -- 172.21.15.3:0/926477819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3630100640 msgr2=0x7f363019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3635b9a700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3630100640 0x7f363019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3635b9a700 1 -- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3630106650 msgr2=0x7f3630198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3635b9a700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3630106650 0x7f3630198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3635b9a700 1 -- 172.21.15.3:0/926477819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36200095f0 con 0x7f3630101000 2023-09-10T01:06:55.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3636b9c700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3630106650 0x7f3630198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:06:55.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.660+0000 7f3635b9a700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630101000 0x7f363019ec20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3620000c00 tx=0x7f362000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.661+0000 7f36277fe700 1 -- 172.21.15.3:0/926477819 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362001d070 con 0x7f3630101000 2023-09-10T01:06:55.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.661+0000 7f36277fe700 1 -- 172.21.15.3:0/926477819 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3620005050 con 0x7f3630101000 2023-09-10T01:06:55.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.661+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3630199370 con 0x7f3630101000 2023-09-10T01:06:55.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.661+0000 7f36277fe700 1 -- 172.21.15.3:0/926477819 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620004a00 con 0x7f3630101000 2023-09-10T01:06:55.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.661+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36301a56d0 con 0x7f3630101000 2023-09-10T01:06:55.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.664+0000 7f36277fe700 1 -- 172.21.15.3:0/926477819 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f36200051c0 con 0x7f3630101000 2023-09-10T01:06:55.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.665+0000 7f364bfc7b80 1 Processor -- start 2023-09-10T01:06:55.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f364bfc7b80 1 -- start start 2023-09-10T01:06:55.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f36277fe700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f361c0a07b0 0x7f361c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f36277fe700 1 -- 172.21.15.3:0/926477819 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f36200cd300 con 0x7f3630101000 2023-09-10T01:06:55.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f3636b9c700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f361c0a07b0 0x7f361c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa0baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa15900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa18230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273f964730 con 0x56273fa0c030 2023-09-10T01:06:55.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273f962050 con 0x56273fa0b710 2023-09-10T01:06:55.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273f95d940 con 0x56273fa15e40 2023-09-10T01:06:55.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f363639b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa0baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f363639b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa0baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43406/0 (socket says 172.21.15.3:43406) 2023-09-10T01:06:55.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.666+0000 7f363639b700 1 -- 172.21.15.3:0/564252085 learned_addr learned my addr 172.21.15.3:0/564252085 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f3636b9c700 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa18230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f3635b9a700 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa15900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f363639b700 1 -- 172.21.15.3:0/564252085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 msgr2=0x56273fa18230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f363639b700 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa18230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f363639b700 1 -- 172.21.15.3:0/564252085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 msgr2=0x56273fa15900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f363639b700 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa15900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f363639b700 1 -- 172.21.15.3:0/564252085 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56273f789de0 con 0x56273fa0b710 2023-09-10T01:06:55.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f3635b9a700 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa15900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:06:55.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f3636b9c700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f361c0a07b0 0x7f361c0a2c70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3628009550 tx=0x7f3628007b00 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f363639b700 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa0baf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f362c00be20 tx=0x7f362c0106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf3496f54e76a87f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f3624bfe700 1 -- 172.21.15.3:0/564252085 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c011440 con 0x56273fa0b710 2023-09-10T01:06:55.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f3624bfe700 1 -- 172.21.15.3:0/564252085 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f362c015070 con 0x56273fa0b710 2023-09-10T01:06:55.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.667+0000 7f3624bfe700 1 -- 172.21.15.3:0/564252085 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0197c0 con 0x56273fa0b710 2023-09-10T01:06:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 -- 172.21.15.3:0/564252085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 msgr2=0x56273fa0baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa0baf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f362c00be20 tx=0x7f362c0106b0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 -- 172.21.15.3:0/564252085 shutdown_connections 2023-09-10T01:06:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa18230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa0baf0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/564252085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa15900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 -- 172.21.15.3:0/564252085 >> 172.21.15.3:0/564252085 conn(0x56273f980090 msgr2=0x56273fa06bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 -- 172.21.15.3:0/564252085 shutdown_connections 2023-09-10T01:06:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.668+0000 7f364bfc7b80 1 -- 172.21.15.3:0/564252085 wait complete. 2023-09-10T01:06:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.669+0000 7f364bfc7b80 1 -- start start 2023-09-10T01:06:55.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.669+0000 7f364bfc7b80 1 Processor -- start 2023-09-10T01:06:55.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.669+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0b710 0x56273fa15830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0c030 0x56273fa12a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa15e40 0x56273fa13050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273f8fbb70 con 0x56273fa15e40 2023-09-10T01:06:55.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273f8fb870 con 0x56273fa0c030 2023-09-10T01:06:55.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273f7df920 con 0x56273fa0b710 2023-09-10T01:06:55.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f364bfc7b80 10 client.0 _ll_register_callbacks cb 0x56273f845a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:55.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0b710 0x56273fa15830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f3635b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0c030 0x56273fa12a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0b710 0x56273fa15830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40206/0 (socket says 172.21.15.3:40206) 2023-09-10T01:06:55.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 -- 172.21.15.3:0/2862387538 learned_addr learned my addr 172.21.15.3:0/2862387538 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f3636b9c700 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa15e40 0x56273fa13050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 -- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0c030 msgr2=0x56273fa12a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0c030 0x56273fa12a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 -- 172.21.15.3:0/2862387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa15e40 msgr2=0x56273fa13050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa15e40 0x56273fa13050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f363639b700 1 -- 172.21.15.3:0/2862387538 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c00ba50 con 0x56273fa0b710 2023-09-10T01:06:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.670+0000 7f3635b9a700 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0c030 0x56273fa12a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f363639b700 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0b710 0x56273fa15830 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f362c000c00 tx=0x7f362c010e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b13b34ba071b2ec0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f3619ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:06:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f3619ffb700 1 -- 172.21.15.3:0/2862387538 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c005350 con 0x56273fa0b710 2023-09-10T01:06:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f3619ffb700 1 -- 172.21.15.3:0/2862387538 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f362c0054b0 con 0x56273fa0b710 2023-09-10T01:06:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56273f8b54f0 con 0x56273fa0b710 2023-09-10T01:06:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f3619ffb700 1 -- 172.21.15.3:0/2862387538 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f362c0056b0 con 0x56273fa0b710 2023-09-10T01:06:55.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f364bfc7b80 10 client.26686 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:55.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.671+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56273f772c60 con 0x56273fa0b710 2023-09-10T01:06:55.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f3619ffb700 1 -- 172.21.15.3:0/2862387538 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56273f8b54f0 con 0x56273fa0b710 2023-09-10T01:06:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f3619ffb700 1 -- 172.21.15.3:0/2862387538 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f362c021070 con 0x56273fa0b710 2023-09-10T01:06:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f364bfc7b80 10 client.26686 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f364bfc7b80 10 client.26686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f364bfc7b80 20 client.26686 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa257d0 0x56273fa27bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f364bfc7b80 4 client.26686 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:55.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56273f9728a0 con 0x56273fa257d0 2023-09-10T01:06:55.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.672+0000 7f3635b9a700 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa257d0 0x56273fa27bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.673+0000 7f3619ffb700 10 client.26686 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:55.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.673+0000 7f3635b9a700 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa257d0 0x56273fa27bf0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d17d9e10d635cb4a server_cookie=4fce883c416baf73 in_seq=0 out_seq=0 2023-09-10T01:06:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f3619ffb700 1 -- 172.21.15.3:0/2862387538 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56273f9728a0 con 0x56273fa257d0 2023-09-10T01:06:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f3619ffb700 10 client.26686 handle_command_reply: tid=0 2023-09-10T01:06:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 client.26686 shutdown 2023-09-10T01:06:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0b710 msgr2=0x56273fa15830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0b710 0x56273fa15830 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f362c000c00 tx=0x7f362c010e00 comp rx=0 tx=0).stop 2023-09-10T01:06:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 shutdown_connections 2023-09-10T01:06:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa257d0 0x56273fa27bf0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0b710 0x56273fa15830 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa15e40 0x56273fa13050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f3619ffb700 0 client.26686 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/2862387538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0c030 0x56273fa12a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 >> 172.21.15.3:0/2862387538 conn(0x56273f980090 msgr2=0x56273fa06780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 shutdown_connections 2023-09-10T01:06:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.674+0000 7f364bfc7b80 1 -- 172.21.15.3:0/2862387538 wait complete. 2023-09-10T01:06:55.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.675+0000 7f364bfc7b80 20 client.26686 trim_cache size 0 max 16384 2023-09-10T01:06:55.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.690+0000 7f364bfc7b80 1 Processor -- start 2023-09-10T01:06:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.690+0000 7f364bfc7b80 1 -- start start 2023-09-10T01:06:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa18230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa21140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa15360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273fa0c4c0 con 0x56273fa0c030 2023-09-10T01:06:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273fa27e40 con 0x56273fa0b710 2023-09-10T01:06:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273fa06dd0 con 0x56273fa15e40 2023-09-10T01:06:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa18230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa18230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43416/0 (socket says 172.21.15.3:43416) 2023-09-10T01:06:55.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 -- 172.21.15.3:0/1831149308 learned_addr learned my addr 172.21.15.3:0/1831149308 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f3635b9a700 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa21140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f3636b9c700 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa15360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 -- 172.21.15.3:0/1831149308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 msgr2=0x56273fa15360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa15360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 -- 172.21.15.3:0/1831149308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 msgr2=0x56273fa21140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa21140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.691+0000 7f363639b700 1 -- 172.21.15.3:0/1831149308 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56273f7b0250 con 0x56273fa0b710 2023-09-10T01:06:55.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.692+0000 7f363639b700 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa18230 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f362c00c010 tx=0x7f362c005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d71ad98cef30d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.692+0000 7f36253ff700 1 -- 172.21.15.3:0/1831149308 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c00b500 con 0x56273fa0b710 2023-09-10T01:06:55.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.692+0000 7f36253ff700 1 -- 172.21.15.3:0/1831149308 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f362c011cb0 con 0x56273fa0b710 2023-09-10T01:06:55.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.692+0000 7f36253ff700 1 -- 172.21.15.3:0/1831149308 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c003970 con 0x56273fa0b710 2023-09-10T01:06:55.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1831149308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 msgr2=0x56273fa18230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa18230 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f362c00c010 tx=0x7f362c005040 comp rx=0 tx=0).stop 2023-09-10T01:06:55.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1831149308 shutdown_connections 2023-09-10T01:06:55.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa15e40 0x56273fa15360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa0b710 0x56273fa18230 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1831149308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0c030 0x56273fa21140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1831149308 >> 172.21.15.3:0/1831149308 conn(0x56273f980090 msgr2=0x56273fa05320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1831149308 shutdown_connections 2023-09-10T01:06:55.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1831149308 wait complete. 2023-09-10T01:06:55.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 -- start start 2023-09-10T01:06:55.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.693+0000 7f364bfc7b80 1 Processor -- start 2023-09-10T01:06:55.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0b710 0x56273fa1c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0c030 0x56273fa1cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f364bfc7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa15e40 0x56273fa1aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273fa1b120 con 0x56273fa0b710 2023-09-10T01:06:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273fa1b290 con 0x56273fa15e40 2023-09-10T01:06:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f364bfc7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56273fa1b450 con 0x56273fa0c030 2023-09-10T01:06:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f364bfc7b80 10 client.0 _ll_register_callbacks cb 0x7f362c00bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f363639b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0b710 0x56273fa1c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f363639b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0b710 0x56273fa1c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33220/0 (socket says 172.21.15.3:33220) 2023-09-10T01:06:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f363639b700 1 -- 172.21.15.3:0/1126295697 learned_addr learned my addr 172.21.15.3:0/1126295697 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f3636b9c700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa15e40 0x56273fa1aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.694+0000 7f3635b9a700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0c030 0x56273fa1cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3636b9c700 1 -- 172.21.15.3:0/1126295697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0c030 msgr2=0x56273fa1cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3636b9c700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0c030 0x56273fa1cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3636b9c700 1 -- 172.21.15.3:0/1126295697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0b710 msgr2=0x56273fa1c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:55.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3636b9c700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0b710 0x56273fa1c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3636b9c700 1 -- 172.21.15.3:0/1126295697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362c011e70 con 0x56273fa15e40 2023-09-10T01:06:55.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3635b9a700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0c030 0x56273fa1cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:55.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3636b9c700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa15e40 0x56273fa1aa10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3628009010 tx=0x7f3628004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=505c4330f10ed831 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3619ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3619ffb700 1 -- 172.21.15.3:0/1126295697 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628019e10 con 0x56273fa15e40 2023-09-10T01:06:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3619ffb700 1 -- 172.21.15.3:0/1126295697 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3628004bb0 con 0x56273fa15e40 2023-09-10T01:06:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f362c015070 con 0x56273fa15e40 2023-09-10T01:06:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.695+0000 7f3619ffb700 1 -- 172.21.15.3:0/1126295697 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3628004d70 con 0x56273fa15e40 2023-09-10T01:06:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 10 client.26771 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56273f7882b0 con 0x56273fa15e40 2023-09-10T01:06:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f3619ffb700 1 -- 172.21.15.3:0/1126295697 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f362c015070 con 0x56273fa15e40 2023-09-10T01:06:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f3619ffb700 1 -- 172.21.15.3:0/1126295697 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3628026970 con 0x56273fa15e40 2023-09-10T01:06:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 10 client.26771 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 10 client.26771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 20 client.26771 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa7a9c0 0x56273fa7cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 4 client.26771 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:06:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56273f956920 con 0x56273fa7a9c0 2023-09-10T01:06:55.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.696+0000 7f363639b700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa7a9c0 0x56273fa7cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f363639b700 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa7a9c0 0x56273fa7cde0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d1b972c8855d7df server_cookie=3a966633e5c6e66d in_seq=0 out_seq=0 2023-09-10T01:06:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f3619ffb700 10 client.26771 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f3619ffb700 1 -- 172.21.15.3:0/1126295697 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56273f956920 con 0x56273fa7a9c0 2023-09-10T01:06:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f3619ffb700 10 client.26771 handle_command_reply: tid=0 2023-09-10T01:06:55.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 client.26771 shutdown 2023-09-10T01:06:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa15e40 msgr2=0x56273fa1aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa15e40 0x56273fa1aa10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3628009010 tx=0x7f3628004b40 comp rx=0 tx=0).stop 2023-09-10T01:06:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 shutdown_connections 2023-09-10T01:06:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56273fa7a9c0 0x56273fa7cde0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56273fa15e40 0x56273fa1aa10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56273fa0b710 0x56273fa1c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 --2- 172.21.15.3:0/1126295697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56273fa0c030 0x56273fa1cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f3619ffb700 0 client.26771 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 >> 172.21.15.3:0/1126295697 conn(0x56273f980090 msgr2=0x56273fa21140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 shutdown_connections 2023-09-10T01:06:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.697+0000 7f364bfc7b80 1 -- 172.21.15.3:0/1126295697 wait complete. 2023-09-10T01:06:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f364bfc7b80 20 client.26771 trim_cache size 0 max 16384 2023-09-10T01:06:55.713 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:06:55.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f361c0a07b0 msgr2=0x7f361c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f361c0a07b0 0x7f361c0a2c70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3628009550 tx=0x7f3628007b00 comp rx=0 tx=0).stop 2023-09-10T01:06:55.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630101000 msgr2=0x7f363019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:55.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630101000 0x7f363019ec20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3620000c00 tx=0x7f362000f690 comp rx=0 tx=0).stop 2023-09-10T01:06:55.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 shutdown_connections 2023-09-10T01:06:55.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f361c0a07b0 0x7f361c0a2c70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3630100640 0x7f363019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3630106650 0x7f3630198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 --2- 172.21.15.3:0/926477819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3630101000 0x7f363019ec20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:55.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 >> 172.21.15.3:0/926477819 conn(0x7f36300fc170 msgr2=0x7f363010ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:55.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 shutdown_connections 2023-09-10T01:06:55.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:55.698+0000 7f36385ff700 1 -- 172.21.15.3:0/926477819 wait complete. 2023-09-10T01:06:55.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 286187 9.50 MB/sec execute 37 sec latency 32.058 ms 2023-09-10T01:06:56.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:56 smithi136 ceph-mon[133708]: pgmap v114: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 411 op/s 2023-09-10T01:06:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:56 smithi092 ceph-mon[134067]: pgmap v114: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 411 op/s 2023-09-10T01:06:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:56 smithi003 ceph-mon[97654]: pgmap v114: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 411 op/s 2023-09-10T01:06:56.720 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:06:56.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:06:56.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 287867 9.57 MB/sec execute 38 sec latency 14.789 ms 2023-09-10T01:06:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 -- 172.21.15.3:0/3879912971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1018a0 msgr2=0x7efe0c101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 --2- 172.21.15.3:0/3879912971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1018a0 0x7efe0c101c80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009990 tx=0x7efdfc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 -- 172.21.15.3:0/3879912971 shutdown_connections 2023-09-10T01:06:56.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 --2- 172.21.15.3:0/3879912971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe0c10ac70 0x7efe0c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 --2- 172.21.15.3:0/3879912971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1018a0 0x7efe0c101c80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 --2- 172.21.15.3:0/3879912971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe0c1021c0 0x7efe0c10a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 -- 172.21.15.3:0/3879912971 >> 172.21.15.3:0/3879912971 conn(0x7efe0c076710 msgr2=0x7efe0c076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:56.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 -- 172.21.15.3:0/3879912971 shutdown_connections 2023-09-10T01:06:56.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.941+0000 7efe121ff700 1 -- 172.21.15.3:0/3879912971 wait complete. 2023-09-10T01:06:56.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.942+0000 7efe121ff700 1 Processor -- start 2023-09-10T01:06:56.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe121ff700 1 -- start start 2023-09-10T01:06:56.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe121ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe0c1018a0 0x7efe0c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe121ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1021c0 0x7efe0c19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe121ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe0c10ac70 0x7efe0c1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe121ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c196c20 con 0x7efe0c1021c0 2023-09-10T01:06:56.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe121ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c196d90 con 0x7efe0c10ac70 2023-09-10T01:06:56.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe121ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c196f50 con 0x7efe0c1018a0 2023-09-10T01:06:56.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe0affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1021c0 0x7efe0c19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe0affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1021c0 0x7efe0c19ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33238/0 (socket says 172.21.15.3:33238) 2023-09-10T01:06:56.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe0affd700 1 -- 172.21.15.3:0/3213282006 learned_addr learned my addr 172.21.15.3:0/3213282006 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:56.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe0bfff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe0c10ac70 0x7efe0c1965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.943+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe0c1018a0 0x7efe0c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe0bfff700 1 -- 172.21.15.3:0/3213282006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe0c1018a0 msgr2=0x7efe0c19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:56.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe0bfff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe0c1018a0 0x7efe0c19c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe0bfff700 1 -- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1021c0 msgr2=0x7efe0c19ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:56.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe0bfff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1021c0 0x7efe0c19ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe0bfff700 1 -- 172.21.15.3:0/3213282006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc0095f0 con 0x7efe0c10ac70 2023-09-10T01:06:56.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe0bfff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe0c10ac70 0x7efe0c1965b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efe0400efc0 tx=0x7efe0400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:56.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe08ff9700 1 -- 172.21.15.3:0/3213282006 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04018070 con 0x7efe0c10ac70 2023-09-10T01:06:56.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe08ff9700 1 -- 172.21.15.3:0/3213282006 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe0400f040 con 0x7efe0c10ac70 2023-09-10T01:06:56.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c1971d0 con 0x7efe0c10ac70 2023-09-10T01:06:56.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe08ff9700 1 -- 172.21.15.3:0/3213282006 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0400cd40 con 0x7efe0c10ac70 2023-09-10T01:06:56.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.944+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c1a3530 con 0x7efe0c10ac70 2023-09-10T01:06:56.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.948+0000 7efe08ff9700 1 -- 172.21.15.3:0/3213282006 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7efe04003a00 con 0x7efe0c10ac70 2023-09-10T01:06:56.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.949+0000 7efe25a31b80 1 Processor -- start 2023-09-10T01:06:56.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe08ff9700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efdf40a07b0 0x7efdf40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe25a31b80 1 -- start start 2023-09-10T01:06:56.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe08ff9700 1 -- 172.21.15.3:0/3213282006 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efe040cbf00 con 0x7efe0c10ac70 2023-09-10T01:06:56.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe0affd700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efdf40a07b0 0x7efdf40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b4db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b57980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050b5a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.950+0000 7efe25a31b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050aa67b0 con 0x556050b4d790 2023-09-10T01:06:56.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe25a31b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050aa40d0 con 0x556050b4e0b0 2023-09-10T01:06:56.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe25a31b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050a9f9c0 con 0x556050b57ec0 2023-09-10T01:06:56.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b4db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b4db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33244/0 (socket says 172.21.15.3:33244) 2023-09-10T01:06:56.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 -- 172.21.15.3:0/625811807 learned_addr learned my addr 172.21.15.3:0/625811807 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:56.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0affd700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efdf40a07b0 0x7efdf40a2c70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0ff1a0 tx=0x7efe00009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:56.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0affd700 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b57980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0bfff700 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050b5a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 -- 172.21.15.3:0/625811807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 msgr2=0x556050b5a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050b5a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 -- 172.21.15.3:0/625811807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 msgr2=0x556050b57980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b57980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0b7fe700 1 -- 172.21.15.3:0/625811807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560508cbe60 con 0x556050b4d790 2023-09-10T01:06:56.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.951+0000 7efe0affd700 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b57980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:56.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.952+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b4db70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efdfc005ea0 tx=0x7efdfc005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4e593db251a79 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:56.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.952+0000 7efdf1ffb700 1 -- 172.21.15.3:0/625811807 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc01d070 con 0x556050b4d790 2023-09-10T01:06:56.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.952+0000 7efdf1ffb700 1 -- 172.21.15.3:0/625811807 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdfc01e040 con 0x556050b4d790 2023-09-10T01:06:56.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.952+0000 7efdf1ffb700 1 -- 172.21.15.3:0/625811807 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0048a0 con 0x556050b4d790 2023-09-10T01:06:56.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.952+0000 7efe25a31b80 1 -- 172.21.15.3:0/625811807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 msgr2=0x556050b4db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:56.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.952+0000 7efe25a31b80 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b4db70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efdfc005ea0 tx=0x7efdfc005ed0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 -- 172.21.15.3:0/625811807 shutdown_connections 2023-09-10T01:06:56.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050b5a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b4db70 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 --2- 172.21.15.3:0/625811807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b57980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 -- 172.21.15.3:0/625811807 >> 172.21.15.3:0/625811807 conn(0x556050ac2110 msgr2=0x556050b48c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:56.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 -- 172.21.15.3:0/625811807 shutdown_connections 2023-09-10T01:06:56.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 -- 172.21.15.3:0/625811807 wait complete. 2023-09-10T01:06:56.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.953+0000 7efe25a31b80 1 -- start start 2023-09-10T01:06:56.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.954+0000 7efe25a31b80 1 Processor -- start 2023-09-10T01:06:56.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.954+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b57870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.954+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4e0b0 0x556050b54a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.954+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b55040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.954+0000 7efe25a31b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050a3dbf0 con 0x556050b4d790 2023-09-10T01:06:56.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.954+0000 7efe25a31b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050a3d8f0 con 0x556050b57ec0 2023-09-10T01:06:56.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.954+0000 7efe25a31b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560509219a0 con 0x556050b4e0b0 2023-09-10T01:06:56.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe25a31b80 10 client.0 _ll_register_callbacks cb 0x556050987b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:56.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4e0b0 0x556050b54a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b55040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b55040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43466/0 (socket says 172.21.15.3:43466) 2023-09-10T01:06:56.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4e0b0 0x556050b54a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40270/0 (socket says 172.21.15.3:40270) 2023-09-10T01:06:56.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 -- 172.21.15.3:0/657313886 learned_addr learned my addr 172.21.15.3:0/657313886 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:56.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b57870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 -- 172.21.15.3:0/657313886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4e0b0 msgr2=0x556050b54a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4e0b0 0x556050b54a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 -- 172.21.15.3:0/657313886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 msgr2=0x556050b57870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b57870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 -- 172.21.15.3:0/657313886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe00017070 con 0x556050b57ec0 2023-09-10T01:06:56.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe0bfff700 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b55040 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efe0400c6f0 tx=0x7efe040989b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b522bc61228af90 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:56.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efdeaffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:56.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efdeaffd700 1 -- 172.21.15.3:0/657313886 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe040b0070 con 0x556050b57ec0 2023-09-10T01:06:56.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efdeaffd700 1 -- 172.21.15.3:0/657313886 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efe0401b320 con 0x556050b57ec0 2023-09-10T01:06:56.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.955+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560509f7570 con 0x556050b57ec0 2023-09-10T01:06:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efdeaffd700 1 -- 172.21.15.3:0/657313886 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe04099750 con 0x556050b57ec0 2023-09-10T01:06:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 10 client.26783 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7efe0c10ddb0 con 0x556050b57ec0 2023-09-10T01:06:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efdeaffd700 1 -- 172.21.15.3:0/657313886 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5560509f7570 con 0x556050b57ec0 2023-09-10T01:06:56.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efdeaffd700 1 -- 172.21.15.3:0/657313886 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efe040b1260 con 0x556050b57ec0 2023-09-10T01:06:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 10 client.26783 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 10 client.26783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 20 client.26783 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050b67920 0x556050b69d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 4 client.26783 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556050ab4920 con 0x556050b67920 2023-09-10T01:06:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.956+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050b67920 0x556050b69d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.957+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050b67920 0x556050b69d40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=442fb838a111765f server_cookie=12501db7ca5e13a0 in_seq=0 out_seq=0 2023-09-10T01:06:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.957+0000 7efdeaffd700 10 client.26783 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.957+0000 7efdeaffd700 1 -- 172.21.15.3:0/657313886 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556050ab4920 con 0x556050b67920 2023-09-10T01:06:56.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.957+0000 7efdeaffd700 10 client.26783 handle_command_reply: tid=0 2023-09-10T01:06:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.957+0000 7efe25a31b80 1 client.26783 shutdown 2023-09-10T01:06:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 msgr2=0x556050b55040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b55040 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efe0400c6f0 tx=0x7efe040989b0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 shutdown_connections 2023-09-10T01:06:56.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050b67920 0x556050b69d40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b55040 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b57870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 --2- 172.21.15.3:0/657313886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4e0b0 0x556050b54a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 >> 172.21.15.3:0/657313886 conn(0x556050ac2110 msgr2=0x556050b48800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:56.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efdeaffd700 0 client.26783 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:56.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 shutdown_connections 2023-09-10T01:06:56.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 1 -- 172.21.15.3:0/657313886 wait complete. 2023-09-10T01:06:56.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.958+0000 7efe25a31b80 20 client.26783 trim_cache size 0 max 16384 2023-09-10T01:06:56.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.978+0000 7efe25a31b80 1 Processor -- start 2023-09-10T01:06:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.978+0000 7efe25a31b80 1 -- start start 2023-09-10T01:06:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4d790 0x556050b5ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4e0b0 0x556050b573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b4fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe25a31b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050b4e4d0 con 0x556050b4e0b0 2023-09-10T01:06:56.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe25a31b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050b49220 con 0x556050b57ec0 2023-09-10T01:06:56.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe25a31b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050b5ef50 con 0x556050b4d790 2023-09-10T01:06:56.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4d790 0x556050b5ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4d790 0x556050b5ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40284/0 (socket says 172.21.15.3:40284) 2023-09-10T01:06:56.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 -- 172.21.15.3:0/1795882025 learned_addr learned my addr 172.21.15.3:0/1795882025 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:56.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0bfff700 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b4fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0affd700 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4e0b0 0x556050b573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 -- 172.21.15.3:0/1795882025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 msgr2=0x556050b4fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b4fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 -- 172.21.15.3:0/1795882025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4e0b0 msgr2=0x556050b573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4e0b0 0x556050b573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.979+0000 7efe0b7fe700 1 -- 172.21.15.3:0/1795882025 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560508b4ce0 con 0x556050b4d790 2023-09-10T01:06:56.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.980+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4d790 0x556050b5ea10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efdfc005b00 tx=0x7efdfc0058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee76d5c23dbfd6d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:56.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.980+0000 7efdf27fc700 1 -- 172.21.15.3:0/1795882025 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc031070 con 0x556050b4d790 2023-09-10T01:06:56.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.980+0000 7efdf27fc700 1 -- 172.21.15.3:0/1795882025 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdfc00b310 con 0x556050b4d790 2023-09-10T01:06:56.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.980+0000 7efdf27fc700 1 -- 172.21.15.3:0/1795882025 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc01d040 con 0x556050b4d790 2023-09-10T01:06:56.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 -- 172.21.15.3:0/1795882025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4d790 msgr2=0x556050b5ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:56.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4d790 0x556050b5ea10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efdfc005b00 tx=0x7efdfc0058e0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 -- 172.21.15.3:0/1795882025 shutdown_connections 2023-09-10T01:06:56.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b4d790 0x556050b5ea10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b57ec0 0x556050b4fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1795882025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4e0b0 0x556050b573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 -- 172.21.15.3:0/1795882025 >> 172.21.15.3:0/1795882025 conn(0x556050ac2110 msgr2=0x556050b48770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:56.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 -- 172.21.15.3:0/1795882025 shutdown_connections 2023-09-10T01:06:56.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.981+0000 7efe25a31b80 1 -- 172.21.15.3:0/1795882025 wait complete. 2023-09-10T01:06:56.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.982+0000 7efe25a31b80 1 -- start start 2023-09-10T01:06:56.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.982+0000 7efe25a31b80 1 Processor -- start 2023-09-10T01:06:56.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b5d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b5db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe25a31b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050bbfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe25a31b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050b5e220 con 0x556050b4d790 2023-09-10T01:06:56.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe25a31b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050bc00b0 con 0x556050b4e0b0 2023-09-10T01:06:56.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe25a31b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556050bc01f0 con 0x556050b57ec0 2023-09-10T01:06:56.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe25a31b80 10 client.0 _ll_register_callbacks cb 0x7efe04098e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:56.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe0affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b5db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe0bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050bbfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b5d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe0affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b5db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43488/0 (socket says 172.21.15.3:43488) 2023-09-10T01:06:56.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.983+0000 7efe0affd700 1 -- 172.21.15.3:0/1172352570 learned_addr learned my addr 172.21.15.3:0/1172352570 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:56.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe0affd700 1 -- 172.21.15.3:0/1172352570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 msgr2=0x556050bbfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe0affd700 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050bbfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe0affd700 1 -- 172.21.15.3:0/1172352570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 msgr2=0x556050b5d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:56.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe0affd700 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b5d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:56.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe0affd700 1 -- 172.21.15.3:0/1172352570 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560508cbe60 con 0x556050b4e0b0 2023-09-10T01:06:56.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe0bfff700 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050bbfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:56.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe0affd700 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b5db30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efe00009be0 tx=0x7efe000047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c06bdf8652938499 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:56.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efdeaffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:56.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efdeaffd700 1 -- 172.21.15.3:0/1172352570 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe00022040 con 0x556050b4e0b0 2023-09-10T01:06:56.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556050a989a0 con 0x556050b4e0b0 2023-09-10T01:06:56.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.984+0000 7efdeaffd700 1 -- 172.21.15.3:0/1172352570 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7efe0001cbf0 con 0x556050b4e0b0 2023-09-10T01:06:56.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.985+0000 7efdeaffd700 1 -- 172.21.15.3:0/1172352570 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe0001cdb0 con 0x556050b4e0b0 2023-09-10T01:06:56.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.985+0000 7efe25a31b80 10 client.26789 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:56.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.985+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5560508f22d0 con 0x556050b4e0b0 2023-09-10T01:06:56.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.985+0000 7efdeaffd700 1 -- 172.21.15.3:0/1172352570 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556050a989a0 con 0x556050b4e0b0 2023-09-10T01:06:56.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efdeaffd700 1 -- 172.21.15.3:0/1172352570 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efe00028430 con 0x556050b4e0b0 2023-09-10T01:06:56.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efe25a31b80 10 client.26789 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:56.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efe25a31b80 10 client.26789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:56.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efe25a31b80 20 client.26789 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:56.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050bc1ab0 0x556050bc3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:56.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efe25a31b80 4 client.26789 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:06:56.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556050a59130 con 0x556050bc1ab0 2023-09-10T01:06:56.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.986+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050bc1ab0 0x556050bc3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:56.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.987+0000 7efe0b7fe700 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050bc1ab0 0x556050bc3ed0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=644776048e0e19e8 server_cookie=169b04f35c5861ba in_seq=0 out_seq=0 2023-09-10T01:06:56.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.987+0000 7efdeaffd700 10 client.26789 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:56.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efdeaffd700 1 -- 172.21.15.3:0/1172352570 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556050a59130 con 0x556050bc1ab0 2023-09-10T01:06:56.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efdeaffd700 10 client.26789 handle_command_reply: tid=0 2023-09-10T01:06:57.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 client.26789 shutdown 2023-09-10T01:06:57.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 msgr2=0x556050b5db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b5db30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efe00009be0 tx=0x7efe000047c0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 shutdown_connections 2023-09-10T01:06:57.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556050bc1ab0 0x556050bc3ed0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556050b57ec0 0x556050bbfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556050b4d790 0x556050b5d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 --2- 172.21.15.3:0/1172352570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556050b4e0b0 0x556050b5db30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efdeaffd700 0 client.26789 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:57.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 >> 172.21.15.3:0/1172352570 conn(0x556050ac2110 msgr2=0x556050b4fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 shutdown_connections 2023-09-10T01:06:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 1 -- 172.21.15.3:0/1172352570 wait complete. 2023-09-10T01:06:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.988+0000 7efe25a31b80 20 client.26789 trim_cache size 0 max 16384 2023-09-10T01:06:57.003 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:06:57.003 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:06:57.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "a1254f15-5165-4848-b676-623a7dc67570", 2023-09-10T01:06:57.003 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:06:57.003 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:06:57.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efdf40a07b0 msgr2=0x7efdf40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efdf40a07b0 0x7efdf40a2c70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0ff1a0 tx=0x7efe00009260 comp rx=0 tx=0).stop 2023-09-10T01:06:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe0c10ac70 msgr2=0x7efe0c1965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe0c10ac70 0x7efe0c1965b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efe0400efc0 tx=0x7efe0400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 shutdown_connections 2023-09-10T01:06:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efdf40a07b0 0x7efdf40a2c70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe0c1018a0 0x7efe0c19c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efe0c10ac70 0x7efe0c1965b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 --2- 172.21.15.3:0/3213282006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe0c1021c0 0x7efe0c19ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.989+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 >> 172.21.15.3:0/3213282006 conn(0x7efe0c076710 msgr2=0x7efe0c0fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.990+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 shutdown_connections 2023-09-10T01:06:57.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:56.990+0000 7efe121ff700 1 -- 172.21.15.3:0/3213282006 wait complete. 2023-09-10T01:06:57.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:06:57.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 -- 172.21.15.3:0/2780608236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 msgr2=0x7f3d94106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 --2- 172.21.15.3:0/2780608236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d94106a30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3d84009990 tx=0x7f3d84009ca0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 -- 172.21.15.3:0/2780608236 shutdown_connections 2023-09-10T01:06:57.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 --2- 172.21.15.3:0/2780608236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d94106a30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 --2- 172.21.15.3:0/2780608236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d94101000 0x7f3d9410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 --2- 172.21.15.3:0/2780608236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d94100640 0x7f3d94100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 -- 172.21.15.3:0/2780608236 >> 172.21.15.3:0/2780608236 conn(0x7f3d940fc170 msgr2=0x7f3d940fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 -- 172.21.15.3:0/2780608236 shutdown_connections 2023-09-10T01:06:57.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.268+0000 7f3d995ff700 1 -- 172.21.15.3:0/2780608236 wait complete. 2023-09-10T01:06:57.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.269+0000 7f3d995ff700 1 Processor -- start 2023-09-10T01:06:57.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.269+0000 7f3d995ff700 1 -- start start 2023-09-10T01:06:57.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d995ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d94100640 0x7f3d94072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d995ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d94101000 0x7f3d94072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d995ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d941a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d995ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d94073670 con 0x7f3d94100640 2023-09-10T01:06:57.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d995ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d941a4df0 con 0x7f3d94101000 2023-09-10T01:06:57.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d995ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d941a4f30 con 0x7f3d94106650 2023-09-10T01:06:57.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d92ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d94100640 0x7f3d94072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d937fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d941a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d94101000 0x7f3d94072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d937fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d941a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40320/0 (socket says 172.21.15.3:40320) 2023-09-10T01:06:57.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d937fe700 1 -- 172.21.15.3:0/1705374809 learned_addr learned my addr 172.21.15.3:0/1705374809 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:57.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d927fc700 1 -- 172.21.15.3:0/1705374809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 msgr2=0x7f3d941a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d927fc700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d941a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d927fc700 1 -- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d94100640 msgr2=0x7f3d94072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d927fc700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d94100640 0x7f3d94072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d927fc700 1 -- 172.21.15.3:0/1705374809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d840095f0 con 0x7f3d94101000 2023-09-10T01:06:57.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.270+0000 7f3d937fe700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d941a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:57.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.271+0000 7f3d927fc700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d94101000 0x7f3d94072e40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800d750 tx=0x7f3d8800da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:57.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.271+0000 7f3d7bfff700 1 -- 172.21.15.3:0/1705374809 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d88004020 con 0x7f3d94101000 2023-09-10T01:06:57.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.271+0000 7f3d7bfff700 1 -- 172.21.15.3:0/1705374809 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d88004830 con 0x7f3d94101000 2023-09-10T01:06:57.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.271+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d941a5190 con 0x7f3d94101000 2023-09-10T01:06:57.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.271+0000 7f3d7bfff700 1 -- 172.21.15.3:0/1705374809 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d880054c0 con 0x7f3d94101000 2023-09-10T01:06:57.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.271+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d941a5670 con 0x7f3d94101000 2023-09-10T01:06:57.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.274+0000 7f3d7bfff700 1 -- 172.21.15.3:0/1705374809 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3d880049a0 con 0x7f3d94101000 2023-09-10T01:06:57.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 Processor -- start 2023-09-10T01:06:57.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 -- start start 2023-09-10T01:06:57.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3d7bfff700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d7c0a07b0 0x7f3d7c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3d7bfff700 1 -- 172.21.15.3:0/1705374809 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3d880cbc90 con 0x7f3d94101000 2023-09-10T01:06:57.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c65083af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65090230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64fdc730 con 0x564c65083710 2023-09-10T01:06:57.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64fda050 con 0x564c6508de40 2023-09-10T01:06:57.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64fd5940 con 0x564c65084030 2023-09-10T01:06:57.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.276+0000 7f3d937fe700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d7c0a07b0 0x7f3d7c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d92ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c65083af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d937fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65090230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40332/0 (socket says 172.21.15.3:40332) 2023-09-10T01:06:57.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 -- 172.21.15.3:0/2514755187 learned_addr learned my addr 172.21.15.3:0/2514755187 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:57.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 -- 172.21.15.3:0/2514755187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 msgr2=0x564c65090230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65090230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 -- 172.21.15.3:0/2514755187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 msgr2=0x564c65083af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c65083af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 -- 172.21.15.3:0/2514755187 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c64e01de0 con 0x564c65084030 2023-09-10T01:06:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c65083af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d937fe700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d7c0a07b0 0x7f3d7c0a2c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009fd0 tx=0x7f3d8c009150 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d927fc700 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508d900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d880957c0 tx=0x7f3d880994a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40c63b6129b11614 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d797fa700 1 -- 172.21.15.3:0/2514755187 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8809cb80 con 0x564c65084030 2023-09-10T01:06:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d797fa700 1 -- 172.21.15.3:0/2514755187 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d880a0dc0 con 0x564c65084030 2023-09-10T01:06:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.277+0000 7f3d797fa700 1 -- 172.21.15.3:0/2514755187 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d88098240 con 0x564c65084030 2023-09-10T01:06:57.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2514755187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 msgr2=0x564c6508d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508d900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d880957c0 tx=0x7f3d880994a0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2514755187 shutdown_connections 2023-09-10T01:06:57.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65090230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c65083af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2514755187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508d900 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2514755187 >> 172.21.15.3:0/2514755187 conn(0x564c64ff8090 msgr2=0x564c6507ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2514755187 shutdown_connections 2023-09-10T01:06:57.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.278+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2514755187 wait complete. 2023-09-10T01:06:57.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.279+0000 7f3dacf98b80 1 -- start start 2023-09-10T01:06:57.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.279+0000 7f3dacf98b80 1 Processor -- start 2023-09-10T01:06:57.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65083710 0x564c6508ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c6508de40 0x564c6508b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64f73b70 con 0x564c6508de40 2023-09-10T01:06:57.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64f73870 con 0x564c65083710 2023-09-10T01:06:57.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64e57920 con 0x564c65084030 2023-09-10T01:06:57.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3dacf98b80 10 client.0 _ll_register_callbacks cb 0x564c64ebda80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:57.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65083710 0x564c6508ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65083710 0x564c6508ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43532/0 (socket says 172.21.15.3:43532) 2023-09-10T01:06:57.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 -- 172.21.15.3:0/1677472710 learned_addr learned my addr 172.21.15.3:0/1677472710 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:57.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d927fc700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d937fe700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c6508de40 0x564c6508b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 -- 172.21.15.3:0/1677472710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 msgr2=0x564c6508b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 -- 172.21.15.3:0/1677472710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c6508de40 msgr2=0x564c6508b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c6508de40 0x564c6508b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d92ffd700 1 -- 172.21.15.3:0/1677472710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9410de10 con 0x564c65083710 2023-09-10T01:06:57.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.280+0000 7f3d927fc700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:57.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65083710 0x564c6508ac30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3d84015000 tx=0x7f3d84005c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9774d5169107d9d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:57.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3d76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:57.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3d76ff5700 1 -- 172.21.15.3:0/1677472710 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8401d070 con 0x564c65083710 2023-09-10T01:06:57.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564c64f2d4f0 con 0x564c65083710 2023-09-10T01:06:57.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3d76ff5700 1 -- 172.21.15.3:0/1677472710 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3d84004700 con 0x564c65083710 2023-09-10T01:06:57.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3d76ff5700 1 -- 172.21.15.3:0/1677472710 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d840056f0 con 0x564c65083710 2023-09-10T01:06:57.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3dacf98b80 10 client.26801 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:57.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.281+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f3d941a5670 con 0x564c65083710 2023-09-10T01:06:57.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3d76ff5700 1 -- 172.21.15.3:0/1677472710 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564c64f2d4f0 con 0x564c65083710 2023-09-10T01:06:57.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3d76ff5700 1 -- 172.21.15.3:0/1677472710 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3d8400fa30 con 0x564c65083710 2023-09-10T01:06:57.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3dacf98b80 10 client.26801 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:57.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3dacf98b80 10 client.26801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:57.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3dacf98b80 20 client.26801 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:57.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c6509da50 0x564c6509fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3dacf98b80 4 client.26801 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:06:57.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c64fea8a0 con 0x564c6509da50 2023-09-10T01:06:57.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.282+0000 7f3d927fc700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c6509da50 0x564c6509fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.283+0000 7f3d927fc700 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c6509da50 0x564c6509fe70 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aff18851f4961f5e server_cookie=6c3bd6af6829947f in_seq=0 out_seq=0 2023-09-10T01:06:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.283+0000 7f3d76ff5700 10 client.26801 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.283+0000 7f3d76ff5700 1 -- 172.21.15.3:0/1677472710 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564c64fea8a0 con 0x564c6509da50 2023-09-10T01:06:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.283+0000 7f3d76ff5700 10 client.26801 handle_command_reply: tid=0 2023-09-10T01:06:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.283+0000 7f3dacf98b80 1 client.26801 shutdown 2023-09-10T01:06:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65083710 msgr2=0x564c6508ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65083710 0x564c6508ac30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3d84015000 tx=0x7f3d84005c20 comp rx=0 tx=0).stop 2023-09-10T01:06:57.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 shutdown_connections 2023-09-10T01:06:57.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c6509da50 0x564c6509fe70 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c6508de40 0x564c6508b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65083710 0x564c6508ac30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3d76ff5700 0 client.26801 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:57.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/1677472710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65084030 0x564c6508b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 >> 172.21.15.3:0/1677472710 conn(0x564c64ff8090 msgr2=0x564c6507e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 shutdown_connections 2023-09-10T01:06:57.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 1 -- 172.21.15.3:0/1677472710 wait complete. 2023-09-10T01:06:57.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.284+0000 7f3dacf98b80 20 client.26801 trim_cache size 0 max 16384 2023-09-10T01:06:57.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.302+0000 7f3dacf98b80 1 Processor -- start 2023-09-10T01:06:57.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.302+0000 7f3dacf98b80 1 -- start start 2023-09-10T01:06:57.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65083710 0x564c6508b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65084030 0x564c65082e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65087ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c65085ae0 con 0x564c65084030 2023-09-10T01:06:57.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64fea8a0 con 0x564c6508de40 2023-09-10T01:06:57.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c64f2d4f0 con 0x564c65083710 2023-09-10T01:06:57.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d92ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65083710 0x564c6508b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d927fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65084030 0x564c65082e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d92ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65083710 0x564c6508b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40350/0 (socket says 172.21.15.3:40350) 2023-09-10T01:06:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d92ffd700 1 -- 172.21.15.3:0/2675999934 learned_addr learned my addr 172.21.15.3:0/2675999934 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d937fe700 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65087ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d927fc700 1 -- 172.21.15.3:0/2675999934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65083710 msgr2=0x564c6508b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d927fc700 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65083710 0x564c6508b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d927fc700 1 -- 172.21.15.3:0/2675999934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 msgr2=0x564c65087ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d927fc700 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65087ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d927fc700 1 -- 172.21.15.3:0/2675999934 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c64deac60 con 0x564c65084030 2023-09-10T01:06:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.303+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65083710 0x564c6508b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:06:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.304+0000 7f3d927fc700 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65084030 0x564c65082e10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d88098ce0 tx=0x7f3d8809b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6356010cb9676140 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:57.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.304+0000 7f3d79ffb700 1 -- 172.21.15.3:0/2675999934 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d88064070 con 0x564c65084030 2023-09-10T01:06:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.304+0000 7f3d79ffb700 1 -- 172.21.15.3:0/2675999934 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d88005620 con 0x564c65084030 2023-09-10T01:06:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.304+0000 7f3d79ffb700 1 -- 172.21.15.3:0/2675999934 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d880984a0 con 0x564c65084030 2023-09-10T01:06:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2675999934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65084030 msgr2=0x564c65082e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65084030 0x564c65082e10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d88098ce0 tx=0x7f3d8809b040 comp rx=0 tx=0).stop 2023-09-10T01:06:57.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2675999934 shutdown_connections 2023-09-10T01:06:57.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c65083710 0x564c6508b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c6508de40 0x564c65087ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/2675999934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65084030 0x564c65082e10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2675999934 >> 172.21.15.3:0/2675999934 conn(0x564c64ff8090 msgr2=0x564c65082700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2675999934 shutdown_connections 2023-09-10T01:06:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 -- 172.21.15.3:0/2675999934 wait complete. 2023-09-10T01:06:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 -- start start 2023-09-10T01:06:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.305+0000 7f3dacf98b80 1 Processor -- start 2023-09-10T01:06:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c650946d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65084030 0x564c65092950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3dacf98b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c6508de40 0x564c65092e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c65094fa0 con 0x564c65083710 2023-09-10T01:06:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c650933d0 con 0x564c65084030 2023-09-10T01:06:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3dacf98b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c65093590 con 0x564c6508de40 2023-09-10T01:06:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3dacf98b80 10 client.0 _ll_register_callbacks cb 0x7f3d8400bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:06:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3d937fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c6508de40 0x564c65092e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3d927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65084030 0x564c65092950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3d927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65084030 0x564c65092950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43538/0 (socket says 172.21.15.3:43538) 2023-09-10T01:06:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3d927fc700 1 -- 172.21.15.3:0/26412137 learned_addr learned my addr 172.21.15.3:0/26412137 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:06:57.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3d937fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c6508de40 0x564c65092e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40362/0 (socket says 172.21.15.3:40362) 2023-09-10T01:06:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c650946d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.306+0000 7f3d927fc700 1 -- 172.21.15.3:0/26412137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c6508de40 msgr2=0x564c65092e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d927fc700 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c6508de40 0x564c65092e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d927fc700 1 -- 172.21.15.3:0/26412137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 msgr2=0x564c650946d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:06:57.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d927fc700 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c650946d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d927fc700 1 -- 172.21.15.3:0/26412137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d941a5670 con 0x564c65084030 2023-09-10T01:06:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d927fc700 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65084030 0x564c65092950 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3d880957c0 tx=0x7f3d88098be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5831bbc271762ec9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:06:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:06:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d76ff5700 1 -- 172.21.15.3:0/26412137 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d88064040 con 0x564c65084030 2023-09-10T01:06:57.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d880a0dc0 con 0x564c65084030 2023-09-10T01:06:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d76ff5700 1 -- 172.21.15.3:0/26412137 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3d88098fb0 con 0x564c65084030 2023-09-10T01:06:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d76ff5700 1 -- 172.21.15.3:0/26412137 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d880991b0 con 0x564c65084030 2023-09-10T01:06:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3dacf98b80 10 client.26807 fetch_fsmap learned FSMap version 24 2023-09-10T01:06:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564c64e01de0 con 0x564c65084030 2023-09-10T01:06:57.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.307+0000 7f3d76ff5700 1 -- 172.21.15.3:0/26412137 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d880a0dc0 con 0x564c65084030 2023-09-10T01:06:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3d76ff5700 1 -- 172.21.15.3:0/26412137 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3d880b6ab0 con 0x564c65084030 2023-09-10T01:06:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3dacf98b80 10 client.26807 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:06:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3dacf98b80 10 client.26807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:06:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3dacf98b80 20 client.26807 populate_metadata read hostname 'smithi003' 2023-09-10T01:06:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c650f27c0 0x564c650f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:06:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3dacf98b80 4 client.26807 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:06:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564c64fce920 con 0x564c650f27c0 2023-09-10T01:06:57.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.308+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c650f27c0 0x564c650f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:06:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3d92ffd700 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c650f27c0 0x564c650f4bb0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2b4d2c2b0223ed0 server_cookie=bdfec5974ac4745f in_seq=0 out_seq=0 2023-09-10T01:06:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3d76ff5700 10 client.26807 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3d76ff5700 1 -- 172.21.15.3:0/26412137 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564c64fce920 con 0x564c650f27c0 2023-09-10T01:06:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3d76ff5700 10 client.26807 handle_command_reply: tid=0 2023-09-10T01:06:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 client.26807 shutdown 2023-09-10T01:06:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65084030 msgr2=0x564c65092950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65084030 0x564c65092950 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3d880957c0 tx=0x7f3d88098be0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 shutdown_connections 2023-09-10T01:06:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564c650f27c0 0x564c650f4bb0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c6508de40 0x564c65092e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564c65083710 0x564c650946d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 --2- 172.21.15.3:0/26412137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564c65084030 0x564c65092950 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3d76ff5700 0 client.26807 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:06:57.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 >> 172.21.15.3:0/26412137 conn(0x564c64ff8090 msgr2=0x564c65082f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 shutdown_connections 2023-09-10T01:06:57.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.309+0000 7f3dacf98b80 1 -- 172.21.15.3:0/26412137 wait complete. 2023-09-10T01:06:57.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3dacf98b80 20 client.26807 trim_cache size 0 max 16384 2023-09-10T01:06:57.328 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:06:57.328 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:06:57.329 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:06:57.329 INFO:teuthology.orchestra.run.smithi003.stdout: "a1254f15-5165-4848-b676-623a7dc67570": { 2023-09-10T01:06:57.329 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:06:57.329 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "a1254f15-5165-4848-b676-623a7dc67570", 2023-09-10T01:06:57.329 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:06:57.330 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:06:57.330 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:06:57.330 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:06:57.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d7c0a07b0 msgr2=0x7f3d7c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d7c0a07b0 0x7f3d7c0a2c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009fd0 tx=0x7f3d8c009150 comp rx=0 tx=0).stop 2023-09-10T01:06:57.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d94101000 msgr2=0x7f3d94072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:06:57.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d94101000 0x7f3d94072e40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800d750 tx=0x7f3d8800da60 comp rx=0 tx=0).stop 2023-09-10T01:06:57.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 shutdown_connections 2023-09-10T01:06:57.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d7c0a07b0 0x7f3d7c0a2c70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d94106650 0x7f3d941a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d94100640 0x7f3d94072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 --2- 172.21.15.3:0/1705374809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d94101000 0x7f3d94072e40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:06:57.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 >> 172.21.15.3:0/1705374809 conn(0x7f3d940fc170 msgr2=0x7f3d940fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:06:57.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 shutdown_connections 2023-09-10T01:06:57.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:06:57.310+0000 7f3d995ff700 1 -- 172.21.15.3:0/1705374809 wait complete. 2023-09-10T01:06:57.334 INFO:tasks.cephfs.filesystem:scrub status for tag:a1254f15-5165-4848-b676-623a7dc67570 - {'path': '/', 'tag': 'a1254f15-5165-4848-b676-623a7dc67570', 'options': 'recursive,force'} 2023-09-10T01:06:57.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 289750 9.51 MB/sec execute 39 sec latency 17.299 ms 2023-09-10T01:06:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: pgmap v115: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2023-09-10T01:06:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:06:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:06:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:06:58.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:06:58.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:06:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: pgmap v115: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2023-09-10T01:06:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:06:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:06:58.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:06:58.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:06:58.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:06:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: pgmap v115: 129 pgs: 129 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2023-09-10T01:06:58.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:06:58.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:06:58.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:06:58.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:06:58.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.564 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:06:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:06:58.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 291271 9.57 MB/sec execute 40 sec latency 37.505 ms 2023-09-10T01:06:59.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 293125 9.48 MB/sec execute 41 sec latency 14.881 ms 2023-09-10T01:07:00.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:00 smithi136 ceph-mon[133708]: pgmap v116: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 411 op/s 2023-09-10T01:07:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:00 smithi092 ceph-mon[134067]: pgmap v116: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 411 op/s 2023-09-10T01:07:00.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:00 smithi003 ceph-mon[97654]: pgmap v116: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 411 op/s 2023-09-10T01:07:00.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 294132 9.45 MB/sec execute 42 sec latency 63.373 ms 2023-09-10T01:07:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 -- 172.21.15.3:0/800577636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 msgr2=0x7f8338100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 --2- 172.21.15.3:0/800577636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f8338100a00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f832c0099c0 tx=0x7f832c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 -- 172.21.15.3:0/800577636 shutdown_connections 2023-09-10T01:07:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 --2- 172.21.15.3:0/800577636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8338106640 0x7f8338106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 --2- 172.21.15.3:0/800577636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8338100f40 0x7f833810d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 --2- 172.21.15.3:0/800577636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f8338100a00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 -- 172.21.15.3:0/800577636 >> 172.21.15.3:0/800577636 conn(0x7f83380fc050 msgr2=0x7f83380fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 -- 172.21.15.3:0/800577636 shutdown_connections 2023-09-10T01:07:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.343+0000 7f833e5ff700 1 -- 172.21.15.3:0/800577636 wait complete. 2023-09-10T01:07:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.344+0000 7f833e5ff700 1 Processor -- start 2023-09-10T01:07:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.344+0000 7f833e5ff700 1 -- start start 2023-09-10T01:07:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f833e5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f833819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f833e5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8338100f40 0x7f833819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f833e5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8338106640 0x7f83381985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f833e5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338198c50 con 0x7f8338100f40 2023-09-10T01:07:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f833e5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338198dc0 con 0x7f8338100580 2023-09-10T01:07:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f833e5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338198f80 con 0x7f8338106640 2023-09-10T01:07:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f8337fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f833819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f8337fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f833819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43566/0 (socket says 172.21.15.3:43566) 2023-09-10T01:07:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f8337fff700 1 -- 172.21.15.3:0/125168446 learned_addr learned my addr 172.21.15.3:0/125168446 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f83377fe700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8338100f40 0x7f833819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f833cb9c700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8338106640 0x7f83381985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f8337fff700 1 -- 172.21.15.3:0/125168446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8338106640 msgr2=0x7f83381985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.345+0000 7f8337fff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8338106640 0x7f83381985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f8337fff700 1 -- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8338100f40 msgr2=0x7f833819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f8337fff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8338100f40 0x7f833819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f8337fff700 1 -- 172.21.15.3:0/125168446 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330009c40 con 0x7f8338100580 2023-09-10T01:07:01.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f8337fff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f833819e570 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f832800e9a0 tx=0x7f832800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:01.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f83357fa700 1 -- 172.21.15.3:0/125168446 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328009790 con 0x7f8338100580 2023-09-10T01:07:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f83357fa700 1 -- 172.21.15.3:0/125168446 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f832800ce80 con 0x7f8338100580 2023-09-10T01:07:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f83357fa700 1 -- 172.21.15.3:0/125168446 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328018680 con 0x7f8338100580 2023-09-10T01:07:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c0095f0 con 0x7f8338100580 2023-09-10T01:07:01.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.346+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83381a5380 con 0x7f8338100580 2023-09-10T01:07:01.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.350+0000 7f83357fa700 1 -- 172.21.15.3:0/125168446 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f8328010430 con 0x7f8338100580 2023-09-10T01:07:01.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 Processor -- start 2023-09-10T01:07:01.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 -- start start 2023-09-10T01:07:01.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f83357fa700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f83200a0900 0x7f83200a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d32580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f83357fa700 1 -- 172.21.15.3:0/125168446 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f8328014070 con 0x7f8338100580 2023-09-10T01:07:01.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d2c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d38f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0c856d0 con 0x5596d0d321a0 2023-09-10T01:07:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0c82ff0 con 0x5596d0d2c300 2023-09-10T01:07:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.351+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0c7e8e0 con 0x5596d0d2ccc0 2023-09-10T01:07:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f833cb9c700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f83200a0900 0x7f83200a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f8337fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d32580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f83377fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d2c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f8337fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d32580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33358/0 (socket says 172.21.15.3:33358) 2023-09-10T01:07:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f8337fff700 1 -- 172.21.15.3:0/3420738357 learned_addr learned my addr 172.21.15.3:0/3420738357 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f833cb9c700 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d38f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f83377fe700 1 -- 172.21.15.3:0/3420738357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 msgr2=0x5596d0d38f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f83377fe700 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d38f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f83377fe700 1 -- 172.21.15.3:0/3420738357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 msgr2=0x5596d0d32580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f83377fe700 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d32580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f83377fe700 1 -- 172.21.15.3:0/3420738357 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596d0aaad80 con 0x5596d0d2c300 2023-09-10T01:07:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f8337fff700 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d32580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.352+0000 7f833cb9c700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f83200a0900 0x7f83200a2dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f833819f270 tx=0x7f833000fdd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.353+0000 7f83377fe700 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d2c780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f832c0098e0 tx=0x7f832c005ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88842d94198a405b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.353+0000 7f831e7fc700 1 -- 172.21.15.3:0/3420738357 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c005270 con 0x5596d0d2c300 2023-09-10T01:07:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.353+0000 7f831e7fc700 1 -- 172.21.15.3:0/3420738357 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f832c01e040 con 0x5596d0d2c300 2023-09-10T01:07:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.353+0000 7f831e7fc700 1 -- 172.21.15.3:0/3420738357 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c00b9f0 con 0x5596d0d2c300 2023-09-10T01:07:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3420738357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 msgr2=0x5596d0d2c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d2c780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f832c0098e0 tx=0x7f832c005ee0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3420738357 shutdown_connections 2023-09-10T01:07:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d38f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d32580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3420738357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d2c780 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3420738357 >> 172.21.15.3:0/3420738357 conn(0x5596d0ca1030 msgr2=0x5596d0d27560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3420738357 shutdown_connections 2023-09-10T01:07:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3420738357 wait complete. 2023-09-10T01:07:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.354+0000 7f8351f0fb80 1 -- start start 2023-09-10T01:07:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.355+0000 7f8351f0fb80 1 Processor -- start 2023-09-10T01:07:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d38e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d36770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0c1cb10 con 0x5596d0d321a0 2023-09-10T01:07:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0c1c810 con 0x5596d0d2ccc0 2023-09-10T01:07:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0b008c0 con 0x5596d0d2c300 2023-09-10T01:07:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8351f0fb80 10 client.0 _ll_register_callbacks cb 0x5596d0b66a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8337fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d38e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f833cb9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d36770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43584/0 (socket says 172.21.15.3:43584) 2023-09-10T01:07:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 -- 172.21.15.3:0/3239034697 learned_addr learned my addr 172.21.15.3:0/3239034697 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8337fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d38e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40412/0 (socket says 172.21.15.3:40412) 2023-09-10T01:07:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 -- 172.21.15.3:0/3239034697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 msgr2=0x5596d0d38e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d38e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 -- 172.21.15.3:0/3239034697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 msgr2=0x5596d0d36770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d36770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f83377fe700 1 -- 172.21.15.3:0/3239034697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83381a5380 con 0x5596d0d2ccc0 2023-09-10T01:07:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f8337fff700 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d38e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.356+0000 7f833cb9c700 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d36770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f83377fe700 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d360f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f832c0036c0 tx=0x7f832c0055e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7787dada9244d583 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f83177fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f83177fe700 1 -- 172.21.15.3:0/3239034697 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c003be0 con 0x5596d0d2ccc0 2023-09-10T01:07:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5596d0bd6490 con 0x5596d0d2ccc0 2023-09-10T01:07:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f83177fe700 1 -- 172.21.15.3:0/3239034697 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f832c003d40 con 0x5596d0d2ccc0 2023-09-10T01:07:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f83177fe700 1 -- 172.21.15.3:0/3239034697 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f832c00b790 con 0x5596d0d2ccc0 2023-09-10T01:07:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f8351f0fb80 10 client.26819 fetch_fsmap learned FSMap version 24 2023-09-10T01:07:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f833810dd00 con 0x5596d0d2ccc0 2023-09-10T01:07:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.357+0000 7f83177fe700 1 -- 172.21.15.3:0/3239034697 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5596d0bd6490 con 0x5596d0d2ccc0 2023-09-10T01:07:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f83177fe700 1 -- 172.21.15.3:0/3239034697 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f832c02b9a0 con 0x5596d0d2ccc0 2023-09-10T01:07:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8351f0fb80 10 client.26819 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:07:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8351f0fb80 10 client.26819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8351f0fb80 20 client.26819 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d45ff0 0x5596d0d48410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8351f0fb80 4 client.26819 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596d0c93840 con 0x5596d0d45ff0 2023-09-10T01:07:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8337fff700 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d45ff0 0x5596d0d48410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f83177fe700 10 client.26819 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.358+0000 7f8337fff700 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d45ff0 0x5596d0d48410 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e91c161ec176d56 server_cookie=5b7231d15e81ff75 in_seq=0 out_seq=0 2023-09-10T01:07:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.359+0000 7f83177fe700 1 -- 172.21.15.3:0/3239034697 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5596d0c93840 con 0x5596d0d45ff0 2023-09-10T01:07:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.359+0000 7f83177fe700 10 client.26819 handle_command_reply: tid=0 2023-09-10T01:07:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.359+0000 7f8351f0fb80 1 client.26819 shutdown 2023-09-10T01:07:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 msgr2=0x5596d0d360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d360f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f832c0036c0 tx=0x7f832c0055e0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 shutdown_connections 2023-09-10T01:07:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d45ff0 0x5596d0d48410 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d38e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d36770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f83177fe700 0 client.26819 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/3239034697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d360f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 >> 172.21.15.3:0/3239034697 conn(0x5596d0ca1030 msgr2=0x5596d0d267a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 shutdown_connections 2023-09-10T01:07:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 1 -- 172.21.15.3:0/3239034697 wait complete. 2023-09-10T01:07:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.360+0000 7f8351f0fb80 20 client.26819 trim_cache size 0 max 16384 2023-09-10T01:07:01.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.376+0000 7f8351f0fb80 1 Processor -- start 2023-09-10T01:07:01.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8351f0fb80 1 -- start start 2023-09-10T01:07:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d3d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d31170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0bd6490 con 0x5596d0d321a0 2023-09-10T01:07:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0d31c20 con 0x5596d0d2ccc0 2023-09-10T01:07:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0d42da0 con 0x5596d0d2c300 2023-09-10T01:07:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8337fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d3d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8337fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d3d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40422/0 (socket says 172.21.15.3:40422) 2023-09-10T01:07:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f8337fff700 1 -- 172.21.15.3:0/1584610307 learned_addr learned my addr 172.21.15.3:0/1584610307 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f83377fe700 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d31170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.377+0000 7f833cb9c700 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f8337fff700 1 -- 172.21.15.3:0/1584610307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 msgr2=0x5596d0d31170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f8337fff700 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d31170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f8337fff700 1 -- 172.21.15.3:0/1584610307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 msgr2=0x5596d0d301f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f8337fff700 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d301f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f8337fff700 1 -- 172.21.15.3:0/1584610307 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596d0a93c00 con 0x5596d0d2c300 2023-09-10T01:07:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f833cb9c700 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:07:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f8337fff700 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d3d480 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f832800c5b0 tx=0x7f8328010b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8db889f9e5f52d2e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f831effd700 1 -- 172.21.15.3:0/1584610307 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83280995a0 con 0x5596d0d2c300 2023-09-10T01:07:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f831effd700 1 -- 172.21.15.3:0/1584610307 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f832809d3d0 con 0x5596d0d2c300 2023-09-10T01:07:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.378+0000 7f831effd700 1 -- 172.21.15.3:0/1584610307 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328099800 con 0x5596d0d2c300 2023-09-10T01:07:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 -- 172.21.15.3:0/1584610307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 msgr2=0x5596d0d3d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d3d480 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f832800c5b0 tx=0x7f8328010b10 comp rx=0 tx=0).stop 2023-09-10T01:07:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 -- 172.21.15.3:0/1584610307 shutdown_connections 2023-09-10T01:07:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2c300 0x5596d0d3d480 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/1584610307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2ccc0 0x5596d0d31170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 -- 172.21.15.3:0/1584610307 >> 172.21.15.3:0/1584610307 conn(0x5596d0ca1030 msgr2=0x5596d0d42690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 -- 172.21.15.3:0/1584610307 shutdown_connections 2023-09-10T01:07:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 -- 172.21.15.3:0/1584610307 wait complete. 2023-09-10T01:07:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.379+0000 7f8351f0fb80 1 -- start start 2023-09-10T01:07:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 1 Processor -- start 2023-09-10T01:07:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d3ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d3b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d3b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0d98e70 con 0x5596d0d321a0 2023-09-10T01:07:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0d98fe0 con 0x5596d0d2c300 2023-09-10T01:07:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596d0d991a0 con 0x5596d0d2ccc0 2023-09-10T01:07:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8351f0fb80 10 client.0 _ll_register_callbacks cb 0x7f83280958e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8337fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d3ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8337fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d3ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43594/0 (socket says 172.21.15.3:43594) 2023-09-10T01:07:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f8337fff700 1 -- 172.21.15.3:0/2663364377 learned_addr learned my addr 172.21.15.3:0/2663364377 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f83377fe700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d3b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.380+0000 7f833cb9c700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d3b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8337fff700 1 -- 172.21.15.3:0/2663364377 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 msgr2=0x5596d0d3b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8337fff700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d3b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8337fff700 1 -- 172.21.15.3:0/2663364377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 msgr2=0x5596d0d3b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8337fff700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d3b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8337fff700 1 -- 172.21.15.3:0/2663364377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c0095f0 con 0x5596d0d2c300 2023-09-10T01:07:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f833cb9c700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d3b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:07:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8337fff700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d3ac20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8328010ac0 tx=0x7f832809d1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ebe6eeaf5f2eb0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f83177fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f83177fe700 1 -- 172.21.15.3:0/2663364377 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328098a90 con 0x5596d0d2c300 2023-09-10T01:07:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f83177fe700 1 -- 172.21.15.3:0/2663364377 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f8328098bf0 con 0x5596d0d2c300 2023-09-10T01:07:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f832c01e040 con 0x5596d0d2c300 2023-09-10T01:07:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f83177fe700 1 -- 172.21.15.3:0/2663364377 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8328099800 con 0x5596d0d2c300 2023-09-10T01:07:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8351f0fb80 10 client.26825 fetch_fsmap learned FSMap version 24 2023-09-10T01:07:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5596d0ad11f0 con 0x5596d0d2c300 2023-09-10T01:07:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.381+0000 7f83177fe700 1 -- 172.21.15.3:0/2663364377 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f832c01e040 con 0x5596d0d2c300 2023-09-10T01:07:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f83177fe700 1 -- 172.21.15.3:0/2663364377 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 24) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f83280b1ba0 con 0x5596d0d2c300 2023-09-10T01:07:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f8351f0fb80 10 client.26825 fetch_fsmap finished waiting for FSMap version 24 2023-09-10T01:07:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f8351f0fb80 10 client.26825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f8351f0fb80 20 client.26825 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d9a580 0x5596d0d9c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f8351f0fb80 4 client.26825 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:07:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5596d0c778c0 con 0x5596d0d9a580 2023-09-10T01:07:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f83377fe700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d9a580 0x5596d0d9c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f83177fe700 10 client.26825 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.382+0000 7f83377fe700 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d9a580 0x5596d0d9c9a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=134ba621c9e1c146 server_cookie=b08fb46112da8956 in_seq=0 out_seq=0 2023-09-10T01:07:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f83177fe700 1 -- 172.21.15.3:0/2663364377 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5596d0c778c0 con 0x5596d0d9a580 2023-09-10T01:07:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f83177fe700 10 client.26825 handle_command_reply: tid=0 2023-09-10T01:07:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 client.26825 shutdown 2023-09-10T01:07:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 msgr2=0x5596d0d3ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d3ac20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8328010ac0 tx=0x7f832809d1f0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 shutdown_connections 2023-09-10T01:07:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5596d0d9a580 0x5596d0d9c9a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5596d0d321a0 0x5596d0d3b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f83177fe700 0 client.26825 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5596d0d2c300 0x5596d0d3ac20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 --2- 172.21.15.3:0/2663364377 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5596d0d2ccc0 0x5596d0d3b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 >> 172.21.15.3:0/2663364377 conn(0x5596d0ca1030 msgr2=0x5596d0d31170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 shutdown_connections 2023-09-10T01:07:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.477+0000 7f8351f0fb80 1 -- 172.21.15.3:0/2663364377 wait complete. 2023-09-10T01:07:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f8351f0fb80 20 client.26825 trim_cache size 0 max 16384 2023-09-10T01:07:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:07:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:07:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:07:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:07:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f83200a0900 msgr2=0x7f83200a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f83200a0900 0x7f83200a2dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f833819f270 tx=0x7f833000fdd0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 msgr2=0x7f833819e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f833819e570 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f832800e9a0 tx=0x7f832800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 shutdown_connections 2023-09-10T01:07:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f83200a0900 0x7f83200a2dc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8338106640 0x7f83381985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8338100580 0x7f833819e570 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 --2- 172.21.15.3:0/125168446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8338100f40 0x7f833819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 >> 172.21.15.3:0/125168446 conn(0x7f83380fc050 msgr2=0x7f83380fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.478+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 shutdown_connections 2023-09-10T01:07:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:01.479+0000 7f833e5ff700 1 -- 172.21.15.3:0/125168446 wait complete. 2023-09-10T01:07:01.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 294988 9.34 MB/sec execute 43 sec latency 62.937 ms 2023-09-10T01:07:02.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:02 smithi136 ceph-mon[133708]: pgmap v117: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:07:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:02 smithi092 ceph-mon[134067]: pgmap v117: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:07:02.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:02 smithi003 ceph-mon[97654]: pgmap v117: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:07:02.748 INFO:tasks.workunit.client.0.smithi003.stdout: 1 297413 9.42 MB/sec execute 44 sec latency 9.782 ms 2023-09-10T01:07:03.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 298572 9.35 MB/sec execute 45 sec latency 61.331 ms 2023-09-10T01:07:04.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:04 smithi136 ceph-mon[133708]: pgmap v118: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:07:04.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:04 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:07:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:04 smithi092 ceph-mon[134067]: pgmap v118: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:07:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:04 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:07:04.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:04 smithi003 ceph-mon[97654]: pgmap v118: 129 pgs: 129 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2023-09-10T01:07:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:04 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:07:04.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 301148 9.44 MB/sec execute 46 sec latency 18.868 ms 2023-09-10T01:07:05.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:05 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:07:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:05 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:07:05.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:05 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:07:05.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:05 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:07:05.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:05 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:07:05.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:05 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:07:05.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:05 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:07:05.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:05 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:07:05.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:05 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:07:05.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 302425 9.38 MB/sec execute 47 sec latency 15.300 ms 2023-09-10T01:07:06.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:06 smithi136 ceph-mon[133708]: pgmap v119: 129 pgs: 129 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 410 op/s 2023-09-10T01:07:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:06 smithi092 ceph-mon[134067]: pgmap v119: 129 pgs: 129 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 410 op/s 2023-09-10T01:07:06.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:06 smithi003 ceph-mon[97654]: pgmap v119: 129 pgs: 129 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 410 op/s 2023-09-10T01:07:06.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 305011 9.53 MB/sec execute 48 sec latency 13.772 ms 2023-09-10T01:07:07.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 306201 9.41 MB/sec execute 49 sec latency 15.971 ms 2023-09-10T01:07:08.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:08 smithi136 ceph-mon[133708]: pgmap v120: 129 pgs: 129 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 2.4 MiB/s wr, 285 op/s 2023-09-10T01:07:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:08 smithi092 ceph-mon[134067]: pgmap v120: 129 pgs: 129 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 2.4 MiB/s wr, 285 op/s 2023-09-10T01:07:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:08 smithi003 ceph-mon[97654]: pgmap v120: 129 pgs: 129 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 2.4 MiB/s wr, 285 op/s 2023-09-10T01:07:08.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 308574 9.53 MB/sec execute 50 sec latency 52.369 ms 2023-09-10T01:07:09.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 309652 9.42 MB/sec execute 51 sec latency 30.069 ms 2023-09-10T01:07:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:10 smithi136 ceph-mon[133708]: pgmap v121: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 398 op/s 2023-09-10T01:07:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:10 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:07:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:10 smithi092 ceph-mon[134067]: pgmap v121: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 398 op/s 2023-09-10T01:07:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:10 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:07:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:10 smithi003 ceph-mon[97654]: pgmap v121: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 398 op/s 2023-09-10T01:07:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:10 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:07:10.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 312245 9.54 MB/sec execute 52 sec latency 10.092 ms 2023-09-10T01:07:11.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 313339 9.42 MB/sec execute 53 sec latency 15.879 ms 2023-09-10T01:07:12.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:12 smithi136 ceph-mon[133708]: pgmap v122: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:07:12.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:12 smithi092 ceph-mon[134067]: pgmap v122: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:07:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:12 smithi003 ceph-mon[97654]: pgmap v122: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:07:12.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 315818 9.54 MB/sec execute 54 sec latency 94.729 ms 2023-09-10T01:07:13.749 INFO:tasks.workunit.client.0.smithi003.stdout: 1 316859 9.43 MB/sec execute 55 sec latency 17.233 ms 2023-09-10T01:07:14.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:14 smithi136 ceph-mon[133708]: pgmap v123: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:07:14.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:14 smithi092 ceph-mon[134067]: pgmap v123: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:07:14.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:14 smithi003 ceph-mon[97654]: pgmap v123: 129 pgs: 129 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:07:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 -- 172.21.15.3:0/4016663714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0067b60 msgr2=0x7f3ee0067fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/4016663714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0067b60 0x7f3ee0067fe0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009990 tx=0x7f3ed4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 -- 172.21.15.3:0/4016663714 shutdown_connections 2023-09-10T01:07:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/4016663714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee0068520 0x7f3ee010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/4016663714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee01066a0 0x7f3ee0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/4016663714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0067b60 0x7f3ee0067fe0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 -- 172.21.15.3:0/4016663714 >> 172.21.15.3:0/4016663714 conn(0x7f3ee0074fe0 msgr2=0x7f3ee00753f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 -- 172.21.15.3:0/4016663714 shutdown_connections 2023-09-10T01:07:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.651+0000 7f3ee5dff700 1 -- 172.21.15.3:0/4016663714 wait complete. 2023-09-10T01:07:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.652+0000 7f3ee5dff700 1 Processor -- start 2023-09-10T01:07:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.652+0000 7f3ee5dff700 1 -- start start 2023-09-10T01:07:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3ee5dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee0067b60 0x7f3ee00ff740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3ee5dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0068520 0x7f3ee00ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3ee5dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee01066a0 0x7f3ee01001c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3ee5dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01a9040 con 0x7f3ee0067b60 2023-09-10T01:07:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3ee5dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01a9180 con 0x7f3ee01066a0 2023-09-10T01:07:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3ee5dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee01a92c0 con 0x7f3ee0068520 2023-09-10T01:07:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0068520 0x7f3ee00ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3edf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee0067b60 0x7f3ee00ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3edffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee01066a0 0x7f3ee01001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0068520 0x7f3ee00ffc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47452/0 (socket says 172.21.15.3:47452) 2023-09-10T01:07:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.653+0000 7f3edeffd700 1 -- 172.21.15.3:0/3219620941 learned_addr learned my addr 172.21.15.3:0/3219620941 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edeffd700 1 -- 172.21.15.3:0/3219620941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee01066a0 msgr2=0x7f3ee01001c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee01066a0 0x7f3ee01001c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edeffd700 1 -- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee0067b60 msgr2=0x7f3ee00ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee0067b60 0x7f3ee00ff740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edeffd700 1 -- 172.21.15.3:0/3219620941 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed0009520 con 0x7f3ee0068520 2023-09-10T01:07:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0068520 0x7f3ee00ffc80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40095c0 tx=0x7f3ed4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edcff9700 1 -- 172.21.15.3:0/3219620941 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed401d070 con 0x7f3ee0068520 2023-09-10T01:07:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edcff9700 1 -- 172.21.15.3:0/3219620941 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ed4022470 con 0x7f3ee0068520 2023-09-10T01:07:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed40095f0 con 0x7f3ee0068520 2023-09-10T01:07:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3edcff9700 1 -- 172.21.15.3:0/3219620941 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4017650 con 0x7f3ee0068520 2023-09-10T01:07:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.654+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee01a9830 con 0x7f3ee0068520 2023-09-10T01:07:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.657+0000 7f3edcff9700 1 -- 172.21.15.3:0/3219620941 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3ed40225e0 con 0x7f3ee0068520 2023-09-10T01:07:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.659+0000 7f3edcff9700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3ec80a0830 0x7f3ec80a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.659+0000 7f3ef9739b80 1 Processor -- start 2023-09-10T01:07:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.659+0000 7f3edcff9700 1 -- 172.21.15.3:0/3219620941 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3ed40cd6a0 con 0x7f3ee0068520 2023-09-10T01:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.659+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3ec80a0830 0x7f3ec80a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.659+0000 7f3ef9739b80 1 -- start start 2023-09-10T01:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63e51a0 0x55f2f63e5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63df300 0x55f2f63df780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63dfcc0 0x55f2f63ebf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f63386d0 con 0x55f2f63e51a0 2023-09-10T01:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f6335ff0 con 0x55f2f63dfcc0 2023-09-10T01:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f63318e0 con 0x55f2f63df300 2023-09-10T01:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3ec80a0830 0x7f3ec80a2cf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009d70 tx=0x7f3ed0009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:14.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63dfcc0 0x55f2f63ebf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63df300 0x55f2f63df780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63dfcc0 0x55f2f63ebf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36468/0 (socket says 172.21.15.3:36468) 2023-09-10T01:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63df300 0x55f2f63df780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47464/0 (socket says 172.21.15.3:47464) 2023-09-10T01:07:14.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edffff700 1 -- 172.21.15.3:0/2803790812 learned_addr learned my addr 172.21.15.3:0/2803790812 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63e51a0 0x55f2f63e5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edffff700 1 -- 172.21.15.3:0/2803790812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63df300 msgr2=0x55f2f63df780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edffff700 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63df300 0x55f2f63df780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.660+0000 7f3edffff700 1 -- 172.21.15.3:0/2803790812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63e51a0 msgr2=0x55f2f63e5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.661+0000 7f3edffff700 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63e51a0 0x55f2f63e5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.661+0000 7f3edffff700 1 -- 172.21.15.3:0/2803790812 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2f615dd80 con 0x55f2f63dfcc0 2023-09-10T01:07:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.661+0000 7f3edffff700 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63dfcc0 0x55f2f63ebf30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8011680 tx=0x7f3ed8009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e918b2550468ccce server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.661+0000 7f3ec5ffb700 1 -- 172.21.15.3:0/2803790812 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed800e040 con 0x55f2f63dfcc0 2023-09-10T01:07:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.661+0000 7f3ec5ffb700 1 -- 172.21.15.3:0/2803790812 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ed800f040 con 0x55f2f63dfcc0 2023-09-10T01:07:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.661+0000 7f3ec5ffb700 1 -- 172.21.15.3:0/2803790812 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80048c0 con 0x55f2f63dfcc0 2023-09-10T01:07:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 -- 172.21.15.3:0/2803790812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63dfcc0 msgr2=0x55f2f63ebf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63dfcc0 0x55f2f63ebf30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8011680 tx=0x7f3ed8009920 comp rx=0 tx=0).stop 2023-09-10T01:07:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 -- 172.21.15.3:0/2803790812 shutdown_connections 2023-09-10T01:07:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63dfcc0 0x55f2f63ebf30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63e51a0 0x55f2f63e5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/2803790812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63df300 0x55f2f63df780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 -- 172.21.15.3:0/2803790812 >> 172.21.15.3:0/2803790812 conn(0x55f2f6354030 msgr2=0x55f2f63da560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 -- 172.21.15.3:0/2803790812 shutdown_connections 2023-09-10T01:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.662+0000 7f3ef9739b80 1 -- 172.21.15.3:0/2803790812 wait complete. 2023-09-10T01:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.663+0000 7f3ef9739b80 1 -- start start 2023-09-10T01:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.663+0000 7f3ef9739b80 1 Processor -- start 2023-09-10T01:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63df300 0x55f2f63ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63dfcc0 0x55f2f63e9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63e51a0 0x55f2f63e9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f62cfb10 con 0x55f2f63dfcc0 2023-09-10T01:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f62cf810 con 0x55f2f63df300 2023-09-10T01:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f61b38c0 con 0x55f2f63e51a0 2023-09-10T01:07:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3ef9739b80 10 client.0 _ll_register_callbacks cb 0x55f2f6219a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63df300 0x55f2f63ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63df300 0x55f2f63ebd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36480/0 (socket says 172.21.15.3:36480) 2023-09-10T01:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 -- 172.21.15.3:0/3343793729 learned_addr learned my addr 172.21.15.3:0/3343793729 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63dfcc0 0x55f2f63e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edffff700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63e51a0 0x55f2f63e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 -- 172.21.15.3:0/3343793729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63e51a0 msgr2=0x55f2f63e9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63e51a0 0x55f2f63e9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 -- 172.21.15.3:0/3343793729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63dfcc0 msgr2=0x55f2f63e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63dfcc0 0x55f2f63e9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edf7fe700 1 -- 172.21.15.3:0/3343793729 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed8011330 con 0x55f2f63df300 2023-09-10T01:07:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.664+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63dfcc0 0x55f2f63e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:07:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63df300 0x55f2f63ebd10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000ed30 tx=0x7f3ed00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1b19147faaf3e65 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3ebeffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3ebeffd700 1 -- 172.21.15.3:0/3343793729 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0022070 con 0x55f2f63df300 2023-09-10T01:07:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2f6289490 con 0x55f2f63df300 2023-09-10T01:07:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3ebeffd700 1 -- 172.21.15.3:0/3343793729 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3ed000ee10 con 0x55f2f63df300 2023-09-10T01:07:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3ebeffd700 1 -- 172.21.15.3:0/3343793729 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ed001ddc0 con 0x55f2f63df300 2023-09-10T01:07:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3ef9739b80 10 client.26837 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.665+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f2f6146c00 con 0x55f2f63df300 2023-09-10T01:07:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ebeffd700 1 -- 172.21.15.3:0/3343793729 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f2f6289490 con 0x55f2f63df300 2023-09-10T01:07:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ebeffd700 1 -- 172.21.15.3:0/3343793729 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3ed001c4a0 con 0x55f2f63df300 2023-09-10T01:07:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ef9739b80 10 client.26837 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:14.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ef9739b80 10 client.26837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:14.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ef9739b80 20 client.26837 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:14.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f63f93e0 0x55f2f63fb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ef9739b80 4 client.26837 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2f6346840 con 0x55f2f63f93e0 2023-09-10T01:07:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.666+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f63f93e0 0x55f2f63fb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.667+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f63f93e0 0x55f2f63fb800 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4f75d40643bbcf6 server_cookie=dc53ba3d8884c0c4 in_seq=0 out_seq=0 2023-09-10T01:07:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.667+0000 7f3ebeffd700 10 client.26837 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.667+0000 7f3ebeffd700 1 -- 172.21.15.3:0/3343793729 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f2f6346840 con 0x55f2f63f93e0 2023-09-10T01:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.667+0000 7f3ebeffd700 10 client.26837 handle_command_reply: tid=0 2023-09-10T01:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.667+0000 7f3ef9739b80 1 client.26837 shutdown 2023-09-10T01:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63df300 msgr2=0x55f2f63ebd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63df300 0x55f2f63ebd10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000ed30 tx=0x7f3ed00047c0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 shutdown_connections 2023-09-10T01:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f63f93e0 0x55f2f63fb800 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63e51a0 0x55f2f63e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63df300 0x55f2f63ebd10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ebeffd700 0 client.26837 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3343793729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63dfcc0 0x55f2f63e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 >> 172.21.15.3:0/3343793729 conn(0x55f2f6354030 msgr2=0x55f2f63d97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 shutdown_connections 2023-09-10T01:07:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3343793729 wait complete. 2023-09-10T01:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.668+0000 7f3ef9739b80 20 client.26837 trim_cache size 0 max 16384 2023-09-10T01:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.684+0000 7f3ef9739b80 1 Processor -- start 2023-09-10T01:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.684+0000 7f3ef9739b80 1 -- start start 2023-09-10T01:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f63f0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f63dea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f63eb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f63ea650 con 0x55f2f63df300 2023-09-10T01:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f63db930 con 0x55f2f63e51a0 2023-09-10T01:07:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f63f0a70 con 0x55f2f63dfcc0 2023-09-10T01:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f63eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f63dea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f63eb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36486/0 (socket says 172.21.15.3:36486) 2023-09-10T01:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edffff700 1 -- 172.21.15.3:0/3051884978 learned_addr learned my addr 172.21.15.3:0/3051884978 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f63dea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47478/0 (socket says 172.21.15.3:47478) 2023-09-10T01:07:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f63f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edeffd700 1 -- 172.21.15.3:0/3051884978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 msgr2=0x55f2f63eb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f63eb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edeffd700 1 -- 172.21.15.3:0/3051884978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 msgr2=0x55f2f63f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f63f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.685+0000 7f3edeffd700 1 -- 172.21.15.3:0/3051884978 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2f61841f0 con 0x55f2f63dfcc0 2023-09-10T01:07:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.686+0000 7f3edeffd700 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f63dea00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40caa00 tx=0x7f3ed401a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abd675c9ea4727d8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.686+0000 7f3ec67fc700 1 -- 172.21.15.3:0/3051884978 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40b6070 con 0x55f2f63dfcc0 2023-09-10T01:07:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.686+0000 7f3ec67fc700 1 -- 172.21.15.3:0/3051884978 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ed409e740 con 0x55f2f63dfcc0 2023-09-10T01:07:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.686+0000 7f3ec67fc700 1 -- 172.21.15.3:0/3051884978 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40cd230 con 0x55f2f63dfcc0 2023-09-10T01:07:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3051884978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 msgr2=0x55f2f63dea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f63dea00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40caa00 tx=0x7f3ed401a040 comp rx=0 tx=0).stop 2023-09-10T01:07:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3051884978 shutdown_connections 2023-09-10T01:07:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f63eb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f63f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/3051884978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f63dea00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3051884978 >> 172.21.15.3:0/3051884978 conn(0x55f2f6354030 msgr2=0x55f2f63de2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3051884978 shutdown_connections 2023-09-10T01:07:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 -- 172.21.15.3:0/3051884978 wait complete. 2023-09-10T01:07:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 -- start start 2023-09-10T01:07:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.687+0000 7f3ef9739b80 1 Processor -- start 2023-09-10T01:07:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f6441370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f64418b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3ef9739b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f644c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f644c740 con 0x55f2f63df300 2023-09-10T01:07:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f644c880 con 0x55f2f63e51a0 2023-09-10T01:07:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3ef9739b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2f644ca10 con 0x55f2f63dfcc0 2023-09-10T01:07:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3ef9739b80 10 client.0 _ll_register_callbacks cb 0x55f2f6219a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f64418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3edf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f6441370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3edeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f64418b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:47486/0 (socket says 172.21.15.3:47486) 2023-09-10T01:07:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3edeffd700 1 -- 172.21.15.3:0/65406187 learned_addr learned my addr 172.21.15.3:0/65406187 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.688+0000 7f3edffff700 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f644c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3edeffd700 1 -- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 msgr2=0x55f2f644c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3edeffd700 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f644c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3edeffd700 1 -- 172.21.15.3:0/65406187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 msgr2=0x55f2f6441370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:14.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3edeffd700 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f6441370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3edeffd700 1 -- 172.21.15.3:0/65406187 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee01a9830 con 0x55f2f63dfcc0 2023-09-10T01:07:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3edeffd700 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f64418b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006010 tx=0x7f3ed4096530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0519271f5881120 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ebeffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:14.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ebeffd700 1 -- 172.21.15.3:0/65406187 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40b6030 con 0x55f2f63dfcc0 2023-09-10T01:07:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ed800f040 con 0x55f2f63dfcc0 2023-09-10T01:07:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ebeffd700 1 -- 172.21.15.3:0/65406187 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f3ed40b7070 con 0x55f2f63dfcc0 2023-09-10T01:07:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ebeffd700 1 -- 172.21.15.3:0/65406187 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ed4022ac0 con 0x55f2f63dfcc0 2023-09-10T01:07:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ef9739b80 10 client.26764 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f2f615c250 con 0x55f2f63dfcc0 2023-09-10T01:07:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.689+0000 7f3ebeffd700 1 -- 172.21.15.3:0/65406187 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ed800f040 con 0x55f2f63dfcc0 2023-09-10T01:07:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ebeffd700 1 -- 172.21.15.3:0/65406187 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3ed40b1070 con 0x55f2f63dfcc0 2023-09-10T01:07:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ef9739b80 10 client.26764 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ef9739b80 10 client.26764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:14.708 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ef9739b80 20 client.26764 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f644de10 0x55f2f6450230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ef9739b80 4 client.26764 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:07:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f2f632a8c0 con 0x55f2f644de10 2023-09-10T01:07:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f644de10 0x55f2f6450230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3edf7fe700 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f644de10 0x55f2f6450230 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3aedc482b47ee6f server_cookie=80220547f28b6c24 in_seq=0 out_seq=0 2023-09-10T01:07:14.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.690+0000 7f3ebeffd700 10 client.26764 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:14.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 319326 9.54 MB/sec execute 56 sec latency 58.815 ms 2023-09-10T01:07:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.793+0000 7f3ebeffd700 1 -- 172.21.15.3:0/65406187 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f2f632a8c0 con 0x55f2f644de10 2023-09-10T01:07:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.793+0000 7f3ebeffd700 10 client.26764 handle_command_reply: tid=0 2023-09-10T01:07:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.793+0000 7f3ef9739b80 1 client.26764 shutdown 2023-09-10T01:07:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.793+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 msgr2=0x55f2f64418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.793+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f64418b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006010 tx=0x7f3ed4096530 comp rx=0 tx=0).stop 2023-09-10T01:07:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 shutdown_connections 2023-09-10T01:07:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f2f644de10 0x55f2f6450230 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2f63e51a0 0x55f2f644c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2f63df300 0x55f2f6441370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ebeffd700 0 client.26764 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 --2- 172.21.15.3:0/65406187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f2f63dfcc0 0x55f2f64418b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 >> 172.21.15.3:0/65406187 conn(0x55f2f6354030 msgr2=0x55f2f63deb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 shutdown_connections 2023-09-10T01:07:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 1 -- 172.21.15.3:0/65406187 wait complete. 2023-09-10T01:07:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.794+0000 7f3ef9739b80 20 client.26764 trim_cache size 0 max 16384 2023-09-10T01:07:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:07:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:07:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:07:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:07:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3ec80a0830 msgr2=0x7f3ec80a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3ec80a0830 0x7f3ec80a2cf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009d70 tx=0x7f3ed0009260 comp rx=0 tx=0).stop 2023-09-10T01:07:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0068520 msgr2=0x7f3ee00ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0068520 0x7f3ee00ffc80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40095c0 tx=0x7f3ed4004730 comp rx=0 tx=0).stop 2023-09-10T01:07:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 shutdown_connections 2023-09-10T01:07:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3ec80a0830 0x7f3ec80a2cf0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee01066a0 0x7f3ee01001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee0067b60 0x7f3ee00ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 --2- 172.21.15.3:0/3219620941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ee0068520 0x7f3ee00ffc80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 >> 172.21.15.3:0/3219620941 conn(0x7f3ee0074fe0 msgr2=0x7f3ee01041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 shutdown_connections 2023-09-10T01:07:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:14.795+0000 7f3ee5dff700 1 -- 172.21.15.3:0/3219620941 wait complete. 2023-09-10T01:07:15.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 320401 9.43 MB/sec execute 57 sec latency 89.261 ms 2023-09-10T01:07:16.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:16 smithi136 ceph-mon[133708]: pgmap v124: 129 pgs: 129 active+clean; 88 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 389 op/s 2023-09-10T01:07:16.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:16 smithi092 ceph-mon[134067]: pgmap v124: 129 pgs: 129 active+clean; 88 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 389 op/s 2023-09-10T01:07:16.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:16 smithi003 ceph-mon[97654]: pgmap v124: 129 pgs: 129 active+clean; 88 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 389 op/s 2023-09-10T01:07:16.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 322931 9.54 MB/sec execute 58 sec latency 11.024 ms 2023-09-10T01:07:17.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 323978 9.44 MB/sec execute 59 sec latency 39.998 ms 2023-09-10T01:07:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:18 smithi136 ceph-mon[133708]: pgmap v125: 129 pgs: 129 active+clean; 88 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-09-10T01:07:18.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:18 smithi092 ceph-mon[134067]: pgmap v125: 129 pgs: 129 active+clean; 88 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-09-10T01:07:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:18 smithi003 ceph-mon[97654]: pgmap v125: 129 pgs: 129 active+clean; 88 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 213 op/s 2023-09-10T01:07:18.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 326410 9.54 MB/sec execute 60 sec latency 78.907 ms 2023-09-10T01:07:19.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 327445 9.44 MB/sec execute 61 sec latency 15.162 ms 2023-09-10T01:07:20.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:20 smithi136 ceph-mon[133708]: pgmap v126: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 379 op/s 2023-09-10T01:07:20.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:20 smithi092 ceph-mon[134067]: pgmap v126: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 379 op/s 2023-09-10T01:07:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:20 smithi003 ceph-mon[97654]: pgmap v126: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 379 op/s 2023-09-10T01:07:20.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 329971 9.54 MB/sec execute 62 sec latency 122.755 ms 2023-09-10T01:07:21.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 330708 9.42 MB/sec execute 63 sec latency 281.512 ms 2023-09-10T01:07:22.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:22 smithi136 ceph-mon[133708]: pgmap v127: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:07:22.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:22 smithi092 ceph-mon[134067]: pgmap v127: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:07:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:22 smithi003 ceph-mon[97654]: pgmap v127: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:07:22.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 333419 9.54 MB/sec execute 64 sec latency 11.840 ms 2023-09-10T01:07:23.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 333985 9.42 MB/sec execute 65 sec latency 193.056 ms 2023-09-10T01:07:24.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:24 smithi136 ceph-mon[133708]: pgmap v128: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:07:24.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:24 smithi092 ceph-mon[134067]: pgmap v128: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:07:24.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:24 smithi003 ceph-mon[97654]: pgmap v128: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:07:24.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 335935 9.39 MB/sec execute 66 sec latency 41.856 ms 2023-09-10T01:07:25.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:25 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:07:25.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:25 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:07:25.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:25 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:07:25.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 337522 9.42 MB/sec execute 67 sec latency 21.468 ms 2023-09-10T01:07:26.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:26 smithi136 ceph-mon[133708]: pgmap v129: 129 pgs: 129 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2023-09-10T01:07:26.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:26 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:07:26.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:26 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:07:26.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:26 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:07:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:26 smithi092 ceph-mon[134067]: pgmap v129: 129 pgs: 129 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2023-09-10T01:07:26.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:26 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:07:26.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:26 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:07:26.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:26 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:07:26.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:26 smithi003 ceph-mon[97654]: pgmap v129: 129 pgs: 129 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2023-09-10T01:07:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:26 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:07:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:26 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:07:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:26 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:07:26.750 INFO:tasks.workunit.client.0.smithi003.stdout: 1 339573 9.40 MB/sec execute 68 sec latency 16.157 ms 2023-09-10T01:07:27.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:07:27.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 -- 172.21.15.3:0/3535736760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 msgr2=0x7fedb810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/3535736760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 0x7fedb810d510 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fedb000b0a0 tx=0x7fedb000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 -- 172.21.15.3:0/3535736760 shutdown_connections 2023-09-10T01:07:27.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/3535736760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb8106650 0x7fedb8106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/3535736760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 0x7fedb810d510 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/3535736760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 0x7fedb8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 -- 172.21.15.3:0/3535736760 >> 172.21.15.3:0/3535736760 conn(0x7fedb80fc170 msgr2=0x7fedb80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 -- 172.21.15.3:0/3535736760 shutdown_connections 2023-09-10T01:07:27.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.582+0000 7fedbf3ff700 1 -- 172.21.15.3:0/3535736760 wait complete. 2023-09-10T01:07:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.583+0000 7fedbf3ff700 1 Processor -- start 2023-09-10T01:07:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbf3ff700 1 -- start start 2023-09-10T01:07:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbf3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 0x7fedb8196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbf3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 0x7fedb8196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbf3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb8106650 0x7fedb819b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbf3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb819b650 con 0x7fedb8101000 2023-09-10T01:07:27.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbf3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb819b7c0 con 0x7fedb8100640 2023-09-10T01:07:27.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbf3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb819b980 con 0x7fedb8106650 2023-09-10T01:07:27.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 0x7fedb8196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 0x7fedb8196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44294/0 (socket says 172.21.15.3:44294) 2023-09-10T01:07:27.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbd19b700 1 -- 172.21.15.3:0/2535254021 learned_addr learned my addr 172.21.15.3:0/2535254021 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbc99a700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 0x7fedb8196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.584+0000 7fedbd99c700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb8106650 0x7fedb819b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbd19b700 1 -- 172.21.15.3:0/2535254021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb8106650 msgr2=0x7fedb819b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbd19b700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb8106650 0x7fedb819b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbd19b700 1 -- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 msgr2=0x7fedb8196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbd19b700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 0x7fedb8196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbd19b700 1 -- 172.21.15.3:0/2535254021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda8009520 con 0x7fedb8100640 2023-09-10T01:07:27.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbd99c700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb8106650 0x7fedb819b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:07:27.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbd19b700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 0x7fedb8196670 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fedb400e9c0 tx=0x7fedb400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedae7fc700 1 -- 172.21.15.3:0/2535254021 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40097b0 con 0x7fedb8100640 2023-09-10T01:07:27.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedae7fc700 1 -- 172.21.15.3:0/2535254021 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedb4004cf0 con 0x7fedb8100640 2023-09-10T01:07:27.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb0009d00 con 0x7fedb8100640 2023-09-10T01:07:27.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedae7fc700 1 -- 172.21.15.3:0/2535254021 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4005600 con 0x7fedb8100640 2023-09-10T01:07:27.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.585+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb81a32c0 con 0x7fedb8100640 2023-09-10T01:07:27.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.589+0000 7fedae7fc700 1 -- 172.21.15.3:0/2535254021 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fedb400cc60 con 0x7fedb8100640 2023-09-10T01:07:27.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.590+0000 7fedd2c5bb80 1 Processor -- start 2023-09-10T01:07:27.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.590+0000 7fedd2c5bb80 1 -- start start 2023-09-10T01:07:27.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedae7fc700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feda40a07b0 0x7feda40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedae7fc700 1 -- 172.21.15.3:0/2535254021 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fedb4014070 con 0x7fedb8100640 2023-09-10T01:07:27.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x563892284af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd99c700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feda40a07b0 0x7feda40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x563892291230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638921dd730 con 0x56389228ee40 2023-09-10T01:07:27.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638921db050 con 0x563892284710 2023-09-10T01:07:27.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638921d6940 con 0x563892285030 2023-09-10T01:07:27.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x563892284af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x563892284af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44310/0 (socket says 172.21.15.3:44310) 2023-09-10T01:07:27.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd19b700 1 -- 172.21.15.3:0/367062732 learned_addr learned my addr 172.21.15.3:0/367062732 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbc99a700 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd99c700 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x563892291230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd19b700 1 -- 172.21.15.3:0/367062732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 msgr2=0x56389228e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd19b700 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd19b700 1 -- 172.21.15.3:0/367062732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 msgr2=0x563892291230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.591+0000 7fedbd19b700 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x563892291230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.592+0000 7fedbd19b700 1 -- 172.21.15.3:0/367062732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563892002de0 con 0x563892284710 2023-09-10T01:07:27.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.592+0000 7fedbd19b700 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x563892284af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fedb40a11c0 tx=0x7fedb4099240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a688d6e629a333f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.592+0000 7fedbd99c700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feda40a07b0 0x7feda40a2c70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009610 tx=0x7fedb0006210 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.592+0000 7feda37fe700 1 -- 172.21.15.3:0/367062732 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4099d80 con 0x563892284710 2023-09-10T01:07:27.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.592+0000 7feda37fe700 1 -- 172.21.15.3:0/367062732 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedb4099680 con 0x563892284710 2023-09-10T01:07:27.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.592+0000 7feda37fe700 1 -- 172.21.15.3:0/367062732 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40cc190 con 0x563892284710 2023-09-10T01:07:27.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/367062732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 msgr2=0x563892284af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x563892284af0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fedb40a11c0 tx=0x7fedb4099240 comp rx=0 tx=0).stop 2023-09-10T01:07:27.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/367062732 shutdown_connections 2023-09-10T01:07:27.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x563892291230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x563892284af0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/367062732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/367062732 >> 172.21.15.3:0/367062732 conn(0x5638921f9090 msgr2=0x56389227fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/367062732 shutdown_connections 2023-09-10T01:07:27.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.593+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/367062732 wait complete. 2023-09-10T01:07:27.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.594+0000 7fedd2c5bb80 1 -- start start 2023-09-10T01:07:27.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.594+0000 7fedd2c5bb80 1 Processor -- start 2023-09-10T01:07:27.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x56389228e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x56389228c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563892174b70 con 0x56389228ee40 2023-09-10T01:07:27.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563892174870 con 0x563892284710 2023-09-10T01:07:27.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563892058920 con 0x563892285030 2023-09-10T01:07:27.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedd2c5bb80 10 client.0 _ll_register_callbacks cb 0x5638920bea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:27.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedbc99a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedbc99a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45708/0 (socket says 172.21.15.3:45708) 2023-09-10T01:07:27.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedbc99a700 1 -- 172.21.15.3:0/2411718233 learned_addr learned my addr 172.21.15.3:0/2411718233 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedbd19b700 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x56389228e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.595+0000 7fedbd99c700 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x56389228c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedbd19b700 1 -- 172.21.15.3:0/2411718233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 msgr2=0x56389228bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedbd19b700 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedbd19b700 1 -- 172.21.15.3:0/2411718233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 msgr2=0x56389228c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedbd19b700 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x56389228c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedbd19b700 1 -- 172.21.15.3:0/2411718233 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb81a32c0 con 0x563892284710 2023-09-10T01:07:27.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedbd19b700 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x56389228e690 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fedb40c9ea0 tx=0x7fedb40c9910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9765c8ecd93f80c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7feda0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:27.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7feda0ff9700 1 -- 172.21.15.3:0/2411718233 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb409d4b0 con 0x563892284710 2023-09-10T01:07:27.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56389212e4f0 con 0x563892284710 2023-09-10T01:07:27.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7feda0ff9700 1 -- 172.21.15.3:0/2411718233 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fedb4065220 con 0x563892284710 2023-09-10T01:07:27.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7feda0ff9700 1 -- 172.21.15.3:0/2411718233 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedb4065420 con 0x563892284710 2023-09-10T01:07:27.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedd2c5bb80 10 client.26855 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:27.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.596+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563891febc60 con 0x563892284710 2023-09-10T01:07:27.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7feda0ff9700 1 -- 172.21.15.3:0/2411718233 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56389212e4f0 con 0x563892284710 2023-09-10T01:07:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7feda0ff9700 1 -- 172.21.15.3:0/2411718233 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fedb40998e0 con 0x563892284710 2023-09-10T01:07:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7fedd2c5bb80 10 client.26855 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7fedd2c5bb80 10 client.26855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7fedd2c5bb80 20 client.26855 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:27.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922a6da0 0x5638922a9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7fedd2c5bb80 4 client.26855 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.597+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638921eb8a0 con 0x5638922a6da0 2023-09-10T01:07:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.598+0000 7fedbc99a700 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922a6da0 0x5638922a9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.598+0000 7fedbc99a700 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922a6da0 0x5638922a9190 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b99767581fa8f4a5 server_cookie=1d3423621f30922 in_seq=0 out_seq=0 2023-09-10T01:07:27.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.598+0000 7feda0ff9700 10 client.26855 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:27.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7feda0ff9700 1 -- 172.21.15.3:0/2411718233 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5638921eb8a0 con 0x5638922a6da0 2023-09-10T01:07:27.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7feda0ff9700 10 client.26855 handle_command_reply: tid=0 2023-09-10T01:07:27.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 client.26855 shutdown 2023-09-10T01:07:27.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 msgr2=0x56389228e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x56389228e690 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fedb40c9ea0 tx=0x7fedb40c9910 comp rx=0 tx=0).stop 2023-09-10T01:07:27.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 shutdown_connections 2023-09-10T01:07:27.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922a6da0 0x5638922a9190 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56389228ee40 0x56389228c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x56389228e690 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/2411718233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7feda0ff9700 0 client.26855 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:27.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 >> 172.21.15.3:0/2411718233 conn(0x5638921f9090 msgr2=0x56389227f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 shutdown_connections 2023-09-10T01:07:27.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/2411718233 wait complete. 2023-09-10T01:07:27.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.600+0000 7fedd2c5bb80 20 client.26855 trim_cache size 0 max 16384 2023-09-10T01:07:27.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.620+0000 7fedd2c5bb80 1 Processor -- start 2023-09-10T01:07:27.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.620+0000 7fedd2c5bb80 1 -- start start 2023-09-10T01:07:27.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.620+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892284710 0x5638922948c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.620+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.620+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56389228ee40 0x563892288d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.620+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638922811e0 con 0x563892284710 2023-09-10T01:07:27.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56389227e9a0 con 0x56389228ee40 2023-09-10T01:07:27.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563892294e00 con 0x563892285030 2023-09-10T01:07:27.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892284710 0x5638922948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbd99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56389228ee40 0x563892288d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45720/0 (socket says 172.21.15.3:45720) 2023-09-10T01:07:27.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892284710 0x5638922948c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41958/0 (socket says 172.21.15.3:41958) 2023-09-10T01:07:27.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 -- 172.21.15.3:0/4104113363 learned_addr learned my addr 172.21.15.3:0/4104113363 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbd99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56389228ee40 0x563892288d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44332/0 (socket says 172.21.15.3:44332) 2023-09-10T01:07:27.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 -- 172.21.15.3:0/4104113363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56389228ee40 msgr2=0x563892288d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56389228ee40 0x563892288d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 -- 172.21.15.3:0/4104113363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892284710 msgr2=0x5638922948c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892284710 0x5638922948c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbc99a700 1 -- 172.21.15.3:0/4104113363 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563892029250 con 0x563892285030 2023-09-10T01:07:27.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.621+0000 7fedbd19b700 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892284710 0x5638922948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:27.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.622+0000 7fedbc99a700 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x56389228d1b0 tx=0x7feda8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d971f1422973be7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.622+0000 7feda3fff700 1 -- 172.21.15.3:0/4104113363 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8017040 con 0x563892285030 2023-09-10T01:07:27.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.622+0000 7feda3fff700 1 -- 172.21.15.3:0/4104113363 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feda8023070 con 0x563892285030 2023-09-10T01:07:27.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.622+0000 7feda3fff700 1 -- 172.21.15.3:0/4104113363 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8011ba0 con 0x563892285030 2023-09-10T01:07:27.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4104113363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 msgr2=0x56389228e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x56389228d1b0 tx=0x7feda8009260 comp rx=0 tx=0).stop 2023-09-10T01:07:27.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4104113363 shutdown_connections 2023-09-10T01:07:27.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56389228ee40 0x563892288d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892284710 0x5638922948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4104113363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563892285030 0x56389228e420 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4104113363 >> 172.21.15.3:0/4104113363 conn(0x5638921f9090 msgr2=0x56389227f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4104113363 shutdown_connections 2023-09-10T01:07:27.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.623+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4104113363 wait complete. 2023-09-10T01:07:27.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.624+0000 7fedd2c5bb80 1 -- start start 2023-09-10T01:07:27.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.624+0000 7fedd2c5bb80 1 Processor -- start 2023-09-10T01:07:27.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x5638922a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892285030 0x5638922a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedd2c5bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56389228ee40 0x5638922a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638922fa9f0 con 0x563892285030 2023-09-10T01:07:27.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638922fab30 con 0x563892284710 2023-09-10T01:07:27.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedd2c5bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638922fac70 con 0x56389228ee40 2023-09-10T01:07:27.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedd2c5bb80 10 client.0 _ll_register_callbacks cb 0x7fedb4099450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:27.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x5638922a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedbd19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x5638922a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44346/0 (socket says 172.21.15.3:44346) 2023-09-10T01:07:27.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedbd19b700 1 -- 172.21.15.3:0/4096542905 learned_addr learned my addr 172.21.15.3:0/4096542905 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedbd99c700 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56389228ee40 0x5638922a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.625+0000 7fedbc99a700 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892285030 0x5638922a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedbd19b700 1 -- 172.21.15.3:0/4096542905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56389228ee40 msgr2=0x5638922a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedbd19b700 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56389228ee40 0x5638922a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedbd19b700 1 -- 172.21.15.3:0/4096542905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892285030 msgr2=0x5638922a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedbd19b700 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892285030 0x5638922a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedbd19b700 1 -- 172.21.15.3:0/4096542905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563891febc60 con 0x563892284710 2023-09-10T01:07:27.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedbd19b700 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x5638922a0470 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fedb4099610 tx=0x7fedb4097c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e122fac87a748ee1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7feda0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:27.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7feda0ff9700 1 -- 172.21.15.3:0/4096542905 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb40cc390 con 0x563892284710 2023-09-10T01:07:27.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7feda0ff9700 1 -- 172.21.15.3:0/4096542905 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fedb40a16f0 con 0x563892284710 2023-09-10T01:07:27.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedb4099680 con 0x563892284710 2023-09-10T01:07:27.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7feda0ff9700 1 -- 172.21.15.3:0/4096542905 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedb4098c10 con 0x563892284710 2023-09-10T01:07:27.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedd2c5bb80 10 client.26861 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:27.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7feda801f9f0 con 0x563892284710 2023-09-10T01:07:27.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.626+0000 7feda0ff9700 1 -- 172.21.15.3:0/4096542905 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedb4099680 con 0x563892284710 2023-09-10T01:07:27.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7feda0ff9700 1 -- 172.21.15.3:0/4096542905 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fedb40840d0 con 0x563892284710 2023-09-10T01:07:27.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedd2c5bb80 10 client.26861 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:27.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedd2c5bb80 10 client.26861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:27.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedd2c5bb80 20 client.26861 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:27.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922fbea0 0x5638922fe290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedd2c5bb80 4 client.26861 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:07:27.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5638921cf920 con 0x5638922fbea0 2023-09-10T01:07:27.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedbc99a700 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922fbea0 0x5638922fe290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7fedbc99a700 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922fbea0 0x5638922fe290 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7085f2f0920b06af server_cookie=42e64e58b0ab3a4d in_seq=0 out_seq=0 2023-09-10T01:07:27.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.627+0000 7feda0ff9700 10 client.26861 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:27.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7feda0ff9700 1 -- 172.21.15.3:0/4096542905 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5638921cf920 con 0x5638922fbea0 2023-09-10T01:07:27.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7feda0ff9700 10 client.26861 handle_command_reply: tid=0 2023-09-10T01:07:27.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 client.26861 shutdown 2023-09-10T01:07:27.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 msgr2=0x5638922a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x5638922a0470 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fedb4099610 tx=0x7fedb4097c30 comp rx=0 tx=0).stop 2023-09-10T01:07:27.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 shutdown_connections 2023-09-10T01:07:27.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638922fbea0 0x5638922fe290 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56389228ee40 0x5638922a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7feda0ff9700 0 client.26861 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:27.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563892284710 0x5638922a0470 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 --2- 172.21.15.3:0/4096542905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563892285030 0x5638922a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 >> 172.21.15.3:0/4096542905 conn(0x5638921f9090 msgr2=0x563892283700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 shutdown_connections 2023-09-10T01:07:27.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.628+0000 7fedd2c5bb80 1 -- 172.21.15.3:0/4096542905 wait complete. 2023-09-10T01:07:27.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedd2c5bb80 20 client.26861 trim_cache size 0 max 16384 2023-09-10T01:07:27.645 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:07:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:07:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:07:27.645 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:07:27.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feda40a07b0 msgr2=0x7feda40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feda40a07b0 0x7feda40a2c70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009610 tx=0x7fedb0006210 comp rx=0 tx=0).stop 2023-09-10T01:07:27.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 msgr2=0x7fedb8196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 0x7fedb8196670 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fedb400e9c0 tx=0x7fedb400ecd0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 shutdown_connections 2023-09-10T01:07:27.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feda40a07b0 0x7feda40a2c70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb8106650 0x7fedb819b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fedb8100640 0x7fedb8196670 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 --2- 172.21.15.3:0/2535254021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedb8101000 0x7fedb8196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.629+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 >> 172.21.15.3:0/2535254021 conn(0x7fedb80fc170 msgr2=0x7fedb80fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.630+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 shutdown_connections 2023-09-10T01:07:27.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.630+0000 7fedbf3ff700 1 -- 172.21.15.3:0/2535254021 wait complete. 2023-09-10T01:07:27.650 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:07:27.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:07:27.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 341071 9.43 MB/sec execute 69 sec latency 18.805 ms 2023-09-10T01:07:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 -- 172.21.15.3:0/2267333271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0228104020 msgr2=0x7f022810bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 --2- 172.21.15.3:0/2267333271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0228104020 0x7f022810bba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f022000b0a0 tx=0x7f022000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 -- 172.21.15.3:0/2267333271 shutdown_connections 2023-09-10T01:07:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 --2- 172.21.15.3:0/2267333271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0228104020 0x7f022810bba0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 --2- 172.21.15.3:0/2267333271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02280ff110 0x7f02280ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 --2- 172.21.15.3:0/2267333271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02280ffac0 0x7f02281038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 -- 172.21.15.3:0/2267333271 >> 172.21.15.3:0/2267333271 conn(0x7f0228075860 msgr2=0x7f0228075c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 -- 172.21.15.3:0/2267333271 shutdown_connections 2023-09-10T01:07:27.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.860+0000 7f022f1ff700 1 -- 172.21.15.3:0/2267333271 wait complete. 2023-09-10T01:07:27.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.861+0000 7f022f1ff700 1 Processor -- start 2023-09-10T01:07:27.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f022f1ff700 1 -- start start 2023-09-10T01:07:27.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f022f1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02280ff110 0x7f022819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f022f1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0228104020 0x7f022819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f022f1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02281988c0 0x7f0228198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f022f1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0228199340 con 0x7f02280ff110 2023-09-10T01:07:27.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f022f1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0228199480 con 0x7f02281988c0 2023-09-10T01:07:27.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f022f1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0228199640 con 0x7f0228104020 2023-09-10T01:07:27.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f0227fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0228104020 0x7f022819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f022cf9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02280ff110 0x7f022819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.862+0000 7f0227fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0228104020 0x7f022819ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45762/0 (socket says 172.21.15.3:45762) 2023-09-10T01:07:27.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f0227fff700 1 -- 172.21.15.3:0/909231176 learned_addr learned my addr 172.21.15.3:0/909231176 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f022d79c700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02281988c0 0x7f0228198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f0227fff700 1 -- 172.21.15.3:0/909231176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02281988c0 msgr2=0x7f0228198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f0227fff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02281988c0 0x7f0228198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f0227fff700 1 -- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02280ff110 msgr2=0x7f022819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f0227fff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02280ff110 0x7f022819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f0227fff700 1 -- 172.21.15.3:0/909231176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220009d00 con 0x7f0228104020 2023-09-10T01:07:27.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f022cf9b700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02280ff110 0x7f022819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:27.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.863+0000 7f0227fff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0228104020 0x7f022819ed90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f021400e970 tx=0x7f021400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.864+0000 7f0225ffb700 1 -- 172.21.15.3:0/909231176 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214009790 con 0x7f0228104020 2023-09-10T01:07:27.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.864+0000 7f0225ffb700 1 -- 172.21.15.3:0/909231176 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f021400c7e0 con 0x7f0228104020 2023-09-10T01:07:27.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.864+0000 7f0225ffb700 1 -- 172.21.15.3:0/909231176 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140054e0 con 0x7f0228104020 2023-09-10T01:07:27.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.864+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02281a56c0 con 0x7f0228104020 2023-09-10T01:07:27.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.864+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02281a5b80 con 0x7f0228104020 2023-09-10T01:07:27.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.865+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0228066bf0 con 0x7f0228104020 2023-09-10T01:07:27.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.867+0000 7f0225ffb700 1 -- 172.21.15.3:0/909231176 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f021400c950 con 0x7f0228104020 2023-09-10T01:07:27.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.869+0000 7f0225ffb700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f02180a07b0 0x7f02180a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.869+0000 7f0225ffb700 1 -- 172.21.15.3:0/909231176 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0214014070 con 0x7f0228104020 2023-09-10T01:07:27.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.869+0000 7f022cf9b700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f02180a07b0 0x7f02180a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.870+0000 7f022cf9b700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f02180a07b0 0x7f02180a2c70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f021c011490 tx=0x7f021c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:27.871+0000 7f0225ffb700 1 -- 172.21.15.3:0/909231176 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0214095d90 con 0x7f0228104020 2023-09-10T01:07:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 -- 172.21.15.3:0/3176685566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 msgr2=0x7fd02810d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/3176685566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02810d400 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd020009fc0 tx=0x7fd02000b330 comp rx=0 tx=0).stop 2023-09-10T01:07:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 -- 172.21.15.3:0/3176685566 shutdown_connections 2023-09-10T01:07:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/3176685566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02810d400 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/3176685566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd028106640 0x7fd028106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/3176685566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd028100580 0x7fd028100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 -- 172.21.15.3:0/3176685566 >> 172.21.15.3:0/3176685566 conn(0x7fd0280fc050 msgr2=0x7fd0280fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.958+0000 7fd02e9ff700 1 -- 172.21.15.3:0/3176685566 shutdown_connections 2023-09-10T01:07:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.959+0000 7fd02e9ff700 1 -- 172.21.15.3:0/3176685566 wait complete. 2023-09-10T01:07:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.960+0000 7fd02e9ff700 1 Processor -- start 2023-09-10T01:07:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.960+0000 7fd02e9ff700 1 -- start start 2023-09-10T01:07:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd02e9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd028100580 0x7fd02819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd02e9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd02e9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd028106640 0x7fd0281966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd02e9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd028196d60 con 0x7fd028100580 2023-09-10T01:07:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd02e9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd028196ed0 con 0x7fd028100f40 2023-09-10T01:07:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd02e9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd028197090 con 0x7fd028106640 2023-09-10T01:07:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd027fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd028100580 0x7fd02819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02819cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44398/0 (socket says 172.21.15.3:44398) 2023-09-10T01:07:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd0277fe700 1 -- 172.21.15.3:0/2739462559 learned_addr learned my addr 172.21.15.3:0/2739462559 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.961+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd028106640 0x7fd0281966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0277fe700 1 -- 172.21.15.3:0/2739462559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd028106640 msgr2=0x7fd0281966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0277fe700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd028106640 0x7fd0281966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0277fe700 1 -- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd028100580 msgr2=0x7fd02819c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0277fe700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd028100580 0x7fd02819c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0277fe700 1 -- 172.21.15.3:0/2739462559 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd020009c40 con 0x7fd028100f40 2023-09-10T01:07:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd028106640 0x7fd0281966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd027fff700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd028100580 0x7fd02819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0277fe700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02819cbc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00e9c0 tx=0x7fd01c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0257fa700 1 -- 172.21.15.3:0/2739462559 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c004a30 con 0x7fd028100f40 2023-09-10T01:07:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0257fa700 1 -- 172.21.15.3:0/2739462559 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd01c00cc80 con 0x7fd028100f40 2023-09-10T01:07:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd0257fa700 1 -- 172.21.15.3:0/2739462559 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c0054e0 con 0x7fd028100f40 2023-09-10T01:07:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd028197370 con 0x7fd028100f40 2023-09-10T01:07:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.962+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0281a3670 con 0x7fd028100f40 2023-09-10T01:07:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.966+0000 7fd0257fa700 1 -- 172.21.15.3:0/2739462559 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd01c004b90 con 0x7fd028100f40 2023-09-10T01:07:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.967+0000 7fd0423f2b80 1 Processor -- start 2023-09-10T01:07:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0423f2b80 1 -- start start 2023-09-10T01:07:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c857b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c864340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c7ac6d0 con 0x56334c8580d0 2023-09-10T01:07:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c7a9ff0 con 0x56334c85d5b0 2023-09-10T01:07:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c7a58e0 con 0x56334c857710 2023-09-10T01:07:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd027fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c857b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd027fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c857b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45790/0 (socket says 172.21.15.3:45790) 2023-09-10T01:07:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd02cf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd027fff700 1 -- 172.21.15.3:0/2734986307 learned_addr learned my addr 172.21.15.3:0/2734986307 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.968+0000 7fd0257fa700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0100a07b0 0x7fd0100a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd0277fe700 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c864340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd0257fa700 1 -- 172.21.15.3:0/2739462559 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd01c014070 con 0x7fd028100f40 2023-09-10T01:07:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0100a07b0 0x7fd0100a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd027fff700 1 -- 172.21.15.3:0/2734986307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 msgr2=0x56334c85d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd027fff700 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd027fff700 1 -- 172.21.15.3:0/2734986307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 msgr2=0x56334c864340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd027fff700 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c864340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd027fff700 1 -- 172.21.15.3:0/2734986307 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56334c5d1d80 con 0x56334c857710 2023-09-10T01:07:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.969+0000 7fd027fff700 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c857b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd0180115f0 tx=0x7fd018011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c22838a2b49ede1e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.970+0000 7fd00e7fc700 1 -- 172.21.15.3:0/2734986307 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018019630 con 0x56334c857710 2023-09-10T01:07:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.970+0000 7fd00e7fc700 1 -- 172.21.15.3:0/2734986307 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd018019c50 con 0x56334c857710 2023-09-10T01:07:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.970+0000 7fd00e7fc700 1 -- 172.21.15.3:0/2734986307 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01800ebc0 con 0x56334c857710 2023-09-10T01:07:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2734986307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 msgr2=0x56334c857b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0100a07b0 0x7fd0100a2c70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd020009c10 tx=0x7fd02000b330 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c857b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd0180115f0 tx=0x7fd018011900 comp rx=0 tx=0).stop 2023-09-10T01:07:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2734986307 shutdown_connections 2023-09-10T01:07:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c864340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2734986307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c857b90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2734986307 >> 172.21.15.3:0/2734986307 conn(0x56334c7c8030 msgr2=0x56334c84e560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2734986307 shutdown_connections 2023-09-10T01:07:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.971+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2734986307 wait complete. 2023-09-10T01:07:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 -- start start 2023-09-10T01:07:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 Processor -- start 2023-09-10T01:07:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c8641a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c8580d0 0x56334c861490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c85d5b0 0x56334c861a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c743b10 con 0x56334c857710 2023-09-10T01:07:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c743810 con 0x56334c8580d0 2023-09-10T01:07:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c6278c0 con 0x56334c85d5b0 2023-09-10T01:07:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.972+0000 7fd0423f2b80 10 client.0 _ll_register_callbacks cb 0x56334c68da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c8580d0 0x56334c861490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd02cf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c85d5b0 0x56334c861a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c8580d0 0x56334c861490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44416/0 (socket says 172.21.15.3:44416) 2023-09-10T01:07:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 -- 172.21.15.3:0/2308472897 learned_addr learned my addr 172.21.15.3:0/2308472897 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd027fff700 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c8641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 -- 172.21.15.3:0/2308472897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c85d5b0 msgr2=0x56334c861a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c85d5b0 0x56334c861a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 -- 172.21.15.3:0/2308472897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 msgr2=0x56334c8641a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c8641a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 -- 172.21.15.3:0/2308472897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd018009d50 con 0x56334c8580d0 2023-09-10T01:07:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0277fe700 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c8580d0 0x56334c861490 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd01c020000 tx=0x7fd01c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36b8b6c2d6eb58cc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0077fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.973+0000 7fd0077fe700 1 -- 172.21.15.3:0/2308472897 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c095040 con 0x56334c8580d0 2023-09-10T01:07:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56334c6fd490 con 0x56334c8580d0 2023-09-10T01:07:27.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0077fe700 1 -- 172.21.15.3:0/2308472897 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd01c0ace40 con 0x56334c8580d0 2023-09-10T01:07:27.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0077fe700 1 -- 172.21.15.3:0/2308472897 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd01c00cdf0 con 0x56334c8580d0 2023-09-10T01:07:27.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 10 client.26879 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:27.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56334c5bac00 con 0x56334c8580d0 2023-09-10T01:07:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0077fe700 1 -- 172.21.15.3:0/2308472897 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd01c0b5080 con 0x56334c8580d0 2023-09-10T01:07:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0077fe700 1 -- 172.21.15.3:0/2308472897 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd01c095c30 con 0x56334c8580d0 2023-09-10T01:07:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 10 client.26879 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 10 client.26879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 20 client.26879 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c871810 0x56334c873c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 4 client.26879 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.974+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56334c7ba840 con 0x56334c871810 2023-09-10T01:07:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.975+0000 7fd027fff700 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c871810 0x56334c873c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.975+0000 7fd027fff700 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c871810 0x56334c873c30 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c99d626bd0fa454 server_cookie=571b7312b9d35c71 in_seq=0 out_seq=0 2023-09-10T01:07:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.975+0000 7fd0077fe700 10 client.26879 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0077fe700 1 -- 172.21.15.3:0/2308472897 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56334c7ba840 con 0x56334c871810 2023-09-10T01:07:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0077fe700 10 client.26879 handle_command_reply: tid=0 2023-09-10T01:07:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 client.26879 shutdown 2023-09-10T01:07:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c8580d0 msgr2=0x56334c861490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c8580d0 0x56334c861490 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd01c020000 tx=0x7fd01c01d000 comp rx=0 tx=0).stop 2023-09-10T01:07:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 shutdown_connections 2023-09-10T01:07:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c871810 0x56334c873c30 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c85d5b0 0x56334c861a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c8641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/2308472897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c8580d0 0x56334c861490 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0077fe700 0 client.26879 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.976+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 >> 172.21.15.3:0/2308472897 conn(0x56334c7c8030 msgr2=0x56334c84d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.977+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 shutdown_connections 2023-09-10T01:07:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.977+0000 7fd0423f2b80 1 -- 172.21.15.3:0/2308472897 wait complete. 2023-09-10T01:07:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.977+0000 7fd0423f2b80 20 client.26879 trim_cache size 0 max 16384 2023-09-10T01:07:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.994+0000 7fd0423f2b80 1 Processor -- start 2023-09-10T01:07:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0423f2b80 1 -- start start 2023-09-10T01:07:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c868930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c863d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c859f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c858990 con 0x56334c8580d0 2023-09-10T01:07:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c858740 con 0x56334c85d5b0 2023-09-10T01:07:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c868e70 con 0x56334c857710 2023-09-10T01:07:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd027fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c868930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd02cf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c859f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c863d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd027fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c868930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45814/0 (socket says 172.21.15.3:45814) 2023-09-10T01:07:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd02cf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c859f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44428/0 (socket says 172.21.15.3:44428) 2023-09-10T01:07:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.995+0000 7fd027fff700 1 -- 172.21.15.3:0/1251026960 learned_addr learned my addr 172.21.15.3:0/1251026960 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd027fff700 1 -- 172.21.15.3:0/1251026960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 msgr2=0x56334c859f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd027fff700 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c859f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd027fff700 1 -- 172.21.15.3:0/1251026960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 msgr2=0x56334c863d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd027fff700 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c863d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd027fff700 1 -- 172.21.15.3:0/1251026960 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56334c5f81f0 con 0x56334c857710 2023-09-10T01:07:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c859f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd0277fe700 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c863d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd027fff700 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c868930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56334c84e060 tx=0x7fd018011540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cffa73bd47c46c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd00effd700 1 -- 172.21.15.3:0/1251026960 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018005440 con 0x56334c857710 2023-09-10T01:07:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd00effd700 1 -- 172.21.15.3:0/1251026960 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd018005ce0 con 0x56334c857710 2023-09-10T01:07:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.996+0000 7fd00effd700 1 -- 172.21.15.3:0/1251026960 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018003a00 con 0x56334c857710 2023-09-10T01:07:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 -- 172.21.15.3:0/1251026960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 msgr2=0x56334c868930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c868930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56334c84e060 tx=0x7fd018011540 comp rx=0 tx=0).stop 2023-09-10T01:07:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 -- 172.21.15.3:0/1251026960 shutdown_connections 2023-09-10T01:07:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c857710 0x56334c868930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c859f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/1251026960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c8580d0 0x56334c863d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 -- 172.21.15.3:0/1251026960 >> 172.21.15.3:0/1251026960 conn(0x56334c7c8030 msgr2=0x56334c84e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 -- 172.21.15.3:0/1251026960 shutdown_connections 2023-09-10T01:07:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 -- 172.21.15.3:0/1251026960 wait complete. 2023-09-10T01:07:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.997+0000 7fd0423f2b80 1 -- start start 2023-09-10T01:07:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 1 Processor -- start 2023-09-10T01:07:28.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c860840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c8580d0 0x56334c860d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c85fc90 con 0x56334c857710 2023-09-10T01:07:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c85fe00 con 0x56334c85d5b0 2023-09-10T01:07:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334c85ff70 con 0x56334c8580d0 2023-09-10T01:07:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0423f2b80 10 client.0 _ll_register_callbacks cb 0x7fd01c0958a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd027fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c860840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c8580d0 0x56334c860d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd02cf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0277fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c8580d0 0x56334c860d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45822/0 (socket says 172.21.15.3:45822) 2023-09-10T01:07:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd0277fe700 1 -- 172.21.15.3:0/129799771 learned_addr learned my addr 172.21.15.3:0/129799771 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.998+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44430/0 (socket says 172.21.15.3:44430) 2023-09-10T01:07:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd02cf9c700 1 -- 172.21.15.3:0/129799771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c8580d0 msgr2=0x56334c860d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c8580d0 0x56334c860d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd02cf9c700 1 -- 172.21.15.3:0/129799771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 msgr2=0x56334c860840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c860840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd02cf9c700 1 -- 172.21.15.3:0/129799771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56334c5bac00 con 0x56334c85d5b0 2023-09-10T01:07:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd02cf9c700 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85f750 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd0200074d0 tx=0x7fd020009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f5040235e067f99 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd0077fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd0077fe700 1 -- 172.21.15.3:0/129799771 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02000e040 con 0x56334c85d5b0 2023-09-10T01:07:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd018019c50 con 0x56334c85d5b0 2023-09-10T01:07:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:27.999+0000 7fd0077fe700 1 -- 172.21.15.3:0/129799771 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd02002c450 con 0x56334c85d5b0 2023-09-10T01:07:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0077fe700 1 -- 172.21.15.3:0/129799771 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0200368d0 con 0x56334c85d5b0 2023-09-10T01:07:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 10 client.26885 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fd020025040 con 0x56334c85d5b0 2023-09-10T01:07:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0077fe700 1 -- 172.21.15.3:0/129799771 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd02003a020 con 0x56334c85d5b0 2023-09-10T01:07:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0077fe700 1 -- 172.21.15.3:0/129799771 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd020020580 con 0x56334c85d5b0 2023-09-10T01:07:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 10 client.26885 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 10 client.26885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 20 client.26885 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c8c6630 0x56334c8c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 4 client.26885 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:07:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56334c79e8c0 con 0x56334c8c6630 2023-09-10T01:07:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.000+0000 7fd027fff700 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c8c6630 0x56334c8c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.001+0000 7fd0077fe700 10 client.26885 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.001+0000 7fd027fff700 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c8c6630 0x56334c8c8a50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d1817f2bd3ab1ed server_cookie=2d6b3dfb01b80d71 in_seq=0 out_seq=0 2023-09-10T01:07:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0077fe700 1 -- 172.21.15.3:0/129799771 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56334c79e8c0 con 0x56334c8c6630 2023-09-10T01:07:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0077fe700 10 client.26885 handle_command_reply: tid=0 2023-09-10T01:07:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 client.26885 shutdown 2023-09-10T01:07:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 msgr2=0x56334c85f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85f750 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd0200074d0 tx=0x7fd020009920 comp rx=0 tx=0).stop 2023-09-10T01:07:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 shutdown_connections 2023-09-10T01:07:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56334c8c6630 0x56334c8c8a50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56334c85d5b0 0x56334c85f750 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0077fe700 0 client.26885 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56334c857710 0x56334c860840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 --2- 172.21.15.3:0/129799771 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56334c8580d0 0x56334c860d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.129+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 >> 172.21.15.3:0/129799771 conn(0x56334c7c8030 msgr2=0x56334c856700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.131+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 shutdown_connections 2023-09-10T01:07:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.131+0000 7fd0423f2b80 1 -- 172.21.15.3:0/129799771 wait complete. 2023-09-10T01:07:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.131+0000 7fd0423f2b80 20 client.26885 trim_cache size 0 max 16384 2023-09-10T01:07:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:07:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:07:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:07:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:07:28.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.131+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0100a07b0 msgr2=0x7fd0100a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.131+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0100a07b0 0x7fd0100a2c70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd020009c10 tx=0x7fd02000b330 comp rx=0 tx=0).stop 2023-09-10T01:07:28.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.131+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 msgr2=0x7fd02819cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.131+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02819cbc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00e9c0 tx=0x7fd01c00ecd0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 shutdown_connections 2023-09-10T01:07:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0100a07b0 0x7fd0100a2c70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd028106640 0x7fd0281966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd028100580 0x7fd02819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 --2- 172.21.15.3:0/2739462559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd028100f40 0x7fd02819cbc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 >> 172.21.15.3:0/2739462559 conn(0x7fd0280fc050 msgr2=0x7fd0280fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 shutdown_connections 2023-09-10T01:07:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:28.132+0000 7fd02e9ff700 1 -- 172.21.15.3:0/2739462559 wait complete. 2023-09-10T01:07:28.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.222+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0228199dd0 con 0x7f0228104020 2023-09-10T01:07:28.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.223+0000 7f0225ffb700 1 -- 172.21.15.3:0/909231176 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11293 (secure 0 0 0) 0x7f02140993b0 con 0x7f0228104020 2023-09-10T01:07:28.227 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:07:28.227 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:07:08.821454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:07:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f02180a07b0 msgr2=0x7f02180a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f02180a07b0 0x7f02180a2c70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f021c011490 tx=0x7f021c009260 comp rx=0 tx=0).stop 2023-09-10T01:07:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0228104020 msgr2=0x7f022819ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0228104020 0x7f022819ed90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f021400e970 tx=0x7f021400ec80 comp rx=0 tx=0).stop 2023-09-10T01:07:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 shutdown_connections 2023-09-10T01:07:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f02180a07b0 0x7f02180a2c70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02281988c0 0x7f0228198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02280ff110 0x7f022819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 --2- 172.21.15.3:0/909231176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0228104020 0x7f022819ed90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 >> 172.21.15.3:0/909231176 conn(0x7f0228075860 msgr2=0x7f02280fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 shutdown_connections 2023-09-10T01:07:28.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.225+0000 7f022f1ff700 1 -- 172.21.15.3:0/909231176 wait complete. 2023-09-10T01:07:28.231 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2023-09-10T01:07:28.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:07:28.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:28 smithi136 ceph-mon[133708]: pgmap v130: 129 pgs: 129 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 270 op/s 2023-09-10T01:07:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:28 smithi092 ceph-mon[134067]: pgmap v130: 129 pgs: 129 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 270 op/s 2023-09-10T01:07:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 -- 172.21.15.3:0/2866520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c106650 msgr2=0x7f0d1c106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 --2- 172.21.15.3:0/2866520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c106650 0x7f0d1c106a30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009990 tx=0x7f0d18009ca0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 -- 172.21.15.3:0/2866520577 shutdown_connections 2023-09-10T01:07:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 --2- 172.21.15.3:0/2866520577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d1c101000 0x7f0d1c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 --2- 172.21.15.3:0/2866520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c106650 0x7f0d1c106a30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 --2- 172.21.15.3:0/2866520577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c100640 0x7f0d1c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 -- 172.21.15.3:0/2866520577 >> 172.21.15.3:0/2866520577 conn(0x7f0d1c0fc170 msgr2=0x7f0d1c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 -- 172.21.15.3:0/2866520577 shutdown_connections 2023-09-10T01:07:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.474+0000 7f0d241ff700 1 -- 172.21.15.3:0/2866520577 wait complete. 2023-09-10T01:07:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.475+0000 7f0d241ff700 1 Processor -- start 2023-09-10T01:07:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.475+0000 7f0d241ff700 1 -- start start 2023-09-10T01:07:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.475+0000 7f0d241ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c100640 0x7f0d1c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d241ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c101000 0x7f0d1c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d241ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d1c106650 0x7f0d1c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d241ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c198de0 con 0x7f0d1c106650 2023-09-10T01:07:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d241ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c198f50 con 0x7f0d1c100640 2023-09-10T01:07:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d241ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c199110 con 0x7f0d1c101000 2023-09-10T01:07:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c100640 0x7f0d1c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c100640 0x7f0d1c19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44452/0 (socket says 172.21.15.3:44452) 2023-09-10T01:07:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 -- 172.21.15.3:0/4023397163 learned_addr learned my addr 172.21.15.3:0/4023397163 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d2179a700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c101000 0x7f0d1c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d2279c700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d1c106650 0x7f0d1c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 -- 172.21.15.3:0/4023397163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c101000 msgr2=0x7f0d1c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c101000 0x7f0d1c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 -- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d1c106650 msgr2=0x7f0d1c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d1c106650 0x7f0d1c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 -- 172.21.15.3:0/4023397163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d180095f0 con 0x7f0d1c100640 2023-09-10T01:07:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.476+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c100640 0x7f0d1c19e6a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0d18000c00 tx=0x7f0d180047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.477+0000 7f0d12ffd700 1 -- 172.21.15.3:0/4023397163 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1801d070 con 0x7f0d1c100640 2023-09-10T01:07:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.477+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c1993f0 con 0x7f0d1c100640 2023-09-10T01:07:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.477+0000 7f0d12ffd700 1 -- 172.21.15.3:0/4023397163 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d18021410 con 0x7f0d1c100640 2023-09-10T01:07:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.477+0000 7f0d12ffd700 1 -- 172.21.15.3:0/4023397163 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18017650 con 0x7f0d1c100640 2023-09-10T01:07:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.477+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c1a56d0 con 0x7f0d1c100640 2023-09-10T01:07:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.480+0000 7f0d12ffd700 1 -- 172.21.15.3:0/4023397163 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f0d180217f0 con 0x7f0d1c100640 2023-09-10T01:07:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 Processor -- start 2023-09-10T01:07:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d12ffd700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d080a07b0 0x7f0d080a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 -- start start 2023-09-10T01:07:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d12ffd700 1 -- 172.21.15.3:0/4023397163 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0d180cd710 con 0x7f0d1c100640 2023-09-10T01:07:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d2279c700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d080a07b0 0x7f0d080a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c502f710 0x55c6c502faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5030030 0x55c6c5039900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c503c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c4f88730 con 0x55c6c5030030 2023-09-10T01:07:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c4f86050 con 0x55c6c5039e40 2023-09-10T01:07:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.482+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c4f81940 con 0x55c6c502f710 2023-09-10T01:07:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c502f710 0x55c6c502faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d2179a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5030030 0x55c6c5039900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c502f710 0x55c6c502faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45838/0 (socket says 172.21.15.3:45838) 2023-09-10T01:07:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3762753271 learned_addr learned my addr 172.21.15.3:0/3762753271 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d2279c700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d080a07b0 0x7f0d080a2c70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009550 tx=0x7f0d14009130 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3762753271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 msgr2=0x55c6c503c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3762753271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c503c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3762753271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5030030 msgr2=0x55c6c5039900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3762753271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5030030 0x55c6c5039900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3762753271 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6c4dadde0 con 0x55c6c502f710 2023-09-10T01:07:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3762753271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c502f710 0x55c6c502faf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0d18096d60 tx=0x7f0d1809a700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0790afb739f24c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d07fff700 1 -- 172.21.15.3:0/3762753271 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1809af60 con 0x55c6c502f710 2023-09-10T01:07:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.483+0000 7f0d07fff700 1 -- 172.21.15.3:0/3762753271 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d180b0030 con 0x55c6c502f710 2023-09-10T01:07:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d07fff700 1 -- 172.21.15.3:0/3762753271 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d180cd4e0 con 0x55c6c502f710 2023-09-10T01:07:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3762753271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c502f710 msgr2=0x55c6c502faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3762753271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c502f710 0x55c6c502faf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0d18096d60 tx=0x7f0d1809a700 comp rx=0 tx=0).stop 2023-09-10T01:07:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3762753271 shutdown_connections 2023-09-10T01:07:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3762753271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c502f710 0x55c6c502faf0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3762753271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c503c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3762753271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5030030 0x55c6c5039900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3762753271 >> 172.21.15.3:0/3762753271 conn(0x55c6c4fa4090 msgr2=0x55c6c502abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3762753271 shutdown_connections 2023-09-10T01:07:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.484+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3762753271 wait complete. 2023-09-10T01:07:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.485+0000 7f0d37af7b80 1 -- start start 2023-09-10T01:07:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.485+0000 7f0d37af7b80 1 Processor -- start 2023-09-10T01:07:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.485+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c502f710 0x55c6c503a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.485+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.485+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c50452a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c4f1fb70 con 0x55c6c502f710 2023-09-10T01:07:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c4f1f870 con 0x55c6c5039e40 2023-09-10T01:07:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c4e03920 con 0x55c6c5030030 2023-09-10T01:07:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d37af7b80 10 client.0 _ll_register_callbacks cb 0x55c6c4e69a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c502f710 0x55c6c503a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c502f710 0x55c6c503a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42066/0 (socket says 172.21.15.3:42066) 2023-09-10T01:07:28.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d2179a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 -- 172.21.15.3:0/2130594585 learned_addr learned my addr 172.21.15.3:0/2130594585 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d2279c700 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c50452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 -- 172.21.15.3:0/2130594585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 msgr2=0x55c6c503a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 -- 172.21.15.3:0/2130594585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 msgr2=0x55c6c50452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c50452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d21f9b700 1 -- 172.21.15.3:0/2130594585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c1a56d0 con 0x55c6c502f710 2023-09-10T01:07:28.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.486+0000 7f0d2279c700 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c50452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c502f710 0x55c6c503a220 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006010 tx=0x7f0d1809a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16895f30cf6befef server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d057fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:07:28.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d057fa700 1 -- 172.21.15.3:0/2130594585 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1809acc0 con 0x55c6c502f710 2023-09-10T01:07:28.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d057fa700 1 -- 172.21.15.3:0/2130594585 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0d1809e690 con 0x55c6c502f710 2023-09-10T01:07:28.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6c4ed94f0 con 0x55c6c502f710 2023-09-10T01:07:28.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d057fa700 1 -- 172.21.15.3:0/2130594585 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d1809a860 con 0x55c6c502f710 2023-09-10T01:07:28.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d37af7b80 10 client.16833 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:28.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c6c4d96c60 con 0x55c6c502f710 2023-09-10T01:07:28.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.487+0000 7f0d057fa700 1 -- 172.21.15.3:0/2130594585 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c6c4ed94f0 con 0x55c6c502f710 2023-09-10T01:07:28.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d057fa700 1 -- 172.21.15.3:0/2130594585 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0d18060cf0 con 0x55c6c502f710 2023-09-10T01:07:28.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d37af7b80 10 client.16833 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:28.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d37af7b80 10 client.16833 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:07:28.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d37af7b80 20 client.16833 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:28.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c5049140 0x55c6c504b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d37af7b80 4 client.16833 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:28.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6c4f968a0 con 0x55c6c5049140 2023-09-10T01:07:28.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d2179a700 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c5049140 0x55c6c504b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d2179a700 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c5049140 0x55c6c504b560 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=277645226d914f40 server_cookie=eadea736d3f0fc4e in_seq=0 out_seq=0 2023-09-10T01:07:28.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.488+0000 7f0d057fa700 10 client.16833 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:07:28.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d057fa700 1 -- 172.21.15.3:0/2130594585 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c6c4f968a0 con 0x55c6c5049140 2023-09-10T01:07:28.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d057fa700 10 client.16833 handle_command_reply: tid=0 2023-09-10T01:07:28.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 client.16833 shutdown 2023-09-10T01:07:28.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c502f710 msgr2=0x55c6c503a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c502f710 0x55c6c503a220 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006010 tx=0x7f0d1809a2c0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 shutdown_connections 2023-09-10T01:07:28.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c5049140 0x55c6c504b560 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c5039e40 0x55c6c50452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c502f710 0x55c6c503a220 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d057fa700 0 client.16833 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:07:28.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/2130594585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.489+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 >> 172.21.15.3:0/2130594585 conn(0x55c6c4fa4090 msgr2=0x55c6c502a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.490+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 shutdown_connections 2023-09-10T01:07:28.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.490+0000 7f0d37af7b80 1 -- 172.21.15.3:0/2130594585 wait complete. 2023-09-10T01:07:28.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.490+0000 7f0d37af7b80 20 client.16833 trim_cache size 0 max 16384 2023-09-10T01:07:28.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.504+0000 7f0d37af7b80 1 Processor -- start 2023-09-10T01:07:28.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d37af7b80 1 -- start start 2023-09-10T01:07:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c50342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c5033de0 con 0x55c6c5039e40 2023-09-10T01:07:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c503f780 con 0x55c6c502f710 2023-09-10T01:07:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c50374e0 con 0x55c6c5030030 2023-09-10T01:07:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d2179a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c50342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44472/0 (socket says 172.21.15.3:44472) 2023-09-10T01:07:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d2279c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d2179a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c50342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45854/0 (socket says 172.21.15.3:45854) 2023-09-10T01:07:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.505+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3289597648 learned_addr learned my addr 172.21.15.3:0/3289597648 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3289597648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 msgr2=0x55c6c50342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3289597648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c50342c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3289597648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 msgr2=0x55c6c503a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3289597648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3289597648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6c4dd4250 con 0x55c6c502f710 2023-09-10T01:07:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3289597648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503f240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0d1809a560 tx=0x7f0d18098e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=407ac772778e45b0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d10ff9700 1 -- 172.21.15.3:0/3289597648 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18099f50 con 0x55c6c502f710 2023-09-10T01:07:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d10ff9700 1 -- 172.21.15.3:0/3289597648 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d18021e90 con 0x55c6c502f710 2023-09-10T01:07:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.506+0000 7f0d10ff9700 1 -- 172.21.15.3:0/3289597648 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18060e30 con 0x55c6c502f710 2023-09-10T01:07:28.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3289597648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 msgr2=0x55c6c503f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3289597648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503f240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0d1809a560 tx=0x7f0d18098e10 comp rx=0 tx=0).stop 2023-09-10T01:07:28.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3289597648 shutdown_connections 2023-09-10T01:07:28.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3289597648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3289597648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503f240 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3289597648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c50342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3289597648 >> 172.21.15.3:0/3289597648 conn(0x55c6c4fa4090 msgr2=0x55c6c5029320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3289597648 shutdown_connections 2023-09-10T01:07:28.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3289597648 wait complete. 2023-09-10T01:07:28.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.507+0000 7f0d37af7b80 1 -- start start 2023-09-10T01:07:28.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 1 Processor -- start 2023-09-10T01:07:28.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c503dc50 con 0x55c6c5039e40 2023-09-10T01:07:28.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c503ddc0 con 0x55c6c502f710 2023-09-10T01:07:28.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6c503df80 con 0x55c6c5030030 2023-09-10T01:07:28.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d37af7b80 10 client.0 _ll_register_callbacks cb 0x7f0d0c009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:28.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d2179a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d2279c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d21f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44476/0 (socket says 172.21.15.3:44476) 2023-09-10T01:07:28.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.508+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3097203003 learned_addr learned my addr 172.21.15.3:0/3097203003 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 msgr2=0x55c6c503d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3097203003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 msgr2=0x55c6c503d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d21f9b700 1 -- 172.21.15.3:0/3097203003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6c4d96c60 con 0x55c6c502f710 2023-09-10T01:07:28.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d2179a700 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d21f9b700 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503eee0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0d1809a7f0 tx=0x7f0d18099f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=989aea33e2a2b71c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d057fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:28.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d057fa700 1 -- 172.21.15.3:0/3097203003 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1809a3a0 con 0x55c6c502f710 2023-09-10T01:07:28.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d180b0030 con 0x55c6c502f710 2023-09-10T01:07:28.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d057fa700 1 -- 172.21.15.3:0/3097203003 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f0d1809e690 con 0x55c6c502f710 2023-09-10T01:07:28.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d057fa700 1 -- 172.21.15.3:0/3097203003 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d1809a860 con 0x55c6c502f710 2023-09-10T01:07:28.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d37af7b80 10 client.26900 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:28.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.509+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c6c4dac2b0 con 0x55c6c502f710 2023-09-10T01:07:28.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d057fa700 1 -- 172.21.15.3:0/3097203003 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d180b0030 con 0x55c6c502f710 2023-09-10T01:07:28.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d057fa700 1 -- 172.21.15.3:0/3097203003 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0d180961d0 con 0x55c6c502f710 2023-09-10T01:07:28.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d37af7b80 10 client.26900 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:28.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d37af7b80 10 client.26900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:07:28.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d37af7b80 20 client.26900 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:28.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c509de60 0x55c6c50a0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d37af7b80 4 client.26900 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:07:28.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c6c4f7a920 con 0x55c6c509de60 2023-09-10T01:07:28.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.510+0000 7f0d2179a700 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c509de60 0x55c6c50a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d2179a700 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c509de60 0x55c6c50a0250 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2f55ee34a7fa39d server_cookie=23d90543ea6db814 in_seq=0 out_seq=0 2023-09-10T01:07:28.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d057fa700 10 client.26900 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:07:28.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d057fa700 1 -- 172.21.15.3:0/3097203003 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c6c4f7a920 con 0x55c6c509de60 2023-09-10T01:07:28.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d057fa700 10 client.26900 handle_command_reply: tid=0 2023-09-10T01:07:28.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 client.26900 shutdown 2023-09-10T01:07:28.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 msgr2=0x55c6c503eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503eee0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0d1809a7f0 tx=0x7f0d18099f80 comp rx=0 tx=0).stop 2023-09-10T01:07:28.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 shutdown_connections 2023-09-10T01:07:28.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55c6c509de60 0x55c6c50a0250 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c6c5039e40 0x55c6c503d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c6c502f710 0x55c6c503eee0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 --2- 172.21.15.3:0/3097203003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c6c5030030 0x55c6c503d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 >> 172.21.15.3:0/3097203003 conn(0x55c6c4fa4090 msgr2=0x55c6c5038820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d057fa700 0 client.26900 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:07:28.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 shutdown_connections 2023-09-10T01:07:28.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 1 -- 172.21.15.3:0/3097203003 wait complete. 2023-09-10T01:07:28.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.511+0000 7f0d37af7b80 20 client.26900 trim_cache size 0 max 16384 2023-09-10T01:07:28.531 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:07:28.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d080a07b0 msgr2=0x7f0d080a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d080a07b0 0x7f0d080a2c70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009550 tx=0x7f0d14009130 comp rx=0 tx=0).stop 2023-09-10T01:07:28.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c100640 msgr2=0x7f0d1c19e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c100640 0x7f0d1c19e6a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0d18000c00 tx=0x7f0d180047b0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 shutdown_connections 2023-09-10T01:07:28.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d080a07b0 0x7f0d080a2c70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d1c106650 0x7f0d1c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d1c100640 0x7f0d1c19e6a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 --2- 172.21.15.3:0/4023397163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c101000 0x7f0d1c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 >> 172.21.15.3:0/4023397163 conn(0x7f0d1c0fc170 msgr2=0x7f0d1c10b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 shutdown_connections 2023-09-10T01:07:28.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.512+0000 7f0d241ff700 1 -- 172.21.15.3:0/4023397163 wait complete. 2023-09-10T01:07:28.534 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:28 smithi003 ceph-mon[97654]: pgmap v130: 129 pgs: 129 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 270 op/s 2023-09-10T01:07:28.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:07:28.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 342210 9.37 MB/sec execute 70 sec latency 63.316 ms 2023-09-10T01:07:28.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.812+0000 7f8c565ff700 1 -- 172.21.15.3:0/2940416988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 msgr2=0x7f8c50113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.812+0000 7f8c565ff700 1 --2- 172.21.15.3:0/2940416988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 0x7f8c50113a40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800b0a0 tx=0x7f8c4800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.812+0000 7f8c565ff700 1 -- 172.21.15.3:0/2940416988 shutdown_connections 2023-09-10T01:07:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.812+0000 7f8c565ff700 1 --2- 172.21.15.3:0/2940416988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c500729a0 0x7f8c50072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.812+0000 7f8c565ff700 1 --2- 172.21.15.3:0/2940416988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 0x7f8c50113a40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.812+0000 7f8c565ff700 1 --2- 172.21.15.3:0/2940416988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 0x7f8c50111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.812+0000 7f8c565ff700 1 -- 172.21.15.3:0/2940416988 >> 172.21.15.3:0/2940416988 conn(0x7f8c500fc170 msgr2=0x7f8c500fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.813+0000 7f8c565ff700 1 -- 172.21.15.3:0/2940416988 shutdown_connections 2023-09-10T01:07:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.813+0000 7f8c565ff700 1 -- 172.21.15.3:0/2940416988 wait complete. 2023-09-10T01:07:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.813+0000 7f8c565ff700 1 Processor -- start 2023-09-10T01:07:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c565ff700 1 -- start start 2023-09-10T01:07:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c565ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c500729a0 0x7f8c5019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c565ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 0x7f8c5019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c565ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 0x7f8c501a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c565ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c501a1e70 con 0x7f8c500732c0 2023-09-10T01:07:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c565ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c501a1fe0 con 0x7f8c50111650 2023-09-10T01:07:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c565ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c501a21a0 con 0x7f8c500729a0 2023-09-10T01:07:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 0x7f8c5019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 0x7f8c5019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42108/0 (socket says 172.21.15.3:42108) 2023-09-10T01:07:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c47fff700 1 -- 172.21.15.3:0/3317997479 learned_addr learned my addr 172.21.15.3:0/3317997479 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c500729a0 0x7f8c5019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.814+0000 7f8c54b9c700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 0x7f8c501a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c47fff700 1 -- 172.21.15.3:0/3317997479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c500729a0 msgr2=0x7f8c5019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c47fff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c500729a0 0x7f8c5019ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c47fff700 1 -- 172.21.15.3:0/3317997479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 msgr2=0x7f8c501a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c47fff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 0x7f8c501a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c47fff700 1 -- 172.21.15.3:0/3317997479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48009d00 con 0x7f8c500732c0 2023-09-10T01:07:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c47fff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 0x7f8c5019d3d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d950 tx=0x7f8c3800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c4dffb700 1 -- 172.21.15.3:0/3317997479 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800f890 con 0x7f8c500732c0 2023-09-10T01:07:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c501a2480 con 0x7f8c500732c0 2023-09-10T01:07:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c4dffb700 1 -- 172.21.15.3:0/3317997479 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c38010460 con 0x7f8c500732c0 2023-09-10T01:07:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c4dffb700 1 -- 172.21.15.3:0/3317997479 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800e610 con 0x7f8c500732c0 2023-09-10T01:07:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.815+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c501a9cc0 con 0x7f8c500732c0 2023-09-10T01:07:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.819+0000 7f8c4dffb700 1 -- 172.21.15.3:0/3317997479 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f8c380108d0 con 0x7f8c500732c0 2023-09-10T01:07:28.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.820+0000 7f8c69fdeb80 1 Processor -- start 2023-09-10T01:07:28.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.820+0000 7f8c4dffb700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c3c0a07b0 0x7f8c3c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c4dffb700 1 -- 172.21.15.3:0/3317997479 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f8c380cc0b0 con 0x7f8c500732c0 2023-09-10T01:07:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c69fdeb80 1 -- start start 2023-09-10T01:07:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c54b9c700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c3c0a07b0 0x7f8c3c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bbdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21b16730 con 0x557a21bbd710 2023-09-10T01:07:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21b14050 con 0x557a21bc7e40 2023-09-10T01:07:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.821+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21b0f940 con 0x557a21bbe030 2023-09-10T01:07:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bbdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bbdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42110/0 (socket says 172.21.15.3:42110) 2023-09-10T01:07:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 -- 172.21.15.3:0/2321195279 learned_addr learned my addr 172.21.15.3:0/2321195279 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c47fff700 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c54b9c700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c3c0a07b0 0x7f8c3c0a2c70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800b780 tx=0x7f8c48006230 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c54b9c700 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 -- 172.21.15.3:0/2321195279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 msgr2=0x557a21bc7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 -- 172.21.15.3:0/2321195279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 msgr2=0x557a21bca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 -- 172.21.15.3:0/2321195279 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a2193bde0 con 0x557a21bbd710 2023-09-10T01:07:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c47fff700 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bbdaf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8c40009d80 tx=0x7f8c40011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf85558d37792df server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.822+0000 7f8c467fc700 1 -- 172.21.15.3:0/2321195279 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c40004230 con 0x557a21bbd710 2023-09-10T01:07:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c467fc700 1 -- 172.21.15.3:0/2321195279 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c40004390 con 0x557a21bbd710 2023-09-10T01:07:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c467fc700 1 -- 172.21.15.3:0/2321195279 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c40019860 con 0x557a21bbd710 2023-09-10T01:07:28.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/2321195279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 msgr2=0x557a21bbdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bbdaf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8c40009d80 tx=0x7f8c40011720 comp rx=0 tx=0).stop 2023-09-10T01:07:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/2321195279 shutdown_connections 2023-09-10T01:07:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bbdaf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/2321195279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.823+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/2321195279 >> 172.21.15.3:0/2321195279 conn(0x557a21b32090 msgr2=0x557a21bb8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.824+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/2321195279 shutdown_connections 2023-09-10T01:07:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.824+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/2321195279 wait complete. 2023-09-10T01:07:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.824+0000 7f8c69fdeb80 1 -- start start 2023-09-10T01:07:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.824+0000 7f8c69fdeb80 1 Processor -- start 2023-09-10T01:07:28.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bc7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bc4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21aadb70 con 0x557a21bbd710 2023-09-10T01:07:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21aad870 con 0x557a21bc7e40 2023-09-10T01:07:28.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21991920 con 0x557a21bbe030 2023-09-10T01:07:28.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c69fdeb80 10 client.0 _ll_register_callbacks cb 0x557a219f7a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:28.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bc7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c54b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bc4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c54b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bc4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44506/0 (socket says 172.21.15.3:44506) 2023-09-10T01:07:28.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bc7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42114/0 (socket says 172.21.15.3:42114) 2023-09-10T01:07:28.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c54b9c700 1 -- 172.21.15.3:0/1567920194 learned_addr learned my addr 172.21.15.3:0/1567920194 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c47fff700 1 -- 172.21.15.3:0/1567920194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 msgr2=0x557a21bc4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c47fff700 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bc4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c47fff700 1 -- 172.21.15.3:0/1567920194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 msgr2=0x557a21bc7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.825+0000 7f8c47fff700 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bc7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c47fff700 1 -- 172.21.15.3:0/1567920194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c501a9cc0 con 0x557a21bbe030 2023-09-10T01:07:28.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c47fff700 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc49e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8c380c9410 tx=0x7f8c380991f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b76d38c0699fb27c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:28.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1567920194 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c380995d0 con 0x557a21bbe030 2023-09-10T01:07:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a21a674f0 con 0x557a21bbe030 2023-09-10T01:07:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1567920194 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f8c38098660 con 0x557a21bbe030 2023-09-10T01:07:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1567920194 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c380b71a0 con 0x557a21bbe030 2023-09-10T01:07:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c69fdeb80 10 client.26836 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:28.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.826+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f8c400099e0 con 0x557a21bbe030 2023-09-10T01:07:28.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1567920194 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557a21a674f0 con 0x557a21bbe030 2023-09-10T01:07:28.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1567920194 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f8c380c9b60 con 0x557a21bbe030 2023-09-10T01:07:28.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c69fdeb80 10 client.26836 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:28.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c69fdeb80 10 client.26836 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:07:28.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c69fdeb80 20 client.26836 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:28.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21bd7790 0x557a21bd9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c69fdeb80 4 client.26836 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:28.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a21b248a0 con 0x557a21bd7790 2023-09-10T01:07:28.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.827+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21bd7790 0x557a21bd9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.828+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21bd7790 0x557a21bd9bb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57bc5484bbb1de0a server_cookie=dcfda849ae73f690 in_seq=0 out_seq=0 2023-09-10T01:07:28.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.828+0000 7f8c2f7fe700 10 client.26836 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:07:28.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.828+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1567920194 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557a21b248a0 con 0x557a21bd7790 2023-09-10T01:07:28.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.828+0000 7f8c2f7fe700 10 client.26836 handle_command_reply: tid=0 2023-09-10T01:07:28.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.828+0000 7f8c69fdeb80 1 client.26836 shutdown 2023-09-10T01:07:28.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 msgr2=0x557a21bc49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc49e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8c380c9410 tx=0x7f8c380991f0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 shutdown_connections 2023-09-10T01:07:28.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bc7e40 0x557a21bc4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21bc7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21bd7790 0x557a21bd9bb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1567920194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbe030 0x557a21bc49e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c2f7fe700 0 client.26836 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:07:28.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 >> 172.21.15.3:0/1567920194 conn(0x557a21b32090 msgr2=0x557a21bb87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 shutdown_connections 2023-09-10T01:07:28.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1567920194 wait complete. 2023-09-10T01:07:28.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.829+0000 7f8c69fdeb80 20 client.26836 trim_cache size 0 max 16384 2023-09-10T01:07:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 Processor -- start 2023-09-10T01:07:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 -- start start 2023-09-10T01:07:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbd710 0x557a21bca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21bce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bc7e40 0x557a21bc7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21bd4fa0 con 0x557a21bc7e40 2023-09-10T01:07:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21bcef30 con 0x557a21bbe030 2023-09-10T01:07:28.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21bd9dd0 con 0x557a21bbd710 2023-09-10T01:07:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21bce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbd710 0x557a21bca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c54b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bc7e40 0x557a21bc7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbd710 0x557a21bca230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45908/0 (socket says 172.21.15.3:45908) 2023-09-10T01:07:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21bce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44508/0 (socket says 172.21.15.3:44508) 2023-09-10T01:07:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.844+0000 7f8c4ffff700 1 -- 172.21.15.3:0/1782061548 learned_addr learned my addr 172.21.15.3:0/1782061548 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c47fff700 1 -- 172.21.15.3:0/1782061548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbd710 msgr2=0x557a21bca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c47fff700 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbd710 0x557a21bca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c47fff700 1 -- 172.21.15.3:0/1782061548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bc7e40 msgr2=0x557a21bc7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c47fff700 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bc7e40 0x557a21bc7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c47fff700 1 -- 172.21.15.3:0/1782061548 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a21924c60 con 0x557a21bbe030 2023-09-10T01:07:28.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbd710 0x557a21bca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c47fff700 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21bce9f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8c38099a20 tx=0x7f8c38098c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca70658c08775e9c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c46ffd700 1 -- 172.21.15.3:0/1782061548 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38014070 con 0x557a21bbe030 2023-09-10T01:07:28.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c46ffd700 1 -- 172.21.15.3:0/1782061548 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c380b71a0 con 0x557a21bbe030 2023-09-10T01:07:28.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.845+0000 7f8c46ffd700 1 -- 172.21.15.3:0/1782061548 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c380988c0 con 0x557a21bbe030 2023-09-10T01:07:28.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1782061548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 msgr2=0x557a21bce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21bce9f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8c38099a20 tx=0x7f8c38098c00 comp rx=0 tx=0).stop 2023-09-10T01:07:28.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1782061548 shutdown_connections 2023-09-10T01:07:28.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bbd710 0x557a21bca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bc7e40 0x557a21bc7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1782061548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21bce9f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1782061548 >> 172.21.15.3:0/1782061548 conn(0x557a21b32090 msgr2=0x557a21bd3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1782061548 shutdown_connections 2023-09-10T01:07:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.846+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1782061548 wait complete. 2023-09-10T01:07:28.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.847+0000 7f8c69fdeb80 1 -- start start 2023-09-10T01:07:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.847+0000 7f8c69fdeb80 1 Processor -- start 2023-09-10T01:07:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.847+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21c1f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.847+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21c1fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c69fdeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bc7e40 0x557a21c2a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21c20370 con 0x557a21bbd710 2023-09-10T01:07:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21c2ab30 con 0x557a21bbe030 2023-09-10T01:07:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c69fdeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a21c2ac70 con 0x557a21bc7e40 2023-09-10T01:07:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c69fdeb80 10 client.0 _ll_register_callbacks cb 0x7f8c38098d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c54b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bc7e40 0x557a21c2a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21c1fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21c1f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21c1fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44520/0 (socket says 172.21.15.3:44520) 2023-09-10T01:07:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c54b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bc7e40 0x557a21c2a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45922/0 (socket says 172.21.15.3:45922) 2023-09-10T01:07:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c47fff700 1 -- 172.21.15.3:0/1271023534 learned_addr learned my addr 172.21.15.3:0/1271023534 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:28.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c54b9c700 1 -- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 msgr2=0x557a21c1fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c54b9c700 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21c1fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c54b9c700 1 -- 172.21.15.3:0/1271023534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 msgr2=0x557a21c1f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c54b9c700 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21c1f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c54b9c700 1 -- 172.21.15.3:0/1271023534 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a2193bde0 con 0x557a21bc7e40 2023-09-10T01:07:28.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c47fff700 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21c1fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.848+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21c1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c54b9c700 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bc7e40 0x557a21c2a5f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009fd0 tx=0x7f8c480099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ba14842e4a1f06 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1271023534 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800e040 con 0x557a21bc7e40 2023-09-10T01:07:28.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c40004390 con 0x557a21bc7e40 2023-09-10T01:07:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1271023534 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f8c48022040 con 0x557a21bc7e40 2023-09-10T01:07:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1271023534 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c48029480 con 0x557a21bc7e40 2023-09-10T01:07:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c69fdeb80 10 client.26842 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f8c400099e0 con 0x557a21bc7e40 2023-09-10T01:07:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.849+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1271023534 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c40004390 con 0x557a21bc7e40 2023-09-10T01:07:28.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1271023534 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f8c4801c740 con 0x557a21bc7e40 2023-09-10T01:07:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c69fdeb80 10 client.26842 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c69fdeb80 10 client.26842 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:07:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c69fdeb80 20 client.26842 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21c2c3f0 0x557a21c2e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:28.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c69fdeb80 4 client.26842 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:07:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557a21b08920 con 0x557a21c2c3f0 2023-09-10T01:07:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21c2c3f0 0x557a21c2e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c4ffff700 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21c2c3f0 0x557a21c2e810 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b8d2108aeb592b4 server_cookie=178d1a45033b40c1 in_seq=0 out_seq=0 2023-09-10T01:07:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.850+0000 7f8c2f7fe700 10 client.26842 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:07:28.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c2f7fe700 1 -- 172.21.15.3:0/1271023534 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557a21b08920 con 0x557a21c2c3f0 2023-09-10T01:07:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c2f7fe700 10 client.26842 handle_command_reply: tid=0 2023-09-10T01:07:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 client.26842 shutdown 2023-09-10T01:07:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bc7e40 msgr2=0x557a21c2a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bc7e40 0x557a21c2a5f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009fd0 tx=0x7f8c480099e0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 shutdown_connections 2023-09-10T01:07:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557a21c2c3f0 0x557a21c2e810 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a21bc7e40 0x557a21c2a5f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557a21bbd710 0x557a21c1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 --2- 172.21.15.3:0/1271023534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a21bbe030 0x557a21c1fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 >> 172.21.15.3:0/1271023534 conn(0x557a21b32090 msgr2=0x557a21bbfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c2f7fe700 0 client.26842 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:07:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 shutdown_connections 2023-09-10T01:07:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 1 -- 172.21.15.3:0/1271023534 wait complete. 2023-09-10T01:07:28.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.851+0000 7f8c69fdeb80 20 client.26842 trim_cache size 0 max 16384 2023-09-10T01:07:28.875 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:07:28.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c3c0a07b0 msgr2=0x7f8c3c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c3c0a07b0 0x7f8c3c0a2c70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800b780 tx=0x7f8c48006230 comp rx=0 tx=0).stop 2023-09-10T01:07:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 msgr2=0x7f8c5019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 0x7f8c5019d3d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800d950 tx=0x7f8c3800dc60 comp rx=0 tx=0).stop 2023-09-10T01:07:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 shutdown_connections 2023-09-10T01:07:28.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c3c0a07b0 0x7f8c3c0a2c70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c500729a0 0x7f8c5019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c50111650 0x7f8c501a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 --2- 172.21.15.3:0/3317997479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c500732c0 0x7f8c5019d3d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:28.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 >> 172.21.15.3:0/3317997479 conn(0x7f8c500fc170 msgr2=0x7f8c500fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:28.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 shutdown_connections 2023-09-10T01:07:28.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:28.852+0000 7f8c565ff700 1 -- 172.21.15.3:0/3317997479 wait complete. 2023-09-10T01:07:28.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:07:29.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 -- 172.21.15.3:0/1903194313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00106650 msgr2=0x7f7a00106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 --2- 172.21.15.3:0/1903194313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00106650 0x7f7a00106a30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009990 tx=0x7f79f4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 -- 172.21.15.3:0/1903194313 shutdown_connections 2023-09-10T01:07:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 --2- 172.21.15.3:0/1903194313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a00101000 0x7f7a0010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 --2- 172.21.15.3:0/1903194313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00106650 0x7f7a00106a30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 --2- 172.21.15.3:0/1903194313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a00100640 0x7f7a00100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 -- 172.21.15.3:0/1903194313 >> 172.21.15.3:0/1903194313 conn(0x7f7a000fc170 msgr2=0x7f7a000fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 -- 172.21.15.3:0/1903194313 shutdown_connections 2023-09-10T01:07:29.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.103+0000 7f7a071ff700 1 -- 172.21.15.3:0/1903194313 wait complete. 2023-09-10T01:07:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.104+0000 7f7a071ff700 1 Processor -- start 2023-09-10T01:07:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.104+0000 7f7a071ff700 1 -- start start 2023-09-10T01:07:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.104+0000 7f7a071ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a00100640 0x7f7a00072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.104+0000 7f7a071ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00101000 0x7f7a00072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f7a071ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a00106650 0x7f7a001a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f7a071ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a00073670 con 0x7f7a00100640 2023-09-10T01:07:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f7a071ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a001a4de0 con 0x7f7a00101000 2023-09-10T01:07:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f7a071ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a001a4f20 con 0x7f7a00106650 2023-09-10T01:07:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f7a04f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a00100640 0x7f7a00072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00101000 0x7f7a00072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00101000 0x7f7a00072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44534/0 (socket says 172.21.15.3:44534) 2023-09-10T01:07:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 -- 172.21.15.3:0/3736707632 learned_addr learned my addr 172.21.15.3:0/3736707632 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f7a0579c700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a00106650 0x7f7a001a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 -- 172.21.15.3:0/3736707632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a00106650 msgr2=0x7f7a001a48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a00106650 0x7f7a001a48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 -- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a00100640 msgr2=0x7f7a00072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a00100640 0x7f7a00072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 -- 172.21.15.3:0/3736707632 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f40095f0 con 0x7f7a00101000 2023-09-10T01:07:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.105+0000 7f79fffff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00101000 0x7f7a00072e40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00d980 tx=0x7f79ec00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.106+0000 7f79fdffb700 1 -- 172.21.15.3:0/3736707632 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec004220 con 0x7f7a00101000 2023-09-10T01:07:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.106+0000 7f79fdffb700 1 -- 172.21.15.3:0/3736707632 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79ec007640 con 0x7f7a00101000 2023-09-10T01:07:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.106+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a001a5180 con 0x7f7a00101000 2023-09-10T01:07:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.106+0000 7f79fdffb700 1 -- 172.21.15.3:0/3736707632 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec005610 con 0x7f7a00101000 2023-09-10T01:07:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.106+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a001a56b0 con 0x7f7a00101000 2023-09-10T01:07:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.109+0000 7f79fdffb700 1 -- 172.21.15.3:0/3736707632 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f79ec004750 con 0x7f7a00101000 2023-09-10T01:07:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.110+0000 7f7a1aa7db80 1 Processor -- start 2023-09-10T01:07:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a1aa7db80 1 -- start start 2023-09-10T01:07:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f79fdffb700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f00a0830 0x7f79f00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f79fdffb700 1 -- 172.21.15.3:0/3736707632 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f79ec0cc180 con 0x7f7a00101000 2023-09-10T01:07:29.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a0579c700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f00a0830 0x7f79f00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3df230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f32b730 con 0x55cf2f3d3030 2023-09-10T01:07:29.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f329050 con 0x55cf2f3dce40 2023-09-10T01:07:29.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f324940 con 0x55cf2f3d2710 2023-09-10T01:07:29.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f79fffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.111+0000 7f7a04f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3d2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45956/0 (socket says 172.21.15.3:45956) 2023-09-10T01:07:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a0579c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3df230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f79fffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42164/0 (socket says 172.21.15.3:42164) 2023-09-10T01:07:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 -- 172.21.15.3:0/923156354 learned_addr learned my addr 172.21.15.3:0/923156354 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 -- 172.21.15.3:0/923156354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 msgr2=0x55cf2f3df230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3df230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 -- 172.21.15.3:0/923156354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 msgr2=0x55cf2f3dc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 -- 172.21.15.3:0/923156354 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf2f150de0 con 0x55cf2f3d2710 2023-09-10T01:07:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f79fffff700 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a0579c700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f00a0830 0x7f79f00a2cf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7a00074a10 tx=0x7f79f8007a80 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3d2af0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f79f4005f50 tx=0x7f79f400b610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ef0f0a37cf2c3c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.112+0000 7f79eaffd700 1 -- 172.21.15.3:0/923156354 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f401d070 con 0x55cf2f3d2710 2023-09-10T01:07:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f79eaffd700 1 -- 172.21.15.3:0/923156354 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79f401e040 con 0x55cf2f3d2710 2023-09-10T01:07:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f79eaffd700 1 -- 172.21.15.3:0/923156354 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f40048a0 con 0x55cf2f3d2710 2023-09-10T01:07:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/923156354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 msgr2=0x55cf2f3d2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3d2af0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f79f4005f50 tx=0x7f79f400b610 comp rx=0 tx=0).stop 2023-09-10T01:07:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/923156354 shutdown_connections 2023-09-10T01:07:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3d2af0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3df230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/923156354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/923156354 >> 172.21.15.3:0/923156354 conn(0x55cf2f347090 msgr2=0x55cf2f3cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:29.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/923156354 shutdown_connections 2023-09-10T01:07:29.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.113+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/923156354 wait complete. 2023-09-10T01:07:29.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.114+0000 7f7a1aa7db80 1 -- start start 2023-09-10T01:07:29.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.114+0000 7f7a1aa7db80 1 Processor -- start 2023-09-10T01:07:29.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d2710 0x55cf2f3dc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d3030 0x55cf2f3d9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3da0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f2c2b70 con 0x55cf2f3d2710 2023-09-10T01:07:29.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f2c2870 con 0x55cf2f3dce40 2023-09-10T01:07:29.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f1a6920 con 0x55cf2f3d3030 2023-09-10T01:07:29.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a1aa7db80 10 client.0 _ll_register_callbacks cb 0x55cf2f20ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:29.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a0579c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3da0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a0579c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3da0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44548/0 (socket says 172.21.15.3:44548) 2023-09-10T01:07:29.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a0579c700 1 -- 172.21.15.3:0/3269620177 learned_addr learned my addr 172.21.15.3:0/3269620177 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:29.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f79fffff700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d3030 0x55cf2f3d9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d2710 0x55cf2f3dc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f79fffff700 1 -- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 msgr2=0x55cf2f3da0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f79fffff700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3da0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f79fffff700 1 -- 172.21.15.3:0/3269620177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d2710 msgr2=0x55cf2f3dc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f79fffff700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d2710 0x55cf2f3dc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f79fffff700 1 -- 172.21.15.3:0/3269620177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a001a56b0 con 0x55cf2f3d3030 2023-09-10T01:07:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.115+0000 7f7a0579c700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3da0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f79fffff700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d3030 0x55cf2f3d9ab0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0c94e0 tx=0x7f79ec099a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f1c2fee6ad3bf5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f79dffff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f79dffff700 1 -- 172.21.15.3:0/3269620177 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0b0070 con 0x55cf2f3d3030 2023-09-10T01:07:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f79dffff700 1 -- 172.21.15.3:0/3269620177 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f79ec0a12b0 con 0x55cf2f3d3030 2023-09-10T01:07:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf2f27c4f0 con 0x55cf2f3d3030 2023-09-10T01:07:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f79dffff700 1 -- 172.21.15.3:0/3269620177 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79ec004380 con 0x55cf2f3d3030 2023-09-10T01:07:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f7a1aa7db80 10 client.26854 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f7a0010de10 con 0x55cf2f3d3030 2023-09-10T01:07:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.116+0000 7f79dffff700 1 -- 172.21.15.3:0/3269620177 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79ec07c080 con 0x55cf2f3d3030 2023-09-10T01:07:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f79dffff700 1 -- 172.21.15.3:0/3269620177 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f79ec098730 con 0x55cf2f3d3030 2023-09-10T01:07:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f7a1aa7db80 10 client.26854 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f7a1aa7db80 10 client.26854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f7a1aa7db80 20 client.26854 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f3ec830 0x55cf2f3eec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f7a1aa7db80 4 client.26854 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf2f3398a0 con 0x55cf2f3ec830 2023-09-10T01:07:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.117+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f3ec830 0x55cf2f3eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.118+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f3ec830 0x55cf2f3eec50 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bcec19ff6da8f14 server_cookie=bf0275bb78f7cbd1 in_seq=0 out_seq=0 2023-09-10T01:07:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.118+0000 7f79dffff700 10 client.26854 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.118+0000 7f79dffff700 1 -- 172.21.15.3:0/3269620177 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cf2f3398a0 con 0x55cf2f3ec830 2023-09-10T01:07:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.118+0000 7f79dffff700 10 client.26854 handle_command_reply: tid=0 2023-09-10T01:07:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.118+0000 7f7a1aa7db80 1 client.26854 shutdown 2023-09-10T01:07:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d3030 msgr2=0x55cf2f3d9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d3030 0x55cf2f3d9ab0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0c94e0 tx=0x7f79ec099a80 comp rx=0 tx=0).stop 2023-09-10T01:07:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 shutdown_connections 2023-09-10T01:07:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f3ec830 0x55cf2f3eec50 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3da0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d2710 0x55cf2f3dc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f79dffff700 0 client.26854 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/3269620177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d3030 0x55cf2f3d9ab0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 >> 172.21.15.3:0/3269620177 conn(0x55cf2f347090 msgr2=0x55cf2f3cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 shutdown_connections 2023-09-10T01:07:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/3269620177 wait complete. 2023-09-10T01:07:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.119+0000 7f7a1aa7db80 20 client.26854 trim_cache size 0 max 16384 2023-09-10T01:07:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 Processor -- start 2023-09-10T01:07:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 -- start start 2023-09-10T01:07:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3df230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3d4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f3d8050 con 0x55cf2f3d3030 2023-09-10T01:07:29.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f3eec20 con 0x55cf2f3dce40 2023-09-10T01:07:29.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f3cddd0 con 0x55cf2f3d2710 2023-09-10T01:07:29.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a0579c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3d4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a04f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3df230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a0579c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3d4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44564/0 (socket says 172.21.15.3:44564) 2023-09-10T01:07:29.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.135+0000 7f7a0579c700 1 -- 172.21.15.3:0/1050282793 learned_addr learned my addr 172.21.15.3:0/1050282793 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:29.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f79fffff700 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f7a0579c700 1 -- 172.21.15.3:0/1050282793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 msgr2=0x55cf2f3df230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f7a0579c700 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3df230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f7a0579c700 1 -- 172.21.15.3:0/1050282793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 msgr2=0x55cf2f3dc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f7a0579c700 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f7a0579c700 1 -- 172.21.15.3:0/1050282793 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf2f139c60 con 0x55cf2f3dce40 2023-09-10T01:07:29.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3df230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:29.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f79fffff700 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:29.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f7a0579c700 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3d4dc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f79f801aa10 tx=0x7f79f801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15da01f9e007d855 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:29.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.136+0000 7f79eb7fe700 1 -- 172.21.15.3:0/1050282793 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f800e050 con 0x55cf2f3dce40 2023-09-10T01:07:29.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f79eb7fe700 1 -- 172.21.15.3:0/1050282793 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79f800f040 con 0x55cf2f3dce40 2023-09-10T01:07:29.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f79eb7fe700 1 -- 172.21.15.3:0/1050282793 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8025b60 con 0x55cf2f3dce40 2023-09-10T01:07:29.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1050282793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 msgr2=0x55cf2f3d4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3d4dc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f79f801aa10 tx=0x7f79f801d000 comp rx=0 tx=0).stop 2023-09-10T01:07:29.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1050282793 shutdown_connections 2023-09-10T01:07:29.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3df230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3d4dc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1050282793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3dc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1050282793 >> 172.21.15.3:0/1050282793 conn(0x55cf2f347090 msgr2=0x55cf2f3cc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:29.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1050282793 shutdown_connections 2023-09-10T01:07:29.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.137+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1050282793 wait complete. 2023-09-10T01:07:29.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.138+0000 7f7a1aa7db80 1 -- start start 2023-09-10T01:07:29.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.138+0000 7f7a1aa7db80 1 Processor -- start 2023-09-10T01:07:29.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.138+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3e3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.138+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3e1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a1aa7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3e1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f3e24b0 con 0x55cf2f3d3030 2023-09-10T01:07:29.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f3e2620 con 0x55cf2f3dce40 2023-09-10T01:07:29.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a1aa7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2f3e27e0 con 0x55cf2f3d2710 2023-09-10T01:07:29.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a1aa7db80 10 client.0 _ll_register_callbacks cb 0x7f79ec098ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:29.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3e3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45976/0 (socket says 172.21.15.3:45976) 2023-09-10T01:07:29.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 -- 172.21.15.3:0/1480590643 learned_addr learned my addr 172.21.15.3:0/1480590643 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:29.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f79fffff700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3e1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a0579c700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3e1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 -- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 msgr2=0x55cf2f3e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3e1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 -- 172.21.15.3:0/1480590643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 msgr2=0x55cf2f3e1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:29.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3e1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 -- 172.21.15.3:0/1480590643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf2f150de0 con 0x55cf2f3d2710 2023-09-10T01:07:29.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a0579c700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3e1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:07:29.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f7a04f9b700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3e3770 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009990 tx=0x7f79f400b280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad8aa2a61b50b52f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:29.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.139+0000 7f79dffff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:29.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f79dffff700 1 -- 172.21.15.3:0/1480590643 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4031040 con 0x55cf2f3d2710 2023-09-10T01:07:29.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f79dffff700 1 -- 172.21.15.3:0/1480590643 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f79f4033810 con 0x55cf2f3d2710 2023-09-10T01:07:29.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79f401e040 con 0x55cf2f3d2710 2023-09-10T01:07:29.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f79dffff700 1 -- 172.21.15.3:0/1480590643 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79f401d040 con 0x55cf2f3d2710 2023-09-10T01:07:29.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f7a1aa7db80 10 client.26860 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:29.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f7a0010de10 con 0x55cf2f3d2710 2023-09-10T01:07:29.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f79dffff700 1 -- 172.21.15.3:0/1480590643 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79f401e040 con 0x55cf2f3d2710 2023-09-10T01:07:29.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f79dffff700 1 -- 172.21.15.3:0/1480590643 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f79f4005550 con 0x55cf2f3d2710 2023-09-10T01:07:29.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f7a1aa7db80 10 client.26860 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:29.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f7a1aa7db80 10 client.26860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:29.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f7a1aa7db80 20 client.26860 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:29.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.140+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f4416a0 0x55cf2f443ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:29.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f7a1aa7db80 4 client.26860 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:07:29.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cf2f31d920 con 0x55cf2f4416a0 2023-09-10T01:07:29.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f79fffff700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f4416a0 0x55cf2f443ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:29.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f79fffff700 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f4416a0 0x55cf2f443ac0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=280495abe9eaf411 server_cookie=9c6ae912ec2e847b in_seq=0 out_seq=0 2023-09-10T01:07:29.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f79dffff700 10 client.26860 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:29.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f79dffff700 1 -- 172.21.15.3:0/1480590643 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cf2f31d920 con 0x55cf2f4416a0 2023-09-10T01:07:29.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f79dffff700 10 client.26860 handle_command_reply: tid=0 2023-09-10T01:07:29.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f7a1aa7db80 1 client.26860 shutdown 2023-09-10T01:07:29.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 msgr2=0x55cf2f3e3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.141+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3e3770 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009990 tx=0x7f79f400b280 comp rx=0 tx=0).stop 2023-09-10T01:07:29.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 shutdown_connections 2023-09-10T01:07:29.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55cf2f4416a0 0x55cf2f443ac0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55cf2f3d2710 0x55cf2f3e3770 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cf2f3dce40 0x55cf2f3e1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f79dffff700 0 client.26860 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:29.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 --2- 172.21.15.3:0/1480590643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cf2f3d3030 0x55cf2f3e1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 >> 172.21.15.3:0/1480590643 conn(0x55cf2f347090 msgr2=0x55cf2f3d4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:29.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 shutdown_connections 2023-09-10T01:07:29.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 1 -- 172.21.15.3:0/1480590643 wait complete. 2023-09-10T01:07:29.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a1aa7db80 20 client.26860 trim_cache size 0 max 16384 2023-09-10T01:07:29.160 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:07:29.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f00a0830 msgr2=0x7f79f00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a071ff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f00a0830 0x7f79f00a2cf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7a00074a10 tx=0x7f79f8007a80 comp rx=0 tx=0).stop 2023-09-10T01:07:29.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00101000 msgr2=0x7f7a00072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:29.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.142+0000 7f7a071ff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00101000 0x7f7a00072e40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00d980 tx=0x7f79ec00dc90 comp rx=0 tx=0).stop 2023-09-10T01:07:29.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 shutdown_connections 2023-09-10T01:07:29.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f00a0830 0x7f79f00a2cf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a00106650 0x7f7a001a48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a00100640 0x7f7a00072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 --2- 172.21.15.3:0/3736707632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a00101000 0x7f7a00072e40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:29.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 >> 172.21.15.3:0/3736707632 conn(0x7f7a000fc170 msgr2=0x7f7a00104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:29.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 shutdown_connections 2023-09-10T01:07:29.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:29.143+0000 7f7a071ff700 1 -- 172.21.15.3:0/3736707632 wait complete. 2023-09-10T01:07:29.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:29 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/909231176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:07:29.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:29 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/909231176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:07:29.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:29 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/909231176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:07:29.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 344423 9.43 MB/sec execute 71 sec latency 37.297 ms 2023-09-10T01:07:30.164 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:07:30.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:07:30.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.389+0000 7f99a85ff700 1 -- 172.21.15.3:0/1084964988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0100650 msgr2=0x7f99a0100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.389+0000 7f99a85ff700 1 --2- 172.21.15.3:0/1084964988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0100650 0x7f99a0100ad0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f99900098f0 tx=0x7f9990009c00 comp rx=0 tx=0).stop 2023-09-10T01:07:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.389+0000 7f99a85ff700 1 -- 172.21.15.3:0/1084964988 shutdown_connections 2023-09-10T01:07:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.389+0000 7f99a85ff700 1 --2- 172.21.15.3:0/1084964988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0106660 0x7f99a0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.389+0000 7f99a85ff700 1 --2- 172.21.15.3:0/1084964988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99a0101010 0x7f99a010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.389+0000 7f99a85ff700 1 --2- 172.21.15.3:0/1084964988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0100650 0x7f99a0100ad0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.389+0000 7f99a85ff700 1 -- 172.21.15.3:0/1084964988 >> 172.21.15.3:0/1084964988 conn(0x7f99a00fc140 msgr2=0x7f99a00fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.390+0000 7f99a85ff700 1 -- 172.21.15.3:0/1084964988 shutdown_connections 2023-09-10T01:07:30.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.390+0000 7f99a85ff700 1 -- 172.21.15.3:0/1084964988 wait complete. 2023-09-10T01:07:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 Processor -- start 2023-09-10T01:07:30.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 -- start start 2023-09-10T01:07:30.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99a0100650 0x7f99a019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0101010 0x7f99a019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0106660 0x7f99a0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a0198db0 con 0x7f99a0100650 2023-09-10T01:07:30.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a0198f20 con 0x7f99a0106660 2023-09-10T01:07:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a85ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a01990e0 con 0x7f99a0101010 2023-09-10T01:07:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a6b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0106660 0x7f99a0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.391+0000 7f99a5b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0101010 0x7f99a019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a6b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0106660 0x7f99a0198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44586/0 (socket says 172.21.15.3:44586) 2023-09-10T01:07:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a5b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0101010 0x7f99a019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46002/0 (socket says 172.21.15.3:46002) 2023-09-10T01:07:30.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a5b9a700 1 -- 172.21.15.3:0/4052185694 learned_addr learned my addr 172.21.15.3:0/4052185694 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a639b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99a0100650 0x7f99a019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a6b9c700 1 -- 172.21.15.3:0/4052185694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0101010 msgr2=0x7f99a019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a6b9c700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0101010 0x7f99a019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a6b9c700 1 -- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99a0100650 msgr2=0x7f99a019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a6b9c700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99a0100650 0x7f99a019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a6b9c700 1 -- 172.21.15.3:0/4052185694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c0095f0 con 0x7f99a0106660 2023-09-10T01:07:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0101010 0x7f99a019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a6b9c700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0106660 0x7f99a0198700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f999800efc0 tx=0x7f999800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99977fe700 1 -- 172.21.15.3:0/4052185694 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998018070 con 0x7f99a0106660 2023-09-10T01:07:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99977fe700 1 -- 172.21.15.3:0/4052185694 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f999800f040 con 0x7f99a0106660 2023-09-10T01:07:30.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9990009520 con 0x7f99a0106660 2023-09-10T01:07:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99977fe700 1 -- 172.21.15.3:0/4052185694 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999800cd40 con 0x7f99a0106660 2023-09-10T01:07:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.392+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a01a54c0 con 0x7f99a0106660 2023-09-10T01:07:30.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.396+0000 7f99977fe700 1 -- 172.21.15.3:0/4052185694 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9998003a00 con 0x7f99a0106660 2023-09-10T01:07:30.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:30 smithi136 ceph-mon[133708]: pgmap v131: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 433 op/s 2023-09-10T01:07:30.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 Processor -- start 2023-09-10T01:07:30.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 -- start start 2023-09-10T01:07:30.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fa3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fad980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d11fb02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11efc7b0 con 0x557d11fa3790 2023-09-10T01:07:30.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11efa0d0 con 0x557d11fadec0 2023-09-10T01:07:30.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.397+0000 7f99bc015b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11ef59c0 con 0x557d11fa40b0 2023-09-10T01:07:30.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99977fe700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f998c0a07b0 0x7f998c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a5b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fa3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fad980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46016/0 (socket says 172.21.15.3:46016) 2023-09-10T01:07:30.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a5b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fa3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42222/0 (socket says 172.21.15.3:42222) 2023-09-10T01:07:30.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 -- 172.21.15.3:0/3498744564 learned_addr learned my addr 172.21.15.3:0/3498744564 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d11fb02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99977fe700 1 -- 172.21.15.3:0/4052185694 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f99980cbf00 con 0x7f99a0106660 2023-09-10T01:07:30.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 -- 172.21.15.3:0/3498744564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 msgr2=0x557d11fb02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a6b9c700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f998c0a07b0 0x7f998c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d11fb02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 -- 172.21.15.3:0/3498744564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 msgr2=0x557d11fa3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fa3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 -- 172.21.15.3:0/3498744564 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d11d21e60 con 0x557d11fa40b0 2023-09-10T01:07:30.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.398+0000 7f99a639b700 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fad980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f999c009f40 tx=0x7f999c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=862b4ce7f0e1483b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99a6b9c700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f998c0a07b0 0x7f998c0a2c70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f999800c6f0 tx=0x7f9998000f40 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f9994bfe700 1 -- 172.21.15.3:0/3498744564 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c004310 con 0x557d11fa40b0 2023-09-10T01:07:30.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f9994bfe700 1 -- 172.21.15.3:0/3498744564 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f999c004470 con 0x557d11fa40b0 2023-09-10T01:07:30.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f9994bfe700 1 -- 172.21.15.3:0/3498744564 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c019850 con 0x557d11fa40b0 2023-09-10T01:07:30.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99bc015b80 1 -- 172.21.15.3:0/3498744564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 msgr2=0x557d11fad980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99bc015b80 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fad980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f999c009f40 tx=0x7f999c011720 comp rx=0 tx=0).stop 2023-09-10T01:07:30.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99bc015b80 1 -- 172.21.15.3:0/3498744564 shutdown_connections 2023-09-10T01:07:30.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99bc015b80 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d11fb02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99bc015b80 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fa3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99bc015b80 1 --2- 172.21.15.3:0/3498744564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fad980 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.399+0000 7f99bc015b80 1 -- 172.21.15.3:0/3498744564 >> 172.21.15.3:0/3498744564 conn(0x557d11f18110 msgr2=0x557d11f9ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.400+0000 7f99bc015b80 1 -- 172.21.15.3:0/3498744564 shutdown_connections 2023-09-10T01:07:30.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.400+0000 7f99bc015b80 1 -- 172.21.15.3:0/3498744564 wait complete. 2023-09-10T01:07:30.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.400+0000 7f99bc015b80 1 -- start start 2023-09-10T01:07:30.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.400+0000 7f99bc015b80 1 Processor -- start 2023-09-10T01:07:30.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa3790 0x557d11fad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11faaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fadec0 0x557d11faafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99bc015b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11e93bf0 con 0x557d11fadec0 2023-09-10T01:07:30.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99bc015b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11e938f0 con 0x557d11fa3790 2023-09-10T01:07:30.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99bc015b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11d779a0 con 0x557d11fa40b0 2023-09-10T01:07:30.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99bc015b80 10 client.0 _ll_register_callbacks cb 0x557d11dddb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:30.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99a639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11faaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99a639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11faaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46020/0 (socket says 172.21.15.3:46020) 2023-09-10T01:07:30.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99a639b700 1 -- 172.21.15.3:0/1056727718 learned_addr learned my addr 172.21.15.3:0/1056727718 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa3790 0x557d11fad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.401+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fadec0 0x557d11faafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99a639b700 1 -- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa3790 msgr2=0x557d11fad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99a639b700 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa3790 0x557d11fad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99a639b700 1 -- 172.21.15.3:0/1056727718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fadec0 msgr2=0x557d11faafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99a639b700 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fadec0 0x557d11faafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99a639b700 1 -- 172.21.15.3:0/1056727718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c009ba0 con 0x557d11fa40b0 2023-09-10T01:07:30.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99a639b700 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11faaa40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f999c000c00 tx=0x7f999c004660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b6944c76aa0a7ff server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f9989ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:30.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f9989ffb700 1 -- 172.21.15.3:0/1056727718 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c003a10 con 0x557d11fa40b0 2023-09-10T01:07:30.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d11e4d570 con 0x557d11fa40b0 2023-09-10T01:07:30.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f9989ffb700 1 -- 172.21.15.3:0/1056727718 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f999c003b70 con 0x557d11fa40b0 2023-09-10T01:07:30.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f9989ffb700 1 -- 172.21.15.3:0/1056727718 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f999c003d30 con 0x557d11fa40b0 2023-09-10T01:07:30.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.402+0000 7f99bc015b80 10 client.26872 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:30.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557d11d0ace0 con 0x557d11fa40b0 2023-09-10T01:07:30.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f9989ffb700 1 -- 172.21.15.3:0/1056727718 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d11e4d570 con 0x557d11fa40b0 2023-09-10T01:07:30.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f9989ffb700 1 -- 172.21.15.3:0/1056727718 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f999c01a4d0 con 0x557d11fa40b0 2023-09-10T01:07:30.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99bc015b80 10 client.26872 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:30.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99bc015b80 10 client.26872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:30.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99bc015b80 20 client.26872 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:30.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99bc015b80 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d11fbd850 0x557d11fbfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99bc015b80 4 client.26872 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:30.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d11f0a920 con 0x557d11fbd850 2023-09-10T01:07:30.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.403+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d11fbd850 0x557d11fbfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.404+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d11fbd850 0x557d11fbfc70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2fd4641582d9ed1 server_cookie=7080d4f2f237a2e3 in_seq=0 out_seq=0 2023-09-10T01:07:30.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.404+0000 7f9989ffb700 10 client.26872 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f9989ffb700 1 -- 172.21.15.3:0/1056727718 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557d11f0a920 con 0x557d11fbd850 2023-09-10T01:07:30.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f9989ffb700 10 client.26872 handle_command_reply: tid=0 2023-09-10T01:07:30.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 client.26872 shutdown 2023-09-10T01:07:30.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 msgr2=0x557d11faaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11faaa40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f999c000c00 tx=0x7f999c004660 comp rx=0 tx=0).stop 2023-09-10T01:07:30.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 shutdown_connections 2023-09-10T01:07:30.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d11fbd850 0x557d11fbfc70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fadec0 0x557d11faafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa3790 0x557d11fad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 --2- 172.21.15.3:0/1056727718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11faaa40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f9989ffb700 0 client.26872 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 >> 172.21.15.3:0/1056727718 conn(0x557d11f18110 msgr2=0x557d11f9e860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 shutdown_connections 2023-09-10T01:07:30.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 1 -- 172.21.15.3:0/1056727718 wait complete. 2023-09-10T01:07:30.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.405+0000 7f99bc015b80 20 client.26872 trim_cache size 0 max 16384 2023-09-10T01:07:30.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 Processor -- start 2023-09-10T01:07:30.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 -- start start 2023-09-10T01:07:30.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11faa200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa40b0 0x557d11faa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fadec0 0x557d11fa7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11fa56c0 con 0x557d11fa3790 2023-09-10T01:07:30.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11fa8fa0 con 0x557d11fa40b0 2023-09-10T01:07:30.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99bc015b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d11f9da20 con 0x557d11fadec0 2023-09-10T01:07:30.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99a639b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa40b0 0x557d11faa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99a5b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11faa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99a639b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa40b0 0x557d11faa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44624/0 (socket says 172.21.15.3:44624) 2023-09-10T01:07:30.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99a639b700 1 -- 172.21.15.3:0/592187931 learned_addr learned my addr 172.21.15.3:0/592187931 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.424+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fadec0 0x557d11fa7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99a639b700 1 -- 172.21.15.3:0/592187931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fadec0 msgr2=0x557d11fa7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99a639b700 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fadec0 0x557d11fa7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99a639b700 1 -- 172.21.15.3:0/592187931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 msgr2=0x557d11faa200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99a639b700 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11faa200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99a639b700 1 -- 172.21.15.3:0/592187931 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d11d482d0 con 0x557d11fa40b0 2023-09-10T01:07:30.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99a639b700 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa40b0 0x557d11faa740 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f999c009e90 tx=0x7f999c005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6304d3cf17c9d4a6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99953ff700 1 -- 172.21.15.3:0/592187931 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c003680 con 0x557d11fa40b0 2023-09-10T01:07:30.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99953ff700 1 -- 172.21.15.3:0/592187931 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f999c003ed0 con 0x557d11fa40b0 2023-09-10T01:07:30.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.425+0000 7f99953ff700 1 -- 172.21.15.3:0/592187931 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c0194a0 con 0x557d11fa40b0 2023-09-10T01:07:30.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 -- 172.21.15.3:0/592187931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa40b0 msgr2=0x557d11faa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa40b0 0x557d11faa740 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f999c009e90 tx=0x7f999c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 -- 172.21.15.3:0/592187931 shutdown_connections 2023-09-10T01:07:30.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fadec0 0x557d11fa7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11faa200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 --2- 172.21.15.3:0/592187931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fa40b0 0x557d11faa740 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 -- 172.21.15.3:0/592187931 >> 172.21.15.3:0/592187931 conn(0x557d11f18110 msgr2=0x557d11fa5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 -- 172.21.15.3:0/592187931 shutdown_connections 2023-09-10T01:07:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 -- 172.21.15.3:0/592187931 wait complete. 2023-09-10T01:07:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.426+0000 7f99bc015b80 1 -- start start 2023-09-10T01:07:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 1 Processor -- start 2023-09-10T01:07:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fb3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fb3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d1200b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d1200bbd0 con 0x557d11fa3790 2023-09-10T01:07:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d1200bd10 con 0x557d11fadec0 2023-09-10T01:07:30.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d1200bed0 con 0x557d11fa40b0 2023-09-10T01:07:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99bc015b80 10 client.0 _ll_register_callbacks cb 0x7f99980097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99a639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fb3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99a5b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fb3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99a639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fb3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46042/0 (socket says 172.21.15.3:46042) 2023-09-10T01:07:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99a639b700 1 -- 172.21.15.3:0/2788137355 learned_addr learned my addr 172.21.15.3:0/2788137355 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.427+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d1200b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99a639b700 1 -- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 msgr2=0x557d1200b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99a639b700 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d1200b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99a639b700 1 -- 172.21.15.3:0/2788137355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 msgr2=0x557d11fb3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99a639b700 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fb3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99a639b700 1 -- 172.21.15.3:0/2788137355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d11d0ace0 con 0x557d11fa40b0 2023-09-10T01:07:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fb3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99a639b700 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fb3ec0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f999c01a9a0 tx=0x7f999c009990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aebd3e7559aceee5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f9989ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f9989ffb700 1 -- 172.21.15.3:0/2788137355 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c01aa70 con 0x557d11fa40b0 2023-09-10T01:07:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d11eee9a0 con 0x557d11fa40b0 2023-09-10T01:07:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.428+0000 7f9989ffb700 1 -- 172.21.15.3:0/2788137355 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f999c000e40 con 0x557d11fa40b0 2023-09-10T01:07:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f9989ffb700 1 -- 172.21.15.3:0/2788137355 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f999c01abd0 con 0x557d11fa40b0 2023-09-10T01:07:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 10 client.26878 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557d11d20330 con 0x557d11fa40b0 2023-09-10T01:07:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f9989ffb700 1 -- 172.21.15.3:0/2788137355 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d11eee9a0 con 0x557d11fa40b0 2023-09-10T01:07:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f9989ffb700 1 -- 172.21.15.3:0/2788137355 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f999c005370 con 0x557d11fa40b0 2023-09-10T01:07:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 10 client.26878 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 10 client.26878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 20 client.26878 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d12013260 0x557d12015650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 4 client.26878 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:07:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557d11eaf130 con 0x557d12013260 2023-09-10T01:07:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.429+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d12013260 0x557d12015650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.430+0000 7f99a5b9a700 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d12013260 0x557d12015650 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a12b69afcda40dbd server_cookie=cca05ee136ecf91 in_seq=0 out_seq=0 2023-09-10T01:07:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.430+0000 7f9989ffb700 10 client.26878 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f9989ffb700 1 -- 172.21.15.3:0/2788137355 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557d11eaf130 con 0x557d12013260 2023-09-10T01:07:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f9989ffb700 10 client.26878 handle_command_reply: tid=0 2023-09-10T01:07:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 client.26878 shutdown 2023-09-10T01:07:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 msgr2=0x557d11fb3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fb3ec0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f999c01a9a0 tx=0x7f999c009990 comp rx=0 tx=0).stop 2023-09-10T01:07:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 shutdown_connections 2023-09-10T01:07:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557d12013260 0x557d12015650 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d11fadec0 0x557d1200b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f9989ffb700 0 client.26878 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d11fa3790 0x557d11fb3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 --2- 172.21.15.3:0/2788137355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557d11fa40b0 0x557d11fb3ec0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 >> 172.21.15.3:0/2788137355 conn(0x557d11f18110 msgr2=0x557d11fa2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 shutdown_connections 2023-09-10T01:07:30.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 1 -- 172.21.15.3:0/2788137355 wait complete. 2023-09-10T01:07:30.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.431+0000 7f99bc015b80 20 client.26878 trim_cache size 0 max 16384 2023-09-10T01:07:30.469 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:07:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:07:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "4531f59d-4cce-4609-b994-508bd3e9393a", 2023-09-10T01:07:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:07:30.470 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:07:30.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:30 smithi092 ceph-mon[134067]: pgmap v131: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 433 op/s 2023-09-10T01:07:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f998c0a07b0 msgr2=0x7f998c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f998c0a07b0 0x7f998c0a2c70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f999800c6f0 tx=0x7f9998000f40 comp rx=0 tx=0).stop 2023-09-10T01:07:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0106660 msgr2=0x7f99a0198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0106660 0x7f99a0198700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f999800efc0 tx=0x7f999800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 shutdown_connections 2023-09-10T01:07:30.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f998c0a07b0 0x7f998c0a2c70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99a0106660 0x7f99a0198700 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99a0100650 0x7f99a019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 --2- 172.21.15.3:0/4052185694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99a0101010 0x7f99a019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 >> 172.21.15.3:0/4052185694 conn(0x7f99a00fc140 msgr2=0x7f99a010ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 shutdown_connections 2023-09-10T01:07:30.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.432+0000 7f99a85ff700 1 -- 172.21.15.3:0/4052185694 wait complete. 2023-09-10T01:07:30.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:30 smithi003 ceph-mon[97654]: pgmap v131: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 433 op/s 2023-09-10T01:07:30.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:07:30.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 -- 172.21.15.3:0/2746399674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100640 msgr2=0x7f14ac100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 --2- 172.21.15.3:0/2746399674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100640 0x7f14ac100ac0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1498009990 tx=0x7f1498009ca0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 -- 172.21.15.3:0/2746399674 shutdown_connections 2023-09-10T01:07:30.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 --2- 172.21.15.3:0/2746399674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac106650 0x7f14ac106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 --2- 172.21.15.3:0/2746399674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac101000 0x7f14ac10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 --2- 172.21.15.3:0/2746399674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac100640 0x7f14ac100ac0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 -- 172.21.15.3:0/2746399674 >> 172.21.15.3:0/2746399674 conn(0x7f14ac0fc170 msgr2=0x7f14ac0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.693+0000 7f14b31ff700 1 -- 172.21.15.3:0/2746399674 shutdown_connections 2023-09-10T01:07:30.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.694+0000 7f14b31ff700 1 -- 172.21.15.3:0/2746399674 wait complete. 2023-09-10T01:07:30.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.694+0000 7f14b31ff700 1 Processor -- start 2023-09-10T01:07:30.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b31ff700 1 -- start start 2023-09-10T01:07:30.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b31ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac100640 0x7f14ac196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b31ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac101000 0x7f14ac196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b31ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac106650 0x7f14ac19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b31ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac19a600 con 0x7f14ac101000 2023-09-10T01:07:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b31ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac19a770 con 0x7f14ac106650 2023-09-10T01:07:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b31ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac19a930 con 0x7f14ac100640 2023-09-10T01:07:30.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b179c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac106650 0x7f14ac19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14abfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac101000 0x7f14ac196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac100640 0x7f14ac196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b179c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac106650 0x7f14ac19a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44658/0 (socket says 172.21.15.3:44658) 2023-09-10T01:07:30.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.695+0000 7f14b179c700 1 -- 172.21.15.3:0/81375839 learned_addr learned my addr 172.21.15.3:0/81375839 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b0f9b700 1 -- 172.21.15.3:0/81375839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac106650 msgr2=0x7f14ac19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac106650 0x7f14ac19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b0f9b700 1 -- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac101000 msgr2=0x7f14ac196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac101000 0x7f14ac196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b0f9b700 1 -- 172.21.15.3:0/81375839 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980095f0 con 0x7f14ac100640 2023-09-10T01:07:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14abfff700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac101000 0x7f14ac196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:30.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac100640 0x7f14ac196710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f14a000ef10 tx=0x7f14a000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14a9ffb700 1 -- 172.21.15.3:0/81375839 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a000c9c0 con 0x7f14ac100640 2023-09-10T01:07:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac19abb0 con 0x7f14ac100640 2023-09-10T01:07:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14a9ffb700 1 -- 172.21.15.3:0/81375839 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a000f040 con 0x7f14ac100640 2023-09-10T01:07:30.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14a9ffb700 1 -- 172.21.15.3:0/81375839 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0014480 con 0x7f14ac100640 2023-09-10T01:07:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.696+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac1a3570 con 0x7f14ac100640 2023-09-10T01:07:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.699+0000 7f14a9ffb700 1 -- 172.21.15.3:0/81375839 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f14a000cb20 con 0x7f14ac100640 2023-09-10T01:07:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 Processor -- start 2023-09-10T01:07:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 -- start start 2023-09-10T01:07:30.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14a9ffb700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f149c0a07b0 0x7f149c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937ed900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56049373c730 con 0x5604937ede40 2023-09-10T01:07:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56049373a050 con 0x5604937e3710 2023-09-10T01:07:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560493735940 con 0x5604937e4030 2023-09-10T01:07:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14a9ffb700 1 -- 172.21.15.3:0/81375839 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f14a00cc5f0 con 0x7f14ac100640 2023-09-10T01:07:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14abfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937e3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44664/0 (socket says 172.21.15.3:44664) 2023-09-10T01:07:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.701+0000 7f14b0f9b700 1 -- 172.21.15.3:0/4186023660 learned_addr learned my addr 172.21.15.3:0/4186023660 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b179c700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f149c0a07b0 0x7f149c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b179c700 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b0f9b700 1 -- 172.21.15.3:0/4186023660 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 msgr2=0x5604937ed900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937ed900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b0f9b700 1 -- 172.21.15.3:0/4186023660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 msgr2=0x5604937f0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b0f9b700 1 -- 172.21.15.3:0/4186023660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560493561de0 con 0x5604937e3710 2023-09-10T01:07:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937e3af0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f14a0096120 tx=0x7f14a0099480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4172b1cb1c238f69 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f14b179c700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f149c0a07b0 0x7f149c0a2c70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14a40098a0 tx=0x7f14a40095c0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f1496ffd700 1 -- 172.21.15.3:0/4186023660 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0099db0 con 0x5604937e3710 2023-09-10T01:07:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f1496ffd700 1 -- 172.21.15.3:0/4186023660 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a00cc020 con 0x5604937e3710 2023-09-10T01:07:30.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.702+0000 7f1496ffd700 1 -- 172.21.15.3:0/4186023660 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0007690 con 0x5604937e3710 2023-09-10T01:07:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/4186023660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 msgr2=0x5604937e3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937e3af0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f14a0096120 tx=0x7f14a0099480 comp rx=0 tx=0).stop 2023-09-10T01:07:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/4186023660 shutdown_connections 2023-09-10T01:07:30.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937e3af0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/4186023660 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937ed900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/4186023660 >> 172.21.15.3:0/4186023660 conn(0x560493758090 msgr2=0x5604937debf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/4186023660 shutdown_connections 2023-09-10T01:07:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.703+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/4186023660 wait complete. 2023-09-10T01:07:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.704+0000 7f14c6aa9b80 1 -- start start 2023-09-10T01:07:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.704+0000 7f14c6aa9b80 1 Processor -- start 2023-09-10T01:07:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.704+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e3710 0x5604937ed7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.704+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e4030 0x5604937eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.704+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937eb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604936d3b70 con 0x5604937ede40 2023-09-10T01:07:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604936d3870 con 0x5604937e4030 2023-09-10T01:07:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604935b7920 con 0x5604937e3710 2023-09-10T01:07:30.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14c6aa9b80 10 client.0 _ll_register_callbacks cb 0x56049361da80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e3710 0x5604937ed7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14abfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e4030 0x5604937eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e3710 0x5604937ed7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46092/0 (socket says 172.21.15.3:46092) 2023-09-10T01:07:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 -- 172.21.15.3:0/40101426 learned_addr learned my addr 172.21.15.3:0/40101426 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b179c700 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 -- 172.21.15.3:0/40101426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e4030 msgr2=0x5604937eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e4030 0x5604937eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 -- 172.21.15.3:0/40101426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 msgr2=0x5604937eb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937eb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 -- 172.21.15.3:0/40101426 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac1a3570 con 0x5604937e3710 2023-09-10T01:07:30.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e3710 0x5604937ed7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f14a0000f80 tx=0x7f14a0099130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=722612477a5d9c80 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f148bfff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:07:30.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.705+0000 7f148bfff700 1 -- 172.21.15.3:0/40101426 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a009d4e0 con 0x5604937e3710 2023-09-10T01:07:30.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f148bfff700 1 -- 172.21.15.3:0/40101426 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f14a009d170 con 0x5604937e3710 2023-09-10T01:07:30.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56049368d4f0 con 0x5604937e3710 2023-09-10T01:07:30.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f148bfff700 1 -- 172.21.15.3:0/40101426 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14a00cc190 con 0x5604937e3710 2023-09-10T01:07:30.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 10 client.26890 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:30.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56049354ac60 con 0x5604937e3710 2023-09-10T01:07:30.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f148bfff700 1 -- 172.21.15.3:0/40101426 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56049368d4f0 con 0x5604937e3710 2023-09-10T01:07:30.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f148bfff700 1 -- 172.21.15.3:0/40101426 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f14a0098ba0 con 0x5604937e3710 2023-09-10T01:07:30.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 10 client.26890 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:30.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 10 client.26890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:30.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 20 client.26890 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:30.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604937fd440 0x5604937ff830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 4 client.26890 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:30.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.706+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56049374a8a0 con 0x5604937fd440 2023-09-10T01:07:30.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.707+0000 7f14abfff700 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604937fd440 0x5604937ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.707+0000 7f14abfff700 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604937fd440 0x5604937ff830 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e36330d77cae202c server_cookie=ca62a3b07a4a029b in_seq=0 out_seq=0 2023-09-10T01:07:30.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.707+0000 7f148bfff700 10 client.26890 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f148bfff700 1 -- 172.21.15.3:0/40101426 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56049374a8a0 con 0x5604937fd440 2023-09-10T01:07:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f148bfff700 10 client.26890 handle_command_reply: tid=0 2023-09-10T01:07:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 client.26890 shutdown 2023-09-10T01:07:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e3710 msgr2=0x5604937ed7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e3710 0x5604937ed7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f14a0000f80 tx=0x7f14a0099130 comp rx=0 tx=0).stop 2023-09-10T01:07:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 shutdown_connections 2023-09-10T01:07:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604937fd440 0x5604937ff830 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e3710 0x5604937ed7a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/40101426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e4030 0x5604937eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f148bfff700 0 client.26890 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 >> 172.21.15.3:0/40101426 conn(0x560493758090 msgr2=0x5604937de780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 shutdown_connections 2023-09-10T01:07:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/40101426 wait complete. 2023-09-10T01:07:30.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.708+0000 7f14c6aa9b80 20 client.26890 trim_cache size 0 max 16384 2023-09-10T01:07:30.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 Processor -- start 2023-09-10T01:07:30.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 -- start start 2023-09-10T01:07:30.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937dd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604937ff830 con 0x5604937ede40 2023-09-10T01:07:30.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604937ff9a0 con 0x5604937e3710 2023-09-10T01:07:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.726+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604937fa140 con 0x5604937e4030 2023-09-10T01:07:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14abfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14abfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46106/0 (socket says 172.21.15.3:46106) 2023-09-10T01:07:30.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14abfff700 1 -- 172.21.15.3:0/2062873954 learned_addr learned my addr 172.21.15.3:0/2062873954 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b179c700 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937dd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b0f9b700 1 -- 172.21.15.3:0/2062873954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 msgr2=0x5604937f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b0f9b700 1 -- 172.21.15.3:0/2062873954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 msgr2=0x5604937dd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937dd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b0f9b700 1 -- 172.21.15.3:0/2062873954 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560493588250 con 0x5604937e3710 2023-09-10T01:07:30.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14abfff700 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:30.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f49f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f14a0096120 tx=0x7f14a00078a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=780bbeb0b5fb758c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.727+0000 7f14977fe700 1 -- 172.21.15.3:0/2062873954 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0099480 con 0x5604937e3710 2023-09-10T01:07:30.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14977fe700 1 -- 172.21.15.3:0/2062873954 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a009d4e0 con 0x5604937e3710 2023-09-10T01:07:30.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14977fe700 1 -- 172.21.15.3:0/2062873954 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a00cc190 con 0x5604937e3710 2023-09-10T01:07:30.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/2062873954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 msgr2=0x5604937f49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f49f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f14a0096120 tx=0x7f14a00078a0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/2062873954 shutdown_connections 2023-09-10T01:07:30.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937dd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f49f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/2062873954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/2062873954 >> 172.21.15.3:0/2062873954 conn(0x560493758090 msgr2=0x560493804150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/2062873954 shutdown_connections 2023-09-10T01:07:30.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.728+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/2062873954 wait complete. 2023-09-10T01:07:30.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 -- start start 2023-09-10T01:07:30.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 Processor -- start 2023-09-10T01:07:30.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604937f4ec0 con 0x5604937ede40 2023-09-10T01:07:30.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604937f3360 con 0x5604937e3710 2023-09-10T01:07:30.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604937f34d0 con 0x5604937e4030 2023-09-10T01:07:30.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.729+0000 7f14c6aa9b80 10 client.0 _ll_register_callbacks cb 0x7f14a4005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:30.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44696/0 (socket says 172.21.15.3:44696) 2023-09-10T01:07:30.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 -- 172.21.15.3:0/699719580 learned_addr learned my addr 172.21.15.3:0/699719580 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:30.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b179c700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14abfff700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 -- 172.21.15.3:0/699719580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 msgr2=0x5604937f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 -- 172.21.15.3:0/699719580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 msgr2=0x5604937f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:30.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 -- 172.21.15.3:0/699719580 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac10de10 con 0x5604937e3710 2023-09-10T01:07:30.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b179c700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:07:30.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14abfff700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:07:30.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14b0f9b700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f45f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f14a0097a00 tx=0x7f14a0007420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e0fd6ae0b701ce7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:30.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f148bfff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:30.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f148bfff700 1 -- 172.21.15.3:0/699719580 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0099480 con 0x5604937e3710 2023-09-10T01:07:30.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f148bfff700 1 -- 172.21.15.3:0/699719580 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f14a00cc190 con 0x5604937e3710 2023-09-10T01:07:30.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.730+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14a00cc020 con 0x5604937e3710 2023-09-10T01:07:30.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f148bfff700 1 -- 172.21.15.3:0/699719580 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14a00cc350 con 0x5604937e3710 2023-09-10T01:07:30.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 10 client.26972 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:30.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56049354ac60 con 0x5604937e3710 2023-09-10T01:07:30.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f148bfff700 1 -- 172.21.15.3:0/699719580 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a00cc020 con 0x5604937e3710 2023-09-10T01:07:30.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f148bfff700 1 -- 172.21.15.3:0/699719580 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f14a00a13b0 con 0x5604937e3710 2023-09-10T01:07:30.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 10 client.26972 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:30.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 10 client.26972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:30.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 20 client.26972 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:30.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604938564b0 0x5604938588a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:30.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 4 client.26972 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:07:30.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56049372e920 con 0x5604938564b0 2023-09-10T01:07:30.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.731+0000 7f14abfff700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604938564b0 0x5604938588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:30.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.732+0000 7f148bfff700 10 client.26972 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.732+0000 7f14abfff700 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604938564b0 0x5604938588a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0123ea01b55b23a server_cookie=c3459bee10ed3596 in_seq=0 out_seq=0 2023-09-10T01:07:30.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.732+0000 7f148bfff700 1 -- 172.21.15.3:0/699719580 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56049372e920 con 0x5604938564b0 2023-09-10T01:07:30.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.732+0000 7f148bfff700 10 client.26972 handle_command_reply: tid=0 2023-09-10T01:07:30.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 client.26972 shutdown 2023-09-10T01:07:30.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 msgr2=0x5604937f45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f45f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f14a0097a00 tx=0x7f14a0007420 comp rx=0 tx=0).stop 2023-09-10T01:07:30.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 shutdown_connections 2023-09-10T01:07:30.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604938564b0 0x5604938588a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604937ede40 0x5604937f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604937e3710 0x5604937f45f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 --2- 172.21.15.3:0/699719580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604937e4030 0x5604937f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f148bfff700 0 client.26972 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:30.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 >> 172.21.15.3:0/699719580 conn(0x560493758090 msgr2=0x5604937f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 shutdown_connections 2023-09-10T01:07:30.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 1 -- 172.21.15.3:0/699719580 wait complete. 2023-09-10T01:07:30.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.733+0000 7f14c6aa9b80 20 client.26972 trim_cache size 0 max 16384 2023-09-10T01:07:30.753 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:07:30.753 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:07:30.753 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:07:30.753 INFO:teuthology.orchestra.run.smithi003.stdout: "4531f59d-4cce-4609-b994-508bd3e9393a": { 2023-09-10T01:07:30.754 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:07:30.754 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "4531f59d-4cce-4609-b994-508bd3e9393a", 2023-09-10T01:07:30.754 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:07:30.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:07:30.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:07:30.755 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:07:30.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f149c0a07b0 msgr2=0x7f149c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f149c0a07b0 0x7f149c0a2c70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14a40098a0 tx=0x7f14a40095c0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac100640 msgr2=0x7f14ac196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:30.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac100640 0x7f14ac196710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f14a000ef10 tx=0x7f14a000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 shutdown_connections 2023-09-10T01:07:30.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f149c0a07b0 0x7f149c0a2c70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14ac100640 0x7f14ac196710 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14ac106650 0x7f14ac19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 --2- 172.21.15.3:0/81375839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14ac101000 0x7f14ac196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:30.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 >> 172.21.15.3:0/81375839 conn(0x7f14ac0fc170 msgr2=0x7f14ac0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:30.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 shutdown_connections 2023-09-10T01:07:30.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:07:30.734+0000 7f14b31ff700 1 -- 172.21.15.3:0/81375839 wait complete. 2023-09-10T01:07:30.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 345772 9.37 MB/sec execute 72 sec latency 17.334 ms 2023-09-10T01:07:30.759 INFO:tasks.cephfs.filesystem:scrub status for tag:4531f59d-4cce-4609-b994-508bd3e9393a - {'path': '/', 'tag': '4531f59d-4cce-4609-b994-508bd3e9393a', 'options': 'recursive,force'} 2023-09-10T01:07:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:31 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:07:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:31 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:31 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:07:31.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:31 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:07:31.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:31 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:31 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:31 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:07:31.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:31 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:31 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:07:31.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:31 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:07:31.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:31 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:31 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:31 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:07:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:31 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:31 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:07:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:31 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:07:31.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:31 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:31 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:31.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347996 9.43 MB/sec execute 73 sec latency 10.674 ms 2023-09-10T01:07:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:32 smithi136 ceph-mon[133708]: pgmap v132: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:07:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:32 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:07:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:32 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:32 smithi092 ceph-mon[134067]: pgmap v132: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:07:32.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:32 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:07:32.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:32 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:32.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:32 smithi003 ceph-mon[97654]: pgmap v132: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:07:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:32 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:07:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:32 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:07:32.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 349421 9.38 MB/sec execute 74 sec latency 16.531 ms 2023-09-10T01:07:33.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 351581 9.44 MB/sec execute 75 sec latency 10.466 ms 2023-09-10T01:07:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:34 smithi092 ceph-mon[134067]: pgmap v133: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:07:34.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:34 smithi003 ceph-mon[97654]: pgmap v133: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:07:34.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:34 smithi136 ceph-mon[133708]: pgmap v133: 129 pgs: 129 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 267 op/s 2023-09-10T01:07:34.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 352688 9.38 MB/sec execute 76 sec latency 180.800 ms 2023-09-10T01:07:35.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 355088 9.44 MB/sec execute 77 sec latency 14.502 ms 2023-09-10T01:07:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:36 smithi092 ceph-mon[134067]: pgmap v134: 129 pgs: 129 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 388 op/s 2023-09-10T01:07:36.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:36 smithi003 ceph-mon[97654]: pgmap v134: 129 pgs: 129 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 388 op/s 2023-09-10T01:07:36.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:36 smithi136 ceph-mon[133708]: pgmap v134: 129 pgs: 129 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 388 op/s 2023-09-10T01:07:36.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 356735 9.39 MB/sec execute 78 sec latency 10.779 ms 2023-09-10T01:07:37.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 358664 9.44 MB/sec execute 79 sec latency 60.351 ms 2023-09-10T01:07:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:38 smithi092 ceph-mon[134067]: pgmap v135: 129 pgs: 129 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 284 op/s 2023-09-10T01:07:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:38 smithi003 ceph-mon[97654]: pgmap v135: 129 pgs: 129 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 284 op/s 2023-09-10T01:07:38.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:38 smithi136 ceph-mon[133708]: pgmap v135: 129 pgs: 129 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 284 op/s 2023-09-10T01:07:38.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 360450 9.40 MB/sec execute 80 sec latency 14.010 ms 2023-09-10T01:07:39.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:39 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:07:39.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:39 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:07:39.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:39 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:07:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:39 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:07:39.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:39 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:07:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:39 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:07:39.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 362163 9.45 MB/sec execute 81 sec latency 64.057 ms 2023-09-10T01:07:40.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:40 smithi092 ceph-mon[134067]: pgmap v136: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 384 op/s 2023-09-10T01:07:40.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:40 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:07:40.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:40 smithi003 ceph-mon[97654]: pgmap v136: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 384 op/s 2023-09-10T01:07:40.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:40 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:07:40.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:40 smithi136 ceph-mon[133708]: pgmap v136: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 384 op/s 2023-09-10T01:07:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:40 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:07:40.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 363882 9.40 MB/sec execute 82 sec latency 15.567 ms 2023-09-10T01:07:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 -- 172.21.15.3:0/4147826385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724100f80 msgr2=0x7f472410d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 --2- 172.21.15.3:0/4147826385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724100f80 0x7f472410d440 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f471c009fc0 tx=0x7f471c00b330 comp rx=0 tx=0).stop 2023-09-10T01:07:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 -- 172.21.15.3:0/4147826385 shutdown_connections 2023-09-10T01:07:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 --2- 172.21.15.3:0/4147826385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724106680 0x7f4724106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 --2- 172.21.15.3:0/4147826385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724100f80 0x7f472410d440 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 --2- 172.21.15.3:0/4147826385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47241005c0 0x7f4724100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 -- 172.21.15.3:0/4147826385 >> 172.21.15.3:0/4147826385 conn(0x7f47240fc0b0 msgr2=0x7f47240fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 -- 172.21.15.3:0/4147826385 shutdown_connections 2023-09-10T01:07:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.285+0000 7f472a3ff700 1 -- 172.21.15.3:0/4147826385 wait complete. 2023-09-10T01:07:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.286+0000 7f472a3ff700 1 Processor -- start 2023-09-10T01:07:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.287+0000 7f472a3ff700 1 -- start start 2023-09-10T01:07:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.287+0000 7f472a3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47241005c0 0x7f4724196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.287+0000 7f472a3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724100f80 0x7f4724196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.287+0000 7f472a3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724106680 0x7f4724199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.287+0000 7f472a3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472419a5e0 con 0x7f4724106680 2023-09-10T01:07:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.287+0000 7f472a3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472419a750 con 0x7f47241005c0 2023-09-10T01:07:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.287+0000 7f472a3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472419a910 con 0x7f4724100f80 2023-09-10T01:07:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724100f80 0x7f4724196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724100f80 0x7f4724196c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49972/0 (socket says 172.21.15.3:49972) 2023-09-10T01:07:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f4723fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47241005c0 0x7f4724196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 -- 172.21.15.3:0/3774734415 learned_addr learned my addr 172.21.15.3:0/3774734415 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f472899c700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724106680 0x7f4724199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 -- 172.21.15.3:0/3774734415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47241005c0 msgr2=0x7f4724196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47241005c0 0x7f4724196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 -- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724106680 msgr2=0x7f4724199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724106680 0x7f4724199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 -- 172.21.15.3:0/3774734415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c009c40 con 0x7f4724100f80 2023-09-10T01:07:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f4723fff700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47241005c0 0x7f4724196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f472899c700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724106680 0x7f4724199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47237fe700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724100f80 0x7f4724196c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f471800e9a0 tx=0x7f471800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.288+0000 7f47217fa700 1 -- 172.21.15.3:0/3774734415 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718009790 con 0x7f4724100f80 2023-09-10T01:07:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.289+0000 7f47217fa700 1 -- 172.21.15.3:0/3774734415 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f471800ce80 con 0x7f4724100f80 2023-09-10T01:07:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.289+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472419abf0 con 0x7f4724100f80 2023-09-10T01:07:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.289+0000 7f47217fa700 1 -- 172.21.15.3:0/3774734415 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718018680 con 0x7f4724100f80 2023-09-10T01:07:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.289+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47241a3550 con 0x7f4724100f80 2023-09-10T01:07:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.292+0000 7f47217fa700 1 -- 172.21.15.3:0/3774734415 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f4718010430 con 0x7f4724100f80 2023-09-10T01:07:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.293+0000 7f473dc53b80 1 Processor -- start 2023-09-10T01:07:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.293+0000 7f473dc53b80 1 -- start start 2023-09-10T01:07:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b44990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b3eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b4b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f473dc53b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856a936d0 con 0x559856b445b0 2023-09-10T01:07:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f473dc53b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856a90ff0 con 0x559856b3f0d0 2023-09-10T01:07:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f473dc53b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856a8c8e0 con 0x559856b3e710 2023-09-10T01:07:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f4723fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b3eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f47237fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b4b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f4723fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b3eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49982/0 (socket says 172.21.15.3:49982) 2023-09-10T01:07:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f4723fff700 1 -- 172.21.15.3:0/338156324 learned_addr learned my addr 172.21.15.3:0/338156324 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f47217fa700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f470c0a07b0 0x7f470c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f472899c700 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b44990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f4723fff700 1 -- 172.21.15.3:0/338156324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 msgr2=0x559856b4b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f4723fff700 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b4b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f4723fff700 1 -- 172.21.15.3:0/338156324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 msgr2=0x559856b44990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f4723fff700 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b44990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.294+0000 7f472899c700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f470c0a07b0 0x7f470c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.295+0000 7f47217fa700 1 -- 172.21.15.3:0/3774734415 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4718014070 con 0x7f4724100f80 2023-09-10T01:07:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.295+0000 7f4723fff700 1 -- 172.21.15.3:0/338156324 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598568b8d80 con 0x559856b3e710 2023-09-10T01:07:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.295+0000 7f4723fff700 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b3eb90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f47140115a0 tx=0x7f47140118b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=120c37b2f80906e8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.295+0000 7f470a7fc700 1 -- 172.21.15.3:0/338156324 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714004230 con 0x559856b3e710 2023-09-10T01:07:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.295+0000 7f470a7fc700 1 -- 172.21.15.3:0/338156324 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4714004390 con 0x559856b3e710 2023-09-10T01:07:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.295+0000 7f470a7fc700 1 -- 172.21.15.3:0/338156324 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714019890 con 0x559856b3e710 2023-09-10T01:07:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 -- 172.21.15.3:0/338156324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 msgr2=0x559856b3eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b3eb90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f47140115a0 tx=0x7f47140118b0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f472899c700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f470c0a07b0 0x7f470c0a2c70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f471c00b6c0 tx=0x7f471c00b330 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 -- 172.21.15.3:0/338156324 shutdown_connections 2023-09-10T01:07:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b4b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b44990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 --2- 172.21.15.3:0/338156324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b3eb90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 -- 172.21.15.3:0/338156324 >> 172.21.15.3:0/338156324 conn(0x559856aaf030 msgr2=0x559856b35560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 -- 172.21.15.3:0/338156324 shutdown_connections 2023-09-10T01:07:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.296+0000 7f473dc53b80 1 -- 172.21.15.3:0/338156324 wait complete. 2023-09-10T01:07:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 -- start start 2023-09-10T01:07:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 Processor -- start 2023-09-10T01:07:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b4b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3f0d0 0x559856b48430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856b489f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856a2ab10 con 0x559856b3f0d0 2023-09-10T01:07:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856a2a810 con 0x559856b445b0 2023-09-10T01:07:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55985690e8c0 con 0x559856b3e710 2023-09-10T01:07:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.297+0000 7f473dc53b80 10 client.0 _ll_register_callbacks cb 0x559856974a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856b489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f47237fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3f0d0 0x559856b48430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856b489f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48606/0 (socket says 172.21.15.3:48606) 2023-09-10T01:07:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 -- 172.21.15.3:0/3515639908 learned_addr learned my addr 172.21.15.3:0/3515639908 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f4723fff700 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b4b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 -- 172.21.15.3:0/3515639908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 msgr2=0x559856b4b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b4b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 -- 172.21.15.3:0/3515639908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3f0d0 msgr2=0x559856b48430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3f0d0 0x559856b48430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 -- 172.21.15.3:0/3515639908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4714009d90 con 0x559856b445b0 2023-09-10T01:07:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f47237fe700 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3f0d0 0x559856b48430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f472899c700 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856b489f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f471c003dd0 tx=0x7f471c009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb959e52a947bf22 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.298+0000 7f47037fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f47037fe700 1 -- 172.21.15.3:0/3515639908 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c00e040 con 0x559856b445b0 2023-09-10T01:07:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5598569e4490 con 0x559856b445b0 2023-09-10T01:07:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f47037fe700 1 -- 172.21.15.3:0/3515639908 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f471c028650 con 0x559856b445b0 2023-09-10T01:07:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f47037fe700 1 -- 172.21.15.3:0/3515639908 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f471c00ce90 con 0x559856b445b0 2023-09-10T01:07:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 10 client.26984 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5598568a1c00 con 0x559856b445b0 2023-09-10T01:07:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f47037fe700 1 -- 172.21.15.3:0/3515639908 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5598569e4490 con 0x559856b445b0 2023-09-10T01:07:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f47037fe700 1 -- 172.21.15.3:0/3515639908 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f471c025050 con 0x559856b445b0 2023-09-10T01:07:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 10 client.26984 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 10 client.26984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 20 client.26984 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856b58660 0x559856b5aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 4 client.26984 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.299+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559856aa1840 con 0x559856b58660 2023-09-10T01:07:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.300+0000 7f4723fff700 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856b58660 0x559856b5aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.300+0000 7f4723fff700 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856b58660 0x559856b5aa80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=352302d54692cc05 server_cookie=3909df268a0bf26f in_seq=0 out_seq=0 2023-09-10T01:07:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.300+0000 7f47037fe700 10 client.26984 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f47037fe700 1 -- 172.21.15.3:0/3515639908 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559856aa1840 con 0x559856b58660 2023-09-10T01:07:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f47037fe700 10 client.26984 handle_command_reply: tid=0 2023-09-10T01:07:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 client.26984 shutdown 2023-09-10T01:07:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 msgr2=0x559856b489f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856b489f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f471c003dd0 tx=0x7f471c009f30 comp rx=0 tx=0).stop 2023-09-10T01:07:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 shutdown_connections 2023-09-10T01:07:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856b58660 0x559856b5aa80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b4b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856b489f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 --2- 172.21.15.3:0/3515639908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3f0d0 0x559856b48430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f47037fe700 0 client.26984 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 >> 172.21.15.3:0/3515639908 conn(0x559856aaf030 msgr2=0x559856b347a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 shutdown_connections 2023-09-10T01:07:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 1 -- 172.21.15.3:0/3515639908 wait complete. 2023-09-10T01:07:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.301+0000 7f473dc53b80 20 client.26984 trim_cache size 0 max 16384 2023-09-10T01:07:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.316+0000 7f473dc53b80 1 Processor -- start 2023-09-10T01:07:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.316+0000 7f473dc53b80 1 -- start start 2023-09-10T01:07:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b40f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b42600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f473dc53b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856b44210 con 0x559856b445b0 2023-09-10T01:07:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f473dc53b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856b49a10 con 0x559856b3f0d0 2023-09-10T01:07:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f473dc53b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856b469f0 con 0x559856b3e710 2023-09-10T01:07:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f472899c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b42600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f47237fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f4723fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b40f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f472899c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b42600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57734/0 (socket says 172.21.15.3:57734) 2023-09-10T01:07:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f47237fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48616/0 (socket says 172.21.15.3:48616) 2023-09-10T01:07:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.317+0000 7f472899c700 1 -- 172.21.15.3:0/4219983725 learned_addr learned my addr 172.21.15.3:0/4219983725 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f47237fe700 1 -- 172.21.15.3:0/4219983725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 msgr2=0x559856b40f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f47237fe700 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b40f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f47237fe700 1 -- 172.21.15.3:0/4219983725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 msgr2=0x559856b42600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f47237fe700 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b42600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f47237fe700 1 -- 172.21.15.3:0/4219983725 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598568df1f0 con 0x559856b3f0d0 2023-09-10T01:07:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f472899c700 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b42600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f47237fe700 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b414b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f47180c9990 tx=0x7f47180995f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8471a7fef1d4601c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f470affd700 1 -- 172.21.15.3:0/4219983725 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718099df0 con 0x559856b3f0d0 2023-09-10T01:07:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f470affd700 1 -- 172.21.15.3:0/4219983725 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47180996f0 con 0x559856b3f0d0 2023-09-10T01:07:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.318+0000 7f470affd700 1 -- 172.21.15.3:0/4219983725 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471809d2f0 con 0x559856b3f0d0 2023-09-10T01:07:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 -- 172.21.15.3:0/4219983725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 msgr2=0x559856b414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b414b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f47180c9990 tx=0x7f47180995f0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 -- 172.21.15.3:0/4219983725 shutdown_connections 2023-09-10T01:07:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3e710 0x559856b40f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b445b0 0x559856b42600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4219983725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b3f0d0 0x559856b414b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 -- 172.21.15.3:0/4219983725 >> 172.21.15.3:0/4219983725 conn(0x559856aaf030 msgr2=0x559856b54090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 -- 172.21.15.3:0/4219983725 shutdown_connections 2023-09-10T01:07:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 -- 172.21.15.3:0/4219983725 wait complete. 2023-09-10T01:07:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 -- start start 2023-09-10T01:07:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.319+0000 7f473dc53b80 1 Processor -- start 2023-09-10T01:07:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3e710 0x559856b4eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3f0d0 0x559856b4f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f473dc53b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856bab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f473dc53b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856bab9d0 con 0x559856b3e710 2023-09-10T01:07:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f473dc53b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856babb10 con 0x559856b445b0 2023-09-10T01:07:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f473dc53b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559856babc50 con 0x559856b3f0d0 2023-09-10T01:07:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f473dc53b80 10 client.0 _ll_register_callbacks cb 0x7f471c003a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f4723fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3e710 0x559856b4eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f4723fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3e710 0x559856b4eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57750/0 (socket says 172.21.15.3:57750) 2023-09-10T01:07:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f4723fff700 1 -- 172.21.15.3:0/4026022790 learned_addr learned my addr 172.21.15.3:0/4026022790 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f47237fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3f0d0 0x559856b4f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.320+0000 7f472899c700 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856bab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f4723fff700 1 -- 172.21.15.3:0/4026022790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3f0d0 msgr2=0x559856b4f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f4723fff700 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3f0d0 0x559856b4f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f4723fff700 1 -- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 msgr2=0x559856bab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f4723fff700 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856bab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f4723fff700 1 -- 172.21.15.3:0/4026022790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598568a1c00 con 0x559856b3e710 2023-09-10T01:07:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f472899c700 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856bab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f4723fff700 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3e710 0x559856b4eee0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4714014040 tx=0x7f4714009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa63199a16c34922 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f47037fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:07:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f47037fe700 1 -- 172.21.15.3:0/4026022790 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714005040 con 0x559856b3e710 2023-09-10T01:07:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4714004390 con 0x559856b3e710 2023-09-10T01:07:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f47037fe700 1 -- 172.21.15.3:0/4026022790 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f47140051a0 con 0x559856b3e710 2023-09-10T01:07:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f47037fe700 1 -- 172.21.15.3:0/4026022790 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4714005360 con 0x559856b3e710 2023-09-10T01:07:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f473dc53b80 10 client.16923 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.321+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f47241a3550 con 0x559856b3e710 2023-09-10T01:07:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f47037fe700 1 -- 172.21.15.3:0/4026022790 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4714004390 con 0x559856b3e710 2023-09-10T01:07:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f47037fe700 1 -- 172.21.15.3:0/4026022790 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f47140115d0 con 0x559856b3e710 2023-09-10T01:07:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f473dc53b80 10 client.16923 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f473dc53b80 10 client.16923 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f473dc53b80 20 client.16923 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856bad330 0x559856baf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f473dc53b80 4 client.16923 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:07:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559856a858c0 con 0x559856bad330 2023-09-10T01:07:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.322+0000 7f47237fe700 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856bad330 0x559856baf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.323+0000 7f47237fe700 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856bad330 0x559856baf750 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94ec57647d0c14da server_cookie=987b74a4c0628ea6 in_seq=0 out_seq=0 2023-09-10T01:07:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.323+0000 7f47037fe700 10 client.16923 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f47037fe700 1 -- 172.21.15.3:0/4026022790 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559856a858c0 con 0x559856bad330 2023-09-10T01:07:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f47037fe700 10 client.16923 handle_command_reply: tid=0 2023-09-10T01:07:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 client.16923 shutdown 2023-09-10T01:07:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3e710 msgr2=0x559856b4eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3e710 0x559856b4eee0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4714014040 tx=0x7f4714009350 comp rx=0 tx=0).stop 2023-09-10T01:07:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 shutdown_connections 2023-09-10T01:07:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559856bad330 0x559856baf750 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559856b445b0 0x559856bab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559856b3e710 0x559856b4eee0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 --2- 172.21.15.3:0/4026022790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559856b3f0d0 0x559856b4f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f47037fe700 0 client.16923 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.440+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 >> 172.21.15.3:0/4026022790 conn(0x559856aaf030 msgr2=0x559856b42600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.441+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 shutdown_connections 2023-09-10T01:07:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.441+0000 7f473dc53b80 1 -- 172.21.15.3:0/4026022790 wait complete. 2023-09-10T01:07:41.445 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.441+0000 7f473dc53b80 20 client.16923 trim_cache size 0 max 16384 2023-09-10T01:07:41.445 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:07:41.445 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:07:41.446 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:07:41.446 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:07:41.446 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f470c0a07b0 msgr2=0x7f470c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f470c0a07b0 0x7f470c0a2c70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f471c00b6c0 tx=0x7f471c00b330 comp rx=0 tx=0).stop 2023-09-10T01:07:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724100f80 msgr2=0x7f4724196c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724100f80 0x7f4724196c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f471800e9a0 tx=0x7f471800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 shutdown_connections 2023-09-10T01:07:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f470c0a07b0 0x7f470c0a2c70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4724106680 0x7f4724199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47241005c0 0x7f4724196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 --2- 172.21.15.3:0/3774734415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4724100f80 0x7f4724196c80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 >> 172.21.15.3:0/3774734415 conn(0x7f47240fc0b0 msgr2=0x7f4724104250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 shutdown_connections 2023-09-10T01:07:41.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:41.442+0000 7f472a3ff700 1 -- 172.21.15.3:0/3774734415 wait complete. 2023-09-10T01:07:41.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 365755 9.45 MB/sec execute 83 sec latency 56.515 ms 2023-09-10T01:07:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:42 smithi092 ceph-mon[134067]: pgmap v137: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 221 op/s 2023-09-10T01:07:42.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:42 smithi003 ceph-mon[97654]: pgmap v137: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 221 op/s 2023-09-10T01:07:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:42 smithi136 ceph-mon[133708]: pgmap v137: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 221 op/s 2023-09-10T01:07:42.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 367540 9.40 MB/sec execute 84 sec latency 14.449 ms 2023-09-10T01:07:43.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 369385 9.45 MB/sec execute 85 sec latency 13.631 ms 2023-09-10T01:07:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:44 smithi092 ceph-mon[134067]: pgmap v138: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 221 op/s 2023-09-10T01:07:44.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:44 smithi003 ceph-mon[97654]: pgmap v138: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 221 op/s 2023-09-10T01:07:44.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:44 smithi136 ceph-mon[133708]: pgmap v138: 129 pgs: 129 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 221 op/s 2023-09-10T01:07:44.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 371308 9.42 MB/sec execute 86 sec latency 44.834 ms 2023-09-10T01:07:45.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 372989 9.45 MB/sec execute 87 sec latency 20.968 ms 2023-09-10T01:07:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:46 smithi092 ceph-mon[134067]: pgmap v139: 129 pgs: 129 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 391 op/s 2023-09-10T01:07:46.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:46 smithi003 ceph-mon[97654]: pgmap v139: 129 pgs: 129 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 391 op/s 2023-09-10T01:07:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:46 smithi136 ceph-mon[133708]: pgmap v139: 129 pgs: 129 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 391 op/s 2023-09-10T01:07:46.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 375150 9.47 MB/sec execute 88 sec latency 14.479 ms 2023-09-10T01:07:47.753 INFO:tasks.workunit.client.0.smithi003.stdout: 1 376636 9.46 MB/sec execute 89 sec latency 20.423 ms 2023-09-10T01:07:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:48 smithi092 ceph-mon[134067]: pgmap v140: 129 pgs: 129 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 270 op/s 2023-09-10T01:07:48.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:48 smithi003 ceph-mon[97654]: pgmap v140: 129 pgs: 129 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 270 op/s 2023-09-10T01:07:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:48 smithi136 ceph-mon[133708]: pgmap v140: 129 pgs: 129 active+clean; 97 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 270 op/s 2023-09-10T01:07:48.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 378891 9.48 MB/sec execute 90 sec latency 9.071 ms 2023-09-10T01:07:49.753 INFO:tasks.workunit.client.0.smithi003.stdout: 1 380045 9.46 MB/sec execute 91 sec latency 44.406 ms 2023-09-10T01:07:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:50 smithi092 ceph-mon[134067]: pgmap v141: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 382 op/s 2023-09-10T01:07:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:50 smithi003 ceph-mon[97654]: pgmap v141: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 382 op/s 2023-09-10T01:07:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:50 smithi136 ceph-mon[133708]: pgmap v141: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 382 op/s 2023-09-10T01:07:50.752 INFO:tasks.workunit.client.0.smithi003.stdout: 1 382159 9.44 MB/sec execute 92 sec latency 24.046 ms 2023-09-10T01:07:51.753 INFO:tasks.workunit.client.0.smithi003.stdout: 1 383771 9.46 MB/sec execute 93 sec latency 66.853 ms 2023-09-10T01:07:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:52 smithi092 ceph-mon[134067]: pgmap v142: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:07:52.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:52 smithi003 ceph-mon[97654]: pgmap v142: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:07:52.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:52 smithi136 ceph-mon[133708]: pgmap v142: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:07:52.753 INFO:tasks.workunit.client.0.smithi003.stdout: 1 385906 9.48 MB/sec execute 94 sec latency 10.977 ms 2023-09-10T01:07:53.753 INFO:tasks.workunit.client.0.smithi003.stdout: 1 387358 9.47 MB/sec execute 95 sec latency 23.845 ms 2023-09-10T01:07:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:54 smithi092 ceph-mon[134067]: pgmap v143: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:07:54.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:54 smithi003 ceph-mon[97654]: pgmap v143: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:07:54.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.607+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3685151036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81027c0 msgr2=0x7fdef8102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.607+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3685151036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81027c0 0x7fdef8102c40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009990 tx=0x7fdee0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.607+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3685151036 shutdown_connections 2023-09-10T01:07:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.607+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3685151036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdef8103180 0x7fdef810f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.607+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3685151036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef81087d0 0x7fdef8108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.607+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3685151036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81027c0 0x7fdef8102c40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.607+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3685151036 >> 172.21.15.3:0/3685151036 conn(0x7fdef80fe290 msgr2=0x7fdef81006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.608+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3685151036 shutdown_connections 2023-09-10T01:07:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.608+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3685151036 wait complete. 2023-09-10T01:07:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.609+0000 7fdefd1ff700 1 Processor -- start 2023-09-10T01:07:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.610+0000 7fdefd1ff700 1 -- start start 2023-09-10T01:07:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.610+0000 7fdefd1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef81027c0 0x7fdef810da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.610+0000 7fdefd1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdef8103180 0x7fdef81197b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.611+0000 7fdefd1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81087d0 0x7fdef810df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.611+0000 7fdefd1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef810e500 con 0x7fdef81087d0 2023-09-10T01:07:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.611+0000 7fdefd1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef810e670 con 0x7fdef81027c0 2023-09-10T01:07:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.611+0000 7fdefd1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef810e830 con 0x7fdef8103180 2023-09-10T01:07:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.611+0000 7fdef759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81087d0 0x7fdef810df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.611+0000 7fdef759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81087d0 0x7fdef810df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45532/0 (socket says 172.21.15.3:45532) 2023-09-10T01:07:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.611+0000 7fdef759e700 1 -- 172.21.15.3:0/3446503465 learned_addr learned my addr 172.21.15.3:0/3446503465 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.612+0000 7fdef759e700 1 -- 172.21.15.3:0/3446503465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdef8103180 msgr2=0x7fdef81197b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-10T01:07:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.612+0000 7fdef759e700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdef8103180 0x7fdef81197b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.612+0000 7fdef759e700 1 -- 172.21.15.3:0/3446503465 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef81027c0 msgr2=0x7fdef810da50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-10T01:07:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.612+0000 7fdef759e700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef81027c0 0x7fdef810da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.612+0000 7fdef759e700 1 -- 172.21.15.3:0/3446503465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00095f0 con 0x7fdef81087d0 2023-09-10T01:07:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.613+0000 7fdef759e700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81087d0 0x7fdef810df90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdef0007ee0 tx=0x7fdef000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.614+0000 7fdef4d99700 1 -- 172.21.15.3:0/3446503465 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0017070 con 0x7fdef81087d0 2023-09-10T01:07:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.614+0000 7fdef4d99700 1 -- 172.21.15.3:0/3446503465 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdef000f040 con 0x7fdef81087d0 2023-09-10T01:07:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.614+0000 7fdef4d99700 1 -- 172.21.15.3:0/3446503465 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef00057d0 con 0x7fdef81087d0 2023-09-10T01:07:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.614+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef81b1d70 con 0x7fdef81087d0 2023-09-10T01:07:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.615+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef81b21d0 con 0x7fdef81087d0 2023-09-10T01:07:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.621+0000 7fdef4d99700 1 -- 172.21.15.3:0/3446503465 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fdef0005930 con 0x7fdef81087d0 2023-09-10T01:07:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.622+0000 7fdf10bd9b80 1 Processor -- start 2023-09-10T01:07:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.622+0000 7fdf10bd9b80 1 -- start start 2023-09-10T01:07:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.623+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a79f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.623+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a799780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.623+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.623+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a6f26d0 con 0x56550a799300 2023-09-10T01:07:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.623+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a6efff0 con 0x56550a799cc0 2023-09-10T01:07:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.623+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a6eb8e0 con 0x56550a79f1a0 2023-09-10T01:07:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdeee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdeee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59014/0 (socket says 172.21.15.3:59014) 2023-09-10T01:07:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdeee59c700 1 -- 172.21.15.3:0/2254065382 learned_addr learned my addr 172.21.15.3:0/2254065382 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdeee59c700 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a79f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdeee59c700 1 -- 172.21.15.3:0/2254065382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 msgr2=0x56550a79f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdeee59c700 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a79f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a799780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.624+0000 7fdeee59c700 1 -- 172.21.15.3:0/2254065382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 msgr2=0x56550a799780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.625+0000 7fdeee59c700 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a799780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.625+0000 7fdeee59c700 1 -- 172.21.15.3:0/2254065382 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56550a517d80 con 0x56550a799cc0 2023-09-10T01:07:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.625+0000 7fdeee59c700 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a5f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdee0005850 tx=0x7fdee0005ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e38b641c1d5b7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.625+0000 7fdef4d99700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdee40a0940 0x7fdee40a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.625+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a799780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:07:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.625+0000 7fdef4d99700 1 -- 172.21.15.3:0/3446503465 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdef00cbb70 con 0x7fdef81087d0 2023-09-10T01:07:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.625+0000 7fdeed59a700 1 -- 172.21.15.3:0/2254065382 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee000bab0 con 0x56550a799cc0 2023-09-10T01:07:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.626+0000 7fdeed59a700 1 -- 172.21.15.3:0/2254065382 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdee00054f0 con 0x56550a799cc0 2023-09-10T01:07:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.626+0000 7fdeed59a700 1 -- 172.21.15.3:0/2254065382 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0004930 con 0x56550a799cc0 2023-09-10T01:07:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.626+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdee40a0940 0x7fdee40a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2254065382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 msgr2=0x56550a7a5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a5f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdee0005850 tx=0x7fdee0005ab0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdee40a0940 0x7fdee40a2e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x56550a794060 tx=0x7fdee8009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2254065382 shutdown_connections 2023-09-10T01:07:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a79f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a5f30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2254065382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a799780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.627+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2254065382 >> 172.21.15.3:0/2254065382 conn(0x56550a70e030 msgr2=0x56550a794560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.628+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2254065382 shutdown_connections 2023-09-10T01:07:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.628+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2254065382 wait complete. 2023-09-10T01:07:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.629+0000 7fdf10bd9b80 1 -- start start 2023-09-10T01:07:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.629+0000 7fdf10bd9b80 1 Processor -- start 2023-09-10T01:07:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.630+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.630+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.630+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a7a3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.630+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a689b10 con 0x56550a799300 2023-09-10T01:07:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.630+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a689810 con 0x56550a799cc0 2023-09-10T01:07:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.630+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a56d8c0 con 0x56550a79f1a0 2023-09-10T01:07:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.630+0000 7fdf10bd9b80 10 client.0 _ll_register_callbacks cb 0x56550a5d3a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.631+0000 7fdeee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.631+0000 7fdeee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59026/0 (socket says 172.21.15.3:59026) 2023-09-10T01:07:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.631+0000 7fdeee59c700 1 -- 172.21.15.3:0/2490918770 learned_addr learned my addr 172.21.15.3:0/2490918770 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.631+0000 7fdef759e700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a7a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.632+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.632+0000 7fdeee59c700 1 -- 172.21.15.3:0/2490918770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 msgr2=0x56550a7a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.632+0000 7fdeee59c700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a7a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.632+0000 7fdeee59c700 1 -- 172.21.15.3:0/2490918770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 msgr2=0x56550a7a5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.632+0000 7fdeee59c700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.632+0000 7fdeee59c700 1 -- 172.21.15.3:0/2490918770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef810ff90 con 0x56550a799cc0 2023-09-10T01:07:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.632+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:07:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.633+0000 7fdeee59c700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a3150 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdee00056a0 tx=0x7fdee00056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=890e07ce4c0f03ac server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.633+0000 7fdede7fc700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.634+0000 7fdede7fc700 1 -- 172.21.15.3:0/2490918770 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee002e070 con 0x56550a799cc0 2023-09-10T01:07:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.634+0000 7fdede7fc700 1 -- 172.21.15.3:0/2490918770 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdee000bb40 con 0x56550a799cc0 2023-09-10T01:07:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.634+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56550a643490 con 0x56550a799cc0 2023-09-10T01:07:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.635+0000 7fdede7fc700 1 -- 172.21.15.3:0/2490918770 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdee000bd00 con 0x56550a799cc0 2023-09-10T01:07:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.635+0000 7fdf10bd9b80 10 client.26999 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.635+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56550a500c00 con 0x56550a799cc0 2023-09-10T01:07:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.636+0000 7fdede7fc700 1 -- 172.21.15.3:0/2490918770 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56550a643490 con 0x56550a799cc0 2023-09-10T01:07:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.636+0000 7fdede7fc700 1 -- 172.21.15.3:0/2490918770 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fdee0028750 con 0x56550a799cc0 2023-09-10T01:07:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.637+0000 7fdf10bd9b80 10 client.26999 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.637+0000 7fdf10bd9b80 10 client.26999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.637+0000 7fdf10bd9b80 20 client.26999 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.637+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a7b2ee0 0x56550a7b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.637+0000 7fdf10bd9b80 4 client.26999 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:07:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.637+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56550a700840 con 0x56550a7b2ee0 2023-09-10T01:07:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.637+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a7b2ee0 0x56550a7b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.638+0000 7fdede7fc700 10 client.26999 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.639+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a7b2ee0 0x56550a7b5300 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85dffa2a4ffbc8c server_cookie=bab1feb78cce543c in_seq=0 out_seq=0 2023-09-10T01:07:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.640+0000 7fdede7fc700 1 -- 172.21.15.3:0/2490918770 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56550a700840 con 0x56550a7b2ee0 2023-09-10T01:07:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.640+0000 7fdede7fc700 10 client.26999 handle_command_reply: tid=0 2023-09-10T01:07:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.641+0000 7fdf10bd9b80 1 client.26999 shutdown 2023-09-10T01:07:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.641+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 msgr2=0x56550a7a3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.641+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a3150 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdee00056a0 tx=0x7fdee00056d0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.642+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 shutdown_connections 2023-09-10T01:07:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.642+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a7b2ee0 0x56550a7b5300 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.642+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a79f1a0 0x56550a7a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.642+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.642+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/2490918770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799cc0 0x56550a7a3150 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.642+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 >> 172.21.15.3:0/2490918770 conn(0x56550a70e030 msgr2=0x56550a7937a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.642+0000 7fdede7fc700 0 client.26999 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.644+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 shutdown_connections 2023-09-10T01:07:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.644+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/2490918770 wait complete. 2023-09-10T01:07:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.645+0000 7fdf10bd9b80 20 client.26999 trim_cache size 0 max 16384 2023-09-10T01:07:54.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:54 smithi136 ceph-mon[133708]: pgmap v143: 129 pgs: 129 active+clean; 102 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 282 op/s 2023-09-10T01:07:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.662+0000 7fdf10bd9b80 1 Processor -- start 2023-09-10T01:07:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.662+0000 7fdf10bd9b80 1 -- start start 2023-09-10T01:07:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a79d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a79f1a0 0x56550a79d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a700840 con 0x56550a799300 2023-09-10T01:07:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a643490 con 0x56550a79f1a0 2023-09-10T01:07:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a78e8a0 con 0x56550a799cc0 2023-09-10T01:07:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a79f1a0 0x56550a79d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48744/0 (socket says 172.21.15.3:48744) 2023-09-10T01:07:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 -- 172.21.15.3:0/3126408324 learned_addr learned my addr 172.21.15.3:0/3126408324 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdeee59c700 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a79d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 -- 172.21.15.3:0/3126408324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 msgr2=0x56550a79d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a79d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 -- 172.21.15.3:0/3126408324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a79f1a0 msgr2=0x56550a79d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a79f1a0 0x56550a79d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef6d9d700 1 -- 172.21.15.3:0/3126408324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56550a53e1f0 con 0x56550a799300 2023-09-10T01:07:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.663+0000 7fdef759e700 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a79f1a0 0x56550a79d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:07:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.664+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a9b20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdee800f9c0 tx=0x7fdee8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6274f1644d55a024 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.664+0000 7fdeedd9b700 1 -- 172.21.15.3:0/3126408324 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8024070 con 0x56550a799300 2023-09-10T01:07:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.664+0000 7fdeedd9b700 1 -- 172.21.15.3:0/3126408324 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdee80044e0 con 0x56550a799300 2023-09-10T01:07:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.664+0000 7fdeedd9b700 1 -- 172.21.15.3:0/3126408324 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8024e60 con 0x56550a799300 2023-09-10T01:07:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/3126408324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 msgr2=0x56550a7a9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a9b20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdee800f9c0 tx=0x7fdee8015000 comp rx=0 tx=0).stop 2023-09-10T01:07:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/3126408324 shutdown_connections 2023-09-10T01:07:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a79f1a0 0x56550a79d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a799300 0x56550a7a9b20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/3126408324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a79d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/3126408324 >> 172.21.15.3:0/3126408324 conn(0x56550a70e030 msgr2=0x56550a79e170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/3126408324 shutdown_connections 2023-09-10T01:07:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/3126408324 wait complete. 2023-09-10T01:07:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 -- start start 2023-09-10T01:07:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.665+0000 7fdf10bd9b80 1 Processor -- start 2023-09-10T01:07:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799300 0x56550a7a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a7a8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdf10bd9b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a79f1a0 0x56550a7a8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a805ec0 con 0x56550a79f1a0 2023-09-10T01:07:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a806030 con 0x56550a799300 2023-09-10T01:07:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdf10bd9b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56550a8061f0 con 0x56550a799cc0 2023-09-10T01:07:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdf10bd9b80 10 client.0 _ll_register_callbacks cb 0x7fdee8005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:07:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdef759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a79f1a0 0x56550a7a8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdef6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799300 0x56550a7a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdeee59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a7a8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdef6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799300 0x56550a7a7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57558/0 (socket says 172.21.15.3:57558) 2023-09-10T01:07:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.666+0000 7fdef6d9d700 1 -- 172.21.15.3:0/1460324483 learned_addr learned my addr 172.21.15.3:0/1460324483 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:07:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdef6d9d700 1 -- 172.21.15.3:0/1460324483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 msgr2=0x56550a7a8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a7a8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdef6d9d700 1 -- 172.21.15.3:0/1460324483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a79f1a0 msgr2=0x56550a7a8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:07:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a79f1a0 0x56550a7a8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdef6d9d700 1 -- 172.21.15.3:0/1460324483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef81b21d0 con 0x56550a799300 2023-09-10T01:07:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdef6d9d700 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799300 0x56550a7a7af0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdee800fa90 tx=0x7fdee8005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f0b5d486a06ca9b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:07:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdede7fc700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:07:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdede7fc700 1 -- 172.21.15.3:0/1460324483 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8024050 con 0x56550a799300 2023-09-10T01:07:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdee00054f0 con 0x56550a799300 2023-09-10T01:07:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdede7fc700 1 -- 172.21.15.3:0/1460324483 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdee8024a60 con 0x56550a799300 2023-09-10T01:07:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdede7fc700 1 -- 172.21.15.3:0/1460324483 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdee8024c20 con 0x56550a799300 2023-09-10T01:07:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdf10bd9b80 10 client.27005 fetch_fsmap learned FSMap version 26 2023-09-10T01:07:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.667+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56550a516250 con 0x56550a799300 2023-09-10T01:07:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdede7fc700 1 -- 172.21.15.3:0/1460324483 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdee00054f0 con 0x56550a799300 2023-09-10T01:07:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdede7fc700 1 -- 172.21.15.3:0/1460324483 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fdee802b020 con 0x56550a799300 2023-09-10T01:07:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdf10bd9b80 10 client.27005 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:07:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdf10bd9b80 10 client.27005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:07:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdf10bd9b80 20 client.27005 populate_metadata read hostname 'smithi003' 2023-09-10T01:07:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a807700 0x56550a809af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:07:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdf10bd9b80 4 client.27005 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:07:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56550a6e48c0 con 0x56550a807700 2023-09-10T01:07:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdeee59c700 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a807700 0x56550a809af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:07:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdeee59c700 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a807700 0x56550a809af0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f09992c022984dc server_cookie=4d20cc90942a68fb in_seq=0 out_seq=0 2023-09-10T01:07:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.668+0000 7fdede7fc700 10 client.27005 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:54.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 389065 9.44 MB/sec execute 96 sec latency 89.188 ms 2023-09-10T01:07:54.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.780+0000 7fdede7fc700 1 -- 172.21.15.3:0/1460324483 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56550a6e48c0 con 0x56550a807700 2023-09-10T01:07:54.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.780+0000 7fdede7fc700 10 client.27005 handle_command_reply: tid=0 2023-09-10T01:07:54.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.780+0000 7fdf10bd9b80 1 client.27005 shutdown 2023-09-10T01:07:54.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.782+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799300 msgr2=0x56550a7a7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:54.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.782+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799300 0x56550a7a7af0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdee800fa90 tx=0x7fdee8005760 comp rx=0 tx=0).stop 2023-09-10T01:07:54.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 shutdown_connections 2023-09-10T01:07:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56550a807700 0x56550a809af0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56550a79f1a0 0x56550a7a8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56550a799300 0x56550a7a7af0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 --2- 172.21.15.3:0/1460324483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56550a799cc0 0x56550a7a8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 >> 172.21.15.3:0/1460324483 conn(0x56550a70e030 msgr2=0x56550a79d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdede7fc700 0 client.27005 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 shutdown_connections 2023-09-10T01:07:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.783+0000 7fdf10bd9b80 1 -- 172.21.15.3:0/1460324483 wait complete. 2023-09-10T01:07:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:07:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:07:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:07:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:07:54.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.784+0000 7fdf10bd9b80 20 client.27005 trim_cache size 0 max 16384 2023-09-10T01:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.785+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdee40a0940 msgr2=0x7fdee40a2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.785+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdee40a0940 0x7fdee40a2e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x56550a794060 tx=0x7fdee8009260 comp rx=0 tx=0).stop 2023-09-10T01:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.785+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81087d0 msgr2=0x7fdef810df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.785+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81087d0 0x7fdef810df90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdef0007ee0 tx=0x7fdef000d3b0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.786+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 shutdown_connections 2023-09-10T01:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.786+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdee40a0940 0x7fdee40a2e00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.786+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdef81087d0 0x7fdef810df90 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.786+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef81027c0 0x7fdef810da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.786+0000 7fdefd1ff700 1 --2- 172.21.15.3:0/3446503465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdef8103180 0x7fdef81197b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:07:54.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.786+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 >> 172.21.15.3:0/3446503465 conn(0x7fdef80fe290 msgr2=0x7fdef8100210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:07:54.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.787+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 shutdown_connections 2023-09-10T01:07:54.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:07:54.787+0000 7fdefd1ff700 1 -- 172.21.15.3:0/3446503465 wait complete. 2023-09-10T01:07:55.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 390654 9.47 MB/sec execute 97 sec latency 33.758 ms 2023-09-10T01:07:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:56 smithi092 ceph-mon[134067]: pgmap v144: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 390 op/s 2023-09-10T01:07:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:56 smithi003 ceph-mon[97654]: pgmap v144: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 390 op/s 2023-09-10T01:07:56.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:56 smithi136 ceph-mon[133708]: pgmap v144: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 390 op/s 2023-09-10T01:07:56.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 392361 9.43 MB/sec execute 98 sec latency 86.150 ms 2023-09-10T01:07:57.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 394206 9.47 MB/sec execute 99 sec latency 12.464 ms 2023-09-10T01:07:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:07:58 smithi092 ceph-mon[134067]: pgmap v145: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 219 op/s 2023-09-10T01:07:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:07:58 smithi003 ceph-mon[97654]: pgmap v145: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 219 op/s 2023-09-10T01:07:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:07:58 smithi136 ceph-mon[133708]: pgmap v145: 129 pgs: 129 active+clean; 107 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 219 op/s 2023-09-10T01:07:58.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 396163 9.44 MB/sec execute 100 sec latency 18.526 ms 2023-09-10T01:07:59.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 397712 9.47 MB/sec execute 101 sec latency 54.382 ms 2023-09-10T01:08:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:00 smithi092 ceph-mon[134067]: pgmap v146: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 372 op/s 2023-09-10T01:08:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:00 smithi003 ceph-mon[97654]: pgmap v146: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 372 op/s 2023-09-10T01:08:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:00 smithi136 ceph-mon[133708]: pgmap v146: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 372 op/s 2023-09-10T01:08:00.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 398339 9.39 MB/sec execute 102 sec latency 59.318 ms 2023-09-10T01:08:00.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:08:01.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 -- 172.21.15.3:0/967094810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40400ffdd0 msgr2=0x7f404010ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 --2- 172.21.15.3:0/967094810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40400ffdd0 0x7f404010ca90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4030009990 tx=0x7f4030009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 -- 172.21.15.3:0/967094810 shutdown_connections 2023-09-10T01:08:01.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 --2- 172.21.15.3:0/967094810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f404010cfd0 0x7f404010f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 --2- 172.21.15.3:0/967094810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40400ff420 0x7f40400ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 --2- 172.21.15.3:0/967094810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40400ffdd0 0x7f404010ca90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 -- 172.21.15.3:0/967094810 >> 172.21.15.3:0/967094810 conn(0x7f4040076710 msgr2=0x7f4040076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 -- 172.21.15.3:0/967094810 shutdown_connections 2023-09-10T01:08:01.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.017+0000 7f40481ff700 1 -- 172.21.15.3:0/967094810 wait complete. 2023-09-10T01:08:01.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 Processor -- start 2023-09-10T01:08:01.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 -- start start 2023-09-10T01:08:01.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40400ff420 0x7f40401989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40400ffdd0 0x7f4040198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404010cfd0 0x7f404019d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404019d950 con 0x7f40400ff420 2023-09-10T01:08:01.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404019dac0 con 0x7f404010cfd0 2023-09-10T01:08:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f40481ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404019dc80 con 0x7f40400ffdd0 2023-09-10T01:08:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f404579a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40400ffdd0 0x7f4040198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f4045f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40400ff420 0x7f40401989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f404579a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40400ffdd0 0x7f4040198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43786/0 (socket says 172.21.15.3:43786) 2023-09-10T01:08:01.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.019+0000 7f404579a700 1 -- 172.21.15.3:0/3383383040 learned_addr learned my addr 172.21.15.3:0/3383383040 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.020+0000 7f404679c700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404010cfd0 0x7f404019d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.020+0000 7f404579a700 1 -- 172.21.15.3:0/3383383040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404010cfd0 msgr2=0x7f404019d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.020+0000 7f404579a700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404010cfd0 0x7f404019d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.020+0000 7f404579a700 1 -- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40400ff420 msgr2=0x7f40401989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.020+0000 7f404579a700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40400ff420 0x7f40401989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.020+0000 7f404579a700 1 -- 172.21.15.3:0/3383383040 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40300095f0 con 0x7f40400ffdd0 2023-09-10T01:08:01.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.020+0000 7f404579a700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40400ffdd0 0x7f4040198ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4030005910 tx=0x7f4030005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.021+0000 7f4036ffd700 1 -- 172.21.15.3:0/3383383040 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403001d070 con 0x7f40400ffdd0 2023-09-10T01:08:01.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.021+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404019df00 con 0x7f40400ffdd0 2023-09-10T01:08:01.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.021+0000 7f4036ffd700 1 -- 172.21.15.3:0/3383383040 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4030022470 con 0x7f40400ffdd0 2023-09-10T01:08:01.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.021+0000 7f4036ffd700 1 -- 172.21.15.3:0/3383383040 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030017650 con 0x7f40400ffdd0 2023-09-10T01:08:01.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.021+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40401a5810 con 0x7f40400ffdd0 2023-09-10T01:08:01.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.025+0000 7f4036ffd700 1 -- 172.21.15.3:0/3383383040 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f40300225e0 con 0x7f40400ffdd0 2023-09-10T01:08:01.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.025+0000 7f405ba51b80 1 Processor -- start 2023-09-10T01:08:01.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.026+0000 7f405ba51b80 1 -- start start 2023-09-10T01:08:01.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.026+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff0710 0x55acc7ff0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.026+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ffa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.026+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc7ffd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.026+0000 7f405ba51b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7f49730 con 0x55acc7ff1030 2023-09-10T01:08:01.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.026+0000 7f405ba51b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7f47050 con 0x55acc7ffae40 2023-09-10T01:08:01.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.026+0000 7f405ba51b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7f42940 con 0x55acc7ff0710 2023-09-10T01:08:01.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff0710 0x55acc7ff0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff0710 0x55acc7ff0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43794/0 (socket says 172.21.15.3:43794) 2023-09-10T01:08:01.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 -- 172.21.15.3:0/1611107786 learned_addr learned my addr 172.21.15.3:0/1611107786 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f4045f9b700 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ffa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404579a700 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc7ffd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f4036ffd700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f402c0a07b0 0x7f402c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 -- 172.21.15.3:0/1611107786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 msgr2=0x55acc7ffd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc7ffd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f4036ffd700 1 -- 172.21.15.3:0/3383383040 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f40300cd7c0 con 0x7f40400ffdd0 2023-09-10T01:08:01.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 -- 172.21.15.3:0/1611107786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 msgr2=0x55acc7ffa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ffa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 -- 172.21.15.3:0/1611107786 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acc7d6ede0 con 0x55acc7ff0710 2023-09-10T01:08:01.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.027+0000 7f404679c700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f402c0a07b0 0x7f402c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.028+0000 7f404679c700 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff0710 0x55acc7ff0af0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4038011690 tx=0x7f40380119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=590bffe8616d54a5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.028+0000 7f402bfff700 1 -- 172.21.15.3:0/1611107786 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403800e040 con 0x55acc7ff0710 2023-09-10T01:08:01.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.028+0000 7f402bfff700 1 -- 172.21.15.3:0/1611107786 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f403800f040 con 0x55acc7ff0710 2023-09-10T01:08:01.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.028+0000 7f402bfff700 1 -- 172.21.15.3:0/1611107786 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038020700 con 0x55acc7ff0710 2023-09-10T01:08:01.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 -- 172.21.15.3:0/1611107786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff0710 msgr2=0x55acc7ff0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f404679c700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f402c0a07b0 0x7f402c0a2c70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4038022000 tx=0x7f403801f000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff0710 0x55acc7ff0af0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4038011690 tx=0x7f40380119a0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 -- 172.21.15.3:0/1611107786 shutdown_connections 2023-09-10T01:08:01.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff0710 0x55acc7ff0af0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc7ffd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1611107786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ffa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 -- 172.21.15.3:0/1611107786 >> 172.21.15.3:0/1611107786 conn(0x55acc7f65090 msgr2=0x55acc7febbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 -- 172.21.15.3:0/1611107786 shutdown_connections 2023-09-10T01:08:01.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.029+0000 7f405ba51b80 1 -- 172.21.15.3:0/1611107786 wait complete. 2023-09-10T01:08:01.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 -- start start 2023-09-10T01:08:01.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 Processor -- start 2023-09-10T01:08:01.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7ee0b70 con 0x55acc7ff1030 2023-09-10T01:08:01.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7ee0870 con 0x55acc7ff0710 2023-09-10T01:08:01.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7dc4920 con 0x55acc7ffae40 2023-09-10T01:08:01.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.030+0000 7f405ba51b80 10 client.0 _ll_register_callbacks cb 0x55acc7e2aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:01.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57596/0 (socket says 172.21.15.3:57596) 2023-09-10T01:08:01.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 -- 172.21.15.3:0/1999326053 learned_addr learned my addr 172.21.15.3:0/1999326053 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f404679c700 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f404579a700 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 -- 172.21.15.3:0/1999326053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 msgr2=0x55acc7ff8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 -- 172.21.15.3:0/1999326053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 msgr2=0x55acc7ff7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 -- 172.21.15.3:0/1999326053 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40380112c0 con 0x55acc7ff0710 2023-09-10T01:08:01.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f4045f9b700 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa8d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f403c0117e0 tx=0x7f403c011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=213beb17449f5f09 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f40297fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:01.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.031+0000 7f40297fa700 1 -- 172.21.15.3:0/1999326053 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c016070 con 0x55acc7ff0710 2023-09-10T01:08:01.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.032+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acc7e9a4f0 con 0x55acc7ff0710 2023-09-10T01:08:01.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.032+0000 7f40297fa700 1 -- 172.21.15.3:0/1999326053 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f403c00f830 con 0x55acc7ff0710 2023-09-10T01:08:01.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.032+0000 7f40297fa700 1 -- 172.21.15.3:0/1999326053 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f403c00f9f0 con 0x55acc7ff0710 2023-09-10T01:08:01.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.032+0000 7f405ba51b80 10 client.27017 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:01.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.032+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55acc7d57c60 con 0x55acc7ff0710 2023-09-10T01:08:01.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.032+0000 7f40297fa700 1 -- 172.21.15.3:0/1999326053 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55acc7e9a4f0 con 0x55acc7ff0710 2023-09-10T01:08:01.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.032+0000 7f40297fa700 1 -- 172.21.15.3:0/1999326053 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f403c01ac60 con 0x55acc7ff0710 2023-09-10T01:08:01.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f405ba51b80 10 client.27017 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:01.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f405ba51b80 10 client.27017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:01.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f405ba51b80 20 client.27017 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:01.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc800aab0 0x55acc800ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f405ba51b80 4 client.27017 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:01.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acc7f578a0 con 0x55acc800aab0 2023-09-10T01:08:01.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f404579a700 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc800aab0 0x55acc800ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f404579a700 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc800aab0 0x55acc800ced0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2842cb55515bf22a server_cookie=d8067dadc433cc60 in_seq=0 out_seq=0 2023-09-10T01:08:01.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.033+0000 7f40297fa700 10 client.27017 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:01.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f40297fa700 1 -- 172.21.15.3:0/1999326053 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55acc7f578a0 con 0x55acc800aab0 2023-09-10T01:08:01.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f40297fa700 10 client.27017 handle_command_reply: tid=0 2023-09-10T01:08:01.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 client.27017 shutdown 2023-09-10T01:08:01.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 msgr2=0x55acc7ffa8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa8d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f403c0117e0 tx=0x7f403c011af0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 shutdown_connections 2023-09-10T01:08:01.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc800aab0 0x55acc800ced0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa8d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 --2- 172.21.15.3:0/1999326053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 >> 172.21.15.3:0/1999326053 conn(0x55acc7f65090 msgr2=0x55acc7feb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.035+0000 7f40297fa700 0 client.27017 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:01.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.036+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 shutdown_connections 2023-09-10T01:08:01.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.036+0000 7f405ba51b80 1 -- 172.21.15.3:0/1999326053 wait complete. 2023-09-10T01:08:01.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.036+0000 7f405ba51b80 20 client.27017 trim_cache size 0 max 16384 2023-09-10T01:08:01.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.054+0000 7f405ba51b80 1 Processor -- start 2023-09-10T01:08:01.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.054+0000 7f405ba51b80 1 -- start start 2023-09-10T01:08:01.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.054+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.054+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f405ba51b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7ffd750 con 0x55acc7ff1030 2023-09-10T01:08:01.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f405ba51b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc8008100 con 0x55acc7ff0710 2023-09-10T01:08:01.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f405ba51b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc7fed230 con 0x55acc7ffae40 2023-09-10T01:08:01.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f4045f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f4045f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57604/0 (socket says 172.21.15.3:57604) 2023-09-10T01:08:01.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f4045f9b700 1 -- 172.21.15.3:0/2016011341 learned_addr learned my addr 172.21.15.3:0/2016011341 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48808/0 (socket says 172.21.15.3:48808) 2023-09-10T01:08:01.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404679c700 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 -- 172.21.15.3:0/2016011341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 msgr2=0x55acc7ff78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 -- 172.21.15.3:0/2016011341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 msgr2=0x55acc7ffa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 -- 172.21.15.3:0/2016011341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acc7d95250 con 0x55acc7ff1030 2023-09-10T01:08:01.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.055+0000 7f404579a700 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7380 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f40300cab20 tx=0x7f403001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2adb9c5ec4dcd2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f4034ff9700 1 -- 172.21.15.3:0/2016011341 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403009a570 con 0x55acc7ff1030 2023-09-10T01:08:01.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f4034ff9700 1 -- 172.21.15.3:0/2016011341 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f403009a390 con 0x55acc7ff1030 2023-09-10T01:08:01.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f4034ff9700 1 -- 172.21.15.3:0/2016011341 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40300a2a10 con 0x55acc7ff1030 2023-09-10T01:08:01.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f405ba51b80 1 -- 172.21.15.3:0/2016011341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 msgr2=0x55acc7ff7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7380 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f40300cab20 tx=0x7f403001a040 comp rx=0 tx=0).stop 2023-09-10T01:08:01.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f405ba51b80 1 -- 172.21.15.3:0/2016011341 shutdown_connections 2023-09-10T01:08:01.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ffae40 0x55acc7ff78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ff0710 0x55acc7ffa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2016011341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff1030 0x55acc7ff7380 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.056+0000 7f405ba51b80 1 -- 172.21.15.3:0/2016011341 >> 172.21.15.3:0/2016011341 conn(0x55acc7f65090 msgr2=0x55acc7ff2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.057+0000 7f405ba51b80 1 -- 172.21.15.3:0/2016011341 shutdown_connections 2023-09-10T01:08:01.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.057+0000 7f405ba51b80 1 -- 172.21.15.3:0/2016011341 wait complete. 2023-09-10T01:08:01.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.057+0000 7f405ba51b80 1 -- start start 2023-09-10T01:08:01.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.057+0000 7f405ba51b80 1 Processor -- start 2023-09-10T01:08:01.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff0710 0x55acc8001940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff1030 0x55acc7fffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f405ba51b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc8000170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f405ba51b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc8000750 con 0x55acc7ff0710 2023-09-10T01:08:01.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f405ba51b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc8000890 con 0x55acc7ffae40 2023-09-10T01:08:01.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f405ba51b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc8000a50 con 0x55acc7ff1030 2023-09-10T01:08:01.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f405ba51b80 10 client.0 _ll_register_callbacks cb 0x7f403c009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:01.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f4045f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff0710 0x55acc8001940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f4045f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff0710 0x55acc8001940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48812/0 (socket says 172.21.15.3:48812) 2023-09-10T01:08:01.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f4045f9b700 1 -- 172.21.15.3:0/2859404695 learned_addr learned my addr 172.21.15.3:0/2859404695 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.058+0000 7f404579a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff1030 0x55acc7fffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f404679c700 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc8000170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f4045f9b700 1 -- 172.21.15.3:0/2859404695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff1030 msgr2=0x55acc7fffc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f4045f9b700 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff1030 0x55acc7fffc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f4045f9b700 1 -- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 msgr2=0x55acc8000170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f4045f9b700 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc8000170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f4045f9b700 1 -- 172.21.15.3:0/2859404695 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40401a5810 con 0x55acc7ff0710 2023-09-10T01:08:01.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f4045f9b700 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff0710 0x55acc8001940 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f403c0095c0 tx=0x7f403c0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13355419d776c51 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f40297fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:08:01.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f40297fa700 1 -- 172.21.15.3:0/2859404695 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c016040 con 0x55acc7ff0710 2023-09-10T01:08:01.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f40297fa700 1 -- 172.21.15.3:0/2859404695 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f403c000e40 con 0x55acc7ff0710 2023-09-10T01:08:01.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.059+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f403800f040 con 0x55acc7ff0710 2023-09-10T01:08:01.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f40297fa700 1 -- 172.21.15.3:0/2859404695 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f403c019c90 con 0x55acc7ff0710 2023-09-10T01:08:01.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 10 client.16953 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:01.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55acc7d6d2b0 con 0x55acc7ff0710 2023-09-10T01:08:01.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f40297fa700 1 -- 172.21.15.3:0/2859404695 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f403800f040 con 0x55acc7ff0710 2023-09-10T01:08:01.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f40297fa700 1 -- 172.21.15.3:0/2859404695 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f403c005390 con 0x55acc7ff0710 2023-09-10T01:08:01.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 10 client.16953 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:01.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 10 client.16953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:01.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 20 client.16953 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:01.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc805fe70 0x55acc8062290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 4 client.16953 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:08:01.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55acc7f3b920 con 0x55acc805fe70 2023-09-10T01:08:01.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.060+0000 7f404579a700 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc805fe70 0x55acc8062290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f404579a700 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc805fe70 0x55acc8062290 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44673013136eeb41 server_cookie=658124db6d32aa50 in_seq=0 out_seq=0 2023-09-10T01:08:01.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f40297fa700 10 client.16953 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:01.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f40297fa700 1 -- 172.21.15.3:0/2859404695 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55acc7f3b920 con 0x55acc805fe70 2023-09-10T01:08:01.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f40297fa700 10 client.16953 handle_command_reply: tid=0 2023-09-10T01:08:01.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 client.16953 shutdown 2023-09-10T01:08:01.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff0710 msgr2=0x55acc8001940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff0710 0x55acc8001940 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f403c0095c0 tx=0x7f403c0197f0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 shutdown_connections 2023-09-10T01:08:01.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55acc805fe70 0x55acc8062290 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55acc7ffae40 0x55acc8000170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55acc7ff0710 0x55acc8001940 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f40297fa700 0 client.16953 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:01.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 --2- 172.21.15.3:0/2859404695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55acc7ff1030 0x55acc7fffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 >> 172.21.15.3:0/2859404695 conn(0x55acc7f65090 msgr2=0x55acc7fef700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.061+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 shutdown_connections 2023-09-10T01:08:01.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f405ba51b80 1 -- 172.21.15.3:0/2859404695 wait complete. 2023-09-10T01:08:01.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f405ba51b80 20 client.16953 trim_cache size 0 max 16384 2023-09-10T01:08:01.078 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:08:01.078 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:08:01.078 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:08:01.078 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:08:01.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f402c0a07b0 msgr2=0x7f402c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f402c0a07b0 0x7f402c0a2c70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4038022000 tx=0x7f403801f000 comp rx=0 tx=0).stop 2023-09-10T01:08:01.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40400ffdd0 msgr2=0x7f4040198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40400ffdd0 0x7f4040198ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4030005910 tx=0x7f4030005720 comp rx=0 tx=0).stop 2023-09-10T01:08:01.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 shutdown_connections 2023-09-10T01:08:01.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f402c0a07b0 0x7f402c0a2c70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404010cfd0 0x7f404019d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40400ff420 0x7f40401989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 --2- 172.21.15.3:0/3383383040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40400ffdd0 0x7f4040198ef0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.062+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 >> 172.21.15.3:0/3383383040 conn(0x7f4040076710 msgr2=0x7f404010e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.063+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 shutdown_connections 2023-09-10T01:08:01.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.063+0000 7f40481ff700 1 -- 172.21.15.3:0/3383383040 wait complete. 2023-09-10T01:08:01.082 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:08:01.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:08:01.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 -- 172.21.15.3:0/1814014963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 msgr2=0x7fbcd4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/1814014963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 0x7fbcd4106a10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009990 tx=0x7fbcc4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 -- 172.21.15.3:0/1814014963 shutdown_connections 2023-09-10T01:08:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/1814014963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 0x7fbcd4106a10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/1814014963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100fe0 0x7fbcd410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/1814014963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcd4100620 0x7fbcd4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 -- 172.21.15.3:0/1814014963 >> 172.21.15.3:0/1814014963 conn(0x7fbcd40fc110 msgr2=0x7fbcd40fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 -- 172.21.15.3:0/1814014963 shutdown_connections 2023-09-10T01:08:01.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.292+0000 7fbcda3ff700 1 -- 172.21.15.3:0/1814014963 wait complete. 2023-09-10T01:08:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 Processor -- start 2023-09-10T01:08:01.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 -- start start 2023-09-10T01:08:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100620 0x7fbcd41966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcd4100fe0 0x7fbcd4196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 0x7fbcd4199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd419a5a0 con 0x7fbcd4100fe0 2023-09-10T01:08:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd419a710 con 0x7fbcd4100620 2023-09-10T01:08:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcda3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd419a8d0 con 0x7fbcd4106630 2023-09-10T01:08:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcd3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100620 0x7fbcd41966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcd3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100620 0x7fbcd41966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57642/0 (socket says 172.21.15.3:57642) 2023-09-10T01:08:01.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcd3fff700 1 -- 172.21.15.3:0/108650538 learned_addr learned my addr 172.21.15.3:0/108650538 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbcd899c700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 0x7fbcd4199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.294+0000 7fbccbfff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcd4100fe0 0x7fbcd4196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd3fff700 1 -- 172.21.15.3:0/108650538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 msgr2=0x7fbcd4199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd3fff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 0x7fbcd4199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd3fff700 1 -- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcd4100fe0 msgr2=0x7fbcd4196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd3fff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcd4100fe0 0x7fbcd4196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd3fff700 1 -- 172.21.15.3:0/108650538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40095f0 con 0x7fbcd4100620 2023-09-10T01:08:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd899c700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 0x7fbcd4199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd3fff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100620 0x7fbcd41966b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4000c00 tx=0x7fbcc4004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd1ffb700 1 -- 172.21.15.3:0/108650538 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc401d070 con 0x7fbcd4100620 2023-09-10T01:08:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd1ffb700 1 -- 172.21.15.3:0/108650538 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcc4021410 con 0x7fbcd4100620 2023-09-10T01:08:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd419abb0 con 0x7fbcd4100620 2023-09-10T01:08:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.295+0000 7fbcd1ffb700 1 -- 172.21.15.3:0/108650538 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4017650 con 0x7fbcd4100620 2023-09-10T01:08:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.296+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd41a3510 con 0x7fbcd4100620 2023-09-10T01:08:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.296+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd4067480 con 0x7fbcd4100620 2023-09-10T01:08:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.299+0000 7fbcd1ffb700 1 -- 172.21.15.3:0/108650538 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fbcc4021580 con 0x7fbcd4100620 2023-09-10T01:08:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.300+0000 7fbcd1ffb700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbcc00a07b0 0x7fbcc00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.300+0000 7fbcd1ffb700 1 -- 172.21.15.3:0/108650538 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fbcc40cdd20 con 0x7fbcd4100620 2023-09-10T01:08:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.301+0000 7fbccbfff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbcc00a07b0 0x7fbcc00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.301+0000 7fbccbfff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbcc00a07b0 0x7fbcc00a2c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4197cc0 tx=0x7fbcbc009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.303+0000 7fbcd1ffb700 1 -- 172.21.15.3:0/108650538 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbcc40971b0 con 0x7fbcd4100620 2023-09-10T01:08:01.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.582+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcd4197a00 con 0x7fbcd4100620 2023-09-10T01:08:01.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.583+0000 7fbcd1ffb700 1 -- 172.21.15.3:0/108650538 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11293 (secure 0 0 0) 0x7fbcc409a7d0 con 0x7fbcd4100620 2023-09-10T01:08:01.585 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:08:01.585 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:07:08.821454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:08:01.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbcc00a07b0 msgr2=0x7fbcc00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbcc00a07b0 0x7fbcc00a2c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4197cc0 tx=0x7fbcbc009260 comp rx=0 tx=0).stop 2023-09-10T01:08:01.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100620 msgr2=0x7fbcd41966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100620 0x7fbcd41966b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4000c00 tx=0x7fbcc4004790 comp rx=0 tx=0).stop 2023-09-10T01:08:01.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 shutdown_connections 2023-09-10T01:08:01.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbcc00a07b0 0x7fbcc00a2c70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd4106630 0x7fbcd4199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcd4100620 0x7fbcd41966b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 --2- 172.21.15.3:0/108650538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbcd4100fe0 0x7fbcd4196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 >> 172.21.15.3:0/108650538 conn(0x7fbcd40fc110 msgr2=0x7fbcd40fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 shutdown_connections 2023-09-10T01:08:01.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.586+0000 7fbcda3ff700 1 -- 172.21.15.3:0/108650538 wait complete. 2023-09-10T01:08:01.590 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2023-09-10T01:08:01.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:08:01.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 399860 9.36 MB/sec execute 103 sec latency 58.082 ms 2023-09-10T01:08:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 -- 172.21.15.3:0/3974537296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088106650 msgr2=0x7fd088106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 --2- 172.21.15.3:0/3974537296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088106650 0x7fd088106a30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd0780098f0 tx=0x7fd078009c00 comp rx=0 tx=0).stop 2023-09-10T01:08:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 -- 172.21.15.3:0/3974537296 shutdown_connections 2023-09-10T01:08:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 --2- 172.21.15.3:0/3974537296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd088101000 0x7fd08810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 --2- 172.21.15.3:0/3974537296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088106650 0x7fd088106a30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 --2- 172.21.15.3:0/3974537296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd088100640 0x7fd088100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 -- 172.21.15.3:0/3974537296 >> 172.21.15.3:0/3974537296 conn(0x7fd0880fc170 msgr2=0x7fd0880fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 -- 172.21.15.3:0/3974537296 shutdown_connections 2023-09-10T01:08:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.831+0000 7fd08cdff700 1 -- 172.21.15.3:0/3974537296 wait complete. 2023-09-10T01:08:01.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 Processor -- start 2023-09-10T01:08:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 -- start start 2023-09-10T01:08:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088100640 0x7fd088072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd088101000 0x7fd088072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd088106650 0x7fd088073380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0881a90b0 con 0x7fd088101000 2023-09-10T01:08:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0881a91f0 con 0x7fd088100640 2023-09-10T01:08:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.832+0000 7fd08cdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0881a9330 con 0x7fd088106650 2023-09-10T01:08:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088100640 0x7fd088072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd085d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd088101000 0x7fd088072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088100640 0x7fd088072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57654/0 (socket says 172.21.15.3:57654) 2023-09-10T01:08:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 -- 172.21.15.3:0/2255897519 learned_addr learned my addr 172.21.15.3:0/2255897519 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd086d9d700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd088106650 0x7fd088073380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 -- 172.21.15.3:0/2255897519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd088106650 msgr2=0x7fd088073380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd088106650 0x7fd088073380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 -- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd088101000 msgr2=0x7fd088072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd088101000 0x7fd088072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 -- 172.21.15.3:0/2255897519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0095f0 con 0x7fd088100640 2023-09-10T01:08:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd085d9b700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd088101000 0x7fd088072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:01.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd08659c700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088100640 0x7fd088072900 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd078009230 tx=0x7fd07800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd0777fe700 1 -- 172.21.15.3:0/2255897519 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07801c070 con 0x7fd088100640 2023-09-10T01:08:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.833+0000 7fd0777fe700 1 -- 172.21.15.3:0/2255897519 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0780058e0 con 0x7fd088100640 2023-09-10T01:08:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.834+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd078009520 con 0x7fd088100640 2023-09-10T01:08:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.834+0000 7fd0777fe700 1 -- 172.21.15.3:0/2255897519 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780038e0 con 0x7fd088100640 2023-09-10T01:08:01.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.834+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0881a9990 con 0x7fd088100640 2023-09-10T01:08:01.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.837+0000 7fd0777fe700 1 -- 172.21.15.3:0/2255897519 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd078020030 con 0x7fd088100640 2023-09-10T01:08:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.838+0000 7fd0a0792b80 1 Processor -- start 2023-09-10T01:08:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.838+0000 7fd0777fe700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0700a07b0 0x7fd0700a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.838+0000 7fd0a0792b80 1 -- start start 2023-09-10T01:08:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd0777fe700 1 -- 172.21.15.3:0/2255897519 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd0780cd100 con 0x7fd088100640 2023-09-10T01:08:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd086d9d700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0700a07b0 0x7fd0700a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e2624daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624e030 0x555e26257900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e261a6730 con 0x555e2624d710 2023-09-10T01:08:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e261a4050 con 0x555e26257e40 2023-09-10T01:08:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2619f940 con 0x555e2624e030 2023-09-10T01:08:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd08659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e2624daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd08659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e2624daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48854/0 (socket says 172.21.15.3:48854) 2023-09-10T01:08:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd08659c700 1 -- 172.21.15.3:0/1160473876 learned_addr learned my addr 172.21.15.3:0/1160473876 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd085d9b700 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624e030 0x555e26257900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.839+0000 7fd086d9d700 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd086d9d700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0700a07b0 0x7fd0700a2c70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd08000f540 tx=0x7fd080009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd08659c700 1 -- 172.21.15.3:0/1160473876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624e030 msgr2=0x555e26257900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd08659c700 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624e030 0x555e26257900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd08659c700 1 -- 172.21.15.3:0/1160473876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 msgr2=0x555e2625a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd08659c700 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd08659c700 1 -- 172.21.15.3:0/1160473876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e25fcbde0 con 0x555e2624d710 2023-09-10T01:08:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd085d9b700 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624e030 0x555e26257900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd08659c700 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e2624daf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd07809a240 tx=0x7fd07800fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f611dff646e24d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd074ff9700 1 -- 172.21.15.3:0/1160473876 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07809a9b0 con 0x555e2624d710 2023-09-10T01:08:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd074ff9700 1 -- 172.21.15.3:0/1160473876 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd07809a2b0 con 0x555e2624d710 2023-09-10T01:08:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.840+0000 7fd074ff9700 1 -- 172.21.15.3:0/1160473876 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780ca740 con 0x555e2624d710 2023-09-10T01:08:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 -- 172.21.15.3:0/1160473876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 msgr2=0x555e2624daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e2624daf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd07809a240 tx=0x7fd07800fa70 comp rx=0 tx=0).stop 2023-09-10T01:08:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 -- 172.21.15.3:0/1160473876 shutdown_connections 2023-09-10T01:08:01.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e2624daf0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/1160473876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624e030 0x555e26257900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 -- 172.21.15.3:0/1160473876 >> 172.21.15.3:0/1160473876 conn(0x555e261c2090 msgr2=0x555e26248bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 -- 172.21.15.3:0/1160473876 shutdown_connections 2023-09-10T01:08:01.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 -- 172.21.15.3:0/1160473876 wait complete. 2023-09-10T01:08:01.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.841+0000 7fd0a0792b80 1 -- start start 2023-09-10T01:08:01.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 1 Processor -- start 2023-09-10T01:08:01.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e26258220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624e030 0x555e26258760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e262632a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2613db70 con 0x555e2624d710 2023-09-10T01:08:01.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2613d870 con 0x555e2624e030 2023-09-10T01:08:01.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e26021920 con 0x555e26257e40 2023-09-10T01:08:01.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.842+0000 7fd0a0792b80 10 client.0 _ll_register_callbacks cb 0x555e26087a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:01.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624e030 0x555e26258760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624e030 0x555e26258760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57672/0 (socket says 172.21.15.3:57672) 2023-09-10T01:08:01.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 -- 172.21.15.3:0/2568116482 learned_addr learned my addr 172.21.15.3:0/2568116482 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd086d9d700 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e262632a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd08659c700 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e26258220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 -- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 msgr2=0x555e262632a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e262632a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 -- 172.21.15.3:0/2568116482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 msgr2=0x555e26258220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e26258220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 -- 172.21.15.3:0/2568116482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c011440 con 0x555e2624e030 2023-09-10T01:08:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd085d9b700 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624e030 0x555e26258760 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd07c012e90 tx=0x7fd07c00f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a22a7c079dc26433 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd06dffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd06dffb700 1 -- 172.21.15.3:0/2568116482 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c017070 con 0x555e2624e030 2023-09-10T01:08:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.843+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e260f74f0 con 0x555e2624e030 2023-09-10T01:08:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd06dffb700 1 -- 172.21.15.3:0/2568116482 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd07c004050 con 0x555e2624e030 2023-09-10T01:08:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd06dffb700 1 -- 172.21.15.3:0/2568116482 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd07c00fe50 con 0x555e2624e030 2023-09-10T01:08:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 10 client.27041 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fd0881a9990 con 0x555e2624e030 2023-09-10T01:08:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd06dffb700 1 -- 172.21.15.3:0/2568116482 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e260f74f0 con 0x555e2624e030 2023-09-10T01:08:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd06dffb700 1 -- 172.21.15.3:0/2568116482 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd07c0050d0 con 0x555e2624e030 2023-09-10T01:08:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 10 client.27041 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 10 client.27041 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:08:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 20 client.27041 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262676a0 0x555e26269ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 4 client.27041 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.844+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e261b48a0 con 0x555e262676a0 2023-09-10T01:08:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.845+0000 7fd08659c700 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262676a0 0x555e26269ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.845+0000 7fd08659c700 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262676a0 0x555e26269ac0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d41110a688776da5 server_cookie=818d31adb33e7515 in_seq=0 out_seq=0 2023-09-10T01:08:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.845+0000 7fd06dffb700 10 client.27041 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd06dffb700 1 -- 172.21.15.3:0/2568116482 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555e261b48a0 con 0x555e262676a0 2023-09-10T01:08:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd06dffb700 10 client.27041 handle_command_reply: tid=0 2023-09-10T01:08:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 client.27041 shutdown 2023-09-10T01:08:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624e030 msgr2=0x555e26258760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624e030 0x555e26258760 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd07c012e90 tx=0x7fd07c00f6a0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 shutdown_connections 2023-09-10T01:08:01.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262676a0 0x555e26269ac0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e262632a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd06dffb700 0 client.27041 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:01.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624d710 0x555e26258220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/2568116482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624e030 0x555e26258760 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 >> 172.21.15.3:0/2568116482 conn(0x555e261c2090 msgr2=0x555e26248780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 shutdown_connections 2023-09-10T01:08:01.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.846+0000 7fd0a0792b80 1 -- 172.21.15.3:0/2568116482 wait complete. 2023-09-10T01:08:01.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.847+0000 7fd0a0792b80 20 client.27041 trim_cache size 0 max 16384 2023-09-10T01:08:01.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 Processor -- start 2023-09-10T01:08:01.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 -- start start 2023-09-10T01:08:01.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624d710 0x555e26262990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e26262ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e26254260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e260f74f0 con 0x555e2624e030 2023-09-10T01:08:01.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e26257670 con 0x555e2624d710 2023-09-10T01:08:01.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.861+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e26258ab0 con 0x555e26257e40 2023-09-10T01:08:01.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd08659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624d710 0x555e26262990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd08659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624d710 0x555e26262990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57686/0 (socket says 172.21.15.3:57686) 2023-09-10T01:08:01.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd086d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e26254260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd08659c700 1 -- 172.21.15.3:0/3326815709 learned_addr learned my addr 172.21.15.3:0/3326815709 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd085d9b700 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e26262ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd086d9d700 1 -- 172.21.15.3:0/3326815709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624d710 msgr2=0x555e26262990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd086d9d700 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624d710 0x555e26262990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd086d9d700 1 -- 172.21.15.3:0/3326815709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 msgr2=0x555e26262ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd086d9d700 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e26262ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd086d9d700 1 -- 172.21.15.3:0/3326815709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e25fb4c60 con 0x555e26257e40 2023-09-10T01:08:01.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd08659c700 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624d710 0x555e26262990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:01.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.862+0000 7fd086d9d700 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e26254260 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd08000fc90 tx=0x7fd08001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e4d2ddc80c7da71 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0757fa700 1 -- 172.21.15.3:0/3326815709 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08000e040 con 0x555e26257e40 2023-09-10T01:08:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0757fa700 1 -- 172.21.15.3:0/3326815709 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd08000f040 con 0x555e26257e40 2023-09-10T01:08:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0757fa700 1 -- 172.21.15.3:0/3326815709 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080004280 con 0x555e26257e40 2023-09-10T01:08:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3326815709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 msgr2=0x555e26254260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e26254260 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd08000fc90 tx=0x7fd08001c000 comp rx=0 tx=0).stop 2023-09-10T01:08:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3326815709 shutdown_connections 2023-09-10T01:08:01.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e26257e40 0x555e26254260 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e2624d710 0x555e26262990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3326815709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e26262ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3326815709 >> 172.21.15.3:0/3326815709 conn(0x555e261c2090 msgr2=0x555e262522d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.863+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3326815709 shutdown_connections 2023-09-10T01:08:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.864+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3326815709 wait complete. 2023-09-10T01:08:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.864+0000 7fd0a0792b80 1 -- start start 2023-09-10T01:08:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.864+0000 7fd0a0792b80 1 Processor -- start 2023-09-10T01:08:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624d710 0x555e2625cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e2625b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd0a0792b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2625bd90 con 0x555e2624e030 2023-09-10T01:08:01.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2625bf00 con 0x555e26257e40 2023-09-10T01:08:01.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd0a0792b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2625c0c0 con 0x555e2624d710 2023-09-10T01:08:01.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd0a0792b80 10 client.0 _ll_register_callbacks cb 0x7fd07c012980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:01.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e2625b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e2625b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:48880/0 (socket says 172.21.15.3:48880) 2023-09-10T01:08:01.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 -- 172.21.15.3:0/3516719932 learned_addr learned my addr 172.21.15.3:0/3516719932 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:01.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd08659c700 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624d710 0x555e2625cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd086d9d700 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 -- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624d710 msgr2=0x555e2625cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624d710 0x555e2625cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 -- 172.21.15.3:0/3516719932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 msgr2=0x555e2625b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.865+0000 7fd085d9b700 1 -- 172.21.15.3:0/3516719932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e25fcbde0 con 0x555e2624e030 2023-09-10T01:08:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd085d9b700 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e2625b310 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x555e26257590 tx=0x7fd07c01bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f7b75e9072b3d4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd06dffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:08:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd06dffb700 1 -- 172.21.15.3:0/3516719932 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c017070 con 0x555e2624e030 2023-09-10T01:08:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd07809a2b0 con 0x555e2624e030 2023-09-10T01:08:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd06dffb700 1 -- 172.21.15.3:0/3516719932 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fd07c000e40 con 0x555e2624e030 2023-09-10T01:08:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd06dffb700 1 -- 172.21.15.3:0/3516719932 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd07c009380 con 0x555e2624e030 2023-09-10T01:08:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd0a0792b80 10 client.16977 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.866+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fd0800195f0 con 0x555e2624e030 2023-09-10T01:08:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd06dffb700 1 -- 172.21.15.3:0/3516719932 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd07809a2b0 con 0x555e2624e030 2023-09-10T01:08:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd06dffb700 1 -- 172.21.15.3:0/3516719932 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd07c003b40 con 0x555e2624e030 2023-09-10T01:08:01.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd0a0792b80 10 client.16977 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd0a0792b80 10 client.16977 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:08:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd0a0792b80 20 client.16977 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262bca10 0x555e262bee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd0a0792b80 4 client.16977 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:08:01.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e26198920 con 0x555e262bca10 2023-09-10T01:08:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.867+0000 7fd08659c700 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262bca10 0x555e262bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd08659c700 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262bca10 0x555e262bee30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=625fb746ea75bb07 server_cookie=b57cd7175829faa5 in_seq=0 out_seq=0 2023-09-10T01:08:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd06dffb700 10 client.16977 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:01.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd06dffb700 1 -- 172.21.15.3:0/3516719932 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e26198920 con 0x555e262bca10 2023-09-10T01:08:01.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd06dffb700 10 client.16977 handle_command_reply: tid=0 2023-09-10T01:08:01.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 client.16977 shutdown 2023-09-10T01:08:01.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 msgr2=0x555e2625b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e2625b310 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x555e26257590 tx=0x7fd07c01bbf0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 shutdown_connections 2023-09-10T01:08:01.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x555e262bca10 0x555e262bee30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e2624d710 0x555e2625cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e26257e40 0x555e2625b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 --2- 172.21.15.3:0/3516719932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e2624e030 0x555e2625b310 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd06dffb700 0 client.16977 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:01.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 >> 172.21.15.3:0/3516719932 conn(0x555e261c2090 msgr2=0x555e262557a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 shutdown_connections 2023-09-10T01:08:01.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.868+0000 7fd0a0792b80 1 -- 172.21.15.3:0/3516719932 wait complete. 2023-09-10T01:08:01.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd0a0792b80 20 client.16977 trim_cache size 0 max 16384 2023-09-10T01:08:01.886 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:08:01.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0700a07b0 msgr2=0x7fd0700a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0700a07b0 0x7fd0700a2c70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd08000f540 tx=0x7fd080009920 comp rx=0 tx=0).stop 2023-09-10T01:08:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088100640 msgr2=0x7fd088072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088100640 0x7fd088072900 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd078009230 tx=0x7fd07800fa00 comp rx=0 tx=0).stop 2023-09-10T01:08:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 shutdown_connections 2023-09-10T01:08:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0700a07b0 0x7fd0700a2c70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd088106650 0x7fd088073380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd088100640 0x7fd088072900 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 --2- 172.21.15.3:0/2255897519 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd088101000 0x7fd088072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 >> 172.21.15.3:0/2255897519 conn(0x7fd0880fc170 msgr2=0x7fd088104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 shutdown_connections 2023-09-10T01:08:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:01.869+0000 7fd08cdff700 1 -- 172.21.15.3:0/2255897519 wait complete. 2023-09-10T01:08:01.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:08:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 -- 172.21.15.3:0/2396287991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 msgr2=0x7f74b010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 --2- 172.21.15.3:0/2396287991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 0x7f74b010d510 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f74a800b0a0 tx=0x7f74a800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 -- 172.21.15.3:0/2396287991 shutdown_connections 2023-09-10T01:08:02.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 --2- 172.21.15.3:0/2396287991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 0x7f74b010d510 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 --2- 172.21.15.3:0/2396287991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b0106650 0x7f74b0106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 --2- 172.21.15.3:0/2396287991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0100640 0x7f74b0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 -- 172.21.15.3:0/2396287991 >> 172.21.15.3:0/2396287991 conn(0x7f74b00fc170 msgr2=0x7f74b00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 -- 172.21.15.3:0/2396287991 shutdown_connections 2023-09-10T01:08:02.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.109+0000 7f74b71ff700 1 -- 172.21.15.3:0/2396287991 wait complete. 2023-09-10T01:08:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 Processor -- start 2023-09-10T01:08:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 -- start start 2023-09-10T01:08:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b0100640 0x7f74b019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 0x7f74b019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0106650 0x7f74b01986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0198d20 con 0x7f74b0100640 2023-09-10T01:08:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0198e90 con 0x7f74b0101000 2023-09-10T01:08:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.110+0000 7f74b71ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0199050 con 0x7f74b0106650 2023-09-10T01:08:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0106650 0x7f74b01986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0106650 0x7f74b01986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43900/0 (socket says 172.21.15.3:43900) 2023-09-10T01:08:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 -- 172.21.15.3:0/954298827 learned_addr learned my addr 172.21.15.3:0/954298827 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74affff700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 0x7f74b019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b0100640 0x7f74b019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 -- 172.21.15.3:0/954298827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 msgr2=0x7f74b019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 0x7f74b019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 -- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b0100640 msgr2=0x7f74b019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b0100640 0x7f74b019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 -- 172.21.15.3:0/954298827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8009d00 con 0x7f74b0106650 2023-09-10T01:08:02.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.111+0000 7f74b579c700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0106650 0x7f74b01986b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009850 tx=0x7f74a80077f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.112+0000 7f74adffb700 1 -- 172.21.15.3:0/954298827 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a800e040 con 0x7f74b0106650 2023-09-10T01:08:02.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.112+0000 7f74adffb700 1 -- 172.21.15.3:0/954298827 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74a80079e0 con 0x7f74b0106650 2023-09-10T01:08:02.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.112+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b01992d0 con 0x7f74b0106650 2023-09-10T01:08:02.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.112+0000 7f74adffb700 1 -- 172.21.15.3:0/954298827 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a80052a0 con 0x7f74b0106650 2023-09-10T01:08:02.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.112+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b01a5630 con 0x7f74b0106650 2023-09-10T01:08:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.116+0000 7f74adffb700 1 -- 172.21.15.3:0/954298827 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f74a801d470 con 0x7f74b0106650 2023-09-10T01:08:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.117+0000 7f74cab95b80 1 Processor -- start 2023-09-10T01:08:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74cab95b80 1 -- start start 2023-09-10T01:08:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869279710 0x558869279af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x558869283900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869286230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74cab95b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588691d2730 con 0x558869283e40 2023-09-10T01:08:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74cab95b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588691d0050 con 0x558869279710 2023-09-10T01:08:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74cab95b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588691cb940 con 0x55886927a030 2023-09-10T01:08:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74adffb700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a00a07b0 0x7f74a00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x558869283900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74affff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869279710 0x558869279af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74adffb700 1 -- 172.21.15.3:0/954298827 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f74a80cd6e0 con 0x7f74b0106650 2023-09-10T01:08:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x558869283900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43906/0 (socket says 172.21.15.3:43906) 2023-09-10T01:08:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74b4f9b700 1 -- 172.21.15.3:0/3742624215 learned_addr learned my addr 172.21.15.3:0/3742624215 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74affff700 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869286230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.118+0000 7f74b579c700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a00a07b0 0x7f74a00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f74affff700 1 -- 172.21.15.3:0/3742624215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 msgr2=0x558869283900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f74affff700 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x558869283900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f74affff700 1 -- 172.21.15.3:0/3742624215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 msgr2=0x558869286230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f74affff700 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869286230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f74affff700 1 -- 172.21.15.3:0/3742624215 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558868ff7de0 con 0x558869279710 2023-09-10T01:08:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f74b579c700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a00a07b0 0x7f74a00a2c70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f74a800b780 tx=0x7f74a800b3b0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f74affff700 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869279710 0x558869279af0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f74b0105ac0 tx=0x7f749c009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7cce7a37bc77014 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f749affd700 1 -- 172.21.15.3:0/3742624215 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c004050 con 0x558869279710 2023-09-10T01:08:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f749affd700 1 -- 172.21.15.3:0/3742624215 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f749c011e90 con 0x558869279710 2023-09-10T01:08:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.119+0000 7f749affd700 1 -- 172.21.15.3:0/3742624215 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c005280 con 0x558869279710 2023-09-10T01:08:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 -- 172.21.15.3:0/3742624215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869279710 msgr2=0x558869279af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869279710 0x558869279af0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f74b0105ac0 tx=0x7f749c009750 comp rx=0 tx=0).stop 2023-09-10T01:08:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 -- 172.21.15.3:0/3742624215 shutdown_connections 2023-09-10T01:08:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869286230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869279710 0x558869279af0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 --2- 172.21.15.3:0/3742624215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x558869283900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 -- 172.21.15.3:0/3742624215 >> 172.21.15.3:0/3742624215 conn(0x5588691ee090 msgr2=0x558869274bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 -- 172.21.15.3:0/3742624215 shutdown_connections 2023-09-10T01:08:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.120+0000 7f74cab95b80 1 -- 172.21.15.3:0/3742624215 wait complete. 2023-09-10T01:08:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.121+0000 7f74cab95b80 1 -- start start 2023-09-10T01:08:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.121+0000 7f74cab95b80 1 Processor -- start 2023-09-10T01:08:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x5588692838d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869280ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869283e40 0x558869281220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74cab95b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558869169b70 con 0x558869279710 2023-09-10T01:08:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74cab95b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558869169870 con 0x55886927a030 2023-09-10T01:08:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74cab95b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55886904d920 con 0x558869283e40 2023-09-10T01:08:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74cab95b80 10 client.0 _ll_register_callbacks cb 0x5588690b3a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869280ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74affff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x5588692838d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869280ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57748/0 (socket says 172.21.15.3:57748) 2023-09-10T01:08:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 -- 172.21.15.3:0/794066638 learned_addr learned my addr 172.21.15.3:0/794066638 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74affff700 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869283e40 0x558869281220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 -- 172.21.15.3:0/794066638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869283e40 msgr2=0x558869281220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869283e40 0x558869281220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 -- 172.21.15.3:0/794066638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 msgr2=0x5588692838d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x5588692838d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74b4f9b700 1 -- 172.21.15.3:0/794066638 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c019040 con 0x55886927a030 2023-09-10T01:08:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.122+0000 7f74affff700 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869283e40 0x558869281220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:08:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869280ae0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f74a4011830 tx=0x7f74a4011b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fda12d554acdb183 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f748ffff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f748ffff700 1 -- 172.21.15.3:0/794066638 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4016070 con 0x55886927a030 2023-09-10T01:08:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588691234f0 con 0x55886927a030 2023-09-10T01:08:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f748ffff700 1 -- 172.21.15.3:0/794066638 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f74a4004320 con 0x55886927a030 2023-09-10T01:08:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f748ffff700 1 -- 172.21.15.3:0/794066638 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a40044e0 con 0x55886927a030 2023-09-10T01:08:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f74cab95b80 10 client.27059 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.123+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558868fe0c60 con 0x55886927a030 2023-09-10T01:08:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f748ffff700 1 -- 172.21.15.3:0/794066638 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5588691234f0 con 0x55886927a030 2023-09-10T01:08:02.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f748ffff700 1 -- 172.21.15.3:0/794066638 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f74a400f990 con 0x55886927a030 2023-09-10T01:08:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f74cab95b80 10 client.27059 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f74cab95b80 10 client.27059 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:08:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f74cab95b80 20 client.27059 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f74cab95b80 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x558869293a20 0x558869295e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f74cab95b80 4 client.27059 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:02.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588691e08a0 con 0x558869293a20 2023-09-10T01:08:02.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.124+0000 7f74affff700 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x558869293a20 0x558869295e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.125+0000 7f748ffff700 10 client.27059 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:02.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.125+0000 7f74affff700 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x558869293a20 0x558869295e40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7592fdd8b4652c93 server_cookie=5d48a184a1a725bb in_seq=0 out_seq=0 2023-09-10T01:08:02.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.125+0000 7f748ffff700 1 -- 172.21.15.3:0/794066638 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5588691e08a0 con 0x558869293a20 2023-09-10T01:08:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.125+0000 7f748ffff700 10 client.27059 handle_command_reply: tid=0 2023-09-10T01:08:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.125+0000 7f74cab95b80 1 client.27059 shutdown 2023-09-10T01:08:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.125+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 msgr2=0x558869280ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.125+0000 7f74cab95b80 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869280ae0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f74a4011830 tx=0x7f74a4011b40 comp rx=0 tx=0).stop 2023-09-10T01:08:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 shutdown_connections 2023-09-10T01:08:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x558869293a20 0x558869295e40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869283e40 0x558869281220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f748ffff700 0 client.27059 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x5588692838d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 --2- 172.21.15.3:0/794066638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869280ae0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 >> 172.21.15.3:0/794066638 conn(0x5588691ee090 msgr2=0x5588692747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 shutdown_connections 2023-09-10T01:08:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 1 -- 172.21.15.3:0/794066638 wait complete. 2023-09-10T01:08:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.126+0000 7f74cab95b80 20 client.27059 trim_cache size 0 max 16384 2023-09-10T01:08:02.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.139+0000 7f74cab95b80 1 Processor -- start 2023-09-10T01:08:02.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.139+0000 7f74cab95b80 1 -- start start 2023-09-10T01:08:02.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x558869283400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x55886927bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869283e40 0x55886927c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74cab95b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558869291070 con 0x558869279710 2023-09-10T01:08:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74cab95b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55886927b890 con 0x558869283e40 2023-09-10T01:08:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74cab95b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55886929ab60 con 0x55886927a030 2023-09-10T01:08:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x55886927bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x55886927bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43926/0 (socket says 172.21.15.3:43926) 2023-09-10T01:08:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 -- 172.21.15.3:0/583776910 learned_addr learned my addr 172.21.15.3:0/583776910 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74affff700 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x558869283400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74affff700 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869283e40 0x55886927c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 -- 172.21.15.3:0/583776910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869283e40 msgr2=0x55886927c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869283e40 0x55886927c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 -- 172.21.15.3:0/583776910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 msgr2=0x558869283400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x558869283400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74b4f9b700 1 -- 172.21.15.3:0/583776910 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55886901e250 con 0x55886927a030 2023-09-10T01:08:02.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.140+0000 7f74affff700 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x558869283400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:02.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.141+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x55886927bdc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f74a4004020 tx=0x7f74a4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a235c41066e19033 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.141+0000 7f749b7fe700 1 -- 172.21.15.3:0/583776910 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4016070 con 0x55886927a030 2023-09-10T01:08:02.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.141+0000 7f749b7fe700 1 -- 172.21.15.3:0/583776910 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74a400fe90 con 0x55886927a030 2023-09-10T01:08:02.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.141+0000 7f749b7fe700 1 -- 172.21.15.3:0/583776910 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a40038e0 con 0x55886927a030 2023-09-10T01:08:02.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 -- 172.21.15.3:0/583776910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 msgr2=0x55886927bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x55886927bdc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f74a4004020 tx=0x7f74a4000bc0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 -- 172.21.15.3:0/583776910 shutdown_connections 2023-09-10T01:08:02.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558869283e40 0x55886927c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869279710 0x558869283400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 --2- 172.21.15.3:0/583776910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55886927a030 0x55886927bdc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 -- 172.21.15.3:0/583776910 >> 172.21.15.3:0/583776910 conn(0x5588691ee090 msgr2=0x55886928fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 -- 172.21.15.3:0/583776910 shutdown_connections 2023-09-10T01:08:02.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 -- 172.21.15.3:0/583776910 wait complete. 2023-09-10T01:08:02.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.142+0000 7f74cab95b80 1 -- start start 2023-09-10T01:08:02.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 1 Processor -- start 2023-09-10T01:08:02.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869279710 0x55886928a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869288be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869289120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558869289700 con 0x558869283e40 2023-09-10T01:08:02.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558869289870 con 0x55886927a030 2023-09-10T01:08:02.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558869289a30 con 0x558869279710 2023-09-10T01:08:02.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74cab95b80 10 client.0 _ll_register_callbacks cb 0x7f74a4009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:02.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.143+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869288be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869288be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57780/0 (socket says 172.21.15.3:57780) 2023-09-10T01:08:02.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 -- 172.21.15.3:0/1892649624 learned_addr learned my addr 172.21.15.3:0/1892649624 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74affff700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869279710 0x55886928a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74affff700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869289120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 -- 172.21.15.3:0/1892649624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869279710 msgr2=0x55886928a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869279710 0x55886928a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 -- 172.21.15.3:0/1892649624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 msgr2=0x558869289120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869289120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 -- 172.21.15.3:0/1892649624 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558868fe0c60 con 0x55886927a030 2023-09-10T01:08:02.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74affff700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869289120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:08:02.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f74b4f9b700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869288be0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f74a4011b80 tx=0x7f74a400f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=213fa06fab7e1422 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.144+0000 7f748ffff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:02.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f748ffff700 1 -- 172.21.15.3:0/1892649624 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4016040 con 0x55886927a030 2023-09-10T01:08:02.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f748ffff700 1 -- 172.21.15.3:0/1892649624 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f74a4000e40 con 0x55886927a030 2023-09-10T01:08:02.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f749c011e90 con 0x55886927a030 2023-09-10T01:08:02.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f748ffff700 1 -- 172.21.15.3:0/1892649624 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a400fb70 con 0x55886927a030 2023-09-10T01:08:02.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 10 client.27065 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:02.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558868ff62b0 con 0x55886927a030 2023-09-10T01:08:02.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f748ffff700 1 -- 172.21.15.3:0/1892649624 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f749c011e90 con 0x55886927a030 2023-09-10T01:08:02.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f748ffff700 1 -- 172.21.15.3:0/1892649624 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f74a4011860 con 0x55886927a030 2023-09-10T01:08:02.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 10 client.27065 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:02.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 10 client.27065 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:08:02.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 20 client.27065 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:02.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5588692e8b20 0x5588692eaf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 4 client.27065 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:08:02.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.145+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5588691c4920 con 0x5588692e8b20 2023-09-10T01:08:02.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.146+0000 7f74affff700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5588692e8b20 0x5588692eaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.146+0000 7f74affff700 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5588692e8b20 0x5588692eaf10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=abc413a37cbd1524 server_cookie=3d14f461e11d8c48 in_seq=0 out_seq=0 2023-09-10T01:08:02.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.146+0000 7f748ffff700 10 client.27065 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:02.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f748ffff700 1 -- 172.21.15.3:0/1892649624 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5588691c4920 con 0x5588692e8b20 2023-09-10T01:08:02.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f748ffff700 10 client.27065 handle_command_reply: tid=0 2023-09-10T01:08:02.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 client.27065 shutdown 2023-09-10T01:08:02.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 msgr2=0x558869288be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869288be0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f74a4011b80 tx=0x7f74a400f7f0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 shutdown_connections 2023-09-10T01:08:02.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5588692e8b20 0x5588692eaf10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558869279710 0x55886928a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558869283e40 0x558869289120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f748ffff700 0 client.27065 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:02.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 --2- 172.21.15.3:0/1892649624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55886927a030 0x558869288be0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 >> 172.21.15.3:0/1892649624 conn(0x5588691ee090 msgr2=0x55886927bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 shutdown_connections 2023-09-10T01:08:02.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.148+0000 7f74cab95b80 1 -- 172.21.15.3:0/1892649624 wait complete. 2023-09-10T01:08:02.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74cab95b80 20 client.27065 trim_cache size 0 max 16384 2023-09-10T01:08:02.165 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:08:02.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a00a07b0 msgr2=0x7f74a00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a00a07b0 0x7f74a00a2c70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f74a800b780 tx=0x7f74a800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0106650 msgr2=0x7f74b01986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0106650 0x7f74b01986b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009850 tx=0x7f74a80077f0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 shutdown_connections 2023-09-10T01:08:02.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a00a07b0 0x7f74a00a2c70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b0106650 0x7f74b01986b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b0100640 0x7f74b019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 --2- 172.21.15.3:0/954298827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b0101000 0x7f74b019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 >> 172.21.15.3:0/954298827 conn(0x7f74b00fc170 msgr2=0x7f74b010ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 shutdown_connections 2023-09-10T01:08:02.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.149+0000 7f74b71ff700 1 -- 172.21.15.3:0/954298827 wait complete. 2023-09-10T01:08:02.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:08:02.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 -- 172.21.15.3:0/999627366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc0729a0 msgr2=0x7fe9fc072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 --2- 172.21.15.3:0/999627366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc0729a0 0x7fe9fc072d80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009990 tx=0x7fe9ec009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 -- 172.21.15.3:0/999627366 shutdown_connections 2023-09-10T01:08:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 --2- 172.21.15.3:0/999627366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9fc111650 0x7fe9fc113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 --2- 172.21.15.3:0/999627366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc0729a0 0x7fe9fc072d80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 --2- 172.21.15.3:0/999627366 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9fc0732c0 0x7fe9fc111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 -- 172.21.15.3:0/999627366 >> 172.21.15.3:0/999627366 conn(0x7fe9fc0fc170 msgr2=0x7fe9fc0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 -- 172.21.15.3:0/999627366 shutdown_connections 2023-09-10T01:08:02.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.389+0000 7fea00bff700 1 -- 172.21.15.3:0/999627366 wait complete. 2023-09-10T01:08:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.390+0000 7fea00bff700 1 Processor -- start 2023-09-10T01:08:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.390+0000 7fea00bff700 1 -- start start 2023-09-10T01:08:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fea00bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9fc0729a0 0x7fe9fc19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fea00bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9fc0732c0 0x7fe9fc19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fea00bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc111650 0x7fe9fc1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fea00bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc1a1d70 con 0x7fe9fc0729a0 2023-09-10T01:08:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fea00bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc1a1ee0 con 0x7fe9fc111650 2023-09-10T01:08:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fea00bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc1a20a0 con 0x7fe9fc0732c0 2023-09-10T01:08:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9fc0729a0 0x7fe9fc19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9f9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9fc0732c0 0x7fe9fc19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fad9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc111650 0x7fe9fc1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9f9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9fc0732c0 0x7fe9fc19d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43948/0 (socket says 172.21.15.3:43948) 2023-09-10T01:08:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fad9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc111650 0x7fe9fc1a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57798/0 (socket says 172.21.15.3:57798) 2023-09-10T01:08:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9f9d9b700 1 -- 172.21.15.3:0/297497823 learned_addr learned my addr 172.21.15.3:0/297497823 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fad9d700 1 -- 172.21.15.3:0/297497823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9fc0732c0 msgr2=0x7fe9fc19d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9fc0732c0 0x7fe9fc19d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fad9d700 1 -- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9fc0729a0 msgr2=0x7fe9fc19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9fc0729a0 0x7fe9fc19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.391+0000 7fe9fad9d700 1 -- 172.21.15.3:0/297497823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec0095f0 con 0x7fe9fc111650 2023-09-10T01:08:02.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.392+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc111650 0x7fe9fc1a1730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4007ee0 tx=0x7fe9f400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.392+0000 7fe9eb7fe700 1 -- 172.21.15.3:0/297497823 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4009ea0 con 0x7fe9fc111650 2023-09-10T01:08:02.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.392+0000 7fe9eb7fe700 1 -- 172.21.15.3:0/297497823 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9f400f040 con 0x7fe9fc111650 2023-09-10T01:08:02.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.392+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc1a2380 con 0x7fe9fc111650 2023-09-10T01:08:02.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.392+0000 7fe9eb7fe700 1 -- 172.21.15.3:0/297497823 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4004280 con 0x7fe9fc111650 2023-09-10T01:08:02.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.392+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc1a9bc0 con 0x7fe9fc111650 2023-09-10T01:08:02.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.395+0000 7fe9eb7fe700 1 -- 172.21.15.3:0/297497823 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe9f400db20 con 0x7fe9fc111650 2023-09-10T01:08:02.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.396+0000 7fea14472b80 1 Processor -- start 2023-09-10T01:08:02.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.396+0000 7fe9eb7fe700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe9e40a0830 0x7fe9e40a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fe9eb7fe700 1 -- 172.21.15.3:0/297497823 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe9f40cbb30 con 0x7fe9fc111650 2023-09-10T01:08:02.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fea14472b80 1 -- start start 2023-09-10T01:08:02.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe9e40a0830 0x7fe9e40a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fea14472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b5f710 0x55bed1b5faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fea14472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b60030 0x55bed1b69900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fea14472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b6c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fea14472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1ab8730 con 0x55bed1b5f710 2023-09-10T01:08:02.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fea14472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1ab6050 con 0x55bed1b69e40 2023-09-10T01:08:02.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fea14472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1ab1940 con 0x55bed1b60030 2023-09-10T01:08:02.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fe9f9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b60030 0x55bed1b69900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b5f710 0x55bed1b5faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fe9f9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b60030 0x55bed1b69900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43956/0 (socket says 172.21.15.3:43956) 2023-09-10T01:08:02.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.397+0000 7fe9f9d9b700 1 -- 172.21.15.3:0/3322654484 learned_addr learned my addr 172.21.15.3:0/3322654484 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b6c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9f9d9b700 1 -- 172.21.15.3:0/3322654484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 msgr2=0x55bed1b6c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b6c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9f9d9b700 1 -- 172.21.15.3:0/3322654484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b5f710 msgr2=0x55bed1b5faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b5f710 0x55bed1b5faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9f9d9b700 1 -- 172.21.15.3:0/3322654484 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bed18ddde0 con 0x55bed1b60030 2023-09-10T01:08:02.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe9e40a0830 0x7fe9e40a2cf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0011b80 tx=0x7fe9f0011670 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b60030 0x55bed1b69900 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000fad0 tx=0x7fe9f001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c734e1cda4f22df server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.398+0000 7fe9e8ff9700 1 -- 172.21.15.3:0/3322654484 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0011810 con 0x55bed1b60030 2023-09-10T01:08:02.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fe9e8ff9700 1 -- 172.21.15.3:0/3322654484 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9f0018070 con 0x55bed1b60030 2023-09-10T01:08:02.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fe9e8ff9700 1 -- 172.21.15.3:0/3322654484 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f000e6c0 con 0x55bed1b60030 2023-09-10T01:08:02.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 -- 172.21.15.3:0/3322654484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b60030 msgr2=0x55bed1b69900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b60030 0x55bed1b69900 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000fad0 tx=0x7fe9f001d000 comp rx=0 tx=0).stop 2023-09-10T01:08:02.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 -- 172.21.15.3:0/3322654484 shutdown_connections 2023-09-10T01:08:02.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b6c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b5f710 0x55bed1b5faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 --2- 172.21.15.3:0/3322654484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b60030 0x55bed1b69900 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 -- 172.21.15.3:0/3322654484 >> 172.21.15.3:0/3322654484 conn(0x55bed1ad4090 msgr2=0x55bed1b5abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 -- 172.21.15.3:0/3322654484 shutdown_connections 2023-09-10T01:08:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.399+0000 7fea14472b80 1 -- 172.21.15.3:0/3322654484 wait complete. 2023-09-10T01:08:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.400+0000 7fea14472b80 1 -- start start 2023-09-10T01:08:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.400+0000 7fea14472b80 1 Processor -- start 2023-09-10T01:08:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fea14472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fea14472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b60030 0x55bed1b66b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fea14472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b69e40 0x55bed1b671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fea14472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1a4fb70 con 0x55bed1b69e40 2023-09-10T01:08:02.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fea14472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1a4f870 con 0x55bed1b60030 2023-09-10T01:08:02.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fea14472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1933920 con 0x55bed1b5f710 2023-09-10T01:08:02.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fea14472b80 10 client.0 _ll_register_callbacks cb 0x55bed1999a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:02.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43972/0 (socket says 172.21.15.3:43972) 2023-09-10T01:08:02.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2892912513 learned_addr learned my addr 172.21.15.3:0/2892912513 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b69e40 0x55bed1b671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b60030 0x55bed1b66b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b60030 msgr2=0x55bed1b66b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b60030 0x55bed1b66b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2892912513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b69e40 msgr2=0x55bed1b671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.401+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b69e40 0x55bed1b671f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2892912513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f000f890 con 0x55bed1b5f710 2023-09-10T01:08:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b69e40 0x55bed1b671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69860 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec000c00 tx=0x7fe9ec00bfc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f53f6f2e5a40c9e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fe9e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/2892912513 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec01d070 con 0x55bed1b5f710 2023-09-10T01:08:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bed1a094f0 con 0x55bed1b5f710 2023-09-10T01:08:02.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/2892912513 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe9ec004700 con 0x55bed1b5f710 2023-09-10T01:08:02.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/2892912513 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9ec004900 con 0x55bed1b5f710 2023-09-10T01:08:02.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fea14472b80 10 client.26992 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:02.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.402+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55bed18c6c60 con 0x55bed1b5f710 2023-09-10T01:08:02.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/2892912513 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bed1a094f0 con 0x55bed1b5f710 2023-09-10T01:08:02.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/2892912513 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe9ec00fa60 con 0x55bed1b5f710 2023-09-10T01:08:02.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fea14472b80 10 client.26992 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:02.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fea14472b80 10 client.26992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:02.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fea14472b80 20 client.26992 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:02.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fea14472b80 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1b798c0 0x55bed1b7bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fea14472b80 4 client.26992 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:02.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bed1ac68a0 con 0x55bed1b798c0 2023-09-10T01:08:02.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.403+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1b798c0 0x55bed1b7bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.404+0000 7fe9e1ffb700 10 client.26992 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.404+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1b798c0 0x55bed1b7bce0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b0fc95046aed7f server_cookie=76a1eb85a65568ab in_seq=0 out_seq=0 2023-09-10T01:08:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/2892912513 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bed1ac68a0 con 0x55bed1b798c0 2023-09-10T01:08:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fe9e1ffb700 10 client.26992 handle_command_reply: tid=0 2023-09-10T01:08:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 client.26992 shutdown 2023-09-10T01:08:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 msgr2=0x55bed1b69860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69860 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec000c00 tx=0x7fe9ec00bfc0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 shutdown_connections 2023-09-10T01:08:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1b798c0 0x55bed1b7bce0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69860 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b69e40 0x55bed1b671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fe9e1ffb700 0 client.26992 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 --2- 172.21.15.3:0/2892912513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b60030 0x55bed1b66b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 >> 172.21.15.3:0/2892912513 conn(0x55bed1ad4090 msgr2=0x55bed1b5a780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 shutdown_connections 2023-09-10T01:08:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.405+0000 7fea14472b80 1 -- 172.21.15.3:0/2892912513 wait complete. 2023-09-10T01:08:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.406+0000 7fea14472b80 20 client.26992 trim_cache size 0 max 16384 2023-09-10T01:08:02.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 Processor -- start 2023-09-10T01:08:02.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 -- start start 2023-09-10T01:08:02.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b61e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1b606b0 con 0x55bed1b60030 2023-09-10T01:08:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1b5c250 con 0x55bed1b69e40 2023-09-10T01:08:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fea14472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1b68110 con 0x55bed1b5f710 2023-09-10T01:08:02.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43976/0 (socket says 172.21.15.3:43976) 2023-09-10T01:08:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2207193080 learned_addr learned my addr 172.21.15.3:0/2207193080 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.420+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b61e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2207193080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 msgr2=0x55bed1b623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2207193080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 msgr2=0x55bed1b61e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b61e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9fa59c700 1 -- 172.21.15.3:0/2207193080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bed1904250 con 0x55bed1b5f710 2023-09-10T01:08:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec005f50 tx=0x7fe9ec009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f4796270c92b48 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9e97fa700 1 -- 172.21.15.3:0/2207193080 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec01d070 con 0x55bed1b5f710 2023-09-10T01:08:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9e97fa700 1 -- 172.21.15.3:0/2207193080 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9ec01e040 con 0x55bed1b5f710 2023-09-10T01:08:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.421+0000 7fe9e97fa700 1 -- 172.21.15.3:0/2207193080 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec00f900 con 0x55bed1b5f710 2023-09-10T01:08:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 -- 172.21.15.3:0/2207193080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 msgr2=0x55bed1b69400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec005f50 tx=0x7fe9ec009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 -- 172.21.15.3:0/2207193080 shutdown_connections 2023-09-10T01:08:02.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b5f710 0x55bed1b69400 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b69e40 0x55bed1b623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 --2- 172.21.15.3:0/2207193080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b61e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 -- 172.21.15.3:0/2207193080 >> 172.21.15.3:0/2207193080 conn(0x55bed1ad4090 msgr2=0x55bed1b75c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 -- 172.21.15.3:0/2207193080 shutdown_connections 2023-09-10T01:08:02.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 -- 172.21.15.3:0/2207193080 wait complete. 2023-09-10T01:08:02.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.422+0000 7fea14472b80 1 -- start start 2023-09-10T01:08:02.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 1 Processor -- start 2023-09-10T01:08:02.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b5f710 0x55bed1b70250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b70790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b69e40 0x55bed1b6e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1b6ecf0 con 0x55bed1b60030 2023-09-10T01:08:02.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1b6ee60 con 0x55bed1b5f710 2023-09-10T01:08:02.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed1b6f020 con 0x55bed1b69e40 2023-09-10T01:08:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fea14472b80 10 client.0 _ll_register_callbacks cb 0x7fe9ec00bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.423+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b5f710 0x55bed1b70250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b5f710 0x55bed1b70250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57838/0 (socket says 172.21.15.3:57838) 2023-09-10T01:08:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 -- 172.21.15.3:0/3597783408 learned_addr learned my addr 172.21.15.3:0/3597783408 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:02.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b70790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fad9d700 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b69e40 0x55bed1b6e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 -- 172.21.15.3:0/3597783408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b69e40 msgr2=0x55bed1b6e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b69e40 0x55bed1b6e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 -- 172.21.15.3:0/3597783408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 msgr2=0x55bed1b70790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b70790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 -- 172.21.15.3:0/3597783408 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bed18c6c60 con 0x55bed1b5f710 2023-09-10T01:08:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9fa59c700 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b5f710 0x55bed1b70250 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0098e0 tx=0x7fe9ec024750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71aa3cbc784bc0d3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9e1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/3597783408 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec01d070 con 0x55bed1b5f710 2023-09-10T01:08:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9f0018070 con 0x55bed1b5f710 2023-09-10T01:08:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.424+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/3597783408 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fe9ec000e40 con 0x55bed1b5f710 2023-09-10T01:08:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/3597783408 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9ec024cc0 con 0x55bed1b5f710 2023-09-10T01:08:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 10 client.27083 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55bed18dc2b0 con 0x55bed1b5f710 2023-09-10T01:08:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/3597783408 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9f0018070 con 0x55bed1b5f710 2023-09-10T01:08:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/3597783408 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe9ec0046a0 con 0x55bed1b5f710 2023-09-10T01:08:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 10 client.27083 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 10 client.27083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 20 client.27083 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1bce2f0 0x55bed1bd06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 4 client.27083 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:08:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bed1aaa920 con 0x55bed1bce2f0 2023-09-10T01:08:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.425+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1bce2f0 0x55bed1bd06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:02.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fe9e1ffb700 10 client.27083 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:02.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fe9f9d9b700 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1bce2f0 0x55bed1bd06e0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4173e26b3895651 server_cookie=41b7890aea2f468a in_seq=0 out_seq=0 2023-09-10T01:08:02.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fe9e1ffb700 1 -- 172.21.15.3:0/3597783408 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bed1aaa920 con 0x55bed1bce2f0 2023-09-10T01:08:02.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fe9e1ffb700 10 client.27083 handle_command_reply: tid=0 2023-09-10T01:08:02.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 client.27083 shutdown 2023-09-10T01:08:02.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b5f710 msgr2=0x55bed1b70250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b5f710 0x55bed1b70250 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec0098e0 tx=0x7fe9ec024750 comp rx=0 tx=0).stop 2023-09-10T01:08:02.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 shutdown_connections 2023-09-10T01:08:02.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55bed1bce2f0 0x55bed1bd06e0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bed1b69e40 0x55bed1b6e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bed1b5f710 0x55bed1b70250 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fe9e1ffb700 0 client.27083 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 --2- 172.21.15.3:0/3597783408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bed1b60030 0x55bed1b70790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.426+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 >> 172.21.15.3:0/3597783408 conn(0x55bed1ad4090 msgr2=0x55bed1b61e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 shutdown_connections 2023-09-10T01:08:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea14472b80 1 -- 172.21.15.3:0/3597783408 wait complete. 2023-09-10T01:08:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea14472b80 20 client.27083 trim_cache size 0 max 16384 2023-09-10T01:08:02.444 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:08:02.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe9e40a0830 msgr2=0x7fe9e40a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe9e40a0830 0x7fe9e40a2cf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0011b80 tx=0x7fe9f0011670 comp rx=0 tx=0).stop 2023-09-10T01:08:02.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc111650 msgr2=0x7fe9fc1a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:02.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc111650 0x7fe9fc1a1730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4007ee0 tx=0x7fe9f400d3b0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 shutdown_connections 2023-09-10T01:08:02.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe9e40a0830 0x7fe9e40a2cf0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe9fc111650 0x7fe9fc1a1730 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9fc0729a0 0x7fe9fc19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 --2- 172.21.15.3:0/297497823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9fc0732c0 0x7fe9fc19d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:02.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.427+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 >> 172.21.15.3:0/297497823 conn(0x7fe9fc0fc170 msgr2=0x7fe9fc0fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:02.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.428+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 shutdown_connections 2023-09-10T01:08:02.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:02.428+0000 7fea00bff700 1 -- 172.21.15.3:0/297497823 wait complete. 2023-09-10T01:08:02.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:02 smithi003 ceph-mon[97654]: pgmap v147: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2023-09-10T01:08:02.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:02 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/108650538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:08:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:02 smithi092 ceph-mon[134067]: pgmap v147: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2023-09-10T01:08:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:02 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/108650538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:08:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:02 smithi136 ceph-mon[133708]: pgmap v147: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2023-09-10T01:08:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:02 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/108650538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:08:02.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 401340 9.38 MB/sec execute 104 sec latency 44.085 ms 2023-09-10T01:08:03.449 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:08:03.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:08:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 -- 172.21.15.3:0/409309187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1018a0 msgr2=0x7f7c7c101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 --2- 172.21.15.3:0/409309187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1018a0 0x7f7c7c101c80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009990 tx=0x7f7c78009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 -- 172.21.15.3:0/409309187 shutdown_connections 2023-09-10T01:08:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 --2- 172.21.15.3:0/409309187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c7c10ac70 0x7f7c7c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 --2- 172.21.15.3:0/409309187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1018a0 0x7f7c7c101c80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 --2- 172.21.15.3:0/409309187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c7c1021c0 0x7f7c7c10a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 -- 172.21.15.3:0/409309187 >> 172.21.15.3:0/409309187 conn(0x7f7c7c076710 msgr2=0x7f7c7c076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.669+0000 7f7c83dff700 1 -- 172.21.15.3:0/409309187 shutdown_connections 2023-09-10T01:08:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.670+0000 7f7c83dff700 1 -- 172.21.15.3:0/409309187 wait complete. 2023-09-10T01:08:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.670+0000 7f7c83dff700 1 Processor -- start 2023-09-10T01:08:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c83dff700 1 -- start start 2023-09-10T01:08:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c83dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c7c1018a0 0x7f7c7c19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c83dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1021c0 0x7f7c7c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c83dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c7c10ac70 0x7f7c7c196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c83dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c196cf0 con 0x7f7c7c1018a0 2023-09-10T01:08:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c83dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c196e60 con 0x7f7c7c1021c0 2023-09-10T01:08:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c83dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c197020 con 0x7f7c7c10ac70 2023-09-10T01:08:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c81b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c7c1018a0 0x7f7c7c19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c8139a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1021c0 0x7f7c7c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c8139a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1021c0 0x7f7c7c19cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57856/0 (socket says 172.21.15.3:57856) 2023-09-10T01:08:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c81b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c7c1018a0 0x7f7c7c19c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49010/0 (socket says 172.21.15.3:49010) 2023-09-10T01:08:03.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c8139a700 1 -- 172.21.15.3:0/2834837785 learned_addr learned my addr 172.21.15.3:0/2834837785 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.671+0000 7f7c8239c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c7c10ac70 0x7f7c7c196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c8139a700 1 -- 172.21.15.3:0/2834837785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c7c10ac70 msgr2=0x7f7c7c196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c8139a700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c7c10ac70 0x7f7c7c196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c8139a700 1 -- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c7c1018a0 msgr2=0x7f7c7c19c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c8139a700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c7c1018a0 0x7f7c7c19c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c8139a700 1 -- 172.21.15.3:0/2834837785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780095f0 con 0x7f7c7c1021c0 2023-09-10T01:08:03.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c8139a700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1021c0 0x7f7c7c19cb50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00d950 tx=0x7f7c6c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c72ffd700 1 -- 172.21.15.3:0/2834837785 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00f890 con 0x7f7c7c1021c0 2023-09-10T01:08:03.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.672+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c197300 con 0x7f7c7c1021c0 2023-09-10T01:08:03.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.673+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c0ff600 con 0x7f7c7c1021c0 2023-09-10T01:08:03.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.673+0000 7f7c72ffd700 1 -- 172.21.15.3:0/2834837785 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c6c010460 con 0x7f7c7c1021c0 2023-09-10T01:08:03.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.673+0000 7f7c72ffd700 1 -- 172.21.15.3:0/2834837785 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00e610 con 0x7f7c7c1021c0 2023-09-10T01:08:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.676+0000 7f7c72ffd700 1 -- 172.21.15.3:0/2834837785 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7c6c0105d0 con 0x7f7c7c1021c0 2023-09-10T01:08:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.677+0000 7f7c97818b80 1 Processor -- start 2023-09-10T01:08:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.677+0000 7f7c97818b80 1 -- start start 2023-09-10T01:08:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344bb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c97818b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626344077b0 con 0x5626344b8ec0 2023-09-10T01:08:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c97818b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626344050d0 con 0x5626344ae790 2023-09-10T01:08:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c97818b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626344009c0 con 0x5626344af0b0 2023-09-10T01:08:03.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c8239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c8239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344aeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57866/0 (socket says 172.21.15.3:57866) 2023-09-10T01:08:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c8239c700 1 -- 172.21.15.3:0/4250422809 learned_addr learned my addr 172.21.15.3:0/4250422809 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c72ffd700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c680a07b0 0x7f7c680a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c8139a700 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344bb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.678+0000 7f7c72ffd700 1 -- 172.21.15.3:0/2834837785 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7c6c0cc1b0 con 0x7f7c7c1021c0 2023-09-10T01:08:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c680a07b0 0x7f7c680a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 -- 172.21.15.3:0/4250422809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 msgr2=0x5626344b8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 -- 172.21.15.3:0/4250422809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 msgr2=0x5626344bb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344bb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 -- 172.21.15.3:0/4250422809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56263422ce60 con 0x5626344ae790 2023-09-10T01:08:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344aeb70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c74013c70 tx=0x7f7c74013f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30a199cd406b5316 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c8239c700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c680a07b0 0x7f7c680a2c70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009000 tx=0x7f7c7401f000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c67fff700 1 -- 172.21.15.3:0/4250422809 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400e040 con 0x5626344ae790 2023-09-10T01:08:03.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c67fff700 1 -- 172.21.15.3:0/4250422809 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c7400f040 con 0x5626344ae790 2023-09-10T01:08:03.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.679+0000 7f7c67fff700 1 -- 172.21.15.3:0/4250422809 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7401a6a0 con 0x5626344ae790 2023-09-10T01:08:03.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 -- 172.21.15.3:0/4250422809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 msgr2=0x5626344aeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344aeb70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c74013c70 tx=0x7f7c74013f80 comp rx=0 tx=0).stop 2023-09-10T01:08:03.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 -- 172.21.15.3:0/4250422809 shutdown_connections 2023-09-10T01:08:03.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344bb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344aeb70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 --2- 172.21.15.3:0/4250422809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 -- 172.21.15.3:0/4250422809 >> 172.21.15.3:0/4250422809 conn(0x562634423110 msgr2=0x5626344a9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 -- 172.21.15.3:0/4250422809 shutdown_connections 2023-09-10T01:08:03.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.680+0000 7f7c97818b80 1 -- 172.21.15.3:0/4250422809 wait complete. 2023-09-10T01:08:03.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.681+0000 7f7c97818b80 1 -- start start 2023-09-10T01:08:03.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.681+0000 7f7c97818b80 1 Processor -- start 2023-09-10T01:08:03.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c97818b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56263439ebf0 con 0x5626344b8ec0 2023-09-10T01:08:03.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c97818b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56263439e8f0 con 0x5626344ae790 2023-09-10T01:08:03.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c97818b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626342829a0 con 0x5626344af0b0 2023-09-10T01:08:03.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c97818b80 10 client.0 _ll_register_callbacks cb 0x5626342e8b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:03.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57876/0 (socket says 172.21.15.3:57876) 2023-09-10T01:08:03.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 -- 172.21.15.3:0/1703155607 learned_addr learned my addr 172.21.15.3:0/1703155607 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c8139a700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c8239c700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 -- 172.21.15.3:0/1703155607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 msgr2=0x5626344b6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 -- 172.21.15.3:0/1703155607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 msgr2=0x5626344b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c81b9b700 1 -- 172.21.15.3:0/1703155607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c74013850 con 0x5626344ae790 2023-09-10T01:08:03.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.682+0000 7f7c8139a700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:03.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344b5bd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c78015000 tx=0x7f7c7800bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2f43f4c37023cf6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c657fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:03.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c657fa700 1 -- 172.21.15.3:0/1703155607 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78021070 con 0x5626344ae790 2023-09-10T01:08:03.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562634358570 con 0x5626344ae790 2023-09-10T01:08:03.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c657fa700 1 -- 172.21.15.3:0/1703155607 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7c7801be20 con 0x5626344ae790 2023-09-10T01:08:03.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c657fa700 1 -- 172.21.15.3:0/1703155607 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c78005230 con 0x5626344ae790 2023-09-10T01:08:03.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c97818b80 10 client.27095 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:03.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562634215ce0 con 0x5626344ae790 2023-09-10T01:08:03.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c657fa700 1 -- 172.21.15.3:0/1703155607 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562634358570 con 0x5626344ae790 2023-09-10T01:08:03.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.683+0000 7f7c657fa700 1 -- 172.21.15.3:0/1703155607 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f7c78004ce0 con 0x5626344ae790 2023-09-10T01:08:03.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c97818b80 10 client.27095 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:03.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c97818b80 10 client.27095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:03.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c97818b80 20 client.27095 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:03.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c97818b80 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5626344c89c0 0x5626344cade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c97818b80 4 client.27095 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:03.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562634415920 con 0x5626344c89c0 2023-09-10T01:08:03.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c8139a700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5626344c89c0 0x5626344cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c8139a700 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5626344c89c0 0x5626344cade0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b534d54bea6e5308 server_cookie=5e3f05837e0b49c8 in_seq=0 out_seq=0 2023-09-10T01:08:03.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.684+0000 7f7c657fa700 10 client.27095 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:03.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c657fa700 1 -- 172.21.15.3:0/1703155607 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562634415920 con 0x5626344c89c0 2023-09-10T01:08:03.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c657fa700 10 client.27095 handle_command_reply: tid=0 2023-09-10T01:08:03.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 client.27095 shutdown 2023-09-10T01:08:03.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 msgr2=0x5626344b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344b5bd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c78015000 tx=0x7f7c7800bf60 comp rx=0 tx=0).stop 2023-09-10T01:08:03.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 shutdown_connections 2023-09-10T01:08:03.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5626344c89c0 0x5626344cade0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344b8ec0 0x5626344b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344ae790 0x5626344b5bd0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c657fa700 0 client.27095 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:03.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 --2- 172.21.15.3:0/1703155607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x5626344b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 >> 172.21.15.3:0/1703155607 conn(0x562634423110 msgr2=0x5626344a9860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 shutdown_connections 2023-09-10T01:08:03.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.685+0000 7f7c97818b80 1 -- 172.21.15.3:0/1703155607 wait complete. 2023-09-10T01:08:03.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.686+0000 7f7c97818b80 20 client.27095 trim_cache size 0 max 16384 2023-09-10T01:08:03.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 Processor -- start 2023-09-10T01:08:03.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 -- start start 2023-09-10T01:08:03.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344ae790 0x5626344bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344af0b0 0x5626344ade90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x5626344b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626344c60f0 con 0x5626344af0b0 2023-09-10T01:08:03.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626344a3e20 con 0x5626344b8ec0 2023-09-10T01:08:03.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c97818b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626344bff80 con 0x5626344ae790 2023-09-10T01:08:03.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c8239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x5626344b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c8139a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344af0b0 0x5626344ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c8239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x5626344b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57886/0 (socket says 172.21.15.3:57886) 2023-09-10T01:08:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c8239c700 1 -- 172.21.15.3:0/3477762084 learned_addr learned my addr 172.21.15.3:0/3477762084 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.705+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344ae790 0x5626344bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c8239c700 1 -- 172.21.15.3:0/3477762084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344ae790 msgr2=0x5626344bf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c8239c700 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344ae790 0x5626344bf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c8239c700 1 -- 172.21.15.3:0/3477762084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344af0b0 msgr2=0x5626344ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c8239c700 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344af0b0 0x5626344ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c8239c700 1 -- 172.21.15.3:0/3477762084 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626344cfa10 con 0x5626344b8ec0 2023-09-10T01:08:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344ae790 0x5626344bf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c8239c700 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x5626344b2e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c74023000 tx=0x7f7c74018f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2494a2a0222f8b12 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c70bff700 1 -- 172.21.15.3:0/3477762084 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400e070 con 0x5626344b8ec0 2023-09-10T01:08:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c70bff700 1 -- 172.21.15.3:0/3477762084 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c74009520 con 0x5626344b8ec0 2023-09-10T01:08:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.706+0000 7f7c70bff700 1 -- 172.21.15.3:0/3477762084 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74004040 con 0x5626344b8ec0 2023-09-10T01:08:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 -- 172.21.15.3:0/3477762084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 msgr2=0x5626344b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x5626344b2e80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c74023000 tx=0x7f7c74018f80 comp rx=0 tx=0).stop 2023-09-10T01:08:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 -- 172.21.15.3:0/3477762084 shutdown_connections 2023-09-10T01:08:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344ae790 0x5626344bf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x5626344b2e80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 --2- 172.21.15.3:0/3477762084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344af0b0 0x5626344ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 -- 172.21.15.3:0/3477762084 >> 172.21.15.3:0/3477762084 conn(0x562634423110 msgr2=0x5626344ad780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 -- 172.21.15.3:0/3477762084 shutdown_connections 2023-09-10T01:08:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 -- 172.21.15.3:0/3477762084 wait complete. 2023-09-10T01:08:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.707+0000 7f7c97818b80 1 -- start start 2023-09-10T01:08:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 1 Processor -- start 2023-09-10T01:08:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344ae790 0x5626345167f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x562634516d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x562634510b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562634511170 con 0x5626344ae790 2023-09-10T01:08:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626345112e0 con 0x5626344b8ec0 2023-09-10T01:08:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626345114a0 con 0x5626344af0b0 2023-09-10T01:08:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c97818b80 10 client.0 _ll_register_callbacks cb 0x5626342e8b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c8239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x562634510b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c81b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344ae790 0x5626345167f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c8239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x562634510b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57890/0 (socket says 172.21.15.3:57890) 2023-09-10T01:08:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.708+0000 7f7c8239c700 1 -- 172.21.15.3:0/2448341246 learned_addr learned my addr 172.21.15.3:0/2448341246 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c8139a700 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x562634516d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c8239c700 1 -- 172.21.15.3:0/2448341246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 msgr2=0x562634516d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c8239c700 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x562634516d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c8239c700 1 -- 172.21.15.3:0/2448341246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344ae790 msgr2=0x5626345167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c8239c700 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344ae790 0x5626345167f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c8239c700 1 -- 172.21.15.3:0/2448341246 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56263422ce60 con 0x5626344b8ec0 2023-09-10T01:08:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c8239c700 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x562634510b90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7c74019010 tx=0x7f7c74007730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a64a0d81f65a4cf server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c657fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c657fa700 1 -- 172.21.15.3:0/2448341246 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400e040 con 0x5626344b8ec0 2023-09-10T01:08:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c7400f040 con 0x5626344b8ec0 2023-09-10T01:08:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c657fa700 1 -- 172.21.15.3:0/2448341246 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7c7401e070 con 0x5626344b8ec0 2023-09-10T01:08:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c657fa700 1 -- 172.21.15.3:0/2448341246 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c74003c00 con 0x5626344b8ec0 2023-09-10T01:08:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.709+0000 7f7c97818b80 10 client.27101 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5626342532d0 con 0x5626344b8ec0 2023-09-10T01:08:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c657fa700 1 -- 172.21.15.3:0/2448341246 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c7400f040 con 0x5626344b8ec0 2023-09-10T01:08:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c657fa700 1 -- 172.21.15.3:0/2448341246 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f7c74004020 con 0x5626344b8ec0 2023-09-10T01:08:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c97818b80 10 client.27101 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c97818b80 10 client.27101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c97818b80 20 client.27101 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:03.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c97818b80 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56263451e390 0x562634520780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c97818b80 4 client.27101 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:08:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5626343f99a0 con 0x56263451e390 2023-09-10T01:08:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.710+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56263451e390 0x562634520780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.711+0000 7f7c81b9b700 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56263451e390 0x562634520780 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9d98bd18b72717a server_cookie=4c661ac65538644 in_seq=0 out_seq=0 2023-09-10T01:08:03.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.711+0000 7f7c657fa700 10 client.27101 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.713+0000 7f7c657fa700 1 -- 172.21.15.3:0/2448341246 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5626343f99a0 con 0x56263451e390 2023-09-10T01:08:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.713+0000 7f7c657fa700 10 client.27101 handle_command_reply: tid=0 2023-09-10T01:08:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.713+0000 7f7c97818b80 1 client.27101 shutdown 2023-09-10T01:08:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 msgr2=0x562634510b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x562634510b90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7c74019010 tx=0x7f7c74007730 comp rx=0 tx=0).stop 2023-09-10T01:08:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 shutdown_connections 2023-09-10T01:08:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56263451e390 0x562634520780 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626344b8ec0 0x562634510b90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626344ae790 0x5626345167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c657fa700 0 client.27101 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 --2- 172.21.15.3:0/2448341246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626344af0b0 0x562634516d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 >> 172.21.15.3:0/2448341246 conn(0x562634423110 msgr2=0x5626344ae010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 shutdown_connections 2023-09-10T01:08:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 1 -- 172.21.15.3:0/2448341246 wait complete. 2023-09-10T01:08:03.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.714+0000 7f7c97818b80 20 client.27101 trim_cache size 0 max 16384 2023-09-10T01:08:03.735 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:08:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:08:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "c8ee6556-1545-4b59-ae42-ae33b7bdb6fc", 2023-09-10T01:08:03.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:08:03.736 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:08:03.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c680a07b0 msgr2=0x7f7c680a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c680a07b0 0x7f7c680a2c70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009000 tx=0x7f7c7401f000 comp rx=0 tx=0).stop 2023-09-10T01:08:03.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1021c0 msgr2=0x7f7c7c19cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1021c0 0x7f7c7c19cb50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00d950 tx=0x7f7c6c00dc60 comp rx=0 tx=0).stop 2023-09-10T01:08:03.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 shutdown_connections 2023-09-10T01:08:03.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c680a07b0 0x7f7c680a2c70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c7c10ac70 0x7f7c7c196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c7c1018a0 0x7f7c7c19c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 --2- 172.21.15.3:0/2834837785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c7c1021c0 0x7f7c7c19cb50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 >> 172.21.15.3:0/2834837785 conn(0x7f7c7c076710 msgr2=0x7f7c7c0fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 shutdown_connections 2023-09-10T01:08:03.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.715+0000 7f7c83dff700 1 -- 172.21.15.3:0/2834837785 wait complete. 2023-09-10T01:08:03.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:08:03.754 INFO:tasks.workunit.client.0.smithi003.stdout: 1 403254 9.36 MB/sec execute 105 sec latency 16.730 ms 2023-09-10T01:08:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 -- 172.21.15.3:0/4157399007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0101000 msgr2=0x7fdbe010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/4157399007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0101000 0x7fdbe010d510 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009f90 tx=0x7fdbd800b330 comp rx=0 tx=0).stop 2023-09-10T01:08:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 -- 172.21.15.3:0/4157399007 shutdown_connections 2023-09-10T01:08:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/4157399007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0101000 0x7fdbe010d510 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/4157399007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0106650 0x7fdbe0106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/4157399007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0100640 0x7fdbe0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 -- 172.21.15.3:0/4157399007 >> 172.21.15.3:0/4157399007 conn(0x7fdbe00fc170 msgr2=0x7fdbe00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 -- 172.21.15.3:0/4157399007 shutdown_connections 2023-09-10T01:08:03.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.962+0000 7fdbe5dff700 1 -- 172.21.15.3:0/4157399007 wait complete. 2023-09-10T01:08:03.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.963+0000 7fdbe5dff700 1 Processor -- start 2023-09-10T01:08:03.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.963+0000 7fdbe5dff700 1 -- start start 2023-09-10T01:08:03.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbe5dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0100640 0x7fdbe0072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbe5dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101000 0x7fdbe0072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbe5dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0106650 0x7fdbe01a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbe5dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe0073670 con 0x7fdbe0106650 2023-09-10T01:08:03.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbe5dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe01a4da0 con 0x7fdbe0101000 2023-09-10T01:08:03.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbe5dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe01a4ee0 con 0x7fdbe0100640 2023-09-10T01:08:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101000 0x7fdbe0072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101000 0x7fdbe0072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57900/0 (socket says 172.21.15.3:57900) 2023-09-10T01:08:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbdeffd700 1 -- 172.21.15.3:0/1811330231 learned_addr learned my addr 172.21.15.3:0/1811330231 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0100640 0x7fdbe0072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.964+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0106650 0x7fdbe01a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdeffd700 1 -- 172.21.15.3:0/1811330231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0100640 msgr2=0x7fdbe0072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0100640 0x7fdbe0072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdeffd700 1 -- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0106650 msgr2=0x7fdbe01a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0106650 0x7fdbe01a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdeffd700 1 -- 172.21.15.3:0/1811330231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd8009c40 con 0x7fdbe0101000 2023-09-10T01:08:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101000 0x7fdbe0072e40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400e9c0 tx=0x7fdbd400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdcff9700 1 -- 172.21.15.3:0/1811330231 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4004a30 con 0x7fdbe0101000 2023-09-10T01:08:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe01a5140 con 0x7fdbe0101000 2023-09-10T01:08:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdcff9700 1 -- 172.21.15.3:0/1811330231 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbd400cc80 con 0x7fdbe0101000 2023-09-10T01:08:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbdcff9700 1 -- 172.21.15.3:0/1811330231 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40054e0 con 0x7fdbe0101000 2023-09-10T01:08:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.965+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe01a5670 con 0x7fdbe0101000 2023-09-10T01:08:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.969+0000 7fdbdcff9700 1 -- 172.21.15.3:0/1811330231 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fdbd4004b90 con 0x7fdbe0101000 2023-09-10T01:08:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.971+0000 7fdbf96ffb80 1 Processor -- start 2023-09-10T01:08:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.971+0000 7fdbf96ffb80 1 -- start start 2023-09-10T01:08:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.971+0000 7fdbdcff9700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdbc80a07b0 0x7fdbc80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.971+0000 7fdbdcff9700 1 -- 172.21.15.3:0/1811330231 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdbd4014070 con 0x7fdbe0101000 2023-09-10T01:08:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5e6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c53f730 con 0x555c4c5f0e40 2023-09-10T01:08:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c53d050 con 0x555c4c5e6710 2023-09-10T01:08:03.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c538940 con 0x555c4c5e7030 2023-09-10T01:08:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdbc80a07b0 0x7fdbc80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5e6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5e6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57908/0 (socket says 172.21.15.3:57908) 2023-09-10T01:08:03.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1453143756 learned_addr learned my addr 172.21.15.3:0/1453143756 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1453143756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 msgr2=0x555c4c5f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1453143756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 msgr2=0x555c4c5f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1453143756 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c4c364de0 con 0x555c4c5e6710 2023-09-10T01:08:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.972+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:08:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.973+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5e6af0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009d00 tx=0x7fdbd0011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=583131952a9936c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.973+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdbc80a07b0 0x7fdbc80a2c70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009550 tx=0x7fdbd8007bb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.973+0000 7fdbc5ffb700 1 -- 172.21.15.3:0/1453143756 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0004230 con 0x555c4c5e6710 2023-09-10T01:08:03.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.973+0000 7fdbc5ffb700 1 -- 172.21.15.3:0/1453143756 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbd0004390 con 0x555c4c5e6710 2023-09-10T01:08:03.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.973+0000 7fdbc5ffb700 1 -- 172.21.15.3:0/1453143756 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0019880 con 0x555c4c5e6710 2023-09-10T01:08:03.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1453143756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 msgr2=0x555c4c5e6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:03.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5e6af0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009d00 tx=0x7fdbd0011720 comp rx=0 tx=0).stop 2023-09-10T01:08:03.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1453143756 shutdown_connections 2023-09-10T01:08:03.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5e6af0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1453143756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1453143756 >> 172.21.15.3:0/1453143756 conn(0x555c4c55b090 msgr2=0x555c4c5e1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:03.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1453143756 shutdown_connections 2023-09-10T01:08:03.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1453143756 wait complete. 2023-09-10T01:08:03.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.974+0000 7fdbf96ffb80 1 -- start start 2023-09-10T01:08:03.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 1 Processor -- start 2023-09-10T01:08:03.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e6710 0x555c4c5f0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5edf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c4d6b70 con 0x555c4c5e6710 2023-09-10T01:08:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c4d6870 con 0x555c4c5f0e40 2023-09-10T01:08:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c3ba920 con 0x555c4c5e7030 2023-09-10T01:08:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbf96ffb80 10 client.0 _ll_register_callbacks cb 0x555c4c420a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:03.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e6710 0x555c4c5f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5ed9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44070/0 (socket says 172.21.15.3:44070) 2023-09-10T01:08:03.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.975+0000 7fdbdeffd700 1 -- 172.21.15.3:0/2989356360 learned_addr learned my addr 172.21.15.3:0/2989356360 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:03.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdffff700 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/2989356360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 msgr2=0x555c4c5ed9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5ed9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 msgr2=0x555c4c5edf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5edf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:03.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/2989356360 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd0009960 con 0x555c4c5e6710 2023-09-10T01:08:03.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5ed9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:03.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e6710 0x555c4c5f0740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009230 tx=0x7fdbd00038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99863d3c92261202 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:03.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbbeffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:08:03.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.976+0000 7fdbbeffd700 1 -- 172.21.15.3:0/2989356360 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0005040 con 0x555c4c5e6710 2023-09-10T01:08:03.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbbeffd700 1 -- 172.21.15.3:0/2989356360 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdbd00051a0 con 0x555c4c5e6710 2023-09-10T01:08:03.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c4c4904f0 con 0x555c4c5e6710 2023-09-10T01:08:03.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbbeffd700 1 -- 172.21.15.3:0/2989356360 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbd0005360 con 0x555c4c5e6710 2023-09-10T01:08:03.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 10 client.17037 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:03.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x555c4c34dc60 con 0x555c4c5e6710 2023-09-10T01:08:03.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbbeffd700 1 -- 172.21.15.3:0/2989356360 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555c4c4904f0 con 0x555c4c5e6710 2023-09-10T01:08:03.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbbeffd700 1 -- 172.21.15.3:0/2989356360 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fdbd0028020 con 0x555c4c5e6710 2023-09-10T01:08:03.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 10 client.17037 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:03.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 10 client.17037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:03.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 20 client.17037 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:03.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c600730 0x555c4c602b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:03.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 4 client.17037 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:03.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.977+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c4c54d8a0 con 0x555c4c600730 2023-09-10T01:08:03.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.978+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c600730 0x555c4c602b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:03.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.978+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c600730 0x555c4c602b20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e74410032de1e939 server_cookie=9bb3d48787bf30ec in_seq=0 out_seq=0 2023-09-10T01:08:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.978+0000 7fdbbeffd700 10 client.17037 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbbeffd700 1 -- 172.21.15.3:0/2989356360 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555c4c54d8a0 con 0x555c4c600730 2023-09-10T01:08:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbbeffd700 10 client.17037 handle_command_reply: tid=0 2023-09-10T01:08:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 client.17037 shutdown 2023-09-10T01:08:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e6710 msgr2=0x555c4c5f0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e6710 0x555c4c5f0740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009230 tx=0x7fdbd00038b0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 shutdown_connections 2023-09-10T01:08:04.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c600730 0x555c4c602b20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5edf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e6710 0x555c4c5f0740 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2989356360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5ed9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 >> 172.21.15.3:0/2989356360 conn(0x555c4c55b090 msgr2=0x555c4c5e17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:04.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbbeffd700 0 client.17037 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:04.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.979+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 shutdown_connections 2023-09-10T01:08:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.980+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2989356360 wait complete. 2023-09-10T01:08:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.980+0000 7fdbf96ffb80 20 client.17037 trim_cache size 0 max 16384 2023-09-10T01:08:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.999+0000 7fdbf96ffb80 1 Processor -- start 2023-09-10T01:08:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:03.999+0000 7fdbf96ffb80 1 -- start start 2023-09-10T01:08:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e6710 0x555c4c5f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:04.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e7030 0x555c4c5ed1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:04.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5ed700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:04.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c5e2130 con 0x555c4c5e7030 2023-09-10T01:08:04.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c5eef70 con 0x555c4c5f0e40 2023-09-10T01:08:04.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c5fd150 con 0x555c4c5e6710 2023-09-10T01:08:04.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e7030 0x555c4c5ed1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:04.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e6710 0x555c4c5f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:04.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e7030 0x555c4c5ed1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49086/0 (socket says 172.21.15.3:49086) 2023-09-10T01:08:04.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdeffd700 1 -- 172.21.15.3:0/2666988282 learned_addr learned my addr 172.21.15.3:0/2666988282 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:04.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdffff700 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5ed700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:04.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/2666988282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 msgr2=0x555c4c5ed700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:04.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5ed700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/2666988282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e7030 msgr2=0x555c4c5ed1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:04.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e7030 0x555c4c5ed1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.000+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/2666988282 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c4c38b250 con 0x555c4c5e6710 2023-09-10T01:08:04.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.001+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e6710 0x555c4c5f02e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0004f30 tx=0x7fdbd0011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92d4f254e991edba server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:04.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.001+0000 7fdbc67fc700 1 -- 172.21.15.3:0/2666988282 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0003920 con 0x555c4c5e6710 2023-09-10T01:08:04.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.001+0000 7fdbc67fc700 1 -- 172.21.15.3:0/2666988282 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbd0004fb0 con 0x555c4c5e6710 2023-09-10T01:08:04.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.001+0000 7fdbc67fc700 1 -- 172.21.15.3:0/2666988282 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0004760 con 0x555c4c5e6710 2023-09-10T01:08:04.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.001+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2666988282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e6710 msgr2=0x555c4c5f02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:04.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.001+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e6710 0x555c4c5f02e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0004f30 tx=0x7fdbd0011720 comp rx=0 tx=0).stop 2023-09-10T01:08:04.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2666988282 shutdown_connections 2023-09-10T01:08:04.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e6710 0x555c4c5f02e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5f0e40 0x555c4c5ed700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/2666988282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5e7030 0x555c4c5ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2666988282 >> 172.21.15.3:0/2666988282 conn(0x555c4c55b090 msgr2=0x555c4c5e5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:04.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2666988282 shutdown_connections 2023-09-10T01:08:04.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/2666988282 wait complete. 2023-09-10T01:08:04.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 -- start start 2023-09-10T01:08:04.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.002+0000 7fdbf96ffb80 1 Processor -- start 2023-09-10T01:08:04.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:04.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:04.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbf96ffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:04.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c5f5fb0 con 0x555c4c5f0e40 2023-09-10T01:08:04.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c5f6120 con 0x555c4c5e6710 2023-09-10T01:08:04.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbf96ffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c4c5f62e0 con 0x555c4c5e7030 2023-09-10T01:08:04.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbf96ffb80 10 client.0 _ll_register_callbacks cb 0x7fdbd4004f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:04.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.003+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:04.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5f72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57936/0 (socket says 172.21.15.3:57936) 2023-09-10T01:08:04.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1764696587 learned_addr learned my addr 172.21.15.3:0/1764696587 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:04.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:04.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:04.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1764696587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 msgr2=0x555c4c5f7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:04.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1764696587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 msgr2=0x555c4c5f59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:04.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 -- 172.21.15.3:0/1764696587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c4c34dc60 con 0x555c4c5e6710 2023-09-10T01:08:04.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:08:04.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdffff700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:04.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbdf7fe700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5f72c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0005e60 tx=0x7fdbd0019420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a5df908f5aedab server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:04.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbbeffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:04.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.004+0000 7fdbbeffd700 1 -- 172.21.15.3:0/1764696587 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0005120 con 0x555c4c5e6710 2023-09-10T01:08:04.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbbeffd700 1 -- 172.21.15.3:0/1764696587 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fdbd0000e40 con 0x555c4c5e6710 2023-09-10T01:08:04.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbd0004390 con 0x555c4c5e6710 2023-09-10T01:08:04.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbbeffd700 1 -- 172.21.15.3:0/1764696587 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbd0005280 con 0x555c4c5e6710 2023-09-10T01:08:04.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 10 client.27119 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:04.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x555c4c3632b0 con 0x555c4c5e6710 2023-09-10T01:08:04.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbbeffd700 1 -- 172.21.15.3:0/1764696587 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbd0004390 con 0x555c4c5e6710 2023-09-10T01:08:04.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbbeffd700 1 -- 172.21.15.3:0/1764696587 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fdbd0028070 con 0x555c4c5e6710 2023-09-10T01:08:04.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 10 client.27119 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 10 client.27119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 20 client.27119 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c655220 0x555c4c657610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 4 client.27119 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:08:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.005+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555c4c531920 con 0x555c4c655220 2023-09-10T01:08:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c655220 0x555c4c657610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbdeffd700 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c655220 0x555c4c657610 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3aa09995213cce3 server_cookie=133fda96bd50ca0c in_seq=0 out_seq=0 2023-09-10T01:08:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbbeffd700 10 client.27119 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbbeffd700 1 -- 172.21.15.3:0/1764696587 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555c4c531920 con 0x555c4c655220 2023-09-10T01:08:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbbeffd700 10 client.27119 handle_command_reply: tid=0 2023-09-10T01:08:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 client.27119 shutdown 2023-09-10T01:08:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 msgr2=0x555c4c5f72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5f72c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0005e60 tx=0x7fdbd0019420 comp rx=0 tx=0).stop 2023-09-10T01:08:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 shutdown_connections 2023-09-10T01:08:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555c4c655220 0x555c4c657610 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c4c5f0e40 0x555c4c5f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555c4c5e6710 0x555c4c5f72c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 --2- 172.21.15.3:0/1764696587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555c4c5e7030 0x555c4c5f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbbeffd700 0 client.27119 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.006+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 >> 172.21.15.3:0/1764696587 conn(0x555c4c55b090 msgr2=0x555c4c5fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 shutdown_connections 2023-09-10T01:08:04.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbf96ffb80 1 -- 172.21.15.3:0/1764696587 wait complete. 2023-09-10T01:08:04.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbf96ffb80 20 client.27119 trim_cache size 0 max 16384 2023-09-10T01:08:04.031 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:08:04.031 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:08:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:08:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: "c8ee6556-1545-4b59-ae42-ae33b7bdb6fc": { 2023-09-10T01:08:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:08:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "c8ee6556-1545-4b59-ae42-ae33b7bdb6fc", 2023-09-10T01:08:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:08:04.033 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:08:04.033 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:08:04.033 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:08:04.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdbc80a07b0 msgr2=0x7fdbc80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:04.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdbc80a07b0 0x7fdbc80a2c70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009550 tx=0x7fdbd8007bb0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101000 msgr2=0x7fdbe0072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:04.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101000 0x7fdbe0072e40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdbd400e9c0 tx=0x7fdbd400ecd0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 shutdown_connections 2023-09-10T01:08:04.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdbc80a07b0 0x7fdbc80a2c70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbe0100640 0x7fdbe0072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdbe0106650 0x7fdbe01a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 --2- 172.21.15.3:0/1811330231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101000 0x7fdbe0072e40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:04.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.007+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 >> 172.21.15.3:0/1811330231 conn(0x7fdbe00fc170 msgr2=0x7fdbe0104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:04.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.008+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 shutdown_connections 2023-09-10T01:08:04.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:04.008+0000 7fdbe5dff700 1 -- 172.21.15.3:0/1811330231 wait complete. 2023-09-10T01:08:04.037 INFO:tasks.cephfs.filesystem:scrub status for tag:c8ee6556-1545-4b59-ae42-ae33b7bdb6fc - {'path': '/', 'tag': 'c8ee6556-1545-4b59-ae42-ae33b7bdb6fc', 'options': 'recursive,force'} 2023-09-10T01:08:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:04 smithi092 ceph-mon[134067]: pgmap v148: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2023-09-10T01:08:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:04 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:08:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:04 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:04 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:08:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:04 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:08:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:04 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:04 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:04 smithi003 ceph-mon[97654]: pgmap v148: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2023-09-10T01:08:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:04 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:08:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:04 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:04 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:08:04.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:04 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:08:04.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:04 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:04 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:04 smithi136 ceph-mon[133708]: pgmap v148: 129 pgs: 129 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 261 op/s 2023-09-10T01:08:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:04 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:08:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:04 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:04 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:08:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:04 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:08:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:04 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:04 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:04.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 404788 9.38 MB/sec execute 106 sec latency 50.875 ms 2023-09-10T01:08:05.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:05 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:08:05.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:05 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:05.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:05 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:08:05.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:05 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:05.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:05 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:08:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:05 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:05.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 406747 9.36 MB/sec execute 107 sec latency 17.143 ms 2023-09-10T01:08:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:06 smithi092 ceph-mon[134067]: pgmap v149: 129 pgs: 129 active+clean; 108 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 387 op/s 2023-09-10T01:08:06.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:06 smithi003 ceph-mon[97654]: pgmap v149: 129 pgs: 129 active+clean; 108 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 387 op/s 2023-09-10T01:08:06.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:06 smithi136 ceph-mon[133708]: pgmap v149: 129 pgs: 129 active+clean; 108 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 387 op/s 2023-09-10T01:08:06.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 408363 9.39 MB/sec execute 108 sec latency 50.268 ms 2023-09-10T01:08:07.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 410326 9.36 MB/sec execute 109 sec latency 12.293 ms 2023-09-10T01:08:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 -- 172.21.15.3:0/4260420602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec4106680 msgr2=0x7f9ec4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/4260420602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec4106680 0x7f9ec4106a60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00099c0 tx=0x7f9ec0009cd0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 -- 172.21.15.3:0/4260420602 shutdown_connections 2023-09-10T01:08:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/4260420602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ec4100f80 0x7f9ec410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/4260420602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec4106680 0x7f9ec4106a60 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/4260420602 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec41005c0 0x7f9ec4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 -- 172.21.15.3:0/4260420602 >> 172.21.15.3:0/4260420602 conn(0x7f9ec40fc0b0 msgr2=0x7f9ec40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 -- 172.21.15.3:0/4260420602 shutdown_connections 2023-09-10T01:08:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.942+0000 7f9ecbdff700 1 -- 172.21.15.3:0/4260420602 wait complete. 2023-09-10T01:08:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.943+0000 7f9ecbdff700 1 Processor -- start 2023-09-10T01:08:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.943+0000 7f9ecbdff700 1 -- start start 2023-09-10T01:08:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ecbdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec41005c0 0x7f9ec4196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ecbdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec4100f80 0x7f9ec4196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ecbdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ec4106680 0x7f9ec4199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ecbdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec419a580 con 0x7f9ec41005c0 2023-09-10T01:08:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ecbdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec419a6f0 con 0x7f9ec4106680 2023-09-10T01:08:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ecbdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec419a8b0 con 0x7f9ec4100f80 2023-09-10T01:08:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec41005c0 0x7f9ec4196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ec939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec4100f80 0x7f9ec4196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9eca39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ec4106680 0x7f9ec4199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ec939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec4100f80 0x7f9ec4196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37270/0 (socket says 172.21.15.3:37270) 2023-09-10T01:08:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.944+0000 7f9ec939a700 1 -- 172.21.15.3:0/1274109700 learned_addr learned my addr 172.21.15.3:0/1274109700 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9eca39c700 1 -- 172.21.15.3:0/1274109700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec4100f80 msgr2=0x7f9ec4196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9eca39c700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec4100f80 0x7f9ec4196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9eca39c700 1 -- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec41005c0 msgr2=0x7f9ec4196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9eca39c700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec41005c0 0x7f9ec4196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9eca39c700 1 -- 172.21.15.3:0/1274109700 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec00095f0 con 0x7f9ec4106680 2023-09-10T01:08:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9ec939a700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec4100f80 0x7f9ec4196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec41005c0 0x7f9ec4196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9eca39c700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ec4106680 0x7f9ec4199f80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00efc0 tx=0x7f9ebc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9ebaffd700 1 -- 172.21.15.3:0/1274109700 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc018070 con 0x7f9ec4106680 2023-09-10T01:08:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9ebaffd700 1 -- 172.21.15.3:0/1274109700 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ebc00f040 con 0x7f9ec4106680 2023-09-10T01:08:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec419ab30 con 0x7f9ec4106680 2023-09-10T01:08:07.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9ebaffd700 1 -- 172.21.15.3:0/1274109700 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc00cd40 con 0x7f9ec4106680 2023-09-10T01:08:07.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.945+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec4072a10 con 0x7f9ec4106680 2023-09-10T01:08:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.948+0000 7f9ebaffd700 1 -- 172.21.15.3:0/1274109700 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9ebc003a00 con 0x7f9ec4106680 2023-09-10T01:08:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.950+0000 7f9ebaffd700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9eb00a07b0 0x7f9eb00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.950+0000 7f9ebaffd700 1 -- 172.21.15.3:0/1274109700 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9ebc0cbeb0 con 0x7f9ec4106680 2023-09-10T01:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.950+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9eb00a07b0 0x7f9eb00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.950+0000 7f9edf636b80 1 Processor -- start 2023-09-10T01:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.950+0000 7f9edf636b80 1 -- start start 2023-09-10T01:08:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9eb00a07b0 0x7f9eb00a2c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009fd0 tx=0x7f9ec001e040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe344580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe33e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9edf636b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe2976d0 con 0x55b7fe33ecc0 2023-09-10T01:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9edf636b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe294ff0 con 0x55b7fe33e300 2023-09-10T01:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9edf636b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe2908e0 con 0x55b7fe3441a0 2023-09-10T01:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe344580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec939a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe33e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe344580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37278/0 (socket says 172.21.15.3:37278) 2023-09-10T01:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3742259595 learned_addr learned my addr 172.21.15.3:0/3742259595 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec939a700 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe33e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47614/0 (socket says 172.21.15.3:47614) 2023-09-10T01:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9eca39c700 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3742259595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 msgr2=0x55b7fe33e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe33e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3742259595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 msgr2=0x55b7fe34af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.951+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3742259595 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7fe0bcd80 con 0x55b7fe3441a0 2023-09-10T01:08:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.952+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe344580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00099c0 tx=0x7f9ec0023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=465015174df5c38 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.952+0000 7f9eaffff700 1 -- 172.21.15.3:0/3742259595 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0003b50 con 0x55b7fe3441a0 2023-09-10T01:08:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.952+0000 7f9eaffff700 1 -- 172.21.15.3:0/3742259595 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ec0022040 con 0x55b7fe3441a0 2023-09-10T01:08:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.952+0000 7f9eaffff700 1 -- 172.21.15.3:0/3742259595 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000f710 con 0x55b7fe3441a0 2023-09-10T01:08:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 -- 172.21.15.3:0/3742259595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 msgr2=0x55b7fe344580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe344580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00099c0 tx=0x7f9ec0023000 comp rx=0 tx=0).stop 2023-09-10T01:08:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 -- 172.21.15.3:0/3742259595 shutdown_connections 2023-09-10T01:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe344580 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3742259595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe33e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 -- 172.21.15.3:0/3742259595 >> 172.21.15.3:0/3742259595 conn(0x55b7fe2b3030 msgr2=0x55b7fe339560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 -- 172.21.15.3:0/3742259595 shutdown_connections 2023-09-10T01:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 -- 172.21.15.3:0/3742259595 wait complete. 2023-09-10T01:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.953+0000 7f9edf636b80 1 -- start start 2023-09-10T01:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 1 Processor -- start 2023-09-10T01:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe348060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe3441a0 0x55b7fe348690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe22eb10 con 0x55b7fe33ecc0 2023-09-10T01:08:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe22e810 con 0x55b7fe3441a0 2023-09-10T01:08:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe1128c0 con 0x55b7fe33e300 2023-09-10T01:08:07.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.954+0000 7f9edf636b80 10 client.0 _ll_register_callbacks cb 0x55b7fe178a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9eca39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe3441a0 0x55b7fe348690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37288/0 (socket says 172.21.15.3:37288) 2023-09-10T01:08:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3759414058 learned_addr learned my addr 172.21.15.3:0/3759414058 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec939a700 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe348060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3759414058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe3441a0 msgr2=0x55b7fe348690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe3441a0 0x55b7fe348690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3759414058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 msgr2=0x55b7fe348060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe348060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/3759414058 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec410dd40 con 0x55b7fe33e300 2023-09-10T01:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34ade0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00050a0 tx=0x7f9ec000b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e698eb43e89b8c54 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.955+0000 7f9ead7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9ead7fa700 1 -- 172.21.15.3:0/3759414058 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0003d60 con 0x55b7fe33e300 2023-09-10T01:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b7fe1e8490 con 0x55b7fe33e300 2023-09-10T01:08:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9ead7fa700 1 -- 172.21.15.3:0/3759414058 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9ec000f990 con 0x55b7fe33e300 2023-09-10T01:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9ead7fa700 1 -- 172.21.15.3:0/3759414058 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ec000bb10 con 0x55b7fe33e300 2023-09-10T01:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 10 client.27043 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55b7fe0a5c00 con 0x55b7fe33e300 2023-09-10T01:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9ead7fa700 1 -- 172.21.15.3:0/3759414058 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b7fe1e8490 con 0x55b7fe33e300 2023-09-10T01:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9ead7fa700 1 -- 172.21.15.3:0/3759414058 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f9ec0031820 con 0x55b7fe33e300 2023-09-10T01:08:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 10 client.27043 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 10 client.27043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 20 client.27043 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe357e30 0x55b7fe35a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 4 client.27043 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.956+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b7fe2a5840 con 0x55b7fe357e30 2023-09-10T01:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.957+0000 7f9ec939a700 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe357e30 0x55b7fe35a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.957+0000 7f9ec939a700 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe357e30 0x55b7fe35a220 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1970dea81ccbed2 server_cookie=18d617a53bfce178 in_seq=0 out_seq=0 2023-09-10T01:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.957+0000 7f9ead7fa700 10 client.27043 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9ead7fa700 1 -- 172.21.15.3:0/3759414058 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b7fe2a5840 con 0x55b7fe357e30 2023-09-10T01:08:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9ead7fa700 10 client.27043 handle_command_reply: tid=0 2023-09-10T01:08:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 client.27043 shutdown 2023-09-10T01:08:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 msgr2=0x55b7fe34ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34ade0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00050a0 tx=0x7f9ec000b9a0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 shutdown_connections 2023-09-10T01:08:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe357e30 0x55b7fe35a220 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34ade0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe3441a0 0x55b7fe348690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 --2- 172.21.15.3:0/3759414058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe348060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 >> 172.21.15.3:0/3759414058 conn(0x55b7fe2b3030 msgr2=0x55b7fe3387a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.958+0000 7f9ead7fa700 0 client.27043 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.959+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 shutdown_connections 2023-09-10T01:08:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.959+0000 7f9edf636b80 1 -- 172.21.15.3:0/3759414058 wait complete. 2023-09-10T01:08:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.959+0000 7f9edf636b80 20 client.27043 trim_cache size 0 max 16384 2023-09-10T01:08:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 Processor -- start 2023-09-10T01:08:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 -- start start 2023-09-10T01:08:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe34a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe353b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe3421f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe339ae0 con 0x55b7fe33ecc0 2023-09-10T01:08:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe335fc0 con 0x55b7fe33e300 2023-09-10T01:08:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9edf636b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe349640 con 0x55b7fe3441a0 2023-09-10T01:08:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe34a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9eca39c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe3421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe34a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47636/0 (socket says 172.21.15.3:47636) 2023-09-10T01:08:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/796353072 learned_addr learned my addr 172.21.15.3:0/796353072 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9eca39c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe3421f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37304/0 (socket says 172.21.15.3:37304) 2023-09-10T01:08:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.975+0000 7f9ec939a700 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe353b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eca39c700 1 -- 172.21.15.3:0/796353072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 msgr2=0x55b7fe34a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eca39c700 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe34a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eca39c700 1 -- 172.21.15.3:0/796353072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 msgr2=0x55b7fe353b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eca39c700 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe353b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eca39c700 1 -- 172.21.15.3:0/796353072 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7fe0e31f0 con 0x55b7fe3441a0 2023-09-10T01:08:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe34a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eca39c700 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe3421f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc0c9210 tx=0x7f9ebc098b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca060fa87fc71fec server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eb8bff700 1 -- 172.21.15.3:0/796353072 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc004ad0 con 0x55b7fe3441a0 2023-09-10T01:08:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eb8bff700 1 -- 172.21.15.3:0/796353072 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ebc099700 con 0x55b7fe3441a0 2023-09-10T01:08:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.976+0000 7f9eb8bff700 1 -- 172.21.15.3:0/796353072 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc099260 con 0x55b7fe3441a0 2023-09-10T01:08:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 -- 172.21.15.3:0/796353072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 msgr2=0x55b7fe3421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe3421f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc0c9210 tx=0x7f9ebc098b40 comp rx=0 tx=0).stop 2023-09-10T01:08:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 -- 172.21.15.3:0/796353072 shutdown_connections 2023-09-10T01:08:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe3441a0 0x55b7fe3421f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33e300 0x55b7fe34a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 --2- 172.21.15.3:0/796353072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe33ecc0 0x55b7fe353b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 -- 172.21.15.3:0/796353072 >> 172.21.15.3:0/796353072 conn(0x55b7fe2b3030 msgr2=0x55b7fe339090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 -- 172.21.15.3:0/796353072 shutdown_connections 2023-09-10T01:08:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.977+0000 7f9edf636b80 1 -- 172.21.15.3:0/796353072 wait complete. 2023-09-10T01:08:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.978+0000 7f9edf636b80 1 -- start start 2023-09-10T01:08:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.978+0000 7f9edf636b80 1 Processor -- start 2023-09-10T01:08:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.978+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.978+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.978+0000 7f9edf636b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe3441a0 0x55b7fe34de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9edf636b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe3aac30 con 0x55b7fe3441a0 2023-09-10T01:08:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9edf636b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe3aad70 con 0x55b7fe33ecc0 2023-09-10T01:08:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9edf636b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7fe3aaee0 con 0x55b7fe33e300 2023-09-10T01:08:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9edf636b80 10 client.0 _ll_register_callbacks cb 0x55b7fe178a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec939a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37314/0 (socket says 172.21.15.3:37314) 2023-09-10T01:08:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/82364278 learned_addr learned my addr 172.21.15.3:0/82364278 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec939a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47642/0 (socket says 172.21.15.3:47642) 2023-09-10T01:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9eca39c700 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe3441a0 0x55b7fe34de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/82364278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33ecc0 msgr2=0x55b7fe34d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/82364278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe3441a0 msgr2=0x55b7fe34de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe3441a0 0x55b7fe34de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 -- 172.21.15.3:0/82364278 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b7fe0a5c00 con 0x55b7fe33e300 2023-09-10T01:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ec9b9b700 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34d3d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00058e0 tx=0x7f9ec00037d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47da690988e5d87 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.979+0000 7f9ead7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9ead7fa700 1 -- 172.21.15.3:0/82364278 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000bee0 con 0x55b7fe33e300 2023-09-10T01:08:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9ead7fa700 1 -- 172.21.15.3:0/82364278 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9ec0031440 con 0x55b7fe33e300 2023-09-10T01:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ec0022040 con 0x55b7fe33e300 2023-09-10T01:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9ead7fa700 1 -- 172.21.15.3:0/82364278 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ec0031600 con 0x55b7fe33e300 2023-09-10T01:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9edf636b80 10 client.27049 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f9ec4072a10 con 0x55b7fe33e300 2023-09-10T01:08:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9ead7fa700 1 -- 172.21.15.3:0/82364278 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ec0022040 con 0x55b7fe33e300 2023-09-10T01:08:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9ead7fa700 1 -- 172.21.15.3:0/82364278 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f9ec003a5e0 con 0x55b7fe33e300 2023-09-10T01:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9edf636b80 10 client.27049 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9edf636b80 10 client.27049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.980+0000 7f9edf636b80 20 client.27049 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.981+0000 7f9edf636b80 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe3ac230 0x55b7fe3ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.981+0000 7f9edf636b80 4 client.27049 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.981+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b7fe2898c0 con 0x55b7fe3ac230 2023-09-10T01:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.981+0000 7f9ec939a700 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe3ac230 0x55b7fe3ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.981+0000 7f9ec939a700 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe3ac230 0x55b7fe3ae620 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a25cb3aa5edb598 server_cookie=1fcbc91178cc75d3 in_seq=0 out_seq=0 2023-09-10T01:08:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:07.981+0000 7f9ead7fa700 10 client.27049 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.100+0000 7f9ead7fa700 1 -- 172.21.15.3:0/82364278 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b7fe2898c0 con 0x55b7fe3ac230 2023-09-10T01:08:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.100+0000 7f9ead7fa700 10 client.27049 handle_command_reply: tid=0 2023-09-10T01:08:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.100+0000 7f9edf636b80 1 client.27049 shutdown 2023-09-10T01:08:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 msgr2=0x55b7fe34d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34d3d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00058e0 tx=0x7f9ec00037d0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 shutdown_connections 2023-09-10T01:08:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b7fe3ac230 0x55b7fe3ae620 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7fe33e300 0x55b7fe34d3d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b7fe3441a0 0x55b7fe34de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9ead7fa700 0 client.27049 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 --2- 172.21.15.3:0/82364278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7fe33ecc0 0x55b7fe34d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 >> 172.21.15.3:0/82364278 conn(0x55b7fe2b3030 msgr2=0x55b7fe353b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 shutdown_connections 2023-09-10T01:08:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.101+0000 7f9edf636b80 1 -- 172.21.15.3:0/82364278 wait complete. 2023-09-10T01:08:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.102+0000 7f9edf636b80 20 client.27049 trim_cache size 0 max 16384 2023-09-10T01:08:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:08:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:08:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:08:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:08:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.102+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9eb00a07b0 msgr2=0x7f9eb00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.102+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9eb00a07b0 0x7f9eb00a2c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009fd0 tx=0x7f9ec001e040 comp rx=0 tx=0).stop 2023-09-10T01:08:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ec4106680 msgr2=0x7f9ec4199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ec4106680 0x7f9ec4199f80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00efc0 tx=0x7f9ebc00c3a0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 shutdown_connections 2023-09-10T01:08:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9eb00a07b0 0x7f9eb00a2c70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ec4106680 0x7f9ec4199f80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec41005c0 0x7f9ec4196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 --2- 172.21.15.3:0/1274109700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec4100f80 0x7f9ec4196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 >> 172.21.15.3:0/1274109700 conn(0x7f9ec40fc0b0 msgr2=0x7f9ec40fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.103+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 shutdown_connections 2023-09-10T01:08:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:08.104+0000 7f9ecbdff700 1 -- 172.21.15.3:0/1274109700 wait complete. 2023-09-10T01:08:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:08 smithi092 ceph-mon[134067]: pgmap v150: 129 pgs: 129 active+clean; 108 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2023-09-10T01:08:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:08 smithi003 ceph-mon[97654]: pgmap v150: 129 pgs: 129 active+clean; 108 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2023-09-10T01:08:08.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:08 smithi136 ceph-mon[133708]: pgmap v150: 129 pgs: 129 active+clean; 108 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 278 op/s 2023-09-10T01:08:08.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 411882 9.39 MB/sec execute 110 sec latency 58.987 ms 2023-09-10T01:08:09.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:09 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:08:09.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:09 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:08:09.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:09 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:08:09.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:09 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:08:09.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:09 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:08:09.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:09 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:08:09.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 413239 9.35 MB/sec execute 111 sec latency 34.511 ms 2023-09-10T01:08:10.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:10 smithi003 ceph-mon[97654]: pgmap v151: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 442 op/s 2023-09-10T01:08:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:10 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:08:10.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:10 smithi136 ceph-mon[133708]: pgmap v151: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 442 op/s 2023-09-10T01:08:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:10 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:08:10.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:10 smithi092 ceph-mon[134067]: pgmap v151: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 442 op/s 2023-09-10T01:08:10.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:10 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:08:10.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 415396 9.39 MB/sec execute 112 sec latency 37.146 ms 2023-09-10T01:08:11.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 416866 9.36 MB/sec execute 113 sec latency 13.502 ms 2023-09-10T01:08:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:12 smithi003 ceph-mon[97654]: pgmap v152: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:08:12.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:12 smithi136 ceph-mon[133708]: pgmap v152: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:08:12.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:12 smithi092 ceph-mon[134067]: pgmap v152: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:08:12.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 418988 9.40 MB/sec execute 114 sec latency 22.944 ms 2023-09-10T01:08:13.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 420960 9.37 MB/sec execute 115 sec latency 10.742 ms 2023-09-10T01:08:14.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:14 smithi003 ceph-mon[97654]: pgmap v153: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:08:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:14 smithi136 ceph-mon[133708]: pgmap v153: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:08:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:14 smithi092 ceph-mon[134067]: pgmap v153: 129 pgs: 129 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-09-10T01:08:14.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 422558 9.40 MB/sec execute 116 sec latency 30.073 ms 2023-09-10T01:08:15.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 424224 9.37 MB/sec execute 117 sec latency 40.767 ms 2023-09-10T01:08:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:16 smithi003 ceph-mon[97654]: pgmap v154: 129 pgs: 129 active+clean; 108 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 402 op/s 2023-09-10T01:08:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:16 smithi136 ceph-mon[133708]: pgmap v154: 129 pgs: 129 active+clean; 108 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 402 op/s 2023-09-10T01:08:16.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:16 smithi092 ceph-mon[134067]: pgmap v154: 129 pgs: 129 active+clean; 108 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 402 op/s 2023-09-10T01:08:16.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 426108 9.40 MB/sec execute 118 sec latency 13.146 ms 2023-09-10T01:08:17.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 427878 9.37 MB/sec execute 119 sec latency 15.459 ms 2023-09-10T01:08:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:18 smithi003 ceph-mon[97654]: pgmap v155: 129 pgs: 129 active+clean; 108 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 277 op/s 2023-09-10T01:08:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:18 smithi136 ceph-mon[133708]: pgmap v155: 129 pgs: 129 active+clean; 108 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 277 op/s 2023-09-10T01:08:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:18 smithi092 ceph-mon[134067]: pgmap v155: 129 pgs: 129 active+clean; 108 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 277 op/s 2023-09-10T01:08:18.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 429611 9.40 MB/sec execute 120 sec latency 72.796 ms 2023-09-10T01:08:19.755 INFO:tasks.workunit.client.0.smithi003.stdout: 1 431624 9.38 MB/sec execute 121 sec latency 9.905 ms 2023-09-10T01:08:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:20 smithi003 ceph-mon[97654]: pgmap v156: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 382 op/s 2023-09-10T01:08:20.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:20 smithi136 ceph-mon[133708]: pgmap v156: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 382 op/s 2023-09-10T01:08:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:20 smithi092 ceph-mon[134067]: pgmap v156: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 382 op/s 2023-09-10T01:08:20.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 433179 9.41 MB/sec execute 122 sec latency 46.449 ms 2023-09-10T01:08:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.286+0000 7f91929ff700 1 -- 172.21.15.3:0/1979135414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c106ea0 msgr2=0x7f918c107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.286+0000 7f91929ff700 1 --2- 172.21.15.3:0/1979135414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c106ea0 0x7f918c107280 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f917c009990 tx=0x7f917c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.287+0000 7f91929ff700 1 -- 172.21.15.3:0/1979135414 shutdown_connections 2023-09-10T01:08:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.287+0000 7f91929ff700 1 --2- 172.21.15.3:0/1979135414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c106ea0 0x7f918c107280 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.287+0000 7f91929ff700 1 --2- 172.21.15.3:0/1979135414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f918c10bf20 0x7f918c113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.287+0000 7f91929ff700 1 --2- 172.21.15.3:0/1979135414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f918c107850 0x7f918c10b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.287+0000 7f91929ff700 1 -- 172.21.15.3:0/1979135414 >> 172.21.15.3:0/1979135414 conn(0x7f918c076360 msgr2=0x7f918c105420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.288+0000 7f91929ff700 1 -- 172.21.15.3:0/1979135414 shutdown_connections 2023-09-10T01:08:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.288+0000 7f91929ff700 1 -- 172.21.15.3:0/1979135414 wait complete. 2023-09-10T01:08:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.289+0000 7f91929ff700 1 Processor -- start 2023-09-10T01:08:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.290+0000 7f91929ff700 1 -- start start 2023-09-10T01:08:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.290+0000 7f91929ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f918c106ea0 0x7f918c079860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.290+0000 7f91929ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f918c10bf20 0x7f918c079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.290+0000 7f91929ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c1af2b0 0x7f918c1b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.290+0000 7f91929ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c1b1be0 con 0x7f918c106ea0 2023-09-10T01:08:21.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.290+0000 7f91929ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c06a020 con 0x7f918c10bf20 2023-09-10T01:08:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.290+0000 7f91929ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c06a190 con 0x7f918c1af2b0 2023-09-10T01:08:21.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.292+0000 7f918b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f918c10bf20 0x7f918c079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.293+0000 7f918b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f918c10bf20 0x7f918c079da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43734/0 (socket says 172.21.15.3:43734) 2023-09-10T01:08:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.293+0000 7f918b7fe700 1 -- 172.21.15.3:0/1122336152 learned_addr learned my addr 172.21.15.3:0/1122336152 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.293+0000 7f918bfff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f918c106ea0 0x7f918c079860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.293+0000 7f9190f9c700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c1af2b0 0x7f918c1b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.294+0000 7f918bfff700 1 -- 172.21.15.3:0/1122336152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c1af2b0 msgr2=0x7f918c1b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.294+0000 7f918bfff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c1af2b0 0x7f918c1b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.294+0000 7f918bfff700 1 -- 172.21.15.3:0/1122336152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f918c10bf20 msgr2=0x7f918c079da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.294+0000 7f918bfff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f918c10bf20 0x7f918c079da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.294+0000 7f918bfff700 1 -- 172.21.15.3:0/1122336152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c0095f0 con 0x7f918c106ea0 2023-09-10T01:08:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.295+0000 7f918bfff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f918c106ea0 0x7f918c079860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f917c006010 tx=0x7f917c004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.297+0000 7f91897fa700 1 -- 172.21.15.3:0/1122336152 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c01d070 con 0x7f918c106ea0 2023-09-10T01:08:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.297+0000 7f91897fa700 1 -- 172.21.15.3:0/1122336152 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f917c0040a0 con 0x7f918c106ea0 2023-09-10T01:08:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.297+0000 7f91897fa700 1 -- 172.21.15.3:0/1122336152 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c004a20 con 0x7f918c106ea0 2023-09-10T01:08:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.298+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c06a410 con 0x7f918c106ea0 2023-09-10T01:08:21.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.298+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c06a870 con 0x7f918c106ea0 2023-09-10T01:08:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.304+0000 7f91897fa700 1 -- 172.21.15.3:0/1122336152 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f917c0050c0 con 0x7f918c106ea0 2023-09-10T01:08:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.304+0000 7f91a63d2b80 1 Processor -- start 2023-09-10T01:08:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f91a63d2b80 1 -- start start 2023-09-10T01:08:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x55826988d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x558269887780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887cc0 0x558269893f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582697e06d0 con 0x55826988d1a0 2023-09-10T01:08:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582697ddff0 con 0x558269887cc0 2023-09-10T01:08:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582697d98e0 con 0x558269887300 2023-09-10T01:08:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.305+0000 7f918bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x558269887780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887cc0 0x558269893f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x558269887780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51688/0 (socket says 172.21.15.3:51688) 2023-09-10T01:08:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918bfff700 1 -- 172.21.15.3:0/2776129740 learned_addr learned my addr 172.21.15.3:0/2776129740 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f9190f9c700 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x55826988d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f91897fa700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f91740a0940 0x7f91740a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918b7fe700 1 -- 172.21.15.3:0/2776129740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 msgr2=0x558269887780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918b7fe700 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x558269887780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f91897fa700 1 -- 172.21.15.3:0/1122336152 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f917c0cd270 con 0x7f918c106ea0 2023-09-10T01:08:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918b7fe700 1 -- 172.21.15.3:0/2776129740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 msgr2=0x55826988d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918b7fe700 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x55826988d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918b7fe700 1 -- 172.21.15.3:0/2776129740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558269605d80 con 0x558269887cc0 2023-09-10T01:08:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f9190f9c700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f91740a0940 0x7f91740a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.306+0000 7f918b7fe700 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887cc0 0x558269893f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f918c077230 tx=0x7f918000f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4808e1b2de10e595 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.307+0000 7f91727fc700 1 -- 172.21.15.3:0/2776129740 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000fe70 con 0x558269887cc0 2023-09-10T01:08:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.307+0000 7f91727fc700 1 -- 172.21.15.3:0/2776129740 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9180004bb0 con 0x558269887cc0 2023-09-10T01:08:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.307+0000 7f91727fc700 1 -- 172.21.15.3:0/2776129740 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180005040 con 0x558269887cc0 2023-09-10T01:08:21.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2776129740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887cc0 msgr2=0x558269893f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887cc0 0x558269893f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f918c077230 tx=0x7f918000f850 comp rx=0 tx=0).stop 2023-09-10T01:08:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f9190f9c700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f91740a0940 0x7f91740a2e00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9184009b80 tx=0x7f9184008040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2776129740 shutdown_connections 2023-09-10T01:08:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887cc0 0x558269893f30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x55826988d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2776129740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x558269887780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2776129740 >> 172.21.15.3:0/2776129740 conn(0x5582697fc030 msgr2=0x558269882560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2776129740 shutdown_connections 2023-09-10T01:08:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.308+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2776129740 wait complete. 2023-09-10T01:08:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.309+0000 7f91a63d2b80 1 -- start start 2023-09-10T01:08:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.309+0000 7f91a63d2b80 1 Processor -- start 2023-09-10T01:08:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.309+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269893da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.309+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887cc0 0x558269891130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x5582698916f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558269777b10 con 0x55826988d1a0 2023-09-10T01:08:21.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558269777810 con 0x558269887300 2023-09-10T01:08:21.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55826965b8c0 con 0x558269887cc0 2023-09-10T01:08:21.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f91a63d2b80 10 client.0 _ll_register_callbacks cb 0x5582696c1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:21.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269893da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269893da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43746/0 (socket says 172.21.15.3:43746) 2023-09-10T01:08:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 -- 172.21.15.3:0/2821437022 learned_addr learned my addr 172.21.15.3:0/2821437022 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918b7fe700 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887cc0 0x558269891130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f9190f9c700 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x5582698916f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 -- 172.21.15.3:0/2821437022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887cc0 msgr2=0x558269891130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887cc0 0x558269891130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 -- 172.21.15.3:0/2821437022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 msgr2=0x5582698916f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x5582698916f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.310+0000 7f918bfff700 1 -- 172.21.15.3:0/2821437022 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918000f1d0 con 0x558269887300 2023-09-10T01:08:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f918bfff700 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269893da0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f917c009fd0 tx=0x7f917c005460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d605120a79181 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f916b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f916b7fe700 1 -- 172.21.15.3:0/2821437022 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c09a1f0 con 0x558269887300 2023-09-10T01:08:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f916b7fe700 1 -- 172.21.15.3:0/2821437022 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f917c06bd40 con 0x558269887300 2023-09-10T01:08:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558269731490 con 0x558269887300 2023-09-10T01:08:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f916b7fe700 1 -- 172.21.15.3:0/2821437022 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f917c099f60 con 0x558269887300 2023-09-10T01:08:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f91a63d2b80 10 client.27146 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5582695eec00 con 0x558269887300 2023-09-10T01:08:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.311+0000 7f916b7fe700 1 -- 172.21.15.3:0/2821437022 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558269731490 con 0x558269887300 2023-09-10T01:08:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f916b7fe700 1 -- 172.21.15.3:0/2821437022 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f917c099910 con 0x558269887300 2023-09-10T01:08:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f91a63d2b80 10 client.27146 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f91a63d2b80 10 client.27146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f91a63d2b80 20 client.27146 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698a1070 0x5582698a3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f91a63d2b80 4 client.27146 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582697ee840 con 0x5582698a1070 2023-09-10T01:08:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.312+0000 7f918b7fe700 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698a1070 0x5582698a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.313+0000 7f918b7fe700 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698a1070 0x5582698a3490 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=831ca6bb8f9c76bb server_cookie=72b9ff55576e4d24 in_seq=0 out_seq=0 2023-09-10T01:08:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.313+0000 7f916b7fe700 10 client.27146 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.313+0000 7f916b7fe700 1 -- 172.21.15.3:0/2821437022 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5582697ee840 con 0x5582698a1070 2023-09-10T01:08:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.313+0000 7f916b7fe700 10 client.27146 handle_command_reply: tid=0 2023-09-10T01:08:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 client.27146 shutdown 2023-09-10T01:08:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 msgr2=0x558269893da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269893da0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f917c009fd0 tx=0x7f917c005460 comp rx=0 tx=0).stop 2023-09-10T01:08:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 shutdown_connections 2023-09-10T01:08:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698a1070 0x5582698a3490 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55826988d1a0 0x5582698916f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269893da0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/2821437022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887cc0 0x558269891130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f916b7fe700 0 client.27146 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 >> 172.21.15.3:0/2821437022 conn(0x5582697fc030 msgr2=0x5582698817a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 shutdown_connections 2023-09-10T01:08:21.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.314+0000 7f91a63d2b80 1 -- 172.21.15.3:0/2821437022 wait complete. 2023-09-10T01:08:21.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.315+0000 7f91a63d2b80 20 client.27146 trim_cache size 0 max 16384 2023-09-10T01:08:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 Processor -- start 2023-09-10T01:08:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 -- start start 2023-09-10T01:08:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269898520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x5582698938d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55826988d1a0 0x558269889b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582698885c0 con 0x558269887cc0 2023-09-10T01:08:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55826988ce00 con 0x558269887300 2023-09-10T01:08:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.332+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582698926d0 con 0x55826988d1a0 2023-09-10T01:08:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269898520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269898520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43760/0 (socket says 172.21.15.3:43760) 2023-09-10T01:08:21.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 -- 172.21.15.3:0/3709187923 learned_addr learned my addr 172.21.15.3:0/3709187923 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:21.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918b7fe700 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x5582698938d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f9190f9c700 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55826988d1a0 0x558269889b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 -- 172.21.15.3:0/3709187923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55826988d1a0 msgr2=0x558269889b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55826988d1a0 0x558269889b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 -- 172.21.15.3:0/3709187923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 msgr2=0x5582698938d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x5582698938d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 -- 172.21.15.3:0/3709187923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55826962c1f0 con 0x558269887300 2023-09-10T01:08:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.333+0000 7f918bfff700 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269898520 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f917c000c00 tx=0x7f917c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=758adc6b08f038be server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f9172ffd700 1 -- 172.21.15.3:0/3709187923 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c096040 con 0x558269887300 2023-09-10T01:08:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f9172ffd700 1 -- 172.21.15.3:0/3709187923 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f917c099f60 con 0x558269887300 2023-09-10T01:08:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f9172ffd700 1 -- 172.21.15.3:0/3709187923 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c06bd40 con 0x558269887300 2023-09-10T01:08:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f91a63d2b80 1 -- 172.21.15.3:0/3709187923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 msgr2=0x558269898520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269898520 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f917c000c00 tx=0x7f917c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f91a63d2b80 1 -- 172.21.15.3:0/3709187923 shutdown_connections 2023-09-10T01:08:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55826988d1a0 0x558269889b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558269887300 0x558269898520 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/3709187923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x5582698938d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.334+0000 7f91a63d2b80 1 -- 172.21.15.3:0/3709187923 >> 172.21.15.3:0/3709187923 conn(0x5582697fc030 msgr2=0x558269886b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.335+0000 7f91a63d2b80 1 -- 172.21.15.3:0/3709187923 shutdown_connections 2023-09-10T01:08:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.335+0000 7f91a63d2b80 1 -- 172.21.15.3:0/3709187923 wait complete. 2023-09-10T01:08:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.335+0000 7f91a63d2b80 1 -- start start 2023-09-10T01:08:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.335+0000 7f91a63d2b80 1 Processor -- start 2023-09-10T01:08:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x55826988f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x55826988fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f91a63d2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55826988d1a0 0x558269890360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582698908a0 con 0x558269887cc0 2023-09-10T01:08:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558269890a10 con 0x55826988d1a0 2023-09-10T01:08:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f91a63d2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558269890bd0 con 0x558269887300 2023-09-10T01:08:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f91a63d2b80 10 client.0 _ll_register_callbacks cb 0x5582696c1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f918bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x55826988f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f918bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x55826988f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51708/0 (socket says 172.21.15.3:51708) 2023-09-10T01:08:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f918bfff700 1 -- 172.21.15.3:0/705840420 learned_addr learned my addr 172.21.15.3:0/705840420 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f918b7fe700 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x55826988fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.336+0000 7f9190f9c700 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55826988d1a0 0x558269890360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f9190f9c700 1 -- 172.21.15.3:0/705840420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 msgr2=0x55826988f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f9190f9c700 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x55826988f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f9190f9c700 1 -- 172.21.15.3:0/705840420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 msgr2=0x55826988fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f9190f9c700 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x55826988fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f9190f9c700 1 -- 172.21.15.3:0/705840420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582695eec00 con 0x55826988d1a0 2023-09-10T01:08:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f9190f9c700 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55826988d1a0 0x558269890360 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9184012550 tx=0x7f9184009110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a92168fa7c634296 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f916b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f916b7fe700 1 -- 172.21.15.3:0/705840420 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184024dc0 con 0x55826988d1a0 2023-09-10T01:08:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f916b7fe700 1 -- 172.21.15.3:0/705840420 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f918402d440 con 0x55826988d1a0 2023-09-10T01:08:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.337+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9180004bb0 con 0x55826988d1a0 2023-09-10T01:08:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f916b7fe700 1 -- 172.21.15.3:0/705840420 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f918402d600 con 0x55826988d1a0 2023-09-10T01:08:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 10 client.27152 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558269604250 con 0x55826988d1a0 2023-09-10T01:08:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f916b7fe700 1 -- 172.21.15.3:0/705840420 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9180004bb0 con 0x55826988d1a0 2023-09-10T01:08:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f916b7fe700 1 -- 172.21.15.3:0/705840420 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f91840214b0 con 0x55826988d1a0 2023-09-10T01:08:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 10 client.27152 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 10 client.27152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 20 client.27152 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698f5b50 0x5582698f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 4 client.27152 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:08:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5582697d28c0 con 0x5582698f5b50 2023-09-10T01:08:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.338+0000 7f918bfff700 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698f5b50 0x5582698f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.339+0000 7f918bfff700 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698f5b50 0x5582698f7f70 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4845add80c1b192b server_cookie=d5d0b1eb99625c62 in_seq=0 out_seq=0 2023-09-10T01:08:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.339+0000 7f916b7fe700 10 client.27152 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.445+0000 7f916b7fe700 1 -- 172.21.15.3:0/705840420 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5582697d28c0 con 0x5582698f5b50 2023-09-10T01:08:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.445+0000 7f916b7fe700 10 client.27152 handle_command_reply: tid=0 2023-09-10T01:08:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.445+0000 7f91a63d2b80 1 client.27152 shutdown 2023-09-10T01:08:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.445+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55826988d1a0 msgr2=0x558269890360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.445+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55826988d1a0 0x558269890360 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9184012550 tx=0x7f9184009110 comp rx=0 tx=0).stop 2023-09-10T01:08:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 shutdown_connections 2023-09-10T01:08:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5582698f5b50 0x5582698f7f70 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558269887300 0x55826988f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55826988d1a0 0x558269890360 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f916b7fe700 0 client.27152 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 --2- 172.21.15.3:0/705840420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558269887cc0 0x55826988fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 >> 172.21.15.3:0/705840420 conn(0x5582697fc030 msgr2=0x55826989cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 shutdown_connections 2023-09-10T01:08:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 1 -- 172.21.15.3:0/705840420 wait complete. 2023-09-10T01:08:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.446+0000 7f91a63d2b80 20 client.27152 trim_cache size 0 max 16384 2023-09-10T01:08:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:08:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:08:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:08:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:08:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f91740a0940 msgr2=0x7f91740a2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f91740a0940 0x7f91740a2e00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9184009b80 tx=0x7f9184008040 comp rx=0 tx=0).stop 2023-09-10T01:08:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f918c106ea0 msgr2=0x7f918c079860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f918c106ea0 0x7f918c079860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f917c006010 tx=0x7f917c004710 comp rx=0 tx=0).stop 2023-09-10T01:08:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 shutdown_connections 2023-09-10T01:08:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f91740a0940 0x7f91740a2e00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f918c1af2b0 0x7f918c1b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f918c106ea0 0x7f918c079860 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 --2- 172.21.15.3:0/1122336152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f918c10bf20 0x7f918c079da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.447+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 >> 172.21.15.3:0/1122336152 conn(0x7f918c076360 msgr2=0x7f918c0776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.448+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 shutdown_connections 2023-09-10T01:08:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:21.448+0000 7f91929ff700 1 -- 172.21.15.3:0/1122336152 wait complete. 2023-09-10T01:08:21.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 434722 9.37 MB/sec execute 123 sec latency 49.525 ms 2023-09-10T01:08:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:22 smithi003 ceph-mon[97654]: pgmap v157: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 218 op/s 2023-09-10T01:08:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:22 smithi136 ceph-mon[133708]: pgmap v157: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 218 op/s 2023-09-10T01:08:22.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:22 smithi092 ceph-mon[134067]: pgmap v157: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 218 op/s 2023-09-10T01:08:22.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 436736 9.41 MB/sec execute 124 sec latency 16.021 ms 2023-09-10T01:08:23.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 438668 9.39 MB/sec execute 125 sec latency 36.150 ms 2023-09-10T01:08:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:24 smithi003 ceph-mon[97654]: pgmap v158: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 218 op/s 2023-09-10T01:08:24.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:24 smithi136 ceph-mon[133708]: pgmap v158: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 218 op/s 2023-09-10T01:08:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:24 smithi092 ceph-mon[134067]: pgmap v158: 129 pgs: 129 active+clean; 114 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 218 op/s 2023-09-10T01:08:24.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 440444 9.41 MB/sec execute 126 sec latency 11.097 ms 2023-09-10T01:08:25.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 442341 9.40 MB/sec execute 127 sec latency 39.533 ms 2023-09-10T01:08:26.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:26 smithi003 ceph-mon[97654]: pgmap v159: 129 pgs: 129 active+clean; 111 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 379 op/s 2023-09-10T01:08:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:26 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:08:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:26 smithi136 ceph-mon[133708]: pgmap v159: 129 pgs: 129 active+clean; 111 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 379 op/s 2023-09-10T01:08:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:26 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:08:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:26 smithi092 ceph-mon[134067]: pgmap v159: 129 pgs: 129 active+clean; 111 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 379 op/s 2023-09-10T01:08:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:26 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:08:26.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 443867 9.41 MB/sec execute 128 sec latency 28.331 ms 2023-09-10T01:08:27.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:27 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:08:27.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:27 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:08:27.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:27 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:08:27.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:27 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:08:27.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:27 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:08:27.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:27 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:08:27.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:27 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:08:27.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:27 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:08:27.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:27 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:08:27.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 445846 9.40 MB/sec execute 129 sec latency 12.960 ms 2023-09-10T01:08:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:28 smithi003 ceph-mon[97654]: pgmap v160: 129 pgs: 129 active+clean; 111 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 266 op/s 2023-09-10T01:08:28.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:28 smithi136 ceph-mon[133708]: pgmap v160: 129 pgs: 129 active+clean; 111 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 266 op/s 2023-09-10T01:08:28.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:28 smithi092 ceph-mon[134067]: pgmap v160: 129 pgs: 129 active+clean; 111 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 266 op/s 2023-09-10T01:08:28.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 447473 9.42 MB/sec execute 130 sec latency 45.132 ms 2023-09-10T01:08:29.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 449275 9.40 MB/sec execute 131 sec latency 39.756 ms 2023-09-10T01:08:30.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:30 smithi003 ceph-mon[97654]: pgmap v161: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 380 op/s 2023-09-10T01:08:30.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:30 smithi136 ceph-mon[133708]: pgmap v161: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 380 op/s 2023-09-10T01:08:30.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:30 smithi092 ceph-mon[134067]: pgmap v161: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 380 op/s 2023-09-10T01:08:30.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 450886 9.42 MB/sec execute 132 sec latency 24.238 ms 2023-09-10T01:08:31.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 452679 9.39 MB/sec execute 133 sec latency 15.642 ms 2023-09-10T01:08:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:32 smithi003 ceph-mon[97654]: pgmap v162: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 275 op/s 2023-09-10T01:08:32.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:32 smithi136 ceph-mon[133708]: pgmap v162: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 275 op/s 2023-09-10T01:08:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:32 smithi092 ceph-mon[134067]: pgmap v162: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 275 op/s 2023-09-10T01:08:32.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 454459 9.42 MB/sec execute 134 sec latency 18.012 ms 2023-09-10T01:08:33.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:33 smithi003 ceph-mon[97654]: pgmap v163: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 275 op/s 2023-09-10T01:08:33.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:33 smithi136 ceph-mon[133708]: pgmap v163: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 275 op/s 2023-09-10T01:08:33.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:33 smithi092 ceph-mon[134067]: pgmap v163: 129 pgs: 129 active+clean; 117 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 275 op/s 2023-09-10T01:08:33.756 INFO:tasks.workunit.client.0.smithi003.stdout: 1 456018 9.39 MB/sec execute 135 sec latency 13.194 ms 2023-09-10T01:08:34.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:08:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2252652343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288101000 msgr2=0x7fb28810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2252652343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288101000 0x7fb28810d510 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb280009f90 tx=0x7fb28000b330 comp rx=0 tx=0).stop 2023-09-10T01:08:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2252652343 shutdown_connections 2023-09-10T01:08:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2252652343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288106650 0x7fb288106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2252652343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288101000 0x7fb28810d510 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2252652343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb288100640 0x7fb288100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2252652343 >> 172.21.15.3:0/2252652343 conn(0x7fb2880fc170 msgr2=0x7fb2880fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2252652343 shutdown_connections 2023-09-10T01:08:34.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.259+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2252652343 wait complete. 2023-09-10T01:08:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 Processor -- start 2023-09-10T01:08:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 -- start start 2023-09-10T01:08:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288100640 0x7fb288072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288101000 0x7fb288072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb288106650 0x7fb2881a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb288073670 con 0x7fb288106650 2023-09-10T01:08:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2881a4d50 con 0x7fb288100640 2023-09-10T01:08:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28f5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2881a4e90 con 0x7fb288101000 2023-09-10T01:08:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.261+0000 7fb28cb9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288101000 0x7fb288072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288101000 0x7fb288072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49882/0 (socket says 172.21.15.3:49882) 2023-09-10T01:08:34.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 -- 172.21.15.3:0/2344690666 learned_addr learned my addr 172.21.15.3:0/2344690666 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28d39b700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288100640 0x7fb288072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28db9c700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb288106650 0x7fb2881a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 -- 172.21.15.3:0/2344690666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288100640 msgr2=0x7fb288072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288100640 0x7fb288072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 -- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb288106650 msgr2=0x7fb2881a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb288106650 0x7fb2881a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 -- 172.21.15.3:0/2344690666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb280009c40 con 0x7fb288101000 2023-09-10T01:08:34.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288101000 0x7fb288072e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb27800e9a0 tx=0x7fb27800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb27e7fc700 1 -- 172.21.15.3:0/2344690666 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278009790 con 0x7fb288101000 2023-09-10T01:08:34.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2881a50f0 con 0x7fb288101000 2023-09-10T01:08:34.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb27e7fc700 1 -- 172.21.15.3:0/2344690666 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb27800ce80 con 0x7fb288101000 2023-09-10T01:08:34.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.262+0000 7fb27e7fc700 1 -- 172.21.15.3:0/2344690666 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278018680 con 0x7fb288101000 2023-09-10T01:08:34.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.263+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2881a55d0 con 0x7fb288101000 2023-09-10T01:08:34.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.266+0000 7fb27e7fc700 1 -- 172.21.15.3:0/2344690666 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fb278010430 con 0x7fb288101000 2023-09-10T01:08:34.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.267+0000 7fb27e7fc700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb2740a07b0 0x7fb2740a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.268+0000 7fb2a2e30b80 1 Processor -- start 2023-09-10T01:08:34.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.268+0000 7fb27e7fc700 1 -- 172.21.15.3:0/2344690666 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fb278014070 con 0x7fb288101000 2023-09-10T01:08:34.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.268+0000 7fb28d39b700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb2740a07b0 0x7fb2740a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.268+0000 7fb2a2e30b80 1 -- start start 2023-09-10T01:08:34.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839a6710 0x5555839a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839b0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555838ff730 con 0x5555839b0e40 2023-09-10T01:08:34.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555838fd050 con 0x5555839a6710 2023-09-10T01:08:34.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555838f8940 con 0x5555839a7030 2023-09-10T01:08:34.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28d39b700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb2740a07b0 0x7fb2740a2c70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb284005fd0 tx=0x7fb284009310 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28d39b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839a6710 0x5555839a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49894/0 (socket says 172.21.15.3:49894) 2023-09-10T01:08:34.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3769529645 learned_addr learned my addr 172.21.15.3:0/3769529645 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28d39b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839a6710 0x5555839a6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45440/0 (socket says 172.21.15.3:45440) 2023-09-10T01:08:34.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28db9c700 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3769529645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839a6710 msgr2=0x5555839a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839a6710 0x5555839a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3769529645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 msgr2=0x5555839b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3769529645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555583724de0 con 0x5555839a7030 2023-09-10T01:08:34.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.269+0000 7fb28d39b700 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839a6710 0x5555839a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:34.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.270+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839b0900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb2780a1230 tx=0x7fb278099360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b42ad28d7e220b66 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.270+0000 7fb2737fe700 1 -- 172.21.15.3:0/3769529645 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27809d520 con 0x5555839a7030 2023-09-10T01:08:34.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.270+0000 7fb2737fe700 1 -- 172.21.15.3:0/3769529645 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2780a1760 con 0x5555839a7030 2023-09-10T01:08:34.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.270+0000 7fb2737fe700 1 -- 172.21.15.3:0/3769529645 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2780cc420 con 0x5555839a7030 2023-09-10T01:08:34.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.270+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3769529645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 msgr2=0x5555839b0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.270+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839b0900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb2780a1230 tx=0x7fb278099360 comp rx=0 tx=0).stop 2023-09-10T01:08:34.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3769529645 shutdown_connections 2023-09-10T01:08:34.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839a6710 0x5555839a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3769529645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839b0900 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3769529645 >> 172.21.15.3:0/3769529645 conn(0x55558391b090 msgr2=0x5555839a1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3769529645 shutdown_connections 2023-09-10T01:08:34.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3769529645 wait complete. 2023-09-10T01:08:34.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 -- start start 2023-09-10T01:08:34.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.271+0000 7fb2a2e30b80 1 Processor -- start 2023-09-10T01:08:34.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839a6710 0x5555839adb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x5555839ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555583896b70 con 0x5555839a6710 2023-09-10T01:08:34.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555583896870 con 0x5555839b0e40 2023-09-10T01:08:34.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558377a920 con 0x5555839a7030 2023-09-10T01:08:34.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb2a2e30b80 10 client.0 _ll_register_callbacks cb 0x5555837e0a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:34.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb28cb9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb28cb9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839ae0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49902/0 (socket says 172.21.15.3:49902) 2023-09-10T01:08:34.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3580152156 learned_addr learned my addr 172.21.15.3:0/3580152156 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.272+0000 7fb28d39b700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839a6710 0x5555839adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28db9c700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x5555839ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 msgr2=0x5555839ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x5555839ae5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3580152156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839a6710 msgr2=0x5555839adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839a6710 0x5555839adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28cb9a700 1 -- 172.21.15.3:0/3580152156 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb284017070 con 0x5555839a7030 2023-09-10T01:08:34.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28db9c700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x5555839ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:08:34.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839ae0a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb278099680 tx=0x7fb278010940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9a06ca8bffdb60b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb270ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:34.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb270ff9700 1 -- 172.21.15.3:0/3580152156 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27809d5a0 con 0x5555839a7030 2023-09-10T01:08:34.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5555838504f0 con 0x5555839a7030 2023-09-10T01:08:34.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb270ff9700 1 -- 172.21.15.3:0/3580152156 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fb278099950 con 0x5555839a7030 2023-09-10T01:08:34.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb270ff9700 1 -- 172.21.15.3:0/3580152156 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb278099b10 con 0x5555839a7030 2023-09-10T01:08:34.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb2a2e30b80 10 client.27079 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.273+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fb2881a55d0 con 0x5555839a7030 2023-09-10T01:08:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb270ff9700 1 -- 172.21.15.3:0/3580152156 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5555838504f0 con 0x5555839a7030 2023-09-10T01:08:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb270ff9700 1 -- 172.21.15.3:0/3580152156 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb2780a12d0 con 0x5555839a7030 2023-09-10T01:08:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb2a2e30b80 10 client.27079 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb2a2e30b80 10 client.27079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb2a2e30b80 20 client.27079 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:34.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5555839c05d0 0x5555839c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb2a2e30b80 4 client.27079 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:34.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55558390d8a0 con 0x5555839c05d0 2023-09-10T01:08:34.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.274+0000 7fb28d39b700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5555839c05d0 0x5555839c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.275+0000 7fb28d39b700 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5555839c05d0 0x5555839c29f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcb3579d51afbac7 server_cookie=261fb2963d8d0bc3 in_seq=0 out_seq=0 2023-09-10T01:08:34.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.275+0000 7fb270ff9700 10 client.27079 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.276+0000 7fb270ff9700 1 -- 172.21.15.3:0/3580152156 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55558390d8a0 con 0x5555839c05d0 2023-09-10T01:08:34.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.276+0000 7fb270ff9700 10 client.27079 handle_command_reply: tid=0 2023-09-10T01:08:34.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.276+0000 7fb2a2e30b80 1 client.27079 shutdown 2023-09-10T01:08:34.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.276+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 msgr2=0x5555839ae0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.276+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839ae0a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb278099680 tx=0x7fb278010940 comp rx=0 tx=0).stop 2023-09-10T01:08:34.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.276+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 shutdown_connections 2023-09-10T01:08:34.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.276+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5555839c05d0 0x5555839c29f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x5555839ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839a6710 0x5555839adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/3580152156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a7030 0x5555839ae0a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 >> 172.21.15.3:0/3580152156 conn(0x55558391b090 msgr2=0x5555839a1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb270ff9700 0 client.27079 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 shutdown_connections 2023-09-10T01:08:34.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/3580152156 wait complete. 2023-09-10T01:08:34.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.277+0000 7fb2a2e30b80 20 client.27079 trim_cache size 0 max 16384 2023-09-10T01:08:34.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 Processor -- start 2023-09-10T01:08:34.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 -- start start 2023-09-10T01:08:34.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839ad400 0x5555839ab310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555839b7f20 con 0x5555839b0e40 2023-09-10T01:08:34.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555839c3070 con 0x5555839ad400 2023-09-10T01:08:34.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.296+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555839bc850 con 0x5555839a6710 2023-09-10T01:08:34.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49916/0 (socket says 172.21.15.3:49916) 2023-09-10T01:08:34.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 -- 172.21.15.3:0/541032106 learned_addr learned my addr 172.21.15.3:0/541032106 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28db9c700 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839ad400 0x5555839ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 -- 172.21.15.3:0/541032106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839ad400 msgr2=0x5555839ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839ad400 0x5555839ab310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 -- 172.21.15.3:0/541032106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 msgr2=0x5555839aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 -- 172.21.15.3:0/541032106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55558370dc60 con 0x5555839a6710 2023-09-10T01:08:34.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28db9c700 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839ad400 0x5555839ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:34.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:34.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.297+0000 7fb28d39b700 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b79e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x5555839c32d0 tx=0x7fb284014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2753927ca21eb920 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb273fff700 1 -- 172.21.15.3:0/541032106 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28402e070 con 0x5555839a6710 2023-09-10T01:08:34.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb273fff700 1 -- 172.21.15.3:0/541032106 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb284011cd0 con 0x5555839a6710 2023-09-10T01:08:34.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb273fff700 1 -- 172.21.15.3:0/541032106 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284024040 con 0x5555839a6710 2023-09-10T01:08:34.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/541032106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 msgr2=0x5555839b79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b79e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x5555839c32d0 tx=0x7fb284014000 comp rx=0 tx=0).stop 2023-09-10T01:08:34.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/541032106 shutdown_connections 2023-09-10T01:08:34.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b79e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839ad400 0x5555839ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/541032106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839b0e40 0x5555839aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/541032106 >> 172.21.15.3:0/541032106 conn(0x55558391b090 msgr2=0x5555839bc140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.298+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/541032106 shutdown_connections 2023-09-10T01:08:34.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.299+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/541032106 wait complete. 2023-09-10T01:08:34.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.299+0000 7fb2a2e30b80 1 -- start start 2023-09-10T01:08:34.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.299+0000 7fb2a2e30b80 1 Processor -- start 2023-09-10T01:08:34.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839ad400 0x5555839b6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb2a2e30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x555583a14410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555839b7080 con 0x5555839ad400 2023-09-10T01:08:34.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555583a14950 con 0x5555839b0e40 2023-09-10T01:08:34.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb2a2e30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555583a14ac0 con 0x5555839a6710 2023-09-10T01:08:34.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb2a2e30b80 10 client.0 _ll_register_callbacks cb 0x7fb278005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:34.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb28d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb28d39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49924/0 (socket says 172.21.15.3:49924) 2023-09-10T01:08:34.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb28d39b700 1 -- 172.21.15.3:0/1084847341 learned_addr learned my addr 172.21.15.3:0/1084847341 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb28db9c700 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x555583a14410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839ad400 0x5555839b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb28d39b700 1 -- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 msgr2=0x555583a14410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.300+0000 7fb28d39b700 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x555583a14410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb28d39b700 1 -- 172.21.15.3:0/1084847341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839ad400 msgr2=0x5555839b6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb28d39b700 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839ad400 0x5555839b6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb28d39b700 1 -- 172.21.15.3:0/1084847341 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28810de10 con 0x5555839a6710 2023-09-10T01:08:34.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb28d39b700 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b6230 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb284029e60 tx=0x7fb2840057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d596b4335bc9dcf7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb270ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:34.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb270ff9700 1 -- 172.21.15.3:0/1084847341 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284022070 con 0x5555839a6710 2023-09-10T01:08:34.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb270ff9700 1 -- 172.21.15.3:0/1084847341 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fb284003a90 con 0x5555839a6710 2023-09-10T01:08:34.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2780a1760 con 0x5555839a6710 2023-09-10T01:08:34.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb270ff9700 1 -- 172.21.15.3:0/1084847341 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb28402e040 con 0x5555839a6710 2023-09-10T01:08:34.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb2a2e30b80 10 client.27085 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:34.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.301+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fb2881a55d0 con 0x5555839a6710 2023-09-10T01:08:34.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb270ff9700 1 -- 172.21.15.3:0/1084847341 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2780a1760 con 0x5555839a6710 2023-09-10T01:08:34.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb270ff9700 1 -- 172.21.15.3:0/1084847341 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb28401ec30 con 0x5555839a6710 2023-09-10T01:08:34.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb2a2e30b80 10 client.27085 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb2a2e30b80 10 client.27085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb2a2e30b80 20 client.27085 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555583a16220 0x555583a18640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb2a2e30b80 4 client.27085 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:08:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5555838f1920 con 0x555583a16220 2023-09-10T01:08:34.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.302+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555583a16220 0x555583a18640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.303+0000 7fb28cb9a700 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555583a16220 0x555583a18640 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918ced243018593f server_cookie=439b50ba8536115b in_seq=0 out_seq=0 2023-09-10T01:08:34.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.303+0000 7fb270ff9700 10 client.27085 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.303+0000 7fb270ff9700 1 -- 172.21.15.3:0/1084847341 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5555838f1920 con 0x555583a16220 2023-09-10T01:08:34.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.303+0000 7fb270ff9700 10 client.27085 handle_command_reply: tid=0 2023-09-10T01:08:34.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.303+0000 7fb2a2e30b80 1 client.27085 shutdown 2023-09-10T01:08:34.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 msgr2=0x5555839b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b6230 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb284029e60 tx=0x7fb2840057f0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 shutdown_connections 2023-09-10T01:08:34.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555583a16220 0x555583a18640 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5555839a6710 0x5555839b6230 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5555839b0e40 0x555583a14410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb270ff9700 0 client.27085 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 --2- 172.21.15.3:0/1084847341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5555839ad400 0x5555839b6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 >> 172.21.15.3:0/1084847341 conn(0x55558391b090 msgr2=0x5555839adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 shutdown_connections 2023-09-10T01:08:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 1 -- 172.21.15.3:0/1084847341 wait complete. 2023-09-10T01:08:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.304+0000 7fb2a2e30b80 20 client.27085 trim_cache size 0 max 16384 2023-09-10T01:08:34.319 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:08:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:08:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:08:34.320 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:08:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb2740a07b0 msgr2=0x7fb2740a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb2740a07b0 0x7fb2740a2c70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb284005fd0 tx=0x7fb284009310 comp rx=0 tx=0).stop 2023-09-10T01:08:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288101000 msgr2=0x7fb288072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288101000 0x7fb288072e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb27800e9a0 tx=0x7fb27800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 shutdown_connections 2023-09-10T01:08:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb2740a07b0 0x7fb2740a2c70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb288106650 0x7fb2881a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb288100640 0x7fb288072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 --2- 172.21.15.3:0/2344690666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb288101000 0x7fb288072e40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 >> 172.21.15.3:0/2344690666 conn(0x7fb2880fc170 msgr2=0x7fb2880fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 shutdown_connections 2023-09-10T01:08:34.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.305+0000 7fb28f5ff700 1 -- 172.21.15.3:0/2344690666 wait complete. 2023-09-10T01:08:34.325 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:08:34.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:08:34.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 -- 172.21.15.3:0/403534813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b100732c0 msgr2=0x7f9b101110d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 --2- 172.21.15.3:0/403534813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b100732c0 0x7f9b101110d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009990 tx=0x7f9b04009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 -- 172.21.15.3:0/403534813 shutdown_connections 2023-09-10T01:08:34.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 --2- 172.21.15.3:0/403534813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b10111610 0x7f9b10113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 --2- 172.21.15.3:0/403534813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100729a0 0x7f9b10072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 --2- 172.21.15.3:0/403534813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b100732c0 0x7f9b101110d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 -- 172.21.15.3:0/403534813 >> 172.21.15.3:0/403534813 conn(0x7f9b100fc0d0 msgr2=0x7f9b100fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 -- 172.21.15.3:0/403534813 shutdown_connections 2023-09-10T01:08:34.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.546+0000 7f9b165ff700 1 -- 172.21.15.3:0/403534813 wait complete. 2023-09-10T01:08:34.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.547+0000 7f9b165ff700 1 Processor -- start 2023-09-10T01:08:34.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.547+0000 7f9b165ff700 1 -- start start 2023-09-10T01:08:34.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b165ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b100729a0 0x7f9b1019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b165ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100732c0 0x7f9b1019d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b165ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b10111610 0x7f9b101a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b165ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b101a1d80 con 0x7f9b100732c0 2023-09-10T01:08:34.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b165ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b101a1ef0 con 0x7f9b10111610 2023-09-10T01:08:34.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b165ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b101a20b0 con 0x7f9b100729a0 2023-09-10T01:08:34.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b0ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b100729a0 0x7f9b1019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b0ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b100729a0 0x7f9b1019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49946/0 (socket says 172.21.15.3:49946) 2023-09-10T01:08:34.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b0ffff700 1 -- 172.21.15.3:0/3708148436 learned_addr learned my addr 172.21.15.3:0/3708148436 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b0f7fe700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100732c0 0x7f9b1019d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b14b9c700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b10111610 0x7f9b101a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.548+0000 7f9b0ffff700 1 -- 172.21.15.3:0/3708148436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b10111610 msgr2=0x7f9b101a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0ffff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b10111610 0x7f9b101a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0ffff700 1 -- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100732c0 msgr2=0x7f9b1019d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0ffff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100732c0 0x7f9b1019d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0ffff700 1 -- 172.21.15.3:0/3708148436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040095f0 con 0x7f9b100729a0 2023-09-10T01:08:34.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b14b9c700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b10111610 0x7f9b101a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:34.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0ffff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b100729a0 0x7f9b1019cda0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000e9a0 tx=0x7f9b0000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0d7fa700 1 -- 172.21.15.3:0/3708148436 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00009790 con 0x7f9b100729a0 2023-09-10T01:08:34.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0d7fa700 1 -- 172.21.15.3:0/3708148436 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b00010430 con 0x7f9b100729a0 2023-09-10T01:08:34.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b0d7fa700 1 -- 172.21.15.3:0/3708148436 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00018680 con 0x7f9b100729a0 2023-09-10T01:08:34.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b101a2390 con 0x7f9b100729a0 2023-09-10T01:08:34.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.549+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b101a9bd0 con 0x7f9b100729a0 2023-09-10T01:08:34.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.551+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b10103af0 con 0x7f9b100729a0 2023-09-10T01:08:34.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.553+0000 7f9b0d7fa700 1 -- 172.21.15.3:0/3708148436 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9b000105a0 con 0x7f9b100729a0 2023-09-10T01:08:34.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.554+0000 7f9b0d7fa700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9af80a07b0 0x7f9af80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.554+0000 7f9b0d7fa700 1 -- 172.21.15.3:0/3708148436 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9b00014070 con 0x7f9b100729a0 2023-09-10T01:08:34.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.555+0000 7f9b0f7fe700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9af80a07b0 0x7f9af80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.555+0000 7f9b0f7fe700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9af80a07b0 0x7f9af80a2c70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b1019e3b0 tx=0x7f9b04005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.559+0000 7f9b0d7fa700 1 -- 172.21.15.3:0/3708148436 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9b00095c20 con 0x7f9b100729a0 2023-09-10T01:08:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.611+0000 7f05ebfff700 1 -- 172.21.15.3:0/2333335901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40729a0 msgr2=0x7f05e4072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.611+0000 7f05ebfff700 1 --2- 172.21.15.3:0/2333335901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40729a0 0x7f05e4072d80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f05e00098f0 tx=0x7f05e0009c00 comp rx=0 tx=0).stop 2023-09-10T01:08:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.612+0000 7f05ebfff700 1 -- 172.21.15.3:0/2333335901 shutdown_connections 2023-09-10T01:08:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.612+0000 7f05ebfff700 1 --2- 172.21.15.3:0/2333335901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e41115b0 0x7f05e41139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.612+0000 7f05ebfff700 1 --2- 172.21.15.3:0/2333335901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40729a0 0x7f05e4072d80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.612+0000 7f05ebfff700 1 --2- 172.21.15.3:0/2333335901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e40732c0 0x7f05e4111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.612+0000 7f05ebfff700 1 -- 172.21.15.3:0/2333335901 >> 172.21.15.3:0/2333335901 conn(0x7f05e40fc0b0 msgr2=0x7f05e40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.612+0000 7f05ebfff700 1 -- 172.21.15.3:0/2333335901 shutdown_connections 2023-09-10T01:08:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.612+0000 7f05ebfff700 1 -- 172.21.15.3:0/2333335901 wait complete. 2023-09-10T01:08:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 Processor -- start 2023-09-10T01:08:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 -- start start 2023-09-10T01:08:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e40729a0 0x7f05e419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40732c0 0x7f05e419d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e41115b0 0x7f05e41a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e41a1d50 con 0x7f05e40732c0 2023-09-10T01:08:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e41a1ec0 con 0x7f05e41115b0 2023-09-10T01:08:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.613+0000 7f05ebfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e41a2080 con 0x7f05e40729a0 2023-09-10T01:08:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05e9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e40729a0 0x7f05e419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05ea59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e41115b0 0x7f05e41a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05e9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e40729a0 0x7f05e419cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49962/0 (socket says 172.21.15.3:49962) 2023-09-10T01:08:34.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05e9d9b700 1 -- 172.21.15.3:0/4241507623 learned_addr learned my addr 172.21.15.3:0/4241507623 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05e959a700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40732c0 0x7f05e419d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05ea59c700 1 -- 172.21.15.3:0/4241507623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e40729a0 msgr2=0x7f05e419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05ea59c700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e40729a0 0x7f05e419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05ea59c700 1 -- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40732c0 msgr2=0x7f05e419d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05ea59c700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40732c0 0x7f05e419d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05ea59c700 1 -- 172.21.15.3:0/4241507623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d40095f0 con 0x7f05e41115b0 2023-09-10T01:08:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e40729a0 0x7f05e419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05e959a700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40732c0 0x7f05e419d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.614+0000 7f05ea59c700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e41115b0 0x7f05e41a1710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00ef90 tx=0x7f05dc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.615+0000 7f05daffd700 1 -- 172.21.15.3:0/4241507623 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc018070 con 0x7f05e41115b0 2023-09-10T01:08:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.615+0000 7f05daffd700 1 -- 172.21.15.3:0/4241507623 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05dc00f040 con 0x7f05e41115b0 2023-09-10T01:08:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.615+0000 7f05daffd700 1 -- 172.21.15.3:0/4241507623 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc00cc30 con 0x7f05e41115b0 2023-09-10T01:08:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.615+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e0009520 con 0x7f05e41115b0 2023-09-10T01:08:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.615+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e41a99c0 con 0x7f05e41115b0 2023-09-10T01:08:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.619+0000 7f05daffd700 1 -- 172.21.15.3:0/4241507623 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f05dc003a00 con 0x7f05e41115b0 2023-09-10T01:08:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.621+0000 7f05daffd700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05d00a0830 0x7f05d00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.621+0000 7f05daffd700 1 -- 172.21.15.3:0/4241507623 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f05dc0cbe80 con 0x7f05e41115b0 2023-09-10T01:08:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.621+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05d00a0830 0x7f05d00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.621+0000 7f05ff829b80 1 Processor -- start 2023-09-10T01:08:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ff829b80 1 -- start start 2023-09-10T01:08:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1da9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1da3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1daff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ff829b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1cfc6d0 con 0x564ec1da3300 2023-09-10T01:08:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ff829b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1cf9ff0 con 0x564ec1da91a0 2023-09-10T01:08:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ff829b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1cf58e0 con 0x564ec1da3cc0 2023-09-10T01:08:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05d00a0830 0x7f05d00a2cf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c010 tx=0x7f05e0019040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.622+0000 7f05ea59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1daff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1daff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49970/0 (socket says 172.21.15.3:49970) 2023-09-10T01:08:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 -- 172.21.15.3:0/1659684696 learned_addr learned my addr 172.21.15.3:0/1659684696 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05e959a700 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1da3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1da9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 -- 172.21.15.3:0/1659684696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 msgr2=0x564ec1da9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1da9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 -- 172.21.15.3:0/1659684696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 msgr2=0x564ec1da3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1da3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 -- 172.21.15.3:0/1659684696 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ec1b21d80 con 0x564ec1da3cc0 2023-09-10T01:08:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05e959a700 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1da3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05ea59c700 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1daff30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0a0a80 tx=0x7f05dc098be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37dc2a55cf4f7ed server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.623+0000 7f05cffff700 1 -- 172.21.15.3:0/1659684696 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc098fd0 con 0x564ec1da3cc0 2023-09-10T01:08:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05cffff700 1 -- 172.21.15.3:0/1659684696 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05dc09ce00 con 0x564ec1da3cc0 2023-09-10T01:08:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05cffff700 1 -- 172.21.15.3:0/1659684696 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc093be0 con 0x564ec1da3cc0 2023-09-10T01:08:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 -- 172.21.15.3:0/1659684696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 msgr2=0x564ec1daff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1daff30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0a0a80 tx=0x7f05dc098be0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 -- 172.21.15.3:0/1659684696 shutdown_connections 2023-09-10T01:08:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1daff30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1da9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1659684696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1da3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 -- 172.21.15.3:0/1659684696 >> 172.21.15.3:0/1659684696 conn(0x564ec1d18030 msgr2=0x564ec1d9e560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 -- 172.21.15.3:0/1659684696 shutdown_connections 2023-09-10T01:08:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.624+0000 7f05ff829b80 1 -- 172.21.15.3:0/1659684696 wait complete. 2023-09-10T01:08:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.625+0000 7f05ff829b80 1 -- start start 2023-09-10T01:08:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.625+0000 7f05ff829b80 1 Processor -- start 2023-09-10T01:08:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1dafe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dad120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1dad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ff829b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1c93b10 con 0x564ec1da91a0 2023-09-10T01:08:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ff829b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1c93810 con 0x564ec1da3300 2023-09-10T01:08:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ff829b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1b778c0 con 0x564ec1da3cc0 2023-09-10T01:08:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ff829b80 10 client.0 _ll_register_callbacks cb 0x564ec1bdda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05e959a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1dafe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05e959a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dad120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:49978/0 (socket says 172.21.15.3:49978) 2023-09-10T01:08:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1dafe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45534/0 (socket says 172.21.15.3:45534) 2023-09-10T01:08:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05e959a700 1 -- 172.21.15.3:0/1948181185 learned_addr learned my addr 172.21.15.3:0/1948181185 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.626+0000 7f05ea59c700 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1dad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05e959a700 1 -- 172.21.15.3:0/1948181185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 msgr2=0x564ec1dafe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05e959a700 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1dafe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05e959a700 1 -- 172.21.15.3:0/1948181185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 msgr2=0x564ec1dad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05e959a700 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1dad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05e959a700 1 -- 172.21.15.3:0/1948181185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e41a99c0 con 0x564ec1da3cc0 2023-09-10T01:08:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05e959a700 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dad120 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f05d4012cf0 tx=0x7f05d4009e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e16580e89851f60b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05cd7fa700 1 -- 172.21.15.3:0/1948181185 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4017070 con 0x564ec1da3cc0 2023-09-10T01:08:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ec1c4d490 con 0x564ec1da3cc0 2023-09-10T01:08:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05cd7fa700 1 -- 172.21.15.3:0/1948181185 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f05d400f970 con 0x564ec1da3cc0 2023-09-10T01:08:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05cd7fa700 1 -- 172.21.15.3:0/1948181185 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05d400fb70 con 0x564ec1da3cc0 2023-09-10T01:08:34.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05ff829b80 10 client.27103 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:34.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.627+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564ec1b0ac00 con 0x564ec1da3cc0 2023-09-10T01:08:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05cd7fa700 1 -- 172.21.15.3:0/1948181185 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564ec1c4d490 con 0x564ec1da3cc0 2023-09-10T01:08:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05cd7fa700 1 -- 172.21.15.3:0/1948181185 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f05d40038f0 con 0x564ec1da3cc0 2023-09-10T01:08:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05ff829b80 10 client.27103 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05ff829b80 10 client.27103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05ff829b80 20 client.27103 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1dbd520 0x564ec1dbf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05ff829b80 4 client.27103 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.628+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ec1d0a840 con 0x564ec1dbd520 2023-09-10T01:08:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.629+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1dbd520 0x564ec1dbf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.629+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1dbd520 0x564ec1dbf940 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee0088ad921c10eb server_cookie=82bb15aa862cd46d in_seq=0 out_seq=0 2023-09-10T01:08:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.629+0000 7f05cd7fa700 10 client.27103 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05cd7fa700 1 -- 172.21.15.3:0/1948181185 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564ec1d0a840 con 0x564ec1dbd520 2023-09-10T01:08:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05cd7fa700 10 client.27103 handle_command_reply: tid=0 2023-09-10T01:08:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 client.27103 shutdown 2023-09-10T01:08:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 msgr2=0x564ec1dad120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dad120 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f05d4012cf0 tx=0x7f05d4009e80 comp rx=0 tx=0).stop 2023-09-10T01:08:34.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 shutdown_connections 2023-09-10T01:08:34.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1dbd520 0x564ec1dbf940 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1dad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1dafe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05cd7fa700 0 client.27103 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 --2- 172.21.15.3:0/1948181185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dad120 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 >> 172.21.15.3:0/1948181185 conn(0x564ec1d18030 msgr2=0x564ec1d9d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 shutdown_connections 2023-09-10T01:08:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 1 -- 172.21.15.3:0/1948181185 wait complete. 2023-09-10T01:08:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.630+0000 7f05ff829b80 20 client.27103 trim_cache size 0 max 16384 2023-09-10T01:08:34.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.649+0000 7f05ff829b80 1 Processor -- start 2023-09-10T01:08:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.649+0000 7f05ff829b80 1 -- start start 2023-09-10T01:08:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1dad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1daf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ff829b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1dbab50 con 0x564ec1da3300 2023-09-10T01:08:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ff829b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1dbffa0 con 0x564ec1da91a0 2023-09-10T01:08:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ff829b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1dab680 con 0x564ec1da3cc0 2023-09-10T01:08:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ea59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1daf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ea59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1daf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49274/0 (socket says 172.21.15.3:49274) 2023-09-10T01:08:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05ea59c700 1 -- 172.21.15.3:0/875641023 learned_addr learned my addr 172.21.15.3:0/875641023 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1dad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.650+0000 7f05e959a700 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05ea59c700 1 -- 172.21.15.3:0/875641023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 msgr2=0x564ec1dac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05ea59c700 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05ea59c700 1 -- 172.21.15.3:0/875641023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 msgr2=0x564ec1dad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05ea59c700 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1dad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05ea59c700 1 -- 172.21.15.3:0/875641023 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ec1b481f0 con 0x564ec1da91a0 2023-09-10T01:08:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05ea59c700 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1daf930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0090a0 tx=0x7f05dc098b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca5b2b7525c5e3d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05d8ff9700 1 -- 172.21.15.3:0/875641023 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0a0fb0 con 0x564ec1da91a0 2023-09-10T01:08:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05d8ff9700 1 -- 172.21.15.3:0/875641023 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05dc0092e0 con 0x564ec1da91a0 2023-09-10T01:08:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.651+0000 7f05d8ff9700 1 -- 172.21.15.3:0/875641023 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0b1380 con 0x564ec1da91a0 2023-09-10T01:08:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 -- 172.21.15.3:0/875641023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 msgr2=0x564ec1daf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1daf930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0090a0 tx=0x7f05dc098b10 comp rx=0 tx=0).stop 2023-09-10T01:08:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 -- 172.21.15.3:0/875641023 shutdown_connections 2023-09-10T01:08:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da91a0 0x564ec1daf930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da3300 0x564ec1dad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 --2- 172.21.15.3:0/875641023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1dac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 -- 172.21.15.3:0/875641023 >> 172.21.15.3:0/875641023 conn(0x564ec1d18030 msgr2=0x564ec1da71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 -- 172.21.15.3:0/875641023 shutdown_connections 2023-09-10T01:08:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 -- 172.21.15.3:0/875641023 wait complete. 2023-09-10T01:08:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.652+0000 7f05ff829b80 1 -- start start 2023-09-10T01:08:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 1 Processor -- start 2023-09-10T01:08:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1db41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1db4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1db2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1db2e20 con 0x564ec1da91a0 2023-09-10T01:08:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1db2f90 con 0x564ec1da3300 2023-09-10T01:08:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec1db3150 con 0x564ec1da3cc0 2023-09-10T01:08:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ff829b80 10 client.0 _ll_register_callbacks cb 0x7f05d40127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05e959a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1db4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1db41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1db41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49288/0 (socket says 172.21.15.3:49288) 2023-09-10T01:08:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05e9d9b700 1 -- 172.21.15.3:0/46957409 learned_addr learned my addr 172.21.15.3:0/46957409 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.653+0000 7f05ea59c700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1db2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.654+0000 7f05e9d9b700 1 -- 172.21.15.3:0/46957409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 msgr2=0x564ec1db4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.654+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1db4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.654+0000 7f05e9d9b700 1 -- 172.21.15.3:0/46957409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 msgr2=0x564ec1db2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:34.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.654+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1db2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.654+0000 7f05e9d9b700 1 -- 172.21.15.3:0/46957409 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ec1b21d80 con 0x564ec1da3300 2023-09-10T01:08:34.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.654+0000 7f05e959a700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1db4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:34.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.654+0000 7f05ea59c700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1db2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.655+0000 7f05e9d9b700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1db41d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009230 tx=0x7f05e0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68a227f1b951d3e1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.655+0000 7f05cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.655+0000 7f05cd7fa700 1 -- 172.21.15.3:0/46957409 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0034d20 con 0x564ec1da3300 2023-09-10T01:08:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.655+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05dc09ce00 con 0x564ec1da3300 2023-09-10T01:08:34.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.655+0000 7f05cd7fa700 1 -- 172.21.15.3:0/46957409 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f05e000b5e0 con 0x564ec1da3300 2023-09-10T01:08:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05cd7fa700 1 -- 172.21.15.3:0/46957409 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05e0034e80 con 0x564ec1da3300 2023-09-10T01:08:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 10 client.27191 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564ec1b20250 con 0x564ec1da3300 2023-09-10T01:08:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05cd7fa700 1 -- 172.21.15.3:0/46957409 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05dc09ce00 con 0x564ec1da3300 2023-09-10T01:08:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05cd7fa700 1 -- 172.21.15.3:0/46957409 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f05e0003680 con 0x564ec1da3300 2023-09-10T01:08:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 10 client.27191 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 10 client.27191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 20 client.27191 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1e126b0 0x564ec1e14ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 4 client.27191 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:08:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.656+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564ec1cee8c0 con 0x564ec1e126b0 2023-09-10T01:08:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.657+0000 7f05e959a700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1e126b0 0x564ec1e14ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.657+0000 7f05cd7fa700 10 client.27191 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.657+0000 7f05e959a700 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1e126b0 0x564ec1e14ad0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84553152d8dfcca1 server_cookie=255030c2522207a6 in_seq=0 out_seq=0 2023-09-10T01:08:34.761 INFO:tasks.workunit.client.0.smithi003.stdout: 1 457911 9.42 MB/sec execute 136 sec latency 94.946 ms 2023-09-10T01:08:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.840+0000 7f05cd7fa700 1 -- 172.21.15.3:0/46957409 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564ec1cee8c0 con 0x564ec1e126b0 2023-09-10T01:08:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.840+0000 7f05cd7fa700 10 client.27191 handle_command_reply: tid=0 2023-09-10T01:08:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.840+0000 7f05ff829b80 1 client.27191 shutdown 2023-09-10T01:08:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.841+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 msgr2=0x564ec1db41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.841+0000 7f05ff829b80 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1db41d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009230 tx=0x7f05e0009260 comp rx=0 tx=0).stop 2023-09-10T01:08:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.841+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 shutdown_connections 2023-09-10T01:08:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.841+0000 7f05ff829b80 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564ec1e126b0 0x564ec1e14ad0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.842+0000 7f05ff829b80 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ec1da91a0 0x564ec1db2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.842+0000 7f05ff829b80 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ec1da3300 0x564ec1db41d0 secure :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009230 tx=0x7f05e0009260 comp rx=0 tx=0).stop 2023-09-10T01:08:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.842+0000 7f05ff829b80 1 --2- 172.21.15.3:0/46957409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564ec1da3cc0 0x564ec1db4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.842+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 >> 172.21.15.3:0/46957409 conn(0x564ec1d18030 msgr2=0x564ec1da22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.842+0000 7f05cd7fa700 0 client.27191 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.843+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 shutdown_connections 2023-09-10T01:08:34.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.843+0000 7f05ff829b80 1 -- 172.21.15.3:0/46957409 wait complete. 2023-09-10T01:08:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.846+0000 7f05ff829b80 20 client.27191 trim_cache size 0 max 16384 2023-09-10T01:08:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:08:34.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:08:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:08:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:08:34.848 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.847+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05d00a0830 msgr2=0x7f05d00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.847+0000 7f05ebfff700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05d00a0830 0x7f05d00a2cf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c010 tx=0x7f05e0019040 comp rx=0 tx=0).stop 2023-09-10T01:08:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.847+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e41115b0 msgr2=0x7f05e41a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.847+0000 7f05ebfff700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e41115b0 0x7f05e41a1710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00ef90 tx=0x7f05dc00c3a0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 shutdown_connections 2023-09-10T01:08:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05d00a0830 0x7f05d00a2cf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05e40729a0 0x7f05e419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05e41115b0 0x7f05e41a1710 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 --2- 172.21.15.3:0/4241507623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05e40732c0 0x7f05e419d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 >> 172.21.15.3:0/4241507623 conn(0x7f05e40fc0b0 msgr2=0x7f05e40fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 shutdown_connections 2023-09-10T01:08:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:34.848+0000 7f05ebfff700 1 -- 172.21.15.3:0/4241507623 wait complete. 2023-09-10T01:08:34.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.920+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b100678d0 con 0x7f9b100729a0 2023-09-10T01:08:34.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.920+0000 7f9b0d7fa700 1 -- 172.21.15.3:0/3708148436 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11293 (secure 0 0 0) 0x7f9b00099240 con 0x7f9b100729a0 2023-09-10T01:08:34.922 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:08:34.922 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:07:08.821454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:08:34.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.923+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9af80a07b0 msgr2=0x7f9af80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.923+0000 7f9b165ff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9af80a07b0 0x7f9af80a2c70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b1019e3b0 tx=0x7f9b04005fb0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.923+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b100729a0 msgr2=0x7f9b1019cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:34.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.923+0000 7f9b165ff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b100729a0 0x7f9b1019cda0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000e9a0 tx=0x7f9b0000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 shutdown_connections 2023-09-10T01:08:34.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9af80a07b0 0x7f9af80a2c70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b100729a0 0x7f9b1019cda0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b10111610 0x7f9b101a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 --2- 172.21.15.3:0/3708148436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100732c0 0x7f9b1019d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 >> 172.21.15.3:0/3708148436 conn(0x7f9b100fc0d0 msgr2=0x7f9b100fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:34.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 shutdown_connections 2023-09-10T01:08:34.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:34.924+0000 7f9b165ff700 1 -- 172.21.15.3:0/3708148436 wait complete. 2023-09-10T01:08:34.927 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 26 2023-09-10T01:08:34.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:08:35.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 -- 172.21.15.3:0/1147855372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c106650 msgr2=0x7f7f6c106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 --2- 172.21.15.3:0/1147855372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c106650 0x7f7f6c106a30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009990 tx=0x7f7f60009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 -- 172.21.15.3:0/1147855372 shutdown_connections 2023-09-10T01:08:35.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 --2- 172.21.15.3:0/1147855372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f6c101000 0x7f7f6c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 --2- 172.21.15.3:0/1147855372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c106650 0x7f7f6c106a30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 --2- 172.21.15.3:0/1147855372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f6c100640 0x7f7f6c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 -- 172.21.15.3:0/1147855372 >> 172.21.15.3:0/1147855372 conn(0x7f7f6c0fc170 msgr2=0x7f7f6c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 -- 172.21.15.3:0/1147855372 shutdown_connections 2023-09-10T01:08:35.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.165+0000 7f7f72bff700 1 -- 172.21.15.3:0/1147855372 wait complete. 2023-09-10T01:08:35.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.166+0000 7f7f72bff700 1 Processor -- start 2023-09-10T01:08:35.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f72bff700 1 -- start start 2023-09-10T01:08:35.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f72bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f6c100640 0x7f7f6c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f72bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c101000 0x7f7f6c198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f72bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f6c106650 0x7f7f6c19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f72bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c19d900 con 0x7f7f6c106650 2023-09-10T01:08:35.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f72bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c19da70 con 0x7f7f6c101000 2023-09-10T01:08:35.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f72bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c19dc30 con 0x7f7f6c100640 2023-09-10T01:08:35.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c101000 0x7f7f6c198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f7099b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f6c100640 0x7f7f6c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c101000 0x7f7f6c198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49312/0 (socket says 172.21.15.3:49312) 2023-09-10T01:08:35.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f6bfff700 1 -- 172.21.15.3:0/4137221212 learned_addr learned my addr 172.21.15.3:0/4137221212 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.167+0000 7f7f7119c700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f6c106650 0x7f7f6c19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f6bfff700 1 -- 172.21.15.3:0/4137221212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f6c100640 msgr2=0x7f7f6c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f6bfff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f6c100640 0x7f7f6c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f6bfff700 1 -- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f6c106650 msgr2=0x7f7f6c19d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f6bfff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f6c106650 0x7f7f6c19d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f6bfff700 1 -- 172.21.15.3:0/4137221212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64009c40 con 0x7f7f6c101000 2023-09-10T01:08:35.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f7099b700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f6c100640 0x7f7f6c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:35.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f7119c700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f6c106650 0x7f7f6c19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:35.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f6bfff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c101000 0x7f7f6c198ea0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800e9a0 tx=0x7f7f5800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f69ffb700 1 -- 172.21.15.3:0/4137221212 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58009790 con 0x7f7f6c101000 2023-09-10T01:08:35.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f69ffb700 1 -- 172.21.15.3:0/4137221212 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f5800ce80 con 0x7f7f6c101000 2023-09-10T01:08:35.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f69ffb700 1 -- 172.21.15.3:0/4137221212 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58018680 con 0x7f7f6c101000 2023-09-10T01:08:35.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.168+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f600095f0 con 0x7f7f6c101000 2023-09-10T01:08:35.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.169+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c1a55e0 con 0x7f7f6c101000 2023-09-10T01:08:35.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.172+0000 7f7f69ffb700 1 -- 172.21.15.3:0/4137221212 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7f58010430 con 0x7f7f6c101000 2023-09-10T01:08:35.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.173+0000 7f7f865ceb80 1 Processor -- start 2023-09-10T01:08:35.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.173+0000 7f7f865ceb80 1 -- start start 2023-09-10T01:08:35.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f69ffb700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7f5c0a0830 0x7f7f5c0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc3710 0x559160cc3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160ccde40 0x559160cd0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f69ffb700 1 -- 172.21.15.3:0/4137221212 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7f58014070 con 0x7f7f6c101000 2023-09-10T01:08:35.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160c1c730 con 0x559160cc4030 2023-09-10T01:08:35.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160c1a050 con 0x559160cc3710 2023-09-10T01:08:35.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160c15940 con 0x559160ccde40 2023-09-10T01:08:35.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f7119c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160ccde40 0x559160cd0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f7119c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160ccde40 0x559160cd0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55408/0 (socket says 172.21.15.3:55408) 2023-09-10T01:08:35.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f7119c700 1 -- 172.21.15.3:0/2943105449 learned_addr learned my addr 172.21.15.3:0/2943105449 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f7099b700 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc3710 0x559160cc3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.174+0000 7f7f7119c700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7f5c0a0830 0x7f7f5c0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f7099b700 1 -- 172.21.15.3:0/2943105449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160ccde40 msgr2=0x559160cd0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f7099b700 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160ccde40 0x559160cd0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f7099b700 1 -- 172.21.15.3:0/2943105449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 msgr2=0x559160ccd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f7099b700 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f7099b700 1 -- 172.21.15.3:0/2943105449 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559160a41de0 con 0x559160cc3710 2023-09-10T01:08:35.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f7099b700 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc3710 0x559160cc3af0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7f60000c00 tx=0x7f7f60005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc9c17e9655e34a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f7119c700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7f5c0a0830 0x7f7f5c0a2cf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7f64008f70 tx=0x7f7f6400b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f56ffd700 1 -- 172.21.15.3:0/2943105449 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6001d050 con 0x559160cc3710 2023-09-10T01:08:35.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f56ffd700 1 -- 172.21.15.3:0/2943105449 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f6001e040 con 0x559160cc3710 2023-09-10T01:08:35.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.175+0000 7f7f56ffd700 1 -- 172.21.15.3:0/2943105449 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60005420 con 0x559160cc3710 2023-09-10T01:08:35.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2943105449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc3710 msgr2=0x559160cc3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc3710 0x559160cc3af0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7f60000c00 tx=0x7f7f60005ea0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2943105449 shutdown_connections 2023-09-10T01:08:35.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160ccde40 0x559160cd0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc3710 0x559160cc3af0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2943105449 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2943105449 >> 172.21.15.3:0/2943105449 conn(0x559160c38090 msgr2=0x559160cbebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2943105449 shutdown_connections 2023-09-10T01:08:35.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2943105449 wait complete. 2023-09-10T01:08:35.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.176+0000 7f7f865ceb80 1 -- start start 2023-09-10T01:08:35.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 1 Processor -- start 2023-09-10T01:08:35.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160ccab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160bb3b70 con 0x559160cc4030 2023-09-10T01:08:35.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160bb3870 con 0x559160ccde40 2023-09-10T01:08:35.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160a97920 con 0x559160cc3710 2023-09-10T01:08:35.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.177+0000 7f7f865ceb80 10 client.0 _ll_register_callbacks cb 0x559160afda80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:35.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160ccab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160ccab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55416/0 (socket says 172.21.15.3:55416) 2023-09-10T01:08:35.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 -- 172.21.15.3:0/2100967146 learned_addr learned my addr 172.21.15.3:0/2100967146 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f6bfff700 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7119c700 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 -- 172.21.15.3:0/2100967146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 msgr2=0x559160ccb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 -- 172.21.15.3:0/2100967146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 msgr2=0x559160ccb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 -- 172.21.15.3:0/2100967146 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c10de10 con 0x559160cc3710 2023-09-10T01:08:35.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f7099b700 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160ccab40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000b2e0 tx=0x7f7f60004a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9302f7230392057 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.178+0000 7f7f4bfff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:35.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f4bfff700 1 -- 172.21.15.3:0/2100967146 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60005190 con 0x559160cc3710 2023-09-10T01:08:35.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f4bfff700 1 -- 172.21.15.3:0/2100967146 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7f600052f0 con 0x559160cc3710 2023-09-10T01:08:35.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559160b6d4f0 con 0x559160cc3710 2023-09-10T01:08:35.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f4bfff700 1 -- 172.21.15.3:0/2100967146 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f60005980 con 0x559160cc3710 2023-09-10T01:08:35.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f865ceb80 10 client.27121 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:35.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x559160a2ac60 con 0x559160cc3710 2023-09-10T01:08:35.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f4bfff700 1 -- 172.21.15.3:0/2100967146 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559160b6d4f0 con 0x559160cc3710 2023-09-10T01:08:35.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f4bfff700 1 -- 172.21.15.3:0/2100967146 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f7f60023460 con 0x559160cc3710 2023-09-10T01:08:35.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f865ceb80 10 client.27121 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:35.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f865ceb80 10 client.27121 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:08:35.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f865ceb80 20 client.27121 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:35.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.179+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160cdd440 0x559160cdf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.180+0000 7f7f865ceb80 4 client.27121 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:35.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.180+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559160c2a8a0 con 0x559160cdd440 2023-09-10T01:08:35.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.180+0000 7f7f6bfff700 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160cdd440 0x559160cdf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.180+0000 7f7f6bfff700 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160cdd440 0x559160cdf830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=32debcf8af1680cf server_cookie=15235f3335f136b9 in_seq=0 out_seq=0 2023-09-10T01:08:35.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.180+0000 7f7f4bfff700 10 client.27121 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:35.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f4bfff700 1 -- 172.21.15.3:0/2100967146 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559160c2a8a0 con 0x559160cdd440 2023-09-10T01:08:35.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f4bfff700 10 client.27121 handle_command_reply: tid=0 2023-09-10T01:08:35.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 client.27121 shutdown 2023-09-10T01:08:35.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 msgr2=0x559160ccab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160ccab40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000b2e0 tx=0x7f7f60004a90 comp rx=0 tx=0).stop 2023-09-10T01:08:35.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 shutdown_connections 2023-09-10T01:08:35.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160cdd440 0x559160cdf830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160ccab40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/2100967146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc4030 0x559160ccb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f4bfff700 0 client.27121 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:35.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 >> 172.21.15.3:0/2100967146 conn(0x559160c38090 msgr2=0x559160cbe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 shutdown_connections 2023-09-10T01:08:35.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.181+0000 7f7f865ceb80 1 -- 172.21.15.3:0/2100967146 wait complete. 2023-09-10T01:08:35.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.182+0000 7f7f865ceb80 20 client.27121 trim_cache size 0 max 16384 2023-09-10T01:08:35.203 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.195+0000 7f7f865ceb80 1 Processor -- start 2023-09-10T01:08:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.195+0000 7f7f865ceb80 1 -- start start 2023-09-10T01:08:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc3710 0x559160cd49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc4030 0x559160cd9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.204 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160cd4f20 con 0x559160cc3710 2023-09-10T01:08:35.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160cc90a0 con 0x559160ccde40 2023-09-10T01:08:35.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160cc66d0 con 0x559160cc4030 2023-09-10T01:08:35.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7099b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc3710 0x559160cd49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.205 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc4030 0x559160cd9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7099b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc3710 0x559160cd49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43098/0 (socket says 172.21.15.3:43098) 2023-09-10T01:08:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49334/0 (socket says 172.21.15.3:49334) 2023-09-10T01:08:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 -- 172.21.15.3:0/1639171641 learned_addr learned my addr 172.21.15.3:0/1639171641 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc4030 0x559160cd9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55426/0 (socket says 172.21.15.3:55426) 2023-09-10T01:08:35.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 -- 172.21.15.3:0/1639171641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc4030 msgr2=0x559160cd9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 --2- 172.21.15.3:0/1639171641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc4030 0x559160cd9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 -- 172.21.15.3:0/1639171641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc3710 msgr2=0x559160cd49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 --2- 172.21.15.3:0/1639171641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc3710 0x559160cd49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.196+0000 7f7f7119c700 1 -- 172.21.15.3:0/1639171641 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559160a68250 con 0x559160ccde40 2023-09-10T01:08:35.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.197+0000 7f7f7119c700 1 --2- 172.21.15.3:0/1639171641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb3f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400ecf0 tx=0x7f7f6400c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b736822cda617121 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.197+0000 7f7f577fe700 1 -- 172.21.15.3:0/1639171641 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64022050 con 0x559160ccde40 2023-09-10T01:08:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.197+0000 7f7f577fe700 1 -- 172.21.15.3:0/1639171641 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f64023040 con 0x559160ccde40 2023-09-10T01:08:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.197+0000 7f7f577fe700 1 -- 172.21.15.3:0/1639171641 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64004750 con 0x559160ccde40 2023-09-10T01:08:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.198+0000 7f7f865ceb80 1 -- 172.21.15.3:0/1639171641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 msgr2=0x559160ccb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.198+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/1639171641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb3f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400ecf0 tx=0x7f7f6400c000 comp rx=0 tx=0).stop 2023-09-10T01:08:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.198+0000 7f7f865ceb80 1 -- 172.21.15.3:0/1639171641 shutdown_connections 2023-09-10T01:08:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.198+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/1639171641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160ccde40 0x559160ccb3f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.198+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/1639171641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160cc3710 0x559160cd49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.198+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/1639171641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc4030 0x559160cd9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.198+0000 7f7f865ceb80 1 -- 172.21.15.3:0/1639171641 >> 172.21.15.3:0/1639171641 conn(0x559160c38090 msgr2=0x559160ce4180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.199+0000 7f7f865ceb80 1 -- 172.21.15.3:0/1639171641 shutdown_connections 2023-09-10T01:08:35.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.199+0000 7f7f865ceb80 1 -- 172.21.15.3:0/1639171641 wait complete. 2023-09-10T01:08:35.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.199+0000 7f7f865ceb80 1 -- start start 2023-09-10T01:08:35.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.200+0000 7f7f865ceb80 1 Processor -- start 2023-09-10T01:08:35.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.200+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160cd4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.200+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc4030 0x559160cd29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f865ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160ccde40 0x559160cd2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160cd3520 con 0x559160ccde40 2023-09-10T01:08:35.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160cd3690 con 0x559160cc4030 2023-09-10T01:08:35.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f865ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559160cd3850 con 0x559160cc3710 2023-09-10T01:08:35.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f865ceb80 10 client.0 _ll_register_callbacks cb 0x7f7f58095970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:35.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc4030 0x559160cd29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f7119c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160ccde40 0x559160cd2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f7119c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160ccde40 0x559160cd2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43108/0 (socket says 172.21.15.3:43108) 2023-09-10T01:08:35.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.201+0000 7f7f7119c700 1 -- 172.21.15.3:0/756540950 learned_addr learned my addr 172.21.15.3:0/756540950 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.205+0000 7f7f7119c700 1 -- 172.21.15.3:0/756540950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 msgr2=0x559160cd4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.205+0000 7f7f7119c700 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160cd4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.205+0000 7f7f7119c700 1 -- 172.21.15.3:0/756540950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc4030 msgr2=0x559160cd29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.205+0000 7f7f7119c700 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc4030 0x559160cd29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.205+0000 7f7f7119c700 1 -- 172.21.15.3:0/756540950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f64008ba0 con 0x559160ccde40 2023-09-10T01:08:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.207+0000 7f7f7119c700 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160ccde40 0x559160cd2f10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7f64029040 tx=0x7f7f64003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb72c751cb46238 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.208+0000 7f7f4bfff700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:08:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.208+0000 7f7f4bfff700 1 -- 172.21.15.3:0/756540950 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64004580 con 0x559160ccde40 2023-09-10T01:08:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.208+0000 7f7f4bfff700 1 -- 172.21.15.3:0/756540950 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f7f640046e0 con 0x559160ccde40 2023-09-10T01:08:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.209+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f6001e040 con 0x559160ccde40 2023-09-10T01:08:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.209+0000 7f7f4bfff700 1 -- 172.21.15.3:0/756540950 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f64022020 con 0x559160ccde40 2023-09-10T01:08:35.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.209+0000 7f7f865ceb80 10 client.17136 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.209+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x559160a402b0 con 0x559160ccde40 2023-09-10T01:08:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.211+0000 7f7f4bfff700 1 -- 172.21.15.3:0/756540950 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f6001e040 con 0x559160ccde40 2023-09-10T01:08:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.211+0000 7f7f4bfff700 1 -- 172.21.15.3:0/756540950 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f7f64030480 con 0x559160ccde40 2023-09-10T01:08:35.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.212+0000 7f7f865ceb80 10 client.17136 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:35.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.212+0000 7f7f865ceb80 10 client.17136 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:08:35.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.212+0000 7f7f865ceb80 20 client.17136 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:35.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.212+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160d327e0 0x559160d34bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.212+0000 7f7f865ceb80 4 client.17136 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:08:35.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.212+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559160c0e920 con 0x559160d327e0 2023-09-10T01:08:35.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.216+0000 7f7f7099b700 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160d327e0 0x559160d34bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.218+0000 7f7f7099b700 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160d327e0 0x559160d34bf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1efc949db510311c server_cookie=d177f97cf15f905 in_seq=0 out_seq=0 2023-09-10T01:08:35.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.219+0000 7f7f4bfff700 10 client.17136 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:35.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.222+0000 7f7f4bfff700 1 -- 172.21.15.3:0/756540950 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559160c0e920 con 0x559160d327e0 2023-09-10T01:08:35.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.222+0000 7f7f4bfff700 10 client.17136 handle_command_reply: tid=0 2023-09-10T01:08:35.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.224+0000 7f7f865ceb80 1 client.17136 shutdown 2023-09-10T01:08:35.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.235+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160ccde40 msgr2=0x559160cd2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.235+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160ccde40 0x559160cd2f10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7f64029040 tx=0x7f7f64003ef0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.237+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 shutdown_connections 2023-09-10T01:08:35.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.237+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559160d327e0 0x559160d34bf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.237+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559160cc3710 0x559160cd4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.237+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559160ccde40 0x559160cd2f10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.237+0000 7f7f865ceb80 1 --2- 172.21.15.3:0/756540950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559160cc4030 0x559160cd29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.237+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 >> 172.21.15.3:0/756540950 conn(0x559160c38090 msgr2=0x559160cd9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.239+0000 7f7f4bfff700 0 client.17136 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:08:35.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.242+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 shutdown_connections 2023-09-10T01:08:35.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.244+0000 7f7f865ceb80 1 -- 172.21.15.3:0/756540950 wait complete. 2023-09-10T01:08:35.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.250+0000 7f7f865ceb80 20 client.17136 trim_cache size 0 max 16384 2023-09-10T01:08:35.252 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:08:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.253+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7f5c0a0830 msgr2=0x7f7f5c0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.253+0000 7f7f72bff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7f5c0a0830 0x7f7f5c0a2cf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7f64008f70 tx=0x7f7f6400b040 comp rx=0 tx=0).stop 2023-09-10T01:08:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.253+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c101000 msgr2=0x7f7f6c198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.253+0000 7f7f72bff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c101000 0x7f7f6c198ea0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800e9a0 tx=0x7f7f5800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.256+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 shutdown_connections 2023-09-10T01:08:35.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.256+0000 7f7f72bff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7f5c0a0830 0x7f7f5c0a2cf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.256+0000 7f7f72bff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f6c100640 0x7f7f6c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.256+0000 7f7f72bff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f6c106650 0x7f7f6c19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.256+0000 7f7f72bff700 1 --2- 172.21.15.3:0/4137221212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f6c101000 0x7f7f6c198ea0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.256+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 >> 172.21.15.3:0/4137221212 conn(0x7f7f6c0fc170 msgr2=0x7f7f6c10bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.260+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 shutdown_connections 2023-09-10T01:08:35.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.261+0000 7f7f72bff700 1 -- 172.21.15.3:0/4137221212 wait complete. 2023-09-10T01:08:35.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:08:35.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/3729990430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 msgr2=0x7f4f58106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/3729990430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 0x7f4f58106a30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4f54009990 tx=0x7f4f54009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/3729990430 shutdown_connections 2023-09-10T01:08:35.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/3729990430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 0x7f4f5810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/3729990430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 0x7f4f58106a30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/3729990430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f58100640 0x7f4f58100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/3729990430 >> 172.21.15.3:0/3729990430 conn(0x7f4f580fc170 msgr2=0x7f4f580fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/3729990430 shutdown_connections 2023-09-10T01:08:35.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.513+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/3729990430 wait complete. 2023-09-10T01:08:35.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.514+0000 7f4f5f7ff700 1 Processor -- start 2023-09-10T01:08:35.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5f7ff700 1 -- start start 2023-09-10T01:08:35.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5f7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f58100640 0x7f4f581966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5f7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 0x7f4f58196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5f7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 0x7f4f58199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5f7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5819a5c0 con 0x7f4f58100640 2023-09-10T01:08:35.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5f7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5819a730 con 0x7f4f58106650 2023-09-10T01:08:35.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5f7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5819a8f0 con 0x7f4f58101000 2023-09-10T01:08:35.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5cd9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 0x7f4f58196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5d59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f58100640 0x7f4f581966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5dd9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 0x7f4f58199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5cd9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 0x7f4f58196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55454/0 (socket says 172.21.15.3:55454) 2023-09-10T01:08:35.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.515+0000 7f4f5cd9a700 1 -- 172.21.15.3:0/4078919037 learned_addr learned my addr 172.21.15.3:0/4078919037 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5cd9a700 1 -- 172.21.15.3:0/4078919037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 msgr2=0x7f4f58199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 0x7f4f58199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5cd9a700 1 -- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f58100640 msgr2=0x7f4f581966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f58100640 0x7f4f581966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5cd9a700 1 -- 172.21.15.3:0/4078919037 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f540095f0 con 0x7f4f58101000 2023-09-10T01:08:35.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5dd9c700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 0x7f4f58199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:35.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 0x7f4f58196c10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800e9a0 tx=0x7f4f4800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f4e7fc700 1 -- 172.21.15.3:0/4078919037 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48009790 con 0x7f4f58101000 2023-09-10T01:08:35.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5819abd0 con 0x7f4f58101000 2023-09-10T01:08:35.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f4e7fc700 1 -- 172.21.15.3:0/4078919037 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f48010430 con 0x7f4f58101000 2023-09-10T01:08:35.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f4e7fc700 1 -- 172.21.15.3:0/4078919037 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48018680 con 0x7f4f58101000 2023-09-10T01:08:35.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.516+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f581a3530 con 0x7f4f58101000 2023-09-10T01:08:35.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.520+0000 7f4f4e7fc700 1 -- 172.21.15.3:0/4078919037 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f4f480105a0 con 0x7f4f58101000 2023-09-10T01:08:35.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.521+0000 7f4f730ceb80 1 Processor -- start 2023-09-10T01:08:35.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.521+0000 7f4f730ceb80 1 -- start start 2023-09-10T01:08:35.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.521+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8981710 0x557ec8981af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8982030 0x557ec898fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8992640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec88da730 con 0x557ec8990250 2023-09-10T01:08:35.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec88d8050 con 0x557ec8982030 2023-09-10T01:08:35.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec88d3940 con 0x557ec8981710 2023-09-10T01:08:35.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5dd9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8981710 0x557ec8981af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f4e7fc700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4f440a07b0 0x7f4f440a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8982030 0x557ec898fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8982030 0x557ec898fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49374/0 (socket says 172.21.15.3:49374) 2023-09-10T01:08:35.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5dd9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8981710 0x557ec8981af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55466/0 (socket says 172.21.15.3:55466) 2023-09-10T01:08:35.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4133384307 learned_addr learned my addr 172.21.15.3:0/4133384307 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8992640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f4e7fc700 1 -- 172.21.15.3:0/4078919037 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4f48014070 con 0x7f4f58101000 2023-09-10T01:08:35.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4133384307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8981710 msgr2=0x557ec8981af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8981710 0x557ec8981af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4133384307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 msgr2=0x557ec8992640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8992640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4133384307 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ec86ffde0 con 0x557ec8982030 2023-09-10T01:08:35.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.522+0000 7f4f5dd9c700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4f440a07b0 0x7f4f440a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.523+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8982030 0x557ec898fd10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4f54005ea0 tx=0x7f4f54005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba1888f85943c28 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.523+0000 7f4f437fe700 1 -- 172.21.15.3:0/4133384307 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5401d070 con 0x557ec8982030 2023-09-10T01:08:35.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.523+0000 7f4f437fe700 1 -- 172.21.15.3:0/4133384307 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f5401e040 con 0x557ec8982030 2023-09-10T01:08:35.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.523+0000 7f4f437fe700 1 -- 172.21.15.3:0/4133384307 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f540048f0 con 0x557ec8982030 2023-09-10T01:08:35.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4133384307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8982030 msgr2=0x557ec898fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f5dd9c700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4f440a07b0 0x7f4f440a2c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4f500060b0 tx=0x7f4f50004ad0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8982030 0x557ec898fd10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4f54005ea0 tx=0x7f4f54005ed0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4133384307 shutdown_connections 2023-09-10T01:08:35.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8981710 0x557ec8981af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8992640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4133384307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8982030 0x557ec898fd10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4133384307 >> 172.21.15.3:0/4133384307 conn(0x557ec88f6090 msgr2=0x557ec897cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4133384307 shutdown_connections 2023-09-10T01:08:35.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.524+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4133384307 wait complete. 2023-09-10T01:08:35.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.525+0000 7f4f730ceb80 1 -- start start 2023-09-10T01:08:35.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.525+0000 7f4f730ceb80 1 Processor -- start 2023-09-10T01:08:35.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.525+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.525+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8982030 0x557ec898ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.525+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8990250 0x557ec898d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec8871b70 con 0x557ec8982030 2023-09-10T01:08:35.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec8871870 con 0x557ec8981710 2023-09-10T01:08:35.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec8755920 con 0x557ec8990250 2023-09-10T01:08:35.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f730ceb80 10 client.0 _ll_register_callbacks cb 0x557ec87bba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:35.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49376/0 (socket says 172.21.15.3:49376) 2023-09-10T01:08:35.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4250654552 learned_addr learned my addr 172.21.15.3:0/4250654552 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5dd9c700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8990250 0x557ec898d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8982030 0x557ec898ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4250654552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8990250 msgr2=0x557ec898d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8990250 0x557ec898d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4250654552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8982030 msgr2=0x557ec898ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8982030 0x557ec898ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5d59b700 1 -- 172.21.15.3:0/4250654552 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5810de10 con 0x557ec8981710 2023-09-10T01:08:35.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.526+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8982030 0x557ec898ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:35.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.527+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898fc00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400bb30 tx=0x7f4f5400fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5276fb60b908a83a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.528+0000 7f4f40ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:35.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.529+0000 7f4f40ff9700 1 -- 172.21.15.3:0/4250654552 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54031070 con 0x557ec8981710 2023-09-10T01:08:35.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.529+0000 7f4f40ff9700 1 -- 172.21.15.3:0/4250654552 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4f54005980 con 0x557ec8981710 2023-09-10T01:08:35.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.529+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ec882b4f0 con 0x557ec8981710 2023-09-10T01:08:35.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.530+0000 7f4f40ff9700 1 -- 172.21.15.3:0/4250654552 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f5401d040 con 0x557ec8981710 2023-09-10T01:08:35.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.530+0000 7f4f730ceb80 10 client.27221 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.530+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557ec86e8c60 con 0x557ec8981710 2023-09-10T01:08:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f40ff9700 1 -- 172.21.15.3:0/4250654552 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557ec882b4f0 con 0x557ec8981710 2023-09-10T01:08:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f40ff9700 1 -- 172.21.15.3:0/4250654552 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4f54034030 con 0x557ec8981710 2023-09-10T01:08:35.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f730ceb80 10 client.27221 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f730ceb80 10 client.27221 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:08:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f730ceb80 20 client.27221 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec899f770 0x557ec89a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f730ceb80 4 client.27221 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:35.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.531+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ec88e88a0 con 0x557ec899f770 2023-09-10T01:08:35.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.535+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec899f770 0x557ec89a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.536+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec899f770 0x557ec89a1b90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=407a9da2621cbf90 server_cookie=1be209a5ba471a3d in_seq=0 out_seq=0 2023-09-10T01:08:35.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.536+0000 7f4f40ff9700 10 client.27221 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:35.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.540+0000 7f4f40ff9700 1 -- 172.21.15.3:0/4250654552 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557ec88e88a0 con 0x557ec899f770 2023-09-10T01:08:35.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.540+0000 7f4f40ff9700 10 client.27221 handle_command_reply: tid=0 2023-09-10T01:08:35.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.540+0000 7f4f730ceb80 1 client.27221 shutdown 2023-09-10T01:08:35.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 msgr2=0x557ec898fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898fc00 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400bb30 tx=0x7f4f5400fc00 comp rx=0 tx=0).stop 2023-09-10T01:08:35.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 shutdown_connections 2023-09-10T01:08:35.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec899f770 0x557ec89a1b90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8990250 0x557ec898d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898fc00 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f40ff9700 0 client.27221 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:35.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/4250654552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8982030 0x557ec898ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 >> 172.21.15.3:0/4250654552 conn(0x557ec88f6090 msgr2=0x557ec897c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 shutdown_connections 2023-09-10T01:08:35.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 1 -- 172.21.15.3:0/4250654552 wait complete. 2023-09-10T01:08:35.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.541+0000 7f4f730ceb80 20 client.27221 trim_cache size 0 max 16384 2023-09-10T01:08:35.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:35 smithi003 ceph-mon[97654]: pgmap v164: 129 pgs: 129 active+clean; 123 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 378 op/s 2023-09-10T01:08:35.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:35 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3708148436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:08:35.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.568+0000 7f4f730ceb80 1 Processor -- start 2023-09-10T01:08:35.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.568+0000 7f4f730ceb80 1 -- start start 2023-09-10T01:08:35.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec89881b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec898da40 con 0x557ec8990250 2023-09-10T01:08:35.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec8997310 con 0x557ec8981710 2023-09-10T01:08:35.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec89a6890 con 0x557ec8982030 2023-09-10T01:08:35.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f5cd9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f5d59b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.569+0000 7f4f5d59b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49390/0 (socket says 172.21.15.3:49390) 2023-09-10T01:08:35.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5d59b700 1 -- 172.21.15.3:0/31154339 learned_addr learned my addr 172.21.15.3:0/31154339 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5dd9c700 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec89881b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5d59b700 1 -- 172.21.15.3:0/31154339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 msgr2=0x557ec8996dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5d59b700 1 -- 172.21.15.3:0/31154339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 msgr2=0x557ec89881b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec89881b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5d59b700 1 -- 172.21.15.3:0/31154339 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ec8726250 con 0x557ec8981710 2023-09-10T01:08:35.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:35.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.570+0000 7f4f5dd9c700 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec89881b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:35.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.571+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898f730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4f540098e0 tx=0x7f4f5400b280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a6e676ba2d78c94 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.573+0000 7f4f43fff700 1 -- 172.21.15.3:0/31154339 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54031030 con 0x557ec8981710 2023-09-10T01:08:35.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.573+0000 7f4f43fff700 1 -- 172.21.15.3:0/31154339 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f54005bc0 con 0x557ec8981710 2023-09-10T01:08:35.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.573+0000 7f4f43fff700 1 -- 172.21.15.3:0/31154339 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54005720 con 0x557ec8981710 2023-09-10T01:08:35.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.574+0000 7f4f730ceb80 1 -- 172.21.15.3:0/31154339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 msgr2=0x557ec898f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.574+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898f730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4f540098e0 tx=0x7f4f5400b280 comp rx=0 tx=0).stop 2023-09-10T01:08:35.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.574+0000 7f4f730ceb80 1 -- 172.21.15.3:0/31154339 shutdown_connections 2023-09-10T01:08:35.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.574+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec89881b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.574+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec898f730 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.574+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/31154339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.574+0000 7f4f730ceb80 1 -- 172.21.15.3:0/31154339 >> 172.21.15.3:0/31154339 conn(0x557ec88f6090 msgr2=0x557ec899bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.575+0000 7f4f730ceb80 1 -- 172.21.15.3:0/31154339 shutdown_connections 2023-09-10T01:08:35.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.575+0000 7f4f730ceb80 1 -- 172.21.15.3:0/31154339 wait complete. 2023-09-10T01:08:35.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.575+0000 7f4f730ceb80 1 -- start start 2023-09-10T01:08:35.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.575+0000 7f4f730ceb80 1 Processor -- start 2023-09-10T01:08:35.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.576+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec8996830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.576+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.576+0000 7f4f730ceb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8994fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.576+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec8987cf0 con 0x557ec8990250 2023-09-10T01:08:35.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.576+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec8995500 con 0x557ec8981710 2023-09-10T01:08:35.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.576+0000 7f4f730ceb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ec8995670 con 0x557ec8982030 2023-09-10T01:08:35.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.576+0000 7f4f730ceb80 10 client.0 _ll_register_callbacks cb 0x7f4f50009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:35.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.577+0000 7f4f5dd9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8994fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.577+0000 7f4f5dd9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8994fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43158/0 (socket says 172.21.15.3:43158) 2023-09-10T01:08:35.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.577+0000 7f4f5dd9c700 1 -- 172.21.15.3:0/1148901821 learned_addr learned my addr 172.21.15.3:0/1148901821 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.577+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec8996830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.577+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.579+0000 7f4f5d59b700 1 -- 172.21.15.3:0/1148901821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 msgr2=0x557ec8996d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.579+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.579+0000 7f4f5d59b700 1 -- 172.21.15.3:0/1148901821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 msgr2=0x557ec8994fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.579+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8994fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.579+0000 7f4f5d59b700 1 -- 172.21.15.3:0/1148901821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ec86e8c60 con 0x557ec8981710 2023-09-10T01:08:35.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.584+0000 7f4f5d59b700 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec8996830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400bb30 tx=0x7f4f54004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b5784d0504e492 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.584+0000 7f4f40ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:35.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.585+0000 7f4f40ff9700 1 -- 172.21.15.3:0/1148901821 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54044030 con 0x557ec8981710 2023-09-10T01:08:35.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.585+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ec88cc920 con 0x557ec8981710 2023-09-10T01:08:35.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.585+0000 7f4f40ff9700 1 -- 172.21.15.3:0/1148901821 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4f54045070 con 0x557ec8981710 2023-09-10T01:08:35.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.585+0000 7f4f40ff9700 1 -- 172.21.15.3:0/1148901821 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f54031040 con 0x557ec8981710 2023-09-10T01:08:35.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 10 client.27227 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:35.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557ec86fe2b0 con 0x557ec8981710 2023-09-10T01:08:35.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f40ff9700 1 -- 172.21.15.3:0/1148901821 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557ec88cc920 con 0x557ec8981710 2023-09-10T01:08:35.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f40ff9700 1 -- 172.21.15.3:0/1148901821 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4f5403e020 con 0x557ec8981710 2023-09-10T01:08:35.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 10 client.27227 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:35.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 10 client.27227 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:08:35.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 20 client.27227 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:35.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec89f86d0 0x557ec89faaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 4 client.27227 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:08:35.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.586+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557ec888d0b0 con 0x557ec89f86d0 2023-09-10T01:08:35.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.587+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec89f86d0 0x557ec89faaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.590+0000 7f4f40ff9700 10 client.27227 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:35.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.590+0000 7f4f5cd9a700 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec89f86d0 0x557ec89faaf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ace4c9d70d76d3c server_cookie=ec2d1628d763273f in_seq=0 out_seq=0 2023-09-10T01:08:35.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f40ff9700 1 -- 172.21.15.3:0/1148901821 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557ec888d0b0 con 0x557ec89f86d0 2023-09-10T01:08:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f40ff9700 10 client.27227 handle_command_reply: tid=0 2023-09-10T01:08:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 client.27227 shutdown 2023-09-10T01:08:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 msgr2=0x557ec8996830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec8996830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400bb30 tx=0x7f4f54004760 comp rx=0 tx=0).stop 2023-09-10T01:08:35.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 shutdown_connections 2023-09-10T01:08:35.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557ec8990250 0x557ec8994fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ec8981710 0x557ec8996830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x557ec89f86d0 0x557ec89faaf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 --2- 172.21.15.3:0/1148901821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557ec8982030 0x557ec8996d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 >> 172.21.15.3:0/1148901821 conn(0x557ec88f6090 msgr2=0x557ec89881b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f40ff9700 0 client.27227 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:08:35.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.591+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 shutdown_connections 2023-09-10T01:08:35.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.592+0000 7f4f730ceb80 1 -- 172.21.15.3:0/1148901821 wait complete. 2023-09-10T01:08:35.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.592+0000 7f4f730ceb80 20 client.27227 trim_cache size 0 max 16384 2023-09-10T01:08:35.596 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:08:35.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.592+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4f440a07b0 msgr2=0x7f4f440a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.592+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4f440a07b0 0x7f4f440a2c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4f500060b0 tx=0x7f4f50004ad0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.592+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 msgr2=0x7f4f58196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.592+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 0x7f4f58196c10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800e9a0 tx=0x7f4f4800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 shutdown_connections 2023-09-10T01:08:35.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4f440a07b0 0x7f4f440a2c70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f58106650 0x7f4f58199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f58100640 0x7f4f581966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 --2- 172.21.15.3:0/4078919037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f58101000 0x7f4f58196c10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 >> 172.21.15.3:0/4078919037 conn(0x7f4f580fc170 msgr2=0x7f4f58104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 shutdown_connections 2023-09-10T01:08:35.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.593+0000 7f4f5f7ff700 1 -- 172.21.15.3:0/4078919037 wait complete. 2023-09-10T01:08:35.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:08:35.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:35 smithi136 ceph-mon[133708]: pgmap v164: 129 pgs: 129 active+clean; 123 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 378 op/s 2023-09-10T01:08:35.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:35 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3708148436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:08:35.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:35 smithi092 ceph-mon[134067]: pgmap v164: 129 pgs: 129 active+clean; 123 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 378 op/s 2023-09-10T01:08:35.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:35 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3708148436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:08:35.757 INFO:tasks.workunit.client.0.smithi003.stdout: 1 458345 9.35 MB/sec execute 137 sec latency 170.689 ms 2023-09-10T01:08:35.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 -- 172.21.15.3:0/4144654345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 msgr2=0x7f9858105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 --2- 172.21.15.3:0/4144654345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 0x7f9858105a10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9854009920 tx=0x7f9854009c30 comp rx=0 tx=0).stop 2023-09-10T01:08:35.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 -- 172.21.15.3:0/4144654345 shutdown_connections 2023-09-10T01:08:35.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 --2- 172.21.15.3:0/4144654345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98580fff30 0x7f985810c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 --2- 172.21.15.3:0/4144654345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 0x7f9858105a10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 --2- 172.21.15.3:0/4144654345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 0x7f98580ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 -- 172.21.15.3:0/4144654345 >> 172.21.15.3:0/4144654345 conn(0x7f9858076710 msgr2=0x7f9858076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 -- 172.21.15.3:0/4144654345 shutdown_connections 2023-09-10T01:08:35.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.889+0000 7f985fbff700 1 -- 172.21.15.3:0/4144654345 wait complete. 2023-09-10T01:08:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.890+0000 7f985fbff700 1 Processor -- start 2023-09-10T01:08:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.891+0000 7f985fbff700 1 -- start start 2023-09-10T01:08:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.891+0000 7f985fbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 0x7f985819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.891+0000 7f985fbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98580fff30 0x7f985819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.891+0000 7f985fbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 0x7f9858198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.891+0000 7f985fbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9858199020 con 0x7f98580fff30 2023-09-10T01:08:35.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.891+0000 7f985fbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9858199190 con 0x7f9858105630 2023-09-10T01:08:35.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.891+0000 7f985fbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9858199350 con 0x7f98580ff570 2023-09-10T01:08:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 0x7f985819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d19a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98580fff30 0x7f985819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 0x7f985819e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55528/0 (socket says 172.21.15.3:55528) 2023-09-10T01:08:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 -- 172.21.15.3:0/3592557751 learned_addr learned my addr 172.21.15.3:0/3592557751 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985e19c700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 0x7f9858198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 -- 172.21.15.3:0/3592557751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 msgr2=0x7f9858198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 0x7f9858198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 -- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98580fff30 msgr2=0x7f985819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98580fff30 0x7f985819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 -- 172.21.15.3:0/3592557751 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98480095f0 con 0x7f98580ff570 2023-09-10T01:08:35.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.892+0000 7f985d99b700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 0x7f985819e870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9854009840 tx=0x7f985400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.893+0000 7f984effd700 1 -- 172.21.15.3:0/3592557751 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985401c070 con 0x7f98580ff570 2023-09-10T01:08:35.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.893+0000 7f984effd700 1 -- 172.21.15.3:0/3592557751 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f985400bba0 con 0x7f98580ff570 2023-09-10T01:08:35.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.893+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9850009c40 con 0x7f98580ff570 2023-09-10T01:08:35.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.893+0000 7f984effd700 1 -- 172.21.15.3:0/3592557751 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98540038e0 con 0x7f98580ff570 2023-09-10T01:08:35.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.893+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9854009520 con 0x7f98580ff570 2023-09-10T01:08:35.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.898+0000 7f984effd700 1 -- 172.21.15.3:0/3592557751 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9854020030 con 0x7f98580ff570 2023-09-10T01:08:35.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 Processor -- start 2023-09-10T01:08:35.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 -- start start 2023-09-10T01:08:35.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac249710 0x558dac249af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac24a030 0x558dac253900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac256230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac1a2730 con 0x558dac249710 2023-09-10T01:08:35.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac1a0050 con 0x558dac24a030 2023-09-10T01:08:35.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f9873449b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac19b940 con 0x558dac253e40 2023-09-10T01:08:35.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f985d99b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac24a030 0x558dac253900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f985e19c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac249710 0x558dac249af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f985d99b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac24a030 0x558dac253900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49432/0 (socket says 172.21.15.3:49432) 2023-09-10T01:08:35.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f985d19a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac256230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.899+0000 7f985d99b700 1 -- 172.21.15.3:0/1564565174 learned_addr learned my addr 172.21.15.3:0/1564565174 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f984effd700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98440a0830 0x7f98440a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f985d99b700 1 -- 172.21.15.3:0/1564565174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 msgr2=0x558dac256230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f985d99b700 1 --2- 172.21.15.3:0/1564565174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac256230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f985d99b700 1 -- 172.21.15.3:0/1564565174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac249710 msgr2=0x558dac249af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f985d99b700 1 --2- 172.21.15.3:0/1564565174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac249710 0x558dac249af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f985d99b700 1 -- 172.21.15.3:0/1564565174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dabfc7de0 con 0x558dac24a030 2023-09-10T01:08:35.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f984effd700 1 -- 172.21.15.3:0/3592557751 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f98540cd290 con 0x7f98580ff570 2023-09-10T01:08:35.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f985e19c700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98440a0830 0x7f98440a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.900+0000 7f985d99b700 1 --2- 172.21.15.3:0/1564565174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac24a030 0x558dac253900 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9854096eb0 tx=0x7f985402d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a77b5e0d0acc8fbe server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.901+0000 7f9843fff700 1 -- 172.21.15.3:0/1564565174 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854085110 con 0x558dac24a030 2023-09-10T01:08:35.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.901+0000 7f9843fff700 1 -- 172.21.15.3:0/1564565174 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9854099f80 con 0x558dac24a030 2023-09-10T01:08:35.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.901+0000 7f9843fff700 1 -- 172.21.15.3:0/1564565174 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98540146e0 con 0x558dac24a030 2023-09-10T01:08:35.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.901+0000 7f9873449b80 1 -- 172.21.15.3:0/1564565174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac24a030 msgr2=0x558dac253900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.901+0000 7f985e19c700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98440a0830 0x7f98440a2cf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f985819f500 tx=0x7f9850004ad0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.901+0000 7f9873449b80 1 --2- 172.21.15.3:0/1564565174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac24a030 0x558dac253900 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9854096eb0 tx=0x7f985402d000 comp rx=0 tx=0).stop 2023-09-10T01:08:35.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 -- 172.21.15.3:0/1564565174 shutdown_connections 2023-09-10T01:08:35.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 --2- 172.21.15.3:0/1564565174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac256230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 --2- 172.21.15.3:0/1564565174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac249710 0x558dac249af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 --2- 172.21.15.3:0/1564565174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac24a030 0x558dac253900 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 -- 172.21.15.3:0/1564565174 >> 172.21.15.3:0/1564565174 conn(0x558dac1be090 msgr2=0x558dac244bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 -- 172.21.15.3:0/1564565174 shutdown_connections 2023-09-10T01:08:35.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 -- 172.21.15.3:0/1564565174 wait complete. 2023-09-10T01:08:35.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 -- start start 2023-09-10T01:08:35.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.902+0000 7f9873449b80 1 Processor -- start 2023-09-10T01:08:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f9873449b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac254220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f9873449b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac24a030 0x558dac254760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f9873449b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac253e40 0x558dac254ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f9873449b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac139b70 con 0x558dac253e40 2023-09-10T01:08:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f9873449b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac139870 con 0x558dac249710 2023-09-10T01:08:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f9873449b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac01d920 con 0x558dac24a030 2023-09-10T01:08:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f9873449b80 10 client.0 _ll_register_callbacks cb 0x558dac083a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f985d19a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac24a030 0x558dac254760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f985d19a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac24a030 0x558dac254760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55546/0 (socket says 172.21.15.3:55546) 2023-09-10T01:08:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f985d19a700 1 -- 172.21.15.3:0/3376980401 learned_addr learned my addr 172.21.15.3:0/3376980401 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.903+0000 7f985d99b700 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac254220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f985e19c700 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac253e40 0x558dac254ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f985d19a700 1 -- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 msgr2=0x558dac254220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f985d19a700 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac254220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f985d19a700 1 -- 172.21.15.3:0/3376980401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac253e40 msgr2=0x558dac254ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f985d19a700 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac253e40 0x558dac254ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f985d19a700 1 -- 172.21.15.3:0/3376980401 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9854009520 con 0x558dac24a030 2023-09-10T01:08:35.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f985d19a700 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac24a030 0x558dac254760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9848005950 tx=0x7f9848012af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b54d0188d027755 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f98417fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:35.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f98417fa700 1 -- 172.21.15.3:0/3376980401 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848017070 con 0x558dac24a030 2023-09-10T01:08:35.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558dac0f34f0 con 0x558dac24a030 2023-09-10T01:08:35.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.904+0000 7f98417fa700 1 -- 172.21.15.3:0/3376980401 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f9848004030 con 0x558dac24a030 2023-09-10T01:08:35.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f98417fa700 1 -- 172.21.15.3:0/3376980401 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9848004230 con 0x558dac24a030 2023-09-10T01:08:35.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 10 client.27154 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:35.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558dabfb0c60 con 0x558dac24a030 2023-09-10T01:08:35.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f98417fa700 1 -- 172.21.15.3:0/3376980401 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558dac0f34f0 con 0x558dac24a030 2023-09-10T01:08:35.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f98417fa700 1 -- 172.21.15.3:0/3376980401 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f984800f5d0 con 0x558dac24a030 2023-09-10T01:08:35.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 10 client.27154 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:35.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 10 client.27154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:35.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 20 client.27154 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:35.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac26bdd0 0x558dac26e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 4 client.27154 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.905+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558dac1b08a0 con 0x558dac26bdd0 2023-09-10T01:08:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.906+0000 7f985d99b700 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac26bdd0 0x558dac26e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.906+0000 7f985d99b700 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac26bdd0 0x558dac26e1f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b8debe316be055e server_cookie=195308e54f8e5e78 in_seq=0 out_seq=0 2023-09-10T01:08:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.906+0000 7f98417fa700 10 client.27154 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f98417fa700 1 -- 172.21.15.3:0/3376980401 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558dac1b08a0 con 0x558dac26bdd0 2023-09-10T01:08:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f98417fa700 10 client.27154 handle_command_reply: tid=0 2023-09-10T01:08:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 client.27154 shutdown 2023-09-10T01:08:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac24a030 msgr2=0x558dac254760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac24a030 0x558dac254760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9848005950 tx=0x7f9848012af0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 shutdown_connections 2023-09-10T01:08:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac26bdd0 0x558dac26e1f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac253e40 0x558dac254ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac254220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f98417fa700 0 client.27154 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 --2- 172.21.15.3:0/3376980401 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac24a030 0x558dac254760 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 >> 172.21.15.3:0/3376980401 conn(0x558dac1be090 msgr2=0x558dac244780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 shutdown_connections 2023-09-10T01:08:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 1 -- 172.21.15.3:0/3376980401 wait complete. 2023-09-10T01:08:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.907+0000 7f9873449b80 20 client.27154 trim_cache size 0 max 16384 2023-09-10T01:08:35.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 Processor -- start 2023-09-10T01:08:35.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 -- start start 2023-09-10T01:08:35.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac249710 0x558dac2526b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac250160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac253e40 0x558dac2506a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac245fe0 con 0x558dac24a030 2023-09-10T01:08:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac24a4a0 con 0x558dac253e40 2023-09-10T01:08:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f9873449b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac245730 con 0x558dac249710 2023-09-10T01:08:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f985d99b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac249710 0x558dac2526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f985d19a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac250160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f985e19c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac253e40 0x558dac2506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f985d99b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac249710 0x558dac2526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55556/0 (socket says 172.21.15.3:55556) 2023-09-10T01:08:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.924+0000 7f985d99b700 1 -- 172.21.15.3:0/1309616445 learned_addr learned my addr 172.21.15.3:0/1309616445 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f985d99b700 1 -- 172.21.15.3:0/1309616445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac253e40 msgr2=0x558dac2506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f985d99b700 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac253e40 0x558dac2506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f985d99b700 1 -- 172.21.15.3:0/1309616445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 msgr2=0x558dac250160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f985d99b700 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac250160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f985d99b700 1 -- 172.21.15.3:0/1309616445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dabfee250 con 0x558dac249710 2023-09-10T01:08:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f985e19c700 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac253e40 0x558dac2506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:35.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f985d99b700 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac249710 0x558dac2526b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f985409a3d0 tx=0x7f9854009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3c6299cde99fcc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f984cbff700 1 -- 172.21.15.3:0/1309616445 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98540144f0 con 0x558dac249710 2023-09-10T01:08:35.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f984cbff700 1 -- 172.21.15.3:0/1309616445 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9854099da0 con 0x558dac249710 2023-09-10T01:08:35.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.925+0000 7f984cbff700 1 -- 172.21.15.3:0/1309616445 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98540cd0b0 con 0x558dac249710 2023-09-10T01:08:35.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 -- 172.21.15.3:0/1309616445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac249710 msgr2=0x558dac2526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac249710 0x558dac2526b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f985409a3d0 tx=0x7f9854009c30 comp rx=0 tx=0).stop 2023-09-10T01:08:35.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 -- 172.21.15.3:0/1309616445 shutdown_connections 2023-09-10T01:08:35.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac249710 0x558dac2526b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac253e40 0x558dac2506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 --2- 172.21.15.3:0/1309616445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac250160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 -- 172.21.15.3:0/1309616445 >> 172.21.15.3:0/1309616445 conn(0x558dac1be090 msgr2=0x558dac24e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 -- 172.21.15.3:0/1309616445 shutdown_connections 2023-09-10T01:08:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.926+0000 7f9873449b80 1 -- 172.21.15.3:0/1309616445 wait complete. 2023-09-10T01:08:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.927+0000 7f9873449b80 1 -- start start 2023-09-10T01:08:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.927+0000 7f9873449b80 1 Processor -- start 2023-09-10T01:08:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f9873449b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac258150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f9873449b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac256480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f9873449b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac2569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f9873449b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac256f90 con 0x558dac24a030 2023-09-10T01:08:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f9873449b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac2570d0 con 0x558dac249710 2023-09-10T01:08:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f9873449b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dac257290 con 0x558dac253e40 2023-09-10T01:08:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f9873449b80 10 client.0 _ll_register_callbacks cb 0x7f9848009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac258150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac258150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49458/0 (socket says 172.21.15.3:49458) 2023-09-10T01:08:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 -- 172.21.15.3:0/1212212587 learned_addr learned my addr 172.21.15.3:0/1212212587 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985e19c700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac2569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d19a700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac256480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 -- 172.21.15.3:0/1212212587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 msgr2=0x558dac2569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac2569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 -- 172.21.15.3:0/1212212587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 msgr2=0x558dac256480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac256480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 -- 172.21.15.3:0/1212212587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9850017440 con 0x558dac249710 2023-09-10T01:08:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d19a700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac256480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:08:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985e19c700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac2569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f985d99b700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac258150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f98540094b0 tx=0x7f985400bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c19a0229c5b4621 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f98417fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.928+0000 7f98417fa700 1 -- 172.21.15.3:0/1212212587 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985409e040 con 0x558dac249710 2023-09-10T01:08:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f98417fa700 1 -- 172.21.15.3:0/1212212587 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f985409ab40 con 0x558dac249710 2023-09-10T01:08:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9854099f80 con 0x558dac249710 2023-09-10T01:08:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f98417fa700 1 -- 172.21.15.3:0/1212212587 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98540cd0b0 con 0x558dac249710 2023-09-10T01:08:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 10 client.27245 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558dabfc7de0 con 0x558dac249710 2023-09-10T01:08:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f98417fa700 1 -- 172.21.15.3:0/1212212587 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9854099f80 con 0x558dac249710 2023-09-10T01:08:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f98417fa700 1 -- 172.21.15.3:0/1212212587 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f98540ca8d0 con 0x558dac249710 2023-09-10T01:08:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 10 client.27245 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 10 client.27245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 20 client.27245 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac2c0c80 0x558dac2c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 4 client.27245 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:08:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.929+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558dac194920 con 0x558dac2c0c80 2023-09-10T01:08:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f985d19a700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac2c0c80 0x558dac2c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f985d19a700 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac2c0c80 0x558dac2c3070 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31cbb649a32dd65 server_cookie=beae13f6328bb989 in_seq=0 out_seq=0 2023-09-10T01:08:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f98417fa700 10 client.27245 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f98417fa700 1 -- 172.21.15.3:0/1212212587 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558dac194920 con 0x558dac2c0c80 2023-09-10T01:08:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f98417fa700 10 client.27245 handle_command_reply: tid=0 2023-09-10T01:08:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f9873449b80 1 client.27245 shutdown 2023-09-10T01:08:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 msgr2=0x558dac258150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.930+0000 7f9873449b80 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac258150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f98540094b0 tx=0x7f985400bd10 comp rx=0 tx=0).stop 2023-09-10T01:08:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 shutdown_connections 2023-09-10T01:08:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558dac2c0c80 0x558dac2c3070 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558dac253e40 0x558dac2569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558dac249710 0x558dac258150 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f98417fa700 0 client.27245 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 --2- 172.21.15.3:0/1212212587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558dac24a030 0x558dac256480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 >> 172.21.15.3:0/1212212587 conn(0x558dac1be090 msgr2=0x558dac248700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 shutdown_connections 2023-09-10T01:08:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 1 -- 172.21.15.3:0/1212212587 wait complete. 2023-09-10T01:08:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f9873449b80 20 client.27245 trim_cache size 0 max 16384 2023-09-10T01:08:35.953 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:08:35.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98440a0830 msgr2=0x7f98440a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f985fbff700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98440a0830 0x7f98440a2cf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f985819f500 tx=0x7f9850004ad0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 msgr2=0x7f985819e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.931+0000 7f985fbff700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 0x7f985819e870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9854009840 tx=0x7f985400fa00 comp rx=0 tx=0).stop 2023-09-10T01:08:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 shutdown_connections 2023-09-10T01:08:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98440a0830 0x7f98440a2cf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98580ff570 0x7f985819e870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9858105630 0x7f9858198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 --2- 172.21.15.3:0/3592557751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98580fff30 0x7f985819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 >> 172.21.15.3:0/3592557751 conn(0x7f9858076710 msgr2=0x7f98580fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 shutdown_connections 2023-09-10T01:08:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:35.932+0000 7f985fbff700 1 -- 172.21.15.3:0/3592557751 wait complete. 2023-09-10T01:08:36.757 INFO:tasks.workunit.client.0.smithi003.stdout: 1 459960 9.48 MB/sec execute 138 sec latency 141.184 ms 2023-09-10T01:08:36.958 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:08:36.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:08:37.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 -- 172.21.15.3:0/4247748609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4101010 msgr2=0x7f68d410d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 --2- 172.21.15.3:0/4247748609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4101010 0x7f68d410d520 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00b0a0 tx=0x7f68cc00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 -- 172.21.15.3:0/4247748609 shutdown_connections 2023-09-10T01:08:37.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 --2- 172.21.15.3:0/4247748609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4101010 0x7f68d410d520 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 --2- 172.21.15.3:0/4247748609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d4106660 0x7f68d4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 --2- 172.21.15.3:0/4247748609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4100650 0x7f68d4100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 -- 172.21.15.3:0/4247748609 >> 172.21.15.3:0/4247748609 conn(0x7f68d40fc160 msgr2=0x7f68d40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 -- 172.21.15.3:0/4247748609 shutdown_connections 2023-09-10T01:08:37.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.185+0000 7f68d8fff700 1 -- 172.21.15.3:0/4247748609 wait complete. 2023-09-10T01:08:37.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 Processor -- start 2023-09-10T01:08:37.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 -- start start 2023-09-10T01:08:37.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4100650 0x7f68d419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d4101010 0x7f68d419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4106660 0x7f68d4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4198e00 con 0x7f68d4101010 2023-09-10T01:08:37.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4198f70 con 0x7f68d4106660 2023-09-10T01:08:37.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.187+0000 7f68d8fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4199130 con 0x7f68d4100650 2023-09-10T01:08:37.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4106660 0x7f68d4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d1d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d4101010 0x7f68d419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4106660 0x7f68d4198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49472/0 (socket says 172.21.15.3:49472) 2023-09-10T01:08:37.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 -- 172.21.15.3:0/2137370452 learned_addr learned my addr 172.21.15.3:0/2137370452 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d259c700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4100650 0x7f68d419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 -- 172.21.15.3:0/2137370452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4100650 msgr2=0x7f68d419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4100650 0x7f68d419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 -- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d4101010 msgr2=0x7f68d419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d4101010 0x7f68d419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 -- 172.21.15.3:0/2137370452 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c4009520 con 0x7f68d4106660 2023-09-10T01:08:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.188+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4106660 0x7f68d4198750 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009850 tx=0x7f68cc0077f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.189+0000 7f68c37fe700 1 -- 172.21.15.3:0/2137370452 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00e040 con 0x7f68d4106660 2023-09-10T01:08:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.189+0000 7f68c37fe700 1 -- 172.21.15.3:0/2137370452 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68cc0079e0 con 0x7f68d4106660 2023-09-10T01:08:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.189+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc009d00 con 0x7f68d4106660 2023-09-10T01:08:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.189+0000 7f68c37fe700 1 -- 172.21.15.3:0/2137370452 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0052a0 con 0x7f68d4106660 2023-09-10T01:08:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.189+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d41a5510 con 0x7f68d4106660 2023-09-10T01:08:37.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.192+0000 7f68c37fe700 1 -- 172.21.15.3:0/2137370452 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f68cc01d470 con 0x7f68d4106660 2023-09-10T01:08:37.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.194+0000 7f68c37fe700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a0830 0x7f68bc0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.194+0000 7f68c37fe700 1 -- 172.21.15.3:0/2137370452 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f68cc0cd600 con 0x7f68d4106660 2023-09-10T01:08:37.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.194+0000 7f68ec921b80 1 Processor -- start 2023-09-10T01:08:37.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.194+0000 7f68d259c700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a0830 0x7f68bc0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.194+0000 7f68ec921b80 1 -- start start 2023-09-10T01:08:37.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.194+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b0790 0x5629073b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.194+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073ba980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073baec0 0x5629073bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68ec921b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073097b0 con 0x5629073baec0 2023-09-10T01:08:37.198 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68ec921b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073070d0 con 0x5629073b0790 2023-09-10T01:08:37.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68ec921b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073029c0 con 0x5629073b10b0 2023-09-10T01:08:37.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a0830 0x7f68bc0a2cf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009d70 tx=0x7f68c4009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b0790 0x5629073b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b0790 0x5629073b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49474/0 (socket says 172.21.15.3:49474) 2023-09-10T01:08:37.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 -- 172.21.15.3:0/265925422 learned_addr learned my addr 172.21.15.3:0/265925422 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d1d9b700 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073ba980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073baec0 0x5629073bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 -- 172.21.15.3:0/265925422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 msgr2=0x5629073ba980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073ba980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 -- 172.21.15.3:0/265925422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073baec0 msgr2=0x5629073bd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073baec0 0x5629073bd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 -- 172.21.15.3:0/265925422 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56290712ee60 con 0x5629073b0790 2023-09-10T01:08:37.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d1d9b700 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073ba980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:37.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073baec0 0x5629073bd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:37.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.195+0000 7f68d259c700 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b0790 0x5629073b0b70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f68c400e5b0 tx=0x7f68c4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffbeea261f053b19 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.196+0000 7f68c0ff9700 1 -- 172.21.15.3:0/265925422 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c400ee00 con 0x5629073b0790 2023-09-10T01:08:37.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.196+0000 7f68c0ff9700 1 -- 172.21.15.3:0/265925422 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68c401e8e0 con 0x5629073b0790 2023-09-10T01:08:37.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.196+0000 7f68c0ff9700 1 -- 172.21.15.3:0/265925422 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c401d820 con 0x5629073b0790 2023-09-10T01:08:37.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.196+0000 7f68ec921b80 1 -- 172.21.15.3:0/265925422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b0790 msgr2=0x5629073b0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.196+0000 7f68ec921b80 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b0790 0x5629073b0b70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f68c400e5b0 tx=0x7f68c4014000 comp rx=0 tx=0).stop 2023-09-10T01:08:37.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 -- 172.21.15.3:0/265925422 shutdown_connections 2023-09-10T01:08:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073baec0 0x5629073bd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b0790 0x5629073b0b70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 --2- 172.21.15.3:0/265925422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073ba980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 -- 172.21.15.3:0/265925422 >> 172.21.15.3:0/265925422 conn(0x562907325110 msgr2=0x5629073abc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 -- 172.21.15.3:0/265925422 shutdown_connections 2023-09-10T01:08:37.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 -- 172.21.15.3:0/265925422 wait complete. 2023-09-10T01:08:37.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 -- start start 2023-09-10T01:08:37.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.197+0000 7f68ec921b80 1 Processor -- start 2023-09-10T01:08:37.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073ba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b10b0 0x5629073b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073baec0 0x5629073b8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68ec921b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629072a0bf0 con 0x5629073b0790 2023-09-10T01:08:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68ec921b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629072a08f0 con 0x5629073b10b0 2023-09-10T01:08:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68ec921b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629071849a0 con 0x5629073baec0 2023-09-10T01:08:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68ec921b80 10 client.0 _ll_register_callbacks cb 0x5629071eab00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68d1d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b10b0 0x5629073b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68d2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073baec0 0x5629073b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073ba920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43234/0 (socket says 172.21.15.3:43234) 2023-09-10T01:08:37.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68d259c700 1 -- 172.21.15.3:0/1366934006 learned_addr learned my addr 172.21.15.3:0/1366934006 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.198+0000 7f68d1d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b10b0 0x5629073b7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49476/0 (socket says 172.21.15.3:49476) 2023-09-10T01:08:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68d1d9b700 1 -- 172.21.15.3:0/1366934006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073baec0 msgr2=0x5629073b8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68d1d9b700 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073baec0 0x5629073b8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68d1d9b700 1 -- 172.21.15.3:0/1366934006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 msgr2=0x5629073ba920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68d1d9b700 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073ba920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68d1d9b700 1 -- 172.21.15.3:0/1366934006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c4017070 con 0x5629073b10b0 2023-09-10T01:08:37.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68d1d9b700 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b10b0 0x5629073b7b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f68d4199f40 tx=0x7f68c8012990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b51eac070ce815ff server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:37.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1366934006 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8017070 con 0x5629073b10b0 2023-09-10T01:08:37.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56290725a570 con 0x5629073b10b0 2023-09-10T01:08:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.199+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1366934006 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f68c8004030 con 0x5629073b10b0 2023-09-10T01:08:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1366934006 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68c80041f0 con 0x5629073b10b0 2023-09-10T01:08:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 10 client.27257 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562907117ce0 con 0x5629073b10b0 2023-09-10T01:08:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1366934006 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56290725a570 con 0x5629073b10b0 2023-09-10T01:08:37.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1366934006 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f68c80104c0 con 0x5629073b10b0 2023-09-10T01:08:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 10 client.27257 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 10 client.27257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 20 client.27257 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629073caa60 0x5629073cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 4 client.27257 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.200+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562907317920 con 0x5629073caa60 2023-09-10T01:08:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.201+0000 7f68d259c700 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629073caa60 0x5629073cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.201+0000 7f68d259c700 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629073caa60 0x5629073cce80 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a597dfa8955c9741 server_cookie=df20eaaa579c8a26 in_seq=0 out_seq=0 2023-09-10T01:08:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.201+0000 7f68b9ffb700 10 client.27257 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.203+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1366934006 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562907317920 con 0x5629073caa60 2023-09-10T01:08:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.203+0000 7f68b9ffb700 10 client.27257 handle_command_reply: tid=0 2023-09-10T01:08:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.203+0000 7f68ec921b80 1 client.27257 shutdown 2023-09-10T01:08:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b10b0 msgr2=0x5629073b7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b10b0 0x5629073b7b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f68d4199f40 tx=0x7f68c8012990 comp rx=0 tx=0).stop 2023-09-10T01:08:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 shutdown_connections 2023-09-10T01:08:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629073caa60 0x5629073cce80 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073baec0 0x5629073b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073ba920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1366934006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073b10b0 0x5629073b7b30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68b9ffb700 0 client.27257 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 >> 172.21.15.3:0/1366934006 conn(0x562907325110 msgr2=0x5629073ab860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 shutdown_connections 2023-09-10T01:08:37.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 1 -- 172.21.15.3:0/1366934006 wait complete. 2023-09-10T01:08:37.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.204+0000 7f68ec921b80 20 client.27257 trim_cache size 0 max 16384 2023-09-10T01:08:37.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.224+0000 7f68ec921b80 1 Processor -- start 2023-09-10T01:08:37.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68ec921b80 1 -- start start 2023-09-10T01:08:37.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073ba450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68ec921b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073b8870 con 0x5629073b0790 2023-09-10T01:08:37.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68ec921b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073b2310 con 0x5629073baec0 2023-09-10T01:08:37.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68ec921b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073b1810 con 0x5629073b10b0 2023-09-10T01:08:37.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68d2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073ba450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68d2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073ba450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49490/0 (socket says 172.21.15.3:49490) 2023-09-10T01:08:37.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68d2d9d700 1 -- 172.21.15.3:0/3675029438 learned_addr learned my addr 172.21.15.3:0/3675029438 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68d259c700 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.225+0000 7f68d1d9b700 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68d2d9d700 1 -- 172.21.15.3:0/3675029438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 msgr2=0x5629073afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68d2d9d700 1 -- 172.21.15.3:0/3675029438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 msgr2=0x5629073c1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073c1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68d2d9d700 1 -- 172.21.15.3:0/3675029438 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629071552d0 con 0x5629073baec0 2023-09-10T01:08:37.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073ba450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009fd0 tx=0x7f68cc085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb722ecf9d126a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68c17fa700 1 -- 172.21.15.3:0/3675029438 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc004e60 con 0x5629073baec0 2023-09-10T01:08:37.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68c17fa700 1 -- 172.21.15.3:0/3675029438 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68cc09aeb0 con 0x5629073baec0 2023-09-10T01:08:37.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.226+0000 7f68c17fa700 1 -- 172.21.15.3:0/3675029438 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc098be0 con 0x5629073baec0 2023-09-10T01:08:37.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 -- 172.21.15.3:0/3675029438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 msgr2=0x5629073ba450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073ba450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009fd0 tx=0x7f68cc085040 comp rx=0 tx=0).stop 2023-09-10T01:08:37.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 -- 172.21.15.3:0/3675029438 shutdown_connections 2023-09-10T01:08:37.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073ba450 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x5629073c1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 --2- 172.21.15.3:0/3675029438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629073afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 -- 172.21.15.3:0/3675029438 >> 172.21.15.3:0/3675029438 conn(0x562907325110 msgr2=0x5629073af780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 -- 172.21.15.3:0/3675029438 shutdown_connections 2023-09-10T01:08:37.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 -- 172.21.15.3:0/3675029438 wait complete. 2023-09-10T01:08:37.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.227+0000 7f68ec921b80 1 -- start start 2023-09-10T01:08:37.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 1 Processor -- start 2023-09-10T01:08:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x562907421b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629074220b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073c14b0 con 0x5629073b0790 2023-09-10T01:08:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073c1620 con 0x5629073baec0 2023-09-10T01:08:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629073c17e0 con 0x5629073b10b0 2023-09-10T01:08:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68ec921b80 10 client.0 _ll_register_callbacks cb 0x7f68c8009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68d259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x562907421b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.228+0000 7f68d2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d1d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629074220b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49506/0 (socket says 172.21.15.3:49506) 2023-09-10T01:08:37.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 -- 172.21.15.3:0/1059472051 learned_addr learned my addr 172.21.15.3:0/1059472051 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 -- 172.21.15.3:0/1059472051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 msgr2=0x5629074220b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629074220b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 -- 172.21.15.3:0/1059472051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 msgr2=0x562907421b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x562907421b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 -- 172.21.15.3:0/1059472051 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d41a5510 con 0x5629073baec0 2023-09-10T01:08:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d1d9b700 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629074220b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:37.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d259c700 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x562907421b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68d2d9d700 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073c0eb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f68cc097220 tx=0x7f68cc09a1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3986edc9acafd8b3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1059472051 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a2790 con 0x5629073baec0 2023-09-10T01:08:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68c401e8e0 con 0x5629073baec0 2023-09-10T01:08:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.229+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1059472051 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f68cc09e5e0 con 0x5629073baec0 2023-09-10T01:08:37.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1059472051 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68cc09aa10 con 0x5629073baec0 2023-09-10T01:08:37.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 10 client.27263 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:37.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56290712d330 con 0x5629073baec0 2023-09-10T01:08:37.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1059472051 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68c401e8e0 con 0x5629073baec0 2023-09-10T01:08:37.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1059472051 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f68cc0b2d20 con 0x5629073baec0 2023-09-10T01:08:37.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 10 client.27263 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:37.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 10 client.27263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:37.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 20 client.27263 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:37.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629074295d0 0x56290742b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 4 client.27263 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:08:37.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5629072fb9a0 con 0x5629074295d0 2023-09-10T01:08:37.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.230+0000 7f68d259c700 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629074295d0 0x56290742b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.231+0000 7f68d259c700 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629074295d0 0x56290742b9c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad0a5b5fd1c34870 server_cookie=3ca23725091ca130 in_seq=0 out_seq=0 2023-09-10T01:08:37.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.231+0000 7f68b9ffb700 10 client.27263 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68b9ffb700 1 -- 172.21.15.3:0/1059472051 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5629072fb9a0 con 0x5629074295d0 2023-09-10T01:08:37.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68b9ffb700 10 client.27263 handle_command_reply: tid=0 2023-09-10T01:08:37.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 client.27263 shutdown 2023-09-10T01:08:37.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 msgr2=0x5629073c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073c0eb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f68cc097220 tx=0x7f68cc09a1d0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 shutdown_connections 2023-09-10T01:08:37.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5629074295d0 0x56290742b9c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5629073baec0 0x5629073c0eb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629073b0790 0x562907421b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68b9ffb700 0 client.27263 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 --2- 172.21.15.3:0/1059472051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5629073b10b0 0x5629074220b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 >> 172.21.15.3:0/1059472051 conn(0x562907325110 msgr2=0x5629073b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 shutdown_connections 2023-09-10T01:08:37.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 1 -- 172.21.15.3:0/1059472051 wait complete. 2023-09-10T01:08:37.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.233+0000 7f68ec921b80 20 client.27263 trim_cache size 0 max 16384 2023-09-10T01:08:37.246 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:08:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:08:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "b8b95580-55cf-4b81-8f47-eca1e1a4e4c3", 2023-09-10T01:08:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:08:37.247 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:08:37.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a0830 msgr2=0x7f68bc0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a0830 0x7f68bc0a2cf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009d70 tx=0x7f68c4009260 comp rx=0 tx=0).stop 2023-09-10T01:08:37.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4106660 msgr2=0x7f68d4198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4106660 0x7f68d4198750 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009850 tx=0x7f68cc0077f0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 shutdown_connections 2023-09-10T01:08:37.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a0830 0x7f68bc0a2cf0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4100650 0x7f68d419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4106660 0x7f68d4198750 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 --2- 172.21.15.3:0/2137370452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d4101010 0x7f68d419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 >> 172.21.15.3:0/2137370452 conn(0x7f68d40fc160 msgr2=0x7f68d410b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 shutdown_connections 2023-09-10T01:08:37.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.234+0000 7f68d8fff700 1 -- 172.21.15.3:0/2137370452 wait complete. 2023-09-10T01:08:37.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:08:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.485+0000 7ff40f3ff700 1 -- 172.21.15.3:0/1042339714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408106660 msgr2=0x7ff408106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.485+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/1042339714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408106660 0x7ff408106a40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff404009990 tx=0x7ff404009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.486+0000 7ff40f3ff700 1 -- 172.21.15.3:0/1042339714 shutdown_connections 2023-09-10T01:08:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.486+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/1042339714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff408101010 0x7ff40810d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.486+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/1042339714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408106660 0x7ff408106a40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.486+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/1042339714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408100650 0x7ff408100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.486+0000 7ff40f3ff700 1 -- 172.21.15.3:0/1042339714 >> 172.21.15.3:0/1042339714 conn(0x7ff4080fc160 msgr2=0x7ff4080fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.486+0000 7ff40f3ff700 1 -- 172.21.15.3:0/1042339714 shutdown_connections 2023-09-10T01:08:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.486+0000 7ff40f3ff700 1 -- 172.21.15.3:0/1042339714 wait complete. 2023-09-10T01:08:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 Processor -- start 2023-09-10T01:08:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 -- start start 2023-09-10T01:08:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408100650 0x7ff40819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff408101010 0x7ff40819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408106660 0x7ff408198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff408198ed0 con 0x7ff408101010 2023-09-10T01:08:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff408199040 con 0x7ff408100650 2023-09-10T01:08:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.487+0000 7ff40f3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff408199200 con 0x7ff408106660 2023-09-10T01:08:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408106660 0x7ff408198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40c99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff408101010 0x7ff40819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408100650 0x7ff40819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408106660 0x7ff408198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55642/0 (socket says 172.21.15.3:55642) 2023-09-10T01:08:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408100650 0x7ff40819e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49526/0 (socket says 172.21.15.3:49526) 2023-09-10T01:08:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 -- 172.21.15.3:0/2314698392 learned_addr learned my addr 172.21.15.3:0/2314698392 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 -- 172.21.15.3:0/2314698392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408100650 msgr2=0x7ff40819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408100650 0x7ff40819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 -- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff408101010 msgr2=0x7ff40819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff408101010 0x7ff40819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 -- 172.21.15.3:0/2314698392 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f8009520 con 0x7ff408106660 2023-09-10T01:08:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408100650 0x7ff40819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40c99a700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff408101010 0x7ff40819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.488+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408106660 0x7ff408198860 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff40000ef30 tx=0x7ff40000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.489+0000 7ff3fe7fc700 1 -- 172.21.15.3:0/2314698392 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400018070 con 0x7ff408106660 2023-09-10T01:08:37.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.489+0000 7ff3fe7fc700 1 -- 172.21.15.3:0/2314698392 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40000f040 con 0x7ff408106660 2023-09-10T01:08:37.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.489+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4040095f0 con 0x7ff408106660 2023-09-10T01:08:37.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.489+0000 7ff3fe7fc700 1 -- 172.21.15.3:0/2314698392 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40000cd00 con 0x7ff408106660 2023-09-10T01:08:37.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.489+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4081a5600 con 0x7ff408106660 2023-09-10T01:08:37.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.492+0000 7ff3fe7fc700 1 -- 172.21.15.3:0/2314698392 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ff400003a00 con 0x7ff408106660 2023-09-10T01:08:37.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.494+0000 7ff422d7db80 1 Processor -- start 2023-09-10T01:08:37.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.494+0000 7ff3fe7fc700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff3f40a0830 0x7ff3f40a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.494+0000 7ff422d7db80 1 -- start start 2023-09-10T01:08:37.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.494+0000 7ff3fe7fc700 1 -- 172.21.15.3:0/2314698392 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff4000cbd20 con 0x7ff408106660 2023-09-10T01:08:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.494+0000 7ff40c99a700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff3f40a0830 0x7ff3f40a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff422d7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be1533730 con 0x556be15db030 2023-09-10T01:08:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff422d7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be1531050 con 0x556be15da710 2023-09-10T01:08:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff422d7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be152c940 con 0x556be15e4e40 2023-09-10T01:08:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15daaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49540/0 (socket says 172.21.15.3:49540) 2023-09-10T01:08:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 -- 172.21.15.3:0/2079583107 learned_addr learned my addr 172.21.15.3:0/2079583107 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40c99a700 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 -- 172.21.15.3:0/2079583107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 msgr2=0x556be15e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 -- 172.21.15.3:0/2079583107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 msgr2=0x556be15e4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.495+0000 7ff40d19b700 1 -- 172.21.15.3:0/2079583107 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556be1358de0 con 0x556be15da710 2023-09-10T01:08:37.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.496+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15daaf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff404005ee0 tx=0x7ff404005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39514ff82d6e056d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.496+0000 7ff40c99a700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff3f40a0830 0x7ff3f40a2cf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009fd0 tx=0x7ff3f8013e30 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.496+0000 7ff3f37fe700 1 -- 172.21.15.3:0/2079583107 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40401d070 con 0x556be15da710 2023-09-10T01:08:37.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.496+0000 7ff3f37fe700 1 -- 172.21.15.3:0/2079583107 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40401e040 con 0x556be15da710 2023-09-10T01:08:37.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.496+0000 7ff3f37fe700 1 -- 172.21.15.3:0/2079583107 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404004d50 con 0x556be15da710 2023-09-10T01:08:37.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 -- 172.21.15.3:0/2079583107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 msgr2=0x556be15daaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15daaf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff404005ee0 tx=0x7ff404005f10 comp rx=0 tx=0).stop 2023-09-10T01:08:37.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 -- 172.21.15.3:0/2079583107 shutdown_connections 2023-09-10T01:08:37.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15daaf0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2079583107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 -- 172.21.15.3:0/2079583107 >> 172.21.15.3:0/2079583107 conn(0x556be154f090 msgr2=0x556be15d5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 -- 172.21.15.3:0/2079583107 shutdown_connections 2023-09-10T01:08:37.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 -- 172.21.15.3:0/2079583107 wait complete. 2023-09-10T01:08:37.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.497+0000 7ff422d7db80 1 -- start start 2023-09-10T01:08:37.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 1 Processor -- start 2023-09-10T01:08:37.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15e2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be14cab70 con 0x556be15db030 2023-09-10T01:08:37.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be14ca870 con 0x556be15e4e40 2023-09-10T01:08:37.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be13ae920 con 0x556be15da710 2023-09-10T01:08:37.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.498+0000 7ff422d7db80 10 client.0 _ll_register_callbacks cb 0x556be1414a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:37.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15e2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40c99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15e2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49542/0 (socket says 172.21.15.3:49542) 2023-09-10T01:08:37.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55666/0 (socket says 172.21.15.3:55666) 2023-09-10T01:08:37.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 -- 172.21.15.3:0/2854579775 learned_addr learned my addr 172.21.15.3:0/2854579775 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 -- 172.21.15.3:0/2854579775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 msgr2=0x556be15e48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 -- 172.21.15.3:0/2854579775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 msgr2=0x556be15e1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 -- 172.21.15.3:0/2854579775 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40810de20 con 0x556be15e4e40 2023-09-10T01:08:37.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15e2220 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c6f0 tx=0x7ff400098920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af9f20ffb1e509b0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff3f0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:37.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/2854579775 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4000b0070 con 0x556be15e4e40 2023-09-10T01:08:37.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/2854579775 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff4000995d0 con 0x556be15e4e40 2023-09-10T01:08:37.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.499+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556be14844f0 con 0x556be15e4e40 2023-09-10T01:08:37.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/2854579775 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4000b1260 con 0x556be15e4e40 2023-09-10T01:08:37.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 10 client.27275 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:37.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556be1341c60 con 0x556be15e4e40 2023-09-10T01:08:37.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/2854579775 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40007b020 con 0x556be15e4e40 2023-09-10T01:08:37.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/2854579775 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff4000983c0 con 0x556be15e4e40 2023-09-10T01:08:37.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 10 client.27275 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:37.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 10 client.27275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:37.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 20 client.27275 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:37.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be15f4830 0x556be15f6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 4 client.27275 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:37.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556be15418a0 con 0x556be15f4830 2023-09-10T01:08:37.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.500+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be15f4830 0x556be15f6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.501+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be15f4830 0x556be15f6c50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc06cd2ad4f82415 server_cookie=7c88f25a939c674b in_seq=0 out_seq=0 2023-09-10T01:08:37.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.501+0000 7ff3f0ff9700 10 client.27275 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.501+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/2854579775 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556be15418a0 con 0x556be15f4830 2023-09-10T01:08:37.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.501+0000 7ff3f0ff9700 10 client.27275 handle_command_reply: tid=0 2023-09-10T01:08:37.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.501+0000 7ff422d7db80 1 client.27275 shutdown 2023-09-10T01:08:37.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 msgr2=0x556be15e2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15e2220 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c6f0 tx=0x7ff400098920 comp rx=0 tx=0).stop 2023-09-10T01:08:37.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 shutdown_connections 2023-09-10T01:08:37.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be15f4830 0x556be15f6c50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15e2220 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff3f0ff9700 0 client.27275 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2854579775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 >> 172.21.15.3:0/2854579775 conn(0x556be154f090 msgr2=0x556be15d5780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 shutdown_connections 2023-09-10T01:08:37.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 1 -- 172.21.15.3:0/2854579775 wait complete. 2023-09-10T01:08:37.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.502+0000 7ff422d7db80 20 client.27275 trim_cache size 0 max 16384 2023-09-10T01:08:37.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.521+0000 7ff422d7db80 1 Processor -- start 2023-09-10T01:08:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff422d7db80 1 -- start start 2023-09-10T01:08:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15dcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15dd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff422d7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be15dc8d0 con 0x556be15db030 2023-09-10T01:08:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff422d7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be15cd400 con 0x556be15e4e40 2023-09-10T01:08:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff422d7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be15e2f70 con 0x556be15da710 2023-09-10T01:08:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55678/0 (socket says 172.21.15.3:55678) 2023-09-10T01:08:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff40d19b700 1 -- 172.21.15.3:0/2364594008 learned_addr learned my addr 172.21.15.3:0/2364594008 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15dd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.522+0000 7ff40c99a700 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15dcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff40d19b700 1 -- 172.21.15.3:0/2364594008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 msgr2=0x556be15dd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15dd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff40d19b700 1 -- 172.21.15.3:0/2364594008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 msgr2=0x556be15dcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15dcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff40d19b700 1 -- 172.21.15.3:0/2364594008 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556be137f250 con 0x556be15da710 2023-09-10T01:08:37.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff40d99c700 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15dd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:08:37.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff40d19b700 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e4400 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff40400bee0 tx=0x7ff40401a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e705450279facc6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff3f3fff700 1 -- 172.21.15.3:0/2364594008 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40401d040 con 0x556be15da710 2023-09-10T01:08:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff3f3fff700 1 -- 172.21.15.3:0/2364594008 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff404005bc0 con 0x556be15da710 2023-09-10T01:08:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.523+0000 7ff3f3fff700 1 -- 172.21.15.3:0/2364594008 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40400f6c0 con 0x556be15da710 2023-09-10T01:08:37.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 -- 172.21.15.3:0/2364594008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 msgr2=0x556be15e4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e4400 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff40400bee0 tx=0x7ff40401a040 comp rx=0 tx=0).stop 2023-09-10T01:08:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 -- 172.21.15.3:0/2364594008 shutdown_connections 2023-09-10T01:08:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15da710 0x556be15e4400 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15e4e40 0x556be15dd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 --2- 172.21.15.3:0/2364594008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15dcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 -- 172.21.15.3:0/2364594008 >> 172.21.15.3:0/2364594008 conn(0x556be154f090 msgr2=0x556be15d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 -- 172.21.15.3:0/2364594008 shutdown_connections 2023-09-10T01:08:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.524+0000 7ff422d7db80 1 -- 172.21.15.3:0/2364594008 wait complete. 2023-09-10T01:08:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.525+0000 7ff422d7db80 1 -- start start 2023-09-10T01:08:37.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.525+0000 7ff422d7db80 1 Processor -- start 2023-09-10T01:08:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.525+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff422d7db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff422d7db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be15ebf80 con 0x556be15db030 2023-09-10T01:08:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff422d7db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be15ea450 con 0x556be15da710 2023-09-10T01:08:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff422d7db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be15ea610 con 0x556be15e4e40 2023-09-10T01:08:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff422d7db80 10 client.0 _ll_register_callbacks cb 0x7ff400098d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40c99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15eb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49556/0 (socket says 172.21.15.3:49556) 2023-09-10T01:08:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 -- 172.21.15.3:0/3643872729 learned_addr learned my addr 172.21.15.3:0/3643872729 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40c99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43306/0 (socket says 172.21.15.3:43306) 2023-09-10T01:08:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d99c700 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 -- 172.21.15.3:0/3643872729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 msgr2=0x556be15e9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 -- 172.21.15.3:0/3643872729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 msgr2=0x556be15e99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:37.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 -- 172.21.15.3:0/3643872729 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556be1358de0 con 0x556be15da710 2023-09-10T01:08:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff40d19b700 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15eb6b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff404005a00 tx=0x7ff404005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ce731234922d64 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.526+0000 7ff3f0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/3643872729 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40401d040 con 0x556be15da710 2023-09-10T01:08:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556be1525920 con 0x556be15da710 2023-09-10T01:08:37.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/3643872729 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7ff40400f770 con 0x556be15da710 2023-09-10T01:08:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/3643872729 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff40400f950 con 0x556be15da710 2023-09-10T01:08:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff422d7db80 10 client.27281 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556be13572b0 con 0x556be15da710 2023-09-10T01:08:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/3643872729 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556be1525920 con 0x556be15da710 2023-09-10T01:08:37.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.527+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/3643872729 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff404022630 con 0x556be15da710 2023-09-10T01:08:37.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff422d7db80 10 client.27281 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:37.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff422d7db80 10 client.27281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:37.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff422d7db80 20 client.27281 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:37.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff422d7db80 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be16495d0 0x556be164b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:37.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff422d7db80 4 client.27281 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:08:37.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556be14e60b0 con 0x556be16495d0 2023-09-10T01:08:37.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff40c99a700 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be16495d0 0x556be164b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:37.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff40c99a700 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be16495d0 0x556be164b9c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74b70d78e8d08ff1 server_cookie=1dc2cfdb4aafd56 in_seq=0 out_seq=0 2023-09-10T01:08:37.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff3f0ff9700 10 client.27281 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff3f0ff9700 1 -- 172.21.15.3:0/3643872729 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556be14e60b0 con 0x556be16495d0 2023-09-10T01:08:37.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff3f0ff9700 10 client.27281 handle_command_reply: tid=0 2023-09-10T01:08:37.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.528+0000 7ff422d7db80 1 client.27281 shutdown 2023-09-10T01:08:37.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 msgr2=0x556be15eb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15eb6b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff404005a00 tx=0x7ff404005c50 comp rx=0 tx=0).stop 2023-09-10T01:08:37.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 shutdown_connections 2023-09-10T01:08:37.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x556be16495d0 0x556be164b9c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556be15e4e40 0x556be15e9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556be15da710 0x556be15eb6b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 --2- 172.21.15.3:0/3643872729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556be15db030 0x556be15e99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 >> 172.21.15.3:0/3643872729 conn(0x556be154f090 msgr2=0x556be15dcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff3f0ff9700 0 client.27281 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:37.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 shutdown_connections 2023-09-10T01:08:37.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 1 -- 172.21.15.3:0/3643872729 wait complete. 2023-09-10T01:08:37.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.529+0000 7ff422d7db80 20 client.27281 trim_cache size 0 max 16384 2023-09-10T01:08:37.546 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:08:37.546 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:08:37.546 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:08:37.547 INFO:teuthology.orchestra.run.smithi003.stdout: "b8b95580-55cf-4b81-8f47-eca1e1a4e4c3": { 2023-09-10T01:08:37.547 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:08:37.547 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "b8b95580-55cf-4b81-8f47-eca1e1a4e4c3", 2023-09-10T01:08:37.547 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:08:37.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:08:37.548 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:08:37.548 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:08:37.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff3f40a0830 msgr2=0x7ff3f40a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff3f40a0830 0x7ff3f40a2cf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009fd0 tx=0x7ff3f8013e30 comp rx=0 tx=0).stop 2023-09-10T01:08:37.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408106660 msgr2=0x7ff408198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:37.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408106660 0x7ff408198860 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff40000ef30 tx=0x7ff40000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 shutdown_connections 2023-09-10T01:08:37.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff3f40a0830 0x7ff3f40a2cf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff408106660 0x7ff408198860 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff408100650 0x7ff40819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 --2- 172.21.15.3:0/2314698392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff408101010 0x7ff40819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:37.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 >> 172.21.15.3:0/2314698392 conn(0x7ff4080fc160 msgr2=0x7ff4080fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:37.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 shutdown_connections 2023-09-10T01:08:37.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:08:37.530+0000 7ff40f3ff700 1 -- 172.21.15.3:0/2314698392 wait complete. 2023-09-10T01:08:37.552 INFO:tasks.cephfs.filesystem:scrub status for tag:b8b95580-55cf-4b81-8f47-eca1e1a4e4c3 - {'path': '/', 'tag': 'b8b95580-55cf-4b81-8f47-eca1e1a4e4c3', 'options': 'recursive,force'} 2023-09-10T01:08:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:37 smithi136 ceph-mon[133708]: pgmap v165: 129 pgs: 129 active+clean; 123 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2023-09-10T01:08:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:37 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:08:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:37 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:37 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:08:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:37 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:37.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:37 smithi092 ceph-mon[134067]: pgmap v165: 129 pgs: 129 active+clean; 123 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2023-09-10T01:08:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:37 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:08:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:37 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:37 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:08:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:37 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:37.757 INFO:tasks.workunit.client.0.smithi003.stdout: 1 462222 9.51 MB/sec execute 139 sec latency 13.378 ms 2023-09-10T01:08:37.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:37 smithi003 ceph-mon[97654]: pgmap v165: 129 pgs: 129 active+clean; 123 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 216 op/s 2023-09-10T01:08:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:37 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:08:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:37 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:37 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:08:37.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:37 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:38 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:08:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:38 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:38 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:08:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:38 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:08:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:38 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:38 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:38 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:08:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:38 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:38 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:08:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:38 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:08:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:38 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:38 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.757 INFO:tasks.workunit.client.0.smithi003.stdout: 1 463764 9.48 MB/sec execute 140 sec latency 37.873 ms 2023-09-10T01:08:38.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:38 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:08:38.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:38 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:38 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:08:38.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:38 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:08:38.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:38 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:38.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:38 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:08:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:39 smithi136 ceph-mon[133708]: pgmap v166: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 384 op/s 2023-09-10T01:08:39.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:39 smithi092 ceph-mon[134067]: pgmap v166: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 384 op/s 2023-09-10T01:08:39.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 465890 9.51 MB/sec execute 141 sec latency 11.106 ms 2023-09-10T01:08:39.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:39 smithi003 ceph-mon[97654]: pgmap v166: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 384 op/s 2023-09-10T01:08:40.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 467292 9.48 MB/sec execute 142 sec latency 78.121 ms 2023-09-10T01:08:41.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 469421 9.51 MB/sec execute 143 sec latency 20.491 ms 2023-09-10T01:08:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:42 smithi136 ceph-mon[133708]: pgmap v167: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-09-10T01:08:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:42 smithi092 ceph-mon[134067]: pgmap v167: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-09-10T01:08:42.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:42 smithi003 ceph-mon[97654]: pgmap v167: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-09-10T01:08:42.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 471414 9.49 MB/sec execute 144 sec latency 19.234 ms 2023-09-10T01:08:43.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 473228 9.51 MB/sec execute 145 sec latency 10.756 ms 2023-09-10T01:08:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:44 smithi136 ceph-mon[133708]: pgmap v168: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-09-10T01:08:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:44 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:08:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:44 smithi092 ceph-mon[134067]: pgmap v168: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-09-10T01:08:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:44 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:08:44.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:44 smithi003 ceph-mon[97654]: pgmap v168: 129 pgs: 129 active+clean; 116 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-09-10T01:08:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:44 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:08:44.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 475529 9.53 MB/sec execute 146 sec latency 16.993 ms 2023-09-10T01:08:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:45 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:08:45.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:45 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:08:45.469 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:45 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:08:45.470 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:45 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:08:45.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:45 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:08:45.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:45 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:08:45.759 INFO:tasks.workunit.client.0.smithi003.stdout: 1 476662 9.51 MB/sec execute 147 sec latency 45.099 ms 2023-09-10T01:08:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:46 smithi136 ceph-mon[133708]: pgmap v169: 129 pgs: 129 active+clean; 122 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 390 op/s 2023-09-10T01:08:46.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:46 smithi092 ceph-mon[134067]: pgmap v169: 129 pgs: 129 active+clean; 122 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 390 op/s 2023-09-10T01:08:46.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:46 smithi003 ceph-mon[97654]: pgmap v169: 129 pgs: 129 active+clean; 122 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 390 op/s 2023-09-10T01:08:46.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 478618 9.50 MB/sec execute 148 sec latency 29.851 ms 2023-09-10T01:08:47.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 480379 9.51 MB/sec execute 149 sec latency 16.473 ms 2023-09-10T01:08:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.017+0000 7feb773ff700 1 -- 172.21.15.3:0/275241656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700732c0 msgr2=0x7feb70111070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.017+0000 7feb773ff700 1 --2- 172.21.15.3:0/275241656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700732c0 0x7feb70111070 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feb60009990 tx=0x7feb60009ca0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.018+0000 7feb773ff700 1 -- 172.21.15.3:0/275241656 shutdown_connections 2023-09-10T01:08:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.018+0000 7feb773ff700 1 --2- 172.21.15.3:0/275241656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb701115b0 0x7feb701139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.018+0000 7feb773ff700 1 --2- 172.21.15.3:0/275241656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb700729a0 0x7feb70072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.018+0000 7feb773ff700 1 --2- 172.21.15.3:0/275241656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700732c0 0x7feb70111070 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.018+0000 7feb773ff700 1 -- 172.21.15.3:0/275241656 >> 172.21.15.3:0/275241656 conn(0x7feb700fc0b0 msgr2=0x7feb700fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:48.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.018+0000 7feb773ff700 1 -- 172.21.15.3:0/275241656 shutdown_connections 2023-09-10T01:08:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.018+0000 7feb773ff700 1 -- 172.21.15.3:0/275241656 wait complete. 2023-09-10T01:08:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 Processor -- start 2023-09-10T01:08:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 -- start start 2023-09-10T01:08:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700729a0 0x7feb7019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb700732c0 0x7feb7019d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb701115b0 0x7feb701a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb701a1d50 con 0x7feb701115b0 2023-09-10T01:08:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb701a1ec0 con 0x7feb700729a0 2023-09-10T01:08:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.019+0000 7feb773ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb701a2080 con 0x7feb700732c0 2023-09-10T01:08:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7519b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700729a0 0x7feb7019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb700732c0 0x7feb7019d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7519b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700729a0 0x7feb7019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:51162/0 (socket says 172.21.15.3:51162) 2023-09-10T01:08:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb700732c0 0x7feb7019d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41264/0 (socket says 172.21.15.3:41264) 2023-09-10T01:08:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7519b700 1 -- 172.21.15.3:0/827528292 learned_addr learned my addr 172.21.15.3:0/827528292 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7599c700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb701115b0 0x7feb701a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 -- 172.21.15.3:0/827528292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700729a0 msgr2=0x7feb7019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700729a0 0x7feb7019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 -- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb701115b0 msgr2=0x7feb701a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb701115b0 0x7feb701a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 -- 172.21.15.3:0/827528292 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb600095f0 con 0x7feb700732c0 2023-09-10T01:08:48.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7519b700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700729a0 0x7feb7019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.020+0000 7feb7499a700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb700732c0 0x7feb7019d2b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feb60000c00 tx=0x7feb60005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.021+0000 7feb667fc700 1 -- 172.21.15.3:0/827528292 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6001d070 con 0x7feb700732c0 2023-09-10T01:08:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.021+0000 7feb667fc700 1 -- 172.21.15.3:0/827528292 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb60021410 con 0x7feb700732c0 2023-09-10T01:08:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.021+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb701a2300 con 0x7feb700732c0 2023-09-10T01:08:48.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.021+0000 7feb667fc700 1 -- 172.21.15.3:0/827528292 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60017650 con 0x7feb700732c0 2023-09-10T01:08:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.021+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb701a9ba0 con 0x7feb700732c0 2023-09-10T01:08:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.024+0000 7feb667fc700 1 -- 172.21.15.3:0/827528292 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7feb600217f0 con 0x7feb700732c0 2023-09-10T01:08:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.026+0000 7feb8adfdb80 1 Processor -- start 2023-09-10T01:08:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.026+0000 7feb8adfdb80 1 -- start start 2023-09-10T01:08:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.026+0000 7feb667fc700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb5c0a0830 0x7feb5c0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.026+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a908e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9088780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a9094f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a8fe16d0 con 0x55b6a9088300 2023-09-10T01:08:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a8fdeff0 con 0x55b6a9088cc0 2023-09-10T01:08:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a8fda8e0 con 0x55b6a908e1a0 2023-09-10T01:08:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb667fc700 1 -- 172.21.15.3:0/827528292 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7feb600cd600 con 0x7feb700732c0 2023-09-10T01:08:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb7599c700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb5c0a0830 0x7feb5c0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb7599c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a9094f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb7599c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a9094f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:51174/0 (socket says 172.21.15.3:51174) 2023-09-10T01:08:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb7599c700 1 -- 172.21.15.3:0/3630981058 learned_addr learned my addr 172.21.15.3:0/3630981058 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb7499a700 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9088780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.027+0000 7feb7519b700 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a908e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7599c700 1 -- 172.21.15.3:0/3630981058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 msgr2=0x55b6a908e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7599c700 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a908e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7599c700 1 -- 172.21.15.3:0/3630981058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 msgr2=0x55b6a9088780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7599c700 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9088780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7599c700 1 -- 172.21.15.3:0/3630981058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6a8e06d80 con 0x55b6a9088cc0 2023-09-10T01:08:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7499a700 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9088780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7599c700 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a9094f30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb6801a510 tx=0x7feb6801a820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5596d7476dc52f8f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb7599c700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb5c0a0830 0x7feb5c0a2cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feb6801ab70 tx=0x7feb68011000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.028+0000 7feb5b7fe700 1 -- 172.21.15.3:0/3630981058 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800e040 con 0x55b6a9088cc0 2023-09-10T01:08:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb5b7fe700 1 -- 172.21.15.3:0/3630981058 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6801f070 con 0x55b6a9088cc0 2023-09-10T01:08:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb5b7fe700 1 -- 172.21.15.3:0/3630981058 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb68008510 con 0x55b6a9088cc0 2023-09-10T01:08:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb8adfdb80 1 -- 172.21.15.3:0/3630981058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 msgr2=0x55b6a9094f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a9094f30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb6801a510 tx=0x7feb6801a820 comp rx=0 tx=0).stop 2023-09-10T01:08:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb8adfdb80 1 -- 172.21.15.3:0/3630981058 shutdown_connections 2023-09-10T01:08:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a908e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a9094f30 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/3630981058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9088780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.029+0000 7feb8adfdb80 1 -- 172.21.15.3:0/3630981058 >> 172.21.15.3:0/3630981058 conn(0x55b6a8ffd030 msgr2=0x55b6a9083560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.030+0000 7feb8adfdb80 1 -- 172.21.15.3:0/3630981058 shutdown_connections 2023-09-10T01:08:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.030+0000 7feb8adfdb80 1 -- 172.21.15.3:0/3630981058 wait complete. 2023-09-10T01:08:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.030+0000 7feb8adfdb80 1 -- start start 2023-09-10T01:08:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.030+0000 7feb8adfdb80 1 Processor -- start 2023-09-10T01:08:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.031+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9097e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.031+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90983b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.031+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a9092100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.031+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a8f78b10 con 0x55b6a9088300 2023-09-10T01:08:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.031+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a8f78810 con 0x55b6a9088cc0 2023-09-10T01:08:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.031+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a8e5c8c0 con 0x55b6a908e1a0 2023-09-10T01:08:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.031+0000 7feb8adfdb80 10 client.0 _ll_register_callbacks cb 0x55b6a8ec2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a9092100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7499a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a9092100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41282/0 (socket says 172.21.15.3:41282) 2023-09-10T01:08:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 -- 172.21.15.3:0/4097042437 learned_addr learned my addr 172.21.15.3:0/4097042437 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7519b700 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9097e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 -- 172.21.15.3:0/4097042437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 msgr2=0x55b6a90983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 -- 172.21.15.3:0/4097042437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 msgr2=0x55b6a9097e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9097e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 -- 172.21.15.3:0/4097042437 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb701a9ba0 con 0x55b6a908e1a0 2023-09-10T01:08:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7519b700 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9097e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:08:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb7599c700 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a9092100 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feb68013ca0 tx=0x7feb6801bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9a35cd941bff593 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.032+0000 7feb58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:08:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb58ff9700 1 -- 172.21.15.3:0/4097042437 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800e050 con 0x55b6a908e1a0 2023-09-10T01:08:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b6a8f32490 con 0x55b6a908e1a0 2023-09-10T01:08:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb58ff9700 1 -- 172.21.15.3:0/4097042437 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7feb68021070 con 0x55b6a908e1a0 2023-09-10T01:08:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb58ff9700 1 -- 172.21.15.3:0/4097042437 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb68004d10 con 0x55b6a908e1a0 2023-09-10T01:08:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb8adfdb80 10 client.27208 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7feb68013cd0 con 0x55b6a908e1a0 2023-09-10T01:08:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb58ff9700 1 -- 172.21.15.3:0/4097042437 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b6a8f32490 con 0x55b6a908e1a0 2023-09-10T01:08:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.033+0000 7feb58ff9700 1 -- 172.21.15.3:0/4097042437 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7feb6802d3f0 con 0x55b6a908e1a0 2023-09-10T01:08:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.034+0000 7feb8adfdb80 10 client.27208 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.034+0000 7feb8adfdb80 10 client.27208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.034+0000 7feb8adfdb80 20 client.27208 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.034+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90a2380 0x55b6a90a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.034+0000 7feb8adfdb80 4 client.27208 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:08:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.034+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6a8fef840 con 0x55b6a90a2380 2023-09-10T01:08:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.034+0000 7feb7519b700 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90a2380 0x55b6a90a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.035+0000 7feb58ff9700 10 client.27208 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.035+0000 7feb7519b700 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90a2380 0x55b6a90a47a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=967acc19d5f2b36a server_cookie=acd69852c5fcfdd3 in_seq=0 out_seq=0 2023-09-10T01:08:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.035+0000 7feb58ff9700 1 -- 172.21.15.3:0/4097042437 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b6a8fef840 con 0x55b6a90a2380 2023-09-10T01:08:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.035+0000 7feb58ff9700 10 client.27208 handle_command_reply: tid=0 2023-09-10T01:08:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 client.27208 shutdown 2023-09-10T01:08:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 msgr2=0x55b6a9092100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a9092100 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feb68013ca0 tx=0x7feb6801bda0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 shutdown_connections 2023-09-10T01:08:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90a2380 0x55b6a90a47a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a908e1a0 0x55b6a9092100 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a9088300 0x55b6a9097e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/4097042437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 >> 172.21.15.3:0/4097042437 conn(0x55b6a8ffd030 msgr2=0x55b6a90827a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb58ff9700 0 client.27208 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 shutdown_connections 2023-09-10T01:08:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 1 -- 172.21.15.3:0/4097042437 wait complete. 2023-09-10T01:08:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.036+0000 7feb8adfdb80 20 client.27208 trim_cache size 0 max 16384 2023-09-10T01:08:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 Processor -- start 2023-09-10T01:08:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 -- start start 2023-09-10T01:08:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088300 0x55b6a9092230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088cc0 0x55b6a9087a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a9082f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a90890a0 con 0x55b6a908e1a0 2023-09-10T01:08:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a907ffc0 con 0x55b6a9088300 2023-09-10T01:08:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.054+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a908de00 con 0x55b6a9088cc0 2023-09-10T01:08:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088300 0x55b6a9092230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088300 0x55b6a9092230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:51190/0 (socket says 172.21.15.3:51190) 2023-09-10T01:08:48.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7499a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088cc0 0x55b6a9087a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 -- 172.21.15.3:0/836000689 learned_addr learned my addr 172.21.15.3:0/836000689 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7599c700 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a9082f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 -- 172.21.15.3:0/836000689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088cc0 msgr2=0x55b6a9087a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088cc0 0x55b6a9087a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 -- 172.21.15.3:0/836000689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 msgr2=0x55b6a9082f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a9082f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 -- 172.21.15.3:0/836000689 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6a9092d00 con 0x55b6a9088300 2023-09-10T01:08:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7499a700 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088cc0 0x55b6a9087a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb7519b700 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088300 0x55b6a9092230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb6c004490 tx=0x7feb6c01c590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c56481a1fd66b9cb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb5bfff700 1 -- 172.21.15.3:0/836000689 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c021070 con 0x55b6a9088300 2023-09-10T01:08:48.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.055+0000 7feb5bfff700 1 -- 172.21.15.3:0/836000689 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6c01cd70 con 0x55b6a9088300 2023-09-10T01:08:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb5bfff700 1 -- 172.21.15.3:0/836000689 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c017040 con 0x55b6a9088300 2023-09-10T01:08:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 -- 172.21.15.3:0/836000689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088300 msgr2=0x55b6a9092230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088300 0x55b6a9092230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb6c004490 tx=0x7feb6c01c590 comp rx=0 tx=0).stop 2023-09-10T01:08:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 -- 172.21.15.3:0/836000689 shutdown_connections 2023-09-10T01:08:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a9082f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088300 0x55b6a9092230 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/836000689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088cc0 0x55b6a9087a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 -- 172.21.15.3:0/836000689 >> 172.21.15.3:0/836000689 conn(0x55b6a8ffd030 msgr2=0x55b6a90872f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 -- 172.21.15.3:0/836000689 shutdown_connections 2023-09-10T01:08:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.056+0000 7feb8adfdb80 1 -- 172.21.15.3:0/836000689 wait complete. 2023-09-10T01:08:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 -- start start 2023-09-10T01:08:48.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 Processor -- start 2023-09-10T01:08:48.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088300 0x55b6a90ea310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90ea850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a90f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a90f56e0 con 0x55b6a908e1a0 2023-09-10T01:08:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a90f5820 con 0x55b6a9088cc0 2023-09-10T01:08:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6a90f59e0 con 0x55b6a9088300 2023-09-10T01:08:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.057+0000 7feb8adfdb80 10 client.0 _ll_register_callbacks cb 0x7feb60005be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:08:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7599c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a90f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90ea850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:51206/0 (socket says 172.21.15.3:51206) 2023-09-10T01:08:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 -- 172.21.15.3:0/2050628569 learned_addr learned my addr 172.21.15.3:0/2050628569 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:08:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7519b700 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088300 0x55b6a90ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 -- 172.21.15.3:0/2050628569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088300 msgr2=0x55b6a90ea310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088300 0x55b6a90ea310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 -- 172.21.15.3:0/2050628569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 msgr2=0x55b6a90f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:08:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a90f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 -- 172.21.15.3:0/2050628569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb701a2300 con 0x55b6a9088cc0 2023-09-10T01:08:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7599c700 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a90f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:08:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb7499a700 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90ea850 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb600a2200 tx=0x7feb600caec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a1758ec7d4e6c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:08:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.058+0000 7feb58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:08:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb58ff9700 1 -- 172.21.15.3:0/2050628569 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60099e40 con 0x55b6a9088cc0 2023-09-10T01:08:48.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb58ff9700 1 -- 172.21.15.3:0/2050628569 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7feb6009ae50 con 0x55b6a9088cc0 2023-09-10T01:08:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb6801f070 con 0x55b6a9088cc0 2023-09-10T01:08:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb58ff9700 1 -- 172.21.15.3:0/2050628569 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb6009a750 con 0x55b6a9088cc0 2023-09-10T01:08:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 10 client.27299 fetch_fsmap learned FSMap version 26 2023-09-10T01:08:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7feb6c009ca0 con 0x55b6a9088cc0 2023-09-10T01:08:48.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb58ff9700 1 -- 172.21.15.3:0/2050628569 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6801f070 con 0x55b6a9088cc0 2023-09-10T01:08:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb58ff9700 1 -- 172.21.15.3:0/2050628569 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7feb600a23a0 con 0x55b6a9088cc0 2023-09-10T01:08:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 10 client.27299 fetch_fsmap finished waiting for FSMap version 26 2023-09-10T01:08:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 10 client.27299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:08:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 20 client.27299 populate_metadata read hostname 'smithi003' 2023-09-10T01:08:48.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90f6e30 0x55b6a90f9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:08:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 4 client.27299 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:08:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.059+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b6a8fd38c0 con 0x55b6a90f6e30 2023-09-10T01:08:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.060+0000 7feb7519b700 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90f6e30 0x55b6a90f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:08:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.060+0000 7feb7519b700 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90f6e30 0x55b6a90f9250 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6100ee7f84862b7 server_cookie=c8afb5c0eccfab22 in_seq=0 out_seq=0 2023-09-10T01:08:48.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.060+0000 7feb58ff9700 10 client.27299 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.193+0000 7feb58ff9700 1 -- 172.21.15.3:0/2050628569 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b6a8fd38c0 con 0x55b6a90f6e30 2023-09-10T01:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.193+0000 7feb58ff9700 10 client.27299 handle_command_reply: tid=0 2023-09-10T01:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.193+0000 7feb8adfdb80 1 client.27299 shutdown 2023-09-10T01:08:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.194+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 msgr2=0x55b6a90ea850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.194+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90ea850 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb600a2200 tx=0x7feb600caec0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.195+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 shutdown_connections 2023-09-10T01:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.195+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b6a90f6e30 0x55b6a90f9250 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.195+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b6a9088300 0x55b6a90ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.195+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b6a908e1a0 0x55b6a90f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.195+0000 7feb8adfdb80 1 --2- 172.21.15.3:0/2050628569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9088cc0 0x55b6a90ea850 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.195+0000 7feb58ff9700 0 client.27299 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.195+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 >> 172.21.15.3:0/2050628569 conn(0x55b6a8ffd030 msgr2=0x55b6a908ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.196+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 shutdown_connections 2023-09-10T01:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.196+0000 7feb8adfdb80 1 -- 172.21.15.3:0/2050628569 wait complete. 2023-09-10T01:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb8adfdb80 20 client.27299 trim_cache size 0 max 16384 2023-09-10T01:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb5c0a0830 msgr2=0x7feb5c0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb773ff700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb5c0a0830 0x7feb5c0a2cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feb6801ab70 tx=0x7feb68011000 comp rx=0 tx=0).stop 2023-09-10T01:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb700732c0 msgr2=0x7feb7019d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb773ff700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb700732c0 0x7feb7019d2b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feb60000c00 tx=0x7feb60005740 comp rx=0 tx=0).stop 2023-09-10T01:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 shutdown_connections 2023-09-10T01:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb773ff700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7feb5c0a0830 0x7feb5c0a2cf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.197+0000 7feb773ff700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb701115b0 0x7feb701a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.198+0000 7feb773ff700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb700729a0 0x7feb7019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.198+0000 7feb773ff700 1 --2- 172.21.15.3:0/827528292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb700732c0 0x7feb7019d2b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.198+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 >> 172.21.15.3:0/827528292 conn(0x7feb700fc0b0 msgr2=0x7feb700fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.198+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 shutdown_connections 2023-09-10T01:08:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:08:48.198+0000 7feb773ff700 1 -- 172.21.15.3:0/827528292 wait complete. 2023-09-10T01:08:48.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:48 smithi136 ceph-mon[133708]: pgmap v170: 129 pgs: 129 active+clean; 122 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-09-10T01:08:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:48 smithi092 ceph-mon[134067]: pgmap v170: 129 pgs: 129 active+clean; 122 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-09-10T01:08:48.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:48 smithi003 ceph-mon[97654]: pgmap v170: 129 pgs: 129 active+clean; 122 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-09-10T01:08:48.758 INFO:tasks.workunit.client.0.smithi003.stdout: 1 482098 9.50 MB/sec execute 150 sec latency 115.923 ms 2023-09-10T01:08:49.759 INFO:tasks.workunit.client.0.smithi003.stdout: 1 483895 9.51 MB/sec execute 151 sec latency 11.104 ms 2023-09-10T01:08:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:50 smithi136 ceph-mon[133708]: pgmap v171: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 453 op/s 2023-09-10T01:08:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:50 smithi092 ceph-mon[134067]: pgmap v171: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 453 op/s 2023-09-10T01:08:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:50 smithi003 ceph-mon[97654]: pgmap v171: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 453 op/s 2023-09-10T01:08:50.759 INFO:tasks.workunit.client.0.smithi003.stdout: 1 485636 9.50 MB/sec execute 152 sec latency 70.876 ms 2023-09-10T01:08:51.760 INFO:tasks.workunit.client.0.smithi003.stdout: 1 487194 9.51 MB/sec execute 153 sec latency 29.656 ms 2023-09-10T01:08:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:52 smithi136 ceph-mon[133708]: pgmap v172: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 285 op/s 2023-09-10T01:08:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:52 smithi092 ceph-mon[134067]: pgmap v172: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 285 op/s 2023-09-10T01:08:52.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:52 smithi003 ceph-mon[97654]: pgmap v172: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 285 op/s 2023-09-10T01:08:52.760 INFO:tasks.workunit.client.0.smithi003.stdout: 1 488740 9.49 MB/sec execute 154 sec latency 26.203 ms 2023-09-10T01:08:53.760 INFO:tasks.workunit.client.0.smithi003.stdout: 1 490717 9.51 MB/sec execute 155 sec latency 15.134 ms 2023-09-10T01:08:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:54 smithi136 ceph-mon[133708]: pgmap v173: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 285 op/s 2023-09-10T01:08:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:54 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:08:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:54 smithi092 ceph-mon[134067]: pgmap v173: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 285 op/s 2023-09-10T01:08:54.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:54 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:08:54.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:54 smithi003 ceph-mon[97654]: pgmap v173: 129 pgs: 129 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 285 op/s 2023-09-10T01:08:54.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:54 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:08:54.760 INFO:tasks.workunit.client.0.smithi003.stdout: 1 492399 9.49 MB/sec execute 156 sec latency 14.897 ms 2023-09-10T01:08:55.761 INFO:tasks.workunit.client.0.smithi003.stdout: 1 494227 9.51 MB/sec execute 157 sec latency 64.892 ms 2023-09-10T01:08:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:56 smithi136 ceph-mon[133708]: pgmap v174: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 396 op/s 2023-09-10T01:08:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:56 smithi092 ceph-mon[134067]: pgmap v174: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 396 op/s 2023-09-10T01:08:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:56 smithi003 ceph-mon[97654]: pgmap v174: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.0 MiB/s wr, 396 op/s 2023-09-10T01:08:56.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 494960 9.47 MB/sec execute 158 sec latency 186.795 ms 2023-09-10T01:08:57.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 496830 9.47 MB/sec execute 159 sec latency 48.523 ms 2023-09-10T01:08:58.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:08:58 smithi136 ceph-mon[133708]: pgmap v175: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 275 op/s 2023-09-10T01:08:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:08:58 smithi092 ceph-mon[134067]: pgmap v175: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 275 op/s 2023-09-10T01:08:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:08:58 smithi003 ceph-mon[97654]: pgmap v175: 129 pgs: 129 active+clean; 126 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 275 op/s 2023-09-10T01:08:58.761 INFO:tasks.workunit.client.0.smithi003.stdout: 1 498266 9.46 MB/sec execute 160 sec latency 14.354 ms 2023-09-10T01:08:59.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 500569 9.48 MB/sec execute 161 sec latency 15.741 ms 2023-09-10T01:09:00.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:00 smithi136 ceph-mon[133708]: pgmap v176: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.2 MiB/s wr, 450 op/s 2023-09-10T01:09:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:00 smithi092 ceph-mon[134067]: pgmap v176: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.2 MiB/s wr, 450 op/s 2023-09-10T01:09:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:00 smithi003 ceph-mon[97654]: pgmap v176: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.2 MiB/s wr, 450 op/s 2023-09-10T01:09:00.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 501681 9.46 MB/sec execute 162 sec latency 68.781 ms 2023-09-10T01:09:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 -- 172.21.15.3:0/1136169112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1944106680 msgr2=0x7f1944106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 --2- 172.21.15.3:0/1136169112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1944106680 0x7f1944106a60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f19400099c0 tx=0x7f1940009cd0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 -- 172.21.15.3:0/1136169112 shutdown_connections 2023-09-10T01:09:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 --2- 172.21.15.3:0/1136169112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1944100f80 0x7f194410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 --2- 172.21.15.3:0/1136169112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1944106680 0x7f1944106a60 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 --2- 172.21.15.3:0/1136169112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19441005c0 0x7f1944100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 -- 172.21.15.3:0/1136169112 >> 172.21.15.3:0/1136169112 conn(0x7f19440fc0b0 msgr2=0x7f19440fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 -- 172.21.15.3:0/1136169112 shutdown_connections 2023-09-10T01:09:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.382+0000 7f194c5ff700 1 -- 172.21.15.3:0/1136169112 wait complete. 2023-09-10T01:09:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.383+0000 7f194c5ff700 1 Processor -- start 2023-09-10T01:09:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194c5ff700 1 -- start start 2023-09-10T01:09:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194c5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19441005c0 0x7f1944072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194c5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1944100f80 0x7f1944072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194c5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1944106680 0x7f19441a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194c5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1944073670 con 0x7f19441005c0 2023-09-10T01:09:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194c5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19441a4d10 con 0x7f1944100f80 2023-09-10T01:09:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194c5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19441a4e50 con 0x7f1944106680 2023-09-10T01:09:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194a39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19441005c0 0x7f1944072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194a39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19441005c0 0x7f1944072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39708/0 (socket says 172.21.15.3:39708) 2023-09-10T01:09:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194a39b700 1 -- 172.21.15.3:0/2400748438 learned_addr learned my addr 172.21.15.3:0/2400748438 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f194ab9c700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1944106680 0x7f19441a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.384+0000 7f1949b9a700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1944100f80 0x7f1944072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194a39b700 1 -- 172.21.15.3:0/2400748438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1944106680 msgr2=0x7f19441a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194a39b700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1944106680 0x7f19441a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194a39b700 1 -- 172.21.15.3:0/2400748438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1944100f80 msgr2=0x7f1944072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194a39b700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1944100f80 0x7f1944072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194a39b700 1 -- 172.21.15.3:0/2400748438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400095f0 con 0x7f19441005c0 2023-09-10T01:09:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194a39b700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19441005c0 0x7f1944072900 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1940000c00 tx=0x7f194000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f193b7fe700 1 -- 172.21.15.3:0/2400748438 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194001d070 con 0x7f19441005c0 2023-09-10T01:09:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f193b7fe700 1 -- 172.21.15.3:0/2400748438 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1940005050 con 0x7f19441005c0 2023-09-10T01:09:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19441a5050 con 0x7f19441005c0 2023-09-10T01:09:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f193b7fe700 1 -- 172.21.15.3:0/2400748438 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940004ab0 con 0x7f19441005c0 2023-09-10T01:09:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.385+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19441a54d0 con 0x7f19441005c0 2023-09-10T01:09:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.389+0000 7f193b7fe700 1 -- 172.21.15.3:0/2400748438 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f19400051c0 con 0x7f19441005c0 2023-09-10T01:09:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.391+0000 7f195feaeb80 1 Processor -- start 2023-09-10T01:09:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.391+0000 7f195feaeb80 1 -- start start 2023-09-10T01:09:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.391+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce1367580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f193b7fe700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f19300a0830 0x7f19300a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce1361780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f195feaeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce12ba6d0 con 0x55dce1361cc0 2023-09-10T01:09:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f195feaeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce12b7ff0 con 0x55dce13671a0 2023-09-10T01:09:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f195feaeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce12b38e0 con 0x55dce1361300 2023-09-10T01:09:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f193b7fe700 1 -- 172.21.15.3:0/2400748438 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f19400ccfe0 con 0x7f19441005c0 2023-09-10T01:09:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce1361780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce1361780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:56706/0 (socket says 172.21.15.3:56706) 2023-09-10T01:09:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 -- 172.21.15.3:0/1038026916 learned_addr learned my addr 172.21.15.3:0/1038026916 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f194a39b700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f19300a0830 0x7f19300a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f194ab9c700 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce1367580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f194ab9c700 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 -- 172.21.15.3:0/1038026916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 msgr2=0x55dce1367580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce1367580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 -- 172.21.15.3:0/1038026916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 msgr2=0x55dce136df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f1949b9a700 1 -- 172.21.15.3:0/1038026916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dce10dfd80 con 0x55dce1361300 2023-09-10T01:09:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.392+0000 7f194ab9c700 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.393+0000 7f1949b9a700 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce1361780 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1934009d20 tx=0x7f1934011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a801323b2d257d6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.393+0000 7f194a39b700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f19300a0830 0x7f19300a2cf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f194009a100 tx=0x7f1940009cd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.393+0000 7f1938bfe700 1 -- 172.21.15.3:0/1038026916 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934019630 con 0x55dce1361300 2023-09-10T01:09:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.393+0000 7f1938bfe700 1 -- 172.21.15.3:0/1038026916 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1934019c50 con 0x55dce1361300 2023-09-10T01:09:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.393+0000 7f1938bfe700 1 -- 172.21.15.3:0/1038026916 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193400ec40 con 0x55dce1361300 2023-09-10T01:09:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 -- 172.21.15.3:0/1038026916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 msgr2=0x55dce1361780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce1361780 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1934009d20 tx=0x7f1934011720 comp rx=0 tx=0).stop 2023-09-10T01:09:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 -- 172.21.15.3:0/1038026916 shutdown_connections 2023-09-10T01:09:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce1367580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1038026916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce1361780 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 -- 172.21.15.3:0/1038026916 >> 172.21.15.3:0/1038026916 conn(0x55dce12d6030 msgr2=0x55dce135c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 -- 172.21.15.3:0/1038026916 shutdown_connections 2023-09-10T01:09:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.394+0000 7f195feaeb80 1 -- 172.21.15.3:0/1038026916 wait complete. 2023-09-10T01:09:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.395+0000 7f195feaeb80 1 -- start start 2023-09-10T01:09:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.395+0000 7f195feaeb80 1 Processor -- start 2023-09-10T01:09:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce136dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce136b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f195feaeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce1251b10 con 0x55dce1361cc0 2023-09-10T01:09:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f195feaeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce1251810 con 0x55dce13671a0 2023-09-10T01:09:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f195feaeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce11358c0 con 0x55dce1361300 2023-09-10T01:09:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f195feaeb80 10 client.0 _ll_register_callbacks cb 0x55dce119ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce136b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce136b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44604/0 (socket says 172.21.15.3:44604) 2023-09-10T01:09:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 -- 172.21.15.3:0/1670022353 learned_addr learned my addr 172.21.15.3:0/1670022353 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce136dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f1949b9a700 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 -- 172.21.15.3:0/1670022353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 msgr2=0x55dce136b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce136b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 -- 172.21.15.3:0/1670022353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 msgr2=0x55dce136b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.396+0000 7f194ab9c700 1 -- 172.21.15.3:0/1670022353 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1934009980 con 0x55dce1361300 2023-09-10T01:09:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f194ab9c700 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce136dda0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f193c01c730 tx=0x7f193c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1350a0d1d7cf5d7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f192dffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:09:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f192dffb700 1 -- 172.21.15.3:0/1670022353 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c00e040 con 0x55dce1361300 2023-09-10T01:09:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dce120b490 con 0x55dce1361300 2023-09-10T01:09:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f192dffb700 1 -- 172.21.15.3:0/1670022353 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f193c01ce10 con 0x55dce1361300 2023-09-10T01:09:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f192dffb700 1 -- 172.21.15.3:0/1670022353 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f193c008950 con 0x55dce1361300 2023-09-10T01:09:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f195feaeb80 10 client.27226 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dce10c8c00 con 0x55dce1361300 2023-09-10T01:09:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.397+0000 7f192dffb700 1 -- 172.21.15.3:0/1670022353 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dce120b490 con 0x55dce1361300 2023-09-10T01:09:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f192dffb700 1 -- 172.21.15.3:0/1670022353 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f193c004720 con 0x55dce1361300 2023-09-10T01:09:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f195feaeb80 10 client.27226 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f195feaeb80 10 client.27226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f195feaeb80 20 client.27226 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce137b3d0 0x55dce137d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f195feaeb80 4 client.27226 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dce12c8840 con 0x55dce137b3d0 2023-09-10T01:09:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.398+0000 7f1949b9a700 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce137b3d0 0x55dce137d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.399+0000 7f1949b9a700 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce137b3d0 0x55dce137d7f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee2d52bd8a74a9b9 server_cookie=f33fdf5a7fc5ab4f in_seq=0 out_seq=0 2023-09-10T01:09:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.399+0000 7f192dffb700 10 client.27226 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.400+0000 7f192dffb700 1 -- 172.21.15.3:0/1670022353 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dce12c8840 con 0x55dce137b3d0 2023-09-10T01:09:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.400+0000 7f192dffb700 10 client.27226 handle_command_reply: tid=0 2023-09-10T01:09:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.400+0000 7f195feaeb80 1 client.27226 shutdown 2023-09-10T01:09:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.400+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 msgr2=0x55dce136dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.400+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce136dda0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f193c01c730 tx=0x7f193c00b040 comp rx=0 tx=0).stop 2023-09-10T01:09:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 shutdown_connections 2023-09-10T01:09:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce137b3d0 0x55dce137d7f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361300 0x55dce136dda0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce13671a0 0x55dce136b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 --2- 172.21.15.3:0/1670022353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 >> 172.21.15.3:0/1670022353 conn(0x55dce12d6030 msgr2=0x55dce135b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f192dffb700 0 client.27226 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 shutdown_connections 2023-09-10T01:09:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 1 -- 172.21.15.3:0/1670022353 wait complete. 2023-09-10T01:09:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.401+0000 7f195feaeb80 20 client.27226 trim_cache size 0 max 16384 2023-09-10T01:09:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 Processor -- start 2023-09-10T01:09:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 -- start start 2023-09-10T01:09:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce13671a0 0x55dce1363b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce135db40 con 0x55dce1361cc0 2023-09-10T01:09:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce12c8840 con 0x55dce1361300 2023-09-10T01:09:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.419+0000 7f195feaeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce1362580 con 0x55dce13671a0 2023-09-10T01:09:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39718/0 (socket says 172.21.15.3:39718) 2023-09-10T01:09:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f194ab9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 -- 172.21.15.3:0/3436536205 learned_addr learned my addr 172.21.15.3:0/3436536205 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f194ab9c700 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce13671a0 0x55dce1363b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 -- 172.21.15.3:0/3436536205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce13671a0 msgr2=0x55dce1363b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce13671a0 0x55dce1363b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 -- 172.21.15.3:0/3436536205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 msgr2=0x55dce1372520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 -- 172.21.15.3:0/3436536205 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dce11061f0 con 0x55dce1361cc0 2023-09-10T01:09:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f194ab9c700 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce13671a0 0x55dce1363b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.420+0000 7f1949b9a700 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136d940 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1934009d20 tx=0x7f1934011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55009bc161e26839 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f19393ff700 1 -- 172.21.15.3:0/3436536205 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193402a300 con 0x55dce1361cc0 2023-09-10T01:09:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f19393ff700 1 -- 172.21.15.3:0/3436536205 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f193402a460 con 0x55dce1361cc0 2023-09-10T01:09:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f19393ff700 1 -- 172.21.15.3:0/3436536205 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19340054a0 con 0x55dce1361cc0 2023-09-10T01:09:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f195feaeb80 1 -- 172.21.15.3:0/3436536205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 msgr2=0x55dce136d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f195feaeb80 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136d940 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1934009d20 tx=0x7f1934011720 comp rx=0 tx=0).stop 2023-09-10T01:09:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f195feaeb80 1 -- 172.21.15.3:0/3436536205 shutdown_connections 2023-09-10T01:09:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f195feaeb80 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce13671a0 0x55dce1363b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f195feaeb80 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f195feaeb80 1 --2- 172.21.15.3:0/3436536205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce1361cc0 0x55dce136d940 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.421+0000 7f195feaeb80 1 -- 172.21.15.3:0/3436536205 >> 172.21.15.3:0/3436536205 conn(0x55dce12d6030 msgr2=0x55dce1365190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.422+0000 7f195feaeb80 1 -- 172.21.15.3:0/3436536205 shutdown_connections 2023-09-10T01:09:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.422+0000 7f195feaeb80 1 -- 172.21.15.3:0/3436536205 wait complete. 2023-09-10T01:09:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.422+0000 7f195feaeb80 1 -- start start 2023-09-10T01:09:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 1 Processor -- start 2023-09-10T01:09:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361cc0 0x55dce1369840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce13671a0 0x55dce1369e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce136a510 con 0x55dce13671a0 2023-09-10T01:09:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce136a680 con 0x55dce1361300 2023-09-10T01:09:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dce136a840 con 0x55dce1361cc0 2023-09-10T01:09:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.423+0000 7f195feaeb80 10 client.0 _ll_register_callbacks cb 0x7f193c008b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:44622/0 (socket says 172.21.15.3:44622) 2023-09-10T01:09:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 -- 172.21.15.3:0/2415241585 learned_addr learned my addr 172.21.15.3:0/2415241585 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce13671a0 0x55dce1369e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f1949b9a700 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361cc0 0x55dce1369840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 -- 172.21.15.3:0/2415241585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361cc0 msgr2=0x55dce1369840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361cc0 0x55dce1369840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 -- 172.21.15.3:0/2415241585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce13671a0 msgr2=0x55dce1369e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce13671a0 0x55dce1369e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 -- 172.21.15.3:0/2415241585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dce10dfd80 con 0x55dce1361300 2023-09-10T01:09:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f194ab9c700 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372720 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f193c008c80 tx=0x7f193c0094d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565af72702c5c573 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.424+0000 7f192dffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.425+0000 7f192dffb700 1 -- 172.21.15.3:0/2415241585 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c00e040 con 0x55dce1361300 2023-09-10T01:09:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.425+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1934019c50 con 0x55dce1361300 2023-09-10T01:09:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.425+0000 7f192dffb700 1 -- 172.21.15.3:0/2415241585 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f193c008830 con 0x55dce1361300 2023-09-10T01:09:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.425+0000 7f192dffb700 1 -- 172.21.15.3:0/2415241585 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f193c01c020 con 0x55dce1361300 2023-09-10T01:09:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.425+0000 7f195feaeb80 10 client.27317 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.425+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dce10de250 con 0x55dce1361300 2023-09-10T01:09:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f192dffb700 1 -- 172.21.15.3:0/2415241585 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f193c03f080 con 0x55dce1361300 2023-09-10T01:09:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f192dffb700 1 -- 172.21.15.3:0/2415241585 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f193c0050d0 con 0x55dce1361300 2023-09-10T01:09:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f195feaeb80 10 client.27317 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f195feaeb80 10 client.27317 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f195feaeb80 20 client.27317 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f195feaeb80 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce13d82b0 0x55dce13da6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:01.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f195feaeb80 4 client.27317 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:09:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dce12ac8c0 con 0x55dce13d82b0 2023-09-10T01:09:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.426+0000 7f1949b9a700 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce13d82b0 0x55dce13da6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.427+0000 7f1949b9a700 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce13d82b0 0x55dce13da6d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f26512d1ca02e72 server_cookie=40a5cdb288311602 in_seq=0 out_seq=0 2023-09-10T01:09:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.427+0000 7f192dffb700 10 client.27317 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:01.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f192dffb700 1 -- 172.21.15.3:0/2415241585 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dce12ac8c0 con 0x55dce13d82b0 2023-09-10T01:09:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f192dffb700 10 client.27317 handle_command_reply: tid=0 2023-09-10T01:09:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 client.27317 shutdown 2023-09-10T01:09:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 msgr2=0x55dce1372720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372720 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f193c008c80 tx=0x7f193c0094d0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 shutdown_connections 2023-09-10T01:09:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dce13d82b0 0x55dce13da6d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dce13671a0 0x55dce1369e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dce1361300 0x55dce1372720 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.559+0000 7f195feaeb80 1 --2- 172.21.15.3:0/2415241585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dce1361cc0 0x55dce1369840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.560+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 >> 172.21.15.3:0/2415241585 conn(0x55dce12d6030 msgr2=0x55dce1366110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.560+0000 7f192dffb700 0 client.27317 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.560+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 shutdown_connections 2023-09-10T01:09:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.560+0000 7f195feaeb80 1 -- 172.21.15.3:0/2415241585 wait complete. 2023-09-10T01:09:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.560+0000 7f195feaeb80 20 client.27317 trim_cache size 0 max 16384 2023-09-10T01:09:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:09:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:09:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:09:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:09:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f19300a0830 msgr2=0x7f19300a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f19300a0830 0x7f19300a2cf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f194009a100 tx=0x7f1940009cd0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19441005c0 msgr2=0x7f1944072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19441005c0 0x7f1944072900 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1940000c00 tx=0x7f194000bfd0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 shutdown_connections 2023-09-10T01:09:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f19300a0830 0x7f19300a2cf0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1944106680 0x7f19441a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19441005c0 0x7f1944072900 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 --2- 172.21.15.3:0/2400748438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1944100f80 0x7f1944072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 >> 172.21.15.3:0/2400748438 conn(0x7f19440fc0b0 msgr2=0x7f19440fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 shutdown_connections 2023-09-10T01:09:01.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:01.561+0000 7f194c5ff700 1 -- 172.21.15.3:0/2400748438 wait complete. 2023-09-10T01:09:01.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 501969 9.41 MB/sec execute 163 sec latency 196.519 ms 2023-09-10T01:09:02.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:02 smithi136 ceph-mon[133708]: pgmap v177: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 285 op/s 2023-09-10T01:09:02.467 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:02 smithi092 ceph-mon[134067]: pgmap v177: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 285 op/s 2023-09-10T01:09:02.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:02 smithi003 ceph-mon[97654]: pgmap v177: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 285 op/s 2023-09-10T01:09:02.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 502650 9.37 MB/sec execute 164 sec latency 114.376 ms 2023-09-10T01:09:03.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 504810 9.40 MB/sec execute 165 sec latency 29.574 ms 2023-09-10T01:09:04.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:04 smithi136 ceph-mon[133708]: pgmap v178: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 285 op/s 2023-09-10T01:09:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:04 smithi092 ceph-mon[134067]: pgmap v178: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 285 op/s 2023-09-10T01:09:04.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:04 smithi003 ceph-mon[97654]: pgmap v178: 129 pgs: 129 active+clean; 147 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 285 op/s 2023-09-10T01:09:04.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 506186 9.38 MB/sec execute 166 sec latency 23.693 ms 2023-09-10T01:09:05.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 508419 9.40 MB/sec execute 167 sec latency 14.290 ms 2023-09-10T01:09:06.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:06 smithi136 ceph-mon[133708]: pgmap v179: 129 pgs: 129 active+clean; 138 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.5 MiB/s wr, 509 op/s 2023-09-10T01:09:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:06 smithi092 ceph-mon[134067]: pgmap v179: 129 pgs: 129 active+clean; 138 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.5 MiB/s wr, 509 op/s 2023-09-10T01:09:06.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:06 smithi003 ceph-mon[97654]: pgmap v179: 129 pgs: 129 active+clean; 138 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.5 MiB/s wr, 509 op/s 2023-09-10T01:09:06.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 509950 9.38 MB/sec execute 168 sec latency 19.741 ms 2023-09-10T01:09:07.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:09:07.762 INFO:tasks.workunit.client.0.smithi003.stdout: 1 511900 9.40 MB/sec execute 169 sec latency 50.388 ms 2023-09-10T01:09:07.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 -- 172.21.15.3:0/2936622951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400106650 msgr2=0x7f4400106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 --2- 172.21.15.3:0/2936622951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400106650 0x7f4400106a30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009990 tx=0x7f43fc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 -- 172.21.15.3:0/2936622951 shutdown_connections 2023-09-10T01:09:07.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 --2- 172.21.15.3:0/2936622951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101000 0x7f440010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 --2- 172.21.15.3:0/2936622951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400106650 0x7f4400106a30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 --2- 172.21.15.3:0/2936622951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400100640 0x7f4400100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 -- 172.21.15.3:0/2936622951 >> 172.21.15.3:0/2936622951 conn(0x7f44000fc170 msgr2=0x7f44000fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:07.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.826+0000 7f44079ff700 1 -- 172.21.15.3:0/2936622951 shutdown_connections 2023-09-10T01:09:07.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.827+0000 7f44079ff700 1 -- 172.21.15.3:0/2936622951 wait complete. 2023-09-10T01:09:07.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 Processor -- start 2023-09-10T01:09:07.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 -- start start 2023-09-10T01:09:07.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400100640 0x7f440019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400101000 0x7f440019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400106650 0x7f4400196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400196be0 con 0x7f4400100640 2023-09-10T01:09:07.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400196d50 con 0x7f4400101000 2023-09-10T01:09:07.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f44079ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400196f10 con 0x7f4400106650 2023-09-10T01:09:07.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f4405f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400106650 0x7f4400196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f4405f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400106650 0x7f4400196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40496/0 (socket says 172.21.15.3:40496) 2023-09-10T01:09:07.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f4405f9c700 1 -- 172.21.15.3:0/3268582525 learned_addr learned my addr 172.21.15.3:0/3268582525 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:07.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.828+0000 7f440579b700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400100640 0x7f440019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4404f9a700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400101000 0x7f440019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4405f9c700 1 -- 172.21.15.3:0/3268582525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400101000 msgr2=0x7f440019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4405f9c700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400101000 0x7f440019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4405f9c700 1 -- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400100640 msgr2=0x7f440019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4405f9c700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400100640 0x7f440019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4405f9c700 1 -- 172.21.15.3:0/3268582525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0095f0 con 0x7f4400106650 2023-09-10T01:09:07.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4404f9a700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400101000 0x7f440019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:07.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f4405f9c700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400106650 0x7f4400196570 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f43f8007ee0 tx=0x7f43f800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f43f67fc700 1 -- 172.21.15.3:0/3268582525 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f800ddd0 con 0x7f4400106650 2023-09-10T01:09:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f43f67fc700 1 -- 172.21.15.3:0/3268582525 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43f800f040 con 0x7f4400106650 2023-09-10T01:09:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44001971f0 con 0x7f4400106650 2023-09-10T01:09:07.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.829+0000 7f43f67fc700 1 -- 172.21.15.3:0/3268582525 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f80048e0 con 0x7f4400106650 2023-09-10T01:09:07.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.830+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44001a34f0 con 0x7f4400106650 2023-09-10T01:09:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.833+0000 7f43f67fc700 1 -- 172.21.15.3:0/3268582525 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f43f8003a00 con 0x7f4400106650 2023-09-10T01:09:07.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.834+0000 7f43f67fc700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f43ec0a0830 0x7f43ec0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.834+0000 7f441b3efb80 1 Processor -- start 2023-09-10T01:09:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.834+0000 7f43f67fc700 1 -- 172.21.15.3:0/3268582525 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f43f80cbb90 con 0x7f4400106650 2023-09-10T01:09:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f440579b700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f43ec0a0830 0x7f43ec0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f441b3efb80 1 -- start start 2023-09-10T01:09:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e328710 0x555e5e328af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e337250 0x555e5e339640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f441b3efb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e281730 con 0x555e5e329030 2023-09-10T01:09:07.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f441b3efb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e27f050 con 0x555e5e328710 2023-09-10T01:09:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.835+0000 7f441b3efb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e27a940 con 0x555e5e337250 2023-09-10T01:09:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e328710 0x555e5e328af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e328710 0x555e5e328af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35494/0 (socket says 172.21.15.3:35494) 2023-09-10T01:09:07.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 -- 172.21.15.3:0/331899431 learned_addr learned my addr 172.21.15.3:0/331899431 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:07.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f4405f9c700 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e337250 0x555e5e339640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f4404f9a700 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f43ec0a0830 0x7f43ec0a2cf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00b2e0 tx=0x7f43fc005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:07.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 -- 172.21.15.3:0/331899431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e337250 msgr2=0x555e5e339640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e337250 0x555e5e339640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 -- 172.21.15.3:0/331899431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 msgr2=0x555e5e336d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 -- 172.21.15.3:0/331899431 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e5e0a6de0 con 0x555e5e328710 2023-09-10T01:09:07.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f440579b700 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e328710 0x555e5e328af0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f43fc015000 tx=0x7f43fc020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4206a12e7a9cc92 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:07.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f43eb7fe700 1 -- 172.21.15.3:0/331899431 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc004700 con 0x555e5e328710 2023-09-10T01:09:07.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.836+0000 7f43eb7fe700 1 -- 172.21.15.3:0/331899431 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43fc01e040 con 0x555e5e328710 2023-09-10T01:09:07.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f43eb7fe700 1 -- 172.21.15.3:0/331899431 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc00fa60 con 0x555e5e328710 2023-09-10T01:09:07.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 -- 172.21.15.3:0/331899431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e328710 msgr2=0x555e5e328af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e328710 0x555e5e328af0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f43fc015000 tx=0x7f43fc020000 comp rx=0 tx=0).stop 2023-09-10T01:09:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 -- 172.21.15.3:0/331899431 shutdown_connections 2023-09-10T01:09:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e337250 0x555e5e339640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e328710 0x555e5e328af0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 --2- 172.21.15.3:0/331899431 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 -- 172.21.15.3:0/331899431 >> 172.21.15.3:0/331899431 conn(0x555e5e29d090 msgr2=0x555e5e323bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 -- 172.21.15.3:0/331899431 shutdown_connections 2023-09-10T01:09:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.837+0000 7f441b3efb80 1 -- 172.21.15.3:0/331899431 wait complete. 2023-09-10T01:09:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.838+0000 7f441b3efb80 1 -- start start 2023-09-10T01:09:07.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.838+0000 7f441b3efb80 1 Processor -- start 2023-09-10T01:09:07.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e336c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e333e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e334440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f441b3efb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e218b70 con 0x555e5e329030 2023-09-10T01:09:07.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f441b3efb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e218870 con 0x555e5e337250 2023-09-10T01:09:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f441b3efb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e0fc920 con 0x555e5e328710 2023-09-10T01:09:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f441b3efb80 10 client.0 _ll_register_callbacks cb 0x555e5e162a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e336c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e336c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40506/0 (socket says 172.21.15.3:40506) 2023-09-10T01:09:07.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 -- 172.21.15.3:0/4264671798 learned_addr learned my addr 172.21.15.3:0/4264671798 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f4405f9c700 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e334440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f4404f9a700 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e333e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 -- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 msgr2=0x555e5e334440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e334440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 -- 172.21.15.3:0/4264671798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 msgr2=0x555e5e333e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e333e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.839+0000 7f440579b700 1 -- 172.21.15.3:0/4264671798 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440010de10 con 0x555e5e328710 2023-09-10T01:09:07.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f440579b700 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e336c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009350 tx=0x7f43fc00f8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eadc60c4a94063c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:07.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f43e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:09:07.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f43e8ff9700 1 -- 172.21.15.3:0/4264671798 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc00fae0 con 0x555e5e328710 2023-09-10T01:09:07.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e5e1d24f0 con 0x555e5e328710 2023-09-10T01:09:07.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f43e8ff9700 1 -- 172.21.15.3:0/4264671798 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f43fc005940 con 0x555e5e328710 2023-09-10T01:09:07.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f43e8ff9700 1 -- 172.21.15.3:0/4264671798 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43fc004910 con 0x555e5e328710 2023-09-10T01:09:07.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f441b3efb80 10 client.27244 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:07.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.840+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555e5e08fc60 con 0x555e5e328710 2023-09-10T01:09:07.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f43e8ff9700 1 -- 172.21.15.3:0/4264671798 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e5e1d24f0 con 0x555e5e328710 2023-09-10T01:09:07.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f43e8ff9700 1 -- 172.21.15.3:0/4264671798 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f43fc024bc0 con 0x555e5e328710 2023-09-10T01:09:07.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f441b3efb80 10 client.27244 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:07.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f441b3efb80 10 client.27244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:07.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f441b3efb80 20 client.27244 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:07.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e346780 0x555e5e348ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f441b3efb80 4 client.27244 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:07.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e5e28f8a0 con 0x555e5e346780 2023-09-10T01:09:07.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.841+0000 7f4404f9a700 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e346780 0x555e5e348ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.842+0000 7f4404f9a700 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e346780 0x555e5e348ba0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25ec3a30d8104db7 server_cookie=dfa14299a9b58465 in_seq=0 out_seq=0 2023-09-10T01:09:07.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.842+0000 7f43e8ff9700 10 client.27244 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:07.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f43e8ff9700 1 -- 172.21.15.3:0/4264671798 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555e5e28f8a0 con 0x555e5e346780 2023-09-10T01:09:07.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f43e8ff9700 10 client.27244 handle_command_reply: tid=0 2023-09-10T01:09:07.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 client.27244 shutdown 2023-09-10T01:09:07.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 msgr2=0x555e5e336c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e336c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009350 tx=0x7f43fc00f8b0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 shutdown_connections 2023-09-10T01:09:07.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e346780 0x555e5e348ba0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e336c20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e334440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f43e8ff9700 0 client.27244 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:07.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4264671798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e333e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 >> 172.21.15.3:0/4264671798 conn(0x555e5e29d090 msgr2=0x555e5e323780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:07.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 shutdown_connections 2023-09-10T01:09:07.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 1 -- 172.21.15.3:0/4264671798 wait complete. 2023-09-10T01:09:07.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.843+0000 7f441b3efb80 20 client.27244 trim_cache size 0 max 16384 2023-09-10T01:09:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.861+0000 7f441b3efb80 1 Processor -- start 2023-09-10T01:09:07.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.861+0000 7f441b3efb80 1 -- start start 2023-09-10T01:09:07.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e32f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f441b3efb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e334ac0 con 0x555e5e329030 2023-09-10T01:09:07.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f441b3efb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e33e2b0 con 0x555e5e337250 2023-09-10T01:09:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f441b3efb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e329660 con 0x555e5e328710 2023-09-10T01:09:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f4404f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40512/0 (socket says 172.21.15.3:40512) 2023-09-10T01:09:07.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 -- 172.21.15.3:0/4115862434 learned_addr learned my addr 172.21.15.3:0/4115862434 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f4405f9c700 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e32f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 -- 172.21.15.3:0/4115862434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 msgr2=0x555e5e32f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e32f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 -- 172.21.15.3:0/4115862434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 msgr2=0x555e5e336750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.862+0000 7f440579b700 1 -- 172.21.15.3:0/4115862434 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e5e0cd250 con 0x555e5e328710 2023-09-10T01:09:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.863+0000 7f4404f9a700 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:07.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.863+0000 7f440579b700 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dd40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00c010 tx=0x7f43fc005680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b95da3b17fd7f123 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.863+0000 7f43ebfff700 1 -- 172.21.15.3:0/4115862434 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc00fb20 con 0x555e5e328710 2023-09-10T01:09:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.863+0000 7f43ebfff700 1 -- 172.21.15.3:0/4115862434 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43fc004bd0 con 0x555e5e328710 2023-09-10T01:09:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.863+0000 7f43ebfff700 1 -- 172.21.15.3:0/4115862434 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc018c90 con 0x555e5e328710 2023-09-10T01:09:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 -- 172.21.15.3:0/4115862434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 msgr2=0x555e5e33dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dd40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00c010 tx=0x7f43fc005680 comp rx=0 tx=0).stop 2023-09-10T01:09:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 -- 172.21.15.3:0/4115862434 shutdown_connections 2023-09-10T01:09:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dd40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e337250 0x555e5e32f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 --2- 172.21.15.3:0/4115862434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e329030 0x555e5e336750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 -- 172.21.15.3:0/4115862434 >> 172.21.15.3:0/4115862434 conn(0x555e5e29d090 msgr2=0x555e5e342490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:07.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 -- 172.21.15.3:0/4115862434 shutdown_connections 2023-09-10T01:09:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 -- 172.21.15.3:0/4115862434 wait complete. 2023-09-10T01:09:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 -- start start 2023-09-10T01:09:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.864+0000 7f441b3efb80 1 Processor -- start 2023-09-10T01:09:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e329030 0x555e5e33bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f441b3efb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e337250 0x555e5e33c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f441b3efb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e33caf0 con 0x555e5e337250 2023-09-10T01:09:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f441b3efb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e33cc60 con 0x555e5e329030 2023-09-10T01:09:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f441b3efb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5e39a580 con 0x555e5e328710 2023-09-10T01:09:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f441b3efb80 10 client.0 _ll_register_callbacks cb 0x7f43f8007f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f4405f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e337250 0x555e5e33c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f4404f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e329030 0x555e5e33bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f4404f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e329030 0x555e5e33bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35524/0 (socket says 172.21.15.3:35524) 2023-09-10T01:09:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f4405f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e337250 0x555e5e33c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35372/0 (socket says 172.21.15.3:35372) 2023-09-10T01:09:07.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f4404f9a700 1 -- 172.21.15.3:0/2617619043 learned_addr learned my addr 172.21.15.3:0/2617619043 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:07.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f440579b700 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f4404f9a700 1 -- 172.21.15.3:0/2617619043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 msgr2=0x555e5e33dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.865+0000 7f4404f9a700 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f4404f9a700 1 -- 172.21.15.3:0/2617619043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e337250 msgr2=0x555e5e33c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:07.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f4404f9a700 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e337250 0x555e5e33c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f4404f9a700 1 -- 172.21.15.3:0/2617619043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f0009a30 con 0x555e5e329030 2023-09-10T01:09:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f4404f9a700 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e329030 0x555e5e33bee0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x555e5e31e280 tx=0x7f43f001dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef7331f92a0a644 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f43e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f43e8ff9700 1 -- 172.21.15.3:0/2617619043 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0018040 con 0x555e5e329030 2023-09-10T01:09:07.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43fc01e040 con 0x555e5e329030 2023-09-10T01:09:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f43e8ff9700 1 -- 172.21.15.3:0/2617619043 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f43f0004250 con 0x555e5e329030 2023-09-10T01:09:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f43e8ff9700 1 -- 172.21.15.3:0/2617619043 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43f00096d0 con 0x555e5e329030 2023-09-10T01:09:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f441b3efb80 10 client.27335 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.866+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555e5e08fc60 con 0x555e5e329030 2023-09-10T01:09:07.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f43e8ff9700 1 -- 172.21.15.3:0/2617619043 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43fc01e040 con 0x555e5e329030 2023-09-10T01:09:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f43e8ff9700 1 -- 172.21.15.3:0/2617619043 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f43f00038e0 con 0x555e5e329030 2023-09-10T01:09:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f441b3efb80 10 client.27335 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f441b3efb80 10 client.27335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f441b3efb80 20 client.27335 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f441b3efb80 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e39bd20 0x555e5e39e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:07.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f441b3efb80 4 client.27335 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:09:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555e5e273920 con 0x555e5e39bd20 2023-09-10T01:09:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f440579b700 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e39bd20 0x555e5e39e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f440579b700 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e39bd20 0x555e5e39e140 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2767ddd38a8a055b server_cookie=51172545c8c4f203 in_seq=0 out_seq=0 2023-09-10T01:09:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.867+0000 7f43e8ff9700 10 client.27335 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f43e8ff9700 1 -- 172.21.15.3:0/2617619043 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555e5e273920 con 0x555e5e39bd20 2023-09-10T01:09:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f43e8ff9700 10 client.27335 handle_command_reply: tid=0 2023-09-10T01:09:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 client.27335 shutdown 2023-09-10T01:09:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e329030 msgr2=0x555e5e33bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e329030 0x555e5e33bee0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x555e5e31e280 tx=0x7f43f001dd40 comp rx=0 tx=0).stop 2023-09-10T01:09:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 shutdown_connections 2023-09-10T01:09:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5e39bd20 0x555e5e39e140 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5e328710 0x555e5e33dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5e337250 0x555e5e33c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f43e8ff9700 0 client.27335 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:07.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 --2- 172.21.15.3:0/2617619043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5e329030 0x555e5e33bee0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 >> 172.21.15.3:0/2617619043 conn(0x555e5e29d090 msgr2=0x555e5e32f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 shutdown_connections 2023-09-10T01:09:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 1 -- 172.21.15.3:0/2617619043 wait complete. 2023-09-10T01:09:07.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.868+0000 7f441b3efb80 20 client.27335 trim_cache size 0 max 16384 2023-09-10T01:09:07.889 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:09:07.889 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:09:07.889 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:09:07.889 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:09:07.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f43ec0a0830 msgr2=0x7f43ec0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f43ec0a0830 0x7f43ec0a2cf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00b2e0 tx=0x7f43fc005fd0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400106650 msgr2=0x7f4400196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:07.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400106650 0x7f4400196570 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f43f8007ee0 tx=0x7f43f800d3b0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 shutdown_connections 2023-09-10T01:09:07.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f43ec0a0830 0x7f43ec0a2cf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4400106650 0x7f4400196570 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400100640 0x7f440019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 --2- 172.21.15.3:0/3268582525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4400101000 0x7f440019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:07.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 >> 172.21.15.3:0/3268582525 conn(0x7f44000fc170 msgr2=0x7f44000fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:07.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 shutdown_connections 2023-09-10T01:09:07.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:07.869+0000 7f44079ff700 1 -- 172.21.15.3:0/3268582525 wait complete. 2023-09-10T01:09:07.894 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:09:07.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:09:08.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.110+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/460673628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98106630 msgr2=0x7fbe98106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.110+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/460673628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98106630 0x7fbe98106a10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009990 tx=0x7fbe88009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.110+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/460673628 shutdown_connections 2023-09-10T01:09:08.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.110+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/460673628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe98100fe0 0x7fbe9810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.110+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/460673628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98106630 0x7fbe98106a10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.110+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/460673628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe98100620 0x7fbe98100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.110+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/460673628 >> 172.21.15.3:0/460673628 conn(0x7fbe980fc110 msgr2=0x7fbe980fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.111+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/460673628 shutdown_connections 2023-09-10T01:09:08.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.111+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/460673628 wait complete. 2023-09-10T01:09:08.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 Processor -- start 2023-09-10T01:09:08.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 -- start start 2023-09-10T01:09:08.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98100620 0x7fbe98196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe98100fe0 0x7fbe98196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe98106630 0x7fbe9819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9819a600 con 0x7fbe98106630 2023-09-10T01:09:08.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9819a770 con 0x7fbe98100620 2023-09-10T01:09:08.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe9e1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9819a930 con 0x7fbe98100fe0 2023-09-10T01:09:08.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe977fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98100620 0x7fbe98196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe977fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98100620 0x7fbe98196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35532/0 (socket says 172.21.15.3:35532) 2023-09-10T01:09:08.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.112+0000 7fbe977fe700 1 -- 172.21.15.3:0/620248464 learned_addr learned my addr 172.21.15.3:0/620248464 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe97fff700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe98106630 0x7fbe9819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe96ffd700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe98100fe0 0x7fbe98196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe977fe700 1 -- 172.21.15.3:0/620248464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe98100fe0 msgr2=0x7fbe98196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe977fe700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe98100fe0 0x7fbe98196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe977fe700 1 -- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe98106630 msgr2=0x7fbe9819a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe977fe700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe98106630 0x7fbe9819a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe977fe700 1 -- 172.21.15.3:0/620248464 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe880095f0 con 0x7fbe98100620 2023-09-10T01:09:08.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe977fe700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98100620 0x7fbe98196710 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbe88005910 tx=0x7fbe88004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe94ff9700 1 -- 172.21.15.3:0/620248464 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8801d070 con 0x7fbe98100620 2023-09-10T01:09:08.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe94ff9700 1 -- 172.21.15.3:0/620248464 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe88022470 con 0x7fbe98100620 2023-09-10T01:09:08.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe94ff9700 1 -- 172.21.15.3:0/620248464 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88017650 con 0x7fbe98100620 2023-09-10T01:09:08.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.113+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9819abb0 con 0x7fbe98100620 2023-09-10T01:09:08.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.114+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9819ae60 con 0x7fbe98100620 2023-09-10T01:09:08.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.115+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe98067480 con 0x7fbe98100620 2023-09-10T01:09:08.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.117+0000 7fbe94ff9700 1 -- 172.21.15.3:0/620248464 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fbe8800fad0 con 0x7fbe98100620 2023-09-10T01:09:08.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.118+0000 7fbe94ff9700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbe800a07b0 0x7fbe800a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.118+0000 7fbe94ff9700 1 -- 172.21.15.3:0/620248464 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fbe880cddd0 con 0x7fbe98100620 2023-09-10T01:09:08.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.118+0000 7fbe96ffd700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbe800a07b0 0x7fbe800a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.119+0000 7fbe96ffd700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbe800a07b0 0x7fbe800a2c70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe98197d20 tx=0x7fbe8c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.121+0000 7fbe94ff9700 1 -- 172.21.15.3:0/620248464 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fbe88097260 con 0x7fbe98100620 2023-09-10T01:09:08.308 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:08 smithi003 ceph-mon[97654]: pgmap v180: 129 pgs: 129 active+clean; 138 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 398 op/s 2023-09-10T01:09:08.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:08 smithi136 ceph-mon[133708]: pgmap v180: 129 pgs: 129 active+clean; 138 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 398 op/s 2023-09-10T01:09:08.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.415+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe98197a60 con 0x7fbe98100620 2023-09-10T01:09:08.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.416+0000 7fbe94ff9700 1 -- 172.21.15.3:0/620248464 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11292 (secure 0 0 0) 0x7fbe8809a880 con 0x7fbe98100620 2023-09-10T01:09:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:09:08.418 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:08:53.072858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:09:08.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbe800a07b0 msgr2=0x7fbe800a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbe800a07b0 0x7fbe800a2c70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe98197d20 tx=0x7fbe8c009260 comp rx=0 tx=0).stop 2023-09-10T01:09:08.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98100620 msgr2=0x7fbe98196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98100620 0x7fbe98196710 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbe88005910 tx=0x7fbe88004730 comp rx=0 tx=0).stop 2023-09-10T01:09:08.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 shutdown_connections 2023-09-10T01:09:08.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbe800a07b0 0x7fbe800a2c70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe98106630 0x7fbe9819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe98100620 0x7fbe98196710 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 --2- 172.21.15.3:0/620248464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe98100fe0 0x7fbe98196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 >> 172.21.15.3:0/620248464 conn(0x7fbe980fc110 msgr2=0x7fbe980fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 shutdown_connections 2023-09-10T01:09:08.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.419+0000 7fbe9e1ff700 1 -- 172.21.15.3:0/620248464 wait complete. 2023-09-10T01:09:08.423 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 27 2023-09-10T01:09:08.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:09:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:08 smithi092 ceph-mon[134067]: pgmap v180: 129 pgs: 129 active+clean; 138 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 7.1 MiB/s wr, 398 op/s 2023-09-10T01:09:08.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 -- 172.21.15.3:0/1699760249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70101000 msgr2=0x7fab7010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 --2- 172.21.15.3:0/1699760249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70101000 0x7fab7010d510 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fab6800b0a0 tx=0x7fab6800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 -- 172.21.15.3:0/1699760249 shutdown_connections 2023-09-10T01:09:08.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 --2- 172.21.15.3:0/1699760249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70101000 0x7fab7010d510 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 --2- 172.21.15.3:0/1699760249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70106650 0x7fab70106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 --2- 172.21.15.3:0/1699760249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab70100640 0x7fab70100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 -- 172.21.15.3:0/1699760249 >> 172.21.15.3:0/1699760249 conn(0x7fab700fc170 msgr2=0x7fab700fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 -- 172.21.15.3:0/1699760249 shutdown_connections 2023-09-10T01:09:08.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.657+0000 7fab753ff700 1 -- 172.21.15.3:0/1699760249 wait complete. 2023-09-10T01:09:08.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.658+0000 7fab753ff700 1 Processor -- start 2023-09-10T01:09:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.658+0000 7fab753ff700 1 -- start start 2023-09-10T01:09:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.658+0000 7fab753ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab70100640 0x7fab70072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.658+0000 7fab753ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70101000 0x7fab70072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab753ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70106650 0x7fab701a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab753ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70073670 con 0x7fab70101000 2023-09-10T01:09:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab753ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab701a4da0 con 0x7fab70106650 2023-09-10T01:09:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab753ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab701a4ee0 con 0x7fab70100640 2023-09-10T01:09:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab6e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70101000 0x7fab70072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab6effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab70100640 0x7fab70072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab6e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70101000 0x7fab70072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35400/0 (socket says 172.21.15.3:35400) 2023-09-10T01:09:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab6e7fc700 1 -- 172.21.15.3:0/3816262527 learned_addr learned my addr 172.21.15.3:0/3816262527 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.659+0000 7fab6f7fe700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70106650 0x7fab701a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab6effd700 1 -- 172.21.15.3:0/3816262527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70106650 msgr2=0x7fab701a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab6effd700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70106650 0x7fab701a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab6effd700 1 -- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70101000 msgr2=0x7fab70072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab6effd700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70101000 0x7fab70072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab6effd700 1 -- 172.21.15.3:0/3816262527 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68009d00 con 0x7fab70100640 2023-09-10T01:09:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab6effd700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab70100640 0x7fab70072900 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fab6000e9f0 tx=0x7fab6000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab57fff700 1 -- 172.21.15.3:0/3816262527 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6000c9c0 con 0x7fab70100640 2023-09-10T01:09:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab57fff700 1 -- 172.21.15.3:0/3816262527 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab60010430 con 0x7fab70100640 2023-09-10T01:09:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.660+0000 7fab57fff700 1 -- 172.21.15.3:0/3816262527 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60018680 con 0x7fab70100640 2023-09-10T01:09:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.661+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab701a5140 con 0x7fab70100640 2023-09-10T01:09:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.661+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab701a55d0 con 0x7fab70100640 2023-09-10T01:09:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.664+0000 7fab57fff700 1 -- 172.21.15.3:0/3816262527 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fab60010860 con 0x7fab70100640 2023-09-10T01:09:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.665+0000 7fab57fff700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fab580a0830 0x7fab580a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 Processor -- start 2023-09-10T01:09:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab57fff700 1 -- 172.21.15.3:0/3816262527 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fab60014070 con 0x7fab70100640 2023-09-10T01:09:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fab580a0830 0x7fab580a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 -- start start 2023-09-10T01:09:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d369710 0x562b1d369af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d36a030 0x562b1d373900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d376230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d2c2730 con 0x562b1d373e40 2023-09-10T01:09:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d2c0050 con 0x562b1d369710 2023-09-10T01:09:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab88db3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d2bb940 con 0x562b1d36a030 2023-09-10T01:09:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab6effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d369710 0x562b1d369af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab6effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d369710 0x562b1d369af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35546/0 (socket says 172.21.15.3:35546) 2023-09-10T01:09:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.666+0000 7fab6effd700 1 -- 172.21.15.3:0/3850155126 learned_addr learned my addr 172.21.15.3:0/3850155126 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d36a030 0x562b1d373900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6f7fe700 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d376230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6effd700 1 -- 172.21.15.3:0/3850155126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d36a030 msgr2=0x562b1d373900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6effd700 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d36a030 0x562b1d373900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6effd700 1 -- 172.21.15.3:0/3850155126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 msgr2=0x562b1d376230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6effd700 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d376230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6effd700 1 -- 172.21.15.3:0/3850155126 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b1d0e7de0 con 0x562b1d369710 2023-09-10T01:09:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fab580a0830 0x7fab580a2cf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab700740f0 tx=0x7fab64009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab6effd700 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d369710 0x562b1d369af0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fab60095f30 tx=0x7fab600990d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebd1c27b812b2e3a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab557fa700 1 -- 172.21.15.3:0/3850155126 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab600994c0 con 0x562b1d369710 2023-09-10T01:09:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab557fa700 1 -- 172.21.15.3:0/3850155126 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab6009d2f0 con 0x562b1d369710 2023-09-10T01:09:08.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.667+0000 7fab557fa700 1 -- 172.21.15.3:0/3850155126 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60099720 con 0x562b1d369710 2023-09-10T01:09:08.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 -- 172.21.15.3:0/3850155126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d369710 msgr2=0x562b1d369af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d369710 0x562b1d369af0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fab60095f30 tx=0x7fab600990d0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 -- 172.21.15.3:0/3850155126 shutdown_connections 2023-09-10T01:09:08.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d376230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d369710 0x562b1d369af0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 --2- 172.21.15.3:0/3850155126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d36a030 0x562b1d373900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 -- 172.21.15.3:0/3850155126 >> 172.21.15.3:0/3850155126 conn(0x562b1d2de090 msgr2=0x562b1d364bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 -- 172.21.15.3:0/3850155126 shutdown_connections 2023-09-10T01:09:08.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.668+0000 7fab88db3b80 1 -- 172.21.15.3:0/3850155126 wait complete. 2023-09-10T01:09:08.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 -- start start 2023-09-10T01:09:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 Processor -- start 2023-09-10T01:09:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d369710 0x562b1d370b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d3710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d373e40 0x562b1d3715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d259b70 con 0x562b1d369710 2023-09-10T01:09:08.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d259870 con 0x562b1d36a030 2023-09-10T01:09:08.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d13d920 con 0x562b1d373e40 2023-09-10T01:09:08.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.669+0000 7fab88db3b80 10 client.0 _ll_register_callbacks cb 0x562b1d1a3a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:08.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d3710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d369710 0x562b1d370b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d3710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35560/0 (socket says 172.21.15.3:35560) 2023-09-10T01:09:08.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2513479744 learned_addr learned my addr 172.21.15.3:0/2513479744 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6f7fe700 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d373e40 0x562b1d3715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d373e40 msgr2=0x562b1d3715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d373e40 0x562b1d3715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2513479744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d369710 msgr2=0x562b1d370b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d369710 0x562b1d370b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2513479744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab701a55d0 con 0x562b1d36a030 2023-09-10T01:09:08.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6effd700 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d369710 0x562b1d370b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:08.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d3710a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fab64009b50 tx=0x7fab640047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=582da0e0d68795a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.670+0000 7fab52ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:08.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab52ff5700 1 -- 172.21.15.3:0/2513479744 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6402a470 con 0x562b1d36a030 2023-09-10T01:09:08.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b1d2134f0 con 0x562b1d36a030 2023-09-10T01:09:08.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab52ff5700 1 -- 172.21.15.3:0/2513479744 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fab6402a5d0 con 0x562b1d36a030 2023-09-10T01:09:08.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab52ff5700 1 -- 172.21.15.3:0/2513479744 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab6402a790 con 0x562b1d36a030 2023-09-10T01:09:08.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab88db3b80 10 client.27353 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:08.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562b1d0d0c60 con 0x562b1d36a030 2023-09-10T01:09:08.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab52ff5700 1 -- 172.21.15.3:0/2513479744 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562b1d2134f0 con 0x562b1d36a030 2023-09-10T01:09:08.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.671+0000 7fab52ff5700 1 -- 172.21.15.3:0/2513479744 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fab64029920 con 0x562b1d36a030 2023-09-10T01:09:08.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab88db3b80 10 client.27353 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:08.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab88db3b80 10 client.27353 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:09:08.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab88db3b80 20 client.27353 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:08.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d383a00 0x562b1d385e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab88db3b80 4 client.27353 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:08.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b1d2d08a0 con 0x562b1d383a00 2023-09-10T01:09:08.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab6effd700 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d383a00 0x562b1d385e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab6effd700 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d383a00 0x562b1d385e20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa2a5233f85031c6 server_cookie=528d873a491d61b0 in_seq=0 out_seq=0 2023-09-10T01:09:08.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.672+0000 7fab52ff5700 10 client.27353 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:09:08.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab52ff5700 1 -- 172.21.15.3:0/2513479744 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562b1d2d08a0 con 0x562b1d383a00 2023-09-10T01:09:08.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab52ff5700 10 client.27353 handle_command_reply: tid=0 2023-09-10T01:09:08.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 client.27353 shutdown 2023-09-10T01:09:08.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 msgr2=0x562b1d3710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d3710a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fab64009b50 tx=0x7fab640047c0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 shutdown_connections 2023-09-10T01:09:08.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d383a00 0x562b1d385e20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d373e40 0x562b1d3715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d369710 0x562b1d370b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab52ff5700 0 client.27353 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:09:08.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2513479744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d3710a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.673+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 >> 172.21.15.3:0/2513479744 conn(0x562b1d2de090 msgr2=0x562b1d3647e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.674+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 shutdown_connections 2023-09-10T01:09:08.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.674+0000 7fab88db3b80 1 -- 172.21.15.3:0/2513479744 wait complete. 2023-09-10T01:09:08.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.674+0000 7fab88db3b80 20 client.27353 trim_cache size 0 max 16384 2023-09-10T01:09:08.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.688+0000 7fab88db3b80 1 Processor -- start 2023-09-10T01:09:08.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.688+0000 7fab88db3b80 1 -- start start 2023-09-10T01:09:08.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d371410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d36be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d36c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab88db3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d36a5e0 con 0x562b1d373e40 2023-09-10T01:09:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab88db3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d36b280 con 0x562b1d36a030 2023-09-10T01:09:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab88db3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d373bf0 con 0x562b1d369710 2023-09-10T01:09:08.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d36be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d36be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35574/0 (socket says 172.21.15.3:35574) 2023-09-10T01:09:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2241081706 learned_addr learned my addr 172.21.15.3:0/2241081706 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6effd700 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d371410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6f7fe700 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d36c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2241081706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 msgr2=0x562b1d371410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d371410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2241081706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 msgr2=0x562b1d36c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d36c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.689+0000 7fab6e7fc700 1 -- 172.21.15.3:0/2241081706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b1d10e250 con 0x562b1d36a030 2023-09-10T01:09:08.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.690+0000 7fab6e7fc700 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d36be40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fab64020fd0 tx=0x7fab640095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef769f8d6bf34f2b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.690+0000 7fab55ffb700 1 -- 172.21.15.3:0/2241081706 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400f980 con 0x562b1d36a030 2023-09-10T01:09:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.690+0000 7fab55ffb700 1 -- 172.21.15.3:0/2241081706 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab64004cf0 con 0x562b1d36a030 2023-09-10T01:09:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.690+0000 7fab55ffb700 1 -- 172.21.15.3:0/2241081706 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64005040 con 0x562b1d36a030 2023-09-10T01:09:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.690+0000 7fab88db3b80 1 -- 172.21.15.3:0/2241081706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 msgr2=0x562b1d36be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.690+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d36be40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fab64020fd0 tx=0x7fab640095c0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 -- 172.21.15.3:0/2241081706 shutdown_connections 2023-09-10T01:09:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d371410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d373e40 0x562b1d36c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 --2- 172.21.15.3:0/2241081706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d36a030 0x562b1d36be40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 -- 172.21.15.3:0/2241081706 >> 172.21.15.3:0/2241081706 conn(0x562b1d2de090 msgr2=0x562b1d3646f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 -- 172.21.15.3:0/2241081706 shutdown_connections 2023-09-10T01:09:08.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 -- 172.21.15.3:0/2241081706 wait complete. 2023-09-10T01:09:08.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 -- start start 2023-09-10T01:09:08.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.691+0000 7fab88db3b80 1 Processor -- start 2023-09-10T01:09:08.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d37a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d36a030 0x562b1d37aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab88db3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d373e40 0x562b1d378a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab88db3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d379150 con 0x562b1d36a030 2023-09-10T01:09:08.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab88db3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d3792c0 con 0x562b1d373e40 2023-09-10T01:09:08.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab88db3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1d379480 con 0x562b1d369710 2023-09-10T01:09:08.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab88db3b80 10 client.0 _ll_register_callbacks cb 0x7fab64020e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:08.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d373e40 0x562b1d378a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6e7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d36a030 0x562b1d37aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d373e40 0x562b1d378a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35590/0 (socket says 172.21.15.3:35590) 2023-09-10T01:09:08.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 -- 172.21.15.3:0/559867819 learned_addr learned my addr 172.21.15.3:0/559867819 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6effd700 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d37a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 -- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 msgr2=0x562b1d37a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d37a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 -- 172.21.15.3:0/559867819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d36a030 msgr2=0x562b1d37aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d36a030 0x562b1d37aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.692+0000 7fab6f7fe700 1 -- 172.21.15.3:0/559867819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b1d0d0c60 con 0x562b1d373e40 2023-09-10T01:09:08.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab6f7fe700 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d373e40 0x562b1d378a40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fab6800b0a0 tx=0x7fab68004c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d27ae671bb97b1b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab52ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab52ff5700 1 -- 172.21.15.3:0/559867819 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800e040 con 0x562b1d373e40 2023-09-10T01:09:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab6009d2f0 con 0x562b1d373e40 2023-09-10T01:09:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab52ff5700 1 -- 172.21.15.3:0/559867819 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fab680076e0 con 0x562b1d373e40 2023-09-10T01:09:08.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab52ff5700 1 -- 172.21.15.3:0/559867819 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab680078a0 con 0x562b1d373e40 2023-09-10T01:09:08.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab88db3b80 10 client.27359 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:08.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562b1d0e62b0 con 0x562b1d373e40 2023-09-10T01:09:08.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.693+0000 7fab52ff5700 1 -- 172.21.15.3:0/559867819 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab6009d2f0 con 0x562b1d373e40 2023-09-10T01:09:08.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab52ff5700 1 -- 172.21.15.3:0/559867819 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fab68023020 con 0x562b1d373e40 2023-09-10T01:09:08.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab88db3b80 10 client.27359 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:08.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab88db3b80 10 client.27359 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:09:08.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab88db3b80 20 client.27359 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:08.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab88db3b80 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d3d8b80 0x562b1d3dafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab88db3b80 4 client.27359 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:09:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562b1d2b4920 con 0x562b1d3d8b80 2023-09-10T01:09:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab6effd700 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d3d8b80 0x562b1d3dafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab52ff5700 10 client.27359 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:09:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.694+0000 7fab6effd700 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d3d8b80 0x562b1d3dafa0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86b175863e71e070 server_cookie=eb7463809e93484c in_seq=0 out_seq=0 2023-09-10T01:09:08.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab52ff5700 1 -- 172.21.15.3:0/559867819 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562b1d2b4920 con 0x562b1d3d8b80 2023-09-10T01:09:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab52ff5700 10 client.27359 handle_command_reply: tid=0 2023-09-10T01:09:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 client.27359 shutdown 2023-09-10T01:09:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d373e40 msgr2=0x562b1d378a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d373e40 0x562b1d378a40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fab6800b0a0 tx=0x7fab68004c40 comp rx=0 tx=0).stop 2023-09-10T01:09:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 shutdown_connections 2023-09-10T01:09:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562b1d3d8b80 0x562b1d3dafa0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562b1d369710 0x562b1d37a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562b1d373e40 0x562b1d378a40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 --2- 172.21.15.3:0/559867819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b1d36a030 0x562b1d37aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab52ff5700 0 client.27359 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:09:08.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 >> 172.21.15.3:0/559867819 conn(0x562b1d2de090 msgr2=0x562b1d364530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 shutdown_connections 2023-09-10T01:09:08.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 1 -- 172.21.15.3:0/559867819 wait complete. 2023-09-10T01:09:08.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.695+0000 7fab88db3b80 20 client.27359 trim_cache size 0 max 16384 2023-09-10T01:09:08.716 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:09:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fab580a0830 msgr2=0x7fab580a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fab580a0830 0x7fab580a2cf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab700740f0 tx=0x7fab64009260 comp rx=0 tx=0).stop 2023-09-10T01:09:08.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab70100640 msgr2=0x7fab70072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab70100640 0x7fab70072900 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fab6000e9f0 tx=0x7fab6000ed00 comp rx=0 tx=0).stop 2023-09-10T01:09:08.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 shutdown_connections 2023-09-10T01:09:08.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fab580a0830 0x7fab580a2cf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab70100640 0x7fab70072900 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab70106650 0x7fab701a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 --2- 172.21.15.3:0/3816262527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70101000 0x7fab70072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 >> 172.21.15.3:0/3816262527 conn(0x7fab700fc170 msgr2=0x7fab700fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 shutdown_connections 2023-09-10T01:09:08.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.696+0000 7fab753ff700 1 -- 172.21.15.3:0/3816262527 wait complete. 2023-09-10T01:09:08.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:09:08.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 513220 9.38 MB/sec execute 170 sec latency 33.638 ms 2023-09-10T01:09:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.935+0000 7f68993ff700 1 -- 172.21.15.3:0/3362946930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6894100fd0 msgr2=0x7f689410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.935+0000 7f68993ff700 1 --2- 172.21.15.3:0/3362946930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6894100fd0 0x7f689410d4e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f688c009f90 tx=0x7f688c00b330 comp rx=0 tx=0).stop 2023-09-10T01:09:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.935+0000 7f68993ff700 1 -- 172.21.15.3:0/3362946930 shutdown_connections 2023-09-10T01:09:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.935+0000 7f68993ff700 1 --2- 172.21.15.3:0/3362946930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6894100fd0 0x7f689410d4e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.935+0000 7f68993ff700 1 --2- 172.21.15.3:0/3362946930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68941066b0 0x7f6894106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.935+0000 7f68993ff700 1 --2- 172.21.15.3:0/3362946930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100610 0x7f6894100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.935+0000 7f68993ff700 1 -- 172.21.15.3:0/3362946930 >> 172.21.15.3:0/3362946930 conn(0x7f68940fc100 msgr2=0x7f68940fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.936+0000 7f68993ff700 1 -- 172.21.15.3:0/3362946930 shutdown_connections 2023-09-10T01:09:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.936+0000 7f68993ff700 1 -- 172.21.15.3:0/3362946930 wait complete. 2023-09-10T01:09:08.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.936+0000 7f68993ff700 1 Processor -- start 2023-09-10T01:09:08.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68993ff700 1 -- start start 2023-09-10T01:09:08.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68993ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6894100610 0x7f6894072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68993ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100fd0 0x7f6894072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68993ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68941066b0 0x7f6894073380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68993ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941a9000 con 0x7f6894100610 2023-09-10T01:09:08.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68993ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941a9140 con 0x7f6894100fd0 2023-09-10T01:09:08.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68993ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941a9280 con 0x7f68941066b0 2023-09-10T01:09:08.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f6892ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6894100610 0x7f6894072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100fd0 0x7f6894072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100fd0 0x7f6894072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35604/0 (socket says 172.21.15.3:35604) 2023-09-10T01:09:08.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68927fc700 1 -- 172.21.15.3:0/2431302082 learned_addr learned my addr 172.21.15.3:0/2431302082 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.937+0000 7f68937fe700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68941066b0 0x7f6894073380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f68927fc700 1 -- 172.21.15.3:0/2431302082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68941066b0 msgr2=0x7f6894073380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f68927fc700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68941066b0 0x7f6894073380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f68927fc700 1 -- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6894100610 msgr2=0x7f6894072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f68927fc700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6894100610 0x7f6894072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f68927fc700 1 -- 172.21.15.3:0/2431302082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c009c40 con 0x7f6894100fd0 2023-09-10T01:09:08.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f68927fc700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100fd0 0x7f6894072e40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f688800e9c0 tx=0x7f688800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f687bfff700 1 -- 172.21.15.3:0/2431302082 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888004a30 con 0x7f6894100fd0 2023-09-10T01:09:08.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f687bfff700 1 -- 172.21.15.3:0/2431302082 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f688800cc80 con 0x7f6894100fd0 2023-09-10T01:09:08.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68941a9560 con 0x7f6894100fd0 2023-09-10T01:09:08.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.938+0000 7f687bfff700 1 -- 172.21.15.3:0/2431302082 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68880054e0 con 0x7f6894100fd0 2023-09-10T01:09:08.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.939+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68941a9a70 con 0x7f6894100fd0 2023-09-10T01:09:08.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.942+0000 7f687bfff700 1 -- 172.21.15.3:0/2431302082 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6888004b90 con 0x7f6894100fd0 2023-09-10T01:09:08.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.943+0000 7f68ace1cb80 1 Processor -- start 2023-09-10T01:09:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.943+0000 7f68ace1cb80 1 -- start start 2023-09-10T01:09:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f687bfff700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f687c0a07b0 0x7f687c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56191831d730 con 0x5619183c4710 2023-09-10T01:09:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56191831b050 con 0x5619183cee40 2023-09-10T01:09:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561918316940 con 0x5619183c5030 2023-09-10T01:09:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f687bfff700 1 -- 172.21.15.3:0/2431302082 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6888014070 con 0x7f6894100fd0 2023-09-10T01:09:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68937fe700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f687c0a07b0 0x7f687c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f6892ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68927fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f6892ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35462/0 (socket says 172.21.15.3:35462) 2023-09-10T01:09:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f68937fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.944+0000 7f6892ffd700 1 -- 172.21.15.3:0/1021787675 learned_addr learned my addr 172.21.15.3:0/1021787675 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68937fe700 1 -- 172.21.15.3:0/1021787675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 msgr2=0x5619183ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68937fe700 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68937fe700 1 -- 172.21.15.3:0/1021787675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 msgr2=0x5619183c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68937fe700 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68937fe700 1 -- 172.21.15.3:0/1021787675 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561918142de0 con 0x5619183cee40 2023-09-10T01:09:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68927fc700 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68937fe700 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183d1230 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f688c00b6c0 tx=0x7f688c004e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7bf9908a8b5a7de server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.945+0000 7f68937fe700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f687c0a07b0 0x7f687c0a2c70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f688c009f90 tx=0x7f688c01c000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68797fa700 1 -- 172.21.15.3:0/1021787675 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c02e070 con 0x5619183cee40 2023-09-10T01:09:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68797fa700 1 -- 172.21.15.3:0/1021787675 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f688c004bb0 con 0x5619183cee40 2023-09-10T01:09:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68797fa700 1 -- 172.21.15.3:0/1021787675 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c026070 con 0x5619183cee40 2023-09-10T01:09:08.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 -- 172.21.15.3:0/1021787675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 msgr2=0x5619183d1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183d1230 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f688c00b6c0 tx=0x7f688c004e20 comp rx=0 tx=0).stop 2023-09-10T01:09:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 -- 172.21.15.3:0/1021787675 shutdown_connections 2023-09-10T01:09:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183d1230 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/1021787675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 -- 172.21.15.3:0/1021787675 >> 172.21.15.3:0/1021787675 conn(0x561918339090 msgr2=0x5619183bfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 -- 172.21.15.3:0/1021787675 shutdown_connections 2023-09-10T01:09:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.946+0000 7f68ace1cb80 1 -- 172.21.15.3:0/1021787675 wait complete. 2023-09-10T01:09:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.947+0000 7f68ace1cb80 1 -- start start 2023-09-10T01:09:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.947+0000 7f68ace1cb80 1 Processor -- start 2023-09-10T01:09:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183cc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619182b4b70 con 0x5619183cee40 2023-09-10T01:09:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619182b4870 con 0x5619183c5030 2023-09-10T01:09:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561918198920 con 0x5619183c4710 2023-09-10T01:09:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68ace1cb80 10 client.0 _ll_register_callbacks cb 0x5619181fea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35630/0 (socket says 172.21.15.3:35630) 2023-09-10T01:09:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 -- 172.21.15.3:0/3399125162 learned_addr learned my addr 172.21.15.3:0/3399125162 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68937fe700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f6892ffd700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 -- 172.21.15.3:0/3399125162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 msgr2=0x5619183ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 -- 172.21.15.3:0/3399125162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 msgr2=0x5619183cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f68927fc700 1 -- 172.21.15.3:0/3399125162 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68941a9a70 con 0x5619183c5030 2023-09-10T01:09:08.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.948+0000 7f6892ffd700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:08.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f68927fc700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183cbb10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f68880994b0 tx=0x7f6888098f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=637946cdd04b4e2e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f6876ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:08.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f6876ff5700 1 -- 172.21.15.3:0/3399125162 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888095040 con 0x5619183c5030 2023-09-10T01:09:08.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56191826e4f0 con 0x5619183c5030 2023-09-10T01:09:08.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f6876ff5700 1 -- 172.21.15.3:0/3399125162 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f68880ace40 con 0x5619183c5030 2023-09-10T01:09:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f6876ff5700 1 -- 172.21.15.3:0/3399125162 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f688809d160 con 0x5619183c5030 2023-09-10T01:09:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f68ace1cb80 10 client.27371 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f689410dde0 con 0x5619183c5030 2023-09-10T01:09:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f6876ff5700 1 -- 172.21.15.3:0/3399125162 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56191826e4f0 con 0x5619183c5030 2023-09-10T01:09:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f6876ff5700 1 -- 172.21.15.3:0/3399125162 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6888095c30 con 0x5619183c5030 2023-09-10T01:09:08.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f68ace1cb80 10 client.27371 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:08.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f68ace1cb80 10 client.27371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:09:08.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.949+0000 7f68ace1cb80 20 client.27371 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:08.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.950+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5619183de930 0x5619183e0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.950+0000 7f68ace1cb80 4 client.27371 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:08.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.950+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56191832b8a0 con 0x5619183de930 2023-09-10T01:09:08.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.950+0000 7f6892ffd700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5619183de930 0x5619183e0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.950+0000 7f6892ffd700 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5619183de930 0x5619183e0d50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19cdcdee45deee90 server_cookie=bad1436b96008e0a in_seq=0 out_seq=0 2023-09-10T01:09:08.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.950+0000 7f6876ff5700 10 client.27371 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:09:08.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f6876ff5700 1 -- 172.21.15.3:0/3399125162 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56191832b8a0 con 0x5619183de930 2023-09-10T01:09:08.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f6876ff5700 10 client.27371 handle_command_reply: tid=0 2023-09-10T01:09:08.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 client.27371 shutdown 2023-09-10T01:09:08.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 msgr2=0x5619183cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183cbb10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f68880994b0 tx=0x7f6888098f30 comp rx=0 tx=0).stop 2023-09-10T01:09:08.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 shutdown_connections 2023-09-10T01:09:08.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5619183de930 0x5619183e0d50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f6876ff5700 0 client.27371 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:09:08.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/3399125162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183cbb10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 >> 172.21.15.3:0/3399125162 conn(0x561918339090 msgr2=0x5619183bf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 shutdown_connections 2023-09-10T01:09:08.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.951+0000 7f68ace1cb80 1 -- 172.21.15.3:0/3399125162 wait complete. 2023-09-10T01:09:08.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.952+0000 7f68ace1cb80 20 client.27371 trim_cache size 0 max 16384 2023-09-10T01:09:08.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.966+0000 7f68ace1cb80 1 Processor -- start 2023-09-10T01:09:08.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.967+0000 7f68ace1cb80 1 -- start start 2023-09-10T01:09:08.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.967+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183d5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.967+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.967+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183da0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.967+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619183de750 con 0x5619183c4710 2023-09-10T01:09:08.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.967+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619183ccac0 con 0x5619183cee40 2023-09-10T01:09:08.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.967+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619183db350 con 0x5619183c5030 2023-09-10T01:09:08.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183bedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40620/0 (socket says 172.21.15.3:40620) 2023-09-10T01:09:08.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 -- 172.21.15.3:0/4182092445 learned_addr learned my addr 172.21.15.3:0/4182092445 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f6892ffd700 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183d5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68937fe700 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183da0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 -- 172.21.15.3:0/4182092445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 msgr2=0x5619183da0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183da0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 -- 172.21.15.3:0/4182092445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 msgr2=0x5619183d5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183d5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 -- 172.21.15.3:0/4182092445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56191812bc60 con 0x5619183c5030 2023-09-10T01:09:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.968+0000 7f68927fc700 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183bedc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f688800cfd0 tx=0x7f6888095970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=facfc318d0e803a2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.969+0000 7f6879ffb700 1 -- 172.21.15.3:0/4182092445 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688809d160 con 0x5619183c5030 2023-09-10T01:09:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.969+0000 7f6879ffb700 1 -- 172.21.15.3:0/4182092445 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6888095030 con 0x5619183c5030 2023-09-10T01:09:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.969+0000 7f6879ffb700 1 -- 172.21.15.3:0/4182092445 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888016540 con 0x5619183c5030 2023-09-10T01:09:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 -- 172.21.15.3:0/4182092445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 msgr2=0x5619183bedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183bedc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f688800cfd0 tx=0x7f6888095970 comp rx=0 tx=0).stop 2023-09-10T01:09:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 -- 172.21.15.3:0/4182092445 shutdown_connections 2023-09-10T01:09:08.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183cee40 0x5619183da0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183c4710 0x5619183d5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/4182092445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c5030 0x5619183bedc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 -- 172.21.15.3:0/4182092445 >> 172.21.15.3:0/4182092445 conn(0x561918339090 msgr2=0x5619183dac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 -- 172.21.15.3:0/4182092445 shutdown_connections 2023-09-10T01:09:08.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 -- 172.21.15.3:0/4182092445 wait complete. 2023-09-10T01:09:08.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.970+0000 7f68ace1cb80 1 -- start start 2023-09-10T01:09:08.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.971+0000 7f68ace1cb80 1 Processor -- start 2023-09-10T01:09:08.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.971+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183d5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183d5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68ace1cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619183d4240 con 0x5619183cee40 2023-09-10T01:09:08.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619183d43b0 con 0x5619183c5030 2023-09-10T01:09:08.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68ace1cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619183d4570 con 0x5619183c4710 2023-09-10T01:09:08.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68ace1cb80 10 client.0 _ll_register_callbacks cb 0x7f68880992f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:08.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183d5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183d5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35648/0 (socket says 172.21.15.3:35648) 2023-09-10T01:09:08.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 -- 172.21.15.3:0/90152937 learned_addr learned my addr 172.21.15.3:0/90152937 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:08.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68937fe700 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 -- 172.21.15.3:0/90152937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 msgr2=0x5619183d5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183d5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 -- 172.21.15.3:0/90152937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 msgr2=0x5619183d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:08.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.972+0000 7f68927fc700 1 -- 172.21.15.3:0/90152937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689410dde0 con 0x5619183c5030 2023-09-10T01:09:08.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f68927fc700 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183d5990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6888000c00 tx=0x7f6888099110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be1a49ec6770340d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:08.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f6876ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:08.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f6876ff5700 1 -- 172.21.15.3:0/90152937 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68880652b0 con 0x5619183c5030 2023-09-10T01:09:08.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f6876ff5700 1 -- 172.21.15.3:0/90152937 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f68880ace40 con 0x5619183c5030 2023-09-10T01:09:08.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f688c004bb0 con 0x5619183c5030 2023-09-10T01:09:08.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f6876ff5700 1 -- 172.21.15.3:0/90152937 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68880c9c50 con 0x5619183c5030 2023-09-10T01:09:08.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f68ace1cb80 10 client.27377 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:08.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.973+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561918169250 con 0x5619183c5030 2023-09-10T01:09:08.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f6876ff5700 1 -- 172.21.15.3:0/90152937 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f688c004bb0 con 0x5619183c5030 2023-09-10T01:09:08.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f6876ff5700 1 -- 172.21.15.3:0/90152937 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6888099780 con 0x5619183c5030 2023-09-10T01:09:08.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f68ace1cb80 10 client.27377 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:08.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f68ace1cb80 10 client.27377 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:09:08.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f68ace1cb80 20 client.27377 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:08.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x561918433590 0x5619184359b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:08.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f68ace1cb80 4 client.27377 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:09:08.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56191830f920 con 0x561918433590 2023-09-10T01:09:08.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f6892ffd700 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x561918433590 0x5619184359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:08.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f6892ffd700 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x561918433590 0x5619184359b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e766cf7d7fe8f246 server_cookie=36ea6ff7d7110151 in_seq=0 out_seq=0 2023-09-10T01:09:08.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.974+0000 7f6876ff5700 10 client.27377 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:09:08.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f6876ff5700 1 -- 172.21.15.3:0/90152937 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56191830f920 con 0x561918433590 2023-09-10T01:09:08.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f6876ff5700 10 client.27377 handle_command_reply: tid=0 2023-09-10T01:09:08.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 client.27377 shutdown 2023-09-10T01:09:08.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 msgr2=0x5619183d5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183d5990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6888000c00 tx=0x7f6888099110 comp rx=0 tx=0).stop 2023-09-10T01:09:08.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 shutdown_connections 2023-09-10T01:09:08.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x561918433590 0x5619184359b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619183c4710 0x5619183d5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5619183cee40 0x5619183d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 --2- 172.21.15.3:0/90152937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619183c5030 0x5619183d5990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f6876ff5700 0 client.27377 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:09:08.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 >> 172.21.15.3:0/90152937 conn(0x561918339090 msgr2=0x5619183dac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 shutdown_connections 2023-09-10T01:09:08.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 1 -- 172.21.15.3:0/90152937 wait complete. 2023-09-10T01:09:08.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.975+0000 7f68ace1cb80 20 client.27377 trim_cache size 0 max 16384 2023-09-10T01:09:08.991 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:09:08.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f687c0a07b0 msgr2=0x7f687c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f687c0a07b0 0x7f687c0a2c70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f688c009f90 tx=0x7f688c01c000 comp rx=0 tx=0).stop 2023-09-10T01:09:08.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100fd0 msgr2=0x7f6894072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:08.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100fd0 0x7f6894072e40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f688800e9c0 tx=0x7f688800ecd0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 shutdown_connections 2023-09-10T01:09:08.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f687c0a07b0 0x7f687c0a2c70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68941066b0 0x7f6894073380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6894100610 0x7f6894072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 --2- 172.21.15.3:0/2431302082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6894100fd0 0x7f6894072e40 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:08.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 >> 172.21.15.3:0/2431302082 conn(0x7f68940fc100 msgr2=0x7f68940fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:08.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 shutdown_connections 2023-09-10T01:09:08.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:08.976+0000 7f68993ff700 1 -- 172.21.15.3:0/2431302082 wait complete. 2023-09-10T01:09:08.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:09:09.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.252+0000 7f6f33dff700 1 -- 172.21.15.3:0/1432547474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c102200 msgr2=0x7f6f2c10a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.252+0000 7f6f33dff700 1 --2- 172.21.15.3:0/1432547474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c102200 0x7f6f2c10a6e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009990 tx=0x7f6f1c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.253+0000 7f6f33dff700 1 -- 172.21.15.3:0/1432547474 shutdown_connections 2023-09-10T01:09:09.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.253+0000 7f6f33dff700 1 --2- 172.21.15.3:0/1432547474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f2c1018e0 0x7f6f2c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.253+0000 7f6f33dff700 1 --2- 172.21.15.3:0/1432547474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c10acb0 0x7f6f2c10d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.253+0000 7f6f33dff700 1 --2- 172.21.15.3:0/1432547474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c102200 0x7f6f2c10a6e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.253+0000 7f6f33dff700 1 -- 172.21.15.3:0/1432547474 >> 172.21.15.3:0/1432547474 conn(0x7f6f2c076710 msgr2=0x7f6f2c076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:09.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.253+0000 7f6f33dff700 1 -- 172.21.15.3:0/1432547474 shutdown_connections 2023-09-10T01:09:09.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.253+0000 7f6f33dff700 1 -- 172.21.15.3:0/1432547474 wait complete. 2023-09-10T01:09:09.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.254+0000 7f6f33dff700 1 Processor -- start 2023-09-10T01:09:09.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.254+0000 7f6f33dff700 1 -- start start 2023-09-10T01:09:09.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f33dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c1018e0 0x7f6f2c19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f33dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f2c102200 0x7f6f2c19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f33dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c10acb0 0x7f6f2c1965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f33dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c196c50 con 0x7f6f2c10acb0 2023-09-10T01:09:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f33dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c196dc0 con 0x7f6f2c1018e0 2023-09-10T01:09:09.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f33dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c196f80 con 0x7f6f2c102200 2023-09-10T01:09:09.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f31b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c1018e0 0x7f6f2c19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f31b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c1018e0 0x7f6f2c19c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35662/0 (socket says 172.21.15.3:35662) 2023-09-10T01:09:09.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f31b9b700 1 -- 172.21.15.3:0/825997417 learned_addr learned my addr 172.21.15.3:0/825997417 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:09.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f3139a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f2c102200 0x7f6f2c19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.255+0000 7f6f3239c700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c10acb0 0x7f6f2c1965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f31b9b700 1 -- 172.21.15.3:0/825997417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f2c102200 msgr2=0x7f6f2c19cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f31b9b700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f2c102200 0x7f6f2c19cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f31b9b700 1 -- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c10acb0 msgr2=0x7f6f2c1965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f31b9b700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c10acb0 0x7f6f2c1965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f31b9b700 1 -- 172.21.15.3:0/825997417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28009520 con 0x7f6f2c1018e0 2023-09-10T01:09:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f3239c700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c10acb0 0x7f6f2c1965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f31b9b700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c1018e0 0x7f6f2c19c570 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6f2800eb30 tx=0x7f6f2800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f22ffd700 1 -- 172.21.15.3:0/825997417 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f28009880 con 0x7f6f2c1018e0 2023-09-10T01:09:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f22ffd700 1 -- 172.21.15.3:0/825997417 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f280044e0 con 0x7f6f2c1018e0 2023-09-10T01:09:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f22ffd700 1 -- 172.21.15.3:0/825997417 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f28003e90 con 0x7f6f2c1018e0 2023-09-10T01:09:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f24009c40 con 0x7f6f2c1018e0 2023-09-10T01:09:09.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.256+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1c0095f0 con 0x7f6f2c1018e0 2023-09-10T01:09:09.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.261+0000 7f6f22ffd700 1 -- 172.21.15.3:0/825997417 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6f28004000 con 0x7f6f2c1018e0 2023-09-10T01:09:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.262+0000 7f6f47634b80 1 Processor -- start 2023-09-10T01:09:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.262+0000 7f6f47634b80 1 -- start start 2023-09-10T01:09:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca613eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca613f030 0x563ca614cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca614d250 0x563ca614f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f47634b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca6097730 con 0x563ca613f030 2023-09-10T01:09:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f47634b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca6095050 con 0x563ca613e710 2023-09-10T01:09:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f47634b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca6090940 con 0x563ca614d250 2023-09-10T01:09:09.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca613eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca613eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35674/0 (socket says 172.21.15.3:35674) 2023-09-10T01:09:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 -- 172.21.15.3:0/3793633769 learned_addr learned my addr 172.21.15.3:0/3793633769 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3139a700 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca614d250 0x563ca614f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f31b9b700 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca613f030 0x563ca614cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 -- 172.21.15.3:0/3793633769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca614d250 msgr2=0x563ca614f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca614d250 0x563ca614f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 -- 172.21.15.3:0/3793633769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca613f030 msgr2=0x563ca614cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca613f030 0x563ca614cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3239c700 1 -- 172.21.15.3:0/3793633769 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ca5ebcde0 con 0x563ca613e710 2023-09-10T01:09:09.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.263+0000 7f6f3139a700 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca614d250 0x563ca614f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.264+0000 7f6f3239c700 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca613eaf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6f24007550 tx=0x7f6f24009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90aa0df6d64fce55 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.264+0000 7f6f22ffd700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6f180a0830 0x7f6f180a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.264+0000 7f6f17fff700 1 -- 172.21.15.3:0/3793633769 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400e070 con 0x563ca613e710 2023-09-10T01:09:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.264+0000 7f6f17fff700 1 -- 172.21.15.3:0/3793633769 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f2400f040 con 0x563ca613e710 2023-09-10T01:09:09.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.264+0000 7f6f17fff700 1 -- 172.21.15.3:0/3793633769 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f24014930 con 0x563ca613e710 2023-09-10T01:09:09.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.264+0000 7f6f22ffd700 1 -- 172.21.15.3:0/825997417 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6f28014070 con 0x7f6f2c1018e0 2023-09-10T01:09:09.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.264+0000 7f6f31b9b700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6f180a0830 0x7f6f180a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 -- 172.21.15.3:0/3793633769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 msgr2=0x563ca613eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f31b9b700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6f180a0830 0x7f6f180a2cf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x563ca6134280 tx=0x7f6f28003930 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:09.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca613eaf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6f24007550 tx=0x7f6f24009920 comp rx=0 tx=0).stop 2023-09-10T01:09:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 -- 172.21.15.3:0/3793633769 shutdown_connections 2023-09-10T01:09:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca614d250 0x563ca614f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca613eaf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3793633769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca613f030 0x563ca614cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 -- 172.21.15.3:0/3793633769 >> 172.21.15.3:0/3793633769 conn(0x563ca60b3090 msgr2=0x563ca6139bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:09.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 -- 172.21.15.3:0/3793633769 shutdown_connections 2023-09-10T01:09:09.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.265+0000 7f6f47634b80 1 -- 172.21.15.3:0/3793633769 wait complete. 2023-09-10T01:09:09.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 -- start start 2023-09-10T01:09:09.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 Processor -- start 2023-09-10T01:09:09.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca614cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca602eb70 con 0x563ca614d250 2023-09-10T01:09:09.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca602e870 con 0x563ca613f030 2023-09-10T01:09:09.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.266+0000 7f6f47634b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca5f12920 con 0x563ca613e710 2023-09-10T01:09:09.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f47634b80 10 client.0 _ll_register_callbacks cb 0x563ca5f78a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:09.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca614cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35682/0 (socket says 172.21.15.3:35682) 2023-09-10T01:09:09.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 -- 172.21.15.3:0/2251394568 learned_addr learned my addr 172.21.15.3:0/2251394568 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:09.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3239c700 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 -- 172.21.15.3:0/2251394568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 msgr2=0x563ca614cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca614cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 -- 172.21.15.3:0/2251394568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 msgr2=0x563ca614a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.267+0000 7f6f3139a700 1 -- 172.21.15.3:0/2251394568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f24009280 con 0x563ca613f030 2023-09-10T01:09:09.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f3139a700 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149dc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c015000 tx=0x7f6f1c00bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a221fb1c41549d97 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:09.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f157fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:09.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f157fa700 1 -- 172.21.15.3:0/2251394568 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c01d070 con 0x563ca613f030 2023-09-10T01:09:09.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f157fa700 1 -- 172.21.15.3:0/2251394568 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6f1c004700 con 0x563ca613f030 2023-09-10T01:09:09.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ca5fe84f0 con 0x563ca613f030 2023-09-10T01:09:09.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f157fa700 1 -- 172.21.15.3:0/2251394568 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f1c0056e0 con 0x563ca613f030 2023-09-10T01:09:09.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f47634b80 10 client.27389 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:09.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563ca5ea5c60 con 0x563ca613f030 2023-09-10T01:09:09.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.268+0000 7f6f157fa700 1 -- 172.21.15.3:0/2251394568 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563ca5fe84f0 con 0x563ca613f030 2023-09-10T01:09:09.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f157fa700 1 -- 172.21.15.3:0/2251394568 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6f1c0247c0 con 0x563ca613f030 2023-09-10T01:09:09.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f47634b80 10 client.27389 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:09.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f47634b80 10 client.27389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:09.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f47634b80 20 client.27389 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:09.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca615cdb0 0x563ca615f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f47634b80 4 client.27389 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:09.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ca60a58a0 con 0x563ca615cdb0 2023-09-10T01:09:09.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f3239c700 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca615cdb0 0x563ca615f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f3239c700 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca615cdb0 0x563ca615f1d0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=631f3fb224a22d5f server_cookie=2d68b29533a68ff2 in_seq=0 out_seq=0 2023-09-10T01:09:09.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.269+0000 7f6f157fa700 10 client.27389 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:09.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f157fa700 1 -- 172.21.15.3:0/2251394568 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563ca60a58a0 con 0x563ca615cdb0 2023-09-10T01:09:09.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f157fa700 10 client.27389 handle_command_reply: tid=0 2023-09-10T01:09:09.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 client.27389 shutdown 2023-09-10T01:09:09.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 msgr2=0x563ca6149dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149dc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c015000 tx=0x7f6f1c00bf60 comp rx=0 tx=0).stop 2023-09-10T01:09:09.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 shutdown_connections 2023-09-10T01:09:09.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca615cdb0 0x563ca615f1d0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca614cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f157fa700 0 client.27389 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:09.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2251394568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149dc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 >> 172.21.15.3:0/2251394568 conn(0x563ca60b3090 msgr2=0x563ca61397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:09.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 shutdown_connections 2023-09-10T01:09:09.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.270+0000 7f6f47634b80 1 -- 172.21.15.3:0/2251394568 wait complete. 2023-09-10T01:09:09.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.271+0000 7f6f47634b80 20 client.27389 trim_cache size 0 max 16384 2023-09-10T01:09:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.284+0000 7f6f47634b80 1 Processor -- start 2023-09-10T01:09:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.284+0000 7f6f47634b80 1 -- start start 2023-09-10T01:09:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca6149580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f47634b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca5fe84f0 con 0x563ca614d250 2023-09-10T01:09:09.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f47634b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca613b0b0 con 0x563ca613f030 2023-09-10T01:09:09.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f47634b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca613f9f0 con 0x563ca613e710 2023-09-10T01:09:09.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35686/0 (socket says 172.21.15.3:35686) 2023-09-10T01:09:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35544/0 (socket says 172.21.15.3:35544) 2023-09-10T01:09:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 -- 172.21.15.3:0/3500675601 learned_addr learned my addr 172.21.15.3:0/3500675601 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3239c700 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca6149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 -- 172.21.15.3:0/3500675601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 msgr2=0x563ca6149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca6149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 -- 172.21.15.3:0/3500675601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 msgr2=0x563ca614a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3139a700 1 -- 172.21.15.3:0/3500675601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ca5ee3250 con 0x563ca613f030 2023-09-10T01:09:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.285+0000 7f6f3239c700 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.286+0000 7f6f3139a700 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149d80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00f480 tx=0x7f6f1c00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20a58ea527da4f7b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.286+0000 7f6f20bff700 1 -- 172.21.15.3:0/3500675601 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c01d040 con 0x563ca613f030 2023-09-10T01:09:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.286+0000 7f6f20bff700 1 -- 172.21.15.3:0/3500675601 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f1c024e90 con 0x563ca613f030 2023-09-10T01:09:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.286+0000 7f6f20bff700 1 -- 172.21.15.3:0/3500675601 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c00fa60 con 0x563ca613f030 2023-09-10T01:09:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 -- 172.21.15.3:0/3500675601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 msgr2=0x563ca6149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149d80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00f480 tx=0x7f6f1c00bfd0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 -- 172.21.15.3:0/3500675601 shutdown_connections 2023-09-10T01:09:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613e710 0x563ca6149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca614a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 --2- 172.21.15.3:0/3500675601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613f030 0x563ca6149d80 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 -- 172.21.15.3:0/3500675601 >> 172.21.15.3:0/3500675601 conn(0x563ca60b3090 msgr2=0x563ca61450d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 -- 172.21.15.3:0/3500675601 shutdown_connections 2023-09-10T01:09:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 -- 172.21.15.3:0/3500675601 wait complete. 2023-09-10T01:09:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 -- start start 2023-09-10T01:09:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.287+0000 7f6f47634b80 1 Processor -- start 2023-09-10T01:09:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca6153c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613f030 0x563ca6151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f47634b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca6152400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f47634b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca6152940 con 0x563ca614d250 2023-09-10T01:09:09.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f47634b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca6152ab0 con 0x563ca613e710 2023-09-10T01:09:09.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f47634b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ca6152c70 con 0x563ca613f030 2023-09-10T01:09:09.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f47634b80 10 client.0 _ll_register_callbacks cb 0x7f6f1c009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:09.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca6153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca6152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f3139a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613f030 0x563ca6151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f3239c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca6153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35696/0 (socket says 172.21.15.3:35696) 2023-09-10T01:09:09.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.288+0000 7f6f3239c700 1 -- 172.21.15.3:0/2800772726 learned_addr learned my addr 172.21.15.3:0/2800772726 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:09.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f3239c700 1 -- 172.21.15.3:0/2800772726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613f030 msgr2=0x563ca6151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f3239c700 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613f030 0x563ca6151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f3239c700 1 -- 172.21.15.3:0/2800772726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 msgr2=0x563ca6152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:09.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f3239c700 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca6152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f3239c700 1 -- 172.21.15.3:0/2800772726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f24027040 con 0x563ca613e710 2023-09-10T01:09:09.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f3239c700 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca6153c00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6f240144a0 tx=0x7f6f24009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f708f0987bb7651d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:09.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f157fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:09.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f157fa700 1 -- 172.21.15.3:0/2800772726 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400e070 con 0x563ca613e710 2023-09-10T01:09:09.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f2400f040 con 0x563ca613e710 2023-09-10T01:09:09.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f157fa700 1 -- 172.21.15.3:0/2800772726 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6f24015a40 con 0x563ca613e710 2023-09-10T01:09:09.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.289+0000 7f6f157fa700 1 -- 172.21.15.3:0/2800772726 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f24015c00 con 0x563ca613e710 2023-09-10T01:09:09.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 10 client.27395 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:09.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563ca5ea5c60 con 0x563ca613e710 2023-09-10T01:09:09.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f157fa700 1 -- 172.21.15.3:0/2800772726 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f24034030 con 0x563ca613e710 2023-09-10T01:09:09.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f157fa700 1 -- 172.21.15.3:0/2800772726 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6f24005030 con 0x563ca613e710 2023-09-10T01:09:09.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 10 client.27395 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:09.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 10 client.27395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:09.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 20 client.27395 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:09.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca61b1f40 0x563ca61b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:09.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 4 client.27395 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:09:09.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563ca6089920 con 0x563ca61b1f40 2023-09-10T01:09:09.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.290+0000 7f6f3139a700 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca61b1f40 0x563ca61b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f157fa700 10 client.27395 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f3139a700 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca61b1f40 0x563ca61b4360 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83dac46f8fc42986 server_cookie=e145d810dbc49f7a in_seq=0 out_seq=0 2023-09-10T01:09:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f157fa700 1 -- 172.21.15.3:0/2800772726 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563ca6089920 con 0x563ca61b1f40 2023-09-10T01:09:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f157fa700 10 client.27395 handle_command_reply: tid=0 2023-09-10T01:09:09.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 client.27395 shutdown 2023-09-10T01:09:09.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 msgr2=0x563ca6153c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca6153c00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6f240144a0 tx=0x7f6f24009920 comp rx=0 tx=0).stop 2023-09-10T01:09:09.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 shutdown_connections 2023-09-10T01:09:09.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563ca61b1f40 0x563ca61b4360 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563ca614d250 0x563ca6152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563ca613e710 0x563ca6153c00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 --2- 172.21.15.3:0/2800772726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563ca613f030 0x563ca6151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f157fa700 0 client.27395 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:09.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 >> 172.21.15.3:0/2800772726 conn(0x563ca60b3090 msgr2=0x563ca614b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:09.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 shutdown_connections 2023-09-10T01:09:09.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 1 -- 172.21.15.3:0/2800772726 wait complete. 2023-09-10T01:09:09.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.291+0000 7f6f47634b80 20 client.27395 trim_cache size 0 max 16384 2023-09-10T01:09:09.316 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:09:09.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6f180a0830 msgr2=0x7f6f180a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6f180a0830 0x7f6f180a2cf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x563ca6134280 tx=0x7f6f28003930 comp rx=0 tx=0).stop 2023-09-10T01:09:09.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c1018e0 msgr2=0x7f6f2c19c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:09.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c1018e0 0x7f6f2c19c570 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6f2800eb30 tx=0x7f6f2800ee40 comp rx=0 tx=0).stop 2023-09-10T01:09:09.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 shutdown_connections 2023-09-10T01:09:09.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6f180a0830 0x7f6f180a2cf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f2c10acb0 0x7f6f2c1965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f2c1018e0 0x7f6f2c19c570 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 --2- 172.21.15.3:0/825997417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f2c102200 0x7f6f2c19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:09.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 >> 172.21.15.3:0/825997417 conn(0x7f6f2c076710 msgr2=0x7f6f2c0fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:09.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 shutdown_connections 2023-09-10T01:09:09.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:09.292+0000 7f6f33dff700 1 -- 172.21.15.3:0/825997417 wait complete. 2023-09-10T01:09:09.319 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:09 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/620248464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:09:09.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:09 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/620248464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:09:09.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:09 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/620248464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:09:09.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 515379 9.40 MB/sec execute 171 sec latency 58.061 ms 2023-09-10T01:09:10.321 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:09:10.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:09:10.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:10 smithi136 ceph-mon[133708]: pgmap v181: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 486 op/s 2023-09-10T01:09:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:10 smithi092 ceph-mon[134067]: pgmap v181: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 486 op/s 2023-09-10T01:09:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 -- 172.21.15.3:0/308054275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c072a30 msgr2=0x7f6c0c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 --2- 172.21.15.3:0/308054275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c072a30 0x7f6c0c072e10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009990 tx=0x7f6c00009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 -- 172.21.15.3:0/308054275 shutdown_connections 2023-09-10T01:09:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 --2- 172.21.15.3:0/308054275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c072a30 0x7f6c0c072e10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 --2- 172.21.15.3:0/308054275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c0c10ce80 0x7f6c0c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 --2- 172.21.15.3:0/308054275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c073350 0x7f6c0c10c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 -- 172.21.15.3:0/308054275 >> 172.21.15.3:0/308054275 conn(0x7f6c0c0fc170 msgr2=0x7f6c0c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 -- 172.21.15.3:0/308054275 shutdown_connections 2023-09-10T01:09:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.543+0000 7f6c12dff700 1 -- 172.21.15.3:0/308054275 wait complete. 2023-09-10T01:09:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.544+0000 7f6c12dff700 1 Processor -- start 2023-09-10T01:09:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c12dff700 1 -- start start 2023-09-10T01:09:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c12dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c072a30 0x7f6c0c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c12dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c073350 0x7f6c0c198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c12dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c0c10ce80 0x7f6c0c19d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c12dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19d920 con 0x7f6c0c10ce80 2023-09-10T01:09:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c12dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19da90 con 0x7f6c0c072a30 2023-09-10T01:09:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c12dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19dc50 con 0x7f6c0c073350 2023-09-10T01:09:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c0bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c073350 0x7f6c0c198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c1139c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c0c10ce80 0x7f6c0c19d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c10b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c072a30 0x7f6c0c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c0bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c073350 0x7f6c0c198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40710/0 (socket says 172.21.15.3:40710) 2023-09-10T01:09:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.545+0000 7f6c0bfff700 1 -- 172.21.15.3:0/200923205 learned_addr learned my addr 172.21.15.3:0/200923205 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c0bfff700 1 -- 172.21.15.3:0/200923205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c072a30 msgr2=0x7f6c0c198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c072a30 0x7f6c0c198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c0bfff700 1 -- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c0c10ce80 msgr2=0x7f6c0c19d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c0c10ce80 0x7f6c0c19d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c0bfff700 1 -- 172.21.15.3:0/200923205 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c000095f0 con 0x7f6c0c073350 2023-09-10T01:09:10.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c072a30 0x7f6c0c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:10.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c073350 0x7f6c0c198ec0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800e9a0 tx=0x7f6bf800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c09ffb700 1 -- 172.21.15.3:0/200923205 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8009790 con 0x7f6c0c073350 2023-09-10T01:09:10.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c09ffb700 1 -- 172.21.15.3:0/200923205 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6bf8010430 con 0x7f6c0c073350 2023-09-10T01:09:10.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c19df30 con 0x7f6c0c073350 2023-09-10T01:09:10.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c09ffb700 1 -- 172.21.15.3:0/200923205 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8018680 con 0x7f6c0c073350 2023-09-10T01:09:10.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.546+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c1a57e0 con 0x7f6c0c073350 2023-09-10T01:09:10.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.550+0000 7f6c09ffb700 1 -- 172.21.15.3:0/200923205 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6bf80105a0 con 0x7f6c0c073350 2023-09-10T01:09:10.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c09ffb700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6bfc0a0830 0x7f6bfc0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 Processor -- start 2023-09-10T01:09:10.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c09ffb700 1 -- 172.21.15.3:0/200923205 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6bf8014070 con 0x7f6c0c073350 2023-09-10T01:09:10.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 -- start start 2023-09-10T01:09:10.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6bfc0a0830 0x7f6bfc0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5bb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5bbec0 0x563dcb5be2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb50a7b0 con 0x563dcb5b20b0 2023-09-10T01:09:10.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5080d0 con 0x563dcb5bbec0 2023-09-10T01:09:10.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.553+0000 7f6c26820b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5039c0 con 0x563dcb5b1790 2023-09-10T01:09:10.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5bbec0 0x563dcb5be2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c0bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5bb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5bbec0 0x563dcb5be2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35726/0 (socket says 172.21.15.3:35726) 2023-09-10T01:09:10.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 -- 172.21.15.3:0/2224669302 learned_addr learned my addr 172.21.15.3:0/2224669302 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6bfc0a0830 0x7f6bfc0a2cf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000b2e0 tx=0x7f6c00005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 -- 172.21.15.3:0/2224669302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 msgr2=0x563dcb5b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 -- 172.21.15.3:0/2224669302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 msgr2=0x563dcb5bb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5bb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 -- 172.21.15.3:0/2224669302 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563dcb32fe60 con 0x563dcb5bbec0 2023-09-10T01:09:10.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:10.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.554+0000 7f6c1139c700 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5bbec0 0x563dcb5be2b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6c04011680 tx=0x7f6c04009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c999ac4fe9b4396 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/2224669302 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400e040 con 0x563dcb5bbec0 2023-09-10T01:09:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/2224669302 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c0400f040 con 0x563dcb5bbec0 2023-09-10T01:09:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/2224669302 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04004850 con 0x563dcb5bbec0 2023-09-10T01:09:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6c26820b80 1 -- 172.21.15.3:0/2224669302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5bbec0 msgr2=0x563dcb5be2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6c26820b80 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5bbec0 0x563dcb5be2b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6c04011680 tx=0x7f6c04009920 comp rx=0 tx=0).stop 2023-09-10T01:09:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6c26820b80 1 -- 172.21.15.3:0/2224669302 shutdown_connections 2023-09-10T01:09:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6c26820b80 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6c26820b80 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5bbec0 0x563dcb5be2b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6c26820b80 1 --2- 172.21.15.3:0/2224669302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5bb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.555+0000 7f6c26820b80 1 -- 172.21.15.3:0/2224669302 >> 172.21.15.3:0/2224669302 conn(0x563dcb526110 msgr2=0x563dcb5acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.556+0000 7f6c26820b80 1 -- 172.21.15.3:0/2224669302 shutdown_connections 2023-09-10T01:09:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.556+0000 7f6c26820b80 1 -- 172.21.15.3:0/2224669302 wait complete. 2023-09-10T01:09:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.556+0000 7f6c26820b80 1 -- start start 2023-09-10T01:09:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.556+0000 7f6c26820b80 1 Processor -- start 2023-09-10T01:09:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb5bb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c26820b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb4a1bf0 con 0x563dcb5b20b0 2023-09-10T01:09:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c26820b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb4a18f0 con 0x563dcb5b1790 2023-09-10T01:09:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c26820b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb3859a0 con 0x563dcb5bbec0 2023-09-10T01:09:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c26820b80 10 client.0 _ll_register_callbacks cb 0x563dcb3ebb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb5bb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb5bb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35738/0 (socket says 172.21.15.3:35738) 2023-09-10T01:09:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 -- 172.21.15.3:0/389720497 learned_addr learned my addr 172.21.15.3:0/389720497 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c1139c700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 -- 172.21.15.3:0/389720497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 msgr2=0x563dcb5b90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5b90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 -- 172.21.15.3:0/389720497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 msgr2=0x563dcb5b8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5b8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c10b9b700 1 -- 172.21.15.3:0/389720497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c04011330 con 0x563dcb5b1790 2023-09-10T01:09:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.557+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5b8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb5bb8c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015000 tx=0x7f6c00009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5146796e8eb4fa5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6bebfff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:10.571 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:10 smithi003 ceph-mon[97654]: pgmap v181: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 486 op/s 2023-09-10T01:09:10.573 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6bebfff700 1 -- 172.21.15.3:0/389720497 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0000f660 con 0x563dcb5b1790 2023-09-10T01:09:10.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6bebfff700 1 -- 172.21.15.3:0/389720497 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6c0000f7c0 con 0x563dcb5b1790 2023-09-10T01:09:10.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563dcb45b570 con 0x563dcb5b1790 2023-09-10T01:09:10.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6bebfff700 1 -- 172.21.15.3:0/389720497 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c0000f980 con 0x563dcb5b1790 2023-09-10T01:09:10.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6c26820b80 10 client.27407 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:10.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.558+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563dcb318ce0 con 0x563dcb5b1790 2023-09-10T01:09:10.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6bebfff700 1 -- 172.21.15.3:0/389720497 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563dcb45b570 con 0x563dcb5b1790 2023-09-10T01:09:10.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6bebfff700 1 -- 172.21.15.3:0/389720497 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6c0002fcc0 con 0x563dcb5b1790 2023-09-10T01:09:10.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6c26820b80 10 client.27407 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:10.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6c26820b80 10 client.27407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:10.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6c26820b80 20 client.27407 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:10.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6c26820b80 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb5cb990 0x563dcb5cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6c26820b80 4 client.27407 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:10.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563dcb518920 con 0x563dcb5cb990 2023-09-10T01:09:10.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.559+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb5cb990 0x563dcb5cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.560+0000 7f6bebfff700 10 client.27407 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.560+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb5cb990 0x563dcb5cddb0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f498ed7cf5d8cd6 server_cookie=35496c63f9b2c4d in_seq=0 out_seq=0 2023-09-10T01:09:10.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.560+0000 7f6bebfff700 1 -- 172.21.15.3:0/389720497 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563dcb518920 con 0x563dcb5cb990 2023-09-10T01:09:10.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.560+0000 7f6bebfff700 10 client.27407 handle_command_reply: tid=0 2023-09-10T01:09:10.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.560+0000 7f6c26820b80 1 client.27407 shutdown 2023-09-10T01:09:10.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.560+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 msgr2=0x563dcb5bb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.560+0000 7f6c26820b80 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb5bb8c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015000 tx=0x7f6c00009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 shutdown_connections 2023-09-10T01:09:10.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb5cb990 0x563dcb5cddb0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5b90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb5bb8c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 --2- 172.21.15.3:0/389720497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb5b8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6bebfff700 0 client.27407 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 >> 172.21.15.3:0/389720497 conn(0x563dcb526110 msgr2=0x563dcb5ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 shutdown_connections 2023-09-10T01:09:10.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 1 -- 172.21.15.3:0/389720497 wait complete. 2023-09-10T01:09:10.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.561+0000 7f6c26820b80 20 client.27407 trim_cache size 0 max 16384 2023-09-10T01:09:10.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.579+0000 7f6c26820b80 1 Processor -- start 2023-09-10T01:09:10.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.579+0000 7f6c26820b80 1 -- start start 2023-09-10T01:09:10.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b20b0 0x563dcb5c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5bbec0 0x563dcb5bb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c26820b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5b9d80 con 0x563dcb5bbec0 2023-09-10T01:09:10.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c26820b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5c2fb0 con 0x563dcb5b20b0 2023-09-10T01:09:10.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c26820b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5ad760 con 0x563dcb5b1790 2023-09-10T01:09:10.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c0bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b20b0 0x563dcb5c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c10b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c0bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b20b0 0x563dcb5c71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35748/0 (socket says 172.21.15.3:35748) 2023-09-10T01:09:10.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c0bfff700 1 -- 172.21.15.3:0/1620668221 learned_addr learned my addr 172.21.15.3:0/1620668221 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c1139c700 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5bbec0 0x563dcb5bb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c0bfff700 1 -- 172.21.15.3:0/1620668221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 msgr2=0x563dcb5c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.580+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.581+0000 7f6c0bfff700 1 -- 172.21.15.3:0/1620668221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5bbec0 msgr2=0x563dcb5bb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.581+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5bbec0 0x563dcb5bb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.581+0000 7f6c0bfff700 1 -- 172.21.15.3:0/1620668221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563dcb5d29e0 con 0x563dcb5b20b0 2023-09-10T01:09:10.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.581+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b20b0 0x563dcb5c71c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8099540 tx=0x7f6bf8010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd02798a66ebd09a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.581+0000 7f6bf77fe700 1 -- 172.21.15.3:0/1620668221 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf809d3e0 con 0x563dcb5b20b0 2023-09-10T01:09:10.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.581+0000 7f6bf77fe700 1 -- 172.21.15.3:0/1620668221 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6bf80a1620 con 0x563dcb5b20b0 2023-09-10T01:09:10.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.581+0000 7f6bf77fe700 1 -- 172.21.15.3:0/1620668221 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8010960 con 0x563dcb5b20b0 2023-09-10T01:09:10.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 -- 172.21.15.3:0/1620668221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b20b0 msgr2=0x563dcb5c71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b20b0 0x563dcb5c71c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8099540 tx=0x7f6bf8010c80 comp rx=0 tx=0).stop 2023-09-10T01:09:10.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 -- 172.21.15.3:0/1620668221 shutdown_connections 2023-09-10T01:09:10.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5b1790 0x563dcb5c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5bbec0 0x563dcb5bb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 --2- 172.21.15.3:0/1620668221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b20b0 0x563dcb5c71c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 -- 172.21.15.3:0/1620668221 >> 172.21.15.3:0/1620668221 conn(0x563dcb526110 msgr2=0x563dcb5b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 -- 172.21.15.3:0/1620668221 shutdown_connections 2023-09-10T01:09:10.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 -- 172.21.15.3:0/1620668221 wait complete. 2023-09-10T01:09:10.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.582+0000 7f6c26820b80 1 -- start start 2023-09-10T01:09:10.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 1 Processor -- start 2023-09-10T01:09:10.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb618ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb6190e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5c1f90 con 0x563dcb5b20b0 2023-09-10T01:09:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5c2100 con 0x563dcb5b1790 2023-09-10T01:09:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dcb5c22c0 con 0x563dcb5bbec0 2023-09-10T01:09:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.583+0000 7f6c26820b80 10 client.0 _ll_register_callbacks cb 0x7f6c000054b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb618ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb618ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35760/0 (socket says 172.21.15.3:35760) 2023-09-10T01:09:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 -- 172.21.15.3:0/3745003379 learned_addr learned my addr 172.21.15.3:0/3745003379 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c1139c700 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb6190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 -- 172.21.15.3:0/3745003379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 msgr2=0x563dcb5c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 -- 172.21.15.3:0/3745003379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 msgr2=0x563dcb6190e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb6190e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 -- 172.21.15.3:0/3745003379 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c1a57e0 con 0x563dcb5b1790 2023-09-10T01:09:10.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6c10b9b700 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb618ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015000 tx=0x7f6c00024880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0b34b34b3524cb4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6bebfff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:10.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.584+0000 7f6bebfff700 1 -- 172.21.15.3:0/3745003379 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c00005a00 con 0x563dcb5b1790 2023-09-10T01:09:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c0400f040 con 0x563dcb5b1790 2023-09-10T01:09:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6bebfff700 1 -- 172.21.15.3:0/3745003379 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f6c0000f460 con 0x563dcb5b1790 2023-09-10T01:09:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6bebfff700 1 -- 172.21.15.3:0/3745003379 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c00024e90 con 0x563dcb5b1790 2023-09-10T01:09:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 10 client.27413 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563dcb3562d0 con 0x563dcb5b1790 2023-09-10T01:09:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6bebfff700 1 -- 172.21.15.3:0/3745003379 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c0400f040 con 0x563dcb5b1790 2023-09-10T01:09:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6bebfff700 1 -- 172.21.15.3:0/3745003379 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6c00037070 con 0x563dcb5b1790 2023-09-10T01:09:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 10 client.27413 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 10 client.27413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:10.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 20 client.27413 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb621750 0x563dcb623b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 4 client.27413 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:09:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563dcb4fc9a0 con 0x563dcb621750 2023-09-10T01:09:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.585+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb621750 0x563dcb623b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.586+0000 7f6c0bfff700 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb621750 0x563dcb623b70 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e2bfa17b9b0f38a server_cookie=4081318c41b43f3b in_seq=0 out_seq=0 2023-09-10T01:09:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.586+0000 7f6bebfff700 10 client.27413 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6bebfff700 1 -- 172.21.15.3:0/3745003379 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563dcb4fc9a0 con 0x563dcb621750 2023-09-10T01:09:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6bebfff700 10 client.27413 handle_command_reply: tid=0 2023-09-10T01:09:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 client.27413 shutdown 2023-09-10T01:09:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 msgr2=0x563dcb618ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb618ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015000 tx=0x7f6c00024880 comp rx=0 tx=0).stop 2023-09-10T01:09:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 shutdown_connections 2023-09-10T01:09:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563dcb621750 0x563dcb623b70 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563dcb5bbec0 0x563dcb5c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563dcb5b1790 0x563dcb618ba0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 --2- 172.21.15.3:0/3745003379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563dcb5b20b0 0x563dcb6190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6bebfff700 0 client.27413 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 >> 172.21.15.3:0/3745003379 conn(0x563dcb526110 msgr2=0x563dcb5b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 shutdown_connections 2023-09-10T01:09:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.587+0000 7f6c26820b80 1 -- 172.21.15.3:0/3745003379 wait complete. 2023-09-10T01:09:10.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c26820b80 20 client.27413 trim_cache size 0 max 16384 2023-09-10T01:09:10.603 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:09:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:09:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:09:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:09:10.604 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:09:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6bfc0a0830 msgr2=0x7f6bfc0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6bfc0a0830 0x7f6bfc0a2cf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000b2e0 tx=0x7f6c00005fd0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c073350 msgr2=0x7f6c0c198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c073350 0x7f6c0c198ec0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800e9a0 tx=0x7f6bf800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 shutdown_connections 2023-09-10T01:09:10.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6bfc0a0830 0x7f6bfc0a2cf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c0c10ce80 0x7f6c0c19d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c072a30 0x7f6c0c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 --2- 172.21.15.3:0/200923205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c0c073350 0x7f6c0c198ec0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 >> 172.21.15.3:0/200923205 conn(0x7f6c0c0fc170 msgr2=0x7f6c0c10efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 shutdown_connections 2023-09-10T01:09:10.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.588+0000 7f6c12dff700 1 -- 172.21.15.3:0/200923205 wait complete. 2023-09-10T01:09:10.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:09:10.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 516702 9.38 MB/sec execute 172 sec latency 16.984 ms 2023-09-10T01:09:10.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.824+0000 7f1890bff700 1 -- 172.21.15.3:0/2883843504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c106650 msgr2=0x7f188c106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.824+0000 7f1890bff700 1 --2- 172.21.15.3:0/2883843504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c106650 0x7f188c106a30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f187c009990 tx=0x7f187c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.824+0000 7f1890bff700 1 -- 172.21.15.3:0/2883843504 shutdown_connections 2023-09-10T01:09:10.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.824+0000 7f1890bff700 1 --2- 172.21.15.3:0/2883843504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f188c101000 0x7f188c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.824+0000 7f1890bff700 1 --2- 172.21.15.3:0/2883843504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c106650 0x7f188c106a30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.824+0000 7f1890bff700 1 --2- 172.21.15.3:0/2883843504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 0x7f188c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.824+0000 7f1890bff700 1 -- 172.21.15.3:0/2883843504 >> 172.21.15.3:0/2883843504 conn(0x7f188c0fc170 msgr2=0x7f188c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.825+0000 7f1890bff700 1 -- 172.21.15.3:0/2883843504 shutdown_connections 2023-09-10T01:09:10.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.825+0000 7f1890bff700 1 -- 172.21.15.3:0/2883843504 wait complete. 2023-09-10T01:09:10.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.826+0000 7f1890bff700 1 Processor -- start 2023-09-10T01:09:10.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.826+0000 7f1890bff700 1 -- start start 2023-09-10T01:09:10.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f1890bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 0x7f188c1966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f1890bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c101000 0x7f188c196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f1890bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f188c106650 0x7f188c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f1890bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c19a5b0 con 0x7f188c106650 2023-09-10T01:09:10.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f1890bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c19a720 con 0x7f188c101000 2023-09-10T01:09:10.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f1890bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188c19a8e0 con 0x7f188c100640 2023-09-10T01:09:10.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f188a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 0x7f188c1966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f1889d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c101000 0x7f188c196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f188a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 0x7f188c1966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40772/0 (socket says 172.21.15.3:40772) 2023-09-10T01:09:10.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f188a59c700 1 -- 172.21.15.3:0/1921289994 learned_addr learned my addr 172.21.15.3:0/1921289994 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.827+0000 7f188ad9d700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f188c106650 0x7f188c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1889d9b700 1 -- 172.21.15.3:0/1921289994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 msgr2=0x7f188c1966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1889d9b700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 0x7f188c1966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1889d9b700 1 -- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f188c106650 msgr2=0x7f188c199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1889d9b700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f188c106650 0x7f188c199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1889d9b700 1 -- 172.21.15.3:0/1921289994 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c0095f0 con 0x7f188c101000 2023-09-10T01:09:10.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f188a59c700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 0x7f188c1966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:10.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1889d9b700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c101000 0x7f188c196c00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f188000d950 tx=0x7f188000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f187b7fe700 1 -- 172.21.15.3:0/1921289994 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188000f890 con 0x7f188c101000 2023-09-10T01:09:10.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f187b7fe700 1 -- 172.21.15.3:0/1921289994 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1880010460 con 0x7f188c101000 2023-09-10T01:09:10.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188c19abc0 con 0x7f188c101000 2023-09-10T01:09:10.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f187b7fe700 1 -- 172.21.15.3:0/1921289994 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188000e610 con 0x7f188c101000 2023-09-10T01:09:10.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.828+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c1a3520 con 0x7f188c101000 2023-09-10T01:09:10.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.832+0000 7f187b7fe700 1 -- 172.21.15.3:0/1921289994 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f18800108d0 con 0x7f188c101000 2023-09-10T01:09:10.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.833+0000 7f18a44d0b80 1 Processor -- start 2023-09-10T01:09:10.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.833+0000 7f18a44d0b80 1 -- start start 2023-09-10T01:09:10.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622dcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622df640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563462227730 con 0x5634622dd250 2023-09-10T01:09:10.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563462225050 con 0x5634622cf030 2023-09-10T01:09:10.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563462220940 con 0x5634622ce710 2023-09-10T01:09:10.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f188ad9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f188a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622dcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f188ad9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622ceaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40774/0 (socket says 172.21.15.3:40774) 2023-09-10T01:09:10.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f188a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622dcd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35774/0 (socket says 172.21.15.3:35774) 2023-09-10T01:09:10.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f188ad9d700 1 -- 172.21.15.3:0/2598093269 learned_addr learned my addr 172.21.15.3:0/2598093269 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f1889d9b700 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622df640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f187b7fe700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18740a0830 0x7f18740a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f188ad9d700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18740a0830 0x7f18740a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.834+0000 7f187b7fe700 1 -- 172.21.15.3:0/1921289994 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f18800cbf90 con 0x7f188c101000 2023-09-10T01:09:10.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f188a59c700 1 -- 172.21.15.3:0/2598093269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 msgr2=0x5634622ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f188a59c700 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f188a59c700 1 -- 172.21.15.3:0/2598093269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 msgr2=0x5634622df640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f188a59c700 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622df640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f188a59c700 1 -- 172.21.15.3:0/2598093269 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56346204cde0 con 0x5634622cf030 2023-09-10T01:09:10.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f188a59c700 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622dcd10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f187c0059d0 tx=0x7f187c00bc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79808adc6638be1d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f188ad9d700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18740a0830 0x7f18740a2cf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1884003fd0 tx=0x7f1884004b40 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f1878ff9700 1 -- 172.21.15.3:0/2598093269 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c01d050 con 0x5634622cf030 2023-09-10T01:09:10.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f1878ff9700 1 -- 172.21.15.3:0/2598093269 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f187c01e040 con 0x5634622cf030 2023-09-10T01:09:10.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.835+0000 7f1878ff9700 1 -- 172.21.15.3:0/2598093269 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c0056b0 con 0x5634622cf030 2023-09-10T01:09:10.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 -- 172.21.15.3:0/2598093269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 msgr2=0x5634622dcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622dcd10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f187c0059d0 tx=0x7f187c00bc00 comp rx=0 tx=0).stop 2023-09-10T01:09:10.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 -- 172.21.15.3:0/2598093269 shutdown_connections 2023-09-10T01:09:10.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622df640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/2598093269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622dcd10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 -- 172.21.15.3:0/2598093269 >> 172.21.15.3:0/2598093269 conn(0x563462243090 msgr2=0x5634622c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 -- 172.21.15.3:0/2598093269 shutdown_connections 2023-09-10T01:09:10.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.836+0000 7f18a44d0b80 1 -- 172.21.15.3:0/2598093269 wait complete. 2023-09-10T01:09:10.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.837+0000 7f18a44d0b80 1 -- start start 2023-09-10T01:09:10.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.837+0000 7f18a44d0b80 1 Processor -- start 2023-09-10T01:09:10.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622dccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622d9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622da5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634621beb70 con 0x5634622dd250 2023-09-10T01:09:10.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634621be870 con 0x5634622ce710 2023-09-10T01:09:10.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634620a2920 con 0x5634622cf030 2023-09-10T01:09:10.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f18a44d0b80 10 client.0 _ll_register_callbacks cb 0x563462108a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:10.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f188a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622dccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f188a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622dccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35782/0 (socket says 172.21.15.3:35782) 2023-09-10T01:09:10.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f188a59c700 1 -- 172.21.15.3:0/434100792 learned_addr learned my addr 172.21.15.3:0/434100792 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f188ad9d700 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622da5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.838+0000 7f1889d9b700 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622d9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f188a59c700 1 -- 172.21.15.3:0/434100792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 msgr2=0x5634622d9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f188a59c700 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622d9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f188a59c700 1 -- 172.21.15.3:0/434100792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 msgr2=0x5634622da5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f188a59c700 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622da5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f188a59c700 1 -- 172.21.15.3:0/434100792 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c10de10 con 0x5634622ce710 2023-09-10T01:09:10.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f188a59c700 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622dccd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f187c009350 tx=0x7f187c005c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e28304450512456d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f1871ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:10.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f1871ffb700 1 -- 172.21.15.3:0/434100792 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c003ca0 con 0x5634622ce710 2023-09-10T01:09:10.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5634621784f0 con 0x5634622ce710 2023-09-10T01:09:10.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f1871ffb700 1 -- 172.21.15.3:0/434100792 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f187c023460 con 0x5634622ce710 2023-09-10T01:09:10.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f1871ffb700 1 -- 172.21.15.3:0/434100792 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f187c01d020 con 0x5634622ce710 2023-09-10T01:09:10.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f18a44d0b80 10 client.27425 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:10.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.839+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563462035c60 con 0x5634622ce710 2023-09-10T01:09:10.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f1871ffb700 1 -- 172.21.15.3:0/434100792 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5634621784f0 con 0x5634622ce710 2023-09-10T01:09:10.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f1871ffb700 1 -- 172.21.15.3:0/434100792 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f187c036070 con 0x5634622ce710 2023-09-10T01:09:10.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f18a44d0b80 10 client.27425 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:10.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f18a44d0b80 10 client.27425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:10.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f18a44d0b80 20 client.27425 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:10.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634622ec7f0 0x5634622eec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f18a44d0b80 4 client.27425 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:10.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634622358a0 con 0x5634622ec7f0 2023-09-10T01:09:10.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.840+0000 7f1889d9b700 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634622ec7f0 0x5634622eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.841+0000 7f1871ffb700 10 client.27425 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.841+0000 7f1889d9b700 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634622ec7f0 0x5634622eec10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb97d42ecb93830 server_cookie=580dde560cc9a7c7 in_seq=0 out_seq=0 2023-09-10T01:09:10.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.841+0000 7f1871ffb700 1 -- 172.21.15.3:0/434100792 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5634622358a0 con 0x5634622ec7f0 2023-09-10T01:09:10.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.841+0000 7f1871ffb700 10 client.27425 handle_command_reply: tid=0 2023-09-10T01:09:10.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 client.27425 shutdown 2023-09-10T01:09:10.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 msgr2=0x5634622dccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622dccd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f187c009350 tx=0x7f187c005c40 comp rx=0 tx=0).stop 2023-09-10T01:09:10.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 shutdown_connections 2023-09-10T01:09:10.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634622ec7f0 0x5634622eec10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622da5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622dccd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f1871ffb700 0 client.27425 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/434100792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622d9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 >> 172.21.15.3:0/434100792 conn(0x563462243090 msgr2=0x5634622c97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 shutdown_connections 2023-09-10T01:09:10.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 1 -- 172.21.15.3:0/434100792 wait complete. 2023-09-10T01:09:10.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.842+0000 7f18a44d0b80 20 client.27425 trim_cache size 0 max 16384 2023-09-10T01:09:10.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 Processor -- start 2023-09-10T01:09:10.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 -- start start 2023-09-10T01:09:10.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622d9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622dc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622d5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634622db370 con 0x5634622dd250 2023-09-10T01:09:10.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634622c86e0 con 0x5634622ce710 2023-09-10T01:09:10.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.860+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634622c88a0 con 0x5634622cf030 2023-09-10T01:09:10.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622dc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f188a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622d9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622dc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40794/0 (socket says 172.21.15.3:40794) 2023-09-10T01:09:10.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 -- 172.21.15.3:0/56376802 learned_addr learned my addr 172.21.15.3:0/56376802 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f188ad9d700 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622d5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 -- 172.21.15.3:0/56376802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 msgr2=0x5634622d9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622d9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 -- 172.21.15.3:0/56376802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 msgr2=0x5634622d5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622d5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 -- 172.21.15.3:0/56376802 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563462073250 con 0x5634622cf030 2023-09-10T01:09:10.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f1889d9b700 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622dc800 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1880010fd0 tx=0x7f18800ce410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4c7fd1858561d55 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f18797fa700 1 -- 172.21.15.3:0/56376802 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880099050 con 0x5634622cf030 2023-09-10T01:09:10.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f18797fa700 1 -- 172.21.15.3:0/56376802 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18800a10c0 con 0x5634622cf030 2023-09-10T01:09:10.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.861+0000 7f18797fa700 1 -- 172.21.15.3:0/56376802 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880099510 con 0x5634622cf030 2023-09-10T01:09:10.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 -- 172.21.15.3:0/56376802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 msgr2=0x5634622dc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622dc800 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1880010fd0 tx=0x7f18800ce410 comp rx=0 tx=0).stop 2023-09-10T01:09:10.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 -- 172.21.15.3:0/56376802 shutdown_connections 2023-09-10T01:09:10.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622d5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622ce710 0x5634622d9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/56376802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622cf030 0x5634622dc800 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 -- 172.21.15.3:0/56376802 >> 172.21.15.3:0/56376802 conn(0x563462243090 msgr2=0x5634622f3530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 -- 172.21.15.3:0/56376802 shutdown_connections 2023-09-10T01:09:10.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.862+0000 7f18a44d0b80 1 -- 172.21.15.3:0/56376802 wait complete. 2023-09-10T01:09:10.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 -- start start 2023-09-10T01:09:10.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 Processor -- start 2023-09-10T01:09:10.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622e2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622e26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634622e2e10 con 0x5634622dd250 2023-09-10T01:09:10.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634622e2f80 con 0x5634622cf030 2023-09-10T01:09:10.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.863+0000 7f18a44d0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634623405f0 con 0x5634622ce710 2023-09-10T01:09:10.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f18a44d0b80 10 client.0 _ll_register_callbacks cb 0x7f1884003a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:10.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f1889d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622e2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622e26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622e3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40806/0 (socket says 172.21.15.3:40806) 2023-09-10T01:09:10.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 -- 172.21.15.3:0/3992321207 learned_addr learned my addr 172.21.15.3:0/3992321207 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:10.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 -- 172.21.15.3:0/3992321207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 msgr2=0x5634622e2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622e2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 -- 172.21.15.3:0/3992321207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 msgr2=0x5634622e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:10.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622e26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 -- 172.21.15.3:0/3992321207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c1a3520 con 0x5634622ce710 2023-09-10T01:09:10.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f188a59c700 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622e3e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f187c006010 tx=0x7f187c003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a9ac3d1e3052ca server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:10.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f1871ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:09:10.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f1871ffb700 1 -- 172.21.15.3:0/3992321207 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187c01d070 con 0x5634622ce710 2023-09-10T01:09:10.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.864+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f187c01e040 con 0x5634622ce710 2023-09-10T01:09:10.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f1871ffb700 1 -- 172.21.15.3:0/3992321207 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f187c000e40 con 0x5634622ce710 2023-09-10T01:09:10.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f1871ffb700 1 -- 172.21.15.3:0/3992321207 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f187c00b670 con 0x5634622ce710 2023-09-10T01:09:10.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 10 client.27343 fetch_fsmap learned FSMap version 27 2023-09-10T01:09:10.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56346204b2b0 con 0x5634622ce710 2023-09-10T01:09:10.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f1871ffb700 1 -- 172.21.15.3:0/3992321207 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f187c01e040 con 0x5634622ce710 2023-09-10T01:09:10.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f1871ffb700 1 -- 172.21.15.3:0/3992321207 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f187c0099c0 con 0x5634622ce710 2023-09-10T01:09:10.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 10 client.27343 fetch_fsmap finished waiting for FSMap version 27 2023-09-10T01:09:10.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 10 client.27343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:10.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 20 client.27343 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:10.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634623417d0 0x563462343bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:10.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 4 client.27343 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:09:10.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.865+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563462219920 con 0x5634623417d0 2023-09-10T01:09:10.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f1889d9b700 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634623417d0 0x563462343bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f1889d9b700 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634623417d0 0x563462343bc0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b29c9389c0e6e1f server_cookie=e8955a0f2845760b in_seq=0 out_seq=0 2023-09-10T01:09:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f1871ffb700 10 client.27343 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f1871ffb700 1 -- 172.21.15.3:0/3992321207 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x563462219920 con 0x5634623417d0 2023-09-10T01:09:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f1871ffb700 10 client.27343 handle_command_reply: tid=0 2023-09-10T01:09:10.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 client.27343 shutdown 2023-09-10T01:09:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 msgr2=0x5634622e3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622e3e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f187c006010 tx=0x7f187c003cd0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 shutdown_connections 2023-09-10T01:09:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5634623417d0 0x563462343bc0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634622ce710 0x5634622e3e20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634622dd250 0x5634622e26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 --2- 172.21.15.3:0/3992321207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5634622cf030 0x5634622e2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 >> 172.21.15.3:0/3992321207 conn(0x563462243090 msgr2=0x5634622d5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.866+0000 7f1871ffb700 0 client.27343 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 shutdown_connections 2023-09-10T01:09:10.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f18a44d0b80 1 -- 172.21.15.3:0/3992321207 wait complete. 2023-09-10T01:09:10.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f18a44d0b80 20 client.27343 trim_cache size 0 max 16384 2023-09-10T01:09:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:09:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:09:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:09:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:09:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:09:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:09:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:09:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:09:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:09:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:09:10.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18740a0830 msgr2=0x7f18740a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18740a0830 0x7f18740a2cf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1884003fd0 tx=0x7f1884004b40 comp rx=0 tx=0).stop 2023-09-10T01:09:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c101000 msgr2=0x7f188c196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c101000 0x7f188c196c00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f188000d950 tx=0x7f188000dc60 comp rx=0 tx=0).stop 2023-09-10T01:09:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 shutdown_connections 2023-09-10T01:09:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18740a0830 0x7f18740a2cf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f188c100640 0x7f188c1966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f188c106650 0x7f188c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 --2- 172.21.15.3:0/1921289994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f188c101000 0x7f188c196c00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.867+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 >> 172.21.15.3:0/1921289994 conn(0x7f188c0fc170 msgr2=0x7f188c0fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.868+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 shutdown_connections 2023-09-10T01:09:10.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:10.868+0000 7f1890bff700 1 -- 172.21.15.3:0/1921289994 wait complete. 2023-09-10T01:09:10.891 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:09:11.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:11 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:09:11.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:11 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:09:11.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:11 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:09:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:11 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:09:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:11 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:09:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:11 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:09:11.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:11 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:09:11.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:11 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:09:11.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:11 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:09:11.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 518903 9.41 MB/sec execute 173 sec latency 97.151 ms 2023-09-10T01:09:12.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:12 smithi092 ceph-mon[134067]: pgmap v182: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2023-09-10T01:09:12.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:12 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:09:12.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:12 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:09:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:12 smithi003 ceph-mon[97654]: pgmap v182: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2023-09-10T01:09:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:12 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:09:12.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:12 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:09:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:12 smithi136 ceph-mon[133708]: pgmap v182: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2023-09-10T01:09:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:12 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:09:12.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:12 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:09:12.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 520180 9.38 MB/sec execute 174 sec latency 36.407 ms 2023-09-10T01:09:13.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 522434 9.41 MB/sec execute 175 sec latency 76.196 ms 2023-09-10T01:09:14.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:14 smithi092 ceph-mon[134067]: pgmap v183: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2023-09-10T01:09:14.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:14 smithi003 ceph-mon[97654]: pgmap v183: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2023-09-10T01:09:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:14 smithi136 ceph-mon[133708]: pgmap v183: 129 pgs: 129 active+clean; 145 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.3 MiB/s wr, 310 op/s 2023-09-10T01:09:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.727+0000 7f4ef49ff700 1 -- 172.21.15.3:0/3312554427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef010ce40 msgr2=0x7f4ef010f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.727+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/3312554427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef010ce40 0x7f4ef010f270 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800b0a0 tx=0x7f4ee800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.728+0000 7f4ef49ff700 1 -- 172.21.15.3:0/3312554427 shutdown_connections 2023-09-10T01:09:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.728+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/3312554427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef010ce40 0x7f4ef010f270 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.728+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/3312554427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ef00729d0 0x7f4ef0072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.728+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/3312554427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00732f0 0x7f4ef010c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.728+0000 7f4ef49ff700 1 -- 172.21.15.3:0/3312554427 >> 172.21.15.3:0/3312554427 conn(0x7f4ef00fc0b0 msgr2=0x7f4ef00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.728+0000 7f4ef49ff700 1 -- 172.21.15.3:0/3312554427 shutdown_connections 2023-09-10T01:09:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.728+0000 7f4ef49ff700 1 -- 172.21.15.3:0/3312554427 wait complete. 2023-09-10T01:09:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 Processor -- start 2023-09-10T01:09:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 -- start start 2023-09-10T01:09:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00729d0 0x7f4ef019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef00732f0 0x7f4ef019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ef010ce40 0x7f4ef0198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef0198ed0 con 0x7f4ef010ce40 2023-09-10T01:09:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef0199040 con 0x7f4ef00729d0 2023-09-10T01:09:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.729+0000 7f4ef49ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef0199200 con 0x7f4ef00732f0 2023-09-10T01:09:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00729d0 0x7f4ef019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00729d0 0x7f4ef019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:38516/0 (socket says 172.21.15.3:38516) 2023-09-10T01:09:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 -- 172.21.15.3:0/1648921360 learned_addr learned my addr 172.21.15.3:0/1648921360 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eeed9d700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ef010ce40 0x7f4ef0198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef00732f0 0x7f4ef019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 -- 172.21.15.3:0/1648921360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef00732f0 msgr2=0x7f4ef019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef00732f0 0x7f4ef019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 -- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ef010ce40 msgr2=0x7f4ef0198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ef010ce40 0x7f4ef0198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 -- 172.21.15.3:0/1648921360 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8009d00 con 0x7f4ef00729d0 2023-09-10T01:09:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4eee59c700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00729d0 0x7f4ef019e7f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000e9a0 tx=0x7f4ee000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.730+0000 7f4edf7fe700 1 -- 172.21.15.3:0/1648921360 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0009790 con 0x7f4ef00729d0 2023-09-10T01:09:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.731+0000 7f4edf7fe700 1 -- 172.21.15.3:0/1648921360 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ee0010430 con 0x7f4ef00729d0 2023-09-10T01:09:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.731+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef01994e0 con 0x7f4ef00729d0 2023-09-10T01:09:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.731+0000 7f4edf7fe700 1 -- 172.21.15.3:0/1648921360 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0018680 con 0x7f4ef00729d0 2023-09-10T01:09:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.731+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef01a57e0 con 0x7f4ef00729d0 2023-09-10T01:09:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.734+0000 7f4edf7fe700 1 -- 172.21.15.3:0/1648921360 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f4ee00105a0 con 0x7f4ef00729d0 2023-09-10T01:09:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.735+0000 7f4edf7fe700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4ed80a07b0 0x7f4ed80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.735+0000 7f4f08335b80 1 Processor -- start 2023-09-10T01:09:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4edf7fe700 1 -- 172.21.15.3:0/1648921360 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4ee0014070 con 0x7f4ef00729d0 2023-09-10T01:09:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4ed80a07b0 0x7f4ed80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4f08335b80 1 -- start start 2023-09-10T01:09:14.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2a91580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8b300 0x5632c2a8b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8bcc0 0x5632c2a97f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4f08335b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c29e46d0 con 0x5632c2a8bcc0 2023-09-10T01:09:14.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4f08335b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c29e1ff0 con 0x5632c2a911a0 2023-09-10T01:09:14.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4f08335b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c29dd8e0 con 0x5632c2a8b300 2023-09-10T01:09:14.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2a91580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2a91580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:38528/0 (socket says 172.21.15.3:38528) 2023-09-10T01:09:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.736+0000 7f4eee59c700 1 -- 172.21.15.3:0/753114112 learned_addr learned my addr 172.21.15.3:0/753114112 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4ed80a07b0 0x7f4ed80a2c70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4ef010e420 tx=0x7f4ee4009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8b300 0x5632c2a8b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eeed9d700 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8bcc0 0x5632c2a97f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eee59c700 1 -- 172.21.15.3:0/753114112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8b300 msgr2=0x5632c2a8b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eee59c700 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8b300 0x5632c2a8b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eee59c700 1 -- 172.21.15.3:0/753114112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8bcc0 msgr2=0x5632c2a97f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eee59c700 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8bcc0 0x5632c2a97f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eee59c700 1 -- 172.21.15.3:0/753114112 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632c2809d80 con 0x5632c2a911a0 2023-09-10T01:09:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4eee59c700 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2a91580 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0096070 tx=0x7f4ee0099030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32078b45d273cada server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4edcff9700 1 -- 172.21.15.3:0/753114112 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee009d430 con 0x5632c2a911a0 2023-09-10T01:09:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4edcff9700 1 -- 172.21.15.3:0/753114112 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ee00a1670 con 0x5632c2a911a0 2023-09-10T01:09:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.737+0000 7f4edcff9700 1 -- 172.21.15.3:0/753114112 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00842d0 con 0x5632c2a911a0 2023-09-10T01:09:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 -- 172.21.15.3:0/753114112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 msgr2=0x5632c2a91580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2a91580 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0096070 tx=0x7f4ee0099030 comp rx=0 tx=0).stop 2023-09-10T01:09:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 -- 172.21.15.3:0/753114112 shutdown_connections 2023-09-10T01:09:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8bcc0 0x5632c2a97f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2a91580 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 --2- 172.21.15.3:0/753114112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8b300 0x5632c2a8b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 -- 172.21.15.3:0/753114112 >> 172.21.15.3:0/753114112 conn(0x5632c2a00030 msgr2=0x5632c2a86560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:14.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 -- 172.21.15.3:0/753114112 shutdown_connections 2023-09-10T01:09:14.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.738+0000 7f4f08335b80 1 -- 172.21.15.3:0/753114112 wait complete. 2023-09-10T01:09:14.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.739+0000 7f4f08335b80 1 -- start start 2023-09-10T01:09:14.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.739+0000 7f4f08335b80 1 Processor -- start 2023-09-10T01:09:14.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a97e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a95120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a957a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4f08335b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c297bb10 con 0x5632c2a911a0 2023-09-10T01:09:14.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4f08335b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c297b810 con 0x5632c2a8b300 2023-09-10T01:09:14.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4f08335b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c285f8c0 con 0x5632c2a8bcc0 2023-09-10T01:09:14.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4f08335b80 10 client.0 _ll_register_callbacks cb 0x5632c28c5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:14.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a97e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a97e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:38544/0 (socket says 172.21.15.3:38544) 2023-09-10T01:09:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 -- 172.21.15.3:0/1532775929 learned_addr learned my addr 172.21.15.3:0/1532775929 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eeed9d700 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a95120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 -- 172.21.15.3:0/1532775929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 msgr2=0x5632c2a95120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a95120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 -- 172.21.15.3:0/1532775929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 msgr2=0x5632c2a957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.740+0000 7f4eee59c700 1 -- 172.21.15.3:0/1532775929 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef01a57e0 con 0x5632c2a8b300 2023-09-10T01:09:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4eee59c700 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a97e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0099540 tx=0x7f4ee00c9840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d51d99c6404271f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4ed5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/1532775929 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00b5070 con 0x5632c2a8b300 2023-09-10T01:09:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/1532775929 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4ee0082e00 con 0x5632c2a8b300 2023-09-10T01:09:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5632c2935490 con 0x5632c2a8b300 2023-09-10T01:09:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/1532775929 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ee0010880 con 0x5632c2a8b300 2023-09-10T01:09:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4f08335b80 10 client.27443 fetch_fsmap learned FSMap version 28 2023-09-10T01:09:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5632c27f2c00 con 0x5632c2a8b300 2023-09-10T01:09:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/1532775929 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5632c2935490 con 0x5632c2a8b300 2023-09-10T01:09:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.741+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/1532775929 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4ee0098af0 con 0x5632c2a8b300 2023-09-10T01:09:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4f08335b80 10 client.27443 fetch_fsmap finished waiting for FSMap version 28 2023-09-10T01:09:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4f08335b80 10 client.27443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4f08335b80 20 client.27443 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4f08335b80 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2aa5140 0x5632c2aa7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4f08335b80 4 client.27443 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632c29f2840 con 0x5632c2aa5140 2023-09-10T01:09:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2aa5140 0x5632c2aa7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2aa5140 0x5632c2aa7560 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a27071b2feb22a8e server_cookie=cb12945cb54d9907 in_seq=0 out_seq=0 2023-09-10T01:09:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.742+0000 7f4ed5ffb700 10 client.27443 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.743+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/1532775929 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5632c29f2840 con 0x5632c2aa5140 2023-09-10T01:09:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.743+0000 7f4ed5ffb700 10 client.27443 handle_command_reply: tid=0 2023-09-10T01:09:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.743+0000 7f4f08335b80 1 client.27443 shutdown 2023-09-10T01:09:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 msgr2=0x5632c2a97e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a97e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0099540 tx=0x7f4ee00c9840 comp rx=0 tx=0).stop 2023-09-10T01:09:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 shutdown_connections 2023-09-10T01:09:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2aa5140 0x5632c2aa7560 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a97e30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4ed5ffb700 0 client.27443 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 --2- 172.21.15.3:0/1532775929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a95120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 >> 172.21.15.3:0/1532775929 conn(0x5632c2a00030 msgr2=0x5632c2a857a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 shutdown_connections 2023-09-10T01:09:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 1 -- 172.21.15.3:0/1532775929 wait complete. 2023-09-10T01:09:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.744+0000 7f4f08335b80 20 client.27443 trim_cache size 0 max 16384 2023-09-10T01:09:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.760+0000 7f4f08335b80 1 Processor -- start 2023-09-10T01:09:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.760+0000 7f4f08335b80 1 -- start start 2023-09-10T01:09:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a9c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a979d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a84f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4f08335b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c2a8edb0 con 0x5632c2a911a0 2023-09-10T01:09:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4f08335b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c2a96690 con 0x5632c2a8b300 2023-09-10T01:09:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4f08335b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c2a9c980 con 0x5632c2a8bcc0 2023-09-10T01:09:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a9c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eedd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a9c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:38554/0 (socket says 172.21.15.3:38554) 2023-09-10T01:09:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eeed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a84f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eedd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:35168/0 (socket says 172.21.15.3:35168) 2023-09-10T01:09:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eee59c700 1 -- 172.21.15.3:0/728484752 learned_addr learned my addr 172.21.15.3:0/728484752 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eedd9b700 1 -- 172.21.15.3:0/728484752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 msgr2=0x5632c2a9c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/728484752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a9c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eedd9b700 1 -- 172.21.15.3:0/728484752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 msgr2=0x5632c2a84f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/728484752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a84f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.761+0000 7f4eedd9b700 1 -- 172.21.15.3:0/728484752 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632c28301f0 con 0x5632c2a8bcc0 2023-09-10T01:09:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.762+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/728484752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a979d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5632c2a96660 tx=0x7f4ee4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fa9bf4a03f4cd2b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.762+0000 7f4edd7fa700 1 -- 172.21.15.3:0/728484752 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4025070 con 0x5632c2a8bcc0 2023-09-10T01:09:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.762+0000 7f4edd7fa700 1 -- 172.21.15.3:0/728484752 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ee4027de0 con 0x5632c2a8bcc0 2023-09-10T01:09:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.762+0000 7f4edd7fa700 1 -- 172.21.15.3:0/728484752 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40037e0 con 0x5632c2a8bcc0 2023-09-10T01:09:14.775 INFO:tasks.workunit.client.0.smithi003.stdout: 1 523648 9.39 MB/sec execute 176 sec latency 22.002 ms 2023-09-10T01:09:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.762+0000 7f4f08335b80 1 -- 172.21.15.3:0/728484752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 msgr2=0x5632c2a979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.762+0000 7f4f08335b80 1 --2- 172.21.15.3:0/728484752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a979d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x5632c2a96660 tx=0x7f4ee4014000 comp rx=0 tx=0).stop 2023-09-10T01:09:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 -- 172.21.15.3:0/728484752 shutdown_connections 2023-09-10T01:09:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 --2- 172.21.15.3:0/728484752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a911a0 0x5632c2a84f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 --2- 172.21.15.3:0/728484752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a8b300 0x5632c2a9c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 --2- 172.21.15.3:0/728484752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2a979d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 -- 172.21.15.3:0/728484752 >> 172.21.15.3:0/728484752 conn(0x5632c2a00030 msgr2=0x5632c2a86090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 -- 172.21.15.3:0/728484752 shutdown_connections 2023-09-10T01:09:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 -- 172.21.15.3:0/728484752 wait complete. 2023-09-10T01:09:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 -- start start 2023-09-10T01:09:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.763+0000 7f4f08335b80 1 Processor -- start 2023-09-10T01:09:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8b300 0x5632c2af2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2af2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4f08335b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2aecec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4f08335b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c2aed4a0 con 0x5632c2a8b300 2023-09-10T01:09:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4f08335b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c2aed610 con 0x5632c2a911a0 2023-09-10T01:09:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4f08335b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632c2aed7d0 con 0x5632c2a8bcc0 2023-09-10T01:09:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4f08335b80 10 client.0 _ll_register_callbacks cb 0x5632c28c5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8b300 0x5632c2af2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4eee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8b300 0x5632c2af2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47062/0 (socket says 172.21.15.3:47062) 2023-09-10T01:09:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4eee59c700 1 -- 172.21.15.3:0/2658409173 learned_addr learned my addr 172.21.15.3:0/2658409173 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2af2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.764+0000 7f4eeed9d700 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2aecec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4eee59c700 1 -- 172.21.15.3:0/2658409173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 msgr2=0x5632c2af2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4eee59c700 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2af2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4eee59c700 1 -- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 msgr2=0x5632c2aecec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4eee59c700 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2aecec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4eee59c700 1 -- 172.21.15.3:0/2658409173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee4017070 con 0x5632c2a8b300 2023-09-10T01:09:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4eee59c700 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8b300 0x5632c2af2710 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0096070 tx=0x7f4ee0099bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740564074bc07fec server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4ed5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:09:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/2658409173 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0095940 con 0x5632c2a8b300 2023-09-10T01:09:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ee00a1670 con 0x5632c2a8b300 2023-09-10T01:09:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/2658409173 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f4ee009d430 con 0x5632c2a8b300 2023-09-10T01:09:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/2658409173 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ee0010880 con 0x5632c2a8b300 2023-09-10T01:09:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4f08335b80 10 client.17367 fetch_fsmap learned FSMap version 28 2023-09-10T01:09:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.765+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5632c2808250 con 0x5632c2a8b300 2023-09-10T01:09:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/2658409173 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ee00a1670 con 0x5632c2a8b300 2023-09-10T01:09:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/2658409173 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4ee00cc0e0 con 0x5632c2a8b300 2023-09-10T01:09:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4f08335b80 10 client.17367 fetch_fsmap finished waiting for FSMap version 28 2023-09-10T01:09:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4f08335b80 10 client.17367 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4f08335b80 20 client.17367 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4f08335b80 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2afa0a0 0x5632c2afc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4f08335b80 4 client.17367 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:09:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5632c29d68c0 con 0x5632c2afa0a0 2023-09-10T01:09:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.766+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2afa0a0 0x5632c2afc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.767+0000 7f4ed5ffb700 10 client.17367 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.767+0000 7f4eedd9b700 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2afa0a0 0x5632c2afc4c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=342483bdc21a5f server_cookie=99951eb433d3a6c2 in_seq=0 out_seq=0 2023-09-10T01:09:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.867+0000 7f4ed5ffb700 1 -- 172.21.15.3:0/2658409173 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5632c29d68c0 con 0x5632c2afa0a0 2023-09-10T01:09:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.867+0000 7f4ed5ffb700 10 client.17367 handle_command_reply: tid=0 2023-09-10T01:09:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.867+0000 7f4f08335b80 1 client.17367 shutdown 2023-09-10T01:09:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.867+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8b300 msgr2=0x5632c2af2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8b300 0x5632c2af2710 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0096070 tx=0x7f4ee0099bd0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 shutdown_connections 2023-09-10T01:09:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5632c2afa0a0 0x5632c2afc4c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5632c2a911a0 0x5632c2aecec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632c2a8b300 0x5632c2af2710 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 --2- 172.21.15.3:0/2658409173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5632c2a8bcc0 0x5632c2af2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 >> 172.21.15.3:0/2658409173 conn(0x5632c2a00030 msgr2=0x5632c2a90810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4ed5ffb700 0 client.17367 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 shutdown_connections 2023-09-10T01:09:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 1 -- 172.21.15.3:0/2658409173 wait complete. 2023-09-10T01:09:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.868+0000 7f4f08335b80 20 client.17367 trim_cache size 0 max 16384 2023-09-10T01:09:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:09:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:09:14.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:09:14.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:09:14.878 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4ed80a07b0 msgr2=0x7f4ed80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4ed80a07b0 0x7f4ed80a2c70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4ef010e420 tx=0x7f4ee4009260 comp rx=0 tx=0).stop 2023-09-10T01:09:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00729d0 msgr2=0x7f4ef019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:14.879 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00729d0 0x7f4ef019e7f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000e9a0 tx=0x7f4ee000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 shutdown_connections 2023-09-10T01:09:14.880 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4ed80a07b0 0x7f4ed80a2c70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4ef010ce40 0x7f4ef0198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef00729d0 0x7f4ef019e7f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 --2- 172.21.15.3:0/1648921360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ef00732f0 0x7f4ef019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:14.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 >> 172.21.15.3:0/1648921360 conn(0x7f4ef00fc0b0 msgr2=0x7f4ef01048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:14.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 shutdown_connections 2023-09-10T01:09:14.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:14.869+0000 7f4ef49ff700 1 -- 172.21.15.3:0/1648921360 wait complete. 2023-09-10T01:09:15.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:15 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:09:15.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:15 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:09:15.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:15 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:09:15.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 525851 9.40 MB/sec execute 177 sec latency 39.827 ms 2023-09-10T01:09:16.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:16 smithi092 ceph-mon[134067]: pgmap v184: 129 pgs: 129 active+clean; 152 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 391 op/s 2023-09-10T01:09:16.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:16 smithi003 ceph-mon[97654]: pgmap v184: 129 pgs: 129 active+clean; 152 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 391 op/s 2023-09-10T01:09:16.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:16 smithi136 ceph-mon[133708]: pgmap v184: 129 pgs: 129 active+clean; 152 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 391 op/s 2023-09-10T01:09:16.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 527103 9.38 MB/sec execute 178 sec latency 13.265 ms 2023-09-10T01:09:17.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 529424 9.41 MB/sec execute 179 sec latency 17.036 ms 2023-09-10T01:09:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:18 smithi092 ceph-mon[134067]: pgmap v185: 129 pgs: 129 active+clean; 152 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2023-09-10T01:09:18.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:18 smithi003 ceph-mon[97654]: pgmap v185: 129 pgs: 129 active+clean; 152 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2023-09-10T01:09:18.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:18 smithi136 ceph-mon[133708]: pgmap v185: 129 pgs: 129 active+clean; 152 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 168 op/s 2023-09-10T01:09:18.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 530513 9.37 MB/sec execute 180 sec latency 38.139 ms 2023-09-10T01:09:19.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 532678 9.39 MB/sec execute 181 sec latency 73.786 ms 2023-09-10T01:09:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:20 smithi092 ceph-mon[134067]: pgmap v186: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2023-09-10T01:09:20.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:20 smithi003 ceph-mon[97654]: pgmap v186: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2023-09-10T01:09:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:20 smithi136 ceph-mon[133708]: pgmap v186: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2023-09-10T01:09:20.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 533890 9.37 MB/sec execute 182 sec latency 16.632 ms 2023-09-10T01:09:21.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 536257 9.39 MB/sec execute 183 sec latency 16.012 ms 2023-09-10T01:09:22.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:22 smithi092 ceph-mon[134067]: pgmap v187: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2023-09-10T01:09:22.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:22 smithi003 ceph-mon[97654]: pgmap v187: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2023-09-10T01:09:22.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:22 smithi136 ceph-mon[133708]: pgmap v187: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2023-09-10T01:09:22.763 INFO:tasks.workunit.client.0.smithi003.stdout: 1 537524 9.38 MB/sec execute 184 sec latency 12.477 ms 2023-09-10T01:09:23.765 INFO:tasks.workunit.client.0.smithi003.stdout: 1 539957 9.41 MB/sec execute 185 sec latency 17.856 ms 2023-09-10T01:09:24.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:24 smithi092 ceph-mon[134067]: pgmap v188: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2023-09-10T01:09:24.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:24 smithi003 ceph-mon[97654]: pgmap v188: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2023-09-10T01:09:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:24 smithi136 ceph-mon[133708]: pgmap v188: 129 pgs: 129 active+clean; 154 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 209 op/s 2023-09-10T01:09:24.765 INFO:tasks.workunit.client.0.smithi003.stdout: 1 540959 9.37 MB/sec execute 186 sec latency 32.450 ms 2023-09-10T01:09:25.765 INFO:tasks.workunit.client.0.smithi003.stdout: 1 543328 9.39 MB/sec execute 187 sec latency 23.545 ms 2023-09-10T01:09:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:26 smithi092 ceph-mon[134067]: pgmap v189: 129 pgs: 129 active+clean; 161 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 290 op/s 2023-09-10T01:09:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:26 smithi003 ceph-mon[97654]: pgmap v189: 129 pgs: 129 active+clean; 161 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 290 op/s 2023-09-10T01:09:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:26 smithi136 ceph-mon[133708]: pgmap v189: 129 pgs: 129 active+clean; 161 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 290 op/s 2023-09-10T01:09:26.765 INFO:tasks.workunit.client.0.smithi003.stdout: 1 544499 9.37 MB/sec execute 188 sec latency 16.438 ms 2023-09-10T01:09:27.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:27 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:09:27.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:27 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:09:27.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:27 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:09:27.765 INFO:tasks.workunit.client.0.smithi003.stdout: 1 546716 9.39 MB/sec execute 189 sec latency 40.823 ms 2023-09-10T01:09:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.034+0000 7fda2fdff700 1 -- 172.21.15.3:0/2215313232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda28072a30 msgr2=0x7fda28072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.034+0000 7fda2fdff700 1 --2- 172.21.15.3:0/2215313232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda28072a30 0x7fda28072e10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fda24009990 tx=0x7fda24009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.035+0000 7fda2fdff700 1 -- 172.21.15.3:0/2215313232 shutdown_connections 2023-09-10T01:09:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.035+0000 7fda2fdff700 1 --2- 172.21.15.3:0/2215313232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda2810cde0 0x7fda2810f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.035+0000 7fda2fdff700 1 --2- 172.21.15.3:0/2215313232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda28072a30 0x7fda28072e10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.035+0000 7fda2fdff700 1 --2- 172.21.15.3:0/2215313232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28073350 0x7fda2810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.035+0000 7fda2fdff700 1 -- 172.21.15.3:0/2215313232 >> 172.21.15.3:0/2215313232 conn(0x7fda280fc0b0 msgr2=0x7fda280fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.035+0000 7fda2fdff700 1 -- 172.21.15.3:0/2215313232 shutdown_connections 2023-09-10T01:09:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.035+0000 7fda2fdff700 1 -- 172.21.15.3:0/2215313232 wait complete. 2023-09-10T01:09:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.036+0000 7fda2fdff700 1 Processor -- start 2023-09-10T01:09:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.036+0000 7fda2fdff700 1 -- start start 2023-09-10T01:09:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.037+0000 7fda2fdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda28072a30 0x7fda2819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.037+0000 7fda2fdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28073350 0x7fda2819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.037+0000 7fda2fdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda2810cde0 0x7fda28198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.037+0000 7fda2fdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda28198da0 con 0x7fda28072a30 2023-09-10T01:09:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.037+0000 7fda2fdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda28198f10 con 0x7fda2810cde0 2023-09-10T01:09:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.037+0000 7fda2fdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda281990d0 con 0x7fda28073350 2023-09-10T01:09:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.037+0000 7fda2e39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda2810cde0 0x7fda28198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2d39a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28073350 0x7fda2819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2e39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda2810cde0 0x7fda28198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36666/0 (socket says 172.21.15.3:36666) 2023-09-10T01:09:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2e39c700 1 -- 172.21.15.3:0/4116875942 learned_addr learned my addr 172.21.15.3:0/4116875942 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2db9b700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda28072a30 0x7fda2819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2db9b700 1 -- 172.21.15.3:0/4116875942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28073350 msgr2=0x7fda2819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2db9b700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28073350 0x7fda2819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2db9b700 1 -- 172.21.15.3:0/4116875942 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda2810cde0 msgr2=0x7fda28198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2db9b700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda2810cde0 0x7fda28198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2db9b700 1 -- 172.21.15.3:0/4116875942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda240095f0 con 0x7fda28072a30 2023-09-10T01:09:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2e39c700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda2810cde0 0x7fda28198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:28.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.038+0000 7fda2d39a700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28073350 0x7fda2819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.039+0000 7fda2db9b700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda28072a30 0x7fda2819e6c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fda240059d0 tx=0x7fda24005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.039+0000 7fda1effd700 1 -- 172.21.15.3:0/4116875942 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2401d070 con 0x7fda28072a30 2023-09-10T01:09:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.039+0000 7fda1effd700 1 -- 172.21.15.3:0/4116875942 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda24022470 con 0x7fda28072a30 2023-09-10T01:09:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.039+0000 7fda1effd700 1 -- 172.21.15.3:0/4116875942 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24017650 con 0x7fda28072a30 2023-09-10T01:09:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.039+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda28199350 con 0x7fda28072a30 2023-09-10T01:09:28.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.039+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda281995a0 con 0x7fda28072a30 2023-09-10T01:09:28.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.043+0000 7fda1effd700 1 -- 172.21.15.3:0/4116875942 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fda24022890 con 0x7fda28072a30 2023-09-10T01:09:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.044+0000 7fda1effd700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda140a0830 0x7fda140a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.044+0000 7fda43668b80 1 Processor -- start 2023-09-10T01:09:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.044+0000 7fda1effd700 1 -- 172.21.15.3:0/4116875942 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fda240cd6d0 con 0x7fda28072a30 2023-09-10T01:09:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.044+0000 7fda2d39a700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda140a0830 0x7fda140a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.044+0000 7fda43668b80 1 -- start start 2023-09-10T01:09:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda43668b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cf7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda43668b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73cf1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda43668b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73cfdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda43668b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73c4a6d0 con 0x55ff73cf1cc0 2023-09-10T01:09:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda43668b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73c47ff0 con 0x55ff73cf71a0 2023-09-10T01:09:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda43668b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73c438e0 con 0x55ff73cf1300 2023-09-10T01:09:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cf7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cf7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36672/0 (socket says 172.21.15.3:36672) 2023-09-10T01:09:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda2db9b700 1 -- 172.21.15.3:0/4108822182 learned_addr learned my addr 172.21.15.3:0/4108822182 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda2e39c700 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73cfdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda2d39a700 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73cf1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda2d39a700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda140a0830 0x7fda140a2cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda28199ba0 tx=0x7fda18009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.045+0000 7fda2db9b700 1 -- 172.21.15.3:0/4108822182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 msgr2=0x55ff73cf1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda2db9b700 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73cf1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda2db9b700 1 -- 172.21.15.3:0/4108822182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 msgr2=0x55ff73cfdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda2db9b700 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73cfdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda2db9b700 1 -- 172.21.15.3:0/4108822182 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff73a6fd80 con 0x55ff73cf71a0 2023-09-10T01:09:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda2db9b700 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cf7580 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fda240973b0 tx=0x7fda24009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e57c25015cd5f2d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda13fff700 1 -- 172.21.15.3:0/4108822182 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2409a940 con 0x55ff73cf71a0 2023-09-10T01:09:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda13fff700 1 -- 172.21.15.3:0/4108822182 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda2409e770 con 0x55ff73cf71a0 2023-09-10T01:09:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.046+0000 7fda13fff700 1 -- 172.21.15.3:0/4108822182 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda240cd4b0 con 0x55ff73cf71a0 2023-09-10T01:09:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 -- 172.21.15.3:0/4108822182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 msgr2=0x55ff73cf7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cf7580 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fda240973b0 tx=0x7fda24009310 comp rx=0 tx=0).stop 2023-09-10T01:09:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 -- 172.21.15.3:0/4108822182 shutdown_connections 2023-09-10T01:09:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73cfdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cf7580 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 --2- 172.21.15.3:0/4108822182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73cf1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 -- 172.21.15.3:0/4108822182 >> 172.21.15.3:0/4108822182 conn(0x55ff73c66030 msgr2=0x55ff73cec560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 -- 172.21.15.3:0/4108822182 shutdown_connections 2023-09-10T01:09:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.047+0000 7fda43668b80 1 -- 172.21.15.3:0/4108822182 wait complete. 2023-09-10T01:09:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 -- start start 2023-09-10T01:09:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 Processor -- start 2023-09-10T01:09:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73d00b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73d01070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cfadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73be1b10 con 0x55ff73cf1cc0 2023-09-10T01:09:28.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73be1810 con 0x55ff73cf71a0 2023-09-10T01:09:28.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73ac58c0 con 0x55ff73cf1300 2023-09-10T01:09:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.048+0000 7fda43668b80 10 client.0 _ll_register_callbacks cb 0x55ff73b2ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73d00b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2e39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cfadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73d00b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:34082/0 (socket says 172.21.15.3:34082) 2023-09-10T01:09:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 -- 172.21.15.3:0/2873611493 learned_addr learned my addr 172.21.15.3:0/2873611493 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2d39a700 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73d01070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 -- 172.21.15.3:0/2873611493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 msgr2=0x55ff73cfadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cfadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 -- 172.21.15.3:0/2873611493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 msgr2=0x55ff73d01070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:28.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73d01070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 -- 172.21.15.3:0/2873611493 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda28199350 con 0x55ff73cf1300 2023-09-10T01:09:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2d39a700 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73d01070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda2db9b700 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73d00b30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fda240cafb0 tx=0x7fda240cd600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46eae83b34bea165 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.049+0000 7fda117fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:09:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda117fa700 1 -- 172.21.15.3:0/2873611493 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda240a2920 con 0x55ff73cf1300 2023-09-10T01:09:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ff73b9b490 con 0x55ff73cf1300 2023-09-10T01:09:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda117fa700 1 -- 172.21.15.3:0/2873611493 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fda2409a940 con 0x55ff73cf1300 2023-09-10T01:09:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda117fa700 1 -- 172.21.15.3:0/2873611493 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda2409aba0 con 0x55ff73cf1300 2023-09-10T01:09:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 10 client.27373 fetch_fsmap learned FSMap version 28 2023-09-10T01:09:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ff73a58c00 con 0x55ff73cf1300 2023-09-10T01:09:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda117fa700 1 -- 172.21.15.3:0/2873611493 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda240bc080 con 0x55ff73cf1300 2023-09-10T01:09:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda117fa700 1 -- 172.21.15.3:0/2873611493 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fda240a2490 con 0x55ff73cf1300 2023-09-10T01:09:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 10 client.27373 fetch_fsmap finished waiting for FSMap version 28 2023-09-10T01:09:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 10 client.27373 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 20 client.27373 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d0ab40 0x55ff73d0cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 4 client.27373 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.050+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff73c58840 con 0x55ff73d0ab40 2023-09-10T01:09:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.051+0000 7fda2d39a700 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d0ab40 0x55ff73d0cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.051+0000 7fda117fa700 10 client.27373 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.051+0000 7fda2d39a700 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d0ab40 0x55ff73d0cf30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=383a53151960800b server_cookie=899ee395166560c7 in_seq=0 out_seq=0 2023-09-10T01:09:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda117fa700 1 -- 172.21.15.3:0/2873611493 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ff73c58840 con 0x55ff73d0ab40 2023-09-10T01:09:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda117fa700 10 client.27373 handle_command_reply: tid=0 2023-09-10T01:09:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 client.27373 shutdown 2023-09-10T01:09:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 msgr2=0x55ff73d00b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73d00b30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fda240cafb0 tx=0x7fda240cd600 comp rx=0 tx=0).stop 2023-09-10T01:09:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 shutdown_connections 2023-09-10T01:09:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d0ab40 0x55ff73d0cf30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1300 0x55ff73d00b30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf71a0 0x55ff73cfadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda117fa700 0 client.27373 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 --2- 172.21.15.3:0/2873611493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf1cc0 0x55ff73d01070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 >> 172.21.15.3:0/2873611493 conn(0x55ff73c66030 msgr2=0x55ff73ceb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 shutdown_connections 2023-09-10T01:09:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.052+0000 7fda43668b80 1 -- 172.21.15.3:0/2873611493 wait complete. 2023-09-10T01:09:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.053+0000 7fda43668b80 20 client.27373 trim_cache size 0 max 16384 2023-09-10T01:09:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.069+0000 7fda43668b80 1 Processor -- start 2023-09-10T01:09:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda43668b80 1 -- start start 2023-09-10T01:09:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda43668b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73cfad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda43668b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d00700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda43668b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d00c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda43668b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73cf79e0 con 0x55ff73cf71a0 2023-09-10T01:09:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda43668b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73cf7580 con 0x55ff73cf1300 2023-09-10T01:09:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda43668b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73cf5c00 con 0x55ff73cf1cc0 2023-09-10T01:09:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73cfad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73cfad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36698/0 (socket says 172.21.15.3:36698) 2023-09-10T01:09:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda2db9b700 1 -- 172.21.15.3:0/973112812 learned_addr learned my addr 172.21.15.3:0/973112812 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda2d39a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d00700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.070+0000 7fda2e39c700 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d00c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda2d39a700 1 -- 172.21.15.3:0/973112812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 msgr2=0x55ff73cfad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda2d39a700 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73cfad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda2d39a700 1 -- 172.21.15.3:0/973112812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 msgr2=0x55ff73d00c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda2d39a700 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d00c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda2d39a700 1 -- 172.21.15.3:0/973112812 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff73a961f0 con 0x55ff73cf1cc0 2023-09-10T01:09:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda2db9b700 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73cfad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda2d39a700 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d00700 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55ff73d0d810 tx=0x7fda18014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf256a2e66294e33 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda1cbff700 1 -- 172.21.15.3:0/973112812 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1802e070 con 0x55ff73cf1cc0 2023-09-10T01:09:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda1cbff700 1 -- 172.21.15.3:0/973112812 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda180293b0 con 0x55ff73cf1cc0 2023-09-10T01:09:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.071+0000 7fda1cbff700 1 -- 172.21.15.3:0/973112812 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18024040 con 0x55ff73cf1cc0 2023-09-10T01:09:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.072+0000 7fda43668b80 1 -- 172.21.15.3:0/973112812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 msgr2=0x55ff73d00700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.072+0000 7fda43668b80 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d00700 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55ff73d0d810 tx=0x7fda18014000 comp rx=0 tx=0).stop 2023-09-10T01:09:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.073+0000 7fda43668b80 1 -- 172.21.15.3:0/973112812 shutdown_connections 2023-09-10T01:09:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.073+0000 7fda43668b80 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d00c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.073+0000 7fda43668b80 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73cfad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.073+0000 7fda43668b80 1 --2- 172.21.15.3:0/973112812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d00700 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.073+0000 7fda43668b80 1 -- 172.21.15.3:0/973112812 >> 172.21.15.3:0/973112812 conn(0x55ff73c66030 msgr2=0x55ff73cfa660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.074+0000 7fda43668b80 1 -- 172.21.15.3:0/973112812 shutdown_connections 2023-09-10T01:09:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.074+0000 7fda43668b80 1 -- 172.21.15.3:0/973112812 wait complete. 2023-09-10T01:09:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.074+0000 7fda43668b80 1 -- start start 2023-09-10T01:09:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 1 Processor -- start 2023-09-10T01:09:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73d60ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d61010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d5b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73d5b7c0 con 0x55ff73cf71a0 2023-09-10T01:09:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73d5b930 con 0x55ff73cf1300 2023-09-10T01:09:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff73d5baf0 con 0x55ff73cf1cc0 2023-09-10T01:09:28.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.075+0000 7fda43668b80 10 client.0 _ll_register_callbacks cb 0x55ff73b2ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73d60ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73d60ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36710/0 (socket says 172.21.15.3:36710) 2023-09-10T01:09:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 -- 172.21.15.3:0/568869045 learned_addr learned my addr 172.21.15.3:0/568869045 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2e39c700 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d5b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 -- 172.21.15.3:0/568869045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 msgr2=0x55ff73d61010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:09:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d61010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 -- 172.21.15.3:0/568869045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 msgr2=0x55ff73d5b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d5b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 -- 172.21.15.3:0/568869045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff73a58c00 con 0x55ff73cf1300 2023-09-10T01:09:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2e39c700 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d5b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda2db9b700 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73d60ad0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fda24006010 tx=0x7fda2409adb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ce8e06226481dd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.076+0000 7fda117fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda117fa700 1 -- 172.21.15.3:0/568869045 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2409a480 con 0x55ff73cf1300 2023-09-10T01:09:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda2409e770 con 0x55ff73cf1300 2023-09-10T01:09:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda117fa700 1 -- 172.21.15.3:0/568869045 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fda2409a940 con 0x55ff73cf1300 2023-09-10T01:09:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda117fa700 1 -- 172.21.15.3:0/568869045 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda24096ae0 con 0x55ff73cf1300 2023-09-10T01:09:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda43668b80 10 client.27467 fetch_fsmap learned FSMap version 28 2023-09-10T01:09:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fda18017070 con 0x55ff73cf1300 2023-09-10T01:09:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda117fa700 1 -- 172.21.15.3:0/568869045 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda2409e770 con 0x55ff73cf1300 2023-09-10T01:09:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda117fa700 1 -- 172.21.15.3:0/568869045 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fda240aa070 con 0x55ff73cf1300 2023-09-10T01:09:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda43668b80 10 client.27467 fetch_fsmap finished waiting for FSMap version 28 2023-09-10T01:09:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda43668b80 10 client.27467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.077+0000 7fda43668b80 20 client.27467 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.078+0000 7fda43668b80 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d683f0 0x55ff73d6a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.078+0000 7fda43668b80 4 client.27467 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:09:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.078+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ff73c3c8c0 con 0x55ff73d683f0 2023-09-10T01:09:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.078+0000 7fda2d39a700 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d683f0 0x55ff73d6a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.078+0000 7fda2d39a700 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d683f0 0x55ff73d6a810 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93742b5811c4f23b server_cookie=f29987f0b0f52800 in_seq=0 out_seq=0 2023-09-10T01:09:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.078+0000 7fda117fa700 10 client.27467 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.179+0000 7fda117fa700 1 -- 172.21.15.3:0/568869045 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ff73c3c8c0 con 0x55ff73d683f0 2023-09-10T01:09:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.179+0000 7fda117fa700 10 client.27467 handle_command_reply: tid=0 2023-09-10T01:09:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.179+0000 7fda43668b80 1 client.27467 shutdown 2023-09-10T01:09:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.179+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 msgr2=0x55ff73d60ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.179+0000 7fda43668b80 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73d60ad0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fda24006010 tx=0x7fda2409adb0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 shutdown_connections 2023-09-10T01:09:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ff73d683f0 0x55ff73d6a810 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff73cf71a0 0x55ff73d5b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff73cf1300 0x55ff73d60ad0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 --2- 172.21.15.3:0/568869045 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ff73cf1cc0 0x55ff73d61010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda117fa700 0 client.27467 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 >> 172.21.15.3:0/568869045 conn(0x55ff73c66030 msgr2=0x55ff73cf68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 shutdown_connections 2023-09-10T01:09:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.180+0000 7fda43668b80 1 -- 172.21.15.3:0/568869045 wait complete. 2023-09-10T01:09:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.184+0000 7fda43668b80 20 client.27467 trim_cache size 0 max 16384 2023-09-10T01:09:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:09:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:09:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:09:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:09:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.188+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda140a0830 msgr2=0x7fda140a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.188+0000 7fda2fdff700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda140a0830 0x7fda140a2cf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda28199ba0 tx=0x7fda18009260 comp rx=0 tx=0).stop 2023-09-10T01:09:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.188+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda28072a30 msgr2=0x7fda2819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.188+0000 7fda2fdff700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda28072a30 0x7fda2819e6c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fda240059d0 tx=0x7fda24005740 comp rx=0 tx=0).stop 2023-09-10T01:09:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.189+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 shutdown_connections 2023-09-10T01:09:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.189+0000 7fda2fdff700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fda140a0830 0x7fda140a2cf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.189+0000 7fda2fdff700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda2810cde0 0x7fda28198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.189+0000 7fda2fdff700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda28072a30 0x7fda2819e6c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.189+0000 7fda2fdff700 1 --2- 172.21.15.3:0/4116875942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28073350 0x7fda2819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.189+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 >> 172.21.15.3:0/4116875942 conn(0x7fda280fc0b0 msgr2=0x7fda2810efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:28.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.190+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 shutdown_connections 2023-09-10T01:09:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:28.190+0000 7fda2fdff700 1 -- 172.21.15.3:0/4116875942 wait complete. 2023-09-10T01:09:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:28 smithi092 ceph-mon[134067]: pgmap v190: 129 pgs: 129 active+clean; 161 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 209 op/s 2023-09-10T01:09:28.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:28 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:09:28.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:28 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:09:28.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:28 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:09:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:28 smithi003 ceph-mon[97654]: pgmap v190: 129 pgs: 129 active+clean; 161 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 209 op/s 2023-09-10T01:09:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:28 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:09:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:28 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:09:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:28 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:09:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:28 smithi136 ceph-mon[133708]: pgmap v190: 129 pgs: 129 active+clean; 161 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 209 op/s 2023-09-10T01:09:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:28 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:09:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:28 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:09:28.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:28 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:09:28.765 INFO:tasks.workunit.client.0.smithi003.stdout: 1 547814 9.37 MB/sec execute 190 sec latency 92.791 ms 2023-09-10T01:09:29.766 INFO:tasks.workunit.client.0.smithi003.stdout: 1 549679 9.36 MB/sec execute 191 sec latency 33.542 ms 2023-09-10T01:09:30.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:30 smithi092 ceph-mon[134067]: pgmap v191: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 334 op/s 2023-09-10T01:09:30.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:30 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:09:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:30 smithi003 ceph-mon[97654]: pgmap v191: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 334 op/s 2023-09-10T01:09:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:30 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:09:30.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:30 smithi136 ceph-mon[133708]: pgmap v191: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 334 op/s 2023-09-10T01:09:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:30 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:09:30.766 INFO:tasks.workunit.client.0.smithi003.stdout: 1 551295 9.37 MB/sec execute 192 sec latency 18.283 ms 2023-09-10T01:09:31.766 INFO:tasks.workunit.client.0.smithi003.stdout: 1 552592 9.35 MB/sec execute 193 sec latency 192.734 ms 2023-09-10T01:09:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:32 smithi092 ceph-mon[134067]: pgmap v192: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2023-09-10T01:09:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:32 smithi003 ceph-mon[97654]: pgmap v192: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2023-09-10T01:09:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:32 smithi136 ceph-mon[133708]: pgmap v192: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2023-09-10T01:09:32.767 INFO:tasks.workunit.client.0.smithi003.stdout: 1 553823 9.35 MB/sec execute 194 sec latency 367.774 ms 2023-09-10T01:09:33.768 INFO:tasks.workunit.client.0.smithi003.stdout: 1 554754 9.33 MB/sec execute 195 sec latency 30.464 ms 2023-09-10T01:09:34.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:34 smithi092 ceph-mon[134067]: pgmap v193: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2023-09-10T01:09:34.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:34 smithi003 ceph-mon[97654]: pgmap v193: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2023-09-10T01:09:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:34 smithi136 ceph-mon[133708]: pgmap v193: 129 pgs: 129 active+clean; 156 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 206 op/s 2023-09-10T01:09:34.767 INFO:tasks.workunit.client.0.smithi003.stdout: 1 556568 9.32 MB/sec execute 196 sec latency 16.322 ms 2023-09-10T01:09:35.768 INFO:tasks.workunit.client.0.smithi003.stdout: 1 558132 9.33 MB/sec execute 197 sec latency 33.352 ms 2023-09-10T01:09:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:36 smithi092 ceph-mon[134067]: pgmap v194: 129 pgs: 129 active+clean; 157 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2023-09-10T01:09:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:36 smithi003 ceph-mon[97654]: pgmap v194: 129 pgs: 129 active+clean; 157 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2023-09-10T01:09:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:36 smithi136 ceph-mon[133708]: pgmap v194: 129 pgs: 129 active+clean; 157 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 433 op/s 2023-09-10T01:09:36.768 INFO:tasks.workunit.client.0.smithi003.stdout: 1 559481 9.31 MB/sec execute 198 sec latency 73.290 ms 2023-09-10T01:09:37.768 INFO:tasks.workunit.client.0.smithi003.stdout: 1 561639 9.33 MB/sec execute 199 sec latency 15.470 ms 2023-09-10T01:09:38.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:38 smithi092 ceph-mon[134067]: pgmap v195: 129 pgs: 129 active+clean; 157 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 351 op/s 2023-09-10T01:09:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:38 smithi003 ceph-mon[97654]: pgmap v195: 129 pgs: 129 active+clean; 157 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 351 op/s 2023-09-10T01:09:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:38 smithi136 ceph-mon[133708]: pgmap v195: 129 pgs: 129 active+clean; 157 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 351 op/s 2023-09-10T01:09:38.768 INFO:tasks.workunit.client.0.smithi003.stdout: 1 563115 9.31 MB/sec execute 200 sec latency 17.970 ms 2023-09-10T01:09:39.768 INFO:tasks.workunit.client.0.smithi003.stdout: 1 565166 9.33 MB/sec execute 201 sec latency 60.853 ms 2023-09-10T01:09:40.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:40 smithi092 ceph-mon[134067]: pgmap v196: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 431 op/s 2023-09-10T01:09:40.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:40 smithi003 ceph-mon[97654]: pgmap v196: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 431 op/s 2023-09-10T01:09:40.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:40 smithi136 ceph-mon[133708]: pgmap v196: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 431 op/s 2023-09-10T01:09:40.769 INFO:tasks.workunit.client.0.smithi003.stdout: 1 566569 9.31 MB/sec execute 202 sec latency 41.165 ms 2023-09-10T01:09:40.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:09:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.108+0000 7fa71d9ff700 1 -- 172.21.15.3:0/1078862215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 msgr2=0x7fa71810d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.108+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/1078862215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa71810d520 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa71000b0a0 tx=0x7fa71000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.108+0000 7fa71d9ff700 1 -- 172.21.15.3:0/1078862215 shutdown_connections 2023-09-10T01:09:41.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.108+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/1078862215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa71810d520 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.108+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/1078862215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa718106660 0x7fa718106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.108+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/1078862215 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa718100650 0x7fa718100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.108+0000 7fa71d9ff700 1 -- 172.21.15.3:0/1078862215 >> 172.21.15.3:0/1078862215 conn(0x7fa7180fc160 msgr2=0x7fa7180fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.109+0000 7fa71d9ff700 1 -- 172.21.15.3:0/1078862215 shutdown_connections 2023-09-10T01:09:41.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.109+0000 7fa71d9ff700 1 -- 172.21.15.3:0/1078862215 wait complete. 2023-09-10T01:09:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.110+0000 7fa71d9ff700 1 Processor -- start 2023-09-10T01:09:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.110+0000 7fa71d9ff700 1 -- start start 2023-09-10T01:09:41.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa71d9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa718100650 0x7fa718072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa71d9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa718072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa71d9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa718106660 0x7fa7181a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa71d9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa718073670 con 0x7fa718100650 2023-09-10T01:09:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa71d9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7181a4e80 con 0x7fa718101010 2023-09-10T01:09:41.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa71d9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7181a4fc0 con 0x7fa718106660 2023-09-10T01:09:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa718072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa718100650 0x7fa718072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa718072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49718/0 (socket says 172.21.15.3:49718) 2023-09-10T01:09:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa7167fc700 1 -- 172.21.15.3:0/2652233310 learned_addr learned my addr 172.21.15.3:0/2652233310 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.111+0000 7fa7177fe700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa718106660 0x7fa7181a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa716ffd700 1 -- 172.21.15.3:0/2652233310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa718106660 msgr2=0x7fa7181a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa716ffd700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa718106660 0x7fa7181a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa716ffd700 1 -- 172.21.15.3:0/2652233310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 msgr2=0x7fa718072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa716ffd700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa718072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa716ffd700 1 -- 172.21.15.3:0/2652233310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa710009d00 con 0x7fa718100650 2023-09-10T01:09:41.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa718072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:41.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa7177fe700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa718106660 0x7fa7181a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:41.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa716ffd700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa718100650 0x7fa718072900 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa70800e9f0 tx=0x7fa70800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa71c9fd700 1 -- 172.21.15.3:0/2652233310 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70800c9c0 con 0x7fa718100650 2023-09-10T01:09:41.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa71c9fd700 1 -- 172.21.15.3:0/2652233310 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa708010430 con 0x7fa718100650 2023-09-10T01:09:41.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7181a5220 con 0x7fa718100650 2023-09-10T01:09:41.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.112+0000 7fa71c9fd700 1 -- 172.21.15.3:0/2652233310 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708018680 con 0x7fa718100650 2023-09-10T01:09:41.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.113+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7181a5700 con 0x7fa718100650 2023-09-10T01:09:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.116+0000 7fa71c9fd700 1 -- 172.21.15.3:0/2652233310 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa708010860 con 0x7fa718100650 2023-09-10T01:09:41.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.117+0000 7fa71c9fd700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa7000a07b0 0x7fa7000a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.117+0000 7fa731294b80 1 Processor -- start 2023-09-10T01:09:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.117+0000 7fa71c9fd700 1 -- 172.21.15.3:0/2652233310 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa708014070 con 0x7fa718100650 2023-09-10T01:09:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa731294b80 1 -- start start 2023-09-10T01:09:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa7000a07b0 0x7fa7000a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa731294b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa731294b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa731294b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa731294b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55716350e730 con 0x5571635bfe40 2023-09-10T01:09:41.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa731294b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55716350c050 con 0x5571635b6030 2023-09-10T01:09:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa731294b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557163507940 con 0x5571635b5710 2023-09-10T01:09:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa7177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49724/0 (socket says 172.21.15.3:49724) 2023-09-10T01:09:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa7167fc700 1 -- 172.21.15.3:0/3426087053 learned_addr learned my addr 172.21.15.3:0/3426087053 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.118+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa7000a07b0 0x7fa7000a2c70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7180740f0 tx=0x7fa70c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7177fe700 1 -- 172.21.15.3:0/3426087053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 msgr2=0x5571635b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7177fe700 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7177fe700 1 -- 172.21.15.3:0/3426087053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 msgr2=0x5571635bf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7177fe700 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7177fe700 1 -- 172.21.15.3:0/3426087053 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557163333de0 con 0x5571635bfe40 2023-09-10T01:09:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7167fc700 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa7177fe700 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635c2230 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa7100062a0 tx=0x7fa710009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e435747f521465ee server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.119+0000 7fa6fdffb700 1 -- 172.21.15.3:0/3426087053 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71000e040 con 0x5571635bfe40 2023-09-10T01:09:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa6fdffb700 1 -- 172.21.15.3:0/3426087053 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa71000bd30 con 0x5571635bfe40 2023-09-10T01:09:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa6fdffb700 1 -- 172.21.15.3:0/3426087053 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7100052a0 con 0x5571635bfe40 2023-09-10T01:09:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa731294b80 1 -- 172.21.15.3:0/3426087053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 msgr2=0x5571635c2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa731294b80 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635c2230 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa7100062a0 tx=0x7fa710009470 comp rx=0 tx=0).stop 2023-09-10T01:09:41.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa731294b80 1 -- 172.21.15.3:0/3426087053 shutdown_connections 2023-09-10T01:09:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa731294b80 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa731294b80 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635c2230 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa731294b80 1 --2- 172.21.15.3:0/3426087053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.120+0000 7fa731294b80 1 -- 172.21.15.3:0/3426087053 >> 172.21.15.3:0/3426087053 conn(0x55716352a090 msgr2=0x5571635b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.121+0000 7fa731294b80 1 -- 172.21.15.3:0/3426087053 shutdown_connections 2023-09-10T01:09:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.121+0000 7fa731294b80 1 -- 172.21.15.3:0/3426087053 wait complete. 2023-09-10T01:09:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.121+0000 7fa731294b80 1 -- start start 2023-09-10T01:09:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 1 Processor -- start 2023-09-10T01:09:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635bf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b6030 0x5571635bca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635bfe40 0x5571635bcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571634a5b70 con 0x5571635b5710 2023-09-10T01:09:41.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571634a5870 con 0x5571635bfe40 2023-09-10T01:09:41.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557163389920 con 0x5571635b6030 2023-09-10T01:09:41.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa731294b80 10 client.0 _ll_register_callbacks cb 0x5571633efa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:41.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b6030 0x5571635bca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635bf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635bf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35970/0 (socket says 172.21.15.3:35970) 2023-09-10T01:09:41.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa7177fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635bfe40 0x5571635bcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa716ffd700 1 -- 172.21.15.3:0/2659955582 learned_addr learned my addr 172.21.15.3:0/2659955582 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.122+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b6030 0x5571635bca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:59226/0 (socket says 172.21.15.3:59226) 2023-09-10T01:09:41.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa7167fc700 1 -- 172.21.15.3:0/2659955582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635bfe40 msgr2=0x5571635bcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635bfe40 0x5571635bcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa7167fc700 1 -- 172.21.15.3:0/2659955582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 msgr2=0x5571635bf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635bf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa7167fc700 1 -- 172.21.15.3:0/2659955582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c017050 con 0x5571635b6030 2023-09-10T01:09:41.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b6030 0x5571635bca50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa70c011cc0 tx=0x7fa70c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=509518c416c6cde9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa6f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:09:41.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/2659955582 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c00f650 con 0x5571635b6030 2023-09-10T01:09:41.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55716345f4f0 con 0x5571635b6030 2023-09-10T01:09:41.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.123+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/2659955582 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa70c00f7b0 con 0x5571635b6030 2023-09-10T01:09:41.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/2659955582 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa70c00f9b0 con 0x5571635b6030 2023-09-10T01:09:41.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 10 client.27388 fetch_fsmap learned FSMap version 29 2023-09-10T01:09:41.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fa71810de20 con 0x5571635b6030 2023-09-10T01:09:41.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/2659955582 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55716345f4f0 con 0x5571635b6030 2023-09-10T01:09:41.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/2659955582 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa70c005140 con 0x5571635b6030 2023-09-10T01:09:41.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 10 client.27388 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:09:41.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 10 client.27388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:41.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 20 client.27388 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:41.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5571635cf960 0x5571635d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 4 client.27388 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:41.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55716351c8a0 con 0x5571635cf960 2023-09-10T01:09:41.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.124+0000 7fa716ffd700 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5571635cf960 0x5571635d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.125+0000 7fa716ffd700 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5571635cf960 0x5571635d1d80 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecbc1078aace314c server_cookie=73a6cc9648e9e5e1 in_seq=0 out_seq=0 2023-09-10T01:09:41.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.125+0000 7fa6f6ffd700 10 client.27388 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/2659955582 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55716351c8a0 con 0x5571635cf960 2023-09-10T01:09:41.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa6f6ffd700 10 client.27388 handle_command_reply: tid=0 2023-09-10T01:09:41.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 client.27388 shutdown 2023-09-10T01:09:41.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b6030 msgr2=0x5571635bca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b6030 0x5571635bca50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa70c011cc0 tx=0x7fa70c0047c0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 shutdown_connections 2023-09-10T01:09:41.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5571635cf960 0x5571635d1d80 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635bfe40 0x5571635bcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa6f6ffd700 0 client.27388 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635bf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 --2- 172.21.15.3:0/2659955582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b6030 0x5571635bca50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 >> 172.21.15.3:0/2659955582 conn(0x55716352a090 msgr2=0x5571635b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 shutdown_connections 2023-09-10T01:09:41.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 1 -- 172.21.15.3:0/2659955582 wait complete. 2023-09-10T01:09:41.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.126+0000 7fa731294b80 20 client.27388 trim_cache size 0 max 16384 2023-09-10T01:09:41.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 Processor -- start 2023-09-10T01:09:41.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 -- start start 2023-09-10T01:09:41.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635c6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571635c6ed0 con 0x5571635bfe40 2023-09-10T01:09:41.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571635bb000 con 0x5571635b6030 2023-09-10T01:09:41.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa731294b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571635b16e0 con 0x5571635b5710 2023-09-10T01:09:41.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49734/0 (socket says 172.21.15.3:49734) 2023-09-10T01:09:41.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa7167fc700 1 -- 172.21.15.3:0/2159367625 learned_addr learned my addr 172.21.15.3:0/2159367625 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa716ffd700 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.144+0000 7fa7177fe700 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa7167fc700 1 -- 172.21.15.3:0/2159367625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 msgr2=0x5571635c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa7167fc700 1 -- 172.21.15.3:0/2159367625 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 msgr2=0x5571635b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635b7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa7167fc700 1 -- 172.21.15.3:0/2159367625 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55716331cc60 con 0x5571635b6030 2023-09-10T01:09:41.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa7167fc700 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf300 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009880 tx=0x7fa70c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39dce4d2c2f2dca5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa6fe7fc700 1 -- 172.21.15.3:0/2159367625 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c005730 con 0x5571635b6030 2023-09-10T01:09:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa6fe7fc700 1 -- 172.21.15.3:0/2159367625 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa70c00fb10 con 0x5571635b6030 2023-09-10T01:09:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.145+0000 7fa6fe7fc700 1 -- 172.21.15.3:0/2159367625 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c037a00 con 0x5571635b6030 2023-09-10T01:09:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 -- 172.21.15.3:0/2159367625 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 msgr2=0x5571635bf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf300 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009880 tx=0x7fa70c005dc0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 -- 172.21.15.3:0/2159367625 shutdown_connections 2023-09-10T01:09:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635b5710 0x5571635c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635bfe40 0x5571635b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 --2- 172.21.15.3:0/2159367625 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635bf300 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 -- 172.21.15.3:0/2159367625 >> 172.21.15.3:0/2159367625 conn(0x55716352a090 msgr2=0x5571635cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 -- 172.21.15.3:0/2159367625 shutdown_connections 2023-09-10T01:09:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.146+0000 7fa731294b80 1 -- 172.21.15.3:0/2159367625 wait complete. 2023-09-10T01:09:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.147+0000 7fa731294b80 1 -- start start 2023-09-10T01:09:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.147+0000 7fa731294b80 1 Processor -- start 2023-09-10T01:09:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa731294b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa731294b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635c5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa731294b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635bfe40 0x5571635c5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa731294b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571635c6d40 con 0x5571635b5710 2023-09-10T01:09:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa731294b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557163623760 con 0x5571635b6030 2023-09-10T01:09:41.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa731294b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557163623920 con 0x5571635bfe40 2023-09-10T01:09:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa731294b80 10 client.0 _ll_register_callbacks cb 0x7fa70c01c7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49738/0 (socket says 172.21.15.3:49738) 2023-09-10T01:09:41.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa716ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35982/0 (socket says 172.21.15.3:35982) 2023-09-10T01:09:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 -- 172.21.15.3:0/3680765818 learned_addr learned my addr 172.21.15.3:0/3680765818 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7177fe700 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635bfe40 0x5571635c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 -- 172.21.15.3:0/3680765818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635bfe40 msgr2=0x5571635c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635bfe40 0x5571635c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 -- 172.21.15.3:0/3680765818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 msgr2=0x5571635c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa7167fc700 1 -- 172.21.15.3:0/3680765818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71810de20 con 0x5571635b6030 2023-09-10T01:09:41.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.148+0000 7fa716ffd700 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:41.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa7167fc700 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635c5230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa70c011460 tx=0x7fa70c005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2912b8ee740c2ddf server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa6f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/3680765818 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c00f950 con 0x5571635b6030 2023-09-10T01:09:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa71000bd30 con 0x5571635b6030 2023-09-10T01:09:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/3680765818 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fa70c037760 con 0x5571635b6030 2023-09-10T01:09:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/3680765818 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa70c037920 con 0x5571635b6030 2023-09-10T01:09:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa731294b80 10 client.27485 fetch_fsmap learned FSMap version 29 2023-09-10T01:09:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55716335a250 con 0x5571635b6030 2023-09-10T01:09:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.149+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/3680765818 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa71000bd30 con 0x5571635b6030 2023-09-10T01:09:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/3680765818 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa70c040d10 con 0x5571635b6030 2023-09-10T01:09:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa731294b80 10 client.27485 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:09:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa731294b80 10 client.27485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa731294b80 20 client.27485 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa731294b80 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557163624d00 0x557163627120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa731294b80 4 client.27485 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:09:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557163500920 con 0x557163624d00 2023-09-10T01:09:41.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa716ffd700 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557163624d00 0x557163627120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa716ffd700 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557163624d00 0x557163627120 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d1263a75051f194 server_cookie=529c47c659e8ae37 in_seq=0 out_seq=0 2023-09-10T01:09:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.150+0000 7fa6f6ffd700 10 client.27485 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa6f6ffd700 1 -- 172.21.15.3:0/3680765818 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x557163500920 con 0x557163624d00 2023-09-10T01:09:41.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa6f6ffd700 10 client.27485 handle_command_reply: tid=0 2023-09-10T01:09:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 client.27485 shutdown 2023-09-10T01:09:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 msgr2=0x5571635c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635c5230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa70c011460 tx=0x7fa70c005750 comp rx=0 tx=0).stop 2023-09-10T01:09:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 shutdown_connections 2023-09-10T01:09:41.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557163624d00 0x557163627120 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571635bfe40 0x5571635c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5571635b5710 0x5571635c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa6f6ffd700 0 client.27485 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 --2- 172.21.15.3:0/3680765818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5571635b6030 0x5571635c5230 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 >> 172.21.15.3:0/3680765818 conn(0x55716352a090 msgr2=0x5571635b7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 shutdown_connections 2023-09-10T01:09:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 1 -- 172.21.15.3:0/3680765818 wait complete. 2023-09-10T01:09:41.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.151+0000 7fa731294b80 20 client.27485 trim_cache size 0 max 16384 2023-09-10T01:09:41.168 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:09:41.168 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:09:41.168 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:09:41.168 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:09:41.168 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:09:41.169 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:09:41.169 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:09:41.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:09:41.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:09:41.169 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:09:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa7000a07b0 msgr2=0x7fa7000a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa7000a07b0 0x7fa7000a2c70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7180740f0 tx=0x7fa70c009260 comp rx=0 tx=0).stop 2023-09-10T01:09:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa718100650 msgr2=0x7fa718072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa718100650 0x7fa718072900 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa70800e9f0 tx=0x7fa70800ed00 comp rx=0 tx=0).stop 2023-09-10T01:09:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 shutdown_connections 2023-09-10T01:09:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa7000a07b0 0x7fa7000a2c70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa718106660 0x7fa7181a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa718100650 0x7fa718072900 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 --2- 172.21.15.3:0/2652233310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa718101010 0x7fa718072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 >> 172.21.15.3:0/2652233310 conn(0x7fa7180fc160 msgr2=0x7fa718104160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 shutdown_connections 2023-09-10T01:09:41.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:09:41.152+0000 7fa71d9ff700 1 -- 172.21.15.3:0/2652233310 wait complete. 2023-09-10T01:09:41.173 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:09:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.354+0000 7fc0cffff700 1 -- 172.21.15.3:0/3889938899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 msgr2=0x7fc0c8106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.354+0000 7fc0cffff700 1 --2- 172.21.15.3:0/3889938899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 0x7fc0c8106a80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009990 tx=0x7fc0c4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.355+0000 7fc0cffff700 1 -- 172.21.15.3:0/3889938899 shutdown_connections 2023-09-10T01:09:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.355+0000 7fc0cffff700 1 --2- 172.21.15.3:0/3889938899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8068520 0x7fc0c810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.355+0000 7fc0cffff700 1 --2- 172.21.15.3:0/3889938899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 0x7fc0c8106a80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.355+0000 7fc0cffff700 1 --2- 172.21.15.3:0/3889938899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0c8067b60 0x7fc0c8067fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.355+0000 7fc0cffff700 1 -- 172.21.15.3:0/3889938899 >> 172.21.15.3:0/3889938899 conn(0x7fc0c8074fe0 msgr2=0x7fc0c80753f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.355+0000 7fc0cffff700 1 -- 172.21.15.3:0/3889938899 shutdown_connections 2023-09-10T01:09:41.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.355+0000 7fc0cffff700 1 -- 172.21.15.3:0/3889938899 wait complete. 2023-09-10T01:09:41.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 Processor -- start 2023-09-10T01:09:41.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 -- start start 2023-09-10T01:09:41.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8067b60 0x7fc0c81966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0c8068520 0x7fc0c8196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 0x7fc0c8199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c819a5a0 con 0x7fc0c8068520 2023-09-10T01:09:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c819a710 con 0x7fc0c81066a0 2023-09-10T01:09:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.356+0000 7fc0cffff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c819a8d0 con 0x7fc0c8067b60 2023-09-10T01:09:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8067b60 0x7fc0c81966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8067b60 0x7fc0c81966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:59264/0 (socket says 172.21.15.3:59264) 2023-09-10T01:09:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/1640265165 learned_addr learned my addr 172.21.15.3:0/1640265165 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0c8068520 0x7fc0c8196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 0x7fc0c8199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/1640265165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 msgr2=0x7fc0c8199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 0x7fc0c8199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0c8068520 msgr2=0x7fc0c8196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0c8068520 0x7fc0c8196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/1640265165 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c40095f0 con 0x7fc0c8067b60 2023-09-10T01:09:41.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.357+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8067b60 0x7fc0c81966b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4005910 tx=0x7fc0c4005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.358+0000 7fc0beffd700 1 -- 172.21.15.3:0/1640265165 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c401d070 con 0x7fc0c8067b60 2023-09-10T01:09:41.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.358+0000 7fc0beffd700 1 -- 172.21.15.3:0/1640265165 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0c4022470 con 0x7fc0c8067b60 2023-09-10T01:09:41.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.358+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c819ab50 con 0x7fc0c8067b60 2023-09-10T01:09:41.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.358+0000 7fc0beffd700 1 -- 172.21.15.3:0/1640265165 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4017650 con 0x7fc0c8067b60 2023-09-10T01:09:41.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.358+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c819ae20 con 0x7fc0c8067b60 2023-09-10T01:09:41.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.361+0000 7fc0beffd700 1 -- 172.21.15.3:0/1640265165 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fc0c400fad0 con 0x7fc0c8067b60 2023-09-10T01:09:41.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.363+0000 7fc0e390eb80 1 Processor -- start 2023-09-10T01:09:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.363+0000 7fc0e390eb80 1 -- start start 2023-09-10T01:09:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0beffd700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0b40a07b0 0x7fc0b40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679e300 0x56220679e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679ecc0 0x5622067aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0beffd700 1 -- 172.21.15.3:0/1640265165 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc0c40cd980 con 0x7fc0c8067b60 2023-09-10T01:09:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622066f76d0 con 0x56220679e300 2023-09-10T01:09:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622066f4ff0 con 0x5622067a41a0 2023-09-10T01:09:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622066f08e0 con 0x56220679ecc0 2023-09-10T01:09:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0b40a07b0 0x7fc0b40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0cd59a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679e300 0x56220679e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49768/0 (socket says 172.21.15.3:49768) 2023-09-10T01:09:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/334582430 learned_addr learned my addr 172.21.15.3:0/334582430 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.364+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679ecc0 0x5622067aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/334582430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679ecc0 msgr2=0x5622067aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679ecc0 0x5622067aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/334582430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679e300 msgr2=0x56220679e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679e300 0x56220679e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/334582430 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56220651cd80 con 0x5622067a41a0 2023-09-10T01:09:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a4580 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc0c409ab80 tx=0x7fc0c409a7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26b7520f7ffc51fe server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0b40a07b0 0x7fc0b40a2c70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009550 tx=0x7fc0c0003a70 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0b3fff700 1 -- 172.21.15.3:0/334582430 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c409abf0 con 0x5622067a41a0 2023-09-10T01:09:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0b3fff700 1 -- 172.21.15.3:0/334582430 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0c409ea20 con 0x5622067a41a0 2023-09-10T01:09:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.365+0000 7fc0b3fff700 1 -- 172.21.15.3:0/334582430 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c409ae50 con 0x5622067a41a0 2023-09-10T01:09:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 -- 172.21.15.3:0/334582430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 msgr2=0x5622067a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a4580 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc0c409ab80 tx=0x7fc0c409a7b0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 -- 172.21.15.3:0/334582430 shutdown_connections 2023-09-10T01:09:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679ecc0 0x5622067aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a4580 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/334582430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679e300 0x56220679e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 -- 172.21.15.3:0/334582430 >> 172.21.15.3:0/334582430 conn(0x562206713030 msgr2=0x562206799560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 -- 172.21.15.3:0/334582430 shutdown_connections 2023-09-10T01:09:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.366+0000 7fc0e390eb80 1 -- 172.21.15.3:0/334582430 wait complete. 2023-09-10T01:09:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 -- start start 2023-09-10T01:09:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 Processor -- start 2023-09-10T01:09:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56220679ecc0 0x5622067b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5622067a41a0 0x5622067ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56220668eb10 con 0x5622067a41a0 2023-09-10T01:09:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56220668e810 con 0x56220679ecc0 2023-09-10T01:09:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622065728c0 con 0x56220679e300 2023-09-10T01:09:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.367+0000 7fc0e390eb80 10 client.0 _ll_register_callbacks cb 0x5622065d8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56220679ecc0 0x5622067b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56220679ecc0 0x5622067b2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49774/0 (socket says 172.21.15.3:49774) 2023-09-10T01:09:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:59284/0 (socket says 172.21.15.3:59284) 2023-09-10T01:09:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 -- 172.21.15.3:0/1830119969 learned_addr learned my addr 172.21.15.3:0/1830119969 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5622067a41a0 0x5622067ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 -- 172.21.15.3:0/1830119969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 msgr2=0x5622067b1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 -- 172.21.15.3:0/1830119969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5622067a41a0 msgr2=0x5622067ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5622067a41a0 0x5622067ac1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 -- 172.21.15.3:0/1830119969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c819ae20 con 0x56220679ecc0 2023-09-10T01:09:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56220679ecc0 0x5622067b2490 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8011740 tx=0x7fc0b8011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db05bf3fc08a79a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0b17fa700 1 -- 172.21.15.3:0/1830119969 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8016070 con 0x56220679ecc0 2023-09-10T01:09:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.368+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562206648490 con 0x56220679ecc0 2023-09-10T01:09:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0b17fa700 1 -- 172.21.15.3:0/1830119969 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc0b800f7b0 con 0x56220679ecc0 2023-09-10T01:09:41.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0b17fa700 1 -- 172.21.15.3:0/1830119969 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0b800f9b0 con 0x56220679ecc0 2023-09-10T01:09:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 10 client.27497 fetch_fsmap learned FSMap version 29 2023-09-10T01:09:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x562206505c00 con 0x56220679ecc0 2023-09-10T01:09:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0b17fa700 1 -- 172.21.15.3:0/1830119969 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562206648490 con 0x56220679ecc0 2023-09-10T01:09:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0b17fa700 1 -- 172.21.15.3:0/1830119969 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc0b801ac00 con 0x56220679ecc0 2023-09-10T01:09:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 10 client.27497 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:09:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 10 client.27497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 20 client.27497 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5622067bc590 0x5622067be9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 4 client.27497 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.369+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562206705840 con 0x5622067bc590 2023-09-10T01:09:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.370+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5622067bc590 0x5622067be9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.370+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5622067bc590 0x5622067be9b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9df756973ad0fbdf server_cookie=572cfc7d3836d68 in_seq=0 out_seq=0 2023-09-10T01:09:41.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.370+0000 7fc0b17fa700 10 client.27497 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.371+0000 7fc0b17fa700 1 -- 172.21.15.3:0/1830119969 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562206705840 con 0x5622067bc590 2023-09-10T01:09:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.371+0000 7fc0b17fa700 10 client.27497 handle_command_reply: tid=0 2023-09-10T01:09:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.371+0000 7fc0e390eb80 1 client.27497 shutdown 2023-09-10T01:09:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.371+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56220679ecc0 msgr2=0x5622067b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.371+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56220679ecc0 0x5622067b2490 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8011740 tx=0x7fc0b8011a50 comp rx=0 tx=0).stop 2023-09-10T01:09:41.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 shutdown_connections 2023-09-10T01:09:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5622067bc590 0x5622067be9b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5622067a41a0 0x5622067ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0b17fa700 0 client.27497 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/1830119969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56220679ecc0 0x5622067b2490 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 >> 172.21.15.3:0/1830119969 conn(0x562206713030 msgr2=0x5622067987a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 shutdown_connections 2023-09-10T01:09:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 1 -- 172.21.15.3:0/1830119969 wait complete. 2023-09-10T01:09:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.372+0000 7fc0e390eb80 20 client.27497 trim_cache size 0 max 16384 2023-09-10T01:09:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 Processor -- start 2023-09-10T01:09:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 -- start start 2023-09-10T01:09:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067ab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562206795fc0 con 0x56220679ecc0 2023-09-10T01:09:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622067938a0 con 0x5622067a41a0 2023-09-10T01:09:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622067a4cd0 con 0x56220679e300 2023-09-10T01:09:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0ce59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0ce59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49782/0 (socket says 172.21.15.3:49782) 2023-09-10T01:09:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0ce59c700 1 -- 172.21.15.3:0/4100507548 learned_addr learned my addr 172.21.15.3:0/4100507548 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.388+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0ce59c700 1 -- 172.21.15.3:0/4100507548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 msgr2=0x5622067b2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0ce59c700 1 -- 172.21.15.3:0/4100507548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 msgr2=0x5622067ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0ce59c700 1 -- 172.21.15.3:0/4100507548 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622065431f0 con 0x5622067a41a0 2023-09-10T01:09:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a2fe0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0007550 tx=0x7fc0c001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70472213db01d132 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0bcff9700 1 -- 172.21.15.3:0/4100507548 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c001a470 con 0x5622067a41a0 2023-09-10T01:09:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0bcff9700 1 -- 172.21.15.3:0/4100507548 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0c000f040 con 0x5622067a41a0 2023-09-10T01:09:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.389+0000 7fc0bcff9700 1 -- 172.21.15.3:0/4100507548 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c00248b0 con 0x5622067a41a0 2023-09-10T01:09:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 -- 172.21.15.3:0/4100507548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 msgr2=0x5622067a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a2fe0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0007550 tx=0x7fc0c001a000 comp rx=0 tx=0).stop 2023-09-10T01:09:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 -- 172.21.15.3:0/4100507548 shutdown_connections 2023-09-10T01:09:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067a2fe0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/4100507548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 -- 172.21.15.3:0/4100507548 >> 172.21.15.3:0/4100507548 conn(0x562206713030 msgr2=0x5622067b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 -- 172.21.15.3:0/4100507548 shutdown_connections 2023-09-10T01:09:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 -- 172.21.15.3:0/4100507548 wait complete. 2023-09-10T01:09:41.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 -- start start 2023-09-10T01:09:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.390+0000 7fc0e390eb80 1 Processor -- start 2023-09-10T01:09:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0e390eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622067b6ce0 con 0x56220679ecc0 2023-09-10T01:09:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622067b6e50 con 0x5622067a41a0 2023-09-10T01:09:41.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0e390eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622067b7010 con 0x56220679e300 2023-09-10T01:09:41.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0e390eb80 10 client.0 _ll_register_callbacks cb 0x7fc0b8009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:41.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0cdd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:59308/0 (socket says 172.21.15.3:59308) 2023-09-10T01:09:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/3560629109 learned_addr learned my addr 172.21.15.3:0/3560629109 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.391+0000 7fc0ce59c700 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 msgr2=0x5622067b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/3560629109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 msgr2=0x5622067b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0cdd9b700 1 -- 172.21.15.3:0/3560629109 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c40b4040 con 0x56220679e300 2023-09-10T01:09:41.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0cdd9b700 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b8290 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc0c409a8a0 tx=0x7fc0c409a650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3116b9ead1dfc2f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:09:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0b17fa700 1 -- 172.21.15.3:0/3560629109 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c40a2bd0 con 0x56220679e300 2023-09-10T01:09:41.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0c409ea20 con 0x56220679e300 2023-09-10T01:09:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0b17fa700 1 -- 172.21.15.3:0/3560629109 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7fc0c409abf0 con 0x56220679e300 2023-09-10T01:09:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0b17fa700 1 -- 172.21.15.3:0/3560629109 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0c40225e0 con 0x56220679e300 2023-09-10T01:09:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0e390eb80 10 client.27412 fetch_fsmap learned FSMap version 29 2023-09-10T01:09:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.392+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fc0c0009da0 con 0x56220679e300 2023-09-10T01:09:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0b17fa700 1 -- 172.21.15.3:0/3560629109 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0c409ea20 con 0x56220679e300 2023-09-10T01:09:41.410 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0b17fa700 1 -- 172.21.15.3:0/3560629109 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc0c4079be0 con 0x56220679e300 2023-09-10T01:09:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0e390eb80 10 client.27412 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:09:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0e390eb80 10 client.27412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:41.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0e390eb80 20 client.27412 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562206811100 0x562206813520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0e390eb80 4 client.27412 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:09:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5622066e98c0 con 0x562206811100 2023-09-10T01:09:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.393+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562206811100 0x562206813520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.394+0000 7fc0cd59a700 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562206811100 0x562206813520 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed5f955e8738354 server_cookie=4e026ddbcaecac36 in_seq=0 out_seq=0 2023-09-10T01:09:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.394+0000 7fc0b17fa700 10 client.27412 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0b17fa700 1 -- 172.21.15.3:0/3560629109 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5622066e98c0 con 0x562206811100 2023-09-10T01:09:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0b17fa700 10 client.27412 handle_command_reply: tid=0 2023-09-10T01:09:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 client.27412 shutdown 2023-09-10T01:09:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 msgr2=0x5622067b8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b8290 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc0c409a8a0 tx=0x7fc0c409a650 comp rx=0 tx=0).stop 2023-09-10T01:09:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 shutdown_connections 2023-09-10T01:09:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562206811100 0x562206813520 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56220679e300 0x5622067b8290 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0b17fa700 0 client.27412 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5622067a41a0 0x5622067b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 --2- 172.21.15.3:0/3560629109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56220679ecc0 0x5622067b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 >> 172.21.15.3:0/3560629109 conn(0x562206713030 msgr2=0x5622067a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 shutdown_connections 2023-09-10T01:09:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 1 -- 172.21.15.3:0/3560629109 wait complete. 2023-09-10T01:09:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.482+0000 7fc0e390eb80 20 client.27412 trim_cache size 0 max 16384 2023-09-10T01:09:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:09:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:09:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:09:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:09:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0b40a07b0 msgr2=0x7fc0b40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0b40a07b0 0x7fc0b40a2c70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009550 tx=0x7fc0c0003a70 comp rx=0 tx=0).stop 2023-09-10T01:09:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8067b60 msgr2=0x7fc0c81966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8067b60 0x7fc0c81966b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4005910 tx=0x7fc0c4005740 comp rx=0 tx=0).stop 2023-09-10T01:09:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 shutdown_connections 2023-09-10T01:09:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0b40a07b0 0x7fc0b40a2c70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0c8067b60 0x7fc0c81966b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0c81066a0 0x7fc0c8199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 --2- 172.21.15.3:0/1640265165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0c8068520 0x7fc0c8196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 >> 172.21.15.3:0/1640265165 conn(0x7fc0c8074fe0 msgr2=0x7fc0c80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 shutdown_connections 2023-09-10T01:09:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:41.483+0000 7fc0cffff700 1 -- 172.21.15.3:0/1640265165 wait complete. 2023-09-10T01:09:41.769 INFO:tasks.workunit.client.0.smithi003.stdout: 1 568565 9.34 MB/sec execute 203 sec latency 79.961 ms 2023-09-10T01:09:42.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:42 smithi003 ceph-mon[97654]: pgmap v197: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-09-10T01:09:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:42 smithi136 ceph-mon[133708]: pgmap v197: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-09-10T01:09:42.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:42 smithi092 ceph-mon[134067]: pgmap v197: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-09-10T01:09:42.769 INFO:tasks.workunit.client.0.smithi003.stdout: 1 569816 9.32 MB/sec execute 204 sec latency 15.510 ms 2023-09-10T01:09:43.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 572113 9.34 MB/sec execute 205 sec latency 15.132 ms 2023-09-10T01:09:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:44 smithi003 ceph-mon[97654]: pgmap v198: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-09-10T01:09:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:44 smithi136 ceph-mon[133708]: pgmap v198: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-09-10T01:09:44.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:44 smithi092 ceph-mon[134067]: pgmap v198: 129 pgs: 129 active+clean; 164 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 306 op/s 2023-09-10T01:09:44.769 INFO:tasks.workunit.client.0.smithi003.stdout: 1 573375 9.32 MB/sec execute 206 sec latency 20.263 ms 2023-09-10T01:09:45.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 575655 9.34 MB/sec execute 207 sec latency 14.906 ms 2023-09-10T01:09:46.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:46 smithi003 ceph-mon[97654]: pgmap v199: 129 pgs: 129 active+clean; 163 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 418 op/s 2023-09-10T01:09:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:46 smithi136 ceph-mon[133708]: pgmap v199: 129 pgs: 129 active+clean; 163 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 418 op/s 2023-09-10T01:09:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:46 smithi092 ceph-mon[134067]: pgmap v199: 129 pgs: 129 active+clean; 163 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.8 MiB/s wr, 418 op/s 2023-09-10T01:09:46.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 576621 9.31 MB/sec execute 208 sec latency 36.065 ms 2023-09-10T01:09:47.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 579050 9.34 MB/sec execute 209 sec latency 16.242 ms 2023-09-10T01:09:48.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:48 smithi003 ceph-mon[97654]: pgmap v200: 129 pgs: 129 active+clean; 163 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:48.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:48 smithi136 ceph-mon[133708]: pgmap v200: 129 pgs: 129 active+clean; 163 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:48 smithi092 ceph-mon[134067]: pgmap v200: 129 pgs: 129 active+clean; 163 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:48.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 580198 9.31 MB/sec execute 210 sec latency 11.904 ms 2023-09-10T01:09:49.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 582595 9.34 MB/sec execute 211 sec latency 16.245 ms 2023-09-10T01:09:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:50 smithi003 ceph-mon[97654]: pgmap v201: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 272 op/s 2023-09-10T01:09:50.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:50 smithi136 ceph-mon[133708]: pgmap v201: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 272 op/s 2023-09-10T01:09:50.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:50 smithi092 ceph-mon[134067]: pgmap v201: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 272 op/s 2023-09-10T01:09:50.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 583706 9.31 MB/sec execute 212 sec latency 18.196 ms 2023-09-10T01:09:51.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 585542 9.32 MB/sec execute 213 sec latency 77.936 ms 2023-09-10T01:09:52.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:52 smithi003 ceph-mon[97654]: pgmap v202: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:52.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:52 smithi136 ceph-mon[133708]: pgmap v202: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:52.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:52 smithi092 ceph-mon[134067]: pgmap v202: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:52.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 586805 9.31 MB/sec execute 214 sec latency 21.777 ms 2023-09-10T01:09:53.770 INFO:tasks.workunit.client.0.smithi003.stdout: 1 588972 9.32 MB/sec execute 215 sec latency 17.418 ms 2023-09-10T01:09:54.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:54 smithi003 ceph-mon[97654]: pgmap v203: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2491246009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc4106680 msgr2=0x7f5fc4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2491246009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc4106680 0x7f5fc4106a60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009990 tx=0x7f5fb4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2491246009 shutdown_connections 2023-09-10T01:09:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2491246009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc4106680 0x7f5fc4106a60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2491246009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fc4100f80 0x7f5fc410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2491246009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc41005c0 0x7f5fc4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2491246009 >> 172.21.15.3:0/2491246009 conn(0x7f5fc40fc0b0 msgr2=0x7f5fc40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2491246009 shutdown_connections 2023-09-10T01:09:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.648+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2491246009 wait complete. 2023-09-10T01:09:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.649+0000 7f5fca1ff700 1 Processor -- start 2023-09-10T01:09:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fca1ff700 1 -- start start 2023-09-10T01:09:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fca1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc41005c0 0x7f5fc419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fca1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc4100f80 0x7f5fc419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fca1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fc4106680 0x7f5fc41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fca1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc4198d90 con 0x7f5fc4106680 2023-09-10T01:09:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fca1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc4198f00 con 0x7f5fc4100f80 2023-09-10T01:09:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fca1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc41990c0 con 0x7f5fc41005c0 2023-09-10T01:09:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc4100f80 0x7f5fc419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fc37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc41005c0 0x7f5fc419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc4100f80 0x7f5fc419eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35164/0 (socket says 172.21.15.3:35164) 2023-09-10T01:09:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/2738287247 learned_addr learned my addr 172.21.15.3:0/2738287247 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.650+0000 7f5fc37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc41005c0 0x7f5fc419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:32988/0 (socket says 172.21.15.3:32988) 2023-09-10T01:09:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fc4106680 0x7f5fc41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/2738287247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc41005c0 msgr2=0x7f5fc419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc41005c0 0x7f5fc419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fc4106680 msgr2=0x7f5fc41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fc4106680 0x7f5fc41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/2738287247 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40095f0 con 0x7f5fc4100f80 2023-09-10T01:09:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fc4106680 0x7f5fc41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc41005c0 0x7f5fc419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc4100f80 0x7f5fc419eb90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800e9a0 tx=0x7f5fb800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc0ff9700 1 -- 172.21.15.3:0/2738287247 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8009790 con 0x7f5fc4100f80 2023-09-10T01:09:54.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc0ff9700 1 -- 172.21.15.3:0/2738287247 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fb800cbd0 con 0x7f5fc4100f80 2023-09-10T01:09:54.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc41993a0 con 0x7f5fc4100f80 2023-09-10T01:09:54.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.651+0000 7f5fc0ff9700 1 -- 172.21.15.3:0/2738287247 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8018680 con 0x7f5fc4100f80 2023-09-10T01:09:54.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.652+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc41a5680 con 0x7f5fc4100f80 2023-09-10T01:09:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.655+0000 7f5fc0ff9700 1 -- 172.21.15.3:0/2738287247 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f5fb800cd40 con 0x7f5fc4100f80 2023-09-10T01:09:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.657+0000 7f5fddac1b80 1 Processor -- start 2023-09-10T01:09:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.657+0000 7f5fddac1b80 1 -- start start 2023-09-10T01:09:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc0ff9700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5fac0a07b0 0x7f5fac0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b479b9b0 0x55c3b479bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc0ff9700 1 -- 172.21.15.3:0/2738287247 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f5fb8014070 con 0x7f5fc4100f80 2023-09-10T01:09:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b4795f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b47964d0 0x55c3b47a2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b46e66d0 con 0x55c3b47964d0 2023-09-10T01:09:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b46e3ff0 con 0x55c3b479b9b0 2023-09-10T01:09:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b46df8e0 con 0x55c3b4795b10 2023-09-10T01:09:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b4795f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5fac0a07b0 0x7f5fac0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b4795f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:32994/0 (socket says 172.21.15.3:32994) 2023-09-10T01:09:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/1412623020 learned_addr learned my addr 172.21.15.3:0/1412623020 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b479b9b0 0x55c3b479bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.658+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b47964d0 0x55c3b47a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/1412623020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b479b9b0 msgr2=0x55c3b479bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b479b9b0 0x55c3b479bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/1412623020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b47964d0 msgr2=0x55c3b47a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b47964d0 0x55c3b47a2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/1412623020 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3b450bd80 con 0x55c3b4795b10 2023-09-10T01:09:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b479b9b0 0x55c3b479bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b47964d0 0x55c3b47a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:09:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b4795f90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80a1260 tx=0x7f5fb80992e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50b3284f016b6d36 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fa9ffb700 1 -- 172.21.15.3:0/1412623020 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8099720 con 0x55c3b4795b10 2023-09-10T01:09:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fa9ffb700 1 -- 172.21.15.3:0/1412623020 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fb809d550 con 0x55c3b4795b10 2023-09-10T01:09:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.659+0000 7f5fa9ffb700 1 -- 172.21.15.3:0/1412623020 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8099980 con 0x55c3b4795b10 2023-09-10T01:09:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1412623020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 msgr2=0x55c3b4795f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b4795f90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80a1260 tx=0x7f5fb80992e0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5fac0a07b0 0x7f5fac0a2c70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc012740 tx=0x7f5fbc009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1412623020 shutdown_connections 2023-09-10T01:09:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b47964d0 0x55c3b47a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b479b9b0 0x55c3b479bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1412623020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b4795f90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1412623020 >> 172.21.15.3:0/1412623020 conn(0x55c3b4702030 msgr2=0x55c3b4788560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1412623020 shutdown_connections 2023-09-10T01:09:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.660+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1412623020 wait complete. 2023-09-10T01:09:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 -- start start 2023-09-10T01:09:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 Processor -- start 2023-09-10T01:09:54.670 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:54 smithi136 ceph-mon[133708]: pgmap v203: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b4795b10 0x55c3b47a2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b47964d0 0x55c3b479f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b479ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b467db10 con 0x55c3b479b9b0 2023-09-10T01:09:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b467d810 con 0x55c3b4795b10 2023-09-10T01:09:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b45618c0 con 0x55c3b47964d0 2023-09-10T01:09:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.661+0000 7f5fddac1b80 10 client.0 _ll_register_callbacks cb 0x55c3b45c7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b4795b10 0x55c3b47a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b4795b10 0x55c3b47a2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35190/0 (socket says 172.21.15.3:35190) 2023-09-10T01:09:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 -- 172.21.15.3:0/1264323174 learned_addr learned my addr 172.21.15.3:0/1264323174 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b47964d0 0x55c3b479f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b479ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 -- 172.21.15.3:0/1264323174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b47964d0 msgr2=0x55c3b479f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b47964d0 0x55c3b479f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 -- 172.21.15.3:0/1264323174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 msgr2=0x55c3b479ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b479ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 -- 172.21.15.3:0/1264323174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc41a5680 con 0x55c3b4795b10 2023-09-10T01:09:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b47964d0 0x55c3b479f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:09:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b4795b10 0x55c3b47a2660 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400c010 tx=0x7f5fb4005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f4b9d01fd82a6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fa2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:09:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.662+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1264323174 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb401d070 con 0x55c3b4795b10 2023-09-10T01:09:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3b4637490 con 0x55c3b4795b10 2023-09-10T01:09:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1264323174 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f5fb4004910 con 0x55c3b4795b10 2023-09-10T01:09:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1264323174 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fb4004ad0 con 0x55c3b4795b10 2023-09-10T01:09:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fddac1b80 10 client.27515 fetch_fsmap learned FSMap version 29 2023-09-10T01:09:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c3b44f4c00 con 0x55c3b4795b10 2023-09-10T01:09:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1264323174 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c3b4637490 con 0x55c3b4795b10 2023-09-10T01:09:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1264323174 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5fb4024c40 con 0x55c3b4795b10 2023-09-10T01:09:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fddac1b80 10 client.27515 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:09:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.663+0000 7f5fddac1b80 10 client.27515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.664+0000 7f5fddac1b80 20 client.27515 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.664+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b47afc20 0x55c3b47b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.664+0000 7f5fddac1b80 4 client.27515 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:09:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.664+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3b46f4840 con 0x55c3b47afc20 2023-09-10T01:09:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.664+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b47afc20 0x55c3b47b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.664+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b47afc20 0x55c3b47b2040 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db6a9f40fc53bd58 server_cookie=2d313ded42ede5d in_seq=0 out_seq=0 2023-09-10T01:09:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.664+0000 7f5fa2ffd700 10 client.27515 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1264323174 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c3b46f4840 con 0x55c3b47afc20 2023-09-10T01:09:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fa2ffd700 10 client.27515 handle_command_reply: tid=0 2023-09-10T01:09:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 client.27515 shutdown 2023-09-10T01:09:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b4795b10 msgr2=0x55c3b47a2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b4795b10 0x55c3b47a2660 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400c010 tx=0x7f5fb4005270 comp rx=0 tx=0).stop 2023-09-10T01:09:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 shutdown_connections 2023-09-10T01:09:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b47afc20 0x55c3b47b2040 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b479ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b4795b10 0x55c3b47a2660 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1264323174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b47964d0 0x55c3b479f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fa2ffd700 0 client.27515 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 >> 172.21.15.3:0/1264323174 conn(0x55c3b4702030 msgr2=0x55c3b47877a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 shutdown_connections 2023-09-10T01:09:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1264323174 wait complete. 2023-09-10T01:09:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.665+0000 7f5fddac1b80 20 client.27515 trim_cache size 0 max 16384 2023-09-10T01:09:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 Processor -- start 2023-09-10T01:09:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 -- start start 2023-09-10T01:09:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b4795b10 0x55c3b47a2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b479b9b0 0x55c3b479a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b4789960 con 0x55c3b4795b10 2023-09-10T01:09:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b4796b00 con 0x55c3b47964d0 2023-09-10T01:09:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b479a510 con 0x55c3b479b9b0 2023-09-10T01:09:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.683+0000 7f5fc37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b4795b10 0x55c3b47a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b479b9b0 0x55c3b479a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35198/0 (socket says 172.21.15.3:35198) 2023-09-10T01:09:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/4256529689 learned_addr learned my addr 172.21.15.3:0/4256529689 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/4256529689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b479b9b0 msgr2=0x55c3b479a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/4256529689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b479b9b0 0x55c3b479a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/4256529689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b4795b10 msgr2=0x55c3b47a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/4256529689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b4795b10 0x55c3b47a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/4256529689 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3b45321f0 con 0x55c3b47964d0 2023-09-10T01:09:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/4256529689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a2660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8099420 tx=0x7f5fb800eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca9d1d88f78b04ec server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5faa7fc700 1 -- 172.21.15.3:0/4256529689 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8099720 con 0x55c3b47964d0 2023-09-10T01:09:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5faa7fc700 1 -- 172.21.15.3:0/4256529689 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fb8097ac0 con 0x55c3b47964d0 2023-09-10T01:09:54.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.684+0000 7f5faa7fc700 1 -- 172.21.15.3:0/4256529689 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8099980 con 0x55c3b47964d0 2023-09-10T01:09:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 -- 172.21.15.3:0/4256529689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 msgr2=0x55c3b47a2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/4256529689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a2660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8099420 tx=0x7f5fb800eef0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 -- 172.21.15.3:0/4256529689 shutdown_connections 2023-09-10T01:09:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/4256529689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b479b9b0 0x55c3b479a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/4256529689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b4795b10 0x55c3b47a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/4256529689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a2660 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 -- 172.21.15.3:0/4256529689 >> 172.21.15.3:0/4256529689 conn(0x55c3b4702030 msgr2=0x55c3b4799a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 -- 172.21.15.3:0/4256529689 shutdown_connections 2023-09-10T01:09:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.685+0000 7f5fddac1b80 1 -- 172.21.15.3:0/4256529689 wait complete. 2023-09-10T01:09:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.686+0000 7f5fddac1b80 1 -- start start 2023-09-10T01:09:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.686+0000 7f5fddac1b80 1 Processor -- start 2023-09-10T01:09:54.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.686+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b47a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.686+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.686+0000 7f5fddac1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b47a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.686+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b4802a50 con 0x55c3b479b9b0 2023-09-10T01:09:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b4802b90 con 0x55c3b47964d0 2023-09-10T01:09:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fddac1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b4802cd0 con 0x55c3b4795b10 2023-09-10T01:09:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fddac1b80 10 client.0 _ll_register_callbacks cb 0x7f5fb4009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:09:54.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b47a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35206/0 (socket says 172.21.15.3:35206) 2023-09-10T01:09:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc2ffd700 1 -- 172.21.15.3:0/1670551229 learned_addr learned my addr 172.21.15.3:0/1670551229 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:09:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b47a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc37fe700 1 -- 172.21.15.3:0/1670551229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 msgr2=0x55c3b47a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc37fe700 1 -- 172.21.15.3:0/1670551229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 msgr2=0x55c3b47a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:09:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b47a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc37fe700 1 -- 172.21.15.3:0/1670551229 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb80b1040 con 0x55c3b4795b10 2023-09-10T01:09:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc3fff700 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b47a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:09:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fc37fe700 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b47a4d20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400b2e0 tx=0x7f5fb400f5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2a0ccec2b5b68c2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:09:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.687+0000 7f5fa2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:09:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1670551229 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb401d040 con 0x55c3b4795b10 2023-09-10T01:09:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fb809d550 con 0x55c3b4795b10 2023-09-10T01:09:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1670551229 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9439+0+0 (secure 0 0 0) 0x7f5fb4000e40 con 0x55c3b4795b10 2023-09-10T01:09:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1670551229 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fb4005bc0 con 0x55c3b4795b10 2023-09-10T01:09:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 10 client.27430 fetch_fsmap learned FSMap version 29 2023-09-10T01:09:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c3b450a250 con 0x55c3b4795b10 2023-09-10T01:09:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1670551229 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fb809d550 con 0x55c3b4795b10 2023-09-10T01:09:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1670551229 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5fb4004710 con 0x55c3b4795b10 2023-09-10T01:09:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 10 client.27430 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:09:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 10 client.27430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:09:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 20 client.27430 populate_metadata read hostname 'smithi003' 2023-09-10T01:09:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b4803f80 0x55c3b48063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:09:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 4 client.27430 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:09:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.688+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c3b46d88c0 con 0x55c3b4803f80 2023-09-10T01:09:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.689+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b4803f80 0x55c3b48063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:09:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.689+0000 7f5fc2ffd700 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b4803f80 0x55c3b48063a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b9b3b132a1ace94 server_cookie=cd10884abc89299 in_seq=0 out_seq=0 2023-09-10T01:09:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.689+0000 7f5fa2ffd700 10 client.27430 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:54.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:54 smithi092 ceph-mon[134067]: pgmap v203: 129 pgs: 129 active+clean; 170 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 192 op/s 2023-09-10T01:09:54.772 INFO:tasks.workunit.client.0.smithi003.stdout: 1 590214 9.31 MB/sec execute 216 sec latency 43.562 ms 2023-09-10T01:09:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fa2ffd700 1 -- 172.21.15.3:0/1670551229 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c3b46d88c0 con 0x55c3b4803f80 2023-09-10T01:09:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fa2ffd700 10 client.27430 handle_command_reply: tid=0 2023-09-10T01:09:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 client.27430 shutdown 2023-09-10T01:09:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 msgr2=0x55c3b47a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b47a4d20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400b2e0 tx=0x7f5fb400f5d0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 shutdown_connections 2023-09-10T01:09:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c3b4803f80 0x55c3b48063a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3b4795b10 0x55c3b47a4d20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3b479b9b0 0x55c3b47a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 --2- 172.21.15.3:0/1670551229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3b47964d0 0x55c3b47a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fa2ffd700 0 client.27430 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:09:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 >> 172.21.15.3:0/1670551229 conn(0x55c3b4702030 msgr2=0x55c3b479a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.825+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 shutdown_connections 2023-09-10T01:09:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fddac1b80 1 -- 172.21.15.3:0/1670551229 wait complete. 2023-09-10T01:09:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fddac1b80 20 client.27430 trim_cache size 0 max 16384 2023-09-10T01:09:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:09:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:09:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:09:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:09:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5fac0a07b0 msgr2=0x7f5fac0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5fac0a07b0 0x7f5fac0a2c70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc012740 tx=0x7f5fbc009920 comp rx=0 tx=0).stop 2023-09-10T01:09:54.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc4100f80 msgr2=0x7f5fc419eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:09:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc4100f80 0x7f5fc419eb90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800e9a0 tx=0x7f5fb800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 shutdown_connections 2023-09-10T01:09:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5fac0a07b0 0x7f5fac0a2c70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc41005c0 0x7f5fc419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fc4106680 0x7f5fc41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 --2- 172.21.15.3:0/2738287247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc4100f80 0x7f5fc419eb90 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:09:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.826+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 >> 172.21.15.3:0/2738287247 conn(0x7f5fc40fc0b0 msgr2=0x7f5fc4072830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:09:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.827+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 shutdown_connections 2023-09-10T01:09:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:09:54.827+0000 7f5fca1ff700 1 -- 172.21.15.3:0/2738287247 wait complete. 2023-09-10T01:09:55.772 INFO:tasks.workunit.client.0.smithi003.stdout: 1 592039 9.30 MB/sec execute 217 sec latency 59.407 ms 2023-09-10T01:09:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:56 smithi003 ceph-mon[97654]: pgmap v204: 129 pgs: 129 active+clean; 177 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 269 op/s 2023-09-10T01:09:56.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:56 smithi136 ceph-mon[133708]: pgmap v204: 129 pgs: 129 active+clean; 177 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 269 op/s 2023-09-10T01:09:56.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:56 smithi092 ceph-mon[134067]: pgmap v204: 129 pgs: 129 active+clean; 177 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 269 op/s 2023-09-10T01:09:56.772 INFO:tasks.workunit.client.0.smithi003.stdout: 1 593701 9.31 MB/sec execute 218 sec latency 10.806 ms 2023-09-10T01:09:57.772 INFO:tasks.workunit.client.0.smithi003.stdout: 1 595365 9.29 MB/sec execute 219 sec latency 30.112 ms 2023-09-10T01:09:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:09:58 smithi003 ceph-mon[97654]: pgmap v205: 129 pgs: 129 active+clean; 177 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2023-09-10T01:09:58.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:09:58 smithi136 ceph-mon[133708]: pgmap v205: 129 pgs: 129 active+clean; 177 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2023-09-10T01:09:58.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:09:58 smithi092 ceph-mon[134067]: pgmap v205: 129 pgs: 129 active+clean; 177 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 156 op/s 2023-09-10T01:09:58.772 INFO:tasks.workunit.client.0.smithi003.stdout: 1 597136 9.31 MB/sec execute 220 sec latency 14.344 ms 2023-09-10T01:09:59.772 INFO:tasks.workunit.client.0.smithi003.stdout: 1 598690 9.29 MB/sec execute 221 sec latency 16.815 ms 2023-09-10T01:10:00.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:00 smithi003 ceph-mon[97654]: pgmap v206: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 283 op/s 2023-09-10T01:10:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:00 smithi003 ceph-mon[97654]: overall HEALTH_OK 2023-09-10T01:10:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:00 smithi136 ceph-mon[133708]: pgmap v206: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 283 op/s 2023-09-10T01:10:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:00 smithi136 ceph-mon[133708]: overall HEALTH_OK 2023-09-10T01:10:00.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:00 smithi092 ceph-mon[134067]: pgmap v206: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 283 op/s 2023-09-10T01:10:00.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:00 smithi092 ceph-mon[134067]: overall HEALTH_OK 2023-09-10T01:10:00.772 INFO:tasks.workunit.client.0.smithi003.stdout: 1 600435 9.31 MB/sec execute 222 sec latency 66.446 ms 2023-09-10T01:10:01.773 INFO:tasks.workunit.client.0.smithi003.stdout: 1 600764 9.27 MB/sec execute 223 sec latency 65.986 ms 2023-09-10T01:10:02.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:02 smithi003 ceph-mon[97654]: pgmap v207: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 203 op/s 2023-09-10T01:10:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:02 smithi136 ceph-mon[133708]: pgmap v207: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 203 op/s 2023-09-10T01:10:02.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:02 smithi092 ceph-mon[134067]: pgmap v207: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 203 op/s 2023-09-10T01:10:02.773 INFO:tasks.workunit.client.0.smithi003.stdout: 1 602346 9.25 MB/sec execute 224 sec latency 31.072 ms 2023-09-10T01:10:03.773 INFO:tasks.workunit.client.0.smithi003.stdout: 1 603427 9.25 MB/sec execute 225 sec latency 184.902 ms 2023-09-10T01:10:04.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:04 smithi003 ceph-mon[97654]: pgmap v208: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 203 op/s 2023-09-10T01:10:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:04 smithi136 ceph-mon[133708]: pgmap v208: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 203 op/s 2023-09-10T01:10:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:04 smithi092 ceph-mon[134067]: pgmap v208: 129 pgs: 129 active+clean; 173 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 203 op/s 2023-09-10T01:10:04.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 604034 9.23 MB/sec execute 226 sec latency 104.982 ms 2023-09-10T01:10:05.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:05 smithi003 ceph-mon[97654]: osdmap e92: 12 total, 12 up, 12 in 2023-09-10T01:10:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:05 smithi136 ceph-mon[133708]: osdmap e92: 12 total, 12 up, 12 in 2023-09-10T01:10:05.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:05 smithi092 ceph-mon[134067]: osdmap e92: 12 total, 12 up, 12 in 2023-09-10T01:10:05.773 INFO:tasks.workunit.client.0.smithi003.stdout: 1 604225 9.19 MB/sec execute 227 sec latency 80.394 ms 2023-09-10T01:10:06.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:06 smithi003 ceph-mon[97654]: pgmap v210: 129 pgs: 129 active+clean; 220 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 333 op/s 2023-09-10T01:10:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:06 smithi136 ceph-mon[133708]: pgmap v210: 129 pgs: 129 active+clean; 220 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 333 op/s 2023-09-10T01:10:06.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:06 smithi092 ceph-mon[134067]: pgmap v210: 129 pgs: 129 active+clean; 220 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 333 op/s 2023-09-10T01:10:06.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 605615 9.18 MB/sec execute 228 sec latency 35.180 ms 2023-09-10T01:10:07.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 607748 9.19 MB/sec execute 229 sec latency 13.582 ms 2023-09-10T01:10:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 -- 172.21.15.3:0/1039336313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 msgr2=0x7ffac40685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/1039336313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac40685b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009990 tx=0x7ffab8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 -- 172.21.15.3:0/1039336313 shutdown_connections 2023-09-10T01:10:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/1039336313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffac410ce10 0x7ffac410f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/1039336313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffac4101490 0x7ffac4067b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/1039336313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac40685b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 -- 172.21.15.3:0/1039336313 >> 172.21.15.3:0/1039336313 conn(0x7ffac4074fe0 msgr2=0x7ffac40753f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 -- 172.21.15.3:0/1039336313 shutdown_connections 2023-09-10T01:10:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.980+0000 7ffaca7ff700 1 -- 172.21.15.3:0/1039336313 wait complete. 2023-09-10T01:10:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.981+0000 7ffaca7ff700 1 Processor -- start 2023-09-10T01:10:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.981+0000 7ffaca7ff700 1 -- start start 2023-09-10T01:10:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.981+0000 7ffaca7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.981+0000 7ffaca7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffac410ce10 0x7ffac419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.981+0000 7ffaca7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffac41986d0 0x7ffac4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffaca7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac4199160 con 0x7ffac410ce10 2023-09-10T01:10:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffaca7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac41992d0 con 0x7ffac4068130 2023-09-10T01:10:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffaca7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac4199490 con 0x7ffac41986d0 2023-09-10T01:10:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39628/0 (socket says 172.21.15.3:39628) 2023-09-10T01:10:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 -- 172.21.15.3:0/2833138827 learned_addr learned my addr 172.21.15.3:0/2833138827 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffac410ce10 0x7ffac419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac8d9c700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffac41986d0 0x7ffac4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 -- 172.21.15.3:0/2833138827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffac41986d0 msgr2=0x7ffac4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffac41986d0 0x7ffac4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 -- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffac410ce10 msgr2=0x7ffac419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffac410ce10 0x7ffac419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 -- 172.21.15.3:0/2833138827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab80095f0 con 0x7ffac4068130 2023-09-10T01:10:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffac410ce10 0x7ffac419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.982+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac419e640 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ffab400e9a0 tx=0x7ffab400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.983+0000 7ffac17fa700 1 -- 172.21.15.3:0/2833138827 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4009790 con 0x7ffac4068130 2023-09-10T01:10:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.983+0000 7ffac17fa700 1 -- 172.21.15.3:0/2833138827 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffab4010430 con 0x7ffac4068130 2023-09-10T01:10:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.983+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac41a54f0 con 0x7ffac4068130 2023-09-10T01:10:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.983+0000 7ffac17fa700 1 -- 172.21.15.3:0/2833138827 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4018680 con 0x7ffac4068130 2023-09-10T01:10:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.983+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac41a5a50 con 0x7ffac4068130 2023-09-10T01:10:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.987+0000 7ffac17fa700 1 -- 172.21.15.3:0/2833138827 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ffab40105a0 con 0x7ffac4068130 2023-09-10T01:10:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.988+0000 7ffade1b2b80 1 Processor -- start 2023-09-10T01:10:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.988+0000 7ffade1b2b80 1 -- start start 2023-09-10T01:10:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.988+0000 7ffac17fa700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffaac0a0830 0x7ffaac0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.988+0000 7ffac17fa700 1 -- 172.21.15.3:0/2833138827 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9615+0+0 (secure 0 0 0) 0x7ffab4014070 con 0x7ffac4068130 2023-09-10T01:10:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd9e1a0 0x56471bd9e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98300 0x56471bd98780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98cc0 0x56471bda4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bcf16d0 con 0x56471bd98cc0 2023-09-10T01:10:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bceeff0 con 0x56471bd98300 2023-09-10T01:10:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bcea8e0 con 0x56471bd9e1a0 2023-09-10T01:10:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac8d9c700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffaac0a0830 0x7ffaac0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd9e1a0 0x56471bd9e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98300 0x56471bd98780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd9e1a0 0x56471bd9e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50692/0 (socket says 172.21.15.3:50692) 2023-09-10T01:10:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 -- 172.21.15.3:0/2283824178 learned_addr learned my addr 172.21.15.3:0/2283824178 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac8d9c700 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98cc0 0x56471bda4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 -- 172.21.15.3:0/2283824178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98300 msgr2=0x56471bd98780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98300 0x56471bd98780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 -- 172.21.15.3:0/2283824178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98cc0 msgr2=0x56471bda4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98cc0 0x56471bda4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac3fff700 1 -- 172.21.15.3:0/2283824178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56471bb16d80 con 0x56471bd9e1a0 2023-09-10T01:10:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.989+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98300 0x56471bd98780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.990+0000 7ffac8d9c700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffaac0a0830 0x7ffaac0a2cf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffac4068b40 tx=0x7ffabc00fc00 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.990+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd9e1a0 0x56471bd9e580 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffab4096020 tx=0x7ffab4098fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2b8b0bbcbd660ec server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.990+0000 7ffaaa7fc700 1 -- 172.21.15.3:0/2283824178 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab409d3e0 con 0x56471bd9e1a0 2023-09-10T01:10:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.990+0000 7ffaaa7fc700 1 -- 172.21.15.3:0/2283824178 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffab40a1620 con 0x56471bd9e1a0 2023-09-10T01:10:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.990+0000 7ffaaa7fc700 1 -- 172.21.15.3:0/2283824178 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab40cc170 con 0x56471bd9e1a0 2023-09-10T01:10:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2283824178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd9e1a0 msgr2=0x56471bd9e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd9e1a0 0x56471bd9e580 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffab4096020 tx=0x7ffab4098fe0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2283824178 shutdown_connections 2023-09-10T01:10:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd9e1a0 0x56471bd9e580 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98cc0 0x56471bda4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2283824178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98300 0x56471bd98780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2283824178 >> 172.21.15.3:0/2283824178 conn(0x56471bd0d030 msgr2=0x56471bd93560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:07.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2283824178 shutdown_connections 2023-09-10T01:10:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.991+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2283824178 wait complete. 2023-09-10T01:10:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.992+0000 7ffade1b2b80 1 -- start start 2023-09-10T01:10:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.992+0000 7ffade1b2b80 1 Processor -- start 2023-09-10T01:10:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bda7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bda21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bc88b10 con 0x56471bd9e1a0 2023-09-10T01:10:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bc88810 con 0x56471bd98cc0 2023-09-10T01:10:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bb6c8c0 con 0x56471bd98300 2023-09-10T01:10:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffade1b2b80 10 client.0 _ll_register_callbacks cb 0x56471bbd2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bda7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bda7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50704/0 (socket says 172.21.15.3:50704) 2023-09-10T01:10:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac8d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bda21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 -- 172.21.15.3:0/2884685312 learned_addr learned my addr 172.21.15.3:0/2884685312 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 -- 172.21.15.3:0/2884685312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 msgr2=0x56471bda8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 -- 172.21.15.3:0/2884685312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 msgr2=0x56471bda21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bda21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.993+0000 7ffac3fff700 1 -- 172.21.15.3:0/2884685312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac410ff00 con 0x56471bd98300 2023-09-10T01:10:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bda7f10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a10f0 tx=0x7ffab40c9870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3d7896bc7ae4e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffaa37fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:10:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffaa37fe700 1 -- 172.21.15.3:0/2884685312 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab409d3e0 con 0x56471bd98300 2023-09-10T01:10:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffaa37fe700 1 -- 172.21.15.3:0/2884685312 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9615+0+0 (secure 0 0 0) 0x7ffab40cc170 con 0x56471bd98300 2023-09-10T01:10:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56471bc42490 con 0x56471bd98300 2023-09-10T01:10:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffaa37fe700 1 -- 172.21.15.3:0/2884685312 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffab40cc330 con 0x56471bd98300 2023-09-10T01:10:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffade1b2b80 10 client.27442 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ffac41a5a50 con 0x56471bd98300 2023-09-10T01:10:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffaa37fe700 1 -- 172.21.15.3:0/2884685312 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56471bc42490 con 0x56471bd98300 2023-09-10T01:10:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.994+0000 7ffaa37fe700 1 -- 172.21.15.3:0/2884685312 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ffab40a1190 con 0x56471bd98300 2023-09-10T01:10:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffade1b2b80 10 client.27442 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffade1b2b80 10 client.27442 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffade1b2b80 20 client.27442 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471bdb2060 0x56471bdb4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffade1b2b80 4 client.27442 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:10:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56471bcff840 con 0x56471bdb2060 2023-09-10T01:10:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471bdb2060 0x56471bdb4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471bdb2060 0x56471bdb4480 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd0884220048aa1 server_cookie=8fabbf44ea87f7dc in_seq=0 out_seq=0 2023-09-10T01:10:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.995+0000 7ffaa37fe700 10 client.27442 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffaa37fe700 1 -- 172.21.15.3:0/2884685312 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56471bcff840 con 0x56471bdb2060 2023-09-10T01:10:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffaa37fe700 10 client.27442 handle_command_reply: tid=0 2023-09-10T01:10:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 client.27442 shutdown 2023-09-10T01:10:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 msgr2=0x56471bda7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bda7f10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a10f0 tx=0x7ffab40c9870 comp rx=0 tx=0).stop 2023-09-10T01:10:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 shutdown_connections 2023-09-10T01:10:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471bdb2060 0x56471bdb4480 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bda7f10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bda21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2884685312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffaa37fe700 0 client.27442 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 >> 172.21.15.3:0/2884685312 conn(0x56471bd0d030 msgr2=0x56471bd927a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 shutdown_connections 2023-09-10T01:10:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.997+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2884685312 wait complete. 2023-09-10T01:10:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:07.998+0000 7ffade1b2b80 20 client.27442 trim_cache size 0 max 16384 2023-09-10T01:10:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.013+0000 7ffade1b2b80 1 Processor -- start 2023-09-10T01:10:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.013+0000 7ffade1b2b80 1 -- start start 2023-09-10T01:10:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bd9fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bd9ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bda26c0 con 0x56471bd9e1a0 2023-09-10T01:10:08.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bd94d10 con 0x56471bd98cc0 2023-09-10T01:10:08.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bd8fff0 con 0x56471bd98300 2023-09-10T01:10:08.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bd9fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39664/0 (socket says 172.21.15.3:39664) 2023-09-10T01:10:08.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 -- 172.21.15.3:0/2777515425 learned_addr learned my addr 172.21.15.3:0/2777515425 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:08.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac8d9c700 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bd9ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 -- 172.21.15.3:0/2777515425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 msgr2=0x56471bd9fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:08.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bd9fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 -- 172.21.15.3:0/2777515425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 msgr2=0x56471bd9ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:08.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bd9ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac37fe700 1 -- 172.21.15.3:0/2777515425 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56471baffc00 con 0x56471bd98cc0 2023-09-10T01:10:08.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac8d9c700 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bd9ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:10:08.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.014+0000 7ffac3fff700 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bd9fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:08.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.015+0000 7ffac37fe700 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda2180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffab80098e0 tx=0x7ffab8005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f14175643c022628 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:08.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.015+0000 7ffaaaffd700 1 -- 172.21.15.3:0/2777515425 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab801d040 con 0x56471bd98cc0 2023-09-10T01:10:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.015+0000 7ffaaaffd700 1 -- 172.21.15.3:0/2777515425 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffab801e040 con 0x56471bd98cc0 2023-09-10T01:10:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.015+0000 7ffaaaffd700 1 -- 172.21.15.3:0/2777515425 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8004500 con 0x56471bd98cc0 2023-09-10T01:10:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.015+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2777515425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 msgr2=0x56471bda2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda2180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffab80098e0 tx=0x7ffab8005fb0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2777515425 shutdown_connections 2023-09-10T01:10:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98300 0x56471bd9fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd9e1a0 0x56471bd9ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/2777515425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd98cc0 0x56471bda2180 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2777515425 >> 172.21.15.3:0/2777515425 conn(0x56471bd0d030 msgr2=0x56471bda1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2777515425 shutdown_connections 2023-09-10T01:10:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 -- 172.21.15.3:0/2777515425 wait complete. 2023-09-10T01:10:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 -- start start 2023-09-10T01:10:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.016+0000 7ffade1b2b80 1 Processor -- start 2023-09-10T01:10:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98300 0x56471bda8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98cc0 0x56471bda66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffade1b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd9e1a0 0x56471bda6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bda73d0 con 0x56471bd98300 2023-09-10T01:10:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471bda7540 con 0x56471bd9e1a0 2023-09-10T01:10:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffade1b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56471be05e90 con 0x56471bd98cc0 2023-09-10T01:10:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffade1b2b80 10 client.0 _ll_register_callbacks cb 0x56471bbd2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98300 0x56471bda8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffac8d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd9e1a0 0x56471bda6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffac8d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd9e1a0 0x56471bda6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39678/0 (socket says 172.21.15.3:39678) 2023-09-10T01:10:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffac3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98300 0x56471bda8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38738/0 (socket says 172.21.15.3:38738) 2023-09-10T01:10:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffac8d9c700 1 -- 172.21.15.3:0/252471999 learned_addr learned my addr 172.21.15.3:0/252471999 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffac37fe700 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98cc0 0x56471bda66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.017+0000 7ffac37fe700 1 -- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd9e1a0 msgr2=0x56471bda6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffac37fe700 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd9e1a0 0x56471bda6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffac37fe700 1 -- 172.21.15.3:0/252471999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98300 msgr2=0x56471bda8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffac37fe700 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98300 0x56471bda8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffac37fe700 1 -- 172.21.15.3:0/252471999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac41a5a50 con 0x56471bd98cc0 2023-09-10T01:10:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffac8d9c700 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd9e1a0 0x56471bda6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:08.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffac37fe700 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98cc0 0x56471bda66c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x56471bd94f30 tx=0x7ffab80058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c66f3be747d1e1ce server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:08.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffaa37fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:10:08.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffaa37fe700 1 -- 172.21.15.3:0/252471999 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab801d070 con 0x56471bd98cc0 2023-09-10T01:10:08.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffab40a1620 con 0x56471bd98cc0 2023-09-10T01:10:08.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffaa37fe700 1 -- 172.21.15.3:0/252471999 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9615+0+0 (secure 0 0 0) 0x7ffab8029380 con 0x56471bd98cc0 2023-09-10T01:10:08.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffaa37fe700 1 -- 172.21.15.3:0/252471999 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffab8029e80 con 0x56471bd98cc0 2023-09-10T01:10:08.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffade1b2b80 10 client.27448 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:08.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.018+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ffab800be60 con 0x56471bd98cc0 2023-09-10T01:10:08.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffaa37fe700 1 -- 172.21.15.3:0/252471999 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffab40a1620 con 0x56471bd98cc0 2023-09-10T01:10:08.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffaa37fe700 1 -- 172.21.15.3:0/252471999 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ffab8004c90 con 0x56471bd98cc0 2023-09-10T01:10:08.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffade1b2b80 10 client.27448 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:08.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffade1b2b80 10 client.27448 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:08.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffade1b2b80 20 client.27448 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:08.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471be071e0 0x56471be095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:08.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffade1b2b80 4 client.27448 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:10:08.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56471bce38c0 con 0x56471be071e0 2023-09-10T01:10:08.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.019+0000 7ffac3fff700 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471be071e0 0x56471be095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:08.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.020+0000 7ffac3fff700 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471be071e0 0x56471be095d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3996a8c43dea14ac server_cookie=c67c5372fea75454 in_seq=0 out_seq=0 2023-09-10T01:10:08.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.020+0000 7ffaa37fe700 10 client.27448 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.115+0000 7ffaa37fe700 1 -- 172.21.15.3:0/252471999 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56471bce38c0 con 0x56471be071e0 2023-09-10T01:10:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.115+0000 7ffaa37fe700 10 client.27448 handle_command_reply: tid=0 2023-09-10T01:10:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.115+0000 7ffade1b2b80 1 client.27448 shutdown 2023-09-10T01:10:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98cc0 msgr2=0x56471bda66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98cc0 0x56471bda66c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x56471bd94f30 tx=0x7ffab80058e0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 shutdown_connections 2023-09-10T01:10:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56471be071e0 0x56471be095d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56471bd9e1a0 0x56471bda6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56471bd98300 0x56471bda8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 --2- 172.21.15.3:0/252471999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56471bd98cc0 0x56471bda66c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 >> 172.21.15.3:0/252471999 conn(0x56471bd0d030 msgr2=0x56471bd97b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffaa37fe700 0 client.27448 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.116+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 shutdown_connections 2023-09-10T01:10:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.117+0000 7ffade1b2b80 1 -- 172.21.15.3:0/252471999 wait complete. 2023-09-10T01:10:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.117+0000 7ffade1b2b80 20 client.27448 trim_cache size 0 max 16384 2023-09-10T01:10:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:10:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:10:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:10:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:10:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffaac0a0830 msgr2=0x7ffaac0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffaac0a0830 0x7ffaac0a2cf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffac4068b40 tx=0x7ffabc00fc00 comp rx=0 tx=0).stop 2023-09-10T01:10:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 msgr2=0x7ffac419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac419e640 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ffab400e9a0 tx=0x7ffab400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 shutdown_connections 2023-09-10T01:10:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ffaac0a0830 0x7ffaac0a2cf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffac41986d0 0x7ffac4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4068130 0x7ffac419e640 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 --2- 172.21.15.3:0/2833138827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffac410ce10 0x7ffac419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 >> 172.21.15.3:0/2833138827 conn(0x7ffac4074fe0 msgr2=0x7ffac40fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 shutdown_connections 2023-09-10T01:10:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:08.118+0000 7ffaca7ff700 1 -- 172.21.15.3:0/2833138827 wait complete. 2023-09-10T01:10:08.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:08 smithi003 ceph-mon[97654]: pgmap v211: 129 pgs: 129 active+clean; 220 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 333 op/s 2023-09-10T01:10:08.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:08 smithi136 ceph-mon[133708]: pgmap v211: 129 pgs: 129 active+clean; 220 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 333 op/s 2023-09-10T01:10:08.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:08 smithi092 ceph-mon[134067]: pgmap v211: 129 pgs: 129 active+clean; 220 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.9 MiB/s wr, 333 op/s 2023-09-10T01:10:08.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 609003 9.18 MB/sec execute 230 sec latency 48.980 ms 2023-09-10T01:10:09.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 611187 9.20 MB/sec execute 231 sec latency 38.053 ms 2023-09-10T01:10:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:10 smithi003 ceph-mon[97654]: pgmap v212: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 374 op/s 2023-09-10T01:10:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:10 smithi003 ceph-mon[97654]: osdmap e93: 12 total, 12 up, 12 in 2023-09-10T01:10:10.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:10 smithi136 ceph-mon[133708]: pgmap v212: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 374 op/s 2023-09-10T01:10:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:10 smithi136 ceph-mon[133708]: osdmap e93: 12 total, 12 up, 12 in 2023-09-10T01:10:10.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:10 smithi092 ceph-mon[134067]: pgmap v212: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.8 MiB/s wr, 374 op/s 2023-09-10T01:10:10.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:10 smithi092 ceph-mon[134067]: osdmap e93: 12 total, 12 up, 12 in 2023-09-10T01:10:10.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 612543 9.18 MB/sec execute 232 sec latency 17.549 ms 2023-09-10T01:10:11.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:10:11.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 -- 172.21.15.3:0/4284264574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c101000 msgr2=0x7fc35c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 --2- 172.21.15.3:0/4284264574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c101000 0x7fc35c10d510 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc35400b0a0 tx=0x7fc35400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 -- 172.21.15.3:0/4284264574 shutdown_connections 2023-09-10T01:10:11.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 --2- 172.21.15.3:0/4284264574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c101000 0x7fc35c10d510 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 --2- 172.21.15.3:0/4284264574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35c106650 0x7fc35c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 --2- 172.21.15.3:0/4284264574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c100640 0x7fc35c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 -- 172.21.15.3:0/4284264574 >> 172.21.15.3:0/4284264574 conn(0x7fc35c0fc170 msgr2=0x7fc35c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:11.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 -- 172.21.15.3:0/4284264574 shutdown_connections 2023-09-10T01:10:11.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.400+0000 7fc3623ff700 1 -- 172.21.15.3:0/4284264574 wait complete. 2023-09-10T01:10:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.401+0000 7fc3623ff700 1 Processor -- start 2023-09-10T01:10:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.401+0000 7fc3623ff700 1 -- start start 2023-09-10T01:10:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc3623ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c100640 0x7fc35c072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc3623ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c101000 0x7fc35c072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc3623ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35c106650 0x7fc35c1a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc3623ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c073670 con 0x7fc35c106650 2023-09-10T01:10:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc3623ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1a4da0 con 0x7fc35c100640 2023-09-10T01:10:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc3623ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1a4ee0 con 0x7fc35c101000 2023-09-10T01:10:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c100640 0x7fc35c072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc35b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c101000 0x7fc35c072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc36099c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35c106650 0x7fc35c1a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c100640 0x7fc35c072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39692/0 (socket says 172.21.15.3:39692) 2023-09-10T01:10:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.402+0000 7fc35bfff700 1 -- 172.21.15.3:0/366502736 learned_addr learned my addr 172.21.15.3:0/366502736 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc35bfff700 1 -- 172.21.15.3:0/366502736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c101000 msgr2=0x7fc35c072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc35bfff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c101000 0x7fc35c072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc35bfff700 1 -- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35c106650 msgr2=0x7fc35c1a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc35bfff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35c106650 0x7fc35c1a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc35bfff700 1 -- 172.21.15.3:0/366502736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354009d00 con 0x7fc35c100640 2023-09-10T01:10:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc36099c700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35c106650 0x7fc35c1a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:11.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c101000 0x7fc35c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc35bfff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c100640 0x7fc35c072900 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00e9a0 tx=0x7fc34c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc3597fa700 1 -- 172.21.15.3:0/366502736 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c009790 con 0x7fc35c100640 2023-09-10T01:10:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c1a5140 con 0x7fc35c100640 2023-09-10T01:10:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc3597fa700 1 -- 172.21.15.3:0/366502736 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc34c010430 con 0x7fc35c100640 2023-09-10T01:10:11.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc3597fa700 1 -- 172.21.15.3:0/366502736 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c018680 con 0x7fc35c100640 2023-09-10T01:10:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.403+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c1a5620 con 0x7fc35c100640 2023-09-10T01:10:11.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.408+0000 7fc3597fa700 1 -- 172.21.15.3:0/366502736 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fc34c0105a0 con 0x7fc35c100640 2023-09-10T01:10:11.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.409+0000 7fc3597fa700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3440a07b0 0x7fc3440a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.409+0000 7fc375c65b80 1 Processor -- start 2023-09-10T01:10:11.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.409+0000 7fc3597fa700 1 -- 172.21.15.3:0/366502736 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc34c014070 con 0x7fc35c100640 2023-09-10T01:10:11.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.409+0000 7fc375c65b80 1 -- start start 2023-09-10T01:10:11.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.409+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3440a07b0 0x7fc3440a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x55732250caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557322516e40 0x557322519230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc375c65b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557322465730 con 0x557322516e40 2023-09-10T01:10:11.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc375c65b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557322463050 con 0x55732250c710 2023-09-10T01:10:11.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc375c65b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55732245e940 con 0x55732250d030 2023-09-10T01:10:11.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x55732250caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x55732250caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39708/0 (socket says 172.21.15.3:39708) 2023-09-10T01:10:11.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 -- 172.21.15.3:0/2992571240 learned_addr learned my addr 172.21.15.3:0/2992571240 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:11.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc36099c700 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557322516e40 0x557322519230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3440a07b0 0x7fc3440a2c70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0740f0 tx=0x7fc350009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:11.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 -- 172.21.15.3:0/2992571240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 msgr2=0x557322516900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 -- 172.21.15.3:0/2992571240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557322516e40 msgr2=0x557322519230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557322516e40 0x557322519230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc35bfff700 1 -- 172.21.15.3:0/2992571240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55732228ade0 con 0x55732250c710 2023-09-10T01:10:11.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.410+0000 7fc36099c700 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557322516e40 0x557322519230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:10:11.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.411+0000 7fc35bfff700 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x55732250caf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0a10a0 tx=0x7fc34c098f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b0aeab8e758e28 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:11.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.411+0000 7fc3427fc700 1 -- 172.21.15.3:0/2992571240 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c09d170 con 0x55732250c710 2023-09-10T01:10:11.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.411+0000 7fc3427fc700 1 -- 172.21.15.3:0/2992571240 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc34c09d390 con 0x55732250c710 2023-09-10T01:10:11.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.411+0000 7fc3427fc700 1 -- 172.21.15.3:0/2992571240 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0997c0 con 0x55732250c710 2023-09-10T01:10:11.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 -- 172.21.15.3:0/2992571240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 msgr2=0x55732250caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x55732250caf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0a10a0 tx=0x7fc34c098f40 comp rx=0 tx=0).stop 2023-09-10T01:10:11.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 -- 172.21.15.3:0/2992571240 shutdown_connections 2023-09-10T01:10:11.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557322516e40 0x557322519230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x55732250caf0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2992571240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 -- 172.21.15.3:0/2992571240 >> 172.21.15.3:0/2992571240 conn(0x557322481090 msgr2=0x557322507bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:11.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 -- 172.21.15.3:0/2992571240 shutdown_connections 2023-09-10T01:10:11.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 -- 172.21.15.3:0/2992571240 wait complete. 2023-09-10T01:10:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.412+0000 7fc375c65b80 1 -- start start 2023-09-10T01:10:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 1 Processor -- start 2023-09-10T01:10:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x5573225168f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322513b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x5573225142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573223fcb70 con 0x55732250d030 2023-09-10T01:10:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573223fc870 con 0x55732250c710 2023-09-10T01:10:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573222e0920 con 0x557322516e40 2023-09-10T01:10:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc375c65b80 10 client.0 _ll_register_callbacks cb 0x557322346a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:11.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.413+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x5573225168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x5573225168f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39720/0 (socket says 172.21.15.3:39720) 2023-09-10T01:10:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322513b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 -- 172.21.15.3:0/3608055058 learned_addr learned my addr 172.21.15.3:0/3608055058 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc36099c700 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x5573225142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 -- 172.21.15.3:0/3608055058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 msgr2=0x5573225142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x5573225142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 -- 172.21.15.3:0/3608055058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 msgr2=0x557322513b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322513b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 -- 172.21.15.3:0/3608055058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c1a5620 con 0x55732250c710 2023-09-10T01:10:11.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc36099c700 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x5573225142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:10:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc35bfff700 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x5573225168f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0c98b0 tx=0x7fc34c005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=881b982376d67beb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc33b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:10:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc33b7fe700 1 -- 172.21.15.3:0/3608055058 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c098a50 con 0x55732250c710 2023-09-10T01:10:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc33b7fe700 1 -- 172.21.15.3:0/3608055058 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc34c098bb0 con 0x55732250c710 2023-09-10T01:10:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.414+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573223b64f0 con 0x55732250c710 2023-09-10T01:10:11.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc33b7fe700 1 -- 172.21.15.3:0/3608055058 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc34c0997c0 con 0x55732250c710 2023-09-10T01:10:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 10 client.27551 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557322273c60 con 0x55732250c710 2023-09-10T01:10:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc33b7fe700 1 -- 172.21.15.3:0/3608055058 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5573223b64f0 con 0x55732250c710 2023-09-10T01:10:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc33b7fe700 1 -- 172.21.15.3:0/3608055058 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc34c0984c0 con 0x55732250c710 2023-09-10T01:10:11.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 10 client.27551 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 10 client.27551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 20 client.27551 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557322526760 0x557322528b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 4 client.27551 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:10:11.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573224738a0 con 0x557322526760 2023-09-10T01:10:11.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.415+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557322526760 0x557322528b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.416+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557322526760 0x557322528b80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f61b7297334fe6 server_cookie=88397115c57b07fd in_seq=0 out_seq=0 2023-09-10T01:10:11.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.416+0000 7fc33b7fe700 10 client.27551 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:11.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.416+0000 7fc33b7fe700 1 -- 172.21.15.3:0/3608055058 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5573224738a0 con 0x557322526760 2023-09-10T01:10:11.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.416+0000 7fc33b7fe700 10 client.27551 handle_command_reply: tid=0 2023-09-10T01:10:11.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 client.27551 shutdown 2023-09-10T01:10:11.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 msgr2=0x5573225168f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x5573225168f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0c98b0 tx=0x7fc34c005bb0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 shutdown_connections 2023-09-10T01:10:11.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557322526760 0x557322528b80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x5573225142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x5573225168f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc33b7fe700 0 client.27551 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:11.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 --2- 172.21.15.3:0/3608055058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322513b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 >> 172.21.15.3:0/3608055058 conn(0x557322481090 msgr2=0x5573225077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:11.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 shutdown_connections 2023-09-10T01:10:11.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 1 -- 172.21.15.3:0/3608055058 wait complete. 2023-09-10T01:10:11.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.417+0000 7fc375c65b80 20 client.27551 trim_cache size 0 max 16384 2023-09-10T01:10:11.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.436+0000 7fc375c65b80 1 Processor -- start 2023-09-10T01:10:11.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.436+0000 7fc375c65b80 1 -- start start 2023-09-10T01:10:11.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250c710 0x55732251d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557322516e40 0x557322510e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc375c65b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557322508780 con 0x55732250c710 2023-09-10T01:10:11.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc375c65b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557322515150 con 0x557322516e40 2023-09-10T01:10:11.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc375c65b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55732250d5e0 con 0x55732250d030 2023-09-10T01:10:11.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc36099c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557322516e40 0x557322510e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250c710 0x55732251d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc36099c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557322516e40 0x557322510e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39736/0 (socket says 172.21.15.3:39736) 2023-09-10T01:10:11.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc36099c700 1 -- 172.21.15.3:0/2857642183 learned_addr learned my addr 172.21.15.3:0/2857642183 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:11.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.437+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc36099c700 1 -- 172.21.15.3:0/2857642183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 msgr2=0x557322516490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc36099c700 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc36099c700 1 -- 172.21.15.3:0/2857642183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250c710 msgr2=0x55732251d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc36099c700 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250c710 0x55732251d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc36099c700 1 -- 172.21.15.3:0/2857642183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573222b1250 con 0x557322516e40 2023-09-10T01:10:11.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc36099c700 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557322516e40 0x557322510e00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc35400b070 tx=0x7fc3540095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc6821d25fa8a73 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:11.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc342ffd700 1 -- 172.21.15.3:0/2857642183 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35400e040 con 0x557322516e40 2023-09-10T01:10:11.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc342ffd700 1 -- 172.21.15.3:0/2857642183 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc35400be30 con 0x557322516e40 2023-09-10T01:10:11.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.438+0000 7fc342ffd700 1 -- 172.21.15.3:0/2857642183 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354025800 con 0x557322516e40 2023-09-10T01:10:11.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 -- 172.21.15.3:0/2857642183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557322516e40 msgr2=0x557322510e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557322516e40 0x557322510e00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc35400b070 tx=0x7fc3540095b0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 -- 172.21.15.3:0/2857642183 shutdown_connections 2023-09-10T01:10:11.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557322516e40 0x557322510e00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250c710 0x55732251d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 --2- 172.21.15.3:0/2857642183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55732250d030 0x557322516490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 -- 172.21.15.3:0/2857642183 >> 172.21.15.3:0/2857642183 conn(0x557322481090 msgr2=0x55732250eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:11.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 -- 172.21.15.3:0/2857642183 shutdown_connections 2023-09-10T01:10:11.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.439+0000 7fc375c65b80 1 -- 172.21.15.3:0/2857642183 wait complete. 2023-09-10T01:10:11.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 -- start start 2023-09-10T01:10:11.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 Processor -- start 2023-09-10T01:10:11.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x557322573fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322574510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x55732256e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55732256eac0 con 0x55732250d030 2023-09-10T01:10:11.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55732256ec30 con 0x55732250c710 2023-09-10T01:10:11.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55732256edf0 con 0x557322516e40 2023-09-10T01:10:11.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.440+0000 7fc375c65b80 10 client.0 _ll_register_callbacks cb 0x7fc34c099170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:11.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x557322573fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x557322573fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39752/0 (socket says 172.21.15.3:39752) 2023-09-10T01:10:11.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 -- 172.21.15.3:0/149794016 learned_addr learned my addr 172.21.15.3:0/149794016 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:11.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc36099c700 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x55732256e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322574510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 -- 172.21.15.3:0/149794016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 msgr2=0x55732256e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x55732256e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 -- 172.21.15.3:0/149794016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 msgr2=0x557322574510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:11.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322574510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 -- 172.21.15.3:0/149794016 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c10de10 con 0x55732250c710 2023-09-10T01:10:11.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc35bfff700 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x557322573fd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc34c000c00 tx=0x7fc34c0989a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a21c11a13a6df1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:11.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc33b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:10:11.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc33b7fe700 1 -- 172.21.15.3:0/149794016 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c09e0d0 con 0x55732250c710 2023-09-10T01:10:11.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc33b7fe700 1 -- 172.21.15.3:0/149794016 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc34c09d170 con 0x55732250c710 2023-09-10T01:10:11.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.441+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc34c09d390 con 0x55732250c710 2023-09-10T01:10:11.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc33b7fe700 1 -- 172.21.15.3:0/149794016 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc34c099c60 con 0x55732250c710 2023-09-10T01:10:11.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 10 client.27557 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:11.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5573222892b0 con 0x55732250c710 2023-09-10T01:10:11.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc33b7fe700 1 -- 172.21.15.3:0/149794016 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc34c09d390 con 0x55732250c710 2023-09-10T01:10:11.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc33b7fe700 1 -- 172.21.15.3:0/149794016 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc34c098a50 con 0x55732250c710 2023-09-10T01:10:11.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 10 client.27557 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:11.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 10 client.27557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:11.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 20 client.27557 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:11.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55732257b920 0x55732257dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:11.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 4 client.27557 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:10:11.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557322457920 con 0x55732257b920 2023-09-10T01:10:11.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.442+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55732257b920 0x55732257dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:11.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.443+0000 7fc35b7fe700 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55732257b920 0x55732257dd10 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd3c63213a31ed8c server_cookie=f07b391111d5fab9 in_seq=0 out_seq=0 2023-09-10T01:10:11.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.443+0000 7fc33b7fe700 10 client.27557 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:11.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.443+0000 7fc33b7fe700 1 -- 172.21.15.3:0/149794016 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x557322457920 con 0x55732257b920 2023-09-10T01:10:11.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.443+0000 7fc33b7fe700 10 client.27557 handle_command_reply: tid=0 2023-09-10T01:10:11.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.443+0000 7fc375c65b80 1 client.27557 shutdown 2023-09-10T01:10:11.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.443+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 msgr2=0x557322573fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.443+0000 7fc375c65b80 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x557322573fd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc34c000c00 tx=0x7fc34c0989a0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 shutdown_connections 2023-09-10T01:10:11.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55732257b920 0x55732257dd10 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557322516e40 0x55732256e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55732250c710 0x557322573fd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 --2- 172.21.15.3:0/149794016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55732250d030 0x557322574510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc33b7fe700 0 client.27557 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:11.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 >> 172.21.15.3:0/149794016 conn(0x557322481090 msgr2=0x55732250b700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:11.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 shutdown_connections 2023-09-10T01:10:11.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 1 -- 172.21.15.3:0/149794016 wait complete. 2023-09-10T01:10:11.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc375c65b80 20 client.27557 trim_cache size 0 max 16384 2023-09-10T01:10:11.459 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:10:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:10:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:10:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:10:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:10:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:10:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:10:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:10:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:10:11.461 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:10:11.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3440a07b0 msgr2=0x7fc3440a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc3623ff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3440a07b0 0x7fc3440a2c70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0740f0 tx=0x7fc350009260 comp rx=0 tx=0).stop 2023-09-10T01:10:11.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c100640 msgr2=0x7fc35c072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:11.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.444+0000 7fc3623ff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c100640 0x7fc35c072900 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00e9a0 tx=0x7fc34c00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 shutdown_connections 2023-09-10T01:10:11.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc3440a07b0 0x7fc3440a2c70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc35c106650 0x7fc35c1a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc35c100640 0x7fc35c072900 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 --2- 172.21.15.3:0/366502736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c101000 0x7fc35c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:11.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 >> 172.21.15.3:0/366502736 conn(0x7fc35c0fc170 msgr2=0x7fc35c0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:11.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 shutdown_connections 2023-09-10T01:10:11.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:11.445+0000 7fc3623ff700 1 -- 172.21.15.3:0/366502736 wait complete. 2023-09-10T01:10:11.467 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:10:11.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:11 smithi003 ceph-mon[97654]: pgmap v214: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.2 MiB/s wr, 467 op/s 2023-09-10T01:10:11.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:11 smithi136 ceph-mon[133708]: pgmap v214: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.2 MiB/s wr, 467 op/s 2023-09-10T01:10:11.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:11 smithi092 ceph-mon[134067]: pgmap v214: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.2 MiB/s wr, 467 op/s 2023-09-10T01:10:11.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 614675 9.20 MB/sec execute 233 sec latency 55.585 ms 2023-09-10T01:10:12.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 615890 9.18 MB/sec execute 234 sec latency 39.800 ms 2023-09-10T01:10:13.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:13 smithi136 ceph-mon[133708]: pgmap v215: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 908 KiB/s wr, 238 op/s 2023-09-10T01:10:13.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:13 smithi092 ceph-mon[134067]: pgmap v215: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 908 KiB/s wr, 238 op/s 2023-09-10T01:10:13.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 618099 9.20 MB/sec execute 235 sec latency 29.264 ms 2023-09-10T01:10:13.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:13 smithi003 ceph-mon[97654]: pgmap v215: 129 pgs: 129 active+clean; 175 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 908 KiB/s wr, 238 op/s 2023-09-10T01:10:14.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 619138 9.17 MB/sec execute 236 sec latency 68.406 ms 2023-09-10T01:10:15.775 INFO:tasks.workunit.client.0.smithi003.stdout: 1 621503 9.20 MB/sec execute 237 sec latency 18.619 ms 2023-09-10T01:10:16.414 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:16 smithi136 ceph-mon[133708]: pgmap v216: 129 pgs: 129 active+clean; 182 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 298 op/s 2023-09-10T01:10:16.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:16 smithi092 ceph-mon[134067]: pgmap v216: 129 pgs: 129 active+clean; 182 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 298 op/s 2023-09-10T01:10:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:16 smithi003 ceph-mon[97654]: pgmap v216: 129 pgs: 129 active+clean; 182 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 298 op/s 2023-09-10T01:10:16.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 622587 9.18 MB/sec execute 238 sec latency 19.524 ms 2023-09-10T01:10:17.774 INFO:tasks.workunit.client.0.smithi003.stdout: 1 624887 9.19 MB/sec execute 239 sec latency 16.905 ms 2023-09-10T01:10:18.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:18 smithi136 ceph-mon[133708]: pgmap v217: 129 pgs: 129 active+clean; 182 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 298 op/s 2023-09-10T01:10:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:18 smithi092 ceph-mon[134067]: pgmap v217: 129 pgs: 129 active+clean; 182 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 298 op/s 2023-09-10T01:10:18.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:18 smithi003 ceph-mon[97654]: pgmap v217: 129 pgs: 129 active+clean; 182 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 298 op/s 2023-09-10T01:10:18.775 INFO:tasks.workunit.client.0.smithi003.stdout: 1 626135 9.18 MB/sec execute 240 sec latency 14.882 ms 2023-09-10T01:10:19.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 628377 9.19 MB/sec execute 241 sec latency 15.426 ms 2023-09-10T01:10:20.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:20 smithi136 ceph-mon[133708]: pgmap v218: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2023-09-10T01:10:20.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:20 smithi092 ceph-mon[134067]: pgmap v218: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2023-09-10T01:10:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:20 smithi003 ceph-mon[97654]: pgmap v218: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 197 op/s 2023-09-10T01:10:20.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 629429 9.18 MB/sec execute 242 sec latency 33.418 ms 2023-09-10T01:10:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.278+0000 7fd7689ff700 1 -- 172.21.15.3:0/813893627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640732c0 msgr2=0x7fd764111070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.278+0000 7fd7689ff700 1 --2- 172.21.15.3:0/813893627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640732c0 0x7fd764111070 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd758009990 tx=0x7fd758009ca0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.279+0000 7fd7689ff700 1 -- 172.21.15.3:0/813893627 shutdown_connections 2023-09-10T01:10:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.279+0000 7fd7689ff700 1 --2- 172.21.15.3:0/813893627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7640729a0 0x7fd764072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.279+0000 7fd7689ff700 1 --2- 172.21.15.3:0/813893627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7641115b0 0x7fd7641139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.279+0000 7fd7689ff700 1 --2- 172.21.15.3:0/813893627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640732c0 0x7fd764111070 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.279+0000 7fd7689ff700 1 -- 172.21.15.3:0/813893627 >> 172.21.15.3:0/813893627 conn(0x7fd7640fc0b0 msgr2=0x7fd7640fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.279+0000 7fd7689ff700 1 -- 172.21.15.3:0/813893627 shutdown_connections 2023-09-10T01:10:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.279+0000 7fd7689ff700 1 -- 172.21.15.3:0/813893627 wait complete. 2023-09-10T01:10:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 Processor -- start 2023-09-10T01:10:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 -- start start 2023-09-10T01:10:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640729a0 0x7fd76419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7640732c0 0x7fd76419d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7641115b0 0x7fd7641a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641a1d50 con 0x7fd7640732c0 2023-09-10T01:10:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641a1ec0 con 0x7fd7640729a0 2023-09-10T01:10:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.280+0000 7fd7689ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641a2080 con 0x7fd7641115b0 2023-09-10T01:10:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640729a0 0x7fd76419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640729a0 0x7fd76419cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33926/0 (socket says 172.21.15.3:33926) 2023-09-10T01:10:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 -- 172.21.15.3:0/3234569676 learned_addr learned my addr 172.21.15.3:0/3234569676 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd762d9d700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7641115b0 0x7fd7641a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd761d9b700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7640732c0 0x7fd76419d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 -- 172.21.15.3:0/3234569676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7641115b0 msgr2=0x7fd7641a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7641115b0 0x7fd7641a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 -- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7640732c0 msgr2=0x7fd76419d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7640732c0 0x7fd76419d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 -- 172.21.15.3:0/3234569676 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7580095f0 con 0x7fd7640729a0 2023-09-10T01:10:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd762d9d700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7641115b0 0x7fd7641a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd76259c700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640729a0 0x7fd76419cd70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd75400e9a0 tx=0x7fd75400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.281+0000 7fd7537fe700 1 -- 172.21.15.3:0/3234569676 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754009790 con 0x7fd7640729a0 2023-09-10T01:10:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.282+0000 7fd7537fe700 1 -- 172.21.15.3:0/3234569676 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd754010430 con 0x7fd7640729a0 2023-09-10T01:10:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.282+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7641a2360 con 0x7fd7640729a0 2023-09-10T01:10:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.282+0000 7fd7537fe700 1 -- 172.21.15.3:0/3234569676 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754018680 con 0x7fd7640729a0 2023-09-10T01:10:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.282+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7641a9ba0 con 0x7fd7640729a0 2023-09-10T01:10:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.285+0000 7fd7537fe700 1 -- 172.21.15.3:0/3234569676 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd7540105a0 con 0x7fd7640729a0 2023-09-10T01:10:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.286+0000 7fd77c401b80 1 Processor -- start 2023-09-10T01:10:21.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.286+0000 7fd7537fe700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd74c0a07b0 0x7fd74c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd77c401b80 1 -- start start 2023-09-10T01:10:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd7537fe700 1 -- 172.21.15.3:0/3234569676 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd754014070 con 0x7fd7640729a0 2023-09-10T01:10:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd762d9d700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd74c0a07b0 0x7fd74c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bc5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bbf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bcbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd77c401b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391b186d0 con 0x55f391bbf300 2023-09-10T01:10:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd77c401b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391b15ff0 con 0x55f391bc51a0 2023-09-10T01:10:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd77c401b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391b118e0 con 0x55f391bbfcc0 2023-09-10T01:10:21.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd76259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bc5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd761d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bbf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd76259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bc5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33938/0 (socket says 172.21.15.3:33938) 2023-09-10T01:10:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd76259c700 1 -- 172.21.15.3:0/3955638793 learned_addr learned my addr 172.21.15.3:0/3955638793 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:21.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.287+0000 7fd762d9d700 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bcbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd76259c700 1 -- 172.21.15.3:0/3955638793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 msgr2=0x55f391bcbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd76259c700 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bcbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd76259c700 1 -- 172.21.15.3:0/3955638793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 msgr2=0x55f391bbf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd76259c700 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bbf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd76259c700 1 -- 172.21.15.3:0/3955638793 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f39193dd80 con 0x55f391bc51a0 2023-09-10T01:10:21.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd762d9d700 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bcbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:10:21.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd761d9b700 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bbf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:21.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd76259c700 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bc5580 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd754096070 tx=0x7fd754099030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=403e9d221c831a53 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:21.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd762d9d700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd74c0a07b0 0x7fd74c0a2c70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd764069b60 tx=0x7fd75c0045c0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:21.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd750ff9700 1 -- 172.21.15.3:0/3955638793 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75409d430 con 0x55f391bc51a0 2023-09-10T01:10:21.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd750ff9700 1 -- 172.21.15.3:0/3955638793 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7540a1670 con 0x55f391bc51a0 2023-09-10T01:10:21.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.288+0000 7fd750ff9700 1 -- 172.21.15.3:0/3955638793 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754084070 con 0x55f391bc51a0 2023-09-10T01:10:21.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 -- 172.21.15.3:0/3955638793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 msgr2=0x55f391bc5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bc5580 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd754096070 tx=0x7fd754099030 comp rx=0 tx=0).stop 2023-09-10T01:10:21.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 -- 172.21.15.3:0/3955638793 shutdown_connections 2023-09-10T01:10:21.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bcbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bc5580 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3955638793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bbf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 -- 172.21.15.3:0/3955638793 >> 172.21.15.3:0/3955638793 conn(0x55f391b34030 msgr2=0x55f391bba560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 -- 172.21.15.3:0/3955638793 shutdown_connections 2023-09-10T01:10:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.289+0000 7fd77c401b80 1 -- 172.21.15.3:0/3955638793 wait complete. 2023-09-10T01:10:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 -- start start 2023-09-10T01:10:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 Processor -- start 2023-09-10T01:10:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbf300 0x55f391bcbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bc90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bc51a0 0x55f391bc9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391aafb10 con 0x55f391bc51a0 2023-09-10T01:10:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391aaf810 con 0x55f391bbf300 2023-09-10T01:10:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3919938c0 con 0x55f391bbfcc0 2023-09-10T01:10:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.290+0000 7fd77c401b80 10 client.0 _ll_register_callbacks cb 0x55f3919f9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbf300 0x55f391bcbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbf300 0x55f391bcbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33952/0 (socket says 172.21.15.3:33952) 2023-09-10T01:10:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 -- 172.21.15.3:0/2614876678 learned_addr learned my addr 172.21.15.3:0/2614876678 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd761d9b700 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bc90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd762d9d700 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bc51a0 0x55f391bc9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 -- 172.21.15.3:0/2614876678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 msgr2=0x55f391bc90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bc90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 -- 172.21.15.3:0/2614876678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bc51a0 msgr2=0x55f391bc9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bc51a0 0x55f391bc9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 -- 172.21.15.3:0/2614876678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd764114350 con 0x55f391bbf300 2023-09-10T01:10:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd76259c700 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbf300 0x55f391bcbed0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd7540c9e20 tx=0x7fd7540c9810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efa48f36fade52d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd749ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:10:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd749ffb700 1 -- 172.21.15.3:0/2614876678 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7540b5070 con 0x55f391bbf300 2023-09-10T01:10:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f391a69490 con 0x55f391bbf300 2023-09-10T01:10:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.291+0000 7fd749ffb700 1 -- 172.21.15.3:0/2614876678 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd75409d430 con 0x55f391bbf300 2023-09-10T01:10:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd749ffb700 1 -- 172.21.15.3:0/2614876678 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd754082e00 con 0x55f391bbf300 2023-09-10T01:10:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 10 client.27569 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fd7641a9ba0 con 0x55f391bbf300 2023-09-10T01:10:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd749ffb700 1 -- 172.21.15.3:0/2614876678 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd75407c080 con 0x55f391bbf300 2023-09-10T01:10:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd749ffb700 1 -- 172.21.15.3:0/2614876678 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fd754098af0 con 0x55f391bbf300 2023-09-10T01:10:21.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 10 client.27569 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 10 client.27569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 20 client.27569 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391bd9080 0x55f391bdb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 4 client.27569 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:10:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f391b26840 con 0x55f391bd9080 2023-09-10T01:10:21.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.292+0000 7fd761d9b700 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391bd9080 0x55f391bdb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.293+0000 7fd761d9b700 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391bd9080 0x55f391bdb4a0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=928e40e82857dcef server_cookie=579981cadfcbc622 in_seq=0 out_seq=0 2023-09-10T01:10:21.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.293+0000 7fd749ffb700 10 client.27569 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:21.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.293+0000 7fd749ffb700 1 -- 172.21.15.3:0/2614876678 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f391b26840 con 0x55f391bd9080 2023-09-10T01:10:21.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.293+0000 7fd749ffb700 10 client.27569 handle_command_reply: tid=0 2023-09-10T01:10:21.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 client.27569 shutdown 2023-09-10T01:10:21.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbf300 msgr2=0x55f391bcbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbf300 0x55f391bcbed0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd7540c9e20 tx=0x7fd7540c9810 comp rx=0 tx=0).stop 2023-09-10T01:10:21.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 shutdown_connections 2023-09-10T01:10:21.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391bd9080 0x55f391bdb4a0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bc51a0 0x55f391bc9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbf300 0x55f391bcbed0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2614876678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbfcc0 0x55f391bc90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd749ffb700 0 client.27569 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:21.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 >> 172.21.15.3:0/2614876678 conn(0x55f391b34030 msgr2=0x55f391bb97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 shutdown_connections 2023-09-10T01:10:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 1 -- 172.21.15.3:0/2614876678 wait complete. 2023-09-10T01:10:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.294+0000 7fd77c401b80 20 client.27569 trim_cache size 0 max 16384 2023-09-10T01:10:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.310+0000 7fd77c401b80 1 Processor -- start 2023-09-10T01:10:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.310+0000 7fd77c401b80 1 -- start start 2023-09-10T01:10:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bc0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbfcc0 0x55f391bcba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bc51a0 0x55f391bc40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd77c401b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391bc9e60 con 0x55f391bbf300 2023-09-10T01:10:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd77c401b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391bc5a50 con 0x55f391bbfcc0 2023-09-10T01:10:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd77c401b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391bc3a20 con 0x55f391bc51a0 2023-09-10T01:10:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bc0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd761d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbfcc0 0x55f391bcba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd762d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bc51a0 0x55f391bc40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bc0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43308/0 (socket says 172.21.15.3:43308) 2023-09-10T01:10:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 -- 172.21.15.3:0/3202393978 learned_addr learned my addr 172.21.15.3:0/3202393978 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:21.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 -- 172.21.15.3:0/3202393978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bc51a0 msgr2=0x55f391bc40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bc51a0 0x55f391bc40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 -- 172.21.15.3:0/3202393978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbfcc0 msgr2=0x55f391bcba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbfcc0 0x55f391bcba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd76259c700 1 -- 172.21.15.3:0/3202393978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f391926c00 con 0x55f391bbf300 2023-09-10T01:10:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.311+0000 7fd762d9d700 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bc51a0 0x55f391bc40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.312+0000 7fd76259c700 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bc0140 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd754099540 tx=0x7fd754099740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486d40e8d7cd0d98 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.312+0000 7fd7517fa700 1 -- 172.21.15.3:0/3202393978 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754099d80 con 0x55f391bbf300 2023-09-10T01:10:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.312+0000 7fd7517fa700 1 -- 172.21.15.3:0/3202393978 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f391a69490 con 0x55f391bbf300 2023-09-10T01:10:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.312+0000 7fd7517fa700 1 -- 172.21.15.3:0/3202393978 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754098af0 con 0x55f391bbf300 2023-09-10T01:10:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.312+0000 7fd77c401b80 1 -- 172.21.15.3:0/3202393978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 msgr2=0x55f391bc0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.312+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bc0140 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd754099540 tx=0x7fd754099740 comp rx=0 tx=0).stop 2023-09-10T01:10:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 -- 172.21.15.3:0/3202393978 shutdown_connections 2023-09-10T01:10:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bc51a0 0x55f391bc40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbf300 0x55f391bc0140 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 --2- 172.21.15.3:0/3202393978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bbfcc0 0x55f391bcba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 -- 172.21.15.3:0/3202393978 >> 172.21.15.3:0/3202393978 conn(0x55f391b34030 msgr2=0x55f391bd56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 -- 172.21.15.3:0/3202393978 shutdown_connections 2023-09-10T01:10:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 -- 172.21.15.3:0/3202393978 wait complete. 2023-09-10T01:10:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 -- start start 2023-09-10T01:10:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.313+0000 7fd77c401b80 1 Processor -- start 2023-09-10T01:10:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbf300 0x55f391bcdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbfcc0 0x55f391bce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd77c401b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd77c401b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391c2c040 con 0x55f391bbfcc0 2023-09-10T01:10:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd77c401b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391c2c1b0 con 0x55f391bc51a0 2023-09-10T01:10:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd77c401b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f391c2c370 con 0x55f391bbf300 2023-09-10T01:10:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd77c401b80 10 client.0 _ll_register_callbacks cb 0x7fd75c0097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd761d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbfcc0 0x55f391bce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd762d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd76259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbf300 0x55f391bcdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd761d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbfcc0 0x55f391bce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43320/0 (socket says 172.21.15.3:43320) 2023-09-10T01:10:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd761d9b700 1 -- 172.21.15.3:0/2642404765 learned_addr learned my addr 172.21.15.3:0/2642404765 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd761d9b700 1 -- 172.21.15.3:0/2642404765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbf300 msgr2=0x55f391bcdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd761d9b700 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbf300 0x55f391bcdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd761d9b700 1 -- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 msgr2=0x55f391bce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.314+0000 7fd761d9b700 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd761d9b700 1 -- 172.21.15.3:0/2642404765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75c00e630 con 0x55f391bbfcc0 2023-09-10T01:10:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd762d9d700 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd76259c700 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbf300 0x55f391bcdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd761d9b700 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbfcc0 0x55f391bce280 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd758009990 tx=0x7fd75800bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af3b3d3ff79e1c4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd749ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:10:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd749ffb700 1 -- 172.21.15.3:0/2642404765 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75801d040 con 0x55f391bbfcc0 2023-09-10T01:10:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7540a1670 con 0x55f391bbfcc0 2023-09-10T01:10:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd749ffb700 1 -- 172.21.15.3:0/2642404765 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd758029cf0 con 0x55f391bbfcc0 2023-09-10T01:10:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd749ffb700 1 -- 172.21.15.3:0/2642404765 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7580290c0 con 0x55f391bbfcc0 2023-09-10T01:10:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd77c401b80 10 client.17490 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.315+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fd7641a9ba0 con 0x55f391bbfcc0 2023-09-10T01:10:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd749ffb700 1 -- 172.21.15.3:0/2642404765 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7540a1670 con 0x55f391bbfcc0 2023-09-10T01:10:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd749ffb700 1 -- 172.21.15.3:0/2642404765 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fd758004500 con 0x55f391bbfcc0 2023-09-10T01:10:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd77c401b80 10 client.17490 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd77c401b80 10 client.17490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd77c401b80 20 client.17490 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391c2dc70 0x55f391c30090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd77c401b80 4 client.17490 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:10:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f391b0a8c0 con 0x55f391c2dc70 2023-09-10T01:10:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.316+0000 7fd76259c700 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391c2dc70 0x55f391c30090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.317+0000 7fd76259c700 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391c2dc70 0x55f391c30090 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=994c91487e7c813f server_cookie=901097dd22845c33 in_seq=0 out_seq=0 2023-09-10T01:10:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.317+0000 7fd749ffb700 10 client.17490 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd749ffb700 1 -- 172.21.15.3:0/2642404765 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f391b0a8c0 con 0x55f391c2dc70 2023-09-10T01:10:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd749ffb700 10 client.17490 handle_command_reply: tid=0 2023-09-10T01:10:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 client.17490 shutdown 2023-09-10T01:10:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbfcc0 msgr2=0x55f391bce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbfcc0 0x55f391bce280 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd758009990 tx=0x7fd75800bc00 comp rx=0 tx=0).stop 2023-09-10T01:10:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 shutdown_connections 2023-09-10T01:10:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f391c2dc70 0x55f391c30090 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f391bbf300 0x55f391bcdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f391bc51a0 0x55f391bce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 --2- 172.21.15.3:0/2642404765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f391bbfcc0 0x55f391bce280 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd749ffb700 0 client.17490 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.419+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 >> 172.21.15.3:0/2642404765 conn(0x55f391b34030 msgr2=0x55f391bc40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.420+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 shutdown_connections 2023-09-10T01:10:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.420+0000 7fd77c401b80 1 -- 172.21.15.3:0/2642404765 wait complete. 2023-09-10T01:10:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.420+0000 7fd77c401b80 20 client.17490 trim_cache size 0 max 16384 2023-09-10T01:10:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:10:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:10:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:10:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:10:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.421+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd74c0a07b0 msgr2=0x7fd74c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.421+0000 7fd7689ff700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd74c0a07b0 0x7fd74c0a2c70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd764069b60 tx=0x7fd75c0045c0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.421+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640729a0 msgr2=0x7fd76419cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.421+0000 7fd7689ff700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640729a0 0x7fd76419cd70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd75400e9a0 tx=0x7fd75400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 shutdown_connections 2023-09-10T01:10:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd74c0a07b0 0x7fd74c0a2c70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7641115b0 0x7fd7641a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7640729a0 0x7fd76419cd70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 --2- 172.21.15.3:0/3234569676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7640732c0 0x7fd76419d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 >> 172.21.15.3:0/3234569676 conn(0x7fd7640fc0b0 msgr2=0x7fd7640fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 shutdown_connections 2023-09-10T01:10:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:21.422+0000 7fd7689ff700 1 -- 172.21.15.3:0/3234569676 wait complete. 2023-09-10T01:10:21.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 631225 9.17 MB/sec execute 243 sec latency 109.650 ms 2023-09-10T01:10:22.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:22 smithi136 ceph-mon[133708]: pgmap v219: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2023-09-10T01:10:22.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:22 smithi092 ceph-mon[134067]: pgmap v219: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2023-09-10T01:10:22.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:22 smithi003 ceph-mon[97654]: pgmap v219: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 192 op/s 2023-09-10T01:10:22.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 632760 9.18 MB/sec execute 244 sec latency 13.832 ms 2023-09-10T01:10:23.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 634749 9.17 MB/sec execute 245 sec latency 15.599 ms 2023-09-10T01:10:24.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:24 smithi136 ceph-mon[133708]: pgmap v220: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2023-09-10T01:10:24.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:24 smithi092 ceph-mon[134067]: pgmap v220: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2023-09-10T01:10:24.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:24 smithi003 ceph-mon[97654]: pgmap v220: 129 pgs: 129 active+clean; 190 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2023-09-10T01:10:24.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 636386 9.18 MB/sec execute 246 sec latency 16.809 ms 2023-09-10T01:10:25.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 637689 9.17 MB/sec execute 247 sec latency 88.105 ms 2023-09-10T01:10:26.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:26 smithi136 ceph-mon[133708]: pgmap v221: 129 pgs: 129 active+clean; 186 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 284 op/s 2023-09-10T01:10:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:26 smithi092 ceph-mon[134067]: pgmap v221: 129 pgs: 129 active+clean; 186 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 284 op/s 2023-09-10T01:10:26.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:26 smithi003 ceph-mon[97654]: pgmap v221: 129 pgs: 129 active+clean; 186 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 284 op/s 2023-09-10T01:10:26.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 639646 9.18 MB/sec execute 248 sec latency 28.606 ms 2023-09-10T01:10:27.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 641234 9.17 MB/sec execute 249 sec latency 17.636 ms 2023-09-10T01:10:28.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:28 smithi136 ceph-mon[133708]: pgmap v222: 129 pgs: 129 active+clean; 186 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2023-09-10T01:10:28.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:28 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:10:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:28 smithi092 ceph-mon[134067]: pgmap v222: 129 pgs: 129 active+clean; 186 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2023-09-10T01:10:28.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:28 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:10:28.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:28 smithi003 ceph-mon[97654]: pgmap v222: 129 pgs: 129 active+clean; 186 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 196 op/s 2023-09-10T01:10:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:28 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:10:28.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 643182 9.19 MB/sec execute 250 sec latency 14.273 ms 2023-09-10T01:10:29.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:29 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:10:29.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:29 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:10:29.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:29 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:10:29.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:29 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:10:29.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:29 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:10:29.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:29 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:10:29.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:29 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:10:29.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:29 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:10:29.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:29 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:10:29.776 INFO:tasks.workunit.client.0.smithi003.stdout: 1 644564 9.17 MB/sec execute 251 sec latency 21.015 ms 2023-09-10T01:10:30.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:30 smithi136 ceph-mon[133708]: pgmap v223: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2023-09-10T01:10:30.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:30 smithi092 ceph-mon[134067]: pgmap v223: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2023-09-10T01:10:30.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:30 smithi003 ceph-mon[97654]: pgmap v223: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2023-09-10T01:10:30.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 646641 9.19 MB/sec execute 252 sec latency 34.470 ms 2023-09-10T01:10:31.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 647884 9.17 MB/sec execute 253 sec latency 56.092 ms 2023-09-10T01:10:32.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:32 smithi136 ceph-mon[133708]: pgmap v224: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2023-09-10T01:10:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:32 smithi092 ceph-mon[134067]: pgmap v224: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2023-09-10T01:10:32.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:32 smithi003 ceph-mon[97654]: pgmap v224: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2023-09-10T01:10:32.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 650136 9.19 MB/sec execute 254 sec latency 16.963 ms 2023-09-10T01:10:33.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 651397 9.18 MB/sec execute 255 sec latency 18.044 ms 2023-09-10T01:10:34.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:34 smithi136 ceph-mon[133708]: pgmap v225: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2023-09-10T01:10:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:34 smithi092 ceph-mon[134067]: pgmap v225: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2023-09-10T01:10:34.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:34 smithi003 ceph-mon[97654]: pgmap v225: 129 pgs: 129 active+clean; 193 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 202 op/s 2023-09-10T01:10:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 -- 172.21.15.3:0/3255610016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 msgr2=0x7ff7b0100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/3255610016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b0100a40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009990 tx=0x7ff7a4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 -- 172.21.15.3:0/3255610016 shutdown_connections 2023-09-10T01:10:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/3255610016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7b0100f80 0x7ff7b010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/3255610016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7b0106680 0x7ff7b0106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/3255610016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b0100a40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 -- 172.21.15.3:0/3255610016 >> 172.21.15.3:0/3255610016 conn(0x7ff7b00fc0b0 msgr2=0x7ff7b00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 -- 172.21.15.3:0/3255610016 shutdown_connections 2023-09-10T01:10:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.625+0000 7ff7b49ff700 1 -- 172.21.15.3:0/3255610016 wait complete. 2023-09-10T01:10:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.626+0000 7ff7b49ff700 1 Processor -- start 2023-09-10T01:10:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7b49ff700 1 -- start start 2023-09-10T01:10:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7b49ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7b49ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7b0100f80 0x7ff7b019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7b49ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7b0106680 0x7ff7b01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7b49ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0198d60 con 0x7ff7b0100f80 2023-09-10T01:10:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7b49ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0198ed0 con 0x7ff7b0106680 2023-09-10T01:10:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7b49ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0199090 con 0x7ff7b01005c0 2023-09-10T01:10:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7ae59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7ae59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:55998/0 (socket says 172.21.15.3:55998) 2023-09-10T01:10:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7ae59c700 1 -- 172.21.15.3:0/1748569915 learned_addr learned my addr 172.21.15.3:0/1748569915 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7aed9d700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7b0106680 0x7ff7b01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.627+0000 7ff7add9b700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7b0100f80 0x7ff7b019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7ae59c700 1 -- 172.21.15.3:0/1748569915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7b0106680 msgr2=0x7ff7b01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7b0106680 0x7ff7b01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7ae59c700 1 -- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7b0100f80 msgr2=0x7ff7b019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7b0100f80 0x7ff7b019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7ae59c700 1 -- 172.21.15.3:0/1748569915 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40095f0 con 0x7ff7b01005c0 2023-09-10T01:10:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7add9b700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7b0100f80 0x7ff7b019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:10:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b019e680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000e9a0 tx=0x7ff7a000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff79f7fe700 1 -- 172.21.15.3:0/1748569915 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0009790 con 0x7ff7b01005c0 2023-09-10T01:10:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff79f7fe700 1 -- 172.21.15.3:0/1748569915 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7a000cbd0 con 0x7ff7b01005c0 2023-09-10T01:10:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b0199370 con 0x7ff7b01005c0 2023-09-10T01:10:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff79f7fe700 1 -- 172.21.15.3:0/1748569915 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0018680 con 0x7ff7b01005c0 2023-09-10T01:10:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.628+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b01a5670 con 0x7ff7b01005c0 2023-09-10T01:10:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.631+0000 7ff79f7fe700 1 -- 172.21.15.3:0/1748569915 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ff7a000cd40 con 0x7ff7b01005c0 2023-09-10T01:10:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.633+0000 7ff7c825ab80 1 Processor -- start 2023-09-10T01:10:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.633+0000 7ff79f7fe700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff7980a0830 0x7ff7980a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.633+0000 7ff7c825ab80 1 -- start start 2023-09-10T01:10:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.633+0000 7ff79f7fe700 1 -- 172.21.15.3:0/1748569915 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7ff7a0014070 con 0x7ff7b01005c0 2023-09-10T01:10:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.633+0000 7ff7aed9d700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff7980a0830 0x7ff7980a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51f9c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51ef56d0 con 0x563f51fa21a0 2023-09-10T01:10:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51ef2ff0 con 0x563f51f9ccc0 2023-09-10T01:10:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51eee8e0 con 0x563f51f9c300 2023-09-10T01:10:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7add9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51f9c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44108/0 (socket says 172.21.15.3:44108) 2023-09-10T01:10:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7ae59c700 1 -- 172.21.15.3:0/1311467473 learned_addr learned my addr 172.21.15.3:0/1311467473 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7aed9d700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff7980a0830 0x7ff7980a2cf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7b019f380 tx=0x7ff7a800b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.634+0000 7ff7aed9d700 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff7add9b700 1 -- 172.21.15.3:0/1311467473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 msgr2=0x563f51fa8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff7add9b700 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff7add9b700 1 -- 172.21.15.3:0/1311467473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 msgr2=0x563f51fa2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff7add9b700 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff7add9b700 1 -- 172.21.15.3:0/1311467473 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f51d1ad80 con 0x563f51f9c300 2023-09-10T01:10:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff7add9b700 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51f9c780 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0199b60 tx=0x7ff7a4005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74c7967063aece0d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff79cff9700 1 -- 172.21.15.3:0/1311467473 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a400beb0 con 0x563f51f9c300 2023-09-10T01:10:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff79cff9700 1 -- 172.21.15.3:0/1311467473 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7a401e040 con 0x563f51f9c300 2023-09-10T01:10:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.635+0000 7ff79cff9700 1 -- 172.21.15.3:0/1311467473 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4004b00 con 0x563f51f9c300 2023-09-10T01:10:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 -- 172.21.15.3:0/1311467473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 msgr2=0x563f51f9c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51f9c780 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0199b60 tx=0x7ff7a4005c00 comp rx=0 tx=0).stop 2023-09-10T01:10:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 -- 172.21.15.3:0/1311467473 shutdown_connections 2023-09-10T01:10:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/1311467473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51f9c780 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 -- 172.21.15.3:0/1311467473 >> 172.21.15.3:0/1311467473 conn(0x563f51f11030 msgr2=0x563f51f97560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 -- 172.21.15.3:0/1311467473 shutdown_connections 2023-09-10T01:10:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.636+0000 7ff7c825ab80 1 -- 172.21.15.3:0/1311467473 wait complete. 2023-09-10T01:10:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 -- start start 2023-09-10T01:10:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 Processor -- start 2023-09-10T01:10:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9c300 0x563f51fa8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51e8cb10 con 0x563f51f9c300 2023-09-10T01:10:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51e8c810 con 0x563f51f9ccc0 2023-09-10T01:10:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.637+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51d708c0 con 0x563f51fa21a0 2023-09-10T01:10:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7c825ab80 10 client.0 _ll_register_callbacks cb 0x563f51dd6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40926/0 (socket says 172.21.15.3:40926) 2023-09-10T01:10:34.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 -- 172.21.15.3:0/2518595665 learned_addr learned my addr 172.21.15.3:0/2518595665 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9c300 0x563f51fa8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7aed9d700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 -- 172.21.15.3:0/2518595665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 msgr2=0x563f51fa6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 -- 172.21.15.3:0/2518595665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9c300 msgr2=0x563f51fa8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9c300 0x563f51fa8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 -- 172.21.15.3:0/2518595665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b01a5670 con 0x563f51f9ccc0 2023-09-10T01:10:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9c300 0x563f51fa8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:10:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa6110 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009ce0 tx=0x7ff7a4005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=664f61b3b1f2c993 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.638+0000 7ff795ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:10:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff795ffb700 1 -- 172.21.15.3:0/2518595665 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4004700 con 0x563f51f9ccc0 2023-09-10T01:10:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f51e46490 con 0x563f51f9ccc0 2023-09-10T01:10:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff795ffb700 1 -- 172.21.15.3:0/2518595665 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7ff7a4004860 con 0x563f51f9ccc0 2023-09-10T01:10:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff795ffb700 1 -- 172.21.15.3:0/2518595665 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7a4004a20 con 0x563f51f9ccc0 2023-09-10T01:10:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff7c825ab80 10 client.27587 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:34.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ff7b010dd40 con 0x563f51f9ccc0 2023-09-10T01:10:34.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff795ffb700 1 -- 172.21.15.3:0/2518595665 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563f51e46490 con 0x563f51f9ccc0 2023-09-10T01:10:34.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.639+0000 7ff795ffb700 1 -- 172.21.15.3:0/2518595665 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff7a400fbf0 con 0x563f51f9ccc0 2023-09-10T01:10:34.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7c825ab80 10 client.27587 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:34.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7c825ab80 10 client.27587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7c825ab80 20 client.27587 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f51fb5fb0 0x563f51fb83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7c825ab80 4 client.27587 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:10:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f51f03840 con 0x563f51fb5fb0 2023-09-10T01:10:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f51fb5fb0 0x563f51fb83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f51fb5fb0 0x563f51fb83d0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a7a699a4b6f0029 server_cookie=f2723f720b8177c7 in_seq=0 out_seq=0 2023-09-10T01:10:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.640+0000 7ff795ffb700 10 client.27587 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.641+0000 7ff795ffb700 1 -- 172.21.15.3:0/2518595665 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563f51f03840 con 0x563f51fb5fb0 2023-09-10T01:10:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.641+0000 7ff795ffb700 10 client.27587 handle_command_reply: tid=0 2023-09-10T01:10:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.641+0000 7ff7c825ab80 1 client.27587 shutdown 2023-09-10T01:10:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 msgr2=0x563f51fa6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa6110 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009ce0 tx=0x7ff7a4005a00 comp rx=0 tx=0).stop 2023-09-10T01:10:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 shutdown_connections 2023-09-10T01:10:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f51fb5fb0 0x563f51fb83d0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9c300 0x563f51fa8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2518595665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa6110 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff795ffb700 0 client.27587 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 >> 172.21.15.3:0/2518595665 conn(0x563f51f11030 msgr2=0x563f51f967a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 shutdown_connections 2023-09-10T01:10:34.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2518595665 wait complete. 2023-09-10T01:10:34.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.642+0000 7ff7c825ab80 20 client.27587 trim_cache size 0 max 16384 2023-09-10T01:10:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 Processor -- start 2023-09-10T01:10:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 -- start start 2023-09-10T01:10:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51fa1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51f918a0 con 0x563f51fa21a0 2023-09-10T01:10:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51f98930 con 0x563f51f9ccc0 2023-09-10T01:10:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f51fa4670 con 0x563f51f9c300 2023-09-10T01:10:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7aed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55584/0 (socket says 172.21.15.3:55584) 2023-09-10T01:10:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7add9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60964/0 (socket says 172.21.15.3:60964) 2023-09-10T01:10:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7aed9d700 1 -- 172.21.15.3:0/2732979697 learned_addr learned my addr 172.21.15.3:0/2732979697 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.659+0000 7ff7ae59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51fa1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff7add9b700 1 -- 172.21.15.3:0/2732979697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 msgr2=0x563f51fa1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51fa1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff7add9b700 1 -- 172.21.15.3:0/2732979697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 msgr2=0x563f51fa0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff7add9b700 1 -- 172.21.15.3:0/2732979697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f51d03c00 con 0x563f51f9ccc0 2023-09-10T01:10:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff7aed9d700 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa15c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400b2e0 tx=0x7ff7a4009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e74c03b0c1e6498 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:34.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff79d7fa700 1 -- 172.21.15.3:0/2732979697 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4022ac0 con 0x563f51f9ccc0 2023-09-10T01:10:34.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.660+0000 7ff79d7fa700 1 -- 172.21.15.3:0/2732979697 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7a4005c00 con 0x563f51f9ccc0 2023-09-10T01:10:34.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff79d7fa700 1 -- 172.21.15.3:0/2732979697 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4004d60 con 0x563f51f9ccc0 2023-09-10T01:10:34.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2732979697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 msgr2=0x563f51fa15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa15c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400b2e0 tx=0x7ff7a4009290 comp rx=0 tx=0).stop 2023-09-10T01:10:34.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2732979697 shutdown_connections 2023-09-10T01:10:34.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51f9c300 0x563f51fa1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51fa21a0 0x563f51fa0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2732979697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9ccc0 0x563f51fa15c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2732979697 >> 172.21.15.3:0/2732979697 conn(0x563f51f11030 msgr2=0x563f51fb2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:34.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2732979697 shutdown_connections 2023-09-10T01:10:34.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.661+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2732979697 wait complete. 2023-09-10T01:10:34.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 -- start start 2023-09-10T01:10:34.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 Processor -- start 2023-09-10T01:10:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9c300 0x563f51fa9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9ccc0 0x563f51fa9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f52008d90 con 0x563f51f9ccc0 2023-09-10T01:10:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f52008ed0 con 0x563f51f9c300 2023-09-10T01:10:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.662+0000 7ff7c825ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f52009010 con 0x563f51fa21a0 2023-09-10T01:10:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7c825ab80 10 client.0 _ll_register_callbacks cb 0x7ff7a0095920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9c300 0x563f51fa9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7aed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9c300 0x563f51fa9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60970/0 (socket says 172.21.15.3:60970) 2023-09-10T01:10:34.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 -- 172.21.15.3:0/2657196017 learned_addr learned my addr 172.21.15.3:0/2657196017 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9ccc0 0x563f51fa9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 -- 172.21.15.3:0/2657196017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 msgr2=0x563f51fa9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 -- 172.21.15.3:0/2657196017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9ccc0 msgr2=0x563f51fa9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9ccc0 0x563f51fa9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 -- 172.21.15.3:0/2657196017 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a80174c0 con 0x563f51f9c300 2023-09-10T01:10:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7aed9d700 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff7ae59c700 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9c300 0x563f51fa9400 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0095ae0 tx=0x7ff7a0084040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3436a7708f7bb0e0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.663+0000 7ff795ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:10:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff795ffb700 1 -- 172.21.15.3:0/2657196017 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a009d410 con 0x563f51f9c300 2023-09-10T01:10:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff795ffb700 1 -- 172.21.15.3:0/2657196017 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7ff7a00a1650 con 0x563f51f9c300 2023-09-10T01:10:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff7a401e040 con 0x563f51f9c300 2023-09-10T01:10:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff795ffb700 1 -- 172.21.15.3:0/2657196017 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7a00840b0 con 0x563f51f9c300 2023-09-10T01:10:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 10 client.27593 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:34.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ff7b010dd40 con 0x563f51f9c300 2023-09-10T01:10:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff795ffb700 1 -- 172.21.15.3:0/2657196017 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7a401e040 con 0x563f51f9c300 2023-09-10T01:10:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff795ffb700 1 -- 172.21.15.3:0/2657196017 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff7a00b1be0 con 0x563f51f9c300 2023-09-10T01:10:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 10 client.27593 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 10 client.27593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 20 client.27593 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f5200a690 0x563f5200cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 4 client.27593 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:10:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563f51ee78c0 con 0x563f5200a690 2023-09-10T01:10:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.664+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f5200a690 0x563f5200cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.665+0000 7ff795ffb700 10 client.27593 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.665+0000 7ff7add9b700 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f5200a690 0x563f5200cab0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84f1c7eca73423c6 server_cookie=2b1a2ca05a67b768 in_seq=0 out_seq=0 2023-09-10T01:10:34.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 652287 9.15 MB/sec execute 256 sec latency 125.232 ms 2023-09-10T01:10:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.798+0000 7ff795ffb700 1 -- 172.21.15.3:0/2657196017 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563f51ee78c0 con 0x563f5200a690 2023-09-10T01:10:34.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.798+0000 7ff795ffb700 10 client.27593 handle_command_reply: tid=0 2023-09-10T01:10:34.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.798+0000 7ff7c825ab80 1 client.27593 shutdown 2023-09-10T01:10:34.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9c300 msgr2=0x563f51fa9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9c300 0x563f51fa9400 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0095ae0 tx=0x7ff7a0084040 comp rx=0 tx=0).stop 2023-09-10T01:10:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 shutdown_connections 2023-09-10T01:10:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563f5200a690 0x563f5200cab0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563f51fa21a0 0x563f51fa9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563f51f9c300 0x563f51fa9400 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 --2- 172.21.15.3:0/2657196017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f51f9ccc0 0x563f51fa9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 >> 172.21.15.3:0/2657196017 conn(0x563f51f11030 msgr2=0x563f51fa1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:34.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.799+0000 7ff795ffb700 0 client.27593 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:34.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.800+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 shutdown_connections 2023-09-10T01:10:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.800+0000 7ff7c825ab80 1 -- 172.21.15.3:0/2657196017 wait complete. 2023-09-10T01:10:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.801+0000 7ff7c825ab80 20 client.27593 trim_cache size 0 max 16384 2023-09-10T01:10:34.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:10:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:10:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:10:34.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:10:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.802+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff7980a0830 msgr2=0x7ff7980a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.802+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff7980a0830 0x7ff7980a2cf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7b019f380 tx=0x7ff7a800b040 comp rx=0 tx=0).stop 2023-09-10T01:10:34.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.802+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 msgr2=0x7ff7b019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.802+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b019e680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000e9a0 tx=0x7ff7a000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.803+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 shutdown_connections 2023-09-10T01:10:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.803+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff7980a0830 0x7ff7980a2cf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.803+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b01005c0 0x7ff7b019e680 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.803+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7b0106680 0x7ff7b01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.803+0000 7ff7b49ff700 1 --2- 172.21.15.3:0/1748569915 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7b0100f80 0x7ff7b019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.803+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 >> 172.21.15.3:0/1748569915 conn(0x7ff7b00fc0b0 msgr2=0x7ff7b00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:34.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.805+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 shutdown_connections 2023-09-10T01:10:34.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:34.805+0000 7ff7b49ff700 1 -- 172.21.15.3:0/1748569915 wait complete. 2023-09-10T01:10:35.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 653669 9.16 MB/sec execute 257 sec latency 40.583 ms 2023-09-10T01:10:36.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:36 smithi136 ceph-mon[133708]: pgmap v226: 129 pgs: 129 active+clean; 201 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 277 op/s 2023-09-10T01:10:36.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:36 smithi092 ceph-mon[134067]: pgmap v226: 129 pgs: 129 active+clean; 201 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 277 op/s 2023-09-10T01:10:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:36 smithi003 ceph-mon[97654]: pgmap v226: 129 pgs: 129 active+clean; 201 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 277 op/s 2023-09-10T01:10:36.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 654865 9.14 MB/sec execute 258 sec latency 15.352 ms 2023-09-10T01:10:37.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 657128 9.16 MB/sec execute 259 sec latency 21.258 ms 2023-09-10T01:10:38.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:38 smithi136 ceph-mon[133708]: pgmap v227: 129 pgs: 129 active+clean; 201 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:10:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:38 smithi092 ceph-mon[134067]: pgmap v227: 129 pgs: 129 active+clean; 201 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:10:38.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:38 smithi003 ceph-mon[97654]: pgmap v227: 129 pgs: 129 active+clean; 201 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:10:38.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 658215 9.14 MB/sec execute 260 sec latency 14.358 ms 2023-09-10T01:10:39.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 660525 9.16 MB/sec execute 261 sec latency 16.435 ms 2023-09-10T01:10:40.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:40 smithi136 ceph-mon[133708]: pgmap v228: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 418 op/s 2023-09-10T01:10:40.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:40 smithi092 ceph-mon[134067]: pgmap v228: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 418 op/s 2023-09-10T01:10:40.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:40 smithi003 ceph-mon[97654]: pgmap v228: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.7 MiB/s wr, 418 op/s 2023-09-10T01:10:40.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 661579 9.14 MB/sec execute 262 sec latency 18.769 ms 2023-09-10T01:10:41.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:10:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 -- 172.21.15.3:0/519019910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8106650 msgr2=0x7f98d8106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 --2- 172.21.15.3:0/519019910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8106650 0x7f98d8106a30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009990 tx=0x7f98d4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 -- 172.21.15.3:0/519019910 shutdown_connections 2023-09-10T01:10:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 --2- 172.21.15.3:0/519019910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8101000 0x7f98d810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 --2- 172.21.15.3:0/519019910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8106650 0x7f98d8106a30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 --2- 172.21.15.3:0/519019910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98d8100640 0x7f98d8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 -- 172.21.15.3:0/519019910 >> 172.21.15.3:0/519019910 conn(0x7f98d80fc170 msgr2=0x7f98d80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 -- 172.21.15.3:0/519019910 shutdown_connections 2023-09-10T01:10:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.681+0000 7f98df9ff700 1 -- 172.21.15.3:0/519019910 wait complete. 2023-09-10T01:10:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.682+0000 7f98df9ff700 1 Processor -- start 2023-09-10T01:10:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.682+0000 7f98df9ff700 1 -- start start 2023-09-10T01:10:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98df9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8100640 0x7f98d819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98df9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98d8101000 0x7f98d819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98df9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8106650 0x7f98d8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98df9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8198dd0 con 0x7f98d8100640 2023-09-10T01:10:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98df9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8198f40 con 0x7f98d8106650 2023-09-10T01:10:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98df9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8199100 con 0x7f98d8101000 2023-09-10T01:10:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98ddf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8106650 0x7f98d8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98dd79b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8100640 0x7f98d819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98dcf9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98d8101000 0x7f98d819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98ddf9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8106650 0x7f98d8198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60980/0 (socket says 172.21.15.3:60980) 2023-09-10T01:10:41.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.683+0000 7f98ddf9c700 1 -- 172.21.15.3:0/3790596256 learned_addr learned my addr 172.21.15.3:0/3790596256 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:41.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ddf9c700 1 -- 172.21.15.3:0/3790596256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98d8101000 msgr2=0x7f98d819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98d8101000 0x7f98d819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ddf9c700 1 -- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8100640 msgr2=0x7f98d819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8100640 0x7f98d819e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ddf9c700 1 -- 172.21.15.3:0/3790596256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d40095f0 con 0x7f98d8106650 2023-09-10T01:10:41.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8106650 0x7f98d8198760 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f98d000ef30 tx=0x7f98d000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:41.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ce7fc700 1 -- 172.21.15.3:0/3790596256 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0018070 con 0x7f98d8106650 2023-09-10T01:10:41.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d8199380 con 0x7f98d8106650 2023-09-10T01:10:41.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ce7fc700 1 -- 172.21.15.3:0/3790596256 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98d000f040 con 0x7f98d8106650 2023-09-10T01:10:41.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98ce7fc700 1 -- 172.21.15.3:0/3790596256 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d000cd00 con 0x7f98d8106650 2023-09-10T01:10:41.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.684+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d81a5670 con 0x7f98d8106650 2023-09-10T01:10:41.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.688+0000 7f98ce7fc700 1 -- 172.21.15.3:0/3790596256 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f98d0003a00 con 0x7f98d8106650 2023-09-10T01:10:41.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.688+0000 7f98f33b7b80 1 Processor -- start 2023-09-10T01:10:41.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98f33b7b80 1 -- start start 2023-09-10T01:10:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d33af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d34030 0x55e772d3d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d3de40 0x55e772d40230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772c8c730 con 0x55e772d33710 2023-09-10T01:10:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772c8a050 con 0x55e772d3de40 2023-09-10T01:10:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772c85940 con 0x55e772d34030 2023-09-10T01:10:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98dcf9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d33af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98dcf9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d33af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55630/0 (socket says 172.21.15.3:55630) 2023-09-10T01:10:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98dcf9a700 1 -- 172.21.15.3:0/1323418157 learned_addr learned my addr 172.21.15.3:0/1323418157 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.689+0000 7f98dd79b700 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d34030 0x55e772d3d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dcf9a700 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d3de40 0x55e772d40230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dd79b700 1 -- 172.21.15.3:0/1323418157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d3de40 msgr2=0x55e772d40230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:41.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dd79b700 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d3de40 0x55e772d40230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dd79b700 1 -- 172.21.15.3:0/1323418157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 msgr2=0x55e772d33af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dd79b700 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d33af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dd79b700 1 -- 172.21.15.3:0/1323418157 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e772ab1de0 con 0x55e772d34030 2023-09-10T01:10:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98ce7fc700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98c40a07b0 0x7f98c40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dd79b700 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d34030 0x55e772d3d900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74171b5d61b080da server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98ce7fc700 1 -- 172.21.15.3:0/3790596256 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f98d00cbe80 con 0x7f98d8106650 2023-09-10T01:10:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98dcf9a700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98c40a07b0 0x7f98c40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98c37fe700 1 -- 172.21.15.3:0/1323418157 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d401d070 con 0x55e772d34030 2023-09-10T01:10:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98c37fe700 1 -- 172.21.15.3:0/1323418157 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98d401e040 con 0x55e772d34030 2023-09-10T01:10:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.690+0000 7f98c37fe700 1 -- 172.21.15.3:0/1323418157 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4004740 con 0x55e772d34030 2023-09-10T01:10:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98dcf9a700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98c40a07b0 0x7f98c40a2c70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98c8005c50 tx=0x7f98c8015040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1323418157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d34030 msgr2=0x55e772d3d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d34030 0x55e772d3d900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4005ea0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1323418157 shutdown_connections 2023-09-10T01:10:41.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d3de40 0x55e772d40230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d33af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1323418157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d34030 0x55e772d3d900 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1323418157 >> 172.21.15.3:0/1323418157 conn(0x55e772ca8090 msgr2=0x55e772d2ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.691+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1323418157 shutdown_connections 2023-09-10T01:10:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.692+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1323418157 wait complete. 2023-09-10T01:10:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.692+0000 7f98f33b7b80 1 -- start start 2023-09-10T01:10:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.692+0000 7f98f33b7b80 1 Processor -- start 2023-09-10T01:10:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d3d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d34030 0x55e772d3ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d3b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772c23b70 con 0x55e772d33710 2023-09-10T01:10:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772c23870 con 0x55e772d34030 2023-09-10T01:10:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772b07920 con 0x55e772d3de40 2023-09-10T01:10:41.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98f33b7b80 10 client.0 _ll_register_callbacks cb 0x55e772b6da80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d3b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98dcf9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d34030 0x55e772d3ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98dd79b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d3d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d3b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43056/0 (socket says 172.21.15.3:43056) 2023-09-10T01:10:41.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 -- 172.21.15.3:0/1820406279 learned_addr learned my addr 172.21.15.3:0/1820406279 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 -- 172.21.15.3:0/1820406279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d34030 msgr2=0x55e772d3ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d34030 0x55e772d3ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 -- 172.21.15.3:0/1820406279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 msgr2=0x55e772d3d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d3d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.693+0000 7f98ddf9c700 1 -- 172.21.15.3:0/1820406279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d810de10 con 0x55e772d3de40 2023-09-10T01:10:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d3b170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f98d000c6f0 tx=0x7f98d0098950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be039f97c7b9ba09 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:10:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98c0ff9700 1 -- 172.21.15.3:0/1820406279 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d00b8040 con 0x55e772d3de40 2023-09-10T01:10:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e772bdd4f0 con 0x55e772d3de40 2023-09-10T01:10:41.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98c0ff9700 1 -- 172.21.15.3:0/1820406279 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f98d0099670 con 0x55e772d3de40 2023-09-10T01:10:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98c0ff9700 1 -- 172.21.15.3:0/1820406279 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98d000ce60 con 0x55e772d3de40 2023-09-10T01:10:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98f33b7b80 10 client.27514 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e772a9ac60 con 0x55e772d3de40 2023-09-10T01:10:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.694+0000 7f98c0ff9700 1 -- 172.21.15.3:0/1820406279 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e772bdd4f0 con 0x55e772d3de40 2023-09-10T01:10:41.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98c0ff9700 1 -- 172.21.15.3:0/1820406279 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f98d0098460 con 0x55e772d3de40 2023-09-10T01:10:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98f33b7b80 10 client.27514 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98f33b7b80 10 client.27514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98f33b7b80 20 client.27514 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772d4d940 0x55e772d4fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98f33b7b80 4 client.27514 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:10:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e772c9a8a0 con 0x55e772d4d940 2023-09-10T01:10:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98dd79b700 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772d4d940 0x55e772d4fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98dd79b700 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772d4d940 0x55e772d4fd60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a92ca09abd29656 server_cookie=f73051f72ded27bc in_seq=0 out_seq=0 2023-09-10T01:10:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.695+0000 7f98c0ff9700 10 client.27514 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.697+0000 7f98c0ff9700 1 -- 172.21.15.3:0/1820406279 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e772c9a8a0 con 0x55e772d4d940 2023-09-10T01:10:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.697+0000 7f98c0ff9700 10 client.27514 handle_command_reply: tid=0 2023-09-10T01:10:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.697+0000 7f98f33b7b80 1 client.27514 shutdown 2023-09-10T01:10:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.697+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 msgr2=0x55e772d3b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.697+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d3b170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f98d000c6f0 tx=0x7f98d0098950 comp rx=0 tx=0).stop 2023-09-10T01:10:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 shutdown_connections 2023-09-10T01:10:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772d4d940 0x55e772d4fd60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d3b170 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d33710 0x55e772d3d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/1820406279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d34030 0x55e772d3ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98c0ff9700 0 client.27514 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 >> 172.21.15.3:0/1820406279 conn(0x55e772ca8090 msgr2=0x55e772d2e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 shutdown_connections 2023-09-10T01:10:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 1 -- 172.21.15.3:0/1820406279 wait complete. 2023-09-10T01:10:41.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.698+0000 7f98f33b7b80 20 client.27514 trim_cache size 0 max 16384 2023-09-10T01:10:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 Processor -- start 2023-09-10T01:10:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 -- start start 2023-09-10T01:10:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d32e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d37da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772d2b3d0 con 0x55e772d34030 2023-09-10T01:10:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772d35210 con 0x55e772d33710 2023-09-10T01:10:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772d44f10 con 0x55e772d3de40 2023-09-10T01:10:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98dd79b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98dcf9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d32e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98ddf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d37da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98dd79b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d449d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60996/0 (socket says 172.21.15.3:60996) 2023-09-10T01:10:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.718+0000 7f98dd79b700 1 -- 172.21.15.3:0/888206389 learned_addr learned my addr 172.21.15.3:0/888206389 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.719+0000 7f98dd79b700 1 -- 172.21.15.3:0/888206389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 msgr2=0x55e772d37da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.719+0000 7f98dd79b700 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d37da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.719+0000 7f98dd79b700 1 -- 172.21.15.3:0/888206389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 msgr2=0x55e772d32e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.719+0000 7f98dd79b700 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d32e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.719+0000 7f98dd79b700 1 -- 172.21.15.3:0/888206389 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e772ad8250 con 0x55e772d33710 2023-09-10T01:10:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.719+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d37da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.719+0000 7f98dd79b700 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d449d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009350 tx=0x7f98d4005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d154be7fdffdf836 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.722+0000 7f98c3fff700 1 -- 172.21.15.3:0/888206389 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d401d040 con 0x55e772d33710 2023-09-10T01:10:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.722+0000 7f98c3fff700 1 -- 172.21.15.3:0/888206389 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98d4005980 con 0x55e772d33710 2023-09-10T01:10:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.722+0000 7f98c3fff700 1 -- 172.21.15.3:0/888206389 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4033ae0 con 0x55e772d33710 2023-09-10T01:10:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.723+0000 7f98f33b7b80 1 -- 172.21.15.3:0/888206389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 msgr2=0x55e772d449d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.723+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d449d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009350 tx=0x7f98d4005fd0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 -- 172.21.15.3:0/888206389 shutdown_connections 2023-09-10T01:10:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d37da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d449d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/888206389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d32e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 -- 172.21.15.3:0/888206389 >> 172.21.15.3:0/888206389 conn(0x55e772ca8090 msgr2=0x55e772d32700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:41.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 -- 172.21.15.3:0/888206389 shutdown_connections 2023-09-10T01:10:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 -- 172.21.15.3:0/888206389 wait complete. 2023-09-10T01:10:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.724+0000 7f98f33b7b80 1 -- start start 2023-09-10T01:10:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.725+0000 7f98f33b7b80 1 Processor -- start 2023-09-10T01:10:41.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.726+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d9ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.726+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d9b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.726+0000 7f98f33b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d956e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.726+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772d95df0 con 0x55e772d34030 2023-09-10T01:10:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.726+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772d95f60 con 0x55e772d33710 2023-09-10T01:10:41.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.726+0000 7f98f33b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772d96120 con 0x55e772d3de40 2023-09-10T01:10:41.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.726+0000 7f98f33b7b80 10 client.0 _ll_register_callbacks cb 0x7f98d0098d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.729+0000 7f98ddf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d956e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.729+0000 7f98ddf9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d956e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43068/0 (socket says 172.21.15.3:43068) 2023-09-10T01:10:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.729+0000 7f98ddf9c700 1 -- 172.21.15.3:0/959484454 learned_addr learned my addr 172.21.15.3:0/959484454 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:41.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.730+0000 7f98dcf9a700 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d9b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.731+0000 7f98ddf9c700 1 -- 172.21.15.3:0/959484454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 msgr2=0x55e772d9ade0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:10:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.731+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d9ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.731+0000 7f98ddf9c700 1 -- 172.21.15.3:0/959484454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 msgr2=0x55e772d9b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.731+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d9b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.731+0000 7f98ddf9c700 1 -- 172.21.15.3:0/959484454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e772a9ac60 con 0x55e772d3de40 2023-09-10T01:10:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.731+0000 7f98dcf9a700 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d9b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:10:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.732+0000 7f98ddf9c700 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d956e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f98d0095470 tx=0x7f98d0009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d53252d8e0f468e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.732+0000 7f98c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:10:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.732+0000 7f98c0ff9700 1 -- 172.21.15.3:0/959484454 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d00b8040 con 0x55e772d3de40 2023-09-10T01:10:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.732+0000 7f98c0ff9700 1 -- 172.21.15.3:0/959484454 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f98d0099670 con 0x55e772d3de40 2023-09-10T01:10:41.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.732+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98d401e040 con 0x55e772d3de40 2023-09-10T01:10:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.733+0000 7f98c0ff9700 1 -- 172.21.15.3:0/959484454 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98d00b1240 con 0x55e772d3de40 2023-09-10T01:10:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.734+0000 7f98f33b7b80 10 client.27520 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.734+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e772ab1de0 con 0x55e772d3de40 2023-09-10T01:10:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.734+0000 7f98c0ff9700 1 -- 172.21.15.3:0/959484454 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98d401e040 con 0x55e772d3de40 2023-09-10T01:10:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.734+0000 7f98c0ff9700 1 -- 172.21.15.3:0/959484454 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f98d0094d60 con 0x55e772d3de40 2023-09-10T01:10:41.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.735+0000 7f98f33b7b80 10 client.27520 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.735+0000 7f98f33b7b80 10 client.27520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.735+0000 7f98f33b7b80 20 client.27520 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.735+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772da27e0 0x55e772da4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.735+0000 7f98f33b7b80 4 client.27520 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:10:41.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.735+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e772c7e920 con 0x55e772da27e0 2023-09-10T01:10:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.736+0000 7f98dd79b700 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772da27e0 0x55e772da4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.737+0000 7f98dd79b700 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772da27e0 0x55e772da4c00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d015cb704d6ac5f4 server_cookie=b5ad6dcfc355f5ac in_seq=0 out_seq=0 2023-09-10T01:10:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.737+0000 7f98c0ff9700 10 client.27520 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.738+0000 7f98c0ff9700 1 -- 172.21.15.3:0/959484454 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55e772c7e920 con 0x55e772da27e0 2023-09-10T01:10:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.738+0000 7f98c0ff9700 10 client.27520 handle_command_reply: tid=0 2023-09-10T01:10:41.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.738+0000 7f98f33b7b80 1 client.27520 shutdown 2023-09-10T01:10:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 msgr2=0x55e772d956e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d956e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f98d0095470 tx=0x7f98d0009ef0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 shutdown_connections 2023-09-10T01:10:41.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e772da27e0 0x55e772da4c00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e772d3de40 0x55e772d956e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e772d33710 0x55e772d9ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 --2- 172.21.15.3:0/959484454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e772d34030 0x55e772d9b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 >> 172.21.15.3:0/959484454 conn(0x55e772ca8090 msgr2=0x55e772d32f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:41.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.739+0000 7f98c0ff9700 0 client.27520 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.740+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 shutdown_connections 2023-09-10T01:10:41.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.740+0000 7f98f33b7b80 1 -- 172.21.15.3:0/959484454 wait complete. 2023-09-10T01:10:41.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.741+0000 7f98f33b7b80 20 client.27520 trim_cache size 0 max 16384 2023-09-10T01:10:41.743 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:10:41.743 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:10:41.743 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:10:41.744 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:10:41.744 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:10:41.744 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:10:41.744 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:10:41.744 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:10:41.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:10:41.745 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:10:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98c40a07b0 msgr2=0x7f98c40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98c40a07b0 0x7f98c40a2c70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98c8005c50 tx=0x7f98c8015040 comp rx=0 tx=0).stop 2023-09-10T01:10:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8106650 msgr2=0x7f98d8198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8106650 0x7f98d8198760 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f98d000ef30 tx=0x7f98d000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 shutdown_connections 2023-09-10T01:10:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f98c40a07b0 0x7f98c40a2c70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98d8106650 0x7f98d8198760 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98d8100640 0x7f98d819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 --2- 172.21.15.3:0/3790596256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98d8101000 0x7f98d819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.744+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 >> 172.21.15.3:0/3790596256 conn(0x7f98d80fc170 msgr2=0x7f98d810ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:41.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.745+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 shutdown_connections 2023-09-10T01:10:41.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:10:41.747+0000 7f98df9ff700 1 -- 172.21.15.3:0/3790596256 wait complete. 2023-09-10T01:10:41.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 663522 9.14 MB/sec execute 263 sec latency 20.618 ms 2023-09-10T01:10:41.779 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:10:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:42 smithi136 ceph-mon[133708]: pgmap v229: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:42 smithi092 ceph-mon[134067]: pgmap v229: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:42.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:42 smithi003 ceph-mon[97654]: pgmap v229: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:42.777 INFO:tasks.workunit.client.0.smithi003.stdout: 1 664795 9.14 MB/sec execute 264 sec latency 18.291 ms 2023-09-10T01:10:43.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 666551 9.13 MB/sec execute 265 sec latency 35.917 ms 2023-09-10T01:10:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:44 smithi136 ceph-mon[133708]: pgmap v230: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:44 smithi092 ceph-mon[134067]: pgmap v230: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:44.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:44 smithi003 ceph-mon[97654]: pgmap v230: 129 pgs: 129 active+clean; 190 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:44.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 668084 9.14 MB/sec execute 266 sec latency 13.545 ms 2023-09-10T01:10:45.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 669764 9.12 MB/sec execute 267 sec latency 18.505 ms 2023-09-10T01:10:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:46 smithi136 ceph-mon[133708]: pgmap v231: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.6 MiB/s wr, 412 op/s 2023-09-10T01:10:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:46 smithi092 ceph-mon[134067]: pgmap v231: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.6 MiB/s wr, 412 op/s 2023-09-10T01:10:46.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:46 smithi003 ceph-mon[97654]: pgmap v231: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.6 MiB/s wr, 412 op/s 2023-09-10T01:10:46.780 INFO:tasks.workunit.client.0.smithi003.stdout: 1 671566 9.14 MB/sec execute 268 sec latency 12.757 ms 2023-09-10T01:10:47.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 672823 9.13 MB/sec execute 269 sec latency 72.773 ms 2023-09-10T01:10:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/761337838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 msgr2=0x7fd6f4100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/761337838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f4100a40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40099c0 tx=0x7fd6e4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:10:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/761337838 shutdown_connections 2023-09-10T01:10:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/761337838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6f4100f80 0x7fd6f410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/761337838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6f4106680 0x7fd6f4106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/761337838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f4100a40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/761337838 >> 172.21.15.3:0/761337838 conn(0x7fd6f40fc0b0 msgr2=0x7fd6f40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:47.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/761337838 shutdown_connections 2023-09-10T01:10:47.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.985+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/761337838 wait complete. 2023-09-10T01:10:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.986+0000 7fd6fb3ff700 1 Processor -- start 2023-09-10T01:10:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6fb3ff700 1 -- start start 2023-09-10T01:10:47.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6fb3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6fb3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6f4100f80 0x7fd6f419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6fb3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6f4106680 0x7fd6f4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6fb3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f4198cf0 con 0x7fd6f4100f80 2023-09-10T01:10:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6fb3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f4198e60 con 0x7fd6f41005c0 2023-09-10T01:10:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6fb3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f4199020 con 0x7fd6f4106680 2023-09-10T01:10:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40948/0 (socket says 172.21.15.3:40948) 2023-09-10T01:10:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6f919b700 1 -- 172.21.15.3:0/3394768166 learned_addr learned my addr 172.21.15.3:0/3394768166 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6f899a700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6f4100f80 0x7fd6f419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.987+0000 7fd6f999c700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6f4106680 0x7fd6f4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6f919b700 1 -- 172.21.15.3:0/3394768166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6f4106680 msgr2=0x7fd6f4198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6f919b700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6f4106680 0x7fd6f4198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6f919b700 1 -- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6f4100f80 msgr2=0x7fd6f419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6f919b700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6f4100f80 0x7fd6f419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:47.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6f919b700 1 -- 172.21.15.3:0/3394768166 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e40095f0 con 0x7fd6f41005c0 2023-09-10T01:10:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6f919b700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f419e610 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000d950 tx=0x7fd6f000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6ea7fc700 1 -- 172.21.15.3:0/3394768166 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f000f890 con 0x7fd6f41005c0 2023-09-10T01:10:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6ea7fc700 1 -- 172.21.15.3:0/3394768166 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6f0010460 con 0x7fd6f41005c0 2023-09-10T01:10:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6ea7fc700 1 -- 172.21.15.3:0/3394768166 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f000e610 con 0x7fd6f41005c0 2023-09-10T01:10:47.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f4199300 con 0x7fd6f41005c0 2023-09-10T01:10:47.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.988+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f41a5600 con 0x7fd6f41005c0 2023-09-10T01:10:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.992+0000 7fd6ea7fc700 1 -- 172.21.15.3:0/3394768166 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd6f00108d0 con 0x7fd6f41005c0 2023-09-10T01:10:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.993+0000 7fd70ec68b80 1 Processor -- start 2023-09-10T01:10:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd70ec68b80 1 -- start start 2023-09-10T01:10:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b57e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b578780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6ea7fc700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6e00a0830 0x7fd6e00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b584f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b4d16d0 con 0x55786b578cc0 2023-09-10T01:10:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b4ceff0 con 0x55786b57e1a0 2023-09-10T01:10:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b4ca8e0 con 0x55786b578300 2023-09-10T01:10:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6ea7fc700 1 -- 172.21.15.3:0/3394768166 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd6f00cc160 con 0x7fd6f41005c0 2023-09-10T01:10:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b578780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6f999c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b57e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b578780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50904/0 (socket says 172.21.15.3:50904) 2023-09-10T01:10:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6f919b700 1 -- 172.21.15.3:0/3300905216 learned_addr learned my addr 172.21.15.3:0/3300905216 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6f999c700 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b584f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.994+0000 7fd6f899a700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6e00a0830 0x7fd6e00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6f919b700 1 -- 172.21.15.3:0/3300905216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 msgr2=0x55786b57e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6f919b700 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b57e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6f919b700 1 -- 172.21.15.3:0/3300905216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 msgr2=0x55786b584f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6f919b700 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b584f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6f919b700 1 -- 172.21.15.3:0/3300905216 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55786b2f6d80 con 0x55786b578300 2023-09-10T01:10:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6f919b700 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b578780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00a0d00 tx=0x7fd6f00998c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7082d143771a06d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6f899a700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6e00a0830 0x7fd6e00a2cf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009990 tx=0x7fd6e4005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6df7fe700 1 -- 172.21.15.3:0/3300905216 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f00a1230 con 0x55786b578300 2023-09-10T01:10:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6df7fe700 1 -- 172.21.15.3:0/3300905216 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6f0097300 con 0x55786b578300 2023-09-10T01:10:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.995+0000 7fd6df7fe700 1 -- 172.21.15.3:0/3300905216 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0020450 con 0x55786b578300 2023-09-10T01:10:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 -- 172.21.15.3:0/3300905216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 msgr2=0x55786b578780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b578780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00a0d00 tx=0x7fd6f00998c0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 -- 172.21.15.3:0/3300905216 shutdown_connections 2023-09-10T01:10:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b584f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b57e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/3300905216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b578780 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 -- 172.21.15.3:0/3300905216 >> 172.21.15.3:0/3300905216 conn(0x55786b4ed030 msgr2=0x55786b573560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 -- 172.21.15.3:0/3300905216 shutdown_connections 2023-09-10T01:10:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 -- 172.21.15.3:0/3300905216 wait complete. 2023-09-10T01:10:48.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.996+0000 7fd70ec68b80 1 -- start start 2023-09-10T01:10:48.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 1 Processor -- start 2023-09-10T01:10:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b584d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b582080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b582600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b468b10 con 0x55786b578cc0 2023-09-10T01:10:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b468810 con 0x55786b57e1a0 2023-09-10T01:10:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b34c8c0 con 0x55786b578300 2023-09-10T01:10:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd70ec68b80 10 client.0 _ll_register_callbacks cb 0x55786b3b2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b584d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b584d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:50912/0 (socket says 172.21.15.3:50912) 2023-09-10T01:10:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.997+0000 7fd6f919b700 1 -- 172.21.15.3:0/1535220017 learned_addr learned my addr 172.21.15.3:0/1535220017 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f899a700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b582080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f999c700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b582600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f919b700 1 -- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 msgr2=0x55786b582600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f919b700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b582600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f919b700 1 -- 172.21.15.3:0/1535220017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 msgr2=0x55786b582080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f919b700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b582080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f919b700 1 -- 172.21.15.3:0/1535220017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f41a5600 con 0x55786b578300 2023-09-10T01:10:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f999c700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b582600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6f919b700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b584d60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000b2f0 tx=0x7fd6f0099340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaf4f7f3c5b4036f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6dcff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:10:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd6dcff9700 1 -- 172.21.15.3:0/1535220017 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f007a020 con 0x55786b578300 2023-09-10T01:10:48.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.998+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55786b422490 con 0x55786b578300 2023-09-10T01:10:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd6dcff9700 1 -- 172.21.15.3:0/1535220017 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd6f00a1230 con 0x55786b578300 2023-09-10T01:10:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd6dcff9700 1 -- 172.21.15.3:0/1535220017 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6f00c98f0 con 0x55786b578300 2023-09-10T01:10:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 10 client.27532 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55786b2dfc00 con 0x55786b578300 2023-09-10T01:10:48.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd6dcff9700 1 -- 172.21.15.3:0/1535220017 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55786b422490 con 0x55786b578300 2023-09-10T01:10:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd6dcff9700 1 -- 172.21.15.3:0/1535220017 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fd6f00986b0 con 0x55786b578300 2023-09-10T01:10:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 10 client.27532 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 10 client.27532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 20 client.27532 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:48.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b591ff0 0x55786b594410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 4 client.27532 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:10:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:47.999+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55786b4df840 con 0x55786b591ff0 2023-09-10T01:10:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.000+0000 7fd6f899a700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b591ff0 0x55786b594410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.000+0000 7fd6f899a700 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b591ff0 0x55786b594410 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66d0d1401371ee35 server_cookie=f1f304d44bbbed3e in_seq=0 out_seq=0 2023-09-10T01:10:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.000+0000 7fd6dcff9700 10 client.27532 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd6dcff9700 1 -- 172.21.15.3:0/1535220017 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55786b4df840 con 0x55786b591ff0 2023-09-10T01:10:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd6dcff9700 10 client.27532 handle_command_reply: tid=0 2023-09-10T01:10:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 client.27532 shutdown 2023-09-10T01:10:48.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 msgr2=0x55786b584d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b584d60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000b2f0 tx=0x7fd6f0099340 comp rx=0 tx=0).stop 2023-09-10T01:10:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 shutdown_connections 2023-09-10T01:10:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b591ff0 0x55786b594410 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b578300 0x55786b584d60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b57e1a0 0x55786b582600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/1535220017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b582080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.001+0000 7fd6dcff9700 0 client.27532 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.002+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 >> 172.21.15.3:0/1535220017 conn(0x55786b4ed030 msgr2=0x55786b5727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.002+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 shutdown_connections 2023-09-10T01:10:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.002+0000 7fd70ec68b80 1 -- 172.21.15.3:0/1535220017 wait complete. 2023-09-10T01:10:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.002+0000 7fd70ec68b80 20 client.27532 trim_cache size 0 max 16384 2023-09-10T01:10:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.019+0000 7fd70ec68b80 1 Processor -- start 2023-09-10T01:10:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.019+0000 7fd70ec68b80 1 -- start start 2023-09-10T01:10:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b584890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b57d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b57d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b5740d0 con 0x55786b578cc0 2023-09-10T01:10:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b57eb50 con 0x55786b578300 2023-09-10T01:10:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b5742f0 con 0x55786b57e1a0 2023-09-10T01:10:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b584890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b584890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40990/0 (socket says 172.21.15.3:40990) 2023-09-10T01:10:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 -- 172.21.15.3:0/2258975539 learned_addr learned my addr 172.21.15.3:0/2258975539 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f999c700 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b57d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f899a700 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b57d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 -- 172.21.15.3:0/2258975539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 msgr2=0x55786b57d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b57d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 -- 172.21.15.3:0/2258975539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 msgr2=0x55786b57d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b57d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f919b700 1 -- 172.21.15.3:0/2258975539 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55786b31d1f0 con 0x55786b578300 2023-09-10T01:10:48.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.020+0000 7fd6f999c700 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b57d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd6f919b700 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b584890 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0020620 tx=0x7fd6f000dfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6539b9454c7f8d05 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd6dffff700 1 -- 172.21.15.3:0/2258975539 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f007a030 con 0x55786b578300 2023-09-10T01:10:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd6dffff700 1 -- 172.21.15.3:0/2258975539 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6f00c98f0 con 0x55786b578300 2023-09-10T01:10:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd6dffff700 1 -- 172.21.15.3:0/2258975539 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0099500 con 0x55786b578300 2023-09-10T01:10:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd70ec68b80 1 -- 172.21.15.3:0/2258975539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 msgr2=0x55786b584890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b584890 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0020620 tx=0x7fd6f000dfd0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd70ec68b80 1 -- 172.21.15.3:0/2258975539 shutdown_connections 2023-09-10T01:10:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b57d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.021+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b584890 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.022+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/2258975539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b57d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.022+0000 7fd70ec68b80 1 -- 172.21.15.3:0/2258975539 >> 172.21.15.3:0/2258975539 conn(0x55786b4ed030 msgr2=0x55786b57c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.022+0000 7fd70ec68b80 1 -- 172.21.15.3:0/2258975539 shutdown_connections 2023-09-10T01:10:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.022+0000 7fd70ec68b80 1 -- 172.21.15.3:0/2258975539 wait complete. 2023-09-10T01:10:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.022+0000 7fd70ec68b80 1 -- start start 2023-09-10T01:10:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.022+0000 7fd70ec68b80 1 Processor -- start 2023-09-10T01:10:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b587510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b587a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd70ec68b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b5ed5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b5881f0 con 0x55786b578cc0 2023-09-10T01:10:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b588360 con 0x55786b578300 2023-09-10T01:10:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd70ec68b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55786b5edb30 con 0x55786b57e1a0 2023-09-10T01:10:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd70ec68b80 10 client.0 _ll_register_callbacks cb 0x7fd6ec0091e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:10:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b587510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f899a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b587a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b587510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40996/0 (socket says 172.21.15.3:40996) 2023-09-10T01:10:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 -- 172.21.15.3:0/951682973 learned_addr learned my addr 172.21.15.3:0/951682973 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:10:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f999c700 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b5ed5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 -- 172.21.15.3:0/951682973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 msgr2=0x55786b5ed5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b5ed5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 -- 172.21.15.3:0/951682973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 msgr2=0x55786b587a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:10:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b587a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f919b700 1 -- 172.21.15.3:0/951682973 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ec007540 con 0x55786b578300 2023-09-10T01:10:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.023+0000 7fd6f899a700 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b587a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:10:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd6f919b700 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b587510 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000b2f0 tx=0x7fd6f0099300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb6af19886a720a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:10:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd6dcff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:10:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd6dcff9700 1 -- 172.21.15.3:0/951682973 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f007a070 con 0x55786b578300 2023-09-10T01:10:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6f0097300 con 0x55786b578300 2023-09-10T01:10:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd6dcff9700 1 -- 172.21.15.3:0/951682973 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd6f00998c0 con 0x55786b578300 2023-09-10T01:10:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd6dcff9700 1 -- 172.21.15.3:0/951682973 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6f00a0e70 con 0x55786b578300 2023-09-10T01:10:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd70ec68b80 10 client.27626 fetch_fsmap learned FSMap version 29 2023-09-10T01:10:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55786b2dfc00 con 0x55786b578300 2023-09-10T01:10:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd6dcff9700 1 -- 172.21.15.3:0/951682973 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6f0085020 con 0x55786b578300 2023-09-10T01:10:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.024+0000 7fd6dcff9700 1 -- 172.21.15.3:0/951682973 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fd6f00acae0 con 0x55786b578300 2023-09-10T01:10:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd70ec68b80 10 client.27626 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:10:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd70ec68b80 10 client.27626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:10:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd70ec68b80 20 client.27626 populate_metadata read hostname 'smithi003' 2023-09-10T01:10:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b5eed70 0x55786b5f1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:10:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd70ec68b80 4 client.27626 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:10:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55786b4c38c0 con 0x55786b5eed70 2023-09-10T01:10:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd6f899a700 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b5eed70 0x55786b5f1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:10:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd6f899a700 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b5eed70 0x55786b5f1160 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8afe87b76d087ba9 server_cookie=8ce90893fac292dd in_seq=0 out_seq=0 2023-09-10T01:10:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.025+0000 7fd6dcff9700 10 client.27626 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd6dcff9700 1 -- 172.21.15.3:0/951682973 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55786b4c38c0 con 0x55786b5eed70 2023-09-10T01:10:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd6dcff9700 10 client.27626 handle_command_reply: tid=0 2023-09-10T01:10:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 client.27626 shutdown 2023-09-10T01:10:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 msgr2=0x55786b587510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b587510 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000b2f0 tx=0x7fd6f0099300 comp rx=0 tx=0).stop 2023-09-10T01:10:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 shutdown_connections 2023-09-10T01:10:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55786b5eed70 0x55786b5f1160 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55786b57e1a0 0x55786b5ed5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55786b578300 0x55786b587510 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 --2- 172.21.15.3:0/951682973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55786b578cc0 0x55786b587a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd6dcff9700 0 client.27626 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:10:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 >> 172.21.15.3:0/951682973 conn(0x55786b4ed030 msgr2=0x55786b57d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:48.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 shutdown_connections 2023-09-10T01:10:48.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.127+0000 7fd70ec68b80 1 -- 172.21.15.3:0/951682973 wait complete. 2023-09-10T01:10:48.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.128+0000 7fd70ec68b80 20 client.27626 trim_cache size 0 max 16384 2023-09-10T01:10:48.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:10:48.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:10:48.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:10:48.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:10:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6e00a0830 msgr2=0x7fd6e00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6e00a0830 0x7fd6e00a2cf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009990 tx=0x7fd6e4005fb0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 msgr2=0x7fd6f419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:10:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f419e610 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000d950 tx=0x7fd6f000dc60 comp rx=0 tx=0).stop 2023-09-10T01:10:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 shutdown_connections 2023-09-10T01:10:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd6e00a0830 0x7fd6e00a2cf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6f4106680 0x7fd6f4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6f41005c0 0x7fd6f419e610 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 --2- 172.21.15.3:0/3394768166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6f4100f80 0x7fd6f419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:10:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.129+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 >> 172.21.15.3:0/3394768166 conn(0x7fd6f40fc0b0 msgr2=0x7fd6f410b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:10:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.134+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 shutdown_connections 2023-09-10T01:10:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:10:48.134+0000 7fd6fb3ff700 1 -- 172.21.15.3:0/3394768166 wait complete. 2023-09-10T01:10:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:48 smithi136 ceph-mon[133708]: pgmap v232: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:48 smithi092 ceph-mon[134067]: pgmap v232: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:48.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:48 smithi003 ceph-mon[97654]: pgmap v232: 129 pgs: 129 active+clean; 198 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 337 op/s 2023-09-10T01:10:48.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 674992 9.14 MB/sec execute 270 sec latency 46.914 ms 2023-09-10T01:10:49.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 676259 9.13 MB/sec execute 271 sec latency 14.852 ms 2023-09-10T01:10:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:50 smithi136 ceph-mon[133708]: pgmap v233: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 449 op/s 2023-09-10T01:10:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:50 smithi092 ceph-mon[134067]: pgmap v233: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 449 op/s 2023-09-10T01:10:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:50 smithi003 ceph-mon[97654]: pgmap v233: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 449 op/s 2023-09-10T01:10:50.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 678537 9.15 MB/sec execute 272 sec latency 15.587 ms 2023-09-10T01:10:51.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 679815 9.13 MB/sec execute 273 sec latency 18.308 ms 2023-09-10T01:10:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:52 smithi136 ceph-mon[133708]: pgmap v234: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2023-09-10T01:10:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:52 smithi092 ceph-mon[134067]: pgmap v234: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2023-09-10T01:10:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:52 smithi003 ceph-mon[97654]: pgmap v234: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2023-09-10T01:10:52.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 681965 9.15 MB/sec execute 274 sec latency 39.375 ms 2023-09-10T01:10:53.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 683041 9.12 MB/sec execute 275 sec latency 31.275 ms 2023-09-10T01:10:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:54 smithi092 ceph-mon[134067]: pgmap v235: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2023-09-10T01:10:54.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:54 smithi003 ceph-mon[97654]: pgmap v235: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2023-09-10T01:10:54.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:54 smithi136 ceph-mon[133708]: pgmap v235: 129 pgs: 129 active+clean; 193 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2023-09-10T01:10:54.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 685406 9.15 MB/sec execute 276 sec latency 16.858 ms 2023-09-10T01:10:55.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 686555 9.13 MB/sec execute 277 sec latency 13.543 ms 2023-09-10T01:10:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:56 smithi092 ceph-mon[134067]: pgmap v236: 129 pgs: 129 active+clean; 201 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:10:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:56 smithi003 ceph-mon[97654]: pgmap v236: 129 pgs: 129 active+clean; 201 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:10:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:56 smithi136 ceph-mon[133708]: pgmap v236: 129 pgs: 129 active+clean; 201 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-09-10T01:10:56.778 INFO:tasks.workunit.client.0.smithi003.stdout: 1 688957 9.15 MB/sec execute 278 sec latency 15.746 ms 2023-09-10T01:10:57.779 INFO:tasks.workunit.client.0.smithi003.stdout: 1 690180 9.13 MB/sec execute 279 sec latency 18.333 ms 2023-09-10T01:10:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:10:58 smithi092 ceph-mon[134067]: pgmap v237: 129 pgs: 129 active+clean; 201 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2023-09-10T01:10:58.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:10:58 smithi003 ceph-mon[97654]: pgmap v237: 129 pgs: 129 active+clean; 201 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2023-09-10T01:10:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:10:58 smithi136 ceph-mon[133708]: pgmap v237: 129 pgs: 129 active+clean; 201 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2023-09-10T01:10:58.779 INFO:tasks.workunit.client.0.smithi003.stdout: 1 692581 9.15 MB/sec execute 280 sec latency 22.372 ms 2023-09-10T01:10:59.779 INFO:tasks.workunit.client.0.smithi003.stdout: 1 693679 9.13 MB/sec execute 281 sec latency 29.955 ms 2023-09-10T01:11:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:00 smithi092 ceph-mon[134067]: pgmap v238: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 269 op/s 2023-09-10T01:11:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:00 smithi003 ceph-mon[97654]: pgmap v238: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 269 op/s 2023-09-10T01:11:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:00 smithi136 ceph-mon[133708]: pgmap v238: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 269 op/s 2023-09-10T01:11:00.779 INFO:tasks.workunit.client.0.smithi003.stdout: 1 695732 9.14 MB/sec execute 282 sec latency 59.981 ms 2023-09-10T01:11:01.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.304+0000 7f3fb81ff700 1 -- 172.21.15.3:0/784604325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb0067b60 msgr2=0x7f3fb0067fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.304+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/784604325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb0067b60 0x7f3fb0067fe0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009990 tx=0x7f3fa0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.305+0000 7f3fb81ff700 1 -- 172.21.15.3:0/784604325 shutdown_connections 2023-09-10T01:11:01.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.305+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/784604325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fb01066a0 0x7f3fb0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.305+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/784604325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3fb0068520 0x7f3fb010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.305+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/784604325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb0067b60 0x7f3fb0067fe0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.305+0000 7f3fb81ff700 1 -- 172.21.15.3:0/784604325 >> 172.21.15.3:0/784604325 conn(0x7f3fb0074fe0 msgr2=0x7f3fb00753f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.305+0000 7f3fb81ff700 1 -- 172.21.15.3:0/784604325 shutdown_connections 2023-09-10T01:11:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.305+0000 7f3fb81ff700 1 -- 172.21.15.3:0/784604325 wait complete. 2023-09-10T01:11:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.306+0000 7f3fb81ff700 1 Processor -- start 2023-09-10T01:11:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.306+0000 7f3fb81ff700 1 -- start start 2023-09-10T01:11:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb81ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3fb0067b60 0x7f3fb019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb81ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fb0068520 0x7f3fb019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb81ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb01066a0 0x7f3fb01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb81ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb0198d70 con 0x7f3fb0067b60 2023-09-10T01:11:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb81ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb0198ee0 con 0x7f3fb01066a0 2023-09-10T01:11:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb81ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb01990a0 con 0x7f3fb0068520 2023-09-10T01:11:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb679c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb01066a0 0x7f3fb01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb5f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3fb0067b60 0x7f3fb019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb5f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3fb0067b60 0x7f3fb019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35160/0 (socket says 172.21.15.3:35160) 2023-09-10T01:11:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/3265089109 learned_addr learned my addr 172.21.15.3:0/3265089109 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.307+0000 7f3fb579a700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fb0068520 0x7f3fb019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb679c700 1 -- 172.21.15.3:0/3265089109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fb0068520 msgr2=0x7f3fb019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb679c700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fb0068520 0x7f3fb019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb679c700 1 -- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3fb0067b60 msgr2=0x7f3fb019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb679c700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3fb0067b60 0x7f3fb019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb679c700 1 -- 172.21.15.3:0/3265089109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa8009c40 con 0x7f3fb01066a0 2023-09-10T01:11:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb679c700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb01066a0 0x7f3fb01986c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800efb0 tx=0x7f3fa800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fa6ffd700 1 -- 172.21.15.3:0/3265089109 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8003a00 con 0x7f3fb01066a0 2023-09-10T01:11:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fa6ffd700 1 -- 172.21.15.3:0/3265089109 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fa800f040 con 0x7f3fb01066a0 2023-09-10T01:11:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac009520 con 0x7f3fb01066a0 2023-09-10T01:11:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fa6ffd700 1 -- 172.21.15.3:0/3265089109 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8004540 con 0x7f3fb01066a0 2023-09-10T01:11:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.308+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa00095f0 con 0x7f3fb01066a0 2023-09-10T01:11:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.312+0000 7f3fa6ffd700 1 -- 172.21.15.3:0/3265089109 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3fa8003bf0 con 0x7f3fb01066a0 2023-09-10T01:11:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.313+0000 7f3fcbae0b80 1 Processor -- start 2023-09-10T01:11:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fcbae0b80 1 -- start start 2023-09-10T01:11:01.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4144580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413e300 0x55c2c413e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413ecc0 0x55c2c414af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c40976d0 con 0x55c2c413ecc0 2023-09-10T01:11:01.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c4094ff0 con 0x55c2c41441a0 2023-09-10T01:11:01.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c40908e0 con 0x55c2c413e300 2023-09-10T01:11:01.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fa6ffd700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f9c0a0830 0x7f3f9c0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.314+0000 7f3fb579a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4144580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42498/0 (socket says 172.21.15.3:42498) 2023-09-10T01:11:01.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 -- 172.21.15.3:0/1704862776 learned_addr learned my addr 172.21.15.3:0/1704862776 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fa6ffd700 1 -- 172.21.15.3:0/3265089109 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3fa80cbf70 con 0x7f3fb01066a0 2023-09-10T01:11:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413ecc0 0x55c2c414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413e300 0x55c2c413e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb679c700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f9c0a0830 0x7f3f9c0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 -- 172.21.15.3:0/1704862776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413e300 msgr2=0x55c2c413e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413e300 0x55c2c413e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 -- 172.21.15.3:0/1704862776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413ecc0 msgr2=0x55c2c414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413ecc0 0x55c2c414af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb579a700 1 -- 172.21.15.3:0/1704862776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2c3ebcd80 con 0x55c2c41441a0 2023-09-10T01:11:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.315+0000 7f3fb679c700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f9c0a0830 0x7f3f9c0a2cf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8000f80 tx=0x7f3fa8003f90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.316+0000 7f3fb579a700 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4144580 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0015000 tx=0x7f3fa001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1626a6e1ae923bb4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.316+0000 7f3f9bfff700 1 -- 172.21.15.3:0/1704862776 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa000bb60 con 0x55c2c41441a0 2023-09-10T01:11:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.316+0000 7f3f9bfff700 1 -- 172.21.15.3:0/1704862776 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fa000bcc0 con 0x55c2c41441a0 2023-09-10T01:11:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.316+0000 7f3f9bfff700 1 -- 172.21.15.3:0/1704862776 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0017650 con 0x55c2c41441a0 2023-09-10T01:11:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1704862776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 msgr2=0x55c2c4144580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4144580 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0015000 tx=0x7f3fa001a040 comp rx=0 tx=0).stop 2023-09-10T01:11:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1704862776 shutdown_connections 2023-09-10T01:11:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413ecc0 0x55c2c414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4144580 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1704862776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413e300 0x55c2c413e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1704862776 >> 172.21.15.3:0/1704862776 conn(0x55c2c40b3030 msgr2=0x55c2c4139560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1704862776 shutdown_connections 2023-09-10T01:11:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.317+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1704862776 wait complete. 2023-09-10T01:11:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.318+0000 7f3fcbae0b80 1 -- start start 2023-09-10T01:11:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.318+0000 7f3fcbae0b80 1 Processor -- start 2023-09-10T01:11:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.318+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.318+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c41480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4148740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c402eb10 con 0x55c2c413e300 2023-09-10T01:11:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c402e810 con 0x55c2c41441a0 2023-09-10T01:11:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c3f128c0 con 0x55c2c413ecc0 2023-09-10T01:11:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fcbae0b80 10 client.0 _ll_register_callbacks cb 0x55c2c3f78a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb579a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c41480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c41480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57972/0 (socket says 172.21.15.3:57972) 2023-09-10T01:11:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb579a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35184/0 (socket says 172.21.15.3:35184) 2023-09-10T01:11:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/3391573005 learned_addr learned my addr 172.21.15.3:0/3391573005 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb579a700 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 msgr2=0x55c2c4148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/3391573005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 msgr2=0x55c2c414ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.319+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/3391573005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa0020040 con 0x55c2c413ecc0 2023-09-10T01:11:01.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3fb579a700 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:11:01.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c41480c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0128a0 tx=0x7f3fac012bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=401dc470c421a262 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:01.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3f997fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:11:01.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3f997fa700 1 -- 172.21.15.3:0/3391573005 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac017070 con 0x55c2c413ecc0 2023-09-10T01:11:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2c3fe8490 con 0x55c2c413ecc0 2023-09-10T01:11:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3f997fa700 1 -- 172.21.15.3:0/3391573005 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3fac004020 con 0x55c2c413ecc0 2023-09-10T01:11:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3f997fa700 1 -- 172.21.15.3:0/3391573005 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fac0122e0 con 0x55c2c413ecc0 2023-09-10T01:11:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3fcbae0b80 10 client.27550 fetch_fsmap learned FSMap version 29 2023-09-10T01:11:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.320+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c2c3ea5c00 con 0x55c2c413ecc0 2023-09-10T01:11:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3f997fa700 1 -- 172.21.15.3:0/3391573005 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c2c3fe8490 con 0x55c2c413ecc0 2023-09-10T01:11:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3f997fa700 1 -- 172.21.15.3:0/3391573005 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3fac00f670 con 0x55c2c413ecc0 2023-09-10T01:11:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3fcbae0b80 10 client.27550 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:11:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3fcbae0b80 10 client.27550 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3fcbae0b80 20 client.27550 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c4158420 0x55c2c415a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3fcbae0b80 4 client.27550 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:11:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2c40a5840 con 0x55c2c4158420 2023-09-10T01:11:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.321+0000 7f3fb579a700 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c4158420 0x55c2c415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.322+0000 7f3fb579a700 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c4158420 0x55c2c415a840 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c77e620b56bd7ce server_cookie=e2138c495a73a222 in_seq=0 out_seq=0 2023-09-10T01:11:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.322+0000 7f3f997fa700 10 client.27550 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3f997fa700 1 -- 172.21.15.3:0/3391573005 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c2c40a5840 con 0x55c2c4158420 2023-09-10T01:11:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3f997fa700 10 client.27550 handle_command_reply: tid=0 2023-09-10T01:11:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 client.27550 shutdown 2023-09-10T01:11:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 msgr2=0x55c2c41480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c41480c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0128a0 tx=0x7f3fac012bb0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 shutdown_connections 2023-09-10T01:11:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c4158420 0x55c2c415a840 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c4148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3f997fa700 0 client.27550 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/3391573005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c41480c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 >> 172.21.15.3:0/3391573005 conn(0x55c2c40b3030 msgr2=0x55c2c41387a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 shutdown_connections 2023-09-10T01:11:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/3391573005 wait complete. 2023-09-10T01:11:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.323+0000 7f3fcbae0b80 20 client.27550 trim_cache size 0 max 16384 2023-09-10T01:11:01.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.339+0000 7f3fcbae0b80 1 Processor -- start 2023-09-10T01:11:01.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.339+0000 7f3fcbae0b80 1 -- start start 2023-09-10T01:11:01.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c4138f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c41430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c4146640 con 0x55c2c413e300 2023-09-10T01:11:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c4142ca0 con 0x55c2c41441a0 2023-09-10T01:11:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c413f0a0 con 0x55c2c413ecc0 2023-09-10T01:11:01.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb579a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c4138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb5f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb579a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c4138f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35194/0 (socket says 172.21.15.3:35194) 2023-09-10T01:11:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb579a700 1 -- 172.21.15.3:0/362024413 learned_addr learned my addr 172.21.15.3:0/362024413 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:01.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb579a700 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c41430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/362024413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 msgr2=0x55c2c41430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c41430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/362024413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 msgr2=0x55c2c4138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c4138f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.340+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/362024413 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2c3ee31f0 con 0x55c2c413ecc0 2023-09-10T01:11:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.341+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c414a9e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3fac000c00 tx=0x7f3fac005b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86642a569aa0d5e6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.341+0000 7f3fa4ff9700 1 -- 172.21.15.3:0/362024413 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac017070 con 0x55c2c413ecc0 2023-09-10T01:11:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.341+0000 7f3fa4ff9700 1 -- 172.21.15.3:0/362024413 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fac018040 con 0x55c2c413ecc0 2023-09-10T01:11:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.341+0000 7f3fa4ff9700 1 -- 172.21.15.3:0/362024413 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac0038e0 con 0x55c2c413ecc0 2023-09-10T01:11:01.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/362024413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 msgr2=0x55c2c414a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c414a9e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3fac000c00 tx=0x7f3fac005b00 comp rx=0 tx=0).stop 2023-09-10T01:11:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/362024413 shutdown_connections 2023-09-10T01:11:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c41441a0 0x55c2c41430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c4138f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/362024413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c413ecc0 0x55c2c414a9e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/362024413 >> 172.21.15.3:0/362024413 conn(0x55c2c40b3030 msgr2=0x55c2c4142130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/362024413 shutdown_connections 2023-09-10T01:11:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.342+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/362024413 wait complete. 2023-09-10T01:11:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.343+0000 7f3fcbae0b80 1 -- start start 2023-09-10T01:11:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.343+0000 7f3fcbae0b80 1 Processor -- start 2023-09-10T01:11:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.343+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.343+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c413ecc0 0x55c2c414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fcbae0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c41441a0 0x55c2c41a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c41a5830 con 0x55c2c413e300 2023-09-10T01:11:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c41a59a0 con 0x55c2c413ecc0 2023-09-10T01:11:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fcbae0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c41a5b60 con 0x55c2c41441a0 2023-09-10T01:11:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fcbae0b80 10 client.0 _ll_register_callbacks cb 0x7f3fac009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c413ecc0 0x55c2c414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb579a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c413ecc0 0x55c2c414bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42528/0 (socket says 172.21.15.3:42528) 2023-09-10T01:11:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/1441561879 learned_addr learned my addr 172.21.15.3:0/1441561879 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb579a700 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c41441a0 0x55c2c41a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/1441561879 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c41441a0 msgr2=0x55c2c41a52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c41441a0 0x55c2c41a52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/1441561879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 msgr2=0x55c2c414b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.344+0000 7f3fb5f9b700 1 -- 172.21.15.3:0/1441561879 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2c3ea5c00 con 0x55c2c413ecc0 2023-09-10T01:11:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3fb5f9b700 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c413ecc0 0x55c2c414bb70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3fac014fd0 tx=0x7f3fac012350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bddad5f4dcc107c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3f997fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3f997fa700 1 -- 172.21.15.3:0/1441561879 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac017040 con 0x55c2c413ecc0 2023-09-10T01:11:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3fa000bcc0 con 0x55c2c413ecc0 2023-09-10T01:11:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3f997fa700 1 -- 172.21.15.3:0/1441561879 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3fac0052b0 con 0x55c2c413ecc0 2023-09-10T01:11:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3f997fa700 1 -- 172.21.15.3:0/1441561879 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fac00f850 con 0x55c2c413ecc0 2023-09-10T01:11:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3fcbae0b80 10 client.27644 fetch_fsmap learned FSMap version 29 2023-09-10T01:11:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.345+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c2c3ebb250 con 0x55c2c413ecc0 2023-09-10T01:11:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3f997fa700 1 -- 172.21.15.3:0/1441561879 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fa000bcc0 con 0x55c2c413ecc0 2023-09-10T01:11:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3f997fa700 1 -- 172.21.15.3:0/1441561879 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3fac030020 con 0x55c2c413ecc0 2023-09-10T01:11:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3fcbae0b80 10 client.27644 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:11:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3fcbae0b80 10 client.27644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3fcbae0b80 20 client.27644 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c41acf40 0x55c2c41af360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3fcbae0b80 4 client.27644 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:11:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c2c40898c0 con 0x55c2c41acf40 2023-09-10T01:11:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.346+0000 7f3fb579a700 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c41acf40 0x55c2c41af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.347+0000 7f3fb579a700 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c41acf40 0x55c2c41af360 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77a2ebe9f3d2b29a server_cookie=6bc73eff9cd6ab27 in_seq=0 out_seq=0 2023-09-10T01:11:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.347+0000 7f3f997fa700 10 client.27644 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3f997fa700 1 -- 172.21.15.3:0/1441561879 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c2c40898c0 con 0x55c2c41acf40 2023-09-10T01:11:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3f997fa700 10 client.27644 handle_command_reply: tid=0 2023-09-10T01:11:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 client.27644 shutdown 2023-09-10T01:11:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c413ecc0 msgr2=0x55c2c414bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c413ecc0 0x55c2c414bb70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3fac014fd0 tx=0x7f3fac012350 comp rx=0 tx=0).stop 2023-09-10T01:11:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 shutdown_connections 2023-09-10T01:11:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c2c41acf40 0x55c2c41af360 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c2c41441a0 0x55c2c41a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c2c413e300 0x55c2c414b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 --2- 172.21.15.3:0/1441561879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c2c413ecc0 0x55c2c414bb70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 >> 172.21.15.3:0/1441561879 conn(0x55c2c40b3030 msgr2=0x55c2c41430b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:01.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.474+0000 7f3f997fa700 0 client.27644 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.475+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 shutdown_connections 2023-09-10T01:11:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.475+0000 7f3fcbae0b80 1 -- 172.21.15.3:0/1441561879 wait complete. 2023-09-10T01:11:01.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.475+0000 7f3fcbae0b80 20 client.27644 trim_cache size 0 max 16384 2023-09-10T01:11:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:11:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:11:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:11:01.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:11:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.476+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f9c0a0830 msgr2=0x7f3f9c0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.476+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f9c0a0830 0x7f3f9c0a2cf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8000f80 tx=0x7f3fa8003f90 comp rx=0 tx=0).stop 2023-09-10T01:11:01.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.476+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb01066a0 msgr2=0x7f3fb01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.476+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb01066a0 0x7f3fb01986c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800efb0 tx=0x7f3fa800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.476+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 shutdown_connections 2023-09-10T01:11:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.477+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f9c0a0830 0x7f3f9c0a2cf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.477+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fb01066a0 0x7f3fb01986c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.477+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3fb0067b60 0x7f3fb019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.477+0000 7f3fb81ff700 1 --2- 172.21.15.3:0/3265089109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fb0068520 0x7f3fb019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.477+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 >> 172.21.15.3:0/3265089109 conn(0x7f3fb0074fe0 msgr2=0x7f3fb010b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.478+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 shutdown_connections 2023-09-10T01:11:01.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:01.478+0000 7f3fb81ff700 1 -- 172.21.15.3:0/3265089109 wait complete. 2023-09-10T01:11:01.779 INFO:tasks.workunit.client.0.smithi003.stdout: 1 696200 9.13 MB/sec execute 283 sec latency 173.195 ms 2023-09-10T01:11:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:02 smithi092 ceph-mon[134067]: pgmap v239: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 156 op/s 2023-09-10T01:11:02.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:02 smithi003 ceph-mon[97654]: pgmap v239: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 156 op/s 2023-09-10T01:11:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:02 smithi136 ceph-mon[133708]: pgmap v239: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 156 op/s 2023-09-10T01:11:02.781 INFO:tasks.workunit.client.0.smithi003.stdout: 1 697493 9.12 MB/sec execute 284 sec latency 14.699 ms 2023-09-10T01:11:03.779 INFO:tasks.workunit.client.0.smithi003.stdout: 1 699580 9.13 MB/sec execute 285 sec latency 50.661 ms 2023-09-10T01:11:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:04 smithi092 ceph-mon[134067]: pgmap v240: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 156 op/s 2023-09-10T01:11:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:04 smithi003 ceph-mon[97654]: pgmap v240: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 156 op/s 2023-09-10T01:11:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:04 smithi136 ceph-mon[133708]: pgmap v240: 129 pgs: 129 active+clean; 209 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 156 op/s 2023-09-10T01:11:04.780 INFO:tasks.workunit.client.0.smithi003.stdout: 1 699984 9.10 MB/sec execute 286 sec latency 72.319 ms 2023-09-10T01:11:05.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:05 smithi092 ceph-mon[134067]: osdmap e94: 12 total, 12 up, 12 in 2023-09-10T01:11:05.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:05 smithi003 ceph-mon[97654]: osdmap e94: 12 total, 12 up, 12 in 2023-09-10T01:11:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:05 smithi136 ceph-mon[133708]: osdmap e94: 12 total, 12 up, 12 in 2023-09-10T01:11:05.780 INFO:tasks.workunit.client.0.smithi003.stdout: 1 700561 9.07 MB/sec execute 287 sec latency 51.461 ms 2023-09-10T01:11:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:06 smithi092 ceph-mon[134067]: pgmap v242: 129 pgs: 129 active+clean; 208 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 272 op/s 2023-09-10T01:11:06.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:06 smithi003 ceph-mon[97654]: pgmap v242: 129 pgs: 129 active+clean; 208 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 272 op/s 2023-09-10T01:11:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:06 smithi136 ceph-mon[133708]: pgmap v242: 129 pgs: 129 active+clean; 208 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 272 op/s 2023-09-10T01:11:06.780 INFO:tasks.workunit.client.0.smithi003.stdout: 1 702900 9.09 MB/sec execute 288 sec latency 18.245 ms 2023-09-10T01:11:07.780 INFO:tasks.workunit.client.0.smithi003.stdout: 1 704180 9.08 MB/sec execute 289 sec latency 18.840 ms 2023-09-10T01:11:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:08 smithi092 ceph-mon[134067]: pgmap v243: 129 pgs: 129 active+clean; 208 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 272 op/s 2023-09-10T01:11:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:08 smithi003 ceph-mon[97654]: pgmap v243: 129 pgs: 129 active+clean; 208 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 272 op/s 2023-09-10T01:11:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:08 smithi136 ceph-mon[133708]: pgmap v243: 129 pgs: 129 active+clean; 208 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 272 op/s 2023-09-10T01:11:08.780 INFO:tasks.workunit.client.0.smithi003.stdout: 1 706425 9.09 MB/sec execute 290 sec latency 32.397 ms 2023-09-10T01:11:09.782 INFO:tasks.workunit.client.0.smithi003.stdout: 1 707355 9.08 MB/sec execute 291 sec latency 123.129 ms 2023-09-10T01:11:10.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:10 smithi003 ceph-mon[97654]: pgmap v244: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 267 op/s 2023-09-10T01:11:10.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:10 smithi003 ceph-mon[97654]: osdmap e95: 12 total, 12 up, 12 in 2023-09-10T01:11:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:10 smithi136 ceph-mon[133708]: pgmap v244: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 267 op/s 2023-09-10T01:11:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:10 smithi136 ceph-mon[133708]: osdmap e95: 12 total, 12 up, 12 in 2023-09-10T01:11:10.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:10 smithi092 ceph-mon[134067]: pgmap v244: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 267 op/s 2023-09-10T01:11:10.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:10 smithi092 ceph-mon[134067]: osdmap e95: 12 total, 12 up, 12 in 2023-09-10T01:11:10.782 INFO:tasks.workunit.client.0.smithi003.stdout: 1 707912 9.05 MB/sec execute 292 sec latency 132.658 ms 2023-09-10T01:11:11.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:11:11.782 INFO:tasks.workunit.client.0.smithi003.stdout: 1 709281 9.04 MB/sec execute 293 sec latency 136.528 ms 2023-09-10T01:11:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 -- 172.21.15.3:0/2319042000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c1021c0 msgr2=0x7f901c10a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 --2- 172.21.15.3:0/2319042000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c1021c0 0x7f901c10a6a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9010009990 tx=0x7f9010009ca0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 -- 172.21.15.3:0/2319042000 shutdown_connections 2023-09-10T01:11:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 --2- 172.21.15.3:0/2319042000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c10ac70 0x7f901c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 --2- 172.21.15.3:0/2319042000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f901c1018a0 0x7f901c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 --2- 172.21.15.3:0/2319042000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c1021c0 0x7f901c10a6a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 -- 172.21.15.3:0/2319042000 >> 172.21.15.3:0/2319042000 conn(0x7f901c076710 msgr2=0x7f901c076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 -- 172.21.15.3:0/2319042000 shutdown_connections 2023-09-10T01:11:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.016+0000 7f9021dff700 1 -- 172.21.15.3:0/2319042000 wait complete. 2023-09-10T01:11:12.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.017+0000 7f9021dff700 1 Processor -- start 2023-09-10T01:11:12.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.017+0000 7f9021dff700 1 -- start start 2023-09-10T01:11:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f9021dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c1018a0 0x7f901c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f9021dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f901c1021c0 0x7f901c198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f9021dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c10ac70 0x7f901c19d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f9021dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c19d940 con 0x7f901c1021c0 2023-09-10T01:11:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f9021dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c19dab0 con 0x7f901c1018a0 2023-09-10T01:11:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f9021dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c19dc70 con 0x7f901c10ac70 2023-09-10T01:11:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c1018a0 0x7f901c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c1018a0 0x7f901c1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40512/0 (socket says 172.21.15.3:40512) 2023-09-10T01:11:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 -- 172.21.15.3:0/2984110822 learned_addr learned my addr 172.21.15.3:0/2984110822 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901bfff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c10ac70 0x7f901c19d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901affd700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f901c1021c0 0x7f901c198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 -- 172.21.15.3:0/2984110822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c10ac70 msgr2=0x7f901c19d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c10ac70 0x7f901c19d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 -- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f901c1021c0 msgr2=0x7f901c198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f901c1021c0 0x7f901c198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.018+0000 7f901b7fe700 1 -- 172.21.15.3:0/2984110822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90100095f0 con 0x7f901c1018a0 2023-09-10T01:11:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.019+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c1018a0 0x7f901c1989a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f900c00e9a0 tx=0x7f900c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.019+0000 7f9018ff9700 1 -- 172.21.15.3:0/2984110822 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c009790 con 0x7f901c1018a0 2023-09-10T01:11:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.019+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c19df50 con 0x7f901c1018a0 2023-09-10T01:11:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.019+0000 7f9018ff9700 1 -- 172.21.15.3:0/2984110822 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f900c010430 con 0x7f901c1018a0 2023-09-10T01:11:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.019+0000 7f9018ff9700 1 -- 172.21.15.3:0/2984110822 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c018680 con 0x7f901c1018a0 2023-09-10T01:11:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.019+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c1a5800 con 0x7f901c1018a0 2023-09-10T01:11:12.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.023+0000 7f9018ff9700 1 -- 172.21.15.3:0/2984110822 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f900c0105a0 con 0x7f901c1018a0 2023-09-10T01:11:12.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.024+0000 7f90356b2b80 1 Processor -- start 2023-09-10T01:11:12.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.024+0000 7f90356b2b80 1 -- start start 2023-09-10T01:11:12.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f9018ff9700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f90040a07b0 0x7f90040a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f90356b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0945730 con 0x5604e09ec710 2023-09-10T01:11:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f90356b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0943050 con 0x5604e09ed030 2023-09-10T01:11:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f90356b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e093e940 con 0x5604e09f6e40 2023-09-10T01:11:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f9018ff9700 1 -- 172.21.15.3:0/2984110822 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f900c014070 con 0x7f901c1018a0 2023-09-10T01:11:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09ecaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51396/0 (socket says 172.21.15.3:51396) 2023-09-10T01:11:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 -- 172.21.15.3:0/2693167766 learned_addr learned my addr 172.21.15.3:0/2693167766 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901bfff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f90040a07b0 0x7f90040a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901bfff700 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 -- 172.21.15.3:0/2693167766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 msgr2=0x5604e09f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 -- 172.21.15.3:0/2693167766 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 msgr2=0x5604e09f6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901b7fe700 1 -- 172.21.15.3:0/2693167766 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604e076ade0 con 0x5604e09ec710 2023-09-10T01:11:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901bfff700 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:11:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.025+0000 7f901affd700 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.026+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09ecaf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f900c0a1140 tx=0x7f900c098f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a09d83a70a3682 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.026+0000 7f9001ffb700 1 -- 172.21.15.3:0/2693167766 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0a2530 con 0x5604e09ec710 2023-09-10T01:11:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.026+0000 7f901bfff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f90040a07b0 0x7f90040a2c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9014009ec0 tx=0x7f9014009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.026+0000 7f9001ffb700 1 -- 172.21.15.3:0/2693167766 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f900c099600 con 0x5604e09ec710 2023-09-10T01:11:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.026+0000 7f9001ffb700 1 -- 172.21.15.3:0/2693167766 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0840b0 con 0x5604e09ec710 2023-09-10T01:11:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 -- 172.21.15.3:0/2693167766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 msgr2=0x5604e09ecaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09ecaf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f900c0a1140 tx=0x7f900c098f60 comp rx=0 tx=0).stop 2023-09-10T01:11:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 -- 172.21.15.3:0/2693167766 shutdown_connections 2023-09-10T01:11:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09ecaf0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2693167766 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 -- 172.21.15.3:0/2693167766 >> 172.21.15.3:0/2693167766 conn(0x5604e0961090 msgr2=0x5604e09e7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 -- 172.21.15.3:0/2693167766 shutdown_connections 2023-09-10T01:11:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.027+0000 7f90356b2b80 1 -- 172.21.15.3:0/2693167766 wait complete. 2023-09-10T01:11:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.028+0000 7f90356b2b80 1 -- start start 2023-09-10T01:11:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.028+0000 7f90356b2b80 1 Processor -- start 2023-09-10T01:11:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f90356b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e08dcb70 con 0x5604e09ec710 2023-09-10T01:11:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f90356b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e08dc870 con 0x5604e09ed030 2023-09-10T01:11:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f90356b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e07c0920 con 0x5604e09f6e40 2023-09-10T01:11:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f90356b2b80 10 client.0 _ll_register_callbacks cb 0x5604e0826a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f901affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f901affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40516/0 (socket says 172.21.15.3:40516) 2023-09-10T01:11:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f901affd700 1 -- 172.21.15.3:0/2190596371 learned_addr learned my addr 172.21.15.3:0/2190596371 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.029+0000 7f901bfff700 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f901affd700 1 -- 172.21.15.3:0/2190596371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 msgr2=0x5604e09f41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f901affd700 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f901affd700 1 -- 172.21.15.3:0/2190596371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 msgr2=0x5604e09f68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f901affd700 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f901affd700 1 -- 172.21.15.3:0/2190596371 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c10ddb0 con 0x5604e09ed030 2023-09-10T01:11:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f901affd700 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f3ad0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9010009990 tx=0x7f901000b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3420608522c0ae36 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f8ffaffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f8ffaffd700 1 -- 172.21.15.3:0/2190596371 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901001d070 con 0x5604e09ed030 2023-09-10T01:11:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604e08964f0 con 0x5604e09ed030 2023-09-10T01:11:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f8ffaffd700 1 -- 172.21.15.3:0/2190596371 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f9010004500 con 0x5604e09ed030 2023-09-10T01:11:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f8ffaffd700 1 -- 172.21.15.3:0/2190596371 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f901002e8f0 con 0x5604e09ed030 2023-09-10T01:11:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f90356b2b80 10 client.27656 fetch_fsmap learned FSMap version 29 2023-09-10T01:11:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.030+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f901c1a5800 con 0x5604e09ed030 2023-09-10T01:11:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f8ffaffd700 1 -- 172.21.15.3:0/2190596371 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5604e08964f0 con 0x5604e09ed030 2023-09-10T01:11:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f8ffaffd700 1 -- 172.21.15.3:0/2190596371 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9010004d60 con 0x5604e09ed030 2023-09-10T01:11:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f90356b2b80 10 client.27656 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:11:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f90356b2b80 10 client.27656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f90356b2b80 20 client.27656 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a06900 0x5604e0a08d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f90356b2b80 4 client.27656 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:11:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604e09538a0 con 0x5604e0a06900 2023-09-10T01:11:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.031+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a06900 0x5604e0a08d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.032+0000 7f901b7fe700 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a06900 0x5604e0a08d20 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9551f7921b910eb server_cookie=c12c4aad010773a5 in_seq=0 out_seq=0 2023-09-10T01:11:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.032+0000 7f8ffaffd700 10 client.27656 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:12.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.032+0000 7f8ffaffd700 1 -- 172.21.15.3:0/2190596371 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5604e09538a0 con 0x5604e0a06900 2023-09-10T01:11:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.032+0000 7f8ffaffd700 10 client.27656 handle_command_reply: tid=0 2023-09-10T01:11:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.032+0000 7f90356b2b80 1 client.27656 shutdown 2023-09-10T01:11:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 msgr2=0x5604e09f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f3ad0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9010009990 tx=0x7f901000b610 comp rx=0 tx=0).stop 2023-09-10T01:11:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 shutdown_connections 2023-09-10T01:11:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a06900 0x5604e0a08d20 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09f41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2190596371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09f3ad0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 >> 172.21.15.3:0/2190596371 conn(0x5604e0961090 msgr2=0x5604e09e77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f8ffaffd700 0 client.27656 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 shutdown_connections 2023-09-10T01:11:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 1 -- 172.21.15.3:0/2190596371 wait complete. 2023-09-10T01:11:12.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.033+0000 7f90356b2b80 20 client.27656 trim_cache size 0 max 16384 2023-09-10T01:11:12.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.052+0000 7f90356b2b80 1 Processor -- start 2023-09-10T01:11:12.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.052+0000 7f90356b2b80 1 -- start start 2023-09-10T01:11:12.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09eee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09ef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f90356b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e09e8770 con 0x5604e09ec710 2023-09-10T01:11:12.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f90356b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e09df400 con 0x5604e09ed030 2023-09-10T01:11:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f90356b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e09ee210 con 0x5604e09f6e40 2023-09-10T01:11:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09ef340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46924/0 (socket says 172.21.15.3:46924) 2023-09-10T01:11:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51408/0 (socket says 172.21.15.3:51408) 2023-09-10T01:11:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901bfff700 1 -- 172.21.15.3:0/2952029183 learned_addr learned my addr 172.21.15.3:0/2952029183 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09eee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40518/0 (socket says 172.21.15.3:40518) 2023-09-10T01:11:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901affd700 1 -- 172.21.15.3:0/2952029183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 msgr2=0x5604e09ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901affd700 1 --2- 172.21.15.3:0/2952029183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901affd700 1 -- 172.21.15.3:0/2952029183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 msgr2=0x5604e09f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.053+0000 7f901affd700 1 --2- 172.21.15.3:0/2952029183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.054+0000 7f901affd700 1 -- 172.21.15.3:0/2952029183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604e0753c60 con 0x5604e09ed030 2023-09-10T01:11:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.054+0000 7f901affd700 1 --2- 172.21.15.3:0/2952029183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09eee00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f901000c010 tx=0x7f9010005a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfd0054164d4db60 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.054+0000 7f90027fc700 1 -- 172.21.15.3:0/2952029183 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901001d070 con 0x5604e09ed030 2023-09-10T01:11:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.054+0000 7f90027fc700 1 -- 172.21.15.3:0/2952029183 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f901001e040 con 0x5604e09ed030 2023-09-10T01:11:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.054+0000 7f90027fc700 1 -- 172.21.15.3:0/2952029183 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90100049c0 con 0x5604e09ed030 2023-09-10T01:11:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 -- 172.21.15.3:0/2952029183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 msgr2=0x5604e09eee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2952029183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09eee00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f901000c010 tx=0x7f9010005a90 comp rx=0 tx=0).stop 2023-09-10T01:11:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 -- 172.21.15.3:0/2952029183 shutdown_connections 2023-09-10T01:11:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2952029183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09f6e40 0x5604e09ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2952029183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09ec710 0x5604e09f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 --2- 172.21.15.3:0/2952029183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ed030 0x5604e09eee00 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 -- 172.21.15.3:0/2952029183 >> 172.21.15.3:0/2952029183 conn(0x5604e0961090 msgr2=0x5604e09e6320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:12.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 -- 172.21.15.3:0/2952029183 shutdown_connections 2023-09-10T01:11:12.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 -- 172.21.15.3:0/2952029183 wait complete. 2023-09-10T01:11:12.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.055+0000 7f90356b2b80 1 -- start start 2023-09-10T01:11:12.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.056+0000 7f90356b2b80 1 Processor -- start 2023-09-10T01:11:12.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.056+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ec710 0x5604e09fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.056+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09ed030 0x5604e09fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.056+0000 7f90356b2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09f6e40 0x5604e09fc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.056+0000 7f90356b2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e09fc750 con 0x5604e09f6e40 2023-09-10T01:11:12.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.056+0000 7f90356b2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e09fc8c0 con 0x5604e09ec710 2023-09-10T01:11:12.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.056+0000 7f90356b2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e09fca80 con 0x5604e09ed030 2023-09-10T01:11:12.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f90356b2b80 10 client.0 _ll_register_callbacks cb 0x7f901000bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:12.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ec710 0x5604e09fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ec710 0x5604e09fd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40534/0 (socket says 172.21.15.3:40534) 2023-09-10T01:11:12.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 -- 172.21.15.3:0/1466816668 learned_addr learned my addr 172.21.15.3:0/1466816668 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:12.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901affd700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09ed030 0x5604e09fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901bfff700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09f6e40 0x5604e09fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 -- 172.21.15.3:0/1466816668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09ed030 msgr2=0x5604e09fbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09ed030 0x5604e09fbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 -- 172.21.15.3:0/1466816668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09f6e40 msgr2=0x5604e09fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:12.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09f6e40 0x5604e09fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 -- 172.21.15.3:0/1466816668 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c1a5800 con 0x5604e09ec710 2023-09-10T01:11:12.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901bfff700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09f6e40 0x5604e09fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:11:12.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f901b7fe700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ec710 0x5604e09fd8f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f900c099590 tx=0x7f900c0993d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1699842496ff38e2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:12.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f8ffaffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:12.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f8ffaffd700 1 -- 172.21.15.3:0/1466816668 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0a16f0 con 0x5604e09ec710 2023-09-10T01:11:12.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f8ffaffd700 1 -- 172.21.15.3:0/1466816668 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f900c09d150 con 0x5604e09ec710 2023-09-10T01:11:12.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.057+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f900c099600 con 0x5604e09ec710 2023-09-10T01:11:12.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f8ffaffd700 1 -- 172.21.15.3:0/1466816668 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f900c010880 con 0x5604e09ec710 2023-09-10T01:11:12.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 10 client.27662 fetch_fsmap learned FSMap version 29 2023-09-10T01:11:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5604e0791250 con 0x5604e09ec710 2023-09-10T01:11:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f8ffaffd700 1 -- 172.21.15.3:0/1466816668 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f900c099600 con 0x5604e09ec710 2023-09-10T01:11:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f8ffaffd700 1 -- 172.21.15.3:0/1466816668 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f900c098b80 con 0x5604e09ec710 2023-09-10T01:11:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 10 client.27662 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:11:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 10 client.27662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 20 client.27662 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a5bae0 0x5604e0a5df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 4 client.27662 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:11:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5604e0937920 con 0x5604e0a5bae0 2023-09-10T01:11:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.058+0000 7f901affd700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a5bae0 0x5604e0a5df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:12.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f901affd700 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a5bae0 0x5604e0a5df00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=407cec489106af34 server_cookie=77ef5312a2b27fc3 in_seq=0 out_seq=0 2023-09-10T01:11:12.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f8ffaffd700 10 client.27662 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:12.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f8ffaffd700 1 -- 172.21.15.3:0/1466816668 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5604e0937920 con 0x5604e0a5bae0 2023-09-10T01:11:12.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f8ffaffd700 10 client.27662 handle_command_reply: tid=0 2023-09-10T01:11:12.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 client.27662 shutdown 2023-09-10T01:11:12.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ec710 msgr2=0x5604e09fd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ec710 0x5604e09fd8f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f900c099590 tx=0x7f900c0993d0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 shutdown_connections 2023-09-10T01:11:12.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5604e0a5bae0 0x5604e0a5df00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604e09f6e40 0x5604e09fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604e09ec710 0x5604e09fd8f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 --2- 172.21.15.3:0/1466816668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5604e09ed030 0x5604e09fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f8ffaffd700 0 client.27662 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:12.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.059+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 >> 172.21.15.3:0/1466816668 conn(0x5604e0961090 msgr2=0x5604e09eee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:12.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.060+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 shutdown_connections 2023-09-10T01:11:12.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.060+0000 7f90356b2b80 1 -- 172.21.15.3:0/1466816668 wait complete. 2023-09-10T01:11:12.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.060+0000 7f90356b2b80 20 client.27662 trim_cache size 0 max 16384 2023-09-10T01:11:12.079 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:11:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:11:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:11:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:11:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:11:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:11:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:11:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:11:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:11:12.081 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:11:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.060+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f90040a07b0 msgr2=0x7f90040a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.060+0000 7f9021dff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f90040a07b0 0x7f90040a2c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9014009ec0 tx=0x7f9014009920 comp rx=0 tx=0).stop 2023-09-10T01:11:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.060+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c1018a0 msgr2=0x7f901c1989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.060+0000 7f9021dff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c1018a0 0x7f901c1989a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f900c00e9a0 tx=0x7f900c00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 shutdown_connections 2023-09-10T01:11:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f90040a07b0 0x7f90040a2c70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f901c10ac70 0x7f901c19d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f901c1018a0 0x7f901c1989a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 --2- 172.21.15.3:0/2984110822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f901c1021c0 0x7f901c198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 >> 172.21.15.3:0/2984110822 conn(0x7f901c076710 msgr2=0x7f901c107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 shutdown_connections 2023-09-10T01:11:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:12.061+0000 7f9021dff700 1 -- 172.21.15.3:0/2984110822 wait complete. 2023-09-10T01:11:12.085 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:11:12.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:12 smithi003 ceph-mon[97654]: pgmap v246: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 333 op/s 2023-09-10T01:11:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:12 smithi136 ceph-mon[133708]: pgmap v246: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 333 op/s 2023-09-10T01:11:12.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:12 smithi092 ceph-mon[134067]: pgmap v246: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 333 op/s 2023-09-10T01:11:12.782 INFO:tasks.workunit.client.0.smithi003.stdout: 1 710900 9.05 MB/sec execute 294 sec latency 19.209 ms 2023-09-10T01:11:13.782 INFO:tasks.workunit.client.0.smithi003.stdout: 1 712782 9.04 MB/sec execute 295 sec latency 15.208 ms 2023-09-10T01:11:14.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:14 smithi003 ceph-mon[97654]: pgmap v247: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2023-09-10T01:11:14.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 -- 172.21.15.3:0/3979025966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0729a0 msgr2=0x7f84bc072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3979025966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0729a0 0x7f84bc072d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009990 tx=0x7f84b0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 -- 172.21.15.3:0/3979025966 shutdown_connections 2023-09-10T01:11:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3979025966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0729a0 0x7f84bc072d80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3979025966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc1115b0 0x7f84bc1139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3979025966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84bc0732c0 0x7f84bc111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 -- 172.21.15.3:0/3979025966 >> 172.21.15.3:0/3979025966 conn(0x7f84bc0fc0b0 msgr2=0x7f84bc0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 -- 172.21.15.3:0/3979025966 shutdown_connections 2023-09-10T01:11:14.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.645+0000 7f84c2dff700 1 -- 172.21.15.3:0/3979025966 wait complete. 2023-09-10T01:11:14.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.647+0000 7f84c2dff700 1 Processor -- start 2023-09-10T01:11:14.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.647+0000 7f84c2dff700 1 -- start start 2023-09-10T01:11:14.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.647+0000 7f84c2dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84bc0729a0 0x7f84bc19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.647+0000 7f84c2dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0732c0 0x7f84bc19d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.647+0000 7f84c2dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc1115b0 0x7f84bc1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c2dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc1a1d60 con 0x7f84bc0729a0 2023-09-10T01:11:14.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c2dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc1a1ed0 con 0x7f84bc1115b0 2023-09-10T01:11:14.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c2dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc1a2090 con 0x7f84bc0732c0 2023-09-10T01:11:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84bbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0732c0 0x7f84bc19d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c139c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc1115b0 0x7f84bc1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84bc0729a0 0x7f84bc19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84bbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0732c0 0x7f84bc19d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:46936/0 (socket says 172.21.15.3:46936) 2023-09-10T01:11:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84bbfff700 1 -- 172.21.15.3:0/3758471734 learned_addr learned my addr 172.21.15.3:0/3758471734 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c139c700 1 -- 172.21.15.3:0/3758471734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0732c0 msgr2=0x7f84bc19d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c139c700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0732c0 0x7f84bc19d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c139c700 1 -- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84bc0729a0 msgr2=0x7f84bc19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c139c700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84bc0729a0 0x7f84bc19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.648+0000 7f84c139c700 1 -- 172.21.15.3:0/3758471734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00095f0 con 0x7f84bc1115b0 2023-09-10T01:11:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.649+0000 7f84c139c700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc1115b0 0x7f84bc1a1720 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f84b4007ee0 tx=0x7f84b400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.649+0000 7f84b9ffb700 1 -- 172.21.15.3:0/3758471734 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4004db0 con 0x7f84bc1115b0 2023-09-10T01:11:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.649+0000 7f84b9ffb700 1 -- 172.21.15.3:0/3758471734 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84b400f040 con 0x7f84bc1115b0 2023-09-10T01:11:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.649+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc1a2370 con 0x7f84bc1115b0 2023-09-10T01:11:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.649+0000 7f84b9ffb700 1 -- 172.21.15.3:0/3758471734 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4005900 con 0x7f84bc1115b0 2023-09-10T01:11:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.649+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc1a9bb0 con 0x7f84bc1115b0 2023-09-10T01:11:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.652+0000 7f84b9ffb700 1 -- 172.21.15.3:0/3758471734 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f84b400db20 con 0x7f84bc1115b0 2023-09-10T01:11:14.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.654+0000 7f84d67feb80 1 Processor -- start 2023-09-10T01:11:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84d67feb80 1 -- start start 2023-09-10T01:11:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84b9ffb700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84ac0a07b0 0x7f84ac0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8c31a0 0x561c7c8c3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84b9ffb700 1 -- 172.21.15.3:0/3758471734 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f84b40cbbd0 con 0x7f84bc1115b0 2023-09-10T01:11:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bd300 0x561c7c8bd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84d67feb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c8166d0 con 0x561c7c8bdcc0 2023-09-10T01:11:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84d67feb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c813ff0 con 0x561c7c8c31a0 2023-09-10T01:11:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84d67feb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c80f8e0 con 0x561c7c8bd300 2023-09-10T01:11:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84bbfff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84ac0a07b0 0x7f84ac0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.655+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8c31a0 0x561c7c8c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8c31a0 0x561c7c8c3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35316/0 (socket says 172.21.15.3:35316) 2023-09-10T01:11:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 -- 172.21.15.3:0/467483789 learned_addr learned my addr 172.21.15.3:0/467483789 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c139c700 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 -- 172.21.15.3:0/467483789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bd300 msgr2=0x561c7c8bd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bd300 0x561c7c8bd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 -- 172.21.15.3:0/467483789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 msgr2=0x561c7c8c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 -- 172.21.15.3:0/467483789 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c7c63bd80 con 0x561c7c8c31a0 2023-09-10T01:11:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84bbfff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84ac0a07b0 0x7f84ac0a2c70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84bc19e390 tx=0x7f84a8011720 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.656+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8c31a0 0x561c7c8c3580 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005f50 tx=0x7f84b000b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f314052782707f64 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84a6ffd700 1 -- 172.21.15.3:0/467483789 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b001d070 con 0x561c7c8c31a0 2023-09-10T01:11:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84a6ffd700 1 -- 172.21.15.3:0/467483789 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84b001e040 con 0x561c7c8c31a0 2023-09-10T01:11:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84a6ffd700 1 -- 172.21.15.3:0/467483789 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00049a0 con 0x561c7c8c31a0 2023-09-10T01:11:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84d67feb80 1 -- 172.21.15.3:0/467483789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8c31a0 msgr2=0x561c7c8c3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84d67feb80 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8c31a0 0x561c7c8c3580 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005f50 tx=0x7f84b000b610 comp rx=0 tx=0).stop 2023-09-10T01:11:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84d67feb80 1 -- 172.21.15.3:0/467483789 shutdown_connections 2023-09-10T01:11:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84d67feb80 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84d67feb80 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8c31a0 0x561c7c8c3580 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84d67feb80 1 --2- 172.21.15.3:0/467483789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bd300 0x561c7c8bd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.657+0000 7f84d67feb80 1 -- 172.21.15.3:0/467483789 >> 172.21.15.3:0/467483789 conn(0x561c7c832030 msgr2=0x561c7c8b8560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.658+0000 7f84d67feb80 1 -- 172.21.15.3:0/467483789 shutdown_connections 2023-09-10T01:11:14.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.658+0000 7f84d67feb80 1 -- 172.21.15.3:0/467483789 wait complete. 2023-09-10T01:11:14.668 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:14 smithi136 ceph-mon[133708]: pgmap v247: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2023-09-10T01:11:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.658+0000 7f84d67feb80 1 -- start start 2023-09-10T01:11:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.658+0000 7f84d67feb80 1 Processor -- start 2023-09-10T01:11:14.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c8c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84d67feb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c7adb10 con 0x561c7c8bdcc0 2023-09-10T01:11:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84d67feb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c7ad810 con 0x561c7c8bd300 2023-09-10T01:11:14.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84d67feb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c6918c0 con 0x561c7c8c31a0 2023-09-10T01:11:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84d67feb80 10 client.0 _ll_register_callbacks cb 0x561c7c6f7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35318/0 (socket says 172.21.15.3:35318) 2023-09-10T01:11:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 -- 172.21.15.3:0/3739808336 learned_addr learned my addr 172.21.15.3:0/3739808336 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:14.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c139c700 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c8c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84bbfff700 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 -- 172.21.15.3:0/3739808336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 msgr2=0x561c7c8c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c8c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 -- 172.21.15.3:0/3739808336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 msgr2=0x561c7c8c7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.659+0000 7f84c0b9b700 1 -- 172.21.15.3:0/3739808336 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc114350 con 0x561c7c8bd300 2023-09-10T01:11:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8c9d90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009350 tx=0x7f84b0005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f453c2acfcd899c3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f849bfff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f849bfff700 1 -- 172.21.15.3:0/3739808336 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b001d040 con 0x561c7c8bd300 2023-09-10T01:11:14.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c7c767490 con 0x561c7c8bd300 2023-09-10T01:11:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f849bfff700 1 -- 172.21.15.3:0/3739808336 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f84b000fba0 con 0x561c7c8bd300 2023-09-10T01:11:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f849bfff700 1 -- 172.21.15.3:0/3739808336 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84b000fd60 con 0x561c7c8bd300 2023-09-10T01:11:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f84d67feb80 10 client.27674 fetch_fsmap learned FSMap version 29 2023-09-10T01:11:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.660+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561c7c624c00 con 0x561c7c8bd300 2023-09-10T01:11:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f849bfff700 1 -- 172.21.15.3:0/3739808336 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561c7c767490 con 0x561c7c8bd300 2023-09-10T01:11:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f849bfff700 1 -- 172.21.15.3:0/3739808336 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f84b002b480 con 0x561c7c8bd300 2023-09-10T01:11:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f84d67feb80 10 client.27674 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:11:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f84d67feb80 10 client.27674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f84d67feb80 20 client.27674 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f84d67feb80 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c8d7010 0x561c7c8d9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f84d67feb80 4 client.27674 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:11:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c7c824840 con 0x561c7c8d7010 2023-09-10T01:11:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.661+0000 7f84bbfff700 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c8d7010 0x561c7c8d9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.662+0000 7f84bbfff700 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c8d7010 0x561c7c8d9430 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab33ada06ecc0eb2 server_cookie=d813f4332bbe5f85 in_seq=0 out_seq=0 2023-09-10T01:11:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.662+0000 7f849bfff700 10 client.27674 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f849bfff700 1 -- 172.21.15.3:0/3739808336 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561c7c824840 con 0x561c7c8d7010 2023-09-10T01:11:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f849bfff700 10 client.27674 handle_command_reply: tid=0 2023-09-10T01:11:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 client.27674 shutdown 2023-09-10T01:11:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 msgr2=0x561c7c8c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8c9d90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009350 tx=0x7f84b0005c70 comp rx=0 tx=0).stop 2023-09-10T01:11:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 shutdown_connections 2023-09-10T01:11:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c8d7010 0x561c7c8d9430 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c8c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8c9d90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 --2- 172.21.15.3:0/3739808336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f849bfff700 0 client.27674 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 >> 172.21.15.3:0/3739808336 conn(0x561c7c832030 msgr2=0x561c7c8b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 shutdown_connections 2023-09-10T01:11:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 1 -- 172.21.15.3:0/3739808336 wait complete. 2023-09-10T01:11:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.663+0000 7f84d67feb80 20 client.27674 trim_cache size 0 max 16384 2023-09-10T01:11:14.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 Processor -- start 2023-09-10T01:11:14.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 -- start start 2023-09-10T01:11:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8c31a0 0x561c7c8b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c8d3380 con 0x561c7c8c31a0 2023-09-10T01:11:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c8b4fc0 con 0x561c7c8bd300 2023-09-10T01:11:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.681+0000 7f84d67feb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c8c85f0 con 0x561c7c8bdcc0 2023-09-10T01:11:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84bbfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35326/0 (socket says 172.21.15.3:35326) 2023-09-10T01:11:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84c139c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8c31a0 0x561c7c8b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84c0b9b700 1 -- 172.21.15.3:0/867873563 learned_addr learned my addr 172.21.15.3:0/867873563 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:14.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84bbfff700 1 -- 172.21.15.3:0/867873563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 msgr2=0x561c7c8ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84bbfff700 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84bbfff700 1 -- 172.21.15.3:0/867873563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8c31a0 msgr2=0x561c7c8b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84bbfff700 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8c31a0 0x561c7c8b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84bbfff700 1 -- 172.21.15.3:0/867873563 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c7c6621f0 con 0x561c7c8bdcc0 2023-09-10T01:11:14.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:14.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.682+0000 7f84bbfff700 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x561c7c8b8890 tx=0x7f84a801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f79536846c0b5fe5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84a77fe700 1 -- 172.21.15.3:0/867873563 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a800fe00 con 0x561c7c8bdcc0 2023-09-10T01:11:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84a77fe700 1 -- 172.21.15.3:0/867873563 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84a8027d40 con 0x561c7c8bdcc0 2023-09-10T01:11:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84a77fe700 1 -- 172.21.15.3:0/867873563 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a801f830 con 0x561c7c8bdcc0 2023-09-10T01:11:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 -- 172.21.15.3:0/867873563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bdcc0 msgr2=0x561c7c8c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x561c7c8b8890 tx=0x7f84a801a000 comp rx=0 tx=0).stop 2023-09-10T01:11:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 -- 172.21.15.3:0/867873563 shutdown_connections 2023-09-10T01:11:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8c31a0 0x561c7c8b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bd300 0x561c7c8ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 --2- 172.21.15.3:0/867873563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8bdcc0 0x561c7c8c9930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 -- 172.21.15.3:0/867873563 >> 172.21.15.3:0/867873563 conn(0x561c7c832030 msgr2=0x561c7c8c2050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.683+0000 7f84d67feb80 1 -- 172.21.15.3:0/867873563 shutdown_connections 2023-09-10T01:11:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.684+0000 7f84d67feb80 1 -- 172.21.15.3:0/867873563 wait complete. 2023-09-10T01:11:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.684+0000 7f84d67feb80 1 -- start start 2023-09-10T01:11:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.684+0000 7f84d67feb80 1 Processor -- start 2023-09-10T01:11:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bd300 0x561c7c91ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bdcc0 0x561c7c91f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84d67feb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c923f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84d67feb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c924470 con 0x561c7c8bd300 2023-09-10T01:11:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84d67feb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c9245e0 con 0x561c7c8bdcc0 2023-09-10T01:11:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84d67feb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c7c9247a0 con 0x561c7c8c31a0 2023-09-10T01:11:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84d67feb80 10 client.0 _ll_register_callbacks cb 0x7f84b0005ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bdcc0 0x561c7c91f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84c0b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bd300 0x561c7c91ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84c139c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c923f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bdcc0 0x561c7c91f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35340/0 (socket says 172.21.15.3:35340) 2023-09-10T01:11:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 -- 172.21.15.3:0/1324335895 learned_addr learned my addr 172.21.15.3:0/1324335895 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 -- 172.21.15.3:0/1324335895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 msgr2=0x561c7c923f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c923f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 -- 172.21.15.3:0/1324335895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bd300 msgr2=0x561c7c91ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bd300 0x561c7c91ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84bbfff700 1 -- 172.21.15.3:0/1324335895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c7c624c00 con 0x561c7c8bdcc0 2023-09-10T01:11:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84c139c700 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c923f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.685+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bd300 0x561c7c91ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f84bbfff700 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bdcc0 0x561c7c91f4c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f84a8005bf0 tx=0x7f84a8027320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=327c3bd82a898b85 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f849bfff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f849bfff700 1 -- 172.21.15.3:0/1324335895 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8027a90 con 0x561c7c8bdcc0 2023-09-10T01:11:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f849bfff700 1 -- 172.21.15.3:0/1324335895 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f84a8024050 con 0x561c7c8bdcc0 2023-09-10T01:11:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84b001e040 con 0x561c7c8bdcc0 2023-09-10T01:11:14.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f849bfff700 1 -- 172.21.15.3:0/1324335895 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84a8005040 con 0x561c7c8bdcc0 2023-09-10T01:11:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f84d67feb80 10 client.27680 fetch_fsmap learned FSMap version 29 2023-09-10T01:11:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.686+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f84a800f3c0 con 0x561c7c8bdcc0 2023-09-10T01:11:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f849bfff700 1 -- 172.21.15.3:0/1324335895 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84b001e040 con 0x561c7c8bdcc0 2023-09-10T01:11:14.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f849bfff700 1 -- 172.21.15.3:0/1324335895 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f84a8004d20 con 0x561c7c8bdcc0 2023-09-10T01:11:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f84d67feb80 10 client.27680 fetch_fsmap finished waiting for FSMap version 29 2023-09-10T01:11:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f84d67feb80 10 client.27680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:14.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f84d67feb80 20 client.27680 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f84d67feb80 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c92beb0 0x561c7c92e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f84d67feb80 4 client.27680 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:11:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561c7c8088c0 con 0x561c7c92beb0 2023-09-10T01:11:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.687+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c92beb0 0x561c7c92e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.688+0000 7f84c0b9b700 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c92beb0 0x561c7c92e2d0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf12b23d4c9fcf76 server_cookie=84923804a4f0d254 in_seq=0 out_seq=0 2023-09-10T01:11:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.688+0000 7f849bfff700 10 client.27680 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:14.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:14 smithi092 ceph-mon[134067]: pgmap v247: 129 pgs: 129 active+clean; 214 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2023-09-10T01:11:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.770+0000 7f849bfff700 1 -- 172.21.15.3:0/1324335895 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561c7c8088c0 con 0x561c7c92beb0 2023-09-10T01:11:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f849bfff700 10 client.27680 handle_command_reply: tid=0 2023-09-10T01:11:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 client.27680 shutdown 2023-09-10T01:11:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bdcc0 msgr2=0x561c7c91f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bdcc0 0x561c7c91f4c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f84a8005bf0 tx=0x7f84a8027320 comp rx=0 tx=0).stop 2023-09-10T01:11:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 shutdown_connections 2023-09-10T01:11:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x561c7c92beb0 0x561c7c92e2d0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561c7c8c31a0 0x561c7c923f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c7c8bd300 0x561c7c91ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 --2- 172.21.15.3:0/1324335895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561c7c8bdcc0 0x561c7c91f4c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 >> 172.21.15.3:0/1324335895 conn(0x561c7c832030 msgr2=0x561c7c8b6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f849bfff700 0 client.27680 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 shutdown_connections 2023-09-10T01:11:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 1 -- 172.21.15.3:0/1324335895 wait complete. 2023-09-10T01:11:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.771+0000 7f84d67feb80 20 client.27680 trim_cache size 0 max 16384 2023-09-10T01:11:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:11:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:11:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:11:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:11:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84ac0a07b0 msgr2=0x7f84ac0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84ac0a07b0 0x7f84ac0a2c70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84bc19e390 tx=0x7f84a8011720 comp rx=0 tx=0).stop 2023-09-10T01:11:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc1115b0 msgr2=0x7f84bc1a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc1115b0 0x7f84bc1a1720 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f84b4007ee0 tx=0x7f84b400d3b0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 shutdown_connections 2023-09-10T01:11:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f84ac0a07b0 0x7f84ac0a2c70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc1115b0 0x7f84bc1a1720 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84bc0729a0 0x7f84bc19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 --2- 172.21.15.3:0/3758471734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84bc0732c0 0x7f84bc19d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 >> 172.21.15.3:0/3758471734 conn(0x7f84bc0fc0b0 msgr2=0x7f84bc0fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 shutdown_connections 2023-09-10T01:11:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:14.772+0000 7f84c2dff700 1 -- 172.21.15.3:0/3758471734 wait complete. 2023-09-10T01:11:14.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 714231 9.05 MB/sec execute 296 sec latency 34.692 ms 2023-09-10T01:11:15.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 715680 9.04 MB/sec execute 297 sec latency 28.965 ms 2023-09-10T01:11:16.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:16 smithi003 ceph-mon[97654]: pgmap v248: 129 pgs: 129 active+clean; 217 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 344 op/s 2023-09-10T01:11:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:16 smithi136 ceph-mon[133708]: pgmap v248: 129 pgs: 129 active+clean; 217 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 344 op/s 2023-09-10T01:11:16.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:16 smithi092 ceph-mon[134067]: pgmap v248: 129 pgs: 129 active+clean; 217 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 344 op/s 2023-09-10T01:11:16.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 717698 9.05 MB/sec execute 298 sec latency 14.132 ms 2023-09-10T01:11:17.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 719174 9.04 MB/sec execute 299 sec latency 16.323 ms 2023-09-10T01:11:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:18 smithi003 ceph-mon[97654]: pgmap v249: 129 pgs: 129 active+clean; 217 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 344 op/s 2023-09-10T01:11:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:18 smithi136 ceph-mon[133708]: pgmap v249: 129 pgs: 129 active+clean; 217 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 344 op/s 2023-09-10T01:11:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:18 smithi092 ceph-mon[134067]: pgmap v249: 129 pgs: 129 active+clean; 217 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.5 MiB/s wr, 344 op/s 2023-09-10T01:11:18.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 721177 9.06 MB/sec execute 300 sec latency 33.230 ms 2023-09-10T01:11:19.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 722465 9.05 MB/sec execute 301 sec latency 32.050 ms 2023-09-10T01:11:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:20 smithi003 ceph-mon[97654]: pgmap v250: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 390 op/s 2023-09-10T01:11:20.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:20 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:11:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:20 smithi136 ceph-mon[133708]: pgmap v250: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 390 op/s 2023-09-10T01:11:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:20 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:11:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:20 smithi092 ceph-mon[134067]: pgmap v250: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 390 op/s 2023-09-10T01:11:20.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:20 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:11:20.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 724636 9.06 MB/sec execute 302 sec latency 18.287 ms 2023-09-10T01:11:21.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 725789 9.05 MB/sec execute 303 sec latency 35.039 ms 2023-09-10T01:11:22.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:22 smithi003 ceph-mon[97654]: pgmap v251: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.4 MiB/s wr, 384 op/s 2023-09-10T01:11:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:22 smithi136 ceph-mon[133708]: pgmap v251: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.4 MiB/s wr, 384 op/s 2023-09-10T01:11:22.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:22 smithi092 ceph-mon[134067]: pgmap v251: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.4 MiB/s wr, 384 op/s 2023-09-10T01:11:22.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 728163 9.06 MB/sec execute 304 sec latency 18.755 ms 2023-09-10T01:11:23.783 INFO:tasks.workunit.client.0.smithi003.stdout: 1 729399 9.05 MB/sec execute 305 sec latency 17.108 ms 2023-09-10T01:11:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:24 smithi003 ceph-mon[97654]: pgmap v252: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2023-09-10T01:11:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:24 smithi136 ceph-mon[133708]: pgmap v252: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2023-09-10T01:11:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:24 smithi092 ceph-mon[134067]: pgmap v252: 129 pgs: 129 active+clean; 212 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2023-09-10T01:11:24.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 731727 9.07 MB/sec execute 306 sec latency 18.217 ms 2023-09-10T01:11:25.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 732734 9.05 MB/sec execute 307 sec latency 54.169 ms 2023-09-10T01:11:26.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:26 smithi003 ceph-mon[97654]: pgmap v253: 129 pgs: 129 active+clean; 219 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.2 MiB/s wr, 402 op/s 2023-09-10T01:11:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:26 smithi136 ceph-mon[133708]: pgmap v253: 129 pgs: 129 active+clean; 219 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.2 MiB/s wr, 402 op/s 2023-09-10T01:11:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:26 smithi092 ceph-mon[134067]: pgmap v253: 129 pgs: 129 active+clean; 219 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.2 MiB/s wr, 402 op/s 2023-09-10T01:11:26.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 734835 9.06 MB/sec execute 308 sec latency 74.328 ms 2023-09-10T01:11:27.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 735982 9.04 MB/sec execute 309 sec latency 18.540 ms 2023-09-10T01:11:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 -- 172.21.15.3:0/2670879021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec100f80 msgr2=0x7f0aec10d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 --2- 172.21.15.3:0/2670879021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec100f80 0x7f0aec10d440 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400b0d0 tx=0x7f0ae400b3e0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 -- 172.21.15.3:0/2670879021 shutdown_connections 2023-09-10T01:11:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 --2- 172.21.15.3:0/2670879021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec100f80 0x7f0aec10d440 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 --2- 172.21.15.3:0/2670879021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec106680 0x7f0aec106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 --2- 172.21.15.3:0/2670879021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aec1005c0 0x7f0aec100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 -- 172.21.15.3:0/2670879021 >> 172.21.15.3:0/2670879021 conn(0x7f0aec0fc0b0 msgr2=0x7f0aec0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 -- 172.21.15.3:0/2670879021 shutdown_connections 2023-09-10T01:11:27.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.953+0000 7f0af15ff700 1 -- 172.21.15.3:0/2670879021 wait complete. 2023-09-10T01:11:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 Processor -- start 2023-09-10T01:11:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 -- start start 2023-09-10T01:11:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec1005c0 0x7f0aec19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aec100f80 0x7f0aec19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec106680 0x7f0aec198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec198dc0 con 0x7f0aec1005c0 2023-09-10T01:11:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec198f30 con 0x7f0aec106680 2023-09-10T01:11:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.954+0000 7f0af15ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec1990f0 con 0x7f0aec100f80 2023-09-10T01:11:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec1005c0 0x7f0aec19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeb7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec106680 0x7f0aec198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec1005c0 0x7f0aec19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46180/0 (socket says 172.21.15.3:46180) 2023-09-10T01:11:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1349017889 learned_addr learned my addr 172.21.15.3:0/1349017889 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aec100f80 0x7f0aec19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeb7fe700 1 -- 172.21.15.3:0/1349017889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aec100f80 msgr2=0x7f0aec19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeb7fe700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aec100f80 0x7f0aec19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeb7fe700 1 -- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec1005c0 msgr2=0x7f0aec19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeb7fe700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec1005c0 0x7f0aec19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeb7fe700 1 -- 172.21.15.3:0/1349017889 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae4009d00 con 0x7f0aec106680 2023-09-10T01:11:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec1005c0 0x7f0aec19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aec100f80 0x7f0aec19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.955+0000 7f0aeb7fe700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec106680 0x7f0aec198750 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009fd0 tx=0x7f0ae4007860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.956+0000 7f0ad3fff700 1 -- 172.21.15.3:0/1349017889 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae400e040 con 0x7f0aec106680 2023-09-10T01:11:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.956+0000 7f0ad3fff700 1 -- 172.21.15.3:0/1349017889 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ae400bcb0 con 0x7f0aec106680 2023-09-10T01:11:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.956+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec199370 con 0x7f0aec106680 2023-09-10T01:11:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.956+0000 7f0ad3fff700 1 -- 172.21.15.3:0/1349017889 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae40058b0 con 0x7f0aec106680 2023-09-10T01:11:27.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.956+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aec1a56d0 con 0x7f0aec106680 2023-09-10T01:11:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.960+0000 7f0ad3fff700 1 -- 172.21.15.3:0/1349017889 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f0ae4004a50 con 0x7f0aec106680 2023-09-10T01:11:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 Processor -- start 2023-09-10T01:11:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 -- start start 2023-09-10T01:11:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e64aa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e63fd6d0 con 0x5645e64aa1a0 2023-09-10T01:11:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e63faff0 con 0x5645e64a4300 2023-09-10T01:11:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.961+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e63f68e0 con 0x5645e64a4cc0 2023-09-10T01:11:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48042/0 (socket says 172.21.15.3:48042) 2023-09-10T01:11:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1101137822 learned_addr learned my addr 172.21.15.3:0/1101137822 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e64aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0ad3fff700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0ad40a0830 0x7f0ad40a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1101137822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 msgr2=0x5645e64b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1101137822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 msgr2=0x5645e64aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e64aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1101137822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645e6222d80 con 0x5645e64a4300 2023-09-10T01:11:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeb7fe700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0ad40a0830 0x7f0ad40a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0ad3fff700 1 -- 172.21.15.3:0/1349017889 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0ae40cd540 con 0x7f0aec106680 2023-09-10T01:11:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.962+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a4780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0117e0 tx=0x7f0adc011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91bcf0c5cfd54c45 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.963+0000 7f0ad17fa700 1 -- 172.21.15.3:0/1101137822 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc00f480 con 0x5645e64a4300 2023-09-10T01:11:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.963+0000 7f0ad17fa700 1 -- 172.21.15.3:0/1101137822 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0adc00faa0 con 0x5645e64a4300 2023-09-10T01:11:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.963+0000 7f0ad17fa700 1 -- 172.21.15.3:0/1101137822 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc019a20 con 0x5645e64a4300 2023-09-10T01:11:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1101137822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 msgr2=0x5645e64a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0aeb7fe700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0ad40a0830 0x7f0ad40a2cf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4096b90 tx=0x7f0ae4005310 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a4780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0117e0 tx=0x7f0adc011af0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1101137822 shutdown_connections 2023-09-10T01:11:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e64aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1101137822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a4780 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1101137822 >> 172.21.15.3:0/1101137822 conn(0x5645e6419030 msgr2=0x5645e649f560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1101137822 shutdown_connections 2023-09-10T01:11:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1101137822 wait complete. 2023-09-10T01:11:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.964+0000 7f0b04ebbb80 1 -- start start 2023-09-10T01:11:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.965+0000 7f0b04ebbb80 1 Processor -- start 2023-09-10T01:11:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4300 0x5645e64b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64aa1a0 0x5645e64ae670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e6394b10 con 0x5645e64a4cc0 2023-09-10T01:11:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e6394810 con 0x5645e64aa1a0 2023-09-10T01:11:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e62788c0 con 0x5645e64a4300 2023-09-10T01:11:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0b04ebbb80 10 client.0 _ll_register_callbacks cb 0x5645e62dea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aea7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4300 0x5645e64b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aeaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aea7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4300 0x5645e64b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43950/0 (socket says 172.21.15.3:43950) 2023-09-10T01:11:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aea7fc700 1 -- 172.21.15.3:0/2663716194 learned_addr learned my addr 172.21.15.3:0/2663716194 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64aa1a0 0x5645e64ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aeaffd700 1 -- 172.21.15.3:0/2663716194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4300 msgr2=0x5645e64b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4300 0x5645e64b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aeaffd700 1 -- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64aa1a0 msgr2=0x5645e64ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64aa1a0 0x5645e64ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aeaffd700 1 -- 172.21.15.3:0/2663716194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0adc011440 con 0x5645e64a4cc0 2023-09-10T01:11:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.966+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4300 0x5645e64b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.967+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64ae0f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0adc01c040 tx=0x7f0adc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc780cc4093f5ee1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.967+0000 7f0aceff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:11:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.967+0000 7f0aceff5700 1 -- 172.21.15.3:0/2663716194 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc00f920 con 0x5645e64a4cc0 2023-09-10T01:11:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.967+0000 7f0aceff5700 1 -- 172.21.15.3:0/2663716194 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0adc005ce0 con 0x5645e64a4cc0 2023-09-10T01:11:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.967+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5645e634e490 con 0x5645e64a4cc0 2023-09-10T01:11:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0aceff5700 1 -- 172.21.15.3:0/2663716194 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0adc005250 con 0x5645e64a4cc0 2023-09-10T01:11:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 10 client.17610 fetch_fsmap learned FSMap version 30 2023-09-10T01:11:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5645e620bc00 con 0x5645e64a4cc0 2023-09-10T01:11:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0aceff5700 1 -- 172.21.15.3:0/2663716194 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5645e634e490 con 0x5645e64a4cc0 2023-09-10T01:11:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0aceff5700 1 -- 172.21.15.3:0/2663716194 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0adc019600 con 0x5645e64a4cc0 2023-09-10T01:11:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 10 client.17610 fetch_fsmap finished waiting for FSMap version 30 2023-09-10T01:11:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 10 client.17610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 20 client.17610 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e64be3b0 0x5645e64c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 4 client.17610 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:11:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.968+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5645e640b840 con 0x5645e64be3b0 2023-09-10T01:11:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.969+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e64be3b0 0x5645e64c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.969+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e64be3b0 0x5645e64c07d0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2856e95fa28c00c8 server_cookie=4f7ac4dd4c3b98a7 in_seq=0 out_seq=0 2023-09-10T01:11:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.969+0000 7f0aceff5700 10 client.17610 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.970+0000 7f0aceff5700 1 -- 172.21.15.3:0/2663716194 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5645e640b840 con 0x5645e64be3b0 2023-09-10T01:11:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.970+0000 7f0aceff5700 10 client.17610 handle_command_reply: tid=0 2023-09-10T01:11:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 client.17610 shutdown 2023-09-10T01:11:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 msgr2=0x5645e64ae0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64ae0f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0adc01c040 tx=0x7f0adc00f730 comp rx=0 tx=0).stop 2023-09-10T01:11:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 shutdown_connections 2023-09-10T01:11:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e64be3b0 0x5645e64c07d0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4300 0x5645e64b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64aa1a0 0x5645e64ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/2663716194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64ae0f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 >> 172.21.15.3:0/2663716194 conn(0x5645e6419030 msgr2=0x5645e649e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0aceff5700 0 client.17610 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 shutdown_connections 2023-09-10T01:11:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/2663716194 wait complete. 2023-09-10T01:11:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.971+0000 7f0b04ebbb80 20 client.17610 trim_cache size 0 max 16384 2023-09-10T01:11:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 Processor -- start 2023-09-10T01:11:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 -- start start 2023-09-10T01:11:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64a8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64aa1a0 0x5645e64a6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e64998a0 con 0x5645e64a4cc0 2023-09-10T01:11:27.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e64a9db0 con 0x5645e64a4300 2023-09-10T01:11:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e64ac690 con 0x5645e64aa1a0 2023-09-10T01:11:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0aea7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0aea7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48050/0 (socket says 172.21.15.3:48050) 2023-09-10T01:11:27.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0aea7fc700 1 -- 172.21.15.3:0/1485713140 learned_addr learned my addr 172.21.15.3:0/1485713140 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.988+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64aa1a0 0x5645e64a6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1485713140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64aa1a0 msgr2=0x5645e64a6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64aa1a0 0x5645e64a6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1485713140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 msgr2=0x5645e64a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aeaffd700 1 -- 172.21.15.3:0/1485713140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645e62491f0 con 0x5645e64a4cc0 2023-09-10T01:11:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64a8800 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0adc019960 tx=0x7f0adc009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee285ce97f3a1e6c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.989+0000 7f0ad1ffb700 1 -- 172.21.15.3:0/1485713140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc015030 con 0x5645e64a4cc0 2023-09-10T01:11:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.990+0000 7f0ad1ffb700 1 -- 172.21.15.3:0/1485713140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0adc005710 con 0x5645e64a4cc0 2023-09-10T01:11:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.990+0000 7f0ad1ffb700 1 -- 172.21.15.3:0/1485713140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc003680 con 0x5645e64a4cc0 2023-09-10T01:11:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1485713140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 msgr2=0x5645e64a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64a8800 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0adc019960 tx=0x7f0adc009fd0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1485713140 shutdown_connections 2023-09-10T01:11:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64aa1a0 0x5645e64a6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64a82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/1485713140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64a4cc0 0x5645e64a8800 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1485713140 >> 172.21.15.3:0/1485713140 conn(0x5645e6419030 msgr2=0x5645e64a9210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1485713140 shutdown_connections 2023-09-10T01:11:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/1485713140 wait complete. 2023-09-10T01:11:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.991+0000 7f0b04ebbb80 1 -- start start 2023-09-10T01:11:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 1 Processor -- start 2023-09-10T01:11:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:27.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e65111e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e64b43a0 con 0x5645e64aa1a0 2023-09-10T01:11:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e6511720 con 0x5645e64a4300 2023-09-10T01:11:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645e6511860 con 0x5645e64a4cc0 2023-09-10T01:11:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0b04ebbb80 10 client.0 _ll_register_callbacks cb 0x7f0ae000bed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0aea7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0aeaffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0aeaffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43970/0 (socket says 172.21.15.3:43970) 2023-09-10T01:11:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0aea7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48054/0 (socket says 172.21.15.3:48054) 2023-09-10T01:11:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0aeaffd700 1 -- 172.21.15.3:0/82992966 learned_addr learned my addr 172.21.15.3:0/82992966 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.992+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e65111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aeaffd700 1 -- 172.21.15.3:0/82992966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 msgr2=0x5645e64b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aeaffd700 1 -- 172.21.15.3:0/82992966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 msgr2=0x5645e65111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e65111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aeaffd700 1 -- 172.21.15.3:0/82992966 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645e620bc00 con 0x5645e64a4cc0 2023-09-10T01:11:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aeaffd700 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b3a90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0adc010010 tx=0x7f0adc0197b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=746bb5fe454596bb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aceff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:11:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aceff5700 1 -- 172.21.15.3:0/82992966 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc015070 con 0x5645e64a4cc0 2023-09-10T01:11:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0adc00faa0 con 0x5645e64a4cc0 2023-09-10T01:11:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.993+0000 7f0aceff5700 1 -- 172.21.15.3:0/82992966 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0adc016070 con 0x5645e64a4cc0 2023-09-10T01:11:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0aceff5700 1 -- 172.21.15.3:0/82992966 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0adc005250 con 0x5645e64a4cc0 2023-09-10T01:11:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 10 client.27604 fetch_fsmap learned FSMap version 30 2023-09-10T01:11:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5645e6221250 con 0x5645e64a4cc0 2023-09-10T01:11:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0aceff5700 1 -- 172.21.15.3:0/82992966 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0adc00faa0 con 0x5645e64a4cc0 2023-09-10T01:11:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0aceff5700 1 -- 172.21.15.3:0/82992966 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0adc036480 con 0x5645e64a4cc0 2023-09-10T01:11:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 10 client.27604 fetch_fsmap finished waiting for FSMap version 30 2023-09-10T01:11:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 10 client.27604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 20 client.27604 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e6512b90 0x5645e6514fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 4 client.27604 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:11:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5645e63ef8c0 con 0x5645e6512b90 2023-09-10T01:11:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.994+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e6512b90 0x5645e6514fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.995+0000 7f0aea7fc700 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e6512b90 0x5645e6514fb0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ecc5aa7f9779466 server_cookie=16335586af9b2673 in_seq=0 out_seq=0 2023-09-10T01:11:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:27.995+0000 7f0aceff5700 10 client.27604 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.118+0000 7f0aceff5700 1 -- 172.21.15.3:0/82992966 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5645e63ef8c0 con 0x5645e6512b90 2023-09-10T01:11:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.118+0000 7f0aceff5700 10 client.27604 handle_command_reply: tid=0 2023-09-10T01:11:28.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.120+0000 7f0b04ebbb80 1 client.27604 shutdown 2023-09-10T01:11:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 msgr2=0x5645e64b3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b3a90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0adc010010 tx=0x7f0adc0197b0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 shutdown_connections 2023-09-10T01:11:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5645e6512b90 0x5645e6514fb0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5645e64aa1a0 0x5645e65111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5645e64a4300 0x5645e64b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 --2- 172.21.15.3:0/82992966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5645e64a4cc0 0x5645e64b3a90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0aceff5700 0 client.27604 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.121+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 >> 172.21.15.3:0/82992966 conn(0x5645e6419030 msgr2=0x5645e64a9210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.122+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 shutdown_connections 2023-09-10T01:11:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.122+0000 7f0b04ebbb80 1 -- 172.21.15.3:0/82992966 wait complete. 2023-09-10T01:11:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.122+0000 7f0b04ebbb80 20 client.27604 trim_cache size 0 max 16384 2023-09-10T01:11:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:11:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:11:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:11:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:11:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0ad40a0830 msgr2=0x7f0ad40a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0ad40a0830 0x7f0ad40a2cf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4096b90 tx=0x7f0ae4005310 comp rx=0 tx=0).stop 2023-09-10T01:11:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec106680 msgr2=0x7f0aec198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec106680 0x7f0aec198750 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009fd0 tx=0x7f0ae4007860 comp rx=0 tx=0).stop 2023-09-10T01:11:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 shutdown_connections 2023-09-10T01:11:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0ad40a0830 0x7f0ad40a2cf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0aec106680 0x7f0aec198750 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0aec1005c0 0x7f0aec19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 --2- 172.21.15.3:0/1349017889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aec100f80 0x7f0aec19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 >> 172.21.15.3:0/1349017889 conn(0x7f0aec0fc0b0 msgr2=0x7f0aec10b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 shutdown_connections 2023-09-10T01:11:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:28.123+0000 7f0af15ff700 1 -- 172.21.15.3:0/1349017889 wait complete. 2023-09-10T01:11:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:28 smithi003 ceph-mon[97654]: pgmap v254: 129 pgs: 129 active+clean; 219 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 188 op/s 2023-09-10T01:11:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:28 smithi136 ceph-mon[133708]: pgmap v254: 129 pgs: 129 active+clean; 219 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 188 op/s 2023-09-10T01:11:28.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:28 smithi092 ceph-mon[134067]: pgmap v254: 129 pgs: 129 active+clean; 219 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 188 op/s 2023-09-10T01:11:28.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 737806 9.05 MB/sec execute 310 sec latency 112.821 ms 2023-09-10T01:11:29.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:29 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:11:29.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:29 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:11:29.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:29 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:11:29.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 739057 9.05 MB/sec execute 311 sec latency 19.051 ms 2023-09-10T01:11:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:30 smithi003 ceph-mon[97654]: pgmap v255: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2023-09-10T01:11:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:30 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:11:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:30 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:11:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:30 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:11:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:30 smithi136 ceph-mon[133708]: pgmap v255: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2023-09-10T01:11:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:30 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:11:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:30 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:11:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:30 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:11:30.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:30 smithi092 ceph-mon[134067]: pgmap v255: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2023-09-10T01:11:30.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:30 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:11:30.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:30 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:11:30.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:30 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:11:30.784 INFO:tasks.workunit.client.0.smithi003.stdout: 1 740765 9.03 MB/sec execute 312 sec latency 17.814 ms 2023-09-10T01:11:31.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 742510 9.05 MB/sec execute 313 sec latency 23.465 ms 2023-09-10T01:11:32.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:32 smithi003 ceph-mon[97654]: pgmap v256: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2023-09-10T01:11:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:32 smithi136 ceph-mon[133708]: pgmap v256: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2023-09-10T01:11:32.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:32 smithi092 ceph-mon[134067]: pgmap v256: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2023-09-10T01:11:32.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 743796 9.04 MB/sec execute 314 sec latency 35.560 ms 2023-09-10T01:11:33.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 745973 9.05 MB/sec execute 315 sec latency 15.161 ms 2023-09-10T01:11:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:34 smithi003 ceph-mon[97654]: pgmap v257: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2023-09-10T01:11:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:34 smithi136 ceph-mon[133708]: pgmap v257: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2023-09-10T01:11:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:34 smithi092 ceph-mon[134067]: pgmap v257: 129 pgs: 129 active+clean; 215 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2023-09-10T01:11:34.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 747199 9.04 MB/sec execute 316 sec latency 18.538 ms 2023-09-10T01:11:35.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:35 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:11:35.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:35 smithi003 ceph-mon[97654]: pgmap v258: 129 pgs: 129 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2023-09-10T01:11:35.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:35 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:11:35.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:35 smithi136 ceph-mon[133708]: pgmap v258: 129 pgs: 129 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2023-09-10T01:11:35.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:35 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:11:35.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:35 smithi092 ceph-mon[134067]: pgmap v258: 129 pgs: 129 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2023-09-10T01:11:35.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 748879 9.04 MB/sec execute 317 sec latency 91.164 ms 2023-09-10T01:11:36.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 749602 9.03 MB/sec execute 318 sec latency 43.437 ms 2023-09-10T01:11:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:37 smithi136 ceph-mon[133708]: pgmap v259: 129 pgs: 129 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2023-09-10T01:11:37.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:37 smithi092 ceph-mon[134067]: pgmap v259: 129 pgs: 129 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2023-09-10T01:11:37.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 750742 9.01 MB/sec execute 319 sec latency 48.733 ms 2023-09-10T01:11:37.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:37 smithi003 ceph-mon[97654]: pgmap v259: 129 pgs: 129 active+clean; 223 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2023-09-10T01:11:38.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 752975 9.03 MB/sec execute 320 sec latency 20.457 ms 2023-09-10T01:11:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:39 smithi136 ceph-mon[133708]: pgmap v260: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 372 op/s 2023-09-10T01:11:39.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:39 smithi092 ceph-mon[134067]: pgmap v260: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 372 op/s 2023-09-10T01:11:39.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 753947 9.01 MB/sec execute 321 sec latency 18.622 ms 2023-09-10T01:11:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:39 smithi003 ceph-mon[97654]: pgmap v260: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.1 MiB/s wr, 372 op/s 2023-09-10T01:11:40.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 756124 9.02 MB/sec execute 322 sec latency 18.369 ms 2023-09-10T01:11:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 -- 172.21.15.3:0/1198679639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100580 msgr2=0x7fd4e4100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/1198679639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100580 0x7fd4e4100a00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40099c0 tx=0x7fd4d4009cd0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 -- 172.21.15.3:0/1198679639 shutdown_connections 2023-09-10T01:11:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/1198679639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100f40 0x7fd4e410d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/1198679639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e4106640 0x7fd4e4106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/1198679639 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100580 0x7fd4e4100a00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 -- 172.21.15.3:0/1198679639 >> 172.21.15.3:0/1198679639 conn(0x7fd4e40fc050 msgr2=0x7fd4e40fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 -- 172.21.15.3:0/1198679639 shutdown_connections 2023-09-10T01:11:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.286+0000 7fd4ebfff700 1 -- 172.21.15.3:0/1198679639 wait complete. 2023-09-10T01:11:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.288+0000 7fd4ebfff700 1 Processor -- start 2023-09-10T01:11:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.288+0000 7fd4ebfff700 1 -- start start 2023-09-10T01:11:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.288+0000 7fd4ebfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100580 0x7fd4e4196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4ebfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100f40 0x7fd4e4196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4ebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e4106640 0x7fd4e419afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4ebfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e419b5f0 con 0x7fd4e4106640 2023-09-10T01:11:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4ebfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e419b760 con 0x7fd4e4100f40 2023-09-10T01:11:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4ebfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e419b920 con 0x7fd4e4100580 2023-09-10T01:11:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100580 0x7fd4e4196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e959a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100f40 0x7fd4e4196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100580 0x7fd4e4196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40890/0 (socket says 172.21.15.3:40890) 2023-09-10T01:11:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e959a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100f40 0x7fd4e4196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36016/0 (socket says 172.21.15.3:36016) 2023-09-10T01:11:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3368124413 learned_addr learned my addr 172.21.15.3:0/3368124413 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4ea59c700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e4106640 0x7fd4e419afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3368124413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100f40 msgr2=0x7fd4e4196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100f40 0x7fd4e4196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e4106640 msgr2=0x7fd4e419afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e4106640 0x7fd4e419afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3368124413 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc009520 con 0x7fd4e4100580 2023-09-10T01:11:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.289+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100f40 0x7fd4e4196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:11:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.290+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100580 0x7fd4e4196610 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000ef10 tx=0x7fd4e000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.290+0000 7fd4daffd700 1 -- 172.21.15.3:0/3368124413 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e000ca10 con 0x7fd4e4100580 2023-09-10T01:11:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.290+0000 7fd4daffd700 1 -- 172.21.15.3:0/3368124413 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4e000f040 con 0x7fd4e4100580 2023-09-10T01:11:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.290+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d40095f0 con 0x7fd4e4100580 2023-09-10T01:11:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.290+0000 7fd4daffd700 1 -- 172.21.15.3:0/3368124413 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e00146e0 con 0x7fd4e4100580 2023-09-10T01:11:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.290+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e41a3260 con 0x7fd4e4100580 2023-09-10T01:11:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.294+0000 7fd4daffd700 1 -- 172.21.15.3:0/3368124413 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd4e0007500 con 0x7fd4e4100580 2023-09-10T01:11:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.295+0000 7fd4daffd700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4d00a0830 0x7fd4d00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.295+0000 7fd4ff87db80 1 Processor -- start 2023-09-10T01:11:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.295+0000 7fd4daffd700 1 -- 172.21.15.3:0/3368124413 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd4e00cc6d0 con 0x7fd4e4100580 2023-09-10T01:11:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ff87db80 1 -- start start 2023-09-10T01:11:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4d00a0830 0x7fd4d00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bba1a0 0x55fb17bba580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bb4300 0x55fb17bb4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17b0d6d0 con 0x55fb17bb4300 2023-09-10T01:11:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17b0aff0 con 0x55fb17bba1a0 2023-09-10T01:11:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17b068e0 con 0x55fb17bb4cc0 2023-09-10T01:11:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bba1a0 0x55fb17bba580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bba1a0 0x55fb17bba580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36030/0 (socket says 172.21.15.3:36030) 2023-09-10T01:11:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3681736550 learned_addr learned my addr 172.21.15.3:0/3681736550 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4ea59c700 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.296+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bb4300 0x55fb17bb4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3681736550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 msgr2=0x55fb17bc0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3681736550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bb4300 msgr2=0x55fb17bb4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bb4300 0x55fb17bb4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3681736550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb17932d80 con 0x55fb17bba1a0 2023-09-10T01:11:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4d00a0830 0x7fd4d00a2cf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009990 tx=0x7fd4d4005c20 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bba1a0 0x55fb17bba580 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00961a0 tx=0x7fd4e0099090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924fc1641b067487 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4cffff700 1 -- 172.21.15.3:0/3681736550 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e000cb70 con 0x55fb17bba1a0 2023-09-10T01:11:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4cffff700 1 -- 172.21.15.3:0/3681736550 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4e0099730 con 0x55fb17bba1a0 2023-09-10T01:11:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.297+0000 7fd4cffff700 1 -- 172.21.15.3:0/3681736550 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e009d160 con 0x55fb17bba1a0 2023-09-10T01:11:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3681736550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bba1a0 msgr2=0x55fb17bba580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bba1a0 0x55fb17bba580 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00961a0 tx=0x7fd4e0099090 comp rx=0 tx=0).stop 2023-09-10T01:11:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3681736550 shutdown_connections 2023-09-10T01:11:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bba1a0 0x55fb17bba580 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3681736550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bb4300 0x55fb17bb4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3681736550 >> 172.21.15.3:0/3681736550 conn(0x55fb17b29030 msgr2=0x55fb17baf560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3681736550 shutdown_connections 2023-09-10T01:11:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.298+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3681736550 wait complete. 2023-09-10T01:11:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 -- start start 2023-09-10T01:11:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 Processor -- start 2023-09-10T01:11:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17bc0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bbe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bbe610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17aa4b10 con 0x55fb17bba1a0 2023-09-10T01:11:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17aa4810 con 0x55fb17bb4300 2023-09-10T01:11:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.299+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb179888c0 con 0x55fb17bb4cc0 2023-09-10T01:11:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4ff87db80 10 client.0 _ll_register_callbacks cb 0x55fb179eea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17bc0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bbe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bbe090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40918/0 (socket says 172.21.15.3:40918) 2023-09-10T01:11:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 -- 172.21.15.3:0/3315557646 learned_addr learned my addr 172.21.15.3:0/3315557646 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4ea59c700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bbe610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 -- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 msgr2=0x55fb17bc0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17bc0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 -- 172.21.15.3:0/3315557646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 msgr2=0x55fb17bbe610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bbe610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 -- 172.21.15.3:0/3315557646 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e41a3260 con 0x55fb17bb4cc0 2023-09-10T01:11:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17bc0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4ea59c700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bbe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bbe090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4015000 tx=0x7fd4d4020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c9de338b0eb4f7d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.300+0000 7fd4cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:11:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3315557646 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d401d050 con 0x55fb17bb4cc0 2023-09-10T01:11:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb17a5e490 con 0x55fb17bb4cc0 2023-09-10T01:11:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3315557646 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd4d402a040 con 0x55fb17bb4cc0 2023-09-10T01:11:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3315557646 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d402e640 con 0x55fb17bb4cc0 2023-09-10T01:11:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 10 client.27616 fetch_fsmap learned FSMap version 31 2023-09-10T01:11:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fb1791bc00 con 0x55fb17bb4cc0 2023-09-10T01:11:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3315557646 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fb17a5e490 con 0x55fb17bb4cc0 2023-09-10T01:11:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3315557646 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd4d4025980 con 0x55fb17bb4cc0 2023-09-10T01:11:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 10 client.27616 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:11:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 10 client.27616 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 20 client.27616 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17bd26d0 0x55fb17bd4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 4 client.27616 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:11:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.301+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb17b1b840 con 0x55fb17bd26d0 2023-09-10T01:11:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.302+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17bd26d0 0x55fb17bd4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.302+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17bd26d0 0x55fb17bd4af0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37783f709164fad server_cookie=139d655f365f81e1 in_seq=0 out_seq=0 2023-09-10T01:11:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.302+0000 7fd4cd7fa700 10 client.27616 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3315557646 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fb17b1b840 con 0x55fb17bd26d0 2023-09-10T01:11:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4cd7fa700 10 client.27616 handle_command_reply: tid=0 2023-09-10T01:11:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 client.27616 shutdown 2023-09-10T01:11:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 msgr2=0x55fb17bbe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bbe090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4015000 tx=0x7fd4d4020000 comp rx=0 tx=0).stop 2023-09-10T01:11:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 shutdown_connections 2023-09-10T01:11:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17bd26d0 0x55fb17bd4af0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bbe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17bc0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3315557646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bbe090 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 >> 172.21.15.3:0/3315557646 conn(0x55fb17b29030 msgr2=0x55fb17bae7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4cd7fa700 0 client.27616 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 shutdown_connections 2023-09-10T01:11:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3315557646 wait complete. 2023-09-10T01:11:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.303+0000 7fd4ff87db80 20 client.27616 trim_cache size 0 max 16384 2023-09-10T01:11:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 Processor -- start 2023-09-10T01:11:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 -- start start 2023-09-10T01:11:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17baef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17bac000 con 0x55fb17bba1a0 2023-09-10T01:11:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17bb5580 con 0x55fb17bb4300 2023-09-10T01:11:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17bbc670 con 0x55fb17bb4cc0 2023-09-10T01:11:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17baef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17baef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36040/0 (socket says 172.21.15.3:36040) 2023-09-10T01:11:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4ea59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/156333955 learned_addr learned my addr 172.21.15.3:0/156333955 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.321+0000 7fd4e959a700 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/156333955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 msgr2=0x55fb17bc08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/156333955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 msgr2=0x55fb17bc0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/156333955 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb179591f0 con 0x55fb17bb4300 2023-09-10T01:11:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17baef20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0099430 tx=0x7fd4e0007b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef09b4c3107889a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4d8ff9700 1 -- 172.21.15.3:0/156333955 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e009d5e0 con 0x55fb17bb4300 2023-09-10T01:11:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4d8ff9700 1 -- 172.21.15.3:0/156333955 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4e00a17a0 con 0x55fb17bb4300 2023-09-10T01:11:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.322+0000 7fd4d8ff9700 1 -- 172.21.15.3:0/156333955 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0099990 con 0x55fb17bb4300 2023-09-10T01:11:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 -- 172.21.15.3:0/156333955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 msgr2=0x55fb17baef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17baef20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0099430 tx=0x7fd4e0007b30 comp rx=0 tx=0).stop 2023-09-10T01:11:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 -- 172.21.15.3:0/156333955 shutdown_connections 2023-09-10T01:11:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4300 0x55fb17baef20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/156333955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 -- 172.21.15.3:0/156333955 >> 172.21.15.3:0/156333955 conn(0x55fb17b29030 msgr2=0x55fb17bb8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 -- 172.21.15.3:0/156333955 shutdown_connections 2023-09-10T01:11:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.323+0000 7fd4ff87db80 1 -- 172.21.15.3:0/156333955 wait complete. 2023-09-10T01:11:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 -- start start 2023-09-10T01:11:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 Processor -- start 2023-09-10T01:11:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4300 0x55fb17bc92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17bc8020 con 0x55fb17bba1a0 2023-09-10T01:11:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17bc8190 con 0x55fb17bb4cc0 2023-09-10T01:11:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb17bc8350 con 0x55fb17bb4300 2023-09-10T01:11:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4ff87db80 10 client.0 _ll_register_callbacks cb 0x7fd4d400bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4300 0x55fb17bc92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4e959a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4e9d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4300 0x55fb17bc92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40932/0 (socket says 172.21.15.3:40932) 2023-09-10T01:11:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3992868573 learned_addr learned my addr 172.21.15.3:0/3992868573 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.324+0000 7fd4e959a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36046/0 (socket says 172.21.15.3:36046) 2023-09-10T01:11:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4ea59c700 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3992868573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4cc0 msgr2=0x55fb17bc9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3992868573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 msgr2=0x55fb17bc79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4e9d9b700 1 -- 172.21.15.3:0/3992868573 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb17932d80 con 0x55fb17bb4300 2023-09-10T01:11:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4ea59c700 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:11:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4e9d9b700 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4300 0x55fb17bc92e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009c10 tx=0x7fd4e0095a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da6976bd1a42bbce server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4cd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:11:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3992868573 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e009d160 con 0x55fb17bb4300 2023-09-10T01:11:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3992868573 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd4e009d2c0 con 0x55fb17bb4300 2023-09-10T01:11:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4e0099730 con 0x55fb17bb4300 2023-09-10T01:11:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.325+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3992868573 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4e0012340 con 0x55fb17bb4300 2023-09-10T01:11:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 10 client.27622 fetch_fsmap learned FSMap version 31 2023-09-10T01:11:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fb17931250 con 0x55fb17bb4300 2023-09-10T01:11:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3992868573 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4e0099730 con 0x55fb17bb4300 2023-09-10T01:11:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3992868573 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd4e0099ab0 con 0x55fb17bb4300 2023-09-10T01:11:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 10 client.27622 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:11:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 10 client.27622 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 20 client.27622 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17c270a0 0x55fb17c29490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 4 client.27622 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:11:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fb17aff8c0 con 0x55fb17c270a0 2023-09-10T01:11:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.326+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17c270a0 0x55fb17c29490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.327+0000 7fd4e959a700 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17c270a0 0x55fb17c29490 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=386390a2bfcbf542 server_cookie=a6f99fe680fde498 in_seq=0 out_seq=0 2023-09-10T01:11:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.327+0000 7fd4cd7fa700 10 client.27622 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:41.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.425+0000 7fd4cd7fa700 1 -- 172.21.15.3:0/3992868573 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fb17aff8c0 con 0x55fb17c270a0 2023-09-10T01:11:41.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.425+0000 7fd4cd7fa700 10 client.27622 handle_command_reply: tid=0 2023-09-10T01:11:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.425+0000 7fd4ff87db80 1 client.27622 shutdown 2023-09-10T01:11:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4300 msgr2=0x55fb17bc92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4300 0x55fb17bc92e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0009c10 tx=0x7fd4e0095a70 comp rx=0 tx=0).stop 2023-09-10T01:11:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 shutdown_connections 2023-09-10T01:11:41.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fb17c270a0 0x55fb17c29490 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fb17bb4300 0x55fb17bc92e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4cd7fa700 0 client.27622 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fb17bba1a0 0x55fb17bc79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 --2- 172.21.15.3:0/3992868573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fb17bb4cc0 0x55fb17bc9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 >> 172.21.15.3:0/3992868573 conn(0x55fb17b29030 msgr2=0x55fb17bb90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 shutdown_connections 2023-09-10T01:11:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 1 -- 172.21.15.3:0/3992868573 wait complete. 2023-09-10T01:11:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.426+0000 7fd4ff87db80 20 client.27622 trim_cache size 0 max 16384 2023-09-10T01:11:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:11:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:11:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:11:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:11:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.429+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4d00a0830 msgr2=0x7fd4d00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.429+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4d00a0830 0x7fd4d00a2cf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009990 tx=0x7fd4d4005c20 comp rx=0 tx=0).stop 2023-09-10T01:11:41.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.429+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100580 msgr2=0x7fd4e4196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.429+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100580 0x7fd4e4196610 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000ef10 tx=0x7fd4e000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 shutdown_connections 2023-09-10T01:11:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4d00a0830 0x7fd4d00a2cf0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e4100580 0x7fd4e4196610 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e4106640 0x7fd4e419afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 --2- 172.21.15.3:0/3368124413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4e4100f40 0x7fd4e4196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 >> 172.21.15.3:0/3368124413 conn(0x7fd4e40fc050 msgr2=0x7fd4e40fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 shutdown_connections 2023-09-10T01:11:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:41.430+0000 7fd4ebfff700 1 -- 172.21.15.3:0/3368124413 wait complete. 2023-09-10T01:11:41.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 757003 9.01 MB/sec execute 323 sec latency 45.815 ms 2023-09-10T01:11:42.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:11:42.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 -- 172.21.15.3:0/2770248721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84072a30 msgr2=0x7fea84072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 --2- 172.21.15.3:0/2770248721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84072a30 0x7fea84072e10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fea80009f90 tx=0x7fea8000b330 comp rx=0 tx=0).stop 2023-09-10T01:11:42.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 -- 172.21.15.3:0/2770248721 shutdown_connections 2023-09-10T01:11:42.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 --2- 172.21.15.3:0/2770248721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea8410ce80 0x7fea8410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 --2- 172.21.15.3:0/2770248721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84072a30 0x7fea84072e10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 --2- 172.21.15.3:0/2770248721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea84073350 0x7fea8410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 -- 172.21.15.3:0/2770248721 >> 172.21.15.3:0/2770248721 conn(0x7fea840fc170 msgr2=0x7fea840fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:42.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 -- 172.21.15.3:0/2770248721 shutdown_connections 2023-09-10T01:11:42.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.312+0000 7fea8bbff700 1 -- 172.21.15.3:0/2770248721 wait complete. 2023-09-10T01:11:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.313+0000 7fea8bbff700 1 Processor -- start 2023-09-10T01:11:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8bbff700 1 -- start start 2023-09-10T01:11:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8bbff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea84072a30 0x7fea8419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8bbff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84073350 0x7fea8419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8bbff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8410ce80 0x7fea841988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8bbff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84198f50 con 0x7fea84073350 2023-09-10T01:11:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8bbff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea841990c0 con 0x7fea84072a30 2023-09-10T01:11:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8bbff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84199280 con 0x7fea8410ce80 2023-09-10T01:11:42.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea84072a30 0x7fea8419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea84072a30 0x7fea8419e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36054/0 (socket says 172.21.15.3:36054) 2023-09-10T01:11:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8999b700 1 -- 172.21.15.3:0/86450332 learned_addr learned my addr 172.21.15.3:0/86450332 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.314+0000 7fea8a19c700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8410ce80 0x7fea841988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8919a700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84073350 0x7fea8419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8a19c700 1 -- 172.21.15.3:0/86450332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea84072a30 msgr2=0x7fea8419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8a19c700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea84072a30 0x7fea8419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8a19c700 1 -- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84073350 msgr2=0x7fea8419edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8a19c700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84073350 0x7fea8419edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8a19c700 1 -- 172.21.15.3:0/86450332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c009520 con 0x7fea8410ce80 2023-09-10T01:11:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8999b700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea84072a30 0x7fea8419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:42.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.315+0000 7fea8a19c700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8410ce80 0x7fea841988e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00e900 tx=0x7fea7c00ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.316+0000 7fea7affd700 1 -- 172.21.15.3:0/86450332 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c004220 con 0x7fea8410ce80 2023-09-10T01:11:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.316+0000 7fea7affd700 1 -- 172.21.15.3:0/86450332 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea7c004380 con 0x7fea8410ce80 2023-09-10T01:11:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.316+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea80009c40 con 0x7fea8410ce80 2023-09-10T01:11:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.316+0000 7fea7affd700 1 -- 172.21.15.3:0/86450332 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c003cb0 con 0x7fea8410ce80 2023-09-10T01:11:42.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.316+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea84199750 con 0x7fea8410ce80 2023-09-10T01:11:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.319+0000 7fea7affd700 1 -- 172.21.15.3:0/86450332 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fea7c00c7c0 con 0x7fea8410ce80 2023-09-10T01:11:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.320+0000 7fea9f56eb80 1 Processor -- start 2023-09-10T01:11:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea9f56eb80 1 -- start start 2023-09-10T01:11:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea7affd700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fea700a0830 0x7fea700a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea7affd700 1 -- 172.21.15.3:0/86450332 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fea7c014070 con 0x7fea8410ce80 2023-09-10T01:11:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c4570af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4571030 0x5624c457a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c457ae40 0x5624c457d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c44c9730 con 0x5624c4570710 2023-09-10T01:11:42.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c44c7050 con 0x5624c457ae40 2023-09-10T01:11:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c44c2940 con 0x5624c4571030 2023-09-10T01:11:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.321+0000 7fea8919a700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fea700a0830 0x7fea700a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4571030 0x5624c457a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4571030 0x5624c457a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40974/0 (socket says 172.21.15.3:40974) 2023-09-10T01:11:42.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 -- 172.21.15.3:0/1177718611 learned_addr learned my addr 172.21.15.3:0/1177718611 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8a19c700 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c457ae40 0x5624c457d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8999b700 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c4570af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 -- 172.21.15.3:0/1177718611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c457ae40 msgr2=0x5624c457d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c457ae40 0x5624c457d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 -- 172.21.15.3:0/1177718611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 msgr2=0x5624c4570af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c4570af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 -- 172.21.15.3:0/1177718611 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624c42eede0 con 0x5624c4571030 2023-09-10T01:11:42.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8999b700 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c4570af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:11:42.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.322+0000 7fea8919a700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fea700a0830 0x7fea700a2cf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea74013a70 tx=0x7fea740098a0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:42.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.323+0000 7fea8919a700 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4571030 0x5624c457a900 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fea74005900 tx=0x7fea7401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52ac0388eb897c00 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:42.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.323+0000 7fea6ffff700 1 -- 172.21.15.3:0/1177718611 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74009c50 con 0x5624c4571030 2023-09-10T01:11:42.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.323+0000 7fea6ffff700 1 -- 172.21.15.3:0/1177718611 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea740044f0 con 0x5624c4571030 2023-09-10T01:11:42.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.323+0000 7fea6ffff700 1 -- 172.21.15.3:0/1177718611 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74003c50 con 0x5624c4571030 2023-09-10T01:11:42.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1177718611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4571030 msgr2=0x5624c457a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4571030 0x5624c457a900 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fea74005900 tx=0x7fea7401d000 comp rx=0 tx=0).stop 2023-09-10T01:11:42.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1177718611 shutdown_connections 2023-09-10T01:11:42.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c457ae40 0x5624c457d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c4570af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1177718611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4571030 0x5624c457a900 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1177718611 >> 172.21.15.3:0/1177718611 conn(0x5624c44e5090 msgr2=0x5624c456bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:42.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1177718611 shutdown_connections 2023-09-10T01:11:42.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.324+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1177718611 wait complete. 2023-09-10T01:11:42.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 -- start start 2023-09-10T01:11:42.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 Processor -- start 2023-09-10T01:11:42.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4570710 0x5624c457a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4577b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c457ae40 0x5624c45781a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c4460b70 con 0x5624c457ae40 2023-09-10T01:11:42.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c4460870 con 0x5624c4571030 2023-09-10T01:11:42.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.325+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c4344920 con 0x5624c4570710 2023-09-10T01:11:42.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea9f56eb80 10 client.0 _ll_register_callbacks cb 0x5624c43aaa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:42.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4577b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4577b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36072/0 (socket says 172.21.15.3:36072) 2023-09-10T01:11:42.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 -- 172.21.15.3:0/1644613194 learned_addr learned my addr 172.21.15.3:0/1644613194 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:42.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8999b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4570710 0x5624c457a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8a19c700 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c457ae40 0x5624c45781a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 -- 172.21.15.3:0/1644613194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4570710 msgr2=0x5624c457a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4570710 0x5624c457a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 -- 172.21.15.3:0/1644613194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c457ae40 msgr2=0x5624c45781a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c457ae40 0x5624c45781a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 -- 172.21.15.3:0/1644613194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7401b040 con 0x5624c4571030 2023-09-10T01:11:42.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea8919a700 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4577b20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fea740097d0 tx=0x7fea74013760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2841e77d5c828db9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:42.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.326+0000 7fea6d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:42.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea6d7fa700 1 -- 172.21.15.3:0/1644613194 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74005050 con 0x5624c4571030 2023-09-10T01:11:42.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624c441a4f0 con 0x5624c4571030 2023-09-10T01:11:42.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea6d7fa700 1 -- 172.21.15.3:0/1644613194 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fea740051b0 con 0x5624c4571030 2023-09-10T01:11:42.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea6d7fa700 1 -- 172.21.15.3:0/1644613194 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea74005370 con 0x5624c4571030 2023-09-10T01:11:42.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea9f56eb80 10 client.27728 fetch_fsmap learned FSMap version 31 2023-09-10T01:11:42.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5624c42d7c60 con 0x5624c4571030 2023-09-10T01:11:42.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea6d7fa700 1 -- 172.21.15.3:0/1644613194 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5624c441a4f0 con 0x5624c4571030 2023-09-10T01:11:42.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.327+0000 7fea6d7fa700 1 -- 172.21.15.3:0/1644613194 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fea74030020 con 0x5624c4571030 2023-09-10T01:11:42.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea9f56eb80 10 client.27728 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:11:42.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea9f56eb80 10 client.27728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:42.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea9f56eb80 20 client.27728 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:42.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c4592ee0 0x5624c4595300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea9f56eb80 4 client.27728 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:11:42.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624c44d78a0 con 0x5624c4592ee0 2023-09-10T01:11:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea8999b700 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c4592ee0 0x5624c4595300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea6d7fa700 10 client.27728 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.328+0000 7fea8999b700 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c4592ee0 0x5624c4595300 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b38aacee5c96cdcc server_cookie=ffd66e0cae504f52 in_seq=0 out_seq=0 2023-09-10T01:11:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.329+0000 7fea6d7fa700 1 -- 172.21.15.3:0/1644613194 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5624c44d78a0 con 0x5624c4592ee0 2023-09-10T01:11:42.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.329+0000 7fea6d7fa700 10 client.27728 handle_command_reply: tid=0 2023-09-10T01:11:42.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.329+0000 7fea9f56eb80 1 client.27728 shutdown 2023-09-10T01:11:42.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 msgr2=0x5624c4577b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4577b20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fea740097d0 tx=0x7fea74013760 comp rx=0 tx=0).stop 2023-09-10T01:11:42.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 shutdown_connections 2023-09-10T01:11:42.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c4592ee0 0x5624c4595300 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c4570710 0x5624c457a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c457ae40 0x5624c45781a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/1644613194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4577b20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea6d7fa700 0 client.27728 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:42.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 >> 172.21.15.3:0/1644613194 conn(0x5624c44e5090 msgr2=0x5624c456b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:42.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 shutdown_connections 2023-09-10T01:11:42.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 1 -- 172.21.15.3:0/1644613194 wait complete. 2023-09-10T01:11:42.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.330+0000 7fea9f56eb80 20 client.27728 trim_cache size 0 max 16384 2023-09-10T01:11:42.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.349+0000 7fea9f56eb80 1 Processor -- start 2023-09-10T01:11:42.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.349+0000 7fea9f56eb80 1 -- start start 2023-09-10T01:11:42.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4570710 0x5624c458a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4571030 0x5624c4572e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4573380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c4578ab0 con 0x5624c4571030 2023-09-10T01:11:42.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c4579090 con 0x5624c4570710 2023-09-10T01:11:42.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c4571410 con 0x5624c457ae40 2023-09-10T01:11:42.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4570710 0x5624c458a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4570710 0x5624c458a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36084/0 (socket says 172.21.15.3:36084) 2023-09-10T01:11:42.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 -- 172.21.15.3:0/105947038 learned_addr learned my addr 172.21.15.3:0/105947038 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:42.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8919a700 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4571030 0x5624c4572e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8a19c700 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4573380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 -- 172.21.15.3:0/105947038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 msgr2=0x5624c4573380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4573380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 -- 172.21.15.3:0/105947038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4571030 msgr2=0x5624c4572e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4571030 0x5624c4572e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.350+0000 7fea8999b700 1 -- 172.21.15.3:0/105947038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624c4315250 con 0x5624c4570710 2023-09-10T01:11:42.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.351+0000 7fea8999b700 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4570710 0x5624c458a1a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fea8000b6c0 tx=0x7fea800096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69c91a327cc3064 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:42.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.351+0000 7fea78bff700 1 -- 172.21.15.3:0/105947038 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80022070 con 0x5624c4570710 2023-09-10T01:11:42.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.351+0000 7fea78bff700 1 -- 172.21.15.3:0/105947038 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea80009400 con 0x5624c4570710 2023-09-10T01:11:42.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.351+0000 7fea78bff700 1 -- 172.21.15.3:0/105947038 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8000e040 con 0x5624c4570710 2023-09-10T01:11:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 -- 172.21.15.3:0/105947038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4570710 msgr2=0x5624c458a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4570710 0x5624c458a1a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fea8000b6c0 tx=0x7fea800096c0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 -- 172.21.15.3:0/105947038 shutdown_connections 2023-09-10T01:11:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4573380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4570710 0x5624c458a1a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/105947038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4571030 0x5624c4572e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 -- 172.21.15.3:0/105947038 >> 172.21.15.3:0/105947038 conn(0x5624c44e5090 msgr2=0x5624c458e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 -- 172.21.15.3:0/105947038 shutdown_connections 2023-09-10T01:11:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 -- 172.21.15.3:0/105947038 wait complete. 2023-09-10T01:11:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.352+0000 7fea9f56eb80 1 -- start start 2023-09-10T01:11:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 1 Processor -- start 2023-09-10T01:11:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c45887c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4588d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4589240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c458a750 con 0x5624c4570710 2023-09-10T01:11:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c45e6ce0 con 0x5624c4571030 2023-09-10T01:11:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624c45e6e20 con 0x5624c457ae40 2023-09-10T01:11:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea9f56eb80 10 client.0 _ll_register_callbacks cb 0x7fea7c095660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea8919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4588d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea8919a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4588d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36086/0 (socket says 172.21.15.3:36086) 2023-09-10T01:11:42.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea8919a700 1 -- 172.21.15.3:0/536801115 learned_addr learned my addr 172.21.15.3:0/536801115 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:42.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea8999b700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c45887c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.353+0000 7fea8a19c700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4589240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8919a700 1 -- 172.21.15.3:0/536801115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 msgr2=0x5624c4589240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8919a700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4589240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8919a700 1 -- 172.21.15.3:0/536801115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 msgr2=0x5624c45887c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:42.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8919a700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c45887c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8919a700 1 -- 172.21.15.3:0/536801115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8410ffb0 con 0x5624c4571030 2023-09-10T01:11:42.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8a19c700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4589240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:42.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8999b700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c45887c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:42.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea8919a700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4588d00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fea740104b0 tx=0x7fea74010ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b4e536959ad28 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:42.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea6d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:42.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea6d7fa700 1 -- 172.21.15.3:0/536801115 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea74005da0 con 0x5624c4571030 2023-09-10T01:11:42.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea740044f0 con 0x5624c4571030 2023-09-10T01:11:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea6d7fa700 1 -- 172.21.15.3:0/536801115 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fea74005250 con 0x5624c4571030 2023-09-10T01:11:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea6d7fa700 1 -- 172.21.15.3:0/536801115 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea74005410 con 0x5624c4571030 2023-09-10T01:11:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea9f56eb80 10 client.27734 fetch_fsmap learned FSMap version 31 2023-09-10T01:11:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.354+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5624c42ed2b0 con 0x5624c4571030 2023-09-10T01:11:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea6d7fa700 1 -- 172.21.15.3:0/536801115 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea740044f0 con 0x5624c4571030 2023-09-10T01:11:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea6d7fa700 1 -- 172.21.15.3:0/536801115 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fea7400fbc0 con 0x5624c4571030 2023-09-10T01:11:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea9f56eb80 10 client.27734 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:11:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea9f56eb80 10 client.27734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea9f56eb80 20 client.27734 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c45e8200 0x5624c45ea620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:42.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea9f56eb80 4 client.27734 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:11:42.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5624c44bb920 con 0x5624c45e8200 2023-09-10T01:11:42.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.355+0000 7fea8999b700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c45e8200 0x5624c45ea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:42.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea8999b700 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c45e8200 0x5624c45ea620 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28e328698f58583c server_cookie=2f60cba5ba31423c in_seq=0 out_seq=0 2023-09-10T01:11:42.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea6d7fa700 10 client.27734 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:42.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea6d7fa700 1 -- 172.21.15.3:0/536801115 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5624c44bb920 con 0x5624c45e8200 2023-09-10T01:11:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea6d7fa700 10 client.27734 handle_command_reply: tid=0 2023-09-10T01:11:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 client.27734 shutdown 2023-09-10T01:11:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 msgr2=0x5624c4588d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4588d00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fea740104b0 tx=0x7fea74010ce0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 shutdown_connections 2023-09-10T01:11:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5624c45e8200 0x5624c45ea620 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5624c457ae40 0x5624c4589240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5624c4570710 0x5624c45887c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 --2- 172.21.15.3:0/536801115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c4571030 0x5624c4588d00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea6d7fa700 0 client.27734 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 >> 172.21.15.3:0/536801115 conn(0x5624c44e5090 msgr2=0x5624c45793c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 shutdown_connections 2023-09-10T01:11:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.356+0000 7fea9f56eb80 1 -- 172.21.15.3:0/536801115 wait complete. 2023-09-10T01:11:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea9f56eb80 20 client.27734 trim_cache size 0 max 16384 2023-09-10T01:11:42.371 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:11:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:11:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:11:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:11:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:11:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:11:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:11:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:11:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:11:42.373 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:11:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fea700a0830 msgr2=0x7fea700a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fea700a0830 0x7fea700a2cf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea74013a70 tx=0x7fea740098a0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8410ce80 msgr2=0x7fea841988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8410ce80 0x7fea841988e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00e900 tx=0x7fea7c00ec10 comp rx=0 tx=0).stop 2023-09-10T01:11:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 shutdown_connections 2023-09-10T01:11:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fea700a0830 0x7fea700a2cf0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8410ce80 0x7fea841988e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea84072a30 0x7fea8419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 --2- 172.21.15.3:0/86450332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fea84073350 0x7fea8419edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 >> 172.21.15.3:0/86450332 conn(0x7fea840fc170 msgr2=0x7fea8410f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 shutdown_connections 2023-09-10T01:11:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:11:42.357+0000 7fea8bbff700 1 -- 172.21.15.3:0/86450332 wait complete. 2023-09-10T01:11:42.377 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:11:42.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:42 smithi136 ceph-mon[133708]: pgmap v261: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 251 op/s 2023-09-10T01:11:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:42 smithi092 ceph-mon[134067]: pgmap v261: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 251 op/s 2023-09-10T01:11:42.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:42 smithi003 ceph-mon[97654]: pgmap v261: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 251 op/s 2023-09-10T01:11:42.785 INFO:tasks.workunit.client.0.smithi003.stdout: 1 758748 9.00 MB/sec execute 324 sec latency 20.614 ms 2023-09-10T01:11:43.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 760190 9.01 MB/sec execute 325 sec latency 35.176 ms 2023-09-10T01:11:44.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:44 smithi136 ceph-mon[133708]: pgmap v262: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 251 op/s 2023-09-10T01:11:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:44 smithi092 ceph-mon[134067]: pgmap v262: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 251 op/s 2023-09-10T01:11:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:44 smithi003 ceph-mon[97654]: pgmap v262: 129 pgs: 129 active+clean; 227 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 251 op/s 2023-09-10T01:11:44.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 761515 9.00 MB/sec execute 326 sec latency 15.696 ms 2023-09-10T01:11:45.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 763657 9.01 MB/sec execute 327 sec latency 17.685 ms 2023-09-10T01:11:46.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:46 smithi136 ceph-mon[133708]: pgmap v263: 129 pgs: 129 active+clean; 220 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 364 op/s 2023-09-10T01:11:46.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:46 smithi092 ceph-mon[134067]: pgmap v263: 129 pgs: 129 active+clean; 220 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 364 op/s 2023-09-10T01:11:46.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:46 smithi003 ceph-mon[97654]: pgmap v263: 129 pgs: 129 active+clean; 220 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 364 op/s 2023-09-10T01:11:46.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 764862 9.00 MB/sec execute 328 sec latency 18.358 ms 2023-09-10T01:11:47.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 767026 9.01 MB/sec execute 329 sec latency 47.120 ms 2023-09-10T01:11:48.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:48 smithi136 ceph-mon[133708]: pgmap v264: 129 pgs: 129 active+clean; 220 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 288 op/s 2023-09-10T01:11:48.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:48 smithi092 ceph-mon[134067]: pgmap v264: 129 pgs: 129 active+clean; 220 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 288 op/s 2023-09-10T01:11:48.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:48 smithi003 ceph-mon[97654]: pgmap v264: 129 pgs: 129 active+clean; 220 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 288 op/s 2023-09-10T01:11:48.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 767766 8.99 MB/sec execute 330 sec latency 51.474 ms 2023-09-10T01:11:49.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 769850 9.00 MB/sec execute 331 sec latency 18.078 ms 2023-09-10T01:11:50.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:50 smithi136 ceph-mon[133708]: pgmap v265: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 361 op/s 2023-09-10T01:11:50.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:50 smithi092 ceph-mon[134067]: pgmap v265: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 361 op/s 2023-09-10T01:11:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:50 smithi003 ceph-mon[97654]: pgmap v265: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.0 MiB/s wr, 361 op/s 2023-09-10T01:11:50.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 771093 8.99 MB/sec execute 332 sec latency 19.231 ms 2023-09-10T01:11:51.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 772929 8.99 MB/sec execute 333 sec latency 18.338 ms 2023-09-10T01:11:52.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:52 smithi136 ceph-mon[133708]: pgmap v266: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2023-09-10T01:11:52.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:52 smithi092 ceph-mon[134067]: pgmap v266: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2023-09-10T01:11:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:52 smithi003 ceph-mon[97654]: pgmap v266: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2023-09-10T01:11:52.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 774585 8.99 MB/sec execute 334 sec latency 11.746 ms 2023-09-10T01:11:53.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 776140 8.98 MB/sec execute 335 sec latency 36.014 ms 2023-09-10T01:11:54.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:54 smithi136 ceph-mon[133708]: pgmap v267: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2023-09-10T01:11:54.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:54 smithi092 ceph-mon[134067]: pgmap v267: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2023-09-10T01:11:54.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:54 smithi003 ceph-mon[97654]: pgmap v267: 129 pgs: 129 active+clean; 227 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 184 op/s 2023-09-10T01:11:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 -- 172.21.15.3:0/2760848754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c106680 msgr2=0x7f785c106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 --2- 172.21.15.3:0/2760848754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c106680 0x7f785c106a60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f784c0099c0 tx=0x7f784c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 -- 172.21.15.3:0/2760848754 shutdown_connections 2023-09-10T01:11:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 --2- 172.21.15.3:0/2760848754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c100f80 0x7f785c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 --2- 172.21.15.3:0/2760848754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c106680 0x7f785c106a60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 --2- 172.21.15.3:0/2760848754 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f785c1005c0 0x7f785c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 -- 172.21.15.3:0/2760848754 >> 172.21.15.3:0/2760848754 conn(0x7f785c0fc0b0 msgr2=0x7f785c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 -- 172.21.15.3:0/2760848754 shutdown_connections 2023-09-10T01:11:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.586+0000 7f78627ff700 1 -- 172.21.15.3:0/2760848754 wait complete. 2023-09-10T01:11:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.587+0000 7f78627ff700 1 Processor -- start 2023-09-10T01:11:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f78627ff700 1 -- start start 2023-09-10T01:11:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f78627ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c1005c0 0x7f785c196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f78627ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c100f80 0x7f785c196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f78627ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f785c106680 0x7f785c19afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f78627ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c19b5f0 con 0x7f785c1005c0 2023-09-10T01:11:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f78627ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c19b760 con 0x7f785c100f80 2023-09-10T01:11:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f78627ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c19b920 con 0x7f785c106680 2023-09-10T01:11:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f785bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c1005c0 0x7f785c196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f7860d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f785c106680 0x7f785c19afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f785b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c100f80 0x7f785c196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f785bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c1005c0 0x7f785c196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53462/0 (socket says 172.21.15.3:53462) 2023-09-10T01:11:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.588+0000 7f785bfff700 1 -- 172.21.15.3:0/1689458428 learned_addr learned my addr 172.21.15.3:0/1689458428 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f785bfff700 1 -- 172.21.15.3:0/1689458428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f785c106680 msgr2=0x7f785c19afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f785bfff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f785c106680 0x7f785c19afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f785bfff700 1 -- 172.21.15.3:0/1689458428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c100f80 msgr2=0x7f785c196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f785bfff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c100f80 0x7f785c196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f785bfff700 1 -- 172.21.15.3:0/1689458428 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c0095f0 con 0x7f785c1005c0 2023-09-10T01:11:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f785b7fe700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c100f80 0x7f785c196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f7860d9c700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f785c106680 0x7f785c19afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:11:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f785bfff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c1005c0 0x7f785c196610 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f784c00b580 tx=0x7f784c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.589+0000 7f78597fa700 1 -- 172.21.15.3:0/1689458428 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c01d070 con 0x7f785c1005c0 2023-09-10T01:11:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.590+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c19bba0 con 0x7f785c1005c0 2023-09-10T01:11:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.590+0000 7f78597fa700 1 -- 172.21.15.3:0/1689458428 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f784c005050 con 0x7f785c1005c0 2023-09-10T01:11:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.590+0000 7f78597fa700 1 -- 172.21.15.3:0/1689458428 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c004a40 con 0x7f785c1005c0 2023-09-10T01:11:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.590+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c1a3440 con 0x7f785c1005c0 2023-09-10T01:11:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.593+0000 7f78597fa700 1 -- 172.21.15.3:0/1689458428 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f784c0051c0 con 0x7f785c1005c0 2023-09-10T01:11:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.594+0000 7f787607eb80 1 Processor -- start 2023-09-10T01:11:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.594+0000 7f78597fa700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78440a0830 0x7f78440a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f787607eb80 1 -- start start 2023-09-10T01:11:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f78597fa700 1 -- 172.21.15.3:0/1689458428 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f784c0cd0e0 con 0x7f785c1005c0 2023-09-10T01:11:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f7860d9c700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78440a0830 0x7f78440a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f787607eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f787607eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fc5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f787607eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f787607eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12f1e6d0 con 0x55ca12fc5300 2023-09-10T01:11:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f787607eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12f1bff0 con 0x55ca12fc5cc0 2023-09-10T01:11:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f787607eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12f178e0 con 0x55ca12fcb1a0 2023-09-10T01:11:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f785bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f785b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fc5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f785bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:48522/0 (socket says 172.21.15.3:48522) 2023-09-10T01:11:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.595+0000 7f785bfff700 1 -- 172.21.15.3:0/3075790848 learned_addr learned my addr 172.21.15.3:0/3075790848 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f7860d9c700 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f7860d9c700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78440a0830 0x7f78440a2cf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7854003e80 tx=0x7f7854009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f785bfff700 1 -- 172.21.15.3:0/3075790848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 msgr2=0x55ca12fd1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f785bfff700 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f785bfff700 1 -- 172.21.15.3:0/3075790848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 msgr2=0x55ca12fc5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f785bfff700 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fc5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f785bfff700 1 -- 172.21.15.3:0/3075790848 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca12d43d80 con 0x55ca12fcb1a0 2023-09-10T01:11:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f785bfff700 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcb580 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f784c096c40 tx=0x7f784c09a9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f70f10839fd1e011 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f78427fc700 1 -- 172.21.15.3:0/3075790848 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c0a21b0 con 0x55ca12fcb1a0 2023-09-10T01:11:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f78427fc700 1 -- 172.21.15.3:0/3075790848 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f784c098570 con 0x55ca12fcb1a0 2023-09-10T01:11:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.596+0000 7f78427fc700 1 -- 172.21.15.3:0/3075790848 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c099920 con 0x55ca12fcb1a0 2023-09-10T01:11:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 -- 172.21.15.3:0/3075790848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 msgr2=0x55ca12fcb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcb580 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f784c096c40 tx=0x7f784c09a9a0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 -- 172.21.15.3:0/3075790848 shutdown_connections 2023-09-10T01:11:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcb580 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 --2- 172.21.15.3:0/3075790848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fc5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 -- 172.21.15.3:0/3075790848 >> 172.21.15.3:0/3075790848 conn(0x55ca12f3a030 msgr2=0x55ca12fc0560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 -- 172.21.15.3:0/3075790848 shutdown_connections 2023-09-10T01:11:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.597+0000 7f787607eb80 1 -- 172.21.15.3:0/3075790848 wait complete. 2023-09-10T01:11:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 -- start start 2023-09-10T01:11:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 Processor -- start 2023-09-10T01:11:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12eb5b10 con 0x55ca12fc5300 2023-09-10T01:11:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12eb5810 con 0x55ca12fc5cc0 2023-09-10T01:11:54.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.598+0000 7f787607eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12d998c0 con 0x55ca12fcb1a0 2023-09-10T01:11:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f787607eb80 10 client.0 _ll_register_callbacks cb 0x55ca12dffa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55234/0 (socket says 172.21.15.3:55234) 2023-09-10T01:11:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 -- 172.21.15.3:0/2521022405 learned_addr learned my addr 172.21.15.3:0/2521022405 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785bfff700 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f7860d9c700 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 -- 172.21.15.3:0/2521022405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 msgr2=0x55ca12fcee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 -- 172.21.15.3:0/2521022405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 msgr2=0x55ca12fd4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 -- 172.21.15.3:0/2521022405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c1a3440 con 0x55ca12fc5cc0 2023-09-10T01:11:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f785b7fe700 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd5130 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7850012d20 tx=0x7f785000f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74c43466a7895925 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.599+0000 7f783b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:11:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f783b7fe700 1 -- 172.21.15.3:0/2521022405 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850017070 con 0x55ca12fc5cc0 2023-09-10T01:11:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f783b7fe700 1 -- 172.21.15.3:0/2521022405 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f785000fb80 con 0x55ca12fc5cc0 2023-09-10T01:11:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca12e6f490 con 0x55ca12fc5cc0 2023-09-10T01:11:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f783b7fe700 1 -- 172.21.15.3:0/2521022405 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f785000fd40 con 0x55ca12fc5cc0 2023-09-10T01:11:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f787607eb80 10 client.27746 fetch_fsmap learned FSMap version 31 2023-09-10T01:11:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ca12d2cc00 con 0x55ca12fc5cc0 2023-09-10T01:11:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f783b7fe700 1 -- 172.21.15.3:0/2521022405 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ca12e6f490 con 0x55ca12fc5cc0 2023-09-10T01:11:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f783b7fe700 1 -- 172.21.15.3:0/2521022405 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7850003a20 con 0x55ca12fc5cc0 2023-09-10T01:11:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.600+0000 7f787607eb80 10 client.27746 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:11:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f787607eb80 10 client.27746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f787607eb80 20 client.27746 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f787607eb80 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca12fdf180 0x55ca12fe15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f787607eb80 4 client.27746 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:11:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca12f2c840 con 0x55ca12fdf180 2023-09-10T01:11:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f785bfff700 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca12fdf180 0x55ca12fe15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f785bfff700 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca12fdf180 0x55ca12fe15a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d281ba193bc52cd server_cookie=8ed7c333b3442573 in_seq=0 out_seq=0 2023-09-10T01:11:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.601+0000 7f783b7fe700 10 client.27746 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f783b7fe700 1 -- 172.21.15.3:0/2521022405 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ca12f2c840 con 0x55ca12fdf180 2023-09-10T01:11:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f783b7fe700 10 client.27746 handle_command_reply: tid=0 2023-09-10T01:11:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 client.27746 shutdown 2023-09-10T01:11:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 msgr2=0x55ca12fd5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd5130 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7850012d20 tx=0x7f785000f6a0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 shutdown_connections 2023-09-10T01:11:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca12fdf180 0x55ca12fe15a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca12fcee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f787607eb80 1 --2- 172.21.15.3:0/2521022405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd5130 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.602+0000 7f783b7fe700 0 client.27746 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.603+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 >> 172.21.15.3:0/2521022405 conn(0x55ca12f3a030 msgr2=0x55ca12fbf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.603+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 shutdown_connections 2023-09-10T01:11:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.603+0000 7f787607eb80 1 -- 172.21.15.3:0/2521022405 wait complete. 2023-09-10T01:11:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.603+0000 7f787607eb80 20 client.27746 trim_cache size 0 max 16384 2023-09-10T01:11:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.619+0000 7f787607eb80 1 Processor -- start 2023-09-10T01:11:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.619+0000 7f787607eb80 1 -- start start 2023-09-10T01:11:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f787607eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fc5300 0x55ca12fc56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f787607eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5cc0 0x55ca12fbff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f787607eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fcb1a0 0x55ca12fc0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f787607eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12fdc7b0 con 0x55ca12fc5cc0 2023-09-10T01:11:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f787607eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12fbcfc0 con 0x55ca12fcb1a0 2023-09-10T01:11:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f787607eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca12fc9990 con 0x55ca12fc5300 2023-09-10T01:11:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fc5300 0x55ca12fc56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f7860d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fcb1a0 0x55ca12fc0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fc5300 0x55ca12fc56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:48534/0 (socket says 172.21.15.3:48534) 2023-09-10T01:11:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 -- 172.21.15.3:0/2363302212 learned_addr learned my addr 172.21.15.3:0/2363302212 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785b7fe700 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5cc0 0x55ca12fbff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 -- 172.21.15.3:0/2363302212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fcb1a0 msgr2=0x55ca12fc0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fcb1a0 0x55ca12fc0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 -- 172.21.15.3:0/2363302212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5cc0 msgr2=0x55ca12fbff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5cc0 0x55ca12fbff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785bfff700 1 -- 172.21.15.3:0/2363302212 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca12d6a1f0 con 0x55ca12fc5300 2023-09-10T01:11:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.620+0000 7f785b7fe700 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5cc0 0x55ca12fbff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:11:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f785bfff700 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fc5300 0x55ca12fc56e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f784c099ed0 tx=0x7f784c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a490290ad6aef286 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f7842ffd700 1 -- 172.21.15.3:0/2363302212 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c0056a0 con 0x55ca12fc5300 2023-09-10T01:11:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f7842ffd700 1 -- 172.21.15.3:0/2363302212 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f784c0a00c0 con 0x55ca12fc5300 2023-09-10T01:11:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f7842ffd700 1 -- 172.21.15.3:0/2363302212 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c09a690 con 0x55ca12fc5300 2023-09-10T01:11:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f787607eb80 1 -- 172.21.15.3:0/2363302212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fc5300 msgr2=0x55ca12fc56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f787607eb80 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fc5300 0x55ca12fc56e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f784c099ed0 tx=0x7f784c009310 comp rx=0 tx=0).stop 2023-09-10T01:11:54.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f787607eb80 1 -- 172.21.15.3:0/2363302212 shutdown_connections 2023-09-10T01:11:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f787607eb80 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fc5300 0x55ca12fc56e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f787607eb80 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fcb1a0 0x55ca12fc0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f787607eb80 1 --2- 172.21.15.3:0/2363302212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5cc0 0x55ca12fbff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.621+0000 7f787607eb80 1 -- 172.21.15.3:0/2363302212 >> 172.21.15.3:0/2363302212 conn(0x55ca12f3a030 msgr2=0x55ca12fcf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.622+0000 7f787607eb80 1 -- 172.21.15.3:0/2363302212 shutdown_connections 2023-09-10T01:11:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.622+0000 7f787607eb80 1 -- 172.21.15.3:0/2363302212 wait complete. 2023-09-10T01:11:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.622+0000 7f787607eb80 1 -- start start 2023-09-10T01:11:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.622+0000 7f787607eb80 1 Processor -- start 2023-09-10T01:11:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f787607eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f787607eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f787607eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca13031fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f787607eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca130324f0 con 0x55ca12fc5300 2023-09-10T01:11:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f787607eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca13032630 con 0x55ca12fc5cc0 2023-09-10T01:11:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f787607eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca13032770 con 0x55ca12fcb1a0 2023-09-10T01:11:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f787607eb80 10 client.0 _ll_register_callbacks cb 0x7f78500127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:11:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53510/0 (socket says 172.21.15.3:53510) 2023-09-10T01:11:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 -- 172.21.15.3:0/3932652624 learned_addr learned my addr 172.21.15.3:0/3932652624 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:11:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f7860d9c700 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca13031fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785b7fe700 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 -- 172.21.15.3:0/3932652624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 msgr2=0x55ca13031fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca13031fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 -- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 msgr2=0x55ca12fd9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:11:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.623+0000 7f785bfff700 1 -- 172.21.15.3:0/3932652624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca12d43d80 con 0x55ca12fc5300 2023-09-10T01:11:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f785bfff700 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd92f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f784c096c40 tx=0x7f784c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f5c551e907178d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:11:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f783b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:11:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f783b7fe700 1 -- 172.21.15.3:0/3932652624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c0056a0 con 0x55ca12fc5300 2023-09-10T01:11:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f784c098570 con 0x55ca12fc5300 2023-09-10T01:11:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f783b7fe700 1 -- 172.21.15.3:0/3932652624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f784c09a430 con 0x55ca12fc5300 2023-09-10T01:11:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f783b7fe700 1 -- 172.21.15.3:0/3932652624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f784c09a5f0 con 0x55ca12fc5300 2023-09-10T01:11:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f787607eb80 10 client.17667 fetch_fsmap learned FSMap version 31 2023-09-10T01:11:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ca12d42250 con 0x55ca12fc5300 2023-09-10T01:11:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f783b7fe700 1 -- 172.21.15.3:0/3932652624 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f784c098570 con 0x55ca12fc5300 2023-09-10T01:11:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.624+0000 7f783b7fe700 1 -- 172.21.15.3:0/3932652624 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f784c004ba0 con 0x55ca12fc5300 2023-09-10T01:11:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f787607eb80 10 client.17667 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:11:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f787607eb80 10 client.17667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:11:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f787607eb80 20 client.17667 populate_metadata read hostname 'smithi003' 2023-09-10T01:11:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f787607eb80 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca13033a20 0x55ca13035e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:11:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f787607eb80 4 client.17667 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:11:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ca12f108c0 con 0x55ca13033a20 2023-09-10T01:11:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f785b7fe700 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca13033a20 0x55ca13035e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:11:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f785b7fe700 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca13033a20 0x55ca13035e10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b57482f515807f server_cookie=644d396302b1eebf in_seq=0 out_seq=0 2023-09-10T01:11:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.625+0000 7f783b7fe700 10 client.17667 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.735+0000 7f783b7fe700 1 -- 172.21.15.3:0/3932652624 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ca12f108c0 con 0x55ca13033a20 2023-09-10T01:11:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.735+0000 7f783b7fe700 10 client.17667 handle_command_reply: tid=0 2023-09-10T01:11:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.735+0000 7f787607eb80 1 client.17667 shutdown 2023-09-10T01:11:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.735+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 msgr2=0x55ca12fd92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.735+0000 7f787607eb80 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd92f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f784c096c40 tx=0x7f784c005c00 comp rx=0 tx=0).stop 2023-09-10T01:11:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 shutdown_connections 2023-09-10T01:11:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ca13033a20 0x55ca13035e10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ca12fcb1a0 0x55ca13031fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ca12fc5300 0x55ca12fd92f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 --2- 172.21.15.3:0/3932652624 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca12fc5cc0 0x55ca12fd9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 >> 172.21.15.3:0/3932652624 conn(0x55ca12f3a030 msgr2=0x55ca12fcd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f783b7fe700 0 client.17667 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:11:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 shutdown_connections 2023-09-10T01:11:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 1 -- 172.21.15.3:0/3932652624 wait complete. 2023-09-10T01:11:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.736+0000 7f787607eb80 20 client.17667 trim_cache size 0 max 16384 2023-09-10T01:11:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:11:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:11:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:11:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:11:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.737+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78440a0830 msgr2=0x7f78440a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.737+0000 7f78627ff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78440a0830 0x7f78440a2cf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7854003e80 tx=0x7f7854009920 comp rx=0 tx=0).stop 2023-09-10T01:11:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.737+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c1005c0 msgr2=0x7f785c196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:11:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.737+0000 7f78627ff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c1005c0 0x7f785c196610 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f784c00b580 tx=0x7f784c00bfd0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 shutdown_connections 2023-09-10T01:11:54.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f78440a0830 0x7f78440a2cf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f785c106680 0x7f785c19afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f785c1005c0 0x7f785c196610 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 --2- 172.21.15.3:0/1689458428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f785c100f80 0x7f785c196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:11:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 >> 172.21.15.3:0/1689458428 conn(0x7f785c0fc0b0 msgr2=0x7f785c0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:11:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 shutdown_connections 2023-09-10T01:11:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:11:54.742+0000 7f78627ff700 1 -- 172.21.15.3:0/1689458428 wait complete. 2023-09-10T01:11:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 777862 9.00 MB/sec execute 336 sec latency 62.080 ms 2023-09-10T01:11:55.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 779120 8.99 MB/sec execute 337 sec latency 19.236 ms 2023-09-10T01:11:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:56 smithi136 ceph-mon[133708]: pgmap v268: 129 pgs: 129 active+clean; 235 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 259 op/s 2023-09-10T01:11:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:56 smithi092 ceph-mon[134067]: pgmap v268: 129 pgs: 129 active+clean; 235 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 259 op/s 2023-09-10T01:11:56.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:56 smithi003 ceph-mon[97654]: pgmap v268: 129 pgs: 129 active+clean; 235 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 259 op/s 2023-09-10T01:11:56.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 781384 9.00 MB/sec execute 338 sec latency 15.406 ms 2023-09-10T01:11:57.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 782592 8.99 MB/sec execute 339 sec latency 16.470 ms 2023-09-10T01:11:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:11:58 smithi136 ceph-mon[133708]: pgmap v269: 129 pgs: 129 active+clean; 235 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2023-09-10T01:11:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:11:58 smithi092 ceph-mon[134067]: pgmap v269: 129 pgs: 129 active+clean; 235 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2023-09-10T01:11:58.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:11:58 smithi003 ceph-mon[97654]: pgmap v269: 129 pgs: 129 active+clean; 235 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 147 op/s 2023-09-10T01:11:58.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 784803 9.00 MB/sec execute 340 sec latency 59.172 ms 2023-09-10T01:11:59.786 INFO:tasks.workunit.client.0.smithi003.stdout: 1 785671 8.98 MB/sec execute 341 sec latency 68.059 ms 2023-09-10T01:12:00.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:00 smithi136 ceph-mon[133708]: pgmap v270: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 265 op/s 2023-09-10T01:12:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:00 smithi092 ceph-mon[134067]: pgmap v270: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 265 op/s 2023-09-10T01:12:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:00 smithi003 ceph-mon[97654]: pgmap v270: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 265 op/s 2023-09-10T01:12:00.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 787914 8.99 MB/sec execute 342 sec latency 16.190 ms 2023-09-10T01:12:01.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 788209 8.98 MB/sec execute 343 sec latency 72.833 ms 2023-09-10T01:12:02.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:02 smithi136 ceph-mon[133708]: pgmap v271: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2023-09-10T01:12:02.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:02 smithi092 ceph-mon[134067]: pgmap v271: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2023-09-10T01:12:02.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:02 smithi003 ceph-mon[97654]: pgmap v271: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2023-09-10T01:12:02.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 789302 8.96 MB/sec execute 344 sec latency 12.938 ms 2023-09-10T01:12:03.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 791591 8.97 MB/sec execute 345 sec latency 15.295 ms 2023-09-10T01:12:04.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:04 smithi092 ceph-mon[134067]: pgmap v272: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2023-09-10T01:12:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:04 smithi003 ceph-mon[97654]: pgmap v272: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2023-09-10T01:12:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:04 smithi136 ceph-mon[133708]: pgmap v272: 129 pgs: 129 active+clean; 234 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2023-09-10T01:12:04.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 792136 8.96 MB/sec execute 346 sec latency 53.450 ms 2023-09-10T01:12:05.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:05 smithi092 ceph-mon[134067]: osdmap e96: 12 total, 12 up, 12 in 2023-09-10T01:12:05.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:05 smithi003 ceph-mon[97654]: osdmap e96: 12 total, 12 up, 12 in 2023-09-10T01:12:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:05 smithi136 ceph-mon[133708]: osdmap e96: 12 total, 12 up, 12 in 2023-09-10T01:12:05.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 792483 8.93 MB/sec execute 347 sec latency 65.991 ms 2023-09-10T01:12:06.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:06 smithi092 ceph-mon[134067]: pgmap v274: 129 pgs: 129 active+clean; 241 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 255 op/s 2023-09-10T01:12:06.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:06 smithi003 ceph-mon[97654]: pgmap v274: 129 pgs: 129 active+clean; 241 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 255 op/s 2023-09-10T01:12:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:06 smithi136 ceph-mon[133708]: pgmap v274: 129 pgs: 129 active+clean; 241 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 255 op/s 2023-09-10T01:12:06.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 794354 8.93 MB/sec execute 348 sec latency 18.163 ms 2023-09-10T01:12:07.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 795680 8.94 MB/sec execute 349 sec latency 236.922 ms 2023-09-10T01:12:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.910+0000 7f055d5ff700 1 -- 172.21.15.3:0/1236029416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05581005c0 msgr2=0x7f0558100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.910+0000 7f055d5ff700 1 --2- 172.21.15.3:0/1236029416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05581005c0 0x7f0558100a40 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f054c0099c0 tx=0x7f054c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.910+0000 7f055d5ff700 1 -- 172.21.15.3:0/1236029416 shutdown_connections 2023-09-10T01:12:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.910+0000 7f055d5ff700 1 --2- 172.21.15.3:0/1236029416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0558100f80 0x7f055810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.910+0000 7f055d5ff700 1 --2- 172.21.15.3:0/1236029416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0558106680 0x7f0558106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.910+0000 7f055d5ff700 1 --2- 172.21.15.3:0/1236029416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05581005c0 0x7f0558100a40 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.910+0000 7f055d5ff700 1 -- 172.21.15.3:0/1236029416 >> 172.21.15.3:0/1236029416 conn(0x7f05580fc0b0 msgr2=0x7f05580fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.911+0000 7f055d5ff700 1 -- 172.21.15.3:0/1236029416 shutdown_connections 2023-09-10T01:12:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.911+0000 7f055d5ff700 1 -- 172.21.15.3:0/1236029416 wait complete. 2023-09-10T01:12:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 Processor -- start 2023-09-10T01:12:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 -- start start 2023-09-10T01:12:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05581005c0 0x7f055819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0558100f80 0x7f055819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0558106680 0x7f05581986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0558198d60 con 0x7f0558100f80 2023-09-10T01:12:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0558198ed0 con 0x7f0558106680 2023-09-10T01:12:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f055d5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0558199090 con 0x7f05581005c0 2023-09-10T01:12:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.912+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05581005c0 0x7f055819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05581005c0 0x7f055819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42954/0 (socket says 172.21.15.3:42954) 2023-09-10T01:12:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 -- 172.21.15.3:0/3405021791 learned_addr learned my addr 172.21.15.3:0/3405021791 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f05577fe700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0558106680 0x7f05581986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f05567fc700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0558100f80 0x7f055819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 -- 172.21.15.3:0/3405021791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0558106680 msgr2=0x7f05581986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0558106680 0x7f05581986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 -- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0558100f80 msgr2=0x7f055819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0558100f80 0x7f055819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 -- 172.21.15.3:0/3405021791 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0095f0 con 0x7f05581005c0 2023-09-10T01:12:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f0556ffd700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05581005c0 0x7f055819e630 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f054800e9a0 tx=0x7f054800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f053ffff700 1 -- 172.21.15.3:0/3405021791 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548009790 con 0x7f05581005c0 2023-09-10T01:12:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0558199370 con 0x7f05581005c0 2023-09-10T01:12:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f053ffff700 1 -- 172.21.15.3:0/3405021791 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0548010430 con 0x7f05581005c0 2023-09-10T01:12:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.913+0000 7f053ffff700 1 -- 172.21.15.3:0/3405021791 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548018680 con 0x7f05581005c0 2023-09-10T01:12:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.914+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05581a5620 con 0x7f05581005c0 2023-09-10T01:12:07.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.917+0000 7f053ffff700 1 -- 172.21.15.3:0/3405021791 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f05480105a0 con 0x7f05581005c0 2023-09-10T01:12:07.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.918+0000 7f0570ee1b80 1 Processor -- start 2023-09-10T01:12:07.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.918+0000 7f0570ee1b80 1 -- start start 2023-09-10T01:12:07.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.918+0000 7f053ffff700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05400a07b0 0x7f05400a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f053ffff700 1 -- 172.21.15.3:0/3405021791 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f0548014070 con 0x7f05581005c0 2023-09-10T01:12:07.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f05577fe700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05400a07b0 0x7f05400a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227ef580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227e9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9cc0 0x55b5227f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5227426d0 con 0x55b5227e9300 2023-09-10T01:12:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52273fff0 con 0x55b5227e9cc0 2023-09-10T01:12:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52273b8e0 con 0x55b5227ef1a0 2023-09-10T01:12:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f05577fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9cc0 0x55b5227f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f05577fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9cc0 0x55b5227f5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42136/0 (socket says 172.21.15.3:42136) 2023-09-10T01:12:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f05577fe700 1 -- 172.21.15.3:0/3529860315 learned_addr learned my addr 172.21.15.3:0/3529860315 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f05567fc700 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227e9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.919+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227ef580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f05577fe700 1 -- 172.21.15.3:0/3529860315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 msgr2=0x55b5227ef580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f05577fe700 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227ef580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f05577fe700 1 -- 172.21.15.3:0/3529860315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 msgr2=0x55b5227e9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f05577fe700 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227e9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f05577fe700 1 -- 172.21.15.3:0/3529860315 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b522567d80 con 0x55b5227e9cc0 2023-09-10T01:12:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f05577fe700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05400a07b0 0x7f05400a2c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f055000f900 tx=0x7f055000fcc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f05577fe700 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9cc0 0x55b5227f5f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0550018640 tx=0x7f055001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38ea52b9dc11d572 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f053d7fa700 1 -- 172.21.15.3:0/3529860315 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550027070 con 0x55b5227e9cc0 2023-09-10T01:12:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f053d7fa700 1 -- 172.21.15.3:0/3529860315 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f055000f040 con 0x55b5227e9cc0 2023-09-10T01:12:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.920+0000 7f053d7fa700 1 -- 172.21.15.3:0/3529860315 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055000e040 con 0x55b5227e9cc0 2023-09-10T01:12:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 -- 172.21.15.3:0/3529860315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9cc0 msgr2=0x55b5227f5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9cc0 0x55b5227f5f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0550018640 tx=0x7f055001d000 comp rx=0 tx=0).stop 2023-09-10T01:12:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 -- 172.21.15.3:0/3529860315 shutdown_connections 2023-09-10T01:12:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227ef580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9cc0 0x55b5227f5f30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/3529860315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227e9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 -- 172.21.15.3:0/3529860315 >> 172.21.15.3:0/3529860315 conn(0x55b52275e030 msgr2=0x55b5227e4560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 -- 172.21.15.3:0/3529860315 shutdown_connections 2023-09-10T01:12:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.921+0000 7f0570ee1b80 1 -- 172.21.15.3:0/3529860315 wait complete. 2023-09-10T01:12:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.922+0000 7f0570ee1b80 1 -- start start 2023-09-10T01:12:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.922+0000 7f0570ee1b80 1 Processor -- start 2023-09-10T01:12:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.922+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227f3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5226d9b10 con 0x55b5227e9cc0 2023-09-10T01:12:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5226d9810 con 0x55b5227e9300 2023-09-10T01:12:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5225bd8c0 con 0x55b5227ef1a0 2023-09-10T01:12:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0570ee1b80 10 client.0 _ll_register_callbacks cb 0x55b522623a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42144/0 (socket says 172.21.15.3:42144) 2023-09-10T01:12:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 -- 172.21.15.3:0/2855722102 learned_addr learned my addr 172.21.15.3:0/2855722102 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f05567fc700 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f05577fe700 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 -- 172.21.15.3:0/2855722102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 msgr2=0x55b5227f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 -- 172.21.15.3:0/2855722102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 msgr2=0x55b5227f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 -- 172.21.15.3:0/2855722102 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055000fe40 con 0x55b5227e9300 2023-09-10T01:12:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f0556ffd700 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f5e50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f054800e970 tx=0x7f0548099c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7887a742c1cf663f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.923+0000 7f053aff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f053aff5700 1 -- 172.21.15.3:0/2855722102 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05480999b0 con 0x55b5227e9300 2023-09-10T01:12:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f053aff5700 1 -- 172.21.15.3:0/2855722102 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f054809d380 con 0x55b5227e9300 2023-09-10T01:12:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b522693490 con 0x55b5227e9300 2023-09-10T01:12:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f053aff5700 1 -- 172.21.15.3:0/2855722102 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05480a15c0 con 0x55b5227e9300 2023-09-10T01:12:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 10 client.27764 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b522550c00 con 0x55b5227e9300 2023-09-10T01:12:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f053aff5700 1 -- 172.21.15.3:0/2855722102 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b522693490 con 0x55b5227e9300 2023-09-10T01:12:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f053aff5700 1 -- 172.21.15.3:0/2855722102 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f05480a1130 con 0x55b5227e9300 2023-09-10T01:12:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 10 client.27764 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 10 client.27764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 20 client.27764 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b5228033d0 0x55b5228057f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 4 client.27764 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:12:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.924+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b522750840 con 0x55b5228033d0 2023-09-10T01:12:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.925+0000 7f05567fc700 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b5228033d0 0x55b5228057f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.925+0000 7f05567fc700 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b5228033d0 0x55b5228057f0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=535cf6122acbb34f server_cookie=b5e31ef7cc84b0b in_seq=0 out_seq=0 2023-09-10T01:12:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.925+0000 7f053aff5700 10 client.27764 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f053aff5700 1 -- 172.21.15.3:0/2855722102 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b522750840 con 0x55b5228033d0 2023-09-10T01:12:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f053aff5700 10 client.27764 handle_command_reply: tid=0 2023-09-10T01:12:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 client.27764 shutdown 2023-09-10T01:12:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 msgr2=0x55b5227f5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f5e50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f054800e970 tx=0x7f0548099c50 comp rx=0 tx=0).stop 2023-09-10T01:12:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 shutdown_connections 2023-09-10T01:12:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b5228033d0 0x55b5228057f0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f5e50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f053aff5700 0 client.27764 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/2855722102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 >> 172.21.15.3:0/2855722102 conn(0x55b52275e030 msgr2=0x55b5227e37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 shutdown_connections 2023-09-10T01:12:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.926+0000 7f0570ee1b80 1 -- 172.21.15.3:0/2855722102 wait complete. 2023-09-10T01:12:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.927+0000 7f0570ee1b80 20 client.27764 trim_cache size 0 max 16384 2023-09-10T01:12:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.942+0000 7f0570ee1b80 1 Processor -- start 2023-09-10T01:12:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.942+0000 7f0570ee1b80 1 -- start start 2023-09-10T01:12:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227e3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227e4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5227ecdb0 con 0x55b5227e9cc0 2023-09-10T01:12:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5227ea580 con 0x55b5227e9300 2023-09-10T01:12:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5227e5930 con 0x55b5227ef1a0 2023-09-10T01:12:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42152/0 (socket says 172.21.15.3:42152) 2023-09-10T01:12:07.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f0556ffd700 1 -- 172.21.15.3:0/4061970447 learned_addr learned my addr 172.21.15.3:0/4061970447 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f05567fc700 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227e3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.943+0000 7f05577fe700 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f0556ffd700 1 -- 172.21.15.3:0/4061970447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 msgr2=0x55b5227e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f0556ffd700 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227e4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f0556ffd700 1 -- 172.21.15.3:0/4061970447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 msgr2=0x55b5227e3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f0556ffd700 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227e3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f0556ffd700 1 -- 172.21.15.3:0/4061970447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b52258e1f0 con 0x55b5227e9300 2023-09-10T01:12:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f05567fc700 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227e3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:12:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f0556ffd700 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f59f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0548099490 tx=0x7f0548010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce70fe3bfec59597 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f053dffb700 1 -- 172.21.15.3:0/4061970447 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05480a1330 con 0x55b5227e9300 2023-09-10T01:12:07.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f053dffb700 1 -- 172.21.15.3:0/4061970447 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05480a15c0 con 0x55b5227e9300 2023-09-10T01:12:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.944+0000 7f053dffb700 1 -- 172.21.15.3:0/4061970447 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05480997b0 con 0x55b5227e9300 2023-09-10T01:12:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 -- 172.21.15.3:0/4061970447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 msgr2=0x55b5227f59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:07.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f59f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0548099490 tx=0x7f0548010c80 comp rx=0 tx=0).stop 2023-09-10T01:12:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 -- 172.21.15.3:0/4061970447 shutdown_connections 2023-09-10T01:12:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227ef1a0 0x55b5227e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227e9300 0x55b5227f59f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/4061970447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9cc0 0x55b5227e3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 -- 172.21.15.3:0/4061970447 >> 172.21.15.3:0/4061970447 conn(0x55b52275e030 msgr2=0x55b5227e8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 -- 172.21.15.3:0/4061970447 shutdown_connections 2023-09-10T01:12:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.945+0000 7f0570ee1b80 1 -- 172.21.15.3:0/4061970447 wait complete. 2023-09-10T01:12:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.946+0000 7f0570ee1b80 1 -- start start 2023-09-10T01:12:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.946+0000 7f0570ee1b80 1 Processor -- start 2023-09-10T01:12:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.946+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227f8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227e9cc0 0x55b5227f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0570ee1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227ef1a0 0x55b522856200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5227f9250 con 0x55b5227e9300 2023-09-10T01:12:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b522856740 con 0x55b5227ef1a0 2023-09-10T01:12:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0570ee1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b522856880 con 0x55b5227e9cc0 2023-09-10T01:12:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0570ee1b80 10 client.0 _ll_register_callbacks cb 0x7f05480caf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227ef1a0 0x55b522856200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05567fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227e9cc0 0x55b5227f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227ef1a0 0x55b522856200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42168/0 (socket says 172.21.15.3:42168) 2023-09-10T01:12:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05567fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227e9cc0 0x55b5227f8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42982/0 (socket says 172.21.15.3:42982) 2023-09-10T01:12:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f0556ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227f8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43110/0 (socket says 172.21.15.3:43110) 2023-09-10T01:12:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 -- 172.21.15.3:0/708061549 learned_addr learned my addr 172.21.15.3:0/708061549 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:07.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 -- 172.21.15.3:0/708061549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227e9cc0 msgr2=0x55b5227f8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227e9cc0 0x55b5227f8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 -- 172.21.15.3:0/708061549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 msgr2=0x55b5227f8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227f8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05577fe700 1 -- 172.21.15.3:0/708061549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055810dd40 con 0x55b5227ef1a0 2023-09-10T01:12:07.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.947+0000 7f05567fc700 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227e9cc0 0x55b5227f8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:12:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f05577fe700 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227ef1a0 0x55b522856200 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0550026000 tx=0x7f0550009e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7555c8bad0c344f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f053aff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f053aff5700 1 -- 172.21.15.3:0/708061549 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055000e050 con 0x55b5227ef1a0 2023-09-10T01:12:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f055000f040 con 0x55b5227ef1a0 2023-09-10T01:12:07.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f053aff5700 1 -- 172.21.15.3:0/708061549 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f0550008710 con 0x55b5227ef1a0 2023-09-10T01:12:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f053aff5700 1 -- 172.21.15.3:0/708061549 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0550027040 con 0x55b5227ef1a0 2023-09-10T01:12:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f0570ee1b80 10 client.27770 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b522550c00 con 0x55b5227ef1a0 2023-09-10T01:12:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.948+0000 7f053aff5700 1 -- 172.21.15.3:0/708061549 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f055000f040 con 0x55b5227ef1a0 2023-09-10T01:12:07.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f053aff5700 1 -- 172.21.15.3:0/708061549 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0550022070 con 0x55b5227ef1a0 2023-09-10T01:12:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0570ee1b80 10 client.27770 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0570ee1b80 10 client.27770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0570ee1b80 20 client.27770 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b522858060 0x55b52285a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:07.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0570ee1b80 4 client.27770 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:12:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b5227348c0 con 0x55b522858060 2023-09-10T01:12:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0556ffd700 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b522858060 0x55b52285a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f0556ffd700 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b522858060 0x55b52285a480 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f56de920303bcd6 server_cookie=278c89e8593240ed in_seq=0 out_seq=0 2023-09-10T01:12:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:07.949+0000 7f053aff5700 10 client.27770 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.060+0000 7f053aff5700 1 -- 172.21.15.3:0/708061549 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b5227348c0 con 0x55b522858060 2023-09-10T01:12:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.060+0000 7f053aff5700 10 client.27770 handle_command_reply: tid=0 2023-09-10T01:12:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.061+0000 7f0570ee1b80 1 client.27770 shutdown 2023-09-10T01:12:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227ef1a0 msgr2=0x55b522856200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227ef1a0 0x55b522856200 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0550026000 tx=0x7f0550009e50 comp rx=0 tx=0).stop 2023-09-10T01:12:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 shutdown_connections 2023-09-10T01:12:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b522858060 0x55b52285a480 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5227ef1a0 0x55b522856200 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b5227e9300 0x55b5227f8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 --2- 172.21.15.3:0/708061549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5227e9cc0 0x55b5227f8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f053aff5700 0 client.27770 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 >> 172.21.15.3:0/708061549 conn(0x55b52275e030 msgr2=0x55b5227febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 shutdown_connections 2023-09-10T01:12:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.064+0000 7f0570ee1b80 1 -- 172.21.15.3:0/708061549 wait complete. 2023-09-10T01:12:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:12:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:12:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:12:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:12:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.065+0000 7f0570ee1b80 20 client.27770 trim_cache size 0 max 16384 2023-09-10T01:12:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.065+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05400a07b0 msgr2=0x7f05400a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.065+0000 7f055d5ff700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05400a07b0 0x7f05400a2c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f055000f900 tx=0x7f055000fcc0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.065+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05581005c0 msgr2=0x7f055819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.065+0000 7f055d5ff700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05581005c0 0x7f055819e630 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f054800e9a0 tx=0x7f054800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 shutdown_connections 2023-09-10T01:12:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05400a07b0 0x7f05400a2c70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05581005c0 0x7f055819e630 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0558106680 0x7f05581986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 --2- 172.21.15.3:0/3405021791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0558100f80 0x7f055819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 >> 172.21.15.3:0/3405021791 conn(0x7f05580fc0b0 msgr2=0x7f0558072980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 shutdown_connections 2023-09-10T01:12:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:08.066+0000 7f055d5ff700 1 -- 172.21.15.3:0/3405021791 wait complete. 2023-09-10T01:12:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:08 smithi092 ceph-mon[134067]: pgmap v275: 129 pgs: 129 active+clean; 241 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 255 op/s 2023-09-10T01:12:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:08 smithi003 ceph-mon[97654]: pgmap v275: 129 pgs: 129 active+clean; 241 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 255 op/s 2023-09-10T01:12:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:08 smithi136 ceph-mon[133708]: pgmap v275: 129 pgs: 129 active+clean; 241 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 255 op/s 2023-09-10T01:12:08.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 796814 8.93 MB/sec execute 350 sec latency 47.913 ms 2023-09-10T01:12:09.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 799003 8.94 MB/sec execute 351 sec latency 35.049 ms 2023-09-10T01:12:10.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:10 smithi092 ceph-mon[134067]: pgmap v276: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 228 op/s 2023-09-10T01:12:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:10 smithi092 ceph-mon[134067]: osdmap e97: 12 total, 12 up, 12 in 2023-09-10T01:12:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:10 smithi003 ceph-mon[97654]: pgmap v276: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 228 op/s 2023-09-10T01:12:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:10 smithi003 ceph-mon[97654]: osdmap e97: 12 total, 12 up, 12 in 2023-09-10T01:12:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:10 smithi136 ceph-mon[133708]: pgmap v276: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 228 op/s 2023-09-10T01:12:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:10 smithi136 ceph-mon[133708]: osdmap e97: 12 total, 12 up, 12 in 2023-09-10T01:12:10.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 799900 8.92 MB/sec execute 352 sec latency 36.176 ms 2023-09-10T01:12:11.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 800733 8.91 MB/sec execute 353 sec latency 237.871 ms 2023-09-10T01:12:12.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:12:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:12 smithi003 ceph-mon[97654]: pgmap v278: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 285 op/s 2023-09-10T01:12:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:12 smithi136 ceph-mon[133708]: pgmap v278: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 285 op/s 2023-09-10T01:12:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.689+0000 7f5781dff700 1 -- 172.21.15.3:0/3551702647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577c107870 msgr2=0x7f577c107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.689+0000 7f5781dff700 1 --2- 172.21.15.3:0/3551702647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577c107870 0x7f577c107c50 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f577400b0a0 tx=0x7f577400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.690+0000 7f5781dff700 1 -- 172.21.15.3:0/3551702647 shutdown_connections 2023-09-10T01:12:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.690+0000 7f5781dff700 1 --2- 172.21.15.3:0/3551702647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f577c10c220 0x7f577c10c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.690+0000 7f5781dff700 1 --2- 172.21.15.3:0/3551702647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577c107870 0x7f577c107c50 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.690+0000 7f5781dff700 1 --2- 172.21.15.3:0/3551702647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c108190 0x7f577c10bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.690+0000 7f5781dff700 1 -- 172.21.15.3:0/3551702647 >> 172.21.15.3:0/3551702647 conn(0x7f577c072090 msgr2=0x7f577c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.692+0000 7f5781dff700 1 -- 172.21.15.3:0/3551702647 shutdown_connections 2023-09-10T01:12:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.692+0000 7f5781dff700 1 -- 172.21.15.3:0/3551702647 wait complete. 2023-09-10T01:12:12.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.693+0000 7f5781dff700 1 Processor -- start 2023-09-10T01:12:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.694+0000 7f5781dff700 1 -- start start 2023-09-10T01:12:12.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.694+0000 7f5781dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577c107870 0x7f577c1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.694+0000 7f5781dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f577c108190 0x7f577c1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.694+0000 7f5781dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c10c220 0x7f577c1a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.694+0000 7f5781dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c1a6120 con 0x7f577c10c220 2023-09-10T01:12:12.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.694+0000 7f5781dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c1a6290 con 0x7f577c107870 2023-09-10T01:12:12.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.694+0000 7f5781dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c1a6450 con 0x7f577c108190 2023-09-10T01:12:12.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.695+0000 7f577bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c10c220 0x7f577c1a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.695+0000 7f577bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c10c220 0x7f577c1a5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43122/0 (socket says 172.21.15.3:43122) 2023-09-10T01:12:12.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.695+0000 7f577bfff700 1 -- 172.21.15.3:0/2993067005 learned_addr learned my addr 172.21.15.3:0/2993067005 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:12.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.696+0000 7f577affd700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f577c108190 0x7f577c1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.697+0000 7f577bfff700 1 -- 172.21.15.3:0/2993067005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f577c108190 msgr2=0x7f577c1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.697+0000 7f577bfff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f577c108190 0x7f577c1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.697+0000 7f577bfff700 1 -- 172.21.15.3:0/2993067005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577c107870 msgr2=0x7f577c1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.697+0000 7f577bfff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577c107870 0x7f577c1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.697+0000 7f577bfff700 1 -- 172.21.15.3:0/2993067005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5774009d00 con 0x7f577c10c220 2023-09-10T01:12:12.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.697+0000 7f577bfff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c10c220 0x7f577c1a5ae0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f576c00d980 tx=0x7f576c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:12.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.698+0000 7f5778ff9700 1 -- 172.21.15.3:0/2993067005 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c009940 con 0x7f577c10c220 2023-09-10T01:12:12.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.698+0000 7f5778ff9700 1 -- 172.21.15.3:0/2993067005 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f576c004830 con 0x7f577c10c220 2023-09-10T01:12:12.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.698+0000 7f5778ff9700 1 -- 172.21.15.3:0/2993067005 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c0054e0 con 0x7f577c10c220 2023-09-10T01:12:12.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.698+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c1a6730 con 0x7f577c10c220 2023-09-10T01:12:12.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.699+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c1adf70 con 0x7f577c10c220 2023-09-10T01:12:12.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.706+0000 7f57956f8b80 1 Processor -- start 2023-09-10T01:12:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.707+0000 7f5778ff9700 1 -- 172.21.15.3:0/2993067005 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f576c004ca0 con 0x7f577c10c220 2023-09-10T01:12:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.708+0000 7f57956f8b80 1 -- start start 2023-09-10T01:12:12.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.708+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 0x560cf2d3f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.708+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d4a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.708+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d4a660 0x560cf2d4ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.708+0000 7f57956f8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2c90730 con 0x560cf2d3ef10 2023-09-10T01:12:12.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.708+0000 7f57956f8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2c8e050 con 0x560cf2d4a660 2023-09-10T01:12:12.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.708+0000 7f57956f8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2c89940 con 0x560cf2d3f830 2023-09-10T01:12:12.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.709+0000 7f577affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d4a660 0x560cf2d4ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.709+0000 7f577affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d4a660 0x560cf2d4ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42190/0 (socket says 172.21.15.3:42190) 2023-09-10T01:12:12.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.709+0000 7f577affd700 1 -- 172.21.15.3:0/3849338007 learned_addr learned my addr 172.21.15.3:0/3849338007 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:12.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.709+0000 7f577affd700 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 0x560cf2d3f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f577affd700 1 -- 172.21.15.3:0/3849338007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 msgr2=0x560cf2d4a120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:12:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f577affd700 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d4a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f577affd700 1 -- 172.21.15.3:0/3849338007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 msgr2=0x560cf2d3f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f577affd700 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 0x560cf2d3f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f577affd700 1 -- 172.21.15.3:0/3849338007 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cf2ab5de0 con 0x560cf2d4a660 2023-09-10T01:12:12.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f577affd700 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d4a660 0x560cf2d4ca50 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f577000aed0 tx=0x7f577000a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b5a6769cb6587 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:12.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f5761ffb700 1 -- 172.21.15.3:0/3849338007 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770014a90 con 0x560cf2d4a660 2023-09-10T01:12:12.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f5761ffb700 1 -- 172.21.15.3:0/3849338007 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f577000c070 con 0x560cf2d4a660 2023-09-10T01:12:12.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.710+0000 7f5761ffb700 1 -- 172.21.15.3:0/3849338007 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5770012370 con 0x560cf2d4a660 2023-09-10T01:12:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.711+0000 7f5778ff9700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57640a0870 0x7f57640a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.711+0000 7f5778ff9700 1 -- 172.21.15.3:0/2993067005 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f576c0cbfd0 con 0x7f577c10c220 2023-09-10T01:12:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.711+0000 7f577b7fe700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57640a0870 0x7f57640a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.712+0000 7f57956f8b80 1 -- 172.21.15.3:0/3849338007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d4a660 msgr2=0x560cf2d4ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.712+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d4a660 0x560cf2d4ca50 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f577000aed0 tx=0x7f577000a360 comp rx=0 tx=0).stop 2023-09-10T01:12:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.712+0000 7f57956f8b80 1 -- 172.21.15.3:0/3849338007 shutdown_connections 2023-09-10T01:12:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.712+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d4a660 0x560cf2d4ca50 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.712+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 0x560cf2d3f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.712+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3849338007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d4a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.712+0000 7f57956f8b80 1 -- 172.21.15.3:0/3849338007 >> 172.21.15.3:0/3849338007 conn(0x560cf2cac090 msgr2=0x560cf2d32a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.713+0000 7f57956f8b80 1 -- 172.21.15.3:0/3849338007 shutdown_connections 2023-09-10T01:12:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.713+0000 7f57956f8b80 1 -- 172.21.15.3:0/3849338007 wait complete. 2023-09-10T01:12:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.714+0000 7f577b7fe700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57640a0870 0x7f57640a2d30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f577400b830 tx=0x7f5774007df0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:12.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:12 smithi092 ceph-mon[134067]: pgmap v278: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 285 op/s 2023-09-10T01:12:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.714+0000 7f57956f8b80 1 -- start start 2023-09-10T01:12:12.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.715+0000 7f57956f8b80 1 Processor -- start 2023-09-10T01:12:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.715+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 0x560cf2d47420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.715+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2d47960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d4a660 0x560cf2d47ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f57956f8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2c27b70 con 0x560cf2d3ef10 2023-09-10T01:12:12.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f57956f8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2c27870 con 0x560cf2d3f830 2023-09-10T01:12:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f57956f8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2b0b920 con 0x560cf2d4a660 2023-09-10T01:12:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f57956f8b80 10 client.0 _ll_register_callbacks cb 0x560cf2b71a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f577bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d4a660 0x560cf2d47ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f577bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d4a660 0x560cf2d47ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43010/0 (socket says 172.21.15.3:43010) 2023-09-10T01:12:12.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f577bfff700 1 -- 172.21.15.3:0/4234793724 learned_addr learned my addr 172.21.15.3:0/4234793724 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:12.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f577affd700 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2d47960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.716+0000 7f577bfff700 1 -- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 msgr2=0x560cf2d47960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:12.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f577bfff700 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2d47960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f577bfff700 1 -- 172.21.15.3:0/4234793724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 msgr2=0x560cf2d47420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:12.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f577bfff700 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 0x560cf2d47420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f577bfff700 1 -- 172.21.15.3:0/4234793724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577000e040 con 0x560cf2d4a660 2023-09-10T01:12:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f577bfff700 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d4a660 0x560cf2d47ea0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f576c095aa0 tx=0x7f576c09cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bb5b4595dd142a1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f575affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:12:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f575affd700 1 -- 172.21.15.3:0/4234793724 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c0b0070 con 0x560cf2d4a660 2023-09-10T01:12:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f575affd700 1 -- 172.21.15.3:0/4234793724 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f576c095740 con 0x560cf2d4a660 2023-09-10T01:12:12.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.717+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560cf2be14f0 con 0x560cf2d4a660 2023-09-10T01:12:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.718+0000 7f575affd700 1 -- 172.21.15.3:0/4234793724 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f576c005760 con 0x560cf2d4a660 2023-09-10T01:12:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.718+0000 7f57956f8b80 10 client.27685 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.718+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560cf2a9ec60 con 0x560cf2d4a660 2023-09-10T01:12:12.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.718+0000 7f575affd700 1 -- 172.21.15.3:0/4234793724 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560cf2be14f0 con 0x560cf2d4a660 2023-09-10T01:12:12.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.719+0000 7f575affd700 1 -- 172.21.15.3:0/4234793724 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f576c099290 con 0x560cf2d4a660 2023-09-10T01:12:12.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.720+0000 7f57956f8b80 10 client.27685 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:12.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.720+0000 7f57956f8b80 10 client.27685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:12.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.720+0000 7f57956f8b80 20 client.27685 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.720+0000 7f57956f8b80 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2d5a100 0x560cf2d5c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.720+0000 7f57956f8b80 4 client.27685 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:12:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.720+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560cf2c9e8a0 con 0x560cf2d5a100 2023-09-10T01:12:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.720+0000 7f577b7fe700 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2d5a100 0x560cf2d5c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.722+0000 7f577b7fe700 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2d5a100 0x560cf2d5c4f0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956ef709fa7febb7 server_cookie=c63a9a3ec9e1ed63 in_seq=0 out_seq=0 2023-09-10T01:12:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.722+0000 7f575affd700 10 client.27685 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.723+0000 7f575affd700 1 -- 172.21.15.3:0/4234793724 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560cf2c9e8a0 con 0x560cf2d5a100 2023-09-10T01:12:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.723+0000 7f575affd700 10 client.27685 handle_command_reply: tid=0 2023-09-10T01:12:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.723+0000 7f57956f8b80 1 client.27685 shutdown 2023-09-10T01:12:12.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d4a660 msgr2=0x560cf2d47ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d4a660 0x560cf2d47ea0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f576c095aa0 tx=0x7f576c09cf10 comp rx=0 tx=0).stop 2023-09-10T01:12:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 shutdown_connections 2023-09-10T01:12:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2d5a100 0x560cf2d5c4f0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d4a660 0x560cf2d47ea0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d3ef10 0x560cf2d47420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 --2- 172.21.15.3:0/4234793724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2d47960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 >> 172.21.15.3:0/4234793724 conn(0x560cf2cac090 msgr2=0x560cf2d32680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f575affd700 0 client.27685 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 shutdown_connections 2023-09-10T01:12:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 1 -- 172.21.15.3:0/4234793724 wait complete. 2023-09-10T01:12:12.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.724+0000 7f57956f8b80 20 client.27685 trim_cache size 0 max 16384 2023-09-10T01:12:12.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.742+0000 7f57956f8b80 1 Processor -- start 2023-09-10T01:12:12.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.743+0000 7f57956f8b80 1 -- start start 2023-09-10T01:12:12.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.743+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3ef10 0x560cf2d511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.743+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d42780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.743+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2d42cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.743+0000 7f57956f8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2d59f70 con 0x560cf2d4a660 2023-09-10T01:12:12.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.744+0000 7f57956f8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2d57890 con 0x560cf2d3ef10 2023-09-10T01:12:12.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.744+0000 7f57956f8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2d51730 con 0x560cf2d3f830 2023-09-10T01:12:12.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.744+0000 7f577affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d42780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.744+0000 7f577affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d42780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43012/0 (socket says 172.21.15.3:43012) 2023-09-10T01:12:12.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.744+0000 7f577affd700 1 -- 172.21.15.3:0/2497178729 learned_addr learned my addr 172.21.15.3:0/2497178729 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:12.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.745+0000 7f577bfff700 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2d42cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.745+0000 7f577b7fe700 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3ef10 0x560cf2d511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.745+0000 7f577bfff700 1 -- 172.21.15.3:0/2497178729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 msgr2=0x560cf2d42780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:12.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.745+0000 7f577bfff700 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d42780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.745+0000 7f577bfff700 1 -- 172.21.15.3:0/2497178729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3ef10 msgr2=0x560cf2d511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:12.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.745+0000 7f577bfff700 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3ef10 0x560cf2d511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.745+0000 7f577bfff700 1 -- 172.21.15.3:0/2497178729 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cf2adc250 con 0x560cf2d4a660 2023-09-10T01:12:12.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.746+0000 7f577bfff700 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2d42cc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f576c098fc0 tx=0x7f576c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87df38700069f74a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:12.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.746+0000 7f57627fc700 1 -- 172.21.15.3:0/2497178729 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c0b0040 con 0x560cf2d4a660 2023-09-10T01:12:12.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.746+0000 7f57627fc700 1 -- 172.21.15.3:0/2497178729 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f576c005760 con 0x560cf2d4a660 2023-09-10T01:12:12.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.746+0000 7f57627fc700 1 -- 172.21.15.3:0/2497178729 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c0a0c10 con 0x560cf2d4a660 2023-09-10T01:12:12.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.747+0000 7f57956f8b80 1 -- 172.21.15.3:0/2497178729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 msgr2=0x560cf2d42cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.747+0000 7f57956f8b80 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2d42cc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f576c098fc0 tx=0x7f576c004750 comp rx=0 tx=0).stop 2023-09-10T01:12:12.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.748+0000 7f57956f8b80 1 -- 172.21.15.3:0/2497178729 shutdown_connections 2023-09-10T01:12:12.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.748+0000 7f57956f8b80 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2d42cc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.748+0000 7f57956f8b80 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3ef10 0x560cf2d511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.748+0000 7f57956f8b80 1 --2- 172.21.15.3:0/2497178729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3f830 0x560cf2d42780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.748+0000 7f57956f8b80 1 -- 172.21.15.3:0/2497178729 >> 172.21.15.3:0/2497178729 conn(0x560cf2cac090 msgr2=0x560cf2d47c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:12.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.748+0000 7f57956f8b80 1 -- 172.21.15.3:0/2497178729 shutdown_connections 2023-09-10T01:12:12.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.748+0000 7f57956f8b80 1 -- 172.21.15.3:0/2497178729 wait complete. 2023-09-10T01:12:12.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.749+0000 7f57956f8b80 1 -- start start 2023-09-10T01:12:12.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 1 Processor -- start 2023-09-10T01:12:12.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3ef10 0x560cf2da74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2da7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2da1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2da24a0 con 0x560cf2d4a660 2023-09-10T01:12:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2da2610 con 0x560cf2d3f830 2023-09-10T01:12:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf2da27d0 con 0x560cf2d3ef10 2023-09-10T01:12:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.750+0000 7f57956f8b80 10 client.0 _ll_register_callbacks cb 0x7f576c099750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:12.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.751+0000 7f577affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2da7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.751+0000 7f577affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2da7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:42214/0 (socket says 172.21.15.3:42214) 2023-09-10T01:12:12.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.751+0000 7f577affd700 1 -- 172.21.15.3:0/3386922218 learned_addr learned my addr 172.21.15.3:0/3386922218 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:12.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.751+0000 7f577bfff700 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2da1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.752+0000 7f577affd700 1 -- 172.21.15.3:0/3386922218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3ef10 msgr2=0x560cf2da74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:12.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.752+0000 7f577affd700 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3ef10 0x560cf2da74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.752+0000 7f577affd700 1 -- 172.21.15.3:0/3386922218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 msgr2=0x560cf2da1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:12.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.752+0000 7f577affd700 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2da1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.752+0000 7f577affd700 1 -- 172.21.15.3:0/3386922218 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cf2a9ec60 con 0x560cf2d3f830 2023-09-10T01:12:12.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.752+0000 7f577affd700 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2da7a00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5770006b00 tx=0x7f5770014f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea401a5c04087503 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:12.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.752+0000 7f575affd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:12.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.753+0000 7f575affd700 1 -- 172.21.15.3:0/3386922218 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577000a690 con 0x560cf2d3f830 2023-09-10T01:12:12.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.753+0000 7f575affd700 1 -- 172.21.15.3:0/3386922218 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f5770000e40 con 0x560cf2d3f830 2023-09-10T01:12:12.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.753+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560cf2c82920 con 0x560cf2d3f830 2023-09-10T01:12:12.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.754+0000 7f575affd700 1 -- 172.21.15.3:0/3386922218 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f577000a0d0 con 0x560cf2d3f830 2023-09-10T01:12:12.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.754+0000 7f57956f8b80 10 client.27785 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:12.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.754+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560cf2ab42b0 con 0x560cf2d3f830 2023-09-10T01:12:12.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.754+0000 7f575affd700 1 -- 172.21.15.3:0/3386922218 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560cf2c82920 con 0x560cf2d3f830 2023-09-10T01:12:12.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f575affd700 1 -- 172.21.15.3:0/3386922218 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5770028020 con 0x560cf2d3f830 2023-09-10T01:12:12.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f57956f8b80 10 client.27785 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:12.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f57956f8b80 10 client.27785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:12.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f57956f8b80 20 client.27785 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:12.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2daf170 0x560cf2db1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:12.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f57956f8b80 4 client.27785 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:12:12.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560cf2c430b0 con 0x560cf2daf170 2023-09-10T01:12:12.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.756+0000 7f577b7fe700 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2daf170 0x560cf2db1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:12.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.757+0000 7f577b7fe700 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2daf170 0x560cf2db1590 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=149ca42b9d4a0dde server_cookie=d35b045c2b8dbfe1 in_seq=0 out_seq=0 2023-09-10T01:12:12.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.757+0000 7f575affd700 10 client.27785 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:12.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.757+0000 7f575affd700 1 -- 172.21.15.3:0/3386922218 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x560cf2c430b0 con 0x560cf2daf170 2023-09-10T01:12:12.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.757+0000 7f575affd700 10 client.27785 handle_command_reply: tid=0 2023-09-10T01:12:12.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.757+0000 7f57956f8b80 1 client.27785 shutdown 2023-09-10T01:12:12.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 msgr2=0x560cf2da7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2da7a00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5770006b00 tx=0x7f5770014f00 comp rx=0 tx=0).stop 2023-09-10T01:12:12.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 shutdown_connections 2023-09-10T01:12:12.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560cf2daf170 0x560cf2db1590 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560cf2d3ef10 0x560cf2da74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf2d4a660 0x560cf2da1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 --2- 172.21.15.3:0/3386922218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cf2d3f830 0x560cf2da7a00 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 >> 172.21.15.3:0/3386922218 conn(0x560cf2cac090 msgr2=0x560cf2d42780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:12.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f575affd700 0 client.27785 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:12.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 shutdown_connections 2023-09-10T01:12:12.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 1 -- 172.21.15.3:0/3386922218 wait complete. 2023-09-10T01:12:12.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.758+0000 7f57956f8b80 20 client.27785 trim_cache size 0 max 16384 2023-09-10T01:12:12.766 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:12:12.766 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:12:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:12:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:12:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:12:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:12:12.767 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:12:12.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:12:12.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:12:12.768 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:12:12.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57640a0870 msgr2=0x7f57640a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57640a0870 0x7f57640a2d30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f577400b830 tx=0x7f5774007df0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c10c220 msgr2=0x7f577c1a5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:12.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c10c220 0x7f577c1a5ae0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f576c00d980 tx=0x7f576c00dd40 comp rx=0 tx=0).stop 2023-09-10T01:12:12.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 shutdown_connections 2023-09-10T01:12:12.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57640a0870 0x7f57640a2d30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f577c10c220 0x7f577c1a5ae0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f577c107870 0x7f577c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 --2- 172.21.15.3:0/2993067005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f577c108190 0x7f577c1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:12.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 >> 172.21.15.3:0/2993067005 conn(0x7f577c072090 msgr2=0x7f577c116b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:12.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 shutdown_connections 2023-09-10T01:12:12.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:12.759+0000 7f5781dff700 1 -- 172.21.15.3:0/2993067005 wait complete. 2023-09-10T01:12:12.779 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:12:12.787 INFO:tasks.workunit.client.0.smithi003.stdout: 1 802202 8.91 MB/sec execute 354 sec latency 63.521 ms 2023-09-10T01:12:13.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 803201 8.89 MB/sec execute 355 sec latency 64.430 ms 2023-09-10T01:12:14.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:14 smithi003 ceph-mon[97654]: pgmap v279: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2023-09-10T01:12:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:14 smithi136 ceph-mon[133708]: pgmap v279: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2023-09-10T01:12:14.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:14 smithi092 ceph-mon[134067]: pgmap v279: 129 pgs: 129 active+clean; 235 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2023-09-10T01:12:14.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 805309 8.90 MB/sec execute 356 sec latency 18.317 ms 2023-09-10T01:12:15.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 806556 8.90 MB/sec execute 357 sec latency 25.443 ms 2023-09-10T01:12:16.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:16 smithi003 ceph-mon[97654]: pgmap v280: 129 pgs: 129 active+clean; 238 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2023-09-10T01:12:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:16 smithi136 ceph-mon[133708]: pgmap v280: 129 pgs: 129 active+clean; 238 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2023-09-10T01:12:16.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:16 smithi092 ceph-mon[134067]: pgmap v280: 129 pgs: 129 active+clean; 238 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2023-09-10T01:12:16.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 808390 8.89 MB/sec execute 358 sec latency 24.537 ms 2023-09-10T01:12:17.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 810027 8.90 MB/sec execute 359 sec latency 14.377 ms 2023-09-10T01:12:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:18 smithi003 ceph-mon[97654]: pgmap v281: 129 pgs: 129 active+clean; 238 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2023-09-10T01:12:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:18 smithi136 ceph-mon[133708]: pgmap v281: 129 pgs: 129 active+clean; 238 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2023-09-10T01:12:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:18 smithi092 ceph-mon[134067]: pgmap v281: 129 pgs: 129 active+clean; 238 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2023-09-10T01:12:18.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 811643 8.89 MB/sec execute 360 sec latency 57.863 ms 2023-09-10T01:12:19.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 813475 8.90 MB/sec execute 361 sec latency 15.714 ms 2023-09-10T01:12:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:20 smithi003 ceph-mon[97654]: pgmap v282: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 342 op/s 2023-09-10T01:12:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:20 smithi136 ceph-mon[133708]: pgmap v282: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 342 op/s 2023-09-10T01:12:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:20 smithi092 ceph-mon[134067]: pgmap v282: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 342 op/s 2023-09-10T01:12:20.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 814905 8.89 MB/sec execute 362 sec latency 29.572 ms 2023-09-10T01:12:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 -- 172.21.15.3:0/735992779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064106680 msgr2=0x7f3064106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 --2- 172.21.15.3:0/735992779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064106680 0x7f3064106a60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f305400b0a0 tx=0x7f305400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 -- 172.21.15.3:0/735992779 shutdown_connections 2023-09-10T01:12:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 --2- 172.21.15.3:0/735992779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064106680 0x7f3064106a60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 --2- 172.21.15.3:0/735992779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3064100f80 0x7f306410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 --2- 172.21.15.3:0/735992779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30641005c0 0x7f3064100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 -- 172.21.15.3:0/735992779 >> 172.21.15.3:0/735992779 conn(0x7f30640fc0b0 msgr2=0x7f30640fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 -- 172.21.15.3:0/735992779 shutdown_connections 2023-09-10T01:12:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.232+0000 7f306a1ff700 1 -- 172.21.15.3:0/735992779 wait complete. 2023-09-10T01:12:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.234+0000 7f306a1ff700 1 Processor -- start 2023-09-10T01:12:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.234+0000 7f306a1ff700 1 -- start start 2023-09-10T01:12:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f306a1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30641005c0 0x7f3064072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f306a1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064100f80 0x7f3064072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f306a1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064106680 0x7f30641a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f306a1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3064073670 con 0x7f3064106680 2023-09-10T01:12:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f306a1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30641a4ee0 con 0x7f30641005c0 2023-09-10T01:12:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f306a1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30641a5020 con 0x7f3064100f80 2023-09-10T01:12:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30641005c0 0x7f3064072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30641005c0 0x7f3064072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34754/0 (socket says 172.21.15.3:34754) 2023-09-10T01:12:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 -- 172.21.15.3:0/186065926 learned_addr learned my addr 172.21.15.3:0/186065926 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f3062ffd700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064100f80 0x7f3064072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f3063fff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064106680 0x7f30641a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 -- 172.21.15.3:0/186065926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064100f80 msgr2=0x7f3064072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064100f80 0x7f3064072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 -- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064106680 msgr2=0x7f30641a49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064106680 0x7f30641a49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f30637fe700 1 -- 172.21.15.3:0/186065926 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c009520 con 0x7f30641005c0 2023-09-10T01:12:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.235+0000 7f3063fff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064106680 0x7f30641a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:12:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.236+0000 7f30637fe700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30641005c0 0x7f3064072900 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f305400b070 tx=0x7f30540077f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.236+0000 7f3060ff9700 1 -- 172.21.15.3:0/186065926 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400e040 con 0x7f30641005c0 2023-09-10T01:12:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.236+0000 7f3060ff9700 1 -- 172.21.15.3:0/186065926 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f305400be20 con 0x7f30641005c0 2023-09-10T01:12:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.236+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3054009d00 con 0x7f30641005c0 2023-09-10T01:12:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.236+0000 7f3060ff9700 1 -- 172.21.15.3:0/186065926 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305401b6c0 con 0x7f30641005c0 2023-09-10T01:12:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.236+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30641a5510 con 0x7f30641005c0 2023-09-10T01:12:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.239+0000 7f3060ff9700 1 -- 172.21.15.3:0/186065926 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f305401d470 con 0x7f30641005c0 2023-09-10T01:12:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.241+0000 7f3060ff9700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f304c0a0830 0x7f304c0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.241+0000 7f3060ff9700 1 -- 172.21.15.3:0/186065926 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f30540cd420 con 0x7f30641005c0 2023-09-10T01:12:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.241+0000 7f3062ffd700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f304c0a0830 0x7f304c0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.242+0000 7f3062ffd700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f304c0a0830 0x7f304c0a2cf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f30640740f0 tx=0x7f3058009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.242+0000 7f307da66b80 1 Processor -- start 2023-09-10T01:12:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.242+0000 7f307da66b80 1 -- start start 2023-09-10T01:12:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f307da66b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de798580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f307da66b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792300 0x55a8de792780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f307da66b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792cc0 0x55a8de79ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f307da66b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de6eb6d0 con 0x55a8de792cc0 2023-09-10T01:12:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f307da66b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de6e8ff0 con 0x55a8de792300 2023-09-10T01:12:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f307da66b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de6e48e0 con 0x55a8de7981a0 2023-09-10T01:12:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f30637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de798580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f30637fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de798580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:33810/0 (socket says 172.21.15.3:33810) 2023-09-10T01:12:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f30637fe700 1 -- 172.21.15.3:0/326498585 learned_addr learned my addr 172.21.15.3:0/326498585 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f3062ffd700 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792300 0x55a8de792780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.243+0000 7f3063fff700 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792cc0 0x55a8de79ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f30637fe700 1 -- 172.21.15.3:0/326498585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792300 msgr2=0x55a8de792780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f30637fe700 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792300 0x55a8de792780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f30637fe700 1 -- 172.21.15.3:0/326498585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792cc0 msgr2=0x55a8de79ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f30637fe700 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792cc0 0x55a8de79ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f30637fe700 1 -- 172.21.15.3:0/326498585 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8de510d80 con 0x55a8de7981a0 2023-09-10T01:12:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f3063fff700 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792cc0 0x55a8de79ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:12:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f30637fe700 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de798580 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f30540090a0 tx=0x7f305400bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4cce7d81de15315 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f3049ffb700 1 -- 172.21.15.3:0/326498585 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305409e170 con 0x55a8de7981a0 2023-09-10T01:12:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f3049ffb700 1 -- 172.21.15.3:0/326498585 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f305409ac10 con 0x55a8de7981a0 2023-09-10T01:12:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.244+0000 7f3049ffb700 1 -- 172.21.15.3:0/326498585 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30540ca9a0 con 0x55a8de7981a0 2023-09-10T01:12:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 -- 172.21.15.3:0/326498585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 msgr2=0x55a8de798580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de798580 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f30540090a0 tx=0x7f305400bf90 comp rx=0 tx=0).stop 2023-09-10T01:12:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 -- 172.21.15.3:0/326498585 shutdown_connections 2023-09-10T01:12:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de798580 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792cc0 0x55a8de79ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 --2- 172.21.15.3:0/326498585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792300 0x55a8de792780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 -- 172.21.15.3:0/326498585 >> 172.21.15.3:0/326498585 conn(0x55a8de707030 msgr2=0x55a8de78d560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 -- 172.21.15.3:0/326498585 shutdown_connections 2023-09-10T01:12:21.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.245+0000 7f307da66b80 1 -- 172.21.15.3:0/326498585 wait complete. 2023-09-10T01:12:21.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.246+0000 7f307da66b80 1 -- start start 2023-09-10T01:12:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.246+0000 7f307da66b80 1 Processor -- start 2023-09-10T01:12:21.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f307da66b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f307da66b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de7a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f307da66b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de79be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f307da66b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de682b10 con 0x55a8de792300 2023-09-10T01:12:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f307da66b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de682810 con 0x55a8de792cc0 2023-09-10T01:12:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f307da66b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de5668c0 con 0x55a8de7981a0 2023-09-10T01:12:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f307da66b80 10 client.0 _ll_register_callbacks cb 0x55a8de5cca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de7a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de7a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34784/0 (socket says 172.21.15.3:34784) 2023-09-10T01:12:21.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 -- 172.21.15.3:0/4255272165 learned_addr learned my addr 172.21.15.3:0/4255272165 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3063fff700 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de79be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f30637fe700 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 -- 172.21.15.3:0/4255272165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 msgr2=0x55a8de79be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de79be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 -- 172.21.15.3:0/4255272165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 msgr2=0x55a8de7a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.247+0000 7f3062ffd700 1 -- 172.21.15.3:0/4255272165 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30641a5510 con 0x55a8de792cc0 2023-09-10T01:12:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f3062ffd700 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de7a20d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3058004580 tx=0x7f30580047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6689a72414804e5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f3042ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f3042ffd700 1 -- 172.21.15.3:0/4255272165 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058022070 con 0x55a8de792cc0 2023-09-10T01:12:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8de63c490 con 0x55a8de792cc0 2023-09-10T01:12:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f3042ffd700 1 -- 172.21.15.3:0/4255272165 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f305801d410 con 0x55a8de792cc0 2023-09-10T01:12:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f3042ffd700 1 -- 172.21.15.3:0/4255272165 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30580045b0 con 0x55a8de792cc0 2023-09-10T01:12:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f307da66b80 10 client.27797 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.248+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a8de4f9c00 con 0x55a8de792cc0 2023-09-10T01:12:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f3042ffd700 1 -- 172.21.15.3:0/4255272165 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a8de63c490 con 0x55a8de792cc0 2023-09-10T01:12:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f3042ffd700 1 -- 172.21.15.3:0/4255272165 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f30580277b0 con 0x55a8de792cc0 2023-09-10T01:12:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f307da66b80 10 client.27797 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f307da66b80 10 client.27797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f307da66b80 20 client.27797 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f307da66b80 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de7ac0a0 0x55a8de7ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f307da66b80 4 client.27797 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:12:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8de6f9840 con 0x55a8de7ac0a0 2023-09-10T01:12:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.249+0000 7f30637fe700 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de7ac0a0 0x55a8de7ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.250+0000 7f30637fe700 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de7ac0a0 0x55a8de7ae4c0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1fdf8bdc9b69939 server_cookie=f0226f0488c72d41 in_seq=0 out_seq=0 2023-09-10T01:12:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.250+0000 7f3042ffd700 10 client.27797 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.250+0000 7f3042ffd700 1 -- 172.21.15.3:0/4255272165 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a8de6f9840 con 0x55a8de7ac0a0 2023-09-10T01:12:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.250+0000 7f3042ffd700 10 client.27797 handle_command_reply: tid=0 2023-09-10T01:12:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.250+0000 7f307da66b80 1 client.27797 shutdown 2023-09-10T01:12:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 msgr2=0x55a8de7a20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de7a20d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3058004580 tx=0x7f30580047c0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 shutdown_connections 2023-09-10T01:12:21.268 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de7ac0a0 0x55a8de7ae4c0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.268 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de7981a0 0x55a8de79be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.268 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.268 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f3042ffd700 0 client.27797 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 --2- 172.21.15.3:0/4255272165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de7a20d0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 >> 172.21.15.3:0/4255272165 conn(0x55a8de707030 msgr2=0x55a8de78c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 shutdown_connections 2023-09-10T01:12:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 1 -- 172.21.15.3:0/4255272165 wait complete. 2023-09-10T01:12:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.251+0000 7f307da66b80 20 client.27797 trim_cache size 0 max 16384 2023-09-10T01:12:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 Processor -- start 2023-09-10T01:12:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 -- start start 2023-09-10T01:12:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792300 0x55a8de797900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de78cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de7981a0 0x55a8de78d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de797e40 con 0x55a8de7981a0 2023-09-10T01:12:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de799e60 con 0x55a8de792cc0 2023-09-10T01:12:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f307da66b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de784e00 con 0x55a8de792300 2023-09-10T01:12:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f3062ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de78cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f3062ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de78cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34786/0 (socket says 172.21.15.3:34786) 2023-09-10T01:12:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f3062ffd700 1 -- 172.21.15.3:0/2566495776 learned_addr learned my addr 172.21.15.3:0/2566495776 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f3063fff700 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de7981a0 0x55a8de78d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.269+0000 7f30637fe700 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792300 0x55a8de797900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f3062ffd700 1 -- 172.21.15.3:0/2566495776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792300 msgr2=0x55a8de797900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f3062ffd700 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792300 0x55a8de797900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f3062ffd700 1 -- 172.21.15.3:0/2566495776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de7981a0 msgr2=0x55a8de78d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f3062ffd700 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de7981a0 0x55a8de78d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f3062ffd700 1 -- 172.21.15.3:0/2566495776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8de5371f0 con 0x55a8de792cc0 2023-09-10T01:12:21.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f3062ffd700 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de78cf20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3058016040 tx=0x7f3058003cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f93c09bbc5d393 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f304a7fc700 1 -- 172.21.15.3:0/2566495776 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058022070 con 0x55a8de792cc0 2023-09-10T01:12:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f304a7fc700 1 -- 172.21.15.3:0/2566495776 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30580278e0 con 0x55a8de792cc0 2023-09-10T01:12:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.270+0000 7f304a7fc700 1 -- 172.21.15.3:0/2566495776 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058035ae0 con 0x55a8de792cc0 2023-09-10T01:12:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 -- 172.21.15.3:0/2566495776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 msgr2=0x55a8de78cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de78cf20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3058016040 tx=0x7f3058003cc0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 -- 172.21.15.3:0/2566495776 shutdown_connections 2023-09-10T01:12:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792300 0x55a8de797900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de7981a0 0x55a8de78d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 --2- 172.21.15.3:0/2566495776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de792cc0 0x55a8de78cf20 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 -- 172.21.15.3:0/2566495776 >> 172.21.15.3:0/2566495776 conn(0x55a8de707030 msgr2=0x55a8de79bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 -- 172.21.15.3:0/2566495776 shutdown_connections 2023-09-10T01:12:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.271+0000 7f307da66b80 1 -- 172.21.15.3:0/2566495776 wait complete. 2023-09-10T01:12:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 -- start start 2023-09-10T01:12:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 Processor -- start 2023-09-10T01:12:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7f9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792cc0 0x55a8de7f97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de7981a0 0x55a8de7f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de7f4440 con 0x55a8de792300 2023-09-10T01:12:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de7f4580 con 0x55a8de7981a0 2023-09-10T01:12:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.272+0000 7f307da66b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8de7f4740 con 0x55a8de792cc0 2023-09-10T01:12:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f307da66b80 10 client.0 _ll_register_callbacks cb 0x7f30580043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3063fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de7981a0 0x55a8de7f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f30637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7f9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3062ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792cc0 0x55a8de7f97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3063fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de7981a0 0x55a8de7f3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34794/0 (socket says 172.21.15.3:34794) 2023-09-10T01:12:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3063fff700 1 -- 172.21.15.3:0/1511679295 learned_addr learned my addr 172.21.15.3:0/1511679295 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3062ffd700 1 -- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de7981a0 msgr2=0x55a8de7f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3062ffd700 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de7981a0 0x55a8de7f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3062ffd700 1 -- 172.21.15.3:0/1511679295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 msgr2=0x55a8de7f9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3062ffd700 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7f9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3062ffd700 1 -- 172.21.15.3:0/1511679295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8de4f9c00 con 0x55a8de792cc0 2023-09-10T01:12:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.273+0000 7f3063fff700 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de7981a0 0x55a8de7f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:12:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f3062ffd700 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792cc0 0x55a8de7f97c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3058027d70 tx=0x7f3058014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92140a497b10f721 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f3042ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:12:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f3042ffd700 1 -- 172.21.15.3:0/1511679295 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058022030 con 0x55a8de792cc0 2023-09-10T01:12:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f305409ac10 con 0x55a8de792cc0 2023-09-10T01:12:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f3042ffd700 1 -- 172.21.15.3:0/1511679295 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3058023070 con 0x55a8de792cc0 2023-09-10T01:12:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f3042ffd700 1 -- 172.21.15.3:0/1511679295 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3058005240 con 0x55a8de792cc0 2023-09-10T01:12:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f307da66b80 10 client.27706 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.274+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a8de50f250 con 0x55a8de792cc0 2023-09-10T01:12:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f3042ffd700 1 -- 172.21.15.3:0/1511679295 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f305409ac10 con 0x55a8de792cc0 2023-09-10T01:12:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f3042ffd700 1 -- 172.21.15.3:0/1511679295 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f305803c070 con 0x55a8de792cc0 2023-09-10T01:12:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f307da66b80 10 client.27706 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f307da66b80 10 client.27706 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f307da66b80 20 client.27706 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f307da66b80 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de800c40 0x55a8de803060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f307da66b80 4 client.27706 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:12:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a8de6dd8c0 con 0x55a8de800c40 2023-09-10T01:12:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f30637fe700 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de800c40 0x55a8de803060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f30637fe700 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de800c40 0x55a8de803060 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c0e6c481d986c7 server_cookie=c563403cd892e1f2 in_seq=0 out_seq=0 2023-09-10T01:12:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.275+0000 7f3042ffd700 10 client.27706 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f3042ffd700 1 -- 172.21.15.3:0/1511679295 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a8de6dd8c0 con 0x55a8de800c40 2023-09-10T01:12:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f3042ffd700 10 client.27706 handle_command_reply: tid=0 2023-09-10T01:12:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 client.27706 shutdown 2023-09-10T01:12:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792cc0 msgr2=0x55a8de7f97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792cc0 0x55a8de7f97c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3058027d70 tx=0x7f3058014000 comp rx=0 tx=0).stop 2023-09-10T01:12:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 shutdown_connections 2023-09-10T01:12:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a8de800c40 0x55a8de803060 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f3042ffd700 0 client.27706 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a8de7981a0 0x55a8de7f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a8de792300 0x55a8de7f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 --2- 172.21.15.3:0/1511679295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a8de792cc0 0x55a8de7f97c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 >> 172.21.15.3:0/1511679295 conn(0x55a8de707030 msgr2=0x55a8de79a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:21.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 shutdown_connections 2023-09-10T01:12:21.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.377+0000 7f307da66b80 1 -- 172.21.15.3:0/1511679295 wait complete. 2023-09-10T01:12:21.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f307da66b80 20 client.27706 trim_cache size 0 max 16384 2023-09-10T01:12:21.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:12:21.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:12:21.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:12:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:12:21.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f304c0a0830 msgr2=0x7f304c0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f304c0a0830 0x7f304c0a2cf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f30640740f0 tx=0x7f3058009260 comp rx=0 tx=0).stop 2023-09-10T01:12:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30641005c0 msgr2=0x7f3064072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30641005c0 0x7f3064072900 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f305400b070 tx=0x7f30540077f0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 shutdown_connections 2023-09-10T01:12:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f304c0a0830 0x7f304c0a2cf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064106680 0x7f30641a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30641005c0 0x7f3064072900 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 --2- 172.21.15.3:0/186065926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3064100f80 0x7f3064072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.378+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 >> 172.21.15.3:0/186065926 conn(0x7f30640fc0b0 msgr2=0x7f30640fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.379+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 shutdown_connections 2023-09-10T01:12:21.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:21.379+0000 7f306a1ff700 1 -- 172.21.15.3:0/186065926 wait complete. 2023-09-10T01:12:21.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 816865 8.91 MB/sec execute 363 sec latency 34.963 ms 2023-09-10T01:12:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:22 smithi003 ceph-mon[97654]: pgmap v283: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.9 MiB/s wr, 335 op/s 2023-09-10T01:12:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:22 smithi136 ceph-mon[133708]: pgmap v283: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.9 MiB/s wr, 335 op/s 2023-09-10T01:12:22.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:22 smithi092 ceph-mon[134067]: pgmap v283: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.9 MiB/s wr, 335 op/s 2023-09-10T01:12:22.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 818252 8.90 MB/sec execute 364 sec latency 20.703 ms 2023-09-10T01:12:23.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 820450 8.91 MB/sec execute 365 sec latency 24.353 ms 2023-09-10T01:12:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:24 smithi003 ceph-mon[97654]: pgmap v284: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 285 op/s 2023-09-10T01:12:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:24 smithi136 ceph-mon[133708]: pgmap v284: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 285 op/s 2023-09-10T01:12:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:24 smithi092 ceph-mon[134067]: pgmap v284: 129 pgs: 129 active+clean; 245 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 285 op/s 2023-09-10T01:12:24.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 821757 8.90 MB/sec execute 366 sec latency 70.242 ms 2023-09-10T01:12:25.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 823959 8.91 MB/sec execute 367 sec latency 16.079 ms 2023-09-10T01:12:26.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:26 smithi003 ceph-mon[97654]: pgmap v285: 129 pgs: 129 active+clean; 240 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 396 op/s 2023-09-10T01:12:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:26 smithi136 ceph-mon[133708]: pgmap v285: 129 pgs: 129 active+clean; 240 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 396 op/s 2023-09-10T01:12:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:26 smithi092 ceph-mon[134067]: pgmap v285: 129 pgs: 129 active+clean; 240 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.8 MiB/s wr, 396 op/s 2023-09-10T01:12:26.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 825190 8.90 MB/sec execute 368 sec latency 28.727 ms 2023-09-10T01:12:27.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 827255 8.91 MB/sec execute 369 sec latency 46.387 ms 2023-09-10T01:12:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:28 smithi003 ceph-mon[97654]: pgmap v286: 129 pgs: 129 active+clean; 240 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2023-09-10T01:12:28.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:28 smithi136 ceph-mon[133708]: pgmap v286: 129 pgs: 129 active+clean; 240 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2023-09-10T01:12:28.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:28 smithi092 ceph-mon[134067]: pgmap v286: 129 pgs: 129 active+clean; 240 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2023-09-10T01:12:28.788 INFO:tasks.workunit.client.0.smithi003.stdout: 1 828296 8.90 MB/sec execute 370 sec latency 55.473 ms 2023-09-10T01:12:29.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 830522 8.91 MB/sec execute 371 sec latency 20.611 ms 2023-09-10T01:12:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:30 smithi003 ceph-mon[97654]: pgmap v287: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-09-10T01:12:30.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:30 smithi136 ceph-mon[133708]: pgmap v287: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-09-10T01:12:30.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:30 smithi092 ceph-mon[134067]: pgmap v287: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-09-10T01:12:30.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 831561 8.90 MB/sec execute 372 sec latency 45.615 ms 2023-09-10T01:12:31.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:31 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:12:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:31 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:12:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:31 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:12:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:31 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:12:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:31 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:12:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:31 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:12:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:31 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:12:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:31 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:12:31.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:31 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:12:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:31 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:12:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:31 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:12:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:31 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:12:31.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 833231 8.89 MB/sec execute 373 sec latency 29.336 ms 2023-09-10T01:12:32.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:32 smithi003 ceph-mon[97654]: pgmap v288: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2023-09-10T01:12:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:32 smithi136 ceph-mon[133708]: pgmap v288: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2023-09-10T01:12:32.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:32 smithi092 ceph-mon[134067]: pgmap v288: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2023-09-10T01:12:32.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 834752 8.90 MB/sec execute 374 sec latency 20.429 ms 2023-09-10T01:12:33.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 836300 8.89 MB/sec execute 375 sec latency 17.187 ms 2023-09-10T01:12:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 -- 172.21.15.3:0/1763307325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac401005c0 msgr2=0x7fac40100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 --2- 172.21.15.3:0/1763307325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac401005c0 0x7fac40100a40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fac30009990 tx=0x7fac30009ca0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 -- 172.21.15.3:0/1763307325 shutdown_connections 2023-09-10T01:12:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 --2- 172.21.15.3:0/1763307325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac40100f80 0x7fac4010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 --2- 172.21.15.3:0/1763307325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac40106680 0x7fac40106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 --2- 172.21.15.3:0/1763307325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac401005c0 0x7fac40100a40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 -- 172.21.15.3:0/1763307325 >> 172.21.15.3:0/1763307325 conn(0x7fac400fc0b0 msgr2=0x7fac400fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 -- 172.21.15.3:0/1763307325 shutdown_connections 2023-09-10T01:12:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.532+0000 7fac483ff700 1 -- 172.21.15.3:0/1763307325 wait complete. 2023-09-10T01:12:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.533+0000 7fac483ff700 1 Processor -- start 2023-09-10T01:12:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.533+0000 7fac483ff700 1 -- start start 2023-09-10T01:12:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac483ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac401005c0 0x7fac4019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac483ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac40100f80 0x7fac4019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac483ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac40106680 0x7fac40198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac483ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40198cf0 con 0x7fac40106680 2023-09-10T01:12:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac483ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40198e60 con 0x7fac401005c0 2023-09-10T01:12:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac483ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40199020 con 0x7fac40100f80 2023-09-10T01:12:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac4599a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac40100f80 0x7fac4019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac4619b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac401005c0 0x7fac4019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac4619b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac401005c0 0x7fac4019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60988/0 (socket says 172.21.15.3:60988) 2023-09-10T01:12:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac4599a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac40100f80 0x7fac4019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54626/0 (socket says 172.21.15.3:54626) 2023-09-10T01:12:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac4619b700 1 -- 172.21.15.3:0/834195227 learned_addr learned my addr 172.21.15.3:0/834195227 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.534+0000 7fac4699c700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac40106680 0x7fac40198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac4599a700 1 -- 172.21.15.3:0/834195227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac401005c0 msgr2=0x7fac4019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac4599a700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac401005c0 0x7fac4019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac4599a700 1 -- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac40106680 msgr2=0x7fac40198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac4599a700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac40106680 0x7fac40198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac4599a700 1 -- 172.21.15.3:0/834195227 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38009c40 con 0x7fac40100f80 2023-09-10T01:12:34.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac4699c700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac40106680 0x7fac40198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:12:34.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac4599a700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac40100f80 0x7fac4019eb50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fac30009990 tx=0x7fac30009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac377fe700 1 -- 172.21.15.3:0/834195227 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3001d070 con 0x7fac40100f80 2023-09-10T01:12:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac377fe700 1 -- 172.21.15.3:0/834195227 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac30021410 con 0x7fac40100f80 2023-09-10T01:12:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac300095f0 con 0x7fac40100f80 2023-09-10T01:12:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac377fe700 1 -- 172.21.15.3:0/834195227 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30017650 con 0x7fac40100f80 2023-09-10T01:12:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.535+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac401a5420 con 0x7fac40100f80 2023-09-10T01:12:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.539+0000 7fac377fe700 1 -- 172.21.15.3:0/834195227 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fac300217a0 con 0x7fac40100f80 2023-09-10T01:12:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.540+0000 7fac5be0ab80 1 Processor -- start 2023-09-10T01:12:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac5be0ab80 1 -- start start 2023-09-10T01:12:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a6a5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a69f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a6abf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a5f86d0 con 0x56183a69f300 2023-09-10T01:12:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a5f5ff0 con 0x56183a69fcc0 2023-09-10T01:12:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a5f18e0 con 0x56183a6a51a0 2023-09-10T01:12:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac377fe700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fac2c0a07b0 0x7fac2c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac377fe700 1 -- 172.21.15.3:0/834195227 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fac30021e40 con 0x7fac40100f80 2023-09-10T01:12:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac4599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a6abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac4699c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a6a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac4599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a6abf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:60992/0 (socket says 172.21.15.3:60992) 2023-09-10T01:12:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.541+0000 7fac4599a700 1 -- 172.21.15.3:0/806384861 learned_addr learned my addr 172.21.15.3:0/806384861 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4619b700 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a69f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4699c700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fac2c0a07b0 0x7fac2c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4599a700 1 -- 172.21.15.3:0/806384861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 msgr2=0x56183a6a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4599a700 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a6a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4599a700 1 -- 172.21.15.3:0/806384861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 msgr2=0x56183a69f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4599a700 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a69f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4599a700 1 -- 172.21.15.3:0/806384861 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56183a41dd80 con 0x56183a69fcc0 2023-09-10T01:12:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4619b700 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a69f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:12:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4599a700 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a6abf30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fac300a2380 tx=0x7fac300047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1435ef41710603d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac34bfe700 1 -- 172.21.15.3:0/806384861 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3009afd0 con 0x56183a69fcc0 2023-09-10T01:12:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac4699c700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fac2c0a07b0 0x7fac2c0a2c70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fac38009fd0 tx=0x7fac3800b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac34bfe700 1 -- 172.21.15.3:0/806384861 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac3009a8d0 con 0x56183a69fcc0 2023-09-10T01:12:34.553 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.542+0000 7fac34bfe700 1 -- 172.21.15.3:0/806384861 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac300cd370 con 0x56183a69fcc0 2023-09-10T01:12:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 -- 172.21.15.3:0/806384861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 msgr2=0x56183a6abf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a6abf30 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fac300a2380 tx=0x7fac300047c0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 -- 172.21.15.3:0/806384861 shutdown_connections 2023-09-10T01:12:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a6a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a6abf30 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/806384861 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a69f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 -- 172.21.15.3:0/806384861 >> 172.21.15.3:0/806384861 conn(0x56183a614030 msgr2=0x56183a69a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 -- 172.21.15.3:0/806384861 shutdown_connections 2023-09-10T01:12:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.543+0000 7fac5be0ab80 1 -- 172.21.15.3:0/806384861 wait complete. 2023-09-10T01:12:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.544+0000 7fac5be0ab80 1 -- start start 2023-09-10T01:12:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.544+0000 7fac5be0ab80 1 Processor -- start 2023-09-10T01:12:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69f300 0x56183a6aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69fcc0 0x56183a6af070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a6a51a0 0x56183a6a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a58fb10 con 0x56183a6a51a0 2023-09-10T01:12:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a58f810 con 0x56183a69f300 2023-09-10T01:12:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a4738c0 con 0x56183a69fcc0 2023-09-10T01:12:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac5be0ab80 10 client.0 _ll_register_callbacks cb 0x56183a4d9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac4619b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69f300 0x56183a6aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac4599a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69fcc0 0x56183a6af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac4699c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a6a51a0 0x56183a6a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac4619b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69f300 0x56183a6aeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:32768/0 (socket says 172.21.15.3:32768) 2023-09-10T01:12:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac4599a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69fcc0 0x56183a6af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54642/0 (socket says 172.21.15.3:54642) 2023-09-10T01:12:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac4619b700 1 -- 172.21.15.3:0/328669933 learned_addr learned my addr 172.21.15.3:0/328669933 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:34.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.545+0000 7fac4619b700 1 -- 172.21.15.3:0/328669933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69fcc0 msgr2=0x56183a6af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac4619b700 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69fcc0 0x56183a6af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac4619b700 1 -- 172.21.15.3:0/328669933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a6a51a0 msgr2=0x56183a6a8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac4619b700 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a6a51a0 0x56183a6a8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac4619b700 1 -- 172.21.15.3:0/328669933 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac401a5420 con 0x56183a69f300 2023-09-10T01:12:34.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac4599a700 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69fcc0 0x56183a6af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:12:34.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac4619b700 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69f300 0x56183a6aeb30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fac3c012740 tx=0x7fac3c012a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97436a0338bf599 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:34.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:34.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac29ffb700 1 -- 172.21.15.3:0/328669933 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c017070 con 0x56183a69f300 2023-09-10T01:12:34.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56183a549490 con 0x56183a69f300 2023-09-10T01:12:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac29ffb700 1 -- 172.21.15.3:0/328669933 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fac3c00f4e0 con 0x56183a69f300 2023-09-10T01:12:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac29ffb700 1 -- 172.21.15.3:0/328669933 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac3c0122e0 con 0x56183a69f300 2023-09-10T01:12:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac5be0ab80 10 client.27815 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.546+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56183a406c00 con 0x56183a69f300 2023-09-10T01:12:34.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac29ffb700 1 -- 172.21.15.3:0/328669933 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56183a549490 con 0x56183a69f300 2023-09-10T01:12:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac29ffb700 1 -- 172.21.15.3:0/328669933 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fac3c0108f0 con 0x56183a69f300 2023-09-10T01:12:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac5be0ab80 10 client.27815 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac5be0ab80 10 client.27815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:34.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac5be0ab80 20 client.27815 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a6b90c0 0x56183a6bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac5be0ab80 4 client.27815 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:12:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56183a606840 con 0x56183a6b90c0 2023-09-10T01:12:34.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.547+0000 7fac4599a700 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a6b90c0 0x56183a6bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.548+0000 7fac4599a700 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a6b90c0 0x56183a6bb4e0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfda00f2a383c34a server_cookie=cb0858daf5e5210e in_seq=0 out_seq=0 2023-09-10T01:12:34.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.548+0000 7fac29ffb700 10 client.27815 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:34.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.548+0000 7fac29ffb700 1 -- 172.21.15.3:0/328669933 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56183a606840 con 0x56183a6b90c0 2023-09-10T01:12:34.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.548+0000 7fac29ffb700 10 client.27815 handle_command_reply: tid=0 2023-09-10T01:12:34.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.548+0000 7fac5be0ab80 1 client.27815 shutdown 2023-09-10T01:12:34.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69f300 msgr2=0x56183a6aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69f300 0x56183a6aeb30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fac3c012740 tx=0x7fac3c012a50 comp rx=0 tx=0).stop 2023-09-10T01:12:34.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 shutdown_connections 2023-09-10T01:12:34.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a6b90c0 0x56183a6bb4e0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a6a51a0 0x56183a6a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac29ffb700 0 client.27815 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69f300 0x56183a6aeb30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/328669933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69fcc0 0x56183a6af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 >> 172.21.15.3:0/328669933 conn(0x56183a614030 msgr2=0x56183a6997a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 shutdown_connections 2023-09-10T01:12:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 1 -- 172.21.15.3:0/328669933 wait complete. 2023-09-10T01:12:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.549+0000 7fac5be0ab80 20 client.27815 trim_cache size 0 max 16384 2023-09-10T01:12:34.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:34 smithi003 ceph-mon[97654]: pgmap v289: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2023-09-10T01:12:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 Processor -- start 2023-09-10T01:12:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 -- start start 2023-09-10T01:12:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69f300 0x56183a6ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69fcc0 0x56183a6aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a6a51a0 0x56183a698f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a6aa340 con 0x56183a69fcc0 2023-09-10T01:12:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a691ed0 con 0x56183a6a51a0 2023-09-10T01:12:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a6a4db0 con 0x56183a69f300 2023-09-10T01:12:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac4619b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69f300 0x56183a6ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac4599a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69fcc0 0x56183a6aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac4619b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69f300 0x56183a6ae6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54652/0 (socket says 172.21.15.3:54652) 2023-09-10T01:12:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac4619b700 1 -- 172.21.15.3:0/3352257556 learned_addr learned my addr 172.21.15.3:0/3352257556 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.565+0000 7fac4699c700 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a6a51a0 0x56183a698f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac4619b700 1 -- 172.21.15.3:0/3352257556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a6a51a0 msgr2=0x56183a698f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac4619b700 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a6a51a0 0x56183a698f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac4619b700 1 -- 172.21.15.3:0/3352257556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69fcc0 msgr2=0x56183a6aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac4619b700 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69fcc0 0x56183a6aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac4619b700 1 -- 172.21.15.3:0/3352257556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56183a4441f0 con 0x56183a69f300 2023-09-10T01:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac4699c700 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a6a51a0 0x56183a698f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:12:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac4619b700 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69f300 0x56183a6ae6c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fac3c000c00 tx=0x7fac3c0057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63757191d914041 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac353ff700 1 -- 172.21.15.3:0/3352257556 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c017040 con 0x56183a69f300 2023-09-10T01:12:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac353ff700 1 -- 172.21.15.3:0/3352257556 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac3c00f4e0 con 0x56183a69f300 2023-09-10T01:12:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.566+0000 7fac353ff700 1 -- 172.21.15.3:0/3352257556 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c02a6a0 con 0x56183a69f300 2023-09-10T01:12:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 -- 172.21.15.3:0/3352257556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69f300 msgr2=0x56183a6ae6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69f300 0x56183a6ae6c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fac3c000c00 tx=0x7fac3c0057a0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 -- 172.21.15.3:0/3352257556 shutdown_connections 2023-09-10T01:12:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a69f300 0x56183a6ae6c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a6a51a0 0x56183a698f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/3352257556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69fcc0 0x56183a6aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 -- 172.21.15.3:0/3352257556 >> 172.21.15.3:0/3352257556 conn(0x56183a614030 msgr2=0x56183a69eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 -- 172.21.15.3:0/3352257556 shutdown_connections 2023-09-10T01:12:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.567+0000 7fac5be0ab80 1 -- 172.21.15.3:0/3352257556 wait complete. 2023-09-10T01:12:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 -- start start 2023-09-10T01:12:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 Processor -- start 2023-09-10T01:12:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a705fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a7064f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a7002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a7008c0 con 0x56183a69f300 2023-09-10T01:12:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a700a30 con 0x56183a69fcc0 2023-09-10T01:12:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.568+0000 7fac5be0ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56183a700bf0 con 0x56183a6a51a0 2023-09-10T01:12:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac5be0ab80 10 client.0 _ll_register_callbacks cb 0x56183a4d9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a7064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4619b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a705fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a7064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:32784/0 (socket says 172.21.15.3:32784) 2023-09-10T01:12:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 -- 172.21.15.3:0/1741573391 learned_addr learned my addr 172.21.15.3:0/1741573391 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4699c700 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a7002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 -- 172.21.15.3:0/1741573391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 msgr2=0x56183a7002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a7002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 -- 172.21.15.3:0/1741573391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 msgr2=0x56183a705fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a705fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 -- 172.21.15.3:0/1741573391 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac38009110 con 0x56183a69fcc0 2023-09-10T01:12:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac4599a700 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a7064f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fac30096dd0 tx=0x7fac3009a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7eac9022ee4164 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.569+0000 7fac29ffb700 1 -- 172.21.15.3:0/1741573391 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac300a28b0 con 0x56183a69fcc0 2023-09-10T01:12:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac29ffb700 1 -- 172.21.15.3:0/1741573391 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fac300b2e40 con 0x56183a69fcc0 2023-09-10T01:12:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac3009a8d0 con 0x56183a69fcc0 2023-09-10T01:12:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac29ffb700 1 -- 172.21.15.3:0/1741573391 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac3009ab30 con 0x56183a69fcc0 2023-09-10T01:12:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 10 client.27821 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56183a406c00 con 0x56183a69fcc0 2023-09-10T01:12:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac29ffb700 1 -- 172.21.15.3:0/1741573391 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac30064020 con 0x56183a69fcc0 2023-09-10T01:12:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac29ffb700 1 -- 172.21.15.3:0/1741573391 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fac30099e50 con 0x56183a69fcc0 2023-09-10T01:12:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 10 client.27821 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 10 client.27821 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 20 client.27821 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a70db50 0x56183a70ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 4 client.27821 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:12:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56183a5ea8c0 con 0x56183a70db50 2023-09-10T01:12:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.570+0000 7fac4619b700 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a70db50 0x56183a70ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.571+0000 7fac4619b700 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a70db50 0x56183a70ff70 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15d332b362332ac server_cookie=d753709cf0c251be in_seq=0 out_seq=0 2023-09-10T01:12:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.571+0000 7fac29ffb700 10 client.27821 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:34 smithi136 ceph-mon[133708]: pgmap v289: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2023-09-10T01:12:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac29ffb700 1 -- 172.21.15.3:0/1741573391 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56183a5ea8c0 con 0x56183a70db50 2023-09-10T01:12:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac29ffb700 10 client.27821 handle_command_reply: tid=0 2023-09-10T01:12:34.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 client.27821 shutdown 2023-09-10T01:12:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 msgr2=0x56183a7064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a7064f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fac30096dd0 tx=0x7fac3009a510 comp rx=0 tx=0).stop 2023-09-10T01:12:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 shutdown_connections 2023-09-10T01:12:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56183a70db50 0x56183a70ff70 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56183a6a51a0 0x56183a7002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56183a69f300 0x56183a705fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 --2- 172.21.15.3:0/1741573391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56183a69fcc0 0x56183a7064f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac29ffb700 0 client.27821 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:34.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.697+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 >> 172.21.15.3:0/1741573391 conn(0x56183a614030 msgr2=0x56183a6a5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.698+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 shutdown_connections 2023-09-10T01:12:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.698+0000 7fac5be0ab80 1 -- 172.21.15.3:0/1741573391 wait complete. 2023-09-10T01:12:34.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.699+0000 7fac5be0ab80 20 client.27821 trim_cache size 0 max 16384 2023-09-10T01:12:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:12:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:12:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:12:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:12:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.699+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fac2c0a07b0 msgr2=0x7fac2c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.699+0000 7fac483ff700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fac2c0a07b0 0x7fac2c0a2c70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fac38009fd0 tx=0x7fac3800b040 comp rx=0 tx=0).stop 2023-09-10T01:12:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.699+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac40100f80 msgr2=0x7fac4019eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.699+0000 7fac483ff700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac40100f80 0x7fac4019eb50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fac30009990 tx=0x7fac30009580 comp rx=0 tx=0).stop 2023-09-10T01:12:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 shutdown_connections 2023-09-10T01:12:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fac2c0a07b0 0x7fac2c0a2c70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac40106680 0x7fac40198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac401005c0 0x7fac4019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 --2- 172.21.15.3:0/834195227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac40100f80 0x7fac4019eb50 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 >> 172.21.15.3:0/834195227 conn(0x7fac400fc0b0 msgr2=0x7fac400729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 shutdown_connections 2023-09-10T01:12:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:34.700+0000 7fac483ff700 1 -- 172.21.15.3:0/834195227 wait complete. 2023-09-10T01:12:34.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:34 smithi092 ceph-mon[134067]: pgmap v289: 129 pgs: 129 active+clean; 248 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 190 op/s 2023-09-10T01:12:34.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 838166 8.90 MB/sec execute 376 sec latency 63.666 ms 2023-09-10T01:12:35.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 839426 8.90 MB/sec execute 377 sec latency 19.714 ms 2023-09-10T01:12:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:36 smithi003 ceph-mon[97654]: pgmap v290: 129 pgs: 129 active+clean; 256 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 263 op/s 2023-09-10T01:12:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:36 smithi136 ceph-mon[133708]: pgmap v290: 129 pgs: 129 active+clean; 256 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 263 op/s 2023-09-10T01:12:36.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:36 smithi092 ceph-mon[134067]: pgmap v290: 129 pgs: 129 active+clean; 256 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 263 op/s 2023-09-10T01:12:36.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 841672 8.91 MB/sec execute 378 sec latency 17.630 ms 2023-09-10T01:12:37.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 842627 8.89 MB/sec execute 379 sec latency 40.791 ms 2023-09-10T01:12:38.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:38 smithi003 ceph-mon[97654]: pgmap v291: 129 pgs: 129 active+clean; 256 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2023-09-10T01:12:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:38 smithi136 ceph-mon[133708]: pgmap v291: 129 pgs: 129 active+clean; 256 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2023-09-10T01:12:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:38 smithi092 ceph-mon[134067]: pgmap v291: 129 pgs: 129 active+clean; 256 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 152 op/s 2023-09-10T01:12:38.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 844470 8.90 MB/sec execute 380 sec latency 80.525 ms 2023-09-10T01:12:39.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 845742 8.89 MB/sec execute 381 sec latency 14.065 ms 2023-09-10T01:12:40.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:40 smithi003 ceph-mon[97654]: pgmap v292: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2023-09-10T01:12:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:40 smithi136 ceph-mon[133708]: pgmap v292: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2023-09-10T01:12:40.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:40 smithi092 ceph-mon[134067]: pgmap v292: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2023-09-10T01:12:40.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 847506 8.89 MB/sec execute 382 sec latency 26.805 ms 2023-09-10T01:12:41.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 849120 8.89 MB/sec execute 383 sec latency 11.628 ms 2023-09-10T01:12:42.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:42 smithi003 ceph-mon[97654]: pgmap v293: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-09-10T01:12:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:42 smithi136 ceph-mon[133708]: pgmap v293: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-09-10T01:12:42.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:42 smithi092 ceph-mon[134067]: pgmap v293: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-09-10T01:12:42.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:12:42.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 849973 8.88 MB/sec execute 384 sec latency 286.565 ms 2023-09-10T01:12:43.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.080+0000 7fa549dff700 1 -- 172.21.15.3:0/1214007636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544108220 msgr2=0x7fa54410bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.080+0000 7fa549dff700 1 --2- 172.21.15.3:0/1214007636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544108220 0x7fa54410bce0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa538009fc0 tx=0x7fa53800b330 comp rx=0 tx=0).stop 2023-09-10T01:12:43.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.082+0000 7fa549dff700 1 -- 172.21.15.3:0/1214007636 shutdown_connections 2023-09-10T01:12:43.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.082+0000 7fa549dff700 1 --2- 172.21.15.3:0/1214007636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54410c410 0x7fa544113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.082+0000 7fa549dff700 1 --2- 172.21.15.3:0/1214007636 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544107870 0x7fa544107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.082+0000 7fa549dff700 1 --2- 172.21.15.3:0/1214007636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544108220 0x7fa54410bce0 secure :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa538009fc0 tx=0x7fa53800b330 comp rx=0 tx=0).stop 2023-09-10T01:12:43.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.082+0000 7fa549dff700 1 -- 172.21.15.3:0/1214007636 >> 172.21.15.3:0/1214007636 conn(0x7fa544072090 msgr2=0x7fa54406ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:43.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.087+0000 7fa549dff700 1 -- 172.21.15.3:0/1214007636 shutdown_connections 2023-09-10T01:12:43.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.089+0000 7fa549dff700 1 -- 172.21.15.3:0/1214007636 wait complete. 2023-09-10T01:12:43.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.091+0000 7fa549dff700 1 Processor -- start 2023-09-10T01:12:43.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.091+0000 7fa549dff700 1 -- start start 2023-09-10T01:12:43.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.092+0000 7fa549dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544107870 0x7fa544113c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.092+0000 7fa549dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54410c410 0x7fa544110c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.092+0000 7fa549dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544111170 0x7fa544111620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.092+0000 7fa549dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544078860 con 0x7fa544107870 2023-09-10T01:12:43.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.092+0000 7fa549dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5440789d0 con 0x7fa544111170 2023-09-10T01:12:43.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.092+0000 7fa549dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa544078b90 con 0x7fa54410c410 2023-09-10T01:12:43.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.094+0000 7fa543fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544111170 0x7fa544111620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.094+0000 7fa543fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544111170 0x7fa544111620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:37334/0 (socket says 172.21.15.3:37334) 2023-09-10T01:12:43.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.094+0000 7fa543fff700 1 -- 172.21.15.3:0/839574623 learned_addr learned my addr 172.21.15.3:0/839574623 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:43.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.094+0000 7fa5437fe700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544107870 0x7fa544113c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.095+0000 7fa5437fe700 1 -- 172.21.15.3:0/839574623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54410c410 msgr2=0x7fa544110c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.095+0000 7fa5437fe700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54410c410 0x7fa544110c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.095+0000 7fa5437fe700 1 -- 172.21.15.3:0/839574623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544111170 msgr2=0x7fa544111620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.095+0000 7fa5437fe700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544111170 0x7fa544111620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.095+0000 7fa5437fe700 1 -- 172.21.15.3:0/839574623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538009c40 con 0x7fa544107870 2023-09-10T01:12:43.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.096+0000 7fa5437fe700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544107870 0x7fa544113c20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa534007830 tx=0x7fa534007bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:43.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.099+0000 7fa540ff9700 1 -- 172.21.15.3:0/839574623 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53400f040 con 0x7fa544107870 2023-09-10T01:12:43.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.099+0000 7fa540ff9700 1 -- 172.21.15.3:0/839574623 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa534013cd0 con 0x7fa544107870 2023-09-10T01:12:43.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.099+0000 7fa540ff9700 1 -- 172.21.15.3:0/839574623 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53400de50 con 0x7fa544107870 2023-09-10T01:12:43.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.099+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa544078e70 con 0x7fa544107870 2023-09-10T01:12:43.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.099+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa544079330 con 0x7fa544107870 2023-09-10T01:12:43.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.106+0000 7fa55d640b80 1 Processor -- start 2023-09-10T01:12:43.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.106+0000 7fa55d640b80 1 -- start start 2023-09-10T01:12:43.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.107+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f79af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.107+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f7a030 0x560075f83900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.107+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f86230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.107+0000 7fa55d640b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075ed2730 con 0x560075f7a030 2023-09-10T01:12:43.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.107+0000 7fa55d640b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075ed0050 con 0x560075f79710 2023-09-10T01:12:43.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.107+0000 7fa55d640b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075ecb940 con 0x560075f83e40 2023-09-10T01:12:43.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.108+0000 7fa543fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f79af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.108+0000 7fa542ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f86230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.109+0000 7fa5437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f7a030 0x560075f83900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.109+0000 7fa5437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f7a030 0x560075f83900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38794/0 (socket says 172.21.15.3:38794) 2023-09-10T01:12:43.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.109+0000 7fa5437fe700 1 -- 172.21.15.3:0/1899464354 learned_addr learned my addr 172.21.15.3:0/1899464354 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:43.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.110+0000 7fa540ff9700 1 -- 172.21.15.3:0/839574623 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa534013810 con 0x7fa544107870 2023-09-10T01:12:43.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.110+0000 7fa5437fe700 1 -- 172.21.15.3:0/1899464354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 msgr2=0x560075f86230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.110+0000 7fa5437fe700 1 --2- 172.21.15.3:0/1899464354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f86230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.111+0000 7fa5437fe700 1 -- 172.21.15.3:0/1899464354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 msgr2=0x560075f79af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.111+0000 7fa5437fe700 1 --2- 172.21.15.3:0/1899464354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f79af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.111+0000 7fa5437fe700 1 -- 172.21.15.3:0/1899464354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560075cf7de0 con 0x560075f7a030 2023-09-10T01:12:43.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.111+0000 7fa5437fe700 1 --2- 172.21.15.3:0/1899464354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f7a030 0x560075f83900 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa5340c8d10 tx=0x7fa5340cfd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c1c4de9bc212fe server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:43.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.112+0000 7fa529ffb700 1 -- 172.21.15.3:0/1899464354 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340d4ac0 con 0x560075f7a030 2023-09-10T01:12:43.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.112+0000 7fa529ffb700 1 -- 172.21.15.3:0/1899464354 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5340d3040 con 0x560075f7a030 2023-09-10T01:12:43.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.112+0000 7fa529ffb700 1 -- 172.21.15.3:0/1899464354 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340d83e0 con 0x560075f7a030 2023-09-10T01:12:43.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.112+0000 7fa540ff9700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa52c0a09c0 0x7fa52c0a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa540ff9700 1 -- 172.21.15.3:0/839574623 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa5340cbad0 con 0x7fa544107870 2023-09-10T01:12:43.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa55d640b80 1 -- 172.21.15.3:0/1899464354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f7a030 msgr2=0x560075f83900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1899464354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f7a030 0x560075f83900 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa5340c8d10 tx=0x7fa5340cfd90 comp rx=0 tx=0).stop 2023-09-10T01:12:43.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa55d640b80 1 -- 172.21.15.3:0/1899464354 shutdown_connections 2023-09-10T01:12:43.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1899464354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f86230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1899464354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f79af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1899464354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f7a030 0x560075f83900 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.113+0000 7fa55d640b80 1 -- 172.21.15.3:0/1899464354 >> 172.21.15.3:0/1899464354 conn(0x560075eee090 msgr2=0x560075f74bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:43.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.114+0000 7fa543fff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa52c0a09c0 0x7fa52c0a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.114+0000 7fa55d640b80 1 -- 172.21.15.3:0/1899464354 shutdown_connections 2023-09-10T01:12:43.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.114+0000 7fa55d640b80 1 -- 172.21.15.3:0/1899464354 wait complete. 2023-09-10T01:12:43.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.115+0000 7fa55d640b80 1 -- start start 2023-09-10T01:12:43.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.115+0000 7fa55d640b80 1 Processor -- start 2023-09-10T01:12:43.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.115+0000 7fa543fff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa52c0a09c0 0x7fa52c0a2e80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00bc10 tx=0x7fa53c00b2c0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:43.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f83790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 0x560075f809a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f83e40 0x560075f80f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa55d640b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075e69b70 con 0x560075f79710 2023-09-10T01:12:43.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa55d640b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075e69870 con 0x560075f83e40 2023-09-10T01:12:43.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa55d640b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075d4d920 con 0x560075f7a030 2023-09-10T01:12:43.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa55d640b80 10 client.0 _ll_register_callbacks cb 0x560075db3a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:43.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.116+0000 7fa5437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f83790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa5437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f83790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38802/0 (socket says 172.21.15.3:38802) 2023-09-10T01:12:43.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa5437fe700 1 -- 172.21.15.3:0/4156099597 learned_addr learned my addr 172.21.15.3:0/4156099597 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:43.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa543fff700 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f83e40 0x560075f80f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa5437fe700 1 -- 172.21.15.3:0/4156099597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 msgr2=0x560075f809a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa5437fe700 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 0x560075f809a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa5437fe700 1 -- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f83e40 msgr2=0x560075f80f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa5437fe700 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f83e40 0x560075f80f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.117+0000 7fa5437fe700 1 -- 172.21.15.3:0/4156099597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa544079330 con 0x560075f79710 2023-09-10T01:12:43.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.118+0000 7fa5437fe700 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f83790 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa5340d44a0 tx=0x7fa5340d4f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ca29748ba9f27f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:43.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.118+0000 7fa522ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:12:43.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.120+0000 7fa522ffd700 1 -- 172.21.15.3:0/4156099597 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340c8d40 con 0x560075f79710 2023-09-10T01:12:43.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.120+0000 7fa522ffd700 1 -- 172.21.15.3:0/4156099597 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa534013e40 con 0x560075f79710 2023-09-10T01:12:43.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.120+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560075e234f0 con 0x560075f79710 2023-09-10T01:12:43.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.120+0000 7fa522ffd700 1 -- 172.21.15.3:0/4156099597 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5340b2020 con 0x560075f79710 2023-09-10T01:12:43.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.120+0000 7fa55d640b80 10 client.17742 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:43.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.120+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560075ce0c60 con 0x560075f79710 2023-09-10T01:12:43.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.121+0000 7fa522ffd700 1 -- 172.21.15.3:0/4156099597 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560075e234f0 con 0x560075f79710 2023-09-10T01:12:43.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.121+0000 7fa522ffd700 1 -- 172.21.15.3:0/4156099597 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa534095240 con 0x560075f79710 2023-09-10T01:12:43.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.122+0000 7fa55d640b80 10 client.17742 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:43.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.122+0000 7fa55d640b80 10 client.17742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:43.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.122+0000 7fa55d640b80 20 client.17742 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:43.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.122+0000 7fa55d640b80 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075f93380 0x560075f95770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.122+0000 7fa55d640b80 4 client.17742 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:12:43.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.122+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560075ee08a0 con 0x560075f93380 2023-09-10T01:12:43.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.122+0000 7fa542ffd700 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075f93380 0x560075f95770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.123+0000 7fa522ffd700 10 client.17742 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:43.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.123+0000 7fa542ffd700 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075f93380 0x560075f95770 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c4315ed5f9f3a92 server_cookie=3be8e1db7ce55cb3 in_seq=0 out_seq=0 2023-09-10T01:12:43.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.124+0000 7fa522ffd700 1 -- 172.21.15.3:0/4156099597 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560075ee08a0 con 0x560075f93380 2023-09-10T01:12:43.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.124+0000 7fa522ffd700 10 client.17742 handle_command_reply: tid=0 2023-09-10T01:12:43.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.124+0000 7fa55d640b80 1 client.17742 shutdown 2023-09-10T01:12:43.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 msgr2=0x560075f83790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f83790 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa5340d44a0 tx=0x7fa5340d4f80 comp rx=0 tx=0).stop 2023-09-10T01:12:43.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 shutdown_connections 2023-09-10T01:12:43.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075f93380 0x560075f95770 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f83e40 0x560075f80f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f83790 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 --2- 172.21.15.3:0/4156099597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 0x560075f809a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 >> 172.21.15.3:0/4156099597 conn(0x560075eee090 msgr2=0x560075f747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:43.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.125+0000 7fa522ffd700 0 client.17742 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:43.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.126+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 shutdown_connections 2023-09-10T01:12:43.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.127+0000 7fa55d640b80 1 -- 172.21.15.3:0/4156099597 wait complete. 2023-09-10T01:12:43.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.128+0000 7fa55d640b80 20 client.17742 trim_cache size 0 max 16384 2023-09-10T01:12:43.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.165+0000 7fa55d640b80 1 Processor -- start 2023-09-10T01:12:43.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.165+0000 7fa55d640b80 1 -- start start 2023-09-10T01:12:43.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.166+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f8aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.166+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f7a030 0x560075f746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.166+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f7bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.166+0000 7fa55d640b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075f756e0 con 0x560075f79710 2023-09-10T01:12:43.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.166+0000 7fa55d640b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075f7b210 con 0x560075f7a030 2023-09-10T01:12:43.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.166+0000 7fa55d640b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075f8f8f0 con 0x560075f83e40 2023-09-10T01:12:43.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.167+0000 7fa543fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f7bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.167+0000 7fa543fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f7bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:40516/0 (socket says 172.21.15.3:40516) 2023-09-10T01:12:43.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.167+0000 7fa543fff700 1 -- 172.21.15.3:0/3957984612 learned_addr learned my addr 172.21.15.3:0/3957984612 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:43.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.167+0000 7fa542ffd700 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f7a030 0x560075f746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.168+0000 7fa543fff700 1 -- 172.21.15.3:0/3957984612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f7a030 msgr2=0x560075f746f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.168+0000 7fa543fff700 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f7a030 0x560075f746f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.168+0000 7fa543fff700 1 -- 172.21.15.3:0/3957984612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 msgr2=0x560075f8aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.168+0000 7fa543fff700 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f8aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.168+0000 7fa543fff700 1 -- 172.21.15.3:0/3957984612 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560075d1e250 con 0x560075f83e40 2023-09-10T01:12:43.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.170+0000 7fa543fff700 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f7bc80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa53c010e30 tx=0x7fa53c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65e74988cfb8c6e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:43.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.173+0000 7fa52a7fc700 1 -- 172.21.15.3:0/3957984612 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c012dc0 con 0x560075f83e40 2023-09-10T01:12:43.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.173+0000 7fa52a7fc700 1 -- 172.21.15.3:0/3957984612 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa53c010e60 con 0x560075f83e40 2023-09-10T01:12:43.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.173+0000 7fa52a7fc700 1 -- 172.21.15.3:0/3957984612 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c0287c0 con 0x560075f83e40 2023-09-10T01:12:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.174+0000 7fa55d640b80 1 -- 172.21.15.3:0/3957984612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 msgr2=0x560075f7bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.174+0000 7fa55d640b80 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f7bc80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa53c010e30 tx=0x7fa53c013000 comp rx=0 tx=0).stop 2023-09-10T01:12:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.175+0000 7fa55d640b80 1 -- 172.21.15.3:0/3957984612 shutdown_connections 2023-09-10T01:12:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.175+0000 7fa55d640b80 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f83e40 0x560075f7bc80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.175+0000 7fa55d640b80 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f79710 0x560075f8aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.175+0000 7fa55d640b80 1 --2- 172.21.15.3:0/3957984612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f7a030 0x560075f746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.175+0000 7fa55d640b80 1 -- 172.21.15.3:0/3957984612 >> 172.21.15.3:0/3957984612 conn(0x560075eee090 msgr2=0x560075f8f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.175+0000 7fa55d640b80 1 -- 172.21.15.3:0/3957984612 shutdown_connections 2023-09-10T01:12:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.175+0000 7fa55d640b80 1 -- 172.21.15.3:0/3957984612 wait complete. 2023-09-10T01:12:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.176+0000 7fa55d640b80 1 -- start start 2023-09-10T01:12:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.176+0000 7fa55d640b80 1 Processor -- start 2023-09-10T01:12:43.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.177+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f89170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.177+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 0x560075f896b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.177+0000 7fa55d640b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f83e40 0x560075f89c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.177+0000 7fa55d640b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075ff8320 con 0x560075f83e40 2023-09-10T01:12:43.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.177+0000 7fa55d640b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075ff8490 con 0x560075f79710 2023-09-10T01:12:43.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.177+0000 7fa55d640b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560075ff8650 con 0x560075f7a030 2023-09-10T01:12:43.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.177+0000 7fa55d640b80 10 client.0 _ll_register_callbacks cb 0x560075db3a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:43.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.178+0000 7fa543fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f83e40 0x560075f89c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.178+0000 7fa543fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f83e40 0x560075f89c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38814/0 (socket says 172.21.15.3:38814) 2023-09-10T01:12:43.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.178+0000 7fa543fff700 1 -- 172.21.15.3:0/1559991418 learned_addr learned my addr 172.21.15.3:0/1559991418 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:43.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.179+0000 7fa5437fe700 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f89170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.180+0000 7fa542ffd700 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 0x560075f896b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.181+0000 7fa5437fe700 1 -- 172.21.15.3:0/1559991418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 msgr2=0x560075f896b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.181+0000 7fa5437fe700 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 0x560075f896b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.181+0000 7fa5437fe700 1 -- 172.21.15.3:0/1559991418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f83e40 msgr2=0x560075f89c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.181+0000 7fa5437fe700 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f83e40 0x560075f89c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.181+0000 7fa5437fe700 1 -- 172.21.15.3:0/1559991418 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c010ae0 con 0x560075f79710 2023-09-10T01:12:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.181+0000 7fa5437fe700 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f89170 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa534096e80 tx=0x7fa534098b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da416e1e92492f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.181+0000 7fa522ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.182+0000 7fa522ffd700 1 -- 172.21.15.3:0/1559991418 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534098670 con 0x560075f79710 2023-09-10T01:12:43.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.182+0000 7fa522ffd700 1 -- 172.21.15.3:0/1559991418 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa534013e40 con 0x560075f79710 2023-09-10T01:12:43.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.184+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5340d3040 con 0x560075f79710 2023-09-10T01:12:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.187+0000 7fa522ffd700 1 -- 172.21.15.3:0/1559991418 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5340d44d0 con 0x560075f79710 2023-09-10T01:12:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.194+0000 7fa55d640b80 10 client.27836 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:43.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.194+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560075cf62b0 con 0x560075f79710 2023-09-10T01:12:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.197+0000 7fa522ffd700 1 -- 172.21.15.3:0/1559991418 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5340d3040 con 0x560075f79710 2023-09-10T01:12:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.197+0000 7fa522ffd700 1 -- 172.21.15.3:0/1559991418 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa534098020 con 0x560075f79710 2023-09-10T01:12:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.199+0000 7fa55d640b80 10 client.27836 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.199+0000 7fa55d640b80 10 client.27836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.199+0000 7fa55d640b80 20 client.27836 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.199+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075ff9950 0x560075ffbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.199+0000 7fa55d640b80 4 client.27836 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:12:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.199+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560075ec4920 con 0x560075ff9950 2023-09-10T01:12:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.204+0000 7fa542ffd700 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075ff9950 0x560075ffbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.208+0000 7fa542ffd700 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075ff9950 0x560075ffbd40 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc224bcd6eb7e6aa server_cookie=120bbb26b8296270 in_seq=0 out_seq=0 2023-09-10T01:12:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.208+0000 7fa522ffd700 10 client.27836 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.210+0000 7fa522ffd700 1 -- 172.21.15.3:0/1559991418 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x560075ec4920 con 0x560075ff9950 2023-09-10T01:12:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.210+0000 7fa522ffd700 10 client.27836 handle_command_reply: tid=0 2023-09-10T01:12:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.210+0000 7fa55d640b80 1 client.27836 shutdown 2023-09-10T01:12:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.214+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 msgr2=0x560075f89170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.214+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f89170 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa534096e80 tx=0x7fa534098b80 comp rx=0 tx=0).stop 2023-09-10T01:12:43.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.215+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 shutdown_connections 2023-09-10T01:12:43.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.215+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560075ff9950 0x560075ffbd40 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.215+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560075f83e40 0x560075f89c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.215+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560075f79710 0x560075f89170 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.215+0000 7fa55d640b80 1 --2- 172.21.15.3:0/1559991418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560075f7a030 0x560075f896b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.215+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 >> 172.21.15.3:0/1559991418 conn(0x560075eee090 msgr2=0x560075f74670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:43.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.217+0000 7fa522ffd700 0 client.27836 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:43.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.219+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 shutdown_connections 2023-09-10T01:12:43.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.219+0000 7fa55d640b80 1 -- 172.21.15.3:0/1559991418 wait complete. 2023-09-10T01:12:43.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.221+0000 7fa55d640b80 20 client.27836 trim_cache size 0 max 16384 2023-09-10T01:12:43.222 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:12:43.222 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:12:43.223 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:12:43.223 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:12:43.223 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:12:43.223 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:12:43.223 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:12:43.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:12:43.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:12:43.224 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:12:43.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.223+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa52c0a09c0 msgr2=0x7fa52c0a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.223+0000 7fa549dff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa52c0a09c0 0x7fa52c0a2e80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00bc10 tx=0x7fa53c00b2c0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.223+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544107870 msgr2=0x7fa544113c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:43.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.223+0000 7fa549dff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544107870 0x7fa544113c20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa534007830 tx=0x7fa534007bf0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.224+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 shutdown_connections 2023-09-10T01:12:43.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.224+0000 7fa549dff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa52c0a09c0 0x7fa52c0a2e80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.224+0000 7fa549dff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa544111170 0x7fa544111620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.224+0000 7fa549dff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa544107870 0x7fa544113c20 secure :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa534007830 tx=0x7fa534007bf0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.224+0000 7fa549dff700 1 --2- 172.21.15.3:0/839574623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54410c410 0x7fa544110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:43.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.224+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 >> 172.21.15.3:0/839574623 conn(0x7fa544072090 msgr2=0x7fa54410bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:43.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.227+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 shutdown_connections 2023-09-10T01:12:43.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:12:43.227+0000 7fa549dff700 1 -- 172.21.15.3:0/839574623 wait complete. 2023-09-10T01:12:43.267 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:12:43.789 INFO:tasks.workunit.client.0.smithi003.stdout: 1 850705 8.86 MB/sec execute 385 sec latency 90.933 ms 2023-09-10T01:12:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:44 smithi003 ceph-mon[97654]: pgmap v294: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-09-10T01:12:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:44 smithi136 ceph-mon[133708]: pgmap v294: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-09-10T01:12:44.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:44 smithi092 ceph-mon[134067]: pgmap v294: 129 pgs: 129 active+clean; 255 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-09-10T01:12:44.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 852483 8.87 MB/sec execute 386 sec latency 21.198 ms 2023-09-10T01:12:45.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 853803 8.86 MB/sec execute 387 sec latency 18.463 ms 2023-09-10T01:12:46.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:46 smithi003 ceph-mon[97654]: pgmap v295: 129 pgs: 129 active+clean; 283 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 347 op/s 2023-09-10T01:12:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:46 smithi136 ceph-mon[133708]: pgmap v295: 129 pgs: 129 active+clean; 283 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 347 op/s 2023-09-10T01:12:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:46 smithi092 ceph-mon[134067]: pgmap v295: 129 pgs: 129 active+clean; 283 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.6 MiB/s wr, 347 op/s 2023-09-10T01:12:46.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 855902 8.88 MB/sec execute 388 sec latency 16.784 ms 2023-09-10T01:12:47.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 857038 8.87 MB/sec execute 389 sec latency 17.915 ms 2023-09-10T01:12:47.857 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 -- 172.21.15.3:0/1887210804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f40ff510 msgr2=0x7f18f410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:47.857 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 --2- 172.21.15.3:0/1887210804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f40ff510 0x7f18f410d460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00b0a0 tx=0x7f18ec00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 -- 172.21.15.3:0/1887210804 shutdown_connections 2023-09-10T01:12:47.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 --2- 172.21.15.3:0/1887210804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f40ff510 0x7f18f410d460 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.858 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 --2- 172.21.15.3:0/1887210804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18f4106650 0x7f18f4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 --2- 172.21.15.3:0/1887210804 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18f40feb50 0x7f18f40fefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 -- 172.21.15.3:0/1887210804 >> 172.21.15.3:0/1887210804 conn(0x7f18f40fac80 msgr2=0x7f18f40fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:47.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 -- 172.21.15.3:0/1887210804 shutdown_connections 2023-09-10T01:12:47.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.856+0000 7f18f9dff700 1 -- 172.21.15.3:0/1887210804 wait complete. 2023-09-10T01:12:47.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.857+0000 7f18f9dff700 1 Processor -- start 2023-09-10T01:12:47.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.857+0000 7f18f9dff700 1 -- start start 2023-09-10T01:12:47.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.857+0000 7f18f9dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18f40feb50 0x7f18f40ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.857+0000 7f18f9dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18f40ff510 0x7f18f4100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f9dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4106650 0x7f18f41a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f9dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4100aa0 con 0x7f18f40ff510 2023-09-10T01:12:47.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f9dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f41a4e00 con 0x7f18f4106650 2023-09-10T01:12:47.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f9dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f41a4f40 con 0x7f18f40feb50 2023-09-10T01:12:47.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4106650 0x7f18f41a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18f40ff510 0x7f18f4100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4106650 0x7f18f41a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54312/0 (socket says 172.21.15.3:54312) 2023-09-10T01:12:47.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 -- 172.21.15.3:0/204348226 learned_addr learned my addr 172.21.15.3:0/204348226 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:47.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f37fe700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18f40feb50 0x7f18f40ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 -- 172.21.15.3:0/204348226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18f40feb50 msgr2=0x7f18f40ffd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:47.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18f40feb50 0x7f18f40ffd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 -- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18f40ff510 msgr2=0x7f18f4100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:47.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18f40ff510 0x7f18f4100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 -- 172.21.15.3:0/204348226 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e4009520 con 0x7f18f4106650 2023-09-10T01:12:47.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.858+0000 7f18f3fff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4106650 0x7f18f41a48c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00b780 tx=0x7f18ec007990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:47.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.859+0000 7f18f0ff9700 1 -- 172.21.15.3:0/204348226 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec00e040 con 0x7f18f4106650 2023-09-10T01:12:47.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.859+0000 7f18f0ff9700 1 -- 172.21.15.3:0/204348226 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18ec01d470 con 0x7f18f4106650 2023-09-10T01:12:47.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.859+0000 7f18f0ff9700 1 -- 172.21.15.3:0/204348226 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0052a0 con 0x7f18f4106650 2023-09-10T01:12:47.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.859+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ec009d00 con 0x7f18f4106650 2023-09-10T01:12:47.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.859+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f41a53e0 con 0x7f18f4106650 2023-09-10T01:12:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.864+0000 7f18f0ff9700 1 -- 172.21.15.3:0/204348226 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f18ec01d5e0 con 0x7f18f4106650 2023-09-10T01:12:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 Processor -- start 2023-09-10T01:12:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 -- start start 2023-09-10T01:12:47.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d30580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d2a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d36f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556c836d0 con 0x564556d301a0 2023-09-10T01:12:47.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556c80ff0 con 0x564556d2acc0 2023-09-10T01:12:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.865+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556c7c8e0 con 0x564556d2a300 2023-09-10T01:12:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d2a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d2a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58654/0 (socket says 172.21.15.3:58654) 2023-09-10T01:12:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 -- 172.21.15.3:0/3601703853 learned_addr learned my addr 172.21.15.3:0/3601703853 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f0ff9700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18dc0a07b0 0x7f18dc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d30580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d36f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 -- 172.21.15.3:0/3601703853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 msgr2=0x564556d36f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.879 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d36f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 -- 172.21.15.3:0/3601703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 msgr2=0x564556d30580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d30580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 -- 172.21.15.3:0/3601703853 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564556aa8d80 con 0x564556d2a300 2023-09-10T01:12:47.880 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f0ff9700 1 -- 172.21.15.3:0/204348226 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f18ec0cd8c0 con 0x7f18f4106650 2023-09-10T01:12:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d30580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:12:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f3fff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18dc0a07b0 0x7f18dc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.866+0000 7f18f37fe700 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d2a780 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009f10 tx=0x7f18e4011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1565e94555fc576 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.867+0000 7f18d9ffb700 1 -- 172.21.15.3:0/3601703853 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4019630 con 0x564556d2a300 2023-09-10T01:12:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.867+0000 7f18d9ffb700 1 -- 172.21.15.3:0/3601703853 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e4019c50 con 0x564556d2a300 2023-09-10T01:12:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.867+0000 7f18d9ffb700 1 -- 172.21.15.3:0/3601703853 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400eca0 con 0x564556d2a300 2023-09-10T01:12:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.867+0000 7f18f3fff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18dc0a07b0 0x7f18dc0a2c70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009fd0 tx=0x7f18ec01df90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.867+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3601703853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 msgr2=0x564556d2a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.867+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d2a780 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009f10 tx=0x7f18e4011720 comp rx=0 tx=0).stop 2023-09-10T01:12:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3601703853 shutdown_connections 2023-09-10T01:12:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d36f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d30580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3601703853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d2a780 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3601703853 >> 172.21.15.3:0/3601703853 conn(0x564556c9f030 msgr2=0x564556d25560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3601703853 shutdown_connections 2023-09-10T01:12:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3601703853 wait complete. 2023-09-10T01:12:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 -- start start 2023-09-10T01:12:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.868+0000 7f190d6fdb80 1 Processor -- start 2023-09-10T01:12:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d36db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d33fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d34520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556c1ab10 con 0x564556d301a0 2023-09-10T01:12:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556c1a810 con 0x564556d2a300 2023-09-10T01:12:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556afe8c0 con 0x564556d2acc0 2023-09-10T01:12:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f190d6fdb80 10 client.0 _ll_register_callbacks cb 0x564556b64a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d36db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d34520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d36db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54320/0 (socket says 172.21.15.3:54320) 2023-09-10T01:12:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f18f2ffd700 1 -- 172.21.15.3:0/2068193749 learned_addr learned my addr 172.21.15.3:0/2068193749 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.869+0000 7f18f37fe700 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d33fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18f2ffd700 1 -- 172.21.15.3:0/2068193749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 msgr2=0x564556d33fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d33fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18f2ffd700 1 -- 172.21.15.3:0/2068193749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 msgr2=0x564556d34520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d34520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18f2ffd700 1 -- 172.21.15.3:0/2068193749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e4009b70 con 0x564556d2a300 2023-09-10T01:12:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d36db0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f18e8012f10 tx=0x7f18e80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=864ac2fe1366acb7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2068193749 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8016040 con 0x564556d2a300 2023-09-10T01:12:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564556bd4490 con 0x564556d2a300 2023-09-10T01:12:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2068193749 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f18e80120d0 con 0x564556d2a300 2023-09-10T01:12:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2068193749 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e8012290 con 0x564556d2a300 2023-09-10T01:12:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f190d6fdb80 10 client.27848 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.870+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564556a91c00 con 0x564556d2a300 2023-09-10T01:12:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2068193749 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564556bd4490 con 0x564556d2a300 2023-09-10T01:12:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2068193749 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f18e8005320 con 0x564556d2a300 2023-09-10T01:12:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f190d6fdb80 10 client.27848 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f190d6fdb80 10 client.27848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f190d6fdb80 20 client.27848 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d443e0 0x564556d46800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f190d6fdb80 4 client.27848 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:12:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564556c91840 con 0x564556d443e0 2023-09-10T01:12:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.871+0000 7f18f37fe700 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d443e0 0x564556d46800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.872+0000 7f18f37fe700 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d443e0 0x564556d46800 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8a1dbb9b23c7c58 server_cookie=548f5ffa4f904b3f in_seq=0 out_seq=0 2023-09-10T01:12:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.872+0000 7f18d2ffd700 10 client.27848 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.872+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2068193749 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564556c91840 con 0x564556d443e0 2023-09-10T01:12:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.872+0000 7f18d2ffd700 10 client.27848 handle_command_reply: tid=0 2023-09-10T01:12:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 client.27848 shutdown 2023-09-10T01:12:47.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 msgr2=0x564556d36db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d36db0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f18e8012f10 tx=0x7f18e80047c0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 shutdown_connections 2023-09-10T01:12:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d443e0 0x564556d46800 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d34520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d36db0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f18d2ffd700 0 client.27848 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2068193749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d33fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 >> 172.21.15.3:0/2068193749 conn(0x564556c9f030 msgr2=0x564556d247a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:47.896 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 shutdown_connections 2023-09-10T01:12:47.896 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2068193749 wait complete. 2023-09-10T01:12:47.896 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.873+0000 7f190d6fdb80 20 client.27848 trim_cache size 0 max 16384 2023-09-10T01:12:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.889+0000 7f190d6fdb80 1 Processor -- start 2023-09-10T01:12:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.889+0000 7f190d6fdb80 1 -- start start 2023-09-10T01:12:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.889+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d3b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.889+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d2cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d2efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556d41a90 con 0x564556d301a0 2023-09-10T01:12:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556d34b00 con 0x564556d2a300 2023-09-10T01:12:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556d34870 con 0x564556d2acc0 2023-09-10T01:12:47.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d3b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d2cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d2cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58676/0 (socket says 172.21.15.3:58676) 2023-09-10T01:12:47.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 -- 172.21.15.3:0/3786529371 learned_addr learned my addr 172.21.15.3:0/3786529371 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:47.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d3b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54328/0 (socket says 172.21.15.3:54328) 2023-09-10T01:12:47.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d2efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 -- 172.21.15.3:0/3786529371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 msgr2=0x564556d3b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d3b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 -- 172.21.15.3:0/3786529371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 msgr2=0x564556d2efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d2efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f37fe700 1 -- 172.21.15.3:0/3786529371 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564556acf1f0 con 0x564556d2acc0 2023-09-10T01:12:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.890+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d3b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:12:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.891+0000 7f18f37fe700 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d2cb60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009f10 tx=0x7f18e4009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa74c3a705e1bf4f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.891+0000 7f18da7fc700 1 -- 172.21.15.3:0/3786529371 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4029070 con 0x564556d2acc0 2023-09-10T01:12:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.891+0000 7f18da7fc700 1 -- 172.21.15.3:0/3786529371 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e400e4b0 con 0x564556d2acc0 2023-09-10T01:12:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.891+0000 7f18da7fc700 1 -- 172.21.15.3:0/3786529371 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4005710 con 0x564556d2acc0 2023-09-10T01:12:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.891+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3786529371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 msgr2=0x564556d2cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.891+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d2cb60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009f10 tx=0x7f18e4009e40 comp rx=0 tx=0).stop 2023-09-10T01:12:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3786529371 shutdown_connections 2023-09-10T01:12:47.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d2efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2a300 0x564556d3b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/3786529371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2acc0 0x564556d2cb60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3786529371 >> 172.21.15.3:0/3786529371 conn(0x564556c9f030 msgr2=0x564556d25090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:47.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3786529371 shutdown_connections 2023-09-10T01:12:47.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 -- 172.21.15.3:0/3786529371 wait complete. 2023-09-10T01:12:47.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 -- start start 2023-09-10T01:12:47.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.892+0000 7f190d6fdb80 1 Processor -- start 2023-09-10T01:12:47.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d3b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d37500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f190d6fdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d37a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556d37f80 con 0x564556d301a0 2023-09-10T01:12:47.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556d380f0 con 0x564556d2acc0 2023-09-10T01:12:47.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f190d6fdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564556d382b0 con 0x564556d2a300 2023-09-10T01:12:47.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f190d6fdb80 10 client.0 _ll_register_callbacks cb 0x7f18e8012c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:12:47.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d37500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d37500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54332/0 (socket says 172.21.15.3:54332) 2023-09-10T01:12:47.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 -- 172.21.15.3:0/2955367483 learned_addr learned my addr 172.21.15.3:0/2955367483 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:12:47.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d3b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d37a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 -- 172.21.15.3:0/2955367483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 msgr2=0x564556d3b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d3b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 -- 172.21.15.3:0/2955367483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 msgr2=0x564556d37a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:12:47.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d37a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f37fe700 1 -- 172.21.15.3:0/2955367483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564556a91c00 con 0x564556d2acc0 2023-09-10T01:12:47.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.893+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d3b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:12:47.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f18f37fe700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d37500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f18e4013010 tx=0x7f18e4011c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92d2737c21f079b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:12:47.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f18d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:12:47.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2955367483 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4029040 con 0x564556d2acc0 2023-09-10T01:12:47.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18e4019c50 con 0x564556d2acc0 2023-09-10T01:12:47.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2955367483 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f18e4019870 con 0x564556d2acc0 2023-09-10T01:12:47.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2955367483 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e4019a30 con 0x564556d2acc0 2023-09-10T01:12:47.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f190d6fdb80 10 client.27854 fetch_fsmap learned FSMap version 31 2023-09-10T01:12:47.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564556aa8d80 con 0x564556d2acc0 2023-09-10T01:12:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.894+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2955367483 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e4019c50 con 0x564556d2acc0 2023-09-10T01:12:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2955367483 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f18e4040070 con 0x564556d2acc0 2023-09-10T01:12:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f190d6fdb80 10 client.27854 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:12:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f190d6fdb80 10 client.27854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:12:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f190d6fdb80 20 client.27854 populate_metadata read hostname 'smithi003' 2023-09-10T01:12:47.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d99480 0x564556d9b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:12:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f190d6fdb80 4 client.27854 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:12:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564556c758c0 con 0x564556d99480 2023-09-10T01:12:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d99480 0x564556d9b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:12:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f18d2ffd700 10 client.27854 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.895+0000 7f18f2ffd700 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d99480 0x564556d9b8a0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6042e6780f7cc6f server_cookie=cf6a622ae4ffc6ad in_seq=0 out_seq=0 2023-09-10T01:12:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f18d2ffd700 1 -- 172.21.15.3:0/2955367483 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564556c758c0 con 0x564556d99480 2023-09-10T01:12:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f18d2ffd700 10 client.27854 handle_command_reply: tid=0 2023-09-10T01:12:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 client.27854 shutdown 2023-09-10T01:12:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 msgr2=0x564556d37500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d37500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f18e4013010 tx=0x7f18e4011c70 comp rx=0 tx=0).stop 2023-09-10T01:12:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 shutdown_connections 2023-09-10T01:12:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564556d99480 0x564556d9b8a0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564556d2a300 0x564556d3b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564556d301a0 0x564556d37a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f18d2ffd700 0 client.27854 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:12:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 --2- 172.21.15.3:0/2955367483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564556d2acc0 0x564556d37500 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 >> 172.21.15.3:0/2955367483 conn(0x564556c9f030 msgr2=0x564556d2efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 shutdown_connections 2023-09-10T01:12:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.996+0000 7f190d6fdb80 1 -- 172.21.15.3:0/2955367483 wait complete. 2023-09-10T01:12:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f190d6fdb80 20 client.27854 trim_cache size 0 max 16384 2023-09-10T01:12:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:12:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:12:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:12:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:12:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18dc0a07b0 msgr2=0x7f18dc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18dc0a07b0 0x7f18dc0a2c70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009fd0 tx=0x7f18ec01df90 comp rx=0 tx=0).stop 2023-09-10T01:12:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4106650 msgr2=0x7f18f41a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:12:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4106650 0x7f18f41a48c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00b780 tx=0x7f18ec007990 comp rx=0 tx=0).stop 2023-09-10T01:12:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 shutdown_connections 2023-09-10T01:12:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18dc0a07b0 0x7f18dc0a2c70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18f40feb50 0x7f18f40ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4106650 0x7f18f41a48c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 --2- 172.21.15.3:0/204348226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18f40ff510 0x7f18f4100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:12:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.997+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 >> 172.21.15.3:0/204348226 conn(0x7f18f40fac80 msgr2=0x7f18f4107480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:12:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.998+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 shutdown_connections 2023-09-10T01:12:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:12:47.998+0000 7f18f9dff700 1 -- 172.21.15.3:0/204348226 wait complete. 2023-09-10T01:12:48.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:48 smithi003 ceph-mon[97654]: pgmap v296: 129 pgs: 129 active+clean; 283 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.8 MiB/s wr, 274 op/s 2023-09-10T01:12:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:48 smithi136 ceph-mon[133708]: pgmap v296: 129 pgs: 129 active+clean; 283 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.8 MiB/s wr, 274 op/s 2023-09-10T01:12:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:48 smithi092 ceph-mon[134067]: pgmap v296: 129 pgs: 129 active+clean; 283 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.8 MiB/s wr, 274 op/s 2023-09-10T01:12:48.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 859002 8.87 MB/sec execute 390 sec latency 60.172 ms 2023-09-10T01:12:49.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 859888 8.86 MB/sec execute 391 sec latency 32.863 ms 2023-09-10T01:12:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:50 smithi136 ceph-mon[133708]: pgmap v297: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 416 op/s 2023-09-10T01:12:50.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:50 smithi092 ceph-mon[134067]: pgmap v297: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 416 op/s 2023-09-10T01:12:50.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 861657 8.85 MB/sec execute 392 sec latency 18.996 ms 2023-09-10T01:12:50.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:50 smithi003 ceph-mon[97654]: pgmap v297: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.5 MiB/s wr, 416 op/s 2023-09-10T01:12:51.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 863293 8.86 MB/sec execute 393 sec latency 12.111 ms 2023-09-10T01:12:52.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:52 smithi136 ceph-mon[133708]: pgmap v298: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 295 op/s 2023-09-10T01:12:52.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:52 smithi092 ceph-mon[134067]: pgmap v298: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 295 op/s 2023-09-10T01:12:52.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 865091 8.86 MB/sec execute 394 sec latency 18.438 ms 2023-09-10T01:12:52.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:52 smithi003 ceph-mon[97654]: pgmap v298: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 295 op/s 2023-09-10T01:12:53.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:53 smithi136 ceph-mon[133708]: pgmap v299: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 295 op/s 2023-09-10T01:12:53.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:53 smithi092 ceph-mon[134067]: pgmap v299: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 295 op/s 2023-09-10T01:12:53.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 866712 8.86 MB/sec execute 395 sec latency 31.721 ms 2023-09-10T01:12:53.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:53 smithi003 ceph-mon[97654]: pgmap v299: 129 pgs: 129 active+clean; 254 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 295 op/s 2023-09-10T01:12:54.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 867887 8.86 MB/sec execute 396 sec latency 49.246 ms 2023-09-10T01:12:55.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 870075 8.87 MB/sec execute 397 sec latency 16.701 ms 2023-09-10T01:12:56.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:56 smithi136 ceph-mon[133708]: pgmap v300: 129 pgs: 129 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 380 op/s 2023-09-10T01:12:56.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:56 smithi092 ceph-mon[134067]: pgmap v300: 129 pgs: 129 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 380 op/s 2023-09-10T01:12:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:56 smithi003 ceph-mon[97654]: pgmap v300: 129 pgs: 129 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.5 MiB/s wr, 380 op/s 2023-09-10T01:12:56.790 INFO:tasks.workunit.client.0.smithi003.stdout: 1 871270 8.86 MB/sec execute 398 sec latency 18.634 ms 2023-09-10T01:12:57.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 873527 8.87 MB/sec execute 399 sec latency 16.143 ms 2023-09-10T01:12:58.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:12:58 smithi136 ceph-mon[133708]: pgmap v301: 129 pgs: 129 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 226 op/s 2023-09-10T01:12:58.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:12:58 smithi092 ceph-mon[134067]: pgmap v301: 129 pgs: 129 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 226 op/s 2023-09-10T01:12:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:12:58 smithi003 ceph-mon[97654]: pgmap v301: 129 pgs: 129 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 226 op/s 2023-09-10T01:12:58.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 874572 8.85 MB/sec execute 400 sec latency 13.886 ms 2023-09-10T01:12:59.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 876591 8.86 MB/sec execute 401 sec latency 68.039 ms 2023-09-10T01:13:00.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:00 smithi092 ceph-mon[134067]: pgmap v302: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 300 op/s 2023-09-10T01:13:00.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:00 smithi003 ceph-mon[97654]: pgmap v302: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 300 op/s 2023-09-10T01:13:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:00 smithi136 ceph-mon[133708]: pgmap v302: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 300 op/s 2023-09-10T01:13:00.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 877207 8.85 MB/sec execute 402 sec latency 272.183 ms 2023-09-10T01:13:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 -- 172.21.15.3:0/3761013708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a70100f80 msgr2=0x7f8a7010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 --2- 172.21.15.3:0/3761013708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a70100f80 0x7f8a7010d440 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009fc0 tx=0x7f8a6800b330 comp rx=0 tx=0).stop 2023-09-10T01:13:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 -- 172.21.15.3:0/3761013708 shutdown_connections 2023-09-10T01:13:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 --2- 172.21.15.3:0/3761013708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a70106680 0x7f8a70106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 --2- 172.21.15.3:0/3761013708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a70100f80 0x7f8a7010d440 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 --2- 172.21.15.3:0/3761013708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a701005c0 0x7f8a70100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 -- 172.21.15.3:0/3761013708 >> 172.21.15.3:0/3761013708 conn(0x7f8a700fc0b0 msgr2=0x7f8a700fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:01.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 -- 172.21.15.3:0/3761013708 shutdown_connections 2023-09-10T01:13:01.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.153+0000 7f8a783ff700 1 -- 172.21.15.3:0/3761013708 wait complete. 2023-09-10T01:13:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.154+0000 7f8a783ff700 1 Processor -- start 2023-09-10T01:13:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a783ff700 1 -- start start 2023-09-10T01:13:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a783ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a701005c0 0x7f8a7019c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a783ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a70100f80 0x7f8a7019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a783ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a70106680 0x7f8a701965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a783ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70196c20 con 0x7f8a701005c0 2023-09-10T01:13:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a783ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70196d90 con 0x7f8a70106680 2023-09-10T01:13:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a783ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70196f50 con 0x7f8a70100f80 2023-09-10T01:13:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7699c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a70106680 0x7f8a701965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7699c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a70106680 0x7f8a701965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39854/0 (socket says 172.21.15.3:39854) 2023-09-10T01:13:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7699c700 1 -- 172.21.15.3:0/32049928 learned_addr learned my addr 172.21.15.3:0/32049928 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7619b700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a701005c0 0x7f8a7019c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7599a700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a70100f80 0x7f8a7019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7699c700 1 -- 172.21.15.3:0/32049928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a70100f80 msgr2=0x7f8a7019ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7699c700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a70100f80 0x7f8a7019ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.155+0000 7f8a7699c700 1 -- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a701005c0 msgr2=0x7f8a7019c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a7699c700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a701005c0 0x7f8a7019c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a7699c700 1 -- 172.21.15.3:0/32049928 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a600095f0 con 0x7f8a70106680 2023-09-10T01:13:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a7599a700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a70100f80 0x7f8a7019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:13:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a7699c700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a70106680 0x7f8a701965b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009490 tx=0x7f8a680094c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a677fe700 1 -- 172.21.15.3:0/32049928 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800e040 con 0x7f8a70106680 2023-09-10T01:13:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a68009c40 con 0x7f8a70106680 2023-09-10T01:13:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a677fe700 1 -- 172.21.15.3:0/32049928 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a6800bbf0 con 0x7f8a70106680 2023-09-10T01:13:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a677fe700 1 -- 172.21.15.3:0/32049928 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a680052a0 con 0x7f8a70106680 2023-09-10T01:13:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.156+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a701a3350 con 0x7f8a70106680 2023-09-10T01:13:01.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.160+0000 7f8a677fe700 1 -- 172.21.15.3:0/32049928 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f8a6801d470 con 0x7f8a70106680 2023-09-10T01:13:01.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.161+0000 7f8a8bc5cb80 1 Processor -- start 2023-09-10T01:13:01.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.161+0000 7f8a8bc5cb80 1 -- start start 2023-09-10T01:13:01.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b553f1a0 0x55c7b553f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5539780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b5545f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b54926d0 con 0x55c7b553f1a0 2023-09-10T01:13:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b548fff0 con 0x55c7b5539cc0 2023-09-10T01:13:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b548b8e0 con 0x55c7b5539300 2023-09-10T01:13:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a7599a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b553f1a0 0x55c7b553f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a677fe700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8a5c0a07b0 0x7f8a5c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a7599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b5545f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a7599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b5545f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39862/0 (socket says 172.21.15.3:39862) 2023-09-10T01:13:01.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a7599a700 1 -- 172.21.15.3:0/222779188 learned_addr learned my addr 172.21.15.3:0/222779188 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:01.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a7619b700 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5539780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a677fe700 1 -- 172.21.15.3:0/32049928 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8a680cda60 con 0x7f8a70106680 2023-09-10T01:13:01.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a7599a700 1 -- 172.21.15.3:0/222779188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 msgr2=0x55c7b5539780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.162+0000 7f8a7599a700 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5539780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a7699c700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8a5c0a07b0 0x7f8a5c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a7599a700 1 -- 172.21.15.3:0/222779188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b553f1a0 msgr2=0x55c7b553f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a7599a700 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b553f1a0 0x55c7b553f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a7599a700 1 -- 172.21.15.3:0/222779188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7b52b7d80 con 0x55c7b5539cc0 2023-09-10T01:13:01.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a7599a700 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b5545f30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8a60014b60 tx=0x7f8a60009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ae933acfce16ec server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a64bfe700 1 -- 172.21.15.3:0/222779188 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a600142f0 con 0x55c7b5539cc0 2023-09-10T01:13:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a64bfe700 1 -- 172.21.15.3:0/222779188 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a6001a030 con 0x55c7b5539cc0 2023-09-10T01:13:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.163+0000 7f8a64bfe700 1 -- 172.21.15.3:0/222779188 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60005390 con 0x55c7b5539cc0 2023-09-10T01:13:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/222779188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 msgr2=0x55c7b5545f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a7699c700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8a5c0a07b0 0x7f8a5c0a2c70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a6809aae0 tx=0x7f8a6800b870 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b5545f30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8a60014b60 tx=0x7f8a60009310 comp rx=0 tx=0).stop 2023-09-10T01:13:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/222779188 shutdown_connections 2023-09-10T01:13:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b5545f30 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b553f1a0 0x55c7b553f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/222779188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5539780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/222779188 >> 172.21.15.3:0/222779188 conn(0x55c7b54ae030 msgr2=0x55c7b5534560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/222779188 shutdown_connections 2023-09-10T01:13:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/222779188 wait complete. 2023-09-10T01:13:01.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.164+0000 7f8a8bc5cb80 1 -- start start 2023-09-10T01:13:01.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 1 Processor -- start 2023-09-10T01:13:01.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5545da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b55430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b553f1a0 0x55c7b5543680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b5429b10 con 0x55c7b5539cc0 2023-09-10T01:13:01.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b5429810 con 0x55c7b553f1a0 2023-09-10T01:13:01.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b530d8c0 con 0x55c7b5539300 2023-09-10T01:13:01.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.165+0000 7f8a8bc5cb80 10 client.0 _ll_register_callbacks cb 0x55c7b5373a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:01.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b55430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b553f1a0 0x55c7b5543680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b55430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59596/0 (socket says 172.21.15.3:59596) 2023-09-10T01:13:01.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 -- 172.21.15.3:0/252903046 learned_addr learned my addr 172.21.15.3:0/252903046 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:01.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7599a700 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5545da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 -- 172.21.15.3:0/252903046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 msgr2=0x55c7b5545da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5545da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 -- 172.21.15.3:0/252903046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b553f1a0 msgr2=0x55c7b5543680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b553f1a0 0x55c7b5543680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 -- 172.21.15.3:0/252903046 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a60014810 con 0x55c7b5539cc0 2023-09-10T01:13:01.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a7619b700 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b55430c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0127f0 tx=0x7f8a6c012b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98b33151d64345a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:01.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.166+0000 7f8a59ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:13:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.168+0000 7f8a59ffb700 1 -- 172.21.15.3:0/252903046 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c017070 con 0x55c7b5539cc0 2023-09-10T01:13:01.183 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7b53e3490 con 0x55c7b5539cc0 2023-09-10T01:13:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a59ffb700 1 -- 172.21.15.3:0/252903046 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8a6c00f4e0 con 0x55c7b5539cc0 2023-09-10T01:13:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a59ffb700 1 -- 172.21.15.3:0/252903046 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a6c00f6a0 con 0x55c7b5539cc0 2023-09-10T01:13:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 10 client.17775 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c7b52a0c00 con 0x55c7b5539cc0 2023-09-10T01:13:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a59ffb700 1 -- 172.21.15.3:0/252903046 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c7b53e3490 con 0x55c7b5539cc0 2023-09-10T01:13:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a59ffb700 1 -- 172.21.15.3:0/252903046 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8a6c010870 con 0x55c7b5539cc0 2023-09-10T01:13:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 10 client.17775 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 10 client.17775 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 20 client.17775 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55578c0 0x55c7b5559ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 4 client.17775 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:13:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.169+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c7b54a0840 con 0x55c7b55578c0 2023-09-10T01:13:01.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.170+0000 7f8a7599a700 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55578c0 0x55c7b5559ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.170+0000 7f8a7599a700 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55578c0 0x55c7b5559ce0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c33f8b067485dc25 server_cookie=582ee6817d8446fc in_seq=0 out_seq=0 2023-09-10T01:13:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.170+0000 7f8a59ffb700 10 client.17775 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a59ffb700 1 -- 172.21.15.3:0/252903046 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c7b54a0840 con 0x55c7b55578c0 2023-09-10T01:13:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a59ffb700 10 client.17775 handle_command_reply: tid=0 2023-09-10T01:13:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 client.17775 shutdown 2023-09-10T01:13:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 msgr2=0x55c7b55430c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b55430c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0127f0 tx=0x7f8a6c012b00 comp rx=0 tx=0).stop 2023-09-10T01:13:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 shutdown_connections 2023-09-10T01:13:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55578c0 0x55c7b5559ce0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b5539300 0x55c7b5545da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b553f1a0 0x55c7b5543680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/252903046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b55430c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a59ffb700 0 client.17775 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 >> 172.21.15.3:0/252903046 conn(0x55c7b54ae030 msgr2=0x55c7b55337a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:01.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 shutdown_connections 2023-09-10T01:13:01.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/252903046 wait complete. 2023-09-10T01:13:01.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.171+0000 7f8a8bc5cb80 20 client.17775 trim_cache size 0 max 16384 2023-09-10T01:13:01.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 Processor -- start 2023-09-10T01:13:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 -- start start 2023-09-10T01:13:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539300 0x55c7b554e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b55459d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b553bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b54a0840 con 0x55c7b5539300 2023-09-10T01:13:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b53e3490 con 0x55c7b5539cc0 2023-09-10T01:13:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b552e8a0 con 0x55c7b553f1a0 2023-09-10T01:13:01.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a7619b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b55459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a7619b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b55459d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39890/0 (socket says 172.21.15.3:39890) 2023-09-10T01:13:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a7619b700 1 -- 172.21.15.3:0/1672693698 learned_addr learned my addr 172.21.15.3:0/1672693698 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.187+0000 7f8a7599a700 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539300 0x55c7b554e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7599a700 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b553bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7619b700 1 -- 172.21.15.3:0/1672693698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 msgr2=0x55c7b553bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7619b700 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b553bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7619b700 1 -- 172.21.15.3:0/1672693698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539300 msgr2=0x55c7b554e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7619b700 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539300 0x55c7b554e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7619b700 1 -- 172.21.15.3:0/1672693698 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7b52de1f0 con 0x55c7b5539cc0 2023-09-10T01:13:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7599a700 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539300 0x55c7b554e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a7619b700 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b55459d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009750 tx=0x7f8a6c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=685973d3f0236f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a653ff700 1 -- 172.21.15.3:0/1672693698 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c017070 con 0x55c7b5539cc0 2023-09-10T01:13:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a653ff700 1 -- 172.21.15.3:0/1672693698 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a6c018040 con 0x55c7b5539cc0 2023-09-10T01:13:01.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.188+0000 7f8a653ff700 1 -- 172.21.15.3:0/1672693698 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0038e0 con 0x55c7b5539cc0 2023-09-10T01:13:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/1672693698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 msgr2=0x55c7b55459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b55459d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009750 tx=0x7f8a6c009780 comp rx=0 tx=0).stop 2023-09-10T01:13:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/1672693698 shutdown_connections 2023-09-10T01:13:01.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b553bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539300 0x55c7b554e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/1672693698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539cc0 0x55c7b55459d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/1672693698 >> 172.21.15.3:0/1672693698 conn(0x55c7b54ae030 msgr2=0x55c7b553c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/1672693698 shutdown_connections 2023-09-10T01:13:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/1672693698 wait complete. 2023-09-10T01:13:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.189+0000 7f8a8bc5cb80 1 -- start start 2023-09-10T01:13:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 1 Processor -- start 2023-09-10T01:13:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539300 0x55c7b5542980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b5542ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b5541890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b5541dd0 con 0x55c7b5539cc0 2023-09-10T01:13:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b5541f40 con 0x55c7b5539300 2023-09-10T01:13:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7b55420b0 con 0x55c7b553f1a0 2023-09-10T01:13:01.203 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a8bc5cb80 10 client.0 _ll_register_callbacks cb 0x7f8a6c009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a7599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539300 0x55c7b5542980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a7599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539300 0x55c7b5542980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39898/0 (socket says 172.21.15.3:39898) 2023-09-10T01:13:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a7599a700 1 -- 172.21.15.3:0/2313397546 learned_addr learned my addr 172.21.15.3:0/2313397546 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a7619b700 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b5542ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.190+0000 7f8a7599a700 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b5541890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a7599a700 1 -- 172.21.15.3:0/2313397546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 msgr2=0x55c7b5541890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a7599a700 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b5541890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a7599a700 1 -- 172.21.15.3:0/2313397546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 msgr2=0x55c7b5542ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a7599a700 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b5542ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a7599a700 1 -- 172.21.15.3:0/2313397546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c7b52a0c00 con 0x55c7b5539300 2023-09-10T01:13:01.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a7599a700 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539300 0x55c7b5542980 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8a60014b00 tx=0x7f8a60009d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9511c715b00632d0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a59ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:13:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a59ffb700 1 -- 172.21.15.3:0/2313397546 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60037070 con 0x55c7b5539300 2023-09-10T01:13:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a6001a030 con 0x55c7b5539300 2023-09-10T01:13:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a59ffb700 1 -- 172.21.15.3:0/2313397546 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8a60000e40 con 0x55c7b5539300 2023-09-10T01:13:01.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a59ffb700 1 -- 172.21.15.3:0/2313397546 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a600314b0 con 0x55c7b5539300 2023-09-10T01:13:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a8bc5cb80 10 client.27872 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.191+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c7b52b6250 con 0x55c7b5539300 2023-09-10T01:13:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a59ffb700 1 -- 172.21.15.3:0/2313397546 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a6001a030 con 0x55c7b5539300 2023-09-10T01:13:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a59ffb700 1 -- 172.21.15.3:0/2313397546 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8a6003a020 con 0x55c7b5539300 2023-09-10T01:13:01.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a8bc5cb80 10 client.27872 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a8bc5cb80 10 client.27872 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a8bc5cb80 20 client.27872 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55ac460 0x55c7b55ae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a8bc5cb80 4 client.27872 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:13:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c7b54848c0 con 0x55c7b55ac460 2023-09-10T01:13:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.192+0000 7f8a7619b700 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55ac460 0x55c7b55ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.193+0000 7f8a7619b700 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55ac460 0x55c7b55ae850 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2071984f81f22d6 server_cookie=9d43f677bcc7138c in_seq=0 out_seq=0 2023-09-10T01:13:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.193+0000 7f8a59ffb700 10 client.27872 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:01.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a59ffb700 1 -- 172.21.15.3:0/2313397546 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c7b54848c0 con 0x55c7b55ac460 2023-09-10T01:13:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a59ffb700 10 client.27872 handle_command_reply: tid=0 2023-09-10T01:13:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a8bc5cb80 1 client.27872 shutdown 2023-09-10T01:13:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539300 msgr2=0x55c7b5542980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539300 0x55c7b5542980 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8a60014b00 tx=0x7f8a60009d80 comp rx=0 tx=0).stop 2023-09-10T01:13:01.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 shutdown_connections 2023-09-10T01:13:01.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c7b55ac460 0x55c7b55ae850 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.305+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c7b553f1a0 0x55c7b5541890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a59ffb700 0 client.27872 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7b5539300 0x55c7b5542980 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a8bc5cb80 1 --2- 172.21.15.3:0/2313397546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c7b5539cc0 0x55c7b5542ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 >> 172.21.15.3:0/2313397546 conn(0x55c7b54ae030 msgr2=0x55c7b5553a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 shutdown_connections 2023-09-10T01:13:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a8bc5cb80 1 -- 172.21.15.3:0/2313397546 wait complete. 2023-09-10T01:13:01.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a8bc5cb80 20 client.27872 trim_cache size 0 max 16384 2023-09-10T01:13:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:13:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:13:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:13:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:13:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.306+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8a5c0a07b0 msgr2=0x7f8a5c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8a5c0a07b0 0x7f8a5c0a2c70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a6809aae0 tx=0x7f8a6800b870 comp rx=0 tx=0).stop 2023-09-10T01:13:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a70106680 msgr2=0x7f8a701965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a70106680 0x7f8a701965b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009490 tx=0x7f8a680094c0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 shutdown_connections 2023-09-10T01:13:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8a5c0a07b0 0x7f8a5c0a2c70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a70106680 0x7f8a701965b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a701005c0 0x7f8a7019c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 --2- 172.21.15.3:0/32049928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a70100f80 0x7f8a7019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 >> 172.21.15.3:0/32049928 conn(0x7f8a700fc0b0 msgr2=0x7f8a700fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 shutdown_connections 2023-09-10T01:13:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:01.307+0000 7f8a783ff700 1 -- 172.21.15.3:0/32049928 wait complete. 2023-09-10T01:13:01.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 877207 8.83 MB/sec execute 403 sec latency 1243.340 ms 2023-09-10T01:13:02.468 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:02 smithi092 ceph-mon[134067]: pgmap v303: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:02.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:02 smithi003 ceph-mon[97654]: pgmap v303: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:02 smithi136 ceph-mon[133708]: pgmap v303: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:02.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 877207 8.81 MB/sec execute 404 sec latency 2243.434 ms 2023-09-10T01:13:03.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 877207 8.79 MB/sec execute 405 sec latency 3243.558 ms 2023-09-10T01:13:04.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:04 smithi092 ceph-mon[134067]: pgmap v304: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:04 smithi003 ceph-mon[97654]: pgmap v304: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:04 smithi136 ceph-mon[133708]: pgmap v304: 129 pgs: 129 active+clean; 267 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:04.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 877207 8.76 MB/sec execute 406 sec latency 4243.639 ms 2023-09-10T01:13:05.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 877651 8.74 MB/sec execute 407 sec latency 4746.084 ms 2023-09-10T01:13:06.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:06 smithi092 ceph-mon[134067]: pgmap v305: 129 pgs: 129 active+clean; 262 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 230 op/s 2023-09-10T01:13:06.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:06 smithi003 ceph-mon[97654]: pgmap v305: 129 pgs: 129 active+clean; 262 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 230 op/s 2023-09-10T01:13:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:06 smithi136 ceph-mon[133708]: pgmap v305: 129 pgs: 129 active+clean; 262 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 230 op/s 2023-09-10T01:13:06.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 877960 8.73 MB/sec execute 408 sec latency 78.967 ms 2023-09-10T01:13:07.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 879497 8.72 MB/sec execute 409 sec latency 74.282 ms 2023-09-10T01:13:08.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:08 smithi092 ceph-mon[134067]: pgmap v306: 129 pgs: 129 active+clean; 262 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.1 MiB/s wr, 145 op/s 2023-09-10T01:13:08.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:08 smithi003 ceph-mon[97654]: pgmap v306: 129 pgs: 129 active+clean; 262 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.1 MiB/s wr, 145 op/s 2023-09-10T01:13:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:08 smithi136 ceph-mon[133708]: pgmap v306: 129 pgs: 129 active+clean; 262 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 1.1 MiB/s wr, 145 op/s 2023-09-10T01:13:08.791 INFO:tasks.workunit.client.0.smithi003.stdout: 1 881069 8.73 MB/sec execute 410 sec latency 11.992 ms 2023-09-10T01:13:09.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 881886 8.72 MB/sec execute 411 sec latency 90.852 ms 2023-09-10T01:13:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:10 smithi092 ceph-mon[134067]: pgmap v307: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 213 op/s 2023-09-10T01:13:10.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:10 smithi092 ceph-mon[134067]: osdmap e98: 12 total, 12 up, 12 in 2023-09-10T01:13:10.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:10 smithi003 ceph-mon[97654]: pgmap v307: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 213 op/s 2023-09-10T01:13:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:10 smithi003 ceph-mon[97654]: osdmap e98: 12 total, 12 up, 12 in 2023-09-10T01:13:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:10 smithi136 ceph-mon[133708]: pgmap v307: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 213 op/s 2023-09-10T01:13:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:10 smithi136 ceph-mon[133708]: osdmap e98: 12 total, 12 up, 12 in 2023-09-10T01:13:10.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 882931 8.70 MB/sec execute 412 sec latency 73.450 ms 2023-09-10T01:13:11.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 884495 8.71 MB/sec execute 413 sec latency 23.200 ms 2023-09-10T01:13:12.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:12 smithi092 ceph-mon[134067]: pgmap v309: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 986 KiB/s wr, 167 op/s 2023-09-10T01:13:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:12 smithi003 ceph-mon[97654]: pgmap v309: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 986 KiB/s wr, 167 op/s 2023-09-10T01:13:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:12 smithi136 ceph-mon[133708]: pgmap v309: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 986 KiB/s wr, 167 op/s 2023-09-10T01:13:12.796 INFO:tasks.workunit.client.0.smithi003.stdout: 1 886158 8.70 MB/sec execute 414 sec latency 36.938 ms 2023-09-10T01:13:13.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:13:13.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 -- 172.21.15.3:0/1894294572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 msgr2=0x7f314410d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 --2- 172.21.15.3:0/1894294572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314410d510 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b0a0 tx=0x7f313c00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 -- 172.21.15.3:0/1894294572 shutdown_connections 2023-09-10T01:13:13.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 --2- 172.21.15.3:0/1894294572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314410d510 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 --2- 172.21.15.3:0/1894294572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3144106650 0x7f3144106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 --2- 172.21.15.3:0/1894294572 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3144100640 0x7f3144100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 -- 172.21.15.3:0/1894294572 >> 172.21.15.3:0/1894294572 conn(0x7f31440fc170 msgr2=0x7f31440fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 -- 172.21.15.3:0/1894294572 shutdown_connections 2023-09-10T01:13:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.502+0000 7f3149fff700 1 -- 172.21.15.3:0/1894294572 wait complete. 2023-09-10T01:13:13.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.503+0000 7f3149fff700 1 Processor -- start 2023-09-10T01:13:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.503+0000 7f3149fff700 1 -- start start 2023-09-10T01:13:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.503+0000 7f3149fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3144100640 0x7f314419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.503+0000 7f3149fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.503+0000 7f3149fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3144106650 0x7f3144198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3149fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3144198e40 con 0x7f3144100640 2023-09-10T01:13:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3149fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3144198fb0 con 0x7f3144101000 2023-09-10T01:13:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3149fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3144199170 con 0x7f3144106650 2023-09-10T01:13:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f31437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3144100640 0x7f314419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3142ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3142ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314419ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55860/0 (socket says 172.21.15.3:55860) 2023-09-10T01:13:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3142ffd700 1 -- 172.21.15.3:0/1096432475 learned_addr learned my addr 172.21.15.3:0/1096432475 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:13.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3143fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3144106650 0x7f3144198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f31437fe700 1 -- 172.21.15.3:0/1096432475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3144106650 msgr2=0x7f3144198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f31437fe700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3144106650 0x7f3144198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f31437fe700 1 -- 172.21.15.3:0/1096432475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 msgr2=0x7f314419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f31437fe700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f31437fe700 1 -- 172.21.15.3:0/1096432475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c009d00 con 0x7f3144100640 2023-09-10T01:13:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f3142ffd700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.504+0000 7f31437fe700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3144100640 0x7f314419e690 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f313400e9f0 tx=0x7f313400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.505+0000 7f3140ff9700 1 -- 172.21.15.3:0/1096432475 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313400c9c0 con 0x7f3144100640 2023-09-10T01:13:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.505+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3144199450 con 0x7f3144100640 2023-09-10T01:13:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.505+0000 7f3140ff9700 1 -- 172.21.15.3:0/1096432475 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3134010430 con 0x7f3144100640 2023-09-10T01:13:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.505+0000 7f3140ff9700 1 -- 172.21.15.3:0/1096432475 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134018680 con 0x7f3144100640 2023-09-10T01:13:13.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.505+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31441a5660 con 0x7f3144100640 2023-09-10T01:13:13.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.508+0000 7f3140ff9700 1 -- 172.21.15.3:0/1096432475 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3134010860 con 0x7f3144100640 2023-09-10T01:13:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.509+0000 7f315d831b80 1 Processor -- start 2023-09-10T01:13:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.509+0000 7f3140ff9700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f312c0a07b0 0x7f312c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.509+0000 7f315d831b80 1 -- start start 2023-09-10T01:13:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.509+0000 7f3140ff9700 1 -- 172.21.15.3:0/1096432475 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f3134014070 con 0x7f3144100640 2023-09-10T01:13:13.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f3143fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f312c0a07b0 0x7f312c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f315d831b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fb710 0x55750e9fbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f315d831b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fc030 0x55750ea05900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f315d831b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750ea05e40 0x55750ea08230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f315d831b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750e954730 con 0x55750e9fb710 2023-09-10T01:13:13.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f315d831b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750e952050 con 0x55750ea05e40 2023-09-10T01:13:13.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f315d831b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750e94d940 con 0x55750e9fc030 2023-09-10T01:13:13.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f31437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fb710 0x55750e9fbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f31437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fb710 0x55750e9fbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45934/0 (socket says 172.21.15.3:45934) 2023-09-10T01:13:13.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f3143fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750ea05e40 0x55750ea08230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f31437fe700 1 -- 172.21.15.3:0/863129226 learned_addr learned my addr 172.21.15.3:0/863129226 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:13.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.510+0000 7f3142ffd700 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fc030 0x55750ea05900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f31437fe700 1 -- 172.21.15.3:0/863129226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fc030 msgr2=0x55750ea05900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f31437fe700 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fc030 0x55750ea05900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f31437fe700 1 -- 172.21.15.3:0/863129226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750ea05e40 msgr2=0x55750ea08230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f31437fe700 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750ea05e40 0x55750ea08230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f31437fe700 1 -- 172.21.15.3:0/863129226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55750e779de0 con 0x55750e9fb710 2023-09-10T01:13:13.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f3143fff700 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750ea05e40 0x55750ea08230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:13.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f3143fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f312c0a07b0 0x7f312c0a2c70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f314419f320 tx=0x7f313c00bec0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:13.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f31437fe700 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fb710 0x55750e9fbaf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3134009230 tx=0x7f3134098e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d27da37b24bbd6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:13.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f3129ffb700 1 -- 172.21.15.3:0/863129226 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313409d120 con 0x55750e9fb710 2023-09-10T01:13:13.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f3129ffb700 1 -- 172.21.15.3:0/863129226 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f313409d320 con 0x55750e9fb710 2023-09-10T01:13:13.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.511+0000 7f3129ffb700 1 -- 172.21.15.3:0/863129226 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31340cc170 con 0x55750e9fb710 2023-09-10T01:13:13.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 -- 172.21.15.3:0/863129226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fb710 msgr2=0x55750e9fbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fb710 0x55750e9fbaf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3134009230 tx=0x7f3134098e50 comp rx=0 tx=0).stop 2023-09-10T01:13:13.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 -- 172.21.15.3:0/863129226 shutdown_connections 2023-09-10T01:13:13.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750ea05e40 0x55750ea08230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fb710 0x55750e9fbaf0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 --2- 172.21.15.3:0/863129226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fc030 0x55750ea05900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 -- 172.21.15.3:0/863129226 >> 172.21.15.3:0/863129226 conn(0x55750e970090 msgr2=0x55750e9f6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:13.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 -- 172.21.15.3:0/863129226 shutdown_connections 2023-09-10T01:13:13.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.512+0000 7f315d831b80 1 -- 172.21.15.3:0/863129226 wait complete. 2023-09-10T01:13:13.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 -- start start 2023-09-10T01:13:13.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 Processor -- start 2023-09-10T01:13:13.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea02b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea03060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea035a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750e8ebb70 con 0x55750e9fc030 2023-09-10T01:13:13.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750e8eb870 con 0x55750e9fb710 2023-09-10T01:13:13.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750e7cf920 con 0x55750ea05e40 2023-09-10T01:13:13.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.513+0000 7f315d831b80 10 client.0 _ll_register_callbacks cb 0x55750e835a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:13.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3142ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea03060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f31437fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea02b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3143fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3142ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea03060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45946/0 (socket says 172.21.15.3:45946) 2023-09-10T01:13:13.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3142ffd700 1 -- 172.21.15.3:0/4161933882 learned_addr learned my addr 172.21.15.3:0/4161933882 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:13.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3143fff700 1 -- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 msgr2=0x55750ea02b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3143fff700 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea02b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3143fff700 1 -- 172.21.15.3:0/4161933882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 msgr2=0x55750ea03060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3143fff700 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea03060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3143fff700 1 -- 172.21.15.3:0/4161933882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31441a5660 con 0x55750ea05e40 2023-09-10T01:13:13.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f31437fe700 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea02b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:13.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3142ffd700 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea03060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:13.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3143fff700 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea035a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f313c0062a0 tx=0x7f313c00b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3f2fcbb3bcb9d7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:13.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.514+0000 7f3122ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:13:13.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f3122ffd700 1 -- 172.21.15.3:0/4161933882 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c00e040 con 0x55750ea05e40 2023-09-10T01:13:13.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55750e8a54f0 con 0x55750ea05e40 2023-09-10T01:13:13.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f3122ffd700 1 -- 172.21.15.3:0/4161933882 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f313c020ab0 con 0x55750ea05e40 2023-09-10T01:13:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f3122ffd700 1 -- 172.21.15.3:0/4161933882 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f313c020c70 con 0x55750ea05e40 2023-09-10T01:13:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f315d831b80 10 client.27778 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55750e762c60 con 0x55750ea05e40 2023-09-10T01:13:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f3122ffd700 1 -- 172.21.15.3:0/4161933882 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55750e8a54f0 con 0x55750ea05e40 2023-09-10T01:13:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.515+0000 7f3122ffd700 1 -- 172.21.15.3:0/4161933882 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f313c01cd00 con 0x55750ea05e40 2023-09-10T01:13:13.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f315d831b80 10 client.27778 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:13.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f315d831b80 10 client.27778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:13.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f315d831b80 20 client.27778 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:13.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f315d831b80 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea158d0 0x55750ea17cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f315d831b80 4 client.27778 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:13:13.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55750e9628a0 con 0x55750ea158d0 2023-09-10T01:13:13.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f31437fe700 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea158d0 0x55750ea17cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f31437fe700 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea158d0 0x55750ea17cf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6e9d2b0066ac648 server_cookie=31540d13c4fdb9ae in_seq=0 out_seq=0 2023-09-10T01:13:13.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.516+0000 7f3122ffd700 10 client.27778 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:13.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f3122ffd700 1 -- 172.21.15.3:0/4161933882 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55750e9628a0 con 0x55750ea158d0 2023-09-10T01:13:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f3122ffd700 10 client.27778 handle_command_reply: tid=0 2023-09-10T01:13:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 client.27778 shutdown 2023-09-10T01:13:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 msgr2=0x55750ea035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea035a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f313c0062a0 tx=0x7f313c00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 shutdown_connections 2023-09-10T01:13:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea158d0 0x55750ea17cf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea035a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea02b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 --2- 172.21.15.3:0/4161933882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea03060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 >> 172.21.15.3:0/4161933882 conn(0x55750e970090 msgr2=0x55750e9f67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:13.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f3122ffd700 0 client.27778 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:13.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 shutdown_connections 2023-09-10T01:13:13.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.517+0000 7f315d831b80 1 -- 172.21.15.3:0/4161933882 wait complete. 2023-09-10T01:13:13.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.518+0000 7f315d831b80 20 client.27778 trim_cache size 0 max 16384 2023-09-10T01:13:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 Processor -- start 2023-09-10T01:13:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 -- start start 2023-09-10T01:13:13.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fb710 0x55750ea0c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fc030 0x55750e9ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750ea05e40 0x55750ea002d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750ea0ce60 con 0x55750ea05e40 2023-09-10T01:13:13.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750ea17cc0 con 0x55750e9fc030 2023-09-10T01:13:13.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.536+0000 7f315d831b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750ea02090 con 0x55750e9fb710 2023-09-10T01:13:13.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fb710 0x55750ea0c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fb710 0x55750ea0c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38768/0 (socket says 172.21.15.3:38768) 2023-09-10T01:13:13.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 -- 172.21.15.3:0/3952688719 learned_addr learned my addr 172.21.15.3:0/3952688719 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:13.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f3142ffd700 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fc030 0x55750e9ffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f3143fff700 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750ea05e40 0x55750ea002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 -- 172.21.15.3:0/3952688719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fc030 msgr2=0x55750e9ffd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fc030 0x55750e9ffd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 -- 172.21.15.3:0/3952688719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750ea05e40 msgr2=0x55750ea002d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750ea05e40 0x55750ea002d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 -- 172.21.15.3:0/3952688719 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55750e7a0250 con 0x55750e9fb710 2023-09-10T01:13:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f31437fe700 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fb710 0x55750ea0c920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f31340959f0 tx=0x7f3134095960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=446c8bdfd2f519e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:13.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.537+0000 7f312a7fc700 1 -- 172.21.15.3:0/3952688719 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31340a1560 con 0x55750e9fb710 2023-09-10T01:13:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f312a7fc700 1 -- 172.21.15.3:0/3952688719 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31340840b0 con 0x55750e9fb710 2023-09-10T01:13:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f312a7fc700 1 -- 172.21.15.3:0/3952688719 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31340999b0 con 0x55750e9fb710 2023-09-10T01:13:13.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 -- 172.21.15.3:0/3952688719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fb710 msgr2=0x55750ea0c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fb710 0x55750ea0c920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f31340959f0 tx=0x7f3134095960 comp rx=0 tx=0).stop 2023-09-10T01:13:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 -- 172.21.15.3:0/3952688719 shutdown_connections 2023-09-10T01:13:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750e9fb710 0x55750ea0c920 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750ea05e40 0x55750ea002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 --2- 172.21.15.3:0/3952688719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fc030 0x55750e9ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 -- 172.21.15.3:0/3952688719 >> 172.21.15.3:0/3952688719 conn(0x55750e970090 msgr2=0x55750e9fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:13.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 -- 172.21.15.3:0/3952688719 shutdown_connections 2023-09-10T01:13:13.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.538+0000 7f315d831b80 1 -- 172.21.15.3:0/3952688719 wait complete. 2023-09-10T01:13:13.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.539+0000 7f315d831b80 1 -- start start 2023-09-10T01:13:13.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.539+0000 7f315d831b80 1 Processor -- start 2023-09-10T01:13:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f315d831b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea0c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f315d831b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea0ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f315d831b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea0b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f315d831b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750ea0b700 con 0x55750e9fc030 2023-09-10T01:13:13.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f315d831b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750ea0b870 con 0x55750e9fb710 2023-09-10T01:13:13.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f315d831b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55750ea69710 con 0x55750ea05e40 2023-09-10T01:13:13.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f315d831b80 10 client.0 _ll_register_callbacks cb 0x55750e835a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:13.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f31437fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea0c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f31437fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea0c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55902/0 (socket says 172.21.15.3:55902) 2023-09-10T01:13:13.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f31437fe700 1 -- 172.21.15.3:0/1531445125 learned_addr learned my addr 172.21.15.3:0/1531445125 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:13.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f3143fff700 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea0b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.540+0000 7f3142ffd700 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea0ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f31437fe700 1 -- 172.21.15.3:0/1531445125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 msgr2=0x55750ea0b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f31437fe700 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea0b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f31437fe700 1 -- 172.21.15.3:0/1531445125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 msgr2=0x55750ea0ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:13.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f31437fe700 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea0ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f31437fe700 1 -- 172.21.15.3:0/1531445125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55750e779de0 con 0x55750e9fb710 2023-09-10T01:13:13.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f31437fe700 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea0c840 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3134097840 tx=0x7f31340092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e42fb7c8e0da75a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:13.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f3122ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:13:13.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f3122ffd700 1 -- 172.21.15.3:0/1531445125 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31340a1560 con 0x55750e9fb710 2023-09-10T01:13:13.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f3122ffd700 1 -- 172.21.15.3:0/1531445125 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f31340cc170 con 0x55750e9fb710 2023-09-10T01:13:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f313409d320 con 0x55750e9fb710 2023-09-10T01:13:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f3122ffd700 1 -- 172.21.15.3:0/1531445125 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31340989e0 con 0x55750e9fb710 2023-09-10T01:13:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f315d831b80 10 client.27890 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.541+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55750e7782b0 con 0x55750e9fb710 2023-09-10T01:13:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f3122ffd700 1 -- 172.21.15.3:0/1531445125 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f313409d320 con 0x55750e9fb710 2023-09-10T01:13:13.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f3122ffd700 1 -- 172.21.15.3:0/1531445125 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3134099750 con 0x55750e9fb710 2023-09-10T01:13:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f315d831b80 10 client.27890 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f315d831b80 10 client.27890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f315d831b80 20 client.27890 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f315d831b80 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea6a870 0x55750ea6cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f315d831b80 4 client.27890 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:13:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55750e946920 con 0x55750ea6a870 2023-09-10T01:13:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.542+0000 7f3142ffd700 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea6a870 0x55750ea6cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.543+0000 7f3142ffd700 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea6a870 0x55750ea6cc60 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b27fd91d0536ce6 server_cookie=285205632077e446 in_seq=0 out_seq=0 2023-09-10T01:13:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.543+0000 7f3122ffd700 10 client.27890 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.543+0000 7f3122ffd700 1 -- 172.21.15.3:0/1531445125 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55750e946920 con 0x55750ea6a870 2023-09-10T01:13:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.543+0000 7f3122ffd700 10 client.27890 handle_command_reply: tid=0 2023-09-10T01:13:13.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.543+0000 7f315d831b80 1 client.27890 shutdown 2023-09-10T01:13:13.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.543+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 msgr2=0x55750ea0c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.543+0000 7f315d831b80 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea0c840 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3134097840 tx=0x7f31340092d0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 shutdown_connections 2023-09-10T01:13:13.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55750ea6a870 0x55750ea6cc60 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55750ea05e40 0x55750ea0b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55750e9fb710 0x55750ea0c840 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f3122ffd700 0 client.27890 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 --2- 172.21.15.3:0/1531445125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55750e9fc030 0x55750ea0ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 >> 172.21.15.3:0/1531445125 conn(0x55750e970090 msgr2=0x55750e9fa700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 shutdown_connections 2023-09-10T01:13:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 1 -- 172.21.15.3:0/1531445125 wait complete. 2023-09-10T01:13:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f315d831b80 20 client.27890 trim_cache size 0 max 16384 2023-09-10T01:13:13.557 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:13:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (1 inodes in the stack)", 2023-09-10T01:13:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:13:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:13:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:13:13.558 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:13:13.559 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:13:13.559 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:13:13.559 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:13:13.559 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:13:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f312c0a07b0 msgr2=0x7f312c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f3149fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f312c0a07b0 0x7f312c0a2c70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f314419f320 tx=0x7f313c00bec0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3144100640 msgr2=0x7f314419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.544+0000 7f3149fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3144100640 0x7f314419e690 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f313400e9f0 tx=0x7f313400ed00 comp rx=0 tx=0).stop 2023-09-10T01:13:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 shutdown_connections 2023-09-10T01:13:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f312c0a07b0 0x7f312c0a2c70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3144106650 0x7f3144198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3144100640 0x7f314419e690 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 --2- 172.21.15.3:0/1096432475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3144101000 0x7f314419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 >> 172.21.15.3:0/1096432475 conn(0x7f31440fc170 msgr2=0x7f314410ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 shutdown_connections 2023-09-10T01:13:13.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:13.545+0000 7f3149fff700 1 -- 172.21.15.3:0/1096432475 wait complete. 2023-09-10T01:13:13.565 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:13:13.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 887904 8.71 MB/sec execute 415 sec latency 100.361 ms 2023-09-10T01:13:14.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:14 smithi092 ceph-mon[134067]: pgmap v310: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 986 KiB/s wr, 167 op/s 2023-09-10T01:13:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.469+0000 7f7a235ff700 1 -- 172.21.15.3:0/1649474964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c1005c0 msgr2=0x7f7a1c100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.469+0000 7f7a235ff700 1 --2- 172.21.15.3:0/1649474964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c100a40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0099c0 tx=0x7f7a0c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.470+0000 7f7a235ff700 1 -- 172.21.15.3:0/1649474964 shutdown_connections 2023-09-10T01:13:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.470+0000 7f7a235ff700 1 --2- 172.21.15.3:0/1649474964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a1c100f80 0x7f7a1c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.470+0000 7f7a235ff700 1 --2- 172.21.15.3:0/1649474964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c106680 0x7f7a1c106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.470+0000 7f7a235ff700 1 --2- 172.21.15.3:0/1649474964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c100a40 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.470+0000 7f7a235ff700 1 -- 172.21.15.3:0/1649474964 >> 172.21.15.3:0/1649474964 conn(0x7f7a1c0fc0b0 msgr2=0x7f7a1c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.470+0000 7f7a235ff700 1 -- 172.21.15.3:0/1649474964 shutdown_connections 2023-09-10T01:13:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.470+0000 7f7a235ff700 1 -- 172.21.15.3:0/1649474964 wait complete. 2023-09-10T01:13:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.471+0000 7f7a235ff700 1 Processor -- start 2023-09-10T01:13:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.471+0000 7f7a235ff700 1 -- start start 2023-09-10T01:13:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a235ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a235ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c100f80 0x7f7a1c196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a235ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a1c106680 0x7f7a1c19afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a235ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c19b5e0 con 0x7f7a1c1005c0 2023-09-10T01:13:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a235ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c19b750 con 0x7f7a1c100f80 2023-09-10T01:13:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a235ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c19b910 con 0x7f7a1c106680 2023-09-10T01:13:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a2139b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a20b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c100f80 0x7f7a1c196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a2139b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45986/0 (socket says 172.21.15.3:45986) 2023-09-10T01:13:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a2139b700 1 -- 172.21.15.3:0/149910715 learned_addr learned my addr 172.21.15.3:0/149910715 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a20b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c100f80 0x7f7a1c196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55930/0 (socket says 172.21.15.3:55930) 2023-09-10T01:13:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.472+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a1c106680 0x7f7a1c19afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a2139b700 1 -- 172.21.15.3:0/149910715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a1c106680 msgr2=0x7f7a1c19afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a2139b700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a1c106680 0x7f7a1c19afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a2139b700 1 -- 172.21.15.3:0/149910715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c100f80 msgr2=0x7f7a1c196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a2139b700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c100f80 0x7f7a1c196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a2139b700 1 -- 172.21.15.3:0/149910715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c0095f0 con 0x7f7a1c1005c0 2023-09-10T01:13:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c100f80 0x7f7a1c196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a2139b700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c196600 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800e9a0 tx=0x7f7a1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.473+0000 7f7a127fc700 1 -- 172.21.15.3:0/149910715 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18009790 con 0x7f7a1c1005c0 2023-09-10T01:13:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.474+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1c19bbf0 con 0x7f7a1c1005c0 2023-09-10T01:13:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.474+0000 7f7a127fc700 1 -- 172.21.15.3:0/149910715 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a18010430 con 0x7f7a1c1005c0 2023-09-10T01:13:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.474+0000 7f7a127fc700 1 -- 172.21.15.3:0/149910715 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18018680 con 0x7f7a1c1005c0 2023-09-10T01:13:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.474+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c1a3430 con 0x7f7a1c1005c0 2023-09-10T01:13:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.477+0000 7f7a127fc700 1 -- 172.21.15.3:0/149910715 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7a180105a0 con 0x7f7a1c1005c0 2023-09-10T01:13:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.478+0000 7f7a127fc700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7a080a07b0 0x7f7a080a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.479+0000 7f7a127fc700 1 -- 172.21.15.3:0/149910715 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f7a18014070 con 0x7f7a1c1005c0 2023-09-10T01:13:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.479+0000 7f7a36e4fb80 1 Processor -- start 2023-09-10T01:13:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.479+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7a080a07b0 0x7f7a080a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.479+0000 7f7a36e4fb80 1 -- start start 2023-09-10T01:13:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.479+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d61580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.479+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7a080a07b0 0x7f7a080a2c70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009fd0 tx=0x7f7a0c005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.479+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d5b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d67f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78cb46d0 con 0x55af78d5b300 2023-09-10T01:13:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78cb1ff0 con 0x55af78d611a0 2023-09-10T01:13:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78cad8e0 con 0x55af78d5bcc0 2023-09-10T01:13:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d61580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d61580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55942/0 (socket says 172.21.15.3:55942) 2023-09-10T01:13:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 -- 172.21.15.3:0/3094105093 learned_addr learned my addr 172.21.15.3:0/3094105093 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d5b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d67f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 -- 172.21.15.3:0/3094105093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 msgr2=0x55af78d67f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d67f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 -- 172.21.15.3:0/3094105093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 msgr2=0x55af78d5b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d5b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 -- 172.21.15.3:0/3094105093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af78ad9d80 con 0x55af78d611a0 2023-09-10T01:13:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d5b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.480+0000 7f7a2139b700 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d61580 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7a18095fd0 tx=0x7f7a18098fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2474a75ebb5e6582 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a077fe700 1 -- 172.21.15.3:0/3094105093 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1809d170 con 0x55af78d611a0 2023-09-10T01:13:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a077fe700 1 -- 172.21.15.3:0/3094105093 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a1809d390 con 0x55af78d611a0 2023-09-10T01:13:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a077fe700 1 -- 172.21.15.3:0/3094105093 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180997c0 con 0x55af78d611a0 2023-09-10T01:13:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3094105093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 msgr2=0x55af78d61580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d61580 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7a18095fd0 tx=0x7f7a18098fb0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3094105093 shutdown_connections 2023-09-10T01:13:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d67f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d61580 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3094105093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d5b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3094105093 >> 172.21.15.3:0/3094105093 conn(0x55af78cd0030 msgr2=0x55af78d56560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.481+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3094105093 shutdown_connections 2023-09-10T01:13:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.482+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3094105093 wait complete. 2023-09-10T01:13:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.482+0000 7f7a36e4fb80 1 -- start start 2023-09-10T01:13:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.482+0000 7f7a36e4fb80 1 Processor -- start 2023-09-10T01:13:14.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5b300 0x55af78d6aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d611a0 0x55af78d65100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78c4bb10 con 0x55af78d611a0 2023-09-10T01:13:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78c4b810 con 0x55af78d5b300 2023-09-10T01:13:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78b2f8c0 con 0x55af78d5bcc0 2023-09-10T01:13:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a36e4fb80 10 client.0 _ll_register_callbacks cb 0x55af78b95a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a2139b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5b300 0x55af78d6aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a2139b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5b300 0x55af78d6aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55956/0 (socket says 172.21.15.3:55956) 2023-09-10T01:13:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a2139b700 1 -- 172.21.15.3:0/2984786898 learned_addr learned my addr 172.21.15.3:0/2984786898 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.483+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d611a0 0x55af78d65100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a2139b700 1 -- 172.21.15.3:0/2984786898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 msgr2=0x55af78d6b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a2139b700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a2139b700 1 -- 172.21.15.3:0/2984786898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d611a0 msgr2=0x55af78d65100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a2139b700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d611a0 0x55af78d65100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a2139b700 1 -- 172.21.15.3:0/2984786898 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c1a3430 con 0x55af78d5b300 2023-09-10T01:13:14.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d611a0 0x55af78d65100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:14.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a2139b700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5b300 0x55af78d6aeb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7a18099260 tx=0x7f7a18010bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7e28dfb1627455d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:14.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a04ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:13:14.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a04ff9700 1 -- 172.21.15.3:0/2984786898 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180a15d0 con 0x55af78d5b300 2023-09-10T01:13:14.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a04ff9700 1 -- 172.21.15.3:0/2984786898 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f7a18098a50 con 0x55af78d5b300 2023-09-10T01:13:14.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af78c05490 con 0x55af78d5b300 2023-09-10T01:13:14.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a04ff9700 1 -- 172.21.15.3:0/2984786898 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a18098c10 con 0x55af78d5b300 2023-09-10T01:13:14.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a36e4fb80 10 client.27902 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:14.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.484+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55af78ac2c00 con 0x55af78d5b300 2023-09-10T01:13:14.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a04ff9700 1 -- 172.21.15.3:0/2984786898 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55af78c05490 con 0x55af78d5b300 2023-09-10T01:13:14.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a04ff9700 1 -- 172.21.15.3:0/2984786898 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7a180b1b60 con 0x55af78d5b300 2023-09-10T01:13:14.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a36e4fb80 10 client.27902 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:14.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a36e4fb80 10 client.27902 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:14.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a36e4fb80 20 client.27902 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:14.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78d75010 0x55af78d77400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a36e4fb80 4 client.27902 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:13:14.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af78cc2840 con 0x55af78d75010 2023-09-10T01:13:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.485+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78d75010 0x55af78d77400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.486+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78d75010 0x55af78d77400 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e08373caaa43d077 server_cookie=42192183776b0343 in_seq=0 out_seq=0 2023-09-10T01:13:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.486+0000 7f7a04ff9700 10 client.27902 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.486+0000 7f7a04ff9700 1 -- 172.21.15.3:0/2984786898 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55af78cc2840 con 0x55af78d75010 2023-09-10T01:13:14.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.486+0000 7f7a04ff9700 10 client.27902 handle_command_reply: tid=0 2023-09-10T01:13:14.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 client.27902 shutdown 2023-09-10T01:13:14.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5b300 msgr2=0x55af78d6aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5b300 0x55af78d6aeb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f7a18099260 tx=0x7f7a18010bf0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 shutdown_connections 2023-09-10T01:13:14.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78d75010 0x55af78d77400 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d611a0 0x55af78d65100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5b300 0x55af78d6aeb0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a04ff9700 0 client.27902 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:14.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/2984786898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 >> 172.21.15.3:0/2984786898 conn(0x55af78cd0030 msgr2=0x55af78d557a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:14.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 shutdown_connections 2023-09-10T01:13:14.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/2984786898 wait complete. 2023-09-10T01:13:14.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.487+0000 7f7a36e4fb80 20 client.27902 trim_cache size 0 max 16384 2023-09-10T01:13:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.502+0000 7f7a36e4fb80 1 Processor -- start 2023-09-10T01:13:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.502+0000 7f7a36e4fb80 1 -- start start 2023-09-10T01:13:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.502+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d62d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d6af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78d666e0 con 0x55af78d5b300 2023-09-10T01:13:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78d60b00 con 0x55af78d611a0 2023-09-10T01:13:14.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78d5edb0 con 0x55af78d5bcc0 2023-09-10T01:13:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d6af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a2139b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d62d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a20b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d6af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55962/0 (socket says 172.21.15.3:55962) 2023-09-10T01:13:14.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 -- 172.21.15.3:0/361484349 learned_addr learned my addr 172.21.15.3:0/361484349 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 -- 172.21.15.3:0/361484349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 msgr2=0x55af78d6aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 -- 172.21.15.3:0/361484349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 msgr2=0x55af78d62d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d62d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a21b9c700 1 -- 172.21.15.3:0/361484349 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af78b001f0 con 0x55af78d611a0 2023-09-10T01:13:14.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.503+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:14.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.504+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d6af80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7a140079d0 tx=0x7f7a14009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d5f7642d71f68b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:14.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.504+0000 7f7a07fff700 1 -- 172.21.15.3:0/361484349 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1400e040 con 0x55af78d611a0 2023-09-10T01:13:14.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.504+0000 7f7a07fff700 1 -- 172.21.15.3:0/361484349 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a1400f040 con 0x55af78d611a0 2023-09-10T01:13:14.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.504+0000 7f7a07fff700 1 -- 172.21.15.3:0/361484349 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14020aa0 con 0x55af78d611a0 2023-09-10T01:13:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/361484349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 msgr2=0x55af78d6af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d6af80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7a140079d0 tx=0x7f7a14009920 comp rx=0 tx=0).stop 2023-09-10T01:13:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/361484349 shutdown_connections 2023-09-10T01:13:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d611a0 0x55af78d6af80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.519 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78d62d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/361484349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d5bcc0 0x55af78d6aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/361484349 >> 172.21.15.3:0/361484349 conn(0x55af78cd0030 msgr2=0x55af78d5ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/361484349 shutdown_connections 2023-09-10T01:13:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/361484349 wait complete. 2023-09-10T01:13:14.520 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.505+0000 7f7a36e4fb80 1 -- start start 2023-09-10T01:13:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 1 Processor -- start 2023-09-10T01:13:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78dc6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5bcc0 0x55af78dc6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d611a0 0x55af78dc0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78dc12d0 con 0x55af78d5b300 2023-09-10T01:13:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78dc1440 con 0x55af78d5bcc0 2023-09-10T01:13:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af78dc1600 con 0x55af78d611a0 2023-09-10T01:13:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.506+0000 7f7a36e4fb80 10 client.0 _ll_register_callbacks cb 0x7f7a14009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a20b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5bcc0 0x55af78dc6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a21b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d611a0 0x55af78dc0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a20b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5bcc0 0x55af78dc6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55970/0 (socket says 172.21.15.3:55970) 2023-09-10T01:13:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a20b9a700 1 -- 172.21.15.3:0/3185569081 learned_addr learned my addr 172.21.15.3:0/3185569081 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a2139b700 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78dc6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a21b9c700 1 -- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5bcc0 msgr2=0x55af78dc6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5bcc0 0x55af78dc6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a21b9c700 1 -- 172.21.15.3:0/3185569081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 msgr2=0x55af78dc6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:14.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78dc6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a21b9c700 1 -- 172.21.15.3:0/3185569081 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c10dd40 con 0x55af78d611a0 2023-09-10T01:13:14.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a20b9a700 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5bcc0 0x55af78dc6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:14.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a21b9c700 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d611a0 0x55af78dc0cf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7a14000f80 tx=0x7f7a14003cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23dab1aec6245004 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:14.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.507+0000 7f7a04ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:13:14.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a04ff9700 1 -- 172.21.15.3:0/3185569081 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14024070 con 0x55af78d611a0 2023-09-10T01:13:14.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a1809d390 con 0x55af78d611a0 2023-09-10T01:13:14.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a04ff9700 1 -- 172.21.15.3:0/3185569081 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f7a14020930 con 0x55af78d611a0 2023-09-10T01:13:14.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a04ff9700 1 -- 172.21.15.3:0/3185569081 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a1400e040 con 0x55af78d611a0 2023-09-10T01:13:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a36e4fb80 10 client.27802 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f7a140076b0 con 0x55af78d611a0 2023-09-10T01:13:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a04ff9700 1 -- 172.21.15.3:0/3185569081 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a1809d390 con 0x55af78d611a0 2023-09-10T01:13:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a04ff9700 1 -- 172.21.15.3:0/3185569081 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7a14004040 con 0x55af78d611a0 2023-09-10T01:13:14.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a36e4fb80 10 client.27802 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a36e4fb80 10 client.27802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.508+0000 7f7a36e4fb80 20 client.27802 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.509+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78dce790 0x55af78dd0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.509+0000 7f7a36e4fb80 4 client.27802 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:13:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.509+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55af78ca68c0 con 0x55af78dce790 2023-09-10T01:13:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.509+0000 7f7a2139b700 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78dce790 0x55af78dd0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.509+0000 7f7a2139b700 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78dce790 0x55af78dd0bb0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68538e20e9e3abcc server_cookie=31d7780faab95967 in_seq=0 out_seq=0 2023-09-10T01:13:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.509+0000 7f7a04ff9700 10 client.27802 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:14.571 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:14 smithi003 ceph-mon[97654]: pgmap v310: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 986 KiB/s wr, 167 op/s 2023-09-10T01:13:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a04ff9700 1 -- 172.21.15.3:0/3185569081 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55af78ca68c0 con 0x55af78dce790 2023-09-10T01:13:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a04ff9700 10 client.27802 handle_command_reply: tid=0 2023-09-10T01:13:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 client.27802 shutdown 2023-09-10T01:13:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d611a0 msgr2=0x55af78dc0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d611a0 0x55af78dc0cf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7a14000f80 tx=0x7f7a14003cb0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 shutdown_connections 2023-09-10T01:13:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55af78dce790 0x55af78dd0bb0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55af78d611a0 0x55af78dc0cf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55af78d5b300 0x55af78dc6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 --2- 172.21.15.3:0/3185569081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af78d5bcc0 0x55af78dc6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a04ff9700 0 client.27802 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 >> 172.21.15.3:0/3185569081 conn(0x55af78cd0030 msgr2=0x55af78d61d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 shutdown_connections 2023-09-10T01:13:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.604+0000 7f7a36e4fb80 1 -- 172.21.15.3:0/3185569081 wait complete. 2023-09-10T01:13:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a36e4fb80 20 client.27802 trim_cache size 0 max 16384 2023-09-10T01:13:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:13:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:13:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:13:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:13:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7a080a07b0 msgr2=0x7f7a080a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7a080a07b0 0x7f7a080a2c70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009fd0 tx=0x7f7a0c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c1005c0 msgr2=0x7f7a1c196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c196600 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800e9a0 tx=0x7f7a1800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 shutdown_connections 2023-09-10T01:13:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7a080a07b0 0x7f7a080a2c70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a1c106680 0x7f7a1c19afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a1c1005c0 0x7f7a1c196600 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 --2- 172.21.15.3:0/149910715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a1c100f80 0x7f7a1c196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.605+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 >> 172.21.15.3:0/149910715 conn(0x7f7a1c0fc0b0 msgr2=0x7f7a1c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.606+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 shutdown_connections 2023-09-10T01:13:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:14.606+0000 7f7a235ff700 1 -- 172.21.15.3:0/149910715 wait complete. 2023-09-10T01:13:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:14 smithi136 ceph-mon[133708]: pgmap v310: 129 pgs: 129 active+clean; 266 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 986 KiB/s wr, 167 op/s 2023-09-10T01:13:14.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 889019 8.70 MB/sec execute 416 sec latency 62.615 ms 2023-09-10T01:13:15.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 891320 8.72 MB/sec execute 417 sec latency 23.219 ms 2023-09-10T01:13:16.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:16 smithi092 ceph-mon[134067]: pgmap v311: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:16 smithi003 ceph-mon[97654]: pgmap v311: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:16 smithi136 ceph-mon[133708]: pgmap v311: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:16.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 892317 8.70 MB/sec execute 418 sec latency 30.535 ms 2023-09-10T01:13:17.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 894737 8.72 MB/sec execute 419 sec latency 17.759 ms 2023-09-10T01:13:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:18 smithi092 ceph-mon[134067]: pgmap v312: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:18.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:18 smithi003 ceph-mon[97654]: pgmap v312: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:18 smithi136 ceph-mon[133708]: pgmap v312: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 1.5 MiB/s wr, 159 op/s 2023-09-10T01:13:18.792 INFO:tasks.workunit.client.0.smithi003.stdout: 1 895560 8.70 MB/sec execute 420 sec latency 84.418 ms 2023-09-10T01:13:19.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 897577 8.71 MB/sec execute 421 sec latency 21.398 ms 2023-09-10T01:13:20.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:20 smithi092 ceph-mon[134067]: pgmap v313: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:20.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:20 smithi092 ceph-mon[134067]: osdmap e99: 12 total, 12 up, 12 in 2023-09-10T01:13:20.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:20 smithi003 ceph-mon[97654]: pgmap v313: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:20.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:20 smithi003 ceph-mon[97654]: osdmap e99: 12 total, 12 up, 12 in 2023-09-10T01:13:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:20 smithi136 ceph-mon[133708]: pgmap v313: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:20 smithi136 ceph-mon[133708]: osdmap e99: 12 total, 12 up, 12 in 2023-09-10T01:13:20.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 898726 8.71 MB/sec execute 422 sec latency 147.559 ms 2023-09-10T01:13:21.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 900178 8.70 MB/sec execute 423 sec latency 19.294 ms 2023-09-10T01:13:22.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:22 smithi092 ceph-mon[134067]: pgmap v315: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:22.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:22 smithi003 ceph-mon[97654]: pgmap v315: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:22 smithi136 ceph-mon[133708]: pgmap v315: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:22.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 902170 8.71 MB/sec execute 424 sec latency 14.216 ms 2023-09-10T01:13:23.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 903440 8.70 MB/sec execute 425 sec latency 68.786 ms 2023-09-10T01:13:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:24 smithi092 ceph-mon[134067]: pgmap v316: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:24 smithi003 ceph-mon[97654]: pgmap v316: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:24 smithi136 ceph-mon[133708]: pgmap v316: 129 pgs: 129 active+clean; 271 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 224 op/s 2023-09-10T01:13:24.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 905518 8.71 MB/sec execute 426 sec latency 36.205 ms 2023-09-10T01:13:25.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 906426 8.70 MB/sec execute 427 sec latency 133.904 ms 2023-09-10T01:13:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:26 smithi092 ceph-mon[134067]: pgmap v317: 129 pgs: 129 active+clean; 287 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 262 op/s 2023-09-10T01:13:26.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:26 smithi003 ceph-mon[97654]: pgmap v317: 129 pgs: 129 active+clean; 287 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 262 op/s 2023-09-10T01:13:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:26 smithi136 ceph-mon[133708]: pgmap v317: 129 pgs: 129 active+clean; 287 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 262 op/s 2023-09-10T01:13:26.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 908463 8.70 MB/sec execute 428 sec latency 68.614 ms 2023-09-10T01:13:27.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 -- 172.21.15.3:0/4065370816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae01012a0 msgr2=0x7f8ae0067b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 --2- 172.21.15.3:0/4065370816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae01012a0 0x7f8ae0067b60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009990 tx=0x7f8ad0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 -- 172.21.15.3:0/4065370816 shutdown_connections 2023-09-10T01:13:27.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 --2- 172.21.15.3:0/4065370816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae01012a0 0x7f8ae0067b60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 --2- 172.21.15.3:0/4065370816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae010cc20 0x7f8ae010f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 --2- 172.21.15.3:0/4065370816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ae0068130 0x7f8ae00685b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 -- 172.21.15.3:0/4065370816 >> 172.21.15.3:0/4065370816 conn(0x7f8ae0074e00 msgr2=0x7f8ae0075210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 -- 172.21.15.3:0/4065370816 shutdown_connections 2023-09-10T01:13:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.768+0000 7f8adffff700 1 -- 172.21.15.3:0/4065370816 wait complete. 2023-09-10T01:13:27.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 Processor -- start 2023-09-10T01:13:27.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 -- start start 2023-09-10T01:13:27.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae0068130 0x7f8ae019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae010cc20 0x7f8ae019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ae01985b0 0x7f8ae0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01990b0 con 0x7f8ae01985b0 2023-09-10T01:13:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0199220 con 0x7f8ae0068130 2023-09-10T01:13:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adffff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01993e0 con 0x7f8ae010cc20 2023-09-10T01:13:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae0068130 0x7f8ae019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae010cc20 0x7f8ae019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae0068130 0x7f8ae019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49740/0 (socket says 172.21.15.3:49740) 2023-09-10T01:13:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.770+0000 7f8adeffd700 1 -- 172.21.15.3:0/2526042043 learned_addr learned my addr 172.21.15.3:0/2526042043 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ae01985b0 0x7f8ae0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adeffd700 1 -- 172.21.15.3:0/2526042043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae010cc20 msgr2=0x7f8ae019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adeffd700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae010cc20 0x7f8ae019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adeffd700 1 -- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ae01985b0 msgr2=0x7f8ae0198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adeffd700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ae01985b0 0x7f8ae0198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adeffd700 1 -- 172.21.15.3:0/2526042043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad00095f0 con 0x7f8ae0068130 2023-09-10T01:13:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae010cc20 0x7f8ae019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ae01985b0 0x7f8ae0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adeffd700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae0068130 0x7f8ae019e5d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0005910 tx=0x7f8ad0005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8ac7fff700 1 -- 172.21.15.3:0/2526042043 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad001d070 con 0x7f8ae0068130 2023-09-10T01:13:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae01a53d0 con 0x7f8ae0068130 2023-09-10T01:13:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.771+0000 7f8ac7fff700 1 -- 172.21.15.3:0/2526042043 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ad0022470 con 0x7f8ae0068130 2023-09-10T01:13:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.772+0000 7f8ac7fff700 1 -- 172.21.15.3:0/2526042043 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0017650 con 0x7f8ae0068130 2023-09-10T01:13:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.772+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae01a5850 con 0x7f8ae0068130 2023-09-10T01:13:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.775+0000 7f8ac7fff700 1 -- 172.21.15.3:0/2526042043 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f8ad00225e0 con 0x7f8ae0068130 2023-09-10T01:13:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.777+0000 7f8ac7fff700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8ac80a07b0 0x7f8ac80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.777+0000 7f8af90dcb80 1 Processor -- start 2023-09-10T01:13:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.777+0000 7f8ac7fff700 1 -- 172.21.15.3:0/2526042043 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8ad00cd720 con 0x7f8ae0068130 2023-09-10T01:13:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.778+0000 7f8af90dcb80 1 -- start start 2023-09-10T01:13:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.778+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8ac80a07b0 0x7f8ac80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.778+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.778+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c81169fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.778+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8115f06d0 con 0x55c8116a5aa0 2023-09-10T01:13:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8115edff0 con 0x55c8116a63c0 2023-09-10T01:13:27.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8115e98e0 con 0x55c8116a01b0 2023-09-10T01:13:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:33260/0 (socket says 172.21.15.3:33260) 2023-09-10T01:13:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 -- 172.21.15.3:0/2069978706 learned_addr learned my addr 172.21.15.3:0/2069978706 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c81169fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8ac80a07b0 0x7f8ac80a2c70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0199e50 tx=0x7f8ad4009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 -- 172.21.15.3:0/2069978706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 msgr2=0x55c81169fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c81169fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 -- 172.21.15.3:0/2069978706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 msgr2=0x55c8116a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.779+0000 7f8adf7fe700 1 -- 172.21.15.3:0/2069978706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c811415d80 con 0x55c8116a01b0 2023-09-10T01:13:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.780+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a05f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80093f0 tx=0x7f8ad8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8920e3c1784a273 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.780+0000 7f8ac57fa700 1 -- 172.21.15.3:0/2069978706 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800e030 con 0x55c8116a01b0 2023-09-10T01:13:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.780+0000 7f8ac57fa700 1 -- 172.21.15.3:0/2069978706 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ad800f040 con 0x55c8116a01b0 2023-09-10T01:13:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.780+0000 7f8ac57fa700 1 -- 172.21.15.3:0/2069978706 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80149e0 con 0x55c8116a01b0 2023-09-10T01:13:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.781+0000 7f8af90dcb80 1 -- 172.21.15.3:0/2069978706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 msgr2=0x55c8116a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.781+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a05f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80093f0 tx=0x7f8ad8009920 comp rx=0 tx=0).stop 2023-09-10T01:13:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.782+0000 7f8af90dcb80 1 -- 172.21.15.3:0/2069978706 shutdown_connections 2023-09-10T01:13:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.782+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a05f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.782+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.782+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/2069978706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c81169fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.782+0000 7f8af90dcb80 1 -- 172.21.15.3:0/2069978706 >> 172.21.15.3:0/2069978706 conn(0x55c81160c030 msgr2=0x55c8116926c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.782+0000 7f8af90dcb80 1 -- 172.21.15.3:0/2069978706 shutdown_connections 2023-09-10T01:13:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.782+0000 7f8af90dcb80 1 -- 172.21.15.3:0/2069978706 wait complete. 2023-09-10T01:13:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.783+0000 7f8af90dcb80 1 -- start start 2023-09-10T01:13:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.783+0000 7f8af90dcb80 1 Processor -- start 2023-09-10T01:13:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a01b0 0x55c8116ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a5aa0 0x55c8116a9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c811587b10 con 0x55c8116a63c0 2023-09-10T01:13:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c811587810 con 0x55c8116a01b0 2023-09-10T01:13:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c81146b8c0 con 0x55c8116a5aa0 2023-09-10T01:13:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8af90dcb80 10 client.0 _ll_register_callbacks cb 0x55c8114d1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a01b0 0x55c8116ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a01b0 0x55c8116ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49766/0 (socket says 172.21.15.3:49766) 2023-09-10T01:13:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 -- 172.21.15.3:0/3233065522 learned_addr learned my addr 172.21.15.3:0/3233065522 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a5aa0 0x55c8116a9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 -- 172.21.15.3:0/3233065522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a5aa0 msgr2=0x55c8116a9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a5aa0 0x55c8116a9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 -- 172.21.15.3:0/3233065522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 msgr2=0x55c8116a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.784+0000 7f8adeffd700 1 -- 172.21.15.3:0/3233065522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad80090d0 con 0x55c8116a01b0 2023-09-10T01:13:27.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8adeffd700 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a01b0 0x55c8116ac410 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0029000 tx=0x7f8ad009a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f10ed224c3d57719 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:27.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8ac2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:13:27.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/3233065522 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad009a990 con 0x55c8116a01b0 2023-09-10T01:13:27.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/3233065522 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8ad009e7c0 con 0x55c8116a01b0 2023-09-10T01:13:27.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c811541490 con 0x55c8116a01b0 2023-09-10T01:13:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/3233065522 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ad00a2970 con 0x55c8116a01b0 2023-09-10T01:13:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8af90dcb80 10 client.27920 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.785+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c8113fec00 con 0x55c8116a01b0 2023-09-10T01:13:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/3233065522 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ad0064070 con 0x55c8116a01b0 2023-09-10T01:13:27.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/3233065522 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8ad00a24e0 con 0x55c8116a01b0 2023-09-10T01:13:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8af90dcb80 10 client.27920 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8af90dcb80 10 client.27920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8af90dcb80 20 client.27920 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:27.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c8116b9ad0 0x55c8116bbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8af90dcb80 4 client.27920 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:13:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8115fe840 con 0x55c8116b9ad0 2023-09-10T01:13:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.786+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c8116b9ad0 0x55c8116bbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.787+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c8116b9ad0 0x55c8116bbef0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da071c0fa96dbcc2 server_cookie=edc47b5dc5f0478d in_seq=0 out_seq=0 2023-09-10T01:13:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.787+0000 7f8ac2ff5700 10 client.27920 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.787+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/3233065522 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c8115fe840 con 0x55c8116b9ad0 2023-09-10T01:13:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.787+0000 7f8ac2ff5700 10 client.27920 handle_command_reply: tid=0 2023-09-10T01:13:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.787+0000 7f8af90dcb80 1 client.27920 shutdown 2023-09-10T01:13:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a01b0 msgr2=0x55c8116ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a01b0 0x55c8116ac410 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0029000 tx=0x7f8ad009a580 comp rx=0 tx=0).stop 2023-09-10T01:13:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 shutdown_connections 2023-09-10T01:13:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c8116b9ad0 0x55c8116bbef0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8ac2ff5700 0 client.27920 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a01b0 0x55c8116ac410 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/3233065522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a5aa0 0x55c8116a9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 >> 172.21.15.3:0/3233065522 conn(0x55c81160c030 msgr2=0x55c811691900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 shutdown_connections 2023-09-10T01:13:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 1 -- 172.21.15.3:0/3233065522 wait complete. 2023-09-10T01:13:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.788+0000 7f8af90dcb80 20 client.27920 trim_cache size 0 max 16384 2023-09-10T01:13:27.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 909527 8.70 MB/sec execute 429 sec latency 30.668 ms 2023-09-10T01:13:27.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.806+0000 7f8af90dcb80 1 Processor -- start 2023-09-10T01:13:27.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8af90dcb80 1 -- start start 2023-09-10T01:13:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c8116a9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8116ac8d0 con 0x55c8116a5aa0 2023-09-10T01:13:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8116b1360 con 0x55c8116a63c0 2023-09-10T01:13:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8116a0ca0 con 0x55c8116a01b0 2023-09-10T01:13:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116b0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:33268/0 (socket says 172.21.15.3:33268) 2023-09-10T01:13:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8adeffd700 1 -- 172.21.15.3:0/250836275 learned_addr learned my addr 172.21.15.3:0/250836275 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.807+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c8116a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8adeffd700 1 -- 172.21.15.3:0/250836275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 msgr2=0x55c8116a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8adeffd700 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c8116a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8adeffd700 1 -- 172.21.15.3:0/250836275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 msgr2=0x55c8116a8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8adeffd700 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8adeffd700 1 -- 172.21.15.3:0/250836275 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c81143c1f0 con 0x55c8116a01b0 2023-09-10T01:13:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8adeffd700 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116b0e20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0096e90 tx=0x7f8ad000b380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a5388e39b32fd09 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8ac5ffb700 1 -- 172.21.15.3:0/250836275 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad009e7c0 con 0x55c8116a01b0 2023-09-10T01:13:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8ac5ffb700 1 -- 172.21.15.3:0/250836275 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ad0064070 con 0x55c8116a01b0 2023-09-10T01:13:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.808+0000 7f8ac5ffb700 1 -- 172.21.15.3:0/250836275 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad00228c0 con 0x55c8116a01b0 2023-09-10T01:13:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 -- 172.21.15.3:0/250836275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 msgr2=0x55c8116b0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116b0e20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0096e90 tx=0x7f8ad000b380 comp rx=0 tx=0).stop 2023-09-10T01:13:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 -- 172.21.15.3:0/250836275 shutdown_connections 2023-09-10T01:13:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116b0e20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a63c0 0x55c8116a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/250836275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a5aa0 0x55c8116a8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 -- 172.21.15.3:0/250836275 >> 172.21.15.3:0/250836275 conn(0x55c81160c030 msgr2=0x55c8116a36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 -- 172.21.15.3:0/250836275 shutdown_connections 2023-09-10T01:13:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.809+0000 7f8af90dcb80 1 -- 172.21.15.3:0/250836275 wait complete. 2023-09-10T01:13:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.810+0000 7f8af90dcb80 1 -- start start 2023-09-10T01:13:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.810+0000 7f8af90dcb80 1 Processor -- start 2023-09-10T01:13:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.810+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.810+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a5aa0 0x55c8116a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.810+0000 7f8af90dcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8116a8600 con 0x55c8116a63c0 2023-09-10T01:13:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8116a8770 con 0x55c8116a5aa0 2023-09-10T01:13:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8af90dcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8116a8930 con 0x55c8116a01b0 2023-09-10T01:13:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8af90dcb80 10 client.0 _ll_register_callbacks cb 0x7f8ad0096d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a5aa0 0x55c8116a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8adeffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a5aa0 0x55c8116a7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:49784/0 (socket says 172.21.15.3:49784) 2023-09-10T01:13:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 -- 172.21.15.3:0/1361443736 learned_addr learned my addr 172.21.15.3:0/1361443736 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8adf7fe700 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 -- 172.21.15.3:0/1361443736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 msgr2=0x55c8116a9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 -- 172.21.15.3:0/1361443736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 msgr2=0x55c8116a80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8ade7fc700 1 -- 172.21.15.3:0/1361443736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8113fec00 con 0x55c8116a5aa0 2023-09-10T01:13:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.811+0000 7f8adeffd700 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8ade7fc700 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a5aa0 0x55c8116a7b80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4013010 tx=0x7f8ad40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16aa417facbbc7e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8ac2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:13:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/1361443736 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40274b0 con 0x55c8116a5aa0 2023-09-10T01:13:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ad800f040 con 0x55c8116a5aa0 2023-09-10T01:13:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/1361443736 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8ad4003960 con 0x55c8116a5aa0 2023-09-10T01:13:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/1361443736 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ad4003b20 con 0x55c8116a5aa0 2023-09-10T01:13:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8af90dcb80 10 client.27926 fetch_fsmap learned FSMap version 31 2023-09-10T01:13:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c811414250 con 0x55c8116a5aa0 2023-09-10T01:13:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.812+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/1361443736 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ad800f040 con 0x55c8116a5aa0 2023-09-10T01:13:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/1361443736 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8ad4030540 con 0x55c8116a5aa0 2023-09-10T01:13:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8af90dcb80 10 client.27926 fetch_fsmap finished waiting for FSMap version 31 2023-09-10T01:13:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8af90dcb80 10 client.27926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8af90dcb80 20 client.27926 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c81170ed30 0x55c811711150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8af90dcb80 4 client.27926 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:13:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c8115e28c0 con 0x55c81170ed30 2023-09-10T01:13:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8adeffd700 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c81170ed30 0x55c811711150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8adeffd700 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c81170ed30 0x55c811711150 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99235e2bdd6c2558 server_cookie=55e0a76f489fd36b in_seq=0 out_seq=0 2023-09-10T01:13:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.813+0000 7f8ac2ff5700 10 client.27926 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:27.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8ac2ff5700 1 -- 172.21.15.3:0/1361443736 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c8115e28c0 con 0x55c81170ed30 2023-09-10T01:13:27.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8ac2ff5700 10 client.27926 handle_command_reply: tid=0 2023-09-10T01:13:27.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 client.27926 shutdown 2023-09-10T01:13:27.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a5aa0 msgr2=0x55c8116a7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a5aa0 0x55c8116a7b80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4013010 tx=0x7f8ad40047c0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 shutdown_connections 2023-09-10T01:13:27.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55c81170ed30 0x55c811711150 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8116a01b0 0x55c8116a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c8116a63c0 0x55c8116a80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 --2- 172.21.15.3:0/1361443736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c8116a5aa0 0x55c8116a7b80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 >> 172.21.15.3:0/1361443736 conn(0x55c81160c030 msgr2=0x55c8116a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:27.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.940+0000 7f8ac2ff5700 0 client.27926 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:27.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.941+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 shutdown_connections 2023-09-10T01:13:27.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.941+0000 7f8af90dcb80 1 -- 172.21.15.3:0/1361443736 wait complete. 2023-09-10T01:13:27.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.943+0000 7f8af90dcb80 20 client.27926 trim_cache size 0 max 16384 2023-09-10T01:13:27.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:13:27.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:13:27.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:13:27.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:13:27.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8ac80a07b0 msgr2=0x7f8ac80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8ac80a07b0 0x7f8ac80a2c70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0199e50 tx=0x7f8ad4009260 comp rx=0 tx=0).stop 2023-09-10T01:13:27.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae0068130 msgr2=0x7f8ae019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:27.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae0068130 0x7f8ae019e5d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0005910 tx=0x7f8ad0005720 comp rx=0 tx=0).stop 2023-09-10T01:13:27.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 shutdown_connections 2023-09-10T01:13:27.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8ac80a07b0 0x7f8ac80a2c70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8ae01985b0 0x7f8ae0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae0068130 0x7f8ae019e5d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 --2- 172.21.15.3:0/2526042043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae010cc20 0x7f8ae019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:27.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.944+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 >> 172.21.15.3:0/2526042043 conn(0x7f8ae0074e00 msgr2=0x7f8ae00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:27.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.945+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 shutdown_connections 2023-09-10T01:13:27.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:27.945+0000 7f8adffff700 1 -- 172.21.15.3:0/2526042043 wait complete. 2023-09-10T01:13:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:28 smithi092 ceph-mon[134067]: pgmap v318: 129 pgs: 129 active+clean; 287 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 262 op/s 2023-09-10T01:13:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:28 smithi003 ceph-mon[97654]: pgmap v318: 129 pgs: 129 active+clean; 287 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 262 op/s 2023-09-10T01:13:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:28 smithi136 ceph-mon[133708]: pgmap v318: 129 pgs: 129 active+clean; 287 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 262 op/s 2023-09-10T01:13:28.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 910615 8.69 MB/sec execute 430 sec latency 86.055 ms 2023-09-10T01:13:29.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 912621 8.70 MB/sec execute 431 sec latency 27.823 ms 2023-09-10T01:13:30.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:30 smithi092 ceph-mon[134067]: pgmap v319: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 280 op/s 2023-09-10T01:13:30.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:30 smithi003 ceph-mon[97654]: pgmap v319: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 280 op/s 2023-09-10T01:13:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:30 smithi136 ceph-mon[133708]: pgmap v319: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 280 op/s 2023-09-10T01:13:30.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 913766 8.69 MB/sec execute 432 sec latency 23.363 ms 2023-09-10T01:13:31.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 915869 8.70 MB/sec execute 433 sec latency 16.747 ms 2023-09-10T01:13:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:32 smithi092 ceph-mon[134067]: pgmap v320: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 274 op/s 2023-09-10T01:13:32.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:32 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:13:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:32 smithi003 ceph-mon[97654]: pgmap v320: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 274 op/s 2023-09-10T01:13:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:32 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:13:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:32 smithi136 ceph-mon[133708]: pgmap v320: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 274 op/s 2023-09-10T01:13:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:32 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:13:32.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 916651 8.69 MB/sec execute 434 sec latency 42.211 ms 2023-09-10T01:13:33.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 917165 8.70 MB/sec execute 435 sec latency 136.685 ms 2023-09-10T01:13:34.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:34 smithi092 ceph-mon[134067]: pgmap v321: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 233 op/s 2023-09-10T01:13:34.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:34 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:34.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:34 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:34 smithi003 ceph-mon[97654]: pgmap v321: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 233 op/s 2023-09-10T01:13:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:34 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:34 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:34 smithi136 ceph-mon[133708]: pgmap v321: 129 pgs: 129 active+clean; 283 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 233 op/s 2023-09-10T01:13:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:34 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:34.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:34 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:34.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 918386 8.71 MB/sec execute 436 sec latency 36.590 ms 2023-09-10T01:13:35.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 920500 8.72 MB/sec execute 437 sec latency 19.936 ms 2023-09-10T01:13:35.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: pgmap v322: 129 pgs: 129 active+clean; 282 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2023-09-10T01:13:35.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi136 to 196.2M 2023-09-10T01:13:35.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: Unable to set osd_memory_target on smithi136 to 205750988: error parsing value: Value '205750988' is below minimum 939524096 2023-09-10T01:13:35.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi092 to 196.2M 2023-09-10T01:13:35.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: Unable to set osd_memory_target on smithi092 to 205766041: error parsing value: Value '205766041' is below minimum 939524096 2023-09-10T01:13:35.814 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.814 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:35 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: pgmap v322: 129 pgs: 129 active+clean; 282 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2023-09-10T01:13:35.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi136 to 196.2M 2023-09-10T01:13:35.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: Unable to set osd_memory_target on smithi136 to 205750988: error parsing value: Value '205750988' is below minimum 939524096 2023-09-10T01:13:35.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi092 to 196.2M 2023-09-10T01:13:35.917 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: Unable to set osd_memory_target on smithi092 to 205766041: error parsing value: Value '205766041' is below minimum 939524096 2023-09-10T01:13:35.917 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.917 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:35 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: pgmap v322: 129 pgs: 129 active+clean; 282 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 314 op/s 2023-09-10T01:13:35.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi136 to 196.2M 2023-09-10T01:13:35.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: Unable to set osd_memory_target on smithi136 to 205750988: error parsing value: Value '205750988' is below minimum 939524096 2023-09-10T01:13:35.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.969 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.970 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:35.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi092 to 196.2M 2023-09-10T01:13:35.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: Unable to set osd_memory_target on smithi092 to 205766041: error parsing value: Value '205766041' is below minimum 939524096 2023-09-10T01:13:35.971 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:35.972 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:35 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:36.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 921499 8.70 MB/sec execute 438 sec latency 21.128 ms 2023-09-10T01:13:36.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: Adjusting osd_memory_target on smithi003 to 222.0M 2023-09-10T01:13:36.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: Unable to set osd_memory_target on smithi003 to 232784384: error parsing value: Value '232784384' is below minimum 939524096 2023-09-10T01:13:36.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:13:36.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:13:36.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:36.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:36 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:36.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: Adjusting osd_memory_target on smithi003 to 222.0M 2023-09-10T01:13:36.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: Unable to set osd_memory_target on smithi003 to 232784384: error parsing value: Value '232784384' is below minimum 939524096 2023-09-10T01:13:36.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:13:36.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:13:36.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:36.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:36 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:36.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-09-10T01:13:36.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: Adjusting osd_memory_target on smithi003 to 222.0M 2023-09-10T01:13:36.967 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: Unable to set osd_memory_target on smithi003 to 232784384: error parsing value: Value '232784384' is below minimum 939524096 2023-09-10T01:13:36.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:13:36.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:13:36.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:13:36.968 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:36 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:37.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 923863 8.72 MB/sec execute 439 sec latency 15.638 ms 2023-09-10T01:13:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:38 smithi003 ceph-mon[97654]: pgmap v323: 129 pgs: 129 active+clean; 282 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 218 op/s 2023-09-10T01:13:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:38 smithi136 ceph-mon[133708]: pgmap v323: 129 pgs: 129 active+clean; 282 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 218 op/s 2023-09-10T01:13:38.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:38 smithi092 ceph-mon[134067]: pgmap v323: 129 pgs: 129 active+clean; 282 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 218 op/s 2023-09-10T01:13:38.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 924602 8.70 MB/sec execute 440 sec latency 38.163 ms 2023-09-10T01:13:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:39 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:39 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:13:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:39 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:39 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:13:39.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:39 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:39 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:13:39.793 INFO:tasks.workunit.client.0.smithi003.stdout: 1 926760 8.71 MB/sec execute 441 sec latency 25.714 ms 2023-09-10T01:13:40.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:40 smithi003 ceph-mon[97654]: pgmap v324: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 297 op/s 2023-09-10T01:13:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:40 smithi136 ceph-mon[133708]: pgmap v324: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 297 op/s 2023-09-10T01:13:40.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:40 smithi092 ceph-mon[134067]: pgmap v324: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 297 op/s 2023-09-10T01:13:40.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 928126 8.71 MB/sec execute 442 sec latency 11.999 ms 2023-09-10T01:13:41.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/1114484091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100f40 msgr2=0x7f3e8410d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/1114484091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100f40 0x7f3e8410d400 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b0d0 tx=0x7f3e7c00b3e0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/1114484091 shutdown_connections 2023-09-10T01:13:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/1114484091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e84106640 0x7f3e84106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/1114484091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100f40 0x7f3e8410d400 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/1114484091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e84100580 0x7f3e84100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/1114484091 >> 172.21.15.3:0/1114484091 conn(0x7f3e840fc050 msgr2=0x7f3e840fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/1114484091 shutdown_connections 2023-09-10T01:13:41.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.113+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/1114484091 wait complete. 2023-09-10T01:13:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.115+0000 7f3e8b5ff700 1 Processor -- start 2023-09-10T01:13:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.115+0000 7f3e8b5ff700 1 -- start start 2023-09-10T01:13:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.115+0000 7f3e8b5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100580 0x7f3e84072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.115+0000 7f3e8b5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e84100f40 0x7f3e84072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.115+0000 7f3e8b5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e84106640 0x7f3e84073380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8b5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e841a9140 con 0x7f3e84100f40 2023-09-10T01:13:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8b5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e841a9280 con 0x7f3e84100580 2023-09-10T01:13:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8b5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e841a93c0 con 0x7f3e84106640 2023-09-10T01:13:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100580 0x7f3e84072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100580 0x7f3e84072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45366/0 (socket says 172.21.15.3:45366) 2023-09-10T01:13:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 -- 172.21.15.3:0/3947986652 learned_addr learned my addr 172.21.15.3:0/3947986652 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e89b9c700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e84106640 0x7f3e84073380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e84100f40 0x7f3e84072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 -- 172.21.15.3:0/3947986652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e84106640 msgr2=0x7f3e84073380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e84106640 0x7f3e84073380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 -- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e84100f40 msgr2=0x7f3e84072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e84100f40 0x7f3e84072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.116+0000 7f3e8939b700 1 -- 172.21.15.3:0/3947986652 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c009d00 con 0x7f3e84100580 2023-09-10T01:13:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.117+0000 7f3e8939b700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100580 0x7f3e84072900 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000e9a0 tx=0x7f3e8000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.117+0000 7f3e7a7fc700 1 -- 172.21.15.3:0/3947986652 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80009790 con 0x7f3e84100580 2023-09-10T01:13:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.117+0000 7f3e7a7fc700 1 -- 172.21.15.3:0/3947986652 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e80010430 con 0x7f3e84100580 2023-09-10T01:13:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.117+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e841a96a0 con 0x7f3e84100580 2023-09-10T01:13:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.117+0000 7f3e7a7fc700 1 -- 172.21.15.3:0/3947986652 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80018680 con 0x7f3e84100580 2023-09-10T01:13:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.117+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e841a9c00 con 0x7f3e84100580 2023-09-10T01:13:41.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.122+0000 7f3e7a7fc700 1 -- 172.21.15.3:0/3947986652 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3e800105a0 con 0x7f3e84100580 2023-09-10T01:13:41.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.123+0000 7f3e9ef62b80 1 Processor -- start 2023-09-10T01:13:41.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.123+0000 7f3e9ef62b80 1 -- start start 2023-09-10T01:13:41.130 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.123+0000 7f3e7a7fc700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e700a0830 0x7f3e700a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.123+0000 7f3e7a7fc700 1 -- 172.21.15.3:0/3947986652 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3e80014070 con 0x7f3e84100580 2023-09-10T01:13:41.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.123+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a93521a0 0x55b8a9352580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.123+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a934c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.123+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934ccc0 0x55b8a9358f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a92a56d0 con 0x55b8a934c300 2023-09-10T01:13:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a92a2ff0 con 0x55b8a93521a0 2023-09-10T01:13:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a929e8e0 con 0x55b8a934ccc0 2023-09-10T01:13:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a93521a0 0x55b8a9352580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e89b9c700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e700a0830 0x7f3e700a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a93521a0 0x55b8a9352580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45380/0 (socket says 172.21.15.3:45380) 2023-09-10T01:13:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 -- 172.21.15.3:0/3303289073 learned_addr learned my addr 172.21.15.3:0/3303289073 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a934c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 -- 172.21.15.3:0/3303289073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934ccc0 msgr2=0x55b8a9358f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934ccc0 0x55b8a9358f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 -- 172.21.15.3:0/3303289073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 msgr2=0x55b8a934c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a934c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 -- 172.21.15.3:0/3303289073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8a90cad80 con 0x55b8a93521a0 2023-09-10T01:13:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a934c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.124+0000 7f3e8939b700 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a93521a0 0x55b8a9352580 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3e80095fd0 tx=0x7f3e80098fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e53d606aac1d3c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e89b9c700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e700a0830 0x7f3e700a2cf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b770 tx=0x7f3e7c009380 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e6f7fe700 1 -- 172.21.15.3:0/3303289073 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8009d170 con 0x55b8a93521a0 2023-09-10T01:13:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e6f7fe700 1 -- 172.21.15.3:0/3303289073 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e8009d390 con 0x55b8a93521a0 2023-09-10T01:13:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e6f7fe700 1 -- 172.21.15.3:0/3303289073 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e800997c0 con 0x55b8a93521a0 2023-09-10T01:13:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/3303289073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a93521a0 msgr2=0x55b8a9352580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a93521a0 0x55b8a9352580 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3e80095fd0 tx=0x7f3e80098fb0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/3303289073 shutdown_connections 2023-09-10T01:13:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934ccc0 0x55b8a9358f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a93521a0 0x55b8a9352580 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/3303289073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a934c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.125+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/3303289073 >> 172.21.15.3:0/3303289073 conn(0x55b8a92c1030 msgr2=0x55b8a9347560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.126+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/3303289073 shutdown_connections 2023-09-10T01:13:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.126+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/3303289073 wait complete. 2023-09-10T01:13:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.126+0000 7f3e9ef62b80 1 -- start start 2023-09-10T01:13:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.126+0000 7f3e9ef62b80 1 Processor -- start 2023-09-10T01:13:41.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a9356080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a923cb10 con 0x55b8a93521a0 2023-09-10T01:13:41.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a923c810 con 0x55b8a934ccc0 2023-09-10T01:13:41.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a91208c0 con 0x55b8a934c300 2023-09-10T01:13:41.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e9ef62b80 10 client.0 _ll_register_callbacks cb 0x55b8a9186a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:41.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57044/0 (socket says 172.21.15.3:57044) 2023-09-10T01:13:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 -- 172.21.15.3:0/2138548175 learned_addr learned my addr 172.21.15.3:0/2138548175 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e89b9c700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a9356080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 -- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 msgr2=0x55b8a935c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 -- 172.21.15.3:0/2138548175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 msgr2=0x55b8a9356080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a9356080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e8939b700 1 -- 172.21.15.3:0/2138548175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e841a9c00 con 0x55b8a934c300 2023-09-10T01:13:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.127+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e8939b700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935be30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3e80099e10 tx=0x7f3e80010ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2928f6c9dc4bf39 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e6cff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:13:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2138548175 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80097900 con 0x55b8a934c300 2023-09-10T01:13:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b8a91f6490 con 0x55b8a934c300 2023-09-10T01:13:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2138548175 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3e80098a50 con 0x55b8a934c300 2023-09-10T01:13:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2138548175 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e80098c10 con 0x55b8a934c300 2023-09-10T01:13:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e9ef62b80 10 client.27826 fetch_fsmap learned FSMap version 33 2023-09-10T01:13:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b8a90b3c00 con 0x55b8a934c300 2023-09-10T01:13:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.128+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2138548175 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b8a91f6490 con 0x55b8a934c300 2023-09-10T01:13:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2138548175 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 33) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3e800a1260 con 0x55b8a934c300 2023-09-10T01:13:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e9ef62b80 10 client.27826 fetch_fsmap finished waiting for FSMap version 33 2023-09-10T01:13:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e9ef62b80 10 client.27826 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e9ef62b80 20 client.27826 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a9365f40 0x55b8a9368330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e9ef62b80 4 client.27826 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:13:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8a92b3840 con 0x55b8a9365f40 2023-09-10T01:13:41.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.129+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a9365f40 0x55b8a9368330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.130+0000 7f3e6cff9700 10 client.27826 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.130+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a9365f40 0x55b8a9368330 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e782a8b0c8378cc server_cookie=bef469ec44dd74ed in_seq=0 out_seq=0 2023-09-10T01:13:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.130+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2138548175 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b8a92b3840 con 0x55b8a9365f40 2023-09-10T01:13:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.130+0000 7f3e6cff9700 10 client.27826 handle_command_reply: tid=0 2023-09-10T01:13:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.130+0000 7f3e9ef62b80 1 client.27826 shutdown 2023-09-10T01:13:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 msgr2=0x55b8a935be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935be30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3e80099e10 tx=0x7f3e80010ab0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 shutdown_connections 2023-09-10T01:13:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a9365f40 0x55b8a9368330 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935be30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a9356080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e6cff9700 0 client.27826 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2138548175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 >> 172.21.15.3:0/2138548175 conn(0x55b8a92c1030 msgr2=0x55b8a93467a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 shutdown_connections 2023-09-10T01:13:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2138548175 wait complete. 2023-09-10T01:13:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.131+0000 7f3e9ef62b80 20 client.27826 trim_cache size 0 max 16384 2023-09-10T01:13:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.147+0000 7f3e9ef62b80 1 Processor -- start 2023-09-10T01:13:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.147+0000 7f3e9ef62b80 1 -- start start 2023-09-10T01:13:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a9355940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a9355e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a93521a0 0x55b8a935b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a934d560 con 0x55b8a934c300 2023-09-10T01:13:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a9350cd0 con 0x55b8a934ccc0 2023-09-10T01:13:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a9351990 con 0x55b8a93521a0 2023-09-10T01:13:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a9355e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a9355e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45398/0 (socket says 172.21.15.3:45398) 2023-09-10T01:13:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 -- 172.21.15.3:0/476743075 learned_addr learned my addr 172.21.15.3:0/476743075 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e89b9c700 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a93521a0 0x55b8a935b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e8939b700 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a9355940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 -- 172.21.15.3:0/476743075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a93521a0 msgr2=0x55b8a935b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a93521a0 0x55b8a935b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 -- 172.21.15.3:0/476743075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 msgr2=0x55b8a9355940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a9355940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 -- 172.21.15.3:0/476743075 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8a90f11f0 con 0x55b8a934ccc0 2023-09-10T01:13:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.148+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a9355e80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x55b8a9347060 tx=0x7f3e74004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5233a0b02e2af1a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.149+0000 7f3e6ffff700 1 -- 172.21.15.3:0/476743075 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7401d910 con 0x55b8a934ccc0 2023-09-10T01:13:41.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.149+0000 7f3e6ffff700 1 -- 172.21.15.3:0/476743075 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e74018040 con 0x55b8a934ccc0 2023-09-10T01:13:41.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.149+0000 7f3e6ffff700 1 -- 172.21.15.3:0/476743075 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e74010690 con 0x55b8a934ccc0 2023-09-10T01:13:41.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.149+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/476743075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 msgr2=0x55b8a9355e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.149+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a9355e80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x55b8a9347060 tx=0x7f3e74004750 comp rx=0 tx=0).stop 2023-09-10T01:13:41.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/476743075 shutdown_connections 2023-09-10T01:13:41.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a93521a0 0x55b8a935b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a934c300 0x55b8a9355940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/476743075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a9355e80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/476743075 >> 172.21.15.3:0/476743075 conn(0x55b8a92c1030 msgr2=0x55b8a9347090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/476743075 shutdown_connections 2023-09-10T01:13:41.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/476743075 wait complete. 2023-09-10T01:13:41.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 -- start start 2023-09-10T01:13:41.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.150+0000 7f3e9ef62b80 1 Processor -- start 2023-09-10T01:13:41.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e9ef62b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a935a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a935ad90 con 0x55b8a93521a0 2023-09-10T01:13:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a935af00 con 0x55b8a934ccc0 2023-09-10T01:13:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e9ef62b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8a935b0c0 con 0x55b8a934c300 2023-09-10T01:13:41.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e9ef62b80 10 client.0 _ll_register_callbacks cb 0x7f3e80099190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:41.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e88b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e89b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a935a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e88b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45408/0 (socket says 172.21.15.3:45408) 2023-09-10T01:13:41.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57068/0 (socket says 172.21.15.3:57068) 2023-09-10T01:13:41.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 -- 172.21.15.3:0/2697226486 learned_addr learned my addr 172.21.15.3:0/2697226486 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:41.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 -- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 msgr2=0x55b8a935a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 -- 172.21.15.3:0/2697226486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 msgr2=0x55b8a935a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:41.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a935a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e8939b700 1 -- 172.21.15.3:0/2697226486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e74005ce0 con 0x55b8a934c300 2023-09-10T01:13:41.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.151+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:41.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e8939b700 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935bfe0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3e80095a60 tx=0x7f3e800993c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2627c55561b29eb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:41.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e6cff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:13:41.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2697226486 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80098ec0 con 0x55b8a934c300 2023-09-10T01:13:41.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e8009d390 con 0x55b8a934c300 2023-09-10T01:13:41.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2697226486 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3e80098c60 con 0x55b8a934c300 2023-09-10T01:13:41.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2697226486 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e800997c0 con 0x55b8a934c300 2023-09-10T01:13:41.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e9ef62b80 10 client.27832 fetch_fsmap learned FSMap version 33 2023-09-10T01:13:41.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b8a90c9250 con 0x55b8a934c300 2023-09-10T01:13:41.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.152+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2697226486 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e80080030 con 0x55b8a934c300 2023-09-10T01:13:41.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2697226486 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 33) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3e800b1b60 con 0x55b8a934c300 2023-09-10T01:13:41.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e9ef62b80 10 client.27832 fetch_fsmap finished waiting for FSMap version 33 2023-09-10T01:13:41.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e9ef62b80 10 client.27832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:41.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e9ef62b80 20 client.27832 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:41.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a93bad20 0x55b8a93bd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:41.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e9ef62b80 4 client.27832 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:13:41.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b8a92978c0 con 0x55b8a93bad20 2023-09-10T01:13:41.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a93bad20 0x55b8a93bd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:41.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e88b9a700 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a93bad20 0x55b8a93bd110 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22c45b54077f093e server_cookie=4f90359acb1f2804 in_seq=0 out_seq=0 2023-09-10T01:13:41.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.153+0000 7f3e6cff9700 10 client.27832 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e6cff9700 1 -- 172.21.15.3:0/2697226486 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b8a92978c0 con 0x55b8a93bad20 2023-09-10T01:13:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e6cff9700 10 client.27832 handle_command_reply: tid=0 2023-09-10T01:13:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 client.27832 shutdown 2023-09-10T01:13:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 msgr2=0x55b8a935bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935bfe0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3e80095a60 tx=0x7f3e800993c0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 shutdown_connections 2023-09-10T01:13:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b8a93bad20 0x55b8a93bd110 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8a934c300 0x55b8a935bfe0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8a93521a0 0x55b8a935a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 --2- 172.21.15.3:0/2697226486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8a934ccc0 0x55b8a935a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 >> 172.21.15.3:0/2697226486 conn(0x55b8a92c1030 msgr2=0x55b8a934b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e6cff9700 0 client.27832 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 shutdown_connections 2023-09-10T01:13:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 1 -- 172.21.15.3:0/2697226486 wait complete. 2023-09-10T01:13:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.333+0000 7f3e9ef62b80 20 client.27832 trim_cache size 0 max 16384 2023-09-10T01:13:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:13:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:13:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:13:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:13:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e700a0830 msgr2=0x7f3e700a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e700a0830 0x7f3e700a2cf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b770 tx=0x7f3e7c009380 comp rx=0 tx=0).stop 2023-09-10T01:13:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100580 msgr2=0x7f3e84072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100580 0x7f3e84072900 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000e9a0 tx=0x7f3e8000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 shutdown_connections 2023-09-10T01:13:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3e700a0830 0x7f3e700a2cf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e84106640 0x7f3e84073380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e84100580 0x7f3e84072900 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 --2- 172.21.15.3:0/3947986652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e84100f40 0x7f3e84072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.334+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 >> 172.21.15.3:0/3947986652 conn(0x7f3e840fc050 msgr2=0x7f3e840fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.335+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 shutdown_connections 2023-09-10T01:13:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:41.335+0000 7f3e8b5ff700 1 -- 172.21.15.3:0/3947986652 wait complete. 2023-09-10T01:13:41.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 929899 8.70 MB/sec execute 443 sec latency 88.420 ms 2023-09-10T01:13:42.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:42 smithi003 ceph-mon[97654]: pgmap v325: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2023-09-10T01:13:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:42 smithi136 ceph-mon[133708]: pgmap v325: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2023-09-10T01:13:42.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:42 smithi092 ceph-mon[134067]: pgmap v325: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2023-09-10T01:13:42.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 931477 8.71 MB/sec execute 444 sec latency 13.999 ms 2023-09-10T01:13:43.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:13:43.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 -- 172.21.15.3:0/72195126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514100fa0 msgr2=0x7f2514104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 --2- 172.21.15.3:0/72195126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514100fa0 0x7f2514104fe0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2508009990 tx=0x7f2508009ca0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 -- 172.21.15.3:0/72195126 shutdown_connections 2023-09-10T01:13:43.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 --2- 172.21.15.3:0/72195126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25141005f0 0x7f25141009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 --2- 172.21.15.3:0/72195126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2514105710 0x7f251410d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 --2- 172.21.15.3:0/72195126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514100fa0 0x7f2514104fe0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 -- 172.21.15.3:0/72195126 >> 172.21.15.3:0/72195126 conn(0x7f25140fbe50 msgr2=0x7f25140fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:43.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.785+0000 7f251959d700 1 -- 172.21.15.3:0/72195126 shutdown_connections 2023-09-10T01:13:43.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.786+0000 7f251959d700 1 -- 172.21.15.3:0/72195126 wait complete. 2023-09-10T01:13:43.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.787+0000 7f251959d700 1 Processor -- start 2023-09-10T01:13:43.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.787+0000 7f251959d700 1 -- start start 2023-09-10T01:13:43.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.787+0000 7f251959d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25141005f0 0x7f251419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.787+0000 7f251959d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2514105710 0x7f251419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.787+0000 7f251959d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514198540 0x7f25141989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f251959d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2514198f90 con 0x7f25141005f0 2023-09-10T01:13:43.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f251959d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141990d0 con 0x7f2514198540 2023-09-10T01:13:43.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f251959d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2514199290 con 0x7f2514105710 2023-09-10T01:13:43.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514198540 0x7f25141989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2513fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25141005f0 0x7f251419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514198540 0x7f25141989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45426/0 (socket says 172.21.15.3:45426) 2023-09-10T01:13:43.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 -- 172.21.15.3:0/805096240 learned_addr learned my addr 172.21.15.3:0/805096240 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:43.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f25137fe700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2514105710 0x7f251419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 -- 172.21.15.3:0/805096240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2514105710 msgr2=0x7f251419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2514105710 0x7f251419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 -- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25141005f0 msgr2=0x7f251419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25141005f0 0x7f251419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2518d9c700 1 -- 172.21.15.3:0/805096240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080095f0 con 0x7f2514198540 2023-09-10T01:13:43.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.788+0000 7f2513fff700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25141005f0 0x7f251419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:43.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.789+0000 7f2518d9c700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514198540 0x7f25141989c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f250c00ef30 tx=0x7f250c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:43.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.789+0000 7f25117fa700 1 -- 172.21.15.3:0/805096240 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c018070 con 0x7f2514198540 2023-09-10T01:13:43.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.789+0000 7f25117fa700 1 -- 172.21.15.3:0/805096240 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f250c00f040 con 0x7f2514198540 2023-09-10T01:13:43.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.789+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25141a5330 con 0x7f2514198540 2023-09-10T01:13:43.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.789+0000 7f25117fa700 1 -- 172.21.15.3:0/805096240 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c00cd00 con 0x7f2514198540 2023-09-10T01:13:43.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.789+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25141a57c0 con 0x7f2514198540 2023-09-10T01:13:43.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.792+0000 7f25117fa700 1 -- 172.21.15.3:0/805096240 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f250c003a00 con 0x7f2514198540 2023-09-10T01:13:43.796 INFO:tasks.workunit.client.0.smithi003.stdout: 1 932861 8.70 MB/sec execute 445 sec latency 58.865 ms 2023-09-10T01:13:43.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.794+0000 7f252e1d5b80 1 Processor -- start 2023-09-10T01:13:43.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.794+0000 7f252e1d5b80 1 -- start start 2023-09-10T01:13:43.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4b4180 0x5633bd4beaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4bf030 0x5633bd4c1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd40d730 con 0x5633bd4b4180 2023-09-10T01:13:43.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd40b050 con 0x5633bd4b3860 2023-09-10T01:13:43.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd406940 con 0x5633bd4bf030 2023-09-10T01:13:43.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25117fa700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24fc0a07b0 0x7f24fc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45438/0 (socket says 172.21.15.3:45438) 2023-09-10T01:13:43.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 -- 172.21.15.3:0/1370385786 learned_addr learned my addr 172.21.15.3:0/1370385786 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f2513fff700 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4b4180 0x5633bd4beaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4bf030 0x5633bd4c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25117fa700 1 -- 172.21.15.3:0/805096240 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f250c0cbed0 con 0x7f2514198540 2023-09-10T01:13:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f2518d9c700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24fc0a07b0 0x7f24fc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 -- 172.21.15.3:0/1370385786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4bf030 msgr2=0x5633bd4c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4bf030 0x5633bd4c1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 -- 172.21.15.3:0/1370385786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4b4180 msgr2=0x5633bd4beaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.795+0000 7f25137fe700 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4b4180 0x5633bd4beaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.796+0000 7f25137fe700 1 -- 172.21.15.3:0/1370385786 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633bd232de0 con 0x5633bd4b3860 2023-09-10T01:13:43.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.796+0000 7f2513fff700 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4b4180 0x5633bd4beaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:43.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.796+0000 7f25137fe700 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b3c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2508005cd0 tx=0x7f250801a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c948c65aec8d4643 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:43.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.796+0000 7f2518d9c700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24fc0a07b0 0x7f24fc0a2c70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f250c0090a0 tx=0x7f250c009ed0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:43.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.796+0000 7f24fa7fc700 1 -- 172.21.15.3:0/1370385786 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250800bc90 con 0x5633bd4b3860 2023-09-10T01:13:43.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.796+0000 7f24fa7fc700 1 -- 172.21.15.3:0/1370385786 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f250800bdf0 con 0x5633bd4b3860 2023-09-10T01:13:43.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.796+0000 7f24fa7fc700 1 -- 172.21.15.3:0/1370385786 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508017650 con 0x5633bd4b3860 2023-09-10T01:13:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1370385786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 msgr2=0x5633bd4b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b3c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2508005cd0 tx=0x7f250801a040 comp rx=0 tx=0).stop 2023-09-10T01:13:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1370385786 shutdown_connections 2023-09-10T01:13:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4bf030 0x5633bd4c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b3c40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1370385786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4b4180 0x5633bd4beaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1370385786 >> 172.21.15.3:0/1370385786 conn(0x5633bd429090 msgr2=0x5633bd4afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1370385786 shutdown_connections 2023-09-10T01:13:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.797+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1370385786 wait complete. 2023-09-10T01:13:43.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.798+0000 7f252e1d5b80 1 -- start start 2023-09-10T01:13:43.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.798+0000 7f252e1d5b80 1 Processor -- start 2023-09-10T01:13:43.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4bbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4bc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd3a4b70 con 0x5633bd4bf030 2023-09-10T01:13:43.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd3a4870 con 0x5633bd4b3860 2023-09-10T01:13:43.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd288920 con 0x5633bd4b4180 2023-09-10T01:13:43.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f252e1d5b80 10 client.0 _ll_register_callbacks cb 0x5633bd2eea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:43.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f25137fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4bbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f25137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f25137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56700/0 (socket says 172.21.15.3:56700) 2023-09-10T01:13:43.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f25137fe700 1 -- 172.21.15.3:0/3830143065 learned_addr learned my addr 172.21.15.3:0/3830143065 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:43.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.799+0000 7f2513fff700 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4bc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f25137fe700 1 -- 172.21.15.3:0/3830143065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 msgr2=0x5633bd4bc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f25137fe700 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4bc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f25137fe700 1 -- 172.21.15.3:0/3830143065 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 msgr2=0x5633bd4bbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f25137fe700 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4bbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f25137fe700 1 -- 172.21.15.3:0/3830143065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2508020040 con 0x5633bd4bf030 2023-09-10T01:13:43.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f25137fe700 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bc7a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2508009350 tx=0x7f2508009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2052dae33df61312 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:43.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f24f37fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:13:43.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f24f37fe700 1 -- 172.21.15.3:0/3830143065 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508005030 con 0x5633bd4bf030 2023-09-10T01:13:43.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633bd35e4f0 con 0x5633bd4bf030 2023-09-10T01:13:43.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.800+0000 7f24f37fe700 1 -- 172.21.15.3:0/3830143065 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f2508005190 con 0x5633bd4bf030 2023-09-10T01:13:43.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f24f37fe700 1 -- 172.21.15.3:0/3830143065 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2508005350 con 0x5633bd4bf030 2023-09-10T01:13:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 10 client.17865 fetch_fsmap learned FSMap version 33 2023-09-10T01:13:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5633bd21bc60 con 0x5633bd4bf030 2023-09-10T01:13:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f24f37fe700 1 -- 172.21.15.3:0/3830143065 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5633bd35e4f0 con 0x5633bd4bf030 2023-09-10T01:13:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f24f37fe700 1 -- 172.21.15.3:0/3830143065 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 33) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f2508017bb0 con 0x5633bd4bf030 2023-09-10T01:13:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 10 client.17865 fetch_fsmap finished waiting for FSMap version 33 2023-09-10T01:13:43.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 10 client.17865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 20 client.17865 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd4ce5c0 0x5633bd4d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 4 client.17865 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:13:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5633bd41b8a0 con 0x5633bd4ce5c0 2023-09-10T01:13:43.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.801+0000 7f2513fff700 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd4ce5c0 0x5633bd4d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.802+0000 7f2513fff700 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd4ce5c0 0x5633bd4d09b0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5282cc68c86a2c96 server_cookie=defef8746edae467 in_seq=0 out_seq=0 2023-09-10T01:13:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.802+0000 7f24f37fe700 10 client.17865 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.803+0000 7f24f37fe700 1 -- 172.21.15.3:0/3830143065 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5633bd41b8a0 con 0x5633bd4ce5c0 2023-09-10T01:13:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.803+0000 7f24f37fe700 10 client.17865 handle_command_reply: tid=0 2023-09-10T01:13:43.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.803+0000 7f252e1d5b80 1 client.17865 shutdown 2023-09-10T01:13:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 msgr2=0x5633bd4bc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bc7a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2508009350 tx=0x7f2508009ca0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 shutdown_connections 2023-09-10T01:13:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd4ce5c0 0x5633bd4d09b0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bc7a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4bbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/3830143065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4bc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f24f37fe700 0 client.17865 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 >> 172.21.15.3:0/3830143065 conn(0x5633bd429090 msgr2=0x5633bd4af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:43.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 shutdown_connections 2023-09-10T01:13:43.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 1 -- 172.21.15.3:0/3830143065 wait complete. 2023-09-10T01:13:43.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.804+0000 7f252e1d5b80 20 client.17865 trim_cache size 0 max 16384 2023-09-10T01:13:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.822+0000 7f252e1d5b80 1 Processor -- start 2023-09-10T01:13:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.822+0000 7f252e1d5b80 1 -- start start 2023-09-10T01:13:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd4b4560 con 0x5633bd4bf030 2023-09-10T01:13:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd4ac530 con 0x5633bd4b3860 2023-09-10T01:13:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd4b4a80 con 0x5633bd4b4180 2023-09-10T01:13:43.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45466/0 (socket says 172.21.15.3:45466) 2023-09-10T01:13:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 -- 172.21.15.3:0/2381887274 learned_addr learned my addr 172.21.15.3:0/2381887274 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f2513fff700 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 -- 172.21.15.3:0/2381887274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 msgr2=0x5633bd4c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 -- 172.21.15.3:0/2381887274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 msgr2=0x5633bd4bafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.823+0000 7f25137fe700 1 -- 172.21.15.3:0/2381887274 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633bd4ae750 con 0x5633bd4b3860 2023-09-10T01:13:43.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.824+0000 7f25137fe700 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b67e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f250800b980 tx=0x7f2508009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6326c3ee7f795c64 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:43.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.824+0000 7f24faffd700 1 -- 172.21.15.3:0/2381887274 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250800ece0 con 0x5633bd4b3860 2023-09-10T01:13:43.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.824+0000 7f24faffd700 1 -- 172.21.15.3:0/2381887274 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f250800fcf0 con 0x5633bd4b3860 2023-09-10T01:13:43.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.824+0000 7f24faffd700 1 -- 172.21.15.3:0/2381887274 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250800e8c0 con 0x5633bd4b3860 2023-09-10T01:13:43.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.824+0000 7f252e1d5b80 1 -- 172.21.15.3:0/2381887274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 msgr2=0x5633bd4b67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.824+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b67e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f250800b980 tx=0x7f2508009290 comp rx=0 tx=0).stop 2023-09-10T01:13:43.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 -- 172.21.15.3:0/2381887274 shutdown_connections 2023-09-10T01:13:43.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd4bafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b3860 0x5633bd4b67e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/2381887274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b4180 0x5633bd4c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 -- 172.21.15.3:0/2381887274 >> 172.21.15.3:0/2381887274 conn(0x5633bd429090 msgr2=0x5633bd4bc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:43.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 -- 172.21.15.3:0/2381887274 shutdown_connections 2023-09-10T01:13:43.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 -- 172.21.15.3:0/2381887274 wait complete. 2023-09-10T01:13:43.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 -- start start 2023-09-10T01:13:43.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.825+0000 7f252e1d5b80 1 Processor -- start 2023-09-10T01:13:43.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b3860 0x5633bd51bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b4180 0x5633bd51c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f252e1d5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd516310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd5168f0 con 0x5633bd4bf030 2023-09-10T01:13:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd516a60 con 0x5633bd4b4180 2023-09-10T01:13:43.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f252e1d5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633bd516c20 con 0x5633bd4b3860 2023-09-10T01:13:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f252e1d5b80 10 client.0 _ll_register_callbacks cb 0x7f250800bc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f2513fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b4180 0x5633bd51c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f2513fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b4180 0x5633bd51c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:45470/0 (socket says 172.21.15.3:45470) 2023-09-10T01:13:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f2513fff700 1 -- 172.21.15.3:0/1126451553 learned_addr learned my addr 172.21.15.3:0/1126451553 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:43.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f25137fe700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b3860 0x5633bd51bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f25137fe700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd516310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.826+0000 7f2513fff700 1 -- 172.21.15.3:0/1126451553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b3860 msgr2=0x5633bd51bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f2513fff700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b3860 0x5633bd51bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f2513fff700 1 -- 172.21.15.3:0/1126451553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 msgr2=0x5633bd516310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f2513fff700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd516310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f2513fff700 1 -- 172.21.15.3:0/1126451553 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250803c040 con 0x5633bd4b4180 2023-09-10T01:13:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f25137fe700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd516310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f2513fff700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b4180 0x5633bd51c070 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f250400b6e0 tx=0x7f250401e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92df8a9cd88eca95 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f24f37fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:13:43.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f24f37fe700 1 -- 172.21.15.3:0/1126451553 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504017040 con 0x5633bd4b4180 2023-09-10T01:13:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633bd4cb4a0 con 0x5633bd4b4180 2023-09-10T01:13:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f24f37fe700 1 -- 172.21.15.3:0/1126451553 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f2504004d00 con 0x5633bd4b4180 2023-09-10T01:13:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f24f37fe700 1 -- 172.21.15.3:0/1126451553 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2504004020 con 0x5633bd4b4180 2023-09-10T01:13:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.827+0000 7f252e1d5b80 10 client.27962 fetch_fsmap learned FSMap version 33 2023-09-10T01:13:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5633bd259250 con 0x5633bd4b4180 2023-09-10T01:13:43.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f24f37fe700 1 -- 172.21.15.3:0/1126451553 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5633bd4cb4a0 con 0x5633bd4b4180 2023-09-10T01:13:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f24f37fe700 1 -- 172.21.15.3:0/1126451553 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 33) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f2504012670 con 0x5633bd4b4180 2023-09-10T01:13:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f252e1d5b80 10 client.27962 fetch_fsmap finished waiting for FSMap version 33 2023-09-10T01:13:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f252e1d5b80 10 client.27962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f252e1d5b80 20 client.27962 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd523ae0 0x5633bd525f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f252e1d5b80 4 client.27962 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:13:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f250800bdf0 con 0x5633bd523ae0 2023-09-10T01:13:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.828+0000 7f25137fe700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd523ae0 0x5633bd525f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.829+0000 7f24f37fe700 10 client.27962 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:43.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.829+0000 7f25137fe700 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd523ae0 0x5633bd525f00 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13b45d8dfe030ae9 server_cookie=ac6bf8fc97a1ec8a in_seq=0 out_seq=0 2023-09-10T01:13:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f24f37fe700 1 -- 172.21.15.3:0/1126451553 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f250800bdf0 con 0x5633bd523ae0 2023-09-10T01:13:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f24f37fe700 10 client.27962 handle_command_reply: tid=0 2023-09-10T01:13:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 client.27962 shutdown 2023-09-10T01:13:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b4180 msgr2=0x5633bd51c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b4180 0x5633bd51c070 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f250400b6e0 tx=0x7f250401e9c0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 shutdown_connections 2023-09-10T01:13:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5633bd523ae0 0x5633bd525f00 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633bd4b3860 0x5633bd51bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5633bd4bf030 0x5633bd516310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 --2- 172.21.15.3:0/1126451553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5633bd4b4180 0x5633bd51c070 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f24f37fe700 0 client.27962 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.830+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 >> 172.21.15.3:0/1126451553 conn(0x5633bd429090 msgr2=0x5633bd4bd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 shutdown_connections 2023-09-10T01:13:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f252e1d5b80 1 -- 172.21.15.3:0/1126451553 wait complete. 2023-09-10T01:13:43.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f252e1d5b80 20 client.27962 trim_cache size 0 max 16384 2023-09-10T01:13:43.870 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:13:43.870 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:13:43.870 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:13:43.870 INFO:teuthology.orchestra.run.smithi003.stdout: "60c92784-d24c-41fa-8295-389926a05a88": { 2023-09-10T01:13:43.871 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:13:43.871 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "60c92784-d24c-41fa-8295-389926a05a88", 2023-09-10T01:13:43.871 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:13:43.871 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:13:43.871 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:13:43.872 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:13:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24fc0a07b0 msgr2=0x7f24fc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24fc0a07b0 0x7f24fc0a2c70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f250c0090a0 tx=0x7f250c009ed0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514198540 msgr2=0x7f25141989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514198540 0x7f25141989c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f250c00ef30 tx=0x7f250c00c3a0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 shutdown_connections 2023-09-10T01:13:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24fc0a07b0 0x7f24fc0a2c70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2514198540 0x7f25141989c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25141005f0 0x7f251419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 --2- 172.21.15.3:0/805096240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2514105710 0x7f251419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.831+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 >> 172.21.15.3:0/805096240 conn(0x7f25140fbe50 msgr2=0x7f25140fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.832+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 shutdown_connections 2023-09-10T01:13:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:13:43.832+0000 7f251959d700 1 -- 172.21.15.3:0/805096240 wait complete. 2023-09-10T01:13:43.876 INFO:tasks.cephfs.filesystem:scrub status for tag:60c92784-d24c-41fa-8295-389926a05a88 - {'path': '/', 'tag': '60c92784-d24c-41fa-8295-389926a05a88', 'options': 'recursive,force'} 2023-09-10T01:13:44.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:44 smithi003 ceph-mon[97654]: pgmap v326: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2023-09-10T01:13:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:44 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:13:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:44 smithi136 ceph-mon[133708]: pgmap v326: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2023-09-10T01:13:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:44 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:13:44.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:44 smithi092 ceph-mon[134067]: pgmap v326: 129 pgs: 129 active+clean; 290 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 160 op/s 2023-09-10T01:13:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:44 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:13:44.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 934798 8.71 MB/sec execute 446 sec latency 19.192 ms 2023-09-10T01:13:45.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:45 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:13:45.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:45 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:13:45.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:45 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:13:45.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 936136 8.71 MB/sec execute 447 sec latency 19.415 ms 2023-09-10T01:13:46.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:46 smithi003 ceph-mon[97654]: pgmap v327: 129 pgs: 129 active+clean; 284 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 308 op/s 2023-09-10T01:13:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:46 smithi136 ceph-mon[133708]: pgmap v327: 129 pgs: 129 active+clean; 284 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 308 op/s 2023-09-10T01:13:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:46 smithi092 ceph-mon[134067]: pgmap v327: 129 pgs: 129 active+clean; 284 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 308 op/s 2023-09-10T01:13:46.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 938377 8.72 MB/sec execute 448 sec latency 18.705 ms 2023-09-10T01:13:47.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 939879 8.71 MB/sec execute 449 sec latency 13.510 ms 2023-09-10T01:13:48.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:48 smithi003 ceph-mon[97654]: pgmap v328: 129 pgs: 129 active+clean; 284 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 226 op/s 2023-09-10T01:13:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:48 smithi136 ceph-mon[133708]: pgmap v328: 129 pgs: 129 active+clean; 284 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 226 op/s 2023-09-10T01:13:48.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:48 smithi092 ceph-mon[134067]: pgmap v328: 129 pgs: 129 active+clean; 284 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 226 op/s 2023-09-10T01:13:48.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 941863 8.72 MB/sec execute 450 sec latency 68.374 ms 2023-09-10T01:13:49.795 INFO:tasks.workunit.client.0.smithi003.stdout: 1 942842 8.71 MB/sec execute 451 sec latency 69.198 ms 2023-09-10T01:13:50.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:50 smithi003 ceph-mon[97654]: pgmap v329: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 313 op/s 2023-09-10T01:13:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:50 smithi136 ceph-mon[133708]: pgmap v329: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 313 op/s 2023-09-10T01:13:50.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:50 smithi092 ceph-mon[134067]: pgmap v329: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 313 op/s 2023-09-10T01:13:50.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 945315 8.72 MB/sec execute 452 sec latency 18.030 ms 2023-09-10T01:13:51.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 946515 8.72 MB/sec execute 453 sec latency 14.166 ms 2023-09-10T01:13:52.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:52 smithi003 ceph-mon[97654]: pgmap v330: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-09-10T01:13:52.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:52 smithi136 ceph-mon[133708]: pgmap v330: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-09-10T01:13:52.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:52 smithi092 ceph-mon[134067]: pgmap v330: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-09-10T01:13:52.794 INFO:tasks.workunit.client.0.smithi003.stdout: 1 948878 8.73 MB/sec execute 454 sec latency 12.374 ms 2023-09-10T01:13:53.795 INFO:tasks.workunit.client.0.smithi003.stdout: 1 949874 8.71 MB/sec execute 455 sec latency 87.303 ms 2023-09-10T01:13:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.506+0000 7f5e891ff700 1 -- 172.21.15.3:0/2258408566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84106680 msgr2=0x7f5e84106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.506+0000 7f5e891ff700 1 --2- 172.21.15.3:0/2258408566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84106680 0x7f5e84106a60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009920 tx=0x7f5e74009c30 comp rx=0 tx=0).stop 2023-09-10T01:13:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.507+0000 7f5e891ff700 1 -- 172.21.15.3:0/2258408566 shutdown_connections 2023-09-10T01:13:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.507+0000 7f5e891ff700 1 --2- 172.21.15.3:0/2258408566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e84100f80 0x7f5e8410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.507+0000 7f5e891ff700 1 --2- 172.21.15.3:0/2258408566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84106680 0x7f5e84106a60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.507+0000 7f5e891ff700 1 --2- 172.21.15.3:0/2258408566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e841005c0 0x7f5e84100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.507+0000 7f5e891ff700 1 -- 172.21.15.3:0/2258408566 >> 172.21.15.3:0/2258408566 conn(0x7f5e840fc0b0 msgr2=0x7f5e840fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.507+0000 7f5e891ff700 1 -- 172.21.15.3:0/2258408566 shutdown_connections 2023-09-10T01:13:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.507+0000 7f5e891ff700 1 -- 172.21.15.3:0/2258408566 wait complete. 2023-09-10T01:13:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.508+0000 7f5e891ff700 1 Processor -- start 2023-09-10T01:13:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.508+0000 7f5e891ff700 1 -- start start 2023-09-10T01:13:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e891ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e841005c0 0x7f5e8419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e891ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84100f80 0x7f5e8419d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e891ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e84106680 0x7f5e841a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e891ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e841a0bd0 con 0x7f5e84100f80 2023-09-10T01:13:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e891ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e841a0d40 con 0x7f5e84106680 2023-09-10T01:13:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e891ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e841a0f00 con 0x7f5e841005c0 2023-09-10T01:13:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e8359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e84106680 0x7f5e841a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e8359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e84106680 0x7f5e841a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48608/0 (socket says 172.21.15.3:48608) 2023-09-10T01:13:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.509+0000 7f5e8359e700 1 -- 172.21.15.3:0/3970255784 learned_addr learned my addr 172.21.15.3:0/3970255784 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e841005c0 0x7f5e8419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e8259c700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84100f80 0x7f5e8419d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e8359e700 1 -- 172.21.15.3:0/3970255784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e841005c0 msgr2=0x7f5e8419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e8359e700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e841005c0 0x7f5e8419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e8359e700 1 -- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84100f80 msgr2=0x7f5e8419d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e8359e700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84100f80 0x7f5e8419d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e8359e700 1 -- 172.21.15.3:0/3970255784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e74009520 con 0x7f5e84106680 2023-09-10T01:13:54.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.510+0000 7f5e8259c700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84100f80 0x7f5e8419d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:13:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.511+0000 7f5e8359e700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e84106680 0x7f5e841a05d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00b970 tx=0x7f5e7c00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.513+0000 7f5e6bfff700 1 -- 172.21.15.3:0/3970255784 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c00c930 con 0x7f5e84106680 2023-09-10T01:13:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.513+0000 7f5e6bfff700 1 -- 172.21.15.3:0/3970255784 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e7c014440 con 0x7f5e84106680 2023-09-10T01:13:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.513+0000 7f5e6bfff700 1 -- 172.21.15.3:0/3970255784 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c0075d0 con 0x7f5e84106680 2023-09-10T01:13:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.513+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e841a11e0 con 0x7f5e84106680 2023-09-10T01:13:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.513+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e841a9b40 con 0x7f5e84106680 2023-09-10T01:13:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.519+0000 7f5e9cbf0b80 1 Processor -- start 2023-09-10T01:13:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e6bfff700 1 -- 172.21.15.3:0/3970255784 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f5e7c0145b0 con 0x7f5e84106680 2023-09-10T01:13:54.521 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e9cbf0b80 1 -- start start 2023-09-10T01:13:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f1b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f15790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15cd0 0x563132f21f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132e5d6d0 con 0x563132f1b1b0 2023-09-10T01:13:54.522 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132e5aff0 con 0x563132f15cd0 2023-09-10T01:13:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.520+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132e568e0 con 0x563132f15310 2023-09-10T01:13:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.522+0000 7f5e82d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f15790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.522+0000 7f5e82d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f15790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42868/0 (socket says 172.21.15.3:42868) 2023-09-10T01:13:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.522+0000 7f5e82d9d700 1 -- 172.21.15.3:0/1315666773 learned_addr learned my addr 172.21.15.3:0/1315666773 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.522+0000 7f5e8259c700 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15cd0 0x563132f21f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.523+0000 7f5e8259c700 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f1b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.523+0000 7f5e6bfff700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5e6c0a08e0 0x7f5e6c0a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.523+0000 7f5e6bfff700 1 -- 172.21.15.3:0/3970255784 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f5e7c006e40 con 0x7f5e84106680 2023-09-10T01:13:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.524+0000 7f5e82d9d700 1 -- 172.21.15.3:0/1315666773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15cd0 msgr2=0x563132f21f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.524+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15cd0 0x563132f21f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.524+0000 7f5e82d9d700 1 -- 172.21.15.3:0/1315666773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 msgr2=0x563132f1b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.524+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f1b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.524+0000 7f5e82d9d700 1 -- 172.21.15.3:0/1315666773 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563132c82d80 con 0x563132f15310 2023-09-10T01:13:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.524+0000 7f5e8359e700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5e6c0a08e0 0x7f5e6c0a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.526+0000 7f5e8359e700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5e6c0a08e0 0x7f5e6c0a2da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c094f90 tx=0x7f5e7c004be0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.526+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f15790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400ffd0 tx=0x7f5e7400f460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5fb38ff28cc0037 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.526+0000 7f5e697fa700 1 -- 172.21.15.3:0/1315666773 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7400f890 con 0x563132f15310 2023-09-10T01:13:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.526+0000 7f5e697fa700 1 -- 172.21.15.3:0/1315666773 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e7400f9f0 con 0x563132f15310 2023-09-10T01:13:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.526+0000 7f5e697fa700 1 -- 172.21.15.3:0/1315666773 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74003680 con 0x563132f15310 2023-09-10T01:13:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.527+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/1315666773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 msgr2=0x563132f15790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.527+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f15790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400ffd0 tx=0x7f5e7400f460 comp rx=0 tx=0).stop 2023-09-10T01:13:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.528+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/1315666773 shutdown_connections 2023-09-10T01:13:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.528+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15cd0 0x563132f21f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.528+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f1b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.528+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/1315666773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f15790 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.528+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/1315666773 >> 172.21.15.3:0/1315666773 conn(0x563132e79030 msgr2=0x563132eff560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.528+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/1315666773 shutdown_connections 2023-09-10T01:13:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.531+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/1315666773 wait complete. 2023-09-10T01:13:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.531+0000 7f5e9cbf0b80 1 -- start start 2023-09-10T01:13:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.532+0000 7f5e9cbf0b80 1 Processor -- start 2023-09-10T01:13:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.534+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f21e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.534+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f1f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.534+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f1b1b0 0x563132f1f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.534+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132df4b10 con 0x563132f15cd0 2023-09-10T01:13:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.534+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132df4810 con 0x563132f15310 2023-09-10T01:13:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.534+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132cd88c0 con 0x563132f1b1b0 2023-09-10T01:13:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.535+0000 7f5e9cbf0b80 10 client.0 _ll_register_callbacks cb 0x563132d3ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.535+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f21e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.535+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f21e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48624/0 (socket says 172.21.15.3:48624) 2023-09-10T01:13:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.535+0000 7f5e8259c700 1 -- 172.21.15.3:0/720459574 learned_addr learned my addr 172.21.15.3:0/720459574 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.535+0000 7f5e8259c700 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f1b1b0 0x563132f1f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.535+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f1f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.536+0000 7f5e8259c700 1 -- 172.21.15.3:0/720459574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f1b1b0 msgr2=0x563132f1f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.536+0000 7f5e8259c700 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f1b1b0 0x563132f1f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.536+0000 7f5e8259c700 1 -- 172.21.15.3:0/720459574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 msgr2=0x563132f1f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.536+0000 7f5e8259c700 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f1f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.536+0000 7f5e8259c700 1 -- 172.21.15.3:0/720459574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8410dd40 con 0x563132f15310 2023-09-10T01:13:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.536+0000 7f5e8259c700 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f21e60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5e780046e0 tx=0x7f5e780047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c2d1bec7a56213 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.536+0000 7f5e66ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:13:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.538+0000 7f5e66ff5700 1 -- 172.21.15.3:0/720459574 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780142f0 con 0x563132f15310 2023-09-10T01:13:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.538+0000 7f5e66ff5700 1 -- 172.21.15.3:0/720459574 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f5e78003a90 con 0x563132f15310 2023-09-10T01:13:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.539+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563132dae490 con 0x563132f15310 2023-09-10T01:13:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.542+0000 7f5e66ff5700 1 -- 172.21.15.3:0/720459574 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e78003c50 con 0x563132f15310 2023-09-10T01:13:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.543+0000 7f5e9cbf0b80 10 client.27974 fetch_fsmap learned FSMap version 34 2023-09-10T01:13:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.543+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x563132c6bc00 con 0x563132f15310 2023-09-10T01:13:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.546+0000 7f5e66ff5700 1 -- 172.21.15.3:0/720459574 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563132dae490 con 0x563132f15310 2023-09-10T01:13:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.546+0000 7f5e66ff5700 1 -- 172.21.15.3:0/720459574 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 34) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5e7802b420 con 0x563132f15310 2023-09-10T01:13:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.547+0000 7f5e9cbf0b80 10 client.27974 fetch_fsmap finished waiting for FSMap version 34 2023-09-10T01:13:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.547+0000 7f5e9cbf0b80 10 client.27974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.547+0000 7f5e9cbf0b80 20 client.27974 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.547+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f2f310 0x563132f31730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.547+0000 7f5e9cbf0b80 4 client.27974 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:13:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.547+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563132e6b840 con 0x563132f2f310 2023-09-10T01:13:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.547+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f2f310 0x563132f31730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.552 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.551+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f2f310 0x563132f31730 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77f419a015ae0b89 server_cookie=64764d5e13cb2677 in_seq=0 out_seq=0 2023-09-10T01:13:54.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.553+0000 7f5e66ff5700 10 client.27974 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:54.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.553+0000 7f5e66ff5700 1 -- 172.21.15.3:0/720459574 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563132e6b840 con 0x563132f2f310 2023-09-10T01:13:54.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.553+0000 7f5e66ff5700 10 client.27974 handle_command_reply: tid=0 2023-09-10T01:13:54.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.553+0000 7f5e9cbf0b80 1 client.27974 shutdown 2023-09-10T01:13:54.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:54 smithi003 ceph-mon[97654]: pgmap v331: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-09-10T01:13:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.558+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 msgr2=0x563132f21e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.558+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f21e60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5e780046e0 tx=0x7f5e780047c0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.561+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 shutdown_connections 2023-09-10T01:13:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.561+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f2f310 0x563132f31730 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.561+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f1b1b0 0x563132f1f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.561+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f21e60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.561+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/720459574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f1f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.561+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 >> 172.21.15.3:0/720459574 conn(0x563132e79030 msgr2=0x563132efe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.563+0000 7f5e66ff5700 0 client.27974 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.564+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 shutdown_connections 2023-09-10T01:13:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.564+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/720459574 wait complete. 2023-09-10T01:13:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.565+0000 7f5e9cbf0b80 20 client.27974 trim_cache size 0 max 16384 2023-09-10T01:13:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.583+0000 7f5e9cbf0b80 1 Processor -- start 2023-09-10T01:13:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.583+0000 7f5e9cbf0b80 1 -- start start 2023-09-10T01:13:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f26460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15cd0 0x563132f21990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f17b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132ef6f60 con 0x563132f1b1b0 2023-09-10T01:13:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132ef98a0 con 0x563132f15310 2023-09-10T01:13:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132f16590 con 0x563132f15cd0 2023-09-10T01:13:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f26460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f17b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f26460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48626/0 (socket says 172.21.15.3:48626) 2023-09-10T01:13:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e8259c700 1 -- 172.21.15.3:0/63470473 learned_addr learned my addr 172.21.15.3:0/63470473 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.584+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15cd0 0x563132f21990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e8259c700 1 -- 172.21.15.3:0/63470473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15cd0 msgr2=0x563132f21990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e8259c700 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15cd0 0x563132f21990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e8259c700 1 -- 172.21.15.3:0/63470473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 msgr2=0x563132f17b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e8259c700 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f17b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e8259c700 1 -- 172.21.15.3:0/63470473 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563132ca91f0 con 0x563132f15310 2023-09-10T01:13:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e8259c700 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f26460 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5e7802ba10 tx=0x7f5e7800b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f112fa7065c4781 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e69ffb700 1 -- 172.21.15.3:0/63470473 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800b900 con 0x563132f15310 2023-09-10T01:13:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e69ffb700 1 -- 172.21.15.3:0/63470473 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e7801a070 con 0x563132f15310 2023-09-10T01:13:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.585+0000 7f5e69ffb700 1 -- 172.21.15.3:0/63470473 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7802bea0 con 0x563132f15310 2023-09-10T01:13:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/63470473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 msgr2=0x563132f26460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f26460 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5e7802ba10 tx=0x7f5e7800b540 comp rx=0 tx=0).stop 2023-09-10T01:13:54.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/63470473 shutdown_connections 2023-09-10T01:13:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f1b1b0 0x563132f17b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f15310 0x563132f26460 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/63470473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15cd0 0x563132f21990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/63470473 >> 172.21.15.3:0/63470473 conn(0x563132e79030 msgr2=0x563132f1a120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:54.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.586+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/63470473 shutdown_connections 2023-09-10T01:13:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.587+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/63470473 wait complete. 2023-09-10T01:13:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.587+0000 7f5e9cbf0b80 1 -- start start 2023-09-10T01:13:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.587+0000 7f5e9cbf0b80 1 Processor -- start 2023-09-10T01:13:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f16b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f170a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e9cbf0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f1b1b0 0x563132f175e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132f17b60 con 0x563132f15cd0 2023-09-10T01:13:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132f17ca0 con 0x563132f1b1b0 2023-09-10T01:13:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e9cbf0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563132f17e60 con 0x563132f15310 2023-09-10T01:13:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e9cbf0b80 10 client.0 _ll_register_callbacks cb 0x563132d3ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:13:54.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f1b1b0 0x563132f175e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e82d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f170a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e8259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f1b1b0 0x563132f175e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48642/0 (socket says 172.21.15.3:48642) 2023-09-10T01:13:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e8259c700 1 -- 172.21.15.3:0/3580723319 learned_addr learned my addr 172.21.15.3:0/3580723319 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:13:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e8259c700 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f16b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e82d9d700 1 -- 172.21.15.3:0/3580723319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 msgr2=0x563132f16b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f16b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e82d9d700 1 -- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f1b1b0 msgr2=0x563132f175e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:13:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f1b1b0 0x563132f175e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e82d9d700 1 -- 172.21.15.3:0/3580723319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563132c6bc00 con 0x563132f15cd0 2023-09-10T01:13:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.588+0000 7f5e8259c700 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f1b1b0 0x563132f175e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:13:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e82d9d700 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f170a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400ffa0 tx=0x7f5e74009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249f53401da8795d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:13:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e66ff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:13:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e66ff5700 1 -- 172.21.15.3:0/3580723319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74027040 con 0x563132f15cd0 2023-09-10T01:13:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e7400f9f0 con 0x563132f15cd0 2023-09-10T01:13:54.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e66ff5700 1 -- 172.21.15.3:0/3580723319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f5e740294b0 con 0x563132f15cd0 2023-09-10T01:13:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e66ff5700 1 -- 172.21.15.3:0/3580723319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e740097f0 con 0x563132f15cd0 2023-09-10T01:13:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e9cbf0b80 10 client.17889 fetch_fsmap learned FSMap version 34 2023-09-10T01:13:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x563132c81250 con 0x563132f15cd0 2023-09-10T01:13:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.589+0000 7f5e66ff5700 1 -- 172.21.15.3:0/3580723319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e7400f9f0 con 0x563132f15cd0 2023-09-10T01:13:54.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e66ff5700 1 -- 172.21.15.3:0/3580723319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5e74015020 con 0x563132f15cd0 2023-09-10T01:13:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e9cbf0b80 10 client.17889 fetch_fsmap finished waiting for FSMap version 34 2023-09-10T01:13:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e9cbf0b80 10 client.17889 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:13:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e9cbf0b80 20 client.17889 populate_metadata read hostname 'smithi003' 2023-09-10T01:13:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f83c40 0x563132f86060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:13:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e9cbf0b80 4 client.17889 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:13:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563132e4f8c0 con 0x563132f83c40 2023-09-10T01:13:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e8259c700 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f83c40 0x563132f86060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:13:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e8259c700 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f83c40 0x563132f86060 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19eea50f1d73013c server_cookie=b028cf9bce0b98fb in_seq=0 out_seq=0 2023-09-10T01:13:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.590+0000 7f5e66ff5700 10 client.17889 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:54.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:54 smithi136 ceph-mon[133708]: pgmap v331: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-09-10T01:13:54.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:54 smithi092 ceph-mon[134067]: pgmap v331: 129 pgs: 129 active+clean; 291 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-09-10T01:13:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.784+0000 7f5e66ff5700 1 -- 172.21.15.3:0/3580723319 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563132e4f8c0 con 0x563132f83c40 2023-09-10T01:13:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.784+0000 7f5e66ff5700 10 client.17889 handle_command_reply: tid=0 2023-09-10T01:13:54.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.784+0000 7f5e9cbf0b80 1 client.17889 shutdown 2023-09-10T01:13:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.785+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 msgr2=0x563132f170a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.785+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f170a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400ffa0 tx=0x7f5e74009c30 comp rx=0 tx=0).stop 2023-09-10T01:13:54.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.791+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 shutdown_connections 2023-09-10T01:13:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.791+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563132f83c40 0x563132f86060 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.791+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563132f15310 0x563132f16b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.791+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563132f1b1b0 0x563132f175e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.791+0000 7f5e9cbf0b80 1 --2- 172.21.15.3:0/3580723319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563132f15cd0 0x563132f170a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.791+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 >> 172.21.15.3:0/3580723319 conn(0x563132e79030 msgr2=0x563132f191d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:54.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.791+0000 7f5e66ff5700 0 client.17889 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:13:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.793+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 shutdown_connections 2023-09-10T01:13:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.793+0000 7f5e9cbf0b80 1 -- 172.21.15.3:0/3580723319 wait complete. 2023-09-10T01:13:54.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.794+0000 7f5e9cbf0b80 20 client.17889 trim_cache size 0 max 16384 2023-09-10T01:13:54.795 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951150 8.71 MB/sec execute 456 sec latency 183.986 ms 2023-09-10T01:13:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:13:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:13:54.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:13:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:13:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.795+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5e6c0a08e0 msgr2=0x7f5e6c0a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.795+0000 7f5e891ff700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5e6c0a08e0 0x7f5e6c0a2da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c094f90 tx=0x7f5e7c004be0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.795+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e84106680 msgr2=0x7f5e841a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:13:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.795+0000 7f5e891ff700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e84106680 0x7f5e841a05d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00b970 tx=0x7f5e7c00bc80 comp rx=0 tx=0).stop 2023-09-10T01:13:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 shutdown_connections 2023-09-10T01:13:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5e6c0a08e0 0x7f5e6c0a2da0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e841005c0 0x7f5e8419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e84106680 0x7f5e841a05d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 --2- 172.21.15.3:0/3970255784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e84100f80 0x7f5e8419d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:13:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 >> 172.21.15.3:0/3970255784 conn(0x7f5e840fc0b0 msgr2=0x7f5e840fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:13:54.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 shutdown_connections 2023-09-10T01:13:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:13:54.796+0000 7f5e891ff700 1 -- 172.21.15.3:0/3970255784 wait complete. 2023-09-10T01:13:55.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:55 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:55.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:55 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:55.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:55 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:13:55.795 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951552 8.70 MB/sec execute 457 sec latency 819.990 ms 2023-09-10T01:13:56.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:56 smithi003 ceph-mon[97654]: pgmap v332: 129 pgs: 129 active+clean; 297 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 326 op/s 2023-09-10T01:13:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:56 smithi136 ceph-mon[133708]: pgmap v332: 129 pgs: 129 active+clean; 297 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 326 op/s 2023-09-10T01:13:56.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:56 smithi092 ceph-mon[134067]: pgmap v332: 129 pgs: 129 active+clean; 297 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 326 op/s 2023-09-10T01:13:56.795 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951552 8.68 MB/sec execute 458 sec latency 1820.134 ms 2023-09-10T01:13:57.795 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951552 8.66 MB/sec execute 459 sec latency 2820.275 ms 2023-09-10T01:13:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:13:58 smithi003 ceph-mon[97654]: pgmap v333: 129 pgs: 129 active+clean; 297 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 178 op/s 2023-09-10T01:13:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:13:58 smithi136 ceph-mon[133708]: pgmap v333: 129 pgs: 129 active+clean; 297 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 178 op/s 2023-09-10T01:13:58.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:13:58 smithi092 ceph-mon[134067]: pgmap v333: 129 pgs: 129 active+clean; 297 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 178 op/s 2023-09-10T01:13:58.798 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951625 8.64 MB/sec execute 460 sec latency 3124.639 ms 2023-09-10T01:13:59.798 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951625 8.62 MB/sec execute 461 sec latency 1633.448 ms 2023-09-10T01:14:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:00 smithi003 ceph-mon[97654]: pgmap v334: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2023-09-10T01:14:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:00 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:14:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:00 smithi136 ceph-mon[133708]: pgmap v334: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2023-09-10T01:14:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:00 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:14:00.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:00 smithi092 ceph-mon[134067]: pgmap v334: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 307 op/s 2023-09-10T01:14:00.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:00 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:14:00.798 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951826 8.61 MB/sec execute 462 sec latency 2222.456 ms 2023-09-10T01:14:01.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951826 8.59 MB/sec execute 463 sec latency 1312.204 ms 2023-09-10T01:14:02.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:02 smithi003 ceph-mon[97654]: pgmap v335: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-09-10T01:14:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:02 smithi136 ceph-mon[133708]: pgmap v335: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-09-10T01:14:02.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:02 smithi092 ceph-mon[134067]: pgmap v335: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-09-10T01:14:02.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951826 8.57 MB/sec execute 464 sec latency 2312.366 ms 2023-09-10T01:14:03.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 951859 8.55 MB/sec execute 465 sec latency 2615.245 ms 2023-09-10T01:14:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:04 smithi003 ceph-mon[97654]: pgmap v336: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-09-10T01:14:04.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:04 smithi136 ceph-mon[133708]: pgmap v336: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-09-10T01:14:04.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:04 smithi092 ceph-mon[134067]: pgmap v336: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-09-10T01:14:04.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952418 8.54 MB/sec execute 466 sec latency 743.936 ms 2023-09-10T01:14:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:05 smithi136 ceph-mon[133708]: osdmap e100: 12 total, 12 up, 12 in 2023-09-10T01:14:05.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:05 smithi092 ceph-mon[134067]: osdmap e100: 12 total, 12 up, 12 in 2023-09-10T01:14:05.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952750 8.53 MB/sec execute 467 sec latency 78.326 ms 2023-09-10T01:14:05.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:05 smithi003 ceph-mon[97654]: osdmap e100: 12 total, 12 up, 12 in 2023-09-10T01:14:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:06 smithi136 ceph-mon[133708]: pgmap v338: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.7 MiB/s wr, 208 op/s 2023-09-10T01:14:06.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:06 smithi092 ceph-mon[134067]: pgmap v338: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.7 MiB/s wr, 208 op/s 2023-09-10T01:14:06.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952917 8.51 MB/sec execute 468 sec latency 835.607 ms 2023-09-10T01:14:06.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:06 smithi003 ceph-mon[97654]: pgmap v338: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.7 MiB/s wr, 208 op/s 2023-09-10T01:14:07.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952917 8.49 MB/sec execute 469 sec latency 1835.715 ms 2023-09-10T01:14:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 -- 172.21.15.3:0/2492480067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee08106680 msgr2=0x7fee08106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/2492480067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee08106680 0x7fee08106a60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fedf80099c0 tx=0x7fedf8009cd0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 -- 172.21.15.3:0/2492480067 shutdown_connections 2023-09-10T01:14:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/2492480067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee08100f80 0x7fee0810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/2492480067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee08106680 0x7fee08106a60 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/2492480067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee081005c0 0x7fee08100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 -- 172.21.15.3:0/2492480067 >> 172.21.15.3:0/2492480067 conn(0x7fee080fc0b0 msgr2=0x7fee080fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 -- 172.21.15.3:0/2492480067 shutdown_connections 2023-09-10T01:14:07.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.961+0000 7fee0e5ff700 1 -- 172.21.15.3:0/2492480067 wait complete. 2023-09-10T01:14:07.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.962+0000 7fee0e5ff700 1 Processor -- start 2023-09-10T01:14:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.962+0000 7fee0e5ff700 1 -- start start 2023-09-10T01:14:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.962+0000 7fee0e5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee081005c0 0x7fee081966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.962+0000 7fee0e5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee08100f80 0x7fee08196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.962+0000 7fee0e5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee08106680 0x7fee08199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee0e5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0819a590 con 0x7fee081005c0 2023-09-10T01:14:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee0e5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0819a700 con 0x7fee08106680 2023-09-10T01:14:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee0e5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0819a8c0 con 0x7fee08100f80 2023-09-10T01:14:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee081005c0 0x7fee081966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee081005c0 0x7fee081966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50708/0 (socket says 172.21.15.3:50708) 2023-09-10T01:14:07.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 -- 172.21.15.3:0/1243441057 learned_addr learned my addr 172.21.15.3:0/1243441057 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee077fe700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee08100f80 0x7fee08196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee0cb9c700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee08106680 0x7fee08199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 -- 172.21.15.3:0/1243441057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee08100f80 msgr2=0x7fee08196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee08100f80 0x7fee08196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 -- 172.21.15.3:0/1243441057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee08106680 msgr2=0x7fee08199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee08106680 0x7fee08199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee07fff700 1 -- 172.21.15.3:0/1243441057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80095f0 con 0x7fee081005c0 2023-09-10T01:14:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.963+0000 7fee077fe700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee08100f80 0x7fee08196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.964+0000 7fee07fff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee081005c0 0x7fee081966a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fedf80098e0 tx=0x7fedf800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:07.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.964+0000 7fee057fa700 1 -- 172.21.15.3:0/1243441057 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf801d070 con 0x7fee081005c0 2023-09-10T01:14:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.964+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0819ab40 con 0x7fee081005c0 2023-09-10T01:14:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.964+0000 7fee057fa700 1 -- 172.21.15.3:0/1243441057 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedf8022470 con 0x7fee081005c0 2023-09-10T01:14:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.964+0000 7fee057fa700 1 -- 172.21.15.3:0/1243441057 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8017650 con 0x7fee081005c0 2023-09-10T01:14:07.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.964+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0819ae10 con 0x7fee081005c0 2023-09-10T01:14:07.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.967+0000 7fee057fa700 1 -- 172.21.15.3:0/1243441057 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fedf8022850 con 0x7fee081005c0 2023-09-10T01:14:07.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.968+0000 7fee057fa700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fedf00a07b0 0x7fedf00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 Processor -- start 2023-09-10T01:14:07.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee057fa700 1 -- 172.21.15.3:0/1243441057 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fedf80ccec0 con 0x7fee081005c0 2023-09-10T01:14:07.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 -- start start 2023-09-10T01:14:07.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee077fe700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fedf00a07b0 0x7fedf00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7d8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7d2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7def30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef72b6d0 con 0x55d9ef7d81a0 2023-09-10T01:14:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef728ff0 con 0x55d9ef7d2cc0 2023-09-10T01:14:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.969+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef7248e0 con 0x55d9ef7d2300 2023-09-10T01:14:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7d8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7d2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7d8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50718/0 (socket says 172.21.15.3:50718) 2023-09-10T01:14:07.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7d2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:56052/0 (socket says 172.21.15.3:56052) 2023-09-10T01:14:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee07fff700 1 -- 172.21.15.3:0/553501866 learned_addr learned my addr 172.21.15.3:0/553501866 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fedf00a07b0 0x7fedf00a2c70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee08197cb0 tx=0x7fedfc009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:07.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee0cb9c700 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7def30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 -- 172.21.15.3:0/553501866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 msgr2=0x55d9ef7def30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7def30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 -- 172.21.15.3:0/553501866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 msgr2=0x55d9ef7d8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7d8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 -- 172.21.15.3:0/553501866 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9ef550d80 con 0x55d9ef7d2300 2023-09-10T01:14:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.970+0000 7fee077fe700 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7d2780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fedfc004130 tx=0x7fedfc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e524221139e44917 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:07.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.971+0000 7fedee7fc700 1 -- 172.21.15.3:0/553501866 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc003d10 con 0x55d9ef7d2300 2023-09-10T01:14:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.971+0000 7fedee7fc700 1 -- 172.21.15.3:0/553501866 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedfc003e70 con 0x55d9ef7d2300 2023-09-10T01:14:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.971+0000 7fedee7fc700 1 -- 172.21.15.3:0/553501866 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc01b700 con 0x55d9ef7d2300 2023-09-10T01:14:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.971+0000 7fee21ff1b80 1 -- 172.21.15.3:0/553501866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 msgr2=0x55d9ef7d2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.971+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7d2780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fedfc004130 tx=0x7fedfc009f90 comp rx=0 tx=0).stop 2023-09-10T01:14:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 -- 172.21.15.3:0/553501866 shutdown_connections 2023-09-10T01:14:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7def30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7d8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/553501866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7d2780 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 -- 172.21.15.3:0/553501866 >> 172.21.15.3:0/553501866 conn(0x55d9ef747030 msgr2=0x55d9ef7cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 -- 172.21.15.3:0/553501866 shutdown_connections 2023-09-10T01:14:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 -- 172.21.15.3:0/553501866 wait complete. 2023-09-10T01:14:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 -- start start 2023-09-10T01:14:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.972+0000 7fee21ff1b80 1 Processor -- start 2023-09-10T01:14:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7deda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7dc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef6c2b10 con 0x55d9ef7d2300 2023-09-10T01:14:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef6c2810 con 0x55d9ef7d81a0 2023-09-10T01:14:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef5a68c0 con 0x55d9ef7d2cc0 2023-09-10T01:14:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee21ff1b80 10 client.0 _ll_register_callbacks cb 0x55d9ef60ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7deda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7dc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7deda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50732/0 (socket says 172.21.15.3:50732) 2023-09-10T01:14:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee07fff700 1 -- 172.21.15.3:0/560171488 learned_addr learned my addr 172.21.15.3:0/560171488 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.973+0000 7fee0cb9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee07fff700 1 -- 172.21.15.3:0/560171488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2cc0 msgr2=0x55d9ef7dc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee07fff700 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7dc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee07fff700 1 -- 172.21.15.3:0/560171488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d81a0 msgr2=0x55d9ef7dc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee07fff700 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee07fff700 1 -- 172.21.15.3:0/560171488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc017050 con 0x55d9ef7d2300 2023-09-10T01:14:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee077fe700 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7dc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee07fff700 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7deda0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fedf8000c00 tx=0x7fedf8099a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfbc6e28d902293 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fede77fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fede77fe700 1 -- 172.21.15.3:0/560171488 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf80b1070 con 0x55d9ef7d2300 2023-09-10T01:14:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fede77fe700 1 -- 172.21.15.3:0/560171488 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fedf809a070 con 0x55d9ef7d2300 2023-09-10T01:14:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d9ef67c490 con 0x55d9ef7d2300 2023-09-10T01:14:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fede77fe700 1 -- 172.21.15.3:0/560171488 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf809dea0 con 0x55d9ef7d2300 2023-09-10T01:14:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee21ff1b80 10 client.17901 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.974+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55d9ef539c00 con 0x55d9ef7d2300 2023-09-10T01:14:07.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fede77fe700 1 -- 172.21.15.3:0/560171488 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d9ef67c490 con 0x55d9ef7d2300 2023-09-10T01:14:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fede77fe700 1 -- 172.21.15.3:0/560171488 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fedf8099560 con 0x55d9ef7d2300 2023-09-10T01:14:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fee21ff1b80 10 client.17901 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fee21ff1b80 10 client.17901 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fee21ff1b80 20 client.17901 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef7ec050 0x55d9ef7ee470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fee21ff1b80 4 client.17901 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9ef739840 con 0x55d9ef7ec050 2023-09-10T01:14:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.975+0000 7fee077fe700 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef7ec050 0x55d9ef7ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.976+0000 7fee077fe700 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef7ec050 0x55d9ef7ee470 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdf4f75ac366e7f0 server_cookie=fc9ee9b2d789fc01 in_seq=0 out_seq=0 2023-09-10T01:14:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.976+0000 7fede77fe700 10 client.17901 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fede77fe700 1 -- 172.21.15.3:0/560171488 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d9ef739840 con 0x55d9ef7ec050 2023-09-10T01:14:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fede77fe700 10 client.17901 handle_command_reply: tid=0 2023-09-10T01:14:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 client.17901 shutdown 2023-09-10T01:14:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 msgr2=0x55d9ef7deda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7deda0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fedf8000c00 tx=0x7fedf8099a50 comp rx=0 tx=0).stop 2023-09-10T01:14:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 shutdown_connections 2023-09-10T01:14:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef7ec050 0x55d9ef7ee470 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7deda0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/560171488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7dc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 >> 172.21.15.3:0/560171488 conn(0x55d9ef747030 msgr2=0x55d9ef7cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fede77fe700 0 client.17901 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 shutdown_connections 2023-09-10T01:14:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 1 -- 172.21.15.3:0/560171488 wait complete. 2023-09-10T01:14:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.977+0000 7fee21ff1b80 20 client.17901 trim_cache size 0 max 16384 2023-09-10T01:14:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 Processor -- start 2023-09-10T01:14:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 -- start start 2023-09-10T01:14:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7de8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7db6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef7c78e0 con 0x55d9ef7d81a0 2023-09-10T01:14:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef7d30d0 con 0x55d9ef7d2cc0 2023-09-10T01:14:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef7dd6f0 con 0x55d9ef7d2300 2023-09-10T01:14:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee0cb9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee0cb9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50744/0 (socket says 172.21.15.3:50744) 2023-09-10T01:14:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee0cb9c700 1 -- 172.21.15.3:0/4073099118 learned_addr learned my addr 172.21.15.3:0/4073099118 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee07fff700 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7de8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.993+0000 7fee077fe700 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7db6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fee0cb9c700 1 -- 172.21.15.3:0/4073099118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 msgr2=0x55d9ef7de8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fee0cb9c700 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7de8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fee0cb9c700 1 -- 172.21.15.3:0/4073099118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 msgr2=0x55d9ef7db6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fee0cb9c700 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7db6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fee0cb9c700 1 -- 172.21.15.3:0/4073099118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9ef5771f0 con 0x55d9ef7d81a0 2023-09-10T01:14:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fee0cb9c700 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dbc20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fee00011960 tx=0x7fee00009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de341af6e4e2b8a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fedeeffd700 1 -- 172.21.15.3:0/4073099118 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0000e040 con 0x55d9ef7d81a0 2023-09-10T01:14:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fedeeffd700 1 -- 172.21.15.3:0/4073099118 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee0000f040 con 0x55d9ef7d81a0 2023-09-10T01:14:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.994+0000 7fedeeffd700 1 -- 172.21.15.3:0/4073099118 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00017cb0 con 0x55d9ef7d81a0 2023-09-10T01:14:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 -- 172.21.15.3:0/4073099118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 msgr2=0x55d9ef7dbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dbc20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fee00011960 tx=0x7fee00009920 comp rx=0 tx=0).stop 2023-09-10T01:14:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 -- 172.21.15.3:0/4073099118 shutdown_connections 2023-09-10T01:14:08.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d2300 0x55d9ef7de8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7dbc20 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/4073099118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7db6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 -- 172.21.15.3:0/4073099118 >> 172.21.15.3:0/4073099118 conn(0x55d9ef747030 msgr2=0x55d9ef7cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 -- 172.21.15.3:0/4073099118 shutdown_connections 2023-09-10T01:14:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 -- 172.21.15.3:0/4073099118 wait complete. 2023-09-10T01:14:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 -- start start 2023-09-10T01:14:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.995+0000 7fee21ff1b80 1 Processor -- start 2023-09-10T01:14:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7e3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee21ff1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7e1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef7e2410 con 0x55d9ef7d2300 2023-09-10T01:14:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef7e2580 con 0x55d9ef7d2cc0 2023-09-10T01:14:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee21ff1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9ef844260 con 0x55d9ef7d81a0 2023-09-10T01:14:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee21ff1b80 10 client.0 _ll_register_callbacks cb 0x7fedf8099e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7e3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee07fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7e3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50756/0 (socket says 172.21.15.3:50756) 2023-09-10T01:14:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee07fff700 1 -- 172.21.15.3:0/261734665 learned_addr learned my addr 172.21.15.3:0/261734665 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee0cb9c700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7e1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.996+0000 7fee077fe700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee07fff700 1 -- 172.21.15.3:0/261734665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d81a0 msgr2=0x55d9ef7e1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee07fff700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7e1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee07fff700 1 -- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 msgr2=0x55d9ef7e17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee07fff700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7e17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee07fff700 1 -- 172.21.15.3:0/261734665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee00011610 con 0x55d9ef7d2300 2023-09-10T01:14:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee077fe700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7e17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee0cb9c700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7e1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee07fff700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7e3490 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fedf809a000 tx=0x7fedf8022bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93dab349a6f8d2f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fede77fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fede77fe700 1 -- 172.21.15.3:0/261734665 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf80b1070 con 0x55d9ef7d2300 2023-09-10T01:14:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fede77fe700 1 -- 172.21.15.3:0/261734665 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fedf809a070 con 0x55d9ef7d2300 2023-09-10T01:14:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedfc003e70 con 0x55d9ef7d2300 2023-09-10T01:14:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fede77fe700 1 -- 172.21.15.3:0/261734665 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf809dea0 con 0x55d9ef7d2300 2023-09-10T01:14:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee21ff1b80 10 client.17907 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.997+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55d9ef54f250 con 0x55d9ef7d2300 2023-09-10T01:14:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fede77fe700 1 -- 172.21.15.3:0/261734665 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fedfc003e70 con 0x55d9ef7d2300 2023-09-10T01:14:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fede77fe700 1 -- 172.21.15.3:0/261734665 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fedf800fad0 con 0x55d9ef7d2300 2023-09-10T01:14:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fee21ff1b80 10 client.17907 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fee21ff1b80 10 client.17907 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:08.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fee21ff1b80 20 client.17907 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef8453c0 0x55d9ef8477b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fee21ff1b80 4 client.17907 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:14:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d9ef71d8c0 con 0x55d9ef8453c0 2023-09-10T01:14:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.998+0000 7fee077fe700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef8453c0 0x55d9ef8477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.999+0000 7fee077fe700 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef8453c0 0x55d9ef8477b0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20ba2b7eba12f2ef server_cookie=a98142ff24a895c5 in_seq=0 out_seq=0 2023-09-10T01:14:08.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:07.999+0000 7fede77fe700 10 client.17907 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fede77fe700 1 -- 172.21.15.3:0/261734665 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d9ef71d8c0 con 0x55d9ef8453c0 2023-09-10T01:14:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fede77fe700 10 client.17907 handle_command_reply: tid=0 2023-09-10T01:14:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 client.17907 shutdown 2023-09-10T01:14:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 msgr2=0x55d9ef7e3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7e3490 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fedf809a000 tx=0x7fedf8022bf0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 shutdown_connections 2023-09-10T01:14:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d9ef8453c0 0x55d9ef8477b0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9ef7d81a0 0x55d9ef7e1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d9ef7d2300 0x55d9ef7e3490 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.039+0000 7fee21ff1b80 1 --2- 172.21.15.3:0/261734665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ef7d2cc0 0x55d9ef7e17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.040+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 >> 172.21.15.3:0/261734665 conn(0x55d9ef747030 msgr2=0x55d9ef7d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.040+0000 7fede77fe700 0 client.17907 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.040+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 shutdown_connections 2023-09-10T01:14:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.040+0000 7fee21ff1b80 1 -- 172.21.15.3:0/261734665 wait complete. 2023-09-10T01:14:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.040+0000 7fee21ff1b80 20 client.17907 trim_cache size 0 max 16384 2023-09-10T01:14:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:14:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:14:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:14:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:14:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fedf00a07b0 msgr2=0x7fedf00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fedf00a07b0 0x7fedf00a2c70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee08197cb0 tx=0x7fedfc009260 comp rx=0 tx=0).stop 2023-09-10T01:14:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee081005c0 msgr2=0x7fee081966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee081005c0 0x7fee081966a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fedf80098e0 tx=0x7fedf800f690 comp rx=0 tx=0).stop 2023-09-10T01:14:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 shutdown_connections 2023-09-10T01:14:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fedf00a07b0 0x7fedf00a2c70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee08106680 0x7fee08199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee081005c0 0x7fee081966a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 --2- 172.21.15.3:0/1243441057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee08100f80 0x7fee08196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 >> 172.21.15.3:0/1243441057 conn(0x7fee080fc0b0 msgr2=0x7fee08104250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 shutdown_connections 2023-09-10T01:14:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:08.041+0000 7fee0e5ff700 1 -- 172.21.15.3:0/1243441057 wait complete. 2023-09-10T01:14:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:08 smithi136 ceph-mon[133708]: pgmap v339: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.7 MiB/s wr, 208 op/s 2023-09-10T01:14:08.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:08 smithi092 ceph-mon[134067]: pgmap v339: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.7 MiB/s wr, 208 op/s 2023-09-10T01:14:08.799 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952985 8.47 MB/sec execute 470 sec latency 2137.224 ms 2023-09-10T01:14:08.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:08 smithi003 ceph-mon[97654]: pgmap v339: 129 pgs: 129 active+clean; 286 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.7 MiB/s wr, 208 op/s 2023-09-10T01:14:09.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952985 8.45 MB/sec execute 471 sec latency 1641.509 ms 2023-09-10T01:14:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:10 smithi136 ceph-mon[133708]: pgmap v340: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 3.5 MiB/s wr, 117 op/s 2023-09-10T01:14:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:10 smithi136 ceph-mon[133708]: osdmap e101: 12 total, 12 up, 12 in 2023-09-10T01:14:10.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:10 smithi092 ceph-mon[134067]: pgmap v340: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 3.5 MiB/s wr, 117 op/s 2023-09-10T01:14:10.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:10 smithi092 ceph-mon[134067]: osdmap e101: 12 total, 12 up, 12 in 2023-09-10T01:14:10.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952985 8.44 MB/sec execute 472 sec latency 2641.666 ms 2023-09-10T01:14:10.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:10 smithi003 ceph-mon[97654]: pgmap v340: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 3.5 MiB/s wr, 117 op/s 2023-09-10T01:14:10.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:10 smithi003 ceph-mon[97654]: osdmap e101: 12 total, 12 up, 12 in 2023-09-10T01:14:11.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952985 8.42 MB/sec execute 473 sec latency 3641.738 ms 2023-09-10T01:14:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:12 smithi136 ceph-mon[133708]: pgmap v342: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 4.4 MiB/s wr, 147 op/s 2023-09-10T01:14:12.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:12 smithi092 ceph-mon[134067]: pgmap v342: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 4.4 MiB/s wr, 147 op/s 2023-09-10T01:14:12.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952985 8.40 MB/sec execute 474 sec latency 4641.874 ms 2023-09-10T01:14:12.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:12 smithi003 ceph-mon[97654]: pgmap v342: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 4.4 MiB/s wr, 147 op/s 2023-09-10T01:14:13.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952986 8.38 MB/sec execute 475 sec latency 4955.026 ms 2023-09-10T01:14:13.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:14:14.101 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.100+0000 7f52c01ff700 1 -- 172.21.15.3:0/3167500287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8106650 msgr2=0x7f52b8106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.100+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3167500287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8106650 0x7f52b8106a30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f52b4009990 tx=0x7f52b4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.101+0000 7f52c01ff700 1 -- 172.21.15.3:0/3167500287 shutdown_connections 2023-09-10T01:14:14.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.101+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3167500287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8106650 0x7f52b8106a30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.101+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3167500287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52b8101000 0x7f52b810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.101+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3167500287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 0x7f52b8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.101+0000 7f52c01ff700 1 -- 172.21.15.3:0/3167500287 >> 172.21.15.3:0/3167500287 conn(0x7f52b80fc170 msgr2=0x7f52b80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.101+0000 7f52c01ff700 1 -- 172.21.15.3:0/3167500287 shutdown_connections 2023-09-10T01:14:14.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.101+0000 7f52c01ff700 1 -- 172.21.15.3:0/3167500287 wait complete. 2023-09-10T01:14:14.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 Processor -- start 2023-09-10T01:14:14.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 -- start start 2023-09-10T01:14:14.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 0x7f52b81966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8101000 0x7f52b8196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52b8106650 0x7f52b8199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b819a5b0 con 0x7f52b8106650 2023-09-10T01:14:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b819a720 con 0x7f52b8100640 2023-09-10T01:14:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.102+0000 7f52c01ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b819a8e0 con 0x7f52b8101000 2023-09-10T01:14:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 0x7f52b81966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 0x7f52b81966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:33052/0 (socket says 172.21.15.3:33052) 2023-09-10T01:14:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 -- 172.21.15.3:0/3531921798 learned_addr learned my addr 172.21.15.3:0/3531921798 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bd79a700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8101000 0x7f52b8196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52be79c700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52b8106650 0x7f52b8199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 -- 172.21.15.3:0/3531921798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8101000 msgr2=0x7f52b8196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8101000 0x7f52b8196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 -- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52b8106650 msgr2=0x7f52b8199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52b8106650 0x7f52b8199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 -- 172.21.15.3:0/3531921798 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b40095f0 con 0x7f52b8100640 2023-09-10T01:14:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52be79c700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52b8106650 0x7f52b8199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:14.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.103+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 0x7f52b81966c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000c00 tx=0x7f52b4004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.104+0000 7f52aeffd700 1 -- 172.21.15.3:0/3531921798 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b401d070 con 0x7f52b8100640 2023-09-10T01:14:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.104+0000 7f52aeffd700 1 -- 172.21.15.3:0/3531921798 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52b4021410 con 0x7f52b8100640 2023-09-10T01:14:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.104+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b819abc0 con 0x7f52b8100640 2023-09-10T01:14:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.104+0000 7f52aeffd700 1 -- 172.21.15.3:0/3531921798 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4017650 con 0x7f52b8100640 2023-09-10T01:14:14.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.104+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b81a3520 con 0x7f52b8100640 2023-09-10T01:14:14.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.107+0000 7f52aeffd700 1 -- 172.21.15.3:0/3531921798 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f52b4021580 con 0x7f52b8100640 2023-09-10T01:14:14.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.108+0000 7f52d3a30b80 1 Processor -- start 2023-09-10T01:14:14.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.108+0000 7f52aeffd700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f52a40a07b0 0x7f52a40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.108+0000 7f52d3a30b80 1 -- start start 2023-09-10T01:14:14.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52aeffd700 1 -- 172.21.15.3:0/3531921798 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f52b40cd6c0 con 0x7f52b8100640 2023-09-10T01:14:14.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52be79c700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f52a40a07b0 0x7f52a40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef5710 0x55f8e2ef5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef6030 0x55f8e2f03d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f06640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2e4e730 con 0x55f8e2ef5710 2023-09-10T01:14:14.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2e4c050 con 0x55f8e2f04250 2023-09-10T01:14:14.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2e47940 con 0x55f8e2ef6030 2023-09-10T01:14:14.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52bdf9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef5710 0x55f8e2ef5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52bdf9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef5710 0x55f8e2ef5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50778/0 (socket says 172.21.15.3:50778) 2023-09-10T01:14:14.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52bdf9b700 1 -- 172.21.15.3:0/588359999 learned_addr learned my addr 172.21.15.3:0/588359999 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52bd79a700 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef6030 0x55f8e2f03d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.109+0000 7f52be79c700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f52a40a07b0 0x7f52a40a2c70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f52b0003e80 tx=0x7f52b000b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52be79c700 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f06640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52bdf9b700 1 -- 172.21.15.3:0/588359999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef6030 msgr2=0x55f8e2f03d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef6030 0x55f8e2f03d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52bdf9b700 1 -- 172.21.15.3:0/588359999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 msgr2=0x55f8e2f06640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f06640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52bdf9b700 1 -- 172.21.15.3:0/588359999 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8e2c73de0 con 0x55f8e2ef5710 2023-09-10T01:14:14.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52be79c700 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f06640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:14.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52bd79a700 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef6030 0x55f8e2f03d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:14.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef5710 0x55f8e2ef5af0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f52b4096cb0 tx=0x7f52b409a670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2bcb898fd070fe1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52a3fff700 1 -- 172.21.15.3:0/588359999 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b409aeb0 con 0x55f8e2ef5710 2023-09-10T01:14:14.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52a3fff700 1 -- 172.21.15.3:0/588359999 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52b409a7b0 con 0x55f8e2ef5710 2023-09-10T01:14:14.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.110+0000 7f52a3fff700 1 -- 172.21.15.3:0/588359999 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b409e390 con 0x55f8e2ef5710 2023-09-10T01:14:14.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 -- 172.21.15.3:0/588359999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef5710 msgr2=0x55f8e2ef5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef5710 0x55f8e2ef5af0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f52b4096cb0 tx=0x7f52b409a670 comp rx=0 tx=0).stop 2023-09-10T01:14:14.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 -- 172.21.15.3:0/588359999 shutdown_connections 2023-09-10T01:14:14.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f06640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef5710 0x55f8e2ef5af0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/588359999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef6030 0x55f8e2f03d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 -- 172.21.15.3:0/588359999 >> 172.21.15.3:0/588359999 conn(0x55f8e2e6a090 msgr2=0x55f8e2ef0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 -- 172.21.15.3:0/588359999 shutdown_connections 2023-09-10T01:14:14.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.111+0000 7f52d3a30b80 1 -- 172.21.15.3:0/588359999 wait complete. 2023-09-10T01:14:14.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.112+0000 7f52d3a30b80 1 -- start start 2023-09-10T01:14:14.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.112+0000 7f52d3a30b80 1 Processor -- start 2023-09-10T01:14:14.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f04630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2ef6030 0x55f8e2f04b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2f04250 0x55f8e2f0f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2de5b70 con 0x55f8e2f04250 2023-09-10T01:14:14.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2de5870 con 0x55f8e2ef6030 2023-09-10T01:14:14.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2cc9920 con 0x55f8e2ef5710 2023-09-10T01:14:14.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52d3a30b80 10 client.0 _ll_register_callbacks cb 0x55f8e2d2fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:14.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52be79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2f04250 0x55f8e2f0f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52be79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2f04250 0x55f8e2f0f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50790/0 (socket says 172.21.15.3:50790) 2023-09-10T01:14:14.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52be79c700 1 -- 172.21.15.3:0/547224387 learned_addr learned my addr 172.21.15.3:0/547224387 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52bd79a700 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2ef6030 0x55f8e2f04b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.113+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f04630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52be79c700 1 -- 172.21.15.3:0/547224387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 msgr2=0x55f8e2f04630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52be79c700 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f04630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52be79c700 1 -- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2ef6030 msgr2=0x55f8e2f04b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52be79c700 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2ef6030 0x55f8e2f04b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52be79c700 1 -- 172.21.15.3:0/547224387 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a80098e0 con 0x55f8e2f04250 2023-09-10T01:14:14.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52be79c700 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2f04250 0x55f8e2f0f6b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f52b000ef60 tx=0x7f52b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62129443db71560 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52a17fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:14.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52a17fa700 1 -- 172.21.15.3:0/547224387 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0017bb0 con 0x55f8e2f04250 2023-09-10T01:14:14.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8e2d9f4f0 con 0x55f8e2f04250 2023-09-10T01:14:14.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52a17fa700 1 -- 172.21.15.3:0/547224387 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f52b001c040 con 0x55f8e2f04250 2023-09-10T01:14:14.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52a17fa700 1 -- 172.21.15.3:0/547224387 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52b0017d10 con 0x55f8e2f04250 2023-09-10T01:14:14.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.114+0000 7f52d3a30b80 10 client.17919 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:14.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f52b81a3520 con 0x55f8e2f04250 2023-09-10T01:14:14.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52a17fa700 1 -- 172.21.15.3:0/547224387 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f8e2d9f4f0 con 0x55f8e2f04250 2023-09-10T01:14:14.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52a17fa700 1 -- 172.21.15.3:0/547224387 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f52b0015410 con 0x55f8e2f04250 2023-09-10T01:14:14.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52d3a30b80 10 client.17919 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:14.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52d3a30b80 10 client.17919 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:14.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52d3a30b80 20 client.17919 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:14.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f13b30 0x55f8e2f15f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52d3a30b80 4 client.17919 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:14.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8e2e5c8a0 con 0x55f8e2f13b30 2023-09-10T01:14:14.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.115+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f13b30 0x55f8e2f15f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.116+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f13b30 0x55f8e2f15f50 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42229ce2d44a8b68 server_cookie=ced0ff7abb28ed8a in_seq=0 out_seq=0 2023-09-10T01:14:14.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.116+0000 7f52a17fa700 10 client.17919 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:14.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52a17fa700 1 -- 172.21.15.3:0/547224387 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f8e2e5c8a0 con 0x55f8e2f13b30 2023-09-10T01:14:14.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52a17fa700 10 client.17919 handle_command_reply: tid=0 2023-09-10T01:14:14.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 client.17919 shutdown 2023-09-10T01:14:14.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2f04250 msgr2=0x55f8e2f0f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2f04250 0x55f8e2f0f6b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f52b000ef60 tx=0x7f52b0004ad0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 shutdown_connections 2023-09-10T01:14:14.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f13b30 0x55f8e2f15f50 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f04630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2f04250 0x55f8e2f0f6b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/547224387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2ef6030 0x55f8e2f04b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52a17fa700 0 client.17919 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:14.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 >> 172.21.15.3:0/547224387 conn(0x55f8e2e6a090 msgr2=0x55f8e2ef07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 shutdown_connections 2023-09-10T01:14:14.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 1 -- 172.21.15.3:0/547224387 wait complete. 2023-09-10T01:14:14.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.117+0000 7f52d3a30b80 20 client.17919 trim_cache size 0 max 16384 2023-09-10T01:14:14.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 Processor -- start 2023-09-10T01:14:14.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 -- start start 2023-09-10T01:14:14.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2efdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f02b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f0edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2f11090 con 0x55f8e2ef6030 2023-09-10T01:14:14.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2ef69f0 con 0x55f8e2f04250 2023-09-10T01:14:14.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2f039d0 con 0x55f8e2ef5710 2023-09-10T01:14:14.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52bd79a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f02b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52bd79a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f02b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50804/0 (socket says 172.21.15.3:50804) 2023-09-10T01:14:14.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52bd79a700 1 -- 172.21.15.3:0/3117430412 learned_addr learned my addr 172.21.15.3:0/3117430412 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52be79c700 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f0edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.135+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2efdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52be79c700 1 -- 172.21.15.3:0/3117430412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 msgr2=0x55f8e2efdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52be79c700 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2efdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52be79c700 1 -- 172.21.15.3:0/3117430412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 msgr2=0x55f8e2f02b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52be79c700 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f02b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52be79c700 1 -- 172.21.15.3:0/3117430412 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8e2c5cc60 con 0x55f8e2f04250 2023-09-10T01:14:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52be79c700 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f0edd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f52b0000f80 tx=0x7f52b00049d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2855f1f3a67a18 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52acff9700 1 -- 172.21.15.3:0/3117430412 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0015410 con 0x55f8e2f04250 2023-09-10T01:14:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52acff9700 1 -- 172.21.15.3:0/3117430412 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52b000f040 con 0x55f8e2f04250 2023-09-10T01:14:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.136+0000 7f52acff9700 1 -- 172.21.15.3:0/3117430412 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0015d20 con 0x55f8e2f04250 2023-09-10T01:14:14.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 -- 172.21.15.3:0/3117430412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 msgr2=0x55f8e2f0edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f0edd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f52b0000f80 tx=0x7f52b00049d0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 -- 172.21.15.3:0/3117430412 shutdown_connections 2023-09-10T01:14:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2efdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f0edd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/3117430412 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f02b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 -- 172.21.15.3:0/3117430412 >> 172.21.15.3:0/3117430412 conn(0x55f8e2e6a090 msgr2=0x55f8e2f04630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 -- 172.21.15.3:0/3117430412 shutdown_connections 2023-09-10T01:14:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 -- 172.21.15.3:0/3117430412 wait complete. 2023-09-10T01:14:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 -- start start 2023-09-10T01:14:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.137+0000 7f52d3a30b80 1 Processor -- start 2023-09-10T01:14:14.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f09550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f05990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52d3a30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f05ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2f064d0 con 0x55f8e2ef6030 2023-09-10T01:14:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2f06640 con 0x55f8e2f04250 2023-09-10T01:14:14.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52d3a30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e2f06800 con 0x55f8e2ef5710 2023-09-10T01:14:14.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52d3a30b80 10 client.0 _ll_register_callbacks cb 0x7f52b000e890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:14.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52bdf9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f09550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52bdf9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f09550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:56100/0 (socket says 172.21.15.3:56100) 2023-09-10T01:14:14.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52bdf9b700 1 -- 172.21.15.3:0/1172043630 learned_addr learned my addr 172.21.15.3:0/1172043630 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52be79c700 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f05ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.138+0000 7f52bd79a700 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f05990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52bdf9b700 1 -- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 msgr2=0x55f8e2f05ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f05ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52bdf9b700 1 -- 172.21.15.3:0/1172043630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 msgr2=0x55f8e2f05990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f05990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52bdf9b700 1 -- 172.21.15.3:0/1172043630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b81a3520 con 0x55f8e2ef5710 2023-09-10T01:14:14.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52bdf9b700 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f09550 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52b400b6f0 tx=0x7f52b409a670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86af6ad43261b2fa server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52a17fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:14.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52a17fa700 1 -- 172.21.15.3:0/1172043630 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b4021a60 con 0x55f8e2ef5710 2023-09-10T01:14:14.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52b409a7b0 con 0x55f8e2ef5710 2023-09-10T01:14:14.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52a17fa700 1 -- 172.21.15.3:0/1172043630 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f52b40a2790 con 0x55f8e2ef5710 2023-09-10T01:14:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52a17fa700 1 -- 172.21.15.3:0/1172043630 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52b409a2f0 con 0x55f8e2ef5710 2023-09-10T01:14:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52d3a30b80 10 client.27901 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.139+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55f8e2c9a250 con 0x55f8e2ef5710 2023-09-10T01:14:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52a17fa700 1 -- 172.21.15.3:0/1172043630 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52b409a7b0 con 0x55f8e2ef5710 2023-09-10T01:14:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52a17fa700 1 -- 172.21.15.3:0/1172043630 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f52b40b2d20 con 0x55f8e2ef5710 2023-09-10T01:14:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52d3a30b80 10 client.27901 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52d3a30b80 10 client.27901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52d3a30b80 20 client.27901 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f68b10 0x55f8e2f6af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52d3a30b80 4 client.27901 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:14:14.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f8e2e40920 con 0x55f8e2f68b10 2023-09-10T01:14:14.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.140+0000 7f52bd79a700 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f68b10 0x55f8e2f6af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52bd79a700 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f68b10 0x55f8e2f6af30 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c243b29c4bfaccd9 server_cookie=a05cbf7950ccf5a8 in_seq=0 out_seq=0 2023-09-10T01:14:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52a17fa700 10 client.27901 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:14.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52a17fa700 1 -- 172.21.15.3:0/1172043630 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f8e2e40920 con 0x55f8e2f68b10 2023-09-10T01:14:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52a17fa700 10 client.27901 handle_command_reply: tid=0 2023-09-10T01:14:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 client.27901 shutdown 2023-09-10T01:14:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 msgr2=0x55f8e2f09550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f09550 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52b400b6f0 tx=0x7f52b409a670 comp rx=0 tx=0).stop 2023-09-10T01:14:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 shutdown_connections 2023-09-10T01:14:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8e2f68b10 0x55f8e2f6af30 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8e2ef5710 0x55f8e2f09550 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8e2f04250 0x55f8e2f05ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52a17fa700 0 client.27901 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:14.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 --2- 172.21.15.3:0/1172043630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8e2ef6030 0x55f8e2f05990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 >> 172.21.15.3:0/1172043630 conn(0x55f8e2e6a090 msgr2=0x55f8e2ef4700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 shutdown_connections 2023-09-10T01:14:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 1 -- 172.21.15.3:0/1172043630 wait complete. 2023-09-10T01:14:14.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.141+0000 7f52d3a30b80 20 client.27901 trim_cache size 0 max 16384 2023-09-10T01:14:14.156 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:14:14.157 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:14:14.157 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:14:14.157 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:14:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f52a40a07b0 msgr2=0x7f52a40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f52a40a07b0 0x7f52a40a2c70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f52b0003e80 tx=0x7f52b000b040 comp rx=0 tx=0).stop 2023-09-10T01:14:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 msgr2=0x7f52b81966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 0x7f52b81966c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000c00 tx=0x7f52b4004790 comp rx=0 tx=0).stop 2023-09-10T01:14:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 shutdown_connections 2023-09-10T01:14:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f52a40a07b0 0x7f52a40a2c70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52b8106650 0x7f52b8199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52b8100640 0x7f52b81966c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 --2- 172.21.15.3:0/3531921798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b8101000 0x7f52b8196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 >> 172.21.15.3:0/3531921798 conn(0x7f52b80fc170 msgr2=0x7f52b8104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 shutdown_connections 2023-09-10T01:14:14.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.142+0000 7f52c01ff700 1 -- 172.21.15.3:0/3531921798 wait complete. 2023-09-10T01:14:14.166 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:14:14.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:14:14.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 -- 172.21.15.3:0/1366791181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c106640 msgr2=0x7f677c106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 --2- 172.21.15.3:0/1366791181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c106640 0x7f677c106a20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f676c009990 tx=0x7f676c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 -- 172.21.15.3:0/1366791181 shutdown_connections 2023-09-10T01:14:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 --2- 172.21.15.3:0/1366791181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f677c100ff0 0x7f677c10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 --2- 172.21.15.3:0/1366791181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c106640 0x7f677c106a20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 --2- 172.21.15.3:0/1366791181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f677c100630 0x7f677c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 -- 172.21.15.3:0/1366791181 >> 172.21.15.3:0/1366791181 conn(0x7f677c0fc140 msgr2=0x7f677c0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 -- 172.21.15.3:0/1366791181 shutdown_connections 2023-09-10T01:14:14.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.452+0000 7f67823ff700 1 -- 172.21.15.3:0/1366791181 wait complete. 2023-09-10T01:14:14.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.453+0000 7f67823ff700 1 Processor -- start 2023-09-10T01:14:14.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.453+0000 7f67823ff700 1 -- start start 2023-09-10T01:14:14.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f67823ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c100630 0x7f677c196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f67823ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f677c100ff0 0x7f677c196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f67823ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f677c106640 0x7f677c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f67823ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c19a600 con 0x7f677c100630 2023-09-10T01:14:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f67823ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c19a770 con 0x7f677c100ff0 2023-09-10T01:14:14.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f67823ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c19a930 con 0x7f677c106640 2023-09-10T01:14:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f677bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c100630 0x7f677c196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f677bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c100630 0x7f677c196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50830/0 (socket says 172.21.15.3:50830) 2023-09-10T01:14:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f677bfff700 1 -- 172.21.15.3:0/245587240 learned_addr learned my addr 172.21.15.3:0/245587240 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f677b7fe700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f677c100ff0 0x7f677c196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.454+0000 7f678099c700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f677c106640 0x7f677c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f677bfff700 1 -- 172.21.15.3:0/245587240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f677c106640 msgr2=0x7f677c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f677bfff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f677c106640 0x7f677c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f677bfff700 1 -- 172.21.15.3:0/245587240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f677c100ff0 msgr2=0x7f677c196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f677bfff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f677c100ff0 0x7f677c196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f677bfff700 1 -- 172.21.15.3:0/245587240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c0095f0 con 0x7f677c100630 2023-09-10T01:14:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f677bfff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c100630 0x7f677c196710 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f676c000c00 tx=0x7f676c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f67797fa700 1 -- 172.21.15.3:0/245587240 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c01d070 con 0x7f677c100630 2023-09-10T01:14:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f67797fa700 1 -- 172.21.15.3:0/245587240 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f676c022470 con 0x7f677c100630 2023-09-10T01:14:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c19abb0 con 0x7f677c100630 2023-09-10T01:14:14.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f67797fa700 1 -- 172.21.15.3:0/245587240 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c017650 con 0x7f677c100630 2023-09-10T01:14:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.455+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c1a3570 con 0x7f677c100630 2023-09-10T01:14:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.456+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c067480 con 0x7f677c100630 2023-09-10T01:14:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.458+0000 7f67797fa700 1 -- 172.21.15.3:0/245587240 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f676c0225e0 con 0x7f677c100630 2023-09-10T01:14:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.460+0000 7f67797fa700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f67640a07b0 0x7f67640a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.460+0000 7f67797fa700 1 -- 172.21.15.3:0/245587240 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f676c0cd790 con 0x7f677c100630 2023-09-10T01:14:14.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.461+0000 7f677b7fe700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f67640a07b0 0x7f67640a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.461+0000 7f677b7fe700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f67640a07b0 0x7f67640a2c70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f677c197d20 tx=0x7f6770009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.465+0000 7f67797fa700 1 -- 172.21.15.3:0/245587240 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f676c096c80 con 0x7f677c100630 2023-09-10T01:14:14.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:14 smithi003 ceph-mon[97654]: pgmap v343: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 3.5 MiB/s wr, 78 op/s 2023-09-10T01:14:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:14 smithi136 ceph-mon[133708]: pgmap v343: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 3.5 MiB/s wr, 78 op/s 2023-09-10T01:14:14.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:14 smithi092 ceph-mon[134067]: pgmap v343: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 3.5 MiB/s wr, 78 op/s 2023-09-10T01:14:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.734+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f677c110520 con 0x7f677c100630 2023-09-10T01:14:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.734+0000 7f67797fa700 1 -- 172.21.15.3:0/245587240 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+11297 (secure 0 0 0) 0x7f676c09a2a0 con 0x7f677c100630 2023-09-10T01:14:14.735 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:14:14.736 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:13:58.841353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:14:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f67640a07b0 msgr2=0x7f67640a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f67640a07b0 0x7f67640a2c70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f677c197d20 tx=0x7f6770009260 comp rx=0 tx=0).stop 2023-09-10T01:14:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c100630 msgr2=0x7f677c196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c100630 0x7f677c196710 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f676c000c00 tx=0x7f676c005740 comp rx=0 tx=0).stop 2023-09-10T01:14:14.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 shutdown_connections 2023-09-10T01:14:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f67640a07b0 0x7f67640a2c70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f677c106640 0x7f677c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f677c100630 0x7f677c196710 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 --2- 172.21.15.3:0/245587240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f677c100ff0 0x7f677c196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 >> 172.21.15.3:0/245587240 conn(0x7f677c0fc140 msgr2=0x7f677c0fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 shutdown_connections 2023-09-10T01:14:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.737+0000 7f67823ff700 1 -- 172.21.15.3:0/245587240 wait complete. 2023-09-10T01:14:14.740 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2023-09-10T01:14:14.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:14:14.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952986 8.37 MB/sec execute 476 sec latency 1687.054 ms 2023-09-10T01:14:14.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 -- 172.21.15.3:0/63724425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c100640 msgr2=0x7fcb8c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 --2- 172.21.15.3:0/63724425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c100640 0x7fcb8c100ac0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcb740098f0 tx=0x7fcb74009c00 comp rx=0 tx=0).stop 2023-09-10T01:14:14.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 -- 172.21.15.3:0/63724425 shutdown_connections 2023-09-10T01:14:14.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 --2- 172.21.15.3:0/63724425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 0x7fcb8c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 --2- 172.21.15.3:0/63724425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb8c106650 0x7fcb8c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 --2- 172.21.15.3:0/63724425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c100640 0x7fcb8c100ac0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 -- 172.21.15.3:0/63724425 >> 172.21.15.3:0/63724425 conn(0x7fcb8c0fc170 msgr2=0x7fcb8c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 -- 172.21.15.3:0/63724425 shutdown_connections 2023-09-10T01:14:14.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.963+0000 7fcb91fff700 1 -- 172.21.15.3:0/63724425 wait complete. 2023-09-10T01:14:14.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.965+0000 7fcb91fff700 1 Processor -- start 2023-09-10T01:14:14.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.965+0000 7fcb91fff700 1 -- start start 2023-09-10T01:14:14.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb91fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb8c100640 0x7fcb8c196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb91fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 0x7fcb8c196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb91fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c106650 0x7fcb8c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb91fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c19a610 con 0x7fcb8c100640 2023-09-10T01:14:14.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb91fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c19a780 con 0x7fcb8c101000 2023-09-10T01:14:14.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb91fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c19a940 con 0x7fcb8c106650 2023-09-10T01:14:14.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb8c100640 0x7fcb8c196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 0x7fcb8c196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 0x7fcb8c196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50748/0 (socket says 172.21.15.3:50748) 2023-09-10T01:14:14.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 -- 172.21.15.3:0/274373276 learned_addr learned my addr 172.21.15.3:0/274373276 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c106650 0x7fcb8c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 -- 172.21.15.3:0/274373276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c106650 msgr2=0x7fcb8c19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c106650 0x7fcb8c19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 -- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb8c100640 msgr2=0x7fcb8c196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb8c100640 0x7fcb8c196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.966+0000 7fcb83fff700 1 -- 172.21.15.3:0/274373276 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0095f0 con 0x7fcb8c101000 2023-09-10T01:14:14.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.967+0000 7fcb83fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 0x7fcb8c196c60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009230 tx=0x7fcb7400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.967+0000 7fcb897fa700 1 -- 172.21.15.3:0/274373276 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7401c070 con 0x7fcb8c101000 2023-09-10T01:14:14.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.967+0000 7fcb897fa700 1 -- 172.21.15.3:0/274373276 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb740058e0 con 0x7fcb8c101000 2023-09-10T01:14:14.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.967+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb74009520 con 0x7fcb8c101000 2023-09-10T01:14:14.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.967+0000 7fcb897fa700 1 -- 172.21.15.3:0/274373276 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb740038e0 con 0x7fcb8c101000 2023-09-10T01:14:14.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.967+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c1a33a0 con 0x7fcb8c101000 2023-09-10T01:14:14.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.970+0000 7fcb897fa700 1 -- 172.21.15.3:0/274373276 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fcb74020030 con 0x7fcb8c101000 2023-09-10T01:14:14.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.971+0000 7fcb897fa700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcb780a07b0 0x7fcb780a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.971+0000 7fcba5924b80 1 Processor -- start 2023-09-10T01:14:14.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcb897fa700 1 -- 172.21.15.3:0/274373276 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fcb740cd230 con 0x7fcb8c101000 2023-09-10T01:14:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcba5924b80 1 -- start start 2023-09-10T01:14:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcb780a07b0 0x7fcb780a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b6faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b70030 0x5597f2b7dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b7e250 0x5597f2b80640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcba5924b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2ac8730 con 0x5597f2b7e250 2023-09-10T01:14:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcba5924b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2ac6050 con 0x5597f2b6f710 2023-09-10T01:14:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcba5924b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2ac1940 con 0x5597f2b70030 2023-09-10T01:14:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcb83fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b70030 0x5597f2b7dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcb8bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b7e250 0x5597f2b80640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcb83fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b70030 0x5597f2b7dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57786/0 (socket says 172.21.15.3:57786) 2023-09-10T01:14:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.972+0000 7fcb83fff700 1 -- 172.21.15.3:0/3959753513 learned_addr learned my addr 172.21.15.3:0/3959753513 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b6faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb83fff700 1 -- 172.21.15.3:0/3959753513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 msgr2=0x5597f2b6faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb83fff700 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b6faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb83fff700 1 -- 172.21.15.3:0/3959753513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b7e250 msgr2=0x5597f2b80640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb83fff700 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b7e250 0x5597f2b80640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb83fff700 1 -- 172.21.15.3:0/3959753513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597f28edde0 con 0x5597f2b70030 2023-09-10T01:14:14.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b6faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:14:14.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcb780a07b0 0x7fcb780a2c70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c005fd0 tx=0x7fcb7c011720 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb83fff700 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b70030 0x5597f2b7dd10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcb740a1dd0 tx=0x7fcb74009470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61871247cf678b73 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb81ffb700 1 -- 172.21.15.3:0/3959753513 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb740cd030 con 0x5597f2b70030 2023-09-10T01:14:14.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb81ffb700 1 -- 172.21.15.3:0/3959753513 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb7409ee90 con 0x5597f2b70030 2023-09-10T01:14:14.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.973+0000 7fcb81ffb700 1 -- 172.21.15.3:0/3959753513 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb740a2370 con 0x5597f2b70030 2023-09-10T01:14:14.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 -- 172.21.15.3:0/3959753513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b70030 msgr2=0x5597f2b7dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b70030 0x5597f2b7dd10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcb740a1dd0 tx=0x7fcb74009470 comp rx=0 tx=0).stop 2023-09-10T01:14:14.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 -- 172.21.15.3:0/3959753513 shutdown_connections 2023-09-10T01:14:14.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b7e250 0x5597f2b80640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b6faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3959753513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b70030 0x5597f2b7dd10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 -- 172.21.15.3:0/3959753513 >> 172.21.15.3:0/3959753513 conn(0x5597f2ae4090 msgr2=0x5597f2b6abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 -- 172.21.15.3:0/3959753513 shutdown_connections 2023-09-10T01:14:14.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.974+0000 7fcba5924b80 1 -- 172.21.15.3:0/3959753513 wait complete. 2023-09-10T01:14:14.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 -- start start 2023-09-10T01:14:14.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 Processor -- start 2023-09-10T01:14:14.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b896b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2a5fb70 con 0x5597f2b6f710 2023-09-10T01:14:14.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2a5f870 con 0x5597f2b70030 2023-09-10T01:14:14.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2943920 con 0x5597f2b7e250 2023-09-10T01:14:14.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.975+0000 7fcba5924b80 10 client.0 _ll_register_callbacks cb 0x5597f29a9a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:14.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49350/0 (socket says 172.21.15.3:49350) 2023-09-10T01:14:14.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8b7fe700 1 -- 172.21.15.3:0/3729027463 learned_addr learned my addr 172.21.15.3:0/3729027463 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:14.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b896b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb83fff700 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8bfff700 1 -- 172.21.15.3:0/3729027463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b70030 msgr2=0x5597f2b7eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8bfff700 1 -- 172.21.15.3:0/3729027463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b6f710 msgr2=0x5597f2b7e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:14.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8bfff700 1 -- 172.21.15.3:0/3729027463 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c1a33a0 con 0x5597f2b7e250 2023-09-10T01:14:14.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b896b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcb84007b50 tx=0x7fcb84007b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e2bde4f0b72069e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:14.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.976+0000 7fcb6affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:14.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcb6affd700 1 -- 172.21.15.3:0/3729027463 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8400e070 con 0x5597f2b7e250 2023-09-10T01:14:14.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597f2a194f0 con 0x5597f2b7e250 2023-09-10T01:14:14.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcb6affd700 1 -- 172.21.15.3:0/3729027463 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fcb84003a00 con 0x5597f2b7e250 2023-09-10T01:14:14.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcb6affd700 1 -- 172.21.15.3:0/3729027463 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb84003bc0 con 0x5597f2b7e250 2023-09-10T01:14:14.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 10 client.27919 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:14.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5597f28d6c60 con 0x5597f2b7e250 2023-09-10T01:14:14.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcb6affd700 1 -- 172.21.15.3:0/3729027463 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5597f2a194f0 con 0x5597f2b7e250 2023-09-10T01:14:14.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcb6affd700 1 -- 172.21.15.3:0/3729027463 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fcb840167f0 con 0x5597f2b7e250 2023-09-10T01:14:14.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 10 client.27919 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:14.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 10 client.27919 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:14:14.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 20 client.27919 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:14.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2b8da20 0x5597f2b8fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:14.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 4 client.27919 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:14.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.977+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597f2ad68a0 con 0x5597f2b8da20 2023-09-10T01:14:14.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.978+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2b8da20 0x5597f2b8fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:14.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.978+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2b8da20 0x5597f2b8fe40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7832a9c4abb823b server_cookie=1a19fb07d6f804bd in_seq=0 out_seq=0 2023-09-10T01:14:14.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.978+0000 7fcb6affd700 10 client.27919 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:14.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcb6affd700 1 -- 172.21.15.3:0/3729027463 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5597f2ad68a0 con 0x5597f2b8da20 2023-09-10T01:14:14.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcb6affd700 10 client.27919 handle_command_reply: tid=0 2023-09-10T01:14:14.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 client.27919 shutdown 2023-09-10T01:14:14.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 msgr2=0x5597f2b896b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:14.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b896b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcb84007b50 tx=0x7fcb84007b80 comp rx=0 tx=0).stop 2023-09-10T01:14:14.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 shutdown_connections 2023-09-10T01:14:14.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2b8da20 0x5597f2b8fe40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b896b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3729027463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:14.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcb6affd700 0 client.27919 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:14.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 >> 172.21.15.3:0/3729027463 conn(0x5597f2ae4090 msgr2=0x5597f2b6a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:14.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 shutdown_connections 2023-09-10T01:14:14.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.979+0000 7fcba5924b80 1 -- 172.21.15.3:0/3729027463 wait complete. 2023-09-10T01:14:14.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.980+0000 7fcba5924b80 20 client.27919 trim_cache size 0 max 16384 2023-09-10T01:14:15.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 Processor -- start 2023-09-10T01:14:15.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 -- start start 2023-09-10T01:14:15.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b88da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2b7da00 con 0x5597f2b70030 2023-09-10T01:14:15.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2b79020 con 0x5597f2b6f710 2023-09-10T01:14:15.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcba5924b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2b8b080 con 0x5597f2b7e250 2023-09-10T01:14:15.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcb83fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.993+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b88da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50774/0 (socket says 172.21.15.3:50774) 2023-09-10T01:14:15.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb83fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49358/0 (socket says 172.21.15.3:49358) 2023-09-10T01:14:15.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8b7fe700 1 -- 172.21.15.3:0/3456159696 learned_addr learned my addr 172.21.15.3:0/3456159696 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8bfff700 1 -- 172.21.15.3:0/3456159696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 msgr2=0x5597f2b7e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8bfff700 1 -- 172.21.15.3:0/3456159696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 msgr2=0x5597f2b7eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8bfff700 1 -- 172.21.15.3:0/3456159696 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597f2914250 con 0x5597f2b7e250 2023-09-10T01:14:15.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb83fff700 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:15.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b88da0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcb840095b0 tx=0x7fcb84009a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23ddc751db1afa25 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb827fc700 1 -- 172.21.15.3:0/3456159696 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8400e070 con 0x5597f2b7e250 2023-09-10T01:14:15.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb827fc700 1 -- 172.21.15.3:0/3456159696 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb8400f040 con 0x5597f2b7e250 2023-09-10T01:14:15.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.994+0000 7fcb827fc700 1 -- 172.21.15.3:0/3456159696 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8401e560 con 0x5597f2b7e250 2023-09-10T01:14:15.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 -- 172.21.15.3:0/3456159696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 msgr2=0x5597f2b88da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b88da0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcb840095b0 tx=0x7fcb84009a50 comp rx=0 tx=0).stop 2023-09-10T01:14:15.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 -- 172.21.15.3:0/3456159696 shutdown_connections 2023-09-10T01:14:15.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b7e250 0x5597f2b88da0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b6f710 0x5597f2b7e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 --2- 172.21.15.3:0/3456159696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b7eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 -- 172.21.15.3:0/3456159696 >> 172.21.15.3:0/3456159696 conn(0x5597f2ae4090 msgr2=0x5597f2b77cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 -- 172.21.15.3:0/3456159696 shutdown_connections 2023-09-10T01:14:15.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 -- 172.21.15.3:0/3456159696 wait complete. 2023-09-10T01:14:15.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.995+0000 7fcba5924b80 1 -- start start 2023-09-10T01:14:15.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 1 Processor -- start 2023-09-10T01:14:15.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b6f710 0x5597f2b7ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b80490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b7e250 0x5597f2b809f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2be1a10 con 0x5597f2b70030 2023-09-10T01:14:15.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2be1b80 con 0x5597f2b7e250 2023-09-10T01:14:15.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597f2be1d40 con 0x5597f2b6f710 2023-09-10T01:14:15.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcba5924b80 10 client.0 _ll_register_callbacks cb 0x7fcb84007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:15.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b6f710 0x5597f2b7ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcb83fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b80490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b6f710 0x5597f2b7ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57808/0 (socket says 172.21.15.3:57808) 2023-09-10T01:14:15.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcb8b7fe700 1 -- 172.21.15.3:0/2881728811 learned_addr learned my addr 172.21.15.3:0/2881728811 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.996+0000 7fcb8bfff700 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b7e250 0x5597f2b809f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb8b7fe700 1 -- 172.21.15.3:0/2881728811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b7e250 msgr2=0x5597f2b809f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b7e250 0x5597f2b809f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb8b7fe700 1 -- 172.21.15.3:0/2881728811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 msgr2=0x5597f2b80490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b80490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb8b7fe700 1 -- 172.21.15.3:0/2881728811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597f28edde0 con 0x5597f2b6f710 2023-09-10T01:14:15.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb83fff700 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b80490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:15.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb8b7fe700 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b6f710 0x5597f2b7ff50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c005930 tx=0x7fcb7c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6081e472dd744bd5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb6affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:15.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcb6affd700 1 -- 172.21.15.3:0/2881728811 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00fb20 con 0x5597f2b6f710 2023-09-10T01:14:15.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.997+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb7409ee90 con 0x5597f2b6f710 2023-09-10T01:14:15.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcb6affd700 1 -- 172.21.15.3:0/2881728811 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fcb7c00fc80 con 0x5597f2b6f710 2023-09-10T01:14:15.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcb6affd700 1 -- 172.21.15.3:0/2881728811 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb7c00fe40 con 0x5597f2b6f710 2023-09-10T01:14:15.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 10 client.27925 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:15.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5597f28d6c60 con 0x5597f2b6f710 2023-09-10T01:14:15.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcb6affd700 1 -- 172.21.15.3:0/2881728811 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb7409ee90 con 0x5597f2b6f710 2023-09-10T01:14:15.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcb6affd700 1 -- 172.21.15.3:0/2881728811 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fcb7c01f7c0 con 0x5597f2b6f710 2023-09-10T01:14:15.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 10 client.27925 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:15.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 10 client.27925 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:14:15.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 20 client.27925 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:15.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2be34c0 0x5597f2be58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 4 client.27925 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:14:15.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5597f2aba920 con 0x5597f2be34c0 2023-09-10T01:14:15.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.998+0000 7fcb83fff700 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2be34c0 0x5597f2be58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcb83fff700 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2be34c0 0x5597f2be58b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c87a9d60b1fd3d86 server_cookie=8b7bf0dabe15c947 in_seq=0 out_seq=0 2023-09-10T01:14:15.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcb6affd700 10 client.27925 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:15.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcb6affd700 1 -- 172.21.15.3:0/2881728811 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5597f2aba920 con 0x5597f2be34c0 2023-09-10T01:14:15.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcb6affd700 10 client.27925 handle_command_reply: tid=0 2023-09-10T01:14:15.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 client.27925 shutdown 2023-09-10T01:14:15.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b6f710 msgr2=0x5597f2b7ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b6f710 0x5597f2b7ff50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c005930 tx=0x7fcb7c0047c0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 shutdown_connections 2023-09-10T01:14:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5597f2be34c0 0x5597f2be58b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5597f2b6f710 0x5597f2b7ff50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597f2b7e250 0x5597f2b809f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 --2- 172.21.15.3:0/2881728811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597f2b70030 0x5597f2b80490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 >> 172.21.15.3:0/2881728811 conn(0x5597f2ae4090 msgr2=0x5597f2b6e700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcb6affd700 0 client.27925 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:15.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:14.999+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 shutdown_connections 2023-09-10T01:14:15.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcba5924b80 1 -- 172.21.15.3:0/2881728811 wait complete. 2023-09-10T01:14:15.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcba5924b80 20 client.27925 trim_cache size 0 max 16384 2023-09-10T01:14:15.020 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:14:15.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcb780a07b0 msgr2=0x7fcb780a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcb780a07b0 0x7fcb780a2c70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c005fd0 tx=0x7fcb7c011720 comp rx=0 tx=0).stop 2023-09-10T01:14:15.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 msgr2=0x7fcb8c196c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 0x7fcb8c196c60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009230 tx=0x7fcb7400fa00 comp rx=0 tx=0).stop 2023-09-10T01:14:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 shutdown_connections 2023-09-10T01:14:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcb780a07b0 0x7fcb780a2c70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb8c106650 0x7fcb8c19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb8c100640 0x7fcb8c196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 --2- 172.21.15.3:0/274373276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c101000 0x7fcb8c196c60 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 >> 172.21.15.3:0/274373276 conn(0x7fcb8c0fc170 msgr2=0x7fcb8c0fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 shutdown_connections 2023-09-10T01:14:15.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.000+0000 7fcb91fff700 1 -- 172.21.15.3:0/274373276 wait complete. 2023-09-10T01:14:15.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:14:15.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 -- 172.21.15.3:0/2887306179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294100640 msgr2=0x7f6294100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 --2- 172.21.15.3:0/2887306179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294100640 0x7f6294100ac0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f627c009990 tx=0x7f627c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 -- 172.21.15.3:0/2887306179 shutdown_connections 2023-09-10T01:14:15.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 --2- 172.21.15.3:0/2887306179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294106650 0x7f6294106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 --2- 172.21.15.3:0/2887306179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294101000 0x7f629410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 --2- 172.21.15.3:0/2887306179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294100640 0x7f6294100ac0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 -- 172.21.15.3:0/2887306179 >> 172.21.15.3:0/2887306179 conn(0x7f62940fc170 msgr2=0x7f62940fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 -- 172.21.15.3:0/2887306179 shutdown_connections 2023-09-10T01:14:15.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.271+0000 7f6298dff700 1 -- 172.21.15.3:0/2887306179 wait complete. 2023-09-10T01:14:15.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.272+0000 7f6298dff700 1 Processor -- start 2023-09-10T01:14:15.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.272+0000 7f6298dff700 1 -- start start 2023-09-10T01:14:15.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f6298dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294100640 0x7f629419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f6298dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294101000 0x7f629419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f6298dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294106650 0x7f6294198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f6298dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294198dc0 con 0x7f6294106650 2023-09-10T01:14:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f6298dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294198f30 con 0x7f6294100640 2023-09-10T01:14:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f6298dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62941990f0 con 0x7f6294101000 2023-09-10T01:14:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294100640 0x7f629419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294100640 0x7f629419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50792/0 (socket says 172.21.15.3:50792) 2023-09-10T01:14:15.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 -- 172.21.15.3:0/2320422163 learned_addr learned my addr 172.21.15.3:0/2320422163 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f6292d9d700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294106650 0x7f6294198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f628bfff700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294101000 0x7f629419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 -- 172.21.15.3:0/2320422163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294101000 msgr2=0x7f629419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294101000 0x7f629419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 -- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294106650 msgr2=0x7f6294198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294106650 0x7f6294198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.273+0000 7f629259c700 1 -- 172.21.15.3:0/2320422163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c0095f0 con 0x7f6294100640 2023-09-10T01:14:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.274+0000 7f629259c700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294100640 0x7f629419e6e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f628400e9a0 tx=0x7f628400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.274+0000 7f628b7fe700 1 -- 172.21.15.3:0/2320422163 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284009790 con 0x7f6294100640 2023-09-10T01:14:15.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.274+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62941993d0 con 0x7f6294100640 2023-09-10T01:14:15.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.274+0000 7f628b7fe700 1 -- 172.21.15.3:0/2320422163 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6284010430 con 0x7f6294100640 2023-09-10T01:14:15.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.274+0000 7f628b7fe700 1 -- 172.21.15.3:0/2320422163 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284018680 con 0x7f6294100640 2023-09-10T01:14:15.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.274+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62941a56d0 con 0x7f6294100640 2023-09-10T01:14:15.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.277+0000 7f628b7fe700 1 -- 172.21.15.3:0/2320422163 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f62840105a0 con 0x7f6294100640 2023-09-10T01:14:15.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.278+0000 7f62ac6e7b80 1 Processor -- start 2023-09-10T01:14:15.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.278+0000 7f62ac6e7b80 1 -- start start 2023-09-10T01:14:15.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b5710 0x5558e28b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b6030 0x5558e28bf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e280e730 con 0x5558e28b5710 2023-09-10T01:14:15.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e280c050 con 0x5558e28b6030 2023-09-10T01:14:15.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e2807940 con 0x5558e28bfe40 2023-09-10T01:14:15.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b5710 0x5558e28b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b5710 0x5558e28b5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49394/0 (socket says 172.21.15.3:49394) 2023-09-10T01:14:15.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f6292d9d700 1 -- 172.21.15.3:0/308600474 learned_addr learned my addr 172.21.15.3:0/308600474 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f629259c700 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b6030 0x5558e28bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.279+0000 7f628bfff700 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f6292d9d700 1 -- 172.21.15.3:0/308600474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 msgr2=0x5558e28c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f6292d9d700 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f6292d9d700 1 -- 172.21.15.3:0/308600474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b6030 msgr2=0x5558e28bf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f6292d9d700 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b6030 0x5558e28bf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f6292d9d700 1 -- 172.21.15.3:0/308600474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558e2633de0 con 0x5558e28b5710 2023-09-10T01:14:15.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f628b7fe700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f62800a07b0 0x7f62800a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f628b7fe700 1 -- 172.21.15.3:0/2320422163 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f6284014070 con 0x7f6294100640 2023-09-10T01:14:15.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f6292d9d700 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b5710 0x5558e28b5af0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f628c011600 tx=0x7f628c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e262f54a14237fbe server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.280+0000 7f6288ff9700 1 -- 172.21.15.3:0/308600474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c00e040 con 0x5558e28b5710 2023-09-10T01:14:15.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.281+0000 7f6288ff9700 1 -- 172.21.15.3:0/308600474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f628c00f040 con 0x5558e28b5710 2023-09-10T01:14:15.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.281+0000 7f6288ff9700 1 -- 172.21.15.3:0/308600474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c004760 con 0x5558e28b5710 2023-09-10T01:14:15.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.281+0000 7f629259c700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f62800a07b0 0x7f62800a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.281+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/308600474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b5710 msgr2=0x5558e28b5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.281+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b5710 0x5558e28b5af0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f628c011600 tx=0x7f628c009920 comp rx=0 tx=0).stop 2023-09-10T01:14:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.281+0000 7f629259c700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f62800a07b0 0x7f62800a2c70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x5558e28ab280 tx=0x7f62840cfdb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.282+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/308600474 shutdown_connections 2023-09-10T01:14:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.282+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.282+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b5710 0x5558e28b5af0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.282+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/308600474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b6030 0x5558e28bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.282+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/308600474 >> 172.21.15.3:0/308600474 conn(0x5558e282a090 msgr2=0x5558e28b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.282+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/308600474 shutdown_connections 2023-09-10T01:14:15.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.282+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/308600474 wait complete. 2023-09-10T01:14:15.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.283+0000 7f62ac6e7b80 1 -- start start 2023-09-10T01:14:15.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.283+0000 7f62ac6e7b80 1 Processor -- start 2023-09-10T01:14:15.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.284+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b5710 0x5558e28bf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.284+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e28bcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.284+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28bfe40 0x5558e28bd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.284+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e27a5b70 con 0x5558e28b6030 2023-09-10T01:14:15.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.284+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e27a5870 con 0x5558e28bfe40 2023-09-10T01:14:15.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.284+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e2689920 con 0x5558e28b5710 2023-09-10T01:14:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.284+0000 7f62ac6e7b80 10 client.0 _ll_register_callbacks cb 0x5558e26efa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b5710 0x5558e28bf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f628bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e28bcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b5710 0x5558e28bf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57846/0 (socket says 172.21.15.3:57846) 2023-09-10T01:14:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f628bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e28bcb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49396/0 (socket says 172.21.15.3:49396) 2023-09-10T01:14:15.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f6292d9d700 1 -- 172.21.15.3:0/113125246 learned_addr learned my addr 172.21.15.3:0/113125246 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f6292d9d700 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28bfe40 0x5558e28bd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f628bfff700 1 -- 172.21.15.3:0/113125246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b5710 msgr2=0x5558e28bf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f628bfff700 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b5710 0x5558e28bf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f628bfff700 1 -- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28bfe40 msgr2=0x5558e28bd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f628bfff700 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28bfe40 0x5558e28bd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f628bfff700 1 -- 172.21.15.3:0/113125246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c0112b0 con 0x5558e28b6030 2023-09-10T01:14:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.285+0000 7f6292d9d700 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b5710 0x5558e28bf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f628bfff700 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e28bcb40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b2e0 tx=0x7f627c009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5671a28b233521e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f6279ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f6279ffb700 1 -- 172.21.15.3:0/113125246 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c01d030 con 0x5558e28b6030 2023-09-10T01:14:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558e275f4f0 con 0x5558e28b6030 2023-09-10T01:14:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f6279ffb700 1 -- 172.21.15.3:0/113125246 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f627c004500 con 0x5558e28b6030 2023-09-10T01:14:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f6279ffb700 1 -- 172.21.15.3:0/113125246 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f627c02d850 con 0x5558e28b6030 2023-09-10T01:14:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f62ac6e7b80 10 client.17961 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.286+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5558e261cc60 con 0x5558e28b6030 2023-09-10T01:14:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f6279ffb700 1 -- 172.21.15.3:0/113125246 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5558e275f4f0 con 0x5558e28b6030 2023-09-10T01:14:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f6279ffb700 1 -- 172.21.15.3:0/113125246 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f627c0047e0 con 0x5558e28b6030 2023-09-10T01:14:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f62ac6e7b80 10 client.17961 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f62ac6e7b80 10 client.17961 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:14:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f62ac6e7b80 20 client.17961 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e28cfb10 0x5558e28d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f62ac6e7b80 4 client.17961 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558e281c8a0 con 0x5558e28cfb10 2023-09-10T01:14:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.287+0000 7f6292d9d700 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e28cfb10 0x5558e28d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.288+0000 7f6292d9d700 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e28cfb10 0x5558e28d1f30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4f8fb3ff08a6dceb server_cookie=2471d75f4a1e8724 in_seq=0 out_seq=0 2023-09-10T01:14:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.288+0000 7f6279ffb700 10 client.17961 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f6279ffb700 1 -- 172.21.15.3:0/113125246 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5558e281c8a0 con 0x5558e28cfb10 2023-09-10T01:14:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f6279ffb700 10 client.17961 handle_command_reply: tid=0 2023-09-10T01:14:15.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 client.17961 shutdown 2023-09-10T01:14:15.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 msgr2=0x5558e28bcb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e28bcb40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b2e0 tx=0x7f627c009530 comp rx=0 tx=0).stop 2023-09-10T01:14:15.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 shutdown_connections 2023-09-10T01:14:15.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e28cfb10 0x5558e28d1f30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b5710 0x5558e28bf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28bfe40 0x5558e28bd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/113125246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e28bcb40 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f6279ffb700 0 client.17961 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.289+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 >> 172.21.15.3:0/113125246 conn(0x5558e282a090 msgr2=0x5558e28b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.290+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 shutdown_connections 2023-09-10T01:14:15.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.290+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/113125246 wait complete. 2023-09-10T01:14:15.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.290+0000 7f62ac6e7b80 20 client.17961 trim_cache size 0 max 16384 2023-09-10T01:14:15.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.304+0000 7f62ac6e7b80 1 Processor -- start 2023-09-10T01:14:15.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.304+0000 7f62ac6e7b80 1 -- start start 2023-09-10T01:14:15.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.304+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e28cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.304+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b6030 0x5558e28b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28bfe40 0x5558e28b8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e28b69f0 con 0x5558e28bfe40 2023-09-10T01:14:15.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e28ad3d0 con 0x5558e28b5710 2023-09-10T01:14:15.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e28d2590 con 0x5558e28b6030 2023-09-10T01:14:15.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e28cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e28cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50804/0 (socket says 172.21.15.3:50804) 2023-09-10T01:14:15.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 -- 172.21.15.3:0/2950101125 learned_addr learned my addr 172.21.15.3:0/2950101125 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28bfe40 0x5558e28b8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f628bfff700 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b6030 0x5558e28b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 -- 172.21.15.3:0/2950101125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b6030 msgr2=0x5558e28b7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b6030 0x5558e28b7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 -- 172.21.15.3:0/2950101125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28bfe40 msgr2=0x5558e28b8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28bfe40 0x5558e28b8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.305+0000 7f6292d9d700 1 -- 172.21.15.3:0/2950101125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558e265a250 con 0x5558e28b5710 2023-09-10T01:14:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.306+0000 7f6292d9d700 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e28cb140 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f628c009640 tx=0x7f628c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5da7006187796758 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.306+0000 7f62897fa700 1 -- 172.21.15.3:0/2950101125 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c00e040 con 0x5558e28b5710 2023-09-10T01:14:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.306+0000 7f62897fa700 1 -- 172.21.15.3:0/2950101125 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f628c0048d0 con 0x5558e28b5710 2023-09-10T01:14:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.306+0000 7f62897fa700 1 -- 172.21.15.3:0/2950101125 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c024040 con 0x5558e28b5710 2023-09-10T01:14:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/2950101125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 msgr2=0x5558e28cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e28cb140 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f628c009640 tx=0x7f628c009920 comp rx=0 tx=0).stop 2023-09-10T01:14:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/2950101125 shutdown_connections 2023-09-10T01:14:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28bfe40 0x5558e28b8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e28cb140 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/2950101125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28b6030 0x5558e28b7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/2950101125 >> 172.21.15.3:0/2950101125 conn(0x5558e282a090 msgr2=0x5558e28cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/2950101125 shutdown_connections 2023-09-10T01:14:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/2950101125 wait complete. 2023-09-10T01:14:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 -- start start 2023-09-10T01:14:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.307+0000 7f62ac6e7b80 1 Processor -- start 2023-09-10T01:14:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e291cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e291d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f62ac6e7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e28c4f80 con 0x5558e28b6030 2023-09-10T01:14:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e28c50f0 con 0x5558e28b5710 2023-09-10T01:14:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f62ac6e7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558e28c52b0 con 0x5558e28bfe40 2023-09-10T01:14:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f62ac6e7b80 10 client.0 _ll_register_callbacks cb 0x7f627c00bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e291cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e291cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50810/0 (socket says 172.21.15.3:50810) 2023-09-10T01:14:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 -- 172.21.15.3:0/1614685338 learned_addr learned my addr 172.21.15.3:0/1614685338 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f628bfff700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e291d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 -- 172.21.15.3:0/1614685338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 msgr2=0x5558e28c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 -- 172.21.15.3:0/1614685338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 msgr2=0x5558e291d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e291d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.308+0000 7f6292d9d700 1 -- 172.21.15.3:0/1614685338 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c027040 con 0x5558e28b5710 2023-09-10T01:14:15.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f628bfff700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e291d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:15.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f6292d9d700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e291cac0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f628c009320 tx=0x7f628c011580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99a394b39e63812c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f6279ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:14:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f6279ffb700 1 -- 172.21.15.3:0/1614685338 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c00e070 con 0x5558e28b5710 2023-09-10T01:14:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558e2800920 con 0x5558e28b5710 2023-09-10T01:14:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f6279ffb700 1 -- 172.21.15.3:0/1614685338 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f628c004a90 con 0x5558e28b5710 2023-09-10T01:14:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f6279ffb700 1 -- 172.21.15.3:0/1614685338 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f628c024040 con 0x5558e28b5710 2023-09-10T01:14:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f62ac6e7b80 10 client.28055 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.309+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5558e26322b0 con 0x5558e28b5710 2023-09-10T01:14:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f6279ffb700 1 -- 172.21.15.3:0/1614685338 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5558e2800920 con 0x5558e28b5710 2023-09-10T01:14:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f6279ffb700 1 -- 172.21.15.3:0/1614685338 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f628c037060 con 0x5558e28b5710 2023-09-10T01:14:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f62ac6e7b80 10 client.28055 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f62ac6e7b80 10 client.28055 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:14:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f62ac6e7b80 20 client.28055 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:15.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e29246b0 0x5558e2926ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f62ac6e7b80 4 client.28055 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:14:15.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5558e27c10b0 con 0x5558e29246b0 2023-09-10T01:14:15.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.310+0000 7f628bfff700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e29246b0 0x5558e2926ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f628bfff700 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e29246b0 0x5558e2926ad0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=99bb070fa4afa3dc server_cookie=f25dbdd98742f81d in_seq=0 out_seq=0 2023-09-10T01:14:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f6279ffb700 10 client.28055 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f6279ffb700 1 -- 172.21.15.3:0/1614685338 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5558e27c10b0 con 0x5558e29246b0 2023-09-10T01:14:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f6279ffb700 10 client.28055 handle_command_reply: tid=0 2023-09-10T01:14:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 client.28055 shutdown 2023-09-10T01:14:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 msgr2=0x5558e291cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e291cac0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f628c009320 tx=0x7f628c011580 comp rx=0 tx=0).stop 2023-09-10T01:14:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 shutdown_connections 2023-09-10T01:14:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5558e29246b0 0x5558e2926ad0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558e28bfe40 0x5558e28c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558e28b5710 0x5558e291cac0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f6279ffb700 0 client.28055 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 --2- 172.21.15.3:0/1614685338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558e28b6030 0x5558e291d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 >> 172.21.15.3:0/1614685338 conn(0x5558e282a090 msgr2=0x5558e28b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 shutdown_connections 2023-09-10T01:14:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.311+0000 7f62ac6e7b80 1 -- 172.21.15.3:0/1614685338 wait complete. 2023-09-10T01:14:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f62ac6e7b80 20 client.28055 trim_cache size 0 max 16384 2023-09-10T01:14:15.327 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:14:15.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f62800a07b0 msgr2=0x7f62800a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f62800a07b0 0x7f62800a2c70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x5558e28ab280 tx=0x7f62840cfdb0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294100640 msgr2=0x7f629419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294100640 0x7f629419e6e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f628400e9a0 tx=0x7f628400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 shutdown_connections 2023-09-10T01:14:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f62800a07b0 0x7f62800a2c70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6294106650 0x7f6294198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6294100640 0x7f629419e6e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 --2- 172.21.15.3:0/2320422163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294101000 0x7f629419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 >> 172.21.15.3:0/2320422163 conn(0x7f62940fc170 msgr2=0x7f62940fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 shutdown_connections 2023-09-10T01:14:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.312+0000 7f6298dff700 1 -- 172.21.15.3:0/2320422163 wait complete. 2023-09-10T01:14:15.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:14:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 -- 172.21.15.3:0/1787597155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 msgr2=0x7f57181009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 --2- 172.21.15.3:0/1787597155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f57181009f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5708009990 tx=0x7f5708009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 -- 172.21.15.3:0/1787597155 shutdown_connections 2023-09-10T01:14:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 --2- 172.21.15.3:0/1787597155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5718105730 0x7f571810d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 --2- 172.21.15.3:0/1787597155 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f57181009f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 --2- 172.21.15.3:0/1787597155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5718100fc0 0x7f5718105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 -- 172.21.15.3:0/1787597155 >> 172.21.15.3:0/1787597155 conn(0x7f57180fbe90 msgr2=0x7f57180fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 -- 172.21.15.3:0/1787597155 shutdown_connections 2023-09-10T01:14:15.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.537+0000 7f571cf9d700 1 -- 172.21.15.3:0/1787597155 wait complete. 2023-09-10T01:14:15.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.538+0000 7f571cf9d700 1 Processor -- start 2023-09-10T01:14:15.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.538+0000 7f571cf9d700 1 -- start start 2023-09-10T01:14:15.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f571cf9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f571819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f571cf9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5718105730 0x7f571819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f571cf9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57181985d0 0x7f5718198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f571cf9d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718199050 con 0x7f5718100610 2023-09-10T01:14:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f571cf9d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718199190 con 0x7f5718105730 2023-09-10T01:14:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f571cf9d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718199350 con 0x7f57181985d0 2023-09-10T01:14:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f571819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f571819e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49450/0 (socket says 172.21.15.3:49450) 2023-09-10T01:14:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f57177fe700 1 -- 172.21.15.3:0/935939014 learned_addr learned my addr 172.21.15.3:0/935939014 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f5716ffd700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5718105730 0x7f571819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.539+0000 7f5717fff700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57181985d0 0x7f5718198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f57177fe700 1 -- 172.21.15.3:0/935939014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57181985d0 msgr2=0x7f5718198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f57177fe700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57181985d0 0x7f5718198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f57177fe700 1 -- 172.21.15.3:0/935939014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5718105730 msgr2=0x7f571819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f57177fe700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5718105730 0x7f571819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f57177fe700 1 -- 172.21.15.3:0/935939014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57080095f0 con 0x7f5718100610 2023-09-10T01:14:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f57177fe700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f571819e560 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5708005910 tx=0x7f5708004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f5714ff9700 1 -- 172.21.15.3:0/935939014 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570801d070 con 0x7f5718100610 2023-09-10T01:14:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57181a5370 con 0x7f5718100610 2023-09-10T01:14:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f5714ff9700 1 -- 172.21.15.3:0/935939014 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5708022470 con 0x7f5718100610 2023-09-10T01:14:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f5714ff9700 1 -- 172.21.15.3:0/935939014 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708017650 con 0x7f5718100610 2023-09-10T01:14:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.540+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57181a5840 con 0x7f5718100610 2023-09-10T01:14:15.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.544+0000 7f5714ff9700 1 -- 172.21.15.3:0/935939014 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f57080225e0 con 0x7f5718100610 2023-09-10T01:14:15.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.545+0000 7f5731b45b80 1 Processor -- start 2023-09-10T01:14:15.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.545+0000 7f5731b45b80 1 -- start start 2023-09-10T01:14:15.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c170c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c17baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c17e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5731b45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c0ca730 con 0x56018c171180 2023-09-10T01:14:15.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5731b45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c0c8050 con 0x56018c17c030 2023-09-10T01:14:15.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5731b45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c0c3940 con 0x56018c170860 2023-09-10T01:14:15.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5714ff9700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57000a07b0 0x7f57000a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c17baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5716ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c17e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5716ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c17e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50834/0 (socket says 172.21.15.3:50834) 2023-09-10T01:14:15.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c17baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49452/0 (socket says 172.21.15.3:49452) 2023-09-10T01:14:15.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5716ffd700 1 -- 172.21.15.3:0/1832913891 learned_addr learned my addr 172.21.15.3:0/1832913891 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.546+0000 7f5717fff700 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c170c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5714ff9700 1 -- 172.21.15.3:0/935939014 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f57080cd580 con 0x7f5718100610 2023-09-10T01:14:15.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5717fff700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57000a07b0 0x7f57000a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5716ffd700 1 -- 172.21.15.3:0/1832913891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 msgr2=0x56018c170c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5716ffd700 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c170c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5716ffd700 1 -- 172.21.15.3:0/1832913891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 msgr2=0x56018c17baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5716ffd700 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c17baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5716ffd700 1 -- 172.21.15.3:0/1832913891 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56018beefde0 con 0x56018c17c030 2023-09-10T01:14:15.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5716ffd700 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c17e420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f570c009cf0 tx=0x7f570c011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b75c1d0925c6f5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f5717fff700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57000a07b0 0x7f57000a2c70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5710009ec0 tx=0x7f571000b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f56fdffb700 1 -- 172.21.15.3:0/1832913891 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c011d80 con 0x56018c17c030 2023-09-10T01:14:15.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f56fdffb700 1 -- 172.21.15.3:0/1832913891 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f570c004510 con 0x56018c17c030 2023-09-10T01:14:15.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.547+0000 7f56fdffb700 1 -- 172.21.15.3:0/1832913891 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c005280 con 0x56018c17c030 2023-09-10T01:14:15.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 -- 172.21.15.3:0/1832913891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 msgr2=0x56018c17e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c17e420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f570c009cf0 tx=0x7f570c011440 comp rx=0 tx=0).stop 2023-09-10T01:14:15.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 -- 172.21.15.3:0/1832913891 shutdown_connections 2023-09-10T01:14:15.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c170c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c17e420 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1832913891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c17baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 -- 172.21.15.3:0/1832913891 >> 172.21.15.3:0/1832913891 conn(0x56018c0e6090 msgr2=0x56018c16cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 -- 172.21.15.3:0/1832913891 shutdown_connections 2023-09-10T01:14:15.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.548+0000 7f5731b45b80 1 -- 172.21.15.3:0/1832913891 wait complete. 2023-09-10T01:14:15.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.549+0000 7f5731b45b80 1 -- start start 2023-09-10T01:14:15.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.549+0000 7f5731b45b80 1 Processor -- start 2023-09-10T01:14:15.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.549+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c17bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c171180 0x56018c178d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c17c030 0x56018c1794a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5731b45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c061b70 con 0x56018c170860 2023-09-10T01:14:15.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5731b45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c061870 con 0x56018c171180 2023-09-10T01:14:15.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5731b45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018bf45920 con 0x56018c17c030 2023-09-10T01:14:15.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5731b45b80 10 client.0 _ll_register_callbacks cb 0x56018bfaba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:15.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c17bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c17bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49460/0 (socket says 172.21.15.3:49460) 2023-09-10T01:14:15.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 -- 172.21.15.3:0/3502921816 learned_addr learned my addr 172.21.15.3:0/3502921816 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5716ffd700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c171180 0x56018c178d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5717fff700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c17c030 0x56018c1794a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 -- 172.21.15.3:0/3502921816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c17c030 msgr2=0x56018c1794a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c17c030 0x56018c1794a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 -- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c171180 msgr2=0x56018c178d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c171180 0x56018c178d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f57177fe700 1 -- 172.21.15.3:0/3502921816 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c009920 con 0x56018c170860 2023-09-10T01:14:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.550+0000 7f5716ffd700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c171180 0x56018c178d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f57177fe700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c17bae0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f570809a600 tx=0x7f5708009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663f2858dfe8a434 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f56f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:15.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f56f6ffd700 1 -- 172.21.15.3:0/3502921816 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708099fd0 con 0x56018c170860 2023-09-10T01:14:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56018c01b4f0 con 0x56018c170860 2023-09-10T01:14:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f56f6ffd700 1 -- 172.21.15.3:0/3502921816 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f570809ad70 con 0x56018c170860 2023-09-10T01:14:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f56f6ffd700 1 -- 172.21.15.3:0/3502921816 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f570809a670 con 0x56018c170860 2023-09-10T01:14:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f5731b45b80 10 client.17979 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x56018bed8c60 con 0x56018c170860 2023-09-10T01:14:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.551+0000 7f56f6ffd700 1 -- 172.21.15.3:0/3502921816 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56018c01b4f0 con 0x56018c170860 2023-09-10T01:14:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f56f6ffd700 1 -- 172.21.15.3:0/3502921816 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f57080cd120 con 0x56018c170860 2023-09-10T01:14:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f5731b45b80 10 client.17979 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f5731b45b80 10 client.17979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f5731b45b80 20 client.17979 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f5731b45b80 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c18b6b0 0x56018c18daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f5731b45b80 4 client.17979 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56018c0d88a0 con 0x56018c18b6b0 2023-09-10T01:14:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.552+0000 7f5716ffd700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c18b6b0 0x56018c18daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.553+0000 7f5716ffd700 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c18b6b0 0x56018c18daa0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=939e90b7cfe66027 server_cookie=faccf9b2630a587e in_seq=0 out_seq=0 2023-09-10T01:14:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.553+0000 7f56f6ffd700 10 client.17979 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.553+0000 7f56f6ffd700 1 -- 172.21.15.3:0/3502921816 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56018c0d88a0 con 0x56018c18b6b0 2023-09-10T01:14:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.553+0000 7f56f6ffd700 10 client.17979 handle_command_reply: tid=0 2023-09-10T01:14:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 client.17979 shutdown 2023-09-10T01:14:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 msgr2=0x56018c17bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c17bae0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f570809a600 tx=0x7f5708009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 shutdown_connections 2023-09-10T01:14:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c18b6b0 0x56018c18daa0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c17c030 0x56018c1794a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c17bae0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 --2- 172.21.15.3:0/3502921816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c171180 0x56018c178d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f56f6ffd700 0 client.17979 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 >> 172.21.15.3:0/3502921816 conn(0x56018c0e6090 msgr2=0x56018c16c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 shutdown_connections 2023-09-10T01:14:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 1 -- 172.21.15.3:0/3502921816 wait complete. 2023-09-10T01:14:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.554+0000 7f5731b45b80 20 client.17979 trim_cache size 0 max 16384 2023-09-10T01:14:15.574 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 Processor -- start 2023-09-10T01:14:15.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 -- start start 2023-09-10T01:14:15.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c182b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c16c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c16cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.575 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c169580 con 0x56018c171180 2023-09-10T01:14:15.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c17a370 con 0x56018c17c030 2023-09-10T01:14:15.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.568+0000 7f5731b45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c172390 con 0x56018c170860 2023-09-10T01:14:15.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c16cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5716ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c16c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c16cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50856/0 (socket says 172.21.15.3:50856) 2023-09-10T01:14:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 -- 172.21.15.3:0/1866289930 learned_addr learned my addr 172.21.15.3:0/1866289930 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f57177fe700 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c182b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 -- 172.21.15.3:0/1866289930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 msgr2=0x56018c182b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c182b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 -- 172.21.15.3:0/1866289930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 msgr2=0x56018c16c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c16c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 -- 172.21.15.3:0/1866289930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56018bf16250 con 0x56018c17c030 2023-09-10T01:14:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5716ffd700 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c16c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.569+0000 7f5717fff700 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c16cb70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5710018730 tx=0x7f5710009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b1a583710c264d2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f56fe7fc700 1 -- 172.21.15.3:0/1866289930 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710028070 con 0x56018c17c030 2023-09-10T01:14:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f56fe7fc700 1 -- 172.21.15.3:0/1866289930 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f571000f040 con 0x56018c17c030 2023-09-10T01:14:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f56fe7fc700 1 -- 172.21.15.3:0/1866289930 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000e070 con 0x56018c17c030 2023-09-10T01:14:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 -- 172.21.15.3:0/1866289930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 msgr2=0x56018c16cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c16cb70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5710018730 tx=0x7f5710009920 comp rx=0 tx=0).stop 2023-09-10T01:14:15.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 -- 172.21.15.3:0/1866289930 shutdown_connections 2023-09-10T01:14:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c170860 0x56018c182b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c16cb70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1866289930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c171180 0x56018c16c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 -- 172.21.15.3:0/1866289930 >> 172.21.15.3:0/1866289930 conn(0x56018c0e6090 msgr2=0x56018c16bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 -- 172.21.15.3:0/1866289930 shutdown_connections 2023-09-10T01:14:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.570+0000 7f5731b45b80 1 -- 172.21.15.3:0/1866289930 wait complete. 2023-09-10T01:14:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.571+0000 7f5731b45b80 1 -- start start 2023-09-10T01:14:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.571+0000 7f5731b45b80 1 Processor -- start 2023-09-10T01:14:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c180f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c171180 0x56018c1814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5731b45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c181a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5731b45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c1df4e0 con 0x56018c170860 2023-09-10T01:14:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5731b45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c1df620 con 0x56018c17c030 2023-09-10T01:14:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5731b45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018c1df760 con 0x56018c171180 2023-09-10T01:14:15.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5731b45b80 10 client.0 _ll_register_callbacks cb 0x7f570809a510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c180f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f57177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c180f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49484/0 (socket says 172.21.15.3:49484) 2023-09-10T01:14:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f57177fe700 1 -- 172.21.15.3:0/1810727183 learned_addr learned my addr 172.21.15.3:0/1810727183 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5716ffd700 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c171180 0x56018c1814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.572+0000 7f5717fff700 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c181a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f57177fe700 1 -- 172.21.15.3:0/1810727183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c171180 msgr2=0x56018c1814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f57177fe700 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c171180 0x56018c1814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f57177fe700 1 -- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 msgr2=0x56018c181a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f57177fe700 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c181a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f57177fe700 1 -- 172.21.15.3:0/1810727183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c01d040 con 0x56018c170860 2023-09-10T01:14:15.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f57177fe700 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c180f80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f57080970e0 tx=0x7f5708022af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c5f59185e4c031 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:15.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f56f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:15.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f56f6ffd700 1 -- 172.21.15.3:0/1810727183 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570809e2e0 con 0x56018c170860 2023-09-10T01:14:15.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f56f6ffd700 1 -- 172.21.15.3:0/1810727183 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f570809ad70 con 0x56018c170860 2023-09-10T01:14:15.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f570c004510 con 0x56018c170860 2023-09-10T01:14:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f56f6ffd700 1 -- 172.21.15.3:0/1810727183 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57080b6030 con 0x56018c170860 2023-09-10T01:14:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f5731b45b80 10 client.17985 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f5710020040 con 0x56018c170860 2023-09-10T01:14:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.573+0000 7f56f6ffd700 1 -- 172.21.15.3:0/1810727183 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f570c004510 con 0x56018c170860 2023-09-10T01:14:15.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f56f6ffd700 1 -- 172.21.15.3:0/1810727183 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f5708096460 con 0x56018c170860 2023-09-10T01:14:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f5731b45b80 10 client.17985 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f5731b45b80 10 client.17985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f5731b45b80 20 client.17985 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c1e0aa0 0x56018c1e2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:15.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f5731b45b80 4 client.17985 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:14:15.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56018c0bc920 con 0x56018c1e0aa0 2023-09-10T01:14:15.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.574+0000 7f5716ffd700 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c1e0aa0 0x56018c1e2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:15.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5716ffd700 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c1e0aa0 0x56018c1e2ec0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8253cc550dcae0cc server_cookie=cf90ebf6a8a6be27 in_seq=0 out_seq=0 2023-09-10T01:14:15.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f56f6ffd700 10 client.17985 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:15.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f56f6ffd700 1 -- 172.21.15.3:0/1810727183 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56018c0bc920 con 0x56018c1e0aa0 2023-09-10T01:14:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f56f6ffd700 10 client.17985 handle_command_reply: tid=0 2023-09-10T01:14:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 client.17985 shutdown 2023-09-10T01:14:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 msgr2=0x56018c180f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c180f80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f57080970e0 tx=0x7f5708022af0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 shutdown_connections 2023-09-10T01:14:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56018c1e0aa0 0x56018c1e2ec0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56018c17c030 0x56018c181a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56018c170860 0x56018c180f80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 --2- 172.21.15.3:0/1810727183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56018c171180 0x56018c1814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 >> 172.21.15.3:0/1810727183 conn(0x56018c0e6090 msgr2=0x56018c16c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f56f6ffd700 0 client.17985 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 shutdown_connections 2023-09-10T01:14:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 1 -- 172.21.15.3:0/1810727183 wait complete. 2023-09-10T01:14:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.575+0000 7f5731b45b80 20 client.17985 trim_cache size 0 max 16384 2023-09-10T01:14:15.593 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:14:15.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57000a07b0 msgr2=0x7f57000a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57000a07b0 0x7f57000a2c70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5710009ec0 tx=0x7f571000b040 comp rx=0 tx=0).stop 2023-09-10T01:14:15.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 msgr2=0x7f571819e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:15.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f571819e560 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5708005910 tx=0x7f5708004730 comp rx=0 tx=0).stop 2023-09-10T01:14:15.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 shutdown_connections 2023-09-10T01:14:15.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f57000a07b0 0x7f57000a2c70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57181985d0 0x7f5718198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5718100610 0x7f571819e560 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 --2- 172.21.15.3:0/935939014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5718105730 0x7f571819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:15.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 >> 172.21.15.3:0/935939014 conn(0x7f57180fbe90 msgr2=0x7f571810c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:15.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 shutdown_connections 2023-09-10T01:14:15.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:15.576+0000 7f571cf9d700 1 -- 172.21.15.3:0/935939014 wait complete. 2023-09-10T01:14:15.596 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:15 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/245587240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:14:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:15 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/245587240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:14:15.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:15 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/245587240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:14:15.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952986 8.35 MB/sec execute 477 sec latency 2687.165 ms 2023-09-10T01:14:16.598 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:14:16.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:14:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:16 smithi136 ceph-mon[133708]: pgmap v344: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 3.5 MiB/s wr, 96 op/s 2023-09-10T01:14:16.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:16 smithi003 ceph-mon[97654]: pgmap v344: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 3.5 MiB/s wr, 96 op/s 2023-09-10T01:14:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:16 smithi092 ceph-mon[134067]: pgmap v344: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 3.5 MiB/s wr, 96 op/s 2023-09-10T01:14:16.800 INFO:tasks.workunit.client.0.smithi003.stdout: 1 952986 8.33 MB/sec execute 478 sec latency 3687.229 ms 2023-09-10T01:14:16.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 -- 172.21.15.3:0/1494018617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec100640 msgr2=0x7fa4ec100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/1494018617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec100640 0x7fa4ec100ac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009990 tx=0x7fa4e0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 -- 172.21.15.3:0/1494018617 shutdown_connections 2023-09-10T01:14:16.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/1494018617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4ec106650 0x7fa4ec106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/1494018617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec101000 0x7fa4ec10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/1494018617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec100640 0x7fa4ec100ac0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 -- 172.21.15.3:0/1494018617 >> 172.21.15.3:0/1494018617 conn(0x7fa4ec0fc170 msgr2=0x7fa4ec0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:16.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 -- 172.21.15.3:0/1494018617 shutdown_connections 2023-09-10T01:14:16.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.812+0000 7fa4f17ff700 1 -- 172.21.15.3:0/1494018617 wait complete. 2023-09-10T01:14:16.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.813+0000 7fa4f17ff700 1 Processor -- start 2023-09-10T01:14:16.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.813+0000 7fa4f17ff700 1 -- start start 2023-09-10T01:14:16.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4f17ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec100640 0x7fa4ec19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4f17ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4ec101000 0x7fa4ec19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4f17ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec106650 0x7fa4ec198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4f17ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec198dd0 con 0x7fa4ec106650 2023-09-10T01:14:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4f17ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec198f40 con 0x7fa4ec100640 2023-09-10T01:14:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4f17ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec199100 con 0x7fa4ec101000 2023-09-10T01:14:16.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec106650 0x7fa4ec198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec100640 0x7fa4ec19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec100640 0x7fa4ec19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50886/0 (socket says 172.21.15.3:50886) 2023-09-10T01:14:16.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec106650 0x7fa4ec198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49500/0 (socket says 172.21.15.3:49500) 2023-09-10T01:14:16.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 -- 172.21.15.3:0/2522059969 learned_addr learned my addr 172.21.15.3:0/2522059969 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:16.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4ec101000 0x7fa4ec19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 -- 172.21.15.3:0/2522059969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4ec101000 msgr2=0x7fa4ec19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4ec101000 0x7fa4ec19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 -- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec106650 msgr2=0x7fa4ec198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec106650 0x7fa4ec198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4eaffd700 1 -- 172.21.15.3:0/2522059969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e00095f0 con 0x7fa4ec100640 2023-09-10T01:14:16.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.814+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4ec101000 0x7fa4ec19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:16.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.815+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec100640 0x7fa4ec19e690 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00e9a0 tx=0x7fa4dc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:16.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.815+0000 7fa4d3fff700 1 -- 172.21.15.3:0/2522059969 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc009790 con 0x7fa4ec100640 2023-09-10T01:14:16.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.815+0000 7fa4d3fff700 1 -- 172.21.15.3:0/2522059969 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4dc00cbd0 con 0x7fa4ec100640 2023-09-10T01:14:16.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.815+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec1993e0 con 0x7fa4ec100640 2023-09-10T01:14:16.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.815+0000 7fa4d3fff700 1 -- 172.21.15.3:0/2522059969 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc018680 con 0x7fa4ec100640 2023-09-10T01:14:16.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.815+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec1a56c0 con 0x7fa4ec100640 2023-09-10T01:14:16.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.818+0000 7fa4d3fff700 1 -- 172.21.15.3:0/2522059969 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa4dc00cd40 con 0x7fa4ec100640 2023-09-10T01:14:16.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.819+0000 7fa50512eb80 1 Processor -- start 2023-09-10T01:14:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.819+0000 7fa4d3fff700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4d40a0830 0x7fa4d40a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa4d3fff700 1 -- 172.21.15.3:0/2522059969 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa4dc014070 con 0x7fa4ec100640 2023-09-10T01:14:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa50512eb80 1 -- start start 2023-09-10T01:14:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4d40a0830 0x7fa4d40a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8607790 0x55e7e8607b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e8611980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e86142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa50512eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e85607b0 con 0x55e7e8607790 2023-09-10T01:14:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa50512eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e855e0d0 con 0x55e7e8611ec0 2023-09-10T01:14:16.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa50512eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e85599c0 con 0x55e7e86080b0 2023-09-10T01:14:16.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa4eaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8607790 0x55e7e8607b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa4eaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8607790 0x55e7e8607b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49516/0 (socket says 172.21.15.3:49516) 2023-09-10T01:14:16.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa4eaffd700 1 -- 172.21.15.3:0/3661341547 learned_addr learned my addr 172.21.15.3:0/3661341547 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:16.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e8611980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.820+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e86142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4d40a0830 0x7fa4d40a2cf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400f590 tx=0x7fa4e400b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:16.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4eaffd700 1 -- 172.21.15.3:0/3661341547 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 msgr2=0x55e7e8611980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e8611980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4eaffd700 1 -- 172.21.15.3:0/3661341547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 msgr2=0x55e7e86142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e86142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4eaffd700 1 -- 172.21.15.3:0/3661341547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7e8385e60 con 0x55e7e8607790 2023-09-10T01:14:16.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8607790 0x55e7e8607b70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc096050 tx=0x7fa4dc098f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e902615a928534ce server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:16.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4d17fa700 1 -- 172.21.15.3:0/3661341547 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0995e0 con 0x55e7e8607790 2023-09-10T01:14:16.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4d17fa700 1 -- 172.21.15.3:0/3661341547 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4dc09d410 con 0x55e7e8607790 2023-09-10T01:14:16.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.821+0000 7fa4d17fa700 1 -- 172.21.15.3:0/3661341547 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc099840 con 0x55e7e8607790 2023-09-10T01:14:16.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 -- 172.21.15.3:0/3661341547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8607790 msgr2=0x55e7e8607b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8607790 0x55e7e8607b70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc096050 tx=0x7fa4dc098f40 comp rx=0 tx=0).stop 2023-09-10T01:14:16.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 -- 172.21.15.3:0/3661341547 shutdown_connections 2023-09-10T01:14:16.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e86142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8607790 0x55e7e8607b70 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3661341547 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e8611980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 -- 172.21.15.3:0/3661341547 >> 172.21.15.3:0/3661341547 conn(0x55e7e857c110 msgr2=0x55e7e8602c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:16.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 -- 172.21.15.3:0/3661341547 shutdown_connections 2023-09-10T01:14:16.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 -- 172.21.15.3:0/3661341547 wait complete. 2023-09-10T01:14:16.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.822+0000 7fa50512eb80 1 -- start start 2023-09-10T01:14:16.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 1 Processor -- start 2023-09-10T01:14:16.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8607790 0x55e7e860ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e860f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e860f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e84f7bf0 con 0x55e7e8611ec0 2023-09-10T01:14:16.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e84f78f0 con 0x55e7e8607790 2023-09-10T01:14:16.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e83db9a0 con 0x55e7e86080b0 2023-09-10T01:14:16.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa50512eb80 10 client.0 _ll_register_callbacks cb 0x55e7e8441b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:16.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.823+0000 7fa4ea7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e860f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4ea7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e860f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:57958/0 (socket says 172.21.15.3:57958) 2023-09-10T01:14:16.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e860f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/2715654361 learned_addr learned my addr 172.21.15.3:0/2715654361 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:16.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8607790 0x55e7e860ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eb7fe700 1 -- 172.21.15.3:0/2715654361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 msgr2=0x55e7e860f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e860f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eb7fe700 1 -- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8607790 msgr2=0x55e7e860ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8607790 0x55e7e860ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eb7fe700 1 -- 172.21.15.3:0/2715654361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec1a56c0 con 0x55e7e8611ec0 2023-09-10T01:14:16.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8607790 0x55e7e860ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:16.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e860f6a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400f400 tx=0x7fa4e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731b56e7c3619561 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:16.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:16.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.824+0000 7fa4ceff5700 1 -- 172.21.15.3:0/2715654361 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400e040 con 0x55e7e8611ec0 2023-09-10T01:14:16.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7e84b1570 con 0x55e7e8611ec0 2023-09-10T01:14:16.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa4ceff5700 1 -- 172.21.15.3:0/2715654361 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa4e4017e30 con 0x55e7e8611ec0 2023-09-10T01:14:16.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa4ceff5700 1 -- 172.21.15.3:0/2715654361 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4e4009230 con 0x55e7e8611ec0 2023-09-10T01:14:16.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 10 client.17997 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:16.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55e7e836ece0 con 0x55e7e8611ec0 2023-09-10T01:14:16.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa4ceff5700 1 -- 172.21.15.3:0/2715654361 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e7e84b1570 con 0x55e7e8611ec0 2023-09-10T01:14:16.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa4ceff5700 1 -- 172.21.15.3:0/2715654361 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa4e4028020 con 0x55e7e8611ec0 2023-09-10T01:14:16.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 10 client.17997 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:16.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 10 client.17997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:16.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 20 client.17997 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:16.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e8621ac0 0x55e7e8623ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 4 client.17997 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:16.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.825+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7e856e920 con 0x55e7e8621ac0 2023-09-10T01:14:16.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.826+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e8621ac0 0x55e7e8623ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.826+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e8621ac0 0x55e7e8623ee0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c0cd2a98b83fcb2 server_cookie=6e040ca5dfa65314 in_seq=0 out_seq=0 2023-09-10T01:14:16.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.826+0000 7fa4ceff5700 10 client.17997 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:16.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa4ceff5700 1 -- 172.21.15.3:0/2715654361 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e7e856e920 con 0x55e7e8621ac0 2023-09-10T01:14:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa4ceff5700 10 client.17997 handle_command_reply: tid=0 2023-09-10T01:14:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 client.17997 shutdown 2023-09-10T01:14:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 msgr2=0x55e7e860f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e860f6a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400f400 tx=0x7fa4e4004ad0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 shutdown_connections 2023-09-10T01:14:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e8621ac0 0x55e7e8623ee0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e860f6a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8607790 0x55e7e860ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2715654361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e86080b0 0x55e7e860f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa4ceff5700 0 client.17997 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:16.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.827+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 >> 172.21.15.3:0/2715654361 conn(0x55e7e857c110 msgr2=0x55e7e8602860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:16.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.828+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 shutdown_connections 2023-09-10T01:14:16.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.828+0000 7fa50512eb80 1 -- 172.21.15.3:0/2715654361 wait complete. 2023-09-10T01:14:16.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.828+0000 7fa50512eb80 20 client.17997 trim_cache size 0 max 16384 2023-09-10T01:14:16.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.845+0000 7fa50512eb80 1 Processor -- start 2023-09-10T01:14:16.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa50512eb80 1 -- start start 2023-09-10T01:14:16.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e860f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e86080b0 0x55e7e860beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e860c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa50512eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e861f0a0 con 0x55e7e86080b0 2023-09-10T01:14:16.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa50512eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e8608a70 con 0x55e7e8611ec0 2023-09-10T01:14:16.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa50512eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e8602090 con 0x55e7e8607790 2023-09-10T01:14:16.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa4ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e86080b0 0x55e7e860beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa4ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e86080b0 0x55e7e860beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49530/0 (socket says 172.21.15.3:49530) 2023-09-10T01:14:16.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/2414877198 learned_addr learned my addr 172.21.15.3:0/2414877198 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:16.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e860f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.846+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e860c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/2414877198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 msgr2=0x55e7e860f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e860f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/2414877198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 msgr2=0x55e7e860c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e860c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/2414877198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7e8628ad0 con 0x55e7e86080b0 2023-09-10T01:14:16.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e860f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:16.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4eb7fe700 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e860c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:16.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e86080b0 0x55e7e860beb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009990 tx=0x7fa4e000b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f099e6864d9b262 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:16.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4d1ffb700 1 -- 172.21.15.3:0/2414877198 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e000bb10 con 0x55e7e86080b0 2023-09-10T01:14:16.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4d1ffb700 1 -- 172.21.15.3:0/2414877198 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4e001e040 con 0x55e7e86080b0 2023-09-10T01:14:16.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.847+0000 7fa4d1ffb700 1 -- 172.21.15.3:0/2414877198 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0004760 con 0x55e7e86080b0 2023-09-10T01:14:16.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 -- 172.21.15.3:0/2414877198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e86080b0 msgr2=0x55e7e860beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e86080b0 0x55e7e860beb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009990 tx=0x7fa4e000b280 comp rx=0 tx=0).stop 2023-09-10T01:14:16.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 -- 172.21.15.3:0/2414877198 shutdown_connections 2023-09-10T01:14:16.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e860f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e8611ec0 0x55e7e860c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 --2- 172.21.15.3:0/2414877198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e86080b0 0x55e7e860beb0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 -- 172.21.15.3:0/2414877198 >> 172.21.15.3:0/2414877198 conn(0x55e7e857c110 msgr2=0x55e7e8609f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:16.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 -- 172.21.15.3:0/2414877198 shutdown_connections 2023-09-10T01:14:16.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 -- 172.21.15.3:0/2414877198 wait complete. 2023-09-10T01:14:16.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 -- start start 2023-09-10T01:14:16.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.848+0000 7fa50512eb80 1 Processor -- start 2023-09-10T01:14:16.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e86177b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e86080b0 0x55e7e8617cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa50512eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e866f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa50512eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e866fdd0 con 0x55e7e8611ec0 2023-09-10T01:14:16.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa50512eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e866ff40 con 0x55e7e86080b0 2023-09-10T01:14:16.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa50512eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e8670100 con 0x55e7e8607790 2023-09-10T01:14:16.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa50512eb80 10 client.0 _ll_register_callbacks cb 0x7fa4e40177e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:16.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa4ea7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e86080b0 0x55e7e8617cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa4eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e866f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa4ea7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e86080b0 0x55e7e8617cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50916/0 (socket says 172.21.15.3:50916) 2023-09-10T01:14:16.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/3651099788 learned_addr learned my addr 172.21.15.3:0/3651099788 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:16.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.849+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e86177b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/3651099788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 msgr2=0x55e7e86177b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e86177b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/3651099788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 msgr2=0x55e7e866f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:16.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e866f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ea7fc700 1 -- 172.21.15.3:0/3651099788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7e8385e60 con 0x55e7e86080b0 2023-09-10T01:14:16.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e86177b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:16.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ea7fc700 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e86080b0 0x55e7e8617cf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005b40 tx=0x7fa4e0009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccb00532bd8dba8b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:16.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:14:16.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ceff5700 1 -- 172.21.15.3:0/3651099788 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e000f930 con 0x55e7e86080b0 2023-09-10T01:14:16.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4dc09d410 con 0x55e7e86080b0 2023-09-10T01:14:16.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.850+0000 7fa4ceff5700 1 -- 172.21.15.3:0/3651099788 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa4e0033070 con 0x55e7e86080b0 2023-09-10T01:14:16.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa4ceff5700 1 -- 172.21.15.3:0/3651099788 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4e000fa90 con 0x55e7e86080b0 2023-09-10T01:14:16.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 10 client.28091 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:16.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7fa4ec10de10 con 0x55e7e86080b0 2023-09-10T01:14:16.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa4ceff5700 1 -- 172.21.15.3:0/3651099788 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4dc09d410 con 0x55e7e86080b0 2023-09-10T01:14:16.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa4ceff5700 1 -- 172.21.15.3:0/3651099788 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa4e0038d10 con 0x55e7e86080b0 2023-09-10T01:14:16.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 10 client.28091 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:16.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 10 client.28091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 20 client.28091 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e86775d0 0x55e7e86799f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 4 client.28091 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:14:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e7e85529a0 con 0x55e7e86775d0 2023-09-10T01:14:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.851+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e86775d0 0x55e7e86799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:16.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.852+0000 7fa4eaffd700 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e86775d0 0x55e7e86799f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f02d0a7a79657f68 server_cookie=c98458a6f9a27478 in_seq=0 out_seq=0 2023-09-10T01:14:16.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.852+0000 7fa4ceff5700 10 client.28091 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:16.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.852+0000 7fa4ceff5700 1 -- 172.21.15.3:0/3651099788 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e7e85529a0 con 0x55e7e86775d0 2023-09-10T01:14:16.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.852+0000 7fa4ceff5700 10 client.28091 handle_command_reply: tid=0 2023-09-10T01:14:16.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 client.28091 shutdown 2023-09-10T01:14:16.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e86080b0 msgr2=0x55e7e8617cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e86080b0 0x55e7e8617cf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005b40 tx=0x7fa4e0009920 comp rx=0 tx=0).stop 2023-09-10T01:14:16.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 shutdown_connections 2023-09-10T01:14:16.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e7e86775d0 0x55e7e86799f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e7e8607790 0x55e7e86177b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7e8611ec0 0x55e7e866f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa4ceff5700 0 client.28091 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:16.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 --2- 172.21.15.3:0/3651099788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7e86080b0 0x55e7e8617cf0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 >> 172.21.15.3:0/3651099788 conn(0x55e7e857c110 msgr2=0x55e7e860f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:16.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 shutdown_connections 2023-09-10T01:14:16.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 1 -- 172.21.15.3:0/3651099788 wait complete. 2023-09-10T01:14:16.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.853+0000 7fa50512eb80 20 client.28091 trim_cache size 0 max 16384 2023-09-10T01:14:16.868 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:14:16.869 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:14:16.869 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "4ca92f46-966f-4e25-adba-5a6488f820cb", 2023-09-10T01:14:16.869 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:14:16.869 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:14:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4d40a0830 msgr2=0x7fa4d40a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4d40a0830 0x7fa4d40a2cf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400f590 tx=0x7fa4e400b040 comp rx=0 tx=0).stop 2023-09-10T01:14:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec100640 msgr2=0x7fa4ec19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec100640 0x7fa4ec19e690 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00e9a0 tx=0x7fa4dc00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 shutdown_connections 2023-09-10T01:14:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4d40a0830 0x7fa4d40a2cf0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ec106650 0x7fa4ec198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4ec100640 0x7fa4ec19e690 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 --2- 172.21.15.3:0/2522059969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4ec101000 0x7fa4ec19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 >> 172.21.15.3:0/2522059969 conn(0x7fa4ec0fc170 msgr2=0x7fa4ec10ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:16.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 shutdown_connections 2023-09-10T01:14:16.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:16.854+0000 7fa4f17ff700 1 -- 172.21.15.3:0/2522059969 wait complete. 2023-09-10T01:14:16.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:14:17.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 -- 172.21.15.3:0/1552110712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c101000 msgr2=0x7fef1c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 --2- 172.21.15.3:0/1552110712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c101000 0x7fef1c10d510 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fef1400b0a0 tx=0x7fef1400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 -- 172.21.15.3:0/1552110712 shutdown_connections 2023-09-10T01:14:17.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 --2- 172.21.15.3:0/1552110712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c101000 0x7fef1c10d510 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 --2- 172.21.15.3:0/1552110712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c106650 0x7fef1c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 --2- 172.21.15.3:0/1552110712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef1c100640 0x7fef1c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 -- 172.21.15.3:0/1552110712 >> 172.21.15.3:0/1552110712 conn(0x7fef1c0fc170 msgr2=0x7fef1c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:17.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 -- 172.21.15.3:0/1552110712 shutdown_connections 2023-09-10T01:14:17.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.097+0000 7fef239ff700 1 -- 172.21.15.3:0/1552110712 wait complete. 2023-09-10T01:14:17.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.098+0000 7fef239ff700 1 Processor -- start 2023-09-10T01:14:17.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.098+0000 7fef239ff700 1 -- start start 2023-09-10T01:14:17.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef239ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c100640 0x7fef1c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.102 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef239ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef1c101000 0x7fef1c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef239ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c106650 0x7fef1c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef239ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c198d90 con 0x7fef1c100640 2023-09-10T01:14:17.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef239ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c198f00 con 0x7fef1c106650 2023-09-10T01:14:17.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef239ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c1990c0 con 0x7fef1c101000 2023-09-10T01:14:17.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef2179b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c100640 0x7fef1c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef21f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c106650 0x7fef1c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef2179b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c100640 0x7fef1c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49552/0 (socket says 172.21.15.3:49552) 2023-09-10T01:14:17.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef2179b700 1 -- 172.21.15.3:0/240328732 learned_addr learned my addr 172.21.15.3:0/240328732 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:17.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef20f9a700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef1c101000 0x7fef1c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef21f9c700 1 -- 172.21.15.3:0/240328732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef1c101000 msgr2=0x7fef1c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef21f9c700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef1c101000 0x7fef1c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef21f9c700 1 -- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c100640 msgr2=0x7fef1c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef21f9c700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c100640 0x7fef1c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef21f9c700 1 -- 172.21.15.3:0/240328732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef14009d00 con 0x7fef1c106650 2023-09-10T01:14:17.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef2179b700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c100640 0x7fef1c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:17.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.099+0000 7fef20f9a700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef1c101000 0x7fef1c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:17.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.100+0000 7fef21f9c700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c106650 0x7fef1c1986e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fef140062a0 tx=0x7fef140077f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:17.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.100+0000 7fef127fc700 1 -- 172.21.15.3:0/240328732 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1400e040 con 0x7fef1c106650 2023-09-10T01:14:17.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.100+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c199340 con 0x7fef1c106650 2023-09-10T01:14:17.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.100+0000 7fef127fc700 1 -- 172.21.15.3:0/240328732 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef140094c0 con 0x7fef1c106650 2023-09-10T01:14:17.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.100+0000 7fef127fc700 1 -- 172.21.15.3:0/240328732 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14005040 con 0x7fef1c106650 2023-09-10T01:14:17.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.100+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c1a5680 con 0x7fef1c106650 2023-09-10T01:14:17.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.103+0000 7fef127fc700 1 -- 172.21.15.3:0/240328732 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fef1401d470 con 0x7fef1c106650 2023-09-10T01:14:17.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.104+0000 7fef3737fb80 1 Processor -- start 2023-09-10T01:14:17.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.104+0000 7fef3737fb80 1 -- start start 2023-09-10T01:14:17.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadacaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef3737fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbad05730 con 0x555cbadac710 2023-09-10T01:14:17.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef127fc700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef080a07b0 0x7fef080a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef3737fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbad03050 con 0x555cbadb6e40 2023-09-10T01:14:17.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef3737fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbacfe940 con 0x555cbadad030 2023-09-10T01:14:17.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef127fc700 1 -- 172.21.15.3:0/240328732 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fef140cda50 con 0x7fef1c106650 2023-09-10T01:14:17.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef21f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef20f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadacaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef21f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50930/0 (socket says 172.21.15.3:50930) 2023-09-10T01:14:17.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef21f9c700 1 -- 172.21.15.3:0/3338280255 learned_addr learned my addr 172.21.15.3:0/3338280255 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:17.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef2179b700 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.105+0000 7fef20f9a700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef080a07b0 0x7fef080a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef21f9c700 1 -- 172.21.15.3:0/3338280255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 msgr2=0x555cbadb6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef21f9c700 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef21f9c700 1 -- 172.21.15.3:0/3338280255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 msgr2=0x555cbadacaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef21f9c700 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadacaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef21f9c700 1 -- 172.21.15.3:0/3338280255 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cbab2ade0 con 0x555cbadb6e40 2023-09-10T01:14:17.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef21f9c700 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb9230 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fef14097610 tx=0x7fef14007ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbd80f7e9c65e9b5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:17.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef20f9a700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef080a07b0 0x7fef080a2c70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef0c005950 tx=0x7fef0c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:17.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef077fe700 1 -- 172.21.15.3:0/3338280255 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1409b2a0 con 0x555cbadb6e40 2023-09-10T01:14:17.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef077fe700 1 -- 172.21.15.3:0/3338280255 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef1409aba0 con 0x555cbadb6e40 2023-09-10T01:14:17.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.106+0000 7fef077fe700 1 -- 172.21.15.3:0/3338280255 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef140972b0 con 0x555cbadb6e40 2023-09-10T01:14:17.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 -- 172.21.15.3:0/3338280255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 msgr2=0x555cbadb9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb9230 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fef14097610 tx=0x7fef14007ae0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 -- 172.21.15.3:0/3338280255 shutdown_connections 2023-09-10T01:14:17.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb9230 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadacaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3338280255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 -- 172.21.15.3:0/3338280255 >> 172.21.15.3:0/3338280255 conn(0x555cbad21090 msgr2=0x555cbada7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:17.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 -- 172.21.15.3:0/3338280255 shutdown_connections 2023-09-10T01:14:17.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 -- 172.21.15.3:0/3338280255 wait complete. 2023-09-10T01:14:17.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.107+0000 7fef3737fb80 1 -- start start 2023-09-10T01:14:17.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 1 Processor -- start 2023-09-10T01:14:17.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadac710 0x555cbadb6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadad030 0x555cbadb3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadb6e40 0x555cbadb4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbac9cb70 con 0x555cbadb6e40 2023-09-10T01:14:17.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbac9c870 con 0x555cbadad030 2023-09-10T01:14:17.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbab80920 con 0x555cbadac710 2023-09-10T01:14:17.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef3737fb80 10 client.0 _ll_register_callbacks cb 0x555cbabe6a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:17.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef2179b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadac710 0x555cbadb6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.108+0000 7fef21f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadb6e40 0x555cbadb4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadac710 0x555cbadb6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58006/0 (socket says 172.21.15.3:58006) 2023-09-10T01:14:17.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 -- 172.21.15.3:0/3509356358 learned_addr learned my addr 172.21.15.3:0/3509356358 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:17.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef20f9a700 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadad030 0x555cbadb3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 -- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadad030 msgr2=0x555cbadb3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadad030 0x555cbadb3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 -- 172.21.15.3:0/3509356358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadb6e40 msgr2=0x555cbadb4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadb6e40 0x555cbadb4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 -- 172.21.15.3:0/3509356358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c1a5680 con 0x555cbadac710 2023-09-10T01:14:17.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef21f9c700 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadb6e40 0x555cbadb4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:17.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef2179b700 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadac710 0x555cbadb6880 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fef18011d10 tx=0x7fef1800f6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e4788a9db9ff3b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:17.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef04ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:17.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef04ff9700 1 -- 172.21.15.3:0/3509356358 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18016070 con 0x555cbadac710 2023-09-10T01:14:17.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555cbac564f0 con 0x555cbadac710 2023-09-10T01:14:17.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.109+0000 7fef04ff9700 1 -- 172.21.15.3:0/3509356358 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fef1800fd90 con 0x555cbadac710 2023-09-10T01:14:17.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef04ff9700 1 -- 172.21.15.3:0/3509356358 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef18004020 con 0x555cbadac710 2023-09-10T01:14:17.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 10 client.27991 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:17.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x555cbab13c60 con 0x555cbadac710 2023-09-10T01:14:17.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef04ff9700 1 -- 172.21.15.3:0/3509356358 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555cbac564f0 con 0x555cbadac710 2023-09-10T01:14:17.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef04ff9700 1 -- 172.21.15.3:0/3509356358 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fef1801a3f0 con 0x555cbadac710 2023-09-10T01:14:17.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 10 client.27991 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:17.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 10 client.27991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:17.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 20 client.27991 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:17.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbadc6910 0x555cbadc8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 4 client.27991 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:17.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.110+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555cbad138a0 con 0x555cbadc6910 2023-09-10T01:14:17.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.111+0000 7fef20f9a700 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbadc6910 0x555cbadc8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.111+0000 7fef04ff9700 10 client.27991 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:17.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.111+0000 7fef20f9a700 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbadc6910 0x555cbadc8d30 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41e4349352dfb5ad server_cookie=f7fb279cb3e29ae3 in_seq=0 out_seq=0 2023-09-10T01:14:17.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef04ff9700 1 -- 172.21.15.3:0/3509356358 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555cbad138a0 con 0x555cbadc6910 2023-09-10T01:14:17.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef04ff9700 10 client.27991 handle_command_reply: tid=0 2023-09-10T01:14:17.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 client.27991 shutdown 2023-09-10T01:14:17.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadac710 msgr2=0x555cbadb6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadac710 0x555cbadb6880 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fef18011d10 tx=0x7fef1800f6a0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 shutdown_connections 2023-09-10T01:14:17.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbadc6910 0x555cbadc8d30 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadac710 0x555cbadb6880 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadb6e40 0x555cbadb4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef04ff9700 0 client.27991 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:17.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3509356358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadad030 0x555cbadb3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 >> 172.21.15.3:0/3509356358 conn(0x555cbad21090 msgr2=0x555cbada77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:17.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 shutdown_connections 2023-09-10T01:14:17.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.112+0000 7fef3737fb80 1 -- 172.21.15.3:0/3509356358 wait complete. 2023-09-10T01:14:17.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.113+0000 7fef3737fb80 20 client.27991 trim_cache size 0 max 16384 2023-09-10T01:14:17.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 Processor -- start 2023-09-10T01:14:17.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 -- start start 2023-09-10T01:14:17.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadb6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbadad9f0 con 0x555cbadac710 2023-09-10T01:14:17.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbadb50e0 con 0x555cbadb6e40 2023-09-10T01:14:17.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.130+0000 7fef3737fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbada43d0 con 0x555cbadad030 2023-09-10T01:14:17.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef2179b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadb6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58018/0 (socket says 172.21.15.3:58018) 2023-09-10T01:14:17.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 -- 172.21.15.3:0/3208540757 learned_addr learned my addr 172.21.15.3:0/3208540757 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:17.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef2179b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadb6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49570/0 (socket says 172.21.15.3:49570) 2023-09-10T01:14:17.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef21f9c700 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 -- 172.21.15.3:0/3208540757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 msgr2=0x555cbadb3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 -- 172.21.15.3:0/3208540757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 msgr2=0x555cbadb6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadb6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 -- 172.21.15.3:0/3208540757 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cbab51250 con 0x555cbadad030 2023-09-10T01:14:17.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.131+0000 7fef20f9a700 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb3300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fef1c199ed0 tx=0x7fef0c0058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4fb6935d7858062 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:17.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef07fff700 1 -- 172.21.15.3:0/3208540757 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c024040 con 0x555cbadad030 2023-09-10T01:14:17.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef07fff700 1 -- 172.21.15.3:0/3208540757 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef0c025040 con 0x555cbadad030 2023-09-10T01:14:17.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef07fff700 1 -- 172.21.15.3:0/3208540757 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c005280 con 0x555cbadad030 2023-09-10T01:14:17.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 -- 172.21.15.3:0/3208540757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 msgr2=0x555cbadb3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb3300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fef1c199ed0 tx=0x7fef0c0058e0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 -- 172.21.15.3:0/3208540757 shutdown_connections 2023-09-10T01:14:17.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadb6e40 0x555cbadb3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadac710 0x555cbadb6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 --2- 172.21.15.3:0/3208540757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadad030 0x555cbadb3300 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 -- 172.21.15.3:0/3208540757 >> 172.21.15.3:0/3208540757 conn(0x555cbad21090 msgr2=0x555cbadaee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:17.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 -- 172.21.15.3:0/3208540757 shutdown_connections 2023-09-10T01:14:17.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.132+0000 7fef3737fb80 1 -- 172.21.15.3:0/3208540757 wait complete. 2023-09-10T01:14:17.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 -- start start 2023-09-10T01:14:17.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 Processor -- start 2023-09-10T01:14:17.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadac710 0x555cbadbd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadad030 0x555cbadbd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadb6e40 0x555cbadbb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbadbc010 con 0x555cbadad030 2023-09-10T01:14:17.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbadbc180 con 0x555cbadac710 2023-09-10T01:14:17.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cbadbc340 con 0x555cbadb6e40 2023-09-10T01:14:17.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.133+0000 7fef3737fb80 10 client.0 _ll_register_callbacks cb 0x7fef180117e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:17.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef2179b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadac710 0x555cbadbd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef21f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadb6e40 0x555cbadbb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef2179b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadac710 0x555cbadbd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50970/0 (socket says 172.21.15.3:50970) 2023-09-10T01:14:17.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef20f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadad030 0x555cbadbd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef2179b700 1 -- 172.21.15.3:0/2533265251 learned_addr learned my addr 172.21.15.3:0/2533265251 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:17.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef21f9c700 1 -- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadac710 msgr2=0x555cbadbd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef21f9c700 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadac710 0x555cbadbd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef21f9c700 1 -- 172.21.15.3:0/2533265251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadad030 msgr2=0x555cbadbd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:17.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef21f9c700 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadad030 0x555cbadbd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef21f9c700 1 -- 172.21.15.3:0/2533265251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0c019040 con 0x555cbadb6e40 2023-09-10T01:14:17.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef2179b700 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadac710 0x555cbadbd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:17.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef21f9c700 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadb6e40 0x555cbadbb900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fef1409a8a0 tx=0x7fef140097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=893b9de6ad0af070 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:17.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef04ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:17.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef04ff9700 1 -- 172.21.15.3:0/2533265251 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1401d920 con 0x555cbadb6e40 2023-09-10T01:14:17.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef1409aba0 con 0x555cbadb6e40 2023-09-10T01:14:17.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.134+0000 7fef04ff9700 1 -- 172.21.15.3:0/2533265251 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fef1409b2a0 con 0x555cbadb6e40 2023-09-10T01:14:17.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef04ff9700 1 -- 172.21.15.3:0/2533265251 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef1409e9d0 con 0x555cbadb6e40 2023-09-10T01:14:17.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 10 client.27997 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:17.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x555cbab292b0 con 0x555cbadb6e40 2023-09-10T01:14:17.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef04ff9700 1 -- 172.21.15.3:0/2533265251 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef1409aba0 con 0x555cbadb6e40 2023-09-10T01:14:17.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef04ff9700 1 -- 172.21.15.3:0/2533265251 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fef140a26f0 con 0x555cbadb6e40 2023-09-10T01:14:17.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 10 client.27997 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:17.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 10 client.27997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:17.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 20 client.27997 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:17.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbae1b4e0 0x555cbae1d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:17.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 4 client.27997 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:14:17.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555cbacf7920 con 0x555cbae1b4e0 2023-09-10T01:14:17.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.135+0000 7fef2179b700 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbae1b4e0 0x555cbae1d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:17.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef2179b700 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbae1b4e0 0x555cbae1d900 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dedec4416ad5b488 server_cookie=330f7c692f246906 in_seq=0 out_seq=0 2023-09-10T01:14:17.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef04ff9700 10 client.27997 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:17.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef04ff9700 1 -- 172.21.15.3:0/2533265251 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555cbacf7920 con 0x555cbae1b4e0 2023-09-10T01:14:17.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef04ff9700 10 client.27997 handle_command_reply: tid=0 2023-09-10T01:14:17.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 client.27997 shutdown 2023-09-10T01:14:17.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadb6e40 msgr2=0x555cbadbb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadb6e40 0x555cbadbb900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fef1409a8a0 tx=0x7fef140097e0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 shutdown_connections 2023-09-10T01:14:17.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555cbae1b4e0 0x555cbae1d900 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555cbadb6e40 0x555cbadbb900 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555cbadac710 0x555cbadbd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef04ff9700 0 client.27997 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:17.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 --2- 172.21.15.3:0/2533265251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cbadad030 0x555cbadbd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 >> 172.21.15.3:0/2533265251 conn(0x555cbad21090 msgr2=0x555cbadab700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:17.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 shutdown_connections 2023-09-10T01:14:17.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.136+0000 7fef3737fb80 1 -- 172.21.15.3:0/2533265251 wait complete. 2023-09-10T01:14:17.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef3737fb80 20 client.27997 trim_cache size 0 max 16384 2023-09-10T01:14:17.161 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:14:17.161 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:14:17.162 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:14:17.162 INFO:teuthology.orchestra.run.smithi003.stdout: "4ca92f46-966f-4e25-adba-5a6488f820cb": { 2023-09-10T01:14:17.162 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:14:17.162 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "4ca92f46-966f-4e25-adba-5a6488f820cb", 2023-09-10T01:14:17.162 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:14:17.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:14:17.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:14:17.163 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:14:17.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef080a07b0 msgr2=0x7fef080a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef080a07b0 0x7fef080a2c70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef0c005950 tx=0x7fef0c009260 comp rx=0 tx=0).stop 2023-09-10T01:14:17.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c106650 msgr2=0x7fef1c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:17.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c106650 0x7fef1c1986e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fef140062a0 tx=0x7fef140077f0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 shutdown_connections 2023-09-10T01:14:17.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fef080a07b0 0x7fef080a2c70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c106650 0x7fef1c1986e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef1c100640 0x7fef1c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 --2- 172.21.15.3:0/240328732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef1c101000 0x7fef1c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:17.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 >> 172.21.15.3:0/240328732 conn(0x7fef1c0fc170 msgr2=0x7fef1c10ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:17.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 shutdown_connections 2023-09-10T01:14:17.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:17.137+0000 7fef239ff700 1 -- 172.21.15.3:0/240328732 wait complete. 2023-09-10T01:14:17.167 INFO:tasks.cephfs.filesystem:scrub status for tag:4ca92f46-966f-4e25-adba-5a6488f820cb - {'path': '/', 'tag': '4ca92f46-966f-4e25-adba-5a6488f820cb', 'options': 'recursive,force'} 2023-09-10T01:14:17.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: pgmap v345: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 3.5 MiB/s wr, 96 op/s 2023-09-10T01:14:17.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:14:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:17 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: pgmap v345: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 3.5 MiB/s wr, 96 op/s 2023-09-10T01:14:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:14:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:17.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:17 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 953175 8.31 MB/sec execute 479 sec latency 3773.628 ms 2023-09-10T01:14:17.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: pgmap v345: 129 pgs: 129 active+clean; 287 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 3.5 MiB/s wr, 96 op/s 2023-09-10T01:14:17.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:14:17.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:17.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:17 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:18 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:18 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:18.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:18 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:18.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:18 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:18 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:18.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:18 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:18.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:18 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:18.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:18 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:18.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 954398 8.31 MB/sec execute 480 sec latency 1057.876 ms 2023-09-10T01:14:18.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:18 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:18.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:18 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:18.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:18 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:18.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:18 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:19.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:19 smithi136 ceph-mon[133708]: pgmap v346: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.7 MiB/s wr, 99 op/s 2023-09-10T01:14:19.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:19 smithi092 ceph-mon[134067]: pgmap v346: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.7 MiB/s wr, 99 op/s 2023-09-10T01:14:19.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 955964 8.32 MB/sec execute 481 sec latency 52.009 ms 2023-09-10T01:14:19.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:19 smithi003 ceph-mon[97654]: pgmap v346: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 1.7 MiB/s wr, 99 op/s 2023-09-10T01:14:20.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 956657 8.30 MB/sec execute 482 sec latency 99.634 ms 2023-09-10T01:14:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 -- 172.21.15.3:0/3328185251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc106680 msgr2=0x7fd4cc106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/3328185251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc106680 0x7fd4cc106a60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80099c0 tx=0x7fd4c8009cd0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 -- 172.21.15.3:0/3328185251 shutdown_connections 2023-09-10T01:14:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/3328185251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4cc100f80 0x7fd4cc10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/3328185251 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc106680 0x7fd4cc106a60 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/3328185251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4cc1005c0 0x7fd4cc100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 -- 172.21.15.3:0/3328185251 >> 172.21.15.3:0/3328185251 conn(0x7fd4cc0fc0b0 msgr2=0x7fd4cc0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:21.194 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 -- 172.21.15.3:0/3328185251 shutdown_connections 2023-09-10T01:14:21.195 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.191+0000 7fd4d47ff700 1 -- 172.21.15.3:0/3328185251 wait complete. 2023-09-10T01:14:21.196 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 Processor -- start 2023-09-10T01:14:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 -- start start 2023-09-10T01:14:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc1005c0 0x7fd4cc072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4cc100f80 0x7fd4cc072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4cc106680 0x7fd4cc1a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.197 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc073670 con 0x7fd4cc1005c0 2023-09-10T01:14:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc1a4da0 con 0x7fd4cc106680 2023-09-10T01:14:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.192+0000 7fd4d47ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc1a4ee0 con 0x7fd4cc100f80 2023-09-10T01:14:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc1005c0 0x7fd4cc072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d2d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4cc106680 0x7fd4cc1a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d1d9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4cc100f80 0x7fd4cc072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc1005c0 0x7fd4cc072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49598/0 (socket says 172.21.15.3:49598) 2023-09-10T01:14:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d259b700 1 -- 172.21.15.3:0/1515398760 learned_addr learned my addr 172.21.15.3:0/1515398760 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d2d9c700 1 -- 172.21.15.3:0/1515398760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4cc100f80 msgr2=0x7fd4cc072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d2d9c700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4cc100f80 0x7fd4cc072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d2d9c700 1 -- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc1005c0 msgr2=0x7fd4cc072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d2d9c700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc1005c0 0x7fd4cc072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d2d9c700 1 -- 172.21.15.3:0/1515398760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c80095f0 con 0x7fd4cc106680 2023-09-10T01:14:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc1005c0 0x7fd4cc072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4cc100f80 0x7fd4cc072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:21.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.193+0000 7fd4d2d9c700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4cc106680 0x7fd4cc1a4860 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400efc0 tx=0x7fd4c400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.194+0000 7fd4c37fe700 1 -- 172.21.15.3:0/1515398760 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4018070 con 0x7fd4cc106680 2023-09-10T01:14:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.194+0000 7fd4c37fe700 1 -- 172.21.15.3:0/1515398760 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4c400f040 con 0x7fd4cc106680 2023-09-10T01:14:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.194+0000 7fd4c37fe700 1 -- 172.21.15.3:0/1515398760 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c400cd40 con 0x7fd4cc106680 2023-09-10T01:14:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.194+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc1a50e0 con 0x7fd4cc106680 2023-09-10T01:14:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.194+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc1a5570 con 0x7fd4cc106680 2023-09-10T01:14:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.197+0000 7fd4c37fe700 1 -- 172.21.15.3:0/1515398760 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd4c4003a00 con 0x7fd4cc106680 2023-09-10T01:14:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.198+0000 7fd4e816cb80 1 Processor -- start 2023-09-10T01:14:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.198+0000 7fd4c37fe700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4b80a07b0 0x7fd4b80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.198+0000 7fd4e816cb80 1 -- start start 2023-09-10T01:14:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.198+0000 7fd4c37fe700 1 -- 172.21.15.3:0/1515398760 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd4c40cbf10 con 0x7fd4cc106680 2023-09-10T01:14:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4b80a07b0 0x7fd4b80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe846580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe840780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe7996d0 con 0x559bbe8461a0 2023-09-10T01:14:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe796ff0 con 0x559bbe840300 2023-09-10T01:14:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe7928e0 con 0x559bbe840cc0 2023-09-10T01:14:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe846580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe846580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49612/0 (socket says 172.21.15.3:49612) 2023-09-10T01:14:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 -- 172.21.15.3:0/266907285 learned_addr learned my addr 172.21.15.3:0/266907285 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe840780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d2d9c700 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 -- 172.21.15.3:0/266907285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 msgr2=0x559bbe84cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4b80a07b0 0x7fd4b80a2c70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0740f0 tx=0x7fd4bc011830 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 -- 172.21.15.3:0/266907285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 msgr2=0x559bbe840780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe840780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.199+0000 7fd4d259b700 1 -- 172.21.15.3:0/266907285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bbe5bed80 con 0x559bbe8461a0 2023-09-10T01:14:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.200+0000 7fd4d259b700 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe846580 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8000c00 tx=0x7fd4c8005c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7acdcf3ed0b73b7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.200+0000 7fd4c0ff9700 1 -- 172.21.15.3:0/266907285 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c800bec0 con 0x559bbe8461a0 2023-09-10T01:14:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.200+0000 7fd4c0ff9700 1 -- 172.21.15.3:0/266907285 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4c801e040 con 0x559bbe8461a0 2023-09-10T01:14:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.200+0000 7fd4c0ff9700 1 -- 172.21.15.3:0/266907285 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c800f460 con 0x559bbe8461a0 2023-09-10T01:14:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 -- 172.21.15.3:0/266907285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 msgr2=0x559bbe846580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe846580 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8000c00 tx=0x7fd4c8005c40 comp rx=0 tx=0).stop 2023-09-10T01:14:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 -- 172.21.15.3:0/266907285 shutdown_connections 2023-09-10T01:14:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe846580 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/266907285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe840780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 -- 172.21.15.3:0/266907285 >> 172.21.15.3:0/266907285 conn(0x559bbe7b5030 msgr2=0x559bbe83b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:21.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 -- 172.21.15.3:0/266907285 shutdown_connections 2023-09-10T01:14:21.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.201+0000 7fd4e816cb80 1 -- 172.21.15.3:0/266907285 wait complete. 2023-09-10T01:14:21.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 -- start start 2023-09-10T01:14:21.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 Processor -- start 2023-09-10T01:14:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840300 0x559bbe84ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840cc0 0x559bbe84a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe8461a0 0x559bbe84a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe730b10 con 0x559bbe840300 2023-09-10T01:14:21.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe730810 con 0x559bbe840cc0 2023-09-10T01:14:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe6148c0 con 0x559bbe8461a0 2023-09-10T01:14:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.202+0000 7fd4e816cb80 10 client.0 _ll_register_callbacks cb 0x559bbe67aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840300 0x559bbe84ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d2d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe8461a0 0x559bbe84a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840cc0 0x559bbe84a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d2d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe8461a0 0x559bbe84a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58076/0 (socket says 172.21.15.3:58076) 2023-09-10T01:14:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d259b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840300 0x559bbe84ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49616/0 (socket says 172.21.15.3:49616) 2023-09-10T01:14:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d2d9c700 1 -- 172.21.15.3:0/2841166483 learned_addr learned my addr 172.21.15.3:0/2841166483 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840cc0 0x559bbe84a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:51004/0 (socket says 172.21.15.3:51004) 2023-09-10T01:14:21.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/2841166483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe8461a0 msgr2=0x559bbe84a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe8461a0 0x559bbe84a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/2841166483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840300 msgr2=0x559bbe84ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840300 0x559bbe84ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/2841166483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc10dd40 con 0x559bbe840cc0 2023-09-10T01:14:21.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d2d9c700 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe8461a0 0x559bbe84a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:21.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840cc0 0x559bbe84a100 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc01b000 tx=0x7fd4bc0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=284b00ba870e520a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.203+0000 7fd4b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:14:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/2841166483 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc004230 con 0x559bbe840cc0 2023-09-10T01:14:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559bbe6ea490 con 0x559bbe840cc0 2023-09-10T01:14:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/2841166483 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd4bc004390 con 0x559bbe840cc0 2023-09-10T01:14:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/2841166483 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4bc004550 con 0x559bbe840cc0 2023-09-10T01:14:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4e816cb80 10 client.28121 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559bbe5a7c00 con 0x559bbe840cc0 2023-09-10T01:14:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/2841166483 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559bbe6ea490 con 0x559bbe840cc0 2023-09-10T01:14:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/2841166483 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fd4bc003c80 con 0x559bbe840cc0 2023-09-10T01:14:21.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4e816cb80 10 client.28121 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.204+0000 7fd4e816cb80 10 client.28121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.205+0000 7fd4e816cb80 20 client.28121 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.205+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe85a550 0x559bbe85c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.205+0000 7fd4e816cb80 4 client.28121 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:21.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.205+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559bbe7a7840 con 0x559bbe85a550 2023-09-10T01:14:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.205+0000 7fd4d259b700 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe85a550 0x559bbe85c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.205+0000 7fd4d259b700 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe85a550 0x559bbe85c970 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cf422e0cc6b9764 server_cookie=d34b0307763f1fe4 in_seq=0 out_seq=0 2023-09-10T01:14:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.205+0000 7fd4b5ffb700 10 client.28121 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/2841166483 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559bbe7a7840 con 0x559bbe85a550 2023-09-10T01:14:21.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4b5ffb700 10 client.28121 handle_command_reply: tid=0 2023-09-10T01:14:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 client.28121 shutdown 2023-09-10T01:14:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840cc0 msgr2=0x559bbe84a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840cc0 0x559bbe84a100 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc01b000 tx=0x7fd4bc0047c0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 shutdown_connections 2023-09-10T01:14:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe85a550 0x559bbe85c970 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe8461a0 0x559bbe84a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840300 0x559bbe84ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/2841166483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840cc0 0x559bbe84a100 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4b5ffb700 0 client.28121 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.206+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 >> 172.21.15.3:0/2841166483 conn(0x559bbe7b5030 msgr2=0x559bbe83a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.207+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 shutdown_connections 2023-09-10T01:14:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.207+0000 7fd4e816cb80 1 -- 172.21.15.3:0/2841166483 wait complete. 2023-09-10T01:14:21.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.207+0000 7fd4e816cb80 20 client.28121 trim_cache size 0 max 16384 2023-09-10T01:14:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 Processor -- start 2023-09-10T01:14:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 -- start start 2023-09-10T01:14:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840300 0x559bbe845270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840cc0 0x559bbe8457b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe8461a0 0x559bbe844250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe8415f0 con 0x559bbe840cc0 2023-09-10T01:14:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe845e30 con 0x559bbe8461a0 2023-09-10T01:14:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.222+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe848700 con 0x559bbe840300 2023-09-10T01:14:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d1d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840cc0 0x559bbe8457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d259b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840300 0x559bbe845270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d1d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840cc0 0x559bbe8457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49622/0 (socket says 172.21.15.3:49622) 2023-09-10T01:14:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/1566977003 learned_addr learned my addr 172.21.15.3:0/1566977003 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d2d9c700 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe8461a0 0x559bbe844250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d259b700 1 -- 172.21.15.3:0/1566977003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe8461a0 msgr2=0x559bbe844250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe8461a0 0x559bbe844250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d259b700 1 -- 172.21.15.3:0/1566977003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840cc0 msgr2=0x559bbe8457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840cc0 0x559bbe8457b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d259b700 1 -- 172.21.15.3:0/1566977003 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bbe5e51f0 con 0x559bbe840300 2023-09-10T01:14:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840cc0 0x559bbe8457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840300 0x559bbe845270 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8025040 tx=0x7fd4c801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=860f3bb78bf87e99 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4c17fa700 1 -- 172.21.15.3:0/1566977003 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8031b40 con 0x559bbe840300 2023-09-10T01:14:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4c17fa700 1 -- 172.21.15.3:0/1566977003 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4c8031ca0 con 0x559bbe840300 2023-09-10T01:14:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.223+0000 7fd4c17fa700 1 -- 172.21.15.3:0/1566977003 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c80050b0 con 0x559bbe840300 2023-09-10T01:14:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1566977003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840300 msgr2=0x559bbe845270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840300 0x559bbe845270 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8025040 tx=0x7fd4c801a040 comp rx=0 tx=0).stop 2023-09-10T01:14:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1566977003 shutdown_connections 2023-09-10T01:14:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840300 0x559bbe845270 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe8461a0 0x559bbe844250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1566977003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe840cc0 0x559bbe8457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1566977003 >> 172.21.15.3:0/1566977003 conn(0x559bbe7b5030 msgr2=0x559bbe8565d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1566977003 shutdown_connections 2023-09-10T01:14:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1566977003 wait complete. 2023-09-10T01:14:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.224+0000 7fd4e816cb80 1 -- start start 2023-09-10T01:14:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 1 Processor -- start 2023-09-10T01:14:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe84f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe84ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.245 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe8ad380 con 0x559bbe8461a0 2023-09-10T01:14:21.245 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe8ad4c0 con 0x559bbe840300 2023-09-10T01:14:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bbe8ad600 con 0x559bbe840cc0 2023-09-10T01:14:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4e816cb80 10 client.0 _ll_register_callbacks cb 0x7fd4bc00f870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4d1d9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4d2d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe84ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4d1d9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58100/0 (socket says 172.21.15.3:58100) 2023-09-10T01:14:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/1530751912 learned_addr learned my addr 172.21.15.3:0/1530751912 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.225+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe84f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 msgr2=0x559bbe84f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe84f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/1530751912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 msgr2=0x559bbe84ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe84ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4d1d9a700 1 -- 172.21.15.3:0/1530751912 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559bbe5a7c00 con 0x559bbe840cc0 2023-09-10T01:14:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe84f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4d1d9a700 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84f9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc027440 tx=0x7fd4bc01e2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b1d0f4fb53ec7c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/1530751912 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc027df0 con 0x559bbe840cc0 2023-09-10T01:14:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4c801e040 con 0x559bbe840cc0 2023-09-10T01:14:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/1530751912 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd4bc004000 con 0x559bbe840cc0 2023-09-10T01:14:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.226+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/1530751912 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4bc0041c0 con 0x559bbe840cc0 2023-09-10T01:14:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 10 client.28015 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559bbe5bed80 con 0x559bbe840cc0 2023-09-10T01:14:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/1530751912 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4c801e040 con 0x559bbe840cc0 2023-09-10T01:14:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/1530751912 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fd4bc003680 con 0x559bbe840cc0 2023-09-10T01:14:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 10 client.28015 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 10 client.28015 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 20 client.28015 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe8aedb0 0x559bbe8b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 4 client.28015 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:14:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559bbe78b8c0 con 0x559bbe8aedb0 2023-09-10T01:14:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.227+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe8aedb0 0x559bbe8b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.228+0000 7fd4d259b700 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe8aedb0 0x559bbe8b11d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5704111c3befe798 server_cookie=7e38f0709f811d62 in_seq=0 out_seq=0 2023-09-10T01:14:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.228+0000 7fd4b5ffb700 10 client.28015 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4b5ffb700 1 -- 172.21.15.3:0/1530751912 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559bbe78b8c0 con 0x559bbe8aedb0 2023-09-10T01:14:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4b5ffb700 10 client.28015 handle_command_reply: tid=0 2023-09-10T01:14:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 client.28015 shutdown 2023-09-10T01:14:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 msgr2=0x559bbe84f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84f9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc027440 tx=0x7fd4bc01e2f0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 shutdown_connections 2023-09-10T01:14:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559bbe8aedb0 0x559bbe8b11d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559bbe8461a0 0x559bbe84ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559bbe840300 0x559bbe84f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4b5ffb700 0 client.28015 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 --2- 172.21.15.3:0/1530751912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559bbe840cc0 0x559bbe84f9c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 >> 172.21.15.3:0/1530751912 conn(0x559bbe7b5030 msgr2=0x559bbe845270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 shutdown_connections 2023-09-10T01:14:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.277+0000 7fd4e816cb80 1 -- 172.21.15.3:0/1530751912 wait complete. 2023-09-10T01:14:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.278+0000 7fd4e816cb80 20 client.28015 trim_cache size 0 max 16384 2023-09-10T01:14:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:14:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:14:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:14:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:14:21.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.278+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4b80a07b0 msgr2=0x7fd4b80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.278+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4b80a07b0 0x7fd4b80a2c70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0740f0 tx=0x7fd4bc011830 comp rx=0 tx=0).stop 2023-09-10T01:14:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.278+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4cc106680 msgr2=0x7fd4cc1a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.278+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4cc106680 0x7fd4cc1a4860 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400efc0 tx=0x7fd4c400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 shutdown_connections 2023-09-10T01:14:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd4b80a07b0 0x7fd4b80a2c70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4cc106680 0x7fd4cc1a4860 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4cc1005c0 0x7fd4cc072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 --2- 172.21.15.3:0/1515398760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4cc100f80 0x7fd4cc072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:21.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 >> 172.21.15.3:0/1515398760 conn(0x7fd4cc0fc0b0 msgr2=0x7fd4cc0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 shutdown_connections 2023-09-10T01:14:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:21.279+0000 7fd4d47ff700 1 -- 172.21.15.3:0/1515398760 wait complete. 2023-09-10T01:14:21.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 956657 8.28 MB/sec execute 483 sec latency 1099.781 ms 2023-09-10T01:14:22.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:22 smithi092 ceph-mon[134067]: pgmap v347: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2023-09-10T01:14:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:22 smithi003 ceph-mon[97654]: pgmap v347: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2023-09-10T01:14:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:22 smithi136 ceph-mon[133708]: pgmap v347: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2023-09-10T01:14:22.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 956657 8.27 MB/sec execute 484 sec latency 2099.908 ms 2023-09-10T01:14:23.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 957887 8.26 MB/sec execute 485 sec latency 2403.446 ms 2023-09-10T01:14:24.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:24 smithi092 ceph-mon[134067]: pgmap v348: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2023-09-10T01:14:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:24 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:14:24.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:24 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:14:24.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:24 smithi003 ceph-mon[97654]: pgmap v348: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2023-09-10T01:14:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:24 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:14:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:24 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:14:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:24 smithi136 ceph-mon[133708]: pgmap v348: 129 pgs: 129 active+clean; 289 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 1.5 MiB/s wr, 83 op/s 2023-09-10T01:14:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:24 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:14:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:24 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:14:24.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 959438 8.27 MB/sec execute 486 sec latency 44.789 ms 2023-09-10T01:14:25.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:25 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:14:25.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:25 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:14:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:25 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:14:25.801 INFO:tasks.workunit.client.0.smithi003.stdout: 1 959952 8.25 MB/sec execute 487 sec latency 222.885 ms 2023-09-10T01:14:26.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:26 smithi092 ceph-mon[134067]: pgmap v349: 129 pgs: 129 active+clean; 288 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2023-09-10T01:14:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:26 smithi003 ceph-mon[97654]: pgmap v349: 129 pgs: 129 active+clean; 288 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2023-09-10T01:14:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:26 smithi136 ceph-mon[133708]: pgmap v349: 129 pgs: 129 active+clean; 288 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2023-09-10T01:14:26.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 959952 8.24 MB/sec execute 488 sec latency 1223.032 ms 2023-09-10T01:14:27.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 959952 8.22 MB/sec execute 489 sec latency 2223.186 ms 2023-09-10T01:14:28.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:28 smithi092 ceph-mon[134067]: pgmap v350: 129 pgs: 129 active+clean; 288 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.9 MiB/s wr, 135 op/s 2023-09-10T01:14:28.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:28 smithi003 ceph-mon[97654]: pgmap v350: 129 pgs: 129 active+clean; 288 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.9 MiB/s wr, 135 op/s 2023-09-10T01:14:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:28 smithi136 ceph-mon[133708]: pgmap v350: 129 pgs: 129 active+clean; 288 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 1.9 MiB/s wr, 135 op/s 2023-09-10T01:14:28.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 960076 8.20 MB/sec execute 490 sec latency 2525.109 ms 2023-09-10T01:14:29.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 960076 8.19 MB/sec execute 491 sec latency 1553.515 ms 2023-09-10T01:14:30.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:30 smithi092 ceph-mon[134067]: pgmap v351: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 2.6 MiB/s wr, 199 op/s 2023-09-10T01:14:30.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:30 smithi003 ceph-mon[97654]: pgmap v351: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 2.6 MiB/s wr, 199 op/s 2023-09-10T01:14:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:30 smithi136 ceph-mon[133708]: pgmap v351: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 2.6 MiB/s wr, 199 op/s 2023-09-10T01:14:30.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 960076 8.17 MB/sec execute 492 sec latency 2553.626 ms 2023-09-10T01:14:31.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 960076 8.15 MB/sec execute 493 sec latency 3553.745 ms 2023-09-10T01:14:32.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:32 smithi092 ceph-mon[134067]: pgmap v352: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2023-09-10T01:14:32.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:32 smithi003 ceph-mon[97654]: pgmap v352: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2023-09-10T01:14:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:32 smithi136 ceph-mon[133708]: pgmap v352: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2023-09-10T01:14:32.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 960076 8.14 MB/sec execute 494 sec latency 4553.902 ms 2023-09-10T01:14:33.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 961200 8.13 MB/sec execute 495 sec latency 4856.263 ms 2023-09-10T01:14:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 -- 172.21.15.3:0/2486145911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa404100f80 msgr2=0x7fa40410d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 --2- 172.21.15.3:0/2486145911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa404100f80 0x7fa40410d440 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0099c0 tx=0x7fa3fc009cd0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 -- 172.21.15.3:0/2486145911 shutdown_connections 2023-09-10T01:14:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 --2- 172.21.15.3:0/2486145911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa404100f80 0x7fa40410d440 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 --2- 172.21.15.3:0/2486145911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404106680 0x7fa404106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 --2- 172.21.15.3:0/2486145911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4041005c0 0x7fa404100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 -- 172.21.15.3:0/2486145911 >> 172.21.15.3:0/2486145911 conn(0x7fa4040fc0b0 msgr2=0x7fa4040fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 -- 172.21.15.3:0/2486145911 shutdown_connections 2023-09-10T01:14:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.440+0000 7fa40bdff700 1 -- 172.21.15.3:0/2486145911 wait complete. 2023-09-10T01:14:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 Processor -- start 2023-09-10T01:14:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 -- start start 2023-09-10T01:14:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4041005c0 0x7fa40410c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404100f80 0x7fa40410c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa404106680 0x7fa4041a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041a1c30 con 0x7fa404100f80 2023-09-10T01:14:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041a1da0 con 0x7fa4041005c0 2023-09-10T01:14:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.442+0000 7fa40bdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041a1f60 con 0x7fa404106680 2023-09-10T01:14:34.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404100f80 0x7fa40410c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa409b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4041005c0 0x7fa40410c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404100f80 0x7fa40410c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59032/0 (socket says 172.21.15.3:59032) 2023-09-10T01:14:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40a39c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa404106680 0x7fa4041a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 -- 172.21.15.3:0/1703729480 learned_addr learned my addr 172.21.15.3:0/1703729480 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 -- 172.21.15.3:0/1703729480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa404106680 msgr2=0x7fa4041a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa404106680 0x7fa4041a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 -- 172.21.15.3:0/1703729480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4041005c0 msgr2=0x7fa40410c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4041005c0 0x7fa40410c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 -- 172.21.15.3:0/1703729480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0095f0 con 0x7fa404100f80 2023-09-10T01:14:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.443+0000 7fa40939a700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404100f80 0x7fa40410c640 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9a0 tx=0x7fa3f400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.444+0000 7fa3faffd700 1 -- 172.21.15.3:0/1703729480 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4009790 con 0x7fa404100f80 2023-09-10T01:14:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.444+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4041a2240 con 0x7fa404100f80 2023-09-10T01:14:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.444+0000 7fa3faffd700 1 -- 172.21.15.3:0/1703729480 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3f4010430 con 0x7fa404100f80 2023-09-10T01:14:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.444+0000 7fa3faffd700 1 -- 172.21.15.3:0/1703729480 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4018680 con 0x7fa404100f80 2023-09-10T01:14:34.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.444+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4041a9ab0 con 0x7fa404100f80 2023-09-10T01:14:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.447+0000 7fa3faffd700 1 -- 172.21.15.3:0/1703729480 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa3f40105a0 con 0x7fa404100f80 2023-09-10T01:14:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.448+0000 7fa41f6a6b80 1 Processor -- start 2023-09-10T01:14:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa41f6a6b80 1 -- start start 2023-09-10T01:14:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa3faffd700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa3f00a0830 0x7fa3f00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42750580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4274a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274acc0 0x559e42756f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e426a36d0 con 0x559e4274a300 2023-09-10T01:14:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e426a0ff0 con 0x559e4274acc0 2023-09-10T01:14:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4269c8e0 con 0x559e427501a0 2023-09-10T01:14:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa3faffd700 1 -- 172.21.15.3:0/1703729480 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa3f4014070 con 0x7fa404100f80 2023-09-10T01:14:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa40a39c700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa3f00a0830 0x7fa3f00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa40939a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4274a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa409b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42750580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa40a39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274acc0 0x559e42756f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa409b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42750580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45654/0 (socket says 172.21.15.3:45654) 2023-09-10T01:14:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.449+0000 7fa409b9b700 1 -- 172.21.15.3:0/38790272 learned_addr learned my addr 172.21.15.3:0/38790272 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa409b9b700 1 -- 172.21.15.3:0/38790272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274acc0 msgr2=0x559e42756f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa409b9b700 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274acc0 0x559e42756f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa409b9b700 1 -- 172.21.15.3:0/38790272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 msgr2=0x559e4274a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa409b9b700 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4274a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa409b9b700 1 -- 172.21.15.3:0/38790272 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e424c8d80 con 0x559e427501a0 2023-09-10T01:14:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa40939a700 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4274a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa409b9b700 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42750580 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa400011650 tx=0x7fa400009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d111724e5ca683a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa40a39c700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa3f00a0830 0x7fa3f00a2cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0058e0 tx=0x7fa3fc003680 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa3effff700 1 -- 172.21.15.3:0/38790272 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40000e040 con 0x559e427501a0 2023-09-10T01:14:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa3effff700 1 -- 172.21.15.3:0/38790272 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40000f040 con 0x559e427501a0 2023-09-10T01:14:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.450+0000 7fa3effff700 1 -- 172.21.15.3:0/38790272 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400015960 con 0x559e427501a0 2023-09-10T01:14:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/38790272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 msgr2=0x559e42750580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42750580 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa400011650 tx=0x7fa400009920 comp rx=0 tx=0).stop 2023-09-10T01:14:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/38790272 shutdown_connections 2023-09-10T01:14:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42750580 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274acc0 0x559e42756f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/38790272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4274a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/38790272 >> 172.21.15.3:0/38790272 conn(0x559e426bf030 msgr2=0x559e42745560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/38790272 shutdown_connections 2023-09-10T01:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.451+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/38790272 wait complete. 2023-09-10T01:14:34.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 -- start start 2023-09-10T01:14:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 Processor -- start 2023-09-10T01:14:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274a300 0x559e42756d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274acc0 0x559e42754030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42754570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4263ab10 con 0x559e4274acc0 2023-09-10T01:14:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4263a810 con 0x559e4274a300 2023-09-10T01:14:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.452+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4251e8c0 con 0x559e427501a0 2023-09-10T01:14:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa41f6a6b80 10 client.0 _ll_register_callbacks cb 0x559e42584a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274acc0 0x559e42754030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274acc0 0x559e42754030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59050/0 (socket says 172.21.15.3:59050) 2023-09-10T01:14:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 -- 172.21.15.3:0/417828101 learned_addr learned my addr 172.21.15.3:0/417828101 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40a39c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42754570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa409b9b700 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274a300 0x559e42756d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 -- 172.21.15.3:0/417828101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 msgr2=0x559e42754570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42754570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 -- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274a300 msgr2=0x559e42756d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274a300 0x559e42756d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 -- 172.21.15.3:0/417828101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa400011330 con 0x559e4274acc0 2023-09-10T01:14:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa409b9b700 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274a300 0x559e42756d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa40939a700 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274acc0 0x559e42754030 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4095ab0 tx=0x7fa3f405c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140021695922fb2a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.453+0000 7fa3ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/417828101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f409d3e0 con 0x559e4274acc0 2023-09-10T01:14:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e425f4490 con 0x559e4274acc0 2023-09-10T01:14:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/417828101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa3f40a1620 con 0x559e4274acc0 2023-09-10T01:14:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/417828101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f409d120 con 0x559e4274acc0 2023-09-10T01:14:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 10 client.18051 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559e424b1c00 con 0x559e4274acc0 2023-09-10T01:14:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/417828101 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e425f4490 con 0x559e4274acc0 2023-09-10T01:14:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/417828101 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa3f4099810 con 0x559e4274acc0 2023-09-10T01:14:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 10 client.18051 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 10 client.18051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 20 client.18051 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e42764340 0x559e42766760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 4 client.18051 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.454+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e426b1840 con 0x559e42764340 2023-09-10T01:14:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.455+0000 7fa409b9b700 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e42764340 0x559e42766760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.455+0000 7fa409b9b700 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e42764340 0x559e42766760 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1022290e25145bc server_cookie=11c1c018474e6473 in_seq=0 out_seq=0 2023-09-10T01:14:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.455+0000 7fa3ed7fa700 10 client.18051 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.456+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/417828101 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559e426b1840 con 0x559e42764340 2023-09-10T01:14:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.456+0000 7fa3ed7fa700 10 client.18051 handle_command_reply: tid=0 2023-09-10T01:14:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.456+0000 7fa41f6a6b80 1 client.18051 shutdown 2023-09-10T01:14:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.456+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274acc0 msgr2=0x559e42754030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.456+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274acc0 0x559e42754030 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4095ab0 tx=0x7fa3f405c070 comp rx=0 tx=0).stop 2023-09-10T01:14:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.456+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 shutdown_connections 2023-09-10T01:14:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.456+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e42764340 0x559e42766760 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e427501a0 0x559e42754570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4274a300 0x559e42756d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/417828101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274acc0 0x559e42754030 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa3ed7fa700 0 client.18051 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 >> 172.21.15.3:0/417828101 conn(0x559e426bf030 msgr2=0x559e427447a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 shutdown_connections 2023-09-10T01:14:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/417828101 wait complete. 2023-09-10T01:14:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.457+0000 7fa41f6a6b80 20 client.18051 trim_cache size 0 max 16384 2023-09-10T01:14:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 Processor -- start 2023-09-10T01:14:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 -- start start 2023-09-10T01:14:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4275b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e42756840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e42744f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4275ba70 con 0x559e4274a300 2023-09-10T01:14:34.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e427619a0 con 0x559e427501a0 2023-09-10T01:14:34.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e42754f20 con 0x559e4274acc0 2023-09-10T01:14:34.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.473+0000 7fa40939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e42756840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40a39c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e42744f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e42756840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45680/0 (socket says 172.21.15.3:45680) 2023-09-10T01:14:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 -- 172.21.15.3:0/2791233486 learned_addr learned my addr 172.21.15.3:0/2791233486 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa409b9b700 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4275b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 -- 172.21.15.3:0/2791233486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 msgr2=0x559e42744f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e42744f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 -- 172.21.15.3:0/2791233486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 msgr2=0x559e4275b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4275b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 -- 172.21.15.3:0/2791233486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e424ef1f0 con 0x559e4274acc0 2023-09-10T01:14:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa40939a700 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e42756840 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4095fd0 tx=0x7fa3f4097b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616b784d27a1d96e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa3f8bff700 1 -- 172.21.15.3:0/2791233486 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40a1620 con 0x559e4274acc0 2023-09-10T01:14:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa3f8bff700 1 -- 172.21.15.3:0/2791233486 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3f409d3e0 con 0x559e4274acc0 2023-09-10T01:14:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.474+0000 7fa3f8bff700 1 -- 172.21.15.3:0/2791233486 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40cc150 con 0x559e4274acc0 2023-09-10T01:14:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2791233486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 msgr2=0x559e42756840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e42756840 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4095fd0 tx=0x7fa3f4097b60 comp rx=0 tx=0).stop 2023-09-10T01:14:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2791233486 shutdown_connections 2023-09-10T01:14:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e42744f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e4275b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2791233486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e42756840 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2791233486 >> 172.21.15.3:0/2791233486 conn(0x559e426bf030 msgr2=0x559e42749b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2791233486 shutdown_connections 2023-09-10T01:14:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.475+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2791233486 wait complete. 2023-09-10T01:14:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.476+0000 7fa41f6a6b80 1 -- start start 2023-09-10T01:14:34.490 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.476+0000 7fa41f6a6b80 1 Processor -- start 2023-09-10T01:14:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e427ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e427ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa41f6a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e427b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e427b07a0 con 0x559e4274a300 2023-09-10T01:14:34.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e427b0910 con 0x559e427501a0 2023-09-10T01:14:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa41f6a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e427b0ad0 con 0x559e4274acc0 2023-09-10T01:14:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa41f6a6b80 10 client.0 _ll_register_callbacks cb 0x7fa3f40990f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e427ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e427ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45692/0 (socket says 172.21.15.3:45692) 2023-09-10T01:14:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 -- 172.21.15.3:0/2049864508 learned_addr learned my addr 172.21.15.3:0/2049864508 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40a39c700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e427b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa409b9b700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e427ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 -- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 msgr2=0x559e427b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e427b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 -- 172.21.15.3:0/2049864508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 msgr2=0x559e427ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e427ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa40939a700 1 -- 172.21.15.3:0/2049864508 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e424b1c00 con 0x559e4274acc0 2023-09-10T01:14:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.477+0000 7fa409b9b700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e427ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa40939a700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e427ac870 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40992b0 tx=0x7fa3f4099380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b99b6af4aeab1566 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa3ed7fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/2049864508 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4010940 con 0x559e4274acc0 2023-09-10T01:14:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa40000f040 con 0x559e4274acc0 2023-09-10T01:14:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/2049864508 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa3f4052050 con 0x559e4274acc0 2023-09-10T01:14:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/2049864508 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f40a1190 con 0x559e4274acc0 2023-09-10T01:14:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa41f6a6b80 10 client.28033 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559e424c7250 con 0x559e4274acc0 2023-09-10T01:14:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/2049864508 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40000f040 con 0x559e4274acc0 2023-09-10T01:14:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/2049864508 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa3f4098510 con 0x559e4274acc0 2023-09-10T01:14:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa41f6a6b80 10 client.28033 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa41f6a6b80 10 client.28033 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.478+0000 7fa41f6a6b80 20 client.28033 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.479+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e427b9050 0x559e427bb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.479+0000 7fa41f6a6b80 4 client.28033 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:14:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.479+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559e426958c0 con 0x559e427b9050 2023-09-10T01:14:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.479+0000 7fa409b9b700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e427b9050 0x559e427bb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.479+0000 7fa409b9b700 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e427b9050 0x559e427bb470 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ee2215ac46fa70c server_cookie=39966c3d1edf80a6 in_seq=0 out_seq=0 2023-09-10T01:14:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.479+0000 7fa3ed7fa700 10 client.28033 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.531+0000 7fa3ed7fa700 1 -- 172.21.15.3:0/2049864508 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559e426958c0 con 0x559e427b9050 2023-09-10T01:14:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.531+0000 7fa3ed7fa700 10 client.28033 handle_command_reply: tid=0 2023-09-10T01:14:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.531+0000 7fa41f6a6b80 1 client.28033 shutdown 2023-09-10T01:14:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 msgr2=0x559e427ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e427ac870 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40992b0 tx=0x7fa3f4099380 comp rx=0 tx=0).stop 2023-09-10T01:14:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 shutdown_connections 2023-09-10T01:14:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559e427b9050 0x559e427bb470 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e427501a0 0x559e427b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e4274a300 0x559e427ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 --2- 172.21.15.3:0/2049864508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559e4274acc0 0x559e427ac870 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa3ed7fa700 0 client.28033 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 >> 172.21.15.3:0/2049864508 conn(0x559e426bf030 msgr2=0x559e42749b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 shutdown_connections 2023-09-10T01:14:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 1 -- 172.21.15.3:0/2049864508 wait complete. 2023-09-10T01:14:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.532+0000 7fa41f6a6b80 20 client.28033 trim_cache size 0 max 16384 2023-09-10T01:14:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:14:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:14:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:14:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:14:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa3f00a0830 msgr2=0x7fa3f00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa3f00a0830 0x7fa3f00a2cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0058e0 tx=0x7fa3fc003680 comp rx=0 tx=0).stop 2023-09-10T01:14:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404100f80 msgr2=0x7fa40410c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404100f80 0x7fa40410c640 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9a0 tx=0x7fa3f400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 shutdown_connections 2023-09-10T01:14:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa3f00a0830 0x7fa3f00a2cf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.539 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa404106680 0x7fa4041a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4041005c0 0x7fa40410c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 --2- 172.21.15.3:0/1703729480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa404100f80 0x7fa40410c640 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 >> 172.21.15.3:0/1703729480 conn(0x7fa4040fc0b0 msgr2=0x7fa404102160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 shutdown_connections 2023-09-10T01:14:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:34.533+0000 7fa40bdff700 1 -- 172.21.15.3:0/1703729480 wait complete. 2023-09-10T01:14:34.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:34 smithi003 ceph-mon[97654]: pgmap v353: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2023-09-10T01:14:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:34 smithi136 ceph-mon[133708]: pgmap v353: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2023-09-10T01:14:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:34 smithi092 ceph-mon[134067]: pgmap v353: 129 pgs: 129 active+clean; 290 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 1.7 MiB/s wr, 143 op/s 2023-09-10T01:14:34.802 INFO:tasks.workunit.client.0.smithi003.stdout: 1 963068 8.14 MB/sec execute 496 sec latency 24.018 ms 2023-09-10T01:14:35.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 963626 8.13 MB/sec execute 497 sec latency 336.893 ms 2023-09-10T01:14:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:36 smithi003 ceph-mon[97654]: pgmap v354: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.8 MiB/s wr, 153 op/s 2023-09-10T01:14:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:36 smithi136 ceph-mon[133708]: pgmap v354: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.8 MiB/s wr, 153 op/s 2023-09-10T01:14:36.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:36 smithi092 ceph-mon[134067]: pgmap v354: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.8 MiB/s wr, 153 op/s 2023-09-10T01:14:36.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 963626 8.11 MB/sec execute 498 sec latency 1336.971 ms 2023-09-10T01:14:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:37 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:14:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:37 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:37 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:37 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:37 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:37 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:14:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:37 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:37 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:37 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:37 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:37 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:14:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:37 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:37 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:37 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:37 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:37.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 963626 8.09 MB/sec execute 499 sec latency 2337.032 ms 2023-09-10T01:14:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:38 smithi003 ceph-mon[97654]: pgmap v355: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 876 KiB/s wr, 73 op/s 2023-09-10T01:14:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:38 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:14:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:38 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:14:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:38 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:38 smithi136 ceph-mon[133708]: pgmap v355: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 876 KiB/s wr, 73 op/s 2023-09-10T01:14:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:38 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:14:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:38 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:14:38.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:38 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:38.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:38 smithi092 ceph-mon[134067]: pgmap v355: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 876 KiB/s wr, 73 op/s 2023-09-10T01:14:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:38 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:14:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:38 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:14:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:38 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:14:38.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 964696 8.09 MB/sec execute 500 sec latency 2632.650 ms 2023-09-10T01:14:39.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 966595 8.10 MB/sec execute 501 sec latency 19.416 ms 2023-09-10T01:14:40.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:40 smithi003 ceph-mon[97654]: pgmap v356: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.3 MiB/s wr, 127 op/s 2023-09-10T01:14:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:40 smithi136 ceph-mon[133708]: pgmap v356: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.3 MiB/s wr, 127 op/s 2023-09-10T01:14:40.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:40 smithi092 ceph-mon[134067]: pgmap v356: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.3 MiB/s wr, 127 op/s 2023-09-10T01:14:40.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 967532 8.09 MB/sec execute 502 sec latency 19.757 ms 2023-09-10T01:14:41.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 969399 8.09 MB/sec execute 503 sec latency 17.989 ms 2023-09-10T01:14:42.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:42 smithi003 ceph-mon[97654]: pgmap v357: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 536 KiB/s wr, 63 op/s 2023-09-10T01:14:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:42 smithi136 ceph-mon[133708]: pgmap v357: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 536 KiB/s wr, 63 op/s 2023-09-10T01:14:42.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:42 smithi092 ceph-mon[134067]: pgmap v357: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 536 KiB/s wr, 63 op/s 2023-09-10T01:14:42.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970591 8.09 MB/sec execute 504 sec latency 61.211 ms 2023-09-10T01:14:43.803 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970647 8.07 MB/sec execute 505 sec latency 656.301 ms 2023-09-10T01:14:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:44 smithi003 ceph-mon[97654]: pgmap v358: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 536 KiB/s wr, 63 op/s 2023-09-10T01:14:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:44 smithi136 ceph-mon[133708]: pgmap v358: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 536 KiB/s wr, 63 op/s 2023-09-10T01:14:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:44 smithi092 ceph-mon[134067]: pgmap v358: 129 pgs: 129 active+clean; 291 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 536 KiB/s wr, 63 op/s 2023-09-10T01:14:44.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970647 8.06 MB/sec execute 506 sec latency 1656.451 ms 2023-09-10T01:14:45.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970647 8.04 MB/sec execute 507 sec latency 2656.585 ms 2023-09-10T01:14:46.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:46 smithi003 ceph-mon[97654]: pgmap v359: 129 pgs: 129 active+clean; 297 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 1.1 MiB/s wr, 131 op/s 2023-09-10T01:14:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:46 smithi136 ceph-mon[133708]: pgmap v359: 129 pgs: 129 active+clean; 297 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 1.1 MiB/s wr, 131 op/s 2023-09-10T01:14:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:46 smithi092 ceph-mon[134067]: pgmap v359: 129 pgs: 129 active+clean; 297 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 1.1 MiB/s wr, 131 op/s 2023-09-10T01:14:46.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970647 8.03 MB/sec execute 508 sec latency 3656.660 ms 2023-09-10T01:14:47.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:14:47.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 -- 172.21.15.3:0/3896734907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 msgr2=0x7f79d4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 --2- 172.21.15.3:0/3896734907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d4100ac0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f79c8009990 tx=0x7f79c8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 -- 172.21.15.3:0/3896734907 shutdown_connections 2023-09-10T01:14:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 --2- 172.21.15.3:0/3896734907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d4106650 0x7f79d4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 --2- 172.21.15.3:0/3896734907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79d4101000 0x7f79d410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 --2- 172.21.15.3:0/3896734907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d4100ac0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 -- 172.21.15.3:0/3896734907 >> 172.21.15.3:0/3896734907 conn(0x7f79d40fc170 msgr2=0x7f79d40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 -- 172.21.15.3:0/3896734907 shutdown_connections 2023-09-10T01:14:47.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.388+0000 7f79d95ff700 1 -- 172.21.15.3:0/3896734907 wait complete. 2023-09-10T01:14:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 Processor -- start 2023-09-10T01:14:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 -- start start 2023-09-10T01:14:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d4101000 0x7f79d419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79d4106650 0x7f79d4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d4198db0 con 0x7f79d4100640 2023-09-10T01:14:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d4198f20 con 0x7f79d4106650 2023-09-10T01:14:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.389+0000 7f79d95ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d41990e0 con 0x7f79d4101000 2023-09-10T01:14:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d27fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d4101000 0x7f79d419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36276/0 (socket says 172.21.15.3:36276) 2023-09-10T01:14:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 -- 172.21.15.3:0/1241864036 learned_addr learned my addr 172.21.15.3:0/1241864036 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d37fe700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79d4106650 0x7f79d4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 -- 172.21.15.3:0/1241864036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d4101000 msgr2=0x7f79d419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d4101000 0x7f79d419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 -- 172.21.15.3:0/1241864036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79d4106650 msgr2=0x7f79d4198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79d4106650 0x7f79d4198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 -- 172.21.15.3:0/1241864036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80095f0 con 0x7f79d4100640 2023-09-10T01:14:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.390+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d419e6d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e9a0 tx=0x7f79c400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.391+0000 7f79bbfff700 1 -- 172.21.15.3:0/1241864036 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4009790 con 0x7f79d4100640 2023-09-10T01:14:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.391+0000 7f79bbfff700 1 -- 172.21.15.3:0/1241864036 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79c4010430 con 0x7f79d4100640 2023-09-10T01:14:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.391+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d41993c0 con 0x7f79d4100640 2023-09-10T01:14:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.391+0000 7f79bbfff700 1 -- 172.21.15.3:0/1241864036 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4018680 con 0x7f79d4100640 2023-09-10T01:14:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.391+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d41a56c0 con 0x7f79d4100640 2023-09-10T01:14:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.394+0000 7f79bbfff700 1 -- 172.21.15.3:0/1241864036 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f79c40105a0 con 0x7f79d4100640 2023-09-10T01:14:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.395+0000 7f79ed022b80 1 Processor -- start 2023-09-10T01:14:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.395+0000 7f79ed022b80 1 -- start start 2023-09-10T01:14:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7dd710 0x5607fd7ddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7de030 0x5607fd7e7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7ea230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79bbfff700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79bc0a07b0 0x7f79bc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79ed022b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd736730 con 0x5607fd7e7e40 2023-09-10T01:14:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79ed022b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd734050 con 0x5607fd7de030 2023-09-10T01:14:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79ed022b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd72f940 con 0x5607fd7dd710 2023-09-10T01:14:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79bbfff700 1 -- 172.21.15.3:0/1241864036 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f79c4014070 con 0x7f79d4100640 2023-09-10T01:14:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7de030 0x5607fd7e7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7de030 0x5607fd7e7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34030/0 (socket says 172.21.15.3:34030) 2023-09-10T01:14:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 -- 172.21.15.3:0/4023113573 learned_addr learned my addr 172.21.15.3:0/4023113573 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d27fc700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79bc0a07b0 0x7f79bc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d37fe700 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7dd710 0x5607fd7ddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d37fe700 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 -- 172.21.15.3:0/4023113573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7dd710 msgr2=0x5607fd7ddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7dd710 0x5607fd7ddaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 -- 172.21.15.3:0/4023113573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 msgr2=0x5607fd7ea230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7ea230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d2ffd700 1 -- 172.21.15.3:0/4023113573 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607fd55bde0 con 0x5607fd7de030 2023-09-10T01:14:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.396+0000 7f79d37fe700 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7ea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.397+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7de030 0x5607fd7e7900 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f79c4095fd0 tx=0x7f79c40952a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da07df2c0d7d76da server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.397+0000 7f79b97fa700 1 -- 172.21.15.3:0/4023113573 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4099560 con 0x5607fd7de030 2023-09-10T01:14:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.397+0000 7f79b97fa700 1 -- 172.21.15.3:0/4023113573 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79c409d390 con 0x5607fd7de030 2023-09-10T01:14:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.397+0000 7f79b97fa700 1 -- 172.21.15.3:0/4023113573 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c40cc170 con 0x5607fd7de030 2023-09-10T01:14:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.397+0000 7f79d27fc700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79bc0a07b0 0x7f79bc0a2c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f79d4199bb0 tx=0x7f79c8005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 -- 172.21.15.3:0/4023113573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7de030 msgr2=0x5607fd7e7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7de030 0x5607fd7e7900 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f79c4095fd0 tx=0x7f79c40952a0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 -- 172.21.15.3:0/4023113573 shutdown_connections 2023-09-10T01:14:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7dd710 0x5607fd7ddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7ea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 --2- 172.21.15.3:0/4023113573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7de030 0x5607fd7e7900 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 -- 172.21.15.3:0/4023113573 >> 172.21.15.3:0/4023113573 conn(0x5607fd752090 msgr2=0x5607fd7d8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 -- 172.21.15.3:0/4023113573 shutdown_connections 2023-09-10T01:14:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 -- 172.21.15.3:0/4023113573 wait complete. 2023-09-10T01:14:47.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.398+0000 7f79ed022b80 1 -- start start 2023-09-10T01:14:47.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 1 Processor -- start 2023-09-10T01:14:47.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7e5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd6cdb70 con 0x5607fd7de030 2023-09-10T01:14:47.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd6cd870 con 0x5607fd7dd710 2023-09-10T01:14:47.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd5b1920 con 0x5607fd7e7e40 2023-09-10T01:14:47.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.399+0000 7f79ed022b80 10 client.0 _ll_register_callbacks cb 0x5607fd617a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:47.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34036/0 (socket says 172.21.15.3:34036) 2023-09-10T01:14:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 -- 172.21.15.3:0/3144326615 learned_addr learned my addr 172.21.15.3:0/3144326615 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d37fe700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7e5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d27fc700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 -- 172.21.15.3:0/3144326615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 msgr2=0x5607fd7e5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7e5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 -- 172.21.15.3:0/3144326615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 msgr2=0x5607fd7e4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7e4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 -- 172.21.15.3:0/3144326615 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d41a56c0 con 0x5607fd7dd710 2023-09-10T01:14:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d27fc700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7e4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:47.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e78c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f79c40c9e00 tx=0x7f79c4099dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11197e22fceab24f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79b6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:14:47.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3144326615 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4099560 con 0x5607fd7dd710 2023-09-10T01:14:47.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3144326615 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f79c40840f0 con 0x5607fd7dd710 2023-09-10T01:14:47.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.400+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607fd6874f0 con 0x5607fd7dd710 2023-09-10T01:14:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3144326615 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79c40b50f0 con 0x5607fd7dd710 2023-09-10T01:14:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 10 client.28157 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5607fd544c60 con 0x5607fd7dd710 2023-09-10T01:14:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3144326615 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5607fd6874f0 con 0x5607fd7dd710 2023-09-10T01:14:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3144326615 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f79c40650a0 con 0x5607fd7dd710 2023-09-10T01:14:47.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 10 client.28157 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:47.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 10 client.28157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:47.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 20 client.28157 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:47.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd7f76d0 0x5607fd7f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 4 client.28157 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:47.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607fd7448a0 con 0x5607fd7f76d0 2023-09-10T01:14:47.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.401+0000 7f79d27fc700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd7f76d0 0x5607fd7f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.402+0000 7f79b6ff5700 10 client.28157 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.402+0000 7f79d27fc700 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd7f76d0 0x5607fd7f9af0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=208825c892b12a20 server_cookie=36686ce1a9e7da70 in_seq=0 out_seq=0 2023-09-10T01:14:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3144326615 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5607fd7448a0 con 0x5607fd7f76d0 2023-09-10T01:14:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79b6ff5700 10 client.28157 handle_command_reply: tid=0 2023-09-10T01:14:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 client.28157 shutdown 2023-09-10T01:14:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 msgr2=0x5607fd7e78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e78c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f79c40c9e00 tx=0x7f79c4099dd0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 shutdown_connections 2023-09-10T01:14:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd7f76d0 0x5607fd7f9af0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7e5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e78c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79b6ff5700 0 client.28157 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3144326615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7e4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 >> 172.21.15.3:0/3144326615 conn(0x5607fd752090 msgr2=0x5607fd7d8780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 shutdown_connections 2023-09-10T01:14:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.403+0000 7f79ed022b80 1 -- 172.21.15.3:0/3144326615 wait complete. 2023-09-10T01:14:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.404+0000 7f79ed022b80 20 client.28157 trim_cache size 0 max 16384 2023-09-10T01:14:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 Processor -- start 2023-09-10T01:14:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 -- start start 2023-09-10T01:14:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7de030 0x5607fd7e4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7e4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd7d0430 con 0x5607fd7e7e40 2023-09-10T01:14:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd7e30a0 con 0x5607fd7dd710 2023-09-10T01:14:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.421+0000 7f79ed022b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd7d79a0 con 0x5607fd7de030 2023-09-10T01:14:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7e4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36306/0 (socket says 172.21.15.3:36306) 2023-09-10T01:14:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d37fe700 1 -- 172.21.15.3:0/2032012027 learned_addr learned my addr 172.21.15.3:0/2032012027 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d27fc700 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7de030 0x5607fd7e4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d2ffd700 1 -- 172.21.15.3:0/2032012027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7de030 msgr2=0x5607fd7e4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7de030 0x5607fd7e4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d2ffd700 1 -- 172.21.15.3:0/2032012027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 msgr2=0x5607fd7e4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7e4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d2ffd700 1 -- 172.21.15.3:0/2032012027 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607fd582250 con 0x5607fd7dd710 2023-09-10T01:14:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d37fe700 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.422+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e7460 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f79c40978b0 tx=0x7f79c4097b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2be844be8c520ae7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79b9ffb700 1 -- 172.21.15.3:0/2032012027 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4098ec0 con 0x5607fd7dd710 2023-09-10T01:14:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79b9ffb700 1 -- 172.21.15.3:0/2032012027 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79c4099560 con 0x5607fd7dd710 2023-09-10T01:14:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79b9ffb700 1 -- 172.21.15.3:0/2032012027 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4099a20 con 0x5607fd7dd710 2023-09-10T01:14:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 -- 172.21.15.3:0/2032012027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 msgr2=0x5607fd7e7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e7460 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f79c40978b0 tx=0x7f79c4097b10 comp rx=0 tx=0).stop 2023-09-10T01:14:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 -- 172.21.15.3:0/2032012027 shutdown_connections 2023-09-10T01:14:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7e7e40 0x5607fd7e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7e7460 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 --2- 172.21.15.3:0/2032012027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7de030 0x5607fd7e4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 -- 172.21.15.3:0/2032012027 >> 172.21.15.3:0/2032012027 conn(0x5607fd752090 msgr2=0x5607fd7dfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 -- 172.21.15.3:0/2032012027 shutdown_connections 2023-09-10T01:14:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.423+0000 7f79ed022b80 1 -- 172.21.15.3:0/2032012027 wait complete. 2023-09-10T01:14:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 -- start start 2023-09-10T01:14:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 Processor -- start 2023-09-10T01:14:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7ee760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7eca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7ecf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd7ed4b0 con 0x5607fd7de030 2023-09-10T01:14:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd7ed620 con 0x5607fd7dd710 2023-09-10T01:14:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fd7ed7e0 con 0x5607fd7e7e40 2023-09-10T01:14:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.424+0000 7f79ed022b80 10 client.0 _ll_register_callbacks cb 0x7f79c40c9f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7eca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d27fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7eca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36314/0 (socket says 172.21.15.3:36314) 2023-09-10T01:14:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d27fc700 1 -- 172.21.15.3:0/3312812147 learned_addr learned my addr 172.21.15.3:0/3312812147 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d37fe700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7ecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d37fe700 1 -- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 msgr2=0x5607fd7ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d37fe700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d37fe700 1 -- 172.21.15.3:0/3312812147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 msgr2=0x5607fd7eca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d37fe700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7eca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d37fe700 1 -- 172.21.15.3:0/3312812147 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79cc007500 con 0x5607fd7e7e40 2023-09-10T01:14:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d27fc700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7eca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79d37fe700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7ecf70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007bc0 tx=0x7f79cc007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47251eac8f6f37f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.425+0000 7f79b6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3312812147 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00e040 con 0x5607fd7e7e40 2023-09-10T01:14:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79c409d390 con 0x5607fd7e7e40 2023-09-10T01:14:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3312812147 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f79cc003cd0 con 0x5607fd7e7e40 2023-09-10T01:14:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3312812147 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79cc003e90 con 0x5607fd7e7e40 2023-09-10T01:14:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 10 client.28051 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5607fd544c60 con 0x5607fd7e7e40 2023-09-10T01:14:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3312812147 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79c409d390 con 0x5607fd7e7e40 2023-09-10T01:14:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3312812147 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f79cc004150 con 0x5607fd7e7e40 2023-09-10T01:14:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 10 client.28051 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 10 client.28051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 20 client.28051 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd84c9f0 0x5607fd84ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 4 client.28051 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:14:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.426+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5607fd728920 con 0x5607fd84c9f0 2023-09-10T01:14:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd84c9f0 0x5607fd84ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79d2ffd700 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd84c9f0 0x5607fd84ee10 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f01af18e31694578 server_cookie=a70e34d48f26bb79 in_seq=0 out_seq=0 2023-09-10T01:14:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79b6ff5700 10 client.28051 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79b6ff5700 1 -- 172.21.15.3:0/3312812147 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5607fd728920 con 0x5607fd84c9f0 2023-09-10T01:14:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79b6ff5700 10 client.28051 handle_command_reply: tid=0 2023-09-10T01:14:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 client.28051 shutdown 2023-09-10T01:14:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 msgr2=0x5607fd7ecf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7ecf70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007bc0 tx=0x7f79cc007ed0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 shutdown_connections 2023-09-10T01:14:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5607fd84c9f0 0x5607fd84ee10 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5607fd7e7e40 0x5607fd7ecf70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79b6ff5700 0 client.28051 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5607fd7dd710 0x5607fd7ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.427+0000 7f79ed022b80 1 --2- 172.21.15.3:0/3312812147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607fd7de030 0x5607fd7eca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 >> 172.21.15.3:0/3312812147 conn(0x5607fd752090 msgr2=0x5607fd7dc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 shutdown_connections 2023-09-10T01:14:47.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79ed022b80 1 -- 172.21.15.3:0/3312812147 wait complete. 2023-09-10T01:14:47.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79ed022b80 20 client.28051 trim_cache size 0 max 16384 2023-09-10T01:14:47.449 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:14:47.450 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:14:47.450 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:14:47.450 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:14:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79bc0a07b0 msgr2=0x7f79bc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79bc0a07b0 0x7f79bc0a2c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f79d4199bb0 tx=0x7f79c8005fb0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 msgr2=0x7f79d419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d419e6d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f79c400e9a0 tx=0x7f79c400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 shutdown_connections 2023-09-10T01:14:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79bc0a07b0 0x7f79bc0a2c70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79d4106650 0x7f79d4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79d4100640 0x7f79d419e6d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 --2- 172.21.15.3:0/1241864036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d4101000 0x7f79d419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.428+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 >> 172.21.15.3:0/1241864036 conn(0x7f79d40fc170 msgr2=0x7f79d410bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.429+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 shutdown_connections 2023-09-10T01:14:47.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.429+0000 7f79d95ff700 1 -- 172.21.15.3:0/1241864036 wait complete. 2023-09-10T01:14:47.454 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:14:47.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:14:47.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.658+0000 7fdc55bff700 1 -- 172.21.15.3:0/3366471191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 msgr2=0x7fdc50111100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.658+0000 7fdc55bff700 1 --2- 172.21.15.3:0/3366471191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 0x7fdc50111100 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009990 tx=0x7fdc44009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.659+0000 7fdc55bff700 1 -- 172.21.15.3:0/3366471191 shutdown_connections 2023-09-10T01:14:47.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.659+0000 7fdc55bff700 1 --2- 172.21.15.3:0/3366471191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc500729a0 0x7fdc50072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.659+0000 7fdc55bff700 1 --2- 172.21.15.3:0/3366471191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc50111640 0x7fdc50113a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.659+0000 7fdc55bff700 1 --2- 172.21.15.3:0/3366471191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 0x7fdc50111100 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.659+0000 7fdc55bff700 1 -- 172.21.15.3:0/3366471191 >> 172.21.15.3:0/3366471191 conn(0x7fdc500fc140 msgr2=0x7fdc500fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.659+0000 7fdc55bff700 1 -- 172.21.15.3:0/3366471191 shutdown_connections 2023-09-10T01:14:47.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.659+0000 7fdc55bff700 1 -- 172.21.15.3:0/3366471191 wait complete. 2023-09-10T01:14:47.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 Processor -- start 2023-09-10T01:14:47.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 -- start start 2023-09-10T01:14:47.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc500729a0 0x7fdc5019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 0x7fdc5019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc50111640 0x7fdc501a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc501a1d90 con 0x7fdc500732c0 2023-09-10T01:14:47.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc501a1f00 con 0x7fdc500729a0 2023-09-10T01:14:47.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.660+0000 7fdc55bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc501a20c0 con 0x7fdc50111640 2023-09-10T01:14:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc50111640 0x7fdc501a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc500729a0 0x7fdc5019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc50111640 0x7fdc501a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54154/0 (socket says 172.21.15.3:54154) 2023-09-10T01:14:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 -- 172.21.15.3:0/4109061686 learned_addr learned my addr 172.21.15.3:0/4109061686 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4effd700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 0x7fdc5019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 -- 172.21.15.3:0/4109061686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc500729a0 msgr2=0x7fdc5019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc500729a0 0x7fdc5019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 -- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 msgr2=0x7fdc5019d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 0x7fdc5019d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 -- 172.21.15.3:0/4109061686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc440095f0 con 0x7fdc50111640 2023-09-10T01:14:47.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.661+0000 7fdc4ffff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc50111640 0x7fdc501a1750 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdc4800ef90 tx=0x7fdc4800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.662+0000 7fdc4cff9700 1 -- 172.21.15.3:0/4109061686 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48018070 con 0x7fdc50111640 2023-09-10T01:14:47.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.662+0000 7fdc4cff9700 1 -- 172.21.15.3:0/4109061686 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc4800f040 con 0x7fdc50111640 2023-09-10T01:14:47.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.662+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc501a2340 con 0x7fdc50111640 2023-09-10T01:14:47.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.662+0000 7fdc4cff9700 1 -- 172.21.15.3:0/4109061686 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4800cd60 con 0x7fdc50111640 2023-09-10T01:14:47.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.662+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc501a9be0 con 0x7fdc50111640 2023-09-10T01:14:47.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.663+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc50103b40 con 0x7fdc50111640 2023-09-10T01:14:47.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.665+0000 7fdc4cff9700 1 -- 172.21.15.3:0/4109061686 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fdc48003a00 con 0x7fdc50111640 2023-09-10T01:14:47.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.666+0000 7fdc4cff9700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc380a07b0 0x7fdc380a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.666+0000 7fdc4cff9700 1 -- 172.21.15.3:0/4109061686 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fdc480cc860 con 0x7fdc50111640 2023-09-10T01:14:47.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.667+0000 7fdc4f7fe700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc380a07b0 0x7fdc380a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.667+0000 7fdc4f7fe700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc380a07b0 0x7fdc380a2c70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009c60 tx=0x7fdc40009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.669+0000 7fdc4cff9700 1 -- 172.21.15.3:0/4109061686 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fdc48095c90 con 0x7fdc50111640 2023-09-10T01:14:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 -- 172.21.15.3:0/510268597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc10cde0 msgr2=0x7fd0dc10f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/510268597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc10cde0 0x7fd0dc10f250 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400b0a0 tx=0x7fd0d400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 -- 172.21.15.3:0/510268597 shutdown_connections 2023-09-10T01:14:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/510268597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0dc072a30 0x7fd0dc072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/510268597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc10cde0 0x7fd0dc10f250 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/510268597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0dc073350 0x7fd0dc10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 -- 172.21.15.3:0/510268597 >> 172.21.15.3:0/510268597 conn(0x7fd0dc0fc0b0 msgr2=0x7fd0dc0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 -- 172.21.15.3:0/510268597 shutdown_connections 2023-09-10T01:14:47.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.687+0000 7fd0e15ff700 1 -- 172.21.15.3:0/510268597 wait complete. 2023-09-10T01:14:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.688+0000 7fd0e15ff700 1 Processor -- start 2023-09-10T01:14:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.688+0000 7fd0e15ff700 1 -- start start 2023-09-10T01:14:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0e15ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0dc072a30 0x7fd0dc198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0e15ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc073350 0x7fd0dc198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0e15ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0dc10cde0 0x7fd0dc19d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0e15ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc19d930 con 0x7fd0dc073350 2023-09-10T01:14:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0e15ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc19daa0 con 0x7fd0dc10cde0 2023-09-10T01:14:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0e15ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc19dc60 con 0x7fd0dc072a30 2023-09-10T01:14:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc073350 0x7fd0dc198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc073350 0x7fd0dc198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36334/0 (socket says 172.21.15.3:36334) 2023-09-10T01:14:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0da7fc700 1 -- 172.21.15.3:0/2284317129 learned_addr learned my addr 172.21.15.3:0/2284317129 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0daffd700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0dc072a30 0x7fd0dc198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.689+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0dc10cde0 0x7fd0dc19d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0da7fc700 1 -- 172.21.15.3:0/2284317129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0dc072a30 msgr2=0x7fd0dc198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0dc072a30 0x7fd0dc198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0da7fc700 1 -- 172.21.15.3:0/2284317129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0dc10cde0 msgr2=0x7fd0dc19d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0dc10cde0 0x7fd0dc19d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0da7fc700 1 -- 172.21.15.3:0/2284317129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d4009d00 con 0x7fd0dc073350 2023-09-10T01:14:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0daffd700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0dc072a30 0x7fd0dc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc073350 0x7fd0dc198ed0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000e9a0 tx=0x7fd0d000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0c3fff700 1 -- 172.21.15.3:0/2284317129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0009790 con 0x7fd0dc073350 2023-09-10T01:14:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc19df40 con 0x7fd0dc073350 2023-09-10T01:14:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0c3fff700 1 -- 172.21.15.3:0/2284317129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0d0010430 con 0x7fd0dc073350 2023-09-10T01:14:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0c3fff700 1 -- 172.21.15.3:0/2284317129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0018680 con 0x7fd0dc073350 2023-09-10T01:14:47.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.690+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc1a57f0 con 0x7fd0dc073350 2023-09-10T01:14:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.695+0000 7fd0c3fff700 1 -- 172.21.15.3:0/2284317129 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd0d00105a0 con 0x7fd0dc073350 2023-09-10T01:14:47.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.696+0000 7fd0f4fffb80 1 Processor -- start 2023-09-10T01:14:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0f4fffb80 1 -- start start 2023-09-10T01:14:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada3a5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39f300 0x55fada39f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0c3fff700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0c40a07b0 0x7fd0c40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada3abf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada2f86d0 con 0x55fada3a51a0 2023-09-10T01:14:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada2f5ff0 con 0x55fada39f300 2023-09-10T01:14:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada2f18e0 con 0x55fada39fcc0 2023-09-10T01:14:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0c3fff700 1 -- 172.21.15.3:0/2284317129 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd0d0014070 con 0x7fd0dc073350 2023-09-10T01:14:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0db7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada3a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0db7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada3a5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36338/0 (socket says 172.21.15.3:36338) 2023-09-10T01:14:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0db7fe700 1 -- 172.21.15.3:0/590273666 learned_addr learned my addr 172.21.15.3:0/590273666 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.697+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39f300 0x55fada39f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0daffd700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0c40a07b0 0x7fd0c40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada3abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0db7fe700 1 -- 172.21.15.3:0/590273666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 msgr2=0x55fada3abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada3abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0db7fe700 1 -- 172.21.15.3:0/590273666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39f300 msgr2=0x55fada39f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39f300 0x55fada39f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0db7fe700 1 -- 172.21.15.3:0/590273666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fada11dd80 con 0x55fada3a51a0 2023-09-10T01:14:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada3a5580 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400bab0 tx=0x7fd0d4006080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e462dc7e9b85d95 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0daffd700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0c40a07b0 0x7fd0c40a2c70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc011490 tx=0x7fd0cc009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0c17fa700 1 -- 172.21.15.3:0/590273666 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400e040 con 0x55fada3a51a0 2023-09-10T01:14:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0c17fa700 1 -- 172.21.15.3:0/590273666 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0d4009b80 con 0x55fada3a51a0 2023-09-10T01:14:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.698+0000 7fd0c17fa700 1 -- 172.21.15.3:0/590273666 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d401fb00 con 0x55fada3a51a0 2023-09-10T01:14:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/590273666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 msgr2=0x55fada3a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada3a5580 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400bab0 tx=0x7fd0d4006080 comp rx=0 tx=0).stop 2023-09-10T01:14:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/590273666 shutdown_connections 2023-09-10T01:14:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada3abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada3a5580 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/590273666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39f300 0x55fada39f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/590273666 >> 172.21.15.3:0/590273666 conn(0x55fada314030 msgr2=0x55fada39a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/590273666 shutdown_connections 2023-09-10T01:14:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.699+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/590273666 wait complete. 2023-09-10T01:14:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.700+0000 7fd0f4fffb80 1 -- start start 2023-09-10T01:14:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.700+0000 7fd0f4fffb80 1 Processor -- start 2023-09-10T01:14:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada3aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39fcc0 0x55fada3af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3a9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada28fb10 con 0x55fada39fcc0 2023-09-10T01:14:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada28f810 con 0x55fada3a51a0 2023-09-10T01:14:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada1738c0 con 0x55fada39f300 2023-09-10T01:14:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0f4fffb80 10 client.0 _ll_register_callbacks cb 0x55fada1d9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39fcc0 0x55fada3af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0daffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada3aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39fcc0 0x55fada3af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36352/0 (socket says 172.21.15.3:36352) 2023-09-10T01:14:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 -- 172.21.15.3:0/3872228314 learned_addr learned my addr 172.21.15.3:0/3872228314 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 -- 172.21.15.3:0/3872228314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 msgr2=0x55fada3aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada3aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 -- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 msgr2=0x55fada3a9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3a9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.701+0000 7fd0da7fc700 1 -- 172.21.15.3:0/3872228314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc10ff10 con 0x55fada39fcc0 2023-09-10T01:14:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39fcc0 0x55fada3af380 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0099260 tx=0x7fd0d0063470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10960aa992d42429 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0beff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0beff5700 1 -- 172.21.15.3:0/3872228314 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d009d150 con 0x55fada39fcc0 2023-09-10T01:14:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0beff5700 1 -- 172.21.15.3:0/3872228314 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd0d006ba70 con 0x55fada39fcc0 2023-09-10T01:14:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fada306840 con 0x55fada39fcc0 2023-09-10T01:14:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0beff5700 1 -- 172.21.15.3:0/3872228314 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0d006bcb0 con 0x55fada39fcc0 2023-09-10T01:14:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0f4fffb80 10 client.18093 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55fada106c00 con 0x55fada39fcc0 2023-09-10T01:14:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.702+0000 7fd0beff5700 1 -- 172.21.15.3:0/3872228314 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fada306840 con 0x55fada39fcc0 2023-09-10T01:14:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0beff5700 1 -- 172.21.15.3:0/3872228314 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fd0d00650b0 con 0x55fada39fcc0 2023-09-10T01:14:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0f4fffb80 10 client.18093 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0f4fffb80 10 client.18093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0f4fffb80 20 client.18093 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada3b9350 0x55fada3bb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0f4fffb80 4 client.18093 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fada2ea8c0 con 0x55fada3b9350 2023-09-10T01:14:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.703+0000 7fd0daffd700 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada3b9350 0x55fada3bb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.704+0000 7fd0daffd700 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada3b9350 0x55fada3bb770 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c420b6e96f290bb server_cookie=1b8a210ca4d792a9 in_seq=0 out_seq=0 2023-09-10T01:14:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.704+0000 7fd0beff5700 10 client.18093 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.704+0000 7fd0beff5700 1 -- 172.21.15.3:0/3872228314 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fada2ea8c0 con 0x55fada3b9350 2023-09-10T01:14:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.704+0000 7fd0beff5700 10 client.18093 handle_command_reply: tid=0 2023-09-10T01:14:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.704+0000 7fd0f4fffb80 1 client.18093 shutdown 2023-09-10T01:14:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39fcc0 msgr2=0x55fada3af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39fcc0 0x55fada3af380 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0099260 tx=0x7fd0d0063470 comp rx=0 tx=0).stop 2023-09-10T01:14:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 shutdown_connections 2023-09-10T01:14:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada3b9350 0x55fada3bb770 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada3aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3a9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/3872228314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39fcc0 0x55fada3af380 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 >> 172.21.15.3:0/3872228314 conn(0x55fada314030 msgr2=0x55fada3997a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0beff5700 0 client.18093 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 shutdown_connections 2023-09-10T01:14:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/3872228314 wait complete. 2023-09-10T01:14:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.705+0000 7fd0f4fffb80 20 client.18093 trim_cache size 0 max 16384 2023-09-10T01:14:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.720+0000 7fd0f4fffb80 1 Processor -- start 2023-09-10T01:14:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.720+0000 7fd0f4fffb80 1 -- start start 2023-09-10T01:14:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39f300 0x55fada399f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada39a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada3a2df0 con 0x55fada39f300 2023-09-10T01:14:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada3a00d0 con 0x55fada3a51a0 2023-09-10T01:14:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada3aa030 con 0x55fada39fcc0 2023-09-10T01:14:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39f300 0x55fada399f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39f300 0x55fada399f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36368/0 (socket says 172.21.15.3:36368) 2023-09-10T01:14:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 -- 172.21.15.3:0/1124128325 learned_addr learned my addr 172.21.15.3:0/1124128325 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada39a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 -- 172.21.15.3:0/1124128325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 msgr2=0x55fada39a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada39a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 -- 172.21.15.3:0/1124128325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 msgr2=0x55fada3ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0daffd700 1 -- 172.21.15.3:0/1124128325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fada3a8c00 con 0x55fada39f300 2023-09-10T01:14:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.721+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.722+0000 7fd0daffd700 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39f300 0x55fada399f20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc013010 tx=0x7fd0cc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b168e695f89ee9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.722+0000 7fd0c1ffb700 1 -- 172.21.15.3:0/1124128325 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc024070 con 0x55fada39f300 2023-09-10T01:14:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.722+0000 7fd0c1ffb700 1 -- 172.21.15.3:0/1124128325 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0cc026390 con 0x55fada39f300 2023-09-10T01:14:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.722+0000 7fd0c1ffb700 1 -- 172.21.15.3:0/1124128325 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc003da0 con 0x55fada39f300 2023-09-10T01:14:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/1124128325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39f300 msgr2=0x55fada399f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39f300 0x55fada399f20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc013010 tx=0x7fd0cc014000 comp rx=0 tx=0).stop 2023-09-10T01:14:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/1124128325 shutdown_connections 2023-09-10T01:14:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada3a51a0 0x55fada3ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada39f300 0x55fada399f20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/1124128325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39fcc0 0x55fada39a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/1124128325 >> 172.21.15.3:0/1124128325 conn(0x55fada314030 msgr2=0x55fada3a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/1124128325 shutdown_connections 2023-09-10T01:14:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.723+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/1124128325 wait complete. 2023-09-10T01:14:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.724+0000 7fd0f4fffb80 1 -- start start 2023-09-10T01:14:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.724+0000 7fd0f4fffb80 1 Processor -- start 2023-09-10T01:14:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada401300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39fcc0 0x55fada401840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0f4fffb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada40c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada40c6d0 con 0x55fada3a51a0 2023-09-10T01:14:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada40c810 con 0x55fada39fcc0 2023-09-10T01:14:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0f4fffb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fada40c950 con 0x55fada39f300 2023-09-10T01:14:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0f4fffb80 10 client.0 _ll_register_callbacks cb 0x7fd0d0099190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada40c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada40c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36384/0 (socket says 172.21.15.3:36384) 2023-09-10T01:14:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 -- 172.21.15.3:0/2957769680 learned_addr learned my addr 172.21.15.3:0/2957769680 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0da7fc700 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39fcc0 0x55fada401840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0daffd700 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada401300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 -- 172.21.15.3:0/2957769680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 msgr2=0x55fada401300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada401300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 -- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39fcc0 msgr2=0x55fada401840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39fcc0 0x55fada401840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.725+0000 7fd0db7fe700 1 -- 172.21.15.3:0/2957769680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fada11dd80 con 0x55fada3a51a0 2023-09-10T01:14:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0db7fe700 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada40c190 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4022000 tx=0x7fd0d401f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5695a3159c87f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0beff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0beff5700 1 -- 172.21.15.3:0/2957769680 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400e040 con 0x55fada3a51a0 2023-09-10T01:14:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fada249490 con 0x55fada3a51a0 2023-09-10T01:14:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0beff5700 1 -- 172.21.15.3:0/2957769680 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd0d40076a0 con 0x55fada3a51a0 2023-09-10T01:14:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0beff5700 1 -- 172.21.15.3:0/2957769680 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0d4007860 con 0x55fada3a51a0 2023-09-10T01:14:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0f4fffb80 10 client.18099 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.726+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7fd0cc017070 con 0x55fada3a51a0 2023-09-10T01:14:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0beff5700 1 -- 172.21.15.3:0/2957769680 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fada249490 con 0x55fada3a51a0 2023-09-10T01:14:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0beff5700 1 -- 172.21.15.3:0/2957769680 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fd0d4019070 con 0x55fada3a51a0 2023-09-10T01:14:47.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0f4fffb80 10 client.18099 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0f4fffb80 10 client.18099 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0f4fffb80 20 client.18099 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada40de10 0x55fada410230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0f4fffb80 4 client.18099 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:14:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fada2ab050 con 0x55fada40de10 2023-09-10T01:14:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.727+0000 7fd0daffd700 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada40de10 0x55fada410230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.728+0000 7fd0daffd700 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada40de10 0x55fada410230 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad81e6ca20a8d38e server_cookie=d2708c0aec358487 in_seq=0 out_seq=0 2023-09-10T01:14:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.728+0000 7fd0beff5700 10 client.18099 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0beff5700 1 -- 172.21.15.3:0/2957769680 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fada2ab050 con 0x55fada40de10 2023-09-10T01:14:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0beff5700 10 client.18099 handle_command_reply: tid=0 2023-09-10T01:14:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 client.18099 shutdown 2023-09-10T01:14:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 msgr2=0x55fada40c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada40c190 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4022000 tx=0x7fd0d401f660 comp rx=0 tx=0).stop 2023-09-10T01:14:47.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 shutdown_connections 2023-09-10T01:14:47.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fada40de10 0x55fada410230 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fada39f300 0x55fada401300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0beff5700 0 client.18099 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fada3a51a0 0x55fada40c190 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 --2- 172.21.15.3:0/2957769680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fada39fcc0 0x55fada401840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.777+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 >> 172.21.15.3:0/2957769680 conn(0x55fada314030 msgr2=0x55fada3a7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.778+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 shutdown_connections 2023-09-10T01:14:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.778+0000 7fd0f4fffb80 1 -- 172.21.15.3:0/2957769680 wait complete. 2023-09-10T01:14:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.778+0000 7fd0f4fffb80 20 client.18099 trim_cache size 0 max 16384 2023-09-10T01:14:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:14:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:14:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:14:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:14:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.778+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0c40a07b0 msgr2=0x7fd0c40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.778+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0c40a07b0 0x7fd0c40a2c70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc011490 tx=0x7fd0cc009260 comp rx=0 tx=0).stop 2023-09-10T01:14:47.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.778+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc073350 msgr2=0x7fd0dc198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.778+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc073350 0x7fd0dc198ed0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000e9a0 tx=0x7fd0d000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 shutdown_connections 2023-09-10T01:14:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd0c40a07b0 0x7fd0c40a2c70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0dc072a30 0x7fd0dc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0dc10cde0 0x7fd0dc19d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 --2- 172.21.15.3:0/2284317129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd0dc073350 0x7fd0dc198ed0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 >> 172.21.15.3:0/2284317129 conn(0x7fd0dc0fc0b0 msgr2=0x7fd0dc104840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 shutdown_connections 2023-09-10T01:14:47.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:14:47.779+0000 7fd0e15ff700 1 -- 172.21.15.3:0/2284317129 wait complete. 2023-09-10T01:14:47.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970647 8.01 MB/sec execute 509 sec latency 4656.722 ms 2023-09-10T01:14:47.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.961+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc50067480 con 0x7fdc50111640 2023-09-10T01:14:47.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.963+0000 7fdc4cff9700 1 -- 172.21.15.3:0/4109061686 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+11297 (secure 0 0 0) 0x7fdc480992b0 con 0x7fdc50111640 2023-09-10T01:14:47.964 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:14:47.965 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:13:58.841353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:14:47.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc380a07b0 msgr2=0x7fdc380a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc380a07b0 0x7fdc380a2c70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009c60 tx=0x7fdc40009260 comp rx=0 tx=0).stop 2023-09-10T01:14:47.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc50111640 msgr2=0x7fdc501a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:47.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc50111640 0x7fdc501a1750 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdc4800ef90 tx=0x7fdc4800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 shutdown_connections 2023-09-10T01:14:47.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdc380a07b0 0x7fdc380a2c70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc50111640 0x7fdc501a1750 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc500729a0 0x7fdc5019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 --2- 172.21.15.3:0/4109061686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc500732c0 0x7fdc5019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:47.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 >> 172.21.15.3:0/4109061686 conn(0x7fdc500fc140 msgr2=0x7fdc500fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:47.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 shutdown_connections 2023-09-10T01:14:47.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:47.965+0000 7fdc55bff700 1 -- 172.21.15.3:0/4109061686 wait complete. 2023-09-10T01:14:47.970 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2023-09-10T01:14:47.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:14:48.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 -- 172.21.15.3:0/962632649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4100640 msgr2=0x7fbdd4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/962632649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4100640 0x7fbdd4100ac0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40098f0 tx=0x7fbdc4009c00 comp rx=0 tx=0).stop 2023-09-10T01:14:48.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 -- 172.21.15.3:0/962632649 shutdown_connections 2023-09-10T01:14:48.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/962632649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbdd4101000 0x7fbdd410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/962632649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd4106650 0x7fbdd4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/962632649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4100640 0x7fbdd4100ac0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 -- 172.21.15.3:0/962632649 >> 172.21.15.3:0/962632649 conn(0x7fbdd40fc170 msgr2=0x7fbdd40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 -- 172.21.15.3:0/962632649 shutdown_connections 2023-09-10T01:14:48.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.211+0000 7fbddb5ff700 1 -- 172.21.15.3:0/962632649 wait complete. 2023-09-10T01:14:48.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 Processor -- start 2023-09-10T01:14:48.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 -- start start 2023-09-10T01:14:48.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbdd4100640 0x7fbdd4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd4101000 0x7fbdd4198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4106650 0x7fbdd419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd419d900 con 0x7fbdd4101000 2023-09-10T01:14:48.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd419da70 con 0x7fbdd4106650 2023-09-10T01:14:48.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.212+0000 7fbddb5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd419dc30 con 0x7fbdd4100640 2023-09-10T01:14:48.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd8b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd4101000 0x7fbdd4198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbdd4100640 0x7fbdd4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbdd4100640 0x7fbdd4198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54216/0 (socket says 172.21.15.3:54216) 2023-09-10T01:14:48.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd939b700 1 -- 172.21.15.3:0/220450864 learned_addr learned my addr 172.21.15.3:0/220450864 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd9b9c700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4106650 0x7fbdd419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/220450864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbdd4100640 msgr2=0x7fbdd4198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbdd4100640 0x7fbdd4198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/220450864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4106650 msgr2=0x7fbdd419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4106650 0x7fbdd419d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/220450864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd00095f0 con 0x7fbdd4101000 2023-09-10T01:14:48.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.213+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd4101000 0x7fbdd4198ea0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009810 tx=0x7fbdc400fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.214+0000 7fbdca7fc700 1 -- 172.21.15.3:0/220450864 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc401c070 con 0x7fbdd4101000 2023-09-10T01:14:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.214+0000 7fbdca7fc700 1 -- 172.21.15.3:0/220450864 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdc400bba0 con 0x7fbdd4101000 2023-09-10T01:14:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.214+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc4009520 con 0x7fbdd4101000 2023-09-10T01:14:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.214+0000 7fbdca7fc700 1 -- 172.21.15.3:0/220450864 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4003970 con 0x7fbdd4101000 2023-09-10T01:14:48.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.214+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd41a55e0 con 0x7fbdd4101000 2023-09-10T01:14:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.219+0000 7fbdca7fc700 1 -- 172.21.15.3:0/220450864 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fbdc4020030 con 0x7fbdd4101000 2023-09-10T01:14:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 Processor -- start 2023-09-10T01:14:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 -- start start 2023-09-10T01:14:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3c1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc310730 con 0x559fdc3b8030 2023-09-10T01:14:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc30e050 con 0x559fdc3b7710 2023-09-10T01:14:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc309940 con 0x559fdc3c1e40 2023-09-10T01:14:48.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.220+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3c1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3c1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36408/0 (socket says 172.21.15.3:36408) 2023-09-10T01:14:48.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd9b9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd939b700 1 -- 172.21.15.3:0/2714649926 learned_addr learned my addr 172.21.15.3:0/2714649926 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd9b9c700 1 -- 172.21.15.3:0/2714649926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 msgr2=0x559fdc3c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd9b9c700 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd9b9c700 1 -- 172.21.15.3:0/2714649926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 msgr2=0x559fdc3c1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd9b9c700 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3c1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd9b9c700 1 -- 172.21.15.3:0/2714649926 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fdc135de0 con 0x559fdc3b7710 2023-09-10T01:14:48.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdca7fc700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbdc00a0830 0x7fbdc00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdca7fc700 1 -- 172.21.15.3:0/220450864 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fbdc40cd170 con 0x7fbdd4101000 2023-09-10T01:14:48.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd9b9c700 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3b7af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc003f20 tx=0x7fbdcc009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baa79fff3b56cb02 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdd939b700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbdc00a0830 0x7fbdc00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdbf7fe700 1 -- 172.21.15.3:0/2714649926 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc019070 con 0x559fdc3b7710 2023-09-10T01:14:48.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.221+0000 7fbdbf7fe700 1 -- 172.21.15.3:0/2714649926 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdcc00f040 con 0x559fdc3b7710 2023-09-10T01:14:48.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdbf7fe700 1 -- 172.21.15.3:0/2714649926 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc007990 con 0x559fdc3b7710 2023-09-10T01:14:48.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 -- 172.21.15.3:0/2714649926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 msgr2=0x559fdc3b7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdd939b700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbdc00a0830 0x7fbdc00a2cf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x559fdc3ad280 tx=0x7fbdd0009310 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3b7af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc003f20 tx=0x7fbdcc009ba0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 -- 172.21.15.3:0/2714649926 shutdown_connections 2023-09-10T01:14:48.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3b7af0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/2714649926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3c1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 -- 172.21.15.3:0/2714649926 >> 172.21.15.3:0/2714649926 conn(0x559fdc32c090 msgr2=0x559fdc3b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 -- 172.21.15.3:0/2714649926 shutdown_connections 2023-09-10T01:14:48.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.222+0000 7fbdeee44b80 1 -- 172.21.15.3:0/2714649926 wait complete. 2023-09-10T01:14:48.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.223+0000 7fbdeee44b80 1 -- start start 2023-09-10T01:14:48.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.223+0000 7fbdeee44b80 1 Processor -- start 2023-09-10T01:14:48.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b8030 0x559fdc3bea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3c1e40 0x559fdc3bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc2a7b70 con 0x559fdc3c1e40 2023-09-10T01:14:48.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc2a7870 con 0x559fdc3b7710 2023-09-10T01:14:48.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc18b920 con 0x559fdc3b8030 2023-09-10T01:14:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdeee44b80 10 client.0 _ll_register_callbacks cb 0x559fdc1f1a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd9b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3c1e40 0x559fdc3bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b8030 0x559fdc3bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b8030 0x559fdc3bea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54238/0 (socket says 172.21.15.3:54238) 2023-09-10T01:14:48.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3c1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34138/0 (socket says 172.21.15.3:34138) 2023-09-10T01:14:48.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/1439446616 learned_addr learned my addr 172.21.15.3:0/1439446616 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd9b9c700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3c1e40 0x559fdc3bf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36422/0 (socket says 172.21.15.3:36422) 2023-09-10T01:14:48.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/1439446616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 msgr2=0x559fdc3c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/1439446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3c1e40 msgr2=0x559fdc3bf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3c1e40 0x559fdc3bf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.224+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/1439446616 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdcc003b50 con 0x559fdc3b8030 2023-09-10T01:14:48.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdd939b700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:48.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdd9b9c700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3c1e40 0x559fdc3bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:48.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b8030 0x559fdc3bea60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbdc409a1f0 tx=0x7fbdc4009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a1fd0b6d5015cf1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdbcff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:48.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdbcff9700 1 -- 172.21.15.3:0/1439446616 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc409a030 con 0x559fdc3b8030 2023-09-10T01:14:48.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559fdc2614f0 con 0x559fdc3b8030 2023-09-10T01:14:48.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdbcff9700 1 -- 172.21.15.3:0/1439446616 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fbdc4005040 con 0x559fdc3b8030 2023-09-10T01:14:48.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdbcff9700 1 -- 172.21.15.3:0/1439446616 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdc4099da0 con 0x559fdc3b8030 2023-09-10T01:14:48.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdeee44b80 10 client.28087 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:48.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559fdc11ec60 con 0x559fdc3b8030 2023-09-10T01:14:48.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.225+0000 7fbdbcff9700 1 -- 172.21.15.3:0/1439446616 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559fdc2614f0 con 0x559fdc3b8030 2023-09-10T01:14:48.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdbcff9700 1 -- 172.21.15.3:0/1439446616 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fbdc4096970 con 0x559fdc3b8030 2023-09-10T01:14:48.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdeee44b80 10 client.28087 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:48.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdeee44b80 10 client.28087 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:14:48.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdeee44b80 20 client.28087 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:48.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc3d1570 0x559fdc3d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdeee44b80 4 client.28087 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:48.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fdc31e8a0 con 0x559fdc3d1570 2023-09-10T01:14:48.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.226+0000 7fbdd939b700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc3d1570 0x559fdc3d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.227+0000 7fbdd939b700 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc3d1570 0x559fdc3d3990 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a5533c0761e3cdf server_cookie=d9fb705bad30feae in_seq=0 out_seq=0 2023-09-10T01:14:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.227+0000 7fbdbcff9700 10 client.28087 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.227+0000 7fbdbcff9700 1 -- 172.21.15.3:0/1439446616 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559fdc31e8a0 con 0x559fdc3d1570 2023-09-10T01:14:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.227+0000 7fbdbcff9700 10 client.28087 handle_command_reply: tid=0 2023-09-10T01:14:48.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.227+0000 7fbdeee44b80 1 client.28087 shutdown 2023-09-10T01:14:48.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b8030 msgr2=0x559fdc3bea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b8030 0x559fdc3bea60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbdc409a1f0 tx=0x7fbdc4009f50 comp rx=0 tx=0).stop 2023-09-10T01:14:48.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 shutdown_connections 2023-09-10T01:14:48.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc3d1570 0x559fdc3d3990 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3c1e40 0x559fdc3bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b7710 0x559fdc3c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdbcff9700 0 client.28087 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:48.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1439446616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b8030 0x559fdc3bea60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 >> 172.21.15.3:0/1439446616 conn(0x559fdc32c090 msgr2=0x559fdc3b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 shutdown_connections 2023-09-10T01:14:48.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1439446616 wait complete. 2023-09-10T01:14:48.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.228+0000 7fbdeee44b80 20 client.28087 trim_cache size 0 max 16384 2023-09-10T01:14:48.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.242+0000 7fbdeee44b80 1 Processor -- start 2023-09-10T01:14:48.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.242+0000 7fbdeee44b80 1 -- start start 2023-09-10T01:14:48.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.242+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b7710 0x559fdc3c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.242+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3b1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.242+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3c1e40 0x559fdc3cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.242+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc3b40b0 con 0x559fdc3b8030 2023-09-10T01:14:48.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc3b89f0 con 0x559fdc3c1e40 2023-09-10T01:14:48.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc3b86d0 con 0x559fdc3b7710 2023-09-10T01:14:48.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b7710 0x559fdc3c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b7710 0x559fdc3c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54252/0 (socket says 172.21.15.3:54252) 2023-09-10T01:14:48.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 -- 172.21.15.3:0/1189748101 learned_addr learned my addr 172.21.15.3:0/1189748101 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd9b9c700 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3c1e40 0x559fdc3cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 -- 172.21.15.3:0/1189748101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3c1e40 msgr2=0x559fdc3cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3c1e40 0x559fdc3cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 -- 172.21.15.3:0/1189748101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 msgr2=0x559fdc3b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.243+0000 7fbdd939b700 1 -- 172.21.15.3:0/1189748101 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fdc15c250 con 0x559fdc3b7710 2023-09-10T01:14:48.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.244+0000 7fbdd939b700 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b7710 0x559fdc3c89f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00092e0 tx=0x7fbdd001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41d65b3ab8a6988a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.246+0000 7fbdbffff700 1 -- 172.21.15.3:0/1189748101 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0023040 con 0x559fdc3b7710 2023-09-10T01:14:48.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.246+0000 7fbdbffff700 1 -- 172.21.15.3:0/1189748101 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdd0024040 con 0x559fdc3b7710 2023-09-10T01:14:48.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.246+0000 7fbdbffff700 1 -- 172.21.15.3:0/1189748101 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0017070 con 0x559fdc3b7710 2023-09-10T01:14:48.259 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1189748101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b7710 msgr2=0x559fdc3c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b7710 0x559fdc3c89f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00092e0 tx=0x7fbdd001b000 comp rx=0 tx=0).stop 2023-09-10T01:14:48.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1189748101 shutdown_connections 2023-09-10T01:14:48.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3b7710 0x559fdc3c89f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3c1e40 0x559fdc3cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.260 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/1189748101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b8030 0x559fdc3b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1189748101 >> 172.21.15.3:0/1189748101 conn(0x559fdc32c090 msgr2=0x559fdc3d82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1189748101 shutdown_connections 2023-09-10T01:14:48.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.247+0000 7fbdeee44b80 1 -- 172.21.15.3:0/1189748101 wait complete. 2023-09-10T01:14:48.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 -- start start 2023-09-10T01:14:48.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 Processor -- start 2023-09-10T01:14:48.261 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b7710 0x559fdc3c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b8030 0x559fdc3c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc3c7310 con 0x559fdc3b7710 2023-09-10T01:14:48.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc3c7480 con 0x559fdc3b8030 2023-09-10T01:14:48.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fdc3c7640 con 0x559fdc3c1e40 2023-09-10T01:14:48.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.248+0000 7fbdeee44b80 10 client.0 _ll_register_callbacks cb 0x559fdc1f1a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:48.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b8030 0x559fdc3c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd9b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b8030 0x559fdc3c8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34160/0 (socket says 172.21.15.3:34160) 2023-09-10T01:14:48.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd939b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b7710 0x559fdc3c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/782037844 learned_addr learned my addr 172.21.15.3:0/782037844 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd9b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54262/0 (socket says 172.21.15.3:54262) 2023-09-10T01:14:48.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/782037844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 msgr2=0x559fdc3c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/782037844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b7710 msgr2=0x559fdc3c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b7710 0x559fdc3c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 -- 172.21.15.3:0/782037844 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd001ea90 con 0x559fdc3b8030 2023-09-10T01:14:48.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdd8b9a700 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b8030 0x559fdc3c8b70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4005bf0 tx=0x7fbdc40d0790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b079314f8746b1b0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdbcff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:14:48.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdbcff9700 1 -- 172.21.15.3:0/782037844 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40ca8f0 con 0x559fdc3b8030 2023-09-10T01:14:48.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbdcc00f040 con 0x559fdc3b8030 2023-09-10T01:14:48.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.249+0000 7fbdbcff9700 1 -- 172.21.15.3:0/782037844 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fbdc4005040 con 0x559fdc3b8030 2023-09-10T01:14:48.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdbcff9700 1 -- 172.21.15.3:0/782037844 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdc4099da0 con 0x559fdc3b8030 2023-09-10T01:14:48.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 10 client.28205 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:48.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559fdc1342b0 con 0x559fdc3b8030 2023-09-10T01:14:48.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdbcff9700 1 -- 172.21.15.3:0/782037844 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdcc00f040 con 0x559fdc3b8030 2023-09-10T01:14:48.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdbcff9700 1 -- 172.21.15.3:0/782037844 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fbdc4096170 con 0x559fdc3b8030 2023-09-10T01:14:48.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 10 client.28205 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:48.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 10 client.28205 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:14:48.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 20 client.28205 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:48.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc426220 0x559fdc428610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 4 client.28205 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:14:48.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559fdc302920 con 0x559fdc426220 2023-09-10T01:14:48.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.250+0000 7fbdd939b700 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc426220 0x559fdc428610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdd939b700 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc426220 0x559fdc428610 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=48fe36cb2f870926 server_cookie=fafce587f6940d3 in_seq=0 out_seq=0 2023-09-10T01:14:48.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdbcff9700 10 client.28205 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:48.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdbcff9700 1 -- 172.21.15.3:0/782037844 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559fdc302920 con 0x559fdc426220 2023-09-10T01:14:48.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdbcff9700 10 client.28205 handle_command_reply: tid=0 2023-09-10T01:14:48.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 client.28205 shutdown 2023-09-10T01:14:48.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b8030 msgr2=0x559fdc3c8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b8030 0x559fdc3c8b70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4005bf0 tx=0x7fbdc40d0790 comp rx=0 tx=0).stop 2023-09-10T01:14:48.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 shutdown_connections 2023-09-10T01:14:48.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x559fdc426220 0x559fdc428610 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559fdc3c1e40 0x559fdc3c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdbcff9700 0 client.28205 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:14:48.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fdc3b7710 0x559fdc3c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 --2- 172.21.15.3:0/782037844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fdc3b8030 0x559fdc3c8b70 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.251+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 >> 172.21.15.3:0/782037844 conn(0x559fdc32c090 msgr2=0x559fdc3cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 shutdown_connections 2023-09-10T01:14:48.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbdeee44b80 1 -- 172.21.15.3:0/782037844 wait complete. 2023-09-10T01:14:48.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbdeee44b80 20 client.28205 trim_cache size 0 max 16384 2023-09-10T01:14:48.272 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:14:48.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbdc00a0830 msgr2=0x7fbdc00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbdc00a0830 0x7fbdc00a2cf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x559fdc3ad280 tx=0x7fbdd0009310 comp rx=0 tx=0).stop 2023-09-10T01:14:48.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd4101000 msgr2=0x7fbdd4198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd4101000 0x7fbdd4198ea0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009810 tx=0x7fbdc400fa50 comp rx=0 tx=0).stop 2023-09-10T01:14:48.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 shutdown_connections 2023-09-10T01:14:48.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbdc00a0830 0x7fbdc00a2cf0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbdd4100640 0x7fbdd4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdd4106650 0x7fbdd419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 --2- 172.21.15.3:0/220450864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbdd4101000 0x7fbdd4198ea0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.252+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 >> 172.21.15.3:0/220450864 conn(0x7fbdd40fc170 msgr2=0x7fbdd410bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.253+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 shutdown_connections 2023-09-10T01:14:48.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.253+0000 7fbddb5ff700 1 -- 172.21.15.3:0/220450864 wait complete. 2023-09-10T01:14:48.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:14:48.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 -- 172.21.15.3:0/887792706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4101000 msgr2=0x7f38d410d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 --2- 172.21.15.3:0/887792706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4101000 0x7f38d410d510 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00b0a0 tx=0x7f38cc00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 -- 172.21.15.3:0/887792706 shutdown_connections 2023-09-10T01:14:48.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 --2- 172.21.15.3:0/887792706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38d4106650 0x7f38d4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 --2- 172.21.15.3:0/887792706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4101000 0x7f38d410d510 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 --2- 172.21.15.3:0/887792706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d4100640 0x7f38d4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 -- 172.21.15.3:0/887792706 >> 172.21.15.3:0/887792706 conn(0x7f38d40fc170 msgr2=0x7f38d40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 -- 172.21.15.3:0/887792706 shutdown_connections 2023-09-10T01:14:48.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.483+0000 7f38d9fff700 1 -- 172.21.15.3:0/887792706 wait complete. 2023-09-10T01:14:48.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.484+0000 7f38d9fff700 1 Processor -- start 2023-09-10T01:14:48.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.484+0000 7f38d9fff700 1 -- start start 2023-09-10T01:14:48.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d9fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4100640 0x7f38d419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d9fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38d4101000 0x7f38d419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d9fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d4106650 0x7f38d41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d9fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4198d60 con 0x7f38d4100640 2023-09-10T01:14:48.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d9fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4198ed0 con 0x7f38d4106650 2023-09-10T01:14:48.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d9fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4199090 con 0x7f38d4101000 2023-09-10T01:14:48.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4100640 0x7f38d419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4100640 0x7f38d419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36454/0 (socket says 172.21.15.3:36454) 2023-09-10T01:14:48.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d37fe700 1 -- 172.21.15.3:0/2321549872 learned_addr learned my addr 172.21.15.3:0/2321549872 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d3fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d4106650 0x7f38d41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38d4101000 0x7f38d419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d3fff700 1 -- 172.21.15.3:0/2321549872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38d4101000 msgr2=0x7f38d419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d3fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38d4101000 0x7f38d419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d3fff700 1 -- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4100640 msgr2=0x7f38d419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d3fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4100640 0x7f38d419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d3fff700 1 -- 172.21.15.3:0/2321549872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc009d00 con 0x7f38d4106650 2023-09-10T01:14:48.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.485+0000 7f38d37fe700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4100640 0x7f38d419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:48.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.486+0000 7f38d3fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d4106650 0x7f38d41986f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0062a0 tx=0x7f38cc0077f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.486+0000 7f38d0ff9700 1 -- 172.21.15.3:0/2321549872 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc00e040 con 0x7f38d4106650 2023-09-10T01:14:48.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.486+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d4199310 con 0x7f38d4106650 2023-09-10T01:14:48.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.486+0000 7f38d0ff9700 1 -- 172.21.15.3:0/2321549872 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38cc004e60 con 0x7f38d4106650 2023-09-10T01:14:48.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.486+0000 7f38d0ff9700 1 -- 172.21.15.3:0/2321549872 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc0052a0 con 0x7f38d4106650 2023-09-10T01:14:48.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.486+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d41a5670 con 0x7f38d4106650 2023-09-10T01:14:48.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.489+0000 7f38d0ff9700 1 -- 172.21.15.3:0/2321549872 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f38cc01d470 con 0x7f38d4106650 2023-09-10T01:14:48.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 Processor -- start 2023-09-10T01:14:48.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 -- start start 2023-09-10T01:14:48.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a04eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38d0ff9700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f38bc0a07b0 0x7f38bc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a058900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a05b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558219fa7730 con 0x55821a04f030 2023-09-10T01:14:48.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558219fa5050 con 0x55821a04e710 2023-09-10T01:14:48.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.491+0000 7f38ed855b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558219fa0940 con 0x55821a058e40 2023-09-10T01:14:48.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a04eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d0ff9700 1 -- 172.21.15.3:0/2321549872 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f38cc0cdb50 con 0x7f38d4106650 2023-09-10T01:14:48.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a04eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34174/0 (socket says 172.21.15.3:34174) 2023-09-10T01:14:48.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 -- 172.21.15.3:0/1766998909 learned_addr learned my addr 172.21.15.3:0/1766998909 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f38bc0a07b0 0x7f38bc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a058900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d3fff700 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a05b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 -- 172.21.15.3:0/1766998909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 msgr2=0x55821a05b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a05b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 -- 172.21.15.3:0/1766998909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 msgr2=0x55821a058900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a058900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.492+0000 7f38d37fe700 1 -- 172.21.15.3:0/1766998909 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558219dccde0 con 0x55821a04e710 2023-09-10T01:14:48.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.493+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a04eaf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f38c4011650 tx=0x7f38c4011960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92b5fbc8087481d8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.493+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f38bc0a07b0 0x7f38bc0a2c70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55821a044280 tx=0x7f38c8009610 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.493+0000 7f38b9ffb700 1 -- 172.21.15.3:0/1766998909 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c400f680 con 0x55821a04e710 2023-09-10T01:14:48.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.493+0000 7f38b9ffb700 1 -- 172.21.15.3:0/1766998909 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38c400fca0 con 0x55821a04e710 2023-09-10T01:14:48.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.493+0000 7f38b9ffb700 1 -- 172.21.15.3:0/1766998909 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4019d90 con 0x55821a04e710 2023-09-10T01:14:48.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 -- 172.21.15.3:0/1766998909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 msgr2=0x55821a04eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a04eaf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f38c4011650 tx=0x7f38c4011960 comp rx=0 tx=0).stop 2023-09-10T01:14:48.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 -- 172.21.15.3:0/1766998909 shutdown_connections 2023-09-10T01:14:48.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a05b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a04eaf0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1766998909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a058900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 -- 172.21.15.3:0/1766998909 >> 172.21.15.3:0/1766998909 conn(0x558219fc3090 msgr2=0x55821a049bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 -- 172.21.15.3:0/1766998909 shutdown_connections 2023-09-10T01:14:48.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 -- 172.21.15.3:0/1766998909 wait complete. 2023-09-10T01:14:48.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.494+0000 7f38ed855b80 1 -- start start 2023-09-10T01:14:48.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 1 Processor -- start 2023-09-10T01:14:48.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a04e710 0x55821a058880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a055b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a058e40 0x55821a0562b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558219f3eb70 con 0x55821a04f030 2023-09-10T01:14:48.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558219f3e870 con 0x55821a058e40 2023-09-10T01:14:48.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558219e22920 con 0x55821a04e710 2023-09-10T01:14:48.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.495+0000 7f38ed855b80 10 client.0 _ll_register_callbacks cb 0x558219e88a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:48.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a058e40 0x55821a0562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a04e710 0x55821a058880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a04e710 0x55821a058880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54296/0 (socket says 172.21.15.3:54296) 2023-09-10T01:14:48.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a058e40 0x55821a0562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34180/0 (socket says 172.21.15.3:34180) 2023-09-10T01:14:48.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a055b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 -- 172.21.15.3:0/1425886082 learned_addr learned my addr 172.21.15.3:0/1425886082 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 -- 172.21.15.3:0/1425886082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a058e40 msgr2=0x55821a0562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a058e40 0x55821a0562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 -- 172.21.15.3:0/1425886082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 msgr2=0x55821a055b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a055b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 -- 172.21.15.3:0/1425886082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c4009e10 con 0x55821a04e710 2023-09-10T01:14:48.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a04e710 0x55821a058880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f38c4019910 tx=0x7f38c40057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f7cac56e63cc549 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38b2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:48.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.496+0000 7f38b2ffd700 1 -- 172.21.15.3:0/1425886082 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4004450 con 0x55821a04e710 2023-09-10T01:14:48.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.497+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558219ef84f0 con 0x55821a04e710 2023-09-10T01:14:48.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.497+0000 7f38b2ffd700 1 -- 172.21.15.3:0/1425886082 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f38c4019430 con 0x55821a04e710 2023-09-10T01:14:48.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.497+0000 7f38b2ffd700 1 -- 172.21.15.3:0/1425886082 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38c40045b0 con 0x55821a04e710 2023-09-10T01:14:48.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.497+0000 7f38ed855b80 10 client.28105 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:48.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.497+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x558219db5c60 con 0x55821a04e710 2023-09-10T01:14:48.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.497+0000 7f38b2ffd700 1 -- 172.21.15.3:0/1425886082 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558219ef84f0 con 0x55821a04e710 2023-09-10T01:14:48.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.497+0000 7f38b2ffd700 1 -- 172.21.15.3:0/1425886082 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f38c400f580 con 0x55821a04e710 2023-09-10T01:14:48.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38ed855b80 10 client.28105 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:48.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38ed855b80 10 client.28105 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:14:48.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38ed855b80 20 client.28105 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:48.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a068a50 0x55821a06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38ed855b80 4 client.28105 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:48.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558219fb58a0 con 0x55821a068a50 2023-09-10T01:14:48.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a068a50 0x55821a06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a068a50 0x55821a06ae70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7e046054aab85d5 server_cookie=fe92ddb63dec5185 in_seq=0 out_seq=0 2023-09-10T01:14:48.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.498+0000 7f38b2ffd700 10 client.28105 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:48.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38b2ffd700 1 -- 172.21.15.3:0/1425886082 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558219fb58a0 con 0x55821a068a50 2023-09-10T01:14:48.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38b2ffd700 10 client.28105 handle_command_reply: tid=0 2023-09-10T01:14:48.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 client.28105 shutdown 2023-09-10T01:14:48.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a04e710 msgr2=0x55821a058880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a04e710 0x55821a058880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f38c4019910 tx=0x7f38c40057c0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 shutdown_connections 2023-09-10T01:14:48.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a068a50 0x55821a06ae70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a04e710 0x55821a058880 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a058e40 0x55821a0562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38b2ffd700 0 client.28105 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1425886082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a055b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 >> 172.21.15.3:0/1425886082 conn(0x558219fc3090 msgr2=0x55821a049780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 shutdown_connections 2023-09-10T01:14:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.499+0000 7f38ed855b80 1 -- 172.21.15.3:0/1425886082 wait complete. 2023-09-10T01:14:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.500+0000 7f38ed855b80 20 client.28105 trim_cache size 0 max 16384 2023-09-10T01:14:48.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.513+0000 7f38ed855b80 1 Processor -- start 2023-09-10T01:14:48.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.513+0000 7f38ed855b80 1 -- start start 2023-09-10T01:14:48.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a058420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a050e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0513a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38ed855b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55821a04a6e0 con 0x55821a04f030 2023-09-10T01:14:48.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38ed855b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55821a04f9c0 con 0x55821a04e710 2023-09-10T01:14:48.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38ed855b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55821a056c40 con 0x55821a058e40 2023-09-10T01:14:48.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a058420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a058420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34188/0 (socket says 172.21.15.3:34188) 2023-09-10T01:14:48.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 -- 172.21.15.3:0/1770705620 learned_addr learned my addr 172.21.15.3:0/1770705620 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a050e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d3fff700 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 -- 172.21.15.3:0/1770705620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 msgr2=0x55821a0513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 -- 172.21.15.3:0/1770705620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 msgr2=0x55821a050e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a050e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 -- 172.21.15.3:0/1770705620 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558219df3250 con 0x55821a04e710 2023-09-10T01:14:48.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.514+0000 7f38d37fe700 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a058420 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f38c400f940 tx=0x7f38c40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bbd868ac0a3cc5a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ba7fc700 1 -- 172.21.15.3:0/1770705620 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4005d50 con 0x55821a04e710 2023-09-10T01:14:48.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ba7fc700 1 -- 172.21.15.3:0/1770705620 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38c401ae20 con 0x55821a04e710 2023-09-10T01:14:48.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ba7fc700 1 -- 172.21.15.3:0/1770705620 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40038e0 con 0x55821a04e710 2023-09-10T01:14:48.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 -- 172.21.15.3:0/1770705620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 msgr2=0x55821a058420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a058420 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f38c400f940 tx=0x7f38c40095c0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 -- 172.21.15.3:0/1770705620 shutdown_connections 2023-09-10T01:14:48.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a058420 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 --2- 172.21.15.3:0/1770705620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a050e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 -- 172.21.15.3:0/1770705620 >> 172.21.15.3:0/1770705620 conn(0x558219fc3090 msgr2=0x55821a064ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.515+0000 7f38ed855b80 1 -- 172.21.15.3:0/1770705620 shutdown_connections 2023-09-10T01:14:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.516+0000 7f38ed855b80 1 -- 172.21.15.3:0/1770705620 wait complete. 2023-09-10T01:14:48.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.516+0000 7f38ed855b80 1 -- start start 2023-09-10T01:14:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.516+0000 7f38ed855b80 1 Processor -- start 2023-09-10T01:14:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a05d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a05dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38ed855b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0b5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38ed855b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55821a0b5d50 con 0x55821a04f030 2023-09-10T01:14:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38ed855b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55821a0b5ec0 con 0x55821a04e710 2023-09-10T01:14:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38ed855b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55821a0b6080 con 0x55821a058e40 2023-09-10T01:14:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38ed855b80 10 client.0 _ll_register_callbacks cb 0x7f38c40114d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a05dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a05d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a05d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34190/0 (socket says 172.21.15.3:34190) 2023-09-10T01:14:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a05dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36490/0 (socket says 172.21.15.3:36490) 2023-09-10T01:14:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 -- 172.21.15.3:0/2675421258 learned_addr learned my addr 172.21.15.3:0/2675421258 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d3fff700 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 -- 172.21.15.3:0/2675421258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 msgr2=0x55821a0b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 -- 172.21.15.3:0/2675421258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 msgr2=0x55821a05dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a05dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.517+0000 7f38d37fe700 1 -- 172.21.15.3:0/2675421258 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558219db5c60 con 0x55821a04e710 2023-09-10T01:14:48.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38d37fe700 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a05d840 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f38c4011650 tx=0x7f38c401ad00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd0eb356cc08befa server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38b2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:14:48.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38b2ffd700 1 -- 172.21.15.3:0/2675421258 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4011d60 con 0x55821a04e710 2023-09-10T01:14:48.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38c400fca0 con 0x55821a04e710 2023-09-10T01:14:48.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38b2ffd700 1 -- 172.21.15.3:0/2675421258 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f38c4000e40 con 0x55821a04e710 2023-09-10T01:14:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38b2ffd700 1 -- 172.21.15.3:0/2675421258 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38c4004250 con 0x55821a04e710 2023-09-10T01:14:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38ed855b80 10 client.28223 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x558219dcb2b0 con 0x55821a04e710 2023-09-10T01:14:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.518+0000 7f38b2ffd700 1 -- 172.21.15.3:0/2675421258 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38c400fca0 con 0x55821a04e710 2023-09-10T01:14:48.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38b2ffd700 1 -- 172.21.15.3:0/2675421258 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f38c401a0d0 con 0x55821a04e710 2023-09-10T01:14:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38ed855b80 10 client.28223 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38ed855b80 10 client.28223 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:14:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38ed855b80 20 client.28223 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38ed855b80 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a0bd3d0 0x55821a0bf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38ed855b80 4 client.28223 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:14:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558219f99920 con 0x55821a0bd3d0 2023-09-10T01:14:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a0bd3d0 0x55821a0bf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38d2ffd700 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a0bd3d0 0x55821a0bf7f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d4e66acef34637b5 server_cookie=f937b8610d78c0cf in_seq=0 out_seq=0 2023-09-10T01:14:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38b2ffd700 10 client.28223 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:48.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38b2ffd700 1 -- 172.21.15.3:0/2675421258 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558219f99920 con 0x55821a0bd3d0 2023-09-10T01:14:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38b2ffd700 10 client.28223 handle_command_reply: tid=0 2023-09-10T01:14:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.519+0000 7f38ed855b80 1 client.28223 shutdown 2023-09-10T01:14:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 msgr2=0x55821a05d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a05d840 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f38c4011650 tx=0x7f38c401ad00 comp rx=0 tx=0).stop 2023-09-10T01:14:48.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 shutdown_connections 2023-09-10T01:14:48.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55821a0bd3d0 0x55821a0bf7f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55821a058e40 0x55821a0b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55821a04e710 0x55821a05d840 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38b2ffd700 0 client.28223 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:14:48.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 --2- 172.21.15.3:0/2675421258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55821a04f030 0x55821a05dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 >> 172.21.15.3:0/2675421258 conn(0x558219fc3090 msgr2=0x55821a050e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 shutdown_connections 2023-09-10T01:14:48.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 1 -- 172.21.15.3:0/2675421258 wait complete. 2023-09-10T01:14:48.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.520+0000 7f38ed855b80 20 client.28223 trim_cache size 0 max 16384 2023-09-10T01:14:48.540 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:14:48.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f38bc0a07b0 msgr2=0x7f38bc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f38bc0a07b0 0x7f38bc0a2c70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55821a044280 tx=0x7f38c8009610 comp rx=0 tx=0).stop 2023-09-10T01:14:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d4106650 msgr2=0x7f38d41986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d4106650 0x7f38d41986f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0062a0 tx=0x7f38cc0077f0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 shutdown_connections 2023-09-10T01:14:48.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f38bc0a07b0 0x7f38bc0a2c70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38d4106650 0x7f38d41986f0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38d4100640 0x7f38d419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 --2- 172.21.15.3:0/2321549872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38d4101000 0x7f38d419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 >> 172.21.15.3:0/2321549872 conn(0x7f38d40fc170 msgr2=0x7f38d410ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 shutdown_connections 2023-09-10T01:14:48.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.521+0000 7f38d9fff700 1 -- 172.21.15.3:0/2321549872 wait complete. 2023-09-10T01:14:48.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:48 smithi003 ceph-mon[97654]: pgmap v360: 129 pgs: 129 active+clean; 297 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 1.0 MiB/s wr, 121 op/s 2023-09-10T01:14:48.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:48 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4109061686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:14:48.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:14:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:48 smithi136 ceph-mon[133708]: pgmap v360: 129 pgs: 129 active+clean; 297 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 1.0 MiB/s wr, 121 op/s 2023-09-10T01:14:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:48 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4109061686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:14:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:48 smithi092 ceph-mon[134067]: pgmap v360: 129 pgs: 129 active+clean; 297 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 1.0 MiB/s wr, 121 op/s 2023-09-10T01:14:48.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:48 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4109061686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:14:48.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 -- 172.21.15.3:0/1036995464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10100640 msgr2=0x7fba10100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 --2- 172.21.15.3:0/1036995464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10100640 0x7fba10100ac0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fba00009990 tx=0x7fba00009ca0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 -- 172.21.15.3:0/1036995464 shutdown_connections 2023-09-10T01:14:48.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 --2- 172.21.15.3:0/1036995464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba10106650 0x7fba10106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 --2- 172.21.15.3:0/1036995464 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba10101000 0x7fba1010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 --2- 172.21.15.3:0/1036995464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10100640 0x7fba10100ac0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 -- 172.21.15.3:0/1036995464 >> 172.21.15.3:0/1036995464 conn(0x7fba100fc170 msgr2=0x7fba100fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 -- 172.21.15.3:0/1036995464 shutdown_connections 2023-09-10T01:14:48.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.750+0000 7fba187ff700 1 -- 172.21.15.3:0/1036995464 wait complete. 2023-09-10T01:14:48.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.751+0000 7fba187ff700 1 Processor -- start 2023-09-10T01:14:48.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.751+0000 7fba187ff700 1 -- start start 2023-09-10T01:14:48.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba187ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba10100640 0x7fba10072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba187ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10101000 0x7fba10072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba187ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba10106650 0x7fba101a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba187ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10073670 con 0x7fba10101000 2023-09-10T01:14:48.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba187ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba101a4e20 con 0x7fba10100640 2023-09-10T01:14:48.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba187ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba101a4f60 con 0x7fba10106650 2023-09-10T01:14:48.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10101000 0x7fba10072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.758 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba1659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba10100640 0x7fba10072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10101000 0x7fba10072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36506/0 (socket says 172.21.15.3:36506) 2023-09-10T01:14:48.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 -- 172.21.15.3:0/240590049 learned_addr learned my addr 172.21.15.3:0/240590049 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba16d9c700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba10106650 0x7fba101a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 -- 172.21.15.3:0/240590049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba10106650 msgr2=0x7fba101a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba10106650 0x7fba101a48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 -- 172.21.15.3:0/240590049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba10100640 msgr2=0x7fba10072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba10100640 0x7fba10072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba15d9a700 1 -- 172.21.15.3:0/240590049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba000095f0 con 0x7fba10101000 2023-09-10T01:14:48.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.752+0000 7fba1659b700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba10100640 0x7fba10072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:48.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.753+0000 7fba15d9a700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10101000 0x7fba10072e40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fba00005910 tx=0x7fba00005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.753+0000 7fba077fe700 1 -- 172.21.15.3:0/240590049 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0001d070 con 0x7fba10101000 2023-09-10T01:14:48.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.753+0000 7fba077fe700 1 -- 172.21.15.3:0/240590049 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba00022470 con 0x7fba10101000 2023-09-10T01:14:48.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.753+0000 7fba077fe700 1 -- 172.21.15.3:0/240590049 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00017650 con 0x7fba10101000 2023-09-10T01:14:48.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.753+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba101a5160 con 0x7fba10101000 2023-09-10T01:14:48.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.753+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba101a55f0 con 0x7fba10101000 2023-09-10T01:14:48.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.756+0000 7fba077fe700 1 -- 172.21.15.3:0/240590049 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fba000225e0 con 0x7fba10101000 2023-09-10T01:14:48.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.757+0000 7fba2c209b80 1 Processor -- start 2023-09-10T01:14:48.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba2c209b80 1 -- start start 2023-09-10T01:14:48.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba077fe700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb9fc0a07b0 0x7fb9fc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba077fe700 1 -- 172.21.15.3:0/240590049 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fba000cd7b0 con 0x7fba10101000 2023-09-10T01:14:48.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422b710 0x55a99422baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba16d9c700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb9fc0a07b0 0x7fb9fc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a99422c030 0x55a994235900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a994235e40 0x55a994238230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba2c209b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a994184730 con 0x55a994235e40 2023-09-10T01:14:48.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba2c209b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a994182050 con 0x55a99422b710 2023-09-10T01:14:48.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba2c209b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a99417d940 con 0x55a99422c030 2023-09-10T01:14:48.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba1659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422b710 0x55a99422baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba1659b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422b710 0x55a99422baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34218/0 (socket says 172.21.15.3:34218) 2023-09-10T01:14:48.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba1659b700 1 -- 172.21.15.3:0/108374838 learned_addr learned my addr 172.21.15.3:0/108374838 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba16d9c700 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a994235e40 0x55a994238230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.758+0000 7fba15d9a700 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a99422c030 0x55a994235900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba1659b700 1 -- 172.21.15.3:0/108374838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a99422c030 msgr2=0x55a994235900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba1659b700 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a99422c030 0x55a994235900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba1659b700 1 -- 172.21.15.3:0/108374838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a994235e40 msgr2=0x55a994238230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba1659b700 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a994235e40 0x55a994238230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba1659b700 1 -- 172.21.15.3:0/108374838 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a993fa9de0 con 0x55a99422b710 2023-09-10T01:14:48.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba16d9c700 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a994235e40 0x55a994238230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:48.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba16d9c700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb9fc0a07b0 0x7fb9fc0a2c70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fba10074a10 tx=0x7fba08007a80 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba1659b700 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422b710 0x55a99422baf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0115f0 tx=0x7fba0c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b1dd597daf7463 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba04ff9700 1 -- 172.21.15.3:0/108374838 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c019630 con 0x55a99422b710 2023-09-10T01:14:48.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba04ff9700 1 -- 172.21.15.3:0/108374838 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba0c019c50 con 0x55a99422b710 2023-09-10T01:14:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.759+0000 7fba04ff9700 1 -- 172.21.15.3:0/108374838 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c00eb50 con 0x55a99422b710 2023-09-10T01:14:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 -- 172.21.15.3:0/108374838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422b710 msgr2=0x55a99422baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422b710 0x55a99422baf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0115f0 tx=0x7fba0c011900 comp rx=0 tx=0).stop 2023-09-10T01:14:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 -- 172.21.15.3:0/108374838 shutdown_connections 2023-09-10T01:14:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a994235e40 0x55a994238230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422b710 0x55a99422baf0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 --2- 172.21.15.3:0/108374838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a99422c030 0x55a994235900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 -- 172.21.15.3:0/108374838 >> 172.21.15.3:0/108374838 conn(0x55a9941a0090 msgr2=0x55a994226bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 -- 172.21.15.3:0/108374838 shutdown_connections 2023-09-10T01:14:48.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.760+0000 7fba2c209b80 1 -- 172.21.15.3:0/108374838 wait complete. 2023-09-10T01:14:48.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 -- start start 2023-09-10T01:14:48.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 Processor -- start 2023-09-10T01:14:48.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994235790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a994232a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994232ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a99411bb70 con 0x55a99422b710 2023-09-10T01:14:48.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a99411b870 con 0x55a99422c030 2023-09-10T01:14:48.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a993fff920 con 0x55a994235e40 2023-09-10T01:14:48.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.761+0000 7fba2c209b80 10 client.0 _ll_register_callbacks cb 0x55a994065a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:48.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994232ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994232ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54360/0 (socket says 172.21.15.3:54360) 2023-09-10T01:14:48.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba15d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a994232a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba1659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994235790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 -- 172.21.15.3:0/1206789910 learned_addr learned my addr 172.21.15.3:0/1206789910 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 -- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 msgr2=0x55a994232a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a994232a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 -- 172.21.15.3:0/1206789910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 msgr2=0x55a994235790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994235790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 -- 172.21.15.3:0/1206789910 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c009d50 con 0x55a994235e40 2023-09-10T01:14:48.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba15d9a700 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a994232a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:48.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fba16d9c700 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994232ff0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fba0801aba0 tx=0x7fba08004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e352095eb085cc20 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.762+0000 7fb9f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:48.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/1206789910 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0800e050 con 0x55a994235e40 2023-09-10T01:14:48.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9940d54f0 con 0x55a994235e40 2023-09-10T01:14:48.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/1206789910 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fba080199e0 con 0x55a994235e40 2023-09-10T01:14:48.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/1206789910 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba08019ba0 con 0x55a994235e40 2023-09-10T01:14:48.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fba2c209b80 10 client.28123 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:48.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55a993f92c60 con 0x55a994235e40 2023-09-10T01:14:48.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/1206789910 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a9940d54f0 con 0x55a994235e40 2023-09-10T01:14:48.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.763+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/1206789910 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fba08026cd0 con 0x55a994235e40 2023-09-10T01:14:48.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba2c209b80 10 client.28123 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:48.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba2c209b80 10 client.28123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:48.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba2c209b80 20 client.28123 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:48.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba2c209b80 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a9942459a0 0x55a994247dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba2c209b80 4 client.28123 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:48.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9941928a0 con 0x55a9942459a0 2023-09-10T01:14:48.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba1659b700 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a9942459a0 0x55a994247dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fba1659b700 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a9942459a0 0x55a994247dc0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1844084e53eaecbc server_cookie=3df3dbafb1e953b9 in_seq=0 out_seq=0 2023-09-10T01:14:48.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.764+0000 7fb9f9ffb700 10 client.28123 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.765+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/1206789910 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a9941928a0 con 0x55a9942459a0 2023-09-10T01:14:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.765+0000 7fb9f9ffb700 10 client.28123 handle_command_reply: tid=0 2023-09-10T01:14:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 client.28123 shutdown 2023-09-10T01:14:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 msgr2=0x55a994232ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994232ff0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fba0801aba0 tx=0x7fba08004b40 comp rx=0 tx=0).stop 2023-09-10T01:14:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 shutdown_connections 2023-09-10T01:14:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a9942459a0 0x55a994247dc0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994232ff0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994235790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fb9f9ffb700 0 client.28123 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 --2- 172.21.15.3:0/1206789910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a994232a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 >> 172.21.15.3:0/1206789910 conn(0x55a9941a0090 msgr2=0x55a994226780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 shutdown_connections 2023-09-10T01:14:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 1 -- 172.21.15.3:0/1206789910 wait complete. 2023-09-10T01:14:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.766+0000 7fba2c209b80 20 client.28123 trim_cache size 0 max 16384 2023-09-10T01:14:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.779+0000 7fba2c209b80 1 Processor -- start 2023-09-10T01:14:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba2c209b80 1 -- start start 2023-09-10T01:14:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994238230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99423c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994225dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba2c209b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a99423cf30 con 0x55a99422b710 2023-09-10T01:14:48.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba2c209b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a994230f70 con 0x55a99422c030 2023-09-10T01:14:48.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba2c209b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a994247d90 con 0x55a994235e40 2023-09-10T01:14:48.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba1659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994238230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba15d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99423c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba15d9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99423c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34234/0 (socket says 172.21.15.3:34234) 2023-09-10T01:14:48.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba15d9a700 1 -- 172.21.15.3:0/3222190101 learned_addr learned my addr 172.21.15.3:0/3222190101 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba16d9c700 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994225dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba15d9a700 1 -- 172.21.15.3:0/3222190101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 msgr2=0x55a994225dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.780+0000 7fba15d9a700 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994225dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba15d9a700 1 -- 172.21.15.3:0/3222190101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 msgr2=0x55a994238230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba15d9a700 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994238230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba15d9a700 1 -- 172.21.15.3:0/3222190101 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a993fd0250 con 0x55a99422c030 2023-09-10T01:14:48.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba15d9a700 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99423c9f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fba00096ec0 tx=0x7fba0001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3fc2a3eb0aed48 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba057fa700 1 -- 172.21.15.3:0/3222190101 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0009a500 con 0x55a99422c030 2023-09-10T01:14:48.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba057fa700 1 -- 172.21.15.3:0/3222190101 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba0009a320 con 0x55a99422c030 2023-09-10T01:14:48.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba057fa700 1 -- 172.21.15.3:0/3222190101 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00099eb0 con 0x55a99422c030 2023-09-10T01:14:48.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba2c209b80 1 -- 172.21.15.3:0/3222190101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 msgr2=0x55a99423c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.781+0000 7fba2c209b80 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99423c9f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fba00096ec0 tx=0x7fba0001a040 comp rx=0 tx=0).stop 2023-09-10T01:14:48.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 -- 172.21.15.3:0/3222190101 shutdown_connections 2023-09-10T01:14:48.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a994225dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a994238230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 --2- 172.21.15.3:0/3222190101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99423c9f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 -- 172.21.15.3:0/3222190101 >> 172.21.15.3:0/3222190101 conn(0x55a9941a0090 msgr2=0x55a994241c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 -- 172.21.15.3:0/3222190101 shutdown_connections 2023-09-10T01:14:48.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 -- 172.21.15.3:0/3222190101 wait complete. 2023-09-10T01:14:48.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 -- start start 2023-09-10T01:14:48.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.782+0000 7fba2c209b80 1 Processor -- start 2023-09-10T01:14:48.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a99428d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99428de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba2c209b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a9942987e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba2c209b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a99428e620 con 0x55a99422b710 2023-09-10T01:14:48.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba2c209b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a994298d20 con 0x55a99422c030 2023-09-10T01:14:48.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba2c209b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a994298ee0 con 0x55a994235e40 2023-09-10T01:14:48.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba2c209b80 10 client.0 _ll_register_callbacks cb 0x7fba0801a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:48.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a99428d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a99428d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36560/0 (socket says 172.21.15.3:36560) 2023-09-10T01:14:48.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 -- 172.21.15.3:0/2374412166 learned_addr learned my addr 172.21.15.3:0/2374412166 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:48.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba15d9a700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99428de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba16d9c700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a9942987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 -- 172.21.15.3:0/2374412166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 msgr2=0x55a9942987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a9942987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 -- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 msgr2=0x55a99428de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:48.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99428de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba1659b700 1 -- 172.21.15.3:0/2374412166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba101a55f0 con 0x55a99422b710 2023-09-10T01:14:48.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.783+0000 7fba15d9a700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99428de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:48.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fba1659b700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a99428d950 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fba0c01c040 tx=0x7fba0c00e6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14cd7d5f2a8e3173 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:48.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fb9f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:14:48.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/2374412166 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c019dc0 con 0x55a99422b710 2023-09-10T01:14:48.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba0c019c50 con 0x55a99422b710 2023-09-10T01:14:48.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/2374412166 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fba0c005070 con 0x55a99422b710 2023-09-10T01:14:48.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/2374412166 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba0c005230 con 0x55a99422b710 2023-09-10T01:14:48.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fba2c209b80 10 client.18153 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:48.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55a993fa82b0 con 0x55a99422b710 2023-09-10T01:14:48.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.784+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/2374412166 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba0c019c50 con 0x55a99422b710 2023-09-10T01:14:48.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/2374412166 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fba0c02a7a0 con 0x55a99422b710 2023-09-10T01:14:48.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba2c209b80 10 client.18153 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:48.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba2c209b80 10 client.18153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:48.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba2c209b80 20 client.18153 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba2c209b80 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a99429a680 0x55a99429caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba2c209b80 4 client.18153 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:14:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a994176920 con 0x55a99429a680 2023-09-10T01:14:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba15d9a700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a99429a680 0x55a99429caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fba15d9a700 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a99429a680 0x55a99429caa0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=333a4f3aa5668e41 server_cookie=75a6f0176f1f2e38 in_seq=0 out_seq=0 2023-09-10T01:14:48.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.785+0000 7fb9f9ffb700 10 client.18153 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:48.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fb9f9ffb700 1 -- 172.21.15.3:0/2374412166 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a994176920 con 0x55a99429a680 2023-09-10T01:14:48.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fb9f9ffb700 10 client.18153 handle_command_reply: tid=0 2023-09-10T01:14:48.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 client.18153 shutdown 2023-09-10T01:14:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 msgr2=0x55a99428d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a99428d950 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fba0c01c040 tx=0x7fba0c00e6f0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 shutdown_connections 2023-09-10T01:14:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a99429a680 0x55a99429caa0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a994235e40 0x55a9942987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a99422b710 0x55a99428d950 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 --2- 172.21.15.3:0/2374412166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a99422c030 0x55a99428de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fb9f9ffb700 0 client.18153 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 >> 172.21.15.3:0/2374412166 conn(0x55a9941a0090 msgr2=0x55a994241140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 shutdown_connections 2023-09-10T01:14:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 1 -- 172.21.15.3:0/2374412166 wait complete. 2023-09-10T01:14:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.786+0000 7fba2c209b80 20 client.18153 trim_cache size 0 max 16384 2023-09-10T01:14:48.808 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:14:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb9fc0a07b0 msgr2=0x7fb9fc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb9fc0a07b0 0x7fb9fc0a2c70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fba10074a10 tx=0x7fba08007a80 comp rx=0 tx=0).stop 2023-09-10T01:14:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10101000 msgr2=0x7fba10072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10101000 0x7fba10072e40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fba00005910 tx=0x7fba00005720 comp rx=0 tx=0).stop 2023-09-10T01:14:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 shutdown_connections 2023-09-10T01:14:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb9fc0a07b0 0x7fb9fc0a2c70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba10106650 0x7fba101a48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba10100640 0x7fba10072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 --2- 172.21.15.3:0/240590049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10101000 0x7fba10072e40 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 >> 172.21.15.3:0/240590049 conn(0x7fba100fc170 msgr2=0x7fba100fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 shutdown_connections 2023-09-10T01:14:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:48.787+0000 7fba187ff700 1 -- 172.21.15.3:0/240590049 wait complete. 2023-09-10T01:14:48.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970715 8.00 MB/sec execute 510 sec latency 4951.408 ms 2023-09-10T01:14:49.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 970715 7.98 MB/sec execute 511 sec latency 1627.384 ms 2023-09-10T01:14:49.814 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:14:49.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:14:50.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.096+0000 7f69325ff700 1 -- 172.21.15.3:0/1070969661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107fc0 msgr2=0x7f692c078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.096+0000 7f69325ff700 1 --2- 172.21.15.3:0/1070969661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107fc0 0x7f692c078860 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f69240080b0 tx=0x7f692400d650 comp rx=0 tx=0).stop 2023-09-10T01:14:50.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.110+0000 7f69325ff700 1 -- 172.21.15.3:0/1070969661 shutdown_connections 2023-09-10T01:14:50.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.110+0000 7f69325ff700 1 --2- 172.21.15.3:0/1070969661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c10cd80 0x7f692c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.110+0000 7f69325ff700 1 --2- 172.21.15.3:0/1070969661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107fc0 0x7f692c078860 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.110+0000 7f69325ff700 1 --2- 172.21.15.3:0/1070969661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f692c107600 0x7f692c107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.110+0000 7f69325ff700 1 -- 172.21.15.3:0/1070969661 >> 172.21.15.3:0/1070969661 conn(0x7f692c071dd0 msgr2=0x7f692c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.114+0000 7f69325ff700 1 -- 172.21.15.3:0/1070969661 shutdown_connections 2023-09-10T01:14:50.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.124+0000 7f69325ff700 1 -- 172.21.15.3:0/1070969661 wait complete. 2023-09-10T01:14:50.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.125+0000 7f69325ff700 1 Processor -- start 2023-09-10T01:14:50.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.126+0000 7f69325ff700 1 -- start start 2023-09-10T01:14:50.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.126+0000 7f69325ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107600 0x7f692c082c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.126+0000 7f69325ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c107fc0 0x7f692c0831d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.126+0000 7f69325ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f692c10cd80 0x7f692c07cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.126+0000 7f69325ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c07d510 con 0x7f692c10cd80 2023-09-10T01:14:50.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.126+0000 7f69325ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c07d680 con 0x7f692c107600 2023-09-10T01:14:50.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.126+0000 7f69325ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c07d840 con 0x7f692c107fc0 2023-09-10T01:14:50.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.127+0000 7f692bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107600 0x7f692c082c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.127+0000 7f692bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107600 0x7f692c082c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34260/0 (socket says 172.21.15.3:34260) 2023-09-10T01:14:50.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.127+0000 7f692bfff700 1 -- 172.21.15.3:0/3403791814 learned_addr learned my addr 172.21.15.3:0/3403791814 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.128+0000 7f6930b9c700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f692c10cd80 0x7f692c07cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.129+0000 7f6930b9c700 1 -- 172.21.15.3:0/3403791814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c107fc0 msgr2=0x7f692c0831d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:14:50.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.129+0000 7f6930b9c700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c107fc0 0x7f692c0831d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.129+0000 7f6930b9c700 1 -- 172.21.15.3:0/3403791814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107600 msgr2=0x7f692c082c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.129+0000 7f6930b9c700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107600 0x7f692c082c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.129+0000 7f6930b9c700 1 -- 172.21.15.3:0/3403791814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924007ce0 con 0x7f692c10cd80 2023-09-10T01:14:50.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.131+0000 7f6930b9c700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f692c10cd80 0x7f692c07cd40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f69240080b0 tx=0x7f692400f7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.134+0000 7f69297fa700 1 -- 172.21.15.3:0/3403791814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692401d070 con 0x7f692c10cd80 2023-09-10T01:14:50.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.134+0000 7f69297fa700 1 -- 172.21.15.3:0/3403791814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6924018480 con 0x7f692c10cd80 2023-09-10T01:14:50.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.134+0000 7f69297fa700 1 -- 172.21.15.3:0/3403791814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924010ea0 con 0x7f692c10cd80 2023-09-10T01:14:50.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.134+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c07dac0 con 0x7f692c10cd80 2023-09-10T01:14:50.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.134+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c1baa50 con 0x7f692c10cd80 2023-09-10T01:14:50.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.142+0000 7f6945fb6b80 1 Processor -- start 2023-09-10T01:14:50.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.143+0000 7f6945fb6b80 1 -- start start 2023-09-10T01:14:50.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.143+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473329790 0x560473329b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.143+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473333980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.143+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x5604733362b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.143+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604732827b0 con 0x560473333ec0 2023-09-10T01:14:50.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.143+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604732800d0 con 0x560473329790 2023-09-10T01:14:50.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.143+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56047327b9c0 con 0x56047332a0b0 2023-09-10T01:14:50.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.144+0000 7f692b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473329790 0x560473329b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.144+0000 7f692b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473329790 0x560473329b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34276/0 (socket says 172.21.15.3:34276) 2023-09-10T01:14:50.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.144+0000 7f692b7fe700 1 -- 172.21.15.3:0/3422047667 learned_addr learned my addr 172.21.15.3:0/3422047667 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.145+0000 7f692b7fe700 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x5604733362b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.146+0000 7f692b7fe700 1 -- 172.21.15.3:0/3422047667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 msgr2=0x560473333980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.146+0000 7f692b7fe700 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473333980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.146+0000 7f692b7fe700 1 -- 172.21.15.3:0/3422047667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 msgr2=0x5604733362b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.146+0000 7f692b7fe700 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x5604733362b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.146+0000 7f692b7fe700 1 -- 172.21.15.3:0/3422047667 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604730a7e60 con 0x560473329790 2023-09-10T01:14:50.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.146+0000 7f692b7fe700 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473329790 0x560473329b70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f691c00d080 tx=0x7f691c00b6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3419d72348901381 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.147+0000 7f69127fc700 1 -- 172.21.15.3:0/3422047667 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c00d910 con 0x560473329790 2023-09-10T01:14:50.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.147+0000 7f69127fc700 1 -- 172.21.15.3:0/3422047667 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f691c011070 con 0x560473329790 2023-09-10T01:14:50.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.147+0000 7f69127fc700 1 -- 172.21.15.3:0/3422047667 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c006430 con 0x560473329790 2023-09-10T01:14:50.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.148+0000 7f6945fb6b80 1 -- 172.21.15.3:0/3422047667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473329790 msgr2=0x560473329b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.148+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473329790 0x560473329b70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f691c00d080 tx=0x7f691c00b6b0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.150+0000 7f6945fb6b80 1 -- 172.21.15.3:0/3422047667 shutdown_connections 2023-09-10T01:14:50.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.150+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x5604733362b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.150+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473329790 0x560473329b70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.150+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/3422047667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473333980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.150+0000 7f6945fb6b80 1 -- 172.21.15.3:0/3422047667 >> 172.21.15.3:0/3422047667 conn(0x56047329e110 msgr2=0x560473324c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.152+0000 7f69297fa700 1 -- 172.21.15.3:0/3403791814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6924003a00 con 0x7f692c10cd80 2023-09-10T01:14:50.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.155+0000 7f6945fb6b80 1 -- 172.21.15.3:0/3422047667 shutdown_connections 2023-09-10T01:14:50.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.155+0000 7f6945fb6b80 1 -- 172.21.15.3:0/3422047667 wait complete. 2023-09-10T01:14:50.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.156+0000 7f69297fa700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f69140a08d0 0x7f69140a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.156+0000 7f69297fa700 1 -- 172.21.15.3:0/3403791814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f6924013070 con 0x7f692c10cd80 2023-09-10T01:14:50.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.156+0000 7f6945fb6b80 1 -- start start 2023-09-10T01:14:50.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.157+0000 7f6945fb6b80 1 Processor -- start 2023-09-10T01:14:50.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.158+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473329790 0x5604733338b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.158+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473330ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.158+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 0x5604733310d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.158+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560473219bf0 con 0x560473329790 2023-09-10T01:14:50.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.158+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604732198f0 con 0x560473333ec0 2023-09-10T01:14:50.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.158+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604730fd9a0 con 0x56047332a0b0 2023-09-10T01:14:50.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.158+0000 7f6945fb6b80 10 client.0 _ll_register_callbacks cb 0x560473163b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:50.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.160+0000 7f692b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473330ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.160+0000 7f692bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473329790 0x5604733338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.160+0000 7f692bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473329790 0x5604733338b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36594/0 (socket says 172.21.15.3:36594) 2023-09-10T01:14:50.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.160+0000 7f692bfff700 1 -- 172.21.15.3:0/980680288 learned_addr learned my addr 172.21.15.3:0/980680288 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.160+0000 7f692bfff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f69140a08d0 0x7f69140a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.161+0000 7f692b7fe700 1 -- 172.21.15.3:0/980680288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 msgr2=0x5604733310d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.161+0000 7f692b7fe700 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 0x5604733310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.161+0000 7f692b7fe700 1 -- 172.21.15.3:0/980680288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473329790 msgr2=0x5604733338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.161+0000 7f692b7fe700 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473329790 0x5604733338b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.161+0000 7f692b7fe700 1 -- 172.21.15.3:0/980680288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c013040 con 0x56047332a0b0 2023-09-10T01:14:50.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.162+0000 7f692b7fe700 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473330ac0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f691c00d720 tx=0x7f691c00b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9a89ad111f912c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.162+0000 7f690b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:50.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.162+0000 7f692bfff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f69140a08d0 0x7f69140a2d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69200117a0 tx=0x7f6920009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.162+0000 7f690b7fe700 1 -- 172.21.15.3:0/980680288 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c0060d0 con 0x56047332a0b0 2023-09-10T01:14:50.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.162+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604731d3570 con 0x56047332a0b0 2023-09-10T01:14:50.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.162+0000 7f690b7fe700 1 -- 172.21.15.3:0/980680288 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f691c006230 con 0x56047332a0b0 2023-09-10T01:14:50.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.163+0000 7f690b7fe700 1 -- 172.21.15.3:0/980680288 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f691c0063f0 con 0x56047332a0b0 2023-09-10T01:14:50.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.163+0000 7f6945fb6b80 10 client.28135 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:50.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.163+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x560473090ce0 con 0x56047332a0b0 2023-09-10T01:14:50.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f690b7fe700 1 -- 172.21.15.3:0/980680288 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5604731d3570 con 0x56047332a0b0 2023-09-10T01:14:50.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f690b7fe700 1 -- 172.21.15.3:0/980680288 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f691c0173c0 con 0x56047332a0b0 2023-09-10T01:14:50.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f6945fb6b80 10 client.28135 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:50.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f6945fb6b80 10 client.28135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:50.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f6945fb6b80 20 client.28135 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:50.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473343a30 0x560473345e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f6945fb6b80 4 client.28135 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:50.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.164+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560473290920 con 0x560473343a30 2023-09-10T01:14:50.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.167+0000 7f692bfff700 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473343a30 0x560473345e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.173+0000 7f692bfff700 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473343a30 0x560473345e50 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a86828a7faef865 server_cookie=b013f0fd1a17e1ef in_seq=0 out_seq=0 2023-09-10T01:14:50.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.174+0000 7f690b7fe700 10 client.28135 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.175+0000 7f690b7fe700 1 -- 172.21.15.3:0/980680288 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560473290920 con 0x560473343a30 2023-09-10T01:14:50.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.175+0000 7f690b7fe700 10 client.28135 handle_command_reply: tid=0 2023-09-10T01:14:50.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.175+0000 7f6945fb6b80 1 client.28135 shutdown 2023-09-10T01:14:50.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.175+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 msgr2=0x560473330ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.175+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473330ac0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f691c00d720 tx=0x7f691c00b7c0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.177+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 shutdown_connections 2023-09-10T01:14:50.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.177+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473343a30 0x560473345e50 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.178+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 0x5604733310d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.178+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473329790 0x5604733338b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.178+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/980680288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56047332a0b0 0x560473330ac0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.178+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 >> 172.21.15.3:0/980680288 conn(0x56047329e110 msgr2=0x560473324860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.178+0000 7f690b7fe700 0 client.28135 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.183+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 shutdown_connections 2023-09-10T01:14:50.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.187+0000 7f6945fb6b80 1 -- 172.21.15.3:0/980680288 wait complete. 2023-09-10T01:14:50.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.188+0000 7f6945fb6b80 20 client.28135 trim_cache size 0 max 16384 2023-09-10T01:14:50.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 Processor -- start 2023-09-10T01:14:50.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 -- start start 2023-09-10T01:14:50.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473330280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56047332a0b0 0x5604733333e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 0x56047332dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560473340360 con 0x56047332a0b0 2023-09-10T01:14:50.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56047332f020 con 0x560473333ec0 2023-09-10T01:14:50.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604733307c0 con 0x560473329790 2023-09-10T01:14:50.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473330280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473330280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54434/0 (socket says 172.21.15.3:54434) 2023-09-10T01:14:50.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 -- 172.21.15.3:0/920206960 learned_addr learned my addr 172.21.15.3:0/920206960 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692b7fe700 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56047332a0b0 0x5604733333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f6930b9c700 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 0x56047332dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 -- 172.21.15.3:0/920206960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 msgr2=0x56047332dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 0x56047332dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 -- 172.21.15.3:0/920206960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56047332a0b0 msgr2=0x5604733333e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56047332a0b0 0x5604733333e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.217+0000 7f692bfff700 1 -- 172.21.15.3:0/920206960 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56047334aa80 con 0x560473329790 2023-09-10T01:14:50.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.218+0000 7f692b7fe700 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56047332a0b0 0x5604733333e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:50.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.218+0000 7f692bfff700 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473330280 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x560473346730 tx=0x7f6920008000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d5132c866769d5d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.218+0000 7f6912ffd700 1 -- 172.21.15.3:0/920206960 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692002d070 con 0x560473329790 2023-09-10T01:14:50.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.218+0000 7f6912ffd700 1 -- 172.21.15.3:0/920206960 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69200113d0 con 0x560473329790 2023-09-10T01:14:50.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.218+0000 7f6912ffd700 1 -- 172.21.15.3:0/920206960 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920026070 con 0x560473329790 2023-09-10T01:14:50.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 -- 172.21.15.3:0/920206960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 msgr2=0x560473330280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473330280 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x560473346730 tx=0x7f6920008000 comp rx=0 tx=0).stop 2023-09-10T01:14:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 -- 172.21.15.3:0/920206960 shutdown_connections 2023-09-10T01:14:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473330280 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560473333ec0 0x56047332dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/920206960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56047332a0b0 0x5604733333e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 -- 172.21.15.3:0/920206960 >> 172.21.15.3:0/920206960 conn(0x56047329e110 msgr2=0x56047332bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 -- 172.21.15.3:0/920206960 shutdown_connections 2023-09-10T01:14:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 -- 172.21.15.3:0/920206960 wait complete. 2023-09-10T01:14:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.219+0000 7f6945fb6b80 1 -- start start 2023-09-10T01:14:50.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.220+0000 7f6945fb6b80 1 Processor -- start 2023-09-10T01:14:50.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.220+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473391fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56047332a0b0 0x560473392500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x560473339c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56047333a3a0 con 0x560473333ec0 2023-09-10T01:14:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56047333a510 con 0x56047332a0b0 2023-09-10T01:14:50.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56047333a6d0 con 0x560473329790 2023-09-10T01:14:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 10 client.0 _ll_register_callbacks cb 0x7f691c00be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56047332a0b0 0x560473392500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473391fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56047332a0b0 0x560473392500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34306/0 (socket says 172.21.15.3:34306) 2023-09-10T01:14:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 -- 172.21.15.3:0/1543794742 learned_addr learned my addr 172.21.15.3:0/1543794742 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6930b9c700 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x560473339c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 -- 172.21.15.3:0/1543794742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 msgr2=0x560473391fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473391fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 -- 172.21.15.3:0/1543794742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 msgr2=0x560473339c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x560473339c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 -- 172.21.15.3:0/1543794742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560473090ce0 con 0x56047332a0b0 2023-09-10T01:14:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692bfff700 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473391fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:14:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f692b7fe700 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56047332a0b0 0x560473392500 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f691c00d720 tx=0x7f691c00b6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6667faf84f7d24f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f690b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:14:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f690b7fe700 1 -- 172.21.15.3:0/1543794742 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c004ea0 con 0x56047332a0b0 2023-09-10T01:14:50.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f691c011070 con 0x56047332a0b0 2023-09-10T01:14:50.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f690b7fe700 1 -- 172.21.15.3:0/1543794742 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f691c0060d0 con 0x56047332a0b0 2023-09-10T01:14:50.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f690b7fe700 1 -- 172.21.15.3:0/1543794742 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f691c0175d0 con 0x56047332a0b0 2023-09-10T01:14:50.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 10 client.28256 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:50.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f6920019040 con 0x56047332a0b0 2023-09-10T01:14:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.222+0000 7f690b7fe700 1 -- 172.21.15.3:0/1543794742 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f691c011070 con 0x56047332a0b0 2023-09-10T01:14:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f690b7fe700 1 -- 172.21.15.3:0/1543794742 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f691c02a070 con 0x56047332a0b0 2023-09-10T01:14:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f6945fb6b80 10 client.28256 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f6945fb6b80 10 client.28256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:50.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f6945fb6b80 20 client.28256 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473399a90 0x56047339beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f6945fb6b80 4 client.28256 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:14:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5604732749a0 con 0x560473399a90 2023-09-10T01:14:50.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f692bfff700 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473399a90 0x56047339beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f692bfff700 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473399a90 0x56047339beb0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bfaead7bcb52b43 server_cookie=939694ddb7b7051b in_seq=0 out_seq=0 2023-09-10T01:14:50.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.223+0000 7f690b7fe700 10 client.28256 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.225+0000 7f690b7fe700 1 -- 172.21.15.3:0/1543794742 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5604732749a0 con 0x560473399a90 2023-09-10T01:14:50.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.225+0000 7f690b7fe700 10 client.28256 handle_command_reply: tid=0 2023-09-10T01:14:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.225+0000 7f6945fb6b80 1 client.28256 shutdown 2023-09-10T01:14:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56047332a0b0 msgr2=0x560473392500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56047332a0b0 0x560473392500 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f691c00d720 tx=0x7f691c00b6b0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 shutdown_connections 2023-09-10T01:14:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x560473399a90 0x56047339beb0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560473329790 0x560473391fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560473333ec0 0x560473339c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 --2- 172.21.15.3:0/1543794742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56047332a0b0 0x560473392500 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 >> 172.21.15.3:0/1543794742 conn(0x56047329e110 msgr2=0x560473328780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f690b7fe700 0 client.28256 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 shutdown_connections 2023-09-10T01:14:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 1 -- 172.21.15.3:0/1543794742 wait complete. 2023-09-10T01:14:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.226+0000 7f6945fb6b80 20 client.28256 trim_cache size 0 max 16384 2023-09-10T01:14:50.240 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:14:50.241 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:14:50.241 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "e10af8b6-b1f5-41ef-a4f3-f8ce4f140572", 2023-09-10T01:14:50.241 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:14:50.241 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:14:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f69140a08d0 msgr2=0x7f69140a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f69140a08d0 0x7f69140a2d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69200117a0 tx=0x7f6920009260 comp rx=0 tx=0).stop 2023-09-10T01:14:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f692c10cd80 msgr2=0x7f692c07cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f692c10cd80 0x7f692c07cd40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f69240080b0 tx=0x7f692400f7c0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 shutdown_connections 2023-09-10T01:14:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f69140a08d0 0x7f69140a2d90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f692c10cd80 0x7f692c07cd40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f692c107600 0x7f692c082c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 --2- 172.21.15.3:0/3403791814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c107fc0 0x7f692c0831d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 >> 172.21.15.3:0/3403791814 conn(0x7f692c071dd0 msgr2=0x7f692c06d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 shutdown_connections 2023-09-10T01:14:50.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.227+0000 7f69325ff700 1 -- 172.21.15.3:0/3403791814 wait complete. 2023-09-10T01:14:50.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:14:50.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 -- 172.21.15.3:0/2524637803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100fc0 msgr2=0x7f3d78105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/2524637803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100fc0 0x7f3d78105000 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3d680098f0 tx=0x7f3d68009c00 comp rx=0 tx=0).stop 2023-09-10T01:14:50.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 -- 172.21.15.3:0/2524637803 shutdown_connections 2023-09-10T01:14:50.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/2524637803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 0x7f3d7810d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/2524637803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d78100610 0x7f3d781009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/2524637803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100fc0 0x7f3d78105000 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 -- 172.21.15.3:0/2524637803 >> 172.21.15.3:0/2524637803 conn(0x7f3d780fbe90 msgr2=0x7f3d780fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 -- 172.21.15.3:0/2524637803 shutdown_connections 2023-09-10T01:14:50.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.473+0000 7f3d7e59d700 1 -- 172.21.15.3:0/2524637803 wait complete. 2023-09-10T01:14:50.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.475+0000 7f3d7e59d700 1 Processor -- start 2023-09-10T01:14:50.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.475+0000 7f3d7e59d700 1 -- start start 2023-09-10T01:14:50.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7e59d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100610 0x7f3d7819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7e59d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 0x7f3d7819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7e59d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d78198670 0x7f3d78198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7e59d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781990f0 con 0x7f3d78198670 2023-09-10T01:14:50.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7e59d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d78199230 con 0x7f3d78105730 2023-09-10T01:14:50.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7e59d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781993f0 con 0x7f3d78100610 2023-09-10T01:14:50.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 0x7f3d7819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7d59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100610 0x7f3d7819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7dd9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d78198670 0x7f3d78198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 0x7f3d7819eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:34338/0 (socket says 172.21.15.3:34338) 2023-09-10T01:14:50.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7d59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100610 0x7f3d7819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54464/0 (socket says 172.21.15.3:54464) 2023-09-10T01:14:50.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 -- 172.21.15.3:0/108158414 learned_addr learned my addr 172.21.15.3:0/108158414 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 -- 172.21.15.3:0/108158414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100610 msgr2=0x7f3d7819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100610 0x7f3d7819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 -- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d78198670 msgr2=0x7f3d78198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d78198670 0x7f3d78198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.476+0000 7f3d7cd9a700 1 -- 172.21.15.3:0/108158414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d68009520 con 0x7f3d78105730 2023-09-10T01:14:50.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.477+0000 7f3d7cd9a700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 0x7f3d7819eb40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009fd0 tx=0x7f3d6800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.477+0000 7f3d6e7fc700 1 -- 172.21.15.3:0/108158414 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6801c070 con 0x7f3d78105730 2023-09-10T01:14:50.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.477+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d781a5410 con 0x7f3d78105730 2023-09-10T01:14:50.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.477+0000 7f3d6e7fc700 1 -- 172.21.15.3:0/108158414 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d6800ba40 con 0x7f3d78105730 2023-09-10T01:14:50.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.477+0000 7f3d6e7fc700 1 -- 172.21.15.3:0/108158414 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68004950 con 0x7f3d78105730 2023-09-10T01:14:50.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.477+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d781a58e0 con 0x7f3d78105730 2023-09-10T01:14:50.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.480+0000 7f3d6e7fc700 1 -- 172.21.15.3:0/108158414 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3d68020070 con 0x7f3d78105730 2023-09-10T01:14:50.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 Processor -- start 2023-09-10T01:14:50.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 -- start start 2023-09-10T01:14:50.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae10ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae115af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d6e7fc700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d640a0830 0x7f3d640a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae118420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae064730 con 0x5638ae10b180 2023-09-10T01:14:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae062050 con 0x5638ae116030 2023-09-10T01:14:50.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d93132b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae05d940 con 0x5638ae10a860 2023-09-10T01:14:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d6e7fc700 1 -- 172.21.15.3:0/108158414 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3d680ccff0 con 0x7f3d78105730 2023-09-10T01:14:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d7dd9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae10ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d7dd9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae10ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54478/0 (socket says 172.21.15.3:54478) 2023-09-10T01:14:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.482+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/2205690153 learned_addr learned my addr 172.21.15.3:0/2205690153 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae118420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7cd9a700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d640a0830 0x7f3d640a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/2205690153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 msgr2=0x5638ae118420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae118420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/2205690153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 msgr2=0x5638ae115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/2205690153 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638ade89de0 con 0x5638ae10a860 2023-09-10T01:14:50.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae10ac40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3d70008cb0 tx=0x7f3d700099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4276209828d47eb5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.483+0000 7f3d637fe700 1 -- 172.21.15.3:0/2205690153 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7000e050 con 0x5638ae10a860 2023-09-10T01:14:50.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.484+0000 7f3d7cd9a700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d640a0830 0x7f3d640a2cf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3d68096b70 tx=0x7f3d680054e0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.484+0000 7f3d637fe700 1 -- 172.21.15.3:0/2205690153 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d7000f040 con 0x5638ae10a860 2023-09-10T01:14:50.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.484+0000 7f3d637fe700 1 -- 172.21.15.3:0/2205690153 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70018ba0 con 0x5638ae10a860 2023-09-10T01:14:50.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.484+0000 7f3d93132b80 1 -- 172.21.15.3:0/2205690153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 msgr2=0x5638ae10ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.484+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae10ac40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3d70008cb0 tx=0x7f3d700099e0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 -- 172.21.15.3:0/2205690153 shutdown_connections 2023-09-10T01:14:50.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae10ac40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae118420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2205690153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 -- 172.21.15.3:0/2205690153 >> 172.21.15.3:0/2205690153 conn(0x5638ae080090 msgr2=0x5638ae106d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 -- 172.21.15.3:0/2205690153 shutdown_connections 2023-09-10T01:14:50.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 -- 172.21.15.3:0/2205690153 wait complete. 2023-09-10T01:14:50.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.485+0000 7f3d93132b80 1 -- start start 2023-09-10T01:14:50.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.486+0000 7f3d93132b80 1 Processor -- start 2023-09-10T01:14:50.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae115a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae112cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae113430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d93132b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638adffbb70 con 0x5638ae10a860 2023-09-10T01:14:50.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d93132b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638adffb870 con 0x5638ae116030 2023-09-10T01:14:50.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d93132b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638adedf920 con 0x5638ae10b180 2023-09-10T01:14:50.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d93132b80 10 client.0 _ll_register_callbacks cb 0x5638adf45a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:50.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae112cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae112cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54494/0 (socket says 172.21.15.3:54494) 2023-09-10T01:14:50.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2976191800 learned_addr learned my addr 172.21.15.3:0/2976191800 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae113430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 msgr2=0x5638ae113430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae113430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2976191800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 msgr2=0x5638ae115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2976191800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d70008990 con 0x5638ae10b180 2023-09-10T01:14:50.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.487+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae113430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:14:50.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae112cf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3d74012810 tx=0x7f3d74012bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9418e9298e8605f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d60ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:50.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2976191800 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74017070 con 0x5638ae10b180 2023-09-10T01:14:50.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5638adfb54f0 con 0x5638ae10b180 2023-09-10T01:14:50.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2976191800 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3d7400f5c0 con 0x5638ae10b180 2023-09-10T01:14:50.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2976191800 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d740122e0 con 0x5638ae10b180 2023-09-10T01:14:50.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d93132b80 10 client.28153 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:50.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.488+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5638ade72c60 con 0x5638ae10b180 2023-09-10T01:14:50.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2976191800 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5638adfb54f0 con 0x5638ae10b180 2023-09-10T01:14:50.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2976191800 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3d74010960 con 0x5638ae10b180 2023-09-10T01:14:50.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d93132b80 10 client.28153 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:50.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d93132b80 10 client.28153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:50.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d93132b80 20 client.28153 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:50.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae125b40 0x5638ae127f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d93132b80 4 client.28153 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:14:50.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638ae0728a0 con 0x5638ae125b40 2023-09-10T01:14:50.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.489+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae125b40 0x5638ae127f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.490+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae125b40 0x5638ae127f60 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a2d360573b72d10 server_cookie=f362ed5d7a134151 in_seq=0 out_seq=0 2023-09-10T01:14:50.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.490+0000 7f3d60ff9700 10 client.28153 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.490+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2976191800 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5638ae0728a0 con 0x5638ae125b40 2023-09-10T01:14:50.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.490+0000 7f3d60ff9700 10 client.28153 handle_command_reply: tid=0 2023-09-10T01:14:50.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.490+0000 7f3d93132b80 1 client.28153 shutdown 2023-09-10T01:14:50.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 msgr2=0x5638ae112cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae112cf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3d74012810 tx=0x7f3d74012bd0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 shutdown_connections 2023-09-10T01:14:50.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae125b40 0x5638ae127f60 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae113430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d60ff9700 0 client.28153 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2976191800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae112cf0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 >> 172.21.15.3:0/2976191800 conn(0x5638ae080090 msgr2=0x5638ae1068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 shutdown_connections 2023-09-10T01:14:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 1 -- 172.21.15.3:0/2976191800 wait complete. 2023-09-10T01:14:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.491+0000 7f3d93132b80 20 client.28153 trim_cache size 0 max 16384 2023-09-10T01:14:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 Processor -- start 2023-09-10T01:14:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 -- start start 2023-09-10T01:14:50.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae115610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae1124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae112a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae10db80 con 0x5638ae10b180 2023-09-10T01:14:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae111290 con 0x5638ae116030 2023-09-10T01:14:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d93132b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae10d700 con 0x5638ae10a860 2023-09-10T01:14:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d7dd9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae115610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d7dd9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae115610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54510/0 (socket says 172.21.15.3:54510) 2023-09-10T01:14:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/4098200487 learned_addr learned my addr 172.21.15.3:0/4098200487 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae1124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.511+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae112a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/4098200487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 msgr2=0x5638ae112a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae112a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/4098200487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 msgr2=0x5638ae1124f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae1124f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d7dd9c700 1 -- 172.21.15.3:0/4098200487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638adeb0250 con 0x5638ae10a860 2023-09-10T01:14:50.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae115610 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3d70008c80 tx=0x7f3d70009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c4a83a9ceea6bd0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d63fff700 1 -- 172.21.15.3:0/4098200487 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7000e040 con 0x5638ae10a860 2023-09-10T01:14:50.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d63fff700 1 -- 172.21.15.3:0/4098200487 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d70030070 con 0x5638ae10a860 2023-09-10T01:14:50.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.512+0000 7f3d63fff700 1 -- 172.21.15.3:0/4098200487 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d700283c0 con 0x5638ae10a860 2023-09-10T01:14:50.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 -- 172.21.15.3:0/4098200487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 msgr2=0x5638ae115610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae115610 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3d70008c80 tx=0x7f3d70009e50 comp rx=0 tx=0).stop 2023-09-10T01:14:50.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 -- 172.21.15.3:0/4098200487 shutdown_connections 2023-09-10T01:14:50.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10a860 0x5638ae115610 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae112a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 --2- 172.21.15.3:0/4098200487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10b180 0x5638ae1124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 -- 172.21.15.3:0/4098200487 >> 172.21.15.3:0/4098200487 conn(0x5638ae080090 msgr2=0x5638ae10e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 -- 172.21.15.3:0/4098200487 shutdown_connections 2023-09-10T01:14:50.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.513+0000 7f3d93132b80 1 -- 172.21.15.3:0/4098200487 wait complete. 2023-09-10T01:14:50.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.514+0000 7f3d93132b80 1 -- start start 2023-09-10T01:14:50.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.514+0000 7f3d93132b80 1 Processor -- start 2023-09-10T01:14:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae11c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae11ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae11b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae11b6c0 con 0x5638ae10a860 2023-09-10T01:14:50.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae11b800 con 0x5638ae116030 2023-09-10T01:14:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638ae11b9c0 con 0x5638ae10b180 2023-09-10T01:14:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 10 client.0 _ll_register_callbacks cb 0x7f3d74009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:14:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae11ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7dd9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae11c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae11ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54516/0 (socket says 172.21.15.3:54516) 2023-09-10T01:14:50.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7dd9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae11c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36660/0 (socket says 172.21.15.3:36660) 2023-09-10T01:14:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2432371230 learned_addr learned my addr 172.21.15.3:0/2432371230 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:14:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae11b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 msgr2=0x5638ae11b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae11b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2432371230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 msgr2=0x5638ae11c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:14:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae11c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 -- 172.21.15.3:0/2432371230 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638ade89de0 con 0x5638ae10b180 2023-09-10T01:14:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae11b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:14:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d7d59b700 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae11ac40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009a70 tx=0x7f3d74005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda56463d431f6cf server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:14:50.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d60ff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:14:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2432371230 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74017070 con 0x5638ae10b180 2023-09-10T01:14:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d7000f040 con 0x5638ae10b180 2023-09-10T01:14:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2432371230 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3d74000e40 con 0x5638ae10b180 2023-09-10T01:14:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2432371230 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d740092d0 con 0x5638ae10b180 2023-09-10T01:14:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 10 client.28159 fetch_fsmap learned FSMap version 35 2023-09-10T01:14:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5638ade882b0 con 0x5638ae10b180 2023-09-10T01:14:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.516+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2432371230 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d7000f040 con 0x5638ae10b180 2023-09-10T01:14:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2432371230 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3d74033070 con 0x5638ae10b180 2023-09-10T01:14:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d93132b80 10 client.28159 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:14:50.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d93132b80 10 client.28159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:14:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d93132b80 20 client.28159 populate_metadata read hostname 'smithi003' 2023-09-10T01:14:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae17ae70 0x5638ae17d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:14:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d93132b80 4 client.28159 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:14:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5638ae056920 con 0x5638ae17ae70 2023-09-10T01:14:50.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae17ae70 0x5638ae17d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:14:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d7dd9c700 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae17ae70 0x5638ae17d290 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d086ded3cb064bab server_cookie=10062f0ecbde996d in_seq=0 out_seq=0 2023-09-10T01:14:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d60ff9700 10 client.28159 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d60ff9700 1 -- 172.21.15.3:0/2432371230 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f3d7002c110 con 0x5638ae17ae70 2023-09-10T01:14:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d60ff9700 10 client.28159 handle_command_reply: tid=0 2023-09-10T01:14:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.517+0000 7f3d93132b80 1 client.28159 shutdown 2023-09-10T01:14:50.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 msgr2=0x5638ae11ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae11ac40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009a70 tx=0x7f3d74005730 comp rx=0 tx=0).stop 2023-09-10T01:14:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 shutdown_connections 2023-09-10T01:14:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5638ae17ae70 0x5638ae17d290 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5638ae116030 0x5638ae11b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5638ae10a860 0x5638ae11c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d60ff9700 0 client.28159 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:14:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 --2- 172.21.15.3:0/2432371230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5638ae10b180 0x5638ae11ac40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 >> 172.21.15.3:0/2432371230 conn(0x5638ae080090 msgr2=0x5638ae10c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 shutdown_connections 2023-09-10T01:14:50.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 1 -- 172.21.15.3:0/2432371230 wait complete. 2023-09-10T01:14:50.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.518+0000 7f3d93132b80 20 client.28159 trim_cache size 0 max 16384 2023-09-10T01:14:50.534 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:14:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:14:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:14:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "e10af8b6-b1f5-41ef-a4f3-f8ce4f140572": { 2023-09-10T01:14:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:14:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "e10af8b6-b1f5-41ef-a4f3-f8ce4f140572", 2023-09-10T01:14:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:14:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:14:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:14:50.536 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:14:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d640a0830 msgr2=0x7f3d640a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d640a0830 0x7f3d640a2cf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3d68096b70 tx=0x7f3d680054e0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 msgr2=0x7f3d7819eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:14:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 0x7f3d7819eb40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009fd0 tx=0x7f3d6800fa00 comp rx=0 tx=0).stop 2023-09-10T01:14:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 shutdown_connections 2023-09-10T01:14:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d640a0830 0x7f3d640a2cf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d78100610 0x7f3d7819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d78198670 0x7f3d78198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 --2- 172.21.15.3:0/108158414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d78105730 0x7f3d7819eb40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:14:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 >> 172.21.15.3:0/108158414 conn(0x7f3d780fbe90 msgr2=0x7f3d780fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:14:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 shutdown_connections 2023-09-10T01:14:50.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:14:50.519+0000 7f3d7e59d700 1 -- 172.21.15.3:0/108158414 wait complete. 2023-09-10T01:14:50.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:50 smithi003 ceph-mon[97654]: pgmap v361: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2023-09-10T01:14:50.542 INFO:tasks.cephfs.filesystem:scrub status for tag:e10af8b6-b1f5-41ef-a4f3-f8ce4f140572 - {'path': '/', 'tag': 'e10af8b6-b1f5-41ef-a4f3-f8ce4f140572', 'options': 'recursive,force'} 2023-09-10T01:14:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:50 smithi136 ceph-mon[133708]: pgmap v361: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2023-09-10T01:14:50.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:50 smithi092 ceph-mon[134067]: pgmap v361: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2023-09-10T01:14:50.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 971433 7.97 MB/sec execute 512 sec latency 2014.674 ms 2023-09-10T01:14:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:14:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:14:51.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:51 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:14:51.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:14:51.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:51 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:14:51.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:14:51.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:51 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:14:51.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 973219 7.98 MB/sec execute 513 sec latency 34.270 ms 2023-09-10T01:14:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:52 smithi003 ceph-mon[97654]: pgmap v362: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 726 KiB/s wr, 89 op/s 2023-09-10T01:14:52.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:52 smithi136 ceph-mon[133708]: pgmap v362: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 726 KiB/s wr, 89 op/s 2023-09-10T01:14:52.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:52 smithi092 ceph-mon[134067]: pgmap v362: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 726 KiB/s wr, 89 op/s 2023-09-10T01:14:52.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 974201 7.97 MB/sec execute 514 sec latency 45.855 ms 2023-09-10T01:14:53.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 975338 7.96 MB/sec execute 515 sec latency 240.462 ms 2023-09-10T01:14:54.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:54 smithi003 ceph-mon[97654]: pgmap v363: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 726 KiB/s wr, 89 op/s 2023-09-10T01:14:54.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:54 smithi136 ceph-mon[133708]: pgmap v363: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 726 KiB/s wr, 89 op/s 2023-09-10T01:14:54.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:54 smithi092 ceph-mon[134067]: pgmap v363: 129 pgs: 129 active+clean; 294 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 726 KiB/s wr, 89 op/s 2023-09-10T01:14:54.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 977234 7.97 MB/sec execute 516 sec latency 14.459 ms 2023-09-10T01:14:55.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 977807 7.96 MB/sec execute 517 sec latency 203.621 ms 2023-09-10T01:14:56.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:56 smithi003 ceph-mon[97654]: pgmap v364: 129 pgs: 129 active+clean; 296 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2023-09-10T01:14:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:56 smithi136 ceph-mon[133708]: pgmap v364: 129 pgs: 129 active+clean; 296 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2023-09-10T01:14:56.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:56 smithi092 ceph-mon[134067]: pgmap v364: 129 pgs: 129 active+clean; 296 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 187 op/s 2023-09-10T01:14:56.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 977807 7.95 MB/sec execute 518 sec latency 1203.805 ms 2023-09-10T01:14:57.804 INFO:tasks.workunit.client.0.smithi003.stdout: 1 977807 7.93 MB/sec execute 519 sec latency 2203.928 ms 2023-09-10T01:14:58.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:58 smithi003 ceph-mon[97654]: pgmap v365: 129 pgs: 129 active+clean; 296 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2023-09-10T01:14:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:58 smithi136 ceph-mon[133708]: pgmap v365: 129 pgs: 129 active+clean; 296 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2023-09-10T01:14:58.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:58 smithi092 ceph-mon[134067]: pgmap v365: 129 pgs: 129 active+clean; 296 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2023-09-10T01:14:58.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 978712 7.92 MB/sec execute 520 sec latency 2504.572 ms 2023-09-10T01:14:59.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:59 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:14:59.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:14:59 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:14:59.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:59 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:14:59.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:14:59 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:14:59.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:59 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:14:59.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:14:59 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:14:59.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 980622 7.93 MB/sec execute 521 sec latency 17.520 ms 2023-09-10T01:15:00.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:00 smithi003 ceph-mon[97654]: pgmap v366: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.2 MiB/s wr, 159 op/s 2023-09-10T01:15:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:00 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:15:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:00 smithi136 ceph-mon[133708]: pgmap v366: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.2 MiB/s wr, 159 op/s 2023-09-10T01:15:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:00 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:15:00.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:00 smithi092 ceph-mon[134067]: pgmap v366: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.2 MiB/s wr, 159 op/s 2023-09-10T01:15:00.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:00 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:15:00.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 981280 7.92 MB/sec execute 522 sec latency 151.996 ms 2023-09-10T01:15:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.927+0000 7f759adff700 1 -- 172.21.15.3:0/2571227848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75941005c0 msgr2=0x7f7594100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.927+0000 7f759adff700 1 --2- 172.21.15.3:0/2571227848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75941005c0 0x7f7594100a40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f75800099c0 tx=0x7f7580009cd0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.928+0000 7f759adff700 1 -- 172.21.15.3:0/2571227848 shutdown_connections 2023-09-10T01:15:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.928+0000 7f759adff700 1 --2- 172.21.15.3:0/2571227848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7594100f80 0x7f759410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.928+0000 7f759adff700 1 --2- 172.21.15.3:0/2571227848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7594106680 0x7f7594106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.928+0000 7f759adff700 1 --2- 172.21.15.3:0/2571227848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f75941005c0 0x7f7594100a40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.928+0000 7f759adff700 1 -- 172.21.15.3:0/2571227848 >> 172.21.15.3:0/2571227848 conn(0x7f75940fc0b0 msgr2=0x7f75940fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.928+0000 7f759adff700 1 -- 172.21.15.3:0/2571227848 shutdown_connections 2023-09-10T01:15:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.928+0000 7f759adff700 1 -- 172.21.15.3:0/2571227848 wait complete. 2023-09-10T01:15:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 Processor -- start 2023-09-10T01:15:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 -- start start 2023-09-10T01:15:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75941005c0 0x7f759419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7594100f80 0x7f759419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7594106680 0x7f7594198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594198d00 con 0x7f7594100f80 2023-09-10T01:15:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594198e70 con 0x7f7594106680 2023-09-10T01:15:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f759adff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7594199030 con 0x7f75941005c0 2023-09-10T01:15:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75941005c0 0x7f759419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f7593fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7594100f80 0x7f759419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.929+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75941005c0 0x7f759419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41884/0 (socket says 172.21.15.3:41884) 2023-09-10T01:15:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7598b9b700 1 -- 172.21.15.3:0/360831416 learned_addr learned my addr 172.21.15.3:0/360831416 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f759939c700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7594106680 0x7f7594198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7598b9b700 1 -- 172.21.15.3:0/360831416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7594106680 msgr2=0x7f7594198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7598b9b700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7594106680 0x7f7594198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7598b9b700 1 -- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7594100f80 msgr2=0x7f759419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7598b9b700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7594100f80 0x7f759419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7598b9b700 1 -- 172.21.15.3:0/360831416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588009520 con 0x7f75941005c0 2023-09-10T01:15:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7593fff700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7594100f80 0x7f759419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7598b9b700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75941005c0 0x7f759419e620 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f758800eb30 tx=0x7f758800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7591ffb700 1 -- 172.21.15.3:0/360831416 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588009880 con 0x7f75941005c0 2023-09-10T01:15:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75800095f0 con 0x7f75941005c0 2023-09-10T01:15:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7591ffb700 1 -- 172.21.15.3:0/360831416 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75880044e0 con 0x7f75941005c0 2023-09-10T01:15:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.930+0000 7f7591ffb700 1 -- 172.21.15.3:0/360831416 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588003d80 con 0x7f75941005c0 2023-09-10T01:15:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.931+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75941a5430 con 0x7f75941005c0 2023-09-10T01:15:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.934+0000 7f7591ffb700 1 -- 172.21.15.3:0/360831416 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7588003f20 con 0x7f75941005c0 2023-09-10T01:15:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 Processor -- start 2023-09-10T01:15:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 -- start start 2023-09-10T01:15:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f7591ffb700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f75840a07b0 0x7f75840a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb0a02580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb09fc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb09556d0 con 0x55dcb09fccc0 2023-09-10T01:15:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb0952ff0 con 0x55dcb0a021a0 2023-09-10T01:15:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f75ae658b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb094e8e0 con 0x55dcb09fc300 2023-09-10T01:15:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.935+0000 7f7591ffb700 1 -- 172.21.15.3:0/360831416 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7588014070 con 0x7f75941005c0 2023-09-10T01:15:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb0a02580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7593fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb09fc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb0a02580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:46416/0 (socket says 172.21.15.3:46416) 2023-09-10T01:15:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f759939c700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f75840a07b0 0x7f75840a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7598b9b700 1 -- 172.21.15.3:0/1439253634 learned_addr learned my addr 172.21.15.3:0/1439253634 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f759939c700 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7593fff700 1 -- 172.21.15.3:0/1439253634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 msgr2=0x55dcb0a02580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7593fff700 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb0a02580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7593fff700 1 -- 172.21.15.3:0/1439253634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 msgr2=0x55dcb0a08f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7593fff700 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7593fff700 1 -- 172.21.15.3:0/1439253634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcb077ad80 con 0x55dcb09fc300 2023-09-10T01:15:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f7593fff700 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb09fc780 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f758000b580 tx=0x7f7580005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68962f03157b1bb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.936+0000 7f759939c700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f75840a07b0 0x7f75840a2c70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f758c009ec0 tx=0x7f758c009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f757effd700 1 -- 172.21.15.3:0/1439253634 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000be00 con 0x55dcb09fc300 2023-09-10T01:15:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f757effd700 1 -- 172.21.15.3:0/1439253634 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f758001e040 con 0x55dcb09fc300 2023-09-10T01:15:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f757effd700 1 -- 172.21.15.3:0/1439253634 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580003d60 con 0x55dcb09fc300 2023-09-10T01:15:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 -- 172.21.15.3:0/1439253634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 msgr2=0x55dcb09fc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb09fc780 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f758000b580 tx=0x7f7580005fb0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 -- 172.21.15.3:0/1439253634 shutdown_connections 2023-09-10T01:15:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb0a02580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1439253634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb09fc780 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 -- 172.21.15.3:0/1439253634 >> 172.21.15.3:0/1439253634 conn(0x55dcb0971030 msgr2=0x55dcb09f7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 -- 172.21.15.3:0/1439253634 shutdown_connections 2023-09-10T01:15:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.937+0000 7f75ae658b80 1 -- 172.21.15.3:0/1439253634 wait complete. 2023-09-10T01:15:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.938+0000 7f75ae658b80 1 -- start start 2023-09-10T01:15:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.938+0000 7f75ae658b80 1 Processor -- start 2023-09-10T01:15:00.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb0a08de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a06060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb0a021a0 0x55dcb0a06690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f75ae658b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb08ecb10 con 0x55dcb09fccc0 2023-09-10T01:15:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f75ae658b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb08ec810 con 0x55dcb09fc300 2023-09-10T01:15:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f75ae658b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb07d08c0 con 0x55dcb0a021a0 2023-09-10T01:15:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f75ae658b80 10 client.0 _ll_register_callbacks cb 0x55dcb0836a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb0a08de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb0a08de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:46422/0 (socket says 172.21.15.3:46422) 2023-09-10T01:15:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 -- 172.21.15.3:0/2754983601 learned_addr learned my addr 172.21.15.3:0/2754983601 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7593fff700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a06060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f759939c700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb0a021a0 0x55dcb0a06690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 -- 172.21.15.3:0/2754983601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb0a021a0 msgr2=0x55dcb0a06690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb0a021a0 0x55dcb0a06690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 -- 172.21.15.3:0/2754983601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 msgr2=0x55dcb0a06060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a06060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f7598b9b700 1 -- 172.21.15.3:0/2754983601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759410dd40 con 0x55dcb09fc300 2023-09-10T01:15:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.939+0000 7f759939c700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb0a021a0 0x55dcb0a06690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f7598b9b700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb0a08de0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7588099690 tx=0x7f758800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74cebd400e9c348f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f7573fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f7573fff700 1 -- 172.21.15.3:0/2754983601 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880b9070 con 0x55dcb09fc300 2023-09-10T01:15:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dcb08a6490 con 0x55dcb09fc300 2023-09-10T01:15:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f7573fff700 1 -- 172.21.15.3:0/2754983601 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f758809d7c0 con 0x55dcb09fc300 2023-09-10T01:15:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f7573fff700 1 -- 172.21.15.3:0/2754983601 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75880a1a00 con 0x55dcb09fc300 2023-09-10T01:15:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f75ae658b80 10 client.28283 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55dcb0763c00 con 0x55dcb09fc300 2023-09-10T01:15:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f7573fff700 1 -- 172.21.15.3:0/2754983601 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dcb08a6490 con 0x55dcb09fc300 2023-09-10T01:15:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f7573fff700 1 -- 172.21.15.3:0/2754983601 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f75880ae020 con 0x55dcb09fc300 2023-09-10T01:15:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f75ae658b80 10 client.28283 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f75ae658b80 10 client.28283 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f75ae658b80 20 client.28283 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.940+0000 7f75ae658b80 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a163f0 0x55dcb0a18810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.941+0000 7f75ae658b80 4 client.28283 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.941+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dcb0963840 con 0x55dcb0a163f0 2023-09-10T01:15:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.941+0000 7f7593fff700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a163f0 0x55dcb0a18810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.941+0000 7f7573fff700 10 client.28283 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.941+0000 7f7593fff700 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a163f0 0x55dcb0a18810 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baa7801faafe35e7 server_cookie=7d0c6fe682dd14a4 in_seq=0 out_seq=0 2023-09-10T01:15:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f7573fff700 1 -- 172.21.15.3:0/2754983601 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dcb0963840 con 0x55dcb0a163f0 2023-09-10T01:15:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f7573fff700 10 client.28283 handle_command_reply: tid=0 2023-09-10T01:15:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 client.28283 shutdown 2023-09-10T01:15:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 msgr2=0x55dcb0a08de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb0a08de0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7588099690 tx=0x7f758800ccc0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 shutdown_connections 2023-09-10T01:15:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a163f0 0x55dcb0a18810 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb0a021a0 0x55dcb0a06690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f7573fff700 0 client.28283 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb0a08de0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 --2- 172.21.15.3:0/2754983601 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a06060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 >> 172.21.15.3:0/2754983601 conn(0x55dcb0971030 msgr2=0x55dcb09f67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 shutdown_connections 2023-09-10T01:15:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.942+0000 7f75ae658b80 1 -- 172.21.15.3:0/2754983601 wait complete. 2023-09-10T01:15:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.943+0000 7f75ae658b80 20 client.28283 trim_cache size 0 max 16384 2023-09-10T01:15:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.957+0000 7f75ae658b80 1 Processor -- start 2023-09-10T01:15:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f75ae658b80 1 -- start start 2023-09-10T01:15:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb0a0d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb09f7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f75ae658b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb0a07640 con 0x55dcb09fccc0 2023-09-10T01:15:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f75ae658b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb09f8d10 con 0x55dcb0a021a0 2023-09-10T01:15:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f75ae658b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb0a139a0 con 0x55dcb09fc300 2023-09-10T01:15:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f7593fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f759939c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb09f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f7593fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49818/0 (socket says 172.21.15.3:49818) 2023-09-10T01:15:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb0a0d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.958+0000 7f7593fff700 1 -- 172.21.15.3:0/925125835 learned_addr learned my addr 172.21.15.3:0/925125835 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f7593fff700 1 -- 172.21.15.3:0/925125835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 msgr2=0x55dcb0a0d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f7593fff700 1 --2- 172.21.15.3:0/925125835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb0a0d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f7593fff700 1 -- 172.21.15.3:0/925125835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 msgr2=0x55dcb09f7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f7593fff700 1 --2- 172.21.15.3:0/925125835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb09f7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f7593fff700 1 -- 172.21.15.3:0/925125835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcb0a1d440 con 0x55dcb09fccc0 2023-09-10T01:15:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f7593fff700 1 --2- 172.21.15.3:0/925125835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08980 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f758000b580 tx=0x7f75800092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9cf00df434dca3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f757f7fe700 1 -- 172.21.15.3:0/925125835 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580032aa0 con 0x55dcb09fccc0 2023-09-10T01:15:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f757f7fe700 1 -- 172.21.15.3:0/925125835 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7580032c00 con 0x55dcb09fccc0 2023-09-10T01:15:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.959+0000 7f757f7fe700 1 -- 172.21.15.3:0/925125835 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580003d50 con 0x55dcb09fccc0 2023-09-10T01:15:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 -- 172.21.15.3:0/925125835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 msgr2=0x55dcb0a08980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 --2- 172.21.15.3:0/925125835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08980 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f758000b580 tx=0x7f75800092f0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 -- 172.21.15.3:0/925125835 shutdown_connections 2023-09-10T01:15:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 --2- 172.21.15.3:0/925125835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fc300 0x55dcb0a0d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 --2- 172.21.15.3:0/925125835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb0a021a0 0x55dcb09f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 --2- 172.21.15.3:0/925125835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb09fccc0 0x55dcb0a08980 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 -- 172.21.15.3:0/925125835 >> 172.21.15.3:0/925125835 conn(0x55dcb0971030 msgr2=0x55dcb0a00150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:00.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 -- 172.21.15.3:0/925125835 shutdown_connections 2023-09-10T01:15:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 -- 172.21.15.3:0/925125835 wait complete. 2023-09-10T01:15:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 -- start start 2023-09-10T01:15:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.960+0000 7f75ae658b80 1 Processor -- start 2023-09-10T01:15:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb09ff260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fccc0 0x55dcb09fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f75ae658b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb0a021a0 0x55dcb09fe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f75ae658b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb09fe770 con 0x55dcb0a021a0 2023-09-10T01:15:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f75ae658b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb09fe8e0 con 0x55dcb09fc300 2023-09-10T01:15:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f75ae658b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcb0a6a220 con 0x55dcb09fccc0 2023-09-10T01:15:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f75ae658b80 10 client.0 _ll_register_callbacks cb 0x55dcb0836a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb09ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7598b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb09ff260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:46440/0 (socket says 172.21.15.3:46440) 2023-09-10T01:15:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7598b9b700 1 -- 172.21.15.3:0/1869424947 learned_addr learned my addr 172.21.15.3:0/1869424947 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f759939c700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb0a021a0 0x55dcb09fe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7593fff700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fccc0 0x55dcb09fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7598b9b700 1 -- 172.21.15.3:0/1869424947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fccc0 msgr2=0x55dcb09fdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7598b9b700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fccc0 0x55dcb09fdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7598b9b700 1 -- 172.21.15.3:0/1869424947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb0a021a0 msgr2=0x55dcb09fe090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.961+0000 7f7598b9b700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb0a021a0 0x55dcb09fe090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7598b9b700 1 -- 172.21.15.3:0/1869424947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7580028040 con 0x55dcb09fc300 2023-09-10T01:15:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7593fff700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fccc0 0x55dcb09fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:15:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7598b9b700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb09ff260 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7588099920 tx=0x7f7588097f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6229eb1fd3a512e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7573fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7573fff700 1 -- 172.21.15.3:0/1869424947 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880ca0b0 con 0x55dcb09fc300 2023-09-10T01:15:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7573fff700 1 -- 172.21.15.3:0/1869424947 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f758809d7c0 con 0x55dcb09fc300 2023-09-10T01:15:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f758001e040 con 0x55dcb09fc300 2023-09-10T01:15:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7573fff700 1 -- 172.21.15.3:0/1869424947 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7588082e80 con 0x55dcb09fc300 2023-09-10T01:15:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f75ae658b80 10 client.28289 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55dcb07a11f0 con 0x55dcb09fc300 2023-09-10T01:15:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.962+0000 7f7573fff700 1 -- 172.21.15.3:0/1869424947 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f758001e040 con 0x55dcb09fc300 2023-09-10T01:15:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f7573fff700 1 -- 172.21.15.3:0/1869424947 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f75880ae070 con 0x55dcb09fc300 2023-09-10T01:15:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f75ae658b80 10 client.28289 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f75ae658b80 10 client.28289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f75ae658b80 20 client.28289 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a6b4b0 0x55dcb0a6d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f75ae658b80 4 client.28289 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:15:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dcb09478c0 con 0x55dcb0a6b4b0 2023-09-10T01:15:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f7593fff700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a6b4b0 0x55dcb0a6d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f7593fff700 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a6b4b0 0x55dcb0a6d8d0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=363687bb9338e99b server_cookie=8cfa065cb189887 in_seq=0 out_seq=0 2023-09-10T01:15:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:00.963+0000 7f7573fff700 10 client.28289 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:01.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f7573fff700 1 -- 172.21.15.3:0/1869424947 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dcb09478c0 con 0x55dcb0a6b4b0 2023-09-10T01:15:01.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f7573fff700 10 client.28289 handle_command_reply: tid=0 2023-09-10T01:15:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 client.28289 shutdown 2023-09-10T01:15:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 msgr2=0x55dcb09ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb09ff260 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7588099920 tx=0x7f7588097f40 comp rx=0 tx=0).stop 2023-09-10T01:15:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 shutdown_connections 2023-09-10T01:15:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dcb0a6b4b0 0x55dcb0a6d8d0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dcb0a021a0 0x55dcb09fe090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcb09fc300 0x55dcb09ff260 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f7573fff700 0 client.28289 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:01.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 --2- 172.21.15.3:0/1869424947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dcb09fccc0 0x55dcb09fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 >> 172.21.15.3:0/1869424947 conn(0x55dcb0971030 msgr2=0x55dcb09f7090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:01.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 shutdown_connections 2023-09-10T01:15:01.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.034+0000 7f75ae658b80 1 -- 172.21.15.3:0/1869424947 wait complete. 2023-09-10T01:15:01.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f75ae658b80 20 client.28289 trim_cache size 0 max 16384 2023-09-10T01:15:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:15:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:15:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:15:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:15:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f75840a07b0 msgr2=0x7f75840a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f75840a07b0 0x7f75840a2c70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f758c009ec0 tx=0x7f758c009920 comp rx=0 tx=0).stop 2023-09-10T01:15:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75941005c0 msgr2=0x7f759419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75941005c0 0x7f759419e620 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f758800eb30 tx=0x7f758800ee40 comp rx=0 tx=0).stop 2023-09-10T01:15:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 shutdown_connections 2023-09-10T01:15:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f75840a07b0 0x7f75840a2c70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75941005c0 0x7f759419e620 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7594106680 0x7f7594198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 --2- 172.21.15.3:0/360831416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7594100f80 0x7f759419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.035+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 >> 172.21.15.3:0/360831416 conn(0x7f75940fc0b0 msgr2=0x7f759410b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.036+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 shutdown_connections 2023-09-10T01:15:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:01.036+0000 7f759adff700 1 -- 172.21.15.3:0/360831416 wait complete. 2023-09-10T01:15:01.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 981280 7.91 MB/sec execute 523 sec latency 1152.120 ms 2023-09-10T01:15:02.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:02 smithi003 ceph-mon[97654]: pgmap v367: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2023-09-10T01:15:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:02 smithi136 ceph-mon[133708]: pgmap v367: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2023-09-10T01:15:02.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:02 smithi092 ceph-mon[134067]: pgmap v367: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2023-09-10T01:15:02.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 981280 7.89 MB/sec execute 524 sec latency 2152.252 ms 2023-09-10T01:15:03.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 981575 7.88 MB/sec execute 525 sec latency 2450.897 ms 2023-09-10T01:15:04.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:04 smithi003 ceph-mon[97654]: pgmap v368: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2023-09-10T01:15:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:04 smithi136 ceph-mon[133708]: pgmap v368: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2023-09-10T01:15:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:04 smithi092 ceph-mon[134067]: pgmap v368: 129 pgs: 129 active+clean; 300 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2023-09-10T01:15:04.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 981592 7.86 MB/sec execute 526 sec latency 548.734 ms 2023-09-10T01:15:05.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:05 smithi003 ceph-mon[97654]: osdmap e102: 12 total, 12 up, 12 in 2023-09-10T01:15:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:05 smithi136 ceph-mon[133708]: osdmap e102: 12 total, 12 up, 12 in 2023-09-10T01:15:05.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:05 smithi092 ceph-mon[134067]: osdmap e102: 12 total, 12 up, 12 in 2023-09-10T01:15:05.805 INFO:tasks.workunit.client.0.smithi003.stdout: 1 982473 7.86 MB/sec execute 527 sec latency 604.824 ms 2023-09-10T01:15:06.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:06 smithi003 ceph-mon[97654]: pgmap v370: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 911 KiB/s wr, 130 op/s 2023-09-10T01:15:06.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:06 smithi136 ceph-mon[133708]: pgmap v370: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 911 KiB/s wr, 130 op/s 2023-09-10T01:15:06.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:06 smithi092 ceph-mon[134067]: pgmap v370: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 911 KiB/s wr, 130 op/s 2023-09-10T01:15:06.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 984238 7.86 MB/sec execute 528 sec latency 14.043 ms 2023-09-10T01:15:07.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 984831 7.85 MB/sec execute 529 sec latency 243.141 ms 2023-09-10T01:15:08.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:08 smithi003 ceph-mon[97654]: pgmap v371: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 911 KiB/s wr, 130 op/s 2023-09-10T01:15:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:08 smithi136 ceph-mon[133708]: pgmap v371: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 911 KiB/s wr, 130 op/s 2023-09-10T01:15:08.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:08 smithi092 ceph-mon[134067]: pgmap v371: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 911 KiB/s wr, 130 op/s 2023-09-10T01:15:08.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 985872 7.85 MB/sec execute 530 sec latency 534.131 ms 2023-09-10T01:15:09.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 987878 7.86 MB/sec execute 531 sec latency 13.030 ms 2023-09-10T01:15:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:10 smithi003 ceph-mon[97654]: pgmap v372: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 996 KiB/s wr, 153 op/s 2023-09-10T01:15:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:10 smithi003 ceph-mon[97654]: osdmap e103: 12 total, 12 up, 12 in 2023-09-10T01:15:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:10 smithi136 ceph-mon[133708]: pgmap v372: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 996 KiB/s wr, 153 op/s 2023-09-10T01:15:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:10 smithi136 ceph-mon[133708]: osdmap e103: 12 total, 12 up, 12 in 2023-09-10T01:15:10.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:10 smithi092 ceph-mon[134067]: pgmap v372: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 996 KiB/s wr, 153 op/s 2023-09-10T01:15:10.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:10 smithi092 ceph-mon[134067]: osdmap e103: 12 total, 12 up, 12 in 2023-09-10T01:15:10.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 988373 7.84 MB/sec execute 532 sec latency 399.852 ms 2023-09-10T01:15:11.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 988373 7.83 MB/sec execute 533 sec latency 1399.991 ms 2023-09-10T01:15:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:12 smithi136 ceph-mon[133708]: pgmap v374: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.2 MiB/s wr, 191 op/s 2023-09-10T01:15:12.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:12 smithi092 ceph-mon[134067]: pgmap v374: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.2 MiB/s wr, 191 op/s 2023-09-10T01:15:12.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 988373 7.81 MB/sec execute 534 sec latency 2400.104 ms 2023-09-10T01:15:12.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:12 smithi003 ceph-mon[97654]: pgmap v374: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.2 MiB/s wr, 191 op/s 2023-09-10T01:15:13.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 989343 7.81 MB/sec execute 535 sec latency 2709.562 ms 2023-09-10T01:15:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 -- 172.21.15.3:0/4052182369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0400732c0 msgr2=0x7fc040111070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 --2- 172.21.15.3:0/4052182369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0400732c0 0x7fc040111070 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009990 tx=0x7fc02c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 -- 172.21.15.3:0/4052182369 shutdown_connections 2023-09-10T01:15:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 --2- 172.21.15.3:0/4052182369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0400729a0 0x7fc040072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 --2- 172.21.15.3:0/4052182369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0401115b0 0x7fc0401139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 --2- 172.21.15.3:0/4052182369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0400732c0 0x7fc040111070 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 -- 172.21.15.3:0/4052182369 >> 172.21.15.3:0/4052182369 conn(0x7fc0400fc0b0 msgr2=0x7fc0400fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 -- 172.21.15.3:0/4052182369 shutdown_connections 2023-09-10T01:15:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.206+0000 7fc046bff700 1 -- 172.21.15.3:0/4052182369 wait complete. 2023-09-10T01:15:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.207+0000 7fc046bff700 1 Processor -- start 2023-09-10T01:15:14.211 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc046bff700 1 -- start start 2023-09-10T01:15:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc046bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0400729a0 0x7fc04019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc046bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0400732c0 0x7fc04019d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc046bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0401115b0 0x7fc0401a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc046bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0401a1d60 con 0x7fc0400729a0 2023-09-10T01:15:14.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc046bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0401a1ed0 con 0x7fc0401115b0 2023-09-10T01:15:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc046bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0401a2090 con 0x7fc0400732c0 2023-09-10T01:15:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc04499b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0400729a0 0x7fc04019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc03ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0400732c0 0x7fc04019d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc04519c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0401115b0 0x7fc0401a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc03ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0400732c0 0x7fc04019d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44064/0 (socket says 172.21.15.3:44064) 2023-09-10T01:15:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.208+0000 7fc03ffff700 1 -- 172.21.15.3:0/4220288137 learned_addr learned my addr 172.21.15.3:0/4220288137 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc04519c700 1 -- 172.21.15.3:0/4220288137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0400732c0 msgr2=0x7fc04019d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc04519c700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0400732c0 0x7fc04019d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc04519c700 1 -- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0400729a0 msgr2=0x7fc04019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc04519c700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0400729a0 0x7fc04019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc04519c700 1 -- 172.21.15.3:0/4220288137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc034009520 con 0x7fc0401115b0 2023-09-10T01:15:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc04499b700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0400729a0 0x7fc04019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc04519c700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0401115b0 0x7fc0401a1720 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc03800ef90 tx=0x7fc03800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc03dffb700 1 -- 172.21.15.3:0/4220288137 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038018070 con 0x7fc0401115b0 2023-09-10T01:15:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc03dffb700 1 -- 172.21.15.3:0/4220288137 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc03800f040 con 0x7fc0401115b0 2023-09-10T01:15:14.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c0095f0 con 0x7fc0401115b0 2023-09-10T01:15:14.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc03dffb700 1 -- 172.21.15.3:0/4220288137 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03800cc30 con 0x7fc0401115b0 2023-09-10T01:15:14.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.209+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0401a99d0 con 0x7fc0401115b0 2023-09-10T01:15:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.213+0000 7fc03dffb700 1 -- 172.21.15.3:0/4220288137 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fc038003a00 con 0x7fc0401115b0 2023-09-10T01:15:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc03dffb700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0300a07b0 0x7fc0300a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 Processor -- start 2023-09-10T01:15:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc03dffb700 1 -- 172.21.15.3:0/4220288137 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc0380cbe50 con 0x7fc0401115b0 2023-09-10T01:15:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 -- start start 2023-09-10T01:15:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc04499b700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0300a07b0 0x7fc0300a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e566580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e560780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560cc0 0x558a7e56cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e4b96d0 con 0x558a7e560300 2023-09-10T01:15:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e4b6ff0 con 0x558a7e560cc0 2023-09-10T01:15:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc05a472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e4b28e0 con 0x558a7e5661a0 2023-09-10T01:15:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc04499b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e566580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.215+0000 7fc04499b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e566580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44066/0 (socket says 172.21.15.3:44066) 2023-09-10T01:15:14.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc03ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e560780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04519c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560cc0 0x558a7e56cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04499b700 1 -- 172.21.15.3:0/4174971086 learned_addr learned my addr 172.21.15.3:0/4174971086 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04519c700 1 -- 172.21.15.3:0/4174971086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 msgr2=0x558a7e566580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04519c700 1 --2- 172.21.15.3:0/4174971086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e566580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04519c700 1 -- 172.21.15.3:0/4174971086 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 msgr2=0x558a7e560780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04519c700 1 --2- 172.21.15.3:0/4174971086 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e560780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04519c700 1 -- 172.21.15.3:0/4174971086 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a7e2ded80 con 0x558a7e560cc0 2023-09-10T01:15:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04499b700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0300a07b0 0x7fc0300a2c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc034011b90 tx=0x7fc034011f50 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc04519c700 1 --2- 172.21.15.3:0/4174971086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560cc0 0x558a7e56cf30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc0380c91b0 tx=0x7fc038098ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7456221a345b2abd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:14.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc02affd700 1 -- 172.21.15.3:0/4174971086 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03801b320 con 0x558a7e560cc0 2023-09-10T01:15:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc02affd700 1 -- 172.21.15.3:0/4174971086 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0380996a0 con 0x558a7e560cc0 2023-09-10T01:15:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.216+0000 7fc02affd700 1 -- 172.21.15.3:0/4174971086 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0380956b0 con 0x558a7e560cc0 2023-09-10T01:15:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 -- 172.21.15.3:0/4174971086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560cc0 msgr2=0x558a7e56cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4174971086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560cc0 0x558a7e56cf30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc0380c91b0 tx=0x7fc038098ae0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 -- 172.21.15.3:0/4174971086 shutdown_connections 2023-09-10T01:15:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4174971086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e566580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4174971086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560cc0 0x558a7e56cf30 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4174971086 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e560780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 -- 172.21.15.3:0/4174971086 >> 172.21.15.3:0/4174971086 conn(0x558a7e4d5030 msgr2=0x558a7e55b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 -- 172.21.15.3:0/4174971086 shutdown_connections 2023-09-10T01:15:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.217+0000 7fc05a472b80 1 -- 172.21.15.3:0/4174971086 wait complete. 2023-09-10T01:15:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.218+0000 7fc05a472b80 1 -- start start 2023-09-10T01:15:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.218+0000 7fc05a472b80 1 Processor -- start 2023-09-10T01:15:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e56a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e56a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc05a472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e450b10 con 0x558a7e560cc0 2023-09-10T01:15:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc05a472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e450810 con 0x558a7e560300 2023-09-10T01:15:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc05a472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e3348c0 con 0x558a7e5661a0 2023-09-10T01:15:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc05a472b80 10 client.0 _ll_register_callbacks cb 0x558a7e39aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc04499b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc04499b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50732/0 (socket says 172.21.15.3:50732) 2023-09-10T01:15:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc04499b700 1 -- 172.21.15.3:0/3322447542 learned_addr learned my addr 172.21.15.3:0/3322447542 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc03ffff700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e56a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.231 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.219+0000 7fc04519c700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e56a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc04499b700 1 -- 172.21.15.3:0/3322447542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 msgr2=0x558a7e56a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc04499b700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e56a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc04499b700 1 -- 172.21.15.3:0/3322447542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 msgr2=0x558a7e56a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc04499b700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e56a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc04499b700 1 -- 172.21.15.3:0/3322447542 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0380b8040 con 0x558a7e560300 2023-09-10T01:15:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc03ffff700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e56a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc04499b700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56ce00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc03400ffa0 tx=0x7fc0340047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dda0b6194da723b2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc01ffff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc01ffff700 1 -- 172.21.15.3:0/3322447542 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034028600 con 0x558a7e560300 2023-09-10T01:15:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558a7e40a490 con 0x558a7e560300 2023-09-10T01:15:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc01ffff700 1 -- 172.21.15.3:0/3322447542 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc034028760 con 0x558a7e560300 2023-09-10T01:15:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc01ffff700 1 -- 172.21.15.3:0/3322447542 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc034020e90 con 0x558a7e560300 2023-09-10T01:15:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc05a472b80 10 client.28301 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.220+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x558a7e2c7c00 con 0x558a7e560300 2023-09-10T01:15:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc01ffff700 1 -- 172.21.15.3:0/3322447542 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558a7e40a490 con 0x558a7e560300 2023-09-10T01:15:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc01ffff700 1 -- 172.21.15.3:0/3322447542 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc034026030 con 0x558a7e560300 2023-09-10T01:15:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc05a472b80 10 client.28301 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc05a472b80 10 client.28301 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc05a472b80 20 client.28301 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc05a472b80 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e57a500 0x558a7e57c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc05a472b80 4 client.28301 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a7e4c7840 con 0x558a7e57a500 2023-09-10T01:15:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.221+0000 7fc03ffff700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e57a500 0x558a7e57c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.222+0000 7fc03ffff700 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e57a500 0x558a7e57c920 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea6f14781b99154b server_cookie=2755db7ece23d9b6 in_seq=0 out_seq=0 2023-09-10T01:15:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.222+0000 7fc01ffff700 10 client.28301 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc01ffff700 1 -- 172.21.15.3:0/3322447542 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558a7e4c7840 con 0x558a7e57a500 2023-09-10T01:15:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc01ffff700 10 client.28301 handle_command_reply: tid=0 2023-09-10T01:15:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 client.28301 shutdown 2023-09-10T01:15:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 msgr2=0x558a7e56ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56ce00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc03400ffa0 tx=0x7fc0340047c0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 shutdown_connections 2023-09-10T01:15:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e57a500 0x558a7e57c920 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e56a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56ce00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 --2- 172.21.15.3:0/3322447542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e56a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc01ffff700 0 client.28301 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 >> 172.21.15.3:0/3322447542 conn(0x558a7e4d5030 msgr2=0x558a7e55a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 shutdown_connections 2023-09-10T01:15:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 1 -- 172.21.15.3:0/3322447542 wait complete. 2023-09-10T01:15:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.223+0000 7fc05a472b80 20 client.28301 trim_cache size 0 max 16384 2023-09-10T01:15:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 Processor -- start 2023-09-10T01:15:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 -- start start 2023-09-10T01:15:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e5699a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e569ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e561580 con 0x558a7e560cc0 2023-09-10T01:15:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e56b700 con 0x558a7e560300 2023-09-10T01:15:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc05a472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e55cd60 con 0x558a7e5661a0 2023-09-10T01:15:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc04499b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc04499b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50752/0 (socket says 172.21.15.3:50752) 2023-09-10T01:15:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc04499b700 1 -- 172.21.15.3:0/2538734176 learned_addr learned my addr 172.21.15.3:0/2538734176 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.239+0000 7fc03ffff700 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e5699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc04519c700 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e569ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc04499b700 1 -- 172.21.15.3:0/2538734176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 msgr2=0x558a7e569ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc04499b700 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e569ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc04499b700 1 -- 172.21.15.3:0/2538734176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 msgr2=0x558a7e5699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc04499b700 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e5699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc04499b700 1 -- 172.21.15.3:0/2538734176 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a7e3051f0 con 0x558a7e560300 2023-09-10T01:15:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc03ffff700 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e5699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc04499b700 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56c930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc034011480 tx=0x7fc0340046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8be246ff39bf5dbd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc02b7fe700 1 -- 172.21.15.3:0/2538734176 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034016040 con 0x558a7e560300 2023-09-10T01:15:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc02b7fe700 1 -- 172.21.15.3:0/2538734176 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc034020e90 con 0x558a7e560300 2023-09-10T01:15:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.240+0000 7fc02b7fe700 1 -- 172.21.15.3:0/2538734176 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0340289a0 con 0x558a7e560300 2023-09-10T01:15:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 -- 172.21.15.3:0/2538734176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 msgr2=0x558a7e56c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56c930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc034011480 tx=0x7fc0340046d0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 -- 172.21.15.3:0/2538734176 shutdown_connections 2023-09-10T01:15:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e5661a0 0x558a7e569ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e560300 0x558a7e56c930 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 --2- 172.21.15.3:0/2538734176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560cc0 0x558a7e5699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 -- 172.21.15.3:0/2538734176 >> 172.21.15.3:0/2538734176 conn(0x558a7e4d5030 msgr2=0x558a7e55b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 -- 172.21.15.3:0/2538734176 shutdown_connections 2023-09-10T01:15:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.241+0000 7fc05a472b80 1 -- 172.21.15.3:0/2538734176 wait complete. 2023-09-10T01:15:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 -- start start 2023-09-10T01:15:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 Processor -- start 2023-09-10T01:15:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e571550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e560cc0 0x558a7e56f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e5661a0 0x558a7e56fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e5705a0 con 0x558a7e560300 2023-09-10T01:15:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e570710 con 0x558a7e5661a0 2023-09-10T01:15:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a7e5ce300 con 0x558a7e560cc0 2023-09-10T01:15:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.242+0000 7fc05a472b80 10 client.0 _ll_register_callbacks cb 0x558a7e39aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e5661a0 0x558a7e56fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e5661a0 0x558a7e56fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:50754/0 (socket says 172.21.15.3:50754) 2023-09-10T01:15:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04499b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e571550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 -- 172.21.15.3:0/4022404576 learned_addr learned my addr 172.21.15.3:0/4022404576 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc03ffff700 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e560cc0 0x558a7e56f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 -- 172.21.15.3:0/4022404576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e560cc0 msgr2=0x558a7e56f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e560cc0 0x558a7e56f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 -- 172.21.15.3:0/4022404576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 msgr2=0x558a7e571550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e571550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 -- 172.21.15.3:0/4022404576 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a7e2c7c00 con 0x558a7e5661a0 2023-09-10T01:15:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc04519c700 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e5661a0 0x558a7e56fe80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc0380954a0 tx=0x7fc038000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=315c25faec13217c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc01ffff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.243+0000 7fc01ffff700 1 -- 172.21.15.3:0/4022404576 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03801b320 con 0x558a7e5661a0 2023-09-10T01:15:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0380996a0 con 0x558a7e5661a0 2023-09-10T01:15:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc01ffff700 1 -- 172.21.15.3:0/4022404576 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc03809cdd0 con 0x558a7e5661a0 2023-09-10T01:15:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc01ffff700 1 -- 172.21.15.3:0/4022404576 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0380a0f80 con 0x558a7e5661a0 2023-09-10T01:15:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 10 client.28307 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x558a7e2dd250 con 0x558a7e5661a0 2023-09-10T01:15:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc01ffff700 1 -- 172.21.15.3:0/4022404576 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0380996a0 con 0x558a7e5661a0 2023-09-10T01:15:14.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc01ffff700 1 -- 172.21.15.3:0/4022404576 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc038099290 con 0x558a7e5661a0 2023-09-10T01:15:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 10 client.28307 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 10 client.28307 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 20 client.28307 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e5cf600 0x558a7e5d1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:14.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 4 client.28307 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:15:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558a7e4ab8c0 con 0x558a7e5cf600 2023-09-10T01:15:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.244+0000 7fc04499b700 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e5cf600 0x558a7e5d1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.245+0000 7fc04499b700 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e5cf600 0x558a7e5d1a20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47ef982d6622960a server_cookie=8fddff192fb7e757 in_seq=0 out_seq=0 2023-09-10T01:15:14.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.245+0000 7fc01ffff700 10 client.28307 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.303+0000 7fc01ffff700 1 -- 172.21.15.3:0/4022404576 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558a7e4ab8c0 con 0x558a7e5cf600 2023-09-10T01:15:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc01ffff700 10 client.28307 handle_command_reply: tid=0 2023-09-10T01:15:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 client.28307 shutdown 2023-09-10T01:15:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e5661a0 msgr2=0x558a7e56fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e5661a0 0x558a7e56fe80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc0380954a0 tx=0x7fc038000f40 comp rx=0 tx=0).stop 2023-09-10T01:15:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 shutdown_connections 2023-09-10T01:15:14.306 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558a7e5cf600 0x558a7e5d1a20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558a7e5661a0 0x558a7e56fe80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558a7e560300 0x558a7e571550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 --2- 172.21.15.3:0/4022404576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558a7e560cc0 0x558a7e56f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc01ffff700 0 client.28307 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:14.307 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 >> 172.21.15.3:0/4022404576 conn(0x558a7e4d5030 msgr2=0x558a7e559f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 shutdown_connections 2023-09-10T01:15:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 1 -- 172.21.15.3:0/4022404576 wait complete. 2023-09-10T01:15:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.304+0000 7fc05a472b80 20 client.28307 trim_cache size 0 max 16384 2023-09-10T01:15:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:15:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:15:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:15:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:15:14.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0300a07b0 msgr2=0x7fc0300a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0300a07b0 0x7fc0300a2c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc034011b90 tx=0x7fc034011f50 comp rx=0 tx=0).stop 2023-09-10T01:15:14.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0401115b0 msgr2=0x7fc0401a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:14.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0401115b0 0x7fc0401a1720 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc03800ef90 tx=0x7fc03800c3a0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 shutdown_connections 2023-09-10T01:15:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc0300a07b0 0x7fc0300a2c70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0401115b0 0x7fc0401a1720 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0400729a0 0x7fc04019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 --2- 172.21.15.3:0/4220288137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0400732c0 0x7fc04019d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 >> 172.21.15.3:0/4220288137 conn(0x7fc0400fc0b0 msgr2=0x7fc0400fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 shutdown_connections 2023-09-10T01:15:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:14.305+0000 7fc046bff700 1 -- 172.21.15.3:0/4220288137 wait complete. 2023-09-10T01:15:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:14 smithi136 ceph-mon[133708]: pgmap v375: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 634 KiB/s wr, 86 op/s 2023-09-10T01:15:14.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:14 smithi092 ceph-mon[134067]: pgmap v375: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 634 KiB/s wr, 86 op/s 2023-09-10T01:15:14.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 991301 7.82 MB/sec execute 536 sec latency 49.287 ms 2023-09-10T01:15:14.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:14 smithi003 ceph-mon[97654]: pgmap v375: 129 pgs: 129 active+clean; 299 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 634 KiB/s wr, 86 op/s 2023-09-10T01:15:15.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 992221 7.81 MB/sec execute 537 sec latency 15.315 ms 2023-09-10T01:15:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:16 smithi136 ceph-mon[133708]: pgmap v376: 129 pgs: 129 active+clean; 305 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2023-09-10T01:15:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:16 smithi092 ceph-mon[134067]: pgmap v376: 129 pgs: 129 active+clean; 305 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2023-09-10T01:15:16.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 993999 7.81 MB/sec execute 538 sec latency 21.640 ms 2023-09-10T01:15:16.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:16 smithi003 ceph-mon[97654]: pgmap v376: 129 pgs: 129 active+clean; 305 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2023-09-10T01:15:17.806 INFO:tasks.workunit.client.0.smithi003.stdout: 1 995232 7.81 MB/sec execute 539 sec latency 20.302 ms 2023-09-10T01:15:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:18 smithi136 ceph-mon[133708]: pgmap v377: 129 pgs: 129 active+clean; 305 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2023-09-10T01:15:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:18 smithi092 ceph-mon[134067]: pgmap v377: 129 pgs: 129 active+clean; 305 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2023-09-10T01:15:18.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 995497 7.80 MB/sec execute 540 sec latency 654.756 ms 2023-09-10T01:15:18.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:18 smithi003 ceph-mon[97654]: pgmap v377: 129 pgs: 129 active+clean; 305 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2023-09-10T01:15:19.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 995497 7.78 MB/sec execute 541 sec latency 1654.907 ms 2023-09-10T01:15:20.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:15:20.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:20 smithi003 ceph-mon[97654]: pgmap v378: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 154 op/s 2023-09-10T01:15:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:20 smithi136 ceph-mon[133708]: pgmap v378: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 154 op/s 2023-09-10T01:15:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:20 smithi092 ceph-mon[134067]: pgmap v378: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 154 op/s 2023-09-10T01:15:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 -- 172.21.15.3:0/3129129130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca410ce80 msgr2=0x7f1ca410f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/3129129130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca410ce80 0x7f1ca410f2f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00b0a0 tx=0x7f1c9c00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 -- 172.21.15.3:0/3129129130 shutdown_connections 2023-09-10T01:15:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/3129129130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca410ce80 0x7f1ca410f2f0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/3129129130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca4072a30 0x7f1ca4072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/3129129130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ca4073350 0x7f1ca410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 -- 172.21.15.3:0/3129129130 >> 172.21.15.3:0/3129129130 conn(0x7f1ca40fc170 msgr2=0x7f1ca40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 -- 172.21.15.3:0/3129129130 shutdown_connections 2023-09-10T01:15:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.762+0000 7f1ca93ff700 1 -- 172.21.15.3:0/3129129130 wait complete. 2023-09-10T01:15:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.763+0000 7f1ca93ff700 1 Processor -- start 2023-09-10T01:15:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.763+0000 7f1ca93ff700 1 -- start start 2023-09-10T01:15:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca93ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ca4072a30 0x7f1ca41989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca93ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca4073350 0x7f1ca4198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca93ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca410ce80 0x7f1ca419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca93ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca419d900 con 0x7f1ca4073350 2023-09-10T01:15:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca93ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca419da70 con 0x7f1ca410ce80 2023-09-10T01:15:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca93ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca419dc30 con 0x7f1ca4072a30 2023-09-10T01:15:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca4073350 0x7f1ca4198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca410ce80 0x7f1ca419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca410ce80 0x7f1ca419d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53786/0 (socket says 172.21.15.3:53786) 2023-09-10T01:15:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 -- 172.21.15.3:0/1509332416 learned_addr learned my addr 172.21.15.3:0/1509332416 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca4073350 0x7f1ca4198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39798/0 (socket says 172.21.15.3:39798) 2023-09-10T01:15:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ca4072a30 0x7f1ca41989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 -- 172.21.15.3:0/1509332416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ca4072a30 msgr2=0x7f1ca41989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ca4072a30 0x7f1ca41989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 -- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca4073350 msgr2=0x7f1ca4198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca4073350 0x7f1ca4198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.764+0000 7f1ca37fe700 1 -- 172.21.15.3:0/1509332416 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c009d00 con 0x7f1ca410ce80 2023-09-10T01:15:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.765+0000 7f1ca37fe700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca410ce80 0x7f1ca419d330 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00b780 tx=0x7f1c9c007990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.765+0000 7f1ca0ff9700 1 -- 172.21.15.3:0/1509332416 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00e040 con 0x7f1ca410ce80 2023-09-10T01:15:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.765+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca419deb0 con 0x7f1ca410ce80 2023-09-10T01:15:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.765+0000 7f1ca0ff9700 1 -- 172.21.15.3:0/1509332416 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c9c01d470 con 0x7f1ca410ce80 2023-09-10T01:15:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.765+0000 7f1ca0ff9700 1 -- 172.21.15.3:0/1509332416 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0052a0 con 0x7f1ca410ce80 2023-09-10T01:15:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.765+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca419e190 con 0x7f1ca410ce80 2023-09-10T01:15:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.770+0000 7f1ca0ff9700 1 -- 172.21.15.3:0/1509332416 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f1c9c01d5e0 con 0x7f1ca410ce80 2023-09-10T01:15:20.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 Processor -- start 2023-09-10T01:15:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 -- start start 2023-09-10T01:15:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafdeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafeb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aaf37730 con 0x5592aafde710 2023-09-10T01:15:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aaf35050 con 0x5592aafdf030 2023-09-10T01:15:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aaf30940 con 0x5592aafe8e40 2023-09-10T01:15:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.771+0000 7f1ca0ff9700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1c900a07b0 0x7f1c900a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafdeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafdeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39800/0 (socket says 172.21.15.3:39800) 2023-09-10T01:15:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1c9bfff700 1 -- 172.21.15.3:0/3915154865 learned_addr learned my addr 172.21.15.3:0/3915154865 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca0ff9700 1 -- 172.21.15.3:0/1509332416 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f1c9c0cd920 con 0x7f1ca410ce80 2023-09-10T01:15:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafeb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca37fe700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1c900a07b0 0x7f1c900a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/3915154865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 msgr2=0x5592aafeb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafeb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/3915154865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 msgr2=0x5592aafdeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafdeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/3915154865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592aad5cde0 con 0x5592aafdf030 2023-09-10T01:15:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafdeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca37fe700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1c900a07b0 0x7f1c900a2c70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c09a9a0 tx=0x7f1c9c01df90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.772+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe8900 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009cb0 tx=0x7f1c94011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd536e28eb7fa23e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1c997fa700 1 -- 172.21.15.3:0/3915154865 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94011d20 con 0x5592aafdf030 2023-09-10T01:15:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1c997fa700 1 -- 172.21.15.3:0/3915154865 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c94011e80 con 0x5592aafdf030 2023-09-10T01:15:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1c997fa700 1 -- 172.21.15.3:0/3915154865 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94005280 con 0x5592aafdf030 2023-09-10T01:15:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/3915154865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 msgr2=0x5592aafe8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe8900 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009cb0 tx=0x7f1c94011440 comp rx=0 tx=0).stop 2023-09-10T01:15:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/3915154865 shutdown_connections 2023-09-10T01:15:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafeb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafdeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/3915154865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe8900 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/3915154865 >> 172.21.15.3:0/3915154865 conn(0x5592aaf53090 msgr2=0x5592aafd9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/3915154865 shutdown_connections 2023-09-10T01:15:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.773+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/3915154865 wait complete. 2023-09-10T01:15:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.774+0000 7f1cbcc23b80 1 -- start start 2023-09-10T01:15:20.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.774+0000 7f1cbcc23b80 1 Processor -- start 2023-09-10T01:15:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafe8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafe5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafe6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aaeceb70 con 0x5592aafdf030 2023-09-10T01:15:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aaece870 con 0x5592aafe8e40 2023-09-10T01:15:20.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aadb2920 con 0x5592aafde710 2023-09-10T01:15:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1cbcc23b80 10 client.0 _ll_register_callbacks cb 0x5592aae18a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafe8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.775+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafe5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafe8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39264/0 (socket says 172.21.15.3:39264) 2023-09-10T01:15:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 -- 172.21.15.3:0/4128622283 learned_addr learned my addr 172.21.15.3:0/4128622283 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:20.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafe6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 -- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 msgr2=0x5592aafe6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafe6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 -- 172.21.15.3:0/4128622283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 msgr2=0x5592aafe5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafe5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 -- 172.21.15.3:0/4128622283 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c01b040 con 0x5592aafde710 2023-09-10T01:15:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafe8810 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c013a00 tx=0x7f1c8c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b8787214f5e575 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c8a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:15:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/4128622283 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c004030 con 0x5592aafde710 2023-09-10T01:15:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592aaf458a0 con 0x5592aafde710 2023-09-10T01:15:20.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.776+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/4128622283 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f1c8c004190 con 0x5592aafde710 2023-09-10T01:15:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/4128622283 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c8c004350 con 0x5592aafde710 2023-09-10T01:15:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 10 client.28204 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f1c940098e0 con 0x5592aafde710 2023-09-10T01:15:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/4128622283 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5592aaf458a0 con 0x5592aafde710 2023-09-10T01:15:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/4128622283 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f1c8c003710 con 0x5592aafde710 2023-09-10T01:15:20.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 10 client.28204 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 10 client.28204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 20 client.28204 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592aaff8920 0x5592aaffad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 4 client.28204 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.777+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592aaf29920 con 0x5592aaff8920 2023-09-10T01:15:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.778+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592aaff8920 0x5592aaffad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.778+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592aaff8920 0x5592aaffad40 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=346eb9b6510acb56 server_cookie=54a8d624de643322 in_seq=0 out_seq=0 2023-09-10T01:15:20.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.778+0000 7f1c8a7fc700 10 client.28204 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/4128622283 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5592aaf29920 con 0x5592aaff8920 2023-09-10T01:15:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1c8a7fc700 10 client.28204 handle_command_reply: tid=0 2023-09-10T01:15:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 client.28204 shutdown 2023-09-10T01:15:20.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 msgr2=0x5592aafe8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafe8810 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c013a00 tx=0x7f1c8c009260 comp rx=0 tx=0).stop 2023-09-10T01:15:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 shutdown_connections 2023-09-10T01:15:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592aaff8920 0x5592aaffad40 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafe8810 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafe6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/4128622283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafe5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1c8a7fc700 0 client.28204 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 >> 172.21.15.3:0/4128622283 conn(0x5592aaf53090 msgr2=0x5592aafd97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 shutdown_connections 2023-09-10T01:15:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.779+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/4128622283 wait complete. 2023-09-10T01:15:20.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.780+0000 7f1cbcc23b80 20 client.28204 trim_cache size 0 max 16384 2023-09-10T01:15:20.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.797+0000 7f1cbcc23b80 1 Processor -- start 2023-09-10T01:15:20.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.797+0000 7f1cbcc23b80 1 -- start start 2023-09-10T01:15:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.797+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafe8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.797+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafe5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aafd89a0 con 0x5592aafde710 2023-09-10T01:15:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aafe5d40 con 0x5592aafdf030 2023-09-10T01:15:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aafe6610 con 0x5592aafe8e40 2023-09-10T01:15:20.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53804/0 (socket says 172.21.15.3:53804) 2023-09-10T01:15:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2268065092 learned_addr learned my addr 172.21.15.3:0/2268065092 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafe8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafe5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2268065092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 msgr2=0x5592aafe5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafe5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2268065092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 msgr2=0x5592aafe8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafe8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2268065092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592aad45c60 con 0x5592aafdf030 2023-09-10T01:15:20.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafe5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.798+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe52c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011790 tx=0x7f1c94009710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f86f61c3cf3d5fa0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1c99ffb700 1 -- 172.21.15.3:0/2268065092 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94011ba0 con 0x5592aafdf030 2023-09-10T01:15:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1c99ffb700 1 -- 172.21.15.3:0/2268065092 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c94011d00 con 0x5592aafdf030 2023-09-10T01:15:20.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1c99ffb700 1 -- 172.21.15.3:0/2268065092 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940257a0 con 0x5592aafdf030 2023-09-10T01:15:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2268065092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 msgr2=0x5592aafe52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe52c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011790 tx=0x7f1c94009710 comp rx=0 tx=0).stop 2023-09-10T01:15:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2268065092 shutdown_connections 2023-09-10T01:15:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafe8e40 0x5592aafe5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafde710 0x5592aafe8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2268065092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafdf030 0x5592aafe52c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.799+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2268065092 >> 172.21.15.3:0/2268065092 conn(0x5592aaf53090 msgr2=0x5592aafe0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.800+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2268065092 shutdown_connections 2023-09-10T01:15:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.800+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2268065092 wait complete. 2023-09-10T01:15:20.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.800+0000 7f1cbcc23b80 1 -- start start 2023-09-10T01:15:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.800+0000 7f1cbcc23b80 1 Processor -- start 2023-09-10T01:15:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafefa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafedd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1cbcc23b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafee280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aafee950 con 0x5592aafdf030 2023-09-10T01:15:20.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592aafeeac0 con 0x5592aafe8e40 2023-09-10T01:15:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1cbcc23b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ab04c720 con 0x5592aafde710 2023-09-10T01:15:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1cbcc23b80 10 client.0 _ll_register_callbacks cb 0x7f1c8c013730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafedd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1ca2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafedd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39820/0 (socket says 172.21.15.3:39820) 2023-09-10T01:15:20.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2862921508 learned_addr learned my addr 172.21.15.3:0/2862921508 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.801+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafefa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2862921508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 msgr2=0x5592aafefa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafefa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 msgr2=0x5592aafee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafee280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1ca2ffd700 1 -- 172.21.15.3:0/2862921508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9401d040 con 0x5592aafdf030 2023-09-10T01:15:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1ca2ffd700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafedd40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011790 tx=0x7f1c94011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b68a7a93a0cf3bc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1c8a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:20.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/2862921508 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9402e470 con 0x5592aafdf030 2023-09-10T01:15:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592aaeea0b0 con 0x5592aafdf030 2023-09-10T01:15:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.802+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/2862921508 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f1c9402e5d0 con 0x5592aafdf030 2023-09-10T01:15:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/2862921508 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c9402e7b0 con 0x5592aafdf030 2023-09-10T01:15:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 10 client.18240 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5592aad83250 con 0x5592aafdf030 2023-09-10T01:15:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/2862921508 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5592aaeea0b0 con 0x5592aafdf030 2023-09-10T01:15:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/2862921508 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f1c94025b30 con 0x5592aafdf030 2023-09-10T01:15:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 10 client.18240 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 10 client.18240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 20 client.18240 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:20.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592ab04db40 0x5592ab04ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 4 client.18240 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:15:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.803+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5592aad7df70 con 0x5592ab04db40 2023-09-10T01:15:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592ab04db40 0x5592ab04ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1c9bfff700 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592ab04db40 0x5592ab04ff60 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c33d8a46ee0390 server_cookie=31fcea02dedbe471 in_seq=0 out_seq=0 2023-09-10T01:15:20.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1c8a7fc700 10 client.18240 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1c8a7fc700 1 -- 172.21.15.3:0/2862921508 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5592aad7df70 con 0x5592ab04db40 2023-09-10T01:15:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1c8a7fc700 10 client.18240 handle_command_reply: tid=0 2023-09-10T01:15:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 client.18240 shutdown 2023-09-10T01:15:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 msgr2=0x5592aafedd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafedd40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011790 tx=0x7f1c94011c30 comp rx=0 tx=0).stop 2023-09-10T01:15:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 shutdown_connections 2023-09-10T01:15:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5592ab04db40 0x5592ab04ff60 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592aafde710 0x5592aafefa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592aafe8e40 0x5592aafee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 --2- 172.21.15.3:0/2862921508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592aafdf030 0x5592aafedd40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1c8a7fc700 0 client.18240 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:20.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.804+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 >> 172.21.15.3:0/2862921508 conn(0x5592aaf53090 msgr2=0x5592aafe0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:20.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 shutdown_connections 2023-09-10T01:15:20.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1cbcc23b80 1 -- 172.21.15.3:0/2862921508 wait complete. 2023-09-10T01:15:20.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1cbcc23b80 20 client.18240 trim_cache size 0 max 16384 2023-09-10T01:15:20.818 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:15:20.819 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:15:20.819 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:15:20.819 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:15:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1c900a07b0 msgr2=0x7f1c900a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1c900a07b0 0x7f1c900a2c70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c09a9a0 tx=0x7f1c9c01df90 comp rx=0 tx=0).stop 2023-09-10T01:15:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca410ce80 msgr2=0x7f1ca419d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:20.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca410ce80 0x7f1ca419d330 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00b780 tx=0x7f1c9c007990 comp rx=0 tx=0).stop 2023-09-10T01:15:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 shutdown_connections 2023-09-10T01:15:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1c900a07b0 0x7f1c900a2c70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ca4072a30 0x7f1ca41989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ca410ce80 0x7f1ca419d330 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 --2- 172.21.15.3:0/1509332416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca4073350 0x7f1ca4198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.805+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 >> 172.21.15.3:0/1509332416 conn(0x7f1ca40fc170 msgr2=0x7f1ca41048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.806+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 shutdown_connections 2023-09-10T01:15:20.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:20.806+0000 7f1ca93ff700 1 -- 172.21.15.3:0/1509332416 wait complete. 2023-09-10T01:15:20.822 INFO:tasks.workunit.client.0.smithi003.stdout: 1 995497 7.77 MB/sec execute 542 sec latency 2654.977 ms 2023-09-10T01:15:20.824 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:15:20.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:15:21.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 -- 172.21.15.3:0/1837515174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 msgr2=0x7fd654106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 --2- 172.21.15.3:0/1837515174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 0x7fd654106a20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd644009990 tx=0x7fd644009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 -- 172.21.15.3:0/1837515174 shutdown_connections 2023-09-10T01:15:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 --2- 172.21.15.3:0/1837515174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100ff0 0x7fd65410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 --2- 172.21.15.3:0/1837515174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 0x7fd654106a20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 --2- 172.21.15.3:0/1837515174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100630 0x7fd654100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 -- 172.21.15.3:0/1837515174 >> 172.21.15.3:0/1837515174 conn(0x7fd6540fc140 msgr2=0x7fd6540fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 -- 172.21.15.3:0/1837515174 shutdown_connections 2023-09-10T01:15:21.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.029+0000 7fd659dff700 1 -- 172.21.15.3:0/1837515174 wait complete. 2023-09-10T01:15:21.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.030+0000 7fd659dff700 1 Processor -- start 2023-09-10T01:15:21.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.030+0000 7fd659dff700 1 -- start start 2023-09-10T01:15:21.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd659dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100630 0x7fd654072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd659dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100ff0 0x7fd654072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd659dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 0x7fd6541a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd659dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd654073670 con 0x7fd654106640 2023-09-10T01:15:21.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd659dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541a4e20 con 0x7fd654100ff0 2023-09-10T01:15:21.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd659dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6541a4f60 con 0x7fd654100630 2023-09-10T01:15:21.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd652ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100ff0 0x7fd654072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100630 0x7fd654072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd653fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 0x7fd6541a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100630 0x7fd654072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39282/0 (socket says 172.21.15.3:39282) 2023-09-10T01:15:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 -- 172.21.15.3:0/3863957221 learned_addr learned my addr 172.21.15.3:0/3863957221 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd652ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100ff0 0x7fd654072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53832/0 (socket says 172.21.15.3:53832) 2023-09-10T01:15:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 -- 172.21.15.3:0/3863957221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100ff0 msgr2=0x7fd654072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100ff0 0x7fd654072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 -- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 msgr2=0x7fd6541a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 0x7fd6541a48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd6537fe700 1 -- 172.21.15.3:0/3863957221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6440095f0 con 0x7fd654100630 2023-09-10T01:15:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.031+0000 7fd652ffd700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100ff0 0x7fd654072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.032+0000 7fd6537fe700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100630 0x7fd654072900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd644005910 tx=0x7fd644004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.032+0000 7fd650ff9700 1 -- 172.21.15.3:0/3863957221 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64401d070 con 0x7fd654100630 2023-09-10T01:15:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.032+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6541a5160 con 0x7fd654100630 2023-09-10T01:15:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.032+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6541a5690 con 0x7fd654100630 2023-09-10T01:15:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.032+0000 7fd650ff9700 1 -- 172.21.15.3:0/3863957221 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd644022470 con 0x7fd654100630 2023-09-10T01:15:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.032+0000 7fd650ff9700 1 -- 172.21.15.3:0/3863957221 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644017650 con 0x7fd654100630 2023-09-10T01:15:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.033+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd654067480 con 0x7fd654100630 2023-09-10T01:15:21.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.035+0000 7fd650ff9700 1 -- 172.21.15.3:0/3863957221 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd644022830 con 0x7fd654100630 2023-09-10T01:15:21.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.037+0000 7fd650ff9700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd63c0a07b0 0x7fd63c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.037+0000 7fd650ff9700 1 -- 172.21.15.3:0/3863957221 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd6440cd860 con 0x7fd654100630 2023-09-10T01:15:21.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.037+0000 7fd652ffd700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd63c0a07b0 0x7fd63c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.038+0000 7fd652ffd700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd63c0a07b0 0x7fd63c0a2c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd6540740f0 tx=0x7fd648009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.039+0000 7fd650ff9700 1 -- 172.21.15.3:0/3863957221 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd644096c90 con 0x7fd654100630 2023-09-10T01:15:21.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.321+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd654073e30 con 0x7fd654100630 2023-09-10T01:15:21.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.325+0000 7fd650ff9700 1 -- 172.21.15.3:0/3863957221 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+11297 (secure 0 0 0) 0x7fd64409a2b0 con 0x7fd654100630 2023-09-10T01:15:21.325 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:15:21.326 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:13:58.841353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:15:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd63c0a07b0 msgr2=0x7fd63c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd63c0a07b0 0x7fd63c0a2c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd6540740f0 tx=0x7fd648009260 comp rx=0 tx=0).stop 2023-09-10T01:15:21.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100630 msgr2=0x7fd654072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100630 0x7fd654072900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd644005910 tx=0x7fd644004730 comp rx=0 tx=0).stop 2023-09-10T01:15:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 shutdown_connections 2023-09-10T01:15:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd63c0a07b0 0x7fd63c0a2c70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd654100630 0x7fd654072900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd654106640 0x7fd6541a48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 --2- 172.21.15.3:0/3863957221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd654100ff0 0x7fd654072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 >> 172.21.15.3:0/3863957221 conn(0x7fd6540fc140 msgr2=0x7fd654104140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 shutdown_connections 2023-09-10T01:15:21.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.327+0000 7fd659dff700 1 -- 172.21.15.3:0/3863957221 wait complete. 2023-09-10T01:15:21.331 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 35 2023-09-10T01:15:21.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:15:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 -- 172.21.15.3:0/2838909432 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438072a30 msgr2=0x7f5438072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 --2- 172.21.15.3:0/2838909432 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438072a30 0x7f5438072e10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5434009990 tx=0x7f5434009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 -- 172.21.15.3:0/2838909432 shutdown_connections 2023-09-10T01:15:21.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 --2- 172.21.15.3:0/2838909432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f543810ce80 0x7f543810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 --2- 172.21.15.3:0/2838909432 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438072a30 0x7f5438072e10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 --2- 172.21.15.3:0/2838909432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5438073350 0x7f543810c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 -- 172.21.15.3:0/2838909432 >> 172.21.15.3:0/2838909432 conn(0x7f54380fc170 msgr2=0x7f54380fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 -- 172.21.15.3:0/2838909432 shutdown_connections 2023-09-10T01:15:21.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.554+0000 7f54401ff700 1 -- 172.21.15.3:0/2838909432 wait complete. 2023-09-10T01:15:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.555+0000 7f54401ff700 1 Processor -- start 2023-09-10T01:15:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.555+0000 7f54401ff700 1 -- start start 2023-09-10T01:15:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f54401ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5438072a30 0x7f543819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f54401ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438073350 0x7f543819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f54401ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f543810ce80 0x7f5438198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f54401ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438198dd0 con 0x7f5438073350 2023-09-10T01:15:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f54401ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438198f40 con 0x7f5438072a30 2023-09-10T01:15:21.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f54401ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438199100 con 0x7f543810ce80 2023-09-10T01:15:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f543df9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5438072a30 0x7f543819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f543df9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5438072a30 0x7f543819e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53846/0 (socket says 172.21.15.3:53846) 2023-09-10T01:15:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f543df9b700 1 -- 172.21.15.3:0/1679467092 learned_addr learned my addr 172.21.15.3:0/1679467092 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f543d79a700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438073350 0x7f543819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.556+0000 7f543e79c700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f543810ce80 0x7f5438198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.557+0000 7f543df9b700 1 -- 172.21.15.3:0/1679467092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f543810ce80 msgr2=0x7f5438198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.557+0000 7f543df9b700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f543810ce80 0x7f5438198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.557+0000 7f543df9b700 1 -- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438073350 msgr2=0x7f543819ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.557+0000 7f543df9b700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438073350 0x7f543819ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.557+0000 7f543df9b700 1 -- 172.21.15.3:0/1679467092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54340095f0 con 0x7f5438072a30 2023-09-10T01:15:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.557+0000 7f543e79c700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f543810ce80 0x7f5438198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:21.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.557+0000 7f543df9b700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5438072a30 0x7f543819e6f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5434000c00 tx=0x7f543400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.558+0000 7f542effd700 1 -- 172.21.15.3:0/1679467092 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543401d070 con 0x7f5438072a30 2023-09-10T01:15:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.558+0000 7f542effd700 1 -- 172.21.15.3:0/1679467092 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f543400be40 con 0x7f5438072a30 2023-09-10T01:15:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.558+0000 7f542effd700 1 -- 172.21.15.3:0/1679467092 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434017650 con 0x7f5438072a30 2023-09-10T01:15:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.558+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5438199380 con 0x7f5438072a30 2023-09-10T01:15:21.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.558+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54381a56e0 con 0x7f5438072a30 2023-09-10T01:15:21.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.564+0000 7f5453aa2b80 1 Processor -- start 2023-09-10T01:15:21.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.565+0000 7f5453aa2b80 1 -- start start 2023-09-10T01:15:21.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.565+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f25baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.565+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25c030 0x55896f265900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.565+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f265e40 0x55896f268230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.566+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f1b4730 con 0x55896f25b710 2023-09-10T01:15:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.566+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f1b2050 con 0x55896f265e40 2023-09-10T01:15:21.568 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.566+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f1ad940 con 0x55896f25c030 2023-09-10T01:15:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.566+0000 7f543e79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f25baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.566+0000 7f543e79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f25baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39866/0 (socket says 172.21.15.3:39866) 2023-09-10T01:15:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.566+0000 7f543e79c700 1 -- 172.21.15.3:0/4104034855 learned_addr learned my addr 172.21.15.3:0/4104034855 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.566+0000 7f543d79a700 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f265e40 0x55896f268230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f543e79c700 1 -- 172.21.15.3:0/4104034855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25c030 msgr2=0x55896f265900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:15:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f543e79c700 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25c030 0x55896f265900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f543e79c700 1 -- 172.21.15.3:0/4104034855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f265e40 msgr2=0x55896f268230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f543e79c700 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f265e40 0x55896f268230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f543e79c700 1 -- 172.21.15.3:0/4104034855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55896efd9de0 con 0x55896f25b710 2023-09-10T01:15:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f543e79c700 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f25baf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f54300093e0 tx=0x7f5430009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d0e5d69fff897b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f542effd700 1 -- 172.21.15.3:0/1679467092 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f5434022470 con 0x7f5438072a30 2023-09-10T01:15:21.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f5423fff700 1 -- 172.21.15.3:0/4104034855 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543000e070 con 0x55896f25b710 2023-09-10T01:15:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.567+0000 7f5423fff700 1 -- 172.21.15.3:0/4104034855 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f543000f040 con 0x55896f25b710 2023-09-10T01:15:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.568+0000 7f5423fff700 1 -- 172.21.15.3:0/4104034855 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5430015440 con 0x55896f25b710 2023-09-10T01:15:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.568+0000 7f5453aa2b80 1 -- 172.21.15.3:0/4104034855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 msgr2=0x55896f25baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.568+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f25baf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f54300093e0 tx=0x7f5430009920 comp rx=0 tx=0).stop 2023-09-10T01:15:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.569+0000 7f5453aa2b80 1 -- 172.21.15.3:0/4104034855 shutdown_connections 2023-09-10T01:15:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.569+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f265e40 0x55896f268230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.569+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f25baf0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.569+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/4104034855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25c030 0x55896f265900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.576 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.569+0000 7f5453aa2b80 1 -- 172.21.15.3:0/4104034855 >> 172.21.15.3:0/4104034855 conn(0x55896f1d0090 msgr2=0x55896f256bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.569+0000 7f5453aa2b80 1 -- 172.21.15.3:0/4104034855 shutdown_connections 2023-09-10T01:15:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.570+0000 7f5453aa2b80 1 -- 172.21.15.3:0/4104034855 wait complete. 2023-09-10T01:15:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.570+0000 7f542effd700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f54240a0740 0x7f54240a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.577 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.570+0000 7f542effd700 1 -- 172.21.15.3:0/1679467092 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f54340cd890 con 0x7f5438072a30 2023-09-10T01:15:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.570+0000 7f543d79a700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f54240a0740 0x7f54240a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.571+0000 7f5453aa2b80 1 -- start start 2023-09-10T01:15:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.571+0000 7f5453aa2b80 1 Processor -- start 2023-09-10T01:15:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.571+0000 7f543d79a700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f54240a0740 0x7f54240a2c00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5438199bd0 tx=0x7f542800b410 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.571+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f262b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.571+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f263080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.571+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f14bb70 con 0x55896f25b710 2023-09-10T01:15:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f14b870 con 0x55896f25c030 2023-09-10T01:15:21.579 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f02f920 con 0x55896f265e40 2023-09-10T01:15:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f5453aa2b80 10 client.0 _ll_register_callbacks cb 0x55896f095a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f262b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543d79a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f263080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f262b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39872/0 (socket says 172.21.15.3:39872) 2023-09-10T01:15:21.580 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543d79a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f263080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53872/0 (socket says 172.21.15.3:53872) 2023-09-10T01:15:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 -- 172.21.15.3:0/2505809430 learned_addr learned my addr 172.21.15.3:0/2505809430 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543e79c700 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 -- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 msgr2=0x55896f2635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.581 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 -- 172.21.15.3:0/2505809430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 msgr2=0x55896f263080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f263080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 -- 172.21.15.3:0/2505809430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5430009090 con 0x55896f25b710 2023-09-10T01:15:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f543df9b700 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f262b40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5434029000 tx=0x7f5434022f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b3b37568c86a5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.572+0000 7f54217fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:21.582 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f54217fa700 1 -- 172.21.15.3:0/2505809430 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543409a870 con 0x55896f25b710 2023-09-10T01:15:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f54217fa700 1 -- 172.21.15.3:0/2505809430 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f543409b1a0 con 0x55896f25b710 2023-09-10T01:15:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55896f1054f0 con 0x55896f25b710 2023-09-10T01:15:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f54217fa700 1 -- 172.21.15.3:0/2505809430 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f543409a5e0 con 0x55896f25b710 2023-09-10T01:15:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f5453aa2b80 10 client.18258 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:21.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55896efc2c60 con 0x55896f25b710 2023-09-10T01:15:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f54217fa700 1 -- 172.21.15.3:0/2505809430 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55896f1054f0 con 0x55896f25b710 2023-09-10T01:15:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f54217fa700 1 -- 172.21.15.3:0/2505809430 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f54340cd520 con 0x55896f25b710 2023-09-10T01:15:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f5453aa2b80 10 client.18258 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f5453aa2b80 10 client.18258 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:15:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.573+0000 7f5453aa2b80 20 client.18258 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.574+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f27de60 0x55896f280280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.574+0000 7f5453aa2b80 4 client.18258 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.574+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55896f1c28a0 con 0x55896f27de60 2023-09-10T01:15:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.574+0000 7f543d79a700 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f27de60 0x55896f280280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.574+0000 7f543d79a700 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f27de60 0x55896f280280 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41a0ed91521728d4 server_cookie=d82312b4ecb97583 in_seq=0 out_seq=0 2023-09-10T01:15:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.574+0000 7f54217fa700 10 client.18258 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f54217fa700 1 -- 172.21.15.3:0/2505809430 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55896f1c28a0 con 0x55896f27de60 2023-09-10T01:15:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f54217fa700 10 client.18258 handle_command_reply: tid=0 2023-09-10T01:15:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 client.18258 shutdown 2023-09-10T01:15:21.586 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 msgr2=0x55896f262b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f262b40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5434029000 tx=0x7f5434022f90 comp rx=0 tx=0).stop 2023-09-10T01:15:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 shutdown_connections 2023-09-10T01:15:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f27de60 0x55896f280280 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f262b40 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f54217fa700 0 client.18258 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/2505809430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f263080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.575+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 >> 172.21.15.3:0/2505809430 conn(0x55896f1d0090 msgr2=0x55896f2567e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.576+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 shutdown_connections 2023-09-10T01:15:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.576+0000 7f5453aa2b80 1 -- 172.21.15.3:0/2505809430 wait complete. 2023-09-10T01:15:21.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.576+0000 7f5453aa2b80 20 client.18258 trim_cache size 0 max 16384 2023-09-10T01:15:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.589+0000 7f5453aa2b80 1 Processor -- start 2023-09-10T01:15:21.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.589+0000 7f5453aa2b80 1 -- start start 2023-09-10T01:15:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.589+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25b710 0x55896f2633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f25de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f265e40 0x55896f25e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f2571b0 con 0x55896f265e40 2023-09-10T01:15:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f250da0 con 0x55896f25c030 2023-09-10T01:15:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f2610f0 con 0x55896f25b710 2023-09-10T01:15:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f265e40 0x55896f25e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f265e40 0x55896f25e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39880/0 (socket says 172.21.15.3:39880) 2023-09-10T01:15:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 -- 172.21.15.3:0/816951596 learned_addr learned my addr 172.21.15.3:0/816951596 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543d79a700 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f25de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543df9b700 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25b710 0x55896f2633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 -- 172.21.15.3:0/816951596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25b710 msgr2=0x55896f2633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25b710 0x55896f2633f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 -- 172.21.15.3:0/816951596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 msgr2=0x55896f25de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f25de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 -- 172.21.15.3:0/816951596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55896f000250 con 0x55896f265e40 2023-09-10T01:15:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.590+0000 7f543e79c700 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f265e40 0x55896f25e380 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f54300096d0 tx=0x7f54300045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc69a09183b5ad03 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.595 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f542cff9700 1 -- 172.21.15.3:0/816951596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543000e040 con 0x55896f265e40 2023-09-10T01:15:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f542cff9700 1 -- 172.21.15.3:0/816951596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5430004820 con 0x55896f265e40 2023-09-10T01:15:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f542cff9700 1 -- 172.21.15.3:0/816951596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5430004400 con 0x55896f265e40 2023-09-10T01:15:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 -- 172.21.15.3:0/816951596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f265e40 msgr2=0x55896f25e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f265e40 0x55896f25e380 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f54300096d0 tx=0x7f54300045d0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.596 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 -- 172.21.15.3:0/816951596 shutdown_connections 2023-09-10T01:15:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f25b710 0x55896f2633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f265e40 0x55896f25e380 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/816951596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f25de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 -- 172.21.15.3:0/816951596 >> 172.21.15.3:0/816951596 conn(0x55896f1d0090 msgr2=0x55896f2566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.591+0000 7f5453aa2b80 1 -- 172.21.15.3:0/816951596 shutdown_connections 2023-09-10T01:15:21.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.592+0000 7f5453aa2b80 1 -- 172.21.15.3:0/816951596 wait complete. 2023-09-10T01:15:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.592+0000 7f5453aa2b80 1 -- start start 2023-09-10T01:15:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.592+0000 7f5453aa2b80 1 Processor -- start 2023-09-10T01:15:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f273870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f273db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f5453aa2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2742f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f2d1cc0 con 0x55896f25b710 2023-09-10T01:15:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f2d1e00 con 0x55896f25c030 2023-09-10T01:15:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f5453aa2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896f2d1f40 con 0x55896f265e40 2023-09-10T01:15:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f5453aa2b80 10 client.0 _ll_register_callbacks cb 0x55896f095a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:21.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f543df9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f273870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f543df9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f273870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39884/0 (socket says 172.21.15.3:39884) 2023-09-10T01:15:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f543df9b700 1 -- 172.21.15.3:0/1292072356 learned_addr learned my addr 172.21.15.3:0/1292072356 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f543d79a700 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f273db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.593+0000 7f543e79c700 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f543df9b700 1 -- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 msgr2=0x55896f2742f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f543df9b700 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2742f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f543df9b700 1 -- 172.21.15.3:0/1292072356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 msgr2=0x55896f273db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f543df9b700 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f273db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f543df9b700 1 -- 172.21.15.3:0/1292072356 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55896efd9de0 con 0x55896f25b710 2023-09-10T01:15:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f543df9b700 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f273870 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f54340a2690 tx=0x7f5434009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef3175057355d5b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f54217fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f54217fa700 1 -- 172.21.15.3:0/1292072356 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543409a5e0 con 0x55896f25b710 2023-09-10T01:15:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f543000f040 con 0x55896f25b710 2023-09-10T01:15:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f54217fa700 1 -- 172.21.15.3:0/1292072356 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f543409b1a0 con 0x55896f25b710 2023-09-10T01:15:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f54217fa700 1 -- 172.21.15.3:0/1292072356 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f543409a400 con 0x55896f25b710 2023-09-10T01:15:21.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f5453aa2b80 10 client.18264 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.594+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55896efd82b0 con 0x55896f25b710 2023-09-10T01:15:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f54217fa700 1 -- 172.21.15.3:0/1292072356 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5434083030 con 0x55896f25b710 2023-09-10T01:15:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f54217fa700 1 -- 172.21.15.3:0/1292072356 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f54340cd500 con 0x55896f25b710 2023-09-10T01:15:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f5453aa2b80 10 client.18264 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:21.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f5453aa2b80 10 client.18264 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:15:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f5453aa2b80 20 client.18264 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f2d3170 0x55896f2d5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f5453aa2b80 4 client.18264 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:15:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55896f1a6920 con 0x55896f2d3170 2023-09-10T01:15:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.595+0000 7f543d79a700 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f2d3170 0x55896f2d5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f543d79a700 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f2d3170 0x55896f2d5590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=49d9f9d3e3db05a2 server_cookie=4c964bb68c319b69 in_seq=0 out_seq=0 2023-09-10T01:15:21.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f54217fa700 10 client.18264 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:21.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f54217fa700 1 -- 172.21.15.3:0/1292072356 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55896f1a6920 con 0x55896f2d3170 2023-09-10T01:15:21.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f54217fa700 10 client.18264 handle_command_reply: tid=0 2023-09-10T01:15:21.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 client.18264 shutdown 2023-09-10T01:15:21.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 msgr2=0x55896f273870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f273870 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f54340a2690 tx=0x7f5434009390 comp rx=0 tx=0).stop 2023-09-10T01:15:21.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 shutdown_connections 2023-09-10T01:15:21.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55896f2d3170 0x55896f2d5590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55896f265e40 0x55896f2742f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55896f25b710 0x55896f273870 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 --2- 172.21.15.3:0/1292072356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55896f25c030 0x55896f273db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f54217fa700 0 client.18264 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:21.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 >> 172.21.15.3:0/1292072356 conn(0x55896f1d0090 msgr2=0x55896f25de40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 shutdown_connections 2023-09-10T01:15:21.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 1 -- 172.21.15.3:0/1292072356 wait complete. 2023-09-10T01:15:21.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.596+0000 7f5453aa2b80 20 client.18264 trim_cache size 0 max 16384 2023-09-10T01:15:21.611 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:15:21.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f54240a0740 msgr2=0x7f54240a2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f54240a0740 0x7f54240a2c00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5438199bd0 tx=0x7f542800b410 comp rx=0 tx=0).stop 2023-09-10T01:15:21.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5438072a30 msgr2=0x7f543819e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5438072a30 0x7f543819e6f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5434000c00 tx=0x7f543400f690 comp rx=0 tx=0).stop 2023-09-10T01:15:21.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 shutdown_connections 2023-09-10T01:15:21.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f54240a0740 0x7f54240a2c00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f543810ce80 0x7f5438198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5438072a30 0x7f543819e6f0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 --2- 172.21.15.3:0/1679467092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5438073350 0x7f543819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 >> 172.21.15.3:0/1679467092 conn(0x7f54380fc170 msgr2=0x7f543810efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 shutdown_connections 2023-09-10T01:15:21.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.597+0000 7f54401ff700 1 -- 172.21.15.3:0/1679467092 wait complete. 2023-09-10T01:15:21.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:15:21.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 995497 7.76 MB/sec execute 543 sec latency 3655.032 ms 2023-09-10T01:15:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 -- 172.21.15.3:0/3836464885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0100640 msgr2=0x7f4db0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 --2- 172.21.15.3:0/3836464885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0100640 0x7f4db0100ac0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009990 tx=0x7f4da0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 -- 172.21.15.3:0/3836464885 shutdown_connections 2023-09-10T01:15:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 --2- 172.21.15.3:0/3836464885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0106650 0x7f4db0106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 --2- 172.21.15.3:0/3836464885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db0101000 0x7f4db010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 --2- 172.21.15.3:0/3836464885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0100640 0x7f4db0100ac0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 -- 172.21.15.3:0/3836464885 >> 172.21.15.3:0/3836464885 conn(0x7f4db00fc170 msgr2=0x7f4db00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 -- 172.21.15.3:0/3836464885 shutdown_connections 2023-09-10T01:15:21.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.822+0000 7f4db7bff700 1 -- 172.21.15.3:0/3836464885 wait complete. 2023-09-10T01:15:21.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.823+0000 7f4db7bff700 1 Processor -- start 2023-09-10T01:15:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.824+0000 7f4db7bff700 1 -- start start 2023-09-10T01:15:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.824+0000 7f4db7bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0100640 0x7f4db019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.824+0000 7f4db7bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db0101000 0x7f4db019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.824+0000 7f4db7bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0106650 0x7f4db0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.824+0000 7f4db7bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0198dc0 con 0x7f4db0101000 2023-09-10T01:15:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.824+0000 7f4db7bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0198f30 con 0x7f4db0106650 2023-09-10T01:15:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.824+0000 7f4db7bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db01990f0 con 0x7f4db0100640 2023-09-10T01:15:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0100640 0x7f4db019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0100640 0x7f4db019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39326/0 (socket says 172.21.15.3:39326) 2023-09-10T01:15:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 -- 172.21.15.3:0/544473286 learned_addr learned my addr 172.21.15.3:0/544473286 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db519a700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db0101000 0x7f4db019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db619c700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0106650 0x7f4db0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 -- 172.21.15.3:0/544473286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0106650 msgr2=0x7f4db0198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0106650 0x7f4db0198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 -- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db0101000 msgr2=0x7f4db019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db0101000 0x7f4db019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 -- 172.21.15.3:0/544473286 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da00095f0 con 0x7f4db0100640 2023-09-10T01:15:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4db599b700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0100640 0x7f4db019e6e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00e9a0 tx=0x7f4dac00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.825+0000 7f4da6ffd700 1 -- 172.21.15.3:0/544473286 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac009790 con 0x7f4db0100640 2023-09-10T01:15:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.826+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db01993d0 con 0x7f4db0100640 2023-09-10T01:15:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.826+0000 7f4da6ffd700 1 -- 172.21.15.3:0/544473286 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4dac010430 con 0x7f4db0100640 2023-09-10T01:15:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.826+0000 7f4da6ffd700 1 -- 172.21.15.3:0/544473286 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac018680 con 0x7f4db0100640 2023-09-10T01:15:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.826+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db01a56d0 con 0x7f4db0100640 2023-09-10T01:15:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.829+0000 7f4da6ffd700 1 -- 172.21.15.3:0/544473286 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f4dac0105a0 con 0x7f4db0100640 2023-09-10T01:15:21.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.830+0000 7f4dcb47ab80 1 Processor -- start 2023-09-10T01:15:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.830+0000 7f4dcb47ab80 1 -- start start 2023-09-10T01:15:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.830+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.830+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378da900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.830+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378dae40 0x5625378dd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562537829730 con 0x5625378d1030 2023-09-10T01:15:21.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562537827050 con 0x5625378d0710 2023-09-10T01:15:21.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562537822940 con 0x5625378dae40 2023-09-10T01:15:21.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db599b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378da900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db619c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db519a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378dae40 0x5625378dd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db619c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378d0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53912/0 (socket says 172.21.15.3:53912) 2023-09-10T01:15:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db619c700 1 -- 172.21.15.3:0/3497135894 learned_addr learned my addr 172.21.15.3:0/3497135894 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4da6ffd700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d9c0a07b0 0x7f4d9c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db519a700 1 -- 172.21.15.3:0/3497135894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 msgr2=0x5625378d0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db519a700 1 --2- 172.21.15.3:0/3497135894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378d0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db519a700 1 -- 172.21.15.3:0/3497135894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 msgr2=0x5625378da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db519a700 1 --2- 172.21.15.3:0/3497135894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378da900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db519a700 1 -- 172.21.15.3:0/3497135894 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56253764ede0 con 0x5625378dae40 2023-09-10T01:15:21.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4da6ffd700 1 -- 172.21.15.3:0/544473286 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f4dac014070 con 0x7f4db0100640 2023-09-10T01:15:21.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.831+0000 7f4db619c700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d9c0a07b0 0x7f4d9c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.832+0000 7f4db519a700 1 --2- 172.21.15.3:0/3497135894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378dae40 0x5625378dd230 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4da000b2e0 tx=0x7f4da0005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718910f3754d3d97 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.832+0000 7f4d9bfff700 1 -- 172.21.15.3:0/3497135894 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da001d070 con 0x5625378dae40 2023-09-10T01:15:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.832+0000 7f4db619c700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d9c0a07b0 0x7f4d9c0a2c70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009550 tx=0x7f4da8009040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.832+0000 7f4d9bfff700 1 -- 172.21.15.3:0/3497135894 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4da001e040 con 0x5625378dae40 2023-09-10T01:15:21.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.832+0000 7f4d9bfff700 1 -- 172.21.15.3:0/3497135894 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da00047f0 con 0x5625378dae40 2023-09-10T01:15:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3497135894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378dae40 msgr2=0x5625378dd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3497135894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378dae40 0x5625378dd230 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4da000b2e0 tx=0x7f4da0005ea0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3497135894 shutdown_connections 2023-09-10T01:15:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3497135894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378dae40 0x5625378dd230 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3497135894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378d0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3497135894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378da900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3497135894 >> 172.21.15.3:0/3497135894 conn(0x562537845090 msgr2=0x5625378cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3497135894 shutdown_connections 2023-09-10T01:15:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.833+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3497135894 wait complete. 2023-09-10T01:15:21.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 -- start start 2023-09-10T01:15:21.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 Processor -- start 2023-09-10T01:15:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378da7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378d7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378dae40 0x5625378d7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625377c0b70 con 0x5625378dae40 2023-09-10T01:15:21.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625377c0870 con 0x5625378d0710 2023-09-10T01:15:21.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625376a4920 con 0x5625378d1030 2023-09-10T01:15:21.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.834+0000 7f4dcb47ab80 10 client.0 _ll_register_callbacks cb 0x56253770aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:21.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db599b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378da7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378d7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db619c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378dae40 0x5625378d7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378d7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39358/0 (socket says 172.21.15.3:39358) 2023-09-10T01:15:21.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 -- 172.21.15.3:0/884878451 learned_addr learned my addr 172.21.15.3:0/884878451 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db599b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378da7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53916/0 (socket says 172.21.15.3:53916) 2023-09-10T01:15:21.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 -- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 msgr2=0x5625378da7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378da7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 -- 172.21.15.3:0/884878451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378dae40 msgr2=0x5625378d7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378dae40 0x5625378d7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 -- 172.21.15.3:0/884878451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db010de10 con 0x5625378d1030 2023-09-10T01:15:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db599b700 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378da7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:21.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4db519a700 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378d7a20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009990 tx=0x7f4da000f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4b8b9888459e5a5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.835+0000 7f4d997fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:15:21.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4d997fa700 1 -- 172.21.15.3:0/884878451 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da001d040 con 0x5625378d1030 2023-09-10T01:15:21.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56253777a4f0 con 0x5625378d1030 2023-09-10T01:15:21.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4d997fa700 1 -- 172.21.15.3:0/884878451 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f4da0005980 con 0x5625378d1030 2023-09-10T01:15:21.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4d997fa700 1 -- 172.21.15.3:0/884878451 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4da0005460 con 0x5625378d1030 2023-09-10T01:15:21.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4dcb47ab80 10 client.28243 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:21.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x562537637c60 con 0x5625378d1030 2023-09-10T01:15:21.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4d997fa700 1 -- 172.21.15.3:0/884878451 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56253777a4f0 con 0x5625378d1030 2023-09-10T01:15:21.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4d997fa700 1 -- 172.21.15.3:0/884878451 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f4da0004a00 con 0x5625378d1030 2023-09-10T01:15:21.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4dcb47ab80 10 client.28243 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:21.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4dcb47ab80 10 client.28243 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:15:21.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.836+0000 7f4dcb47ab80 20 client.28243 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:21.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.837+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5625378ea320 0x5625378ec710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.837+0000 7f4dcb47ab80 4 client.28243 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:21.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.837+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625378378a0 con 0x5625378ea320 2023-09-10T01:15:21.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.837+0000 7f4db599b700 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5625378ea320 0x5625378ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.837+0000 7f4db599b700 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5625378ea320 0x5625378ec710 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=795e8b8f76c08ab8 server_cookie=edb5155c604a4834 in_seq=0 out_seq=0 2023-09-10T01:15:21.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.837+0000 7f4d997fa700 10 client.28243 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:21.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.838+0000 7f4d997fa700 1 -- 172.21.15.3:0/884878451 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5625378378a0 con 0x5625378ea320 2023-09-10T01:15:21.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.838+0000 7f4d997fa700 10 client.28243 handle_command_reply: tid=0 2023-09-10T01:15:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.838+0000 7f4dcb47ab80 1 client.28243 shutdown 2023-09-10T01:15:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 msgr2=0x5625378d7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378d7a20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009990 tx=0x7f4da000f710 comp rx=0 tx=0).stop 2023-09-10T01:15:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 shutdown_connections 2023-09-10T01:15:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378dae40 0x5625378d7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378d0710 0x5625378da7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5625378ea320 0x5625378ec710 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/884878451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378d7a20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4d997fa700 0 client.28243 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:21.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 >> 172.21.15.3:0/884878451 conn(0x562537845090 msgr2=0x5625378cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 shutdown_connections 2023-09-10T01:15:21.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/884878451 wait complete. 2023-09-10T01:15:21.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.839+0000 7f4dcb47ab80 20 client.28243 trim_cache size 0 max 16384 2023-09-10T01:15:21.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.852+0000 7f4dcb47ab80 1 Processor -- start 2023-09-10T01:15:21.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.852+0000 7f4dcb47ab80 1 -- start start 2023-09-10T01:15:21.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d0710 0x5625378e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378d6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625378dd5f0 con 0x5625378d0710 2023-09-10T01:15:21.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625378d0af0 con 0x5625378dae40 2023-09-10T01:15:21.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625378e7080 con 0x5625378d1030 2023-09-10T01:15:21.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d0710 0x5625378e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d0710 0x5625378e6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39932/0 (socket says 172.21.15.3:39932) 2023-09-10T01:15:21.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 -- 172.21.15.3:0/3253961069 learned_addr learned my addr 172.21.15.3:0/3253961069 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db619c700 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378d6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db519a700 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 -- 172.21.15.3:0/3253961069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 msgr2=0x5625378da2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378da2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 -- 172.21.15.3:0/3253961069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 msgr2=0x5625378d6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378d6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.853+0000 7f4db599b700 1 -- 172.21.15.3:0/3253961069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562537675250 con 0x5625378d0710 2023-09-10T01:15:21.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.854+0000 7f4db599b700 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d0710 0x5625378e6b40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4dac095a60 tx=0x7f4dac0ce7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b54f4097f7a58f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.854+0000 7f4da4bff700 1 -- 172.21.15.3:0/3253961069 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac098ec0 con 0x5625378d0710 2023-09-10T01:15:21.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.854+0000 7f4da4bff700 1 -- 172.21.15.3:0/3253961069 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4dac099c60 con 0x5625378d0710 2023-09-10T01:15:21.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.854+0000 7f4da4bff700 1 -- 172.21.15.3:0/3253961069 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac097990 con 0x5625378d0710 2023-09-10T01:15:21.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.854+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3253961069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d0710 msgr2=0x5625378e6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.854+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d0710 0x5625378e6b40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4dac095a60 tx=0x7f4dac0ce7c0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3253961069 shutdown_connections 2023-09-10T01:15:21.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378d6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d0710 0x5625378e6b40 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3253961069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d1030 0x5625378da2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3253961069 >> 172.21.15.3:0/3253961069 conn(0x562537845090 msgr2=0x5625378f1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3253961069 shutdown_connections 2023-09-10T01:15:21.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3253961069 wait complete. 2023-09-10T01:15:21.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 -- start start 2023-09-10T01:15:21.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.855+0000 7f4dcb47ab80 1 Processor -- start 2023-09-10T01:15:21.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d0710 0x5625378e1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4dcb47ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378dfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625378e0430 con 0x5625378d1030 2023-09-10T01:15:21.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625378e05a0 con 0x5625378dae40 2023-09-10T01:15:21.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4dcb47ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625378e0760 con 0x5625378d0710 2023-09-10T01:15:21.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4dcb47ab80 10 client.0 _ll_register_callbacks cb 0x7f4dac095920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:21.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378df9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39940/0 (socket says 172.21.15.3:39940) 2023-09-10T01:15:21.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 -- 172.21.15.3:0/3093727442 learned_addr learned my addr 172.21.15.3:0/3093727442 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:21.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db619c700 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378dfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db599b700 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d0710 0x5625378e1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 -- 172.21.15.3:0/3093727442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d0710 msgr2=0x5625378e1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d0710 0x5625378e1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 -- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 msgr2=0x5625378dfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:21.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378dfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.856+0000 7f4db519a700 1 -- 172.21.15.3:0/3093727442 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db01a56d0 con 0x5625378d1030 2023-09-10T01:15:21.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4db519a700 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378df9b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4da0000c00 tx=0x7f4da0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8038425eda3bd9e1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:21.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4d997fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:21.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4d997fa700 1 -- 172.21.15.3:0/3093727442 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da0004700 con 0x5625378d1030 2023-09-10T01:15:21.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4da001e040 con 0x5625378d1030 2023-09-10T01:15:21.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4d997fa700 1 -- 172.21.15.3:0/3093727442 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f4da0000e40 con 0x5625378d1030 2023-09-10T01:15:21.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4d997fa700 1 -- 172.21.15.3:0/3093727442 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4da0022610 con 0x5625378d1030 2023-09-10T01:15:21.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4dcb47ab80 10 client.18282 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:21.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x56253764d2b0 con 0x5625378d1030 2023-09-10T01:15:21.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.857+0000 7f4d997fa700 1 -- 172.21.15.3:0/3093727442 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4da001e040 con 0x5625378d1030 2023-09-10T01:15:21.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4d997fa700 1 -- 172.21.15.3:0/3093727442 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f4da000f780 con 0x5625378d1030 2023-09-10T01:15:21.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4dcb47ab80 10 client.18282 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:21.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4dcb47ab80 10 client.18282 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:15:21.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4dcb47ab80 20 client.18282 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56253793f120 0x562537941540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4dcb47ab80 4 client.18282 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:15:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56253781b920 con 0x56253793f120 2023-09-10T01:15:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4db599b700 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56253793f120 0x562537941540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4db599b700 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56253793f120 0x562537941540 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7369265f2b097316 server_cookie=e587e1f8e1ed38e8 in_seq=0 out_seq=0 2023-09-10T01:15:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.858+0000 7f4d997fa700 10 client.18282 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4d997fa700 1 -- 172.21.15.3:0/3093727442 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56253781b920 con 0x56253793f120 2023-09-10T01:15:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4d997fa700 10 client.18282 handle_command_reply: tid=0 2023-09-10T01:15:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 client.18282 shutdown 2023-09-10T01:15:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 msgr2=0x5625378df9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378df9b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4da0000c00 tx=0x7f4da0005e80 comp rx=0 tx=0).stop 2023-09-10T01:15:21.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 shutdown_connections 2023-09-10T01:15:21.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56253793f120 0x562537941540 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5625378d0710 0x5625378e1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625378dae40 0x5625378dfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 --2- 172.21.15.3:0/3093727442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5625378d1030 0x5625378df9b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 >> 172.21.15.3:0/3093727442 conn(0x562537845090 msgr2=0x5625378d2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4d997fa700 0 client.18282 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:21.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 shutdown_connections 2023-09-10T01:15:21.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 1 -- 172.21.15.3:0/3093727442 wait complete. 2023-09-10T01:15:21.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.859+0000 7f4dcb47ab80 20 client.18282 trim_cache size 0 max 16384 2023-09-10T01:15:21.877 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:15:21.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d9c0a07b0 msgr2=0x7f4d9c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d9c0a07b0 0x7f4d9c0a2c70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009550 tx=0x7f4da8009040 comp rx=0 tx=0).stop 2023-09-10T01:15:21.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0100640 msgr2=0x7f4db019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:21.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0100640 0x7f4db019e6e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00e9a0 tx=0x7f4dac00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 shutdown_connections 2023-09-10T01:15:21.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d9c0a07b0 0x7f4d9c0a2c70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db0100640 0x7f4db019e6e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4db0106650 0x7f4db0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 --2- 172.21.15.3:0/544473286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db0101000 0x7f4db019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:21.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 >> 172.21.15.3:0/544473286 conn(0x7f4db00fc170 msgr2=0x7f4db010ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:21.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 shutdown_connections 2023-09-10T01:15:21.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:21.860+0000 7f4db7bff700 1 -- 172.21.15.3:0/544473286 wait complete. 2023-09-10T01:15:21.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:15:22.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.108+0000 7f712b9ff700 1 -- 172.21.15.3:0/4129793342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410edd0 msgr2=0x7f712410ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 --2- 172.21.15.3:0/4129793342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410edd0 0x7f712410ca80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f712000b0a0 tx=0x7f712000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 -- 172.21.15.3:0/4129793342 shutdown_connections 2023-09-10T01:15:22.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 --2- 172.21.15.3:0/4129793342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7124107600 0x7f7124107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 --2- 172.21.15.3:0/4129793342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410edd0 0x7f712410ca80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 --2- 172.21.15.3:0/4129793342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f712410f1b0 0x7f712410cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 -- 172.21.15.3:0/4129793342 >> 172.21.15.3:0/4129793342 conn(0x7f7124071dd0 msgr2=0x7f712406ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 -- 172.21.15.3:0/4129793342 shutdown_connections 2023-09-10T01:15:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.109+0000 7f712b9ff700 1 -- 172.21.15.3:0/4129793342 wait complete. 2023-09-10T01:15:22.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.110+0000 7f712b9ff700 1 Processor -- start 2023-09-10T01:15:22.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712b9ff700 1 -- start start 2023-09-10T01:15:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712b9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7124107600 0x7f71241a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712b9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f712410edd0 0x7f71241a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712b9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410f1b0 0x7f71241a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712b9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a95e0 con 0x7f712410f1b0 2023-09-10T01:15:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712b9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a9750 con 0x7f712410edd0 2023-09-10T01:15:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712b9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a9910 con 0x7f7124107600 2023-09-10T01:15:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.111+0000 7f712979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7124107600 0x7f71241a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7124107600 0x7f71241a56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39406/0 (socket says 172.21.15.3:39406) 2023-09-10T01:15:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 -- 172.21.15.3:0/3727968209 learned_addr learned my addr 172.21.15.3:0/3727968209 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:22.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f7128f9a700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f712410edd0 0x7f71241a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f7129f9c700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410f1b0 0x7f71241a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 -- 172.21.15.3:0/3727968209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f712410edd0 msgr2=0x7f71241a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f712410edd0 0x7f71241a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 -- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410f1b0 msgr2=0x7f71241a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410f1b0 0x7f71241a8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 -- 172.21.15.3:0/3727968209 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120009d00 con 0x7f7124107600 2023-09-10T01:15:22.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f712979b700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7124107600 0x7f71241a56f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7120006070 tx=0x7f71200079b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:22.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.112+0000 7f711a7fc700 1 -- 172.21.15.3:0/3727968209 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712000e040 con 0x7f7124107600 2023-09-10T01:15:22.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.113+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71241a9b90 con 0x7f7124107600 2023-09-10T01:15:22.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.113+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71241b2550 con 0x7f7124107600 2023-09-10T01:15:22.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.113+0000 7f711a7fc700 1 -- 172.21.15.3:0/3727968209 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7120009b80 con 0x7f7124107600 2023-09-10T01:15:22.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.113+0000 7f711a7fc700 1 -- 172.21.15.3:0/3727968209 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120004790 con 0x7f7124107600 2023-09-10T01:15:22.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.116+0000 7f711a7fc700 1 -- 172.21.15.3:0/3727968209 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7120004020 con 0x7f7124107600 2023-09-10T01:15:22.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.117+0000 7f713f261b80 1 Processor -- start 2023-09-10T01:15:22.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.117+0000 7f713f261b80 1 -- start start 2023-09-10T01:15:22.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f713f261b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e350af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f713f261b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e35a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f713f261b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f713f261b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e2a9730 con 0x56272e351030 2023-09-10T01:15:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f713f261b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e2a7050 con 0x56272e350710 2023-09-10T01:15:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f713f261b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e2a2940 con 0x56272e35ae40 2023-09-10T01:15:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e350af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e350af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53950/0 (socket says 172.21.15.3:53950) 2023-09-10T01:15:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 -- 172.21.15.3:0/623698489 learned_addr learned my addr 172.21.15.3:0/623698489 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f711a7fc700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71100a07b0 0x7f71100a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f712979b700 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e35a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7128f9a700 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 -- 172.21.15.3:0/623698489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 msgr2=0x56272e35d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f711a7fc700 1 -- 172.21.15.3:0/3727968209 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f71200cd860 con 0x7f7124107600 2023-09-10T01:15:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 -- 172.21.15.3:0/623698489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 msgr2=0x56272e35a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e35a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 -- 172.21.15.3:0/623698489 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56272e0cede0 con 0x56272e350710 2023-09-10T01:15:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7128f9a700 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.118+0000 7f7129f9c700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71100a07b0 0x7f71100a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.119+0000 7f7129f9c700 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e350af0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b7e0 tx=0x7f711c00baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3886cbbb513893fb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:22.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.119+0000 7f710f7fe700 1 -- 172.21.15.3:0/623698489 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c013b70 con 0x56272e350710 2023-09-10T01:15:22.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.119+0000 7f710f7fe700 1 -- 172.21.15.3:0/623698489 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f711c013cd0 con 0x56272e350710 2023-09-10T01:15:22.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.119+0000 7f710f7fe700 1 -- 172.21.15.3:0/623698489 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c011440 con 0x56272e350710 2023-09-10T01:15:22.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.119+0000 7f7129f9c700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71100a07b0 0x7f71100a2c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f711c021000 tx=0x7f711c01e000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 -- 172.21.15.3:0/623698489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 msgr2=0x56272e350af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e350af0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b7e0 tx=0x7f711c00baf0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 -- 172.21.15.3:0/623698489 shutdown_connections 2023-09-10T01:15:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e350af0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 --2- 172.21.15.3:0/623698489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e35a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 -- 172.21.15.3:0/623698489 >> 172.21.15.3:0/623698489 conn(0x56272e2c5090 msgr2=0x56272e34bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:22.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 -- 172.21.15.3:0/623698489 shutdown_connections 2023-09-10T01:15:22.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.120+0000 7f713f261b80 1 -- 172.21.15.3:0/623698489 wait complete. 2023-09-10T01:15:22.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.121+0000 7f713f261b80 1 -- start start 2023-09-10T01:15:22.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.121+0000 7f713f261b80 1 Processor -- start 2023-09-10T01:15:22.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.121+0000 7f713f261b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e350710 0x56272e35a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.121+0000 7f713f261b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e351030 0x56272e357a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f713f261b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e358000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f713f261b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e240b70 con 0x56272e350710 2023-09-10T01:15:22.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f713f261b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e240870 con 0x56272e351030 2023-09-10T01:15:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f713f261b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e124920 con 0x56272e35ae40 2023-09-10T01:15:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f713f261b80 10 client.0 _ll_register_callbacks cb 0x56272e18aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e350710 0x56272e35a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f7128f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e351030 0x56272e357a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e350710 0x56272e35a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39992/0 (socket says 172.21.15.3:39992) 2023-09-10T01:15:22.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 -- 172.21.15.3:0/2528333791 learned_addr learned my addr 172.21.15.3:0/2528333791 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f7129f9c700 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e358000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 -- 172.21.15.3:0/2528333791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 msgr2=0x56272e358000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e358000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 -- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e351030 msgr2=0x56272e357a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e351030 0x56272e357a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 -- 172.21.15.3:0/2528333791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711c0043a0 con 0x56272e350710 2023-09-10T01:15:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f7128f9a700 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e351030 0x56272e357a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f712979b700 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e350710 0x56272e35a7a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f71200973c0 tx=0x7f712000b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851416e1bf8655bb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.122+0000 7f710cff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:22.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f710cff9700 1 -- 172.21.15.3:0/2528333791 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120003c00 con 0x56272e350710 2023-09-10T01:15:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f710cff9700 1 -- 172.21.15.3:0/2528333791 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f712009a950 con 0x56272e350710 2023-09-10T01:15:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56272e1fa4f0 con 0x56272e350710 2023-09-10T01:15:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f710cff9700 1 -- 172.21.15.3:0/2528333791 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71200caff0 con 0x56272e350710 2023-09-10T01:15:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f713f261b80 10 client.18294 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x56272e0b7c60 con 0x56272e350710 2023-09-10T01:15:22.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f710cff9700 1 -- 172.21.15.3:0/2528333791 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56272e1fa4f0 con 0x56272e350710 2023-09-10T01:15:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f710cff9700 1 -- 172.21.15.3:0/2528333791 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f71200cc070 con 0x56272e350710 2023-09-10T01:15:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f713f261b80 10 client.18294 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.123+0000 7f713f261b80 10 client.18294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.124+0000 7f713f261b80 20 client.18294 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:22.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.124+0000 7f713f261b80 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e36a6f0 0x56272e36cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.124+0000 7f713f261b80 4 client.18294 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.124+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56272e2b78a0 con 0x56272e36a6f0 2023-09-10T01:15:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.124+0000 7f7128f9a700 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e36a6f0 0x56272e36cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.124+0000 7f7128f9a700 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e36a6f0 0x56272e36cb10 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aafcc5d4464f67dc server_cookie=3b3e77fc9775e634 in_seq=0 out_seq=0 2023-09-10T01:15:22.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.124+0000 7f710cff9700 10 client.18294 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f710cff9700 1 -- 172.21.15.3:0/2528333791 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56272e2b78a0 con 0x56272e36a6f0 2023-09-10T01:15:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f710cff9700 10 client.18294 handle_command_reply: tid=0 2023-09-10T01:15:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 client.18294 shutdown 2023-09-10T01:15:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e350710 msgr2=0x56272e35a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e350710 0x56272e35a7a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f71200973c0 tx=0x7f712000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 shutdown_connections 2023-09-10T01:15:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e36a6f0 0x56272e36cb10 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e358000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e350710 0x56272e35a7a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 --2- 172.21.15.3:0/2528333791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e351030 0x56272e357a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 >> 172.21.15.3:0/2528333791 conn(0x56272e2c5090 msgr2=0x56272e34b780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.125+0000 7f710cff9700 0 client.18294 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.126+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 shutdown_connections 2023-09-10T01:15:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.126+0000 7f713f261b80 1 -- 172.21.15.3:0/2528333791 wait complete. 2023-09-10T01:15:22.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.126+0000 7f713f261b80 20 client.18294 trim_cache size 0 max 16384 2023-09-10T01:15:22.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.139+0000 7f713f261b80 1 Processor -- start 2023-09-10T01:15:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.139+0000 7f713f261b80 1 -- start start 2023-09-10T01:15:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f713f261b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e350710 0x56272e35a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f713f261b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e352d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f713f261b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e35ae40 0x56272e353260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f713f261b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e1fa520 con 0x56272e351030 2023-09-10T01:15:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f713f261b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e351440 con 0x56272e35ae40 2023-09-10T01:15:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f713f261b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e3516d0 con 0x56272e350710 2023-09-10T01:15:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7128f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e352d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.144 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7129f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e35ae40 0x56272e353260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7128f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e352d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40002/0 (socket says 172.21.15.3:40002) 2023-09-10T01:15:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7128f9a700 1 -- 172.21.15.3:0/3169487324 learned_addr learned my addr 172.21.15.3:0/3169487324 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f712979b700 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e350710 0x56272e35a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7129f9c700 1 -- 172.21.15.3:0/3169487324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e350710 msgr2=0x56272e35a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.145 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7129f9c700 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e350710 0x56272e35a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7129f9c700 1 -- 172.21.15.3:0/3169487324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 msgr2=0x56272e352d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7129f9c700 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e352d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.140+0000 7f7129f9c700 1 -- 172.21.15.3:0/3169487324 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56272e0f5250 con 0x56272e35ae40 2023-09-10T01:15:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f7129f9c700 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e35ae40 0x56272e353260 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b7b0 tx=0x7f711c015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237de6ee2f913280 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:22.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f710ffff700 1 -- 172.21.15.3:0/3169487324 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c005ee0 con 0x56272e35ae40 2023-09-10T01:15:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f710ffff700 1 -- 172.21.15.3:0/3169487324 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f711c011440 con 0x56272e35ae40 2023-09-10T01:15:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f710ffff700 1 -- 172.21.15.3:0/3169487324 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c011740 con 0x56272e35ae40 2023-09-10T01:15:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 -- 172.21.15.3:0/3169487324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e35ae40 msgr2=0x56272e353260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e35ae40 0x56272e353260 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f711c00b7b0 tx=0x7f711c015040 comp rx=0 tx=0).stop 2023-09-10T01:15:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 -- 172.21.15.3:0/3169487324 shutdown_connections 2023-09-10T01:15:22.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e350710 0x56272e35a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e35ae40 0x56272e353260 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 --2- 172.21.15.3:0/3169487324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e352d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 -- 172.21.15.3:0/3169487324 >> 172.21.15.3:0/3169487324 conn(0x56272e2c5090 msgr2=0x56272e3661e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.141+0000 7f713f261b80 1 -- 172.21.15.3:0/3169487324 shutdown_connections 2023-09-10T01:15:22.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.142+0000 7f713f261b80 1 -- 172.21.15.3:0/3169487324 wait complete. 2023-09-10T01:15:22.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.142+0000 7f713f261b80 1 -- start start 2023-09-10T01:15:22.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.142+0000 7f713f261b80 1 Processor -- start 2023-09-10T01:15:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.142+0000 7f713f261b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e3616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.142+0000 7f713f261b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e361c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.142+0000 7f713f261b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.142+0000 7f713f261b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e360380 con 0x56272e351030 2023-09-10T01:15:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f713f261b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e3604f0 con 0x56272e350710 2023-09-10T01:15:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f713f261b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56272e3606b0 con 0x56272e35ae40 2023-09-10T01:15:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f713f261b80 10 client.0 _ll_register_callbacks cb 0x7f712009a490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e3616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f7128f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e361c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e3616c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53974/0 (socket says 172.21.15.3:53974) 2023-09-10T01:15:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 -- 172.21.15.3:0/910029451 learned_addr learned my addr 172.21.15.3:0/910029451 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f7129f9c700 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 -- 172.21.15.3:0/910029451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 msgr2=0x56272e35fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 -- 172.21.15.3:0/910029451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 msgr2=0x56272e361c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e361c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 -- 172.21.15.3:0/910029451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56272e0cede0 con 0x56272e350710 2023-09-10T01:15:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f7128f9a700 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e361c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:22.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f712979b700 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e3616c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7120007a20 tx=0x7f712009ae40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83b5d97c20dde63 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.143+0000 7f710cff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f710cff9700 1 -- 172.21.15.3:0/910029451 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120009960 con 0x56272e350710 2023-09-10T01:15:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56272e29b920 con 0x56272e350710 2023-09-10T01:15:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f710cff9700 1 -- 172.21.15.3:0/910029451 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f712009a950 con 0x56272e350710 2023-09-10T01:15:22.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f710cff9700 1 -- 172.21.15.3:0/910029451 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71200caff0 con 0x56272e350710 2023-09-10T01:15:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 10 client.28385 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x56272e0cd2b0 con 0x56272e350710 2023-09-10T01:15:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f710cff9700 1 -- 172.21.15.3:0/910029451 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56272e29b920 con 0x56272e350710 2023-09-10T01:15:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f710cff9700 1 -- 172.21.15.3:0/910029451 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f71200cc040 con 0x56272e350710 2023-09-10T01:15:22.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 10 client.28385 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 10 client.28385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 20 client.28385 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e3bf2b0 0x56272e3c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 4 client.28385 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:15:22.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.144+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56272e25c0b0 con 0x56272e3bf2b0 2023-09-10T01:15:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f7128f9a700 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e3bf2b0 0x56272e3c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f7128f9a700 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e3bf2b0 0x56272e3c16a0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b0ab3eecb68b5d server_cookie=a3e83b97b7a2759c in_seq=0 out_seq=0 2023-09-10T01:15:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f710cff9700 10 client.28385 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f710cff9700 1 -- 172.21.15.3:0/910029451 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56272e25c0b0 con 0x56272e3bf2b0 2023-09-10T01:15:22.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f710cff9700 10 client.28385 handle_command_reply: tid=0 2023-09-10T01:15:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 client.28385 shutdown 2023-09-10T01:15:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 msgr2=0x56272e3616c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e3616c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7120007a20 tx=0x7f712009ae40 comp rx=0 tx=0).stop 2023-09-10T01:15:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 shutdown_connections 2023-09-10T01:15:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56272e3bf2b0 0x56272e3c16a0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56272e35ae40 0x56272e35fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56272e350710 0x56272e3616c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f710cff9700 0 client.28385 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 --2- 172.21.15.3:0/910029451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56272e351030 0x56272e361c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.145+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 >> 172.21.15.3:0/910029451 conn(0x56272e2c5090 msgr2=0x56272e352d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:22.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 shutdown_connections 2023-09-10T01:15:22.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f713f261b80 1 -- 172.21.15.3:0/910029451 wait complete. 2023-09-10T01:15:22.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f713f261b80 20 client.28385 trim_cache size 0 max 16384 2023-09-10T01:15:22.162 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:15:22.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71100a07b0 msgr2=0x7f71100a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71100a07b0 0x7f71100a2c70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f711c021000 tx=0x7f711c01e000 comp rx=0 tx=0).stop 2023-09-10T01:15:22.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7124107600 msgr2=0x7f71241a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7124107600 0x7f71241a56f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7120006070 tx=0x7f71200079b0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 shutdown_connections 2023-09-10T01:15:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71100a07b0 0x7f71100a2c70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7124107600 0x7f71241a56f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f712410f1b0 0x7f71241a8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 --2- 172.21.15.3:0/3727968209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f712410edd0 0x7f71241a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 >> 172.21.15.3:0/3727968209 conn(0x7f7124071dd0 msgr2=0x7f7124109480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 shutdown_connections 2023-09-10T01:15:22.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:22.146+0000 7f712b9ff700 1 -- 172.21.15.3:0/3727968209 wait complete. 2023-09-10T01:15:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:22 smithi003 ceph-mon[97654]: pgmap v379: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.4 MiB/s wr, 151 op/s 2023-09-10T01:15:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:22 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3863957221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:15:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:22 smithi136 ceph-mon[133708]: pgmap v379: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.4 MiB/s wr, 151 op/s 2023-09-10T01:15:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:22 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3863957221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:15:22.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:22 smithi092 ceph-mon[134067]: pgmap v379: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.4 MiB/s wr, 151 op/s 2023-09-10T01:15:22.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:22 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3863957221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:15:22.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 995497 7.74 MB/sec execute 544 sec latency 4655.144 ms 2023-09-10T01:15:23.167 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:15:23.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:15:23.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.383+0000 7f45dadff700 1 -- 172.21.15.3:0/3467114366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 msgr2=0x7f45d4100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.383+0000 7f45dadff700 1 --2- 172.21.15.3:0/3467114366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4100ad0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009990 tx=0x7f45c0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.383+0000 7f45dadff700 1 -- 172.21.15.3:0/3467114366 shutdown_connections 2023-09-10T01:15:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.383+0000 7f45dadff700 1 --2- 172.21.15.3:0/3467114366 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45d4101010 0x7f45d410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.383+0000 7f45dadff700 1 --2- 172.21.15.3:0/3467114366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45d4106660 0x7f45d4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.383+0000 7f45dadff700 1 --2- 172.21.15.3:0/3467114366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4100ad0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.383+0000 7f45dadff700 1 -- 172.21.15.3:0/3467114366 >> 172.21.15.3:0/3467114366 conn(0x7f45d40fc160 msgr2=0x7f45d40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.384+0000 7f45dadff700 1 -- 172.21.15.3:0/3467114366 shutdown_connections 2023-09-10T01:15:23.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.384+0000 7f45dadff700 1 -- 172.21.15.3:0/3467114366 wait complete. 2023-09-10T01:15:23.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 Processor -- start 2023-09-10T01:15:23.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 -- start start 2023-09-10T01:15:23.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45d4101010 0x7f45d4072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45d4106660 0x7f45d41a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d4073670 con 0x7f45d4100650 2023-09-10T01:15:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d41a4e30 con 0x7f45d4106660 2023-09-10T01:15:23.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45dadff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d41a4f70 con 0x7f45d4101010 2023-09-10T01:15:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40026/0 (socket says 172.21.15.3:40026) 2023-09-10T01:15:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45d8b9b700 1 -- 172.21.15.3:0/3405988676 learned_addr learned my addr 172.21.15.3:0/3405988676 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.385+0000 7f45d3fff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45d4101010 0x7f45d4072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d939c700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45d4106660 0x7f45d41a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d8b9b700 1 -- 172.21.15.3:0/3405988676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45d4101010 msgr2=0x7f45d4072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45d4101010 0x7f45d4072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d8b9b700 1 -- 172.21.15.3:0/3405988676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45d4106660 msgr2=0x7f45d41a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45d4106660 0x7f45d41a48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d8b9b700 1 -- 172.21.15.3:0/3405988676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8009520 con 0x7f45d4100650 2023-09-10T01:15:23.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d3fff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45d4101010 0x7f45d4072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:23.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4072900 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f45c800eb30 tx=0x7f45c800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d1ffb700 1 -- 172.21.15.3:0/3405988676 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8009880 con 0x7f45d4100650 2023-09-10T01:15:23.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d1ffb700 1 -- 172.21.15.3:0/3405988676 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45c80044e0 con 0x7f45d4100650 2023-09-10T01:15:23.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45d1ffb700 1 -- 172.21.15.3:0/3405988676 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8003d80 con 0x7f45d4100650 2023-09-10T01:15:23.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.386+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c00095f0 con 0x7f45d4100650 2023-09-10T01:15:23.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.387+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d41a5520 con 0x7f45d4100650 2023-09-10T01:15:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.390+0000 7f45d1ffb700 1 -- 172.21.15.3:0/3405988676 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f45c8003f20 con 0x7f45d4100650 2023-09-10T01:15:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.391+0000 7f45ee7eab80 1 Processor -- start 2023-09-10T01:15:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.391+0000 7f45d1ffb700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45c40a0830 0x7f45c40a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.391+0000 7f45ee7eab80 1 -- start start 2023-09-10T01:15:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.391+0000 7f45d1ffb700 1 -- 172.21.15.3:0/3405988676 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f45c8014070 con 0x7f45d4100650 2023-09-10T01:15:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.391+0000 7f45d939c700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45c40a0830 0x7f45c40a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4a790 0x5583b1b4ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4b0b0 0x5583b1b54980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1aa37b0 con 0x5583b1b4b0b0 2023-09-10T01:15:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1aa10d0 con 0x5583b1b4a790 2023-09-10T01:15:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1a9c9c0 con 0x5583b1b54ec0 2023-09-10T01:15:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4b0b0 0x5583b1b54980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4a790 0x5583b1b4ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4b0b0 0x5583b1b54980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40034/0 (socket says 172.21.15.3:40034) 2023-09-10T01:15:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d3fff700 1 -- 172.21.15.3:0/733568070 learned_addr learned my addr 172.21.15.3:0/733568070 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d939c700 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d8b9b700 1 -- 172.21.15.3:0/733568070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 msgr2=0x5583b1b572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d939c700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45c40a0830 0x7f45c40a2cf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45d4074a10 tx=0x7f45cc00b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d8b9b700 1 -- 172.21.15.3:0/733568070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4b0b0 msgr2=0x5583b1b54980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4b0b0 0x5583b1b54980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d8b9b700 1 -- 172.21.15.3:0/733568070 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583b18c8e60 con 0x5583b1b4a790 2023-09-10T01:15:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.392+0000 7f45d3fff700 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4b0b0 0x5583b1b54980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.393+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4a790 0x5583b1b4ab70 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f45c80c9b20 tx=0x7f45c80991e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acfa53ad9930b746 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.393+0000 7f45beffd700 1 -- 172.21.15.3:0/733568070 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8099880 con 0x5583b1b4a790 2023-09-10T01:15:23.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.393+0000 7f45beffd700 1 -- 172.21.15.3:0/733568070 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45c809d6b0 con 0x5583b1b4a790 2023-09-10T01:15:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.393+0000 7f45beffd700 1 -- 172.21.15.3:0/733568070 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8099ae0 con 0x5583b1b4a790 2023-09-10T01:15:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.393+0000 7f45ee7eab80 1 -- 172.21.15.3:0/733568070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4a790 msgr2=0x5583b1b4ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.393+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4a790 0x5583b1b4ab70 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f45c80c9b20 tx=0x7f45c80991e0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 -- 172.21.15.3:0/733568070 shutdown_connections 2023-09-10T01:15:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4a790 0x5583b1b4ab70 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/733568070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4b0b0 0x5583b1b54980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 -- 172.21.15.3:0/733568070 >> 172.21.15.3:0/733568070 conn(0x5583b1abf110 msgr2=0x5583b1b45c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 -- 172.21.15.3:0/733568070 shutdown_connections 2023-09-10T01:15:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 -- 172.21.15.3:0/733568070 wait complete. 2023-09-10T01:15:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 -- start start 2023-09-10T01:15:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.394+0000 7f45ee7eab80 1 Processor -- start 2023-09-10T01:15:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b51c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1b52160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1a3abf0 con 0x5583b1b4a790 2023-09-10T01:15:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1a3a8f0 con 0x5583b1b4b0b0 2023-09-10T01:15:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b191e9a0 con 0x5583b1b54ec0 2023-09-10T01:15:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45ee7eab80 10 client.0 _ll_register_callbacks cb 0x5583b1984b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b51c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b51c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40050/0 (socket says 172.21.15.3:40050) 2023-09-10T01:15:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45d8b9b700 1 -- 172.21.15.3:0/2948770022 learned_addr learned my addr 172.21.15.3:0/2948770022 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45d3fff700 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1b52160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.395+0000 7f45d939c700 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45d8b9b700 1 -- 172.21.15.3:0/2948770022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 msgr2=0x5583b1b526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45d8b9b700 1 -- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 msgr2=0x5583b1b52160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1b52160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45d8b9b700 1 -- 172.21.15.3:0/2948770022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d410de20 con 0x5583b1b4a790 2023-09-10T01:15:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b51c20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f45c80c9b80 tx=0x7f45c8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1857d2e7fa7add0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45b3fff700 1 -- 172.21.15.3:0/2948770022 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c80a1970 con 0x5583b1b4a790 2023-09-10T01:15:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45b3fff700 1 -- 172.21.15.3:0/2948770022 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f45c800cdd0 con 0x5583b1b4a790 2023-09-10T01:15:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583b19f4570 con 0x5583b1b4a790 2023-09-10T01:15:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45b3fff700 1 -- 172.21.15.3:0/2948770022 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45c8098d70 con 0x5583b1b4a790 2023-09-10T01:15:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45ee7eab80 10 client.18312 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.396+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f45d41a5520 con 0x5583b1b4a790 2023-09-10T01:15:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45b3fff700 1 -- 172.21.15.3:0/2948770022 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5583b19f4570 con 0x5583b1b4a790 2023-09-10T01:15:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45b3fff700 1 -- 172.21.15.3:0/2948770022 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f45c807f020 con 0x5583b1b4a790 2023-09-10T01:15:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45ee7eab80 10 client.18312 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45ee7eab80 10 client.18312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45ee7eab80 20 client.18312 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1b64560 0x5583b1b66950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45ee7eab80 4 client.18312 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583b1ab1920 con 0x5583b1b64560 2023-09-10T01:15:23.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.397+0000 7f45d3fff700 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1b64560 0x5583b1b66950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.398+0000 7f45d3fff700 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1b64560 0x5583b1b66950 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50f74c49f387450a server_cookie=1fe8e0df6463c17d in_seq=0 out_seq=0 2023-09-10T01:15:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.398+0000 7f45b3fff700 10 client.18312 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.398+0000 7f45b3fff700 1 -- 172.21.15.3:0/2948770022 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5583b1ab1920 con 0x5583b1b64560 2023-09-10T01:15:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.398+0000 7f45b3fff700 10 client.18312 handle_command_reply: tid=0 2023-09-10T01:15:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 client.18312 shutdown 2023-09-10T01:15:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 msgr2=0x5583b1b51c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b51c20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f45c80c9b80 tx=0x7f45c8005850 comp rx=0 tx=0).stop 2023-09-10T01:15:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 shutdown_connections 2023-09-10T01:15:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1b64560 0x5583b1b66950 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1b526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b51c20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2948770022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1b52160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 >> 172.21.15.3:0/2948770022 conn(0x5583b1abf110 msgr2=0x5583b1b45860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45b3fff700 0 client.18312 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 shutdown_connections 2023-09-10T01:15:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2948770022 wait complete. 2023-09-10T01:15:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.399+0000 7f45ee7eab80 20 client.18312 trim_cache size 0 max 16384 2023-09-10T01:15:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 Processor -- start 2023-09-10T01:15:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 -- start start 2023-09-10T01:15:23.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b4b0b0 0x5583b1b4eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b54ec0 0x5583b1b4f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1b577d0 con 0x5583b1b4a790 2023-09-10T01:15:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1b4b490 con 0x5583b1b54ec0 2023-09-10T01:15:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.417+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1b3fdf0 con 0x5583b1b4b0b0 2023-09-10T01:15:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b4b0b0 0x5583b1b4eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d939c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b54ec0 0x5583b1b4f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b4b0b0 0x5583b1b4eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39482/0 (socket says 172.21.15.3:39482) 2023-09-10T01:15:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40058/0 (socket says 172.21.15.3:40058) 2023-09-10T01:15:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d939c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b54ec0 0x5583b1b4f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54022/0 (socket says 172.21.15.3:54022) 2023-09-10T01:15:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 -- 172.21.15.3:0/2760696008 learned_addr learned my addr 172.21.15.3:0/2760696008 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 -- 172.21.15.3:0/2760696008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b54ec0 msgr2=0x5583b1b4f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b54ec0 0x5583b1b4f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 -- 172.21.15.3:0/2760696008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 msgr2=0x5583b1b524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 -- 172.21.15.3:0/2760696008 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583b18b1ce0 con 0x5583b1b4b0b0 2023-09-10T01:15:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.418+0000 7f45d3fff700 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b4b0b0 0x5583b1b4eeb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009990 tx=0x7f45c0005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83aec2077dd03f49 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45bf7fe700 1 -- 172.21.15.3:0/2760696008 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c001d040 con 0x5583b1b4b0b0 2023-09-10T01:15:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45bf7fe700 1 -- 172.21.15.3:0/2760696008 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45c001e040 con 0x5583b1b4b0b0 2023-09-10T01:15:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45bf7fe700 1 -- 172.21.15.3:0/2760696008 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0004500 con 0x5583b1b4b0b0 2023-09-10T01:15:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2760696008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b4b0b0 msgr2=0x5583b1b4eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b4b0b0 0x5583b1b4eeb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009990 tx=0x7f45c0005fd0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2760696008 shutdown_connections 2023-09-10T01:15:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b54ec0 0x5583b1b4f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1b524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/2760696008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b4b0b0 0x5583b1b4eeb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2760696008 >> 172.21.15.3:0/2760696008 conn(0x5583b1abf110 msgr2=0x5583b1b4a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2760696008 shutdown_connections 2023-09-10T01:15:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.419+0000 7f45ee7eab80 1 -- 172.21.15.3:0/2760696008 wait complete. 2023-09-10T01:15:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 -- start start 2023-09-10T01:15:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 Processor -- start 2023-09-10T01:15:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1bae300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1bae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1baed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1bb9470 con 0x5583b1b4a790 2023-09-10T01:15:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1bb95b0 con 0x5583b1b4b0b0 2023-09-10T01:15:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b1bb9770 con 0x5583b1b54ec0 2023-09-10T01:15:23.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.420+0000 7f45ee7eab80 10 client.0 _ll_register_callbacks cb 0x7f45c8005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1bae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1bae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d939c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1baed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1bae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40070/0 (socket says 172.21.15.3:40070) 2023-09-10T01:15:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1bae840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54038/0 (socket says 172.21.15.3:54038) 2023-09-10T01:15:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 -- 172.21.15.3:0/125031864 learned_addr learned my addr 172.21.15.3:0/125031864 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 -- 172.21.15.3:0/125031864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 msgr2=0x5583b1baed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1baed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 -- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 msgr2=0x5583b1bae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1bae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 -- 172.21.15.3:0/125031864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d41a5520 con 0x5583b1b4a790 2023-09-10T01:15:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d3fff700 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1bae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45d8b9b700 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1bae300 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f45c800c5b0 tx=0x7f45c8099780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439418e3518ca688 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.421+0000 7f45b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45b3fff700 1 -- 172.21.15.3:0/125031864 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c80a1730 con 0x5583b1b4a790 2023-09-10T01:15:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45b3fff700 1 -- 172.21.15.3:0/125031864 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f45c800cdd0 con 0x5583b1b4a790 2023-09-10T01:15:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45c809d6b0 con 0x5583b1b4a790 2023-09-10T01:15:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45b3fff700 1 -- 172.21.15.3:0/125031864 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45c80a18f0 con 0x5583b1b4a790 2023-09-10T01:15:23.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 10 client.18318 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f45c000be80 con 0x5583b1b4a790 2023-09-10T01:15:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45b3fff700 1 -- 172.21.15.3:0/125031864 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45c8088080 con 0x5583b1b4a790 2023-09-10T01:15:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45b3fff700 1 -- 172.21.15.3:0/125031864 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f45c807f070 con 0x5583b1b4a790 2023-09-10T01:15:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 10 client.18318 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 10 client.18318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:23.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 20 client.18318 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1bbab20 0x5583b1bbcf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 4 client.18318 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:15:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.422+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5583b1a959a0 con 0x5583b1bbab20 2023-09-10T01:15:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.423+0000 7f45d3fff700 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1bbab20 0x5583b1bbcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.423+0000 7f45d3fff700 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1bbab20 0x5583b1bbcf40 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8edc3e4a7f8c7f1e server_cookie=1bb9908ff89dd7b7 in_seq=0 out_seq=0 2023-09-10T01:15:23.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.423+0000 7f45b3fff700 10 client.18318 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45b3fff700 1 -- 172.21.15.3:0/125031864 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5583b1a959a0 con 0x5583b1bbab20 2023-09-10T01:15:23.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45b3fff700 10 client.18318 handle_command_reply: tid=0 2023-09-10T01:15:23.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 client.18318 shutdown 2023-09-10T01:15:23.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 msgr2=0x5583b1bae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1bae300 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f45c800c5b0 tx=0x7f45c8099780 comp rx=0 tx=0).stop 2023-09-10T01:15:23.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 shutdown_connections 2023-09-10T01:15:23.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5583b1bbab20 0x5583b1bbcf40 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5583b1b54ec0 0x5583b1baed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5583b1b4a790 0x5583b1bae300 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 --2- 172.21.15.3:0/125031864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5583b1b4b0b0 0x5583b1bae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 >> 172.21.15.3:0/125031864 conn(0x5583b1abf110 msgr2=0x5583b1b52190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45b3fff700 0 client.18318 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 shutdown_connections 2023-09-10T01:15:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.424+0000 7f45ee7eab80 1 -- 172.21.15.3:0/125031864 wait complete. 2023-09-10T01:15:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45ee7eab80 20 client.18318 trim_cache size 0 max 16384 2023-09-10T01:15:23.448 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:15:23.449 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:15:23.449 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "6794c261-e9a3-432f-a56a-5f0cc48c22df", 2023-09-10T01:15:23.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:15:23.449 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:15:23.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45c40a0830 msgr2=0x7f45c40a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45c40a0830 0x7f45c40a2cf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45d4074a10 tx=0x7f45cc00b040 comp rx=0 tx=0).stop 2023-09-10T01:15:23.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 msgr2=0x7f45d4072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4072900 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f45c800eb30 tx=0x7f45c800ee40 comp rx=0 tx=0).stop 2023-09-10T01:15:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 shutdown_connections 2023-09-10T01:15:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f45c40a0830 0x7f45c40a2cf0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45d4106660 0x7f45d41a48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45d4100650 0x7f45d4072900 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 --2- 172.21.15.3:0/3405988676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45d4101010 0x7f45d4072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 >> 172.21.15.3:0/3405988676 conn(0x7f45d40fc160 msgr2=0x7f45d40fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 shutdown_connections 2023-09-10T01:15:23.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.425+0000 7f45dadff700 1 -- 172.21.15.3:0/3405988676 wait complete. 2023-09-10T01:15:23.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:15:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 -- 172.21.15.3:0/3471671072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c073350 msgr2=0x7f921c10c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 --2- 172.21.15.3:0/3471671072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c073350 0x7f921c10c940 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9210009990 tx=0x7f9210009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 -- 172.21.15.3:0/3471671072 shutdown_connections 2023-09-10T01:15:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 --2- 172.21.15.3:0/3471671072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921c10ce80 0x7f921c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 --2- 172.21.15.3:0/3471671072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c072a30 0x7f921c072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 --2- 172.21.15.3:0/3471671072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c073350 0x7f921c10c940 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 -- 172.21.15.3:0/3471671072 >> 172.21.15.3:0/3471671072 conn(0x7f921c0fc170 msgr2=0x7f921c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 -- 172.21.15.3:0/3471671072 shutdown_connections 2023-09-10T01:15:23.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.679+0000 7f9221dff700 1 -- 172.21.15.3:0/3471671072 wait complete. 2023-09-10T01:15:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 Processor -- start 2023-09-10T01:15:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 -- start start 2023-09-10T01:15:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c072a30 0x7f921c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921c073350 0x7f921c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c10ce80 0x7f921c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c198db0 con 0x7f921c072a30 2023-09-10T01:15:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c198f20 con 0x7f921c073350 2023-09-10T01:15:23.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.680+0000 7f9221dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c1990e0 con 0x7f921c10ce80 2023-09-10T01:15:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921c073350 0x7f921c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c10ce80 0x7f921c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c072a30 0x7f921c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921c073350 0x7f921c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54048/0 (socket says 172.21.15.3:54048) 2023-09-10T01:15:23.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921affd700 1 -- 172.21.15.3:0/1914084626 learned_addr learned my addr 172.21.15.3:0/1914084626 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921bfff700 1 -- 172.21.15.3:0/1914084626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921c073350 msgr2=0x7f921c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921bfff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921c073350 0x7f921c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921bfff700 1 -- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c072a30 msgr2=0x7f921c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921bfff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c072a30 0x7f921c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921bfff700 1 -- 172.21.15.3:0/1914084626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c009520 con 0x7f921c10ce80 2023-09-10T01:15:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f921bfff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c10ce80 0x7f921c198740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f921400efc0 tx=0x7f921400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.681+0000 7f9218ff9700 1 -- 172.21.15.3:0/1914084626 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214018070 con 0x7f921c10ce80 2023-09-10T01:15:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.682+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92100095f0 con 0x7f921c10ce80 2023-09-10T01:15:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.682+0000 7f9218ff9700 1 -- 172.21.15.3:0/1914084626 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f921400f040 con 0x7f921c10ce80 2023-09-10T01:15:23.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.682+0000 7f9218ff9700 1 -- 172.21.15.3:0/1914084626 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921400cd40 con 0x7f921c10ce80 2023-09-10T01:15:23.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.682+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921c1a54e0 con 0x7f921c10ce80 2023-09-10T01:15:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.685+0000 7f9218ff9700 1 -- 172.21.15.3:0/1914084626 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9214003a00 con 0x7f921c10ce80 2023-09-10T01:15:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.686+0000 7f9218ff9700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92040a07b0 0x7f92040a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.686+0000 7f9218ff9700 1 -- 172.21.15.3:0/1914084626 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f92140cbf10 con 0x7f921c10ce80 2023-09-10T01:15:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f921b7fe700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92040a07b0 0x7f92040a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 Processor -- start 2023-09-10T01:15:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 -- start start 2023-09-10T01:15:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d2710 0x5570960d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d3030 0x5570960dc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960df230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55709602b730 con 0x5570960dce40 2023-09-10T01:15:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557096029050 con 0x5570960d2710 2023-09-10T01:15:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f92357ccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557096024940 con 0x5570960d3030 2023-09-10T01:15:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.687+0000 7f921b7fe700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92040a07b0 0x7f92040a2c70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f920c009df0 tx=0x7f920c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d2710 0x5570960d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d3030 0x5570960dc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d2710 0x5570960d2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54050/0 (socket says 172.21.15.3:54050) 2023-09-10T01:15:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 -- 172.21.15.3:0/2614171132 learned_addr learned my addr 172.21.15.3:0/2614171132 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921bfff700 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960df230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 -- 172.21.15.3:0/2614171132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d3030 msgr2=0x5570960dc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d3030 0x5570960dc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 -- 172.21.15.3:0/2614171132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 msgr2=0x5570960df230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960df230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 -- 172.21.15.3:0/2614171132 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557095e50de0 con 0x5570960d2710 2023-09-10T01:15:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921bfff700 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960df230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:23.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.688+0000 7f921b7fe700 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d2710 0x5570960d2af0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f920c00e610 tx=0x7f920c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c143f840a5b21ae0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f9201ffb700 1 -- 172.21.15.3:0/2614171132 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c004220 con 0x5570960d2710 2023-09-10T01:15:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f9201ffb700 1 -- 172.21.15.3:0/2614171132 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f920c021070 con 0x5570960d2710 2023-09-10T01:15:23.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f9201ffb700 1 -- 172.21.15.3:0/2614171132 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c01d670 con 0x5570960d2710 2023-09-10T01:15:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 -- 172.21.15.3:0/2614171132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d2710 msgr2=0x5570960d2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d2710 0x5570960d2af0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f920c00e610 tx=0x7f920c014000 comp rx=0 tx=0).stop 2023-09-10T01:15:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 -- 172.21.15.3:0/2614171132 shutdown_connections 2023-09-10T01:15:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960df230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d2710 0x5570960d2af0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 --2- 172.21.15.3:0/2614171132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d3030 0x5570960dc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 -- 172.21.15.3:0/2614171132 >> 172.21.15.3:0/2614171132 conn(0x557096047090 msgr2=0x5570960cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.689+0000 7f92357ccb80 1 -- 172.21.15.3:0/2614171132 shutdown_connections 2023-09-10T01:15:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.690+0000 7f92357ccb80 1 -- 172.21.15.3:0/2614171132 wait complete. 2023-09-10T01:15:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.690+0000 7f92357ccb80 1 -- start start 2023-09-10T01:15:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.690+0000 7f92357ccb80 1 Processor -- start 2023-09-10T01:15:23.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960dc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f92357ccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557095fc2b70 con 0x5570960dce40 2023-09-10T01:15:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f92357ccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557095fc2870 con 0x5570960d3030 2023-09-10T01:15:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f92357ccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557095ea6920 con 0x5570960d2710 2023-09-10T01:15:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f92357ccb80 10 client.0 _ll_register_callbacks cb 0x557095f0ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:23.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960dc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54066/0 (socket says 172.21.15.3:54066) 2023-09-10T01:15:23.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960dc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39532/0 (socket says 172.21.15.3:39532) 2023-09-10T01:15:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 -- 172.21.15.3:0/383492225 learned_addr learned my addr 172.21.15.3:0/383492225 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921bfff700 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 -- 172.21.15.3:0/383492225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 msgr2=0x5570960dc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960dc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 -- 172.21.15.3:0/383492225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 msgr2=0x5570960da2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960da2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921affd700 1 -- 172.21.15.3:0/383492225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920c017070 con 0x5570960d3030 2023-09-10T01:15:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.691+0000 7f921bfff700 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960da2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f921affd700 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9b90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9210005910 tx=0x7f921000b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1229067dbe2eaeb4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f91faffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f91faffd700 1 -- 172.21.15.3:0/383492225 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921001d070 con 0x5570960d3030 2023-09-10T01:15:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557095f7c4f0 con 0x5570960d3030 2023-09-10T01:15:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f91faffd700 1 -- 172.21.15.3:0/383492225 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f9210004700 con 0x5570960d3030 2023-09-10T01:15:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f91faffd700 1 -- 172.21.15.3:0/383492225 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92100048c0 con 0x5570960d3030 2023-09-10T01:15:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f92357ccb80 10 client.28415 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x557095e39c60 con 0x5570960d3030 2023-09-10T01:15:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.692+0000 7f91faffd700 1 -- 172.21.15.3:0/383492225 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557095f7c4f0 con 0x5570960d3030 2023-09-10T01:15:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f91faffd700 1 -- 172.21.15.3:0/383492225 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f921000fa40 con 0x5570960d3030 2023-09-10T01:15:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f92357ccb80 10 client.28415 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:23.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f92357ccb80 10 client.28415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f92357ccb80 20 client.28415 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f92357ccb80 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5570960ecb50 0x5570960eef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f92357ccb80 4 client.28415 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570960398a0 con 0x5570960ecb50 2023-09-10T01:15:23.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f921b7fe700 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5570960ecb50 0x5570960eef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f921b7fe700 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5570960ecb50 0x5570960eef70 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4775d1b3f28c2c1 server_cookie=9ab15a5d7883de8 in_seq=0 out_seq=0 2023-09-10T01:15:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.693+0000 7f91faffd700 10 client.28415 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f91faffd700 1 -- 172.21.15.3:0/383492225 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5570960398a0 con 0x5570960ecb50 2023-09-10T01:15:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f91faffd700 10 client.28415 handle_command_reply: tid=0 2023-09-10T01:15:23.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 client.28415 shutdown 2023-09-10T01:15:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 msgr2=0x5570960d9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9b90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9210005910 tx=0x7f921000b650 comp rx=0 tx=0).stop 2023-09-10T01:15:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 shutdown_connections 2023-09-10T01:15:23.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5570960ecb50 0x5570960eef70 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960dc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960da2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f91faffd700 0 client.28415 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 --2- 172.21.15.3:0/383492225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9b90 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.694+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 >> 172.21.15.3:0/383492225 conn(0x557096047090 msgr2=0x5570960cd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.695+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 shutdown_connections 2023-09-10T01:15:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.695+0000 7f92357ccb80 1 -- 172.21.15.3:0/383492225 wait complete. 2023-09-10T01:15:23.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.695+0000 7f92357ccb80 20 client.28415 trim_cache size 0 max 16384 2023-09-10T01:15:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.713+0000 7f92357ccb80 1 Processor -- start 2023-09-10T01:15:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.713+0000 7f92357ccb80 1 -- start start 2023-09-10T01:15:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.713+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960d9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960d6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f92357ccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570960d37e0 con 0x5570960dce40 2023-09-10T01:15:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f92357ccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570960ca3d0 con 0x5570960d3030 2023-09-10T01:15:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f92357ccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570960dab10 con 0x5570960d2710 2023-09-10T01:15:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960d9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54082/0 (socket says 172.21.15.3:54082) 2023-09-10T01:15:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 -- 172.21.15.3:0/3118980708 learned_addr learned my addr 172.21.15.3:0/3118980708 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921b7fe700 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960d9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:39548/0 (socket says 172.21.15.3:39548) 2023-09-10T01:15:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921bfff700 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960d6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 -- 172.21.15.3:0/3118980708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 msgr2=0x5570960d9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960d9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 -- 172.21.15.3:0/3118980708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 msgr2=0x5570960d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960d6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 -- 172.21.15.3:0/3118980708 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557095e77250 con 0x5570960d3030 2023-09-10T01:15:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921bfff700 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960d6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.714+0000 7f921affd700 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9890 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f921000fe20 tx=0x7f9210005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d06b732a3b012e6a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92027fc700 1 -- 172.21.15.3:0/3118980708 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921001d030 con 0x5570960d3030 2023-09-10T01:15:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92027fc700 1 -- 172.21.15.3:0/3118980708 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f921001e040 con 0x5570960d3030 2023-09-10T01:15:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92027fc700 1 -- 172.21.15.3:0/3118980708 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921000f860 con 0x5570960d3030 2023-09-10T01:15:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92357ccb80 1 -- 172.21.15.3:0/3118980708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 msgr2=0x5570960d9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92357ccb80 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9890 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f921000fe20 tx=0x7f9210005e80 comp rx=0 tx=0).stop 2023-09-10T01:15:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92357ccb80 1 -- 172.21.15.3:0/3118980708 shutdown_connections 2023-09-10T01:15:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92357ccb80 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960d2710 0x5570960d9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92357ccb80 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960dce40 0x5570960d6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92357ccb80 1 --2- 172.21.15.3:0/3118980708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960d9890 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.715+0000 7f92357ccb80 1 -- 172.21.15.3:0/3118980708 >> 172.21.15.3:0/3118980708 conn(0x557096047090 msgr2=0x5570960d4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.716+0000 7f92357ccb80 1 -- 172.21.15.3:0/3118980708 shutdown_connections 2023-09-10T01:15:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.716+0000 7f92357ccb80 1 -- 172.21.15.3:0/3118980708 wait complete. 2023-09-10T01:15:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.716+0000 7f92357ccb80 1 -- start start 2023-09-10T01:15:23.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.716+0000 7f92357ccb80 1 Processor -- start 2023-09-10T01:15:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960d2710 0x5570960e3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f92357ccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960dce40 0x5570960e2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f92357ccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570960e2a00 con 0x5570960d2710 2023-09-10T01:15:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f92357ccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570960e2b70 con 0x5570960d3030 2023-09-10T01:15:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f92357ccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557096140950 con 0x5570960dce40 2023-09-10T01:15:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f92357ccb80 10 client.0 _ll_register_callbacks cb 0x7f921000bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f921b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960d2710 0x5570960e3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f921affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960e1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:54088/0 (socket says 172.21.15.3:54088) 2023-09-10T01:15:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f921affd700 1 -- 172.21.15.3:0/975715677 learned_addr learned my addr 172.21.15.3:0/975715677 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.717+0000 7f921bfff700 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960dce40 0x5570960e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f921affd700 1 -- 172.21.15.3:0/975715677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960dce40 msgr2=0x5570960e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f921affd700 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960dce40 0x5570960e2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f921affd700 1 -- 172.21.15.3:0/975715677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960d2710 msgr2=0x5570960e3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f921affd700 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960d2710 0x5570960e3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f921affd700 1 -- 172.21.15.3:0/975715677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557095e39c60 con 0x5570960d3030 2023-09-10T01:15:23.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f921affd700 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960e1d90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9210031040 tx=0x7f92100053a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c4ad2bcd79faf7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f91faffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f91faffd700 1 -- 172.21.15.3:0/975715677 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921001d040 con 0x5570960d3030 2023-09-10T01:15:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f920c021070 con 0x5570960d3030 2023-09-10T01:15:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f91faffd700 1 -- 172.21.15.3:0/975715677 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f9210000e40 con 0x5570960d3030 2023-09-10T01:15:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f91faffd700 1 -- 172.21.15.3:0/975715677 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9210025ce0 con 0x5570960d3030 2023-09-10T01:15:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f92357ccb80 10 client.28421 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.718+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x557095e4f2b0 con 0x5570960d3030 2023-09-10T01:15:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f91faffd700 1 -- 172.21.15.3:0/975715677 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f920c021070 con 0x5570960d3030 2023-09-10T01:15:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f91faffd700 1 -- 172.21.15.3:0/975715677 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f92100046c0 con 0x5570960d3030 2023-09-10T01:15:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f92357ccb80 10 client.28421 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:23.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f92357ccb80 10 client.28421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f92357ccb80 20 client.28421 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f92357ccb80 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557096141c90 0x5570961440b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f92357ccb80 4 client.28421 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:15:23.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55709601d920 con 0x557096141c90 2023-09-10T01:15:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.719+0000 7f921b7fe700 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557096141c90 0x5570961440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f921b7fe700 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557096141c90 0x5570961440b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d3b788ce2091ef server_cookie=658911967da0c966 in_seq=0 out_seq=0 2023-09-10T01:15:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f91faffd700 10 client.28421 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f91faffd700 1 -- 172.21.15.3:0/975715677 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55709601d920 con 0x557096141c90 2023-09-10T01:15:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f91faffd700 10 client.28421 handle_command_reply: tid=0 2023-09-10T01:15:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 client.28421 shutdown 2023-09-10T01:15:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 msgr2=0x5570960e1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960e1d90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9210031040 tx=0x7f92100053a0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 shutdown_connections 2023-09-10T01:15:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x557096141c90 0x5570961440b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570960dce40 0x5570960e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5570960d2710 0x5570960e3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f91faffd700 0 client.28421 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 --2- 172.21.15.3:0/975715677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5570960d3030 0x5570960e1d90 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 >> 172.21.15.3:0/975715677 conn(0x557096047090 msgr2=0x5570960db430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 shutdown_connections 2023-09-10T01:15:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 1 -- 172.21.15.3:0/975715677 wait complete. 2023-09-10T01:15:23.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.720+0000 7f92357ccb80 20 client.28421 trim_cache size 0 max 16384 2023-09-10T01:15:23.743 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:15:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:15:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:15:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "6794c261-e9a3-432f-a56a-5f0cc48c22df": { 2023-09-10T01:15:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:15:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "6794c261-e9a3-432f-a56a-5f0cc48c22df", 2023-09-10T01:15:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:15:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:15:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:15:23.746 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:15:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92040a07b0 msgr2=0x7f92040a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92040a07b0 0x7f92040a2c70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f920c009df0 tx=0x7f920c009260 comp rx=0 tx=0).stop 2023-09-10T01:15:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c10ce80 msgr2=0x7f921c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:23.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c10ce80 0x7f921c198740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f921400efc0 tx=0x7f921400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 shutdown_connections 2023-09-10T01:15:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f92040a07b0 0x7f92040a2c70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f921c10ce80 0x7f921c198740 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f921c072a30 0x7f921c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 --2- 172.21.15.3:0/1914084626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921c073350 0x7f921c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 >> 172.21.15.3:0/1914084626 conn(0x7f921c0fc170 msgr2=0x7f921c10efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 shutdown_connections 2023-09-10T01:15:23.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:23.721+0000 7f9221dff700 1 -- 172.21.15.3:0/1914084626 wait complete. 2023-09-10T01:15:23.749 INFO:tasks.cephfs.filesystem:scrub status for tag:6794c261-e9a3-432f-a56a-5f0cc48c22df - {'path': '/', 'tag': '6794c261-e9a3-432f-a56a-5f0cc48c22df', 'options': 'recursive,force'} 2023-09-10T01:15:23.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 996006 7.73 MB/sec execute 545 sec latency 4950.860 ms 2023-09-10T01:15:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: pgmap v380: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2023-09-10T01:15:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:15:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:24.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:24 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: pgmap v380: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2023-09-10T01:15:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:15:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:24.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:24 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 997783 7.74 MB/sec execute 546 sec latency 22.044 ms 2023-09-10T01:15:24.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: pgmap v380: 129 pgs: 129 active+clean; 308 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2023-09-10T01:15:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:15:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:24.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:24.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:24.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:24 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:25 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:25 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:25 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:25.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:25 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:25 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:25.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:25 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:25 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:25 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:25 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:25 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:25 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:25.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:25 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 999002 7.73 MB/sec execute 547 sec latency 15.305 ms 2023-09-10T01:15:25.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:25 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:25.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:25 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:25 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:25.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:25 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:25.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:25 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:25.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:25 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:26 smithi136 ceph-mon[133708]: pgmap v381: 129 pgs: 129 active+clean; 309 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2023-09-10T01:15:26.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:26 smithi092 ceph-mon[134067]: pgmap v381: 129 pgs: 129 active+clean; 309 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2023-09-10T01:15:26.807 INFO:tasks.workunit.client.0.smithi003.stdout: 1 999114 7.72 MB/sec execute 548 sec latency 874.466 ms 2023-09-10T01:15:26.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:26 smithi003 ceph-mon[97654]: pgmap v381: 129 pgs: 129 active+clean; 309 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2023-09-10T01:15:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 -- 172.21.15.3:0/3026144050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 msgr2=0x7f76b8100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 --2- 172.21.15.3:0/3026144050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8100a40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0099c0 tx=0x7f76ac009cd0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 -- 172.21.15.3:0/3026144050 shutdown_connections 2023-09-10T01:15:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 --2- 172.21.15.3:0/3026144050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8100f80 0x7f76b810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 --2- 172.21.15.3:0/3026144050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76b8106680 0x7f76b8106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 --2- 172.21.15.3:0/3026144050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8100a40 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 -- 172.21.15.3:0/3026144050 >> 172.21.15.3:0/3026144050 conn(0x7f76b80fc0b0 msgr2=0x7f76b80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 -- 172.21.15.3:0/3026144050 shutdown_connections 2023-09-10T01:15:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.467+0000 7f76bddff700 1 -- 172.21.15.3:0/3026144050 wait complete. 2023-09-10T01:15:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.468+0000 7f76bddff700 1 Processor -- start 2023-09-10T01:15:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.468+0000 7f76bddff700 1 -- start start 2023-09-10T01:15:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76bddff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76bddff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76b8100f80 0x7f76b8196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76bddff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8106680 0x7f76b819b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76bddff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b819b660 con 0x7f76b81005c0 2023-09-10T01:15:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76bddff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b819b7d0 con 0x7f76b8100f80 2023-09-10T01:15:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76bddff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b819b990 con 0x7f76b8106680 2023-09-10T01:15:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76b7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8106680 0x7f76b819b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76b7fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8106680 0x7f76b819b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58662/0 (socket says 172.21.15.3:58662) 2023-09-10T01:15:27.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56210/0 (socket says 172.21.15.3:56210) 2023-09-10T01:15:27.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76b7fff700 1 -- 172.21.15.3:0/3866335194 learned_addr learned my addr 172.21.15.3:0/3866335194 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:27.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.469+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76b8100f80 0x7f76b8196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b7fff700 1 -- 172.21.15.3:0/3866335194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76b8100f80 msgr2=0x7f76b8196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b7fff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76b8100f80 0x7f76b8196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b7fff700 1 -- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 msgr2=0x7f76b8196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b7fff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b7fff700 1 -- 172.21.15.3:0/3866335194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac0095f0 con 0x7f76b8106680 2023-09-10T01:15:27.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b77fe700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:27.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b7fff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8106680 0x7f76b819b020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f76b000ef30 tx=0x7f76b000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:27.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b4ff9700 1 -- 172.21.15.3:0/3866335194 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b0018070 con 0x7f76b8106680 2023-09-10T01:15:27.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b4ff9700 1 -- 172.21.15.3:0/3866335194 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76b000f040 con 0x7f76b8106680 2023-09-10T01:15:27.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76b4ff9700 1 -- 172.21.15.3:0/3866335194 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b000cd00 con 0x7f76b8106680 2023-09-10T01:15:27.476 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b819bc10 con 0x7f76b8106680 2023-09-10T01:15:27.477 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.470+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b81a34b0 con 0x7f76b8106680 2023-09-10T01:15:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.474+0000 7f76b4ff9700 1 -- 172.21.15.3:0/3866335194 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f76b0003a00 con 0x7f76b8106680 2023-09-10T01:15:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.475+0000 7f76d16b8b80 1 Processor -- start 2023-09-10T01:15:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.475+0000 7f76b4ff9700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f76a00a07b0 0x7f76a00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.475+0000 7f76d16b8b80 1 -- start start 2023-09-10T01:15:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.475+0000 7f76b4ff9700 1 -- 172.21.15.3:0/3866335194 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f76b00cbf10 con 0x7f76b8106680 2023-09-10T01:15:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.475+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f76a00a07b0 0x7f76a00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf52911a0 0x55dbf5291580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf528b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf5297f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf51e46d0 con 0x55dbf52911a0 2023-09-10T01:15:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf51e1ff0 con 0x55dbf528b300 2023-09-10T01:15:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf51dd8e0 con 0x55dbf528bcc0 2023-09-10T01:15:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf52911a0 0x55dbf5291580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf52911a0 0x55dbf5291580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56226/0 (socket says 172.21.15.3:56226) 2023-09-10T01:15:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76b77fe700 1 -- 172.21.15.3:0/158937345 learned_addr learned my addr 172.21.15.3:0/158937345 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf528b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76b7fff700 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf5297f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.476+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f76a00a07b0 0x7f76a00a2c70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009fd0 tx=0x7f76ac005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f76b7fff700 1 -- 172.21.15.3:0/158937345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 msgr2=0x55dbf528b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f76b7fff700 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf528b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f76b7fff700 1 -- 172.21.15.3:0/158937345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf52911a0 msgr2=0x55dbf5291580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f76b7fff700 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf52911a0 0x55dbf5291580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f76b7fff700 1 -- 172.21.15.3:0/158937345 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbf5009d80 con 0x55dbf528bcc0 2023-09-10T01:15:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f76b77fe700 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf52911a0 0x55dbf5291580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f76b7fff700 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf5297f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f76b00c9210 tx=0x7f76b0098bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=391f57728ccf9c7d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f769dffb700 1 -- 172.21.15.3:0/158937345 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b001b320 con 0x55dbf528bcc0 2023-09-10T01:15:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f769dffb700 1 -- 172.21.15.3:0/158937345 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76b0099700 con 0x55dbf528bcc0 2023-09-10T01:15:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.477+0000 7f769dffb700 1 -- 172.21.15.3:0/158937345 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b0099260 con 0x55dbf528bcc0 2023-09-10T01:15:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 -- 172.21.15.3:0/158937345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 msgr2=0x55dbf5297f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf5297f30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f76b00c9210 tx=0x7f76b0098bc0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 -- 172.21.15.3:0/158937345 shutdown_connections 2023-09-10T01:15:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf5297f30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf52911a0 0x55dbf5291580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/158937345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf528b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 -- 172.21.15.3:0/158937345 >> 172.21.15.3:0/158937345 conn(0x55dbf5200030 msgr2=0x55dbf5286560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 -- 172.21.15.3:0/158937345 shutdown_connections 2023-09-10T01:15:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 -- 172.21.15.3:0/158937345 wait complete. 2023-09-10T01:15:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.478+0000 7f76d16b8b80 1 -- start start 2023-09-10T01:15:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 1 Processor -- start 2023-09-10T01:15:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf5297df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528bcc0 0x55dbf5295110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf5295740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf517bb10 con 0x55dbf528bcc0 2023-09-10T01:15:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf517b810 con 0x55dbf528b300 2023-09-10T01:15:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf505f8c0 con 0x55dbf52911a0 2023-09-10T01:15:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76d16b8b80 10 client.0 _ll_register_callbacks cb 0x55dbf50c5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.479+0000 7f76b6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528bcc0 0x55dbf5295110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528bcc0 0x55dbf5295110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56230/0 (socket says 172.21.15.3:56230) 2023-09-10T01:15:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf5297df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 -- 172.21.15.3:0/2050157472 learned_addr learned my addr 172.21.15.3:0/2050157472 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b7fff700 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf5295740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 -- 172.21.15.3:0/2050157472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 msgr2=0x55dbf5295740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf5295740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 -- 172.21.15.3:0/2050157472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 msgr2=0x55dbf5297df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf5297df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 -- 172.21.15.3:0/2050157472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a8009d50 con 0x55dbf528bcc0 2023-09-10T01:15:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528bcc0 0x55dbf5295110 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00b580 tx=0x7f76ac009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6b24a55ebb125 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.480+0000 7f7696ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f7696ffd700 1 -- 172.21.15.3:0/2050157472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac00b790 con 0x55dbf528bcc0 2023-09-10T01:15:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dbf5135490 con 0x55dbf528bcc0 2023-09-10T01:15:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f7696ffd700 1 -- 172.21.15.3:0/2050157472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f76ac00b8f0 con 0x55dbf528bcc0 2023-09-10T01:15:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f7696ffd700 1 -- 172.21.15.3:0/2050157472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76ac00baf0 con 0x55dbf528bcc0 2023-09-10T01:15:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f76d16b8b80 10 client.18348 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7f76b81a34b0 con 0x55dbf528bcc0 2023-09-10T01:15:27.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f7696ffd700 1 -- 172.21.15.3:0/2050157472 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dbf5135490 con 0x55dbf528bcc0 2023-09-10T01:15:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f7696ffd700 1 -- 172.21.15.3:0/2050157472 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f76ac00f460 con 0x55dbf528bcc0 2023-09-10T01:15:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f76d16b8b80 10 client.18348 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f76d16b8b80 10 client.18348 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.481+0000 7f76d16b8b80 20 client.18348 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.482+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52a5480 0x55dbf52a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.482+0000 7f76d16b8b80 4 client.18348 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.482+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dbf51f2840 con 0x55dbf52a5480 2023-09-10T01:15:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.482+0000 7f76b77fe700 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52a5480 0x55dbf52a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.482+0000 7f76b77fe700 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52a5480 0x55dbf52a78a0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b4eadf2e7c367f8 server_cookie=839090110184aa69 in_seq=0 out_seq=0 2023-09-10T01:15:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.482+0000 7f7696ffd700 10 client.18348 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f7696ffd700 1 -- 172.21.15.3:0/2050157472 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dbf51f2840 con 0x55dbf52a5480 2023-09-10T01:15:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f7696ffd700 10 client.18348 handle_command_reply: tid=0 2023-09-10T01:15:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 client.18348 shutdown 2023-09-10T01:15:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528bcc0 msgr2=0x55dbf5295110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528bcc0 0x55dbf5295110 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00b580 tx=0x7f76ac009310 comp rx=0 tx=0).stop 2023-09-10T01:15:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 shutdown_connections 2023-09-10T01:15:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52a5480 0x55dbf52a78a0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf5295740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528b300 0x55dbf5297df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f7696ffd700 0 client.18348 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/2050157472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528bcc0 0x55dbf5295110 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 >> 172.21.15.3:0/2050157472 conn(0x55dbf5200030 msgr2=0x55dbf52857a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:27.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 shutdown_connections 2023-09-10T01:15:27.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.483+0000 7f76d16b8b80 1 -- 172.21.15.3:0/2050157472 wait complete. 2023-09-10T01:15:27.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.484+0000 7f76d16b8b80 20 client.18348 trim_cache size 0 max 16384 2023-09-10T01:15:27.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 Processor -- start 2023-09-10T01:15:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 -- start start 2023-09-10T01:15:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf528aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf52911a0 0x55dbf5297920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf5135490 con 0x55dbf528b300 2023-09-10T01:15:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf52808a0 con 0x55dbf52911a0 2023-09-10T01:15:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.499+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf5282fc0 con 0x55dbf528bcc0 2023-09-10T01:15:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf528aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf528aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:58678/0 (socket says 172.21.15.3:58678) 2023-09-10T01:15:27.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf52911a0 0x55dbf5297920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 -- 172.21.15.3:0/1915292475 learned_addr learned my addr 172.21.15.3:0/1915292475 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:27.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 -- 172.21.15.3:0/1915292475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf52911a0 msgr2=0x55dbf5297920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf52911a0 0x55dbf5297920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 -- 172.21.15.3:0/1915292475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 msgr2=0x55dbf529c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.505 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 -- 172.21.15.3:0/1915292475 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbf4ff2c00 con 0x55dbf528bcc0 2023-09-10T01:15:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b77fe700 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.500+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf528aa00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00f890 tx=0x7f76ac0059c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5889e80629c5a0b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f769e7fc700 1 -- 172.21.15.3:0/1915292475 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0056b0 con 0x55dbf528bcc0 2023-09-10T01:15:27.506 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f769e7fc700 1 -- 172.21.15.3:0/1915292475 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76ac046070 con 0x55dbf528bcc0 2023-09-10T01:15:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f769e7fc700 1 -- 172.21.15.3:0/1915292475 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac003f00 con 0x55dbf528bcc0 2023-09-10T01:15:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f76d16b8b80 1 -- 172.21.15.3:0/1915292475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 msgr2=0x55dbf528aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf528aa00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00f890 tx=0x7f76ac0059c0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f76d16b8b80 1 -- 172.21.15.3:0/1915292475 shutdown_connections 2023-09-10T01:15:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf52911a0 0x55dbf5297920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/1915292475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf528bcc0 0x55dbf528aa00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.501+0000 7f76d16b8b80 1 -- 172.21.15.3:0/1915292475 >> 172.21.15.3:0/1915292475 conn(0x55dbf5200030 msgr2=0x55dbf528a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.502+0000 7f76d16b8b80 1 -- 172.21.15.3:0/1915292475 shutdown_connections 2023-09-10T01:15:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.502+0000 7f76d16b8b80 1 -- 172.21.15.3:0/1915292475 wait complete. 2023-09-10T01:15:27.508 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.502+0000 7f76d16b8b80 1 -- start start 2023-09-10T01:15:27.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.502+0000 7f76d16b8b80 1 Processor -- start 2023-09-10T01:15:27.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.511 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528bcc0 0x55dbf529a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76d16b8b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf529ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf529b410 con 0x55dbf528b300 2023-09-10T01:15:27.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf529b580 con 0x55dbf528bcc0 2023-09-10T01:15:27.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76d16b8b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf529b740 con 0x55dbf52911a0 2023-09-10T01:15:27.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76d16b8b80 10 client.0 _ll_register_callbacks cb 0x7f76ac003ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76b77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56244/0 (socket says 172.21.15.3:56244) 2023-09-10T01:15:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76b77fe700 1 -- 172.21.15.3:0/817891421 learned_addr learned my addr 172.21.15.3:0/817891421 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76b7fff700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf529ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.503+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528bcc0 0x55dbf529a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b77fe700 1 -- 172.21.15.3:0/817891421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 msgr2=0x55dbf529ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b77fe700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf529ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b77fe700 1 -- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528bcc0 msgr2=0x55dbf529a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b77fe700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528bcc0 0x55dbf529a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.514 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b77fe700 1 -- 172.21.15.3:0/817891421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b81a34b0 con 0x55dbf528b300 2023-09-10T01:15:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b7fff700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf529ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528bcc0 0x55dbf529a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76b77fe700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c420 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x55dbf52a8110 tx=0x7f76a8010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857f3c55b2a985b1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f7696ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f7696ffd700 1 -- 172.21.15.3:0/817891421 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a8016040 con 0x55dbf528b300 2023-09-10T01:15:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76b0099700 con 0x55dbf528b300 2023-09-10T01:15:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f7696ffd700 1 -- 172.21.15.3:0/817891421 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f76a801db80 con 0x55dbf528b300 2023-09-10T01:15:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.504+0000 7f7696ffd700 1 -- 172.21.15.3:0/817891421 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76a801dd40 con 0x55dbf528b300 2023-09-10T01:15:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 10 client.18354 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55dbf50301f0 con 0x55dbf528b300 2023-09-10T01:15:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f7696ffd700 1 -- 172.21.15.3:0/817891421 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76b0099700 con 0x55dbf528b300 2023-09-10T01:15:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f7696ffd700 1 -- 172.21.15.3:0/817891421 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f76a8023020 con 0x55dbf528b300 2023-09-10T01:15:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 10 client.18354 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 10 client.18354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 20 client.18354 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52fed30 0x55dbf5301150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 4 client.18354 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:15:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dbf51d68c0 con 0x55dbf52fed30 2023-09-10T01:15:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.505+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52fed30 0x55dbf5301150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.506+0000 7f76b6ffd700 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52fed30 0x55dbf5301150 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c4cd8a02135992 server_cookie=b8689dc256faec12 in_seq=0 out_seq=0 2023-09-10T01:15:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.506+0000 7f7696ffd700 10 client.18354 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.571+0000 7f7696ffd700 1 -- 172.21.15.3:0/817891421 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dbf51d68c0 con 0x55dbf52fed30 2023-09-10T01:15:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.571+0000 7f7696ffd700 10 client.18354 handle_command_reply: tid=0 2023-09-10T01:15:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.571+0000 7f76d16b8b80 1 client.18354 shutdown 2023-09-10T01:15:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 msgr2=0x55dbf529c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c420 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x55dbf52a8110 tx=0x7f76a8010c00 comp rx=0 tx=0).stop 2023-09-10T01:15:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 shutdown_connections 2023-09-10T01:15:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dbf52fed30 0x55dbf5301150 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbf52911a0 0x55dbf529ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dbf528b300 0x55dbf529c420 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 --2- 172.21.15.3:0/817891421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dbf528bcc0 0x55dbf529a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f7696ffd700 0 client.18354 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 >> 172.21.15.3:0/817891421 conn(0x55dbf5200030 msgr2=0x55dbf528ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 shutdown_connections 2023-09-10T01:15:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 1 -- 172.21.15.3:0/817891421 wait complete. 2023-09-10T01:15:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.572+0000 7f76d16b8b80 20 client.18354 trim_cache size 0 max 16384 2023-09-10T01:15:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:15:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:15:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:15:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:15:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f76a00a07b0 msgr2=0x7f76a00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f76a00a07b0 0x7f76a00a2c70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009fd0 tx=0x7f76ac005fb0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8106680 msgr2=0x7f76b819b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8106680 0x7f76b819b020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f76b000ef30 tx=0x7f76b000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 shutdown_connections 2023-09-10T01:15:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f76a00a07b0 0x7f76a00a2c70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76b8106680 0x7f76b819b020 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76b81005c0 0x7f76b8196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 --2- 172.21.15.3:0/3866335194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76b8100f80 0x7f76b8196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 >> 172.21.15.3:0/3866335194 conn(0x7f76b80fc0b0 msgr2=0x7f76b80fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:27.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 shutdown_connections 2023-09-10T01:15:27.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:27.573+0000 7f76bddff700 1 -- 172.21.15.3:0/3866335194 wait complete. 2023-09-10T01:15:27.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 999114 7.71 MB/sec execute 549 sec latency 1874.613 ms 2023-09-10T01:15:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:28 smithi136 ceph-mon[133708]: pgmap v382: 129 pgs: 129 active+clean; 309 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 839 KiB/s wr, 100 op/s 2023-09-10T01:15:28.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:28 smithi092 ceph-mon[134067]: pgmap v382: 129 pgs: 129 active+clean; 309 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 839 KiB/s wr, 100 op/s 2023-09-10T01:15:28.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 999982 7.70 MB/sec execute 550 sec latency 2169.739 ms 2023-09-10T01:15:28.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:28 smithi003 ceph-mon[97654]: pgmap v382: 129 pgs: 129 active+clean; 309 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 839 KiB/s wr, 100 op/s 2023-09-10T01:15:29.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:29 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:15:29.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:29 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:15:29.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:29 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:15:29.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:29 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:15:29.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1001831 7.71 MB/sec execute 551 sec latency 25.056 ms 2023-09-10T01:15:29.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:29 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:15:29.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:29 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:15:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:30 smithi136 ceph-mon[133708]: pgmap v383: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2023-09-10T01:15:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:30 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:15:30.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:30 smithi092 ceph-mon[134067]: pgmap v383: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2023-09-10T01:15:30.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:30 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:15:30.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1002531 7.70 MB/sec execute 552 sec latency 196.967 ms 2023-09-10T01:15:30.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:30 smithi003 ceph-mon[97654]: pgmap v383: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.7 MiB/s wr, 174 op/s 2023-09-10T01:15:30.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:30 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:15:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:31 smithi136 ceph-mon[133708]: pgmap v384: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2023-09-10T01:15:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:31 smithi092 ceph-mon[134067]: pgmap v384: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2023-09-10T01:15:31.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1002531 7.68 MB/sec execute 553 sec latency 1197.112 ms 2023-09-10T01:15:31.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:31 smithi003 ceph-mon[97654]: pgmap v384: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2023-09-10T01:15:32.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1002531 7.67 MB/sec execute 554 sec latency 2197.221 ms 2023-09-10T01:15:33.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1003394 7.67 MB/sec execute 555 sec latency 2499.996 ms 2023-09-10T01:15:33.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:33 smithi003 ceph-mon[97654]: pgmap v385: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2023-09-10T01:15:33.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:33 smithi136 ceph-mon[133708]: pgmap v385: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2023-09-10T01:15:33.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:33 smithi092 ceph-mon[134067]: pgmap v385: 129 pgs: 129 active+clean; 315 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 1.0 MiB/s wr, 87 op/s 2023-09-10T01:15:34.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1005400 7.67 MB/sec execute 556 sec latency 23.347 ms 2023-09-10T01:15:35.808 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1006080 7.66 MB/sec execute 557 sec latency 256.225 ms 2023-09-10T01:15:36.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:36 smithi092 ceph-mon[134067]: pgmap v386: 129 pgs: 129 active+clean; 313 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2023-09-10T01:15:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:36 smithi003 ceph-mon[97654]: pgmap v386: 129 pgs: 129 active+clean; 313 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2023-09-10T01:15:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:36 smithi136 ceph-mon[133708]: pgmap v386: 129 pgs: 129 active+clean; 313 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.4 MiB/s wr, 135 op/s 2023-09-10T01:15:36.809 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1006080 7.65 MB/sec execute 558 sec latency 1256.368 ms 2023-09-10T01:15:37.809 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1006080 7.64 MB/sec execute 559 sec latency 2256.528 ms 2023-09-10T01:15:38.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:38 smithi092 ceph-mon[134067]: pgmap v387: 129 pgs: 129 active+clean; 313 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2023-09-10T01:15:38.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:38 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:15:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:38 smithi003 ceph-mon[97654]: pgmap v387: 129 pgs: 129 active+clean; 313 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2023-09-10T01:15:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:38 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:15:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:38 smithi136 ceph-mon[133708]: pgmap v387: 129 pgs: 129 active+clean; 313 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.3 MiB/s wr, 122 op/s 2023-09-10T01:15:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:38 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:15:38.809 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1006204 7.62 MB/sec execute 560 sec latency 2564.315 ms 2023-09-10T01:15:39.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:39 smithi092 ceph-mon[134067]: pgmap v388: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 1.9 MiB/s wr, 171 op/s 2023-09-10T01:15:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:39 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:39 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:39 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:15:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:39 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:15:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:39 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:39 smithi003 ceph-mon[97654]: pgmap v388: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 1.9 MiB/s wr, 171 op/s 2023-09-10T01:15:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:39 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:39 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:39 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:15:39.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:39 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:15:39.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:39 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1007190 7.62 MB/sec execute 561 sec latency 865.216 ms 2023-09-10T01:15:39.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:39 smithi136 ceph-mon[133708]: pgmap v388: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 1.9 MiB/s wr, 171 op/s 2023-09-10T01:15:39.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:39 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:39 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:39.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:39 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:15:39.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:39 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:15:39.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:39 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:15:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 -- 172.21.15.3:0/172318459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 msgr2=0x7f3d70100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 --2- 172.21.15.3:0/172318459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 0x7f3d70100a40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3d640099c0 tx=0x7f3d64009cd0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 -- 172.21.15.3:0/172318459 shutdown_connections 2023-09-10T01:15:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 --2- 172.21.15.3:0/172318459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d70100f80 0x7f3d7010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 --2- 172.21.15.3:0/172318459 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70106680 0x7f3d70106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 --2- 172.21.15.3:0/172318459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 0x7f3d70100a40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 -- 172.21.15.3:0/172318459 >> 172.21.15.3:0/172318459 conn(0x7f3d700fc0b0 msgr2=0x7f3d700fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 -- 172.21.15.3:0/172318459 shutdown_connections 2023-09-10T01:15:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.732+0000 7f3d75dff700 1 -- 172.21.15.3:0/172318459 wait complete. 2023-09-10T01:15:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.733+0000 7f3d75dff700 1 Processor -- start 2023-09-10T01:15:40.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.733+0000 7f3d75dff700 1 -- start start 2023-09-10T01:15:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d75dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 0x7f3d70196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d75dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70100f80 0x7f3d70196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 0x7f3d70196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 0x7f3d70196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52838/0 (socket says 172.21.15.3:52838) 2023-09-10T01:15:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70100f80 0x7f3d70196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70100f80 0x7f3d70196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40540/0 (socket says 172.21.15.3:40540) 2023-09-10T01:15:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d75dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d70106680 0x7f3d7019af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d6f7fe700 1 -- 172.21.15.3:0/1487596201 learned_addr learned my addr 172.21.15.3:0/1487596201 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7019b5b0 con 0x7f3d70100f80 2023-09-10T01:15:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7019b720 con 0x7f3d701005c0 2023-09-10T01:15:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.734+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7019b8e0 con 0x7f3d70106680 2023-09-10T01:15:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d70106680 0x7f3d7019af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6effd700 1 -- 172.21.15.3:0/1487596201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d70106680 msgr2=0x7f3d7019af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6effd700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d70106680 0x7f3d7019af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6effd700 1 -- 172.21.15.3:0/1487596201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 msgr2=0x7f3d70196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6effd700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 0x7f3d70196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6effd700 1 -- 172.21.15.3:0/1487596201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d640095f0 con 0x7f3d70100f80 2023-09-10T01:15:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d70106680 0x7f3d7019af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6effd700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70100f80 0x7f3d70196b80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3d640098e0 tx=0x7f3d6400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6cff9700 1 -- 172.21.15.3:0/1487596201 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6401d070 con 0x7f3d70100f80 2023-09-10T01:15:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6cff9700 1 -- 172.21.15.3:0/1487596201 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d64005510 con 0x7f3d70100f80 2023-09-10T01:15:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d6cff9700 1 -- 172.21.15.3:0/1487596201 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64004a00 con 0x7f3d70100f80 2023-09-10T01:15:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7019bb60 con 0x7f3d70100f80 2023-09-10T01:15:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.735+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7019bde0 con 0x7f3d70100f80 2023-09-10T01:15:40.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.739+0000 7f3d6cff9700 1 -- 172.21.15.3:0/1487596201 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3d64004bc0 con 0x7f3d70100f80 2023-09-10T01:15:40.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 Processor -- start 2023-09-10T01:15:40.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 -- start start 2023-09-10T01:15:40.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d6cff9700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d580a07b0 0x7f3d580a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f8105b0 0x55900f810990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f80a710 0x55900f80ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f817340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f75f6d0 con 0x55900f8105b0 2023-09-10T01:15:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f75cff0 con 0x55900f80b0d0 2023-09-10T01:15:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f7588e0 con 0x55900f80a710 2023-09-10T01:15:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.741+0000 7f3d6cff9700 1 -- 172.21.15.3:0/1487596201 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3d640ccf90 con 0x7f3d70100f80 2023-09-10T01:15:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d580a07b0 0x7f3d580a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f80a710 0x55900f80ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f8105b0 0x55900f810990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f80a710 0x55900f80ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45318/0 (socket says 172.21.15.3:45318) 2023-09-10T01:15:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 -- 172.21.15.3:0/1088311523 learned_addr learned my addr 172.21.15.3:0/1088311523 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f817340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 -- 172.21.15.3:0/1088311523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 msgr2=0x55900f817340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f817340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 -- 172.21.15.3:0/1088311523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f8105b0 msgr2=0x55900f810990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f8105b0 0x55900f810990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 -- 172.21.15.3:0/1088311523 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55900f584d80 con 0x55900f80a710 2023-09-10T01:15:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f817340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6effd700 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f80a710 0x55900f80ab90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3d64096580 tx=0x7f3d64099a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30d2a3f566cabad8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d580a07b0 0x7f3d580a2c70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d680094f0 tx=0x7f3d68009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.742+0000 7f3d55ffb700 1 -- 172.21.15.3:0/1088311523 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64001b90 con 0x55900f80a710 2023-09-10T01:15:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d55ffb700 1 -- 172.21.15.3:0/1088311523 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d64001810 con 0x55900f80a710 2023-09-10T01:15:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d55ffb700 1 -- 172.21.15.3:0/1088311523 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6400fb80 con 0x55900f80a710 2023-09-10T01:15:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 -- 172.21.15.3:0/1088311523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f80a710 msgr2=0x55900f80ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f80a710 0x55900f80ab90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3d64096580 tx=0x7f3d64099a60 comp rx=0 tx=0).stop 2023-09-10T01:15:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 -- 172.21.15.3:0/1088311523 shutdown_connections 2023-09-10T01:15:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f817340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f8105b0 0x55900f810990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/1088311523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f80a710 0x55900f80ab90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 -- 172.21.15.3:0/1088311523 >> 172.21.15.3:0/1088311523 conn(0x55900f77b030 msgr2=0x55900f801560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.743+0000 7f3d8971bb80 1 -- 172.21.15.3:0/1088311523 shutdown_connections 2023-09-10T01:15:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.744+0000 7f3d8971bb80 1 -- 172.21.15.3:0/1088311523 wait complete. 2023-09-10T01:15:40.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.744+0000 7f3d8971bb80 1 -- start start 2023-09-10T01:15:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.744+0000 7f3d8971bb80 1 Processor -- start 2023-09-10T01:15:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f819eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f81a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f814140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f6f6b10 con 0x55900f80a710 2023-09-10T01:15:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f6f6810 con 0x55900f80b0d0 2023-09-10T01:15:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f5da8c0 con 0x55900f8105b0 2023-09-10T01:15:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d8971bb80 10 client.0 _ll_register_callbacks cb 0x55900f640a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f81a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f81a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52860/0 (socket says 172.21.15.3:52860) 2023-09-10T01:15:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 -- 172.21.15.3:0/3116751224 learned_addr learned my addr 172.21.15.3:0/3116751224 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f819eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f814140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 -- 172.21.15.3:0/3116751224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 msgr2=0x55900f814140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f814140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 -- 172.21.15.3:0/3116751224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 msgr2=0x55900f819eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f819eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6effd700 1 -- 172.21.15.3:0/3116751224 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7019bb60 con 0x55900f80b0d0 2023-09-10T01:15:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f814140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.745+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f819eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d6effd700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f81a3f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d64005890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ca7b2c198f457b8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:40.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d4effd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:40.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d4effd700 1 -- 172.21.15.3:0/3116751224 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6409a780 con 0x55900f80b0d0 2023-09-10T01:15:40.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d4effd700 1 -- 172.21.15.3:0/3116751224 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3d6409a080 con 0x55900f80b0d0 2023-09-10T01:15:40.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55900f6b0490 con 0x55900f80b0d0 2023-09-10T01:15:40.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d4effd700 1 -- 172.21.15.3:0/3116751224 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d6409deb0 con 0x55900f80b0d0 2023-09-10T01:15:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d8971bb80 10 client.28451 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55900f56dc00 con 0x55900f80b0d0 2023-09-10T01:15:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.746+0000 7f3d4effd700 1 -- 172.21.15.3:0/3116751224 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55900f6b0490 con 0x55900f80b0d0 2023-09-10T01:15:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d4effd700 1 -- 172.21.15.3:0/3116751224 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3d64099570 con 0x55900f80b0d0 2023-09-10T01:15:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d8971bb80 10 client.28451 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d8971bb80 10 client.28451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d8971bb80 20 client.28451 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f823ec0 0x55900f8262b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d8971bb80 4 client.28451 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55900f76d840 con 0x55900f823ec0 2023-09-10T01:15:40.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f823ec0 0x55900f8262b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f823ec0 0x55900f8262b0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b91c09e2db2ccfaf server_cookie=deebb0280b491a5c in_seq=0 out_seq=0 2023-09-10T01:15:40.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.747+0000 7f3d4effd700 10 client.28451 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:40.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d4effd700 1 -- 172.21.15.3:0/3116751224 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55900f76d840 con 0x55900f823ec0 2023-09-10T01:15:40.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d4effd700 10 client.28451 handle_command_reply: tid=0 2023-09-10T01:15:40.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 client.28451 shutdown 2023-09-10T01:15:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 msgr2=0x55900f81a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f81a3f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d64005890 comp rx=0 tx=0).stop 2023-09-10T01:15:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 shutdown_connections 2023-09-10T01:15:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f823ec0 0x55900f8262b0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f814140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f819eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3116751224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f81a3f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d4effd700 0 client.28451 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.748+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 >> 172.21.15.3:0/3116751224 conn(0x55900f77b030 msgr2=0x55900f8007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.749+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 shutdown_connections 2023-09-10T01:15:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.749+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3116751224 wait complete. 2023-09-10T01:15:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.749+0000 7f3d8971bb80 20 client.28451 trim_cache size 0 max 16384 2023-09-10T01:15:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.764+0000 7f3d8971bb80 1 Processor -- start 2023-09-10T01:15:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.764+0000 7f3d8971bb80 1 -- start start 2023-09-10T01:15:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80a710 0x55900f80fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80b0d0 0x55900f813050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f800f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f810040 con 0x55900f80b0d0 2023-09-10T01:15:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f802d10 con 0x55900f80a710 2023-09-10T01:15:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f80ee60 con 0x55900f8105b0 2023-09-10T01:15:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80a710 0x55900f80fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80b0d0 0x55900f813050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80b0d0 0x55900f813050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40560/0 (socket says 172.21.15.3:40560) 2023-09-10T01:15:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 -- 172.21.15.3:0/3243742389 learned_addr learned my addr 172.21.15.3:0/3243742389 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f800f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 -- 172.21.15.3:0/3243742389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 msgr2=0x55900f800f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f800f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 -- 172.21.15.3:0/3243742389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80a710 msgr2=0x55900f80fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80a710 0x55900f80fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.765+0000 7f3d6effd700 1 -- 172.21.15.3:0/3243742389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55900f5ab1f0 con 0x55900f80b0d0 2023-09-10T01:15:40.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d6effd700 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80b0d0 0x55900f813050 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3d64096580 tx=0x7f3d64017700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e6b9d2d1ca0ca7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:40.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d567fc700 1 -- 172.21.15.3:0/3243742389 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d640178f0 con 0x55900f80b0d0 2023-09-10T01:15:40.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d567fc700 1 -- 172.21.15.3:0/3243742389 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d6409deb0 con 0x55900f80b0d0 2023-09-10T01:15:40.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d567fc700 1 -- 172.21.15.3:0/3243742389 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6409a080 con 0x55900f80b0d0 2023-09-10T01:15:40.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3243742389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80b0d0 msgr2=0x55900f813050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80b0d0 0x55900f813050 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3d64096580 tx=0x7f3d64017700 comp rx=0 tx=0).stop 2023-09-10T01:15:40.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3243742389 shutdown_connections 2023-09-10T01:15:40.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f800f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80a710 0x55900f80fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/3243742389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80b0d0 0x55900f813050 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3243742389 >> 172.21.15.3:0/3243742389 conn(0x55900f77b030 msgr2=0x55900f812940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:40.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.766+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3243742389 shutdown_connections 2023-09-10T01:15:40.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.767+0000 7f3d8971bb80 1 -- 172.21.15.3:0/3243742389 wait complete. 2023-09-10T01:15:40.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.767+0000 7f3d8971bb80 1 -- start start 2023-09-10T01:15:40.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.767+0000 7f3d8971bb80 1 Processor -- start 2023-09-10T01:15:40.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f8713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f8718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d8971bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f86bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f86c310 con 0x55900f80a710 2023-09-10T01:15:40.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f86c480 con 0x55900f80b0d0 2023-09-10T01:15:40.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d8971bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900f86c640 con 0x55900f8105b0 2023-09-10T01:15:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d8971bb80 10 client.0 _ll_register_callbacks cb 0x7f3d600095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f8718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f8718e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:52874/0 (socket says 172.21.15.3:52874) 2023-09-10T01:15:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d6effd700 1 -- 172.21.15.3:0/293362577 learned_addr learned my addr 172.21.15.3:0/293362577 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d6ffff700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f86bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.768+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f8713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d6effd700 1 -- 172.21.15.3:0/293362577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 msgr2=0x55900f86bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d6effd700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f86bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d6effd700 1 -- 172.21.15.3:0/293362577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 msgr2=0x55900f8713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d6effd700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f8713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d6effd700 1 -- 172.21.15.3:0/293362577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55900f56dc00 con 0x55900f80b0d0 2023-09-10T01:15:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f8713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d6effd700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f8718e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400b580 tx=0x7f3d6400fdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf3048003904127 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d4effd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d4effd700 1 -- 172.21.15.3:0/293362577 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d640ca710 con 0x55900f80b0d0 2023-09-10T01:15:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55900f7518c0 con 0x55900f80b0d0 2023-09-10T01:15:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d4effd700 1 -- 172.21.15.3:0/293362577 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f3d6409deb0 con 0x55900f80b0d0 2023-09-10T01:15:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d4effd700 1 -- 172.21.15.3:0/293362577 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d6409a580 con 0x55900f80b0d0 2023-09-10T01:15:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d8971bb80 10 client.28457 fetch_fsmap learned FSMap version 35 2023-09-10T01:15:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.769+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55900f583250 con 0x55900f80b0d0 2023-09-10T01:15:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d4effd700 1 -- 172.21.15.3:0/293362577 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55900f7518c0 con 0x55900f80b0d0 2023-09-10T01:15:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d4effd700 1 -- 172.21.15.3:0/293362577 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 35) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3d640a3080 con 0x55900f80b0d0 2023-09-10T01:15:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d8971bb80 10 client.28457 fetch_fsmap finished waiting for FSMap version 35 2023-09-10T01:15:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d8971bb80 10 client.28457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d8971bb80 20 client.28457 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f878c10 0x55900f87b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d8971bb80 4 client.28457 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:15:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55900f712050 con 0x55900f878c10 2023-09-10T01:15:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.770+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f878c10 0x55900f87b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.771+0000 7f3d6f7fe700 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f878c10 0x55900f87b000 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6487e51299bb4c3 server_cookie=b41948ff7fac9e48 in_seq=0 out_seq=0 2023-09-10T01:15:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.771+0000 7f3d4effd700 10 client.28457 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:40.809 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1009102 7.63 MB/sec execute 562 sec latency 27.909 ms 2023-09-10T01:15:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.828+0000 7f3d4effd700 1 -- 172.21.15.3:0/293362577 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55900f712050 con 0x55900f878c10 2023-09-10T01:15:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.828+0000 7f3d4effd700 10 client.28457 handle_command_reply: tid=0 2023-09-10T01:15:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.828+0000 7f3d8971bb80 1 client.28457 shutdown 2023-09-10T01:15:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 msgr2=0x55900f8718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f8718e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400b580 tx=0x7f3d6400fdb0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 shutdown_connections 2023-09-10T01:15:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55900f878c10 0x55900f87b000 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55900f8105b0 0x55900f86bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900f80a710 0x55900f8713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 --2- 172.21.15.3:0/293362577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55900f80b0d0 0x55900f8718e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d4effd700 0 client.28457 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 >> 172.21.15.3:0/293362577 conn(0x55900f77b030 msgr2=0x55900f805b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 shutdown_connections 2023-09-10T01:15:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 1 -- 172.21.15.3:0/293362577 wait complete. 2023-09-10T01:15:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.829+0000 7f3d8971bb80 20 client.28457 trim_cache size 0 max 16384 2023-09-10T01:15:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:15:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:15:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:15:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:15:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d580a07b0 msgr2=0x7f3d580a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d580a07b0 0x7f3d580a2c70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d680094f0 tx=0x7f3d68009920 comp rx=0 tx=0).stop 2023-09-10T01:15:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70100f80 msgr2=0x7f3d70196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70100f80 0x7f3d70196b80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3d640098e0 tx=0x7f3d6400f690 comp rx=0 tx=0).stop 2023-09-10T01:15:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 shutdown_connections 2023-09-10T01:15:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d580a07b0 0x7f3d580a2c70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d70106680 0x7f3d7019af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d701005c0 0x7f3d70196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 --2- 172.21.15.3:0/1487596201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d70100f80 0x7f3d70196b80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 >> 172.21.15.3:0/1487596201 conn(0x7f3d700fc0b0 msgr2=0x7f3d700fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 shutdown_connections 2023-09-10T01:15:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:40.830+0000 7f3d75dff700 1 -- 172.21.15.3:0/1487596201 wait complete. 2023-09-10T01:15:41.809 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1009624 7.62 MB/sec execute 563 sec latency 375.489 ms 2023-09-10T01:15:42.466 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:42 smithi092 ceph-mon[134067]: pgmap v389: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2023-09-10T01:15:42.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:42 smithi003 ceph-mon[97654]: pgmap v389: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2023-09-10T01:15:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:42 smithi136 ceph-mon[133708]: pgmap v389: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2023-09-10T01:15:42.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1009624 7.60 MB/sec execute 564 sec latency 1375.609 ms 2023-09-10T01:15:43.809 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1010450 7.60 MB/sec execute 565 sec latency 1670.864 ms 2023-09-10T01:15:44.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:44 smithi092 ceph-mon[134067]: pgmap v390: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2023-09-10T01:15:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:44 smithi003 ceph-mon[97654]: pgmap v390: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2023-09-10T01:15:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:44 smithi136 ceph-mon[133708]: pgmap v390: 129 pgs: 129 active+clean; 317 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.0 MiB/s wr, 96 op/s 2023-09-10T01:15:44.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1012164 7.60 MB/sec execute 566 sec latency 23.239 ms 2023-09-10T01:15:45.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1013171 7.60 MB/sec execute 567 sec latency 20.210 ms 2023-09-10T01:15:46.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:46 smithi003 ceph-mon[97654]: pgmap v391: 129 pgs: 129 active+clean; 318 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 165 op/s 2023-09-10T01:15:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:46 smithi136 ceph-mon[133708]: pgmap v391: 129 pgs: 129 active+clean; 318 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 165 op/s 2023-09-10T01:15:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:46 smithi092 ceph-mon[134067]: pgmap v391: 129 pgs: 129 active+clean; 318 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 165 op/s 2023-09-10T01:15:46.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1014796 7.59 MB/sec execute 568 sec latency 18.614 ms 2023-09-10T01:15:47.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:47 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:15:47.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:47 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:15:47.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:47 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:15:47.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1016168 7.60 MB/sec execute 569 sec latency 38.508 ms 2023-09-10T01:15:48.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:48 smithi003 ceph-mon[97654]: pgmap v392: 129 pgs: 129 active+clean; 318 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2023-09-10T01:15:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:48 smithi136 ceph-mon[133708]: pgmap v392: 129 pgs: 129 active+clean; 318 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2023-09-10T01:15:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:48 smithi092 ceph-mon[134067]: pgmap v392: 129 pgs: 129 active+clean; 318 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2023-09-10T01:15:48.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1016716 7.59 MB/sec execute 570 sec latency 241.518 ms 2023-09-10T01:15:49.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1016716 7.58 MB/sec execute 571 sec latency 1241.643 ms 2023-09-10T01:15:50.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:50 smithi003 ceph-mon[97654]: pgmap v393: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2023-09-10T01:15:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:50 smithi136 ceph-mon[133708]: pgmap v393: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2023-09-10T01:15:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:50 smithi092 ceph-mon[134067]: pgmap v393: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 206 op/s 2023-09-10T01:15:50.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1016716 7.56 MB/sec execute 572 sec latency 2241.806 ms 2023-09-10T01:15:51.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1016716 7.55 MB/sec execute 573 sec latency 3241.987 ms 2023-09-10T01:15:52.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:52 smithi003 ceph-mon[97654]: pgmap v394: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:15:52.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:52 smithi136 ceph-mon[133708]: pgmap v394: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:15:52.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:52 smithi092 ceph-mon[134067]: pgmap v394: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:15:52.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1016716 7.54 MB/sec execute 574 sec latency 4242.040 ms 2023-09-10T01:15:53.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:15:53.810 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1017568 7.53 MB/sec execute 575 sec latency 4543.912 ms 2023-09-10T01:15:53.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.966+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/4055179114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 msgr2=0x7f8b1810c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:53.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.966+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/4055179114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1810c9b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009990 tx=0x7f8b0c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.967+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/4055179114 shutdown_connections 2023-09-10T01:15:53.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.967+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/4055179114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b180729d0 0x7f8b18072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.967+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/4055179114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b1810cef0 0x7f8b1810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.967+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/4055179114 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1810c9b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.967+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/4055179114 >> 172.21.15.3:0/4055179114 conn(0x7f8b180fc160 msgr2=0x7f8b180fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:53.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.967+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/4055179114 shutdown_connections 2023-09-10T01:15:53.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.967+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/4055179114 wait complete. 2023-09-10T01:15:53.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.968+0000 7f8b1e1ff700 1 Processor -- start 2023-09-10T01:15:53.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b1e1ff700 1 -- start start 2023-09-10T01:15:53.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b1e1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b180729d0 0x7f8b1819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b1e1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b1e1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b1810cef0 0x7f8b18198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b1e1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b18198e80 con 0x7f8b180729d0 2023-09-10T01:15:53.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b1e1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b18198ff0 con 0x7f8b180732f0 2023-09-10T01:15:53.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b1e1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b181991b0 con 0x7f8b1810cef0 2023-09-10T01:15:53.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b16ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b180729d0 0x7f8b1819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b180729d0 0x7f8b1819e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59930/0 (socket says 172.21.15.3:59930) 2023-09-10T01:15:53.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b177fe700 1 -- 172.21.15.3:0/2972334673 learned_addr learned my addr 172.21.15.3:0/2972334673 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:53.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.969+0000 7f8b16ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1819ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57414/0 (socket says 172.21.15.3:57414) 2023-09-10T01:15:53.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b17fff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b1810cef0 0x7f8b18198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b16ffd700 1 -- 172.21.15.3:0/2972334673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b1810cef0 msgr2=0x7f8b18198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:53.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b1810cef0 0x7f8b18198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b16ffd700 1 -- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b180729d0 msgr2=0x7f8b1819e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:53.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b180729d0 0x7f8b1819e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b16ffd700 1 -- 172.21.15.3:0/2972334673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10009c40 con 0x7f8b180732f0 2023-09-10T01:15:53.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b180729d0 0x7f8b1819e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:53.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1819ece0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009990 tx=0x7f8b0c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:53.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.970+0000 7f8b14ff9700 1 -- 172.21.15.3:0/2972334673 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c01d070 con 0x7f8b180732f0 2023-09-10T01:15:53.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.971+0000 7f8b14ff9700 1 -- 172.21.15.3:0/2972334673 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b0c021410 con 0x7f8b180732f0 2023-09-10T01:15:53.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.971+0000 7f8b14ff9700 1 -- 172.21.15.3:0/2972334673 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c017650 con 0x7f8b180732f0 2023-09-10T01:15:53.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.971+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0c0095f0 con 0x7f8b180732f0 2023-09-10T01:15:53.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.971+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b181a55b0 con 0x7f8b180732f0 2023-09-10T01:15:53.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.974+0000 7f8b14ff9700 1 -- 172.21.15.3:0/2972334673 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f8b0c0217a0 con 0x7f8b180732f0 2023-09-10T01:15:53.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.975+0000 7f8b31b35b80 1 Processor -- start 2023-09-10T01:15:53.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.975+0000 7f8b31b35b80 1 -- start start 2023-09-10T01:15:53.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bc6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0b1f730 con 0x55f8d0bc6710 2023-09-10T01:15:53.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0b1d050 con 0x55f8d0bc7030 2023-09-10T01:15:53.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0b18940 con 0x55f8d0bd0e40 2023-09-10T01:15:53.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bc6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b177fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bc6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59940/0 (socket says 172.21.15.3:59940) 2023-09-10T01:15:53.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b17fff700 1 -- 172.21.15.3:0/2988487196 learned_addr learned my addr 172.21.15.3:0/2988487196 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:53.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.976+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b177fe700 1 -- 172.21.15.3:0/2988487196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 msgr2=0x55f8d0bd3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:53.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b177fe700 1 -- 172.21.15.3:0/2988487196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 msgr2=0x55f8d0bc6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:53.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bc6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b177fe700 1 -- 172.21.15.3:0/2988487196 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8d0944de0 con 0x55f8d0bc7030 2023-09-10T01:15:53.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b17fff700 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bc6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:53.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b14ff9700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8b000a0830 0x7f8b000a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd0900 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8b08011570 tx=0x7f8b08011880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec1e7a97d9a0ae1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:53.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b14ff9700 1 -- 172.21.15.3:0/2972334673 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8b0c021e40 con 0x7f8b180732f0 2023-09-10T01:15:53.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8b17fff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8b000a0830 0x7f8b000a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8afdffb700 1 -- 172.21.15.3:0/2988487196 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08004230 con 0x55f8d0bc7030 2023-09-10T01:15:53.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8afdffb700 1 -- 172.21.15.3:0/2988487196 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b08004390 con 0x55f8d0bc7030 2023-09-10T01:15:53.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.977+0000 7f8afdffb700 1 -- 172.21.15.3:0/2988487196 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08019880 con 0x55f8d0bc7030 2023-09-10T01:15:53.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2988487196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 msgr2=0x55f8d0bd0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:53.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b17fff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8b000a0830 0x7f8b000a2cf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000ead0 tx=0x7f8b1000b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:53.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd0900 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8b08011570 tx=0x7f8b08011880 comp rx=0 tx=0).stop 2023-09-10T01:15:53.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2988487196 shutdown_connections 2023-09-10T01:15:53.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bc6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2988487196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd0900 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2988487196 >> 172.21.15.3:0/2988487196 conn(0x55f8d0b3b090 msgr2=0x55f8d0bc1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:53.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2988487196 shutdown_connections 2023-09-10T01:15:53.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.978+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2988487196 wait complete. 2023-09-10T01:15:53.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.979+0000 7f8b31b35b80 1 -- start start 2023-09-10T01:15:53.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.979+0000 7f8b31b35b80 1 Processor -- start 2023-09-10T01:15:53.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0ab6b70 con 0x55f8d0bc7030 2023-09-10T01:15:53.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0ab6870 con 0x55f8d0bc6710 2023-09-10T01:15:53.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d099a920 con 0x55f8d0bd0e40 2023-09-10T01:15:53.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b31b35b80 10 client.0 _ll_register_callbacks cb 0x55f8d0a00a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:53.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57444/0 (socket says 172.21.15.3:57444) 2023-09-10T01:15:53.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 -- 172.21.15.3:0/2261854759 learned_addr learned my addr 172.21.15.3:0/2261854759 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:53.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b17fff700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:53.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 -- 172.21.15.3:0/2261854759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 msgr2=0x55f8d0bcdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:53.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 -- 172.21.15.3:0/2261854759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc7030 msgr2=0x55f8d0bcda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:53.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:53.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b177fe700 1 -- 172.21.15.3:0/2261854759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1000e370 con 0x55f8d0bc6710 2023-09-10T01:15:53.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.980+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:53.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd0750 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f8b08000c00 tx=0x7f8b08005320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4469985ccf7ead02 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:53.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8af6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:53.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8af6ffd700 1 -- 172.21.15.3:0/2261854759 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08005d00 con 0x55f8d0bc6710 2023-09-10T01:15:53.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8d0a704f0 con 0x55f8d0bc6710 2023-09-10T01:15:53.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8af6ffd700 1 -- 172.21.15.3:0/2261854759 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8b08004950 con 0x55f8d0bc6710 2023-09-10T01:15:53.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8af6ffd700 1 -- 172.21.15.3:0/2261854759 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b08004b10 con 0x55f8d0bc6710 2023-09-10T01:15:53.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8b31b35b80 10 client.28469 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:53.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.981+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7f8b08009d60 con 0x55f8d0bc6710 2023-09-10T01:15:53.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8af6ffd700 1 -- 172.21.15.3:0/2261854759 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f8d0a704f0 con 0x55f8d0bc6710 2023-09-10T01:15:53.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8af6ffd700 1 -- 172.21.15.3:0/2261854759 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8b08021040 con 0x55f8d0bc6710 2023-09-10T01:15:53.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8b31b35b80 10 client.28469 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:53.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8b31b35b80 10 client.28469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:53.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8b31b35b80 20 client.28469 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0be0880 0x55f8d0be2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8b31b35b80 4 client.28469 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8d0b2d8a0 con 0x55f8d0be0880 2023-09-10T01:15:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.982+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0be0880 0x55f8d0be2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.983+0000 7f8af6ffd700 10 client.28469 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.983+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0be0880 0x55f8d0be2ca0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caf97fd8588f671 server_cookie=b013240fb0ff1cc7 in_seq=0 out_seq=0 2023-09-10T01:15:54.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8af6ffd700 1 -- 172.21.15.3:0/2261854759 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f8d0b2d8a0 con 0x55f8d0be0880 2023-09-10T01:15:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8af6ffd700 10 client.28469 handle_command_reply: tid=0 2023-09-10T01:15:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 client.28469 shutdown 2023-09-10T01:15:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc6710 msgr2=0x55f8d0bd0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd0750 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f8b08000c00 tx=0x7f8b08005320 comp rx=0 tx=0).stop 2023-09-10T01:15:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 shutdown_connections 2023-09-10T01:15:54.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0be0880 0x55f8d0be2ca0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd0750 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8af6ffd700 0 client.28469 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2261854759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 >> 172.21.15.3:0/2261854759 conn(0x55f8d0b3b090 msgr2=0x55f8d0bc17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 shutdown_connections 2023-09-10T01:15:54.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.984+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2261854759 wait complete. 2023-09-10T01:15:54.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:53.985+0000 7f8b31b35b80 20 client.28469 trim_cache size 0 max 16384 2023-09-10T01:15:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 -- 172.21.15.3:0/2836371309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 msgr2=0x7fe5781139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 --2- 172.21.15.3:0/2836371309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781139a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe57000b0a0 tx=0x7fe57000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 -- 172.21.15.3:0/2836371309 shutdown_connections 2023-09-10T01:15:54.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 --2- 172.21.15.3:0/2836371309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781139a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 --2- 172.21.15.3:0/2836371309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5780729a0 0x7fe578072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 --2- 172.21.15.3:0/2836371309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5780732c0 0x7fe578111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 -- 172.21.15.3:0/2836371309 >> 172.21.15.3:0/2836371309 conn(0x7fe5780fc0b0 msgr2=0x7fe5780fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 -- 172.21.15.3:0/2836371309 shutdown_connections 2023-09-10T01:15:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.991+0000 7fe5803ff700 1 -- 172.21.15.3:0/2836371309 wait complete. 2023-09-10T01:15:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.992+0000 7fe5803ff700 1 Processor -- start 2023-09-10T01:15:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe5803ff700 1 -- start start 2023-09-10T01:15:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe5803ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5780729a0 0x7fe57819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe5803ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5780732c0 0x7fe57819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe5803ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe5803ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5781a1e80 con 0x7fe5780732c0 2023-09-10T01:15:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe5803ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5781a1ff0 con 0x7fe5781115b0 2023-09-10T01:15:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe5803ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5781a21b0 con 0x7fe5780729a0 2023-09-10T01:15:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe57e99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe57e99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57452/0 (socket says 172.21.15.3:57452) 2023-09-10T01:15:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe57e99c700 1 -- 172.21.15.3:0/1577174885 learned_addr learned my addr 172.21.15.3:0/1577174885 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe57e19b700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5780729a0 0x7fe57819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.993+0000 7fe57d99a700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5780732c0 0x7fe57819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe57e99c700 1 -- 172.21.15.3:0/1577174885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5780729a0 msgr2=0x7fe57819cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe57e99c700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5780729a0 0x7fe57819cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe57e99c700 1 -- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5780732c0 msgr2=0x7fe57819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe57e99c700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5780732c0 0x7fe57819d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe57e99c700 1 -- 172.21.15.3:0/1577174885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574009520 con 0x7fe5781115b0 2023-09-10T01:15:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe57e99c700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781a1840 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe570009fd0 tx=0x7fe57000bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe56f7fe700 1 -- 172.21.15.3:0/1577174885 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57000e040 con 0x7fe5781115b0 2023-09-10T01:15:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe56f7fe700 1 -- 172.21.15.3:0/1577174885 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5700094a0 con 0x7fe5781115b0 2023-09-10T01:15:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe570009d00 con 0x7fe5781115b0 2023-09-10T01:15:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe56f7fe700 1 -- 172.21.15.3:0/1577174885 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57001b6c0 con 0x7fe5781115b0 2023-09-10T01:15:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.994+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5781a9af0 con 0x7fe5781115b0 2023-09-10T01:15:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.998+0000 7fe56f7fe700 1 -- 172.21.15.3:0/1577174885 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe57001d470 con 0x7fe5781115b0 2023-09-10T01:15:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 Processor -- start 2023-09-10T01:15:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 -- start start 2023-09-10T01:15:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e37580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e31780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61d8a6d0 con 0x558b61e31300 2023-09-10T01:15:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61d87ff0 con 0x558b61e31cc0 2023-09-10T01:15:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:53.999+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61d838e0 con 0x558b61e371a0 2023-09-10T01:15:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57e19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e31780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57454/0 (socket says 172.21.15.3:57454) 2023-09-10T01:15:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57e19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e31780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59968/0 (socket says 172.21.15.3:59968) 2023-09-10T01:15:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57d99a700 1 -- 172.21.15.3:0/1520034675 learned_addr learned my addr 172.21.15.3:0/1520034675 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57d99a700 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e37580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57e19b700 1 -- 172.21.15.3:0/1520034675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 msgr2=0x558b61e37580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57e19b700 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e37580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57e19b700 1 -- 172.21.15.3:0/1520034675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 msgr2=0x558b61e3df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57e19b700 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe57e19b700 1 -- 172.21.15.3:0/1520034675 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b61bafd80 con 0x558b61e31300 2023-09-10T01:15:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe56f7fe700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5640a0830 0x7fe5640a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.000+0000 7fe56f7fe700 1 -- 172.21.15.3:0/1577174885 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe5700cd630 con 0x7fe5781115b0 2023-09-10T01:15:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.001+0000 7fe57e19b700 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e31780 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe574009e40 tx=0x7fe574011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe2f214917dd90d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.001+0000 7fe57d99a700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5640a0830 0x7fe5640a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.001+0000 7fe56cbfe700 1 -- 172.21.15.3:0/1520034675 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574019630 con 0x558b61e31300 2023-09-10T01:15:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.001+0000 7fe56cbfe700 1 -- 172.21.15.3:0/1520034675 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe574019c50 con 0x558b61e31300 2023-09-10T01:15:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.001+0000 7fe56cbfe700 1 -- 172.21.15.3:0/1520034675 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57400ed10 con 0x558b61e31300 2023-09-10T01:15:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 -- 172.21.15.3:0/1520034675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 msgr2=0x558b61e31780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe57d99a700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5640a0830 0x7fe5640a2cf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe568009fd0 tx=0x7fe5680096f0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e31780 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe574009e40 tx=0x7fe574011720 comp rx=0 tx=0).stop 2023-09-10T01:15:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 -- 172.21.15.3:0/1520034675 shutdown_connections 2023-09-10T01:15:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e37580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/1520034675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e31780 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 -- 172.21.15.3:0/1520034675 >> 172.21.15.3:0/1520034675 conn(0x558b61da6030 msgr2=0x558b61e2c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 -- 172.21.15.3:0/1520034675 shutdown_connections 2023-09-10T01:15:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.002+0000 7fe593ceeb80 1 -- 172.21.15.3:0/1520034675 wait complete. 2023-09-10T01:15:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 -- start start 2023-09-10T01:15:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 Processor -- start 2023-09-10T01:15:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e3ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e3b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61d21b10 con 0x558b61e31300 2023-09-10T01:15:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61d21810 con 0x558b61e31cc0 2023-09-10T01:15:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61c058c0 con 0x558b61e371a0 2023-09-10T01:15:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.003+0000 7fe593ceeb80 10 client.0 _ll_register_callbacks cb 0x558b61c6ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57464/0 (socket says 172.21.15.3:57464) 2023-09-10T01:15:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 -- 172.21.15.3:0/3950912455 learned_addr learned my addr 172.21.15.3:0/3950912455 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57e19b700 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e3ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57e99c700 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e3b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 -- 172.21.15.3:0/3950912455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 msgr2=0x558b61e3b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e3b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 -- 172.21.15.3:0/3950912455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 msgr2=0x558b61e3ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e3ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 -- 172.21.15.3:0/3950912455 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe574009a70 con 0x558b61e31cc0 2023-09-10T01:15:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe57d99a700 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3b070 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe568004690 tx=0x7fe568004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9379afcfe6615e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.004+0000 7fe561ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe561ffb700 1 -- 172.21.15.3:0/3950912455 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5680106c0 con 0x558b61e31cc0 2023-09-10T01:15:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b61cdb490 con 0x558b61e31cc0 2023-09-10T01:15:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe561ffb700 1 -- 172.21.15.3:0/3950912455 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe568010820 con 0x558b61e31cc0 2023-09-10T01:15:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe561ffb700 1 -- 172.21.15.3:0/3950912455 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5680109e0 con 0x558b61e31cc0 2023-09-10T01:15:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe593ceeb80 10 client.28481 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x558b61b98c00 con 0x558b61e31cc0 2023-09-10T01:15:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe561ffb700 1 -- 172.21.15.3:0/3950912455 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558b61cdb490 con 0x558b61e31cc0 2023-09-10T01:15:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe561ffb700 1 -- 172.21.15.3:0/3950912455 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe56802cbc0 con 0x558b61e31cc0 2023-09-10T01:15:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.005+0000 7fe593ceeb80 10 client.28481 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe593ceeb80 10 client.28481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:54.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe593ceeb80 20 client.28481 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:54.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61e4b460 0x558b61e4d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe593ceeb80 4 client.28481 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:54.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b61d98840 con 0x558b61e4b460 2023-09-10T01:15:54.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe57e19b700 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61e4b460 0x558b61e4d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe57e19b700 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61e4b460 0x558b61e4d880 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb46afed2e92483 server_cookie=7b7232b51ef74fd6 in_seq=0 out_seq=0 2023-09-10T01:15:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.006+0000 7fe561ffb700 10 client.28481 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.006+0000 7f8b31b35b80 1 Processor -- start 2023-09-10T01:15:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b31b35b80 1 -- start start 2023-09-10T01:15:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe561ffb700 1 -- 172.21.15.3:0/3950912455 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558b61d98840 con 0x558b61e4b460 2023-09-10T01:15:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe561ffb700 10 client.28481 handle_command_reply: tid=0 2023-09-10T01:15:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 client.28481 shutdown 2023-09-10T01:15:54.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 msgr2=0x558b61e3b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3b070 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe568004690 tx=0x7fe568004750 comp rx=0 tx=0).stop 2023-09-10T01:15:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0bc7410 con 0x55f8d0bc6710 2023-09-10T01:15:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0bcbfa0 con 0x55f8d0bd0e40 2023-09-10T01:15:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0bc26e0 con 0x55f8d0bc7030 2023-09-10T01:15:54.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 shutdown_connections 2023-09-10T01:15:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61e4b460 0x558b61e4d880 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e371a0 0x558b61e3b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31300 0x558b61e3ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b16ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/3950912455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3b070 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 >> 172.21.15.3:0/3950912455 conn(0x558b61da6030 msgr2=0x558b61e2b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.007+0000 7fe561ffb700 0 client.28481 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b17fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57468/0 (socket says 172.21.15.3:57468) 2023-09-10T01:15:54.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.007+0000 7f8b17fff700 1 -- 172.21.15.3:0/2981757937 learned_addr learned my addr 172.21.15.3:0/2981757937 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.008+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 shutdown_connections 2023-09-10T01:15:54.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b16ffd700 1 -- 172.21.15.3:0/2981757937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bd0e40 msgr2=0x55f8d0bcd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.008+0000 7fe593ceeb80 1 -- 172.21.15.3:0/3950912455 wait complete. 2023-09-10T01:15:54.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b16ffd700 1 -- 172.21.15.3:0/2981757937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 msgr2=0x55f8d0bd02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b16ffd700 1 -- 172.21.15.3:0/2981757937 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8d092dc60 con 0x55f8d0bc7030 2023-09-10T01:15:54.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b177fe700 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:15:54.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcd200 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0ca970 tx=0x7f8b0c0cf930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b582196e0ab4e88 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.008+0000 7fe593ceeb80 20 client.28481 trim_cache size 0 max 16384 2023-09-10T01:15:54.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8afe7fc700 1 -- 172.21.15.3:0/2981757937 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c096aa0 con 0x55f8d0bc7030 2023-09-10T01:15:54.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8afe7fc700 1 -- 172.21.15.3:0/2981757937 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b0c09a530 con 0x55f8d0bc7030 2023-09-10T01:15:54.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.008+0000 7f8afe7fc700 1 -- 172.21.15.3:0/2981757937 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c09e620 con 0x55f8d0bc7030 2023-09-10T01:15:54.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2981757937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bc7030 msgr2=0x55f8d0bcd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcd200 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0ca970 tx=0x7f8b0c0cf930 comp rx=0 tx=0).stop 2023-09-10T01:15:54.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2981757937 shutdown_connections 2023-09-10T01:15:54.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bcd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/2981757937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bc7030 0x55f8d0bcd200 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2981757937 >> 172.21.15.3:0/2981757937 conn(0x55f8d0b3b090 msgr2=0x55f8d0bc8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2981757937 shutdown_connections 2023-09-10T01:15:54.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.009+0000 7f8b31b35b80 1 -- 172.21.15.3:0/2981757937 wait complete. 2023-09-10T01:15:54.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.010+0000 7f8b31b35b80 1 -- start start 2023-09-10T01:15:54.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.010+0000 7f8b31b35b80 1 Processor -- start 2023-09-10T01:15:54.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b31b35b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0bd6610 con 0x55f8d0bc6710 2023-09-10T01:15:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0bd6750 con 0x55f8d0bc7030 2023-09-10T01:15:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b31b35b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8d0bd6910 con 0x55f8d0bd0e40 2023-09-10T01:15:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b31b35b80 10 client.0 _ll_register_callbacks cb 0x7f8b1000e280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b16ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b16ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57480/0 (socket says 172.21.15.3:57480) 2023-09-10T01:15:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b16ffd700 1 -- 172.21.15.3:0/822108387 learned_addr learned my addr 172.21.15.3:0/822108387 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b17fff700 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b16ffd700 1 -- 172.21.15.3:0/822108387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 msgr2=0x55f8d0bd6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b16ffd700 1 -- 172.21.15.3:0/822108387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 msgr2=0x55f8d0bd7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.011+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8b16ffd700 1 -- 172.21.15.3:0/822108387 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b181a55b0 con 0x55f8d0bc7030 2023-09-10T01:15:54.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8b16ffd700 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd5b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0a2210 tx=0x7f8b0c0d0f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b578f9af5cbfc291 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8af6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:54.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8af6ffd700 1 -- 172.21.15.3:0/822108387 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c09a0c0 con 0x55f8d0bc7030 2023-09-10T01:15:54.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8af6ffd700 1 -- 172.21.15.3:0/822108387 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f8b0c09ae60 con 0x55f8d0bc7030 2023-09-10T01:15:54.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b08004390 con 0x55f8d0bc7030 2023-09-10T01:15:54.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8af6ffd700 1 -- 172.21.15.3:0/822108387 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b0c0a2740 con 0x55f8d0bc7030 2023-09-10T01:15:54.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8b31b35b80 10 client.28487 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:54.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55f8d096b250 con 0x55f8d0bc7030 2023-09-10T01:15:54.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8af6ffd700 1 -- 172.21.15.3:0/822108387 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b08004390 con 0x55f8d0bc7030 2023-09-10T01:15:54.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.012+0000 7f8af6ffd700 1 -- 172.21.15.3:0/822108387 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8b0c099c50 con 0x55f8d0bc7030 2023-09-10T01:15:54.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b31b35b80 10 client.28487 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:54.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b31b35b80 10 client.28487 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:54.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b31b35b80 20 client.28487 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:54.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0c358e0 0x55f8d0c37d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b31b35b80 4 client.28487 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:15:54.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f8d0b11920 con 0x55f8d0c358e0 2023-09-10T01:15:54.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b177fe700 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0c358e0 0x55f8d0c37d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b177fe700 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0c358e0 0x55f8d0c37d00 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=957146062995b491 server_cookie=3caf67f84ce8f8e1 in_seq=0 out_seq=0 2023-09-10T01:15:54.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8af6ffd700 10 client.28487 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.062 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8af6ffd700 1 -- 172.21.15.3:0/822108387 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f8d0b11920 con 0x55f8d0c358e0 2023-09-10T01:15:54.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8af6ffd700 10 client.28487 handle_command_reply: tid=0 2023-09-10T01:15:54.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.013+0000 7f8b31b35b80 1 client.28487 shutdown 2023-09-10T01:15:54.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 msgr2=0x55f8d0bd5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd5b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0a2210 tx=0x7f8b0c0d0f20 comp rx=0 tx=0).stop 2023-09-10T01:15:54.063 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 shutdown_connections 2023-09-10T01:15:54.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f8d0c358e0 0x55f8d0c37d00 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f8d0bd0e40 0x55f8d0bd6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8d0bc6710 0x55f8d0bd7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 --2- 172.21.15.3:0/822108387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8d0bc7030 0x55f8d0bd5b50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 >> 172.21.15.3:0/822108387 conn(0x55f8d0b3b090 msgr2=0x55f8d0bc5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.064 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8af6ffd700 0 client.28487 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 shutdown_connections 2023-09-10T01:15:54.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 1 -- 172.21.15.3:0/822108387 wait complete. 2023-09-10T01:15:54.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.014+0000 7f8b31b35b80 20 client.28487 trim_cache size 0 max 16384 2023-09-10T01:15:54.066 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:15:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:15:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:15:54.066 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:15:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8b000a0830 msgr2=0x7f8b000a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8b000a0830 0x7f8b000a2cf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000ead0 tx=0x7f8b1000b040 comp rx=0 tx=0).stop 2023-09-10T01:15:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 msgr2=0x7f8b1819ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1819ece0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009990 tx=0x7f8b0c009580 comp rx=0 tx=0).stop 2023-09-10T01:15:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 shutdown_connections 2023-09-10T01:15:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8b000a0830 0x7f8b000a2cf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b1810cef0 0x7f8b18198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b180729d0 0x7f8b1819e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 --2- 172.21.15.3:0/2972334673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b180732f0 0x7f8b1819ece0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 >> 172.21.15.3:0/2972334673 conn(0x7f8b180fc160 msgr2=0x7f8b1810f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.015+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 shutdown_connections 2023-09-10T01:15:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.016+0000 7f8b1e1ff700 1 -- 172.21.15.3:0/2972334673 wait complete. 2023-09-10T01:15:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 Processor -- start 2023-09-10T01:15:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 -- start start 2023-09-10T01:15:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31cc0 0x558b61e3ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e371a0 0x558b61e42520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61e42b00 con 0x558b61e31cc0 2023-09-10T01:15:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61e23ea0 con 0x558b61e371a0 2023-09-10T01:15:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61e359b0 con 0x558b61e31300 2023-09-10T01:15:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31cc0 0x558b61e3ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe57e99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e371a0 0x558b61e42520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe57e99c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e371a0 0x558b61e42520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57492/0 (socket says 172.21.15.3:57492) 2023-09-10T01:15:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe57e99c700 1 -- 172.21.15.3:0/2556718621 learned_addr learned my addr 172.21.15.3:0/2556718621 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31cc0 0x558b61e3ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60006/0 (socket says 172.21.15.3:60006) 2023-09-10T01:15:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.026+0000 7fe57e19b700 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe57e99c700 1 -- 172.21.15.3:0/2556718621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 msgr2=0x558b61e3d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe57e99c700 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe57e99c700 1 -- 172.21.15.3:0/2556718621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31cc0 msgr2=0x558b61e3ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe57e99c700 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31cc0 0x558b61e3ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe57e99c700 1 -- 172.21.15.3:0/2556718621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b61bd61f0 con 0x558b61e371a0 2023-09-10T01:15:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe57e99c700 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e371a0 0x558b61e42520 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe5700ca990 tx=0x7fe570009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74bae6ebe2334433 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe56d3ff700 1 -- 172.21.15.3:0/2556718621 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570011e70 con 0x558b61e371a0 2023-09-10T01:15:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe56d3ff700 1 -- 172.21.15.3:0/2556718621 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe570007930 con 0x558b61e371a0 2023-09-10T01:15:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.027+0000 7fe56d3ff700 1 -- 172.21.15.3:0/2556718621 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57009e120 con 0x558b61e371a0 2023-09-10T01:15:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 -- 172.21.15.3:0/2556718621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e371a0 msgr2=0x558b61e42520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e371a0 0x558b61e42520 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe5700ca990 tx=0x7fe570009c70 comp rx=0 tx=0).stop 2023-09-10T01:15:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 -- 172.21.15.3:0/2556718621 shutdown_connections 2023-09-10T01:15:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e371a0 0x558b61e42520 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/2556718621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e31cc0 0x558b61e3ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 -- 172.21.15.3:0/2556718621 >> 172.21.15.3:0/2556718621 conn(0x558b61da6030 msgr2=0x558b61e521e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 -- 172.21.15.3:0/2556718621 shutdown_connections 2023-09-10T01:15:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.028+0000 7fe593ceeb80 1 -- 172.21.15.3:0/2556718621 wait complete. 2023-09-10T01:15:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.029+0000 7fe593ceeb80 1 -- start start 2023-09-10T01:15:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.029+0000 7fe593ceeb80 1 Processor -- start 2023-09-10T01:15:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.029+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.029+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe593ceeb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e371a0 0x558b61e98250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61e98790 con 0x558b61e371a0 2023-09-10T01:15:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61e98900 con 0x558b61e31cc0 2023-09-10T01:15:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe593ceeb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b61e98ac0 con 0x558b61e31300 2023-09-10T01:15:54.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe593ceeb80 10 client.0 _ll_register_callbacks cb 0x7fe568004480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57d99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:57494/0 (socket says 172.21.15.3:57494) 2023-09-10T01:15:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57d99a700 1 -- 172.21.15.3:0/248957955 learned_addr learned my addr 172.21.15.3:0/248957955 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57e99c700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e371a0 0x558b61e98250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57e19b700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57e19b700 1 -- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 msgr2=0x558b61e3ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57e19b700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57e19b700 1 -- 172.21.15.3:0/248957955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e371a0 msgr2=0x558b61e98250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57e19b700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e371a0 0x558b61e98250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57e19b700 1 -- 172.21.15.3:0/248957955 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5781a9af0 con 0x558b61e31300 2023-09-10T01:15:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.030+0000 7fe57d99a700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe57e19b700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3e6f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe574009e10 tx=0x7fe5740092f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be08d6f57493ea server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe561ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:15:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe561ffb700 1 -- 172.21.15.3:0/248957955 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57400ee20 con 0x558b61e31300 2023-09-10T01:15:54.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe574019c50 con 0x558b61e31300 2023-09-10T01:15:54.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe561ffb700 1 -- 172.21.15.3:0/248957955 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe574029070 con 0x558b61e31300 2023-09-10T01:15:54.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe561ffb700 1 -- 172.21.15.3:0/248957955 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe57400e530 con 0x558b61e31300 2023-09-10T01:15:54.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe593ceeb80 10 client.28375 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:54.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x558b61bae250 con 0x558b61e31300 2023-09-10T01:15:54.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.031+0000 7fe561ffb700 1 -- 172.21.15.3:0/248957955 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe574019c50 con 0x558b61e31300 2023-09-10T01:15:54.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe561ffb700 1 -- 172.21.15.3:0/248957955 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe574005490 con 0x558b61e31300 2023-09-10T01:15:54.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe593ceeb80 10 client.28375 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:54.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe593ceeb80 10 client.28375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:54.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe593ceeb80 20 client.28375 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:54.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61ea0330 0x558b61ea2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe593ceeb80 4 client.28375 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:15:54.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558b61d7c8c0 con 0x558b61ea0330 2023-09-10T01:15:54.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.032+0000 7fe57d99a700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61ea0330 0x558b61ea2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.033+0000 7fe57d99a700 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61ea0330 0x558b61ea2750 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92ad5edf1dabeb75 server_cookie=dbbfe72b38c786f9 in_seq=0 out_seq=0 2023-09-10T01:15:54.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.033+0000 7fe561ffb700 10 client.28375 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.090 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:15:54.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:15:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.119+0000 7fe561ffb700 1 -- 172.21.15.3:0/248957955 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558b61d7c8c0 con 0x558b61ea0330 2023-09-10T01:15:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe561ffb700 10 client.28375 handle_command_reply: tid=0 2023-09-10T01:15:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 client.28375 shutdown 2023-09-10T01:15:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 msgr2=0x558b61e3e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3e6f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe574009e10 tx=0x7fe5740092f0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 shutdown_connections 2023-09-10T01:15:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x558b61ea0330 0x558b61ea2750 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558b61e31300 0x558b61e3e6f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b61e371a0 0x558b61e98250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe561ffb700 0 client.28375 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 --2- 172.21.15.3:0/248957955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558b61e31cc0 0x558b61e3ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 >> 172.21.15.3:0/248957955 conn(0x558b61da6030 msgr2=0x558b61e360a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 shutdown_connections 2023-09-10T01:15:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 1 -- 172.21.15.3:0/248957955 wait complete. 2023-09-10T01:15:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.120+0000 7fe593ceeb80 20 client.28375 trim_cache size 0 max 16384 2023-09-10T01:15:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:15:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:15:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:15:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:15:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5640a0830 msgr2=0x7fe5640a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5640a0830 0x7fe5640a2cf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe568009fd0 tx=0x7fe5680096f0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 msgr2=0x7fe5781a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781a1840 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe570009fd0 tx=0x7fe57000bce0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 shutdown_connections 2023-09-10T01:15:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe5640a0830 0x7fe5640a2cf0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5780729a0 0x7fe57819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5781115b0 0x7fe5781a1840 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 --2- 172.21.15.3:0/1577174885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5780732c0 0x7fe57819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 >> 172.21.15.3:0/1577174885 conn(0x7fe5780fc0b0 msgr2=0x7fe5780fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.121+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 shutdown_connections 2023-09-10T01:15:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:15:54.122+0000 7fe5803ff700 1 -- 172.21.15.3:0/1577174885 wait complete. 2023-09-10T01:15:54.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 -- 172.21.15.3:0/630206989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4640ff470 msgr2=0x7ff4640ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 --2- 172.21.15.3:0/630206989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4640ff470 0x7ff4640ff850 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff458009990 tx=0x7ff458009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 -- 172.21.15.3:0/630206989 shutdown_connections 2023-09-10T01:15:54.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 --2- 172.21.15.3:0/630206989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff46410d020 0x7ff46410f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 --2- 172.21.15.3:0/630206989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4640ff470 0x7ff4640ff850 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 --2- 172.21.15.3:0/630206989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4640ffd90 0x7ff46410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 -- 172.21.15.3:0/630206989 >> 172.21.15.3:0/630206989 conn(0x7ff4640766b0 msgr2=0x7ff464076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 -- 172.21.15.3:0/630206989 shutdown_connections 2023-09-10T01:15:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.315+0000 7ff46afff700 1 -- 172.21.15.3:0/630206989 wait complete. 2023-09-10T01:15:54.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.316+0000 7ff46afff700 1 Processor -- start 2023-09-10T01:15:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.316+0000 7ff46afff700 1 -- start start 2023-09-10T01:15:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff46afff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4640ff470 0x7ff46419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff46afff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4640ffd90 0x7ff46419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff46afff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff46410d020 0x7ff4641988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff46afff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff464198f20 con 0x7ff4640ff470 2023-09-10T01:15:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff46afff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff464199090 con 0x7ff46410d020 2023-09-10T01:15:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff46afff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff464199250 con 0x7ff4640ffd90 2023-09-10T01:15:54.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4640ffd90 0x7ff46419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4640ffd90 0x7ff46419ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44250/0 (socket says 172.21.15.3:44250) 2023-09-10T01:15:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 -- 172.21.15.3:0/147381789 learned_addr learned my addr 172.21.15.3:0/147381789 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff468d9b700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4640ff470 0x7ff46419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff46959c700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff46410d020 0x7ff4641988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 -- 172.21.15.3:0/147381789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff46410d020 msgr2=0x7ff4641988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff46410d020 0x7ff4641988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 -- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4640ff470 msgr2=0x7ff46419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4640ff470 0x7ff46419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.317+0000 7ff463fff700 1 -- 172.21.15.3:0/147381789 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450009520 con 0x7ff4640ffd90 2023-09-10T01:15:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.318+0000 7ff463fff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4640ffd90 0x7ff46419ed80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff45000eb30 tx=0x7ff45000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.318+0000 7ff461ffb700 1 -- 172.21.15.3:0/147381789 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450009880 con 0x7ff4640ffd90 2023-09-10T01:15:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.318+0000 7ff461ffb700 1 -- 172.21.15.3:0/147381789 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4500044e0 con 0x7ff4640ffd90 2023-09-10T01:15:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.318+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4580095f0 con 0x7ff4640ffd90 2023-09-10T01:15:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.318+0000 7ff461ffb700 1 -- 172.21.15.3:0/147381789 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450003d80 con 0x7ff4640ffd90 2023-09-10T01:15:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.318+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff464199720 con 0x7ff4640ffd90 2023-09-10T01:15:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.319+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4640678d0 con 0x7ff4640ffd90 2023-09-10T01:15:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.321+0000 7ff461ffb700 1 -- 172.21.15.3:0/147381789 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ff450003f20 con 0x7ff4640ffd90 2023-09-10T01:15:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.323+0000 7ff461ffb700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff4540a0830 0x7ff4540a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.323+0000 7ff461ffb700 1 -- 172.21.15.3:0/147381789 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7ff450014070 con 0x7ff4640ffd90 2023-09-10T01:15:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.323+0000 7ff468d9b700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff4540a0830 0x7ff4540a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.324+0000 7ff468d9b700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff4540a0830 0x7ff4540a2cf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff45800b2e0 tx=0x7ff458005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.325+0000 7ff461ffb700 1 -- 172.21.15.3:0/147381789 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7ff450095e40 con 0x7ff4640ffd90 2023-09-10T01:15:54.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:54 smithi003 ceph-mon[97654]: pgmap v395: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:15:54.608 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.608+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff464199720 con 0x7ff4640ffd90 2023-09-10T01:15:54.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.608+0000 7ff461ffb700 1 -- 172.21.15.3:0/147381789 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+11296 (secure 0 0 0) 0x7ff464199720 con 0x7ff4640ffd90 2023-09-10T01:15:54.609 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:15:54.610 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:15:45.283727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:15:54.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff4540a0830 msgr2=0x7ff4540a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff4540a0830 0x7ff4540a2cf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff45800b2e0 tx=0x7ff458005fb0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4640ffd90 msgr2=0x7ff46419ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4640ffd90 0x7ff46419ed80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff45000eb30 tx=0x7ff45000ee40 comp rx=0 tx=0).stop 2023-09-10T01:15:54.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 shutdown_connections 2023-09-10T01:15:54.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff4540a0830 0x7ff4540a2cf0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff46410d020 0x7ff4641988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff4640ff470 0x7ff46419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 --2- 172.21.15.3:0/147381789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4640ffd90 0x7ff46419ed80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 >> 172.21.15.3:0/147381789 conn(0x7ff4640766b0 msgr2=0x7ff46410e520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 shutdown_connections 2023-09-10T01:15:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.611+0000 7ff46afff700 1 -- 172.21.15.3:0/147381789 wait complete. 2023-09-10T01:15:54.615 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 36 2023-09-10T01:15:54.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:15:54.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:54 smithi136 ceph-mon[133708]: pgmap v395: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:15:54.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:54 smithi092 ceph-mon[134067]: pgmap v395: 129 pgs: 129 active+clean; 320 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2023-09-10T01:15:54.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1019234 7.53 MB/sec execute 576 sec latency 51.275 ms 2023-09-10T01:15:54.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.895+0000 7f744adff700 1 -- 172.21.15.3:0/4248085939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440729a0 msgr2=0x7f7444072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.895+0000 7f744adff700 1 --2- 172.21.15.3:0/4248085939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440729a0 0x7f7444072d80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7438009990 tx=0x7f7438009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.895+0000 7f744adff700 1 -- 172.21.15.3:0/4248085939 shutdown_connections 2023-09-10T01:15:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.895+0000 7f744adff700 1 --2- 172.21.15.3:0/4248085939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440729a0 0x7f7444072d80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.895+0000 7f744adff700 1 --2- 172.21.15.3:0/4248085939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7444111650 0x7f7444113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.895+0000 7f744adff700 1 --2- 172.21.15.3:0/4248085939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74440732c0 0x7f7444111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.895+0000 7f744adff700 1 -- 172.21.15.3:0/4248085939 >> 172.21.15.3:0/4248085939 conn(0x7f74440fc170 msgr2=0x7f74440fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.896+0000 7f744adff700 1 -- 172.21.15.3:0/4248085939 shutdown_connections 2023-09-10T01:15:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.896+0000 7f744adff700 1 -- 172.21.15.3:0/4248085939 wait complete. 2023-09-10T01:15:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.897+0000 7f744adff700 1 Processor -- start 2023-09-10T01:15:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.897+0000 7f744adff700 1 -- start start 2023-09-10T01:15:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744adff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74440729a0 0x7f744419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744adff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440732c0 0x7f744419d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744adff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7444111650 0x7f74441a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744adff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74441a1d60 con 0x7f7444111650 2023-09-10T01:15:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744adff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74441a1ed0 con 0x7f74440729a0 2023-09-10T01:15:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744adff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74441a2090 con 0x7f74440732c0 2023-09-10T01:15:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74440729a0 0x7f744419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74440729a0 0x7f744419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55296/0 (socket says 172.21.15.3:55296) 2023-09-10T01:15:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 -- 172.21.15.3:0/2188290683 learned_addr learned my addr 172.21.15.3:0/2188290683 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744939c700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7444111650 0x7f74441a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7443fff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440732c0 0x7f744419d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 -- 172.21.15.3:0/2188290683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440732c0 msgr2=0x7f744419d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440732c0 0x7f744419d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 -- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7444111650 msgr2=0x7f74441a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7444111650 0x7f74441a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7448b9b700 1 -- 172.21.15.3:0/2188290683 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74380095f0 con 0x7f74440729a0 2023-09-10T01:15:54.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f744939c700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7444111650 0x7f74441a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:54.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.898+0000 7f7443fff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440732c0 0x7f744419d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:54.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.899+0000 7f7448b9b700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74440729a0 0x7f744419cd80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7438005910 tx=0x7f74380047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.899+0000 7f7441ffb700 1 -- 172.21.15.3:0/2188290683 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743801d070 con 0x7f74440729a0 2023-09-10T01:15:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.899+0000 7f7441ffb700 1 -- 172.21.15.3:0/2188290683 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7438022470 con 0x7f74440729a0 2023-09-10T01:15:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.899+0000 7f7441ffb700 1 -- 172.21.15.3:0/2188290683 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438017650 con 0x7f74440729a0 2023-09-10T01:15:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.899+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74441a2310 con 0x7f74440729a0 2023-09-10T01:15:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.899+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74441a2590 con 0x7f74440729a0 2023-09-10T01:15:54.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.903+0000 7f7441ffb700 1 -- 172.21.15.3:0/2188290683 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f74380225e0 con 0x7f74440729a0 2023-09-10T01:15:54.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.904+0000 7f745e6e5b80 1 Processor -- start 2023-09-10T01:15:54.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f745e6e5b80 1 -- start start 2023-09-10T01:15:54.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52ce44af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52ce4e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ce51230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52cd9d730 con 0x55b52ce44710 2023-09-10T01:15:54.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52cd9b050 con 0x55b52ce45030 2023-09-10T01:15:54.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52cd96940 con 0x55b52ce4ee40 2023-09-10T01:15:54.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.905+0000 7f7441ffb700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74340a0830 0x7f74340a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52ce4e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f744939c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52ce44af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7443fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ce51230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52ce4e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55304/0 (socket says 172.21.15.3:55304) 2023-09-10T01:15:54.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 -- 172.21.15.3:0/397470651 learned_addr learned my addr 172.21.15.3:0/397470651 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f744939c700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74340a0830 0x7f74340a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7441ffb700 1 -- 172.21.15.3:0/2188290683 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f74380cd520 con 0x7f74440729a0 2023-09-10T01:15:54.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 -- 172.21.15.3:0/397470651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 msgr2=0x55b52ce51230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 --2- 172.21.15.3:0/397470651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ce51230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 -- 172.21.15.3:0/397470651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 msgr2=0x55b52ce44af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 --2- 172.21.15.3:0/397470651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52ce44af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 -- 172.21.15.3:0/397470651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b52cbc2de0 con 0x55b52ce45030 2023-09-10T01:15:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.906+0000 7f7448b9b700 1 --2- 172.21.15.3:0/397470651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52ce4e900 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f74380ca880 tx=0x7f7438000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a8815388e32cb2b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.907+0000 7f744939c700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74340a0830 0x7f74340a2cf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f743c009fd0 tx=0x7f743c00b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.907+0000 7f742effd700 1 -- 172.21.15.3:0/397470651 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438099f70 con 0x55b52ce45030 2023-09-10T01:15:54.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.907+0000 7f742effd700 1 -- 172.21.15.3:0/397470651 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743809ad10 con 0x55b52ce45030 2023-09-10T01:15:54.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.907+0000 7f742effd700 1 -- 172.21.15.3:0/397470651 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743809aad0 con 0x55b52ce45030 2023-09-10T01:15:54.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.907+0000 7f745e6e5b80 1 -- 172.21.15.3:0/397470651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 msgr2=0x55b52ce4e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.907+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/397470651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52ce4e900 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f74380ca880 tx=0x7f7438000bc0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.908+0000 7f745e6e5b80 1 -- 172.21.15.3:0/397470651 shutdown_connections 2023-09-10T01:15:54.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.908+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/397470651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ce51230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.908+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/397470651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52ce44af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.908+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/397470651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52ce4e900 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.908+0000 7f745e6e5b80 1 -- 172.21.15.3:0/397470651 >> 172.21.15.3:0/397470651 conn(0x55b52cdb9090 msgr2=0x55b52ce3fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.908+0000 7f745e6e5b80 1 -- 172.21.15.3:0/397470651 shutdown_connections 2023-09-10T01:15:54.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.908+0000 7f745e6e5b80 1 -- 172.21.15.3:0/397470651 wait complete. 2023-09-10T01:15:54.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.909+0000 7f745e6e5b80 1 -- start start 2023-09-10T01:15:54.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.909+0000 7f745e6e5b80 1 Processor -- start 2023-09-10T01:15:54.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce4f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce5a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52cd34b70 con 0x55b52ce45030 2023-09-10T01:15:54.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52cd34870 con 0x55b52ce4ee40 2023-09-10T01:15:54.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52cc18920 con 0x55b52ce44710 2023-09-10T01:15:54.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f745e6e5b80 10 client.0 _ll_register_callbacks cb 0x55b52cc7ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:54.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f744939c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce5a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7443fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce4f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f744939c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce5a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55310/0 (socket says 172.21.15.3:55310) 2023-09-10T01:15:54.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce4f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38092/0 (socket says 172.21.15.3:38092) 2023-09-10T01:15:54.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f744939c700 1 -- 172.21.15.3:0/1194686566 learned_addr learned my addr 172.21.15.3:0/1194686566 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7448b9b700 1 -- 172.21.15.3:0/1194686566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 msgr2=0x55b52ce5a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7448b9b700 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce5a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7448b9b700 1 -- 172.21.15.3:0/1194686566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 msgr2=0x55b52ce4f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7448b9b700 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f7448b9b700 1 -- 172.21.15.3:0/1194686566 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74441a2310 con 0x55b52ce44710 2023-09-10T01:15:54.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.910+0000 7f744939c700 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce5a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:54.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f7448b9b700 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce4f220 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x55b52ce3a280 tx=0x7f7438022b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90efbed313f58234 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f7423fff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:15:54.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f7423fff700 1 -- 172.21.15.3:0/1194686566 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380a25f0 con 0x55b52ce44710 2023-09-10T01:15:54.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f7423fff700 1 -- 172.21.15.3:0/1194686566 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f743809e440 con 0x55b52ce44710 2023-09-10T01:15:54.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b52ccee4f0 con 0x55b52ce44710 2023-09-10T01:15:54.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f7423fff700 1 -- 172.21.15.3:0/1194686566 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74380adea0 con 0x55b52ce44710 2023-09-10T01:15:54.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f745e6e5b80 10 client.28393 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:54.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55b52cbabc60 con 0x55b52ce44710 2023-09-10T01:15:54.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.911+0000 7f7423fff700 1 -- 172.21.15.3:0/1194686566 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b52ccee4f0 con 0x55b52ce44710 2023-09-10T01:15:54.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f7423fff700 1 -- 172.21.15.3:0/1194686566 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f74380a2160 con 0x55b52ce44710 2023-09-10T01:15:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f745e6e5b80 10 client.28393 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f745e6e5b80 10 client.28393 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:15:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f745e6e5b80 20 client.28393 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ce5e0a0 0x55b52ce604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f745e6e5b80 4 client.28393 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b52cdab8a0 con 0x55b52ce5e0a0 2023-09-10T01:15:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.912+0000 7f7443fff700 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ce5e0a0 0x55b52ce604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.913+0000 7f7443fff700 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ce5e0a0 0x55b52ce604c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7c10603ebdfdd6b7 server_cookie=31084479e76eeedc in_seq=0 out_seq=0 2023-09-10T01:15:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.913+0000 7f7423fff700 10 client.28393 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f7423fff700 1 -- 172.21.15.3:0/1194686566 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b52cdab8a0 con 0x55b52ce5e0a0 2023-09-10T01:15:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f7423fff700 10 client.28393 handle_command_reply: tid=0 2023-09-10T01:15:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 client.28393 shutdown 2023-09-10T01:15:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 msgr2=0x55b52ce4f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce4f220 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x55b52ce3a280 tx=0x7f7438022b30 comp rx=0 tx=0).stop 2023-09-10T01:15:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 shutdown_connections 2023-09-10T01:15:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ce5e0a0 0x55b52ce604c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce4f220 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce5a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/1194686566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 >> 172.21.15.3:0/1194686566 conn(0x55b52cdb9090 msgr2=0x55b52ce3f780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f7423fff700 0 client.28393 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:54.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 shutdown_connections 2023-09-10T01:15:54.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 1 -- 172.21.15.3:0/1194686566 wait complete. 2023-09-10T01:15:54.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.914+0000 7f745e6e5b80 20 client.28393 trim_cache size 0 max 16384 2023-09-10T01:15:54.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.928+0000 7f745e6e5b80 1 Processor -- start 2023-09-10T01:15:54.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f745e6e5b80 1 -- start start 2023-09-10T01:15:54.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce54240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.933 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce4f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52ce45a30 con 0x55b52ce45030 2023-09-10T01:15:54.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52ce46640 con 0x55b52ce4ee40 2023-09-10T01:15:54.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52ce41000 con 0x55b52ce44710 2023-09-10T01:15:54.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce54240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.934 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f744939c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce4f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f744939c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce4f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55312/0 (socket says 172.21.15.3:55312) 2023-09-10T01:15:54.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce54240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38106/0 (socket says 172.21.15.3:38106) 2023-09-10T01:15:54.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f744939c700 1 -- 172.21.15.3:0/243303856 learned_addr learned my addr 172.21.15.3:0/243303856 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.929+0000 7f7443fff700 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.935 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f744939c700 1 -- 172.21.15.3:0/243303856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 msgr2=0x55b52ce54240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f744939c700 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce54240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f744939c700 1 -- 172.21.15.3:0/243303856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 msgr2=0x55b52ce4f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f744939c700 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f744939c700 1 -- 172.21.15.3:0/243303856 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b52ce650d0 con 0x55b52ce4ee40 2023-09-10T01:15:54.936 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f7443fff700 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f7448b9b700 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce54240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f744939c700 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce4f760 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f743c012380 tx=0x7f743c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31e39e33e5eeb2f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f742f7fe700 1 -- 172.21.15.3:0/243303856 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c00e040 con 0x55b52ce4ee40 2023-09-10T01:15:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f742f7fe700 1 -- 172.21.15.3:0/243303856 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743c022040 con 0x55b52ce4ee40 2023-09-10T01:15:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.930+0000 7f742f7fe700 1 -- 172.21.15.3:0/243303856 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c01aa60 con 0x55b52ce4ee40 2023-09-10T01:15:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 -- 172.21.15.3:0/243303856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 msgr2=0x55b52ce4f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce4f760 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f743c012380 tx=0x7f743c009920 comp rx=0 tx=0).stop 2023-09-10T01:15:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 -- 172.21.15.3:0/243303856 shutdown_connections 2023-09-10T01:15:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce44710 0x55b52ce54240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce4ee40 0x55b52ce4f760 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/243303856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce45030 0x55b52ce4f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 -- 172.21.15.3:0/243303856 >> 172.21.15.3:0/243303856 conn(0x55b52cdb9090 msgr2=0x55b52ce48840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 -- 172.21.15.3:0/243303856 shutdown_connections 2023-09-10T01:15:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.931+0000 7f745e6e5b80 1 -- 172.21.15.3:0/243303856 wait complete. 2023-09-10T01:15:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.932+0000 7f745e6e5b80 1 -- start start 2023-09-10T01:15:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.932+0000 7f745e6e5b80 1 Processor -- start 2023-09-10T01:15:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.932+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52cea6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.932+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52cea6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f745e6e5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ceaae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52ceab3d0 con 0x55b52ce44710 2023-09-10T01:15:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52ceab510 con 0x55b52ce45030 2023-09-10T01:15:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f745e6e5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b52ceab6d0 con 0x55b52ce4ee40 2023-09-10T01:15:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f745e6e5b80 10 client.0 _ll_register_callbacks cb 0x55b52cc7ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52cea6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7448b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52cea6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52cea6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55328/0 (socket says 172.21.15.3:55328) 2023-09-10T01:15:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 -- 172.21.15.3:0/34885461 learned_addr learned my addr 172.21.15.3:0/34885461 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f744939c700 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ceaae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 -- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 msgr2=0x55b52ceaae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ceaae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 -- 172.21.15.3:0/34885461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 msgr2=0x55b52cea6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52cea6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 -- 172.21.15.3:0/34885461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b52cbabc60 con 0x55b52ce45030 2023-09-10T01:15:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7443fff700 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52cea6570 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7430004790 tx=0x7f743001e9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2576340bd0cd978c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.933+0000 7f7423fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:54.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f7423fff700 1 -- 172.21.15.3:0/34885461 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430017040 con 0x55b52ce45030 2023-09-10T01:15:54.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f7423fff700 1 -- 172.21.15.3:0/34885461 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7430004240 con 0x55b52ce45030 2023-09-10T01:15:54.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f743809ad10 con 0x55b52ce45030 2023-09-10T01:15:54.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f7423fff700 1 -- 172.21.15.3:0/34885461 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7430004400 con 0x55b52ce45030 2023-09-10T01:15:54.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f745e6e5b80 10 client.28517 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:54.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7f743c012b50 con 0x55b52ce45030 2023-09-10T01:15:54.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f7423fff700 1 -- 172.21.15.3:0/34885461 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743809ad10 con 0x55b52ce45030 2023-09-10T01:15:54.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f7423fff700 1 -- 172.21.15.3:0/34885461 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7430027620 con 0x55b52ce45030 2023-09-10T01:15:54.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f745e6e5b80 10 client.28517 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:54.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.934+0000 7f745e6e5b80 10 client.28517 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:15:54.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.935+0000 7f745e6e5b80 20 client.28517 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:54.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.935+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ceb2f30 0x55b52ceb5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:54.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.935+0000 7f745e6e5b80 4 client.28517 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:15:54.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.935+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b52cd8f920 con 0x55b52ceb2f30 2023-09-10T01:15:54.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.935+0000 7f7448b9b700 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ceb2f30 0x55b52ceb5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:54.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.935+0000 7f7423fff700 10 client.28517 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:54.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.935+0000 7f7448b9b700 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ceb2f30 0x55b52ceb5350 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89d844800393a5f0 server_cookie=70d1234a2184cd5b in_seq=0 out_seq=0 2023-09-10T01:15:54.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f7423fff700 1 -- 172.21.15.3:0/34885461 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b52cd8f920 con 0x55b52ceb2f30 2023-09-10T01:15:54.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f7423fff700 10 client.28517 handle_command_reply: tid=0 2023-09-10T01:15:54.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 client.28517 shutdown 2023-09-10T01:15:54.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 msgr2=0x55b52cea6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52cea6570 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7430004790 tx=0x7f743001e9f0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 shutdown_connections 2023-09-10T01:15:54.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55b52ceb2f30 0x55b52ceb5350 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b52ce4ee40 0x55b52ceaae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b52ce44710 0x55b52cea6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 --2- 172.21.15.3:0/34885461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b52ce45030 0x55b52cea6570 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f7423fff700 0 client.28517 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:15:54.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 >> 172.21.15.3:0/34885461 conn(0x55b52cdb9090 msgr2=0x55b52ce49310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 shutdown_connections 2023-09-10T01:15:54.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 1 -- 172.21.15.3:0/34885461 wait complete. 2023-09-10T01:15:54.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.936+0000 7f745e6e5b80 20 client.28517 trim_cache size 0 max 16384 2023-09-10T01:15:54.953 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:15:54.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74340a0830 msgr2=0x7f74340a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74340a0830 0x7f74340a2cf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f743c009fd0 tx=0x7f743c00b040 comp rx=0 tx=0).stop 2023-09-10T01:15:54.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74440729a0 msgr2=0x7f744419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:54.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74440729a0 0x7f744419cd80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7438005910 tx=0x7f74380047c0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 shutdown_connections 2023-09-10T01:15:54.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74340a0830 0x7f74340a2cf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7444111650 0x7f74441a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74440729a0 0x7f744419cd80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 --2- 172.21.15.3:0/2188290683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74440732c0 0x7f744419d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 >> 172.21.15.3:0/2188290683 conn(0x7f74440fc170 msgr2=0x7f74440fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 shutdown_connections 2023-09-10T01:15:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:54.937+0000 7f744adff700 1 -- 172.21.15.3:0/2188290683 wait complete. 2023-09-10T01:15:54.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:15:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.186+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2250711576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed010ce80 msgr2=0x7f7ed010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.186+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2250711576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed010ce80 0x7f7ed010f2f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800b0a0 tx=0x7f7ec800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.187+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2250711576 shutdown_connections 2023-09-10T01:15:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.187+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2250711576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed0072a30 0x7f7ed0072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.187+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2250711576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed010ce80 0x7f7ed010f2f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.187+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2250711576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ed0073350 0x7f7ed010c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.187+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2250711576 >> 172.21.15.3:0/2250711576 conn(0x7f7ed00fc170 msgr2=0x7f7ed00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.187+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2250711576 shutdown_connections 2023-09-10T01:15:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.187+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2250711576 wait complete. 2023-09-10T01:15:55.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 Processor -- start 2023-09-10T01:15:55.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 -- start start 2023-09-10T01:15:55.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0072a30 0x7f7ed019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed0073350 0x7f7ed019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ed010ce80 0x7f7ed0198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0198ed0 con 0x7f7ed010ce80 2023-09-10T01:15:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0199040 con 0x7f7ed0072a30 2023-09-10T01:15:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.188+0000 7f7ed6fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0199200 con 0x7f7ed0073350 2023-09-10T01:15:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed559c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ed010ce80 0x7f7ed0198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0072a30 0x7f7ed019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0072a30 0x7f7ed019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55346/0 (socket says 172.21.15.3:55346) 2023-09-10T01:15:55.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed0073350 0x7f7ed019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 -- 172.21.15.3:0/2710841119 learned_addr learned my addr 172.21.15.3:0/2710841119 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 -- 172.21.15.3:0/2710841119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed0073350 msgr2=0x7f7ed019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed0073350 0x7f7ed019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 -- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ed010ce80 msgr2=0x7f7ed0198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ed010ce80 0x7f7ed0198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 -- 172.21.15.3:0/2710841119 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8009d00 con 0x7f7ed0072a30 2023-09-10T01:15:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed559c700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ed010ce80 0x7f7ed0198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:55.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.189+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0072a30 0x7f7ed019e7f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400e9a0 tx=0x7f7ec400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.190+0000 7f7ecdffb700 1 -- 172.21.15.3:0/2710841119 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4009790 con 0x7f7ed0072a30 2023-09-10T01:15:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.190+0000 7f7ecdffb700 1 -- 172.21.15.3:0/2710841119 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec4010430 con 0x7f7ed0072a30 2023-09-10T01:15:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.190+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed01994e0 con 0x7f7ed0072a30 2023-09-10T01:15:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.190+0000 7f7ecdffb700 1 -- 172.21.15.3:0/2710841119 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4018680 con 0x7f7ed0072a30 2023-09-10T01:15:55.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.190+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed01a57e0 con 0x7f7ed0072a30 2023-09-10T01:15:55.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.193+0000 7f7ecdffb700 1 -- 172.21.15.3:0/2710841119 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7ec40105a0 con 0x7f7ed0072a30 2023-09-10T01:15:55.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.195+0000 7f7ecdffb700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7ec00a07b0 0x7f7ec00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.195+0000 7f7eea97eb80 1 Processor -- start 2023-09-10T01:15:55.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.195+0000 7f7ecdffb700 1 -- 172.21.15.3:0/2710841119 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7ec4014070 con 0x7f7ed0072a30 2023-09-10T01:15:55.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.195+0000 7f7ecffff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7ec00a07b0 0x7f7ec00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.195+0000 7f7eea97eb80 1 -- start start 2023-09-10T01:15:55.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfd0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfda900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcf29730 con 0x55c3bcfd0710 2023-09-10T01:15:55.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcf27050 con 0x55c3bcfdae40 2023-09-10T01:15:55.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcf22940 con 0x55c3bcfd1030 2023-09-10T01:15:55.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7ed4d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfd0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfda900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfda900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38144/0 (socket says 172.21.15.3:38144) 2023-09-10T01:15:55.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7ecffff700 1 -- 172.21.15.3:0/150531339 learned_addr learned my addr 172.21.15.3:0/150531339 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7ed559c700 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.196+0000 7f7ecffff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7ec00a07b0 0x7f7ec00a2c70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7ed010e340 tx=0x7f7ebc009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.197+0000 7f7ecffff700 1 -- 172.21.15.3:0/150531339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 msgr2=0x55c3bcfdd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.197+0000 7f7ecffff700 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.197+0000 7f7ecffff700 1 -- 172.21.15.3:0/150531339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 msgr2=0x55c3bcfd0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.197+0000 7f7ecffff700 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfd0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.197+0000 7f7ecffff700 1 -- 172.21.15.3:0/150531339 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3bcd4ede0 con 0x55c3bcfd1030 2023-09-10T01:15:55.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.197+0000 7f7ecffff700 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfda900 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc011ca0 tx=0x7f7ebc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad557e395a50a81c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.197+0000 7f7ebaffd700 1 -- 172.21.15.3:0/150531339 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc00e840 con 0x55c3bcfd1030 2023-09-10T01:15:55.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7ebaffd700 1 -- 172.21.15.3:0/150531339 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ebc021070 con 0x55c3bcfd1030 2023-09-10T01:15:55.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7ebaffd700 1 -- 172.21.15.3:0/150531339 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc005280 con 0x55c3bcfd1030 2023-09-10T01:15:55.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7eea97eb80 1 -- 172.21.15.3:0/150531339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 msgr2=0x55c3bcfda900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfda900 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc011ca0 tx=0x7f7ebc014000 comp rx=0 tx=0).stop 2023-09-10T01:15:55.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7eea97eb80 1 -- 172.21.15.3:0/150531339 shutdown_connections 2023-09-10T01:15:55.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfd0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/150531339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfda900 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.198+0000 7f7eea97eb80 1 -- 172.21.15.3:0/150531339 >> 172.21.15.3:0/150531339 conn(0x55c3bcf45090 msgr2=0x55c3bcfcbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.199+0000 7f7eea97eb80 1 -- 172.21.15.3:0/150531339 shutdown_connections 2023-09-10T01:15:55.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.199+0000 7f7eea97eb80 1 -- 172.21.15.3:0/150531339 wait complete. 2023-09-10T01:15:55.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.199+0000 7f7eea97eb80 1 -- start start 2023-09-10T01:15:55.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.199+0000 7f7eea97eb80 1 Processor -- start 2023-09-10T01:15:55.216 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfda8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcec0b70 con 0x55c3bcfd0710 2023-09-10T01:15:55.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcec0870 con 0x55c3bcfdae40 2023-09-10T01:15:55.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcda4920 con 0x55c3bcfd1030 2023-09-10T01:15:55.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7eea97eb80 10 client.0 _ll_register_callbacks cb 0x55c3bce0aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:55.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38154/0 (socket says 172.21.15.3:38154) 2023-09-10T01:15:55.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7ecffff700 1 -- 172.21.15.3:0/2946160390 learned_addr learned my addr 172.21.15.3:0/2946160390 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfda8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.200+0000 7f7ed559c700 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7ecffff700 1 -- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 msgr2=0x55c3bcfd8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7ecffff700 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7ecffff700 1 -- 172.21.15.3:0/2946160390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 msgr2=0x55c3bcfda8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7ecffff700 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfda8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7ecffff700 1 -- 172.21.15.3:0/2946160390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc017070 con 0x55c3bcfd1030 2023-09-10T01:15:55.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7ecffff700 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd7bc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009fd0 tx=0x7f7ebc0047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f283ce658f1c2f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7eaffff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:15:55.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7eaffff700 1 -- 172.21.15.3:0/2946160390 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc00ea40 con 0x55c3bcfd1030 2023-09-10T01:15:55.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3bce7a4f0 con 0x55c3bcfd1030 2023-09-10T01:15:55.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7eaffff700 1 -- 172.21.15.3:0/2946160390 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7ebc00eba0 con 0x55c3bcfd1030 2023-09-10T01:15:55.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7eaffff700 1 -- 172.21.15.3:0/2946160390 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ebc00ed60 con 0x55c3bcfd1030 2023-09-10T01:15:55.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7eea97eb80 10 client.28411 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:55.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.201+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55c3bcd37c60 con 0x55c3bcfd1030 2023-09-10T01:15:55.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eaffff700 1 -- 172.21.15.3:0/2946160390 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c3bce7a4f0 con 0x55c3bcfd1030 2023-09-10T01:15:55.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eaffff700 1 -- 172.21.15.3:0/2946160390 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7ebc0266c0 con 0x55c3bcfd1030 2023-09-10T01:15:55.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eea97eb80 10 client.28411 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:55.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eea97eb80 10 client.28411 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:15:55.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eea97eb80 20 client.28411 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:55.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bcfea960 0x55c3bcfecd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eea97eb80 4 client.28411 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:55.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3bcf378a0 con 0x55c3bcfea960 2023-09-10T01:15:55.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.202+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bcfea960 0x55c3bcfecd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.223 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.203+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bcfea960 0x55c3bcfecd80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a3c94d4d5adca33 server_cookie=16068005b1646bfb in_seq=0 out_seq=0 2023-09-10T01:15:55.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.203+0000 7f7eaffff700 10 client.28411 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:55.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.203+0000 7f7eaffff700 1 -- 172.21.15.3:0/2946160390 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c3bcf378a0 con 0x55c3bcfea960 2023-09-10T01:15:55.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.203+0000 7f7eaffff700 10 client.28411 handle_command_reply: tid=0 2023-09-10T01:15:55.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.203+0000 7f7eea97eb80 1 client.28411 shutdown 2023-09-10T01:15:55.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 msgr2=0x55c3bcfd7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd7bc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009fd0 tx=0x7f7ebc0047e0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 shutdown_connections 2023-09-10T01:15:55.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfda8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bcfea960 0x55c3bcfecd80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/2946160390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd7bc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eaffff700 0 client.28411 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:55.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 >> 172.21.15.3:0/2946160390 conn(0x55c3bcf45090 msgr2=0x55c3bcfcb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 shutdown_connections 2023-09-10T01:15:55.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 1 -- 172.21.15.3:0/2946160390 wait complete. 2023-09-10T01:15:55.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.204+0000 7f7eea97eb80 20 client.28411 trim_cache size 0 max 16384 2023-09-10T01:15:55.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.218+0000 7f7eea97eb80 1 Processor -- start 2023-09-10T01:15:55.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7eea97eb80 1 -- start start 2023-09-10T01:15:55.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcfdd750 con 0x55c3bcfdae40 2023-09-10T01:15:55.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcfd89c0 con 0x55c3bcfd1030 2023-09-10T01:15:55.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcfcd210 con 0x55c3bcfd0710 2023-09-10T01:15:55.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7ed4d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55374/0 (socket says 172.21.15.3:55374) 2023-09-10T01:15:55.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7ecffff700 1 -- 172.21.15.3:0/409230532 learned_addr learned my addr 172.21.15.3:0/409230532 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.219+0000 7f7ed559c700 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ecffff700 1 -- 172.21.15.3:0/409230532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd0710 msgr2=0x55c3bcfe6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ecffff700 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ecffff700 1 -- 172.21.15.3:0/409230532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfdae40 msgr2=0x55c3bcfd33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ecffff700 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ecffff700 1 -- 172.21.15.3:0/409230532 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3bcd75250 con 0x55c3bcfd1030 2023-09-10T01:15:55.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ecffff700 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd2ea0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc011ca0 tx=0x7f7ebc026f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e668249ae4d962a2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ebb7fe700 1 -- 172.21.15.3:0/409230532 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc026680 con 0x55c3bcfd1030 2023-09-10T01:15:55.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ebb7fe700 1 -- 172.21.15.3:0/409230532 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ebc004850 con 0x55c3bcfd1030 2023-09-10T01:15:55.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.220+0000 7f7ebb7fe700 1 -- 172.21.15.3:0/409230532 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0316c0 con 0x55c3bcfd1030 2023-09-10T01:15:55.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 -- 172.21.15.3:0/409230532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfd1030 msgr2=0x55c3bcfd2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd2ea0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc011ca0 tx=0x7f7ebc026f10 comp rx=0 tx=0).stop 2023-09-10T01:15:55.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 -- 172.21.15.3:0/409230532 shutdown_connections 2023-09-10T01:15:55.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfdae40 0x55c3bcfd33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/409230532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfd1030 0x55c3bcfd2ea0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 -- 172.21.15.3:0/409230532 >> 172.21.15.3:0/409230532 conn(0x55c3bcf45090 msgr2=0x55c3bcfca320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 -- 172.21.15.3:0/409230532 shutdown_connections 2023-09-10T01:15:55.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.221+0000 7f7eea97eb80 1 -- 172.21.15.3:0/409230532 wait complete. 2023-09-10T01:15:55.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.222+0000 7f7eea97eb80 1 -- start start 2023-09-10T01:15:55.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.222+0000 7f7eea97eb80 1 Processor -- start 2023-09-10T01:15:55.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.222+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.222+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfe1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7eea97eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcfdfed0 con 0x55c3bcfd0710 2023-09-10T01:15:55.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcfe0040 con 0x55c3bcfdae40 2023-09-10T01:15:55.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7eea97eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3bcfe0200 con 0x55c3bcfd1030 2023-09-10T01:15:55.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7eea97eb80 10 client.0 _ll_register_callbacks cb 0x7f7ebc005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:55.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfe1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ed4d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfe1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38176/0 (socket says 172.21.15.3:38176) 2023-09-10T01:15:55.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ecffff700 1 -- 172.21.15.3:0/3162586742 learned_addr learned my addr 172.21.15.3:0/3162586742 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ed559c700 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ed559c700 1 -- 172.21.15.3:0/3162586742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 msgr2=0x55c3bcfe1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ed559c700 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfe1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ed559c700 1 -- 172.21.15.3:0/3162586742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 msgr2=0x55c3bcfe1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ed559c700 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.223+0000 7f7ed559c700 1 -- 172.21.15.3:0/3162586742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3bcd37c60 con 0x55c3bcfdae40 2023-09-10T01:15:55.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7ed559c700 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdf7c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80062a0 tx=0x7f7ec8003ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcedad603c7054f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eaffff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:55.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eaffff700 1 -- 172.21.15.3:0/3162586742 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8022070 con 0x55c3bcfdae40 2023-09-10T01:15:55.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3bcf1b920 con 0x55c3bcfdae40 2023-09-10T01:15:55.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eaffff700 1 -- 172.21.15.3:0/3162586742 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7ec80076f0 con 0x55c3bcfdae40 2023-09-10T01:15:55.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eaffff700 1 -- 172.21.15.3:0/3162586742 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ec800e040 con 0x55c3bcfdae40 2023-09-10T01:15:55.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eea97eb80 10 client.28535 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:55.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55c3bcd4d2b0 con 0x55c3bcfdae40 2023-09-10T01:15:55.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eaffff700 1 -- 172.21.15.3:0/3162586742 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec8030080 con 0x55c3bcfdae40 2023-09-10T01:15:55.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.224+0000 7f7eaffff700 1 -- 172.21.15.3:0/3162586742 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7ec800b0d0 con 0x55c3bcfdae40 2023-09-10T01:15:55.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eea97eb80 10 client.28535 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:55.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eea97eb80 10 client.28535 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:15:55.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eea97eb80 20 client.28535 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:55.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bd03f9d0 0x55c3bd041df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eea97eb80 4 client.28535 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:15:55.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c3bcedc0b0 con 0x55c3bd03f9d0 2023-09-10T01:15:55.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bd03f9d0 0x55c3bd041df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7ed4d9b700 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bd03f9d0 0x55c3bd041df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1dc5f50be79e59a9 server_cookie=8ed0de8c2a445681 in_seq=0 out_seq=0 2023-09-10T01:15:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eaffff700 10 client.28535 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eaffff700 1 -- 172.21.15.3:0/3162586742 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c3bcedc0b0 con 0x55c3bd03f9d0 2023-09-10T01:15:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eaffff700 10 client.28535 handle_command_reply: tid=0 2023-09-10T01:15:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.225+0000 7f7eea97eb80 1 client.28535 shutdown 2023-09-10T01:15:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 msgr2=0x55c3bcfdf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdf7c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80062a0 tx=0x7f7ec8003ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 shutdown_connections 2023-09-10T01:15:55.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3bcfdae40 0x55c3bcfdf7c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c3bcfd0710 0x55c3bcfe1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c3bd03f9d0 0x55c3bd041df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 --2- 172.21.15.3:0/3162586742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c3bcfd1030 0x55c3bcfe1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eaffff700 0 client.28535 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:15:55.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 >> 172.21.15.3:0/3162586742 conn(0x55c3bcf45090 msgr2=0x55c3bcfe6140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 shutdown_connections 2023-09-10T01:15:55.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 1 -- 172.21.15.3:0/3162586742 wait complete. 2023-09-10T01:15:55.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.226+0000 7f7eea97eb80 20 client.28535 trim_cache size 0 max 16384 2023-09-10T01:15:55.252 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:15:55.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7ec00a07b0 msgr2=0x7f7ec00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7ec00a07b0 0x7f7ec00a2c70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7ed010e340 tx=0x7f7ebc009260 comp rx=0 tx=0).stop 2023-09-10T01:15:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0072a30 msgr2=0x7f7ed019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0072a30 0x7f7ed019e7f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400e9a0 tx=0x7f7ec400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 shutdown_connections 2023-09-10T01:15:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7ec00a07b0 0x7f7ec00a2c70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ed010ce80 0x7f7ed0198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0072a30 0x7f7ed019e7f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 --2- 172.21.15.3:0/2710841119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed0073350 0x7f7ed019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 >> 172.21.15.3:0/2710841119 conn(0x7f7ed00fc170 msgr2=0x7f7ed01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 shutdown_connections 2023-09-10T01:15:55.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.227+0000 7f7ed6fff700 1 -- 172.21.15.3:0/2710841119 wait complete. 2023-09-10T01:15:55.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:15:55.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/2969730566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24100640 msgr2=0x7f0d24100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/2969730566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24100640 0x7f0d24100ac0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009990 tx=0x7f0d14009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/2969730566 shutdown_connections 2023-09-10T01:15:55.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/2969730566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d24106650 0x7f0d24106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/2969730566 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 0x7f0d2410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/2969730566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24100640 0x7f0d24100ac0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/2969730566 >> 172.21.15.3:0/2969730566 conn(0x7f0d240fc170 msgr2=0x7f0d240fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/2969730566 shutdown_connections 2023-09-10T01:15:55.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.483+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/2969730566 wait complete. 2023-09-10T01:15:55.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.484+0000 7f0d2c5ff700 1 Processor -- start 2023-09-10T01:15:55.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.484+0000 7f0d2c5ff700 1 -- start start 2023-09-10T01:15:55.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2c5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d24100640 0x7f0d2419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2c5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 0x7f0d2419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2c5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24106650 0x7f0d24198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2c5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24198db0 con 0x7f0d24106650 2023-09-10T01:15:55.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2c5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24198f20 con 0x7f0d24101000 2023-09-10T01:15:55.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2c5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d241990e0 con 0x7f0d24100640 2023-09-10T01:15:55.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 0x7f0d2419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 0x7f0d2419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55404/0 (socket says 172.21.15.3:55404) 2023-09-10T01:15:55.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d29b9a700 1 -- 172.21.15.3:0/242860705 learned_addr learned my addr 172.21.15.3:0/242860705 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d24100640 0x7f0d2419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.485+0000 7f0d2ab9c700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24106650 0x7f0d24198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d29b9a700 1 -- 172.21.15.3:0/242860705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d24100640 msgr2=0x7f0d2419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d24100640 0x7f0d2419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d29b9a700 1 -- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24106650 msgr2=0x7f0d24198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24106650 0x7f0d24198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d29b9a700 1 -- 172.21.15.3:0/242860705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140095f0 con 0x7f0d24101000 2023-09-10T01:15:55.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d2ab9c700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24106650 0x7f0d24198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:55.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 0x7f0d2419ec10 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0d14005910 tx=0x7f0d14005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d1b7fe700 1 -- 172.21.15.3:0/242860705 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1401d070 con 0x7f0d24101000 2023-09-10T01:15:55.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d24199360 con 0x7f0d24101000 2023-09-10T01:15:55.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d1b7fe700 1 -- 172.21.15.3:0/242860705 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d14022470 con 0x7f0d24101000 2023-09-10T01:15:55.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d1b7fe700 1 -- 172.21.15.3:0/242860705 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14017650 con 0x7f0d24101000 2023-09-10T01:15:55.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.486+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d241a56c0 con 0x7f0d24101000 2023-09-10T01:15:55.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.490+0000 7f0d1b7fe700 1 -- 172.21.15.3:0/242860705 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f0d140225e0 con 0x7f0d24101000 2023-09-10T01:15:55.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.491+0000 7f0d3fec1b80 1 Processor -- start 2023-09-10T01:15:55.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.491+0000 7f0d1b7fe700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d100a07b0 0x7f0d100a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.491+0000 7f0d3fec1b80 1 -- start start 2023-09-10T01:15:55.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.491+0000 7f0d1b7fe700 1 -- 172.21.15.3:0/242860705 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0d140cd7b0 con 0x7f0d24101000 2023-09-10T01:15:55.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d100a07b0 0x7f0d100a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a710f730 con 0x5587a71c0e40 2023-09-10T01:15:55.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a710d050 con 0x5587a71b7030 2023-09-10T01:15:55.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a7108940 con 0x5587a71b6710 2023-09-10T01:15:55.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d2a39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55410/0 (socket says 172.21.15.3:55410) 2023-09-10T01:15:55.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d29b9a700 1 -- 172.21.15.3:0/3126836836 learned_addr learned my addr 172.21.15.3:0/3126836836 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.492+0000 7f0d2ab9c700 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d29b9a700 1 -- 172.21.15.3:0/3126836836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 msgr2=0x5587a71b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d29b9a700 1 -- 172.21.15.3:0/3126836836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 msgr2=0x5587a71c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d29b9a700 1 -- 172.21.15.3:0/3126836836 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587a6f34de0 con 0x5587a71b7030 2023-09-10T01:15:55.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d2ab9c700 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:55.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d100a07b0 0x7f0d100a2c70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0d20011550 tx=0x7f0d20011910 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c0900 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0d140a2470 tx=0x7f0d14009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc0dc89369a7a5e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d18bfe700 1 -- 172.21.15.3:0/3126836836 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1409b0c0 con 0x5587a71b7030 2023-09-10T01:15:55.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d18bfe700 1 -- 172.21.15.3:0/3126836836 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d1409a9c0 con 0x5587a71b7030 2023-09-10T01:15:55.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.493+0000 7f0d18bfe700 1 -- 172.21.15.3:0/3126836836 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1409ae80 con 0x5587a71b7030 2023-09-10T01:15:55.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/3126836836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 msgr2=0x5587a71c0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c0900 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0d140a2470 tx=0x7f0d14009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/3126836836 shutdown_connections 2023-09-10T01:15:55.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/3126836836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c0900 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/3126836836 >> 172.21.15.3:0/3126836836 conn(0x5587a712b090 msgr2=0x5587a71b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/3126836836 shutdown_connections 2023-09-10T01:15:55.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.494+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/3126836836 wait complete. 2023-09-10T01:15:55.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 -- start start 2023-09-10T01:15:55.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 Processor -- start 2023-09-10T01:15:55.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71c0e40 0x5587a71cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a70a6b70 con 0x5587a71b7030 2023-09-10T01:15:55.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a70a6870 con 0x5587a71c0e40 2023-09-10T01:15:55.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a6f8a920 con 0x5587a71b6710 2023-09-10T01:15:55.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.495+0000 7f0d3fec1b80 10 client.0 _ll_register_callbacks cb 0x5587a6ff0a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:55.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38216/0 (socket says 172.21.15.3:38216) 2023-09-10T01:15:55.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 -- 172.21.15.3:0/4068744893 learned_addr learned my addr 172.21.15.3:0/4068744893 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2ab9c700 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71c0e40 0x5587a71cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 -- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71c0e40 msgr2=0x5587a71cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71c0e40 0x5587a71cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 -- 172.21.15.3:0/4068744893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b7030 msgr2=0x5587a71c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 -- 172.21.15.3:0/4068744893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d241a56c0 con 0x5587a71b6710 2023-09-10T01:15:55.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c1220 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005c50 tx=0x7f0d200047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b58d9de1f6897c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.496+0000 7f0d0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:15:55.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d0dffb700 1 -- 172.21.15.3:0/4068744893 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20003c90 con 0x5587a71b6710 2023-09-10T01:15:55.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5587a70604f0 con 0x5587a71b6710 2023-09-10T01:15:55.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d0dffb700 1 -- 172.21.15.3:0/4068744893 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0d20003df0 con 0x5587a71b6710 2023-09-10T01:15:55.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d0dffb700 1 -- 172.21.15.3:0/4068744893 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d2002c600 con 0x5587a71b6710 2023-09-10T01:15:55.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d3fec1b80 10 client.28429 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:55.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5587a6f1dc60 con 0x5587a71b6710 2023-09-10T01:15:55.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d0dffb700 1 -- 172.21.15.3:0/4068744893 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5587a70604f0 con 0x5587a71b6710 2023-09-10T01:15:55.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d0dffb700 1 -- 172.21.15.3:0/4068744893 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0d20026020 con 0x5587a71b6710 2023-09-10T01:15:55.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.497+0000 7f0d3fec1b80 10 client.28429 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:55.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d3fec1b80 10 client.28429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:55.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d3fec1b80 20 client.28429 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:55.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a71d0590 0x5587a71d29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d3fec1b80 4 client.28429 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:55.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587a711d8a0 con 0x5587a71d0590 2023-09-10T01:15:55.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a71d0590 0x5587a71d29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a71d0590 0x5587a71d29b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3c97c8312f7cc79 server_cookie=eb6509c8bf3b37b in_seq=0 out_seq=0 2023-09-10T01:15:55.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.498+0000 7f0d0dffb700 10 client.28429 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:55.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d0dffb700 1 -- 172.21.15.3:0/4068744893 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5587a711d8a0 con 0x5587a71d0590 2023-09-10T01:15:55.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d0dffb700 10 client.28429 handle_command_reply: tid=0 2023-09-10T01:15:55.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 client.28429 shutdown 2023-09-10T01:15:55.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 msgr2=0x5587a71c1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c1220 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005c50 tx=0x7f0d200047c0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 shutdown_connections 2023-09-10T01:15:55.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a71d0590 0x5587a71d29b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71c0e40 0x5587a71cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/4068744893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d0dffb700 0 client.28429 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:55.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 >> 172.21.15.3:0/4068744893 conn(0x5587a712b090 msgr2=0x5587a71b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 shutdown_connections 2023-09-10T01:15:55.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.499+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/4068744893 wait complete. 2023-09-10T01:15:55.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.500+0000 7f0d3fec1b80 20 client.28429 trim_cache size 0 max 16384 2023-09-10T01:15:55.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.515+0000 7f0d3fec1b80 1 Processor -- start 2023-09-10T01:15:55.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.515+0000 7f0d3fec1b80 1 -- start start 2023-09-10T01:15:55.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71c0e40 0x5587a71bb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a71b79f0 con 0x5587a71b6710 2023-09-10T01:15:55.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a71ae3d0 con 0x5587a71b7030 2023-09-10T01:15:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a71c0570 con 0x5587a71c0e40 2023-09-10T01:15:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d2a39b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d2ab9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71c0e40 0x5587a71bb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55442/0 (socket says 172.21.15.3:55442) 2023-09-10T01:15:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 -- 172.21.15.3:0/1393956107 learned_addr learned my addr 172.21.15.3:0/1393956107 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 -- 172.21.15.3:0/1393956107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71c0e40 msgr2=0x5587a71bb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71c0e40 0x5587a71bb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 -- 172.21.15.3:0/1393956107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b6710 msgr2=0x5587a71c1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d29b9a700 1 -- 172.21.15.3:0/1393956107 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587a6f5b250 con 0x5587a71b7030 2023-09-10T01:15:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.516+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.517+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c1760 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0d14029000 tx=0x7f0d14022f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ffbf36eac689abc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.517+0000 7f0d193ff700 1 -- 172.21.15.3:0/1393956107 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d140b6040 con 0x5587a71b7030 2023-09-10T01:15:55.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.517+0000 7f0d193ff700 1 -- 172.21.15.3:0/1393956107 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d1409e870 con 0x5587a71b7030 2023-09-10T01:15:55.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.517+0000 7f0d193ff700 1 -- 172.21.15.3:0/1393956107 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1409a180 con 0x5587a71b7030 2023-09-10T01:15:55.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.517+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/1393956107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 msgr2=0x5587a71c1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.517+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c1760 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0d14029000 tx=0x7f0d14022f90 comp rx=0 tx=0).stop 2023-09-10T01:15:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/1393956107 shutdown_connections 2023-09-10T01:15:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71c0e40 0x5587a71bb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71b6710 0x5587a71c1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/1393956107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c1760 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/1393956107 >> 172.21.15.3:0/1393956107 conn(0x5587a712b090 msgr2=0x5587a71b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/1393956107 shutdown_connections 2023-09-10T01:15:55.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/1393956107 wait complete. 2023-09-10T01:15:55.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 -- start start 2023-09-10T01:15:55.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.518+0000 7f0d3fec1b80 1 Processor -- start 2023-09-10T01:15:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d3fec1b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a71c2a40 con 0x5587a71c0e40 2023-09-10T01:15:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a71c2bb0 con 0x5587a71b7030 2023-09-10T01:15:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d3fec1b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587a71c2d70 con 0x5587a71b6710 2023-09-10T01:15:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d3fec1b80 10 client.0 _ll_register_callbacks cb 0x7f0d20004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d2a39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55450/0 (socket says 172.21.15.3:55450) 2023-09-10T01:15:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 -- 172.21.15.3:0/2241566343 learned_addr learned my addr 172.21.15.3:0/2241566343 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d2a39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38240/0 (socket says 172.21.15.3:38240) 2023-09-10T01:15:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d2ab9c700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 -- 172.21.15.3:0/2241566343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 msgr2=0x5587a71c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 -- 172.21.15.3:0/2241566343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 msgr2=0x5587a71c2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.535 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d29b9a700 1 -- 172.21.15.3:0/2241566343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587a6f1dc60 con 0x5587a71b7030 2023-09-10T01:15:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d2ab9c700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.519+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d29b9a700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c63b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400f740 tx=0x7f0d140094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7dc60f0cb52d6fc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:15:55.536 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d0dffb700 1 -- 172.21.15.3:0/2241566343 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1407f070 con 0x5587a71b7030 2023-09-10T01:15:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d1409a9c0 con 0x5587a71b7030 2023-09-10T01:15:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d0dffb700 1 -- 172.21.15.3:0/2241566343 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0d14098d60 con 0x5587a71b7030 2023-09-10T01:15:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d0dffb700 1 -- 172.21.15.3:0/2241566343 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d140b6040 con 0x5587a71b7030 2023-09-10T01:15:55.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d3fec1b80 10 client.28553 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:55.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5587a6f34de0 con 0x5587a71b7030 2023-09-10T01:15:55.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.520+0000 7f0d0dffb700 1 -- 172.21.15.3:0/2241566343 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d140a7020 con 0x5587a71b7030 2023-09-10T01:15:55.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d0dffb700 1 -- 172.21.15.3:0/2241566343 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0d14082030 con 0x5587a71b7030 2023-09-10T01:15:55.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d3fec1b80 10 client.28553 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:55.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d3fec1b80 10 client.28553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:55.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d3fec1b80 20 client.28553 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:55.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a72251a0 0x5587a7227590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:55.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d3fec1b80 4 client.28553 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:15:55.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5587a7101920 con 0x5587a72251a0 2023-09-10T01:15:55.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a72251a0 0x5587a7227590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:55.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d2a39b700 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a72251a0 0x5587a7227590 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b04e94b72cd363ef server_cookie=820caa0b92af5ac4 in_seq=0 out_seq=0 2023-09-10T01:15:55.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.521+0000 7f0d0dffb700 10 client.28553 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:55.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d0dffb700 1 -- 172.21.15.3:0/2241566343 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5587a7101920 con 0x5587a72251a0 2023-09-10T01:15:55.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d0dffb700 10 client.28553 handle_command_reply: tid=0 2023-09-10T01:15:55.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 client.28553 shutdown 2023-09-10T01:15:55.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 msgr2=0x5587a71c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c63b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400f740 tx=0x7f0d140094b0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 shutdown_connections 2023-09-10T01:15:55.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587a72251a0 0x5587a7227590 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587a71b6710 0x5587a71c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587a71c0e40 0x5587a71c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 --2- 172.21.15.3:0/2241566343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587a71b7030 0x5587a71c63b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d0dffb700 0 client.28553 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:55.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 >> 172.21.15.3:0/2241566343 conn(0x5587a712b090 msgr2=0x5587a71c1220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 shutdown_connections 2023-09-10T01:15:55.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 1 -- 172.21.15.3:0/2241566343 wait complete. 2023-09-10T01:15:55.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.522+0000 7f0d3fec1b80 20 client.28553 trim_cache size 0 max 16384 2023-09-10T01:15:55.544 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:15:55.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d100a07b0 msgr2=0x7f0d100a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d100a07b0 0x7f0d100a2c70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0d20011550 tx=0x7f0d20011910 comp rx=0 tx=0).stop 2023-09-10T01:15:55.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 msgr2=0x7f0d2419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:55.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 0x7f0d2419ec10 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0d14005910 tx=0x7f0d14005720 comp rx=0 tx=0).stop 2023-09-10T01:15:55.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 shutdown_connections 2023-09-10T01:15:55.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d100a07b0 0x7f0d100a2c70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d24100640 0x7f0d2419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d24106650 0x7f0d24198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 --2- 172.21.15.3:0/242860705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d24101000 0x7f0d2419ec10 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:55.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 >> 172.21.15.3:0/242860705 conn(0x7f0d240fc170 msgr2=0x7f0d2410ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:55.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 shutdown_connections 2023-09-10T01:15:55.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:55.523+0000 7f0d2c5ff700 1 -- 172.21.15.3:0/242860705 wait complete. 2023-09-10T01:15:55.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:55 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/147381789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:15:55.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:55 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/147381789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:15:55.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:55 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/147381789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:15:55.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1020259 7.53 MB/sec execute 577 sec latency 17.733 ms 2023-09-10T01:15:56.549 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:15:56.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:15:56.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:56 smithi003 ceph-mon[97654]: pgmap v396: 129 pgs: 129 active+clean; 323 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 170 op/s 2023-09-10T01:15:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:56 smithi136 ceph-mon[133708]: pgmap v396: 129 pgs: 129 active+clean; 323 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 170 op/s 2023-09-10T01:15:56.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:56 smithi092 ceph-mon[134067]: pgmap v396: 129 pgs: 129 active+clean; 323 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 170 op/s 2023-09-10T01:15:56.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 -- 172.21.15.3:0/1649649186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec100640 msgr2=0x7fc6ec100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/1649649186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec100640 0x7fc6ec100ac0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009990 tx=0x7fc6e0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 -- 172.21.15.3:0/1649649186 shutdown_connections 2023-09-10T01:15:56.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/1649649186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 0x7fc6ec10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/1649649186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec106650 0x7fc6ec106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/1649649186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec100640 0x7fc6ec100ac0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 -- 172.21.15.3:0/1649649186 >> 172.21.15.3:0/1649649186 conn(0x7fc6ec0fc170 msgr2=0x7fc6ec0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:56.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 -- 172.21.15.3:0/1649649186 shutdown_connections 2023-09-10T01:15:56.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.770+0000 7fc6f15ff700 1 -- 172.21.15.3:0/1649649186 wait complete. 2023-09-10T01:15:56.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 Processor -- start 2023-09-10T01:15:56.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 -- start start 2023-09-10T01:15:56.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec100640 0x7fc6ec196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 0x7fc6ec196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec106650 0x7fc6ec199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec19a580 con 0x7fc6ec101000 2023-09-10T01:15:56.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec19a6f0 con 0x7fc6ec100640 2023-09-10T01:15:56.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.771+0000 7fc6f15ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec19a8b0 con 0x7fc6ec106650 2023-09-10T01:15:56.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 0x7fc6ec196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 0x7fc6ec196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35576/0 (socket says 172.21.15.3:35576) 2023-09-10T01:15:56.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2973504124 learned_addr learned my addr 172.21.15.3:0/2973504124 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:56.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec100640 0x7fc6ec196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec106650 0x7fc6ec199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2973504124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec106650 msgr2=0x7fc6ec199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec106650 0x7fc6ec199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2973504124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec100640 msgr2=0x7fc6ec196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec100640 0x7fc6ec196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2973504124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e00095f0 con 0x7fc6ec101000 2023-09-10T01:15:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec106650 0x7fc6ec199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec100640 0x7fc6ec196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.772+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 0x7fc6ec196bd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0000c00 tx=0x7fc6e0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.773+0000 7fc6d3fff700 1 -- 172.21.15.3:0/2973504124 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e001d070 con 0x7fc6ec101000 2023-09-10T01:15:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.773+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec19ab30 con 0x7fc6ec101000 2023-09-10T01:15:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.773+0000 7fc6d3fff700 1 -- 172.21.15.3:0/2973504124 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6e0022470 con 0x7fc6ec101000 2023-09-10T01:15:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.773+0000 7fc6d3fff700 1 -- 172.21.15.3:0/2973504124 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0017650 con 0x7fc6ec101000 2023-09-10T01:15:56.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.773+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec1a34f0 con 0x7fc6ec101000 2023-09-10T01:15:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.776+0000 7fc6d3fff700 1 -- 172.21.15.3:0/2973504124 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fc6e00225e0 con 0x7fc6ec101000 2023-09-10T01:15:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.777+0000 7fc704e48b80 1 Processor -- start 2023-09-10T01:15:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.777+0000 7fc6d3fff700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc6d40a07b0 0x7fc6d40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc704e48b80 1 -- start start 2023-09-10T01:15:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc6d3fff700 1 -- 172.21.15.3:0/2973504124 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc6e00cd790 con 0x7fc6ec101000 2023-09-10T01:15:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc6d40a07b0 0x7fc6d40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca92eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca93cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc704e48b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca8877b0 con 0x5564ca93d2d0 2023-09-10T01:15:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc704e48b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca8850d0 con 0x5564ca92e790 2023-09-10T01:15:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc704e48b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca8809c0 con 0x5564ca92f0b0 2023-09-10T01:15:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc6eaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca92eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35592/0 (socket says 172.21.15.3:35592) 2023-09-10T01:15:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/1000471441 learned_addr learned my addr 172.21.15.3:0/1000471441 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.778+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca93cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.791 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6eaffd700 1 -- 172.21.15.3:0/1000471441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 msgr2=0x5564ca93cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca93cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.792 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6eaffd700 1 -- 172.21.15.3:0/1000471441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 msgr2=0x5564ca93f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6eaffd700 1 -- 172.21.15.3:0/1000471441 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564ca6ace60 con 0x5564ca92e790 2023-09-10T01:15:56.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc6d40a07b0 0x7fc6d40a2c70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4003fd0 tx=0x7fc6e4003dc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:56.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca92eb70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc011440 tx=0x7fc6dc011750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865c22e77c80e2c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:56.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6d17fa700 1 -- 172.21.15.3:0/1000471441 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc004230 con 0x5564ca92e790 2023-09-10T01:15:56.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6d17fa700 1 -- 172.21.15.3:0/1000471441 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6dc004cf0 con 0x5564ca92e790 2023-09-10T01:15:56.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.779+0000 7fc6d17fa700 1 -- 172.21.15.3:0/1000471441 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc005650 con 0x5564ca92e790 2023-09-10T01:15:56.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 -- 172.21.15.3:0/1000471441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 msgr2=0x5564ca92eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca92eb70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc011440 tx=0x7fc6dc011750 comp rx=0 tx=0).stop 2023-09-10T01:15:56.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 -- 172.21.15.3:0/1000471441 shutdown_connections 2023-09-10T01:15:56.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca92eb70 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1000471441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca93cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 -- 172.21.15.3:0/1000471441 >> 172.21.15.3:0/1000471441 conn(0x5564ca8a3110 msgr2=0x5564ca929c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:56.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 -- 172.21.15.3:0/1000471441 shutdown_connections 2023-09-10T01:15:56.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.780+0000 7fc704e48b80 1 -- 172.21.15.3:0/1000471441 wait complete. 2023-09-10T01:15:56.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.781+0000 7fc704e48b80 1 -- start start 2023-09-10T01:15:56.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.781+0000 7fc704e48b80 1 Processor -- start 2023-09-10T01:15:56.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca93cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca939e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc704e48b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca81ebf0 con 0x5564ca93d2d0 2023-09-10T01:15:56.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc704e48b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca81e8f0 con 0x5564ca92e790 2023-09-10T01:15:56.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc704e48b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca7029a0 con 0x5564ca92f0b0 2023-09-10T01:15:56.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc704e48b80 10 client.0 _ll_register_callbacks cb 0x5564ca768b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:56.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35596/0 (socket says 172.21.15.3:35596) 2023-09-10T01:15:56.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/4225562682 learned_addr learned my addr 172.21.15.3:0/4225562682 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:56.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca93cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca939e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/4225562682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 msgr2=0x5564ca939e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca939e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 msgr2=0x5564ca93cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca93cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.782+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/4225562682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc009c90 con 0x5564ca93d2d0 2023-09-10T01:15:56.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93a390 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400efd0 tx=0x7fc6e4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b232af627268ddf3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:56.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc6ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:56.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc6ceff5700 1 -- 172.21.15.3:0/4225562682 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4029070 con 0x5564ca93d2d0 2023-09-10T01:15:56.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564ca7d8570 con 0x5564ca93d2d0 2023-09-10T01:15:56.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc6ceff5700 1 -- 172.21.15.3:0/4225562682 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc6e401be40 con 0x5564ca93d2d0 2023-09-10T01:15:56.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc6ceff5700 1 -- 172.21.15.3:0/4225562682 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6e40040e0 con 0x5564ca93d2d0 2023-09-10T01:15:56.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc704e48b80 10 client.18477 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:56.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5564ca695ce0 con 0x5564ca93d2d0 2023-09-10T01:15:56.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.783+0000 7fc6ceff5700 1 -- 172.21.15.3:0/4225562682 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5564ca7d8570 con 0x5564ca93d2d0 2023-09-10T01:15:56.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc6ceff5700 1 -- 172.21.15.3:0/4225562682 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc6e402c020 con 0x5564ca93d2d0 2023-09-10T01:15:56.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc704e48b80 10 client.18477 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:56.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc704e48b80 10 client.18477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:56.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc704e48b80 20 client.18477 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:56.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc704e48b80 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca94cd80 0x5564ca94f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc704e48b80 4 client.18477 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:56.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564ca895920 con 0x5564ca94cd80 2023-09-10T01:15:56.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.784+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca94cd80 0x5564ca94f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.785+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca94cd80 0x5564ca94f1a0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6123aed356ffd3a server_cookie=26275240a6757294 in_seq=0 out_seq=0 2023-09-10T01:15:56.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.785+0000 7fc6ceff5700 10 client.18477 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:56.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc6ceff5700 1 -- 172.21.15.3:0/4225562682 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5564ca895920 con 0x5564ca94cd80 2023-09-10T01:15:56.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc6ceff5700 10 client.18477 handle_command_reply: tid=0 2023-09-10T01:15:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 client.18477 shutdown 2023-09-10T01:15:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 msgr2=0x5564ca93a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93a390 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400efd0 tx=0x7fc6e4004b40 comp rx=0 tx=0).stop 2023-09-10T01:15:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 shutdown_connections 2023-09-10T01:15:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca94cd80 0x5564ca94f1a0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93a390 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc6ceff5700 0 client.18477 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:56.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92e790 0x5564ca93cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 --2- 172.21.15.3:0/4225562682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92f0b0 0x5564ca939e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 >> 172.21.15.3:0/4225562682 conn(0x5564ca8a3110 msgr2=0x5564ca929860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:56.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 shutdown_connections 2023-09-10T01:15:56.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 1 -- 172.21.15.3:0/4225562682 wait complete. 2023-09-10T01:15:56.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.786+0000 7fc704e48b80 20 client.18477 trim_cache size 0 max 16384 2023-09-10T01:15:56.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 Processor -- start 2023-09-10T01:15:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 -- start start 2023-09-10T01:15:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca943e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92f0b0 0x5564ca92de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca93fbe0 con 0x5564ca93d2d0 2023-09-10T01:15:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca93b410 con 0x5564ca92f0b0 2023-09-10T01:15:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.804+0000 7fc704e48b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca9306c0 con 0x5564ca92e790 2023-09-10T01:15:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eaffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca943e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35598/0 (socket says 172.21.15.3:35598) 2023-09-10T01:15:56.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/1022760967 learned_addr learned my addr 172.21.15.3:0/1022760967 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:56.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92f0b0 0x5564ca92de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/1022760967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 msgr2=0x5564ca943e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca943e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/1022760967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92f0b0 msgr2=0x5564ca92de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92f0b0 0x5564ca92de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 -- 172.21.15.3:0/1022760967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564ca6d32d0 con 0x5564ca93d2d0 2023-09-10T01:15:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93c750 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc6e401af30 tx=0x7fc6e401af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a61f262bf5b2385 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:56.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.805+0000 7fc6d1ffb700 1 -- 172.21.15.3:0/1022760967 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4029070 con 0x5564ca93d2d0 2023-09-10T01:15:56.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc6d1ffb700 1 -- 172.21.15.3:0/1022760967 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6e402a040 con 0x5564ca93d2d0 2023-09-10T01:15:56.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc6d1ffb700 1 -- 172.21.15.3:0/1022760967 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40259b0 con 0x5564ca93d2d0 2023-09-10T01:15:56.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 -- 172.21.15.3:0/1022760967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 msgr2=0x5564ca93c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93c750 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc6e401af30 tx=0x7fc6e401af60 comp rx=0 tx=0).stop 2023-09-10T01:15:56.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 -- 172.21.15.3:0/1022760967 shutdown_connections 2023-09-10T01:15:56.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca943e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca93d2d0 0x5564ca93c750 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 --2- 172.21.15.3:0/1022760967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca92f0b0 0x5564ca92de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 -- 172.21.15.3:0/1022760967 >> 172.21.15.3:0/1022760967 conn(0x5564ca8a3110 msgr2=0x5564ca92d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:56.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 -- 172.21.15.3:0/1022760967 shutdown_connections 2023-09-10T01:15:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.806+0000 7fc704e48b80 1 -- 172.21.15.3:0/1022760967 wait complete. 2023-09-10T01:15:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 -- start start 2023-09-10T01:15:56.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 Processor -- start 2023-09-10T01:15:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca99b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca92f0b0 0x5564ca99b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca93d2d0 0x5564ca942f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca943520 con 0x5564ca92f0b0 2023-09-10T01:15:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca943690 con 0x5564ca93d2d0 2023-09-10T01:15:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ca943850 con 0x5564ca92e790 2023-09-10T01:15:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.807+0000 7fc704e48b80 10 client.0 _ll_register_callbacks cb 0x7fc6e40048c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:56.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca92f0b0 0x5564ca99b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca92f0b0 0x5564ca99b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35600/0 (socket says 172.21.15.3:35600) 2023-09-10T01:15:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2621029865 learned_addr learned my addr 172.21.15.3:0/2621029865 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca99b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca93d2d0 0x5564ca942f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2621029865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 msgr2=0x5564ca99b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca99b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca93d2d0 msgr2=0x5564ca942f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca93d2d0 0x5564ca942f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 -- 172.21.15.3:0/2621029865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564ca695ce0 con 0x5564ca92f0b0 2023-09-10T01:15:56.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6eb7fe700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca93d2d0 0x5564ca942f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ea7fc700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca92f0b0 0x5564ca99b850 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00973b0 tx=0x7fc6e00caff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9403ae00ac7fdb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ceff5700 1 -- 172.21.15.3:0/2621029865 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e009e770 con 0x5564ca92f0b0 2023-09-10T01:15:56.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6dc004cf0 con 0x5564ca92f0b0 2023-09-10T01:15:56.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.808+0000 7fc6ceff5700 1 -- 172.21.15.3:0/2621029865 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc6e00a2920 con 0x5564ca92f0b0 2023-09-10T01:15:56.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc6ceff5700 1 -- 172.21.15.3:0/2621029865 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6e0098ce0 con 0x5564ca92f0b0 2023-09-10T01:15:56.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 10 client.18483 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:56.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5564ca6ab330 con 0x5564ca92f0b0 2023-09-10T01:15:56.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc6ceff5700 1 -- 172.21.15.3:0/2621029865 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6dc004cf0 con 0x5564ca92f0b0 2023-09-10T01:15:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc6ceff5700 1 -- 172.21.15.3:0/2621029865 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc6e00a2490 con 0x5564ca92f0b0 2023-09-10T01:15:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 10 client.18483 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 10 client.18483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 20 client.18483 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:56.832 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca9a2fb0 0x5564ca9a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:56.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 4 client.18483 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:15:56.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5564ca8799a0 con 0x5564ca9a2fb0 2023-09-10T01:15:56.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.809+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca9a2fb0 0x5564ca9a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:56.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.810+0000 7fc6eaffd700 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca9a2fb0 0x5564ca9a53d0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f188aae8c121253 server_cookie=f4a1cd2f02c829a7 in_seq=0 out_seq=0 2023-09-10T01:15:56.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.810+0000 7fc6ceff5700 10 client.18483 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:56.834 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1020264 7.52 MB/sec execute 578 sec latency 997.019 ms 2023-09-10T01:15:56.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc6ceff5700 1 -- 172.21.15.3:0/2621029865 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5564ca8799a0 con 0x5564ca9a2fb0 2023-09-10T01:15:56.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc6ceff5700 10 client.18483 handle_command_reply: tid=0 2023-09-10T01:15:56.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 client.18483 shutdown 2023-09-10T01:15:56.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca92f0b0 msgr2=0x5564ca99b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca92f0b0 0x5564ca99b850 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00973b0 tx=0x7fc6e00caff0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 shutdown_connections 2023-09-10T01:15:56.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5564ca9a2fb0 0x5564ca9a53d0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564ca92e790 0x5564ca99b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564ca93d2d0 0x5564ca942f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 --2- 172.21.15.3:0/2621029865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564ca92f0b0 0x5564ca99b850 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc6ceff5700 0 client.18483 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:56.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 >> 172.21.15.3:0/2621029865 conn(0x5564ca8a3110 msgr2=0x5564ca928e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:56.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 shutdown_connections 2023-09-10T01:15:56.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.811+0000 7fc704e48b80 1 -- 172.21.15.3:0/2621029865 wait complete. 2023-09-10T01:15:56.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc704e48b80 20 client.18483 trim_cache size 0 max 16384 2023-09-10T01:15:56.838 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:15:56.839 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:15:56.839 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "f9592d45-e5aa-458e-8599-5506c3bbafb2", 2023-09-10T01:15:56.839 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:15:56.839 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:15:56.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc6d40a07b0 msgr2=0x7fc6d40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc6d40a07b0 0x7fc6d40a2c70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4003fd0 tx=0x7fc6e4003dc0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 msgr2=0x7fc6ec196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:56.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 0x7fc6ec196bd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0000c00 tx=0x7fc6e0005740 comp rx=0 tx=0).stop 2023-09-10T01:15:56.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 shutdown_connections 2023-09-10T01:15:56.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc6d40a07b0 0x7fc6d40a2c70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6ec106650 0x7fc6ec199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6ec100640 0x7fc6ec196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 --2- 172.21.15.3:0/2973504124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6ec101000 0x7fc6ec196bd0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:56.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 >> 172.21.15.3:0/2973504124 conn(0x7fc6ec0fc170 msgr2=0x7fc6ec104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:56.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 shutdown_connections 2023-09-10T01:15:56.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:56.812+0000 7fc6f15ff700 1 -- 172.21.15.3:0/2973504124 wait complete. 2023-09-10T01:15:56.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:15:57.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 -- 172.21.15.3:0/3504029855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0100650 msgr2=0x7fe2c0100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/3504029855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0100650 0x7fe2c0100ad0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009990 tx=0x7fe2b4009ca0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 -- 172.21.15.3:0/3504029855 shutdown_connections 2023-09-10T01:15:57.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/3504029855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0106660 0x7fe2c0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/3504029855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c0101010 0x7fe2c010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/3504029855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0100650 0x7fe2c0100ad0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 -- 172.21.15.3:0/3504029855 >> 172.21.15.3:0/3504029855 conn(0x7fe2c00fc160 msgr2=0x7fe2c00fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:57.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 -- 172.21.15.3:0/3504029855 shutdown_connections 2023-09-10T01:15:57.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.131+0000 7fe2c4bff700 1 -- 172.21.15.3:0/3504029855 wait complete. 2023-09-10T01:15:57.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.132+0000 7fe2c4bff700 1 Processor -- start 2023-09-10T01:15:57.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2c4bff700 1 -- start start 2023-09-10T01:15:57.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2c4bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0100650 0x7fe2c019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2c4bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c0101010 0x7fe2c019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2c4bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0106660 0x7fe2c0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2c4bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0198de0 con 0x7fe2c0101010 2023-09-10T01:15:57.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2c4bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0198f50 con 0x7fe2c0106660 2023-09-10T01:15:57.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2c4bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0199110 con 0x7fe2c0100650 2023-09-10T01:15:57.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0100650 0x7fe2c019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0100650 0x7fe2c019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38296/0 (socket says 172.21.15.3:38296) 2023-09-10T01:15:57.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2be59c700 1 -- 172.21.15.3:0/74428841 learned_addr learned my addr 172.21.15.3:0/74428841 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:57.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c0101010 0x7fe2c019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.133+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0106660 0x7fe2c0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2be59c700 1 -- 172.21.15.3:0/74428841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0106660 msgr2=0x7fe2c0198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2be59c700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0106660 0x7fe2c0198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2be59c700 1 -- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c0101010 msgr2=0x7fe2c019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2be59c700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c0101010 0x7fe2c019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2be59c700 1 -- 172.21.15.3:0/74428841 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40095f0 con 0x7fe2c0100650 2023-09-10T01:15:57.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2be59c700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0100650 0x7fe2c019e6b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000e9a0 tx=0x7fe2b000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:57.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2af7fe700 1 -- 172.21.15.3:0/74428841 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0009790 con 0x7fe2c0100650 2023-09-10T01:15:57.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2af7fe700 1 -- 172.21.15.3:0/74428841 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2b0010430 con 0x7fe2c0100650 2023-09-10T01:15:57.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c01993f0 con 0x7fe2c0100650 2023-09-10T01:15:57.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.134+0000 7fe2af7fe700 1 -- 172.21.15.3:0/74428841 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0018680 con 0x7fe2c0100650 2023-09-10T01:15:57.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.135+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c01a5750 con 0x7fe2c0100650 2023-09-10T01:15:57.146 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.138+0000 7fe2af7fe700 1 -- 172.21.15.3:0/74428841 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe2b00105a0 con 0x7fe2c0100650 2023-09-10T01:15:57.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.139+0000 7fe2d846cb80 1 Processor -- start 2023-09-10T01:15:57.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.139+0000 7fe2d846cb80 1 -- start start 2023-09-10T01:15:57.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.139+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8361af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.139+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee836b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.147 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee836e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee82ba730 con 0x555ee8361710 2023-09-10T01:15:57.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee82b8050 con 0x555ee836be40 2023-09-10T01:15:57.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee82b3940 con 0x555ee8362030 2023-09-10T01:15:57.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8361af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee836e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee836b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee836e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55524/0 (socket says 172.21.15.3:55524) 2023-09-10T01:15:57.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8361af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35618/0 (socket says 172.21.15.3:35618) 2023-09-10T01:15:57.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 -- 172.21.15.3:0/1986249984 learned_addr learned my addr 172.21.15.3:0/1986249984 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:57.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2af7fe700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2a80a07b0 0x7fe2a80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 -- 172.21.15.3:0/1986249984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 msgr2=0x555ee836b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee836b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 -- 172.21.15.3:0/1986249984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 msgr2=0x555ee8361af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8361af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bdd9b700 1 -- 172.21.15.3:0/1986249984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ee80dfde0 con 0x555ee836be40 2023-09-10T01:15:57.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2af7fe700 1 -- 172.21.15.3:0/74428841 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe2b0014070 con 0x7fe2c0100650 2023-09-10T01:15:57.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8361af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:57.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.140+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2a80a07b0 0x7fe2a80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.141+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee836e230 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4015000 tx=0x7fe2b4005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3d19edb1ee5b820 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:57.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.141+0000 7fe2acff9700 1 -- 172.21.15.3:0/1986249984 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b401d070 con 0x555ee836be40 2023-09-10T01:15:57.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.141+0000 7fe2acff9700 1 -- 172.21.15.3:0/1986249984 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2b401e040 con 0x555ee836be40 2023-09-10T01:15:57.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.141+0000 7fe2acff9700 1 -- 172.21.15.3:0/1986249984 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b40048f0 con 0x555ee836be40 2023-09-10T01:15:57.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1986249984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 msgr2=0x555ee836e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2a80a07b0 0x7fe2a80a2c70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009ec0 tx=0x7fe2b8009040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:57.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee836e230 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4015000 tx=0x7fe2b4005ea0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1986249984 shutdown_connections 2023-09-10T01:15:57.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee836e230 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8361af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1986249984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee836b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1986249984 >> 172.21.15.3:0/1986249984 conn(0x555ee82d6090 msgr2=0x555ee835cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:57.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1986249984 shutdown_connections 2023-09-10T01:15:57.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1986249984 wait complete. 2023-09-10T01:15:57.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.142+0000 7fe2d846cb80 1 -- start start 2023-09-10T01:15:57.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 1 Processor -- start 2023-09-10T01:15:57.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8361710 0x555ee836b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8362030 0x555ee83689c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee8368f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee8251b70 con 0x555ee8362030 2023-09-10T01:15:57.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee8251870 con 0x555ee836be40 2023-09-10T01:15:57.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee8135920 con 0x555ee8361710 2023-09-10T01:15:57.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2d846cb80 10 client.0 _ll_register_callbacks cb 0x555ee819ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:57.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8361710 0x555ee836b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.143+0000 7fe2bdd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8362030 0x555ee83689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8361710 0x555ee836b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:38322/0 (socket says 172.21.15.3:38322) 2023-09-10T01:15:57.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 -- 172.21.15.3:0/2281004218 learned_addr learned my addr 172.21.15.3:0/2281004218 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:57.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee8368f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 -- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 msgr2=0x555ee8368f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee8368f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 -- 172.21.15.3:0/2281004218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8362030 msgr2=0x555ee83689c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8362030 0x555ee83689c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 -- 172.21.15.3:0/2281004218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c010de20 con 0x555ee8361710 2023-09-10T01:15:57.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8362030 0x555ee83689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:57.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2be59c700 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8361710 0x555ee836b7b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000c2d0 tx=0x7fe2b0095630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b3175484b506e6a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:57.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:15:57.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.144+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/2281004218 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0099c60 con 0x555ee8361710 2023-09-10T01:15:57.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/2281004218 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe2b0099560 con 0x555ee8361710 2023-09-10T01:15:57.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ee820b4f0 con 0x555ee8361710 2023-09-10T01:15:57.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/2281004218 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2b009d390 con 0x555ee8361710 2023-09-10T01:15:57.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 10 client.28465 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:57.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x555ee80c8c60 con 0x555ee8361710 2023-09-10T01:15:57.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/2281004218 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555ee820b4f0 con 0x555ee8361710 2023-09-10T01:15:57.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/2281004218 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe2b00997c0 con 0x555ee8361710 2023-09-10T01:15:57.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 10 client.28465 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:57.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 10 client.28465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:57.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 20 client.28465 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:57.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee837b7a0 0x555ee837dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 4 client.28465 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:15:57.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.145+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ee82c88a0 con 0x555ee837b7a0 2023-09-10T01:15:57.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.146+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee837b7a0 0x555ee837dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.146+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee837b7a0 0x555ee837dbc0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76345a5a75602f27 server_cookie=ec661b0d955f7f06 in_seq=0 out_seq=0 2023-09-10T01:15:57.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.146+0000 7fe2a5ffb700 10 client.28465 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:57.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/2281004218 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555ee82c88a0 con 0x555ee837b7a0 2023-09-10T01:15:57.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2a5ffb700 10 client.28465 handle_command_reply: tid=0 2023-09-10T01:15:57.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 client.28465 shutdown 2023-09-10T01:15:57.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8361710 msgr2=0x555ee836b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8361710 0x555ee836b7b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000c2d0 tx=0x7fe2b0095630 comp rx=0 tx=0).stop 2023-09-10T01:15:57.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 shutdown_connections 2023-09-10T01:15:57.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee837b7a0 0x555ee837dbc0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8361710 0x555ee836b7b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee8368f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/2281004218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8362030 0x555ee83689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 >> 172.21.15.3:0/2281004218 conn(0x555ee82d6090 msgr2=0x555ee835c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:57.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2a5ffb700 0 client.28465 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:57.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 shutdown_connections 2023-09-10T01:15:57.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 1 -- 172.21.15.3:0/2281004218 wait complete. 2023-09-10T01:15:57.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.147+0000 7fe2d846cb80 20 client.28465 trim_cache size 0 max 16384 2023-09-10T01:15:57.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.166+0000 7fe2d846cb80 1 Processor -- start 2023-09-10T01:15:57.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.166+0000 7fe2d846cb80 1 -- start start 2023-09-10T01:15:57.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee8361710 0x555ee836b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8363c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee836be40 0x555ee83641c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee83593d0 con 0x555ee836be40 2023-09-10T01:15:57.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee8362410 con 0x555ee8361710 2023-09-10T01:15:57.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee8366f20 con 0x555ee8362030 2023-09-10T01:15:57.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee8361710 0x555ee836b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee8361710 0x555ee836b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:55544/0 (socket says 172.21.15.3:55544) 2023-09-10T01:15:57.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 -- 172.21.15.3:0/1725081869 learned_addr learned my addr 172.21.15.3:0/1725081869 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:57.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee836be40 0x555ee83641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8363c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 -- 172.21.15.3:0/1725081869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 msgr2=0x555ee8363c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8363c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 -- 172.21.15.3:0/1725081869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee836be40 msgr2=0x555ee83641c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee836be40 0x555ee83641c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2be59c700 1 -- 172.21.15.3:0/1725081869 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ee8106250 con 0x555ee8361710 2023-09-10T01:15:57.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.167+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8363c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:57.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.168+0000 7fe2be59c700 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee8361710 0x555ee836b2e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00994f0 tx=0x7fe2b0010a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5d174111cf2f731 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:57.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.168+0000 7fe2ad7fa700 1 -- 172.21.15.3:0/1725081869 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0099c60 con 0x555ee8361710 2023-09-10T01:15:57.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.168+0000 7fe2ad7fa700 1 -- 172.21.15.3:0/1725081869 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2b0099560 con 0x555ee8361710 2023-09-10T01:15:57.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.168+0000 7fe2ad7fa700 1 -- 172.21.15.3:0/1725081869 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00cc150 con 0x555ee8361710 2023-09-10T01:15:57.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1725081869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee8361710 msgr2=0x555ee836b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee8361710 0x555ee836b2e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00994f0 tx=0x7fe2b0010a90 comp rx=0 tx=0).stop 2023-09-10T01:15:57.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1725081869 shutdown_connections 2023-09-10T01:15:57.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee836be40 0x555ee83641c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee8361710 0x555ee836b2e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/1725081869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8363c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1725081869 >> 172.21.15.3:0/1725081869 conn(0x555ee82d6090 msgr2=0x555ee83770e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:57.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1725081869 shutdown_connections 2023-09-10T01:15:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 -- 172.21.15.3:0/1725081869 wait complete. 2023-09-10T01:15:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 -- start start 2023-09-10T01:15:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.169+0000 7fe2d846cb80 1 Processor -- start 2023-09-10T01:15:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8370700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8370c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2d846cb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee83ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee83ceb10 con 0x555ee8361710 2023-09-10T01:15:57.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee83cec50 con 0x555ee836be40 2023-09-10T01:15:57.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2d846cb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ee83ced90 con 0x555ee8362030 2023-09-10T01:15:57.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2d846cb80 10 client.0 _ll_register_callbacks cb 0x555ee819ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:15:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8370700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2be59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8370700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35646/0 (socket says 172.21.15.3:35646) 2023-09-10T01:15:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2be59c700 1 -- 172.21.15.3:0/4138940713 learned_addr learned my addr 172.21.15.3:0/4138940713 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:15:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee83ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.170+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8370c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2be59c700 1 -- 172.21.15.3:0/4138940713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 msgr2=0x555ee8370c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2be59c700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8370c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2be59c700 1 -- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 msgr2=0x555ee83ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:15:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2be59c700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee83ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2be59c700 1 -- 172.21.15.3:0/4138940713 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ee80c8c60 con 0x555ee8361710 2023-09-10T01:15:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2bed9d700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee83ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:15:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8370c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:15:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2be59c700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8370700 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0095fd0 tx=0x7fe2b00997c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b840597ab06698be server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:15:57.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:15:57.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/4138940713 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0098c80 con 0x555ee8361710 2023-09-10T01:15:57.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/4138940713 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe2b00a15d0 con 0x555ee8361710 2023-09-10T01:15:57.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2b401e040 con 0x555ee8361710 2023-09-10T01:15:57.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/4138940713 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2b0099c60 con 0x555ee8361710 2023-09-10T01:15:57.188 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2d846cb80 10 client.18501 fetch_fsmap learned FSMap version 36 2023-09-10T01:15:57.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x555ee80de2b0 con 0x555ee8361710 2023-09-10T01:15:57.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.171+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/4138940713 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2b00b5080 con 0x555ee8361710 2023-09-10T01:15:57.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/4138940713 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe2b009e0d0 con 0x555ee8361710 2023-09-10T01:15:57.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2d846cb80 10 client.18501 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:15:57.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2d846cb80 10 client.18501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:15:57.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2d846cb80 20 client.18501 populate_metadata read hostname 'smithi003' 2023-09-10T01:15:57.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee83cffa0 0x555ee83d23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:15:57.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2d846cb80 4 client.18501 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:15:57.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555ee82ac920 con 0x555ee83cffa0 2023-09-10T01:15:57.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee83cffa0 0x555ee83d23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:15:57.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2bdd9b700 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee83cffa0 0x555ee83d23c0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e2f70fc60a56aa server_cookie=a9f2849a39f7de07 in_seq=0 out_seq=0 2023-09-10T01:15:57.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.172+0000 7fe2a5ffb700 10 client.18501 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:57.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2a5ffb700 1 -- 172.21.15.3:0/4138940713 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555ee82ac920 con 0x555ee83cffa0 2023-09-10T01:15:57.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2a5ffb700 10 client.18501 handle_command_reply: tid=0 2023-09-10T01:15:57.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 client.18501 shutdown 2023-09-10T01:15:57.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 msgr2=0x555ee8370700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8370700 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0095fd0 tx=0x7fe2b00997c0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 shutdown_connections 2023-09-10T01:15:57.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555ee83cffa0 0x555ee83d23c0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555ee836be40 0x555ee83ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ee8361710 0x555ee8370700 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 --2- 172.21.15.3:0/4138940713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555ee8362030 0x555ee8370c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2a5ffb700 0 client.18501 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:15:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 >> 172.21.15.3:0/4138940713 conn(0x555ee82d6090 msgr2=0x555ee8363c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 shutdown_connections 2023-09-10T01:15:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 1 -- 172.21.15.3:0/4138940713 wait complete. 2023-09-10T01:15:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.173+0000 7fe2d846cb80 20 client.18501 trim_cache size 0 max 16384 2023-09-10T01:15:57.195 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:15:57.195 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:15:57.195 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:15:57.196 INFO:teuthology.orchestra.run.smithi003.stdout: "f9592d45-e5aa-458e-8599-5506c3bbafb2": { 2023-09-10T01:15:57.196 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:15:57.196 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "f9592d45-e5aa-458e-8599-5506c3bbafb2", 2023-09-10T01:15:57.196 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:15:57.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:15:57.197 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:15:57.197 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:15:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2a80a07b0 msgr2=0x7fe2a80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2a80a07b0 0x7fe2a80a2c70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009ec0 tx=0x7fe2b8009040 comp rx=0 tx=0).stop 2023-09-10T01:15:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0100650 msgr2=0x7fe2c019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:15:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0100650 0x7fe2c019e6b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000e9a0 tx=0x7fe2b000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 shutdown_connections 2023-09-10T01:15:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe2a80a07b0 0x7fe2a80a2c70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2c0100650 0x7fe2c019e6b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2c0106660 0x7fe2c0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 --2- 172.21.15.3:0/74428841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2c0101010 0x7fe2c019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:15:57.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 >> 172.21.15.3:0/74428841 conn(0x7fe2c00fc160 msgr2=0x7fe2c010ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:15:57.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 shutdown_connections 2023-09-10T01:15:57.201 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:15:57.174+0000 7fe2c4bff700 1 -- 172.21.15.3:0/74428841 wait complete. 2023-09-10T01:15:57.201 INFO:tasks.cephfs.filesystem:scrub status for tag:f9592d45-e5aa-458e-8599-5506c3bbafb2 - {'path': '/', 'tag': 'f9592d45-e5aa-458e-8599-5506c3bbafb2', 'options': 'recursive,force'} 2023-09-10T01:15:57.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:15:57.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:57.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:57.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:57.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:57.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:57 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:15:57.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:57.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:57.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:57.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:57.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:57 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:15:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:57.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:57.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:57.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:57.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:57 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:57.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1020388 7.50 MB/sec execute 579 sec latency 1051.330 ms 2023-09-10T01:15:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: pgmap v397: 129 pgs: 129 active+clean; 323 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2023-09-10T01:15:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:58.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:58.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:58.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:15:58.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:15:58 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: pgmap v397: 129 pgs: 129 active+clean; 323 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2023-09-10T01:15:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:58.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:58.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:58.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:15:58.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:15:58 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: pgmap v397: 129 pgs: 129 active+clean; 323 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2023-09-10T01:15:58.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:58.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:58.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:58.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:58.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:15:58.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:15:58 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:15:58.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1021291 7.50 MB/sec execute 580 sec latency 1075.246 ms 2023-09-10T01:15:59.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1023219 7.51 MB/sec execute 581 sec latency 27.230 ms 2023-09-10T01:16:00.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:00 smithi003 ceph-mon[97654]: pgmap v398: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2023-09-10T01:16:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:00 smithi136 ceph-mon[133708]: pgmap v398: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2023-09-10T01:16:00.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:00 smithi092 ceph-mon[134067]: pgmap v398: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2023-09-10T01:16:00.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1023929 7.50 MB/sec execute 582 sec latency 113.125 ms 2023-09-10T01:16:01.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1024166 7.49 MB/sec execute 583 sec latency 494.053 ms 2023-09-10T01:16:02.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:02 smithi003 ceph-mon[97654]: pgmap v399: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1018 KiB/s wr, 88 op/s 2023-09-10T01:16:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:02 smithi136 ceph-mon[133708]: pgmap v399: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1018 KiB/s wr, 88 op/s 2023-09-10T01:16:02.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:02 smithi092 ceph-mon[134067]: pgmap v399: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1018 KiB/s wr, 88 op/s 2023-09-10T01:16:02.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1024668 7.48 MB/sec execute 584 sec latency 137.649 ms 2023-09-10T01:16:03.811 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1026327 7.48 MB/sec execute 585 sec latency 29.272 ms 2023-09-10T01:16:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:04 smithi003 ceph-mon[97654]: pgmap v400: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1018 KiB/s wr, 88 op/s 2023-09-10T01:16:04.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:04 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:16:04.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:04 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:16:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:04 smithi136 ceph-mon[133708]: pgmap v400: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1018 KiB/s wr, 88 op/s 2023-09-10T01:16:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:04 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:16:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:04 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:16:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:04 smithi092 ceph-mon[134067]: pgmap v400: 129 pgs: 129 active+clean; 324 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 1018 KiB/s wr, 88 op/s 2023-09-10T01:16:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:04 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:16:04.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:04 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:16:04.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1026989 7.48 MB/sec execute 586 sec latency 77.815 ms 2023-09-10T01:16:05.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:05 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:16:05.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:05 smithi003 ceph-mon[97654]: osdmap e104: 12 total, 12 up, 12 in 2023-09-10T01:16:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:05 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:16:05.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:05 smithi136 ceph-mon[133708]: osdmap e104: 12 total, 12 up, 12 in 2023-09-10T01:16:05.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:05 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:16:05.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:05 smithi092 ceph-mon[134067]: osdmap e104: 12 total, 12 up, 12 in 2023-09-10T01:16:05.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1027355 7.47 MB/sec execute 587 sec latency 166.580 ms 2023-09-10T01:16:06.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:06 smithi003 ceph-mon[97654]: pgmap v402: 129 pgs: 129 active+clean; 328 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.9 MiB/s wr, 203 op/s 2023-09-10T01:16:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:06 smithi136 ceph-mon[133708]: pgmap v402: 129 pgs: 129 active+clean; 328 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.9 MiB/s wr, 203 op/s 2023-09-10T01:16:06.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:06 smithi092 ceph-mon[134067]: pgmap v402: 129 pgs: 129 active+clean; 328 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.9 MiB/s wr, 203 op/s 2023-09-10T01:16:06.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1027355 7.45 MB/sec execute 588 sec latency 1166.665 ms 2023-09-10T01:16:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.323+0000 7f68db9ff700 1 -- 172.21.15.3:0/2981420841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d410ed00 msgr2=0x7f68d410cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.323+0000 7f68db9ff700 1 --2- 172.21.15.3:0/2981420841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d410ed00 0x7f68d410cac0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009f90 tx=0x7f68cc00b330 comp rx=0 tx=0).stop 2023-09-10T01:16:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.324+0000 7f68db9ff700 1 -- 172.21.15.3:0/2981420841 shutdown_connections 2023-09-10T01:16:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.324+0000 7f68db9ff700 1 --2- 172.21.15.3:0/2981420841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d410ed00 0x7f68d410cac0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.324+0000 7f68db9ff700 1 --2- 172.21.15.3:0/2981420841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d4107600 0x7f68d4107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.324+0000 7f68db9ff700 1 --2- 172.21.15.3:0/2981420841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d410f170 0x7f68d410d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.324+0000 7f68db9ff700 1 -- 172.21.15.3:0/2981420841 >> 172.21.15.3:0/2981420841 conn(0x7f68d406c6c0 msgr2=0x7f68d406cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.324+0000 7f68db9ff700 1 -- 172.21.15.3:0/2981420841 shutdown_connections 2023-09-10T01:16:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.324+0000 7f68db9ff700 1 -- 172.21.15.3:0/2981420841 wait complete. 2023-09-10T01:16:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.326+0000 7f68db9ff700 1 Processor -- start 2023-09-10T01:16:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.326+0000 7f68db9ff700 1 -- start start 2023-09-10T01:16:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.327+0000 7f68db9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4107600 0x7f68d41ab590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.327+0000 7f68db9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d410ed00 0x7f68d41abad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.327+0000 7f68db9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d410f170 0x7f68d41a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.327+0000 7f68db9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d41a5c70 con 0x7f68d410f170 2023-09-10T01:16:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.327+0000 7f68db9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d41a5de0 con 0x7f68d410ed00 2023-09-10T01:16:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.327+0000 7f68db9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d41a5fa0 con 0x7f68d4107600 2023-09-10T01:16:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.327+0000 7f68d8f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d410ed00 0x7f68d41abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4107600 0x7f68d41ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d8f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d410ed00 0x7f68d41abad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53476/0 (socket says 172.21.15.3:53476) 2023-09-10T01:16:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4107600 0x7f68d41ab590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41066/0 (socket says 172.21.15.3:41066) 2023-09-10T01:16:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d8f9a700 1 -- 172.21.15.3:0/4062109798 learned_addr learned my addr 172.21.15.3:0/4062109798 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d8f9a700 1 -- 172.21.15.3:0/4062109798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4107600 msgr2=0x7f68d41ab590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4107600 0x7f68d41ab590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d8f9a700 1 -- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d410f170 msgr2=0x7f68d41a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d410f170 0x7f68d41a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.328+0000 7f68d8f9a700 1 -- 172.21.15.3:0/4062109798 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc009c40 con 0x7f68d410ed00 2023-09-10T01:16:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.329+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d410ed00 0x7f68d41abad0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f68c800b500 tx=0x7f68c800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.329+0000 7f68d27fc700 1 -- 172.21.15.3:0/4062109798 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8011830 con 0x7f68d410ed00 2023-09-10T01:16:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.329+0000 7f68d27fc700 1 -- 172.21.15.3:0/4062109798 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68c8011e50 con 0x7f68d410ed00 2023-09-10T01:16:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.329+0000 7f68d27fc700 1 -- 172.21.15.3:0/4062109798 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c800f2e0 con 0x7f68d410ed00 2023-09-10T01:16:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.329+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d41a6280 con 0x7f68d410ed00 2023-09-10T01:16:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.329+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d41b2580 con 0x7f68d410ed00 2023-09-10T01:16:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.338+0000 7f68ef317b80 1 Processor -- start 2023-09-10T01:16:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.338+0000 7f68ef317b80 1 -- start start 2023-09-10T01:16:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.339+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a49321a0 0x5602a4932580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.339+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a492c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.339+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a492ccc0 0x5602a4938f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.339+0000 7f68ef317b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a48856d0 con 0x5602a492ccc0 2023-09-10T01:16:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.339+0000 7f68ef317b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a4882ff0 con 0x5602a492c300 2023-09-10T01:16:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.339+0000 7f68ef317b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a487e8e0 con 0x5602a49321a0 2023-09-10T01:16:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.341+0000 7f68d9f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a49321a0 0x5602a4932580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.341+0000 7f68d9f9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a49321a0 0x5602a4932580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41080/0 (socket says 172.21.15.3:41080) 2023-09-10T01:16:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.341+0000 7f68d9f9c700 1 -- 172.21.15.3:0/1199451382 learned_addr learned my addr 172.21.15.3:0/1199451382 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.341+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a492ccc0 0x5602a4938f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.341+0000 7f68d979b700 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a492c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.341+0000 7f68d27fc700 1 -- 172.21.15.3:0/4062109798 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f68c8010580 con 0x7f68d410ed00 2023-09-10T01:16:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.342+0000 7f68d9f9c700 1 -- 172.21.15.3:0/1199451382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 msgr2=0x5602a492c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.342+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a492c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.342+0000 7f68d9f9c700 1 -- 172.21.15.3:0/1199451382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a492ccc0 msgr2=0x5602a4938f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.342+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a492ccc0 0x5602a4938f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.342+0000 7f68d9f9c700 1 -- 172.21.15.3:0/1199451382 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602a46aad80 con 0x5602a49321a0 2023-09-10T01:16:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.342+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a49321a0 0x5602a4932580 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f68c40046e0 tx=0x7f68c4009ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f911dc98230f9f34 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.343+0000 7f68bb7fe700 1 -- 172.21.15.3:0/1199451382 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c400ea80 con 0x5602a49321a0 2023-09-10T01:16:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.343+0000 7f68bb7fe700 1 -- 172.21.15.3:0/1199451382 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68c4004cf0 con 0x5602a49321a0 2023-09-10T01:16:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.343+0000 7f68bb7fe700 1 -- 172.21.15.3:0/1199451382 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c4005550 con 0x5602a49321a0 2023-09-10T01:16:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.344+0000 7f68ef317b80 1 -- 172.21.15.3:0/1199451382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a49321a0 msgr2=0x5602a4932580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.344+0000 7f68ef317b80 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a49321a0 0x5602a4932580 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f68c40046e0 tx=0x7f68c4009ea0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.344+0000 7f68d27fc700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a08b0 0x7f68bc0a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.344+0000 7f68d27fc700 1 -- 172.21.15.3:0/4062109798 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f68c80cbde0 con 0x7f68d410ed00 2023-09-10T01:16:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.345+0000 7f68ef317b80 1 -- 172.21.15.3:0/1199451382 shutdown_connections 2023-09-10T01:16:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.345+0000 7f68ef317b80 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a49321a0 0x5602a4932580 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.345+0000 7f68ef317b80 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a492ccc0 0x5602a4938f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.345+0000 7f68ef317b80 1 --2- 172.21.15.3:0/1199451382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a492c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.345+0000 7f68ef317b80 1 -- 172.21.15.3:0/1199451382 >> 172.21.15.3:0/1199451382 conn(0x5602a48a1030 msgr2=0x5602a4927560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.347+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a08b0 0x7f68bc0a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.350+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a08b0 0x7f68bc0a2d70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68c400e630 tx=0x7f68c4000bc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.351+0000 7f68ef317b80 1 -- 172.21.15.3:0/1199451382 shutdown_connections 2023-09-10T01:16:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.351+0000 7f68ef317b80 1 -- 172.21.15.3:0/1199451382 wait complete. 2023-09-10T01:16:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.352+0000 7f68ef317b80 1 -- start start 2023-09-10T01:16:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.352+0000 7f68ef317b80 1 Processor -- start 2023-09-10T01:16:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a4938f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492ccc0 0x5602a4936180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49368c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68ef317b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a481cb10 con 0x5602a49321a0 2023-09-10T01:16:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68ef317b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a481c810 con 0x5602a492c300 2023-09-10T01:16:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68ef317b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a47008c0 con 0x5602a492ccc0 2023-09-10T01:16:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68ef317b80 10 client.0 _ll_register_callbacks cb 0x5602a4766a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68d979b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a4938f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68d979b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a4938f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53482/0 (socket says 172.21.15.3:53482) 2023-09-10T01:16:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.353+0000 7f68d979b700 1 -- 172.21.15.3:0/949568388 learned_addr learned my addr 172.21.15.3:0/949568388 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492ccc0 0x5602a4936180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d979b700 1 -- 172.21.15.3:0/949568388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492ccc0 msgr2=0x5602a4936180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d979b700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492ccc0 0x5602a4936180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d979b700 1 -- 172.21.15.3:0/949568388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 msgr2=0x5602a49368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d979b700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49368c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d979b700 1 -- 172.21.15.3:0/949568388 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c40098a0 con 0x5602a492c300 2023-09-10T01:16:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:16:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68d979b700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a4938f00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00bc20 tx=0x7f68cc00bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef32c1ef9ded6539 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68b8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.354+0000 7f68b8ff9700 1 -- 172.21.15.3:0/949568388 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00e040 con 0x5602a492c300 2023-09-10T01:16:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602a47d6490 con 0x5602a492c300 2023-09-10T01:16:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68b8ff9700 1 -- 172.21.15.3:0/949568388 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f68cc022070 con 0x5602a492c300 2023-09-10T01:16:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68b8ff9700 1 -- 172.21.15.3:0/949568388 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68cc0078f0 con 0x5602a492c300 2023-09-10T01:16:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68ef317b80 10 client.28601 fetch_fsmap learned FSMap version 36 2023-09-10T01:16:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5602a4693c00 con 0x5602a492c300 2023-09-10T01:16:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68b8ff9700 1 -- 172.21.15.3:0/949568388 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5602a47d6490 con 0x5602a492c300 2023-09-10T01:16:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68b8ff9700 1 -- 172.21.15.3:0/949568388 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f68cc00b050 con 0x5602a492c300 2023-09-10T01:16:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68ef317b80 10 client.28601 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:16:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68ef317b80 10 client.28601 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68ef317b80 20 client.28601 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.355+0000 7f68ef317b80 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a49464d0 0x5602a49488f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.356+0000 7f68ef317b80 4 client.28601 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.356+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602a4893840 con 0x5602a49464d0 2023-09-10T01:16:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.356+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a49464d0 0x5602a49488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.356+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a49464d0 0x5602a49488f0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94764e75745b324b server_cookie=d08539c87318fc2 in_seq=0 out_seq=0 2023-09-10T01:16:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.356+0000 7f68b8ff9700 10 client.28601 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68b8ff9700 1 -- 172.21.15.3:0/949568388 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5602a4893840 con 0x5602a49464d0 2023-09-10T01:16:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68b8ff9700 10 client.28601 handle_command_reply: tid=0 2023-09-10T01:16:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 client.28601 shutdown 2023-09-10T01:16:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 msgr2=0x5602a4938f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a4938f00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00bc20 tx=0x7f68cc00bd00 comp rx=0 tx=0).stop 2023-09-10T01:16:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 shutdown_connections 2023-09-10T01:16:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a49464d0 0x5602a49488f0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492c300 0x5602a4938f00 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 --2- 172.21.15.3:0/949568388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492ccc0 0x5602a4936180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 >> 172.21.15.3:0/949568388 conn(0x5602a48a1030 msgr2=0x5602a49267a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68b8ff9700 0 client.28601 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 shutdown_connections 2023-09-10T01:16:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.357+0000 7f68ef317b80 1 -- 172.21.15.3:0/949568388 wait complete. 2023-09-10T01:16:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.358+0000 7f68ef317b80 20 client.28601 trim_cache size 0 max 16384 2023-09-10T01:16:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.373+0000 7f68ef317b80 1 Processor -- start 2023-09-10T01:16:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.373+0000 7f68ef317b80 1 -- start start 2023-09-10T01:16:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a4938a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a4926f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a4927460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68ef317b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a4928930 con 0x5602a49321a0 2023-09-10T01:16:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68ef317b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a492d290 con 0x5602a492ccc0 2023-09-10T01:16:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68ef317b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a4931a20 con 0x5602a492c300 2023-09-10T01:16:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d8f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a4926f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a4938a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a4938a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41098/0 (socket says 172.21.15.3:41098) 2023-09-10T01:16:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d8f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a4926f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53490/0 (socket says 172.21.15.3:53490) 2023-09-10T01:16:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d979b700 1 -- 172.21.15.3:0/2587514013 learned_addr learned my addr 172.21.15.3:0/2587514013 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d9f9c700 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a4927460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d8f9a700 1 -- 172.21.15.3:0/2587514013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 msgr2=0x5602a4938a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a4938a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d8f9a700 1 -- 172.21.15.3:0/2587514013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 msgr2=0x5602a4927460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a4927460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d8f9a700 1 -- 172.21.15.3:0/2587514013 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602a46d11f0 con 0x5602a492ccc0 2023-09-10T01:16:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.374+0000 7f68d979b700 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a4938a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.375+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a4926f20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f68c80c91a0 tx=0x7f68c80cf810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee982cf85703f035 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.375+0000 7f68bbfff700 1 -- 172.21.15.3:0/2587514013 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8098cd0 con 0x5602a492ccc0 2023-09-10T01:16:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.375+0000 7f68bbfff700 1 -- 172.21.15.3:0/2587514013 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68c8098a40 con 0x5602a492ccc0 2023-09-10T01:16:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.375+0000 7f68bbfff700 1 -- 172.21.15.3:0/2587514013 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c80a0f70 con 0x5602a492ccc0 2023-09-10T01:16:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 -- 172.21.15.3:0/2587514013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 msgr2=0x5602a4926f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a4926f20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f68c80c91a0 tx=0x7f68c80cf810 comp rx=0 tx=0).stop 2023-09-10T01:16:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 -- 172.21.15.3:0/2587514013 shutdown_connections 2023-09-10T01:16:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a4938a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a4927460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 --2- 172.21.15.3:0/2587514013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a4926f20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 -- 172.21.15.3:0/2587514013 >> 172.21.15.3:0/2587514013 conn(0x5602a48a1030 msgr2=0x5602a49301f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 -- 172.21.15.3:0/2587514013 shutdown_connections 2023-09-10T01:16:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 -- 172.21.15.3:0/2587514013 wait complete. 2023-09-10T01:16:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 -- start start 2023-09-10T01:16:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.376+0000 7f68ef317b80 1 Processor -- start 2023-09-10T01:16:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a493b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a493bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68ef317b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49992e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68ef317b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a4999820 con 0x5602a49321a0 2023-09-10T01:16:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68ef317b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a4999960 con 0x5602a492ccc0 2023-09-10T01:16:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68ef317b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602a4999aa0 con 0x5602a492c300 2023-09-10T01:16:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68ef317b80 10 client.0 _ll_register_callbacks cb 0x7f68cc004d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68d979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a493b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68d8f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a493bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68d9f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68d979b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a493b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41108/0 (socket says 172.21.15.3:41108) 2023-09-10T01:16:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.377+0000 7f68d979b700 1 -- 172.21.15.3:0/879962626 learned_addr learned my addr 172.21.15.3:0/879962626 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68d8f9a700 1 -- 172.21.15.3:0/879962626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 msgr2=0x5602a493b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a493b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68d8f9a700 1 -- 172.21.15.3:0/879962626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 msgr2=0x5602a49992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68d8f9a700 1 -- 172.21.15.3:0/879962626 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602a4693c00 con 0x5602a492ccc0 2023-09-10T01:16:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68d8f9a700 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a493bc40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f68c8098bb0 tx=0x7f68c80108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7658339957ea45c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68b8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68b8ff9700 1 -- 172.21.15.3:0/879962626 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c80d4070 con 0x5602a492ccc0 2023-09-10T01:16:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68b8ff9700 1 -- 172.21.15.3:0/879962626 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f68c8099600 con 0x5602a492ccc0 2023-09-10T01:16:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68c4004cf0 con 0x5602a492ccc0 2023-09-10T01:16:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68b8ff9700 1 -- 172.21.15.3:0/879962626 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68c80a0f70 con 0x5602a492ccc0 2023-09-10T01:16:07.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68ef317b80 10 client.28607 fetch_fsmap learned FSMap version 36 2023-09-10T01:16:07.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7f68d41b2580 con 0x5602a492ccc0 2023-09-10T01:16:07.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.378+0000 7f68b8ff9700 1 -- 172.21.15.3:0/879962626 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68c4004cf0 con 0x5602a492ccc0 2023-09-10T01:16:07.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68b8ff9700 1 -- 172.21.15.3:0/879962626 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f68c80172c0 con 0x5602a492ccc0 2023-09-10T01:16:07.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68ef317b80 10 client.28607 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:16:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68ef317b80 10 client.28607 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68ef317b80 20 client.28607 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68ef317b80 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a499aef0 0x5602a499d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68ef317b80 4 client.28607 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:16:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5602a48778c0 con 0x5602a499aef0 2023-09-10T01:16:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68d979b700 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a499aef0 0x5602a499d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68d979b700 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a499aef0 0x5602a499d310 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15164999801b2a27 server_cookie=93504e472d47582c in_seq=0 out_seq=0 2023-09-10T01:16:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.379+0000 7f68b8ff9700 10 client.28607 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68b8ff9700 1 -- 172.21.15.3:0/879962626 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5602a48778c0 con 0x5602a499aef0 2023-09-10T01:16:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68b8ff9700 10 client.28607 handle_command_reply: tid=0 2023-09-10T01:16:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 client.28607 shutdown 2023-09-10T01:16:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 msgr2=0x5602a493bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a493bc40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f68c8098bb0 tx=0x7f68c80108e0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 shutdown_connections 2023-09-10T01:16:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5602a499aef0 0x5602a499d310 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5602a492c300 0x5602a493b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602a49321a0 0x5602a49992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68b8ff9700 0 client.28607 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 --2- 172.21.15.3:0/879962626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602a492ccc0 0x5602a493bc40 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 >> 172.21.15.3:0/879962626 conn(0x5602a48a1030 msgr2=0x5602a492b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 shutdown_connections 2023-09-10T01:16:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 1 -- 172.21.15.3:0/879962626 wait complete. 2023-09-10T01:16:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.445+0000 7f68ef317b80 20 client.28607 trim_cache size 0 max 16384 2023-09-10T01:16:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:16:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:16:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:16:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:16:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a08b0 msgr2=0x7f68bc0a2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a08b0 0x7f68bc0a2d70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68c400e630 tx=0x7f68c4000bc0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d410ed00 msgr2=0x7f68d41abad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d410ed00 0x7f68d41abad0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f68c800b500 tx=0x7f68c800b8c0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 shutdown_connections 2023-09-10T01:16:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f68bc0a08b0 0x7f68bc0a2d70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f68d4107600 0x7f68d41ab590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68d410f170 0x7f68d41a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 --2- 172.21.15.3:0/4062109798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68d410ed00 0x7f68d41abad0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 >> 172.21.15.3:0/4062109798 conn(0x7f68d406c6c0 msgr2=0x7f68d4070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 shutdown_connections 2023-09-10T01:16:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:07.446+0000 7f68db9ff700 1 -- 172.21.15.3:0/4062109798 wait complete. 2023-09-10T01:16:07.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1027355 7.44 MB/sec execute 589 sec latency 2166.736 ms 2023-09-10T01:16:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:08 smithi003 ceph-mon[97654]: pgmap v403: 129 pgs: 129 active+clean; 328 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.9 MiB/s wr, 203 op/s 2023-09-10T01:16:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:08 smithi136 ceph-mon[133708]: pgmap v403: 129 pgs: 129 active+clean; 328 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.9 MiB/s wr, 203 op/s 2023-09-10T01:16:08.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:08 smithi092 ceph-mon[134067]: pgmap v403: 129 pgs: 129 active+clean; 328 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.9 MiB/s wr, 203 op/s 2023-09-10T01:16:08.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1028247 7.44 MB/sec execute 590 sec latency 2457.395 ms 2023-09-10T01:16:09.813 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1030227 7.45 MB/sec execute 591 sec latency 19.963 ms 2023-09-10T01:16:10.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:10 smithi003 ceph-mon[97654]: pgmap v404: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2023-09-10T01:16:10.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:10 smithi003 ceph-mon[97654]: osdmap e105: 12 total, 12 up, 12 in 2023-09-10T01:16:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:10 smithi136 ceph-mon[133708]: pgmap v404: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2023-09-10T01:16:10.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:10 smithi136 ceph-mon[133708]: osdmap e105: 12 total, 12 up, 12 in 2023-09-10T01:16:10.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:10 smithi092 ceph-mon[134067]: pgmap v404: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.7 MiB/s wr, 179 op/s 2023-09-10T01:16:10.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:10 smithi092 ceph-mon[134067]: osdmap e105: 12 total, 12 up, 12 in 2023-09-10T01:16:10.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1031158 7.44 MB/sec execute 592 sec latency 16.679 ms 2023-09-10T01:16:11.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1032827 7.44 MB/sec execute 593 sec latency 18.937 ms 2023-09-10T01:16:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:12 smithi003 ceph-mon[97654]: pgmap v406: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2023-09-10T01:16:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:12 smithi136 ceph-mon[133708]: pgmap v406: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2023-09-10T01:16:12.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:12 smithi092 ceph-mon[134067]: pgmap v406: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2023-09-10T01:16:12.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1034138 7.44 MB/sec execute 594 sec latency 28.367 ms 2023-09-10T01:16:13.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1035203 7.44 MB/sec execute 595 sec latency 16.425 ms 2023-09-10T01:16:14.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:14 smithi003 ceph-mon[97654]: pgmap v407: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 794 KiB/s wr, 81 op/s 2023-09-10T01:16:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:14 smithi136 ceph-mon[133708]: pgmap v407: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 794 KiB/s wr, 81 op/s 2023-09-10T01:16:14.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:14 smithi092 ceph-mon[134067]: pgmap v407: 129 pgs: 129 active+clean; 320 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 794 KiB/s wr, 81 op/s 2023-09-10T01:16:14.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1036761 7.44 MB/sec execute 596 sec latency 150.358 ms 2023-09-10T01:16:15.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1037941 7.44 MB/sec execute 597 sec latency 164.526 ms 2023-09-10T01:16:16.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:16 smithi003 ceph-mon[97654]: pgmap v408: 129 pgs: 129 active+clean; 326 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.8 MiB/s wr, 163 op/s 2023-09-10T01:16:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:16 smithi136 ceph-mon[133708]: pgmap v408: 129 pgs: 129 active+clean; 326 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.8 MiB/s wr, 163 op/s 2023-09-10T01:16:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:16 smithi092 ceph-mon[134067]: pgmap v408: 129 pgs: 129 active+clean; 326 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.8 MiB/s wr, 163 op/s 2023-09-10T01:16:16.812 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1039656 7.44 MB/sec execute 598 sec latency 20.067 ms 2023-09-10T01:16:17.813 INFO:tasks.workunit.client.0.smithi003.stdout: 1 1041094 7.44 MB/sec execute 599 sec latency 67.205 ms 2023-09-10T01:16:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:18 smithi003 ceph-mon[97654]: pgmap v409: 129 pgs: 129 active+clean; 326 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.8 MiB/s wr, 163 op/s 2023-09-10T01:16:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:18 smithi136 ceph-mon[133708]: pgmap v409: 129 pgs: 129 active+clean; 326 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.8 MiB/s wr, 163 op/s 2023-09-10T01:16:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:18 smithi092 ceph-mon[134067]: pgmap v409: 129 pgs: 129 active+clean; 326 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.8 MiB/s wr, 163 op/s 2023-09-10T01:16:18.813 INFO:tasks.workunit.client.0.smithi003.stdout: 1 cleanup 600 sec 2023-09-10T01:16:19.036 INFO:tasks.workunit.client.0.smithi003.stdout: 0 cleanup 600 sec 2023-09-10T01:16:19.036 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-09-10T01:16:19.036 INFO:tasks.workunit.client.0.smithi003.stdout: Operation Count AvgLat MaxLat 2023-09-10T01:16:19.037 INFO:tasks.workunit.client.0.smithi003.stdout: ---------------------------------------- 2023-09-10T01:16:19.037 INFO:tasks.workunit.client.0.smithi003.stdout: NTCreateX 142033 1.936 4951.398 2023-09-10T01:16:19.037 INFO:tasks.workunit.client.0.smithi003.stdout: Close 104311 0.004 0.204 2023-09-10T01:16:19.037 INFO:tasks.workunit.client.0.smithi003.stdout: Rename 6011 6.819 4955.002 2023-09-10T01:16:19.037 INFO:tasks.workunit.client.0.smithi003.stdout: Unlink 28707 1.727 272.171 2023-09-10T01:16:19.038 INFO:tasks.workunit.client.0.smithi003.stdout: Deltree 4 141.063 156.647 2023-09-10T01:16:19.038 INFO:tasks.workunit.client.0.smithi003.stdout: Mkdir 2 0.021 0.026 2023-09-10T01:16:19.038 INFO:tasks.workunit.client.0.smithi003.stdout: Qpathinfo 128709 1.168 2222.441 2023-09-10T01:16:19.038 INFO:tasks.workunit.client.0.smithi003.stdout: Qfileinfo 22560 0.002 0.062 2023-09-10T01:16:19.038 INFO:tasks.workunit.client.0.smithi003.stdout: Qfsinfo 23605 0.212 6.339 2023-09-10T01:16:19.039 INFO:tasks.workunit.client.0.smithi003.stdout: Sfileinfo 11571 0.024 4.056 2023-09-10T01:16:19.039 INFO:tasks.workunit.client.0.smithi003.stdout: Find 49765 0.247 68.771 2023-09-10T01:16:19.039 INFO:tasks.workunit.client.0.smithi003.stdout: WriteX 70868 0.163 604.812 2023-09-10T01:16:19.039 INFO:tasks.workunit.client.0.smithi003.stdout: ReadX 222483 0.009 10.911 2023-09-10T01:16:19.039 INFO:tasks.workunit.client.0.smithi003.stdout: LockX 462 0.675 2.783 2023-09-10T01:16:19.039 INFO:tasks.workunit.client.0.smithi003.stdout: UnlockX 462 0.612 2.122 2023-09-10T01:16:19.040 INFO:tasks.workunit.client.0.smithi003.stdout: Flush 9966 4.866 286.554 2023-09-10T01:16:19.040 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-09-10T01:16:19.040 INFO:tasks.workunit.client.0.smithi003.stdout:Throughput 7.44457 MB/sec 1 clients 1 procs max_latency=4955.026 ms 2023-09-10T01:16:19.040 INFO:tasks.workunit.client.0.smithi003.stderr:+ dbench 10 2023-09-10T01:16:19.041 INFO:tasks.workunit.client.0.smithi003.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2023-09-10T01:16:19.041 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-09-10T01:16:19.041 INFO:tasks.workunit.client.0.smithi003.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2023-09-10T01:16:19.042 INFO:tasks.workunit.client.0.smithi003.stdout:9 of 10 processes prepared for launch 0 sec 2023-09-10T01:16:19.141 INFO:tasks.workunit.client.0.smithi003.stdout:10 of 10 processes prepared for launch 0 sec 2023-09-10T01:16:19.142 INFO:tasks.workunit.client.0.smithi003.stdout:releasing clients 2023-09-10T01:16:20.141 INFO:tasks.workunit.client.0.smithi003.stdout: 10 370 132.26 MB/sec warmup 1 sec latency 49.268 ms 2023-09-10T01:16:20.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:20 smithi003 ceph-mon[97654]: pgmap v410: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2023-09-10T01:16:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 -- 172.21.15.3:0/4271930776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 msgr2=0x7fa968106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/4271930776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 0x7fa968106a60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa9580098f0 tx=0x7fa958009c00 comp rx=0 tx=0).stop 2023-09-10T01:16:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 -- 172.21.15.3:0/4271930776 shutdown_connections 2023-09-10T01:16:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/4271930776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 0x7fa968106a60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/4271930776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa968100f80 0x7fa96810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/4271930776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9681005c0 0x7fa968100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 -- 172.21.15.3:0/4271930776 >> 172.21.15.3:0/4271930776 conn(0x7fa9680fc0b0 msgr2=0x7fa9680fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 -- 172.21.15.3:0/4271930776 shutdown_connections 2023-09-10T01:16:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.615+0000 7fa96d9ff700 1 -- 172.21.15.3:0/4271930776 wait complete. 2023-09-10T01:16:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.617+0000 7fa96d9ff700 1 Processor -- start 2023-09-10T01:16:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.617+0000 7fa96d9ff700 1 -- start start 2023-09-10T01:16:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.617+0000 7fa96d9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9681005c0 0x7fa968196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.617+0000 7fa96d9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa968100f80 0x7fa968196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.617+0000 7fa96d9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 0x7fa96819b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa96d9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96819b640 con 0x7fa9681005c0 2023-09-10T01:16:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa96d9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96819b7b0 con 0x7fa968100f80 2023-09-10T01:16:20.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa96d9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96819b970 con 0x7fa968106680 2023-09-10T01:16:20.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa968100f80 0x7fa968196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa968100f80 0x7fa968196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43872/0 (socket says 172.21.15.3:43872) 2023-09-10T01:16:20.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 -- 172.21.15.3:0/175493140 learned_addr learned my addr 172.21.15.3:0/175493140 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:20.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa966ffd700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9681005c0 0x7fa968196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9677fe700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 0x7fa96819b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 -- 172.21.15.3:0/175493140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 msgr2=0x7fa96819b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 0x7fa96819b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 -- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9681005c0 msgr2=0x7fa968196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9681005c0 0x7fa968196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.618+0000 7fa9667fc700 1 -- 172.21.15.3:0/175493140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0095f0 con 0x7fa968100f80 2023-09-10T01:16:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.619+0000 7fa9677fe700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 0x7fa96819b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.619+0000 7fa9667fc700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa968100f80 0x7fa968196ba0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00ec10 tx=0x7fa95c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.619+0000 7fa96c9fd700 1 -- 172.21.15.3:0/175493140 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00cd40 con 0x7fa968100f80 2023-09-10T01:16:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.619+0000 7fa96c9fd700 1 -- 172.21.15.3:0/175493140 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa95c0044e0 con 0x7fa968100f80 2023-09-10T01:16:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.619+0000 7fa96c9fd700 1 -- 172.21.15.3:0/175493140 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c010690 con 0x7fa968100f80 2023-09-10T01:16:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.619+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa958009520 con 0x7fa968100f80 2023-09-10T01:16:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.619+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9681a32b0 con 0x7fa968100f80 2023-09-10T01:16:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.623+0000 7fa96c9fd700 1 -- 172.21.15.3:0/175493140 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa95c003710 con 0x7fa968100f80 2023-09-10T01:16:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.624+0000 7fa981297b80 1 Processor -- start 2023-09-10T01:16:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa981297b80 1 -- start start 2023-09-10T01:16:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa981297b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8831231a0 0x55b883123580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa981297b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b88311d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa981297b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa981297b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8830766d0 con 0x55b88311dcc0 2023-09-10T01:16:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa96c9fd700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa9500a4cc0 0x7fa9500a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa981297b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b883073ff0 con 0x55b88311d300 2023-09-10T01:16:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa981297b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88306f8e0 con 0x55b8831231a0 2023-09-10T01:16:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.625+0000 7fa96c9fd700 1 -- 172.21.15.3:0/175493140 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa95c014070 con 0x7fa968100f80 2023-09-10T01:16:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8831231a0 0x55b883123580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8831231a0 0x55b883123580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:36094/0 (socket says 172.21.15.3:36094) 2023-09-10T01:16:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 -- 172.21.15.3:0/731406562 learned_addr learned my addr 172.21.15.3:0/731406562 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa966ffd700 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b88311d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa9500a4cc0 0x7fa9500a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9667fc700 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 -- 172.21.15.3:0/731406562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 msgr2=0x55b88311d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b88311d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 -- 172.21.15.3:0/731406562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 msgr2=0x55b883129f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 -- 172.21.15.3:0/731406562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b882e9bd80 con 0x55b8831231a0 2023-09-10T01:16:20.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa966ffd700 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b88311d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:20.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.626+0000 7fa9677fe700 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8831231a0 0x55b883123580 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa960012cf0 tx=0x7fa960017350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba22466dd7f571ea server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:20.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.627+0000 7fa94dffb700 1 -- 172.21.15.3:0/731406562 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96000e040 con 0x55b8831231a0 2023-09-10T01:16:20.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.627+0000 7fa94dffb700 1 -- 172.21.15.3:0/731406562 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa960009360 con 0x55b8831231a0 2023-09-10T01:16:20.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.627+0000 7fa94dffb700 1 -- 172.21.15.3:0/731406562 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960019a10 con 0x55b8831231a0 2023-09-10T01:16:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.627+0000 7fa981297b80 1 -- 172.21.15.3:0/731406562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8831231a0 msgr2=0x55b883123580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.627+0000 7fa9677fe700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa9500a4cc0 0x7fa9500a7180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa960021000 tx=0x7fa96001c000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.627+0000 7fa981297b80 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8831231a0 0x55b883123580 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa960012cf0 tx=0x7fa960017350 comp rx=0 tx=0).stop 2023-09-10T01:16:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 -- 172.21.15.3:0/731406562 shutdown_connections 2023-09-10T01:16:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b8831231a0 0x55b883123580 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 --2- 172.21.15.3:0/731406562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b88311d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 -- 172.21.15.3:0/731406562 >> 172.21.15.3:0/731406562 conn(0x55b883092030 msgr2=0x55b883118560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 -- 172.21.15.3:0/731406562 shutdown_connections 2023-09-10T01:16:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 -- 172.21.15.3:0/731406562 wait complete. 2023-09-10T01:16:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.628+0000 7fa981297b80 1 -- start start 2023-09-10T01:16:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 1 Processor -- start 2023-09-10T01:16:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b883129e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311dcc0 0x55b8831270e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8831231a0 0x55b8831276f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88300db10 con 0x55b8831231a0 2023-09-10T01:16:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88300d810 con 0x55b88311d300 2023-09-10T01:16:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b882ef18c0 con 0x55b88311dcc0 2023-09-10T01:16:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.629+0000 7fa981297b80 10 client.0 _ll_register_callbacks cb 0x55b882f57a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa966ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b883129e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9667fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311dcc0 0x55b8831270e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa966ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b883129e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:43892/0 (socket says 172.21.15.3:43892) 2023-09-10T01:16:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa966ffd700 1 -- 172.21.15.3:0/3535475973 learned_addr learned my addr 172.21.15.3:0/3535475973 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9677fe700 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8831231a0 0x55b8831276f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9667fc700 1 -- 172.21.15.3:0/3535475973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 msgr2=0x55b883129e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9667fc700 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b883129e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9667fc700 1 -- 172.21.15.3:0/3535475973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8831231a0 msgr2=0x55b8831276f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9667fc700 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8831231a0 0x55b8831276f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9667fc700 1 -- 172.21.15.3:0/3535475973 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96810dd40 con 0x55b88311dcc0 2023-09-10T01:16:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9677fe700 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8831231a0 0x55b8831276f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa9667fc700 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311dcc0 0x55b8831270e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0c9c20 tx=0x7fa95c099360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=401739f5e960a9cd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.630+0000 7fa946ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:16:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa946ffd700 1 -- 172.21.15.3:0/3535475973 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c049ca0 con 0x55b88311dcc0 2023-09-10T01:16:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b882fc7490 con 0x55b88311dcc0 2023-09-10T01:16:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa946ffd700 1 -- 172.21.15.3:0/3535475973 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa95c099980 con 0x55b88311dcc0 2023-09-10T01:16:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa946ffd700 1 -- 172.21.15.3:0/3535475973 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa95c09d7b0 con 0x55b88311dcc0 2023-09-10T01:16:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 10 client.28501 fetch_fsmap learned FSMap version 36 2023-09-10T01:16:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7fa9600128a0 con 0x55b88311dcc0 2023-09-10T01:16:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa946ffd700 1 -- 172.21.15.3:0/3535475973 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b882fc7490 con 0x55b88311dcc0 2023-09-10T01:16:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa946ffd700 1 -- 172.21.15.3:0/3535475973 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa95c098e70 con 0x55b88311dcc0 2023-09-10T01:16:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 10 client.28501 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:16:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 10 client.28501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 20 client.28501 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b883137490 0x55b8831398b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 4 client.28501 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.631+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b883084840 con 0x55b883137490 2023-09-10T01:16:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.632+0000 7fa966ffd700 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b883137490 0x55b8831398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.632+0000 7fa966ffd700 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b883137490 0x55b8831398b0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfaaf524a2e0419d server_cookie=6a3d8999b3350817 in_seq=0 out_seq=0 2023-09-10T01:16:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.632+0000 7fa946ffd700 10 client.28501 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.636+0000 7fa946ffd700 1 -- 172.21.15.3:0/3535475973 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b883084840 con 0x55b883137490 2023-09-10T01:16:20.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.636+0000 7fa946ffd700 10 client.28501 handle_command_reply: tid=0 2023-09-10T01:16:20.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.636+0000 7fa981297b80 1 client.28501 shutdown 2023-09-10T01:16:20.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311dcc0 msgr2=0x55b8831270e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:20.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311dcc0 0x55b8831270e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa95c0c9c20 tx=0x7fa95c099360 comp rx=0 tx=0).stop 2023-09-10T01:16:20.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 shutdown_connections 2023-09-10T01:16:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b883137490 0x55b8831398b0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b8831231a0 0x55b8831276f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa946ffd700 0 client.28501 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b88311d300 0x55b883129e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 --2- 172.21.15.3:0/3535475973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311dcc0 0x55b8831270e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 >> 172.21.15.3:0/3535475973 conn(0x55b883092030 msgr2=0x55b8831177a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 shutdown_connections 2023-09-10T01:16:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 1 -- 172.21.15.3:0/3535475973 wait complete. 2023-09-10T01:16:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.637+0000 7fa981297b80 20 client.28501 trim_cache size 0 max 16384 2023-09-10T01:16:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 Processor -- start 2023-09-10T01:16:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 -- start start 2023-09-10T01:16:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b883124d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b883117f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8831229f0 con 0x55b88311dcc0 2023-09-10T01:16:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b883127cf0 con 0x55b8831231a0 2023-09-10T01:16:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.655+0000 7fa981297b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b883139f10 con 0x55b88311d300 2023-09-10T01:16:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa966ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b883124d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9667fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9677fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b883117f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa966ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b883124d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:36114/0 (socket says 172.21.15.3:36114) 2023-09-10T01:16:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa966ffd700 1 -- 172.21.15.3:0/2154042539 learned_addr learned my addr 172.21.15.3:0/2154042539 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9677fe700 1 -- 172.21.15.3:0/2154042539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 msgr2=0x55b883124d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9677fe700 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b883124d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9677fe700 1 -- 172.21.15.3:0/2154042539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 msgr2=0x55b883129990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9677fe700 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9677fe700 1 -- 172.21.15.3:0/2154042539 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b882e84c00 con 0x55b8831231a0 2023-09-10T01:16:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa966ffd700 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b883124d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:20.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.656+0000 7fa9667fc700 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.657+0000 7fa9677fe700 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b883117f20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa960009070 tx=0x7fa960027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea8e06effa9e941b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.657+0000 7fa94e7fc700 1 -- 172.21.15.3:0/2154042539 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96000e030 con 0x55b8831231a0 2023-09-10T01:16:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.657+0000 7fa94e7fc700 1 -- 172.21.15.3:0/2154042539 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa960012680 con 0x55b8831231a0 2023-09-10T01:16:20.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.657+0000 7fa94e7fc700 1 -- 172.21.15.3:0/2154042539 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960003d80 con 0x55b8831231a0 2023-09-10T01:16:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 -- 172.21.15.3:0/2154042539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 msgr2=0x55b883117f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b883117f20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa960009070 tx=0x7fa960027000 comp rx=0 tx=0).stop 2023-09-10T01:16:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 -- 172.21.15.3:0/2154042539 shutdown_connections 2023-09-10T01:16:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b883124d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b883117f20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 --2- 172.21.15.3:0/2154042539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b883129990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 -- 172.21.15.3:0/2154042539 >> 172.21.15.3:0/2154042539 conn(0x55b883092030 msgr2=0x55b88311cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 -- 172.21.15.3:0/2154042539 shutdown_connections 2023-09-10T01:16:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 -- 172.21.15.3:0/2154042539 wait complete. 2023-09-10T01:16:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.658+0000 7fa981297b80 1 -- start start 2023-09-10T01:16:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 1 Processor -- start 2023-09-10T01:16:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b88312e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b88312c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b88312cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88312d050 con 0x55b88311dcc0 2023-09-10T01:16:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88312d1c0 con 0x55b8831231a0 2023-09-10T01:16:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88312d380 con 0x55b88311d300 2023-09-10T01:16:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa981297b80 10 client.0 _ll_register_callbacks cb 0x7fa95c099770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa9667fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b88312c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa966ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b88312e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa9667fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b88312c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45206/0 (socket says 172.21.15.3:45206) 2023-09-10T01:16:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.659+0000 7fa9667fc700 1 -- 172.21.15.3:0/2414690562 learned_addr learned my addr 172.21.15.3:0/2414690562 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa9677fe700 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b88312cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:20 smithi136 ceph-mon[133708]: pgmap v410: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2023-09-10T01:16:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa966ffd700 1 -- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 msgr2=0x55b88312cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa966ffd700 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b88312cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa966ffd700 1 -- 172.21.15.3:0/2414690562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 msgr2=0x55b88312c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa966ffd700 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b88312c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa966ffd700 1 -- 172.21.15.3:0/2414690562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95800f820 con 0x55b88311d300 2023-09-10T01:16:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa9667fc700 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b88312c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa966ffd700 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b88312e350 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa958009810 tx=0x7fa958005900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84d93c9a90d0e7f9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.660+0000 7fa946ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:16:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa946ffd700 1 -- 172.21.15.3:0/2414690562 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958027d90 con 0x55b88311d300 2023-09-10T01:16:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa946ffd700 1 -- 172.21.15.3:0/2414690562 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fa9580270d0 con 0x55b88311d300 2023-09-10T01:16:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa960009360 con 0x55b88311d300 2023-09-10T01:16:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa946ffd700 1 -- 172.21.15.3:0/2414690562 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa958027290 con 0x55b88311d300 2023-09-10T01:16:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa981297b80 10 client.28507 fetch_fsmap learned FSMap version 36 2023-09-10T01:16:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7fa9600128a0 con 0x55b88311d300 2023-09-10T01:16:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa946ffd700 1 -- 172.21.15.3:0/2414690562 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa960009360 con 0x55b88311d300 2023-09-10T01:16:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa946ffd700 1 -- 172.21.15.3:0/2414690562 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 36) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa958009920 con 0x55b88311d300 2023-09-10T01:16:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa981297b80 10 client.28507 fetch_fsmap finished waiting for FSMap version 36 2023-09-10T01:16:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa981297b80 10 client.28507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa981297b80 20 client.28507 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.661+0000 7fa981297b80 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b88318c840 0x55b88318ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.662+0000 7fa981297b80 4 client.28507 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:16:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.662+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b8830688c0 con 0x55b88318c840 2023-09-10T01:16:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.662+0000 7fa9667fc700 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b88318c840 0x55b88318ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.662+0000 7fa946ffd700 10 client.28507 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:20.662+0000 7fa9667fc700 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b88318c840 0x55b88318ec60 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60307af81cf21e99 server_cookie=15690fe2737fd696 in_seq=0 out_seq=0 2023-09-10T01:16:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:20 smithi092 ceph-mon[134067]: pgmap v410: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 200 op/s 2023-09-10T01:16:21.142 INFO:tasks.workunit.client.0.smithi003.stdout: 10 638 100.53 MB/sec warmup 2 sec latency 1032.444 ms 2023-09-10T01:16:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.171+0000 7fa946ffd700 1 -- 172.21.15.3:0/2414690562 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b8830688c0 con 0x55b88318c840 2023-09-10T01:16:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.171+0000 7fa946ffd700 10 client.28507 handle_command_reply: tid=0 2023-09-10T01:16:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.171+0000 7fa981297b80 1 client.28507 shutdown 2023-09-10T01:16:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 msgr2=0x55b88312e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b88312e350 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa958009810 tx=0x7fa958005900 comp rx=0 tx=0).stop 2023-09-10T01:16:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 shutdown_connections 2023-09-10T01:16:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55b88318c840 0x55b88318ec60 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b88311d300 0x55b88312e350 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b8831231a0 0x55b88312cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 --2- 172.21.15.3:0/2414690562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b88311dcc0 0x55b88312c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 >> 172.21.15.3:0/2414690562 conn(0x55b883092030 msgr2=0x55b883125890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.172+0000 7fa946ffd700 0 client.28507 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.173+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 shutdown_connections 2023-09-10T01:16:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.173+0000 7fa981297b80 1 -- 172.21.15.3:0/2414690562 wait complete. 2023-09-10T01:16:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:16:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:16:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:16:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:16:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.173+0000 7fa981297b80 20 client.28507 trim_cache size 0 max 16384 2023-09-10T01:16:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.174+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa9500a4cc0 msgr2=0x7fa9500a7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.174+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa9500a4cc0 0x7fa9500a7180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa960021000 tx=0x7fa96001c000 comp rx=0 tx=0).stop 2023-09-10T01:16:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.174+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa968100f80 msgr2=0x7fa968196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.174+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa968100f80 0x7fa968196ba0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00ec10 tx=0x7fa95c00ef20 comp rx=0 tx=0).stop 2023-09-10T01:16:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 shutdown_connections 2023-09-10T01:16:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa9500a4cc0 0x7fa9500a7180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa968106680 0x7fa96819b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9681005c0 0x7fa968196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 --2- 172.21.15.3:0/175493140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa968100f80 0x7fa968196ba0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 >> 172.21.15.3:0/175493140 conn(0x7fa9680fc0b0 msgr2=0x7fa9680fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 shutdown_connections 2023-09-10T01:16:21.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:21.175+0000 7fa96d9ff700 1 -- 172.21.15.3:0/175493140 wait complete. 2023-09-10T01:16:22.142 INFO:tasks.workunit.client.0.smithi003.stdout: 10 1001 72.33 MB/sec warmup 3 sec latency 2032.517 ms 2023-09-10T01:16:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:22 smithi003 ceph-mon[97654]: pgmap v411: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 196 op/s 2023-09-10T01:16:22.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:22 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:16:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:22 smithi136 ceph-mon[133708]: pgmap v411: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 196 op/s 2023-09-10T01:16:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:22 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:16:22.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:22 smithi092 ceph-mon[134067]: pgmap v411: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 196 op/s 2023-09-10T01:16:22.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:22 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:16:23.142 INFO:tasks.workunit.client.0.smithi003.stdout: 10 1365 61.29 MB/sec warmup 4 sec latency 3032.702 ms 2023-09-10T01:16:24.142 INFO:tasks.workunit.client.0.smithi003.stdout: 10 1836 50.34 MB/sec warmup 5 sec latency 3074.975 ms 2023-09-10T01:16:24.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:24 smithi003 ceph-mon[97654]: pgmap v412: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 166 op/s 2023-09-10T01:16:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:24 smithi136 ceph-mon[133708]: pgmap v412: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 166 op/s 2023-09-10T01:16:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:24 smithi092 ceph-mon[134067]: pgmap v412: 129 pgs: 129 active+clean; 336 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 166 op/s 2023-09-10T01:16:25.142 INFO:tasks.workunit.client.0.smithi003.stdout: 10 2400 46.99 MB/sec warmup 6 sec latency 131.078 ms 2023-09-10T01:16:26.142 INFO:tasks.workunit.client.0.smithi003.stdout: 10 3064 48.38 MB/sec warmup 7 sec latency 116.873 ms 2023-09-10T01:16:26.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:26 smithi003 ceph-mon[97654]: pgmap v413: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 495 op/s 2023-09-10T01:16:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:26 smithi136 ceph-mon[133708]: pgmap v413: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 495 op/s 2023-09-10T01:16:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:26 smithi092 ceph-mon[134067]: pgmap v413: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 495 op/s 2023-09-10T01:16:27.142 INFO:tasks.workunit.client.0.smithi003.stdout: 10 3523 47.60 MB/sec warmup 8 sec latency 54.230 ms 2023-09-10T01:16:27.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:16:27.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.482+0000 7f7c59fff700 1 -- 172.21.15.3:0/436738623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c54105630 msgr2=0x7f7c54105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.482+0000 7f7c59fff700 1 --2- 172.21.15.3:0/436738623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c54105630 0x7f7c54105a10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009990 tx=0x7f7c3c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.483+0000 7f7c59fff700 1 -- 172.21.15.3:0/436738623 shutdown_connections 2023-09-10T01:16:27.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.483+0000 7f7c59fff700 1 --2- 172.21.15.3:0/436738623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c540fff30 0x7f7c540678d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.483+0000 7f7c59fff700 1 --2- 172.21.15.3:0/436738623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c54105630 0x7f7c54105a10 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.483+0000 7f7c59fff700 1 --2- 172.21.15.3:0/436738623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c540ff570 0x7f7c540ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.483+0000 7f7c59fff700 1 -- 172.21.15.3:0/436738623 >> 172.21.15.3:0/436738623 conn(0x7f7c54076710 msgr2=0x7f7c54076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:27.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.483+0000 7f7c59fff700 1 -- 172.21.15.3:0/436738623 shutdown_connections 2023-09-10T01:16:27.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.483+0000 7f7c59fff700 1 -- 172.21.15.3:0/436738623 wait complete. 2023-09-10T01:16:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.485+0000 7f7c59fff700 1 Processor -- start 2023-09-10T01:16:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.485+0000 7f7c59fff700 1 -- start start 2023-09-10T01:16:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.486+0000 7f7c59fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c540ff570 0x7f7c54199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.486+0000 7f7c59fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c540fff30 0x7f7c54199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.486+0000 7f7c59fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54105630 0x7f7c5410cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.486+0000 7f7c59fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5410d3d0 con 0x7f7c540fff30 2023-09-10T01:16:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.486+0000 7f7c59fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5410d540 con 0x7f7c540ff570 2023-09-10T01:16:27.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.486+0000 7f7c59fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5410d700 con 0x7f7c54105630 2023-09-10T01:16:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.486+0000 7f7c52ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c540fff30 0x7f7c54199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.487+0000 7f7c52ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c540fff30 0x7f7c54199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43122/0 (socket says 172.21.15.3:43122) 2023-09-10T01:16:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.487+0000 7f7c52ffd700 1 -- 172.21.15.3:0/3695670716 learned_addr learned my addr 172.21.15.3:0/3695670716 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.487+0000 7f7c53fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54105630 0x7f7c5410cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.487+0000 7f7c537fe700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c540ff570 0x7f7c54199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.490+0000 7f7c53fff700 1 -- 172.21.15.3:0/3695670716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c540ff570 msgr2=0x7f7c54199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.490+0000 7f7c53fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c540ff570 0x7f7c54199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.490+0000 7f7c53fff700 1 -- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c540fff30 msgr2=0x7f7c54199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.490+0000 7f7c53fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c540fff30 0x7f7c54199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.490+0000 7f7c53fff700 1 -- 172.21.15.3:0/3695670716 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c0095f0 con 0x7f7c54105630 2023-09-10T01:16:27.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.492+0000 7f7c53fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54105630 0x7f7c5410cc40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00d720 tx=0x7f7c4c00da30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:27.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.493+0000 7f7c50ff9700 1 -- 172.21.15.3:0/3695670716 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c004220 con 0x7f7c54105630 2023-09-10T01:16:27.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.493+0000 7f7c50ff9700 1 -- 172.21.15.3:0/3695670716 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c4c004cf0 con 0x7f7c54105630 2023-09-10T01:16:27.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.493+0000 7f7c50ff9700 1 -- 172.21.15.3:0/3695670716 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00f690 con 0x7f7c54105630 2023-09-10T01:16:27.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.493+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5410d980 con 0x7f7c54105630 2023-09-10T01:16:27.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.493+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c541a1330 con 0x7f7c54105630 2023-09-10T01:16:27.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.499+0000 7f7c50ff9700 1 -- 172.21.15.3:0/3695670716 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7c4c004410 con 0x7f7c54105630 2023-09-10T01:16:27.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.502+0000 7f7c50ff9700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c340a09c0 0x7f7c340a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.502+0000 7f7c50ff9700 1 -- 172.21.15.3:0/3695670716 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7c4c0cc030 con 0x7f7c54105630 2023-09-10T01:16:27.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.503+0000 7f7c537fe700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c340a09c0 0x7f7c340a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.504+0000 7f7c6d8bcb80 1 Processor -- start 2023-09-10T01:16:27.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.504+0000 7f7c6d8bcb80 1 -- start start 2023-09-10T01:16:27.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.504+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d26fa730 con 0x5612d27a1710 2023-09-10T01:16:27.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d26f8050 con 0x5612d27abe40 2023-09-10T01:16:27.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d26f3940 con 0x5612d27a2030 2023-09-10T01:16:27.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c537fe700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c340a09c0 0x7f7c340a2e80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c005b40 tx=0x7f7c3c005a90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:27.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43128/0 (socket says 172.21.15.3:43128) 2023-09-10T01:16:27.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.505+0000 7f7c537fe700 1 -- 172.21.15.3:0/1701974839 learned_addr learned my addr 172.21.15.3:0/1701974839 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:27.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.506+0000 7f7c53fff700 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.508+0000 7f7c53fff700 1 -- 172.21.15.3:0/1701974839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 msgr2=0x5612d27ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.508+0000 7f7c53fff700 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.508+0000 7f7c53fff700 1 -- 172.21.15.3:0/1701974839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 msgr2=0x5612d27a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.508+0000 7f7c53fff700 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.508+0000 7f7c53fff700 1 -- 172.21.15.3:0/1701974839 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612d251fde0 con 0x5612d27abe40 2023-09-10T01:16:27.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.508+0000 7f7c53fff700 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27ae230 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0c9330 tx=0x7f7c4c098bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44fc9a3f33f42092 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:27.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.509+0000 7f7c49ffb700 1 -- 172.21.15.3:0/1701974839 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c099090 con 0x5612d27abe40 2023-09-10T01:16:27.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.509+0000 7f7c49ffb700 1 -- 172.21.15.3:0/1701974839 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c4c09cec0 con 0x5612d27abe40 2023-09-10T01:16:27.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.509+0000 7f7c49ffb700 1 -- 172.21.15.3:0/1701974839 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0992f0 con 0x5612d27abe40 2023-09-10T01:16:27.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/1701974839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 msgr2=0x5612d27ae230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27ae230 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0c9330 tx=0x7f7c4c098bd0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/1701974839 shutdown_connections 2023-09-10T01:16:27.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27ae230 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/1701974839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/1701974839 >> 172.21.15.3:0/1701974839 conn(0x5612d2716090 msgr2=0x5612d279cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:27.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.510+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/1701974839 shutdown_connections 2023-09-10T01:16:27.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.511+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/1701974839 wait complete. 2023-09-10T01:16:27.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.511+0000 7f7c6d8bcb80 1 -- start start 2023-09-10T01:16:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.512+0000 7f7c6d8bcb80 1 Processor -- start 2023-09-10T01:16:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27a9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d2691b70 con 0x5612d27a1710 2023-09-10T01:16:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d2691870 con 0x5612d27abe40 2023-09-10T01:16:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d2575920 con 0x5612d27a2030 2023-09-10T01:16:27.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c6d8bcb80 10 client.0 _ll_register_callbacks cb 0x5612d25dba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:27.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c52ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c52ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27a8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45446/0 (socket says 172.21.15.3:45446) 2023-09-10T01:16:27.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c52ffd700 1 -- 172.21.15.3:0/121983248 learned_addr learned my addr 172.21.15.3:0/121983248 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:27.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.513+0000 7f7c537fe700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27ab8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c52ffd700 1 -- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 msgr2=0x5612d27a9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c53fff700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27a9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27a9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c52ffd700 1 -- 172.21.15.3:0/121983248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 msgr2=0x5612d27ab8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27ab8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c52ffd700 1 -- 172.21.15.3:0/121983248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c541a1330 con 0x5612d27a2030 2023-09-10T01:16:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c53fff700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27a9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:16:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.514+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27a8b50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7c440070b0 tx=0x7f7c4400eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb27037e3013dd4b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:27.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.515+0000 7f7c42ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:16:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.515+0000 7f7c42ffd700 1 -- 172.21.15.3:0/121983248 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44014890 con 0x5612d27a2030 2023-09-10T01:16:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.515+0000 7f7c42ffd700 1 -- 172.21.15.3:0/121983248 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7c44006e40 con 0x5612d27a2030 2023-09-10T01:16:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.515+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5612d264b4f0 con 0x5612d27a2030 2023-09-10T01:16:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.515+0000 7f7c42ffd700 1 -- 172.21.15.3:0/121983248 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c440149f0 con 0x5612d27a2030 2023-09-10T01:16:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.515+0000 7f7c6d8bcb80 10 client.28516 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.515+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5612d2508c60 con 0x5612d27a2030 2023-09-10T01:16:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.516+0000 7f7c42ffd700 1 -- 172.21.15.3:0/121983248 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5612d264b4f0 con 0x5612d27a2030 2023-09-10T01:16:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.517+0000 7f7c42ffd700 1 -- 172.21.15.3:0/121983248 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7c4401c3a0 con 0x5612d27a2030 2023-09-10T01:16:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.517+0000 7f7c6d8bcb80 10 client.28516 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:27.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.517+0000 7f7c6d8bcb80 10 client.28516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:27.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.517+0000 7f7c6d8bcb80 20 client.28516 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:27.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.517+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d27cc9e0 0x5612d27cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.518+0000 7f7c6d8bcb80 4 client.28516 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:27.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.518+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612d27088a0 con 0x5612d27cc9e0 2023-09-10T01:16:27.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.518+0000 7f7c537fe700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d27cc9e0 0x5612d27cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.519+0000 7f7c42ffd700 10 client.28516 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.519+0000 7f7c537fe700 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d27cc9e0 0x5612d27cee00 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd34998f1bcca7d8 server_cookie=c82d806d25e24454 in_seq=0 out_seq=0 2023-09-10T01:16:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c42ffd700 1 -- 172.21.15.3:0/121983248 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5612d27088a0 con 0x5612d27cc9e0 2023-09-10T01:16:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c42ffd700 10 client.28516 handle_command_reply: tid=0 2023-09-10T01:16:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 client.28516 shutdown 2023-09-10T01:16:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 msgr2=0x5612d27a8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27a8b50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7c440070b0 tx=0x7f7c4400eb90 comp rx=0 tx=0).stop 2023-09-10T01:16:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 shutdown_connections 2023-09-10T01:16:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d27cc9e0 0x5612d27cee00 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27abe40 0x5612d27a9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a1710 0x5612d27ab8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/121983248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27a2030 0x5612d27a8b50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 >> 172.21.15.3:0/121983248 conn(0x5612d2716090 msgr2=0x5612d279c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:27.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c42ffd700 0 client.28516 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 shutdown_connections 2023-09-10T01:16:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.521+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/121983248 wait complete. 2023-09-10T01:16:27.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.522+0000 7f7c6d8bcb80 20 client.28516 trim_cache size 0 max 16384 2023-09-10T01:16:27.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 Processor -- start 2023-09-10T01:16:27.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 -- start start 2023-09-10T01:16:27.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27a8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d27a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d279d900 con 0x5612d27a2030 2023-09-10T01:16:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d27c8a10 con 0x5612d27a1710 2023-09-10T01:16:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.541+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d27ca090 con 0x5612d27abe40 2023-09-10T01:16:27.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27a8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27a8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59352/0 (socket says 172.21.15.3:59352) 2023-09-10T01:16:27.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 -- 172.21.15.3:0/2408509067 learned_addr learned my addr 172.21.15.3:0/2408509067 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:27.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c53fff700 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d27a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 -- 172.21.15.3:0/2408509067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 msgr2=0x5612d27a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d27a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 -- 172.21.15.3:0/2408509067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 msgr2=0x5612d27a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27a8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 -- 172.21.15.3:0/2408509067 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612d2546250 con 0x5612d27a1710 2023-09-10T01:16:27.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:27.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c53fff700 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d27a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:27.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.542+0000 7f7c537fe700 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27a8330 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c006010 tx=0x7f7c3c022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8cfef39d389a61f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.543+0000 7f7c4a7fc700 1 -- 172.21.15.3:0/2408509067 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c02d050 con 0x5612d27a1710 2023-09-10T01:16:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.543+0000 7f7c4a7fc700 1 -- 172.21.15.3:0/2408509067 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c3c02e040 con 0x5612d27a1710 2023-09-10T01:16:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.543+0000 7f7c4a7fc700 1 -- 172.21.15.3:0/2408509067 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c004710 con 0x5612d27a1710 2023-09-10T01:16:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.543+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/2408509067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 msgr2=0x5612d27a8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.543+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27a8330 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c006010 tx=0x7f7c3c022000 comp rx=0 tx=0).stop 2023-09-10T01:16:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/2408509067 shutdown_connections 2023-09-10T01:16:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d27a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27a8330 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/2408509067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/2408509067 >> 172.21.15.3:0/2408509067 conn(0x5612d2716090 msgr2=0x5612d27a3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:27.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/2408509067 shutdown_connections 2023-09-10T01:16:27.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/2408509067 wait complete. 2023-09-10T01:16:27.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 -- start start 2023-09-10T01:16:27.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.544+0000 7f7c6d8bcb80 1 Processor -- start 2023-09-10T01:16:27.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27c2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c6d8bcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d2820820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d27c3020 con 0x5612d27a2030 2023-09-10T01:16:27.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d27c3190 con 0x5612d27a1710 2023-09-10T01:16:27.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c6d8bcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612d2820d60 con 0x5612d27abe40 2023-09-10T01:16:27.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c6d8bcb80 10 client.0 _ll_register_callbacks cb 0x7f7c4400e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:27.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27c2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59368/0 (socket says 172.21.15.3:59368) 2023-09-10T01:16:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c537fe700 1 -- 172.21.15.3:0/383069260 learned_addr learned my addr 172.21.15.3:0/383069260 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c53fff700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d2820820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c537fe700 1 -- 172.21.15.3:0/383069260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 msgr2=0x5612d2820820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c537fe700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d2820820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.545+0000 7f7c537fe700 1 -- 172.21.15.3:0/383069260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 msgr2=0x5612d27c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c537fe700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c537fe700 1 -- 172.21.15.3:0/383069260 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c54068300 con 0x5612d27a1710 2023-09-10T01:16:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c537fe700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27c2350 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009ce0 tx=0x7f7c3c0344a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f826cbf115971757 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c42ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c42ffd700 1 -- 172.21.15.3:0/383069260 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c034b80 con 0x5612d27a1710 2023-09-10T01:16:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c4c09cec0 con 0x5612d27a1710 2023-09-10T01:16:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c42ffd700 1 -- 172.21.15.3:0/383069260 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f7c3c034ce0 con 0x5612d27a1710 2023-09-10T01:16:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c42ffd700 1 -- 172.21.15.3:0/383069260 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c3c02d020 con 0x5612d27a1710 2023-09-10T01:16:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c6d8bcb80 10 client.28640 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.546+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5612d251e2b0 con 0x5612d27a1710 2023-09-10T01:16:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c42ffd700 1 -- 172.21.15.3:0/383069260 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c4c09cec0 con 0x5612d27a1710 2023-09-10T01:16:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c42ffd700 1 -- 172.21.15.3:0/383069260 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7c3c045070 con 0x5612d27a1710 2023-09-10T01:16:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c6d8bcb80 10 client.28640 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c6d8bcb80 10 client.28640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c6d8bcb80 20 client.28640 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d2822090 0x5612d28244b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c6d8bcb80 4 client.28640 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:16:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5612d26ec920 con 0x5612d2822090 2023-09-10T01:16:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.547+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d2822090 0x5612d28244b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.548+0000 7f7c52ffd700 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d2822090 0x5612d28244b0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6693c4c7446a67fc server_cookie=a99dd851cd54f73b in_seq=0 out_seq=0 2023-09-10T01:16:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.548+0000 7f7c42ffd700 10 client.28640 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.549+0000 7f7c42ffd700 1 -- 172.21.15.3:0/383069260 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5612d26ec920 con 0x5612d2822090 2023-09-10T01:16:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.549+0000 7f7c42ffd700 10 client.28640 handle_command_reply: tid=0 2023-09-10T01:16:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 client.28640 shutdown 2023-09-10T01:16:27.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 msgr2=0x5612d27c2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27c2350 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009ce0 tx=0x7f7c3c0344a0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 shutdown_connections 2023-09-10T01:16:27.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5612d2822090 0x5612d28244b0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5612d27abe40 0x5612d2820820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612d27a1710 0x5612d27c2350 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 --2- 172.21.15.3:0/383069260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612d27a2030 0x5612d27c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 >> 172.21.15.3:0/383069260 conn(0x5612d2716090 msgr2=0x5612d27a0700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:27.566 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c42ffd700 0 client.28640 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:27.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 shutdown_connections 2023-09-10T01:16:27.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 1 -- 172.21.15.3:0/383069260 wait complete. 2023-09-10T01:16:27.567 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.550+0000 7f7c6d8bcb80 20 client.28640 trim_cache size 0 max 16384 2023-09-10T01:16:27.568 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:16:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:16:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:16:27.569 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:16:27.569 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c340a09c0 msgr2=0x7f7c340a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c340a09c0 0x7f7c340a2e80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c005b40 tx=0x7f7c3c005a90 comp rx=0 tx=0).stop 2023-09-10T01:16:27.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54105630 msgr2=0x7f7c5410cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54105630 0x7f7c5410cc40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00d720 tx=0x7f7c4c00da30 comp rx=0 tx=0).stop 2023-09-10T01:16:27.570 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 shutdown_connections 2023-09-10T01:16:27.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c340a09c0 0x7f7c340a2e80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54105630 0x7f7c5410cc40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c540ff570 0x7f7c54199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.571 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 --2- 172.21.15.3:0/3695670716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c540fff30 0x7f7c54199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 >> 172.21.15.3:0/3695670716 conn(0x7f7c54076710 msgr2=0x7f7c540fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:27.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 shutdown_connections 2023-09-10T01:16:27.572 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.551+0000 7f7c59fff700 1 -- 172.21.15.3:0/3695670716 wait complete. 2023-09-10T01:16:27.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:27 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:16:27.574 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:16:27.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:16:27.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:27 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:16:27.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:27 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:16:27.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.835+0000 7fc8f03ff700 1 -- 172.21.15.3:0/1474141106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8107b40 msgr2=0x7fc8e8107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:27.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.835+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/1474141106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8107b40 0x7fc8e8107fc0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc8d801a9d0 tx=0x7fc8d801ace0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.838+0000 7fc8f03ff700 1 -- 172.21.15.3:0/1474141106 shutdown_connections 2023-09-10T01:16:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.838+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/1474141106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8e810ee60 0x7fc8e8107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.838+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/1474141106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e8119de0 0x7fc8e811c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.838+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/1474141106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8107b40 0x7fc8e8107fc0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.838+0000 7fc8f03ff700 1 -- 172.21.15.3:0/1474141106 >> 172.21.15.3:0/1474141106 conn(0x7fc8e8071b60 msgr2=0x7fc8e8071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:27.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.840+0000 7fc8f03ff700 1 -- 172.21.15.3:0/1474141106 shutdown_connections 2023-09-10T01:16:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.841+0000 7fc8f03ff700 1 -- 172.21.15.3:0/1474141106 wait complete. 2023-09-10T01:16:27.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.843+0000 7fc8f03ff700 1 Processor -- start 2023-09-10T01:16:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.843+0000 7fc8f03ff700 1 -- start start 2023-09-10T01:16:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.844+0000 7fc8f03ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8e8107b40 0x7fc8e8115300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.844+0000 7fc8f03ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e8119de0 0x7fc8e8115840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.844+0000 7fc8f03ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8118c40 0x7fc8e81190c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.844+0000 7fc8f03ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e81196c0 con 0x7fc8e8118c40 2023-09-10T01:16:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.844+0000 7fc8f03ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e8119830 con 0x7fc8e8119de0 2023-09-10T01:16:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.844+0000 7fc8f03ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e81199f0 con 0x7fc8e8107b40 2023-09-10T01:16:27.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.845+0000 7fc8ee99c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8118c40 0x7fc8e81190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.845+0000 7fc8ee99c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8118c40 0x7fc8e81190c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43170/0 (socket says 172.21.15.3:43170) 2023-09-10T01:16:27.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.845+0000 7fc8ee99c700 1 -- 172.21.15.3:0/152117812 learned_addr learned my addr 172.21.15.3:0/152117812 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:27.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.846+0000 7fc8ee99c700 1 -- 172.21.15.3:0/152117812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8e8107b40 msgr2=0x7fc8e8115300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:16:27.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.846+0000 7fc8ee99c700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8e8107b40 0x7fc8e8115300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.846+0000 7fc8ee99c700 1 -- 172.21.15.3:0/152117812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e8119de0 msgr2=0x7fc8e8115840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-10T01:16:27.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.846+0000 7fc8ee99c700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e8119de0 0x7fc8e8115840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:27.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.846+0000 7fc8ee99c700 1 -- 172.21.15.3:0/152117812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d801a600 con 0x7fc8e8118c40 2023-09-10T01:16:27.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.847+0000 7fc8ee99c700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8118c40 0x7fc8e81190c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000b6d0 tx=0x7fc8e000ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.848+0000 7fc8df7fe700 1 -- 172.21.15.3:0/152117812 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0010860 con 0x7fc8e8118c40 2023-09-10T01:16:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.848+0000 7fc8df7fe700 1 -- 172.21.15.3:0/152117812 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8e0010e80 con 0x7fc8e8118c40 2023-09-10T01:16:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.848+0000 7fc8df7fe700 1 -- 172.21.15.3:0/152117812 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0007370 con 0x7fc8e8118c40 2023-09-10T01:16:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.848+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e81cbc80 con 0x7fc8e8118c40 2023-09-10T01:16:27.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.848+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e81cc190 con 0x7fc8e8118c40 2023-09-10T01:16:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.857+0000 7fc8df7fe700 1 -- 172.21.15.3:0/152117812 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fc8e00109c0 con 0x7fc8e8118c40 2023-09-10T01:16:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.859+0000 7fc8df7fe700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc8d40a09c0 0x7fc8d40a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.860+0000 7fc8df7fe700 1 -- 172.21.15.3:0/152117812 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fc8e00cc010 con 0x7fc8e8118c40 2023-09-10T01:16:27.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.861+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc005320 con 0x7fc8e8118c40 2023-09-10T01:16:27.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.871+0000 7fc8ee19b700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc8d40a09c0 0x7fc8d40a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:27.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.874+0000 7fc8df7fe700 1 -- 172.21.15.3:0/152117812 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fc8e00953b0 con 0x7fc8e8118c40 2023-09-10T01:16:27.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:27.876+0000 7fc8ee19b700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc8d40a09c0 0x7fc8d40a2e80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc8e806ea80 tx=0x7fc8e4006c60 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:28.145 INFO:tasks.workunit.client.0.smithi003.stdout: 10 3663 42.87 MB/sec warmup 9 sec latency 177.687 ms 2023-09-10T01:16:28.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.424+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8cc005190 con 0x7fc8e8118c40 2023-09-10T01:16:28.431 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:16:28.431 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:16:25.285703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:16:28.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.426+0000 7fc8df7fe700 1 -- 172.21.15.3:0/152117812 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+11299 (secure 0 0 0) 0x7fc8e001e070 con 0x7fc8e8118c40 2023-09-10T01:16:28.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.432+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc8d40a09c0 msgr2=0x7fc8d40a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.432+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc8d40a09c0 0x7fc8d40a2e80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc8e806ea80 tx=0x7fc8e4006c60 comp rx=0 tx=0).stop 2023-09-10T01:16:28.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.432+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8118c40 msgr2=0x7fc8e81190c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.432+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8118c40 0x7fc8e81190c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000b6d0 tx=0x7fc8e000ba90 comp rx=0 tx=0).stop 2023-09-10T01:16:28.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.433+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 shutdown_connections 2023-09-10T01:16:28.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.433+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc8d40a09c0 0x7fc8d40a2e80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.433+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8e8107b40 0x7fc8e8115300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.433+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc8e8118c40 0x7fc8e81190c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.433+0000 7fc8f03ff700 1 --2- 172.21.15.3:0/152117812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e8119de0 0x7fc8e8115840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.433+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 >> 172.21.15.3:0/152117812 conn(0x7fc8e8071b60 msgr2=0x7fc8e806ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:28.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.434+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 shutdown_connections 2023-09-10T01:16:28.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.434+0000 7fc8f03ff700 1 -- 172.21.15.3:0/152117812 wait complete. 2023-09-10T01:16:28.440 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 38 2023-09-10T01:16:28.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:16:28.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:28 smithi003 ceph-mon[97654]: pgmap v414: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 10 MiB/s wr, 413 op/s 2023-09-10T01:16:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:28 smithi136 ceph-mon[133708]: pgmap v414: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 10 MiB/s wr, 413 op/s 2023-09-10T01:16:28.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:28 smithi092 ceph-mon[134067]: pgmap v414: 129 pgs: 129 active+clean; 426 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 10 MiB/s wr, 413 op/s 2023-09-10T01:16:28.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 -- 172.21.15.3:0/3110524774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107600 msgr2=0x7f74b41079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 --2- 172.21.15.3:0/3110524774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107600 0x7f74b41079e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009990 tx=0x7f74a8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 -- 172.21.15.3:0/3110524774 shutdown_connections 2023-09-10T01:16:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 --2- 172.21.15.3:0/3110524774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b4113750 0x7f74b4115b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 --2- 172.21.15.3:0/3110524774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107600 0x7f74b41079e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 --2- 172.21.15.3:0/3110524774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b4107f20 0x7f74b4113210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 -- 172.21.15.3:0/3110524774 >> 172.21.15.3:0/3110524774 conn(0x7f74b4071dd0 msgr2=0x7f74b406ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:28.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 -- 172.21.15.3:0/3110524774 shutdown_connections 2023-09-10T01:16:28.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.775+0000 7f74badff700 1 -- 172.21.15.3:0/3110524774 wait complete. 2023-09-10T01:16:28.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 Processor -- start 2023-09-10T01:16:28.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 -- start start 2023-09-10T01:16:28.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b4107600 0x7f74b419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107f20 0x7f74b419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b4113750 0x7f74b41a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b41a3ea0 con 0x7f74b4113750 2023-09-10T01:16:28.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b41a4010 con 0x7f74b4107f20 2023-09-10T01:16:28.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.777+0000 7f74badff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b41a41d0 con 0x7f74b4107600 2023-09-10T01:16:28.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107f20 0x7f74b419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.781 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107f20 0x7f74b419f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59402/0 (socket says 172.21.15.3:59402) 2023-09-10T01:16:28.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b8b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b4107600 0x7f74b419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b3fff700 1 -- 172.21.15.3:0/2118791228 learned_addr learned my addr 172.21.15.3:0/2118791228 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:28.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b939c700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b4113750 0x7f74b41a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b8b9b700 1 -- 172.21.15.3:0/2118791228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107f20 msgr2=0x7f74b419f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.782 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107f20 0x7f74b419f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b8b9b700 1 -- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b4113750 msgr2=0x7f74b41a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b4113750 0x7f74b41a3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b8b9b700 1 -- 172.21.15.3:0/2118791228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80095f0 con 0x7f74b4107600 2023-09-10T01:16:28.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b3fff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107f20 0x7f74b419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:28.783 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.778+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b4107600 0x7f74b419eec0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000c00 tx=0x7f74a8005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:28.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.779+0000 7f74b1ffb700 1 -- 172.21.15.3:0/2118791228 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a801d070 con 0x7f74b4107600 2023-09-10T01:16:28.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.779+0000 7f74b1ffb700 1 -- 172.21.15.3:0/2118791228 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74a8021410 con 0x7f74b4107600 2023-09-10T01:16:28.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.779+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b41a4450 con 0x7f74b4107600 2023-09-10T01:16:28.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.779+0000 7f74b1ffb700 1 -- 172.21.15.3:0/2118791228 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8017650 con 0x7f74b4107600 2023-09-10T01:16:28.784 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.779+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b41abcf0 con 0x7f74b4107600 2023-09-10T01:16:28.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.783+0000 7f74b1ffb700 1 -- 172.21.15.3:0/2118791228 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f74a80217f0 con 0x7f74b4107600 2023-09-10T01:16:28.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.784+0000 7f74b1ffb700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a40a07b0 0x7f74a40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.785+0000 7f74b1ffb700 1 -- 172.21.15.3:0/2118791228 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f74a80cd670 con 0x7f74b4107600 2023-09-10T01:16:28.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.785+0000 7f74b3fff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a40a07b0 0x7f74a40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.785+0000 7f74ce70db80 1 Processor -- start 2023-09-10T01:16:28.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.785+0000 7f74ce70db80 1 -- start start 2023-09-10T01:16:28.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b3fff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a40a07b0 0x7f74a40a2c70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f74b41a04d0 tx=0x7f74a0009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:28.787 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da03c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74ce70db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d947730 con 0x55e84d9ee710 2023-09-10T01:16:28.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74ce70db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d945050 con 0x55e84d9ef0d0 2023-09-10T01:16:28.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74ce70db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d940940 con 0x55e84d9fcf40 2023-09-10T01:16:28.788 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45492/0 (socket says 172.21.15.3:45492) 2023-09-10T01:16:28.789 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 -- 172.21.15.3:0/1224722310 learned_addr learned my addr 172.21.15.3:0/1224722310 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:28.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b939c700 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da03c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 -- 172.21.15.3:0/1224722310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 msgr2=0x55e84da03c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da03c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 -- 172.21.15.3:0/1224722310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 msgr2=0x55e84d9eeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9eeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b8b9b700 1 -- 172.21.15.3:0/1224722310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e84d76cde0 con 0x55e84d9fcf40 2023-09-10T01:16:28.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.786+0000 7f74b939c700 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da03c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:28.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.787+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fd320 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f74a809a4c0 tx=0x7f74a809a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5a817628691f817 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:28.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.787+0000 7f749effd700 1 -- 172.21.15.3:0/1224722310 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a809aec0 con 0x55e84d9fcf40 2023-09-10T01:16:28.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.787+0000 7f749effd700 1 -- 172.21.15.3:0/1224722310 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74a80b1070 con 0x55e84d9fcf40 2023-09-10T01:16:28.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.787+0000 7f749effd700 1 -- 172.21.15.3:0/1224722310 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a809ac60 con 0x55e84d9fcf40 2023-09-10T01:16:28.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 -- 172.21.15.3:0/1224722310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 msgr2=0x55e84d9fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fd320 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f74a809a4c0 tx=0x7f74a809a610 comp rx=0 tx=0).stop 2023-09-10T01:16:28.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 -- 172.21.15.3:0/1224722310 shutdown_connections 2023-09-10T01:16:28.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fd320 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da03c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 --2- 172.21.15.3:0/1224722310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 -- 172.21.15.3:0/1224722310 >> 172.21.15.3:0/1224722310 conn(0x55e84d963090 msgr2=0x55e84d9e9850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:28.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 -- 172.21.15.3:0/1224722310 shutdown_connections 2023-09-10T01:16:28.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.788+0000 7f74ce70db80 1 -- 172.21.15.3:0/1224722310 wait complete. 2023-09-10T01:16:28.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 -- start start 2023-09-10T01:16:28.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 Processor -- start 2023-09-10T01:16:28.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84da068f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da06e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84da00bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d8deb70 con 0x55e84d9ee710 2023-09-10T01:16:28.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d8de870 con 0x55e84d9ef0d0 2023-09-10T01:16:28.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d7c2920 con 0x55e84d9fcf40 2023-09-10T01:16:28.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.789+0000 7f74ce70db80 10 client.0 _ll_register_callbacks cb 0x55e84d828a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:28.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da06e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da06e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59414/0 (socket says 172.21.15.3:59414) 2023-09-10T01:16:28.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 -- 172.21.15.3:0/147147426 learned_addr learned my addr 172.21.15.3:0/147147426 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:28.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b939c700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84da00bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84da068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 -- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 msgr2=0x55e84da00bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84da00bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 -- 172.21.15.3:0/147147426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 msgr2=0x55e84da068f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84da068f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b3fff700 1 -- 172.21.15.3:0/147147426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b41abcf0 con 0x55e84d9ef0d0 2023-09-10T01:16:28.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b939c700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84da00bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:28.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.790+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84da068f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:28.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f74b3fff700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da06e30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009ec0 tx=0x7f74a00047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4295fb76ff09e04c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:28.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f7493fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:28.801 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f7493fff700 1 -- 172.21.15.3:0/147147426 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0022070 con 0x55e84d9ef0d0 2023-09-10T01:16:28.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e84d8984f0 con 0x55e84d9ef0d0 2023-09-10T01:16:28.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f7493fff700 1 -- 172.21.15.3:0/147147426 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f74a001dd10 con 0x55e84d9ef0d0 2023-09-10T01:16:28.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f7493fff700 1 -- 172.21.15.3:0/147147426 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a000ee70 con 0x55e84d9ef0d0 2023-09-10T01:16:28.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f74ce70db80 10 client.28652 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:28.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55e84d755c60 con 0x55e84d9ef0d0 2023-09-10T01:16:28.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.791+0000 7f7493fff700 1 -- 172.21.15.3:0/147147426 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e84d8984f0 con 0x55e84d9ef0d0 2023-09-10T01:16:28.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f7493fff700 1 -- 172.21.15.3:0/147147426 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f74a0005550 con 0x55e84d9ef0d0 2023-09-10T01:16:28.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74ce70db80 10 client.28652 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:28.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74ce70db80 10 client.28652 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:16:28.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74ce70db80 20 client.28652 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:28.803 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74ce70db80 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da10e30 0x55e84da13250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74ce70db80 4 client.28652 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:28.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e84d9558a0 con 0x55e84da10e30 2023-09-10T01:16:28.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da10e30 0x55e84da13250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da10e30 0x55e84da13250 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=317ac62120e0de97 server_cookie=e5ec8fa403e62f63 in_seq=0 out_seq=0 2023-09-10T01:16:28.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.792+0000 7f7493fff700 10 client.28652 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:16:28.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f7493fff700 1 -- 172.21.15.3:0/147147426 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e84d9558a0 con 0x55e84da10e30 2023-09-10T01:16:28.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f7493fff700 10 client.28652 handle_command_reply: tid=0 2023-09-10T01:16:28.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 client.28652 shutdown 2023-09-10T01:16:28.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 msgr2=0x55e84da06e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da06e30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009ec0 tx=0x7f74a00047c0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 shutdown_connections 2023-09-10T01:16:28.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da10e30 0x55e84da13250 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84da00bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84da068f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 --2- 172.21.15.3:0/147147426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84da06e30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 >> 172.21.15.3:0/147147426 conn(0x55e84d963090 msgr2=0x55e84d9ed700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:28.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.793+0000 7f7493fff700 0 client.28652 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:16:28.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.794+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 shutdown_connections 2023-09-10T01:16:28.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.794+0000 7f74ce70db80 1 -- 172.21.15.3:0/147147426 wait complete. 2023-09-10T01:16:28.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.794+0000 7f74ce70db80 20 client.28652 trim_cache size 0 max 16384 2023-09-10T01:16:28.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 Processor -- start 2023-09-10T01:16:28.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 -- start start 2023-09-10T01:16:28.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84d9fe330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d9ea410 con 0x55e84d9ee710 2023-09-10T01:16:28.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d9e14d0 con 0x55e84d9ef0d0 2023-09-10T01:16:28.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.809+0000 7f74ce70db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84d9fa880 con 0x55e84d9fcf40 2023-09-10T01:16:28.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84d9fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84d9fe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59418/0 (socket says 172.21.15.3:59418) 2023-09-10T01:16:28.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b3fff700 1 -- 172.21.15.3:0/3028017715 learned_addr learned my addr 172.21.15.3:0/3028017715 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:28.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fe870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45504/0 (socket says 172.21.15.3:45504) 2023-09-10T01:16:28.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 -- 172.21.15.3:0/3028017715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 msgr2=0x55e84d9fe330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84d9fe330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 -- 172.21.15.3:0/3028017715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 msgr2=0x55e84d9fc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9fc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 -- 172.21.15.3:0/3028017715 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e84d793250 con 0x55e84d9fcf40 2023-09-10T01:16:28.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b3fff700 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84d9fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:28.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.810+0000 7f74b939c700 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fe870 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55e84d9eb140 tx=0x7f74ac009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a1d9309aecc01ed server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.811+0000 7f749f7fe700 1 -- 172.21.15.3:0/3028017715 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac00e040 con 0x55e84d9fcf40 2023-09-10T01:16:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.811+0000 7f749f7fe700 1 -- 172.21.15.3:0/3028017715 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74ac00f040 con 0x55e84d9fcf40 2023-09-10T01:16:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.811+0000 7f749f7fe700 1 -- 172.21.15.3:0/3028017715 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac007df0 con 0x55e84d9fcf40 2023-09-10T01:16:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.811+0000 7f74ce70db80 1 -- 172.21.15.3:0/3028017715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 msgr2=0x55e84d9fe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.811+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fe870 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x55e84d9eb140 tx=0x7f74ac009920 comp rx=0 tx=0).stop 2023-09-10T01:16:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 -- 172.21.15.3:0/3028017715 shutdown_connections 2023-09-10T01:16:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9fcf40 0x55e84d9fe870 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9ee710 0x55e84d9fc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3028017715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ef0d0 0x55e84d9fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 -- 172.21.15.3:0/3028017715 >> 172.21.15.3:0/3028017715 conn(0x55e84d963090 msgr2=0x55e84d9fbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 -- 172.21.15.3:0/3028017715 shutdown_connections 2023-09-10T01:16:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 -- 172.21.15.3:0/3028017715 wait complete. 2023-09-10T01:16:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 -- start start 2023-09-10T01:16:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.812+0000 7f74ce70db80 1 Processor -- start 2023-09-10T01:16:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ee710 0x55e84da5cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9ef0d0 0x55e84da5d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74ce70db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9fcf40 0x55e84da5dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74ce70db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84da5e330 con 0x55e84d9fcf40 2023-09-10T01:16:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74ce70db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84da5e4a0 con 0x55e84d9ee710 2023-09-10T01:16:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74ce70db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e84da5e660 con 0x55e84d9ef0d0 2023-09-10T01:16:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74ce70db80 10 client.0 _ll_register_callbacks cb 0x7f74a000ea50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9ef0d0 0x55e84da5d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74b8b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ee710 0x55e84da5cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74b8b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ee710 0x55e84da5cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59422/0 (socket says 172.21.15.3:59422) 2023-09-10T01:16:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74b3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9ef0d0 0x55e84da5d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45514/0 (socket says 172.21.15.3:45514) 2023-09-10T01:16:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74b8b9b700 1 -- 172.21.15.3:0/3910059768 learned_addr learned my addr 172.21.15.3:0/3910059768 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.813+0000 7f74b939c700 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9fcf40 0x55e84da5dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74b8b9b700 1 -- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9ef0d0 msgr2=0x55e84da5d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9ef0d0 0x55e84da5d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74b8b9b700 1 -- 172.21.15.3:0/3910059768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9fcf40 msgr2=0x55e84da5dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9fcf40 0x55e84da5dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74b8b9b700 1 -- 172.21.15.3:0/3910059768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e84d755c60 con 0x55e84d9ee710 2023-09-10T01:16:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74b3fff700 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9ef0d0 0x55e84da5d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74b8b9b700 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ee710 0x55e84da5cfb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f74a80caa80 tx=0x7f74a8005d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd6497577888f960 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f7493fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f7493fff700 1 -- 172.21.15.3:0/3910059768 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a809aec0 con 0x55e84d9ee710 2023-09-10T01:16:28.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f7493fff700 1 -- 172.21.15.3:0/3910059768 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f74a80b7040 con 0x55e84d9ee710 2023-09-10T01:16:28.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74a80b1070 con 0x55e84d9ee710 2023-09-10T01:16:28.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f7493fff700 1 -- 172.21.15.3:0/3910059768 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a809a7c0 con 0x55e84d9ee710 2023-09-10T01:16:28.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74ce70db80 10 client.28658 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:28.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.814+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7f74ac003a00 con 0x55e84d9ee710 2023-09-10T01:16:28.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f7493fff700 1 -- 172.21.15.3:0/3910059768 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74a8081030 con 0x55e84d9ee710 2023-09-10T01:16:28.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f7493fff700 1 -- 172.21.15.3:0/3910059768 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f74a80cd240 con 0x55e84d9ee710 2023-09-10T01:16:28.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f74ce70db80 10 client.28658 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:28.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f74ce70db80 10 client.28658 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:16:28.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f74ce70db80 20 client.28658 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:28.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da65a90 0x55e84da67eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f74ce70db80 4 client.28658 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:16:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e84d939920 con 0x55e84da65a90 2023-09-10T01:16:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.815+0000 7f74b3fff700 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da65a90 0x55e84da67eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74b3fff700 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da65a90 0x55e84da67eb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a1fda7061fcd0d16 server_cookie=85f1831acde8ca18 in_seq=0 out_seq=0 2023-09-10T01:16:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f7493fff700 10 client.28658 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:16:28.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f7493fff700 1 -- 172.21.15.3:0/3910059768 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e84d939920 con 0x55e84da65a90 2023-09-10T01:16:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f7493fff700 10 client.28658 handle_command_reply: tid=0 2023-09-10T01:16:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 client.28658 shutdown 2023-09-10T01:16:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ee710 msgr2=0x55e84da5cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ee710 0x55e84da5cfb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f74a80caa80 tx=0x7f74a8005d40 comp rx=0 tx=0).stop 2023-09-10T01:16:28.829 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 shutdown_connections 2023-09-10T01:16:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55e84da65a90 0x55e84da67eb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e84d9fcf40 0x55e84da5dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e84d9ee710 0x55e84da5cfb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 --2- 172.21.15.3:0/3910059768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e84d9ef0d0 0x55e84da5d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 >> 172.21.15.3:0/3910059768 conn(0x55e84d963090 msgr2=0x55e84d9edf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f7493fff700 0 client.28658 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:16:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 shutdown_connections 2023-09-10T01:16:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.816+0000 7f74ce70db80 1 -- 172.21.15.3:0/3910059768 wait complete. 2023-09-10T01:16:28.831 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74ce70db80 20 client.28658 trim_cache size 0 max 16384 2023-09-10T01:16:28.832 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:16:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a40a07b0 msgr2=0x7f74a40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a40a07b0 0x7f74a40a2c70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f74b41a04d0 tx=0x7f74a0009260 comp rx=0 tx=0).stop 2023-09-10T01:16:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b4107600 msgr2=0x7f74b419eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b4107600 0x7f74b419eec0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f74a8000c00 tx=0x7f74a8005740 comp rx=0 tx=0).stop 2023-09-10T01:16:28.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 shutdown_connections 2023-09-10T01:16:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f74a40a07b0 0x7f74a40a2c70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74b4107600 0x7f74b419eec0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74b4113750 0x7f74b41a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 --2- 172.21.15.3:0/2118791228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74b4107f20 0x7f74b419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:28.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 >> 172.21.15.3:0/2118791228 conn(0x7f74b4071dd0 msgr2=0x7f74b406c640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 shutdown_connections 2023-09-10T01:16:28.835 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:28.817+0000 7f74badff700 1 -- 172.21.15.3:0/2118791228 wait complete. 2023-09-10T01:16:28.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:16:29.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.080+0000 7ff002dff700 1 -- 172.21.15.3:0/2360872934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 msgr2=0x7feffc10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.080+0000 7ff002dff700 1 --2- 172.21.15.3:0/2360872934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc10d510 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7feff4009f90 tx=0x7feff400b330 comp rx=0 tx=0).stop 2023-09-10T01:16:29.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.081+0000 7ff002dff700 1 -- 172.21.15.3:0/2360872934 shutdown_connections 2023-09-10T01:16:29.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.081+0000 7ff002dff700 1 --2- 172.21.15.3:0/2360872934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc106650 0x7feffc106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.081+0000 7ff002dff700 1 --2- 172.21.15.3:0/2360872934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc10d510 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.081+0000 7ff002dff700 1 --2- 172.21.15.3:0/2360872934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feffc100640 0x7feffc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.081+0000 7ff002dff700 1 -- 172.21.15.3:0/2360872934 >> 172.21.15.3:0/2360872934 conn(0x7feffc0fc170 msgr2=0x7feffc0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.081+0000 7ff002dff700 1 -- 172.21.15.3:0/2360872934 shutdown_connections 2023-09-10T01:16:29.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.081+0000 7ff002dff700 1 -- 172.21.15.3:0/2360872934 wait complete. 2023-09-10T01:16:29.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 Processor -- start 2023-09-10T01:16:29.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 -- start start 2023-09-10T01:16:29.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feffc100640 0x7feffc196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc106650 0x7feffc199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc19a580 con 0x7feffc100640 2023-09-10T01:16:29.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc19a6f0 con 0x7feffc101000 2023-09-10T01:16:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.082+0000 7ff002dff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc19a8b0 con 0x7feffc106650 2023-09-10T01:16:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7ff000b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feffc100640 0x7feffc196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59438/0 (socket says 172.21.15.3:59438) 2023-09-10T01:16:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 -- 172.21.15.3:0/3007658651 learned_addr learned my addr 172.21.15.3:0/3007658651 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7ff00139c700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc106650 0x7feffc199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 -- 172.21.15.3:0/3007658651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc106650 msgr2=0x7feffc199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc106650 0x7feffc199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 -- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feffc100640 msgr2=0x7feffc196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feffc100640 0x7feffc196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 -- 172.21.15.3:0/3007658651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80095f0 con 0x7feffc101000 2023-09-10T01:16:29.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feffbfff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc196bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fefe800ec10 tx=0x7fefe800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.083+0000 7feff9ffb700 1 -- 172.21.15.3:0/3007658651 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe800cc90 con 0x7feffc101000 2023-09-10T01:16:29.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.084+0000 7feff9ffb700 1 -- 172.21.15.3:0/3007658651 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefe80044e0 con 0x7feffc101000 2023-09-10T01:16:29.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.084+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff4009c40 con 0x7feffc101000 2023-09-10T01:16:29.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.084+0000 7feff9ffb700 1 -- 172.21.15.3:0/3007658651 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe8010430 con 0x7feffc101000 2023-09-10T01:16:29.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.084+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc072830 con 0x7feffc101000 2023-09-10T01:16:29.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.088+0000 7feff9ffb700 1 -- 172.21.15.3:0/3007658651 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fefe8003fb0 con 0x7feffc101000 2023-09-10T01:16:29.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.090+0000 7ff016643b80 1 Processor -- start 2023-09-10T01:16:29.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.090+0000 7feff9ffb700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fefec0a0830 0x7fefec0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.090+0000 7ff016643b80 1 -- start start 2023-09-10T01:16:29.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.090+0000 7feff9ffb700 1 -- 172.21.15.3:0/3007658651 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fefe8014070 con 0x7feffc101000 2023-09-10T01:16:29.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.090+0000 7ff00139c700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fefec0a0830 0x7fefec0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.090+0000 7ff016643b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1a9eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.090+0000 7ff016643b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff016643b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aab230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff016643b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f19f7730 con 0x55c5f1a9f030 2023-09-10T01:16:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff016643b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f19f5050 con 0x55c5f1aa8e40 2023-09-10T01:16:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff016643b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f19f0940 con 0x55c5f1a9e710 2023-09-10T01:16:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7feffbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1a9eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1a9eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45546/0 (socket says 172.21.15.3:45546) 2023-09-10T01:16:29.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7feffbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43242/0 (socket says 172.21.15.3:43242) 2023-09-10T01:16:29.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 -- 172.21.15.3:0/1287021491 learned_addr learned my addr 172.21.15.3:0/1287021491 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff00139c700 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.096 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 -- 172.21.15.3:0/1287021491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 msgr2=0x55c5f1aab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 -- 172.21.15.3:0/1287021491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 msgr2=0x55c5f1aa8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff000b9b700 1 -- 172.21.15.3:0/1287021491 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5f181cde0 con 0x55c5f1a9e710 2023-09-10T01:16:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7feffbfff700 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.091+0000 7ff00139c700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fefec0a0830 0x7fefec0a2cf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feff4009790 tx=0x7feff4004af0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.092+0000 7ff000b9b700 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1a9eaf0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7feff0010da0 tx=0x7feff0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b46d0a43f3eb6b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.092+0000 7fefe6ffd700 1 -- 172.21.15.3:0/1287021491 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0015050 con 0x55c5f1a9e710 2023-09-10T01:16:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.092+0000 7fefe6ffd700 1 -- 172.21.15.3:0/1287021491 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feff0016040 con 0x55c5f1a9e710 2023-09-10T01:16:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.092+0000 7fefe6ffd700 1 -- 172.21.15.3:0/1287021491 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff001a400 con 0x55c5f1a9e710 2023-09-10T01:16:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.092+0000 7ff016643b80 1 -- 172.21.15.3:0/1287021491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 msgr2=0x55c5f1a9eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.092+0000 7ff016643b80 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1a9eaf0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7feff0010da0 tx=0x7feff0009260 comp rx=0 tx=0).stop 2023-09-10T01:16:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 -- 172.21.15.3:0/1287021491 shutdown_connections 2023-09-10T01:16:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1a9eaf0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 --2- 172.21.15.3:0/1287021491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 -- 172.21.15.3:0/1287021491 >> 172.21.15.3:0/1287021491 conn(0x55c5f1a13090 msgr2=0x55c5f1a99bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 -- 172.21.15.3:0/1287021491 shutdown_connections 2023-09-10T01:16:29.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 -- 172.21.15.3:0/1287021491 wait complete. 2023-09-10T01:16:29.100 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.093+0000 7ff016643b80 1 -- start start 2023-09-10T01:16:29.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.094+0000 7ff016643b80 1 Processor -- start 2023-09-10T01:16:29.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.094+0000 7ff016643b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9e710 0x55c5f1aa8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.094+0000 7ff016643b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.094+0000 7ff016643b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.094+0000 7ff016643b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f198eb70 con 0x55c5f1aa8e40 2023-09-10T01:16:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.094+0000 7ff016643b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f198e870 con 0x55c5f1a9e710 2023-09-10T01:16:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.094+0000 7ff016643b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f1872920 con 0x55c5f1a9f030 2023-09-10T01:16:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff016643b80 10 client.0 _ll_register_callbacks cb 0x55c5f18d8a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:29.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9e710 0x55c5f1aa8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9e710 0x55c5f1aa8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59448/0 (socket says 172.21.15.3:59448) 2023-09-10T01:16:29.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 -- 172.21.15.3:0/3866571252 learned_addr learned my addr 172.21.15.3:0/3866571252 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7feffbfff700 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff00139c700 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 -- 172.21.15.3:0/3866571252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9f030 msgr2=0x55c5f1aa5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 -- 172.21.15.3:0/3866571252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1aa8e40 msgr2=0x55c5f1aa6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 -- 172.21.15.3:0/3866571252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc072830 con 0x55c5f1a9e710 2023-09-10T01:16:29.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7ff000b9b700 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9e710 0x55c5f1aa8800 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7feff0000c00 tx=0x7feff0005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5786f4f65c5e5322 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.095+0000 7fefdbfff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7fefdbfff700 1 -- 172.21.15.3:0/3866571252 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0005ce0 con 0x55c5f1a9e710 2023-09-10T01:16:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c5f19484f0 con 0x55c5f1a9e710 2023-09-10T01:16:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7fefdbfff700 1 -- 172.21.15.3:0/3866571252 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7feff0004750 con 0x55c5f1a9e710 2023-09-10T01:16:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7fefdbfff700 1 -- 172.21.15.3:0/3866571252 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7feff0015020 con 0x55c5f1a9e710 2023-09-10T01:16:29.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 10 client.28670 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7feff0010a80 con 0x55c5f1a9e710 2023-09-10T01:16:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7fefdbfff700 1 -- 172.21.15.3:0/3866571252 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c5f19484f0 con 0x55c5f1a9e710 2023-09-10T01:16:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7fefdbfff700 1 -- 172.21.15.3:0/3866571252 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7feff002b020 con 0x55c5f1a9e710 2023-09-10T01:16:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 10 client.28670 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:29.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 10 client.28670 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:16:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 20 client.28670 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1ab8730 0x55c5f1abab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 4 client.28670 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.096+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5f1a058a0 con 0x55c5f1ab8730 2023-09-10T01:16:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.097+0000 7feffbfff700 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1ab8730 0x55c5f1abab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.097+0000 7feffbfff700 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1ab8730 0x55c5f1abab50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f26903ed4f6175f1 server_cookie=b91328034d06268b in_seq=0 out_seq=0 2023-09-10T01:16:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.097+0000 7fefdbfff700 10 client.28670 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:16:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7fefdbfff700 1 -- 172.21.15.3:0/3866571252 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c5f1a058a0 con 0x55c5f1ab8730 2023-09-10T01:16:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7fefdbfff700 10 client.28670 handle_command_reply: tid=0 2023-09-10T01:16:29.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 client.28670 shutdown 2023-09-10T01:16:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9e710 msgr2=0x55c5f1aa8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9e710 0x55c5f1aa8800 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7feff0000c00 tx=0x7feff0005710 comp rx=0 tx=0).stop 2023-09-10T01:16:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 shutdown_connections 2023-09-10T01:16:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9e710 0x55c5f1aa8800 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1ab8730 0x55c5f1abab50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 --2- 172.21.15.3:0/3866571252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7fefdbfff700 0 client.28670 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:16:29.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 >> 172.21.15.3:0/3866571252 conn(0x55c5f1a13090 msgr2=0x55c5f1a99780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 shutdown_connections 2023-09-10T01:16:29.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 1 -- 172.21.15.3:0/3866571252 wait complete. 2023-09-10T01:16:29.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.098+0000 7ff016643b80 20 client.28670 trim_cache size 0 max 16384 2023-09-10T01:16:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.115+0000 7ff016643b80 1 Processor -- start 2023-09-10T01:16:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff016643b80 1 -- start start 2023-09-10T01:16:29.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff016643b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1aab230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff016643b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff016643b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff016643b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f1a9b1e0 con 0x55c5f1a9f030 2023-09-10T01:16:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff016643b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f1abab20 con 0x55c5f1aa8e40 2023-09-10T01:16:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff016643b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f1aa8870 con 0x55c5f1a9e710 2023-09-10T01:16:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff00139c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff00139c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59452/0 (socket says 172.21.15.3:59452) 2023-09-10T01:16:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff00139c700 1 -- 172.21.15.3:0/1260681726 learned_addr learned my addr 172.21.15.3:0/1260681726 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.116+0000 7ff000b9b700 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1aab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7feffbfff700 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7ff00139c700 1 -- 172.21.15.3:0/1260681726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 msgr2=0x55c5f1aab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7ff00139c700 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1aab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7ff00139c700 1 -- 172.21.15.3:0/1260681726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 msgr2=0x55c5f1aa8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7ff00139c700 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7ff00139c700 1 -- 172.21.15.3:0/1260681726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5f1805c60 con 0x55c5f1aa8e40 2023-09-10T01:16:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7feffbfff700 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:29.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7ff000b9b700 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1aab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7ff00139c700 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa0da0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7feff4009f90 tx=0x7feff4027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aca401a3d44744b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7fefe77fe700 1 -- 172.21.15.3:0/1260681726 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff400e040 con 0x55c5f1aa8e40 2023-09-10T01:16:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7fefe77fe700 1 -- 172.21.15.3:0/1260681726 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feff400bdd0 con 0x55c5f1aa8e40 2023-09-10T01:16:29.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.117+0000 7fefe77fe700 1 -- 172.21.15.3:0/1260681726 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff401c810 con 0x55c5f1aa8e40 2023-09-10T01:16:29.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 -- 172.21.15.3:0/1260681726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 msgr2=0x55c5f1aa0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa0da0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7feff4009f90 tx=0x7feff4027000 comp rx=0 tx=0).stop 2023-09-10T01:16:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 -- 172.21.15.3:0/1260681726 shutdown_connections 2023-09-10T01:16:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1a9e710 0x55c5f1aab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aa0da0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 --2- 172.21.15.3:0/1260681726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9f030 0x55c5f1aa8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 -- 172.21.15.3:0/1260681726 >> 172.21.15.3:0/1260681726 conn(0x55c5f1a13090 msgr2=0x55c5f1a98320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 -- 172.21.15.3:0/1260681726 shutdown_connections 2023-09-10T01:16:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.118+0000 7ff016643b80 1 -- 172.21.15.3:0/1260681726 wait complete. 2023-09-10T01:16:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.119+0000 7ff016643b80 1 -- start start 2023-09-10T01:16:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.119+0000 7ff016643b80 1 Processor -- start 2023-09-10T01:16:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff016643b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9e710 0x55c5f1aaf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff016643b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9f030 0x55c5f1aade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff016643b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff016643b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f1aaea40 con 0x55c5f1a9e710 2023-09-10T01:16:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff016643b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f1aaebb0 con 0x55c5f1a9f030 2023-09-10T01:16:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff016643b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5f1b0c570 con 0x55c5f1aa8e40 2023-09-10T01:16:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff016643b80 10 client.0 _ll_register_callbacks cb 0x7fefe800cdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9f030 0x55c5f1aade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9f030 0x55c5f1aade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59456/0 (socket says 172.21.15.3:59456) 2023-09-10T01:16:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 -- 172.21.15.3:0/437157615 learned_addr learned my addr 172.21.15.3:0/437157615 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff00139c700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7ff000b9b700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9e710 0x55c5f1aaf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 -- 172.21.15.3:0/437157615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1aa8e40 msgr2=0x55c5f1aae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 -- 172.21.15.3:0/437157615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9e710 msgr2=0x55c5f1aaf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9e710 0x55c5f1aaf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.120+0000 7feffbfff700 1 -- 172.21.15.3:0/437157615 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff0010a80 con 0x55c5f1a9f030 2023-09-10T01:16:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7ff00139c700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7feffbfff700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9f030 0x55c5f1aade20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fefe8099620 tx=0x7fefe800cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8444c78bee00640a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7fefdbfff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7fefdbfff700 1 -- 172.21.15.3:0/437157615 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe8099920 con 0x55c5f1a9f030 2023-09-10T01:16:29.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feff0016040 con 0x55c5f1a9f030 2023-09-10T01:16:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7fefdbfff700 1 -- 172.21.15.3:0/437157615 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fefe809d750 con 0x55c5f1a9f030 2023-09-10T01:16:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7fefdbfff700 1 -- 172.21.15.3:0/437157615 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefe80a1990 con 0x55c5f1a9f030 2023-09-10T01:16:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7ff016643b80 10 client.28676 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7feff4026050 con 0x55c5f1a9f030 2023-09-10T01:16:29.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7fefdbfff700 1 -- 172.21.15.3:0/437157615 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feff0016040 con 0x55c5f1a9f030 2023-09-10T01:16:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.121+0000 7fefdbfff700 1 -- 172.21.15.3:0/437157615 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fefe80c9e40 con 0x55c5f1a9f030 2023-09-10T01:16:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff016643b80 10 client.28676 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff016643b80 10 client.28676 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:16:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff016643b80 20 client.28676 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff016643b80 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1b0dc10 0x55c5f1b10030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff016643b80 4 client.28676 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:16:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c5f19e9920 con 0x55c5f1b0dc10 2023-09-10T01:16:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff000b9b700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1b0dc10 0x55c5f1b10030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7fefdbfff700 10 client.28676 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:16:29.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7ff000b9b700 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1b0dc10 0x55c5f1b10030 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4709587ed0fd9f87 server_cookie=795669ad4b10195c in_seq=0 out_seq=0 2023-09-10T01:16:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7fefdbfff700 1 -- 172.21.15.3:0/437157615 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c5f19e9920 con 0x55c5f1b0dc10 2023-09-10T01:16:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.122+0000 7fefdbfff700 10 client.28676 handle_command_reply: tid=0 2023-09-10T01:16:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 client.28676 shutdown 2023-09-10T01:16:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9f030 msgr2=0x55c5f1aade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9f030 0x55c5f1aade20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fefe8099620 tx=0x7fefe800cea0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 shutdown_connections 2023-09-10T01:16:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55c5f1b0dc10 0x55c5f1b10030 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c5f1aa8e40 0x55c5f1aae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7fefdbfff700 0 client.28676 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:16:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5f1a9e710 0x55c5f1aaf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 --2- 172.21.15.3:0/437157615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5f1a9f030 0x55c5f1aade20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 >> 172.21.15.3:0/437157615 conn(0x55c5f1a13090 msgr2=0x55c5f1aa0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 shutdown_connections 2023-09-10T01:16:29.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 1 -- 172.21.15.3:0/437157615 wait complete. 2023-09-10T01:16:29.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.123+0000 7ff016643b80 20 client.28676 trim_cache size 0 max 16384 2023-09-10T01:16:29.139 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:16:29.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fefec0a0830 msgr2=0x7fefec0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fefec0a0830 0x7fefec0a2cf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feff4009790 tx=0x7feff4004af0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 msgr2=0x7feffc196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc196bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fefe800ec10 tx=0x7fefe800ef20 comp rx=0 tx=0).stop 2023-09-10T01:16:29.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 shutdown_connections 2023-09-10T01:16:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fefec0a0830 0x7fefec0a2cf0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc106650 0x7feffc199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feffc100640 0x7feffc196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 --2- 172.21.15.3:0/3007658651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feffc101000 0x7feffc196bd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 >> 172.21.15.3:0/3007658651 conn(0x7feffc0fc170 msgr2=0x7feffc0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 shutdown_connections 2023-09-10T01:16:29.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.124+0000 7ff002dff700 1 -- 172.21.15.3:0/3007658651 wait complete. 2023-09-10T01:16:29.145 INFO:tasks.workunit.client.0.smithi003.stdout: 10 3754 38.65 MB/sec warmup 10 sec latency 341.718 ms 2023-09-10T01:16:29.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:16:29.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.427+0000 7fbd893ff700 1 -- 172.21.15.3:0/349432325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd84073350 msgr2=0x7fbd8410c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.427+0000 7fbd893ff700 1 --2- 172.21.15.3:0/349432325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd84073350 0x7fbd8410c940 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009990 tx=0x7fbd6c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.427+0000 7fbd893ff700 1 -- 172.21.15.3:0/349432325 shutdown_connections 2023-09-10T01:16:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.427+0000 7fbd893ff700 1 --2- 172.21.15.3:0/349432325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd84072a30 0x7fbd84072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.427+0000 7fbd893ff700 1 --2- 172.21.15.3:0/349432325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd8410ce80 0x7fbd8410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.427+0000 7fbd893ff700 1 --2- 172.21.15.3:0/349432325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd84073350 0x7fbd8410c940 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.427+0000 7fbd893ff700 1 -- 172.21.15.3:0/349432325 >> 172.21.15.3:0/349432325 conn(0x7fbd840fc170 msgr2=0x7fbd840fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.429+0000 7fbd893ff700 1 -- 172.21.15.3:0/349432325 shutdown_connections 2023-09-10T01:16:29.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.429+0000 7fbd893ff700 1 -- 172.21.15.3:0/349432325 wait complete. 2023-09-10T01:16:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.430+0000 7fbd893ff700 1 Processor -- start 2023-09-10T01:16:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.431+0000 7fbd893ff700 1 -- start start 2023-09-10T01:16:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.431+0000 7fbd893ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd84072a30 0x7fbd84105ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.431+0000 7fbd893ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd84073350 0x7fbd84106010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.431+0000 7fbd893ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd8410ce80 0x7fbd84100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.431+0000 7fbd893ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd84101340 con 0x7fbd84072a30 2023-09-10T01:16:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.431+0000 7fbd893ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd841014b0 con 0x7fbd8410ce80 2023-09-10T01:16:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.431+0000 7fbd893ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd84101670 con 0x7fbd84073350 2023-09-10T01:16:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd8410ce80 0x7fbd84100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd8410ce80 0x7fbd84100b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59474/0 (socket says 172.21.15.3:59474) 2023-09-10T01:16:29.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 -- 172.21.15.3:0/3638085071 learned_addr learned my addr 172.21.15.3:0/3638085071 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd827fc700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd84073350 0x7fbd84106010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd84072a30 0x7fbd84105ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 -- 172.21.15.3:0/3638085071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd84073350 msgr2=0x7fbd84106010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd84073350 0x7fbd84106010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 -- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd84072a30 msgr2=0x7fbd84105ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd84072a30 0x7fbd84105ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd837fe700 1 -- 172.21.15.3:0/3638085071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd74009c40 con 0x7fbd8410ce80 2023-09-10T01:16:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.432+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd84072a30 0x7fbd84105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:16:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.433+0000 7fbd837fe700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd8410ce80 0x7fbd84100b70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c370 tx=0x7fbd7400c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.433+0000 7fbd7bfff700 1 -- 172.21.15.3:0/3638085071 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74017070 con 0x7fbd8410ce80 2023-09-10T01:16:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.433+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c0095f0 con 0x7fbd8410ce80 2023-09-10T01:16:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.433+0000 7fbd7bfff700 1 -- 172.21.15.3:0/3638085071 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd7400f040 con 0x7fbd8410ce80 2023-09-10T01:16:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.433+0000 7fbd7bfff700 1 -- 172.21.15.3:0/3638085071 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7400e050 con 0x7fbd8410ce80 2023-09-10T01:16:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.433+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8410ba10 con 0x7fbd8410ce80 2023-09-10T01:16:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.438+0000 7fbd7bfff700 1 -- 172.21.15.3:0/3638085071 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fbd74004520 con 0x7fbd8410ce80 2023-09-10T01:16:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.439+0000 7fbd9cddfb80 1 Processor -- start 2023-09-10T01:16:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.440+0000 7fbd9cddfb80 1 -- start start 2023-09-10T01:16:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.440+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d097710 0x55a67d097af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.440+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d098030 0x55a67d0a1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.440+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d0a1e40 0x55a67d0a4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.440+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67cff0730 con 0x55a67d0a1e40 2023-09-10T01:16:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.440+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67cfee050 con 0x55a67d097710 2023-09-10T01:16:29.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.441+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67cfe9940 con 0x55a67d098030 2023-09-10T01:16:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.441+0000 7fbd82ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d098030 0x55a67d0a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.441+0000 7fbd827fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d097710 0x55a67d097af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.441+0000 7fbd82ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d098030 0x55a67d0a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45618/0 (socket says 172.21.15.3:45618) 2023-09-10T01:16:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.441+0000 7fbd82ffd700 1 -- 172.21.15.3:0/3594083127 learned_addr learned my addr 172.21.15.3:0/3594083127 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.441+0000 7fbd827fc700 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d0a1e40 0x55a67d0a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.442+0000 7fbd82ffd700 1 -- 172.21.15.3:0/3594083127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d097710 msgr2=0x55a67d097af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.442+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d097710 0x55a67d097af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.442+0000 7fbd82ffd700 1 -- 172.21.15.3:0/3594083127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d0a1e40 msgr2=0x55a67d0a4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.442+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d0a1e40 0x55a67d0a4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.442+0000 7fbd82ffd700 1 -- 172.21.15.3:0/3594083127 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a67ce15de0 con 0x55a67d098030 2023-09-10T01:16:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.442+0000 7fbd7bfff700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbd700a0870 0x7fbd700a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.442+0000 7fbd7bfff700 1 -- 172.21.15.3:0/3638085071 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fbd740cbc80 con 0x7fbd8410ce80 2023-09-10T01:16:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.443+0000 7fbd827fc700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbd700a0870 0x7fbd700a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.444+0000 7fbd827fc700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbd700a0870 0x7fbd700a2d30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbd84102340 tx=0x7fbd6c005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.444+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d098030 0x55a67d0a1900 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009ce0 tx=0x7fbd7c00e2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ed28e25e79b4cb9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.444+0000 7fbd797fa700 1 -- 172.21.15.3:0/3594083127 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c00eb70 con 0x55a67d098030 2023-09-10T01:16:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.450+0000 7fbd797fa700 1 -- 172.21.15.3:0/3594083127 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd7c00ecd0 con 0x55a67d098030 2023-09-10T01:16:29.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.450+0000 7fbd797fa700 1 -- 172.21.15.3:0/3594083127 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c0106c0 con 0x55a67d098030 2023-09-10T01:16:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.451+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/3594083127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d098030 msgr2=0x55a67d0a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.451+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d098030 0x55a67d0a1900 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009ce0 tx=0x7fbd7c00e2f0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/3594083127 shutdown_connections 2023-09-10T01:16:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d0a1e40 0x55a67d0a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d097710 0x55a67d097af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/3594083127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d098030 0x55a67d0a1900 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/3594083127 >> 172.21.15.3:0/3594083127 conn(0x55a67d00c090 msgr2=0x55a67d092bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/3594083127 shutdown_connections 2023-09-10T01:16:29.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/3594083127 wait complete. 2023-09-10T01:16:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.452+0000 7fbd9cddfb80 1 -- start start 2023-09-10T01:16:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.453+0000 7fbd9cddfb80 1 Processor -- start 2023-09-10T01:16:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d097710 0x55a67d0a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d098030 0x55a67d09e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d0a1e40 0x55a67d09eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67cf87b70 con 0x55a67d097710 2023-09-10T01:16:29.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67cf87870 con 0x55a67d098030 2023-09-10T01:16:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67ce6b920 con 0x55a67d0a1e40 2023-09-10T01:16:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd9cddfb80 10 client.0 _ll_register_callbacks cb 0x55a67ced1a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd82ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d097710 0x55a67d0a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.454+0000 7fbd82ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d097710 0x55a67d0a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43296/0 (socket says 172.21.15.3:43296) 2023-09-10T01:16:29.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.455+0000 7fbd82ffd700 1 -- 172.21.15.3:0/4092570311 learned_addr learned my addr 172.21.15.3:0/4092570311 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.455+0000 7fbd827fc700 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d098030 0x55a67d09e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.455+0000 7fbd82ffd700 1 -- 172.21.15.3:0/4092570311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d0a1e40 msgr2=0x55a67d09eea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:16:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.455+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d0a1e40 0x55a67d09eea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.455+0000 7fbd82ffd700 1 -- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d098030 msgr2=0x55a67d09e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.455+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d098030 0x55a67d09e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.455+0000 7fbd82ffd700 1 -- 172.21.15.3:0/4092570311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7c009940 con 0x55a67d097710 2023-09-10T01:16:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.456+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d097710 0x55a67d0a16e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000c00 tx=0x7fbd7c011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8837b0cb5f939498 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.456+0000 7fbd6a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:16:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.456+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/4092570311 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c020070 con 0x55a67d097710 2023-09-10T01:16:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.456+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a67cf414f0 con 0x55a67d097710 2023-09-10T01:16:29.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.456+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/4092570311 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fbd7c00ee40 con 0x55a67d097710 2023-09-10T01:16:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.457+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/4092570311 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd7c0092a0 con 0x55a67d097710 2023-09-10T01:16:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.457+0000 7fbd9cddfb80 10 client.18594 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:29.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.457+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55a67cdfec60 con 0x55a67d097710 2023-09-10T01:16:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/4092570311 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a67cf414f0 con 0x55a67d097710 2023-09-10T01:16:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/4092570311 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbd7c004cf0 con 0x55a67d097710 2023-09-10T01:16:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd9cddfb80 10 client.18594 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:29.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd9cddfb80 10 client.18594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd9cddfb80 20 client.18594 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d0b1750 0x55a67d0b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd9cddfb80 4 client.18594 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:29.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.459+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a67cffe8a0 con 0x55a67d0b1750 2023-09-10T01:16:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.466+0000 7fbd827fc700 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d0b1750 0x55a67d0b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.469+0000 7fbd6a7fc700 10 client.18594 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:29.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.469+0000 7fbd827fc700 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d0b1750 0x55a67d0b3bb0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47b6061b5fbee4f5 server_cookie=b088366cbb3c0321 in_seq=0 out_seq=0 2023-09-10T01:16:29.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.472+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/4092570311 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a67cffe8a0 con 0x55a67d0b1750 2023-09-10T01:16:29.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.472+0000 7fbd6a7fc700 10 client.18594 handle_command_reply: tid=0 2023-09-10T01:16:29.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.472+0000 7fbd9cddfb80 1 client.18594 shutdown 2023-09-10T01:16:29.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d097710 msgr2=0x55a67d0a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d097710 0x55a67d0a16e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000c00 tx=0x7fbd7c011750 comp rx=0 tx=0).stop 2023-09-10T01:16:29.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 shutdown_connections 2023-09-10T01:16:29.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d0b1750 0x55a67d0b3bb0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d0a1e40 0x55a67d09eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d097710 0x55a67d0a16e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/4092570311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d098030 0x55a67d09e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd6a7fc700 0 client.18594 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:29.476 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 >> 172.21.15.3:0/4092570311 conn(0x55a67d00c090 msgr2=0x55a67d092780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 shutdown_connections 2023-09-10T01:16:29.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.474+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/4092570311 wait complete. 2023-09-10T01:16:29.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.475+0000 7fbd9cddfb80 20 client.18594 trim_cache size 0 max 16384 2023-09-10T01:16:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.497+0000 7fbd9cddfb80 1 Processor -- start 2023-09-10T01:16:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.497+0000 7fbd9cddfb80 1 -- start start 2023-09-10T01:16:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.498+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 0x55a67d093e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.498+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.498+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d091dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.498+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67d0ae250 con 0x55a67d098030 2023-09-10T01:16:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.498+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67d0aeea0 con 0x55a67d0a1e40 2023-09-10T01:16:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.498+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67d09ff70 con 0x55a67d097710 2023-09-10T01:16:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.499+0000 7fbd837fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d091dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.499+0000 7fbd837fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d091dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59504/0 (socket says 172.21.15.3:59504) 2023-09-10T01:16:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.499+0000 7fbd837fe700 1 -- 172.21.15.3:0/916032877 learned_addr learned my addr 172.21.15.3:0/916032877 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.499+0000 7fbd827fc700 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.500+0000 7fbd827fc700 1 -- 172.21.15.3:0/916032877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 msgr2=0x55a67d093e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.500+0000 7fbd827fc700 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 0x55a67d093e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.500+0000 7fbd827fc700 1 -- 172.21.15.3:0/916032877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 msgr2=0x55a67d091dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.500+0000 7fbd827fc700 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d091dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.500+0000 7fbd827fc700 1 -- 172.21.15.3:0/916032877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a67ce3c250 con 0x55a67d098030 2023-09-10T01:16:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.501+0000 7fbd827fc700 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0a8a30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b2e0 tx=0x7fbd6c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3370b3208a3e20f5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.501+0000 7fbd79ffb700 1 -- 172.21.15.3:0/916032877 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c036050 con 0x55a67d098030 2023-09-10T01:16:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.501+0000 7fbd79ffb700 1 -- 172.21.15.3:0/916032877 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd6c02f040 con 0x55a67d098030 2023-09-10T01:16:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.501+0000 7fbd79ffb700 1 -- 172.21.15.3:0/916032877 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c035d60 con 0x55a67d098030 2023-09-10T01:16:29.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.502+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/916032877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 msgr2=0x55a67d0a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.502+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0a8a30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b2e0 tx=0x7fbd6c009310 comp rx=0 tx=0).stop 2023-09-10T01:16:29.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.502+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/916032877 shutdown_connections 2023-09-10T01:16:29.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.502+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 0x55a67d093e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.502+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d091dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.502+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/916032877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0a8a30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.502+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/916032877 >> 172.21.15.3:0/916032877 conn(0x55a67d00c090 msgr2=0x55a67d0adb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.503+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/916032877 shutdown_connections 2023-09-10T01:16:29.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.503+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/916032877 wait complete. 2023-09-10T01:16:29.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.503+0000 7fbd9cddfb80 1 -- start start 2023-09-10T01:16:29.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.503+0000 7fbd9cddfb80 1 Processor -- start 2023-09-10T01:16:29.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.504+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 0x55a67d0ff990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.504+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0ffed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.504+0000 7fbd9cddfb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d0fe670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.504+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67d0fed80 con 0x55a67d098030 2023-09-10T01:16:29.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.504+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67d0feef0 con 0x55a67d0a1e40 2023-09-10T01:16:29.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.504+0000 7fbd9cddfb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a67d0ff0b0 con 0x55a67d097710 2023-09-10T01:16:29.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.504+0000 7fbd9cddfb80 10 client.0 _ll_register_callbacks cb 0x7fbd740097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:29.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.505+0000 7fbd827fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0ffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.505+0000 7fbd827fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0ffed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43310/0 (socket says 172.21.15.3:43310) 2023-09-10T01:16:29.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.505+0000 7fbd827fc700 1 -- 172.21.15.3:0/883094511 learned_addr learned my addr 172.21.15.3:0/883094511 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:29.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.506+0000 7fbd827fc700 1 -- 172.21.15.3:0/883094511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 msgr2=0x55a67d0ff990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:29.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.506+0000 7fbd827fc700 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 0x55a67d0ff990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.506+0000 7fbd827fc700 1 -- 172.21.15.3:0/883094511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 msgr2=0x55a67d0fe670 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:16:29.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.506+0000 7fbd827fc700 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d0fe670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.506+0000 7fbd827fc700 1 -- 172.21.15.3:0/883094511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c020040 con 0x55a67d098030 2023-09-10T01:16:29.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.507+0000 7fbd827fc700 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0ffed0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0098e0 tx=0x7fbd6c036100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbd17b92fce9365 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:29.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.507+0000 7fbd6a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:16:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.508+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/883094511 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c044070 con 0x55a67d098030 2023-09-10T01:16:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.508+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/883094511 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fbd6c0364a0 con 0x55a67d098030 2023-09-10T01:16:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.508+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd7c00ecd0 con 0x55a67d098030 2023-09-10T01:16:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.508+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/883094511 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd6c02c070 con 0x55a67d098030 2023-09-10T01:16:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.509+0000 7fbd9cddfb80 10 client.18600 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.509+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55a67ce142b0 con 0x55a67d098030 2023-09-10T01:16:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/883094511 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd7c00ecd0 con 0x55a67d098030 2023-09-10T01:16:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/883094511 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbd6c047020 con 0x55a67d098030 2023-09-10T01:16:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd9cddfb80 10 client.18600 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd9cddfb80 10 client.18600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd9cddfb80 20 client.18600 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d117610 0x55a67d119a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd9cddfb80 4 client.18600 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:16:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a67cfe2920 con 0x55a67d117610 2023-09-10T01:16:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.510+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d117610 0x55a67d119a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.511+0000 7fbd6a7fc700 10 client.18600 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.511+0000 7fbd82ffd700 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d117610 0x55a67d119a00 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cce62f9d04ea7d0c server_cookie=c7d8b54ba96d67ce in_seq=0 out_seq=0 2023-09-10T01:16:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd6a7fc700 1 -- 172.21.15.3:0/883094511 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a67cfe2920 con 0x55a67d117610 2023-09-10T01:16:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd6a7fc700 10 client.18600 handle_command_reply: tid=0 2023-09-10T01:16:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 client.18600 shutdown 2023-09-10T01:16:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 msgr2=0x55a67d0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0ffed0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0098e0 tx=0x7fbd6c036100 comp rx=0 tx=0).stop 2023-09-10T01:16:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 shutdown_connections 2023-09-10T01:16:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55a67d117610 0x55a67d119a00 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a67d097710 0x55a67d0ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a67d0a1e40 0x55a67d0fe670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 --2- 172.21.15.3:0/883094511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a67d098030 0x55a67d0ffed0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 >> 172.21.15.3:0/883094511 conn(0x55a67d00c090 msgr2=0x55a67d091dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.516+0000 7fbd6a7fc700 0 client.18600 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.517+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 shutdown_connections 2023-09-10T01:16:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.517+0000 7fbd9cddfb80 1 -- 172.21.15.3:0/883094511 wait complete. 2023-09-10T01:16:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.517+0000 7fbd9cddfb80 20 client.18600 trim_cache size 0 max 16384 2023-09-10T01:16:29.522 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:16:29.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbd700a0870 msgr2=0x7fbd700a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbd700a0870 0x7fbd700a2d30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbd84102340 tx=0x7fbd6c005fd0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd8410ce80 msgr2=0x7fbd84100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:29.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd8410ce80 0x7fbd84100b70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c370 tx=0x7fbd7400c680 comp rx=0 tx=0).stop 2023-09-10T01:16:29.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 shutdown_connections 2023-09-10T01:16:29.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fbd700a0870 0x7fbd700a2d30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd8410ce80 0x7fbd84100b70 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd84072a30 0x7fbd84105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 --2- 172.21.15.3:0/3638085071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd84073350 0x7fbd84106010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 >> 172.21.15.3:0/3638085071 conn(0x7fbd840fc170 msgr2=0x7fbd84197e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 shutdown_connections 2023-09-10T01:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:29.518+0000 7fbd893ff700 1 -- 172.21.15.3:0/3638085071 wait complete. 2023-09-10T01:16:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:29 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/152117812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:16:29.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:29 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/152117812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:16:29.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:29 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/152117812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:16:30.145 INFO:tasks.workunit.client.0.smithi003.stdout: 10 3991 35.32 MB/sec warmup 11 sec latency 185.087 ms 2023-09-10T01:16:30.539 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:16:30.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:16:30.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:30 smithi003 ceph-mon[97654]: pgmap v415: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 20 MiB/s wr, 578 op/s 2023-09-10T01:16:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:30 smithi136 ceph-mon[133708]: pgmap v415: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 20 MiB/s wr, 578 op/s 2023-09-10T01:16:30.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:30 smithi092 ceph-mon[134067]: pgmap v415: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 20 MiB/s wr, 578 op/s 2023-09-10T01:16:30.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 -- 172.21.15.3:0/65816820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d00729b0 msgr2=0x7fd9d0072d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/65816820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d00729b0 0x7fd9d0072d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009990 tx=0x7fd9cc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 -- 172.21.15.3:0/65816820 shutdown_connections 2023-09-10T01:16:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/65816820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d00729b0 0x7fd9d0072d90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/65816820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d010cf10 0x7fd9d010f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/65816820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00732d0 0x7fd9d010c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 -- 172.21.15.3:0/65816820 >> 172.21.15.3:0/65816820 conn(0x7fd9d00fc140 msgr2=0x7fd9d00fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:30.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 -- 172.21.15.3:0/65816820 shutdown_connections 2023-09-10T01:16:30.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.858+0000 7fd9d83ff700 1 -- 172.21.15.3:0/65816820 wait complete. 2023-09-10T01:16:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.859+0000 7fd9d83ff700 1 Processor -- start 2023-09-10T01:16:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.859+0000 7fd9d83ff700 1 -- start start 2023-09-10T01:16:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d83ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00729b0 0x7fd9d01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d83ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d00732d0 0x7fd9d0198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d83ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d010cf10 0x7fd9d019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d83ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d019d940 con 0x7fd9d00729b0 2023-09-10T01:16:30.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d83ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d019dab0 con 0x7fd9d00732d0 2023-09-10T01:16:30.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d83ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d019dc70 con 0x7fd9d010cf10 2023-09-10T01:16:30.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00729b0 0x7fd9d01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00729b0 0x7fd9d01989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43332/0 (socket says 172.21.15.3:43332) 2023-09-10T01:16:30.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 -- 172.21.15.3:0/2231196082 learned_addr learned my addr 172.21.15.3:0/2231196082 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:30.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d599a700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d00732d0 0x7fd9d0198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d699c700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d010cf10 0x7fd9d019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 -- 172.21.15.3:0/2231196082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d010cf10 msgr2=0x7fd9d019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d010cf10 0x7fd9d019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 -- 172.21.15.3:0/2231196082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d00732d0 msgr2=0x7fd9d0198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d00732d0 0x7fd9d0198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d619b700 1 -- 172.21.15.3:0/2231196082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0009520 con 0x7fd9d00729b0 2023-09-10T01:16:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.860+0000 7fd9d699c700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d010cf10 0x7fd9d019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.861+0000 7fd9d619b700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00729b0 0x7fd9d01989a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc005910 tx=0x7fd9cc005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.861+0000 7fd9c77fe700 1 -- 172.21.15.3:0/2231196082 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc01d070 con 0x7fd9d00729b0 2023-09-10T01:16:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.861+0000 7fd9c77fe700 1 -- 172.21.15.3:0/2231196082 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9cc022470 con 0x7fd9d00729b0 2023-09-10T01:16:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.861+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c8009c40 con 0x7fd9d00729b0 2023-09-10T01:16:30.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.861+0000 7fd9c77fe700 1 -- 172.21.15.3:0/2231196082 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc017650 con 0x7fd9d00729b0 2023-09-10T01:16:30.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.861+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc0095f0 con 0x7fd9d00729b0 2023-09-10T01:16:30.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.866+0000 7fd9c77fe700 1 -- 172.21.15.3:0/2231196082 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd9cc0225e0 con 0x7fd9d00729b0 2023-09-10T01:16:30.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.867+0000 7fd9ebca5b80 1 Processor -- start 2023-09-10T01:16:30.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.867+0000 7fd9ebca5b80 1 -- start start 2023-09-10T01:16:30.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534d790 0x55563534db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555635357ec0 0x55563535a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556352a67b0 con 0x55563534d790 2023-09-10T01:16:30.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556352a40d0 con 0x55563534e0b0 2023-09-10T01:16:30.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563529f9c0 con 0x555635357ec0 2023-09-10T01:16:30.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534d790 0x55563534db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d619b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d599a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555635357ec0 0x55563535a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534d790 0x55563534db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43340/0 (socket says 172.21.15.3:43340) 2023-09-10T01:16:30.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d599a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555635357ec0 0x55563535a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45662/0 (socket says 172.21.15.3:45662) 2023-09-10T01:16:30.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 -- 172.21.15.3:0/1435617223 learned_addr learned my addr 172.21.15.3:0/1435617223 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:30.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 -- 172.21.15.3:0/1435617223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555635357ec0 msgr2=0x55563535a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 --2- 172.21.15.3:0/1435617223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555635357ec0 0x55563535a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 -- 172.21.15.3:0/1435617223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 msgr2=0x555635357980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 --2- 172.21.15.3:0/1435617223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.868+0000 7fd9d699c700 1 -- 172.21.15.3:0/1435617223 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556350cbe60 con 0x55563534d790 2023-09-10T01:16:30.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.869+0000 7fd9c77fe700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9bc0a0830 0x7fd9bc0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.869+0000 7fd9c77fe700 1 -- 172.21.15.3:0/2231196082 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd9cc0cd6a0 con 0x7fd9d00729b0 2023-09-10T01:16:30.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.869+0000 7fd9d699c700 1 --2- 172.21.15.3:0/1435617223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534d790 0x55563534db70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8007630 tx=0x7fd9c8009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420ffe86c7dfb0bd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:30.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.869+0000 7fd9d619b700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9bc0a0830 0x7fd9bc0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.869+0000 7fd9c4bfe700 1 -- 172.21.15.3:0/1435617223 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c800e070 con 0x55563534d790 2023-09-10T01:16:30.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.869+0000 7fd9c4bfe700 1 -- 172.21.15.3:0/1435617223 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9c800f040 con 0x55563534d790 2023-09-10T01:16:30.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.869+0000 7fd9c4bfe700 1 -- 172.21.15.3:0/1435617223 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8015b70 con 0x55563534d790 2023-09-10T01:16:30.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1435617223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534d790 msgr2=0x55563534db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1435617223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534d790 0x55563534db70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8007630 tx=0x7fd9c8009920 comp rx=0 tx=0).stop 2023-09-10T01:16:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9d619b700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9bc0a0830 0x7fd9bc0a2cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0caab0 tx=0x7fd9cc022f90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1435617223 shutdown_connections 2023-09-10T01:16:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1435617223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555635357ec0 0x55563535a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1435617223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534d790 0x55563534db70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1435617223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1435617223 >> 172.21.15.3:0/1435617223 conn(0x5556352c2110 msgr2=0x555635348c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:30.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1435617223 shutdown_connections 2023-09-10T01:16:30.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.870+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1435617223 wait complete. 2023-09-10T01:16:30.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.871+0000 7fd9ebca5b80 1 -- start start 2023-09-10T01:16:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.871+0000 7fd9ebca5b80 1 Processor -- start 2023-09-10T01:16:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x555635357930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534e0b0 0x555635354b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555635357ec0 0x555635355230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563523dbf0 con 0x55563534e0b0 2023-09-10T01:16:30.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563523d8f0 con 0x555635357ec0 2023-09-10T01:16:30.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556351219a0 con 0x55563534d790 2023-09-10T01:16:30.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9ebca5b80 10 client.0 _ll_register_callbacks cb 0x555635187b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:30.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x555635357930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x555635357930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45672/0 (socket says 172.21.15.3:45672) 2023-09-10T01:16:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 -- 172.21.15.3:0/1621735323 learned_addr learned my addr 172.21.15.3:0/1621735323 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d599a700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534e0b0 0x555635354b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555635357ec0 0x555635355230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 -- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555635357ec0 msgr2=0x555635355230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555635357ec0 0x555635355230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 -- 172.21.15.3:0/1621735323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534e0b0 msgr2=0x555635354b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534e0b0 0x555635354b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d699c700 1 -- 172.21.15.3:0/1621735323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c80092e0 con 0x55563534d790 2023-09-10T01:16:30.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.872+0000 7fd9d599a700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534e0b0 0x555635354b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:30.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9d699c700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x555635357930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8015cb0 tx=0x7fd9c8009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d5c6efc4984b0da server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:30.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:16:30.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/1621735323 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c800e040 con 0x55563534d790 2023-09-10T01:16:30.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/1621735323 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd9c8013780 con 0x55563534d790 2023-09-10T01:16:30.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556351f7570 con 0x55563534d790 2023-09-10T01:16:30.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/1621735323 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9c8016ea0 con 0x55563534d790 2023-09-10T01:16:30.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9ebca5b80 10 client.28579 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:30.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5556350b4ce0 con 0x55563534d790 2023-09-10T01:16:30.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.873+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/1621735323 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5556351f7570 con 0x55563534d790 2023-09-10T01:16:30.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/1621735323 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd9c801c030 con 0x55563534d790 2023-09-10T01:16:30.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9ebca5b80 10 client.28579 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:30.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9ebca5b80 10 client.28579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9ebca5b80 20 client.28579 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555635367990 0x555635369db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9ebca5b80 4 client.28579 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556352b4920 con 0x555635367990 2023-09-10T01:16:30.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.874+0000 7fd9d599a700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555635367990 0x555635369db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.875+0000 7fd9d599a700 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555635367990 0x555635369db0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5db5f57133336e08 server_cookie=e41094e468ff71fa in_seq=0 out_seq=0 2023-09-10T01:16:30.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.875+0000 7fd9b9ffb700 10 client.28579 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:30.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/1621735323 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5556352b4920 con 0x555635367990 2023-09-10T01:16:30.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9b9ffb700 10 client.28579 handle_command_reply: tid=0 2023-09-10T01:16:30.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 client.28579 shutdown 2023-09-10T01:16:30.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 msgr2=0x555635357930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x555635357930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8015cb0 tx=0x7fd9c8009240 comp rx=0 tx=0).stop 2023-09-10T01:16:30.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 shutdown_connections 2023-09-10T01:16:30.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555635367990 0x555635369db0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x555635357930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555635357ec0 0x555635355230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/1621735323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55563534e0b0 0x555635354b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9b9ffb700 0 client.28579 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 >> 172.21.15.3:0/1621735323 conn(0x5556352c2110 msgr2=0x555635348800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:30.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 shutdown_connections 2023-09-10T01:16:30.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.882+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/1621735323 wait complete. 2023-09-10T01:16:30.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.883+0000 7fd9ebca5b80 20 client.28579 trim_cache size 0 max 16384 2023-09-10T01:16:30.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.902+0000 7fd9ebca5b80 1 Processor -- start 2023-09-10T01:16:30.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9ebca5b80 1 -- start start 2023-09-10T01:16:30.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x55563535ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x555635354430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556351f7570 con 0x555635357ec0 2023-09-10T01:16:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555635345450 con 0x55563534e0b0 2023-09-10T01:16:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563534e750 con 0x55563534d790 2023-09-10T01:16:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9d599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9d699c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x55563535ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9d599a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59548/0 (socket says 172.21.15.3:59548) 2023-09-10T01:16:30.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9d599a700 1 -- 172.21.15.3:0/4211739477 learned_addr learned my addr 172.21.15.3:0/4211739477 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:30.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.903+0000 7fd9d699c700 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x555635354430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9d599a700 1 -- 172.21.15.3:0/4211739477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 msgr2=0x55563535ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9d599a700 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x55563535ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9d599a700 1 -- 172.21.15.3:0/4211739477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 msgr2=0x555635354430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9d599a700 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x555635354430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9d599a700 1 -- 172.21.15.3:0/4211739477 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55563536e9e0 con 0x55563534e0b0 2023-09-10T01:16:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9d699c700 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x55563535ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:16:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9d599a700 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357460 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x5556353561f0 tx=0x7fd9c0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2065efa00716c8c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9c53ff700 1 -- 172.21.15.3:0/4211739477 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0016030 con 0x55563534e0b0 2023-09-10T01:16:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9c53ff700 1 -- 172.21.15.3:0/4211739477 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9c0017040 con 0x55563534e0b0 2023-09-10T01:16:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.904+0000 7fd9c53ff700 1 -- 172.21.15.3:0/4211739477 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0004c50 con 0x55563534e0b0 2023-09-10T01:16:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/4211739477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 msgr2=0x555635357460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357460 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x5556353561f0 tx=0x7fd9c0009260 comp rx=0 tx=0).stop 2023-09-10T01:16:30.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/4211739477 shutdown_connections 2023-09-10T01:16:30.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534d790 0x55563535ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x555635354430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/4211739477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534e0b0 0x555635357460 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/4211739477 >> 172.21.15.3:0/4211739477 conn(0x5556352c2110 msgr2=0x555635348770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:30.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/4211739477 shutdown_connections 2023-09-10T01:16:30.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.905+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/4211739477 wait complete. 2023-09-10T01:16:30.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 -- start start 2023-09-10T01:16:30.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 Processor -- start 2023-09-10T01:16:30.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534d790 0x55563535d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534e0b0 0x55563535dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x5556353b5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556353b5d50 con 0x555635357ec0 2023-09-10T01:16:30.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556353b5e90 con 0x55563534d790 2023-09-10T01:16:30.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.906+0000 7fd9ebca5b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556353b6050 con 0x55563534e0b0 2023-09-10T01:16:30.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9ebca5b80 10 client.0 _ll_register_callbacks cb 0x7fd9c8007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:30.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d699c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534d790 0x55563535d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d599a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534e0b0 0x55563535dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d699c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534d790 0x55563535d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59550/0 (socket says 172.21.15.3:59550) 2023-09-10T01:16:30.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d699c700 1 -- 172.21.15.3:0/2643312899 learned_addr learned my addr 172.21.15.3:0/2643312899 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:30.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d699c700 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x5556353b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d599a700 1 -- 172.21.15.3:0/2643312899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534d790 msgr2=0x55563535d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d599a700 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534d790 0x55563535d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d599a700 1 -- 172.21.15.3:0/2643312899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 msgr2=0x5556353b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:30.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d599a700 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x5556353b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d599a700 1 -- 172.21.15.3:0/2643312899 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c00044e0 con 0x55563534e0b0 2023-09-10T01:16:30.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9d599a700 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534e0b0 0x55563535dc80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0000c00 tx=0x7fd9c001f110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4193e0a220b31eb1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:30.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.907+0000 7fd9b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:16:30.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/2643312899 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0016040 con 0x55563534e0b0 2023-09-10T01:16:30.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9c800f040 con 0x55563534e0b0 2023-09-10T01:16:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/2643312899 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd9c001f4c0 con 0x55563534e0b0 2023-09-10T01:16:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/2643312899 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9c0003960 con 0x55563534e0b0 2023-09-10T01:16:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9ebca5b80 10 client.28585 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5556350f22d0 con 0x55563534e0b0 2023-09-10T01:16:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/2643312899 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9c800f040 con 0x55563534e0b0 2023-09-10T01:16:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.908+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/2643312899 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd9c00050c0 con 0x55563534e0b0 2023-09-10T01:16:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9ebca5b80 10 client.28585 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9ebca5b80 10 client.28585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9ebca5b80 20 client.28585 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5556353bd8e0 0x5556353bfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9ebca5b80 4 client.28585 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:16:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5556352989a0 con 0x5556353bd8e0 2023-09-10T01:16:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9d699c700 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5556353bd8e0 0x5556353bfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9d699c700 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5556353bd8e0 0x5556353bfd00 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f25fe00d1f74713 server_cookie=e5594f447f64b4b1 in_seq=0 out_seq=0 2023-09-10T01:16:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.909+0000 7fd9b9ffb700 10 client.28585 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.913+0000 7fd9b9ffb700 1 -- 172.21.15.3:0/2643312899 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5556352989a0 con 0x5556353bd8e0 2023-09-10T01:16:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.913+0000 7fd9b9ffb700 10 client.28585 handle_command_reply: tid=0 2023-09-10T01:16:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.913+0000 7fd9ebca5b80 1 client.28585 shutdown 2023-09-10T01:16:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534e0b0 msgr2=0x55563535dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534e0b0 0x55563535dc80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0000c00 tx=0x7fd9c001f110 comp rx=0 tx=0).stop 2023-09-10T01:16:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 shutdown_connections 2023-09-10T01:16:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5556353bd8e0 0x5556353bfd00 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555635357ec0 0x5556353b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55563534d790 0x55563535d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 --2- 172.21.15.3:0/2643312899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55563534e0b0 0x55563535dc80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9b9ffb700 0 client.28585 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:30.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 >> 172.21.15.3:0/2643312899 conn(0x5556352c2110 msgr2=0x555635354430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:30.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 shutdown_connections 2023-09-10T01:16:30.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 1 -- 172.21.15.3:0/2643312899 wait complete. 2023-09-10T01:16:30.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.914+0000 7fd9ebca5b80 20 client.28585 trim_cache size 0 max 16384 2023-09-10T01:16:30.927 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:16:30.927 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:16:30.928 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "79fafc81-0f62-4416-91fa-026aa8e080bc", 2023-09-10T01:16:30.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:16:30.928 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:16:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9bc0a0830 msgr2=0x7fd9bc0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9bc0a0830 0x7fd9bc0a2cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0caab0 tx=0x7fd9cc022f90 comp rx=0 tx=0).stop 2023-09-10T01:16:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00729b0 msgr2=0x7fd9d01989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00729b0 0x7fd9d01989a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc005910 tx=0x7fd9cc005720 comp rx=0 tx=0).stop 2023-09-10T01:16:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 shutdown_connections 2023-09-10T01:16:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd9bc0a0830 0x7fd9bc0a2cf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d010cf10 0x7fd9d019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd9d00729b0 0x7fd9d01989a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.930 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 --2- 172.21.15.3:0/2231196082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9d00732d0 0x7fd9d0198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 >> 172.21.15.3:0/2231196082 conn(0x7fd9d00fc140 msgr2=0x7fd9d010f000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 shutdown_connections 2023-09-10T01:16:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:30.915+0000 7fd9d83ff700 1 -- 172.21.15.3:0/2231196082 wait complete. 2023-09-10T01:16:30.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:16:31.145 INFO:tasks.workunit.client.0.smithi003.stdout: 10 4290 33.31 MB/sec warmup 12 sec latency 114.783 ms 2023-09-10T01:16:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 -- 172.21.15.3:0/658073061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681021c0 msgr2=0x7fd36810a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 --2- 172.21.15.3:0/658073061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681021c0 0x7fd36810a6a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd35c009990 tx=0x7fd35c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 -- 172.21.15.3:0/658073061 shutdown_connections 2023-09-10T01:16:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 --2- 172.21.15.3:0/658073061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd36810ac70 0x7fd36810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 --2- 172.21.15.3:0/658073061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3681018a0 0x7fd368101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 --2- 172.21.15.3:0/658073061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681021c0 0x7fd36810a6a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 -- 172.21.15.3:0/658073061 >> 172.21.15.3:0/658073061 conn(0x7fd368076710 msgr2=0x7fd368076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 -- 172.21.15.3:0/658073061 shutdown_connections 2023-09-10T01:16:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.212+0000 7fd36cdff700 1 -- 172.21.15.3:0/658073061 wait complete. 2023-09-10T01:16:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.213+0000 7fd36cdff700 1 Processor -- start 2023-09-10T01:16:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36cdff700 1 -- start start 2023-09-10T01:16:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36cdff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681018a0 0x7fd36819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36cdff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3681021c0 0x7fd36819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36cdff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd36810ac70 0x7fd368198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36cdff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368198e80 con 0x7fd3681021c0 2023-09-10T01:16:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36cdff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368198ff0 con 0x7fd3681018a0 2023-09-10T01:16:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36cdff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681991b0 con 0x7fd36810ac70 2023-09-10T01:16:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681018a0 0x7fd36819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681018a0 0x7fd36819e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59570/0 (socket says 172.21.15.3:59570) 2023-09-10T01:16:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd36659c700 1 -- 172.21.15.3:0/129930785 learned_addr learned my addr 172.21.15.3:0/129930785 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd365d9b700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3681021c0 0x7fd36819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.214+0000 7fd366d9d700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd36810ac70 0x7fd368198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd366d9d700 1 -- 172.21.15.3:0/129930785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681018a0 msgr2=0x7fd36819e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd366d9d700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681018a0 0x7fd36819e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd366d9d700 1 -- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3681021c0 msgr2=0x7fd36819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd366d9d700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3681021c0 0x7fd36819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd366d9d700 1 -- 172.21.15.3:0/129930785 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd358009520 con 0x7fd36810ac70 2023-09-10T01:16:31.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd36659c700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681018a0 0x7fd36819e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd365d9b700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3681021c0 0x7fd36819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd366d9d700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd36810ac70 0x7fd368198810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd36000efc0 tx=0x7fd36000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd3577fe700 1 -- 172.21.15.3:0/129930785 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360018070 con 0x7fd36810ac70 2023-09-10T01:16:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd3577fe700 1 -- 172.21.15.3:0/129930785 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd36000f040 con 0x7fd36810ac70 2023-09-10T01:16:31.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c0095f0 con 0x7fd36810ac70 2023-09-10T01:16:31.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd3577fe700 1 -- 172.21.15.3:0/129930785 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000cd40 con 0x7fd36810ac70 2023-09-10T01:16:31.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.215+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3681a55b0 con 0x7fd36810ac70 2023-09-10T01:16:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.219+0000 7fd3577fe700 1 -- 172.21.15.3:0/129930785 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd360003a00 con 0x7fd36810ac70 2023-09-10T01:16:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.220+0000 7fd3577fe700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3500a07b0 0x7fd3500a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.220+0000 7fd3577fe700 1 -- 172.21.15.3:0/129930785 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd3600cbf10 con 0x7fd36810ac70 2023-09-10T01:16:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.220+0000 7fd36659c700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3500a07b0 0x7fd3500a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 Processor -- start 2023-09-10T01:16:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 -- start start 2023-09-10T01:16:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd36659c700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3500a07b0 0x7fd3500a2c70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd358009fd0 tx=0x7fd358009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3ed3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ee0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3e2c730 con 0x55f7b3ed3710 2023-09-10T01:16:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3e2a050 con 0x55f7b3ed4030 2023-09-10T01:16:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.221+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3e25940 con 0x55f7b3edde40 2023-09-10T01:16:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd36659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3ed3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ee0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59584/0 (socket says 172.21.15.3:59584) 2023-09-10T01:16:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd365d9b700 1 -- 172.21.15.3:0/3737522999 learned_addr learned my addr 172.21.15.3:0/3737522999 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd366d9d700 1 -- 172.21.15.3:0/3737522999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 msgr2=0x55f7b3edd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd366d9d700 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd366d9d700 1 -- 172.21.15.3:0/3737522999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 msgr2=0x55f7b3ed3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd366d9d700 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3ed3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd366d9d700 1 -- 172.21.15.3:0/3737522999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7b3c51de0 con 0x55f7b3edde40 2023-09-10T01:16:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.222+0000 7fd36659c700 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3ed3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.223+0000 7fd366d9d700 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ee0230 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd3600c9210 tx=0x7fd360098dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9221136f2a83cc85 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.223+0000 7fd354ff9700 1 -- 172.21.15.3:0/3737522999 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36001b320 con 0x55f7b3edde40 2023-09-10T01:16:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.223+0000 7fd354ff9700 1 -- 172.21.15.3:0/3737522999 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd360004ad0 con 0x55f7b3edde40 2023-09-10T01:16:31.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.223+0000 7fd354ff9700 1 -- 172.21.15.3:0/3737522999 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360099260 con 0x55f7b3edde40 2023-09-10T01:16:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 -- 172.21.15.3:0/3737522999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 msgr2=0x55f7b3ee0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ee0230 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd3600c9210 tx=0x7fd360098dd0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 -- 172.21.15.3:0/3737522999 shutdown_connections 2023-09-10T01:16:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ee0230 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3ed3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/3737522999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 -- 172.21.15.3:0/3737522999 >> 172.21.15.3:0/3737522999 conn(0x55f7b3e48090 msgr2=0x55f7b3ecebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 -- 172.21.15.3:0/3737522999 shutdown_connections 2023-09-10T01:16:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.224+0000 7fd3807a0b80 1 -- 172.21.15.3:0/3737522999 wait complete. 2023-09-10T01:16:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.225+0000 7fd3807a0b80 1 -- start start 2023-09-10T01:16:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.225+0000 7fd3807a0b80 1 Processor -- start 2023-09-10T01:16:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3edaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3edde40 0x55f7b3edb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3dc3b70 con 0x55f7b3edde40 2023-09-10T01:16:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3dc3870 con 0x55f7b3ed4030 2023-09-10T01:16:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3ca7920 con 0x55f7b3ed3710 2023-09-10T01:16:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd3807a0b80 10 client.0 _ll_register_callbacks cb 0x55f7b3d0da80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59594/0 (socket says 172.21.15.3:59594) 2023-09-10T01:16:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd365d9b700 1 -- 172.21.15.3:0/1821754540 learned_addr learned my addr 172.21.15.3:0/1821754540 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd36659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3edaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.226+0000 7fd366d9d700 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3edde40 0x55f7b3edb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd365d9b700 1 -- 172.21.15.3:0/1821754540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 msgr2=0x55f7b3edaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd365d9b700 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3edaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd365d9b700 1 -- 172.21.15.3:0/1821754540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3edde40 msgr2=0x55f7b3edb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd365d9b700 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3edde40 0x55f7b3edb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd365d9b700 1 -- 172.21.15.3:0/1821754540 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd358017050 con 0x55f7b3ed4030 2023-09-10T01:16:31.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd36659c700 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3edaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd365d9b700 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edb0e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0052a0 tx=0x7fd35c0052d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6255c57c511880f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd34dffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd34dffb700 1 -- 172.21.15.3:0/1821754540 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c01d070 con 0x55f7b3ed4030 2023-09-10T01:16:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f7b3d7d4f0 con 0x55f7b3ed4030 2023-09-10T01:16:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.227+0000 7fd34dffb700 1 -- 172.21.15.3:0/1821754540 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd35c004910 con 0x55f7b3ed4030 2023-09-10T01:16:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd34dffb700 1 -- 172.21.15.3:0/1821754540 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd35c004ad0 con 0x55f7b3ed4030 2023-09-10T01:16:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 10 client.28721 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7fd3681a55b0 con 0x55f7b3ed4030 2023-09-10T01:16:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd34dffb700 1 -- 172.21.15.3:0/1821754540 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f7b3d7d4f0 con 0x55f7b3ed4030 2023-09-10T01:16:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd34dffb700 1 -- 172.21.15.3:0/1821754540 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd35c024450 con 0x55f7b3ed4030 2023-09-10T01:16:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 10 client.28721 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 10 client.28721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 20 client.28721 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3ef6230 0x55f7b3ef8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 4 client.28721 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.228+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7b3e3a8a0 con 0x55f7b3ef6230 2023-09-10T01:16:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.229+0000 7fd36659c700 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3ef6230 0x55f7b3ef8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.229+0000 7fd36659c700 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3ef6230 0x55f7b3ef8650 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24ca0014006eead1 server_cookie=c52c049085129e02 in_seq=0 out_seq=0 2023-09-10T01:16:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.229+0000 7fd34dffb700 10 client.28721 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.231+0000 7fd34dffb700 1 -- 172.21.15.3:0/1821754540 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f7b3e3a8a0 con 0x55f7b3ef6230 2023-09-10T01:16:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.231+0000 7fd34dffb700 10 client.28721 handle_command_reply: tid=0 2023-09-10T01:16:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 client.28721 shutdown 2023-09-10T01:16:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 msgr2=0x55f7b3edb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edb0e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0052a0 tx=0x7fd35c0052d0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 shutdown_connections 2023-09-10T01:16:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3ef6230 0x55f7b3ef8650 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3edaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3edde40 0x55f7b3edb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/1821754540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3edb0e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 >> 172.21.15.3:0/1821754540 conn(0x55f7b3e48090 msgr2=0x55f7b3ece7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:31.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd34dffb700 0 client.28721 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 shutdown_connections 2023-09-10T01:16:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 1 -- 172.21.15.3:0/1821754540 wait complete. 2023-09-10T01:16:31.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.232+0000 7fd3807a0b80 20 client.28721 trim_cache size 0 max 16384 2023-09-10T01:16:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 Processor -- start 2023-09-10T01:16:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 -- start start 2023-09-10T01:16:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3eed2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3ed7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ed8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3d7d4f0 con 0x55f7b3ed3710 2023-09-10T01:16:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3eddc50 con 0x55f7b3ed4030 2023-09-10T01:16:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.252+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3ed00b0 con 0x55f7b3edde40 2023-09-10T01:16:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3ed7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd36659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3eed2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ed8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3ed7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59600/0 (socket says 172.21.15.3:59600) 2023-09-10T01:16:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 -- 172.21.15.3:0/4010913819 learned_addr learned my addr 172.21.15.3:0/4010913819 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ed8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45728/0 (socket says 172.21.15.3:45728) 2023-09-10T01:16:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 -- 172.21.15.3:0/4010913819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 msgr2=0x55f7b3ed8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ed8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 -- 172.21.15.3:0/4010913819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 msgr2=0x55f7b3eed2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3eed2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 -- 172.21.15.3:0/4010913819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7b3eda710 con 0x55f7b3ed4030 2023-09-10T01:16:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd36659c700 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3eed2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd365d9b700 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3ed7e30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd35c005270 tx=0x7fd35c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f90dfa5ba52b4433 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.253+0000 7fd3557fa700 1 -- 172.21.15.3:0/4010913819 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c01d040 con 0x55f7b3ed4030 2023-09-10T01:16:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3557fa700 1 -- 172.21.15.3:0/4010913819 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd35c01e040 con 0x55f7b3ed4030 2023-09-10T01:16:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3557fa700 1 -- 172.21.15.3:0/4010913819 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00f6c0 con 0x55f7b3ed4030 2023-09-10T01:16:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 -- 172.21.15.3:0/4010913819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 msgr2=0x55f7b3ed7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3ed7e30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd35c005270 tx=0x7fd35c005e80 comp rx=0 tx=0).stop 2023-09-10T01:16:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 -- 172.21.15.3:0/4010913819 shutdown_connections 2023-09-10T01:16:31.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3edde40 0x55f7b3ed8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed3710 0x55f7b3eed2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/4010913819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3ed4030 0x55f7b3ed7e30 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 -- 172.21.15.3:0/4010913819 >> 172.21.15.3:0/4010913819 conn(0x55f7b3e48090 msgr2=0x55f7b3ed5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 -- 172.21.15.3:0/4010913819 shutdown_connections 2023-09-10T01:16:31.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.254+0000 7fd3807a0b80 1 -- 172.21.15.3:0/4010913819 wait complete. 2023-09-10T01:16:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.255+0000 7fd3807a0b80 1 -- start start 2023-09-10T01:16:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.255+0000 7fd3807a0b80 1 Processor -- start 2023-09-10T01:16:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.255+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3eeb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.255+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed4030 0x55f7b3eeb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd3807a0b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3edde40 0x55f7b3eebd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3f49120 con 0x55f7b3ed4030 2023-09-10T01:16:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3f49290 con 0x55f7b3edde40 2023-09-10T01:16:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd3807a0b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b3f49450 con 0x55f7b3ed3710 2023-09-10T01:16:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd3807a0b80 10 client.0 _ll_register_callbacks cb 0x7fd35c00bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3eeb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd366d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3edde40 0x55f7b3eebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd365d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed4030 0x55f7b3eeb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3eeb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45736/0 (socket says 172.21.15.3:45736) 2023-09-10T01:16:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 -- 172.21.15.3:0/505067149 learned_addr learned my addr 172.21.15.3:0/505067149 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 -- 172.21.15.3:0/505067149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3edde40 msgr2=0x55f7b3eebd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3edde40 0x55f7b3eebd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 -- 172.21.15.3:0/505067149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed4030 msgr2=0x55f7b3eeb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed4030 0x55f7b3eeb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.256+0000 7fd36659c700 1 -- 172.21.15.3:0/505067149 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35800fda0 con 0x55f7b3ed3710 2023-09-10T01:16:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd36659c700 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3eeb2a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd35800fc60 tx=0x7fd3580047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ce7d9b8ff89e6e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd34dffb700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:16:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd34dffb700 1 -- 172.21.15.3:0/505067149 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358021040 con 0x55f7b3ed3710 2023-09-10T01:16:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd360004ad0 con 0x55f7b3ed3710 2023-09-10T01:16:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd34dffb700 1 -- 172.21.15.3:0/505067149 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fd35802b990 con 0x55f7b3ed3710 2023-09-10T01:16:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd34dffb700 1 -- 172.21.15.3:0/505067149 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd35802bb50 con 0x55f7b3ed3710 2023-09-10T01:16:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd3807a0b80 10 client.28603 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7fd3681a55b0 con 0x55f7b3ed3710 2023-09-10T01:16:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.257+0000 7fd34dffb700 1 -- 172.21.15.3:0/505067149 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd360004ad0 con 0x55f7b3ed3710 2023-09-10T01:16:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd34dffb700 1 -- 172.21.15.3:0/505067149 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd358034540 con 0x55f7b3ed3710 2023-09-10T01:16:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd3807a0b80 10 client.28603 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd3807a0b80 10 client.28603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd3807a0b80 20 client.28603 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3f4ac90 0x55f7b3f4d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd3807a0b80 4 client.28603 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:16:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f7b3e1e920 con 0x55f7b3f4ac90 2023-09-10T01:16:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.258+0000 7fd365d9b700 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3f4ac90 0x55f7b3f4d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.259+0000 7fd34dffb700 10 client.28603 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.259+0000 7fd365d9b700 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3f4ac90 0x55f7b3f4d0b0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78f40846d6c87dd6 server_cookie=a13ec755bc2911cd in_seq=0 out_seq=0 2023-09-10T01:16:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd34dffb700 1 -- 172.21.15.3:0/505067149 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+267 (crc 0 0 0) 0x55f7b3e1e920 con 0x55f7b3f4ac90 2023-09-10T01:16:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd34dffb700 10 client.28603 handle_command_reply: tid=0 2023-09-10T01:16:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 client.28603 shutdown 2023-09-10T01:16:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 msgr2=0x55f7b3eeb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3eeb2a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd35800fc60 tx=0x7fd3580047c0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 shutdown_connections 2023-09-10T01:16:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55f7b3f4ac90 0x55f7b3f4d0b0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f7b3ed3710 0x55f7b3eeb2a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f7b3edde40 0x55f7b3eebd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd34dffb700 0 client.28603 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 --2- 172.21.15.3:0/505067149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7b3ed4030 0x55f7b3eeb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 >> 172.21.15.3:0/505067149 conn(0x55f7b3e48090 msgr2=0x55f7b3edaaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:31.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 shutdown_connections 2023-09-10T01:16:31.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.262+0000 7fd3807a0b80 1 -- 172.21.15.3:0/505067149 wait complete. 2023-09-10T01:16:31.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd3807a0b80 20 client.28603 trim_cache size 0 max 16384 2023-09-10T01:16:31.275 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:16:31.276 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "scrub active (449 inodes in the stack)", 2023-09-10T01:16:31.276 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:16:31.276 INFO:teuthology.orchestra.run.smithi003.stdout: "79fafc81-0f62-4416-91fa-026aa8e080bc": { 2023-09-10T01:16:31.276 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:16:31.276 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "79fafc81-0f62-4416-91fa-026aa8e080bc", 2023-09-10T01:16:31.277 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:16:31.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:16:31.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:16:31.277 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:16:31.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3500a07b0 msgr2=0x7fd3500a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3500a07b0 0x7fd3500a2c70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd358009fd0 tx=0x7fd358009260 comp rx=0 tx=0).stop 2023-09-10T01:16:31.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd36810ac70 msgr2=0x7fd368198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:31.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd36810ac70 0x7fd368198810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd36000efc0 tx=0x7fd36000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 shutdown_connections 2023-09-10T01:16:31.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd3500a07b0 0x7fd3500a2c70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd36810ac70 0x7fd368198810 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3681018a0 0x7fd36819e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 --2- 172.21.15.3:0/129930785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3681021c0 0x7fd36819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:31.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.263+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 >> 172.21.15.3:0/129930785 conn(0x7fd368076710 msgr2=0x7fd3681069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:31.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.264+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 shutdown_connections 2023-09-10T01:16:31.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:16:31.264+0000 7fd36cdff700 1 -- 172.21.15.3:0/129930785 wait complete. 2023-09-10T01:16:31.285 INFO:tasks.cephfs.filesystem:scrub status for tag:79fafc81-0f62-4416-91fa-026aa8e080bc - {'path': '/', 'tag': '79fafc81-0f62-4416-91fa-026aa8e080bc', 'options': 'recursive,force'} 2023-09-10T01:16:31.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:16:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:16:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:16:31.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:16:31.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:16:31.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:31 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:16:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:16:31.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:16:31.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:16:31.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:16:31.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:31 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:16:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:16:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:16:31.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:16:31.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:31.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:16:31.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:31 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.145 INFO:tasks.workunit.client.0.smithi003.stdout: 10 4772 33.20 MB/sec warmup 13 sec latency 110.593 ms 2023-09-10T01:16:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:32 smithi003 ceph-mon[97654]: pgmap v416: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 19 MiB/s wr, 492 op/s 2023-09-10T01:16:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:32 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:16:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:32 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:32 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:16:32.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:32 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:32 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:32 smithi136 ceph-mon[133708]: pgmap v416: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 19 MiB/s wr, 492 op/s 2023-09-10T01:16:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:32 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:16:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:32 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:32 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:16:32.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:32 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:32 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:32 smithi092 ceph-mon[134067]: pgmap v416: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 19 MiB/s wr, 492 op/s 2023-09-10T01:16:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:32 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:16:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:32 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:32 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:16:32.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:32 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:32.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:32 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:16:33.146 INFO:tasks.workunit.client.0.smithi003.stdout: 10 5528 32.24 MB/sec warmup 14 sec latency 61.290 ms 2023-09-10T01:16:34.146 INFO:tasks.workunit.client.0.smithi003.stdout: 10 5934 32.26 MB/sec warmup 15 sec latency 215.642 ms 2023-09-10T01:16:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4017158867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58106680 msgr2=0x7f6e58106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4017158867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58106680 0x7f6e58106a60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009990 tx=0x7f6e54009ca0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4017158867 shutdown_connections 2023-09-10T01:16:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4017158867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58106680 0x7f6e58106a60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4017158867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58100f80 0x7f6e5810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4017158867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e581005c0 0x7f6e58100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4017158867 >> 172.21.15.3:0/4017158867 conn(0x7f6e580fc0b0 msgr2=0x7f6e580fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4017158867 shutdown_connections 2023-09-10T01:16:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.362+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4017158867 wait complete. 2023-09-10T01:16:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.363+0000 7f6e5ffff700 1 Processor -- start 2023-09-10T01:16:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.363+0000 7f6e5ffff700 1 -- start start 2023-09-10T01:16:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e581005c0 0x7f6e5819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58100f80 0x7f6e5819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58106680 0x7f6e581986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5ffff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e58198d30 con 0x7f6e581005c0 2023-09-10T01:16:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5ffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e58198ea0 con 0x7f6e58106680 2023-09-10T01:16:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5ffff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e58199060 con 0x7f6e58100f80 2023-09-10T01:16:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5e59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58106680 0x7f6e581986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5e59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58106680 0x7f6e581986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59620/0 (socket says 172.21.15.3:59620) 2023-09-10T01:16:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5e59c700 1 -- 172.21.15.3:0/4119914044 learned_addr learned my addr 172.21.15.3:0/4119914044 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e581005c0 0x7f6e5819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.364+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58100f80 0x7f6e5819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e5e59c700 1 -- 172.21.15.3:0/4119914044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58100f80 msgr2=0x7f6e5819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e5e59c700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58100f80 0x7f6e5819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e5e59c700 1 -- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e581005c0 msgr2=0x7f6e5819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e5e59c700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e581005c0 0x7f6e5819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e5e59c700 1 -- 172.21.15.3:0/4119914044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e540095f0 con 0x7f6e58106680 2023-09-10T01:16:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e5e59c700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58106680 0x7f6e581986c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000ef30 tx=0x7f6e5000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e4effd700 1 -- 172.21.15.3:0/4119914044 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50018070 con 0x7f6e58106680 2023-09-10T01:16:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e4effd700 1 -- 172.21.15.3:0/4119914044 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e5000f040 con 0x7f6e58106680 2023-09-10T01:16:34.370 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e4effd700 1 -- 172.21.15.3:0/4119914044 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50004280 con 0x7f6e58106680 2023-09-10T01:16:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.365+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e581992e0 con 0x7f6e58106680 2023-09-10T01:16:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.366+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e581a5640 con 0x7f6e58106680 2023-09-10T01:16:34.371 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.370+0000 7f6e4effd700 1 -- 172.21.15.3:0/4119914044 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6e50003a00 con 0x7f6e58106680 2023-09-10T01:16:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 Processor -- start 2023-09-10T01:16:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 -- start start 2023-09-10T01:16:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 0x5587ea1c2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea11b6d0 con 0x5587ea1c81a0 2023-09-10T01:16:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea118ff0 con 0x5587ea1c2300 2023-09-10T01:16:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.372+0000 7f6e73888b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea1148e0 con 0x5587ea1c2cc0 2023-09-10T01:16:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e4effd700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6e440a0830 0x7f6e440a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43438/0 (socket says 172.21.15.3:43438) 2023-09-10T01:16:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1558995276 learned_addr learned my addr 172.21.15.3:0/1558995276 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e4effd700 1 -- 172.21.15.3:0/4119914044 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f6e500cba10 con 0x7f6e58106680 2023-09-10T01:16:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5e59c700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6e440a0830 0x7f6e440a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1558995276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 msgr2=0x5587ea1cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1558995276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 msgr2=0x5587ea1c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 0x5587ea1c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1558995276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587e9f40d80 con 0x5587ea1c81a0 2023-09-10T01:16:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.373+0000 7f6e5e59c700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6e440a0830 0x7f6e440a2cf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e500090a0 tx=0x7f6e50020540 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.374+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c8580 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6e480138c0 tx=0x7f6e48009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2ea846c59bc7be server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:34.378 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.374+0000 7f6e43fff700 1 -- 172.21.15.3:0/1558995276 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48013170 con 0x5587ea1c81a0 2023-09-10T01:16:34.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.374+0000 7f6e43fff700 1 -- 172.21.15.3:0/1558995276 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e480132d0 con 0x5587ea1c81a0 2023-09-10T01:16:34.379 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.374+0000 7f6e43fff700 1 -- 172.21.15.3:0/1558995276 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48005600 con 0x5587ea1c81a0 2023-09-10T01:16:34.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 -- 172.21.15.3:0/1558995276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 msgr2=0x5587ea1c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c8580 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6e480138c0 tx=0x7f6e48009260 comp rx=0 tx=0).stop 2023-09-10T01:16:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 -- 172.21.15.3:0/1558995276 shutdown_connections 2023-09-10T01:16:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c8580 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1558995276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 0x5587ea1c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 -- 172.21.15.3:0/1558995276 >> 172.21.15.3:0/1558995276 conn(0x5587ea137030 msgr2=0x5587ea1bd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 -- 172.21.15.3:0/1558995276 shutdown_connections 2023-09-10T01:16:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.375+0000 7f6e73888b80 1 -- 172.21.15.3:0/1558995276 wait complete. 2023-09-10T01:16:34.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.376+0000 7f6e73888b80 1 -- start start 2023-09-10T01:16:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.376+0000 7f6e73888b80 1 Processor -- start 2023-09-10T01:16:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e73888b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea0b2b10 con 0x5587ea1c81a0 2023-09-10T01:16:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e73888b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea0b2810 con 0x5587ea1c2cc0 2023-09-10T01:16:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e73888b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e9f968c0 con 0x5587ea1c2300 2023-09-10T01:16:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e73888b80 10 client.0 _ll_register_callbacks cb 0x5587e9ffca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e5dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e5dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59638/0 (socket says 172.21.15.3:59638) 2023-09-10T01:16:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.377+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/2194871853 learned_addr learned my addr 172.21.15.3:0/2194871853 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.378+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/2194871853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 msgr2=0x5587ea1cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.378+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.378+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/2194871853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 msgr2=0x5587ea1cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.378+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.378+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/2194871853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4801b040 con 0x5587ea1c2cc0 2023-09-10T01:16:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.378+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cc0d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6e540095c0 tx=0x7f6e5400bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e554cd18af7a8ff4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.378+0000 7f6e417fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.379+0000 7f6e417fa700 1 -- 172.21.15.3:0/2194871853 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5401d070 con 0x5587ea1c2cc0 2023-09-10T01:16:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.379+0000 7f6e417fa700 1 -- 172.21.15.3:0/2194871853 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f6e54004500 con 0x5587ea1c2cc0 2023-09-10T01:16:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.379+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5587ea06c490 con 0x5587ea1c2cc0 2023-09-10T01:16:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.379+0000 7f6e417fa700 1 -- 172.21.15.3:0/2194871853 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e540046c0 con 0x5587ea1c2cc0 2023-09-10T01:16:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.379+0000 7f6e73888b80 10 client.28736 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.379+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5587e9f29c00 con 0x5587ea1c2cc0 2023-09-10T01:16:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.379+0000 7f6e417fa700 1 -- 172.21.15.3:0/2194871853 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5587ea06c490 con 0x5587ea1c2cc0 2023-09-10T01:16:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e417fa700 1 -- 172.21.15.3:0/2194871853 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f6e5400f600 con 0x5587ea1c2cc0 2023-09-10T01:16:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e73888b80 10 client.28736 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e73888b80 10 client.28736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e73888b80 20 client.28736 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:34.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e73888b80 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea1e4ce0 0x5587ea1e7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e73888b80 4 client.28736 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:34.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587ea129840 con 0x5587ea1e4ce0 2023-09-10T01:16:34.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.380+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea1e4ce0 0x5587ea1e7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.381+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea1e4ce0 0x5587ea1e7100 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4221afc15aaddb38 server_cookie=66f813dd35e186bf in_seq=0 out_seq=0 2023-09-10T01:16:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.381+0000 7f6e417fa700 10 client.28736 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e417fa700 1 -- 172.21.15.3:0/2194871853 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5587ea129840 con 0x5587ea1e4ce0 2023-09-10T01:16:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e417fa700 10 client.28736 handle_command_reply: tid=0 2023-09-10T01:16:34.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 client.28736 shutdown 2023-09-10T01:16:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 msgr2=0x5587ea1cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cc0d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6e540095c0 tx=0x7f6e5400bfa0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 shutdown_connections 2023-09-10T01:16:34.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea1e4ce0 0x5587ea1e7100 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e417fa700 0 client.28736 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 --2- 172.21.15.3:0/2194871853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1cc0d0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.384+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 >> 172.21.15.3:0/2194871853 conn(0x5587ea137030 msgr2=0x5587ea1bc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.385+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 shutdown_connections 2023-09-10T01:16:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.385+0000 7f6e73888b80 1 -- 172.21.15.3:0/2194871853 wait complete. 2023-09-10T01:16:34.395 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.385+0000 7f6e73888b80 20 client.28736 trim_cache size 0 max 16384 2023-09-10T01:16:34.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 Processor -- start 2023-09-10T01:16:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 -- start start 2023-09-10T01:16:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1c7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea1b78a0 con 0x5587ea1c81a0 2023-09-10T01:16:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea1c7e00 con 0x5587ea1c2cc0 2023-09-10T01:16:34.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.403+0000 7f6e73888b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea1cd6b0 con 0x5587ea1c2300 2023-09-10T01:16:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:59640/0 (socket says 172.21.15.3:59640) 2023-09-10T01:16:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/325695889 learned_addr learned my addr 172.21.15.3:0/325695889 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/325695889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 msgr2=0x5587ea1c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.407 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/325695889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 msgr2=0x5587ea1c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.404+0000 7f6e5dd9b700 1 -- 172.21.15.3:0/325695889 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587e9f671f0 con 0x5587ea1c2cc0 2023-09-10T01:16:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.405+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1c7140 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6e54000c00 tx=0x7f6e540098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29d6a041ec0cc74b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.406+0000 7f6e4cff9700 1 -- 172.21.15.3:0/325695889 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5401d070 con 0x5587ea1c2cc0 2023-09-10T01:16:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.406+0000 7f6e4cff9700 1 -- 172.21.15.3:0/325695889 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e5401e040 con 0x5587ea1c2cc0 2023-09-10T01:16:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.406+0000 7f6e4cff9700 1 -- 172.21.15.3:0/325695889 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54004500 con 0x5587ea1c2cc0 2023-09-10T01:16:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 -- 172.21.15.3:0/325695889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 msgr2=0x5587ea1c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1c7140 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6e54000c00 tx=0x7f6e540098c0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 -- 172.21.15.3:0/325695889 shutdown_connections 2023-09-10T01:16:34.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2300 0x5587ea1c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 --2- 172.21.15.3:0/325695889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2cc0 0x5587ea1c7140 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 -- 172.21.15.3:0/325695889 >> 172.21.15.3:0/325695889 conn(0x5587ea137030 msgr2=0x5587ea1cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 -- 172.21.15.3:0/325695889 shutdown_connections 2023-09-10T01:16:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.407+0000 7f6e73888b80 1 -- 172.21.15.3:0/325695889 wait complete. 2023-09-10T01:16:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.408+0000 7f6e73888b80 1 -- start start 2023-09-10T01:16:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.408+0000 7f6e73888b80 1 Processor -- start 2023-09-10T01:16:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 0x5587ea1d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1d8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e73888b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1d8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e73888b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea237b40 con 0x5587ea1c81a0 2023-09-10T01:16:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e73888b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea237cb0 con 0x5587ea1c2300 2023-09-10T01:16:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e73888b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587ea237e70 con 0x5587ea1c2cc0 2023-09-10T01:16:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e73888b80 10 client.0 _ll_register_callbacks cb 0x7f6e54009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:34.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 0x5587ea1d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e5dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1d8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1d8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e5d59a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1d8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43468/0 (socket says 172.21.15.3:43468) 2023-09-10T01:16:34.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.409+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1265704436 learned_addr learned my addr 172.21.15.3:0/1265704436 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:34.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1265704436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 msgr2=0x5587ea1d8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1d8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1265704436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 msgr2=0x5587ea1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 0x5587ea1d7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e5d59a700 1 -- 172.21.15.3:0/1265704436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587e9f29c00 con 0x5587ea1c81a0 2023-09-10T01:16:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1d8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e5d59a700 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1d8740 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6e48013eb0 tx=0x7f6e48018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b036e34909afd7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.410+0000 7f6e417fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:16:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.411+0000 7f6e417fa700 1 -- 172.21.15.3:0/1265704436 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48031bd0 con 0x5587ea1c81a0 2023-09-10T01:16:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.411+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e480132d0 con 0x5587ea1c81a0 2023-09-10T01:16:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.411+0000 7f6e417fa700 1 -- 172.21.15.3:0/1265704436 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f6e48031d30 con 0x5587ea1c81a0 2023-09-10T01:16:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.411+0000 7f6e417fa700 1 -- 172.21.15.3:0/1265704436 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e48005210 con 0x5587ea1c81a0 2023-09-10T01:16:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.411+0000 7f6e73888b80 10 client.18654 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.411+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5587e9f3f250 con 0x5587ea1c81a0 2023-09-10T01:16:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.411+0000 7f6e417fa700 1 -- 172.21.15.3:0/1265704436 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e480132d0 con 0x5587ea1c81a0 2023-09-10T01:16:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e417fa700 1 -- 172.21.15.3:0/1265704436 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f6e48003bb0 con 0x5587ea1c81a0 2023-09-10T01:16:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e73888b80 10 client.18654 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e73888b80 10 client.18654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e73888b80 20 client.18654 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:34.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea239520 0x5587ea23b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e73888b80 4 client.18654 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:16:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5587ea10d8c0 con 0x5587ea239520 2023-09-10T01:16:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.412+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea239520 0x5587ea23b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.413+0000 7f6e5dd9b700 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea239520 0x5587ea23b940 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48eb6e638d8c24a8 server_cookie=8355313a8ed4835 in_seq=0 out_seq=0 2023-09-10T01:16:34.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:34.413+0000 7f6e417fa700 10 client.18654 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:34.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:34 smithi003 ceph-mon[97654]: pgmap v417: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 19 MiB/s wr, 492 op/s 2023-09-10T01:16:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:34 smithi136 ceph-mon[133708]: pgmap v417: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 19 MiB/s wr, 492 op/s 2023-09-10T01:16:34.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:34 smithi092 ceph-mon[134067]: pgmap v417: 129 pgs: 129 active+clean; 524 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 19 MiB/s wr, 492 op/s 2023-09-10T01:16:35.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.020+0000 7f6e417fa700 1 -- 172.21.15.3:0/1265704436 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5587ea10d8c0 con 0x5587ea239520 2023-09-10T01:16:35.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.020+0000 7f6e417fa700 10 client.18654 handle_command_reply: tid=0 2023-09-10T01:16:35.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.020+0000 7f6e73888b80 1 client.18654 shutdown 2023-09-10T01:16:35.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 msgr2=0x5587ea1d8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1d8740 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6e48013eb0 tx=0x7f6e48018040 comp rx=0 tx=0).stop 2023-09-10T01:16:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 shutdown_connections 2023-09-10T01:16:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5587ea239520 0x5587ea23b940 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5587ea1c81a0 0x5587ea1d8740 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5587ea1c2300 0x5587ea1d7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 --2- 172.21.15.3:0/1265704436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5587ea1c2cc0 0x5587ea1d8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 >> 172.21.15.3:0/1265704436 conn(0x5587ea137030 msgr2=0x5587ea1c61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:35.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e417fa700 0 client.18654 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:35.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 shutdown_connections 2023-09-10T01:16:35.024 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 1 -- 172.21.15.3:0/1265704436 wait complete. 2023-09-10T01:16:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.021+0000 7f6e73888b80 20 client.18654 trim_cache size 0 max 16384 2023-09-10T01:16:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:16:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:16:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:16:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6e440a0830 msgr2=0x7f6e440a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6e440a0830 0x7f6e440a2cf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e500090a0 tx=0x7f6e50020540 comp rx=0 tx=0).stop 2023-09-10T01:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58106680 msgr2=0x7f6e581986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58106680 0x7f6e581986c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000ef30 tx=0x7f6e5000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 shutdown_connections 2023-09-10T01:16:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6e440a0830 0x7f6e440a2cf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e58106680 0x7f6e581986c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e581005c0 0x7f6e5819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 --2- 172.21.15.3:0/4119914044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e58100f80 0x7f6e5819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 >> 172.21.15.3:0/4119914044 conn(0x7f6e580fc0b0 msgr2=0x7f6e580fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 shutdown_connections 2023-09-10T01:16:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:35.022+0000 7f6e5ffff700 1 -- 172.21.15.3:0/4119914044 wait complete. 2023-09-10T01:16:35.146 INFO:tasks.workunit.client.0.smithi003.stdout: 10 6534 33.04 MB/sec warmup 16 sec latency 484.648 ms 2023-09-10T01:16:36.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 7192 33.68 MB/sec warmup 17 sec latency 39.113 ms 2023-09-10T01:16:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:36 smithi003 ceph-mon[97654]: pgmap v418: 129 pgs: 129 active+clean; 538 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 21 MiB/s wr, 670 op/s 2023-09-10T01:16:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:36 smithi136 ceph-mon[133708]: pgmap v418: 129 pgs: 129 active+clean; 538 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 21 MiB/s wr, 670 op/s 2023-09-10T01:16:36.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:36 smithi092 ceph-mon[134067]: pgmap v418: 129 pgs: 129 active+clean; 538 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 21 MiB/s wr, 670 op/s 2023-09-10T01:16:37.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 7645 32.48 MB/sec warmup 18 sec latency 77.051 ms 2023-09-10T01:16:38.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 8778 33.47 MB/sec warmup 19 sec latency 73.179 ms 2023-09-10T01:16:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:38 smithi003 ceph-mon[97654]: pgmap v419: 129 pgs: 129 active+clean; 538 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 12 MiB/s wr, 341 op/s 2023-09-10T01:16:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:38 smithi136 ceph-mon[133708]: pgmap v419: 129 pgs: 129 active+clean; 538 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 12 MiB/s wr, 341 op/s 2023-09-10T01:16:38.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:38 smithi092 ceph-mon[134067]: pgmap v419: 129 pgs: 129 active+clean; 538 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 12 MiB/s wr, 341 op/s 2023-09-10T01:16:39.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 9786 35.07 MB/sec warmup 20 sec latency 129.174 ms 2023-09-10T01:16:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:39 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:16:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:39 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:16:39.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:39 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:16:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:39 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:16:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:39 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:16:39.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:39 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:16:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:39 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:16:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:39 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:16:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:39 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:16:40.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 10540 35.90 MB/sec warmup 21 sec latency 52.021 ms 2023-09-10T01:16:40.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:40 smithi003 ceph-mon[97654]: pgmap v420: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 14 MiB/s wr, 715 op/s 2023-09-10T01:16:40.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:40 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:16:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:40 smithi136 ceph-mon[133708]: pgmap v420: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 14 MiB/s wr, 715 op/s 2023-09-10T01:16:40.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:40 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:16:40.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:40 smithi092 ceph-mon[134067]: pgmap v420: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 14 MiB/s wr, 715 op/s 2023-09-10T01:16:40.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:40 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:16:41.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 10982 34.74 MB/sec warmup 22 sec latency 83.881 ms 2023-09-10T01:16:41.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:41 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:16:41.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:41 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:16:41.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:41 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:16:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:41 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:16:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:41 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:16:41.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:41 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:16:41.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:41 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:16:41.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:41 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:16:41.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:41 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:16:42.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 12021 35.46 MB/sec warmup 23 sec latency 82.574 ms 2023-09-10T01:16:42.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:42 smithi003 ceph-mon[97654]: pgmap v421: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 551 op/s 2023-09-10T01:16:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:42 smithi136 ceph-mon[133708]: pgmap v421: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 551 op/s 2023-09-10T01:16:42.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:42 smithi092 ceph-mon[134067]: pgmap v421: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 551 op/s 2023-09-10T01:16:43.147 INFO:tasks.workunit.client.0.smithi003.stdout: 10 13095 36.41 MB/sec warmup 24 sec latency 107.997 ms 2023-09-10T01:16:44.149 INFO:tasks.workunit.client.0.smithi003.stdout: 10 13611 36.50 MB/sec warmup 25 sec latency 348.174 ms 2023-09-10T01:16:44.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:44 smithi003 ceph-mon[97654]: pgmap v422: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 551 op/s 2023-09-10T01:16:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:44 smithi136 ceph-mon[133708]: pgmap v422: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 551 op/s 2023-09-10T01:16:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:44 smithi092 ceph-mon[134067]: pgmap v422: 129 pgs: 129 active+clean; 547 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 551 op/s 2023-09-10T01:16:45.149 INFO:tasks.workunit.client.0.smithi003.stdout: 10 13746 35.28 MB/sec warmup 26 sec latency 1146.912 ms 2023-09-10T01:16:46.150 INFO:tasks.workunit.client.0.smithi003.stdout: 10 14175 35.03 MB/sec warmup 27 sec latency 98.121 ms 2023-09-10T01:16:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:46 smithi136 ceph-mon[133708]: pgmap v423: 129 pgs: 129 active+clean; 572 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.0 MiB/s wr, 763 op/s 2023-09-10T01:16:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:46 smithi092 ceph-mon[134067]: pgmap v423: 129 pgs: 129 active+clean; 572 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.0 MiB/s wr, 763 op/s 2023-09-10T01:16:46.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:46 smithi003 ceph-mon[97654]: pgmap v423: 129 pgs: 129 active+clean; 572 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.0 MiB/s wr, 763 op/s 2023-09-10T01:16:47.150 INFO:tasks.workunit.client.0.smithi003.stdout: 10 14672 34.48 MB/sec warmup 28 sec latency 89.524 ms 2023-09-10T01:16:48.153 INFO:tasks.workunit.client.0.smithi003.stdout: 10 15495 34.62 MB/sec warmup 29 sec latency 36.633 ms 2023-09-10T01:16:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.248+0000 7f720d5ff700 1 -- 172.21.15.3:0/3154296036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 msgr2=0x7f7208100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.248+0000 7f720d5ff700 1 --2- 172.21.15.3:0/3154296036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f7208100a40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f71fc009920 tx=0x7f71fc009c30 comp rx=0 tx=0).stop 2023-09-10T01:16:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.249+0000 7f720d5ff700 1 -- 172.21.15.3:0/3154296036 shutdown_connections 2023-09-10T01:16:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.249+0000 7f720d5ff700 1 --2- 172.21.15.3:0/3154296036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7208106680 0x7f7208106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.249+0000 7f720d5ff700 1 --2- 172.21.15.3:0/3154296036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7208100f80 0x7f720810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.249+0000 7f720d5ff700 1 --2- 172.21.15.3:0/3154296036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f7208100a40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.249+0000 7f720d5ff700 1 -- 172.21.15.3:0/3154296036 >> 172.21.15.3:0/3154296036 conn(0x7f72080fc0b0 msgr2=0x7f72080fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.252+0000 7f720d5ff700 1 -- 172.21.15.3:0/3154296036 shutdown_connections 2023-09-10T01:16:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.254+0000 7f720d5ff700 1 -- 172.21.15.3:0/3154296036 wait complete. 2023-09-10T01:16:48.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.255+0000 7f720d5ff700 1 Processor -- start 2023-09-10T01:16:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.258+0000 7f720d5ff700 1 -- start start 2023-09-10T01:16:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.259+0000 7f720d5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f72081aa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.259+0000 7f720d5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7208100f80 0x7f72081aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.259+0000 7f720d5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7208106680 0x7f72081a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.259+0000 7f720d5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72081a6220 con 0x7f72081005c0 2023-09-10T01:16:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.259+0000 7f720d5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72081a6390 con 0x7f7208100f80 2023-09-10T01:16:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.259+0000 7f720d5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72081a6550 con 0x7f7208106680 2023-09-10T01:16:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f72081aa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f72077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7208106680 0x7f72081a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f72081aa1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43070/0 (socket says 172.21.15.3:43070) 2023-09-10T01:16:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f7206ffd700 1 -- 172.21.15.3:0/2676229117 learned_addr learned my addr 172.21.15.3:0/2676229117 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f72067fc700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7208100f80 0x7f72081aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f72067fc700 1 -- 172.21.15.3:0/2676229117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7208106680 msgr2=0x7f72081a5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f72067fc700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7208106680 0x7f72081a5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f72067fc700 1 -- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 msgr2=0x7f72081aa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f72067fc700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f72081aa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f72067fc700 1 -- 172.21.15.3:0/2676229117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc009520 con 0x7f7208100f80 2023-09-10T01:16:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.260+0000 7f7206ffd700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f72081aa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.261+0000 7f72067fc700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7208100f80 0x7f72081aa730 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f71fc000c00 tx=0x7f71fc00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.261+0000 7f71effff700 1 -- 172.21.15.3:0/2676229117 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc01c070 con 0x7f7208100f80 2023-09-10T01:16:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.261+0000 7f71effff700 1 -- 172.21.15.3:0/2676229117 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71fc00bbe0 con 0x7f7208100f80 2023-09-10T01:16:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.261+0000 7f71effff700 1 -- 172.21.15.3:0/2676229117 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc004930 con 0x7f7208100f80 2023-09-10T01:16:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.261+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72081a67d0 con 0x7f7208100f80 2023-09-10T01:16:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.262+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72081b2310 con 0x7f7208100f80 2023-09-10T01:16:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.271+0000 7f7220f52b80 1 Processor -- start 2023-09-10T01:16:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.272+0000 7f71effff700 1 -- 172.21.15.3:0/2676229117 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f71fc020070 con 0x7f7208100f80 2023-09-10T01:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.274+0000 7f7220f52b80 1 -- start start 2023-09-10T01:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.274+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfe2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.274+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdc300 0x555e5bfdc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.274+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdccc0 0x555e5bfe8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.274+0000 7f7220f52b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5bf356d0 con 0x555e5bfe21a0 2023-09-10T01:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.275+0000 7f7220f52b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5bf32ff0 con 0x555e5bfdc300 2023-09-10T01:16:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.275+0000 7f7220f52b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5bf2e8e0 con 0x555e5bfdccc0 2023-09-10T01:16:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.275+0000 7f71effff700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71f00a08b0 0x7f71f00a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.275+0000 7f71effff700 1 -- 172.21.15.3:0/2676229117 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f71fc0ccf10 con 0x7f7208100f80 2023-09-10T01:16:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.276+0000 7f72067fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdccc0 0x555e5bfe8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.276+0000 7f72077fe700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71f00a08b0 0x7f71f00a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.276+0000 7f72067fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdccc0 0x555e5bfe8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:53978/0 (socket says 172.21.15.3:53978) 2023-09-10T01:16:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.276+0000 7f72067fc700 1 -- 172.21.15.3:0/1379978022 learned_addr learned my addr 172.21.15.3:0/1379978022 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.277+0000 7f7206ffd700 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdc300 0x555e5bfdc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.277+0000 7f7206ffd700 1 -- 172.21.15.3:0/1379978022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdccc0 msgr2=0x555e5bfe8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.277+0000 7f7206ffd700 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdccc0 0x555e5bfe8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.277+0000 7f7206ffd700 1 -- 172.21.15.3:0/1379978022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 msgr2=0x555e5bfe2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.277+0000 7f7206ffd700 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfe2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.277+0000 7f7206ffd700 1 -- 172.21.15.3:0/1379978022 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e5bd5ad80 con 0x555e5bfdc300 2023-09-10T01:16:48.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.278+0000 7f7206ffd700 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdc300 0x555e5bfdc780 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f71f8011510 tx=0x7f71f8011820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=681963cf156b79dc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:48.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.278+0000 7f72077fe700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71f00a08b0 0x7f71f00a2d70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7200009df0 tx=0x7f720000b040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:48.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.278+0000 7f71ed7fa700 1 -- 172.21.15.3:0/1379978022 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8011ea0 con 0x555e5bfdc300 2023-09-10T01:16:48.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.278+0000 7f71ed7fa700 1 -- 172.21.15.3:0/1379978022 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71f8003680 con 0x555e5bfdc300 2023-09-10T01:16:48.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.278+0000 7f71ed7fa700 1 -- 172.21.15.3:0/1379978022 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f800e6b0 con 0x555e5bfdc300 2023-09-10T01:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.279+0000 7f7220f52b80 1 -- 172.21.15.3:0/1379978022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdc300 msgr2=0x555e5bfdc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.279+0000 7f7220f52b80 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdc300 0x555e5bfdc780 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f71f8011510 tx=0x7f71f8011820 comp rx=0 tx=0).stop 2023-09-10T01:16:48.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.280+0000 7f7220f52b80 1 -- 172.21.15.3:0/1379978022 shutdown_connections 2023-09-10T01:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.280+0000 7f7220f52b80 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdccc0 0x555e5bfe8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.280+0000 7f7220f52b80 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfe2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.280+0000 7f7220f52b80 1 --2- 172.21.15.3:0/1379978022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdc300 0x555e5bfdc780 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.280+0000 7f7220f52b80 1 -- 172.21.15.3:0/1379978022 >> 172.21.15.3:0/1379978022 conn(0x555e5bf51030 msgr2=0x555e5bfd7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.280+0000 7f7220f52b80 1 -- 172.21.15.3:0/1379978022 shutdown_connections 2023-09-10T01:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.282+0000 7f7220f52b80 1 -- 172.21.15.3:0/1379978022 wait complete. 2023-09-10T01:16:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.282+0000 7f7220f52b80 1 -- start start 2023-09-10T01:16:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.283+0000 7f7220f52b80 1 Processor -- start 2023-09-10T01:16:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.285+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfe8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.285+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfe5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.285+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfe6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.285+0000 7f7220f52b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5beccb10 con 0x555e5bfe21a0 2023-09-10T01:16:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.285+0000 7f7220f52b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5becc810 con 0x555e5bfdccc0 2023-09-10T01:16:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.285+0000 7f7220f52b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5bdb08c0 con 0x555e5bfdc300 2023-09-10T01:16:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.285+0000 7f7220f52b80 10 client.0 _ll_register_callbacks cb 0x555e5be16a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.286+0000 7f72067fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfe5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.286+0000 7f72067fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfe5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48166/0 (socket says 172.21.15.3:48166) 2023-09-10T01:16:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.286+0000 7f72067fc700 1 -- 172.21.15.3:0/3583202505 learned_addr learned my addr 172.21.15.3:0/3583202505 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f72067fc700 1 -- 172.21.15.3:0/3583202505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 msgr2=0x555e5bfe8d00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:16:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f72067fc700 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfe8d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f72067fc700 1 -- 172.21.15.3:0/3583202505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 msgr2=0x555e5bfe6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:48.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f72067fc700 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfe6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f72067fc700 1 -- 172.21.15.3:0/3583202505 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f8009d10 con 0x555e5bfdccc0 2023-09-10T01:16:48.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f72067fc700 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfe5ff0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00bd50 tx=0x7f71fc09a830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0c99935c3b0b4bc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:48.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f71eaff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:48.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f71eaff5700 1 -- 172.21.15.3:0/3583202505 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc09a460 con 0x555e5bfdccc0 2023-09-10T01:16:48.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.287+0000 7f71eaff5700 1 -- 172.21.15.3:0/3583202505 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f71fc0a1fe0 con 0x555e5bfdccc0 2023-09-10T01:16:48.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.288+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e5be86490 con 0x555e5bfdccc0 2023-09-10T01:16:48.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.288+0000 7f71eaff5700 1 -- 172.21.15.3:0/3583202505 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71fc0ca6a0 con 0x555e5bfdccc0 2023-09-10T01:16:48.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.288+0000 7f7220f52b80 10 client.28754 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:48.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.288+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555e5bd43c00 con 0x555e5bfdccc0 2023-09-10T01:16:48.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.288+0000 7f71eaff5700 1 -- 172.21.15.3:0/3583202505 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e5be86490 con 0x555e5bfdccc0 2023-09-10T01:16:48.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.289+0000 7f71eaff5700 1 -- 172.21.15.3:0/3583202505 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f71fc0994f0 con 0x555e5bfdccc0 2023-09-10T01:16:48.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.289+0000 7f7220f52b80 10 client.28754 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:48.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.289+0000 7f7220f52b80 10 client.28754 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.289+0000 7f7220f52b80 20 client.28754 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.289+0000 7f7220f52b80 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5bff60c0 0x555e5bff84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.289+0000 7f7220f52b80 4 client.28754 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:16:48.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.290+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e5bf43840 con 0x555e5bff60c0 2023-09-10T01:16:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.290+0000 7f7206ffd700 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5bff60c0 0x555e5bff84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.291+0000 7f7206ffd700 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5bff60c0 0x555e5bff84e0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46014a506567b987 server_cookie=3284a193f90627e6 in_seq=0 out_seq=0 2023-09-10T01:16:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.291+0000 7f71eaff5700 10 client.28754 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.292+0000 7f71eaff5700 1 -- 172.21.15.3:0/3583202505 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555e5bf43840 con 0x555e5bff60c0 2023-09-10T01:16:48.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.292+0000 7f71eaff5700 10 client.28754 handle_command_reply: tid=0 2023-09-10T01:16:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.292+0000 7f7220f52b80 1 client.28754 shutdown 2023-09-10T01:16:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 msgr2=0x555e5bfe5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:48.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfe5ff0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00bd50 tx=0x7f71fc09a830 comp rx=0 tx=0).stop 2023-09-10T01:16:48.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 shutdown_connections 2023-09-10T01:16:48.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5bff60c0 0x555e5bff84e0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfe8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfe6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 --2- 172.21.15.3:0/3583202505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfe5ff0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.295+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 >> 172.21.15.3:0/3583202505 conn(0x555e5bf51030 msgr2=0x555e5bfd67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.296+0000 7f71eaff5700 0 client.28754 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:48.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.297+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 shutdown_connections 2023-09-10T01:16:48.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.298+0000 7f7220f52b80 1 -- 172.21.15.3:0/3583202505 wait complete. 2023-09-10T01:16:48.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.299+0000 7f7220f52b80 20 client.28754 trim_cache size 0 max 16384 2023-09-10T01:16:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.323+0000 7f7220f52b80 1 Processor -- start 2023-09-10T01:16:48.324 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.323+0000 7f7220f52b80 1 -- start start 2023-09-10T01:16:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.323+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfdc300 0x555e5bfe88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfdeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfe21a0 0x555e5bfdf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f7220f52b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5bfe7560 con 0x555e5bfdc300 2023-09-10T01:16:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f7220f52b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5bfd3fc0 con 0x555e5bfdccc0 2023-09-10T01:16:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f7220f52b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5bfcef10 con 0x555e5bfe21a0 2023-09-10T01:16:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfdeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfdeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48170/0 (socket says 172.21.15.3:48170) 2023-09-10T01:16:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 -- 172.21.15.3:0/328433302 learned_addr learned my addr 172.21.15.3:0/328433302 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72077fe700 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfe21a0 0x555e5bfdf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 -- 172.21.15.3:0/328433302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfe21a0 msgr2=0x555e5bfdf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfe21a0 0x555e5bfdf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 -- 172.21.15.3:0/328433302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfdc300 msgr2=0x555e5bfe88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfdc300 0x555e5bfe88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.324+0000 7f72067fc700 1 -- 172.21.15.3:0/328433302 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e5bffd0d0 con 0x555e5bfdccc0 2023-09-10T01:16:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.325+0000 7f72067fc700 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfdeb60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0094f0 tx=0x7f71fc0ca2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=181a600376c1e5e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.325+0000 7f71edffb700 1 -- 172.21.15.3:0/328433302 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0ca6a0 con 0x555e5bfdccc0 2023-09-10T01:16:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.325+0000 7f71edffb700 1 -- 172.21.15.3:0/328433302 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71fc0a1fe0 con 0x555e5bfdccc0 2023-09-10T01:16:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.325+0000 7f71edffb700 1 -- 172.21.15.3:0/328433302 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0997a0 con 0x555e5bfdccc0 2023-09-10T01:16:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 -- 172.21.15.3:0/328433302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 msgr2=0x555e5bfdeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfdeb60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0094f0 tx=0x7f71fc0ca2c0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 -- 172.21.15.3:0/328433302 shutdown_connections 2023-09-10T01:16:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfe21a0 0x555e5bfdf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfdc300 0x555e5bfe88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 --2- 172.21.15.3:0/328433302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfdeb60 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 -- 172.21.15.3:0/328433302 >> 172.21.15.3:0/328433302 conn(0x555e5bf51030 msgr2=0x555e5bfe0190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 -- 172.21.15.3:0/328433302 shutdown_connections 2023-09-10T01:16:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.326+0000 7f7220f52b80 1 -- 172.21.15.3:0/328433302 wait complete. 2023-09-10T01:16:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.327+0000 7f7220f52b80 1 -- start start 2023-09-10T01:16:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.327+0000 7f7220f52b80 1 Processor -- start 2023-09-10T01:16:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfeb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfeba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7220f52b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7220f52b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5c049080 con 0x555e5bfe21a0 2023-09-10T01:16:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7220f52b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5c0491f0 con 0x555e5bfdccc0 2023-09-10T01:16:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7220f52b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e5c05a3c0 con 0x555e5bfdc300 2023-09-10T01:16:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7220f52b80 10 client.0 _ll_register_callbacks cb 0x555e5be16a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:16:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfeb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f72067fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfeba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfeb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:54008/0 (socket says 172.21.15.3:54008) 2023-09-10T01:16:48.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f72067fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfeba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48174/0 (socket says 172.21.15.3:48174) 2023-09-10T01:16:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.328+0000 7f7206ffd700 1 -- 172.21.15.3:0/379762627 learned_addr learned my addr 172.21.15.3:0/379762627 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:16:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f72067fc700 1 -- 172.21.15.3:0/379762627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 msgr2=0x555e5bfeb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f72067fc700 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfeb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f72067fc700 1 -- 172.21.15.3:0/379762627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 msgr2=0x555e5bfebfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:16:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f72067fc700 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f72067fc700 1 -- 172.21.15.3:0/379762627 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e5bd43c00 con 0x555e5bfdccc0 2023-09-10T01:16:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f7206ffd700 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfeb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:16:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f72067fc700 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfeba50 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x555e5bfd1970 tx=0x7f71fc09a830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fbd7a8c627d1d63 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:16:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f71eaff5700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:16:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f71eaff5700 1 -- 172.21.15.3:0/379762627 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0ca6a0 con 0x555e5bfdccc0 2023-09-10T01:16:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.329+0000 7f71eaff5700 1 -- 172.21.15.3:0/379762627 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f71fc09a260 con 0x555e5bfdccc0 2023-09-10T01:16:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.330+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71f8003680 con 0x555e5bfdccc0 2023-09-10T01:16:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.330+0000 7f71eaff5700 1 -- 172.21.15.3:0/379762627 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71fc09a420 con 0x555e5bfdccc0 2023-09-10T01:16:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.330+0000 7f7220f52b80 10 client.28760 fetch_fsmap learned FSMap version 38 2023-09-10T01:16:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.330+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555e5bd811f0 con 0x555e5bfdccc0 2023-09-10T01:16:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.330+0000 7f71eaff5700 1 -- 172.21.15.3:0/379762627 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71f8003680 con 0x555e5bfdccc0 2023-09-10T01:16:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f71eaff5700 1 -- 172.21.15.3:0/379762627 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f71fc095df0 con 0x555e5bfdccc0 2023-09-10T01:16:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f7220f52b80 10 client.28760 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:16:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f7220f52b80 10 client.28760 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:16:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f7220f52b80 20 client.28760 populate_metadata read hostname 'smithi003' 2023-09-10T01:16:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f7220f52b80 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5c05b620 0x555e5c05da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:16:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f7220f52b80 4 client.28760 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:16:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555e5bf278c0 con 0x555e5c05b620 2023-09-10T01:16:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.331+0000 7f7206ffd700 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5c05b620 0x555e5c05da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:16:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.332+0000 7f7206ffd700 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5c05b620 0x555e5c05da10 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7c4c109cdabbb2d server_cookie=36ac0a525703eeba in_seq=0 out_seq=0 2023-09-10T01:16:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:48.332+0000 7f71eaff5700 10 client.28760 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:48.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:48 smithi136 ceph-mon[133708]: pgmap v424: 129 pgs: 129 active+clean; 572 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 586 op/s 2023-09-10T01:16:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:48 smithi092 ceph-mon[134067]: pgmap v424: 129 pgs: 129 active+clean; 572 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 586 op/s 2023-09-10T01:16:48.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:48 smithi003 ceph-mon[97654]: pgmap v424: 129 pgs: 129 active+clean; 572 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 586 op/s 2023-09-10T01:16:49.154 INFO:tasks.workunit.client.0.smithi003.stdout: 10 15737 34.01 MB/sec warmup 30 sec latency 409.083 ms 2023-09-10T01:16:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.747+0000 7f71eaff5700 1 -- 172.21.15.3:0/379762627 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555e5bf278c0 con 0x555e5c05b620 2023-09-10T01:16:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.747+0000 7f71eaff5700 10 client.28760 handle_command_reply: tid=0 2023-09-10T01:16:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.747+0000 7f7220f52b80 1 client.28760 shutdown 2023-09-10T01:16:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 msgr2=0x555e5bfeba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfeba50 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x555e5bfd1970 tx=0x7f71fc09a830 comp rx=0 tx=0).stop 2023-09-10T01:16:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 shutdown_connections 2023-09-10T01:16:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e5c05b620 0x555e5c05da10 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e5bfdc300 0x555e5bfeb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e5bfe21a0 0x555e5bfebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 --2- 172.21.15.3:0/379762627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e5bfdccc0 0x555e5bfeba50 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 >> 172.21.15.3:0/379762627 conn(0x555e5bf51030 msgr2=0x555e5bfd7090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f71eaff5700 0 client.28760 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:16:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.751+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 shutdown_connections 2023-09-10T01:16:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.752+0000 7f7220f52b80 1 -- 172.21.15.3:0/379762627 wait complete. 2023-09-10T01:16:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.755+0000 7f7220f52b80 20 client.28760 trim_cache size 0 max 16384 2023-09-10T01:16:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:16:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:16:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:16:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:16:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71f00a08b0 msgr2=0x7f71f00a2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71f00a08b0 0x7f71f00a2d70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7200009df0 tx=0x7f720000b040 comp rx=0 tx=0).stop 2023-09-10T01:16:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7208100f80 msgr2=0x7f72081aa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:16:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7208100f80 0x7f72081aa730 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f71fc000c00 tx=0x7f71fc00fa00 comp rx=0 tx=0).stop 2023-09-10T01:16:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 shutdown_connections 2023-09-10T01:16:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f71f00a08b0 0x7f71f00a2d70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7208106680 0x7f72081a5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72081005c0 0x7f72081aa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 --2- 172.21.15.3:0/2676229117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7208100f80 0x7f72081aa730 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:16:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.756+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 >> 172.21.15.3:0/2676229117 conn(0x7f72080fc0b0 msgr2=0x7f72080fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:16:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.757+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 shutdown_connections 2023-09-10T01:16:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:16:49.757+0000 7f720d5ff700 1 -- 172.21.15.3:0/2676229117 wait complete. 2023-09-10T01:16:50.155 INFO:tasks.workunit.client.0.smithi003.stdout: 10 15967 33.34 MB/sec warmup 31 sec latency 664.954 ms 2023-09-10T01:16:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:50 smithi136 ceph-mon[133708]: pgmap v425: 129 pgs: 129 active+clean; 592 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 898 op/s 2023-09-10T01:16:50.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:50 smithi092 ceph-mon[134067]: pgmap v425: 129 pgs: 129 active+clean; 592 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 898 op/s 2023-09-10T01:16:50.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:50 smithi003 ceph-mon[97654]: pgmap v425: 129 pgs: 129 active+clean; 592 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 898 op/s 2023-09-10T01:16:51.157 INFO:tasks.workunit.client.0.smithi003.stdout: 10 16320 32.72 MB/sec warmup 32 sec latency 160.525 ms 2023-09-10T01:16:51.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:51 smithi092 ceph-mon[134067]: pgmap v426: 129 pgs: 129 active+clean; 593 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 MiB/s wr, 544 op/s 2023-09-10T01:16:51.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:51 smithi003 ceph-mon[97654]: pgmap v426: 129 pgs: 129 active+clean; 593 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 MiB/s wr, 544 op/s 2023-09-10T01:16:51.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:51 smithi136 ceph-mon[133708]: pgmap v426: 129 pgs: 129 active+clean; 593 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 MiB/s wr, 544 op/s 2023-09-10T01:16:52.157 INFO:tasks.workunit.client.0.smithi003.stdout: 10 16986 33.13 MB/sec warmup 33 sec latency 164.411 ms 2023-09-10T01:16:53.157 INFO:tasks.workunit.client.0.smithi003.stdout: 10 17853 33.58 MB/sec warmup 34 sec latency 54.229 ms 2023-09-10T01:16:53.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:53 smithi003 ceph-mon[97654]: pgmap v427: 129 pgs: 129 active+clean; 593 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 MiB/s wr, 544 op/s 2023-09-10T01:16:53.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:53 smithi136 ceph-mon[133708]: pgmap v427: 129 pgs: 129 active+clean; 593 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 MiB/s wr, 544 op/s 2023-09-10T01:16:53.966 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:53 smithi092 ceph-mon[134067]: pgmap v427: 129 pgs: 129 active+clean; 593 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 MiB/s wr, 544 op/s 2023-09-10T01:16:54.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 18551 33.43 MB/sec warmup 35 sec latency 78.854 ms 2023-09-10T01:16:55.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 19452 33.94 MB/sec warmup 36 sec latency 64.152 ms 2023-09-10T01:16:56.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 20613 34.78 MB/sec warmup 37 sec latency 42.400 ms 2023-09-10T01:16:56.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:56 smithi003 ceph-mon[97654]: pgmap v428: 129 pgs: 129 active+clean; 587 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 16 MiB/s wr, 1.08k op/s 2023-09-10T01:16:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:56 smithi136 ceph-mon[133708]: pgmap v428: 129 pgs: 129 active+clean; 587 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 16 MiB/s wr, 1.08k op/s 2023-09-10T01:16:56.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:56 smithi092 ceph-mon[134067]: pgmap v428: 129 pgs: 129 active+clean; 587 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 16 MiB/s wr, 1.08k op/s 2023-09-10T01:16:57.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 21292 34.90 MB/sec warmup 38 sec latency 54.139 ms 2023-09-10T01:16:58.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 22010 34.81 MB/sec warmup 39 sec latency 51.620 ms 2023-09-10T01:16:58.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:16:58 smithi003 ceph-mon[97654]: pgmap v429: 129 pgs: 129 active+clean; 587 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 13 MiB/s wr, 869 op/s 2023-09-10T01:16:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:16:58 smithi136 ceph-mon[133708]: pgmap v429: 129 pgs: 129 active+clean; 587 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 13 MiB/s wr, 869 op/s 2023-09-10T01:16:58.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:16:58 smithi092 ceph-mon[134067]: pgmap v429: 129 pgs: 129 active+clean; 587 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 13 MiB/s wr, 869 op/s 2023-09-10T01:16:59.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 22701 34.76 MB/sec warmup 40 sec latency 53.361 ms 2023-09-10T01:17:00.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 23748 35.42 MB/sec warmup 41 sec latency 48.113 ms 2023-09-10T01:17:00.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:00 smithi003 ceph-mon[97654]: pgmap v430: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 16 MiB/s wr, 1.08k op/s 2023-09-10T01:17:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:00 smithi136 ceph-mon[133708]: pgmap v430: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 16 MiB/s wr, 1.08k op/s 2023-09-10T01:17:00.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:00 smithi092 ceph-mon[134067]: pgmap v430: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 16 MiB/s wr, 1.08k op/s 2023-09-10T01:17:01.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24153 35.17 MB/sec warmup 42 sec latency 506.166 ms 2023-09-10T01:17:01.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:17:01.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 -- 172.21.15.3:0/4239522383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c101000 msgr2=0x7fdb3c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4239522383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c101000 0x7fdb3c10d510 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400b0a0 tx=0x7fdb3400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 -- 172.21.15.3:0/4239522383 shutdown_connections 2023-09-10T01:17:01.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4239522383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb3c106650 0x7fdb3c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4239522383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c101000 0x7fdb3c10d510 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4239522383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb3c100640 0x7fdb3c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 -- 172.21.15.3:0/4239522383 >> 172.21.15.3:0/4239522383 conn(0x7fdb3c0fc170 msgr2=0x7fdb3c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:01.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 -- 172.21.15.3:0/4239522383 shutdown_connections 2023-09-10T01:17:01.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.495+0000 7fdb43fff700 1 -- 172.21.15.3:0/4239522383 wait complete. 2023-09-10T01:17:01.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.496+0000 7fdb43fff700 1 Processor -- start 2023-09-10T01:17:01.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb43fff700 1 -- start start 2023-09-10T01:17:01.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb43fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb3c100640 0x7fdb3c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb43fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb3c101000 0x7fdb3c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb43fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c106650 0x7fdb3c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb43fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c198df0 con 0x7fdb3c106650 2023-09-10T01:17:01.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb43fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c198f60 con 0x7fdb3c100640 2023-09-10T01:17:01.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb43fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c199120 con 0x7fdb3c101000 2023-09-10T01:17:01.501 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c106650 0x7fdb3c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c106650 0x7fdb3c198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56710/0 (socket says 172.21.15.3:56710) 2023-09-10T01:17:01.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb4259c700 1 -- 172.21.15.3:0/4065087963 learned_addr learned my addr 172.21.15.3:0/4065087963 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:01.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb4159a700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb3c101000 0x7fdb3c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.497+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb3c100640 0x7fdb3c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb4259c700 1 -- 172.21.15.3:0/4065087963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb3c101000 msgr2=0x7fdb3c19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb4259c700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb3c101000 0x7fdb3c19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb4259c700 1 -- 172.21.15.3:0/4065087963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb3c100640 msgr2=0x7fdb3c19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb4259c700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb3c100640 0x7fdb3c19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb4259c700 1 -- 172.21.15.3:0/4065087963 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c009520 con 0x7fdb3c106650 2023-09-10T01:17:01.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb4259c700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c106650 0x7fdb3c198780 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009850 tx=0x7fdb340077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb32ffd700 1 -- 172.21.15.3:0/4065087963 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400e040 con 0x7fdb3c106650 2023-09-10T01:17:01.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb32ffd700 1 -- 172.21.15.3:0/4065087963 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb340079e0 con 0x7fdb3c106650 2023-09-10T01:17:01.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb34009d00 con 0x7fdb3c106650 2023-09-10T01:17:01.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb32ffd700 1 -- 172.21.15.3:0/4065087963 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb340052a0 con 0x7fdb3c106650 2023-09-10T01:17:01.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.498+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c1a5520 con 0x7fdb3c106650 2023-09-10T01:17:01.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.502+0000 7fdb32ffd700 1 -- 172.21.15.3:0/4065087963 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fdb3401d470 con 0x7fdb3c106650 2023-09-10T01:17:01.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.503+0000 7fdb5798eb80 1 Processor -- start 2023-09-10T01:17:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.503+0000 7fdb32ffd700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdb280a0830 0x7fdb280a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.503+0000 7fdb5798eb80 1 -- start start 2023-09-10T01:17:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.503+0000 7fdb32ffd700 1 -- 172.21.15.3:0/4065087963 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fdb340cd6c0 con 0x7fdb3c106650 2023-09-10T01:17:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.503+0000 7fdb4159a700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdb280a0830 0x7fdb280a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba7989faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba797f8730 con 0x55ba798a9e40 2023-09-10T01:17:01.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba797f6050 con 0x55ba798a0030 2023-09-10T01:17:01.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba797f1940 con 0x55ba7989f710 2023-09-10T01:17:01.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4159a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4159a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40096/0 (socket says 172.21.15.3:40096) 2023-09-10T01:17:01.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56726/0 (socket says 172.21.15.3:56726) 2023-09-10T01:17:01.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4159a700 1 -- 172.21.15.3:0/2095592320 learned_addr learned my addr 172.21.15.3:0/2095592320 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:01.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba7989faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4159a700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdb280a0830 0x7fdb280a2cf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c199bf0 tx=0x7fdb2c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4259c700 1 -- 172.21.15.3:0/2095592320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 msgr2=0x55ba7989faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4259c700 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba7989faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4259c700 1 -- 172.21.15.3:0/2095592320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 msgr2=0x55ba798a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.504+0000 7fdb4259c700 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.505+0000 7fdb4259c700 1 -- 172.21.15.3:0/2095592320 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba7961dde0 con 0x55ba798a9e40 2023-09-10T01:17:01.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.505+0000 7fdb4259c700 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ac230 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fdb340ca9c0 tx=0x7fdb340099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb32208163230ada server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.505+0000 7fdb27fff700 1 -- 172.21.15.3:0/2095592320 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34004e60 con 0x55ba798a9e40 2023-09-10T01:17:01.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.505+0000 7fdb27fff700 1 -- 172.21.15.3:0/2095592320 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb3409aeb0 con 0x55ba798a9e40 2023-09-10T01:17:01.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.505+0000 7fdb27fff700 1 -- 172.21.15.3:0/2095592320 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34098c70 con 0x55ba798a9e40 2023-09-10T01:17:01.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2095592320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 msgr2=0x55ba798ac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ac230 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fdb340ca9c0 tx=0x7fdb340099e0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2095592320 shutdown_connections 2023-09-10T01:17:01.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba7989faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.513 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ac230 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2095592320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2095592320 >> 172.21.15.3:0/2095592320 conn(0x55ba79814090 msgr2=0x55ba7989abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:01.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2095592320 shutdown_connections 2023-09-10T01:17:01.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2095592320 wait complete. 2023-09-10T01:17:01.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.506+0000 7fdb5798eb80 1 -- start start 2023-09-10T01:17:01.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 1 Processor -- start 2023-09-10T01:17:01.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba798a6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a0030 0x55ba798a7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a9e40 0x55ba798a75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba7978fb70 con 0x55ba798a0030 2023-09-10T01:17:01.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba7978f870 con 0x55ba798a9e40 2023-09-10T01:17:01.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba79673920 con 0x55ba7989f710 2023-09-10T01:17:01.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb5798eb80 10 client.0 _ll_register_callbacks cb 0x55ba796d9a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:01.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb4159a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a0030 0x55ba798a7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.507+0000 7fdb41d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba798a6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba798a6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44908/0 (socket says 172.21.15.3:44908) 2023-09-10T01:17:01.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb4159a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a0030 0x55ba798a7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56742/0 (socket says 172.21.15.3:56742) 2023-09-10T01:17:01.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 -- 172.21.15.3:0/1260538443 learned_addr learned my addr 172.21.15.3:0/1260538443 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:01.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb4259c700 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a9e40 0x55ba798a75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 -- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a9e40 msgr2=0x55ba798a75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a9e40 0x55ba798a75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 -- 172.21.15.3:0/1260538443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a0030 msgr2=0x55ba798a7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a0030 0x55ba798a7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 -- 172.21.15.3:0/1260538443 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c1a5520 con 0x55ba7989f710 2023-09-10T01:17:01.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb4159a700 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a0030 0x55ba798a7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:01.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba798a6b20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdb38005c50 tx=0x7fdb38012a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=629e80661d86fce3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb257fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:01.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb257fa700 1 -- 172.21.15.3:0/1260538443 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38017070 con 0x55ba7989f710 2023-09-10T01:17:01.522 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.508+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba797494f0 con 0x55ba7989f710 2023-09-10T01:17:01.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb257fa700 1 -- 172.21.15.3:0/1260538443 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fdb38010430 con 0x55ba7989f710 2023-09-10T01:17:01.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb257fa700 1 -- 172.21.15.3:0/1260538443 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb380122e0 con 0x55ba7989f710 2023-09-10T01:17:01.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 10 client.28648 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:01.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55ba79606c60 con 0x55ba7989f710 2023-09-10T01:17:01.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb257fa700 1 -- 172.21.15.3:0/1260538443 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ba797494f0 con 0x55ba7989f710 2023-09-10T01:17:01.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb257fa700 1 -- 172.21.15.3:0/1260538443 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fdb3801c790 con 0x55ba7989f710 2023-09-10T01:17:01.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 10 client.28648 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:01.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 10 client.28648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:01.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 20 client.28648 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:01.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba798b98d0 0x55ba798bbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 4 client.28648 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:01.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.509+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba798068a0 con 0x55ba798b98d0 2023-09-10T01:17:01.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.510+0000 7fdb4159a700 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba798b98d0 0x55ba798bbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.510+0000 7fdb4159a700 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba798b98d0 0x55ba798bbcf0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21699a07b259023d server_cookie=49c4d64e8c917075 in_seq=0 out_seq=0 2023-09-10T01:17:01.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.510+0000 7fdb257fa700 10 client.28648 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:01.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb257fa700 1 -- 172.21.15.3:0/1260538443 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ba798068a0 con 0x55ba798b98d0 2023-09-10T01:17:01.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb257fa700 10 client.28648 handle_command_reply: tid=0 2023-09-10T01:17:01.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 client.28648 shutdown 2023-09-10T01:17:01.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 msgr2=0x55ba798a6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba798a6b20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdb38005c50 tx=0x7fdb38012a20 comp rx=0 tx=0).stop 2023-09-10T01:17:01.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 shutdown_connections 2023-09-10T01:17:01.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba798b98d0 0x55ba798bbcf0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba798a6b20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a9e40 0x55ba798a75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/1260538443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a0030 0x55ba798a7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 >> 172.21.15.3:0/1260538443 conn(0x55ba79814090 msgr2=0x55ba7989a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:01.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb257fa700 0 client.28648 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:01.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 shutdown_connections 2023-09-10T01:17:01.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 1 -- 172.21.15.3:0/1260538443 wait complete. 2023-09-10T01:17:01.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.511+0000 7fdb5798eb80 20 client.28648 trim_cache size 0 max 16384 2023-09-10T01:17:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 Processor -- start 2023-09-10T01:17:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 -- start start 2023-09-10T01:17:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba7989f710 0x55ba7989ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba798a9e40 0x55ba798a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba7989c270 con 0x55ba7989f710 2023-09-10T01:17:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba79892430 con 0x55ba798a0030 2023-09-10T01:17:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba798a1930 con 0x55ba798a9e40 2023-09-10T01:17:01.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb41d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba7989f710 0x55ba7989ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba798a9e40 0x55ba798a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb41d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba7989f710 0x55ba7989ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56752/0 (socket says 172.21.15.3:56752) 2023-09-10T01:17:01.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb4159a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb41d9b700 1 -- 172.21.15.3:0/156773341 learned_addr learned my addr 172.21.15.3:0/156773341 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:01.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.529+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba798a9e40 0x55ba798a3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44924/0 (socket says 172.21.15.3:44924) 2023-09-10T01:17:01.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb4259c700 1 -- 172.21.15.3:0/156773341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 msgr2=0x55ba798a73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb4259c700 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb4259c700 1 -- 172.21.15.3:0/156773341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba7989f710 msgr2=0x55ba7989ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.537 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb4259c700 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba7989f710 0x55ba7989ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb4259c700 1 -- 172.21.15.3:0/156773341 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba79644250 con 0x55ba798a9e40 2023-09-10T01:17:01.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb4159a700 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:01.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba7989f710 0x55ba7989ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:01.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb4259c700 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba798a9e40 0x55ba798a3dd0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdb340ca9f0 tx=0x7fdb3400b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faca6eb7952ffc93 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb30ff9700 1 -- 172.21.15.3:0/156773341 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb340b1070 con 0x55ba798a9e40 2023-09-10T01:17:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb30ff9700 1 -- 172.21.15.3:0/156773341 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb3409a7b0 con 0x55ba798a9e40 2023-09-10T01:17:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.530+0000 7fdb30ff9700 1 -- 172.21.15.3:0/156773341 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34098c70 con 0x55ba798a9e40 2023-09-10T01:17:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 -- 172.21.15.3:0/156773341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba798a9e40 msgr2=0x55ba798a3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba798a9e40 0x55ba798a3dd0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdb340ca9f0 tx=0x7fdb3400b930 comp rx=0 tx=0).stop 2023-09-10T01:17:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 -- 172.21.15.3:0/156773341 shutdown_connections 2023-09-10T01:17:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba798a9e40 0x55ba798a3dd0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba7989f710 0x55ba7989ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/156773341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba798a73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 -- 172.21.15.3:0/156773341 >> 172.21.15.3:0/156773341 conn(0x55ba79814090 msgr2=0x55ba7989e700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 -- 172.21.15.3:0/156773341 shutdown_connections 2023-09-10T01:17:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 -- 172.21.15.3:0/156773341 wait complete. 2023-09-10T01:17:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 -- start start 2023-09-10T01:17:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.531+0000 7fdb5798eb80 1 Processor -- start 2023-09-10T01:17:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba79906a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.541 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba79906f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb5798eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ae810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba798aede0 con 0x55ba798a9e40 2023-09-10T01:17:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba798aef50 con 0x55ba798a0030 2023-09-10T01:17:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb5798eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba798af110 con 0x55ba7989f710 2023-09-10T01:17:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb5798eb80 10 client.0 _ll_register_callbacks cb 0x7fdb38009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56754/0 (socket says 172.21.15.3:56754) 2023-09-10T01:17:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 -- 172.21.15.3:0/2752738164 learned_addr learned my addr 172.21.15.3:0/2752738164 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4159a700 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba79906f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba79906a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 -- 172.21.15.3:0/2752738164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 msgr2=0x55ba79906a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba79906a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 -- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 msgr2=0x55ba79906f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba79906f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.532+0000 7fdb4259c700 1 -- 172.21.15.3:0/2752738164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba79606c60 con 0x55ba798a9e40 2023-09-10T01:17:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb4259c700 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ae810 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdb340090a0 tx=0x7fdb3400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ce9966ae3a7616 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb257fa700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb257fa700 1 -- 172.21.15.3:0/2752738164 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb340b1070 con 0x55ba798a9e40 2023-09-10T01:17:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb3409aeb0 con 0x55ba798a9e40 2023-09-10T01:17:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb257fa700 1 -- 172.21.15.3:0/2752738164 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fdb3409e5e0 con 0x55ba798a9e40 2023-09-10T01:17:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb257fa700 1 -- 172.21.15.3:0/2752738164 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb340a2790 con 0x55ba798a9e40 2023-09-10T01:17:01.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb5798eb80 10 client.18678 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:01.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7fdb2c017050 con 0x55ba798a9e40 2023-09-10T01:17:01.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.533+0000 7fdb257fa700 1 -- 172.21.15.3:0/2752738164 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb340a6080 con 0x55ba798a9e40 2023-09-10T01:17:01.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb257fa700 1 -- 172.21.15.3:0/2752738164 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fdb340b2d20 con 0x55ba798a9e40 2023-09-10T01:17:01.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb5798eb80 10 client.18678 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:01.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb5798eb80 10 client.18678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:01.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb5798eb80 20 client.18678 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:01.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba7990e310 0x55ba79910730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb5798eb80 4 client.18678 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:17:01.547 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ba797ea920 con 0x55ba7990e310 2023-09-10T01:17:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba7990e310 0x55ba79910730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb41d9b700 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba7990e310 0x55ba79910730 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea3e266a83a35cb2 server_cookie=2320bc0067aaff54 in_seq=0 out_seq=0 2023-09-10T01:17:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb257fa700 10 client.18678 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:01.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb257fa700 1 -- 172.21.15.3:0/2752738164 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ba797ea920 con 0x55ba7990e310 2023-09-10T01:17:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb257fa700 10 client.18678 handle_command_reply: tid=0 2023-09-10T01:17:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.534+0000 7fdb5798eb80 1 client.18678 shutdown 2023-09-10T01:17:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 msgr2=0x55ba798ae810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ae810 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdb340090a0 tx=0x7fdb3400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 shutdown_connections 2023-09-10T01:17:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55ba7990e310 0x55ba79910730 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ba7989f710 0x55ba79906a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba798a9e40 0x55ba798ae810 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 --2- 172.21.15.3:0/2752738164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba798a0030 0x55ba79906f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb257fa700 0 client.18678 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 >> 172.21.15.3:0/2752738164 conn(0x55ba79814090 msgr2=0x55ba7989ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 shutdown_connections 2023-09-10T01:17:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 1 -- 172.21.15.3:0/2752738164 wait complete. 2023-09-10T01:17:01.551 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb5798eb80 20 client.18678 trim_cache size 0 max 16384 2023-09-10T01:17:01.552 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:17:01.552 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:17:01.552 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:17:01.552 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:17:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdb280a0830 msgr2=0x7fdb280a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.535+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdb280a0830 0x7fdb280a2cf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c199bf0 tx=0x7fdb2c009260 comp rx=0 tx=0).stop 2023-09-10T01:17:01.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c106650 msgr2=0x7fdb3c198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c106650 0x7fdb3c198780 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009850 tx=0x7fdb340077f0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 shutdown_connections 2023-09-10T01:17:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fdb280a0830 0x7fdb280a2cf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb3c106650 0x7fdb3c198780 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb3c100640 0x7fdb3c19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 --2- 172.21.15.3:0/4065087963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb3c101000 0x7fdb3c19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 >> 172.21.15.3:0/4065087963 conn(0x7fdb3c0fc170 msgr2=0x7fdb3c10cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 shutdown_connections 2023-09-10T01:17:01.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.536+0000 7fdb43fff700 1 -- 172.21.15.3:0/4065087963 wait complete. 2023-09-10T01:17:01.556 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:17:01.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:17:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.761+0000 7f86ed1ff700 1 -- 172.21.15.3:0/370729795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8100fe0 msgr2=0x7f86e810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.761+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/370729795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8100fe0 0x7f86e810d4f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f86e000b0a0 tx=0x7f86e000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.762+0000 7f86ed1ff700 1 -- 172.21.15.3:0/370729795 shutdown_connections 2023-09-10T01:17:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.762+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/370729795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8106630 0x7f86e8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.762+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/370729795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8100fe0 0x7f86e810d4f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.762+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/370729795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86e8100620 0x7f86e8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.762+0000 7f86ed1ff700 1 -- 172.21.15.3:0/370729795 >> 172.21.15.3:0/370729795 conn(0x7f86e80fc110 msgr2=0x7f86e80fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.762+0000 7f86ed1ff700 1 -- 172.21.15.3:0/370729795 shutdown_connections 2023-09-10T01:17:01.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.762+0000 7f86ed1ff700 1 -- 172.21.15.3:0/370729795 wait complete. 2023-09-10T01:17:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 Processor -- start 2023-09-10T01:17:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 -- start start 2023-09-10T01:17:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8100620 0x7f86e819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86e8100fe0 0x7f86e819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8106630 0x7f86e81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e8198d50 con 0x7f86e8106630 2023-09-10T01:17:01.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e8198ec0 con 0x7f86e8100fe0 2023-09-10T01:17:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.763+0000 7f86ed1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e8199080 con 0x7f86e8100620 2023-09-10T01:17:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8100620 0x7f86e819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8100620 0x7f86e819e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44954/0 (socket says 172.21.15.3:44954) 2023-09-10T01:17:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8106630 0x7f86e81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 -- 172.21.15.3:0/3478492106 learned_addr learned my addr 172.21.15.3:0/3478492106 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e659c700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86e8100fe0 0x7f86e819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 -- 172.21.15.3:0/3478492106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86e8100fe0 msgr2=0x7f86e819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86e8100fe0 0x7f86e819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 -- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8106630 msgr2=0x7f86e81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:01.769 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8106630 0x7f86e81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 -- 172.21.15.3:0/3478492106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0009d00 con 0x7f86e8100620 2023-09-10T01:17:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86e6d9d700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8100620 0x7f86e819e670 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f86d800e9a0 tx=0x7f86d800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86cffff700 1 -- 172.21.15.3:0/3478492106 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8009790 con 0x7f86e8100620 2023-09-10T01:17:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86cffff700 1 -- 172.21.15.3:0/3478492106 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86d8010430 con 0x7f86e8100620 2023-09-10T01:17:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86cffff700 1 -- 172.21.15.3:0/3478492106 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8018680 con 0x7f86e8100620 2023-09-10T01:17:01.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.764+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e8199360 con 0x7f86e8100620 2023-09-10T01:17:01.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.765+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e81a5660 con 0x7f86e8100620 2023-09-10T01:17:01.771 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.765+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e8067480 con 0x7f86e8100620 2023-09-10T01:17:01.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.768+0000 7f86cffff700 1 -- 172.21.15.3:0/3478492106 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f86d80105a0 con 0x7f86e8100620 2023-09-10T01:17:01.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.769+0000 7f86cffff700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f86d00a0830 0x7f86d00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:01.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.769+0000 7f86cffff700 1 -- 172.21.15.3:0/3478492106 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f86d8014070 con 0x7f86e8100620 2023-09-10T01:17:01.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.769+0000 7f86e659c700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f86d00a0830 0x7f86d00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:01.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.770+0000 7f86e659c700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f86d00a0830 0x7f86d00a2cf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f86e8199b50 tx=0x7f86dc009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:01.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:01.771+0000 7f86cffff700 1 -- 172.21.15.3:0/3478492106 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f86d8095ad0 con 0x7f86e8100620 2023-09-10T01:17:02.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.028+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86e8199bf0 con 0x7f86e8100620 2023-09-10T01:17:02.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.029+0000 7f86cffff700 1 -- 172.21.15.3:0/3478492106 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+11299 (secure 0 0 0) 0x7f86d8026020 con 0x7f86e8100620 2023-09-10T01:17:02.030 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:17:02.031 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:16:25.285703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:17:02.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f86d00a0830 msgr2=0x7f86d00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f86d00a0830 0x7f86d00a2cf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f86e8199b50 tx=0x7f86dc009260 comp rx=0 tx=0).stop 2023-09-10T01:17:02.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8100620 msgr2=0x7f86e819e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8100620 0x7f86e819e670 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f86d800e9a0 tx=0x7f86d800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 shutdown_connections 2023-09-10T01:17:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f86d00a0830 0x7f86d00a2cf0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e8100620 0x7f86e819e670 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86e8106630 0x7f86e81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 --2- 172.21.15.3:0/3478492106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86e8100fe0 0x7f86e819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 >> 172.21.15.3:0/3478492106 conn(0x7f86e80fc110 msgr2=0x7f86e810bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 shutdown_connections 2023-09-10T01:17:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.032+0000 7f86ed1ff700 1 -- 172.21.15.3:0/3478492106 wait complete. 2023-09-10T01:17:02.035 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 38 2023-09-10T01:17:02.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:17:02.158 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24153 34.35 MB/sec warmup 43 sec latency 1506.251 ms 2023-09-10T01:17:02.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.307+0000 7f03653ff700 1 -- 172.21.15.3:0/1065315920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600fe830 msgr2=0x7f03600fecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.307+0000 7f03653ff700 1 --2- 172.21.15.3:0/1065315920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600fe830 0x7f03600fecb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0354009990 tx=0x7f0354009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.309+0000 7f03653ff700 1 -- 172.21.15.3:0/1065315920 shutdown_connections 2023-09-10T01:17:02.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.309+0000 7f03653ff700 1 --2- 172.21.15.3:0/1065315920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0360104840 0x7f0360104c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.309+0000 7f03653ff700 1 --2- 172.21.15.3:0/1065315920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03600ff1f0 0x7f036010b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.309+0000 7f03653ff700 1 --2- 172.21.15.3:0/1065315920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600fe830 0x7f03600fecb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.309+0000 7f03653ff700 1 -- 172.21.15.3:0/1065315920 >> 172.21.15.3:0/1065315920 conn(0x7f03600fa2e0 msgr2=0x7f03600fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.310+0000 7f03653ff700 1 -- 172.21.15.3:0/1065315920 shutdown_connections 2023-09-10T01:17:02.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:02 smithi003 ceph-mon[97654]: pgmap v431: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.5 MiB/s wr, 770 op/s 2023-09-10T01:17:02.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:02 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3478492106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:17:02.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.310+0000 7f03653ff700 1 -- 172.21.15.3:0/1065315920 wait complete. 2023-09-10T01:17:02.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.311+0000 7f03653ff700 1 Processor -- start 2023-09-10T01:17:02.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.312+0000 7f03653ff700 1 -- start start 2023-09-10T01:17:02.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.312+0000 7f03653ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03600fe830 0x7f03600719f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.312+0000 7f03653ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600ff1f0 0x7f0360071f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.312+0000 7f03653ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0360104840 0x7f036010d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.312+0000 7f03653ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036010db20 con 0x7f0360104840 2023-09-10T01:17:02.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.313+0000 7f03653ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036010dc60 con 0x7f03600ff1f0 2023-09-10T01:17:02.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.313+0000 7f03653ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036010de20 con 0x7f03600fe830 2023-09-10T01:17:02.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.313+0000 7f035e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600ff1f0 0x7f0360071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.313+0000 7f035e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600ff1f0 0x7f0360071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40162/0 (socket says 172.21.15.3:40162) 2023-09-10T01:17:02.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.313+0000 7f035e7fc700 1 -- 172.21.15.3:0/2990636691 learned_addr learned my addr 172.21.15.3:0/2990636691 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.313+0000 7f035f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0360104840 0x7f036010d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.314+0000 7f035e7fc700 1 -- 172.21.15.3:0/2990636691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03600fe830 msgr2=0x7f03600719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.314+0000 7f035e7fc700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03600fe830 0x7f03600719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.314+0000 7f035e7fc700 1 -- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0360104840 msgr2=0x7f036010d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.314+0000 7f035e7fc700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0360104840 0x7f036010d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.314+0000 7f035e7fc700 1 -- 172.21.15.3:0/2990636691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03540095f0 con 0x7f03600ff1f0 2023-09-10T01:17:02.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.314+0000 7f035e7fc700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600ff1f0 0x7f0360071f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0354005910 tx=0x7f03540047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.316+0000 7f0347fff700 1 -- 172.21.15.3:0/2990636691 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035401d070 con 0x7f03600ff1f0 2023-09-10T01:17:02.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.316+0000 7f0347fff700 1 -- 172.21.15.3:0/2990636691 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0354022470 con 0x7f03600ff1f0 2023-09-10T01:17:02.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.316+0000 7f0347fff700 1 -- 172.21.15.3:0/2990636691 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354017650 con 0x7f03600ff1f0 2023-09-10T01:17:02.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.317+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036010e0a0 con 0x7f03600ff1f0 2023-09-10T01:17:02.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.317+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03601a5c30 con 0x7f03600ff1f0 2023-09-10T01:17:02.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.327+0000 7f0378ca2b80 1 Processor -- start 2023-09-10T01:17:02.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.327+0000 7f0347fff700 1 -- 172.21.15.3:0/2990636691 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f03540225e0 con 0x7f03600ff1f0 2023-09-10T01:17:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.329+0000 7f0378ca2b80 1 -- start start 2023-09-10T01:17:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.329+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 0x5626900ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.329+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900f6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.329+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 0x5626900f9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.329+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562690045730 con 0x5626900ec710 2023-09-10T01:17:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.329+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562690043050 con 0x5626900ed030 2023-09-10T01:17:02.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.329+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56269003e940 con 0x5626900f6e40 2023-09-10T01:17:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.330+0000 7f035effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.330+0000 7f035effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900f6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40164/0 (socket says 172.21.15.3:40164) 2023-09-10T01:17:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.330+0000 7f035effd700 1 -- 172.21.15.3:0/903159887 learned_addr learned my addr 172.21.15.3:0/903159887 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.330+0000 7f035effd700 1 -- 172.21.15.3:0/903159887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 msgr2=0x5626900f9230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:02.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.330+0000 7f035effd700 1 --2- 172.21.15.3:0/903159887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 0x5626900f9230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.330+0000 7f035effd700 1 -- 172.21.15.3:0/903159887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 msgr2=0x5626900ecaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.330+0000 7f035effd700 1 --2- 172.21.15.3:0/903159887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 0x5626900ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.331+0000 7f035effd700 1 -- 172.21.15.3:0/903159887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268fe6ade0 con 0x5626900ed030 2023-09-10T01:17:02.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.332+0000 7f035effd700 1 --2- 172.21.15.3:0/903159887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900f6900 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0350009c50 tx=0x7f0350009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f3a98c0831b4c7a server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.334+0000 7f0347fff700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03480a0980 0x7f03480a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.334+0000 7f0347fff700 1 -- 172.21.15.3:0/2990636691 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f03540cd300 con 0x7f03600ff1f0 2023-09-10T01:17:02.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.336+0000 7f035f7fe700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03480a0980 0x7f03480a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.337+0000 7f035f7fe700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03480a0980 0x7f03480a2e40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0358009360 tx=0x7f0358009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.338+0000 7f03457fa700 1 -- 172.21.15.3:0/903159887 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035000ebe0 con 0x5626900ed030 2023-09-10T01:17:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.338+0000 7f03457fa700 1 -- 172.21.15.3:0/903159887 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f035000ed40 con 0x5626900ed030 2023-09-10T01:17:02.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.338+0000 7f03457fa700 1 -- 172.21.15.3:0/903159887 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350010430 con 0x5626900ed030 2023-09-10T01:17:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.339+0000 7f0378ca2b80 1 -- 172.21.15.3:0/903159887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 msgr2=0x5626900f6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.339+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/903159887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900f6900 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0350009c50 tx=0x7f0350009f60 comp rx=0 tx=0).stop 2023-09-10T01:17:02.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.340+0000 7f0378ca2b80 1 -- 172.21.15.3:0/903159887 shutdown_connections 2023-09-10T01:17:02.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.340+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/903159887 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 0x5626900f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.340+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/903159887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 0x5626900ecaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.340+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/903159887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900f6900 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.340+0000 7f0378ca2b80 1 -- 172.21.15.3:0/903159887 >> 172.21.15.3:0/903159887 conn(0x562690061090 msgr2=0x5626900e7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.340+0000 7f0378ca2b80 1 -- 172.21.15.3:0/903159887 shutdown_connections 2023-09-10T01:17:02.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.340+0000 7f0378ca2b80 1 -- 172.21.15.3:0/903159887 wait complete. 2023-09-10T01:17:02.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.341+0000 7f0378ca2b80 1 -- start start 2023-09-10T01:17:02.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.341+0000 7f0378ca2b80 1 Processor -- start 2023-09-10T01:17:02.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.342+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ec710 0x5626900f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.342+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ed030 0x5626900f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900f6e40 0x5626900f4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ffdcb70 con 0x5626900f6e40 2023-09-10T01:17:02.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ffdc870 con 0x5626900ec710 2023-09-10T01:17:02.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268fec0920 con 0x5626900ed030 2023-09-10T01:17:02.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f0378ca2b80 10 client.0 _ll_register_callbacks cb 0x56268ff26a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:02.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f035effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ec710 0x5626900f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f035effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ec710 0x5626900f3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40168/0 (socket says 172.21.15.3:40168) 2023-09-10T01:17:02.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.343+0000 7f035effd700 1 -- 172.21.15.3:0/435944690 learned_addr learned my addr 172.21.15.3:0/435944690 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.345+0000 7f035f7fe700 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900f6e40 0x5626900f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.346+0000 7f035f7fe700 1 -- 172.21.15.3:0/435944690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ed030 msgr2=0x5626900f40e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:02.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.346+0000 7f035f7fe700 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ed030 0x5626900f40e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.346+0000 7f035f7fe700 1 -- 172.21.15.3:0/435944690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ec710 msgr2=0x5626900f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.346+0000 7f035f7fe700 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ec710 0x5626900f3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.346+0000 7f035f7fe700 1 -- 172.21.15.3:0/435944690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0350009900 con 0x5626900f6e40 2023-09-10T01:17:02.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.346+0000 7f035f7fe700 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900f6e40 0x5626900f4620 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0358007b70 tx=0x7f0358004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf388024afeacb08 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.346+0000 7f0342ff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:02.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.347+0000 7f0342ff5700 1 -- 172.21.15.3:0/435944690 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358025070 con 0x5626900f6e40 2023-09-10T01:17:02.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.347+0000 7f0342ff5700 1 -- 172.21.15.3:0/435944690 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0358004bb0 con 0x5626900f6e40 2023-09-10T01:17:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.347+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56268ff964f0 con 0x5626900f6e40 2023-09-10T01:17:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.348+0000 7f0342ff5700 1 -- 172.21.15.3:0/435944690 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035800e050 con 0x5626900f6e40 2023-09-10T01:17:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.348+0000 7f0378ca2b80 10 client.18693 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:02.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.348+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56268fe53c60 con 0x5626900f6e40 2023-09-10T01:17:02.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0342ff5700 1 -- 172.21.15.3:0/435944690 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56268ff964f0 con 0x5626900f6e40 2023-09-10T01:17:02.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0342ff5700 1 -- 172.21.15.3:0/435944690 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0358028020 con 0x5626900f6e40 2023-09-10T01:17:02.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0378ca2b80 10 client.18693 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:02.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0378ca2b80 10 client.18693 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:17:02.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0378ca2b80 20 client.18693 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:02.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562690128b20 0x56269012af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0378ca2b80 4 client.18693 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:02.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.349+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626900538a0 con 0x562690128b20 2023-09-10T01:17:02.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.355+0000 7f035effd700 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562690128b20 0x56269012af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.363+0000 7f035effd700 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562690128b20 0x56269012af60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=32a0a71c57f49af8 server_cookie=ad77a2e8dcf5117a in_seq=0 out_seq=0 2023-09-10T01:17:02.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.364+0000 7f0342ff5700 10 client.18693 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:02.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.367+0000 7f0342ff5700 1 -- 172.21.15.3:0/435944690 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5626900538a0 con 0x562690128b20 2023-09-10T01:17:02.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.367+0000 7f0342ff5700 10 client.18693 handle_command_reply: tid=0 2023-09-10T01:17:02.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.367+0000 7f0378ca2b80 1 client.18693 shutdown 2023-09-10T01:17:02.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900f6e40 msgr2=0x5626900f4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900f6e40 0x5626900f4620 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0358007b70 tx=0x7f0358004b40 comp rx=0 tx=0).stop 2023-09-10T01:17:02.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 shutdown_connections 2023-09-10T01:17:02.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x562690128b20 0x56269012af60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900f6e40 0x5626900f4620 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ec710 0x5626900f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/435944690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ed030 0x5626900f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 >> 172.21.15.3:0/435944690 conn(0x562690061090 msgr2=0x5626900e77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0342ff5700 0 client.18693 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:02.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 shutdown_connections 2023-09-10T01:17:02.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.368+0000 7f0378ca2b80 1 -- 172.21.15.3:0/435944690 wait complete. 2023-09-10T01:17:02.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.369+0000 7f0378ca2b80 20 client.18693 trim_cache size 0 max 16384 2023-09-10T01:17:02.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.397+0000 7f0378ca2b80 1 Processor -- start 2023-09-10T01:17:02.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.398+0000 7f0378ca2b80 1 -- start start 2023-09-10T01:17:02.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.398+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 0x5626900f4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.398+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.398+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 0x5626900ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.398+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626900ee6e0 con 0x5626900ec710 2023-09-10T01:17:02.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.398+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626900f96d0 con 0x5626900ed030 2023-09-10T01:17:02.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.398+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626900e90b0 con 0x5626900f6e40 2023-09-10T01:17:02.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.400+0000 7f035e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.400+0000 7f035e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900eeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40174/0 (socket says 172.21.15.3:40174) 2023-09-10T01:17:02.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.400+0000 7f035e7fc700 1 -- 172.21.15.3:0/790680651 learned_addr learned my addr 172.21.15.3:0/790680651 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.402+0000 7f035e7fc700 1 -- 172.21.15.3:0/790680651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 msgr2=0x5626900ef3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:02.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.402+0000 7f035e7fc700 1 --2- 172.21.15.3:0/790680651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 0x5626900ef3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.402+0000 7f035e7fc700 1 -- 172.21.15.3:0/790680651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 msgr2=0x5626900f4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.402+0000 7f035e7fc700 1 --2- 172.21.15.3:0/790680651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 0x5626900f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.402+0000 7f035e7fc700 1 -- 172.21.15.3:0/790680651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268fe91250 con 0x5626900ed030 2023-09-10T01:17:02.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.405+0000 7f035e7fc700 1 --2- 172.21.15.3:0/790680651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900eeea0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0354096e60 tx=0x7f0354096090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5d3b10c62ce6168 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.408+0000 7f0345ffb700 1 -- 172.21.15.3:0/790680651 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354099d50 con 0x5626900ed030 2023-09-10T01:17:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.409+0000 7f0345ffb700 1 -- 172.21.15.3:0/790680651 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f035409aaf0 con 0x5626900ed030 2023-09-10T01:17:02.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.409+0000 7f0345ffb700 1 -- 172.21.15.3:0/790680651 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03540a00c0 con 0x5626900ed030 2023-09-10T01:17:02.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.410+0000 7f0378ca2b80 1 -- 172.21.15.3:0/790680651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 msgr2=0x5626900eeea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.410+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/790680651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900eeea0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0354096e60 tx=0x7f0354096090 comp rx=0 tx=0).stop 2023-09-10T01:17:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.412+0000 7f0378ca2b80 1 -- 172.21.15.3:0/790680651 shutdown_connections 2023-09-10T01:17:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.412+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/790680651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900f6e40 0x5626900ef3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.412+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/790680651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900ec710 0x5626900f4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.412+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/790680651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900ed030 0x5626900eeea0 secure :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0354096e60 tx=0x7f0354096090 comp rx=0 tx=0).stop 2023-09-10T01:17:02.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.412+0000 7f0378ca2b80 1 -- 172.21.15.3:0/790680651 >> 172.21.15.3:0/790680651 conn(0x562690061090 msgr2=0x56269012ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.417+0000 7f0378ca2b80 1 -- 172.21.15.3:0/790680651 shutdown_connections 2023-09-10T01:17:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.418+0000 7f0378ca2b80 1 -- 172.21.15.3:0/790680651 wait complete. 2023-09-10T01:17:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.419+0000 7f0378ca2b80 1 -- start start 2023-09-10T01:17:02.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.419+0000 7f0378ca2b80 1 Processor -- start 2023-09-10T01:17:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.420+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ec710 0x5626900fd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.420+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900f6e40 0x5626900fdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.420+0000 7f0378ca2b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900fbb30 0x5626900fbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.420+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626900fc6c0 con 0x5626900fbb30 2023-09-10T01:17:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.420+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626900fc830 con 0x5626900f6e40 2023-09-10T01:17:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.420+0000 7f0378ca2b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56269017c980 con 0x5626900ec710 2023-09-10T01:17:02.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.420+0000 7f0378ca2b80 10 client.0 _ll_register_callbacks cb 0x7f0358015440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ec710 0x5626900fd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ec710 0x5626900fd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:44990/0 (socket says 172.21.15.3:44990) 2023-09-10T01:17:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 -- 172.21.15.3:0/3309321653 learned_addr learned my addr 172.21.15.3:0/3309321653 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035f7fe700 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900fbb30 0x5626900fbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 -- 172.21.15.3:0/3309321653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900f6e40 msgr2=0x5626900fdb70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900f6e40 0x5626900fdb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 -- 172.21.15.3:0/3309321653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900fbb30 msgr2=0x5626900fbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900fbb30 0x5626900fbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.421+0000 7f035effd700 1 -- 172.21.15.3:0/3309321653 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03601a5c30 con 0x5626900ec710 2023-09-10T01:17:02.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.429+0000 7f035effd700 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ec710 0x5626900fd630 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0350005950 tx=0x7f0350004720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a07e06aae1e0cb2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.429+0000 7f0342ff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.436+0000 7f0342ff5700 1 -- 172.21.15.3:0/3309321653 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350027b10 con 0x5626900ec710 2023-09-10T01:17:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.436+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f035000ed40 con 0x5626900ec710 2023-09-10T01:17:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.436+0000 7f0342ff5700 1 -- 172.21.15.3:0/3309321653 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f0350014070 con 0x5626900ec710 2023-09-10T01:17:02.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.437+0000 7f0342ff5700 1 -- 172.21.15.3:0/3309321653 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0350027c70 con 0x5626900ec710 2023-09-10T01:17:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.439+0000 7f0378ca2b80 10 client.28666 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.439+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56268fe692b0 con 0x5626900ec710 2023-09-10T01:17:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.442+0000 7f0342ff5700 1 -- 172.21.15.3:0/3309321653 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f035000ed40 con 0x5626900ec710 2023-09-10T01:17:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.442+0000 7f0342ff5700 1 -- 172.21.15.3:0/3309321653 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0350004830 con 0x5626900ec710 2023-09-10T01:17:02.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.442+0000 7f0378ca2b80 10 client.28666 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:02.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.442+0000 7f0378ca2b80 10 client.28666 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:17:02.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.442+0000 7f0378ca2b80 20 client.28666 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:02.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.442+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x56269017e080 0x5626901804a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.443+0000 7f0378ca2b80 4 client.28666 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:17:02.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.443+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562690037920 con 0x56269017e080 2023-09-10T01:17:02.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.444+0000 7f035e7fc700 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x56269017e080 0x5626901804a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.445+0000 7f0342ff5700 10 client.28666 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:02.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.445+0000 7f035e7fc700 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x56269017e080 0x5626901804a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e906b5a64ae83525 server_cookie=65dfc60ef50df39 in_seq=0 out_seq=0 2023-09-10T01:17:02.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.446+0000 7f0342ff5700 1 -- 172.21.15.3:0/3309321653 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562690037920 con 0x56269017e080 2023-09-10T01:17:02.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.446+0000 7f0342ff5700 10 client.28666 handle_command_reply: tid=0 2023-09-10T01:17:02.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.446+0000 7f0378ca2b80 1 client.28666 shutdown 2023-09-10T01:17:02.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ec710 msgr2=0x5626900fd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ec710 0x5626900fd630 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0350005950 tx=0x7f0350004720 comp rx=0 tx=0).stop 2023-09-10T01:17:02.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 shutdown_connections 2023-09-10T01:17:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x56269017e080 0x5626901804a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626900ec710 0x5626900fd630 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5626900fbb30 0x5626900fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 --2- 172.21.15.3:0/3309321653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5626900f6e40 0x5626900fdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 >> 172.21.15.3:0/3309321653 conn(0x562690061090 msgr2=0x5626900eeea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.447+0000 7f0342ff5700 0 client.28666 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.449+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 shutdown_connections 2023-09-10T01:17:02.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.450+0000 7f0378ca2b80 1 -- 172.21.15.3:0/3309321653 wait complete. 2023-09-10T01:17:02.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.450+0000 7f0378ca2b80 20 client.28666 trim_cache size 0 max 16384 2023-09-10T01:17:02.451 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:17:02.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03480a0980 msgr2=0x7f03480a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03480a0980 0x7f03480a2e40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0358009360 tx=0x7f0358009920 comp rx=0 tx=0).stop 2023-09-10T01:17:02.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600ff1f0 msgr2=0x7f0360071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600ff1f0 0x7f0360071f30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0354005910 tx=0x7f03540047c0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 shutdown_connections 2023-09-10T01:17:02.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f03480a0980 0x7f03480a2e40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03600fe830 0x7f03600719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0360104840 0x7f036010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 --2- 172.21.15.3:0/2990636691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03600ff1f0 0x7f0360071f30 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.454+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 >> 172.21.15.3:0/2990636691 conn(0x7f03600fa2e0 msgr2=0x7f03600fa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.455+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 shutdown_connections 2023-09-10T01:17:02.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.455+0000 7f03653ff700 1 -- 172.21.15.3:0/2990636691 wait complete. 2023-09-10T01:17:02.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:17:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:02 smithi136 ceph-mon[133708]: pgmap v431: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.5 MiB/s wr, 770 op/s 2023-09-10T01:17:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:02 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3478492106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:17:02.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:02 smithi092 ceph-mon[134067]: pgmap v431: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.5 MiB/s wr, 770 op/s 2023-09-10T01:17:02.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:02 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3478492106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:17:02.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.859+0000 7f72dafff700 1 -- 172.21.15.3:0/2765483737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72d4107600 msgr2=0x7f72d4107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.859+0000 7f72dafff700 1 --2- 172.21.15.3:0/2765483737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72d4107600 0x7f72d4107a80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00b0a0 tx=0x7f72cc00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.862+0000 7f72dafff700 1 -- 172.21.15.3:0/2765483737 shutdown_connections 2023-09-10T01:17:02.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.862+0000 7f72dafff700 1 --2- 172.21.15.3:0/2765483737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72d4107600 0x7f72d4107a80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.862+0000 7f72dafff700 1 --2- 172.21.15.3:0/2765483737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72d410edd0 0x7f72d410ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.862+0000 7f72dafff700 1 --2- 172.21.15.3:0/2765483737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410f1b0 0x7f72d410cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.864 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.862+0000 7f72dafff700 1 -- 172.21.15.3:0/2765483737 >> 172.21.15.3:0/2765483737 conn(0x7f72d4071dd0 msgr2=0x7f72d406ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.865+0000 7f72dafff700 1 -- 172.21.15.3:0/2765483737 shutdown_connections 2023-09-10T01:17:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.866+0000 7f72dafff700 1 -- 172.21.15.3:0/2765483737 wait complete. 2023-09-10T01:17:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.867+0000 7f72dafff700 1 Processor -- start 2023-09-10T01:17:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.868+0000 7f72dafff700 1 -- start start 2023-09-10T01:17:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.868+0000 7f72dafff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72d4107600 0x7f72d411d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.868+0000 7f72dafff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410edd0 0x7f72d411d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.868+0000 7f72dafff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72d410f1b0 0x7f72d41160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.868+0000 7f72dafff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d41167f0 con 0x7f72d410f1b0 2023-09-10T01:17:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.868+0000 7f72dafff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d4116960 con 0x7f72d410edd0 2023-09-10T01:17:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.869+0000 7f72dafff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d4116b20 con 0x7f72d4107600 2023-09-10T01:17:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.869+0000 7f72d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410edd0 0x7f72d411d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.869+0000 7f72d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410edd0 0x7f72d411d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40186/0 (socket says 172.21.15.3:40186) 2023-09-10T01:17:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.869+0000 7f72d3fff700 1 -- 172.21.15.3:0/1786217372 learned_addr learned my addr 172.21.15.3:0/1786217372 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.870+0000 7f72d3fff700 1 -- 172.21.15.3:0/1786217372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72d4107600 msgr2=0x7f72d411d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.870+0000 7f72d3fff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72d4107600 0x7f72d411d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.870+0000 7f72d3fff700 1 -- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72d410f1b0 msgr2=0x7f72d41160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.870+0000 7f72d3fff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72d410f1b0 0x7f72d41160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.870+0000 7f72d3fff700 1 -- 172.21.15.3:0/1786217372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc009d00 con 0x7f72d410edd0 2023-09-10T01:17:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.871+0000 7f72d3fff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410edd0 0x7f72d411d5e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c890 tx=0x7f72c000cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.871+0000 7f72d1ffb700 1 -- 172.21.15.3:0/1786217372 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0007b00 con 0x7f72d410edd0 2023-09-10T01:17:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.871+0000 7f72d1ffb700 1 -- 172.21.15.3:0/1786217372 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72c000ce90 con 0x7f72d410edd0 2023-09-10T01:17:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.871+0000 7f72d1ffb700 1 -- 172.21.15.3:0/1786217372 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0005450 con 0x7f72d410edd0 2023-09-10T01:17:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.871+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d4116e00 con 0x7f72d410edd0 2023-09-10T01:17:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.871+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d411ae30 con 0x7f72d410edd0 2023-09-10T01:17:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.878+0000 7f72ee96eb80 1 Processor -- start 2023-09-10T01:17:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.879+0000 7f72d1ffb700 1 -- 172.21.15.3:0/1786217372 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f72c00055d0 con 0x7f72d410edd0 2023-09-10T01:17:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.879+0000 7f72ee96eb80 1 -- start start 2023-09-10T01:17:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.880+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93e710 0x5592cf93eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.880+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf948900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.880+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf948e40 0x5592cf94b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.880+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf897730 con 0x5592cf948e40 2023-09-10T01:17:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.880+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf895050 con 0x5592cf93e710 2023-09-10T01:17:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.880+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf890940 con 0x5592cf93f030 2023-09-10T01:17:02.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.881+0000 7f72d8d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf948900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.881+0000 7f72d3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf948e40 0x5592cf94b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.881+0000 7f72d959c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93e710 0x5592cf93eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.881+0000 7f72d8d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf948900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45012/0 (socket says 172.21.15.3:45012) 2023-09-10T01:17:02.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.881+0000 7f72d8d9b700 1 -- 172.21.15.3:0/2677235008 learned_addr learned my addr 172.21.15.3:0/2677235008 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.881+0000 7f72d1ffb700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f72c40a0870 0x7f72c40a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.881+0000 7f72d1ffb700 1 -- 172.21.15.3:0/1786217372 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f72c00cbf40 con 0x7f72d410edd0 2023-09-10T01:17:02.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d959c700 1 -- 172.21.15.3:0/2677235008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 msgr2=0x5592cf948900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d959c700 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf948900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d959c700 1 -- 172.21.15.3:0/2677235008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf948e40 msgr2=0x5592cf94b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d959c700 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf948e40 0x5592cf94b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d959c700 1 -- 172.21.15.3:0/2677235008 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592cf6bcde0 con 0x5592cf93e710 2023-09-10T01:17:02.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf948900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d959c700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f72c40a0870 0x7f72c40a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d3fff700 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf948e40 0x5592cf94b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:17:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.882+0000 7f72d959c700 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93e710 0x5592cf93eaf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00b070 tx=0x7f72cc00cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da70dec6e6a3ab14 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.883+0000 7f72beffd700 1 -- 172.21.15.3:0/2677235008 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc003c60 con 0x5592cf93e710 2023-09-10T01:17:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.883+0000 7f72beffd700 1 -- 172.21.15.3:0/2677235008 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72cc01e940 con 0x5592cf93e710 2023-09-10T01:17:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.883+0000 7f72beffd700 1 -- 172.21.15.3:0/2677235008 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc01c8f0 con 0x5592cf93e710 2023-09-10T01:17:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.884+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2677235008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93e710 msgr2=0x5592cf93eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.884+0000 7f72d959c700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f72c40a0870 0x7f72c40a2d30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f72cc027000 tx=0x7f72cc02a000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.884+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93e710 0x5592cf93eaf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00b070 tx=0x7f72cc00cf10 comp rx=0 tx=0).stop 2023-09-10T01:17:02.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.885+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2677235008 shutdown_connections 2023-09-10T01:17:02.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.885+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf948e40 0x5592cf94b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.885+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93e710 0x5592cf93eaf0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.885+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2677235008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf948900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.885+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2677235008 >> 172.21.15.3:0/2677235008 conn(0x5592cf8b3090 msgr2=0x5592cf939bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.885+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2677235008 shutdown_connections 2023-09-10T01:17:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.885+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2677235008 wait complete. 2023-09-10T01:17:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.886+0000 7f72ee96eb80 1 -- start start 2023-09-10T01:17:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.886+0000 7f72ee96eb80 1 Processor -- start 2023-09-10T01:17:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf948840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf945ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 0x5592cf9460d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf82eb70 con 0x5592cf93e710 2023-09-10T01:17:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf82e870 con 0x5592cf948e40 2023-09-10T01:17:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf712920 con 0x5592cf93f030 2023-09-10T01:17:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72ee96eb80 10 client.0 _ll_register_callbacks cb 0x5592cf778a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72d3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf945ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72d8d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf948840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72d3fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf945ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45014/0 (socket says 172.21.15.3:45014) 2023-09-10T01:17:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.887+0000 7f72d3fff700 1 -- 172.21.15.3:0/2696400102 learned_addr learned my addr 172.21.15.3:0/2696400102 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.888+0000 7f72d8d9b700 1 -- 172.21.15.3:0/2696400102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 msgr2=0x5592cf945ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.888+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf945ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.888+0000 7f72d8d9b700 1 -- 172.21.15.3:0/2696400102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 msgr2=0x5592cf9460d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.888+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 0x5592cf9460d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.888+0000 7f72d8d9b700 1 -- 172.21.15.3:0/2696400102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c80094d0 con 0x5592cf93e710 2023-09-10T01:17:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.888+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf948840 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x5592cf934280 tx=0x7f72c800e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eaac0f7be1c1764 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.888+0000 7f72b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.891+0000 7f72b3fff700 1 -- 172.21.15.3:0/2696400102 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c8013470 con 0x5592cf93e710 2023-09-10T01:17:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.891+0000 7f72b3fff700 1 -- 172.21.15.3:0/2696400102 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f72c80135d0 con 0x5592cf93e710 2023-09-10T01:17:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.892+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592cf7e84f0 con 0x5592cf93e710 2023-09-10T01:17:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.893+0000 7f72b3fff700 1 -- 172.21.15.3:0/2696400102 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72c8013790 con 0x5592cf93e710 2023-09-10T01:17:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.893+0000 7f72ee96eb80 10 client.18702 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.893+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7f72d410b890 con 0x5592cf93e710 2023-09-10T01:17:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.893+0000 7f72b3fff700 1 -- 172.21.15.3:0/2696400102 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5592cf7e84f0 con 0x5592cf93e710 2023-09-10T01:17:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.893+0000 7f72b3fff700 1 -- 172.21.15.3:0/2696400102 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f72c8011070 con 0x5592cf93e710 2023-09-10T01:17:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.894+0000 7f72ee96eb80 10 client.18702 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.894+0000 7f72ee96eb80 10 client.18702 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:17:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.894+0000 7f72ee96eb80 20 client.18702 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:02.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.894+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9589d0 0x5592cf95adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.894+0000 7f72ee96eb80 4 client.18702 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.894+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592cf8a58a0 con 0x5592cf9589d0 2023-09-10T01:17:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.896+0000 7f72d3fff700 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9589d0 0x5592cf95adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.911+0000 7f72d3fff700 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9589d0 0x5592cf95adf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2f4fa5d16f1c13b server_cookie=1ec3848e71599de6 in_seq=0 out_seq=0 2023-09-10T01:17:02.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.911+0000 7f72b3fff700 10 client.18702 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:02.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.917+0000 7f72b3fff700 1 -- 172.21.15.3:0/2696400102 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5592cf8a58a0 con 0x5592cf9589d0 2023-09-10T01:17:02.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.917+0000 7f72b3fff700 10 client.18702 handle_command_reply: tid=0 2023-09-10T01:17:02.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.918+0000 7f72ee96eb80 1 client.18702 shutdown 2023-09-10T01:17:02.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.918+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 msgr2=0x5592cf948840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.918+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf948840 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x5592cf934280 tx=0x7f72c800e730 comp rx=0 tx=0).stop 2023-09-10T01:17:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.919+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 shutdown_connections 2023-09-10T01:17:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.919+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 0x5592cf9460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.919+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf948840 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.919+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9589d0 0x5592cf95adf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.919+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/2696400102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93f030 0x5592cf945ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.919+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 >> 172.21.15.3:0/2696400102 conn(0x5592cf8b3090 msgr2=0x5592cf939780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.919+0000 7f72b3fff700 0 client.18702 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:02.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.920+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 shutdown_connections 2023-09-10T01:17:02.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.921+0000 7f72ee96eb80 1 -- 172.21.15.3:0/2696400102 wait complete. 2023-09-10T01:17:02.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.924+0000 7f72ee96eb80 20 client.18702 trim_cache size 0 max 16384 2023-09-10T01:17:02.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.953+0000 7f72ee96eb80 1 Processor -- start 2023-09-10T01:17:02.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.953+0000 7f72ee96eb80 1 -- start start 2023-09-10T01:17:02.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf94f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93f030 0x5592cf954140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf948e40 0x5592cf948370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf93fa60 con 0x5592cf93e710 2023-09-10T01:17:02.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf946e40 con 0x5592cf93f030 2023-09-10T01:17:02.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf946750 con 0x5592cf948e40 2023-09-10T01:17:02.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93f030 0x5592cf954140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93f030 0x5592cf954140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40220/0 (socket says 172.21.15.3:40220) 2023-09-10T01:17:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.954+0000 7f72d3fff700 1 -- 172.21.15.3:0/1656067647 learned_addr learned my addr 172.21.15.3:0/1656067647 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d959c700 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf948e40 0x5592cf948370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf94f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d3fff700 1 -- 172.21.15.3:0/1656067647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf948e40 msgr2=0x5592cf948370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d3fff700 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf948e40 0x5592cf948370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d3fff700 1 -- 172.21.15.3:0/1656067647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 msgr2=0x5592cf94f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d3fff700 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf94f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d3fff700 1 -- 172.21.15.3:0/1656067647 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592cf6a5c60 con 0x5592cf93f030 2023-09-10T01:17:02.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.955+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf94f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:02.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.956+0000 7f72d3fff700 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93f030 0x5592cf954140 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f72c00954a0 tx=0x7f72c0007d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=461d1778276c7f34 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.956+0000 7f72bf7fe700 1 -- 172.21.15.3:0/1656067647 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0098fa0 con 0x5592cf93f030 2023-09-10T01:17:02.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.956+0000 7f72bf7fe700 1 -- 172.21.15.3:0/1656067647 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72c009cdd0 con 0x5592cf93f030 2023-09-10T01:17:02.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.956+0000 7f72bf7fe700 1 -- 172.21.15.3:0/1656067647 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0099460 con 0x5592cf93f030 2023-09-10T01:17:02.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.957+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1656067647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93f030 msgr2=0x5592cf954140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.957+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93f030 0x5592cf954140 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f72c00954a0 tx=0x7f72c0007d70 comp rx=0 tx=0).stop 2023-09-10T01:17:02.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.957+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1656067647 shutdown_connections 2023-09-10T01:17:02.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.957+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf948e40 0x5592cf948370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.957+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf93e710 0x5592cf94f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.957+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1656067647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf93f030 0x5592cf954140 secure :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f72c00954a0 tx=0x7f72c0007d70 comp rx=0 tx=0).stop 2023-09-10T01:17:02.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.957+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1656067647 >> 172.21.15.3:0/1656067647 conn(0x5592cf8b3090 msgr2=0x5592cf954b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.961+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1656067647 shutdown_connections 2023-09-10T01:17:02.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.962+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1656067647 wait complete. 2023-09-10T01:17:02.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.962+0000 7f72ee96eb80 1 -- start start 2023-09-10T01:17:02.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.963+0000 7f72ee96eb80 1 Processor -- start 2023-09-10T01:17:02.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.963+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93e710 0x5592cf94dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 0x5592cf94e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72ee96eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf94e750 0x5592cf9ac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf94fc30 con 0x5592cf94e750 2023-09-10T01:17:02.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf94fd70 con 0x5592cf948e40 2023-09-10T01:17:02.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72ee96eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cf9acd50 con 0x5592cf93e710 2023-09-10T01:17:02.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72ee96eb80 10 client.0 _ll_register_callbacks cb 0x7f72c8009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:02.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72d8d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93e710 0x5592cf94dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72d8d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93e710 0x5592cf94dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45030/0 (socket says 172.21.15.3:45030) 2023-09-10T01:17:02.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72d8d9b700 1 -- 172.21.15.3:0/1549172338 learned_addr learned my addr 172.21.15.3:0/1549172338 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:02.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.964+0000 7f72d959c700 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf94e750 0x5592cf9ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.965+0000 7f72d959c700 1 -- 172.21.15.3:0/1549172338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93e710 msgr2=0x5592cf94dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:02.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.965+0000 7f72d959c700 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93e710 0x5592cf94dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.965+0000 7f72d959c700 1 -- 172.21.15.3:0/1549172338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 msgr2=0x5592cf94e210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:02.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.965+0000 7f72d959c700 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 0x5592cf94e210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.965+0000 7f72d959c700 1 -- 172.21.15.3:0/1549172338 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d411ae30 con 0x5592cf94e750 2023-09-10T01:17:02.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.965+0000 7f72d959c700 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf94e750 0x5592cf9ac810 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00b0a0 tx=0x7f72cc01cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be842abf34172a78 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:02.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.965+0000 7f72b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:02.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.966+0000 7f72b3fff700 1 -- 172.21.15.3:0/1549172338 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc00e030 con 0x5592cf94e750 2023-09-10T01:17:02.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.966+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72cc01e940 con 0x5592cf94e750 2023-09-10T01:17:02.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.966+0000 7f72b3fff700 1 -- 172.21.15.3:0/1549172338 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f72cc00f070 con 0x5592cf94e750 2023-09-10T01:17:02.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.969+0000 7f72b3fff700 1 -- 172.21.15.3:0/1549172338 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72cc0090d0 con 0x5592cf94e750 2023-09-10T01:17:02.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.969+0000 7f72ee96eb80 10 client.18705 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:02.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.969+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5592cf6e3250 con 0x5592cf94e750 2023-09-10T01:17:02.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72b3fff700 1 -- 172.21.15.3:0/1549172338 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72cc01e940 con 0x5592cf94e750 2023-09-10T01:17:02.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72b3fff700 1 -- 172.21.15.3:0/1549172338 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f72cc03c070 con 0x5592cf94e750 2023-09-10T01:17:02.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72ee96eb80 10 client.18705 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:02.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72ee96eb80 10 client.18705 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:17:02.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72ee96eb80 20 client.18705 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:02.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9ae040 0x5592cf9b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:02.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72ee96eb80 4 client.18705 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:17:02.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.970+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5592cf889920 con 0x5592cf9ae040 2023-09-10T01:17:02.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.974+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9ae040 0x5592cf9b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:02.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.975+0000 7f72d8d9b700 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9ae040 0x5592cf9b0460 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32a3749a47eb7b67 server_cookie=2a4339b39df63709 in_seq=0 out_seq=0 2023-09-10T01:17:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.975+0000 7f72b3fff700 10 client.18705 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:02.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.979+0000 7f72b3fff700 1 -- 172.21.15.3:0/1549172338 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5592cf889920 con 0x5592cf9ae040 2023-09-10T01:17:02.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.980+0000 7f72b3fff700 10 client.18705 handle_command_reply: tid=0 2023-09-10T01:17:02.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.980+0000 7f72ee96eb80 1 client.18705 shutdown 2023-09-10T01:17:02.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.987+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf94e750 msgr2=0x5592cf9ac810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:02.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.987+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf94e750 0x5592cf9ac810 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00b0a0 tx=0x7f72cc01cb10 comp rx=0 tx=0).stop 2023-09-10T01:17:02.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.992+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 shutdown_connections 2023-09-10T01:17:02.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.992+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5592cf9ae040 0x5592cf9b0460 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.992+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5592cf93e710 0x5592cf94dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.992+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592cf94e750 0x5592cf9ac810 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.992+0000 7f72ee96eb80 1 --2- 172.21.15.3:0/1549172338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5592cf948e40 0x5592cf94e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.992+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 >> 172.21.15.3:0/1549172338 conn(0x5592cf8b3090 msgr2=0x5592cf954140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:02.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.993+0000 7f72b3fff700 0 client.18705 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:02.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.993+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 shutdown_connections 2023-09-10T01:17:02.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:02.994+0000 7f72ee96eb80 1 -- 172.21.15.3:0/1549172338 wait complete. 2023-09-10T01:17:03.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.000+0000 7f72ee96eb80 20 client.18705 trim_cache size 0 max 16384 2023-09-10T01:17:03.001 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:17:03.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.001+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f72c40a0870 msgr2=0x7f72c40a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.001+0000 7f72dafff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f72c40a0870 0x7f72c40a2d30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f72cc027000 tx=0x7f72cc02a000 comp rx=0 tx=0).stop 2023-09-10T01:17:03.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.001+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410edd0 msgr2=0x7f72d411d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.001+0000 7f72dafff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410edd0 0x7f72d411d5e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c890 tx=0x7f72c000cc50 comp rx=0 tx=0).stop 2023-09-10T01:17:03.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.009+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 shutdown_connections 2023-09-10T01:17:03.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.009+0000 7f72dafff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f72c40a0870 0x7f72c40a2d30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.009+0000 7f72dafff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72d4107600 0x7f72d411d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.009+0000 7f72dafff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72d410f1b0 0x7f72d41160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.009+0000 7f72dafff700 1 --2- 172.21.15.3:0/1786217372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72d410edd0 0x7f72d411d5e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.009+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 >> 172.21.15.3:0/1786217372 conn(0x7f72d4071dd0 msgr2=0x7f72d4109490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.014+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 shutdown_connections 2023-09-10T01:17:03.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.014+0000 7f72dafff700 1 -- 172.21.15.3:0/1786217372 wait complete. 2023-09-10T01:17:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.034+0000 7f87cabff700 1 -- 172.21.15.3:0/4008805263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87c410ed00 msgr2=0x7f87c410cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.034+0000 7f87cabff700 1 --2- 172.21.15.3:0/4008805263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87c410ed00 0x7f87c410cac0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00b0a0 tx=0x7f87bc00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.035+0000 7f87cabff700 1 -- 172.21.15.3:0/4008805263 shutdown_connections 2023-09-10T01:17:03.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.035+0000 7f87cabff700 1 --2- 172.21.15.3:0/4008805263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87c4107600 0x7f87c4107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.035+0000 7f87cabff700 1 --2- 172.21.15.3:0/4008805263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87c410ed00 0x7f87c410cac0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.035+0000 7f87cabff700 1 --2- 172.21.15.3:0/4008805263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 0x7f87c410d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.035+0000 7f87cabff700 1 -- 172.21.15.3:0/4008805263 >> 172.21.15.3:0/4008805263 conn(0x7f87c406c6c0 msgr2=0x7f87c406cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.037+0000 7f87cabff700 1 -- 172.21.15.3:0/4008805263 shutdown_connections 2023-09-10T01:17:03.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.037+0000 7f87cabff700 1 -- 172.21.15.3:0/4008805263 wait complete. 2023-09-10T01:17:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.039+0000 7f87cabff700 1 Processor -- start 2023-09-10T01:17:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.039+0000 7f87cabff700 1 -- start start 2023-09-10T01:17:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.040+0000 7f87cabff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87c4107600 0x7f87c41ab570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.040+0000 7f87cabff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87c410ed00 0x7f87c41abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.040+0000 7f87cabff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 0x7f87c41a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.040+0000 7f87cabff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c41a5c50 con 0x7f87c4107600 2023-09-10T01:17:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.040+0000 7f87cabff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c41a5dc0 con 0x7f87c410f170 2023-09-10T01:17:03.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.040+0000 7f87cabff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c41a5f80 con 0x7f87c410ed00 2023-09-10T01:17:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 0x7f87c41a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 0x7f87c41a55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40230/0 (socket says 172.21.15.3:40230) 2023-09-10T01:17:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 -- 172.21.15.3:0/2361037907 learned_addr learned my addr 172.21.15.3:0/2361037907 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c3fff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87c410ed00 0x7f87c41abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 -- 172.21.15.3:0/2361037907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87c410ed00 msgr2=0x7f87c41abab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87c410ed00 0x7f87c41abab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 -- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87c4107600 msgr2=0x7f87c41ab570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87c4107600 0x7f87c41ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.041+0000 7f87c919c700 1 -- 172.21.15.3:0/2361037907 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc009d00 con 0x7f87c410f170 2023-09-10T01:17:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.042+0000 7f87c919c700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 0x7f87c41a55e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b4a0 tx=0x7f87b400b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.042+0000 7f87c1ffb700 1 -- 172.21.15.3:0/2361037907 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4011830 con 0x7f87c410f170 2023-09-10T01:17:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.042+0000 7f87c1ffb700 1 -- 172.21.15.3:0/2361037907 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87b4011e50 con 0x7f87c410f170 2023-09-10T01:17:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.042+0000 7f87c1ffb700 1 -- 172.21.15.3:0/2361037907 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4010e40 con 0x7f87c410f170 2023-09-10T01:17:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.042+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c41a6260 con 0x7f87c410f170 2023-09-10T01:17:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.043+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c41b2560 con 0x7f87c410f170 2023-09-10T01:17:03.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.048+0000 7f87c1ffb700 1 -- 172.21.15.3:0/2361037907 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f87b4011990 con 0x7f87c410f170 2023-09-10T01:17:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.052+0000 7f87de569b80 1 Processor -- start 2023-09-10T01:17:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87de569b80 1 -- start start 2023-09-10T01:17:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87c1ffb700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f87ac0a08b0 0x7f87ac0a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87de569b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93cce01a0 0x55d93cce0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87de569b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93ccda780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87de569b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccdacc0 0x55d93cce6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87de569b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cc336d0 con 0x55d93ccdacc0 2023-09-10T01:17:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87de569b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cc30ff0 con 0x55d93ccda300 2023-09-10T01:17:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.053+0000 7f87de569b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cc2c8e0 con 0x55d93cce01a0 2023-09-10T01:17:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c1ffb700 1 -- 172.21.15.3:0/2361037907 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f87b40cbb20 con 0x7f87c410f170 2023-09-10T01:17:03.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c3fff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f87ac0a08b0 0x7f87ac0a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c919c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccdacc0 0x55d93cce6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c899b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93cce01a0 0x55d93cce0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c899b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93cce01a0 0x55d93cce0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45060/0 (socket says 172.21.15.3:45060) 2023-09-10T01:17:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c919c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccdacc0 0x55d93cce6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56880/0 (socket says 172.21.15.3:56880) 2023-09-10T01:17:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c899b700 1 -- 172.21.15.3:0/3928917036 learned_addr learned my addr 172.21.15.3:0/3928917036 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.054+0000 7f87c3fff700 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93ccda780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.055+0000 7f87c899b700 1 -- 172.21.15.3:0/3928917036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 msgr2=0x55d93ccda780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.055+0000 7f87c899b700 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93ccda780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.055+0000 7f87c899b700 1 -- 172.21.15.3:0/3928917036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccdacc0 msgr2=0x55d93cce6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.055+0000 7f87c899b700 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccdacc0 0x55d93cce6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.055+0000 7f87c899b700 1 -- 172.21.15.3:0/3928917036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d93ca58d80 con 0x55d93cce01a0 2023-09-10T01:17:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.056+0000 7f87c899b700 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93cce01a0 0x55d93cce0580 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009c20 tx=0x7f87bc009c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a02ab08ca4b19f98 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.056+0000 7f87aaffd700 1 -- 172.21.15.3:0/3928917036 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc00e040 con 0x55d93cce01a0 2023-09-10T01:17:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.056+0000 7f87aaffd700 1 -- 172.21.15.3:0/3928917036 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87bc0093d0 con 0x55d93cce01a0 2023-09-10T01:17:03.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.056+0000 7f87aaffd700 1 -- 172.21.15.3:0/3928917036 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc005320 con 0x55d93cce01a0 2023-09-10T01:17:03.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.056+0000 7f87c3fff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f87ac0a08b0 0x7f87ac0a2d70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55d93ccd5060 tx=0x7f87b800b830 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:17:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.057+0000 7f87de569b80 1 -- 172.21.15.3:0/3928917036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93cce01a0 msgr2=0x55d93cce0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.057+0000 7f87de569b80 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93cce01a0 0x55d93cce0580 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009c20 tx=0x7f87bc009c50 comp rx=0 tx=0).stop 2023-09-10T01:17:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.061+0000 7f87de569b80 1 -- 172.21.15.3:0/3928917036 shutdown_connections 2023-09-10T01:17:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.061+0000 7f87de569b80 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93cce01a0 0x55d93cce0580 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.061+0000 7f87de569b80 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccdacc0 0x55d93cce6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.061+0000 7f87de569b80 1 --2- 172.21.15.3:0/3928917036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93ccda780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.061+0000 7f87de569b80 1 -- 172.21.15.3:0/3928917036 >> 172.21.15.3:0/3928917036 conn(0x55d93cc4f030 msgr2=0x55d93ccd5560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.062+0000 7f87de569b80 1 -- 172.21.15.3:0/3928917036 shutdown_connections 2023-09-10T01:17:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.064+0000 7f87de569b80 1 -- 172.21.15.3:0/3928917036 wait complete. 2023-09-10T01:17:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.065+0000 7f87de569b80 1 -- start start 2023-09-10T01:17:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.065+0000 7f87de569b80 1 Processor -- start 2023-09-10T01:17:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.066+0000 7f87de569b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93cce6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.066+0000 7f87de569b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.066+0000 7f87de569b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.067+0000 7f87de569b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cbcab10 con 0x55d93cce01a0 2023-09-10T01:17:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.067+0000 7f87de569b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cbca810 con 0x55d93ccda300 2023-09-10T01:17:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.067+0000 7f87de569b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93caae8c0 con 0x55d93ccdacc0 2023-09-10T01:17:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.067+0000 7f87de569b80 10 client.0 _ll_register_callbacks cb 0x55d93cb14a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c919c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c899b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93cce6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c919c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56892/0 (socket says 172.21.15.3:56892) 2023-09-10T01:17:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c919c700 1 -- 172.21.15.3:0/3286197865 learned_addr learned my addr 172.21.15.3:0/3286197865 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c899b700 1 -- 172.21.15.3:0/3286197865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 msgr2=0x55d93cce3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c899b700 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c899b700 1 -- 172.21.15.3:0/3286197865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 msgr2=0x55d93cce4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c899b700 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.068+0000 7f87c899b700 1 -- 172.21.15.3:0/3286197865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c410a7b0 con 0x55d93ccda300 2023-09-10T01:17:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.069+0000 7f87c899b700 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93cce6dc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009bc0 tx=0x7f87bc0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9029e8429df98a24 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.069+0000 7f87a3fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.069+0000 7f87a3fff700 1 -- 172.21.15.3:0/3286197865 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc00e040 con 0x55d93ccda300 2023-09-10T01:17:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.069+0000 7f87a3fff700 1 -- 172.21.15.3:0/3286197865 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f87bc0050c0 con 0x55d93ccda300 2023-09-10T01:17:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.072+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d93cb84490 con 0x55d93ccda300 2023-09-10T01:17:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.076+0000 7f87a3fff700 1 -- 172.21.15.3:0/3286197865 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87bc004ea0 con 0x55d93ccda300 2023-09-10T01:17:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.077+0000 7f87de569b80 10 client.28820 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.077+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55d93ca41c00 con 0x55d93ccda300 2023-09-10T01:17:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87a3fff700 1 -- 172.21.15.3:0/3286197865 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d93cb84490 con 0x55d93ccda300 2023-09-10T01:17:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87a3fff700 1 -- 172.21.15.3:0/3286197865 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f87bc02e050 con 0x55d93ccda300 2023-09-10T01:17:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87de569b80 10 client.28820 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87de569b80 10 client.28820 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87de569b80 20 client.28820 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87de569b80 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93ccf3fd0 0x55d93ccf63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87de569b80 4 client.28820 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.080+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d93cc41840 con 0x55d93ccf3fd0 2023-09-10T01:17:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.081+0000 7f87c3fff700 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93ccf3fd0 0x55d93ccf63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.089+0000 7f87c3fff700 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93ccf3fd0 0x55d93ccf63f0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f637ca2bb8b3b08 server_cookie=fec38467b14a9dd in_seq=0 out_seq=0 2023-09-10T01:17:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.089+0000 7f87a3fff700 10 client.28820 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.095+0000 7f87a3fff700 1 -- 172.21.15.3:0/3286197865 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d93cc41840 con 0x55d93ccf3fd0 2023-09-10T01:17:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.095+0000 7f87a3fff700 10 client.28820 handle_command_reply: tid=0 2023-09-10T01:17:03.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.095+0000 7f87de569b80 1 client.28820 shutdown 2023-09-10T01:17:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 msgr2=0x55d93cce6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93cce6dc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009bc0 tx=0x7f87bc0091d0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 shutdown_connections 2023-09-10T01:17:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93ccf3fd0 0x55d93ccf63f0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93ccda300 0x55d93cce6dc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 --2- 172.21.15.3:0/3286197865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.096+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 >> 172.21.15.3:0/3286197865 conn(0x55d93cc4f030 msgr2=0x55d93ccd47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.097+0000 7f87a3fff700 0 client.28820 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.098+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 shutdown_connections 2023-09-10T01:17:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.098+0000 7f87de569b80 1 -- 172.21.15.3:0/3286197865 wait complete. 2023-09-10T01:17:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.102+0000 7f87de569b80 20 client.28820 trim_cache size 0 max 16384 2023-09-10T01:17:03.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.132+0000 7f87de569b80 1 Processor -- start 2023-09-10T01:17:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.135+0000 7f87de569b80 1 -- start start 2023-09-10T01:17:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.135+0000 7f87de569b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccda300 0x55d93cceb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.135+0000 7f87de569b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.135+0000 7f87de569b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce01a0 0x55d93ccdf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.135+0000 7f87de569b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cce0580 con 0x55d93ccda300 2023-09-10T01:17:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.135+0000 7f87de569b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cceba60 con 0x55d93cce01a0 2023-09-10T01:17:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.135+0000 7f87de569b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cce2530 con 0x55d93ccdacc0 2023-09-10T01:17:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.136+0000 7f87c899b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccda300 0x55d93cceb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.136+0000 7f87c899b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccda300 0x55d93cceb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56898/0 (socket says 172.21.15.3:56898) 2023-09-10T01:17:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.136+0000 7f87c899b700 1 -- 172.21.15.3:0/2295111691 learned_addr learned my addr 172.21.15.3:0/2295111691 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.136+0000 7f87c3fff700 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.137+0000 7f87c919c700 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce01a0 0x55d93ccdf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.137+0000 7f87c899b700 1 -- 172.21.15.3:0/2295111691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 msgr2=0x55d93cce68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.137+0000 7f87c899b700 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.137+0000 7f87c899b700 1 -- 172.21.15.3:0/2295111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce01a0 msgr2=0x55d93ccdf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.137+0000 7f87c899b700 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce01a0 0x55d93ccdf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.137+0000 7f87c899b700 1 -- 172.21.15.3:0/2295111691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d93ca7f1f0 con 0x55d93ccda300 2023-09-10T01:17:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.138+0000 7f87c899b700 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccda300 0x55d93cceb520 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f87bc01d510 tx=0x7f87bc000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bdf5de85ba3976 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.143+0000 7f87ab7fe700 1 -- 172.21.15.3:0/2295111691 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc00e070 con 0x55d93ccda300 2023-09-10T01:17:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.143+0000 7f87ab7fe700 1 -- 172.21.15.3:0/2295111691 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87bc004ea0 con 0x55d93ccda300 2023-09-10T01:17:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.143+0000 7f87ab7fe700 1 -- 172.21.15.3:0/2295111691 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0050c0 con 0x55d93ccda300 2023-09-10T01:17:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.144+0000 7f87de569b80 1 -- 172.21.15.3:0/2295111691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccda300 msgr2=0x55d93cceb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.144+0000 7f87de569b80 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccda300 0x55d93cceb520 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f87bc01d510 tx=0x7f87bc000f40 comp rx=0 tx=0).stop 2023-09-10T01:17:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.145+0000 7f87de569b80 1 -- 172.21.15.3:0/2295111691 shutdown_connections 2023-09-10T01:17:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.145+0000 7f87de569b80 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce01a0 0x55d93ccdf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.145+0000 7f87de569b80 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93ccda300 0x55d93cceb520 secure :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f87bc01d510 tx=0x7f87bc000f40 comp rx=0 tx=0).stop 2023-09-10T01:17:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.145+0000 7f87de569b80 1 --2- 172.21.15.3:0/2295111691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.145+0000 7f87de569b80 1 -- 172.21.15.3:0/2295111691 >> 172.21.15.3:0/2295111691 conn(0x55d93cc4f030 msgr2=0x55d93ccd5090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.145+0000 7f87de569b80 1 -- 172.21.15.3:0/2295111691 shutdown_connections 2023-09-10T01:17:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.146+0000 7f87de569b80 1 -- 172.21.15.3:0/2295111691 wait complete. 2023-09-10T01:17:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.146+0000 7f87de569b80 1 -- start start 2023-09-10T01:17:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.147+0000 7f87de569b80 1 Processor -- start 2023-09-10T01:17:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.147+0000 7f87de569b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.147+0000 7f87de569b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.147+0000 7f87de569b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce9f90 0x55d93cd46e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.147+0000 7f87de569b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cceb930 con 0x55d93cce01a0 2023-09-10T01:17:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.147+0000 7f87de569b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93ccebaa0 con 0x55d93cce9f90 2023-09-10T01:17:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.147+0000 7f87de569b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93cd58370 con 0x55d93ccdacc0 2023-09-10T01:17:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.148+0000 7f87de569b80 10 client.0 _ll_register_callbacks cb 0x7f87bc009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.149+0000 7f87c919c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce9f90 0x55d93cd46e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.149+0000 7f87c919c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce9f90 0x55d93cd46e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40272/0 (socket says 172.21.15.3:40272) 2023-09-10T01:17:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.149+0000 7f87c919c700 1 -- 172.21.15.3:0/420024880 learned_addr learned my addr 172.21.15.3:0/420024880 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.149+0000 7f87c899b700 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.150+0000 7f87c919c700 1 -- 172.21.15.3:0/420024880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 msgr2=0x55d93cce9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.150+0000 7f87c919c700 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.150+0000 7f87c919c700 1 -- 172.21.15.3:0/420024880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 msgr2=0x55d93cce9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.150+0000 7f87c919c700 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.150+0000 7f87c919c700 1 -- 172.21.15.3:0/420024880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d93ca41c00 con 0x55d93cce9f90 2023-09-10T01:17:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.150+0000 7f87c919c700 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce9f90 0x55d93cd46e20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f87b400bbb0 tx=0x7f87b40994e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db3f1b1aab201830 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.156+0000 7f87a3fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.156+0000 7f87a3fff700 1 -- 172.21.15.3:0/420024880 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b40990f0 con 0x55d93cce9f90 2023-09-10T01:17:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.156+0000 7f87a3fff700 1 -- 172.21.15.3:0/420024880 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f87b4098180 con 0x55d93cce9f90 2023-09-10T01:17:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.157+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d93cc258c0 con 0x55d93cce9f90 2023-09-10T01:17:03.161 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24364 33.67 MB/sec warmup 44 sec latency 1578.692 ms 2023-09-10T01:17:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.162+0000 7f87a3fff700 1 -- 172.21.15.3:0/420024880 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87b4098340 con 0x55d93cce9f90 2023-09-10T01:17:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.162+0000 7f87de569b80 10 client.28826 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.162+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55d93ca57250 con 0x55d93cce9f90 2023-09-10T01:17:03.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87a3fff700 1 -- 172.21.15.3:0/420024880 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d93cc258c0 con 0x55d93cce9f90 2023-09-10T01:17:03.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87a3fff700 1 -- 172.21.15.3:0/420024880 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f87b40b80b0 con 0x55d93cce9f90 2023-09-10T01:17:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87de569b80 10 client.28826 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87de569b80 10 client.28826 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87de569b80 20 client.28826 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87de569b80 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93cd59b20 0x55d93cd5bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87de569b80 4 client.28826 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:17:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.171+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d93cbe6050 con 0x55d93cd59b20 2023-09-10T01:17:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.172+0000 7f87c899b700 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93cd59b20 0x55d93cd5bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.176+0000 7f87c899b700 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93cd59b20 0x55d93cd5bf40 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccfcb1250fb48673 server_cookie=5c4202327235df54 in_seq=0 out_seq=0 2023-09-10T01:17:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.176+0000 7f87a3fff700 10 client.28826 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.545+0000 7fe40e7ff700 1 -- 172.21.15.3:0/504088585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe40810f1b0 msgr2=0x7fe40810cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.545+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/504088585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe40810f1b0 0x7fe40810cfc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009990 tx=0x7fe3fc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.548+0000 7fe40e7ff700 1 -- 172.21.15.3:0/504088585 shutdown_connections 2023-09-10T01:17:03.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.548+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/504088585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe40810edd0 0x7fe40810ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.548+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/504088585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe408107600 0x7fe408107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.548+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/504088585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe40810f1b0 0x7fe40810cfc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.550 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.548+0000 7fe40e7ff700 1 -- 172.21.15.3:0/504088585 >> 172.21.15.3:0/504088585 conn(0x7fe408071dd0 msgr2=0x7fe40806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.560+0000 7fe40e7ff700 1 -- 172.21.15.3:0/504088585 shutdown_connections 2023-09-10T01:17:03.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.563+0000 7fe40e7ff700 1 -- 172.21.15.3:0/504088585 wait complete. 2023-09-10T01:17:03.565 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.564+0000 7fe40e7ff700 1 Processor -- start 2023-09-10T01:17:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.574+0000 7fe40e7ff700 1 -- start start 2023-09-10T01:17:03.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.574+0000 7fe40e7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe408107600 0x7fe40811ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.574+0000 7fe40e7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe40810edd0 0x7fe40811b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.574+0000 7fe40e7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe40810f1b0 0x7fe408115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.588 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.574+0000 7fe40e7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408116560 con 0x7fe408107600 2023-09-10T01:17:03.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.574+0000 7fe40e7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4081166d0 con 0x7fe40810edd0 2023-09-10T01:17:03.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.574+0000 7fe40e7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe408116890 con 0x7fe40810f1b0 2023-09-10T01:17:03.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.581+0000 7fe40cd9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe40810f1b0 0x7fe408115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.582+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe40810edd0 0x7fe40811b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.589 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.584+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe40810edd0 0x7fe40811b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40300/0 (socket says 172.21.15.3:40300) 2023-09-10T01:17:03.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.584+0000 7fe4077fe700 1 -- 172.21.15.3:0/821880918 learned_addr learned my addr 172.21.15.3:0/821880918 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.588+0000 7fe407fff700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe408107600 0x7fe40811ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.590+0000 7fe40cd9c700 1 -- 172.21.15.3:0/821880918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe40810edd0 msgr2=0x7fe40811b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.590+0000 7fe40cd9c700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe40810edd0 0x7fe40811b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.592 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.590+0000 7fe40cd9c700 1 -- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe408107600 msgr2=0x7fe40811ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.590+0000 7fe40cd9c700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe408107600 0x7fe40811ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.590+0000 7fe40cd9c700 1 -- 172.21.15.3:0/821880918 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc0095f0 con 0x7fe40810f1b0 2023-09-10T01:17:03.593 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.590+0000 7fe40cd9c700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe40810f1b0 0x7fe408115f90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe40000efc0 tx=0x7fe40000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.599+0000 7fe4057fa700 1 -- 172.21.15.3:0/821880918 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe400018070 con 0x7fe40810f1b0 2023-09-10T01:17:03.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.599+0000 7fe4057fa700 1 -- 172.21.15.3:0/821880918 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe40000f040 con 0x7fe40810f1b0 2023-09-10T01:17:03.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.599+0000 7fe4057fa700 1 -- 172.21.15.3:0/821880918 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40000ce90 con 0x7fe40810f1b0 2023-09-10T01:17:03.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.599+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe408116b10 con 0x7fe40810f1b0 2023-09-10T01:17:03.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.599+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe408119ce0 con 0x7fe40810f1b0 2023-09-10T01:17:03.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.608+0000 7fe4221eab80 1 Processor -- start 2023-09-10T01:17:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.614+0000 7fe4221eab80 1 -- start start 2023-09-10T01:17:03.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x5573471ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471cd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4221eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557347119730 con 0x5573471c0710 2023-09-10T01:17:03.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4221eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557347117050 con 0x5573471cae40 2023-09-10T01:17:03.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4221eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557347112940 con 0x5573471c1030 2023-09-10T01:17:03.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471c0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56928/0 (socket says 172.21.15.3:56928) 2023-09-10T01:17:03.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4077fe700 1 -- 172.21.15.3:0/3158283778 learned_addr learned my addr 172.21.15.3:0/3158283778 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.615+0000 7fe4077fe700 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.620+0000 7fe4057fa700 1 -- 172.21.15.3:0/821880918 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe400004ad0 con 0x7fe40810f1b0 2023-09-10T01:17:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.622+0000 7fe4057fa700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe3f00a09c0 0x7fe3f00a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.622+0000 7fe4057fa700 1 -- 172.21.15.3:0/821880918 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe4000cbcc0 con 0x7fe40810f1b0 2023-09-10T01:17:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.625+0000 7fe407fff700 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x5573471ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.625+0000 7fe4077fe700 1 -- 172.21.15.3:0/3158283778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 msgr2=0x5573471ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.625+0000 7fe4077fe700 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x5573471ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.625+0000 7fe4077fe700 1 -- 172.21.15.3:0/3158283778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 msgr2=0x5573471c0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.625+0000 7fe4077fe700 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471c0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.625+0000 7fe4077fe700 1 -- 172.21.15.3:0/3158283778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557346f3ede0 con 0x5573471cae40 2023-09-10T01:17:03.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.625+0000 7fe4077fe700 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471cd230 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc015000 tx=0x7fe3fc009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7328b6111f25a44 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.634+0000 7fe40cd9c700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe3f00a09c0 0x7fe3f00a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.635+0000 7fe40cd9c700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe3f00a09c0 0x7fe3f00a2e80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe400095820 tx=0x7fe400009040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.638+0000 7fe3ee7fc700 1 -- 172.21.15.3:0/3158283778 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc01d030 con 0x5573471cae40 2023-09-10T01:17:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.638+0000 7fe3ee7fc700 1 -- 172.21.15.3:0/3158283778 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3fc00be00 con 0x5573471cae40 2023-09-10T01:17:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.638+0000 7fe3ee7fc700 1 -- 172.21.15.3:0/3158283778 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc004d50 con 0x5573471cae40 2023-09-10T01:17:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.639+0000 7fe4221eab80 1 -- 172.21.15.3:0/3158283778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 msgr2=0x5573471cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.639+0000 7fe4221eab80 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471cd230 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc015000 tx=0x7fe3fc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.640+0000 7fe4221eab80 1 -- 172.21.15.3:0/3158283778 shutdown_connections 2023-09-10T01:17:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.640+0000 7fe4221eab80 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471cd230 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.640+0000 7fe4221eab80 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471c0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.640+0000 7fe4221eab80 1 --2- 172.21.15.3:0/3158283778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x5573471ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.640+0000 7fe4221eab80 1 -- 172.21.15.3:0/3158283778 >> 172.21.15.3:0/3158283778 conn(0x557347135090 msgr2=0x5573471bbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.640+0000 7fe4221eab80 1 -- 172.21.15.3:0/3158283778 shutdown_connections 2023-09-10T01:17:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.640+0000 7fe4221eab80 1 -- 172.21.15.3:0/3158283778 wait complete. 2023-09-10T01:17:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.641+0000 7fe4221eab80 1 -- start start 2023-09-10T01:17:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.642+0000 7fe4221eab80 1 Processor -- start 2023-09-10T01:17:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.643+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.643+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x5573471c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.643+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.643+0000 7fe4221eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573470b0b70 con 0x5573471c0710 2023-09-10T01:17:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.643+0000 7fe4221eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573470b0870 con 0x5573471cae40 2023-09-10T01:17:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.643+0000 7fe4221eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557346f94920 con 0x5573471c1030 2023-09-10T01:17:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.643+0000 7fe4221eab80 10 client.0 _ll_register_callbacks cb 0x557346ffaa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.645+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.645+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:40326/0 (socket says 172.21.15.3:40326) 2023-09-10T01:17:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.645+0000 7fe4077fe700 1 -- 172.21.15.3:0/2893472759 learned_addr learned my addr 172.21.15.3:0/2893472759 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.646+0000 7fe4077fe700 1 -- 172.21.15.3:0/2893472759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 msgr2=0x5573471c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.646+0000 7fe4077fe700 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x5573471c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.646+0000 7fe4077fe700 1 -- 172.21.15.3:0/2893472759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 msgr2=0x5573471ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.646+0000 7fe4077fe700 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.646+0000 7fe4077fe700 1 -- 172.21.15.3:0/2893472759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3fc020040 con 0x5573471cae40 2023-09-10T01:17:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.646+0000 7fe4077fe700 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471c7fc0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00b9f0 tx=0x7fe3fc009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85b4585aef5a6fff server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.647+0000 7fe3e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.647+0000 7fe3e77fe700 1 -- 172.21.15.3:0/2893472759 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc01d040 con 0x5573471cae40 2023-09-10T01:17:03.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.647+0000 7fe3e77fe700 1 -- 172.21.15.3:0/2893472759 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe3fc01e070 con 0x5573471cae40 2023-09-10T01:17:03.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.648+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55734706a4f0 con 0x5573471cae40 2023-09-10T01:17:03.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.649+0000 7fe3e77fe700 1 -- 172.21.15.3:0/2893472759 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3fc004e10 con 0x5573471cae40 2023-09-10T01:17:03.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.652+0000 7fe4221eab80 10 client.28838 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:03.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.652+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x557346f27c60 con 0x5573471cae40 2023-09-10T01:17:03.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.652+0000 7fe3e77fe700 1 -- 172.21.15.3:0/2893472759 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55734706a4f0 con 0x5573471cae40 2023-09-10T01:17:03.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.652+0000 7fe3e77fe700 1 -- 172.21.15.3:0/2893472759 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe3fc00fba0 con 0x5573471cae40 2023-09-10T01:17:03.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.653+0000 7fe4221eab80 10 client.28838 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:03.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.653+0000 7fe4221eab80 10 client.28838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:03.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.653+0000 7fe4221eab80 20 client.28838 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:03.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.653+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5573471da500 0x5573471dc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.653+0000 7fe4221eab80 4 client.28838 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:03.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.653+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573471278a0 con 0x5573471da500 2023-09-10T01:17:03.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.653+0000 7fe407fff700 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5573471da500 0x5573471dc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.658+0000 7fe407fff700 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5573471da500 0x5573471dc920 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4034e62da146985 server_cookie=85fb00962bdd4255 in_seq=0 out_seq=0 2023-09-10T01:17:03.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.659+0000 7fe3e77fe700 10 client.28838 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.679+0000 7f87a3fff700 1 -- 172.21.15.3:0/420024880 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d93cbe6050 con 0x55d93cd59b20 2023-09-10T01:17:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.679+0000 7f87a3fff700 10 client.28826 handle_command_reply: tid=0 2023-09-10T01:17:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.679+0000 7f87de569b80 1 client.28826 shutdown 2023-09-10T01:17:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.683+0000 7fe3e77fe700 1 -- 172.21.15.3:0/2893472759 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5573471278a0 con 0x5573471da500 2023-09-10T01:17:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.683+0000 7fe3e77fe700 10 client.28838 handle_command_reply: tid=0 2023-09-10T01:17:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.683+0000 7fe4221eab80 1 client.28838 shutdown 2023-09-10T01:17:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 msgr2=0x5573471c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471c7fc0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00b9f0 tx=0x7fe3fc009310 comp rx=0 tx=0).stop 2023-09-10T01:17:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce9f90 msgr2=0x55d93cd46e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce9f90 0x55d93cd46e20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f87b400bbb0 tx=0x7f87b40994e0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 shutdown_connections 2023-09-10T01:17:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55d93cd59b20 0x55d93cd5bf40 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d93ccdacc0 0x55d93cce9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d93cce9f90 0x55d93cd46e20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 --2- 172.21.15.3:0/420024880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d93cce01a0 0x55d93cce9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 >> 172.21.15.3:0/420024880 conn(0x55d93cc4f030 msgr2=0x55d93ccdf010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.686+0000 7f87a3fff700 0 client.28826 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.687+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 shutdown_connections 2023-09-10T01:17:03.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.687+0000 7f87de569b80 1 -- 172.21.15.3:0/420024880 wait complete. 2023-09-10T01:17:03.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.687+0000 7f87de569b80 20 client.28826 trim_cache size 0 max 16384 2023-09-10T01:17:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 shutdown_connections 2023-09-10T01:17:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5573471da500 0x5573471dc920 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471cae40 0x5573471c7fc0 secure :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00b9f0 tx=0x7fe3fc009310 comp rx=0 tx=0).stop 2023-09-10T01:17:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c0710 0x5573471ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2893472759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x5573471c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 >> 172.21.15.3:0/2893472759 conn(0x557347135090 msgr2=0x5573471bb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe3e77fe700 0 client.28838 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 shutdown_connections 2023-09-10T01:17:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.686+0000 7fe4221eab80 1 -- 172.21.15.3:0/2893472759 wait complete. 2023-09-10T01:17:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:17:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:17:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:17:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:17:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.688+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f87ac0a08b0 msgr2=0x7f87ac0a2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.688+0000 7f87cabff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f87ac0a08b0 0x7f87ac0a2d70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55d93ccd5060 tx=0x7f87b800b830 comp rx=0 tx=0).stop 2023-09-10T01:17:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.688+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 msgr2=0x7f87c41a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.688+0000 7f87cabff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 0x7f87c41a55e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b4a0 tx=0x7f87b400b7b0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.691+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 shutdown_connections 2023-09-10T01:17:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.691+0000 7f87cabff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f87ac0a08b0 0x7f87ac0a2d70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.691+0000 7f87cabff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87c410f170 0x7f87c41a55e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.695 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.691+0000 7f87cabff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87c4107600 0x7f87c41ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.691+0000 7f87cabff700 1 --2- 172.21.15.3:0/2361037907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87c410ed00 0x7f87c41abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.691+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 >> 172.21.15.3:0/2361037907 conn(0x7f87c406c6c0 msgr2=0x7f87c4070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.691+0000 7fe4221eab80 20 client.28838 trim_cache size 0 max 16384 2023-09-10T01:17:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.692+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 shutdown_connections 2023-09-10T01:17:03.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:03.692+0000 7f87cabff700 1 -- 172.21.15.3:0/2361037907 wait complete. 2023-09-10T01:17:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.719+0000 7fe4221eab80 1 Processor -- start 2023-09-10T01:17:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4221eab80 1 -- start start 2023-09-10T01:17:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c0710 0x5573471d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c1030 0x5573471c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x5573471c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4221eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573471c1910 con 0x5573471c1030 2023-09-10T01:17:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4221eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573471d1f70 con 0x5573471e1240 2023-09-10T01:17:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4221eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573471c1410 con 0x5573471c0710 2023-09-10T01:17:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c0710 0x5573471d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.720+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c0710 0x5573471d1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45120/0 (socket says 172.21.15.3:45120) 2023-09-10T01:17:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 -- 172.21.15.3:0/2736458580 learned_addr learned my addr 172.21.15.3:0/2736458580 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x5573471c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 -- 172.21.15.3:0/2736458580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 msgr2=0x5573471c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x5573471c3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 -- 172.21.15.3:0/2736458580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c1030 msgr2=0x5573471c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c1030 0x5573471c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 -- 172.21.15.3:0/2736458580 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573471d7290 con 0x5573471c0710 2023-09-10T01:17:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.721+0000 7fe4077fe700 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c0710 0x5573471d1a30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00ff30 tx=0x7fe3fc004930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95f32b54c7796aed server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.722+0000 7fe3eeffd700 1 -- 172.21.15.3:0/2736458580 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc000e80 con 0x5573471c0710 2023-09-10T01:17:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.723+0000 7fe3eeffd700 1 -- 172.21.15.3:0/2736458580 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3fc043070 con 0x5573471c0710 2023-09-10T01:17:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.723+0000 7fe3eeffd700 1 -- 172.21.15.3:0/2736458580 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc00e6b0 con 0x5573471c0710 2023-09-10T01:17:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.725+0000 7fe4221eab80 1 -- 172.21.15.3:0/2736458580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c0710 msgr2=0x5573471d1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.725+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c0710 0x5573471d1a30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00ff30 tx=0x7fe3fc004930 comp rx=0 tx=0).stop 2023-09-10T01:17:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.725+0000 7fe4221eab80 1 -- 172.21.15.3:0/2736458580 shutdown_connections 2023-09-10T01:17:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.725+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c0710 0x5573471d1a30 secure :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00ff30 tx=0x7fe3fc004930 comp rx=0 tx=0).stop 2023-09-10T01:17:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.725+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x5573471c3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.725+0000 7fe4221eab80 1 --2- 172.21.15.3:0/2736458580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5573471c1030 0x5573471c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.725+0000 7fe4221eab80 1 -- 172.21.15.3:0/2736458580 >> 172.21.15.3:0/2736458580 conn(0x557347135090 msgr2=0x5573471d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.731+0000 7fe4221eab80 1 -- 172.21.15.3:0/2736458580 shutdown_connections 2023-09-10T01:17:03.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.731+0000 7fe4221eab80 1 -- 172.21.15.3:0/2736458580 wait complete. 2023-09-10T01:17:03.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.732+0000 7fe4221eab80 1 -- start start 2023-09-10T01:17:03.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.735+0000 7fe4221eab80 1 Processor -- start 2023-09-10T01:17:03.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.738+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x557347226750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.738+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x557347226c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.738+0000 7fe4221eab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557347222280 0x557347222700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.738+0000 7fe4221eab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557347222c80 con 0x557347222280 2023-09-10T01:17:03.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.738+0000 7fe4221eab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x557347222dc0 con 0x5573471e1240 2023-09-10T01:17:03.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.738+0000 7fe4221eab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x557347222f50 con 0x5573471c1030 2023-09-10T01:17:03.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.738+0000 7fe4221eab80 10 client.0 _ll_register_callbacks cb 0x7fe4000097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:03.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.740+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x557347226750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.741+0000 7fe407fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x557347226c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x557347226750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:45126/0 (socket says 172.21.15.3:45126) 2023-09-10T01:17:03.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 -- 172.21.15.3:0/4181149048 learned_addr learned my addr 172.21.15.3:0/4181149048 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:03.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557347222280 0x557347222700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 -- 172.21.15.3:0/4181149048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 msgr2=0x557347226c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x557347226c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 -- 172.21.15.3:0/4181149048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557347222280 msgr2=0x557347222700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:03.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557347222280 0x557347222700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.742+0000 7fe4077fe700 1 -- 172.21.15.3:0/4181149048 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557346f27c60 con 0x5573471c1030 2023-09-10T01:17:03.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.743+0000 7fe4077fe700 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x557347226750 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0097e0 tx=0x7fe3fc01e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dbad7ae1462bf38 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:03.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.743+0000 7fe3e77fe700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:03.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.743+0000 7fe3e77fe700 1 -- 172.21.15.3:0/4181149048 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc01d030 con 0x5573471c1030 2023-09-10T01:17:03.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.743+0000 7fe3e77fe700 1 -- 172.21.15.3:0/4181149048 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9527+0+0 (secure 0 0 0) 0x7fe3fc00e5f0 con 0x5573471c1030 2023-09-10T01:17:03.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.743+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3fc00be00 con 0x5573471c1030 2023-09-10T01:17:03.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.744+0000 7fe3e77fe700 1 -- 172.21.15.3:0/4181149048 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3fc004ec0 con 0x5573471c1030 2023-09-10T01:17:03.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.745+0000 7fe4221eab80 10 client.28705 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:03.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.745+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x557346f65250 con 0x5573471c1030 2023-09-10T01:17:03.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe3e77fe700 1 -- 172.21.15.3:0/4181149048 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3fc051020 con 0x5573471c1030 2023-09-10T01:17:03.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe3e77fe700 1 -- 172.21.15.3:0/4181149048 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe3fc029790 con 0x5573471c1030 2023-09-10T01:17:03.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe4221eab80 10 client.28705 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:03.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe4221eab80 10 client.28705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:03.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe4221eab80 20 client.28705 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:03.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe4221eab80 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55734722f5f0 0x557347231a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:03.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe4221eab80 4 client.28705 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:17:03.750 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.747+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55734710b920 con 0x55734722f5f0 2023-09-10T01:17:03.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.753+0000 7fe407fff700 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55734722f5f0 0x557347231a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:03.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.755+0000 7fe407fff700 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55734722f5f0 0x557347231a10 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f933ff05a4aebd6 server_cookie=584c0cdb70916b1e in_seq=0 out_seq=0 2023-09-10T01:17:03.755 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.755+0000 7fe3e77fe700 10 client.28705 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.755+0000 7fe3e77fe700 1 -- 172.21.15.3:0/4181149048 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55734710b920 con 0x55734722f5f0 2023-09-10T01:17:03.756 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.755+0000 7fe3e77fe700 10 client.28705 handle_command_reply: tid=0 2023-09-10T01:17:03.757 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.755+0000 7fe4221eab80 1 client.28705 shutdown 2023-09-10T01:17:03.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 msgr2=0x557347226750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x557347226750 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0097e0 tx=0x7fe3fc01e040 comp rx=0 tx=0).stop 2023-09-10T01:17:03.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 shutdown_connections 2023-09-10T01:17:03.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55734722f5f0 0x557347231a10 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5573471c1030 0x557347226750 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557347222280 0x557347222700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 --2- 172.21.15.3:0/4181149048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5573471e1240 0x557347226c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 >> 172.21.15.3:0/4181149048 conn(0x557347135090 msgr2=0x5573471c2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.761 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.759+0000 7fe3e77fe700 0 client.28705 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:03.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.760+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 shutdown_connections 2023-09-10T01:17:03.762 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.760+0000 7fe4221eab80 1 -- 172.21.15.3:0/4181149048 wait complete. 2023-09-10T01:17:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.768+0000 7fe4221eab80 20 client.28705 trim_cache size 0 max 16384 2023-09-10T01:17:03.771 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:17:03.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.771+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe3f00a09c0 msgr2=0x7fe3f00a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.772 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.771+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe3f00a09c0 0x7fe3f00a2e80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe400095820 tx=0x7fe400009040 comp rx=0 tx=0).stop 2023-09-10T01:17:03.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.771+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe40810f1b0 msgr2=0x7fe408115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:03.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.771+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe40810f1b0 0x7fe408115f90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe40000efc0 tx=0x7fe40000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.773 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.772+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 shutdown_connections 2023-09-10T01:17:03.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.772+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe3f00a09c0 0x7fe3f00a2e80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.772+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe40810f1b0 0x7fe408115f90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.772+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe408107600 0x7fe40811ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.772+0000 7fe40e7ff700 1 --2- 172.21.15.3:0/821880918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe40810edd0 0x7fe40811b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.772+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 >> 172.21.15.3:0/821880918 conn(0x7fe408071dd0 msgr2=0x7fe4081094f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.773+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 shutdown_connections 2023-09-10T01:17:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:03.774+0000 7fe40e7ff700 1 -- 172.21.15.3:0/821880918 wait complete. 2023-09-10T01:17:04.163 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24448 32.98 MB/sec warmup 45 sec latency 830.319 ms 2023-09-10T01:17:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:04 smithi136 ceph-mon[133708]: pgmap v432: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.0 MiB/s wr, 751 op/s 2023-09-10T01:17:04.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:04 smithi092 ceph-mon[134067]: pgmap v432: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.0 MiB/s wr, 751 op/s 2023-09-10T01:17:04.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:04 smithi003 ceph-mon[97654]: pgmap v432: 129 pgs: 129 active+clean; 612 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.0 MiB/s wr, 751 op/s 2023-09-10T01:17:04.816 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:17:04.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:17:05.168 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24469 32.30 MB/sec warmup 46 sec latency 842.814 ms 2023-09-10T01:17:05.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.225+0000 7f94e3bff700 1 -- 172.21.15.3:0/4149825822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 msgr2=0x7f94dc113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.225+0000 7f94e3bff700 1 --2- 172.21.15.3:0/4149825822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc113b80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009990 tx=0x7f94d8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.230+0000 7f94e3bff700 1 -- 172.21.15.3:0/4149825822 shutdown_connections 2023-09-10T01:17:05.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.230+0000 7f94e3bff700 1 --2- 172.21.15.3:0/4149825822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94dc106ee0 0x7f94dc1072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.230+0000 7f94e3bff700 1 --2- 172.21.15.3:0/4149825822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc113b80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.230+0000 7f94e3bff700 1 --2- 172.21.15.3:0/4149825822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94dc107890 0x7f94dc10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.230+0000 7f94e3bff700 1 -- 172.21.15.3:0/4149825822 >> 172.21.15.3:0/4149825822 conn(0x7f94dc074ba0 msgr2=0x7f94dc076fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:05.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.239+0000 7f94e3bff700 1 -- 172.21.15.3:0/4149825822 shutdown_connections 2023-09-10T01:17:05.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.246+0000 7f94e3bff700 1 -- 172.21.15.3:0/4149825822 wait complete. 2023-09-10T01:17:05.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.248+0000 7f94e3bff700 1 Processor -- start 2023-09-10T01:17:05.258 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.251+0000 7f94e3bff700 1 -- start start 2023-09-10T01:17:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.258+0000 7f94e3bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94dc106ee0 0x7f94dc10f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.258+0000 7f94e3bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc10f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.258+0000 7f94e3bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94dc1af390 0x7f94dc1b1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.258+0000 7f94e3bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc1b1cc0 con 0x7f94dc10c000 2023-09-10T01:17:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.258+0000 7f94e3bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc1b1e00 con 0x7f94dc106ee0 2023-09-10T01:17:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.258+0000 7f94e3bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc1b1f70 con 0x7f94dc1af390 2023-09-10T01:17:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.260+0000 7f94e119a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc10f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.260+0000 7f94e119a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc10f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33948/0 (socket says 172.21.15.3:33948) 2023-09-10T01:17:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.260+0000 7f94e119a700 1 -- 172.21.15.3:0/3578752276 learned_addr learned my addr 172.21.15.3:0/3578752276 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.261+0000 7f94e119a700 1 -- 172.21.15.3:0/3578752276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94dc1af390 msgr2=0x7f94dc1b1780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-10T01:17:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.261+0000 7f94e119a700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94dc1af390 0x7f94dc1b1780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.261+0000 7f94e119a700 1 -- 172.21.15.3:0/3578752276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94dc106ee0 msgr2=0x7f94dc10f320 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-10T01:17:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.261+0000 7f94e119a700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94dc106ee0 0x7f94dc10f320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.261+0000 7f94e119a700 1 -- 172.21.15.3:0/3578752276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80095f0 con 0x7f94dc10c000 2023-09-10T01:17:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.261+0000 7f94e119a700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc10f860 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f94d400b9e0 tx=0x7f94d400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.269+0000 7f94d2ffd700 1 -- 172.21.15.3:0/3578752276 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d400ca30 con 0x7f94dc10c000 2023-09-10T01:17:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.269+0000 7f94d2ffd700 1 -- 172.21.15.3:0/3578752276 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94d400cb90 con 0x7f94dc10c000 2023-09-10T01:17:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.269+0000 7f94d2ffd700 1 -- 172.21.15.3:0/3578752276 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d40125a0 con 0x7f94dc10c000 2023-09-10T01:17:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.269+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94dc1b2250 con 0x7f94dc10c000 2023-09-10T01:17:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.269+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94dc1b2710 con 0x7f94dc10c000 2023-09-10T01:17:05.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.276+0000 7f94f7524b80 1 Processor -- start 2023-09-10T01:17:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.283+0000 7f94f7524b80 1 -- start start 2023-09-10T01:17:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf6233b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf623d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 0x55fcf62402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94f7524b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf618c7b0 con 0x55fcf623dec0 2023-09-10T01:17:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94f7524b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf618a0d0 con 0x55fcf62340b0 2023-09-10T01:17:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94f7524b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf61859c0 con 0x55fcf6233790 2023-09-10T01:17:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94e219c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf6233b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94e219c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf6233b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:52938/0 (socket says 172.21.15.3:52938) 2023-09-10T01:17:05.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.284+0000 7f94e219c700 1 -- 172.21.15.3:0/2952038276 learned_addr learned my addr 172.21.15.3:0/2952038276 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:05.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.285+0000 7f94d2ffd700 1 -- 172.21.15.3:0/3578752276 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f94d400cd00 con 0x7f94dc10c000 2023-09-10T01:17:05.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.285+0000 7f94e219c700 1 -- 172.21.15.3:0/2952038276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 msgr2=0x55fcf623d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.285+0000 7f94e219c700 1 --2- 172.21.15.3:0/2952038276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf623d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.285+0000 7f94e219c700 1 -- 172.21.15.3:0/2952038276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 msgr2=0x55fcf62402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.285+0000 7f94e219c700 1 --2- 172.21.15.3:0/2952038276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 0x55fcf62402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.285+0000 7f94e219c700 1 -- 172.21.15.3:0/2952038276 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcf5fb1e60 con 0x55fcf6233790 2023-09-10T01:17:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.285+0000 7f94e219c700 1 --2- 172.21.15.3:0/2952038276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf6233b70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f94d8005ae0 tx=0x7f94d8005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba9210702ca51e96 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.287+0000 7f94d2ffd700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f94c80a09c0 0x7f94c80a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.287+0000 7f94d2ffd700 1 -- 172.21.15.3:0/3578752276 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f94d40cb550 con 0x7f94dc10c000 2023-09-10T01:17:05.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.288+0000 7f94c7fff700 1 -- 172.21.15.3:0/2952038276 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d801d070 con 0x55fcf6233790 2023-09-10T01:17:05.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.288+0000 7f94c7fff700 1 -- 172.21.15.3:0/2952038276 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94d80055d0 con 0x55fcf6233790 2023-09-10T01:17:05.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.288+0000 7f94c7fff700 1 -- 172.21.15.3:0/2952038276 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d8004500 con 0x55fcf6233790 2023-09-10T01:17:05.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.289+0000 7f94f7524b80 1 -- 172.21.15.3:0/2952038276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 msgr2=0x55fcf6233b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.289+0000 7f94f7524b80 1 --2- 172.21.15.3:0/2952038276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf6233b70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f94d8005ae0 tx=0x7f94d8005be0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.290+0000 7f94f7524b80 1 -- 172.21.15.3:0/2952038276 shutdown_connections 2023-09-10T01:17:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.290+0000 7f94f7524b80 1 --2- 172.21.15.3:0/2952038276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf6233b70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.290+0000 7f94f7524b80 1 --2- 172.21.15.3:0/2952038276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 0x55fcf62402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.290+0000 7f94f7524b80 1 --2- 172.21.15.3:0/2952038276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf623d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.290+0000 7f94f7524b80 1 -- 172.21.15.3:0/2952038276 >> 172.21.15.3:0/2952038276 conn(0x55fcf61a8110 msgr2=0x55fcf622ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:05.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.290+0000 7f94f7524b80 1 -- 172.21.15.3:0/2952038276 shutdown_connections 2023-09-10T01:17:05.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.290+0000 7f94f7524b80 1 -- 172.21.15.3:0/2952038276 wait complete. 2023-09-10T01:17:05.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.291+0000 7f94f7524b80 1 -- start start 2023-09-10T01:17:05.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.291+0000 7f94f7524b80 1 Processor -- start 2023-09-10T01:17:05.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf623acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf623b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 0x55fcf623b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94f7524b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf6123bf0 con 0x55fcf623dec0 2023-09-10T01:17:05.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94f7524b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf61238f0 con 0x55fcf62340b0 2023-09-10T01:17:05.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94f7524b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf60079a0 con 0x55fcf6233790 2023-09-10T01:17:05.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94f7524b80 10 client.0 _ll_register_callbacks cb 0x55fcf606db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:05.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.292+0000 7f94e199b700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f94c80a09c0 0x7f94c80a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.295+0000 7f94e199b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf623acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.295+0000 7f94e199b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf623acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:52954/0 (socket says 172.21.15.3:52954) 2023-09-10T01:17:05.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.295+0000 7f94e199b700 1 -- 172.21.15.3:0/3394659275 learned_addr learned my addr 172.21.15.3:0/3394659275 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:05.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.298+0000 7f94e199b700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f94c80a09c0 0x7f94c80a2e80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94cc005950 tx=0x7f94cc00bba0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:05.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.298+0000 7f94e199b700 1 -- 172.21.15.3:0/3394659275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 msgr2=0x55fcf623b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:05.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.298+0000 7f94e199b700 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf623b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.298+0000 7f94e199b700 1 -- 172.21.15.3:0/3394659275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 msgr2=0x55fcf623b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:05.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.298+0000 7f94e199b700 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 0x55fcf623b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.298+0000 7f94e199b700 1 -- 172.21.15.3:0/3394659275 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94dc1143f0 con 0x55fcf6233790 2023-09-10T01:17:05.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.298+0000 7f94e199b700 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf623acf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00dbc0 tx=0x7f94cc012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9544d3f75e94245 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:05.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.299+0000 7f94c57fa700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:05.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.299+0000 7f94c57fa700 1 -- 172.21.15.3:0/3394659275 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc012d50 con 0x55fcf6233790 2023-09-10T01:17:05.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.299+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fcf60dd570 con 0x55fcf6233790 2023-09-10T01:17:05.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.299+0000 7f94c57fa700 1 -- 172.21.15.3:0/3394659275 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f94cc010450 con 0x55fcf6233790 2023-09-10T01:17:05.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.309+0000 7f94c57fa700 1 -- 172.21.15.3:0/3394659275 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94cc010610 con 0x55fcf6233790 2023-09-10T01:17:05.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.311+0000 7f94f7524b80 10 client.28714 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:05.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.311+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55fcf5f9ace0 con 0x55fcf6233790 2023-09-10T01:17:05.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.321+0000 7f94c57fa700 1 -- 172.21.15.3:0/3394659275 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fcf60dd570 con 0x55fcf6233790 2023-09-10T01:17:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.321+0000 7f94c57fa700 1 -- 172.21.15.3:0/3394659275 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f94cc01d900 con 0x55fcf6233790 2023-09-10T01:17:05.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.324+0000 7f94f7524b80 10 client.28714 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:05.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.324+0000 7f94f7524b80 10 client.28714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:05.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.324+0000 7f94f7524b80 20 client.28714 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:05.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.324+0000 7f94f7524b80 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf626fba0 0x55fcf6271fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.324+0000 7f94f7524b80 4 client.28714 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:05.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.324+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fcf619a920 con 0x55fcf626fba0 2023-09-10T01:17:05.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.329+0000 7f94e119a700 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf626fba0 0x55fcf6271fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.337+0000 7f94e119a700 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf626fba0 0x55fcf6271fc0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d5358abc0791c75 server_cookie=64c89dfd4ad7d1c1 in_seq=0 out_seq=0 2023-09-10T01:17:05.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.337+0000 7f94c57fa700 10 client.28714 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:05.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.341+0000 7f94c57fa700 1 -- 172.21.15.3:0/3394659275 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fcf619a920 con 0x55fcf626fba0 2023-09-10T01:17:05.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.341+0000 7f94c57fa700 10 client.28714 handle_command_reply: tid=0 2023-09-10T01:17:05.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.341+0000 7f94f7524b80 1 client.28714 shutdown 2023-09-10T01:17:05.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.365+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 msgr2=0x55fcf623acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.365+0000 7f94f7524b80 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf623acf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00dbc0 tx=0x7f94cc012000 comp rx=0 tx=0).stop 2023-09-10T01:17:05.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.369+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 shutdown_connections 2023-09-10T01:17:05.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.369+0000 7f94f7524b80 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf626fba0 0x55fcf6271fc0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.369+0000 7f94f7524b80 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf6233790 0x55fcf623acf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.369+0000 7f94f7524b80 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf623dec0 0x55fcf623b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.369+0000 7f94f7524b80 1 --2- 172.21.15.3:0/3394659275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf623b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.369+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 >> 172.21.15.3:0/3394659275 conn(0x55fcf61a8110 msgr2=0x55fcf622e800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:05.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.370+0000 7f94c57fa700 0 client.28714 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:05.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.381+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 shutdown_connections 2023-09-10T01:17:05.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.389+0000 7f94f7524b80 1 -- 172.21.15.3:0/3394659275 wait complete. 2023-09-10T01:17:05.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.392+0000 7f94f7524b80 20 client.28714 trim_cache size 0 max 16384 2023-09-10T01:17:05.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.422+0000 7f94f7524b80 1 Processor -- start 2023-09-10T01:17:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.423+0000 7f94f7524b80 1 -- start start 2023-09-10T01:17:05.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.424+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf6233790 0x55fcf6244ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.424+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf62340b0 0x55fcf623b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.424+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf6236020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.424+0000 7f94f7524b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf6228df0 con 0x55fcf62340b0 2023-09-10T01:17:05.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.424+0000 7f94f7524b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf622f800 con 0x55fcf6233790 2023-09-10T01:17:05.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.424+0000 7f94f7524b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf622f1c0 con 0x55fcf623dec0 2023-09-10T01:17:05.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.427+0000 7f94e219c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf6236020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.427+0000 7f94e219c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf6236020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:52970/0 (socket says 172.21.15.3:52970) 2023-09-10T01:17:05.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.427+0000 7f94e219c700 1 -- 172.21.15.3:0/1969402197 learned_addr learned my addr 172.21.15.3:0/1969402197 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:05.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.429+0000 7f94e119a700 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf62340b0 0x55fcf623b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.430+0000 7f94e119a700 1 -- 172.21.15.3:0/1969402197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 msgr2=0x55fcf6236020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:05.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.430+0000 7f94e119a700 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf6236020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.430+0000 7f94e119a700 1 -- 172.21.15.3:0/1969402197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf6233790 msgr2=0x55fcf6244ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:05.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.430+0000 7f94e119a700 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf6233790 0x55fcf6244ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.430+0000 7f94e119a700 1 -- 172.21.15.3:0/1969402197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcf6276bf0 con 0x55fcf62340b0 2023-09-10T01:17:05.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.431+0000 7f94e119a700 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf62340b0 0x55fcf623b4b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f94d40c8910 tx=0x7f94d4083040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a58a1a4185d40b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:05.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.433+0000 7f94d0bff700 1 -- 172.21.15.3:0/1969402197 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d409c530 con 0x55fcf62340b0 2023-09-10T01:17:05.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.433+0000 7f94d0bff700 1 -- 172.21.15.3:0/1969402197 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94d40a06e0 con 0x55fcf62340b0 2023-09-10T01:17:05.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.433+0000 7f94d0bff700 1 -- 172.21.15.3:0/1969402197 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d4098960 con 0x55fcf62340b0 2023-09-10T01:17:05.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.434+0000 7f94f7524b80 1 -- 172.21.15.3:0/1969402197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf62340b0 msgr2=0x55fcf623b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.434+0000 7f94f7524b80 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf62340b0 0x55fcf623b4b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f94d40c8910 tx=0x7f94d4083040 comp rx=0 tx=0).stop 2023-09-10T01:17:05.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.436+0000 7f94f7524b80 1 -- 172.21.15.3:0/1969402197 shutdown_connections 2023-09-10T01:17:05.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.436+0000 7f94f7524b80 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf6236020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.436+0000 7f94f7524b80 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf6233790 0x55fcf6244ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.436+0000 7f94f7524b80 1 --2- 172.21.15.3:0/1969402197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf62340b0 0x55fcf623b4b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.436+0000 7f94f7524b80 1 -- 172.21.15.3:0/1969402197 >> 172.21.15.3:0/1969402197 conn(0x55fcf61a8110 msgr2=0x55fcf6271bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:05.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.437+0000 7f94f7524b80 1 -- 172.21.15.3:0/1969402197 shutdown_connections 2023-09-10T01:17:05.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.443+0000 7f94f7524b80 1 -- 172.21.15.3:0/1969402197 wait complete. 2023-09-10T01:17:05.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.443+0000 7f94f7524b80 1 -- start start 2023-09-10T01:17:05.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 1 Processor -- start 2023-09-10T01:17:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf6233790 0x55fcf62b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf62b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf62bc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf62b8b30 con 0x55fcf6233790 2023-09-10T01:17:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf62bc960 con 0x55fcf62340b0 2023-09-10T01:17:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf62bcb20 con 0x55fcf623dec0 2023-09-10T01:17:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.444+0000 7f94f7524b80 10 client.0 _ll_register_callbacks cb 0x7f94cc00d7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf62b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf62b82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:39754/0 (socket says 172.21.15.3:39754) 2023-09-10T01:17:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 -- 172.21.15.3:0/658319800 learned_addr learned my addr 172.21.15.3:0/658319800 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e219c700 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf62bc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 -- 172.21.15.3:0/658319800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 msgr2=0x55fcf62bc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf62bc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 -- 172.21.15.3:0/658319800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf6233790 msgr2=0x55fcf62b7db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf6233790 0x55fcf62b7db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.447+0000 7f94e119a700 1 -- 172.21.15.3:0/658319800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94dc1b2710 con 0x55fcf62340b0 2023-09-10T01:17:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.448+0000 7f94e119a700 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf62b82f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f94d4094c00 tx=0x7f94d40c8970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35a0b8803edfd0f4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.448+0000 7f94c57fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:05.450 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.448+0000 7f94c57fa700 1 -- 172.21.15.3:0/658319800 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d409c320 con 0x55fcf62340b0 2023-09-10T01:17:05.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.448+0000 7f94c57fa700 1 -- 172.21.15.3:0/658319800 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f94d4096b60 con 0x55fcf62340b0 2023-09-10T01:17:05.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.449+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94d80055d0 con 0x55fcf62340b0 2023-09-10T01:17:05.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.455+0000 7f94c57fa700 1 -- 172.21.15.3:0/658319800 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94d40cb380 con 0x55fcf62340b0 2023-09-10T01:17:05.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.456+0000 7f94f7524b80 10 client.28847 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:05.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.456+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55fcf5fd82d0 con 0x55fcf62340b0 2023-09-10T01:17:05.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94c57fa700 1 -- 172.21.15.3:0/658319800 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94d80055d0 con 0x55fcf62340b0 2023-09-10T01:17:05.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94c57fa700 1 -- 172.21.15.3:0/658319800 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f94d40a0250 con 0x55fcf62340b0 2023-09-10T01:17:05.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94f7524b80 10 client.28847 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:05.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94f7524b80 10 client.28847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:05.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94f7524b80 20 client.28847 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:05.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94f7524b80 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf62d6510 0x55fcf62d8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:05.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94f7524b80 4 client.28847 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:17:05.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.458+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55fcf617e9a0 con 0x55fcf62d6510 2023-09-10T01:17:05.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.468+0000 7f94e199b700 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf62d6510 0x55fcf62d8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:05.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.475+0000 7f94e199b700 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf62d6510 0x55fcf62d8930 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec8e1afe18e78c34 server_cookie=c6012cf7d65d01a2 in_seq=0 out_seq=0 2023-09-10T01:17:05.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.478+0000 7f94c57fa700 10 client.28847 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:05.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.478+0000 7f94c57fa700 1 -- 172.21.15.3:0/658319800 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55fcf617e9a0 con 0x55fcf62d6510 2023-09-10T01:17:05.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.478+0000 7f94c57fa700 10 client.28847 handle_command_reply: tid=0 2023-09-10T01:17:05.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.478+0000 7f94f7524b80 1 client.28847 shutdown 2023-09-10T01:17:05.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 msgr2=0x55fcf62b82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf62b82f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f94d4094c00 tx=0x7f94d40c8970 comp rx=0 tx=0).stop 2023-09-10T01:17:05.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 shutdown_connections 2023-09-10T01:17:05.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55fcf62d6510 0x55fcf62d8930 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fcf623dec0 0x55fcf62bc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcf6233790 0x55fcf62b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 --2- 172.21.15.3:0/658319800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fcf62340b0 0x55fcf62b82f0 secure :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f94d4094c00 tx=0x7f94d40c8970 comp rx=0 tx=0).stop 2023-09-10T01:17:05.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 >> 172.21.15.3:0/658319800 conn(0x55fcf61a8110 msgr2=0x55fcf623b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:05.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.480+0000 7f94c57fa700 0 client.28847 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:05.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.481+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 shutdown_connections 2023-09-10T01:17:05.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.483+0000 7f94f7524b80 1 -- 172.21.15.3:0/658319800 wait complete. 2023-09-10T01:17:05.491 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:17:05.491 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:17:05.492 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "0e47f32e-255d-43b8-9e93-e96155881931", 2023-09-10T01:17:05.492 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:17:05.492 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:17:05.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.488+0000 7f94f7524b80 20 client.28847 trim_cache size 0 max 16384 2023-09-10T01:17:05.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.491+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f94c80a09c0 msgr2=0x7f94c80a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.491+0000 7f94e3bff700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f94c80a09c0 0x7f94c80a2e80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94cc005950 tx=0x7f94cc00bba0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.491+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 msgr2=0x7f94dc10f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.491+0000 7f94e3bff700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc10f860 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f94d400b9e0 tx=0x7f94d400bda0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.496+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 shutdown_connections 2023-09-10T01:17:05.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.496+0000 7f94e3bff700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f94c80a09c0 0x7f94c80a2e80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.496+0000 7f94e3bff700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94dc1af390 0x7f94dc1b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.496+0000 7f94e3bff700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94dc106ee0 0x7f94dc10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.496+0000 7f94e3bff700 1 --2- 172.21.15.3:0/3578752276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94dc10c000 0x7f94dc10f860 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:05.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.496+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 >> 172.21.15.3:0/3578752276 conn(0x7f94dc074ba0 msgr2=0x7f94dc075590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:05.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.499+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 shutdown_connections 2023-09-10T01:17:05.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.501+0000 7f94e3bff700 1 -- 172.21.15.3:0/3578752276 wait complete. 2023-09-10T01:17:05.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:17:05.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:05 smithi136 ceph-mon[133708]: osdmap e106: 12 total, 12 up, 12 in 2023-09-10T01:17:05.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:05 smithi003 ceph-mon[97654]: osdmap e106: 12 total, 12 up, 12 in 2023-09-10T01:17:05.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:05 smithi092 ceph-mon[134067]: osdmap e106: 12 total, 12 up, 12 in 2023-09-10T01:17:05.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.986+0000 7fe654bff700 1 -- 172.21.15.3:0/4279894079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010f1b0 msgr2=0x7fe65010cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:05.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.986+0000 7fe654bff700 1 --2- 172.21.15.3:0/4279894079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010f1b0 0x7fe65010cfc0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe64800b0a0 tx=0x7fe64800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.993+0000 7fe654bff700 1 -- 172.21.15.3:0/4279894079 shutdown_connections 2023-09-10T01:17:06.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.993+0000 7fe654bff700 1 --2- 172.21.15.3:0/4279894079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 0x7fe650107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.993+0000 7fe654bff700 1 --2- 172.21.15.3:0/4279894079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe65010edd0 0x7fe65010ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.993+0000 7fe654bff700 1 --2- 172.21.15.3:0/4279894079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010f1b0 0x7fe65010cfc0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.993+0000 7fe654bff700 1 -- 172.21.15.3:0/4279894079 >> 172.21.15.3:0/4279894079 conn(0x7fe650071dd0 msgr2=0x7fe65006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:06.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.995+0000 7fe654bff700 1 -- 172.21.15.3:0/4279894079 shutdown_connections 2023-09-10T01:17:06.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.996+0000 7fe654bff700 1 -- 172.21.15.3:0/4279894079 wait complete. 2023-09-10T01:17:06.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.997+0000 7fe654bff700 1 Processor -- start 2023-09-10T01:17:06.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe654bff700 1 -- start start 2023-09-10T01:17:06.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe654bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 0x7fe65011aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe654bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010edd0 0x7fe65011b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe654bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe65010f1b0 0x7fe650116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe654bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6501166d0 con 0x7fe65010f1b0 2023-09-10T01:17:06.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe654bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe650116840 con 0x7fe65010edd0 2023-09-10T01:17:06.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe654bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe650116a00 con 0x7fe650107600 2023-09-10T01:17:06.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe64e59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 0x7fe65011aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe64e59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 0x7fe65011aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:53016/0 (socket says 172.21.15.3:53016) 2023-09-10T01:17:06.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.998+0000 7fe64e59c700 1 -- 172.21.15.3:0/551031483 learned_addr learned my addr 172.21.15.3:0/551031483 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:06.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64dd9b700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010edd0 0x7fe65011b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe65010f1b0 0x7fe650116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64e59c700 1 -- 172.21.15.3:0/551031483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010edd0 msgr2=0x7fe65011b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64e59c700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010edd0 0x7fe65011b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64e59c700 1 -- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe65010f1b0 msgr2=0x7fe650116020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64e59c700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe65010f1b0 0x7fe650116020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64e59c700 1 -- 172.21.15.3:0/551031483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648009d00 con 0x7fe650107600 2023-09-10T01:17:06.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe65010f1b0 0x7fe650116020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:06.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:05.999+0000 7fe64e59c700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 0x7fe65011aea0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b530 tx=0x7fe64400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:06.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.003+0000 7fe63f7fe700 1 -- 172.21.15.3:0/551031483 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644011830 con 0x7fe650107600 2023-09-10T01:17:06.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.003+0000 7fe63f7fe700 1 -- 172.21.15.3:0/551031483 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe644011e50 con 0x7fe650107600 2023-09-10T01:17:06.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.003+0000 7fe63f7fe700 1 -- 172.21.15.3:0/551031483 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400f2e0 con 0x7fe650107600 2023-09-10T01:17:06.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.004+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe650116ce0 con 0x7fe650107600 2023-09-10T01:17:06.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.004+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe650119de0 con 0x7fe650107600 2023-09-10T01:17:06.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.014+0000 7fe66861db80 1 Processor -- start 2023-09-10T01:17:06.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.026+0000 7fe66861db80 1 -- start start 2023-09-10T01:17:06.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.026+0000 7fe66861db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 0x55dccba6baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.030 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.026+0000 7fe66861db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba75900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.026+0000 7fe66861db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba75e40 0x55dccba78230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.026+0000 7fe66861db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccb9c4730 con 0x55dccba6c030 2023-09-10T01:17:06.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.026+0000 7fe66861db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccb9c2050 con 0x55dccba6b710 2023-09-10T01:17:06.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.026+0000 7fe66861db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccb9bd940 con 0x55dccba75e40 2023-09-10T01:17:06.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.029+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba75e40 0x55dccba78230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.029+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba75e40 0x55dccba78230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:53022/0 (socket says 172.21.15.3:53022) 2023-09-10T01:17:06.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.029+0000 7fe64dd9b700 1 -- 172.21.15.3:0/266516049 learned_addr learned my addr 172.21.15.3:0/266516049 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:06.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.030+0000 7fe64e59c700 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba75900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.030+0000 7fe63f7fe700 1 -- 172.21.15.3:0/551031483 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe6440105f0 con 0x7fe650107600 2023-09-10T01:17:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.032+0000 7fe64e59c700 1 -- 172.21.15.3:0/266516049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba75e40 msgr2=0x55dccba78230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.032+0000 7fe64e59c700 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba75e40 0x55dccba78230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.032+0000 7fe64e59c700 1 -- 172.21.15.3:0/266516049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 msgr2=0x55dccba6baf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:06.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.032+0000 7fe64e59c700 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 0x55dccba6baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.032+0000 7fe64e59c700 1 -- 172.21.15.3:0/266516049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dccb7e9de0 con 0x55dccba6c030 2023-09-10T01:17:06.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.032+0000 7fe64e59c700 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba75900 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe6440cb290 tx=0x7fe6440cff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7b70591c2eb167 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.033+0000 7fe63f7fe700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6380a09c0 0x7fe6380a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.033+0000 7fe63f7fe700 1 -- 172.21.15.3:0/551031483 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fe6440cded0 con 0x7fe650107600 2023-09-10T01:17:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.033+0000 7fe63cff9700 1 -- 172.21.15.3:0/266516049 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6440973e0 con 0x55dccba6c030 2023-09-10T01:17:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.034+0000 7fe63cff9700 1 -- 172.21.15.3:0/266516049 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe64409ae70 con 0x55dccba6c030 2023-09-10T01:17:06.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.034+0000 7fe63cff9700 1 -- 172.21.15.3:0/266516049 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644010380 con 0x55dccba6c030 2023-09-10T01:17:06.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.035+0000 7fe66861db80 1 -- 172.21.15.3:0/266516049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 msgr2=0x55dccba75900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.035+0000 7fe66861db80 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba75900 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe6440cb290 tx=0x7fe6440cff10 comp rx=0 tx=0).stop 2023-09-10T01:17:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.036+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6380a09c0 0x7fe6380a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.036+0000 7fe66861db80 1 -- 172.21.15.3:0/266516049 shutdown_connections 2023-09-10T01:17:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.036+0000 7fe66861db80 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba75e40 0x55dccba78230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.036+0000 7fe66861db80 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 0x55dccba6baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.036+0000 7fe66861db80 1 --2- 172.21.15.3:0/266516049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba75900 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.036+0000 7fe66861db80 1 -- 172.21.15.3:0/266516049 >> 172.21.15.3:0/266516049 conn(0x55dccb9e0090 msgr2=0x55dccba66bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:06.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.037+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6380a09c0 0x7fe6380a2e80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe64000a830 tx=0x7fe640008040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:06.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.041+0000 7fe66861db80 1 -- 172.21.15.3:0/266516049 shutdown_connections 2023-09-10T01:17:06.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.044+0000 7fe66861db80 1 -- 172.21.15.3:0/266516049 wait complete. 2023-09-10T01:17:06.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.045+0000 7fe66861db80 1 -- start start 2023-09-10T01:17:06.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.045+0000 7fe66861db80 1 Processor -- start 2023-09-10T01:17:06.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.046+0000 7fe66861db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccba75820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.046+0000 7fe66861db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba72aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.046+0000 7fe66861db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba75e40 0x55dccba73100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.046+0000 7fe66861db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccb95bb70 con 0x55dccba6c030 2023-09-10T01:17:06.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.046+0000 7fe66861db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccb95b870 con 0x55dccba75e40 2023-09-10T01:17:06.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.046+0000 7fe66861db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccb83f920 con 0x55dccba6b710 2023-09-10T01:17:06.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.046+0000 7fe66861db80 10 client.0 _ll_register_callbacks cb 0x55dccb8a5a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:06.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.047+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba72aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.047+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba72aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33998/0 (socket says 172.21.15.3:33998) 2023-09-10T01:17:06.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.047+0000 7fe64dd9b700 1 -- 172.21.15.3:0/3213317908 learned_addr learned my addr 172.21.15.3:0/3213317908 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:06.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.047+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba75e40 0x55dccba73100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.047+0000 7fe64e59c700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccba75820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.048+0000 7fe64ed9d700 1 -- 172.21.15.3:0/3213317908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 msgr2=0x55dccba75820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.048+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccba75820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.048+0000 7fe64ed9d700 1 -- 172.21.15.3:0/3213317908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 msgr2=0x55dccba72aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:06.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.048+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba72aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.048+0000 7fe64ed9d700 1 -- 172.21.15.3:0/3213317908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe650119de0 con 0x55dccba75e40 2023-09-10T01:17:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.048+0000 7fe64e59c700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccba75820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:17:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.048+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba75e40 0x55dccba73100 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe640006fd0 tx=0x7fe6400047e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6526645ee2770ed0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:06.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.049+0000 7fe635ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:06.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.049+0000 7fe635ffb700 1 -- 172.21.15.3:0/3213317908 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640011c40 con 0x55dccba75e40 2023-09-10T01:17:06.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.049+0000 7fe635ffb700 1 -- 172.21.15.3:0/3213317908 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fe640011da0 con 0x55dccba75e40 2023-09-10T01:17:06.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.049+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dccb9154f0 con 0x55dccba75e40 2023-09-10T01:17:06.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.052+0000 7fe635ffb700 1 -- 172.21.15.3:0/3213317908 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe64001e400 con 0x55dccba75e40 2023-09-10T01:17:06.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.055+0000 7fe66861db80 10 client.28856 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:06.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.055+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55dccb7d2c60 con 0x55dccba75e40 2023-09-10T01:17:06.056 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.055+0000 7fe635ffb700 1 -- 172.21.15.3:0/3213317908 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dccb9154f0 con 0x55dccba75e40 2023-09-10T01:17:06.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.057+0000 7fe635ffb700 1 -- 172.21.15.3:0/3213317908 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe640026070 con 0x55dccba75e40 2023-09-10T01:17:06.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.057+0000 7fe66861db80 10 client.28856 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.057+0000 7fe66861db80 10 client.28856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.057+0000 7fe66861db80 20 client.28856 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.057+0000 7fe66861db80 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccba85980 0x55dccba87da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.057+0000 7fe66861db80 4 client.28856 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:06.059 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.057+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dccb9d28a0 con 0x55dccba85980 2023-09-10T01:17:06.060 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.058+0000 7fe64e59c700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccba85980 0x55dccba87da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.059+0000 7fe64e59c700 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccba85980 0x55dccba87da0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3176b84a0cd9ce0 server_cookie=cf554ad013f37549 in_seq=0 out_seq=0 2023-09-10T01:17:06.061 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.061+0000 7fe635ffb700 10 client.28856 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:06.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.068+0000 7fe635ffb700 1 -- 172.21.15.3:0/3213317908 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dccb9d28a0 con 0x55dccba85980 2023-09-10T01:17:06.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.068+0000 7fe635ffb700 10 client.28856 handle_command_reply: tid=0 2023-09-10T01:17:06.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.069+0000 7fe66861db80 1 client.28856 shutdown 2023-09-10T01:17:06.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.077+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba75e40 msgr2=0x55dccba73100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.077+0000 7fe66861db80 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba75e40 0x55dccba73100 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe640006fd0 tx=0x7fe6400047e0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.082+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 shutdown_connections 2023-09-10T01:17:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.082+0000 7fe66861db80 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccba85980 0x55dccba87da0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.083+0000 7fe66861db80 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccba75820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.085 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.083+0000 7fe66861db80 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba75e40 0x55dccba73100 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.083+0000 7fe66861db80 1 --2- 172.21.15.3:0/3213317908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba6c030 0x55dccba72aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.083+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 >> 172.21.15.3:0/3213317908 conn(0x55dccb9e0090 msgr2=0x55dccba667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:06.086 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.083+0000 7fe635ffb700 0 client.28856 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:06.098 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.097+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 shutdown_connections 2023-09-10T01:17:06.099 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.099+0000 7fe66861db80 1 -- 172.21.15.3:0/3213317908 wait complete. 2023-09-10T01:17:06.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.104+0000 7fe66861db80 20 client.28856 trim_cache size 0 max 16384 2023-09-10T01:17:06.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.146+0000 7fe66861db80 1 Processor -- start 2023-09-10T01:17:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe66861db80 1 -- start start 2023-09-10T01:17:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe66861db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 0x55dccba7ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe66861db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6c030 0x55dccba65dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe66861db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccba81180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe66861db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccba78750 con 0x55dccba75e40 2023-09-10T01:17:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe66861db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccba7cf70 con 0x55dccba6b710 2023-09-10T01:17:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe66861db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccba6d210 con 0x55dccba6c030 2023-09-10T01:17:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.147+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6c030 0x55dccba65dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6c030 0x55dccba65dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:53034/0 (socket says 172.21.15.3:53034) 2023-09-10T01:17:06.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64dd9b700 1 -- 172.21.15.3:0/2506089472 learned_addr learned my addr 172.21.15.3:0/2506089472 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64e59c700 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 0x55dccba7ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccba81180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64ed9d700 1 -- 172.21.15.3:0/2506089472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6c030 msgr2=0x55dccba65dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6c030 0x55dccba65dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64ed9d700 1 -- 172.21.15.3:0/2506089472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 msgr2=0x55dccba7ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 0x55dccba7ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.148+0000 7fe64ed9d700 1 -- 172.21.15.3:0/2506089472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dccb810250 con 0x55dccba75e40 2023-09-10T01:17:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.149+0000 7fe64ed9d700 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccba81180 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe64000af80 tx=0x7fe640027420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0ecb3fe44aeb18 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.149+0000 7fe63d7fa700 1 -- 172.21.15.3:0/2506089472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640011c40 con 0x55dccba75e40 2023-09-10T01:17:06.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.149+0000 7fe63d7fa700 1 -- 172.21.15.3:0/2506089472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe640027b10 con 0x55dccba75e40 2023-09-10T01:17:06.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.149+0000 7fe63d7fa700 1 -- 172.21.15.3:0/2506089472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640032ac0 con 0x55dccba75e40 2023-09-10T01:17:06.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.153+0000 7fe66861db80 1 -- 172.21.15.3:0/2506089472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 msgr2=0x55dccba81180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.153+0000 7fe66861db80 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccba81180 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe64000af80 tx=0x7fe640027420 comp rx=0 tx=0).stop 2023-09-10T01:17:06.170 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24474 31.62 MB/sec warmup 47 sec latency 1392.192 ms 2023-09-10T01:17:06.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.162+0000 7fe66861db80 1 -- 172.21.15.3:0/2506089472 shutdown_connections 2023-09-10T01:17:06.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.162+0000 7fe66861db80 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccba81180 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.162+0000 7fe66861db80 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6b710 0x55dccba7ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.162+0000 7fe66861db80 1 --2- 172.21.15.3:0/2506089472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6c030 0x55dccba65dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.162+0000 7fe66861db80 1 -- 172.21.15.3:0/2506089472 >> 172.21.15.3:0/2506089472 conn(0x55dccb9e0090 msgr2=0x55dccba65320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:06.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.169+0000 7fe66861db80 1 -- 172.21.15.3:0/2506089472 shutdown_connections 2023-09-10T01:17:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.169+0000 7fe66861db80 1 -- 172.21.15.3:0/2506089472 wait complete. 2023-09-10T01:17:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.170+0000 7fe66861db80 1 -- start start 2023-09-10T01:17:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.170+0000 7fe66861db80 1 Processor -- start 2023-09-10T01:17:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.171+0000 7fe66861db80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccbad3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.171+0000 7fe66861db80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6c030 0x55dccbad3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.171+0000 7fe66861db80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccbacd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.171+0000 7fe66861db80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccbacdd20 con 0x55dccba75e40 2023-09-10T01:17:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.171+0000 7fe66861db80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccbacde90 con 0x55dccba6c030 2023-09-10T01:17:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.171+0000 7fe66861db80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dccbace050 con 0x55dccba6b710 2023-09-10T01:17:06.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.171+0000 7fe66861db80 10 client.0 _ll_register_callbacks cb 0x7fe640013640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:06.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.177+0000 7fe64dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6c030 0x55dccbad3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.177+0000 7fe64e59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccbad3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.177+0000 7fe64ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccbacd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.177+0000 7fe64ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccbacd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34020/0 (socket says 172.21.15.3:34020) 2023-09-10T01:17:06.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.177+0000 7fe64ed9d700 1 -- 172.21.15.3:0/1445953868 learned_addr learned my addr 172.21.15.3:0/1445953868 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:06.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.178+0000 7fe64dd9b700 1 -- 172.21.15.3:0/1445953868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 msgr2=0x55dccbad3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:06.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.178+0000 7fe64dd9b700 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccbad3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.178+0000 7fe64dd9b700 1 -- 172.21.15.3:0/1445953868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 msgr2=0x55dccbacd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:06.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.178+0000 7fe64dd9b700 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccbacd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.178+0000 7fe64dd9b700 1 -- 172.21.15.3:0/1445953868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dccb7d2c60 con 0x55dccba6c030 2023-09-10T01:17:06.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.178+0000 7fe64dd9b700 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6c030 0x55dccbad3660 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x55dccba63480 tx=0x7fe6480074e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdd5e4f75402ac8d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:06.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.179+0000 7fe635ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:06.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.179+0000 7fe635ffb700 1 -- 172.21.15.3:0/1445953868 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64800e040 con 0x55dccba6c030 2023-09-10T01:17:06.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.179+0000 7fe635ffb700 1 -- 172.21.15.3:0/1445953868 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fe6480079d0 con 0x55dccba6c030 2023-09-10T01:17:06.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.179+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe64409ae70 con 0x55dccba6c030 2023-09-10T01:17:06.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.179+0000 7fe635ffb700 1 -- 172.21.15.3:0/1445953868 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe648005040 con 0x55dccba6c030 2023-09-10T01:17:06.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.179+0000 7fe66861db80 10 client.28862 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:06.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.179+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55dccb7e82b0 con 0x55dccba6c030 2023-09-10T01:17:06.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe635ffb700 1 -- 172.21.15.3:0/1445953868 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe64409ae70 con 0x55dccba6c030 2023-09-10T01:17:06.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe635ffb700 1 -- 172.21.15.3:0/1445953868 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe64800b0d0 con 0x55dccba6c030 2023-09-10T01:17:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe66861db80 10 client.28862 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe66861db80 10 client.28862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe66861db80 20 client.28862 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe66861db80 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccbadb060 0x55dccbadd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe66861db80 4 client.28862 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:17:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.180+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dccb9b6920 con 0x55dccbadb060 2023-09-10T01:17:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.196+0000 7fe64e59c700 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccbadb060 0x55dccbadd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:06.206 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.201+0000 7fe64e59c700 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccbadb060 0x55dccbadd480 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a644420cfd89f34 server_cookie=57091ace515acb11 in_seq=0 out_seq=0 2023-09-10T01:17:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.201+0000 7fe635ffb700 10 client.28862 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.202+0000 7fe635ffb700 1 -- 172.21.15.3:0/1445953868 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55dccb9b6920 con 0x55dccbadb060 2023-09-10T01:17:06.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.202+0000 7fe635ffb700 10 client.28862 handle_command_reply: tid=0 2023-09-10T01:17:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.202+0000 7fe66861db80 1 client.28862 shutdown 2023-09-10T01:17:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.205+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6c030 msgr2=0x55dccbad3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.205+0000 7fe66861db80 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6c030 0x55dccbad3660 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x55dccba63480 tx=0x7fe6480074e0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.209+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 shutdown_connections 2023-09-10T01:17:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.209+0000 7fe66861db80 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dccbadb060 0x55dccbadd480 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.210 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.209+0000 7fe66861db80 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dccba6b710 0x55dccbad3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.209+0000 7fe66861db80 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dccba75e40 0x55dccbacd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.209+0000 7fe66861db80 1 --2- 172.21.15.3:0/1445953868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dccba6c030 0x55dccbad3660 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.211 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.209+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 >> 172.21.15.3:0/1445953868 conn(0x55dccb9e0090 msgr2=0x55dccba81180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:06.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.211+0000 7fe635ffb700 0 client.28862 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:06.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.213+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 shutdown_connections 2023-09-10T01:17:06.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.218+0000 7fe66861db80 1 -- 172.21.15.3:0/1445953868 wait complete. 2023-09-10T01:17:06.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.222+0000 7fe66861db80 20 client.28862 trim_cache size 0 max 16384 2023-09-10T01:17:06.234 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:17:06.235 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:17:06.235 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:17:06.236 INFO:teuthology.orchestra.run.smithi003.stdout: "0e47f32e-255d-43b8-9e93-e96155881931": { 2023-09-10T01:17:06.236 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:17:06.236 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "0e47f32e-255d-43b8-9e93-e96155881931", 2023-09-10T01:17:06.236 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:17:06.237 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:17:06.237 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:17:06.237 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:17:06.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.238+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6380a09c0 msgr2=0x7fe6380a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.238+0000 7fe654bff700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6380a09c0 0x7fe6380a2e80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe64000a830 tx=0x7fe640008040 comp rx=0 tx=0).stop 2023-09-10T01:17:06.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.238+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 msgr2=0x7fe65011aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:06.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.238+0000 7fe654bff700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 0x7fe65011aea0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b530 tx=0x7fe64400b8f0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.243+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 shutdown_connections 2023-09-10T01:17:06.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.243+0000 7fe654bff700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6380a09c0 0x7fe6380a2e80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.243+0000 7fe654bff700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe650107600 0x7fe65011aea0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.243+0000 7fe654bff700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe65010f1b0 0x7fe650116020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.243+0000 7fe654bff700 1 --2- 172.21.15.3:0/551031483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65010edd0 0x7fe65011b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:06.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.243+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 >> 172.21.15.3:0/551031483 conn(0x7fe650071dd0 msgr2=0x7fe650109510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:06.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.248+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 shutdown_connections 2023-09-10T01:17:06.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:06.254+0000 7fe654bff700 1 -- 172.21.15.3:0/551031483 wait complete. 2023-09-10T01:17:06.303 INFO:tasks.cephfs.filesystem:scrub status for tag:0e47f32e-255d-43b8-9e93-e96155881931 - {'path': '/', 'tag': '0e47f32e-255d-43b8-9e93-e96155881931', 'options': 'recursive,force'} 2023-09-10T01:17:06.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:06 smithi003 ceph-mon[97654]: pgmap v434: 129 pgs: 129 active+clean; 627 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.9 MiB/s wr, 556 op/s 2023-09-10T01:17:06.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:06 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:17:06.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:06 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:06.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:06 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:06.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:06 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:06.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:06 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:06 smithi136 ceph-mon[133708]: pgmap v434: 129 pgs: 129 active+clean; 627 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.9 MiB/s wr, 556 op/s 2023-09-10T01:17:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:06 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:17:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:06 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:06 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:06.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:06 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:06.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:06 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:06.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:06 smithi092 ceph-mon[134067]: pgmap v434: 129 pgs: 129 active+clean; 627 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.9 MiB/s wr, 556 op/s 2023-09-10T01:17:06.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:06 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:17:06.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:06 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:06.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:06 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:06.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:06 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:06.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:06 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.170 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24476 30.99 MB/sec warmup 48 sec latency 1320.476 ms 2023-09-10T01:17:07.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:07 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:07.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:07 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:07 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:07.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:07 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:07 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:07 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:07.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:07 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:07 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:07.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:07 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:07 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:07 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:07.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:07 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:07 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:07.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:07 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:07.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:07 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:08.170 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24484 30.40 MB/sec warmup 49 sec latency 480.512 ms 2023-09-10T01:17:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:08 smithi136 ceph-mon[133708]: pgmap v435: 129 pgs: 129 active+clean; 627 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.9 MiB/s wr, 556 op/s 2023-09-10T01:17:08.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:08 smithi092 ceph-mon[134067]: pgmap v435: 129 pgs: 129 active+clean; 627 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.9 MiB/s wr, 556 op/s 2023-09-10T01:17:08.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:08 smithi003 ceph-mon[97654]: pgmap v435: 129 pgs: 129 active+clean; 627 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.9 MiB/s wr, 556 op/s 2023-09-10T01:17:09.170 INFO:tasks.workunit.client.0.smithi003.stdout: 10 24675 30.04 MB/sec warmup 50 sec latency 674.379 ms 2023-09-10T01:17:09.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:09 smithi136 ceph-mon[133708]: pgmap v436: 129 pgs: 129 active+clean; 624 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 358 op/s 2023-09-10T01:17:09.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:09 smithi092 ceph-mon[134067]: pgmap v436: 129 pgs: 129 active+clean; 624 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 358 op/s 2023-09-10T01:17:09.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:09 smithi003 ceph-mon[97654]: pgmap v436: 129 pgs: 129 active+clean; 624 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 358 op/s 2023-09-10T01:17:10.171 INFO:tasks.workunit.client.0.smithi003.stdout: 10 25028 29.89 MB/sec warmup 51 sec latency 553.221 ms 2023-09-10T01:17:11.173 INFO:tasks.workunit.client.0.smithi003.stdout: 10 25111 29.37 MB/sec warmup 52 sec latency 696.352 ms 2023-09-10T01:17:12.173 INFO:tasks.workunit.client.0.smithi003.stdout: 10 25409 29.15 MB/sec warmup 53 sec latency 770.522 ms 2023-09-10T01:17:12.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:12 smithi003 ceph-mon[97654]: pgmap v437: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 616 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 606 op/s 2023-09-10T01:17:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:12 smithi136 ceph-mon[133708]: pgmap v437: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 616 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 606 op/s 2023-09-10T01:17:12.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:12 smithi092 ceph-mon[134067]: pgmap v437: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 616 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 606 op/s 2023-09-10T01:17:13.173 INFO:tasks.workunit.client.0.smithi003.stdout: 10 25801 28.86 MB/sec warmup 54 sec latency 118.211 ms 2023-09-10T01:17:14.173 INFO:tasks.workunit.client.0.smithi003.stdout: 10 26532 29.07 MB/sec warmup 55 sec latency 60.139 ms 2023-09-10T01:17:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:14 smithi136 ceph-mon[133708]: pgmap v438: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 616 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 606 op/s 2023-09-10T01:17:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:14 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:17:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:14 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:17:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:14 smithi092 ceph-mon[134067]: pgmap v438: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 616 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 606 op/s 2023-09-10T01:17:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:14 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:17:14.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:14 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:17:14.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:14 smithi003 ceph-mon[97654]: pgmap v438: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 616 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 606 op/s 2023-09-10T01:17:14.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:14 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:17:14.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:14 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:17:15.173 INFO:tasks.workunit.client.0.smithi003.stdout: 10 27616 29.61 MB/sec warmup 56 sec latency 42.031 ms 2023-09-10T01:17:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:15 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:17:15.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:15 smithi136 ceph-mon[133708]: osdmap e107: 12 total, 12 up, 12 in 2023-09-10T01:17:15.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:15 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:17:15.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:15 smithi003 ceph-mon[97654]: osdmap e107: 12 total, 12 up, 12 in 2023-09-10T01:17:15.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:15 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:17:15.813 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:15 smithi092 ceph-mon[134067]: osdmap e107: 12 total, 12 up, 12 in 2023-09-10T01:17:16.173 INFO:tasks.workunit.client.0.smithi003.stdout: 10 28005 29.49 MB/sec warmup 57 sec latency 161.489 ms 2023-09-10T01:17:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:16 smithi136 ceph-mon[133708]: pgmap v439: 129 pgs: 129 active+clean; 636 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 758 op/s 2023-09-10T01:17:16.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:16 smithi092 ceph-mon[134067]: pgmap v439: 129 pgs: 129 active+clean; 636 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 758 op/s 2023-09-10T01:17:16.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:16 smithi003 ceph-mon[97654]: pgmap v439: 129 pgs: 129 active+clean; 636 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 11 MiB/s wr, 758 op/s 2023-09-10T01:17:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.959+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/3053002933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a9410ed00 msgr2=0x7f1a9410cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.959+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/3053002933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a9410ed00 0x7f1a9410cac0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0098f0 tx=0x7f1a7c009c00 comp rx=0 tx=0).stop 2023-09-10T01:17:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.960+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/3053002933 shutdown_connections 2023-09-10T01:17:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.960+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/3053002933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a94107600 0x7f1a94107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.960+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/3053002933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a9410ed00 0x7f1a9410cac0 secure :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0098f0 tx=0x7f1a7c009c00 comp rx=0 tx=0).stop 2023-09-10T01:17:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.960+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/3053002933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a9410f170 0x7f1a9410d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.960+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/3053002933 >> 172.21.15.3:0/3053002933 conn(0x7f1a9406c6c0 msgr2=0x7f1a9406cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.965+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/3053002933 shutdown_connections 2023-09-10T01:17:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.966+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/3053002933 wait complete. 2023-09-10T01:17:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.967+0000 7f1a9a5ff700 1 Processor -- start 2023-09-10T01:17:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.967+0000 7f1a9a5ff700 1 -- start start 2023-09-10T01:17:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.968+0000 7f1a9a5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a94107600 0x7f1a941ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.968+0000 7f1a9a5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a941abd50 0x7f1a941a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.968+0000 7f1a9a5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a941a5e10 0x7f1a941a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.968+0000 7f1a9a5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a941ac3b0 con 0x7f1a94107600 2023-09-10T01:17:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.968+0000 7f1a9a5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a941b2620 con 0x7f1a941a5e10 2023-09-10T01:17:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.968+0000 7f1a9a5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a941b27e0 con 0x7f1a941abd50 2023-09-10T01:17:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.969+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a941abd50 0x7f1a941a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.969+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a941abd50 0x7f1a941a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43760/0 (socket says 172.21.15.3:43760) 2023-09-10T01:17:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.969+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/1160044687 learned_addr learned my addr 172.21.15.3:0/1160044687 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.970+0000 7f1a93fff700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a94107600 0x7f1a941ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.970+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/1160044687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a941a5e10 msgr2=0x7f1a941a62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.970+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a941a5e10 0x7f1a941a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.970+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a94107600 msgr2=0x7f1a941ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.970+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a94107600 0x7f1a941ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.970+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/1160044687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c009520 con 0x7f1a941abd50 2023-09-10T01:17:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.971+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a941abd50 0x7f1a941a58d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400ef40 tx=0x7f1a8400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.972+0000 7f1a91ffb700 1 -- 172.21.15.3:0/1160044687 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8400c9f0 con 0x7f1a941abd50 2023-09-10T01:17:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.972+0000 7f1a91ffb700 1 -- 172.21.15.3:0/1160044687 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a8400f040 con 0x7f1a941abd50 2023-09-10T01:17:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.972+0000 7f1a91ffb700 1 -- 172.21.15.3:0/1160044687 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84014480 con 0x7f1a941abd50 2023-09-10T01:17:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.974+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a941b2ac0 con 0x7f1a941abd50 2023-09-10T01:17:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.975+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a941b2f50 con 0x7f1a941abd50 2023-09-10T01:17:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.983+0000 7f1a91ffb700 1 -- 172.21.15.3:0/1160044687 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f1a8400cbe0 con 0x7f1a941abd50 2023-09-10T01:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.989+0000 7f1aade3ab80 1 Processor -- start 2023-09-10T01:17:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.990+0000 7f1aade3ab80 1 -- start start 2023-09-10T01:17:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 0x5637750a6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563774fff6d0 con 0x5637750a6300 2023-09-10T01:17:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563774ffcff0 con 0x5637750a6cc0 2023-09-10T01:17:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563774ff88e0 con 0x5637750ac1a0 2023-09-10T01:17:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1a91ffb700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1a800a0940 0x7f1a800a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.994+0000 7f1a91ffb700 1 -- 172.21.15.3:0/1160044687 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f1a840cbf20 con 0x7f1a941abd50 2023-09-10T01:17:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43764/0 (socket says 172.21.15.3:43764) 2023-09-10T01:17:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 -- 172.21.15.3:0/242779124 learned_addr learned my addr 172.21.15.3:0/242779124 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1a800a0940 0x7f1a800a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 -- 172.21.15.3:0/242779124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 msgr2=0x5637750b2f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 --2- 172.21.15.3:0/242779124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b2f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 -- 172.21.15.3:0/242779124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 msgr2=0x5637750a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.995+0000 7f1a98b9c700 1 --2- 172.21.15.3:0/242779124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 0x5637750a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.996+0000 7f1a98b9c700 1 -- 172.21.15.3:0/242779124 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563774e24d80 con 0x5637750ac1a0 2023-09-10T01:17:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.997+0000 7f1a98b9c700 1 --2- 172.21.15.3:0/242779124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750ac580 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c012650 tx=0x7f1a8c012960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbb4ddfaefe2fcff server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.997+0000 7f1a8a7fc700 1 -- 172.21.15.3:0/242779124 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c020070 con 0x5637750ac1a0 2023-09-10T01:17:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.998+0000 7f1a8a7fc700 1 -- 172.21.15.3:0/242779124 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a8c018070 con 0x5637750ac1a0 2023-09-10T01:17:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.998+0000 7f1a8a7fc700 1 -- 172.21.15.3:0/242779124 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c004c50 con 0x5637750ac1a0 2023-09-10T01:17:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.998+0000 7f1a98b9c700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1a800a0940 0x7f1a800a2e00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c012d60 tx=0x7f1a8c01b000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:17.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.999+0000 7f1aade3ab80 1 -- 172.21.15.3:0/242779124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 msgr2=0x5637750ac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.999+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/242779124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750ac580 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c012650 tx=0x7f1a8c012960 comp rx=0 tx=0).stop 2023-09-10T01:17:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.999+0000 7f1aade3ab80 1 -- 172.21.15.3:0/242779124 shutdown_connections 2023-09-10T01:17:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.999+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/242779124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750ac580 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.999+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/242779124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.999+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/242779124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 0x5637750a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:16.999+0000 7f1aade3ab80 1 -- 172.21.15.3:0/242779124 >> 172.21.15.3:0/242779124 conn(0x56377501b030 msgr2=0x5637750a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.000+0000 7f1aade3ab80 1 -- 172.21.15.3:0/242779124 shutdown_connections 2023-09-10T01:17:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.000+0000 7f1aade3ab80 1 -- 172.21.15.3:0/242779124 wait complete. 2023-09-10T01:17:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.000+0000 7f1aade3ab80 1 -- start start 2023-09-10T01:17:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.001+0000 7f1aade3ab80 1 Processor -- start 2023-09-10T01:17:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 0x5637750b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750b0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563774f96b10 con 0x5637750a6300 2023-09-10T01:17:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563774f96810 con 0x5637750a6cc0 2023-09-10T01:17:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563774e7a8c0 con 0x5637750ac1a0 2023-09-10T01:17:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1aade3ab80 10 client.0 _ll_register_callbacks cb 0x563774ee0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.002+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.003+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48546/0 (socket says 172.21.15.3:48546) 2023-09-10T01:17:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.003+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/3031817469 learned_addr learned my addr 172.21.15.3:0/3031817469 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.003+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/3031817469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 msgr2=0x5637750b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.003+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.003+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/3031817469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 msgr2=0x5637750b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.003+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 0x5637750b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.003+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/3031817469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563774e0dc00 con 0x5637750a6cc0 2023-09-10T01:17:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.004+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b0080 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1a84095ae0 tx=0x7f1a840989d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9503db8d8c7943cc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.004+0000 7f1a737fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.004+0000 7f1a737fe700 1 -- 172.21.15.3:0/3031817469 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a840b1400 con 0x5637750a6cc0 2023-09-10T01:17:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.004+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563774f50490 con 0x5637750a6cc0 2023-09-10T01:17:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.004+0000 7f1a737fe700 1 -- 172.21.15.3:0/3031817469 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f1a84099070 con 0x5637750a6cc0 2023-09-10T01:17:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.005+0000 7f1a737fe700 1 -- 172.21.15.3:0/3031817469 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a840b7050 con 0x5637750a6cc0 2023-09-10T01:17:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.005+0000 7f1aade3ab80 10 client.28868 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.005+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x563774e4b1f0 con 0x5637750a6cc0 2023-09-10T01:17:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.005+0000 7f1a737fe700 1 -- 172.21.15.3:0/3031817469 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563774f50490 con 0x5637750a6cc0 2023-09-10T01:17:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1a737fe700 1 -- 172.21.15.3:0/3031817469 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f1a84098560 con 0x5637750a6cc0 2023-09-10T01:17:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1aade3ab80 10 client.28868 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1aade3ab80 10 client.28868 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1aade3ab80 20 client.28868 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5637750d1410 0x5637750d3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1aade3ab80 4 client.28868 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56377500d840 con 0x5637750d1410 2023-09-10T01:17:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.006+0000 7f1a93fff700 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5637750d1410 0x5637750d3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.007+0000 7f1a93fff700 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5637750d1410 0x5637750d3850 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4740b4aab55bd09 server_cookie=6711cde06781165d in_seq=0 out_seq=0 2023-09-10T01:17:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.007+0000 7f1a737fe700 10 client.28868 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.011+0000 7f1a737fe700 1 -- 172.21.15.3:0/3031817469 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56377500d840 con 0x5637750d1410 2023-09-10T01:17:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.011+0000 7f1a737fe700 10 client.28868 handle_command_reply: tid=0 2023-09-10T01:17:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.011+0000 7f1aade3ab80 1 client.28868 shutdown 2023-09-10T01:17:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.011+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 msgr2=0x5637750b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.011+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b0080 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1a84095ae0 tx=0x7f1a840989d0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 shutdown_connections 2023-09-10T01:17:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5637750d1410 0x5637750d3850 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x5637750b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6300 0x5637750b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3031817469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6cc0 0x5637750b0080 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 >> 172.21.15.3:0/3031817469 conn(0x56377501b030 msgr2=0x5637750a07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1a737fe700 0 client.28868 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 shutdown_connections 2023-09-10T01:17:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.012+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3031817469 wait complete. 2023-09-10T01:17:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.013+0000 7f1aade3ab80 20 client.28868 trim_cache size 0 max 16384 2023-09-10T01:17:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.038+0000 7f1aade3ab80 1 Processor -- start 2023-09-10T01:17:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.039+0000 7f1aade3ab80 1 -- start start 2023-09-10T01:17:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.039+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637750b7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.039+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750a6cc0 0x5637750a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.039+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750ac1a0 0x5637750b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.039+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637750b1030 con 0x5637750ac1a0 2023-09-10T01:17:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.039+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637750a2b80 con 0x5637750a6300 2023-09-10T01:17:17.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.039+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637750ac580 con 0x5637750a6cc0 2023-09-10T01:17:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.040+0000 7f1a98b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750ac1a0 0x5637750b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.040+0000 7f1a98b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750ac1a0 0x5637750b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47034/0 (socket says 172.21.15.3:47034) 2023-09-10T01:17:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.040+0000 7f1a98b9c700 1 -- 172.21.15.3:0/3173667036 learned_addr learned my addr 172.21.15.3:0/3173667036 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.040+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750a6cc0 0x5637750a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.040+0000 7f1a93fff700 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637750b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/3173667036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 msgr2=0x5637750b7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:17.043 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637750b7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/3173667036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750ac1a0 msgr2=0x5637750b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750ac1a0 0x5637750b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8b7fe700 1 -- 172.21.15.3:0/3173667036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563774e23250 con 0x5637750a6cc0 2023-09-10T01:17:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a93fff700 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637750b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:17:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750a6cc0 0x5637750a0f20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1a840060b0 tx=0x7f1a84097620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1603d630d3b75566 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8affd700 1 -- 172.21.15.3:0/3173667036 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84099070 con 0x5637750a6cc0 2023-09-10T01:17:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8affd700 1 -- 172.21.15.3:0/3173667036 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a8409cea0 con 0x5637750a6cc0 2023-09-10T01:17:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.041+0000 7f1a8affd700 1 -- 172.21.15.3:0/3173667036 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8409dbe0 con 0x5637750a6cc0 2023-09-10T01:17:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.042+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3173667036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750a6cc0 msgr2=0x5637750a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.042+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750a6cc0 0x5637750a0f20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1a840060b0 tx=0x7f1a84097620 comp rx=0 tx=0).stop 2023-09-10T01:17:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.042+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3173667036 shutdown_connections 2023-09-10T01:17:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.042+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750ac1a0 0x5637750b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.042+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637750b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.042+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/3173667036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750a6cc0 0x5637750a0f20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.042+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3173667036 >> 172.21.15.3:0/3173667036 conn(0x56377501b030 msgr2=0x5637750aa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.043+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3173667036 shutdown_connections 2023-09-10T01:17:17.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.043+0000 7f1aade3ab80 1 -- 172.21.15.3:0/3173667036 wait complete. 2023-09-10T01:17:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.044+0000 7f1aade3ab80 1 -- start start 2023-09-10T01:17:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.044+0000 7f1aade3ab80 1 Processor -- start 2023-09-10T01:17:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637751193e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6cc0 0x563775119920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1aade3ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x56377511e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377511e780 con 0x5637750a6cc0 2023-09-10T01:17:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377511e8c0 con 0x5637750a6300 2023-09-10T01:17:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1aade3ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377511ea80 con 0x5637750ac1a0 2023-09-10T01:17:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1aade3ab80 10 client.0 _ll_register_callbacks cb 0x7f1a84095460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.045+0000 7f1a93fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637751193e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.046+0000 7f1a93fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637751193e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48572/0 (socket says 172.21.15.3:48572) 2023-09-10T01:17:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.046+0000 7f1a93fff700 1 -- 172.21.15.3:0/2090896389 learned_addr learned my addr 172.21.15.3:0/2090896389 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.046+0000 7f1a93fff700 1 -- 172.21.15.3:0/2090896389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 msgr2=0x56377511e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.046+0000 7f1a93fff700 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x56377511e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.046+0000 7f1a93fff700 1 -- 172.21.15.3:0/2090896389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6cc0 msgr2=0x563775119920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.046+0000 7f1a93fff700 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6cc0 0x563775119920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.046+0000 7f1a93fff700 1 -- 172.21.15.3:0/2090896389 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563774e4b1f0 con 0x5637750a6300 2023-09-10T01:17:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.047+0000 7f1a93fff700 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637751193e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009230 tx=0x7f1a7c0234d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed2b70c7c9487594 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.047+0000 7f1a737fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.047+0000 7f1a737fe700 1 -- 172.21.15.3:0/2090896389 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c023d90 con 0x5637750a6300 2023-09-10T01:17:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.047+0000 7f1a737fe700 1 -- 172.21.15.3:0/2090896389 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9527+0+0 (secure 0 0 0) 0x7f1a7c000e40 con 0x5637750a6300 2023-09-10T01:17:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.047+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a8c018070 con 0x5637750a6300 2023-09-10T01:17:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.048+0000 7f1a737fe700 1 -- 172.21.15.3:0/2090896389 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a7c00f460 con 0x5637750a6300 2023-09-10T01:17:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.048+0000 7f1aade3ab80 10 client.28874 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.048+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x563774ef1a20 con 0x5637750a6300 2023-09-10T01:17:17.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.048+0000 7f1a737fe700 1 -- 172.21.15.3:0/2090896389 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a8c018070 con 0x5637750a6300 2023-09-10T01:17:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.048+0000 7f1a737fe700 1 -- 172.21.15.3:0/2090896389 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f1a7c015020 con 0x5637750a6300 2023-09-10T01:17:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.048+0000 7f1aade3ab80 10 client.28874 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.048+0000 7f1aade3ab80 10 client.28874 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.049+0000 7f1aade3ab80 20 client.28874 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.049+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563775126290 0x5637751286d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.049+0000 7f1aade3ab80 4 client.28874 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:17:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.049+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563774ff18c0 con 0x563775126290 2023-09-10T01:17:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.049+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563775126290 0x5637751286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.050+0000 7f1a8b7fe700 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563775126290 0x5637751286d0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b2ec9d00ccc8621 server_cookie=ef924a719c65612f in_seq=0 out_seq=0 2023-09-10T01:17:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.050+0000 7f1a737fe700 10 client.28874 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:17.174 INFO:tasks.workunit.client.0.smithi003.stdout: 10 28547 29.62 MB/sec warmup 58 sec latency 99.034 ms 2023-09-10T01:17:17.385 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.384+0000 7f1a737fe700 1 -- 172.21.15.3:0/2090896389 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563774ff18c0 con 0x563775126290 2023-09-10T01:17:17.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.384+0000 7f1a737fe700 10 client.28874 handle_command_reply: tid=0 2023-09-10T01:17:17.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.384+0000 7f1aade3ab80 1 client.28874 shutdown 2023-09-10T01:17:17.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 msgr2=0x5637751193e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:17.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637751193e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009230 tx=0x7f1a7c0234d0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 shutdown_connections 2023-09-10T01:17:17.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x563775126290 0x5637751286d0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5637750ac1a0 0x56377511e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1a737fe700 0 client.28874 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5637750a6300 0x5637751193e0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 --2- 172.21.15.3:0/2090896389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637750a6cc0 0x563775119920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 >> 172.21.15.3:0/2090896389 conn(0x56377501b030 msgr2=0x5637750a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 shutdown_connections 2023-09-10T01:17:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.387+0000 7f1aade3ab80 1 -- 172.21.15.3:0/2090896389 wait complete. 2023-09-10T01:17:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.388+0000 7f1aade3ab80 20 client.28874 trim_cache size 0 max 16384 2023-09-10T01:17:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:17:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:17:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:17:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:17:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1a800a0940 msgr2=0x7f1a800a2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1a800a0940 0x7f1a800a2e00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c012d60 tx=0x7f1a8c01b000 comp rx=0 tx=0).stop 2023-09-10T01:17:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a941abd50 msgr2=0x7f1a941a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a941abd50 0x7f1a941a58d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400ef40 tx=0x7f1a8400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 shutdown_connections 2023-09-10T01:17:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f1a800a0940 0x7f1a800a2e00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a941a5e10 0x7f1a941a62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1a94107600 0x7f1a941ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 --2- 172.21.15.3:0/1160044687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a941abd50 0x7f1a941a58d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 >> 172.21.15.3:0/1160044687 conn(0x7f1a9406c6c0 msgr2=0x7f1a94070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 shutdown_connections 2023-09-10T01:17:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:17.389+0000 7f1a9a5ff700 1 -- 172.21.15.3:0/1160044687 wait complete. 2023-09-10T01:17:18.174 INFO:tasks.workunit.client.0.smithi003.stdout: 10 29262 29.66 MB/sec warmup 59 sec latency 210.593 ms 2023-09-10T01:17:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:18 smithi136 ceph-mon[133708]: pgmap v441: 129 pgs: 129 active+clean; 636 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 11 MiB/s wr, 769 op/s 2023-09-10T01:17:18.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:18 smithi092 ceph-mon[134067]: pgmap v441: 129 pgs: 129 active+clean; 636 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 11 MiB/s wr, 769 op/s 2023-09-10T01:17:18.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:18 smithi003 ceph-mon[97654]: pgmap v441: 129 pgs: 129 active+clean; 636 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 11 MiB/s wr, 769 op/s 2023-09-10T01:17:19.174 INFO:tasks.workunit.client.0.smithi003.stdout: 10 29770 29.58 MB/sec warmup 60 sec latency 184.728 ms 2023-09-10T01:17:20.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 30682 29.83 MB/sec warmup 61 sec latency 78.736 ms 2023-09-10T01:17:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:20 smithi136 ceph-mon[133708]: pgmap v442: 129 pgs: 129 active+clean; 641 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 801 op/s 2023-09-10T01:17:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:20 smithi136 ceph-mon[133708]: osdmap e108: 12 total, 12 up, 12 in 2023-09-10T01:17:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:20 smithi092 ceph-mon[134067]: pgmap v442: 129 pgs: 129 active+clean; 641 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 801 op/s 2023-09-10T01:17:20.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:20 smithi092 ceph-mon[134067]: osdmap e108: 12 total, 12 up, 12 in 2023-09-10T01:17:20.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:20 smithi003 ceph-mon[97654]: pgmap v442: 129 pgs: 129 active+clean; 641 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 801 op/s 2023-09-10T01:17:20.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:20 smithi003 ceph-mon[97654]: osdmap e108: 12 total, 12 up, 12 in 2023-09-10T01:17:21.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 31586 30.22 MB/sec warmup 62 sec latency 79.385 ms 2023-09-10T01:17:22.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 32386 30.55 MB/sec warmup 63 sec latency 71.704 ms 2023-09-10T01:17:22.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:22 smithi136 ceph-mon[133708]: pgmap v444: 129 pgs: 129 active+clean; 661 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 19 MiB/s wr, 1.06k op/s 2023-09-10T01:17:22.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:22 smithi092 ceph-mon[134067]: pgmap v444: 129 pgs: 129 active+clean; 661 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 19 MiB/s wr, 1.06k op/s 2023-09-10T01:17:22.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:22 smithi003 ceph-mon[97654]: pgmap v444: 129 pgs: 129 active+clean; 661 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 19 MiB/s wr, 1.06k op/s 2023-09-10T01:17:23.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 32966 30.39 MB/sec warmup 64 sec latency 87.563 ms 2023-09-10T01:17:24.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 33667 30.50 MB/sec warmup 65 sec latency 148.850 ms 2023-09-10T01:17:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:24 smithi136 ceph-mon[133708]: pgmap v445: 129 pgs: 129 active+clean; 661 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.7 MiB/s wr, 485 op/s 2023-09-10T01:17:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:24 smithi092 ceph-mon[134067]: pgmap v445: 129 pgs: 129 active+clean; 661 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.7 MiB/s wr, 485 op/s 2023-09-10T01:17:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:24 smithi003 ceph-mon[97654]: pgmap v445: 129 pgs: 129 active+clean; 661 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.7 MiB/s wr, 485 op/s 2023-09-10T01:17:25.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 34552 30.73 MB/sec warmup 66 sec latency 51.555 ms 2023-09-10T01:17:26.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 35427 31.19 MB/sec warmup 67 sec latency 60.267 ms 2023-09-10T01:17:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:26 smithi136 ceph-mon[133708]: pgmap v446: 129 pgs: 129 active+clean; 688 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.7 MiB/s wr, 707 op/s 2023-09-10T01:17:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:26 smithi092 ceph-mon[134067]: pgmap v446: 129 pgs: 129 active+clean; 688 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.7 MiB/s wr, 707 op/s 2023-09-10T01:17:26.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:26 smithi003 ceph-mon[97654]: pgmap v446: 129 pgs: 129 active+clean; 688 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.7 MiB/s wr, 707 op/s 2023-09-10T01:17:27.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 36139 31.17 MB/sec warmup 68 sec latency 38.038 ms 2023-09-10T01:17:28.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 36679 31.10 MB/sec warmup 69 sec latency 101.896 ms 2023-09-10T01:17:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:28 smithi136 ceph-mon[133708]: pgmap v447: 129 pgs: 129 active+clean; 688 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 8.0 MiB/s wr, 650 op/s 2023-09-10T01:17:28.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:28 smithi092 ceph-mon[134067]: pgmap v447: 129 pgs: 129 active+clean; 688 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 8.0 MiB/s wr, 650 op/s 2023-09-10T01:17:28.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:28 smithi003 ceph-mon[97654]: pgmap v447: 129 pgs: 129 active+clean; 688 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 8.0 MiB/s wr, 650 op/s 2023-09-10T01:17:29.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 37706 31.39 MB/sec warmup 70 sec latency 50.138 ms 2023-09-10T01:17:29.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:29 smithi136 ceph-mon[133708]: pgmap v448: 129 pgs: 129 active+clean; 694 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.9 MiB/s wr, 621 op/s 2023-09-10T01:17:29.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:29 smithi092 ceph-mon[134067]: pgmap v448: 129 pgs: 129 active+clean; 694 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.9 MiB/s wr, 621 op/s 2023-09-10T01:17:29.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:29 smithi003 ceph-mon[97654]: pgmap v448: 129 pgs: 129 active+clean; 694 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.9 MiB/s wr, 621 op/s 2023-09-10T01:17:30.175 INFO:tasks.workunit.client.0.smithi003.stdout: 10 38503 31.72 MB/sec warmup 71 sec latency 99.383 ms 2023-09-10T01:17:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.607+0000 7f18421ff700 1 -- 172.21.15.3:0/2539843036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 msgr2=0x7f183c067fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.607+0000 7f18421ff700 1 --2- 172.21.15.3:0/2539843036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 0x7f183c067fe0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1830009990 tx=0x7f1830009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.608+0000 7f18421ff700 1 -- 172.21.15.3:0/2539843036 shutdown_connections 2023-09-10T01:17:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.608+0000 7f18421ff700 1 --2- 172.21.15.3:0/2539843036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f183c068520 0x7f183c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.608+0000 7f18421ff700 1 --2- 172.21.15.3:0/2539843036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f183c1066a0 0x7f183c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.608+0000 7f18421ff700 1 --2- 172.21.15.3:0/2539843036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 0x7f183c067fe0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.608+0000 7f18421ff700 1 -- 172.21.15.3:0/2539843036 >> 172.21.15.3:0/2539843036 conn(0x7f183c074fe0 msgr2=0x7f183c0753f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.608+0000 7f18421ff700 1 -- 172.21.15.3:0/2539843036 shutdown_connections 2023-09-10T01:17:30.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.608+0000 7f18421ff700 1 -- 172.21.15.3:0/2539843036 wait complete. 2023-09-10T01:17:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.609+0000 7f18421ff700 1 Processor -- start 2023-09-10T01:17:30.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.611+0000 7f18421ff700 1 -- start start 2023-09-10T01:17:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.611+0000 7f18421ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 0x7f183c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.611+0000 7f18421ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f183c068520 0x7f183c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.611+0000 7f18421ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f183c1066a0 0x7f183c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.611+0000 7f18421ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c198d00 con 0x7f183c068520 2023-09-10T01:17:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.611+0000 7f18421ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c198e70 con 0x7f183c067b60 2023-09-10T01:17:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.611+0000 7f18421ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c199030 con 0x7f183c1066a0 2023-09-10T01:17:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 0x7f183c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 0x7f183c19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35636/0 (socket says 172.21.15.3:35636) 2023-09-10T01:17:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183b7fe700 1 -- 172.21.15.3:0/2593111307 learned_addr learned my addr 172.21.15.3:0/2593111307 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183affd700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f183c068520 0x7f183c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183bfff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f183c1066a0 0x7f183c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183bfff700 1 -- 172.21.15.3:0/2593111307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 msgr2=0x7f183c19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183bfff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 0x7f183c19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183bfff700 1 -- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f183c068520 msgr2=0x7f183c19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183bfff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f183c068520 0x7f183c19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.612+0000 7f183bfff700 1 -- 172.21.15.3:0/2593111307 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18300095f0 con 0x7f183c1066a0 2023-09-10T01:17:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.613+0000 7f183bfff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f183c1066a0 0x7f183c198690 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1834007f10 tx=0x7f183400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.613+0000 7f1838ff9700 1 -- 172.21.15.3:0/2593111307 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1834004db0 con 0x7f183c1066a0 2023-09-10T01:17:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.613+0000 7f1838ff9700 1 -- 172.21.15.3:0/2593111307 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f183400f040 con 0x7f183c1066a0 2023-09-10T01:17:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.613+0000 7f1838ff9700 1 -- 172.21.15.3:0/2593111307 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1834005990 con 0x7f183c1066a0 2023-09-10T01:17:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.613+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c199310 con 0x7f183c1066a0 2023-09-10T01:17:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.613+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c1a5610 con 0x7f183c1066a0 2023-09-10T01:17:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.618+0000 7f1838ff9700 1 -- 172.21.15.3:0/2593111307 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f183400db20 con 0x7f183c1066a0 2023-09-10T01:17:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.619+0000 7f1855bfcb80 1 Processor -- start 2023-09-10T01:17:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.619+0000 7f1855bfcb80 1 -- start start 2023-09-10T01:17:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3789b0 0x56208a378d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a372f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3734d0 0x56208a37f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a2c36d0 con 0x56208a3789b0 2023-09-10T01:17:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a2c0ff0 con 0x56208a3734d0 2023-09-10T01:17:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a2bc8e0 con 0x56208a372b10 2023-09-10T01:17:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f183affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3734d0 0x56208a37f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f183affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3734d0 0x56208a37f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35640/0 (socket says 172.21.15.3:35640) 2023-09-10T01:17:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f183affd700 1 -- 172.21.15.3:0/3934874494 learned_addr learned my addr 172.21.15.3:0/3934874494 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.620+0000 7f183affd700 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3789b0 0x56208a378d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f1838ff9700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18240a07b0 0x7f18240a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f1838ff9700 1 -- 172.21.15.3:0/2593111307 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f18340cbc60 con 0x7f183c1066a0 2023-09-10T01:17:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f183affd700 1 -- 172.21.15.3:0/3934874494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 msgr2=0x56208a372f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f183affd700 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a372f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f183affd700 1 -- 172.21.15.3:0/3934874494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3789b0 msgr2=0x56208a378d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f183affd700 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3789b0 0x56208a378d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f183affd700 1 -- 172.21.15.3:0/3934874494 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56208a0e8d80 con 0x56208a3734d0 2023-09-10T01:17:30.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f183bfff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18240a07b0 0x7f18240a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.621+0000 7f183affd700 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3734d0 0x56208a37f740 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1830015000 tx=0x7f183001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b19cc83811c94b90 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:30.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.622+0000 7f183bfff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18240a07b0 0x7f18240a2c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f18340a0830 tx=0x7f1834004f10 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:30.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.622+0000 7f1821ffb700 1 -- 172.21.15.3:0/3934874494 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18300056f0 con 0x56208a3734d0 2023-09-10T01:17:30.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.622+0000 7f1821ffb700 1 -- 172.21.15.3:0/3934874494 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f183000bc60 con 0x56208a3734d0 2023-09-10T01:17:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.622+0000 7f1821ffb700 1 -- 172.21.15.3:0/3934874494 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830004c90 con 0x56208a3734d0 2023-09-10T01:17:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3934874494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3734d0 msgr2=0x56208a37f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3734d0 0x56208a37f740 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f1830015000 tx=0x7f183001a040 comp rx=0 tx=0).stop 2023-09-10T01:17:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3934874494 shutdown_connections 2023-09-10T01:17:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3734d0 0x56208a37f740 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3789b0 0x56208a378d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3934874494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a372f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3934874494 >> 172.21.15.3:0/3934874494 conn(0x56208a2df030 msgr2=0x56208a365560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3934874494 shutdown_connections 2023-09-10T01:17:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3934874494 wait complete. 2023-09-10T01:17:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.623+0000 7f1855bfcb80 1 -- start start 2023-09-10T01:17:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 1 Processor -- start 2023-09-10T01:17:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a372b10 0x56208a37f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a37c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3789b0 0x56208a37cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a25ab10 con 0x56208a3734d0 2023-09-10T01:17:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a25a810 con 0x56208a372b10 2023-09-10T01:17:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a13e8c0 con 0x56208a3789b0 2023-09-10T01:17:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f1855bfcb80 10 client.0 _ll_register_callbacks cb 0x56208a1a4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.624+0000 7f183affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a372b10 0x56208a37f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a372b10 0x56208a37f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35642/0 (socket says 172.21.15.3:35642) 2023-09-10T01:17:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 -- 172.21.15.3:0/3369557272 learned_addr learned my addr 172.21.15.3:0/3369557272 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3789b0 0x56208a37cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183b7fe700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a37c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 -- 172.21.15.3:0/3369557272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3789b0 msgr2=0x56208a37cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3789b0 0x56208a37cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 -- 172.21.15.3:0/3369557272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 msgr2=0x56208a37c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a37c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 -- 172.21.15.3:0/3369557272 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1830020040 con 0x56208a372b10 2023-09-10T01:17:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183b7fe700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a37c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f183affd700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a372b10 0x56208a37f5e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f1830006010 tx=0x7f18300047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a25487ce2df05f3f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.625+0000 7f181affd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.626+0000 7f181affd700 1 -- 172.21.15.3:0/3369557272 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183000f7b0 con 0x56208a372b10 2023-09-10T01:17:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.626+0000 7f181affd700 1 -- 172.21.15.3:0/3369557272 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f183000f910 con 0x56208a372b10 2023-09-10T01:17:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.626+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56208a214490 con 0x56208a372b10 2023-09-10T01:17:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.626+0000 7f181affd700 1 -- 172.21.15.3:0/3369557272 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1830005980 con 0x56208a372b10 2023-09-10T01:17:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.626+0000 7f1855bfcb80 10 client.28886 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.626+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56208a0d1c00 con 0x56208a372b10 2023-09-10T01:17:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f181affd700 1 -- 172.21.15.3:0/3369557272 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56208a214490 con 0x56208a372b10 2023-09-10T01:17:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f181affd700 1 -- 172.21.15.3:0/3369557272 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f183004ac90 con 0x56208a372b10 2023-09-10T01:17:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f1855bfcb80 10 client.28886 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f1855bfcb80 10 client.28886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f1855bfcb80 20 client.28886 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a38c6b0 0x56208a38ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f1855bfcb80 4 client.28886 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56208a2d1840 con 0x56208a38c6b0 2023-09-10T01:17:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.627+0000 7f183b7fe700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a38c6b0 0x56208a38ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.628+0000 7f181affd700 10 client.28886 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.628+0000 7f183b7fe700 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a38c6b0 0x56208a38ead0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ca9afb67e18d34 server_cookie=5aa7c85a10075b9a in_seq=0 out_seq=0 2023-09-10T01:17:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.629+0000 7f181affd700 1 -- 172.21.15.3:0/3369557272 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56208a2d1840 con 0x56208a38c6b0 2023-09-10T01:17:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.629+0000 7f181affd700 10 client.28886 handle_command_reply: tid=0 2023-09-10T01:17:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.629+0000 7f1855bfcb80 1 client.28886 shutdown 2023-09-10T01:17:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a372b10 msgr2=0x56208a37f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a372b10 0x56208a37f5e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f1830006010 tx=0x7f18300047d0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 shutdown_connections 2023-09-10T01:17:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a38c6b0 0x56208a38ead0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3789b0 0x56208a37cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a372b10 0x56208a37f5e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/3369557272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a37c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 >> 172.21.15.3:0/3369557272 conn(0x56208a2df030 msgr2=0x56208a3647a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f181affd700 0 client.28886 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 shutdown_connections 2023-09-10T01:17:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.630+0000 7f1855bfcb80 1 -- 172.21.15.3:0/3369557272 wait complete. 2023-09-10T01:17:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.631+0000 7f1855bfcb80 20 client.28886 trim_cache size 0 max 16384 2023-09-10T01:17:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.650+0000 7f1855bfcb80 1 Processor -- start 2023-09-10T01:17:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f1855bfcb80 1 -- start start 2023-09-10T01:17:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a372b10 0x56208a37c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3734d0 0x56208a37f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a3769a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a373d70 con 0x56208a372b10 2023-09-10T01:17:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a366320 con 0x56208a3789b0 2023-09-10T01:17:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a37aef0 con 0x56208a3734d0 2023-09-10T01:17:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3734d0 0x56208a37f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3734d0 0x56208a37f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:59142/0 (socket says 172.21.15.3:59142) 2023-09-10T01:17:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f183b7fe700 1 -- 172.21.15.3:0/336038374 learned_addr learned my addr 172.21.15.3:0/336038374 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f183affd700 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a372b10 0x56208a37c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.651+0000 7f183affd700 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a3769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.652+0000 7f183b7fe700 1 -- 172.21.15.3:0/336038374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 msgr2=0x56208a3769a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.652+0000 7f183b7fe700 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a3769a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.652+0000 7f183b7fe700 1 -- 172.21.15.3:0/336038374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a372b10 msgr2=0x56208a37c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.652+0000 7f183b7fe700 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a372b10 0x56208a37c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.652+0000 7f183b7fe700 1 -- 172.21.15.3:0/336038374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56208a10f1f0 con 0x56208a3734d0 2023-09-10T01:17:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.652+0000 7f183b7fe700 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3734d0 0x56208a37f110 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x56208a35ceb0 tx=0x7f182c01aaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a70075bb826f3d6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f18227fc700 1 -- 172.21.15.3:0/336038374 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c018040 con 0x56208a3734d0 2023-09-10T01:17:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f18227fc700 1 -- 172.21.15.3:0/336038374 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f182c01e070 con 0x56208a3734d0 2023-09-10T01:17:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f18227fc700 1 -- 172.21.15.3:0/336038374 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c004cd0 con 0x56208a3734d0 2023-09-10T01:17:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 -- 172.21.15.3:0/336038374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3734d0 msgr2=0x56208a37f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3734d0 0x56208a37f110 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x56208a35ceb0 tx=0x7f182c01aaf0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 -- 172.21.15.3:0/336038374 shutdown_connections 2023-09-10T01:17:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a3769a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a372b10 0x56208a37c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/336038374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a3734d0 0x56208a37f110 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 -- 172.21.15.3:0/336038374 >> 172.21.15.3:0/336038374 conn(0x56208a2df030 msgr2=0x56208a377910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.653+0000 7f1855bfcb80 1 -- 172.21.15.3:0/336038374 shutdown_connections 2023-09-10T01:17:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.654+0000 7f1855bfcb80 1 -- 172.21.15.3:0/336038374 wait complete. 2023-09-10T01:17:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.654+0000 7f1855bfcb80 1 -- start start 2023-09-10T01:17:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.654+0000 7f1855bfcb80 1 Processor -- start 2023-09-10T01:17:30.661 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a382030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a382570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f1855bfcb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a383440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a383a20 con 0x56208a3734d0 2023-09-10T01:17:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a383b90 con 0x56208a3789b0 2023-09-10T01:17:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f1855bfcb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56208a383d50 con 0x56208a372b10 2023-09-10T01:17:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f1855bfcb80 10 client.0 _ll_register_callbacks cb 0x7f1830004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.655+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a382570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183affd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a382030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a382570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36804/0 (socket says 172.21.15.3:36804) 2023-09-10T01:17:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183b7fe700 1 -- 172.21.15.3:0/4260293795 learned_addr learned my addr 172.21.15.3:0/4260293795 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183affd700 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a383440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183b7fe700 1 -- 172.21.15.3:0/4260293795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 msgr2=0x56208a382030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183b7fe700 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a382030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183b7fe700 1 -- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 msgr2=0x56208a383440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183b7fe700 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a383440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183b7fe700 1 -- 172.21.15.3:0/4260293795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56208a0e7250 con 0x56208a3734d0 2023-09-10T01:17:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.656+0000 7f183affd700 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a382030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.657+0000 7f183b7fe700 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a382570 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f182c009d40 tx=0x7f182c01a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27466934e230f242 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.657+0000 7f181affd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.657+0000 7f181affd700 1 -- 172.21.15.3:0/4260293795 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c004230 con 0x56208a3734d0 2023-09-10T01:17:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.657+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f183000bc60 con 0x56208a3734d0 2023-09-10T01:17:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.657+0000 7f181affd700 1 -- 172.21.15.3:0/4260293795 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f182c004390 con 0x56208a3734d0 2023-09-10T01:17:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f181affd700 1 -- 172.21.15.3:0/4260293795 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f182c004570 con 0x56208a3734d0 2023-09-10T01:17:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 10 client.18774 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56208a1b5a20 con 0x56208a3734d0 2023-09-10T01:17:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f181affd700 1 -- 172.21.15.3:0/4260293795 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f183000bc60 con 0x56208a3734d0 2023-09-10T01:17:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f181affd700 1 -- 172.21.15.3:0/4260293795 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f182c003680 con 0x56208a3734d0 2023-09-10T01:17:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 10 client.18774 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 10 client.18774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 20 client.18774 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a3e17a0 0x56208a3e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 4 client.18774 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:17:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.658+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56208a2b58c0 con 0x56208a3e17a0 2023-09-10T01:17:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.659+0000 7f183affd700 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a3e17a0 0x56208a3e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.659+0000 7f181affd700 10 client.18774 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:30.659+0000 7f183affd700 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a3e17a0 0x56208a3e3bc0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de392f10ffa0c259 server_cookie=911a92888aac9557 in_seq=0 out_seq=0 2023-09-10T01:17:31.177 INFO:tasks.workunit.client.0.smithi003.stdout: 10 38947 31.71 MB/sec warmup 72 sec latency 460.696 ms 2023-09-10T01:17:31.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.350+0000 7f181affd700 1 -- 172.21.15.3:0/4260293795 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56208a2b58c0 con 0x56208a3e17a0 2023-09-10T01:17:31.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.350+0000 7f181affd700 10 client.18774 handle_command_reply: tid=0 2023-09-10T01:17:31.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.350+0000 7f1855bfcb80 1 client.18774 shutdown 2023-09-10T01:17:31.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 msgr2=0x56208a382570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:31.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a382570 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f182c009d40 tx=0x7f182c01a640 comp rx=0 tx=0).stop 2023-09-10T01:17:31.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 shutdown_connections 2023-09-10T01:17:31.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56208a3e17a0 0x56208a3e3bc0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56208a372b10 0x56208a382030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208a3789b0 0x56208a383440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 --2- 172.21.15.3:0/4260293795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56208a3734d0 0x56208a382570 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.355 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 >> 172.21.15.3:0/4260293795 conn(0x56208a2df030 msgr2=0x56208a377910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:31.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f181affd700 0 client.18774 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:31.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 shutdown_connections 2023-09-10T01:17:31.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.353+0000 7f1855bfcb80 1 -- 172.21.15.3:0/4260293795 wait complete. 2023-09-10T01:17:31.356 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.354+0000 7f1855bfcb80 20 client.18774 trim_cache size 0 max 16384 2023-09-10T01:17:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:17:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:17:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:17:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:17:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.354+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18240a07b0 msgr2=0x7f18240a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.354+0000 7f18421ff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18240a07b0 0x7f18240a2c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f18340a0830 tx=0x7f1834004f10 comp rx=0 tx=0).stop 2023-09-10T01:17:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.354+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f183c1066a0 msgr2=0x7f183c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.354+0000 7f18421ff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f183c1066a0 0x7f183c198690 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1834007f10 tx=0x7f183400d3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 shutdown_connections 2023-09-10T01:17:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f18240a07b0 0x7f18240a2c70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f183c1066a0 0x7f183c198690 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f183c067b60 0x7f183c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 --2- 172.21.15.3:0/2593111307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f183c068520 0x7f183c19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 >> 172.21.15.3:0/2593111307 conn(0x7f183c074fe0 msgr2=0x7f183c10b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 shutdown_connections 2023-09-10T01:17:31.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:31.355+0000 7f18421ff700 1 -- 172.21.15.3:0/2593111307 wait complete. 2023-09-10T01:17:32.177 INFO:tasks.workunit.client.0.smithi003.stdout: 10 39631 31.57 MB/sec warmup 73 sec latency 491.051 ms 2023-09-10T01:17:32.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:32 smithi003 ceph-mon[97654]: pgmap v449: 129 pgs: 129 active+clean; 714 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 6.6 MiB/s wr, 508 op/s 2023-09-10T01:17:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:32 smithi136 ceph-mon[133708]: pgmap v449: 129 pgs: 129 active+clean; 714 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 6.6 MiB/s wr, 508 op/s 2023-09-10T01:17:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:32 smithi092 ceph-mon[134067]: pgmap v449: 129 pgs: 129 active+clean; 714 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 6.6 MiB/s wr, 508 op/s 2023-09-10T01:17:33.177 INFO:tasks.workunit.client.0.smithi003.stdout: 10 40337 31.63 MB/sec warmup 74 sec latency 88.949 ms 2023-09-10T01:17:34.177 INFO:tasks.workunit.client.0.smithi003.stdout: 10 41193 31.93 MB/sec warmup 75 sec latency 85.325 ms 2023-09-10T01:17:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:34 smithi003 ceph-mon[97654]: pgmap v450: 129 pgs: 129 active+clean; 714 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.6 MiB/s wr, 432 op/s 2023-09-10T01:17:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:34 smithi136 ceph-mon[133708]: pgmap v450: 129 pgs: 129 active+clean; 714 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.6 MiB/s wr, 432 op/s 2023-09-10T01:17:34.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:34 smithi092 ceph-mon[134067]: pgmap v450: 129 pgs: 129 active+clean; 714 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.6 MiB/s wr, 432 op/s 2023-09-10T01:17:35.178 INFO:tasks.workunit.client.0.smithi003.stdout: 10 41960 32.07 MB/sec warmup 76 sec latency 70.056 ms 2023-09-10T01:17:36.177 INFO:tasks.workunit.client.0.smithi003.stdout: 10 42607 32.15 MB/sec warmup 77 sec latency 89.893 ms 2023-09-10T01:17:36.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:17:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:36 smithi003 ceph-mon[97654]: pgmap v451: 129 pgs: 129 active+clean; 726 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 7.9 MiB/s wr, 788 op/s 2023-09-10T01:17:36.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.596+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/3883011195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d88107600 msgr2=0x7f4d88107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.597 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.596+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/3883011195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d88107600 0x7f4d88107a80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009990 tx=0x7f4d78009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.596+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/3883011195 shutdown_connections 2023-09-10T01:17:36.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.596+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/3883011195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d8810edd0 0x7f4d8810ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.596+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/3883011195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d88107600 0x7f4d88107a80 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.596+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/3883011195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d8810f1b0 0x7f4d8810cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.596+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/3883011195 >> 172.21.15.3:0/3883011195 conn(0x7f4d88071dd0 msgr2=0x7f4d8806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:36.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.597+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/3883011195 shutdown_connections 2023-09-10T01:17:36.599 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.597+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/3883011195 wait complete. 2023-09-10T01:17:36.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.598+0000 7f4d8f1ff700 1 Processor -- start 2023-09-10T01:17:36.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.598+0000 7f4d8f1ff700 1 -- start start 2023-09-10T01:17:36.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.599+0000 7f4d8f1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d88107600 0x7f4d881a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.599+0000 7f4d8f1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d8810edd0 0x7f4d881a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.599+0000 7f4d8f1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8810f1b0 0x7f4d881a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.599+0000 7f4d8f1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d881aa600 con 0x7f4d88107600 2023-09-10T01:17:36.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.599+0000 7f4d8f1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d881aa770 con 0x7f4d8810f1b0 2023-09-10T01:17:36.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.599+0000 7f4d8f1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d881aa930 con 0x7f4d8810edd0 2023-09-10T01:17:36.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.599+0000 7f4d8cf9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d88107600 0x7f4d881a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8cf9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d88107600 0x7f4d881a5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57876/0 (socket says 172.21.15.3:57876) 2023-09-10T01:17:36.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8cf9b700 1 -- 172.21.15.3:0/1660514785 learned_addr learned my addr 172.21.15.3:0/1660514785 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:36.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d87fff700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d8810edd0 0x7f4d881a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8810f1b0 0x7f4d881a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8cf9b700 1 -- 172.21.15.3:0/1660514785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d8810edd0 msgr2=0x7f4d881a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8cf9b700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d8810edd0 0x7f4d881a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8cf9b700 1 -- 172.21.15.3:0/1660514785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8810f1b0 msgr2=0x7f4d881a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8cf9b700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8810f1b0 0x7f4d881a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.600+0000 7f4d8cf9b700 1 -- 172.21.15.3:0/1660514785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780095f0 con 0x7f4d88107600 2023-09-10T01:17:36.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.601+0000 7f4d8cf9b700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d88107600 0x7f4d881a5620 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000b840 tx=0x7f4d8000bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:36.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.602+0000 7f4d85ffb700 1 -- 172.21.15.3:0/1660514785 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80010800 con 0x7f4d88107600 2023-09-10T01:17:36.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.602+0000 7f4d85ffb700 1 -- 172.21.15.3:0/1660514785 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d80010e20 con 0x7f4d88107600 2023-09-10T01:17:36.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.602+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d881aac10 con 0x7f4d88107600 2023-09-10T01:17:36.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.602+0000 7f4d85ffb700 1 -- 172.21.15.3:0/1660514785 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80017360 con 0x7f4d88107600 2023-09-10T01:17:36.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.602+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d881b2450 con 0x7f4d88107600 2023-09-10T01:17:36.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.610+0000 7f4da2ac3b80 1 Processor -- start 2023-09-10T01:17:36.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.613+0000 7f4da2ac3b80 1 -- start start 2023-09-10T01:17:36.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.613+0000 7f4d85ffb700 1 -- 172.21.15.3:0/1660514785 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f4d80010960 con 0x7f4d88107600 2023-09-10T01:17:36.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a6030 0x559ce16af900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16afe40 0x559ce16b2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce15fe730 con 0x559ce16afe40 2023-09-10T01:17:36.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce15fc050 con 0x559ce16a6030 2023-09-10T01:17:36.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce15f7940 con 0x559ce16a5710 2023-09-10T01:17:36.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16a5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:36854/0 (socket says 172.21.15.3:36854) 2023-09-10T01:17:36.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2375509963 learned_addr learned my addr 172.21.15.3:0/2375509963 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:36.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d87fff700 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16afe40 0x559ce16b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2375509963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a6030 msgr2=0x559ce16af900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:36.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a6030 0x559ce16af900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2375509963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16afe40 msgr2=0x559ce16b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:36.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16afe40 0x559ce16b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.614+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2375509963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ce1423de0 con 0x559ce16a5710 2023-09-10T01:17:36.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.615+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16a5af0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d78015000 tx=0x7f4d78009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77a8435d311e1483 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:36.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.616+0000 7f4d6effd700 1 -- 172.21.15.3:0/2375509963 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7801d070 con 0x559ce16a5710 2023-09-10T01:17:36.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.616+0000 7f4d6effd700 1 -- 172.21.15.3:0/2375509963 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d7801e040 con 0x559ce16a5710 2023-09-10T01:17:36.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.616+0000 7f4d6effd700 1 -- 172.21.15.3:0/2375509963 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78005520 con 0x559ce16a5710 2023-09-10T01:17:36.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.617+0000 7f4d85ffb700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d700a0900 0x7f4d700a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.617+0000 7f4d85ffb700 1 -- 172.21.15.3:0/1660514785 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f4d800cbf10 con 0x7f4d88107600 2023-09-10T01:17:36.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.617+0000 7f4d8cf9b700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d700a0900 0x7f4d700a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2375509963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 msgr2=0x559ce16a5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4d8cf9b700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d700a0900 0x7f4d700a2dc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x559ce169b280 tx=0x7f4d80007960 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:36.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16a5af0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d78015000 tx=0x7f4d78009f90 comp rx=0 tx=0).stop 2023-09-10T01:17:36.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2375509963 shutdown_connections 2023-09-10T01:17:36.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16a5af0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16afe40 0x559ce16b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2375509963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a6030 0x559ce16af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.618+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2375509963 >> 172.21.15.3:0/2375509963 conn(0x559ce161a090 msgr2=0x559ce16a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:36.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.619+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2375509963 shutdown_connections 2023-09-10T01:17:36.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.619+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2375509963 wait complete. 2023-09-10T01:17:36.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.619+0000 7f4da2ac3b80 1 -- start start 2023-09-10T01:17:36.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 1 Processor -- start 2023-09-10T01:17:36.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a5710 0x559ce16af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a6030 0x559ce16acb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1595b70 con 0x559ce16a5710 2023-09-10T01:17:36.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1595870 con 0x559ce16afe40 2023-09-10T01:17:36.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1479920 con 0x559ce16a6030 2023-09-10T01:17:36.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4da2ac3b80 10 client.0 _ll_register_callbacks cb 0x559ce14dfa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:36.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a5710 0x559ce16af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.620+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a5710 0x559ce16af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57900/0 (socket says 172.21.15.3:57900) 2023-09-10T01:17:36.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d8d79c700 1 -- 172.21.15.3:0/1365899740 learned_addr learned my addr 172.21.15.3:0/1365899740 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:36.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d87fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a6030 0x559ce16acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d87fff700 1 -- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 msgr2=0x559ce16ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:36.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d87fff700 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d87fff700 1 -- 172.21.15.3:0/1365899740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a5710 msgr2=0x559ce16af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d87fff700 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a5710 0x559ce16af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d87fff700 1 -- 172.21.15.3:0/1365899740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8810b890 con 0x559ce16a6030 2023-09-10T01:17:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d87fff700 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a6030 0x559ce16acb30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c005950 tx=0x7f4d7c00bbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1432f7cfebb63bd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.621+0000 7f4d67fff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.622+0000 7f4d67fff700 1 -- 172.21.15.3:0/1365899740 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c016070 con 0x559ce16a6030 2023-09-10T01:17:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.622+0000 7f4d67fff700 1 -- 172.21.15.3:0/1365899740 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f4d7c003b50 con 0x559ce16a6030 2023-09-10T01:17:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.622+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ce154f4f0 con 0x559ce16a6030 2023-09-10T01:17:36.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.622+0000 7f4d67fff700 1 -- 172.21.15.3:0/1365899740 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d7c003d10 con 0x559ce16a6030 2023-09-10T01:17:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.622+0000 7f4da2ac3b80 10 client.28768 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.622+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x559ce140cc60 con 0x559ce16a6030 2023-09-10T01:17:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.622+0000 7f4d67fff700 1 -- 172.21.15.3:0/1365899740 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559ce154f4f0 con 0x559ce16a6030 2023-09-10T01:17:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4d67fff700 1 -- 172.21.15.3:0/1365899740 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f4d7c011440 con 0x559ce16a6030 2023-09-10T01:17:36.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4da2ac3b80 10 client.28768 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4da2ac3b80 10 client.28768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4da2ac3b80 20 client.28768 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce16bf920 0x559ce16c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4da2ac3b80 4 client.28768 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:36.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ce160c8a0 con 0x559ce16bf920 2023-09-10T01:17:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce16bf920 0x559ce16c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce16bf920 0x559ce16c1d40 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c0c8a59e66d609f server_cookie=b85af920d7cab0d5 in_seq=0 out_seq=0 2023-09-10T01:17:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.623+0000 7f4d67fff700 10 client.28768 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4d67fff700 1 -- 172.21.15.3:0/1365899740 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559ce160c8a0 con 0x559ce16bf920 2023-09-10T01:17:36.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4d67fff700 10 client.28768 handle_command_reply: tid=0 2023-09-10T01:17:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 client.28768 shutdown 2023-09-10T01:17:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a6030 msgr2=0x559ce16acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a6030 0x559ce16acb30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c005950 tx=0x7f4d7c00bbf0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 shutdown_connections 2023-09-10T01:17:36.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce16bf920 0x559ce16c1d40 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a5710 0x559ce16af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1365899740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a6030 0x559ce16acb30 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 >> 172.21.15.3:0/1365899740 conn(0x559ce161a090 msgr2=0x559ce16a07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:36.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.626+0000 7f4d67fff700 0 client.28768 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.627+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 shutdown_connections 2023-09-10T01:17:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.627+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1365899740 wait complete. 2023-09-10T01:17:36.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.627+0000 7f4da2ac3b80 20 client.28768 trim_cache size 0 max 16384 2023-09-10T01:17:36.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.655+0000 7f4da2ac3b80 1 Processor -- start 2023-09-10T01:17:36.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.658+0000 7f4da2ac3b80 1 -- start start 2023-09-10T01:17:36.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.658+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a5710 0x559ce16af450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.658+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.658+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16afe40 0x559ce16ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.658+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce16a68a0 con 0x559ce16a6030 2023-09-10T01:17:36.662 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.658+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce169d3d0 con 0x559ce16a5710 2023-09-10T01:17:36.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.658+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce16bc340 con 0x559ce16afe40 2023-09-10T01:17:36.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a5710 0x559ce16af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a5710 0x559ce16af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53604/0 (socket says 172.21.15.3:53604) 2023-09-10T01:17:36.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2852640443 learned_addr learned my addr 172.21.15.3:0/2852640443 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:36.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16afe40 0x559ce16ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d87fff700 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2852640443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16afe40 msgr2=0x559ce16ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:36.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16afe40 0x559ce16ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2852640443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 msgr2=0x559ce16ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:36.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d8d79c700 1 -- 172.21.15.3:0/2852640443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ce144a250 con 0x559ce16a5710 2023-09-10T01:17:36.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.659+0000 7f4d87fff700 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:36.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.660+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a5710 0x559ce16af450 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800bc60 tx=0x7f4d78003780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25ac02ec69384628 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:36.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:36 smithi136 ceph-mon[133708]: pgmap v451: 129 pgs: 129 active+clean; 726 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 7.9 MiB/s wr, 788 op/s 2023-09-10T01:17:36.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.660+0000 7f4d6f7fe700 1 -- 172.21.15.3:0/2852640443 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7801d040 con 0x559ce16a5710 2023-09-10T01:17:36.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.660+0000 7f4d6f7fe700 1 -- 172.21.15.3:0/2852640443 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d78035e90 con 0x559ce16a5710 2023-09-10T01:17:36.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.660+0000 7f4d6f7fe700 1 -- 172.21.15.3:0/2852640443 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d780052e0 con 0x559ce16a5710 2023-09-10T01:17:36.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2852640443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a5710 msgr2=0x559ce16af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a5710 0x559ce16af450 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800bc60 tx=0x7f4d78003780 comp rx=0 tx=0).stop 2023-09-10T01:17:36.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2852640443 shutdown_connections 2023-09-10T01:17:36.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16afe40 0x559ce16ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16a5710 0x559ce16af450 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/2852640443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2852640443 >> 172.21.15.3:0/2852640443 conn(0x559ce161a090 msgr2=0x559ce16a7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:36.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2852640443 shutdown_connections 2023-09-10T01:17:36.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.661+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/2852640443 wait complete. 2023-09-10T01:17:36.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.662+0000 7f4da2ac3b80 1 -- start start 2023-09-10T01:17:36.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.662+0000 7f4da2ac3b80 1 Processor -- start 2023-09-10T01:17:36.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4da2ac3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce16b5ce0 con 0x559ce16a6030 2023-09-10T01:17:36.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1735770 con 0x559ce16afe40 2023-09-10T01:17:36.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4da2ac3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce17358b0 con 0x559ce16a5710 2023-09-10T01:17:36.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4da2ac3b80 10 client.0 _ll_register_callbacks cb 0x7f4d7c004360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:36.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4d8d79c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53620/0 (socket says 172.21.15.3:53620) 2023-09-10T01:17:36.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4d8d79c700 1 -- 172.21.15.3:0/1515087263 learned_addr learned my addr 172.21.15.3:0/1515087263 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:36.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.663+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.664+0000 7f4d8d79c700 1 -- 172.21.15.3:0/1515087263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 msgr2=0x559ce16b6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:36.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.664+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16b6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.664+0000 7f4d8d79c700 1 -- 172.21.15.3:0/1515087263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 msgr2=0x559ce16b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:36.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.664+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.664+0000 7f4d8d79c700 1 -- 172.21.15.3:0/1515087263 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ce140cc60 con 0x559ce16afe40 2023-09-10T01:17:36.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.664+0000 7f4d8d79c700 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16b55a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800ba40 tx=0x7f4d78034510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebaef4492797bbfd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:36.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.664+0000 7f4d67fff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:36.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.665+0000 7f4d67fff700 1 -- 172.21.15.3:0/1515087263 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7801d030 con 0x559ce16afe40 2023-09-10T01:17:36.674 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.665+0000 7f4d67fff700 1 -- 172.21.15.3:0/1515087263 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f4d78000e40 con 0x559ce16afe40 2023-09-10T01:17:36.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.665+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d7801e040 con 0x559ce16afe40 2023-09-10T01:17:36.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.665+0000 7f4d67fff700 1 -- 172.21.15.3:0/1515087263 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d780340c0 con 0x559ce16afe40 2023-09-10T01:17:36.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.665+0000 7f4da2ac3b80 10 client.28907 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:36.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.665+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7f4d78027040 con 0x559ce16afe40 2023-09-10T01:17:36.675 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4d67fff700 1 -- 172.21.15.3:0/1515087263 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d7801e040 con 0x559ce16afe40 2023-09-10T01:17:36.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4d67fff700 1 -- 172.21.15.3:0/1515087263 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f4d78005470 con 0x559ce16afe40 2023-09-10T01:17:36.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4da2ac3b80 10 client.28907 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:36.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4da2ac3b80 10 client.28907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:36.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4da2ac3b80 20 client.28907 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:36.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce1736b70 0x559ce1738f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:36.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4da2ac3b80 4 client.28907 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:17:36.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559ce15f0920 con 0x559ce1736b70 2023-09-10T01:17:36.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.666+0000 7f4d87fff700 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce1736b70 0x559ce1738f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:36.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.667+0000 7f4d87fff700 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce1736b70 0x559ce1738f90 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f464bf063306387 server_cookie=6941d2399b9be555 in_seq=0 out_seq=0 2023-09-10T01:17:36.677 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.667+0000 7f4d67fff700 10 client.28907 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:36.678 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4d67fff700 1 -- 172.21.15.3:0/1515087263 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559ce15f0920 con 0x559ce1736b70 2023-09-10T01:17:36.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4d67fff700 10 client.28907 handle_command_reply: tid=0 2023-09-10T01:17:36.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 client.28907 shutdown 2023-09-10T01:17:36.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 msgr2=0x559ce16b55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16b55a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800ba40 tx=0x7f4d78034510 comp rx=0 tx=0).stop 2023-09-10T01:17:36.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 shutdown_connections 2023-09-10T01:17:36.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559ce1736b70 0x559ce1738f90 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559ce16a5710 0x559ce16b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce16afe40 0x559ce16b55a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 --2- 172.21.15.3:0/1515087263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559ce16a6030 0x559ce16b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 >> 172.21.15.3:0/1515087263 conn(0x559ce161a090 msgr2=0x559ce16a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:36.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.672+0000 7f4d67fff700 0 client.28907 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:36.681 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:17:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:17:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:17:36.682 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:17:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.673+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 shutdown_connections 2023-09-10T01:17:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.674+0000 7f4da2ac3b80 1 -- 172.21.15.3:0/1515087263 wait complete. 2023-09-10T01:17:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.674+0000 7f4da2ac3b80 20 client.28907 trim_cache size 0 max 16384 2023-09-10T01:17:36.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.675+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d700a0900 msgr2=0x7f4d700a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.675+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d700a0900 0x7f4d700a2dc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x559ce169b280 tx=0x7f4d80007960 comp rx=0 tx=0).stop 2023-09-10T01:17:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.675+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d88107600 msgr2=0x7f4d881a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.675+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d88107600 0x7f4d881a5620 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000b840 tx=0x7f4d8000bb50 comp rx=0 tx=0).stop 2023-09-10T01:17:36.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.676+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 shutdown_connections 2023-09-10T01:17:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.676+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f4d700a0900 0x7f4d700a2dc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.676+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d8810f1b0 0x7f4d881a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.676+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4d88107600 0x7f4d881a5620 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.676+0000 7f4d8f1ff700 1 --2- 172.21.15.3:0/1660514785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4d8810edd0 0x7f4d881a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.676+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 >> 172.21.15.3:0/1660514785 conn(0x7f4d88071dd0 msgr2=0x7f4d88109460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:36.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.677+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 shutdown_connections 2023-09-10T01:17:36.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.677+0000 7f4d8f1ff700 1 -- 172.21.15.3:0/1660514785 wait complete. 2023-09-10T01:17:36.706 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:17:36.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:17:36.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:36 smithi092 ceph-mon[134067]: pgmap v451: 129 pgs: 129 active+clean; 726 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 7.9 MiB/s wr, 788 op/s 2023-09-10T01:17:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.996+0000 7f6b18fff700 1 -- 172.21.15.3:0/2835721866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14106780 msgr2=0x7f6b14106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.996+0000 7f6b18fff700 1 --2- 172.21.15.3:0/2835721866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14106780 0x7f6b14106b60 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6b040099c0 tx=0x7f6b04009cd0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.997+0000 7f6b18fff700 1 -- 172.21.15.3:0/2835721866 shutdown_connections 2023-09-10T01:17:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.997+0000 7f6b18fff700 1 --2- 172.21.15.3:0/2835721866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b14068fd0 0x7f6b1410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.997+0000 7f6b18fff700 1 --2- 172.21.15.3:0/2835721866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14106780 0x7f6b14106b60 secure :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6b040099c0 tx=0x7f6b04009cd0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.997+0000 7f6b18fff700 1 --2- 172.21.15.3:0/2835721866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b14068610 0x7f6b14068a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.997+0000 7f6b18fff700 1 -- 172.21.15.3:0/2835721866 >> 172.21.15.3:0/2835721866 conn(0x7f6b14077480 msgr2=0x7f6b14077890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.997+0000 7f6b18fff700 1 -- 172.21.15.3:0/2835721866 shutdown_connections 2023-09-10T01:17:36.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.997+0000 7f6b18fff700 1 -- 172.21.15.3:0/2835721866 wait complete. 2023-09-10T01:17:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.998+0000 7f6b18fff700 1 Processor -- start 2023-09-10T01:17:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.998+0000 7f6b18fff700 1 -- start start 2023-09-10T01:17:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b18fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b14068610 0x7f6b140ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b18fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b14068fd0 0x7f6b140ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b18fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14100250 0x7f6b141a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b18fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141a52b0 con 0x7f6b14068610 2023-09-10T01:17:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b18fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141a5420 con 0x7f6b14100250 2023-09-10T01:17:37.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b18fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141a55e0 con 0x7f6b14068fd0 2023-09-10T01:17:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14100250 0x7f6b141a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b09d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b14068fd0 0x7f6b140ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14100250 0x7f6b141a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53646/0 (socket says 172.21.15.3:53646) 2023-09-10T01:17:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 -- 172.21.15.3:0/3222598851 learned_addr learned my addr 172.21.15.3:0/3222598851 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b1259c700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b14068610 0x7f6b140ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 -- 172.21.15.3:0/3222598851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b14068fd0 msgr2=0x7f6b140ffd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b14068fd0 0x7f6b140ffd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 -- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b14068610 msgr2=0x7f6b140ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b14068610 0x7f6b140ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:36.999+0000 7f6b12d9d700 1 -- 172.21.15.3:0/3222598851 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b040095f0 con 0x7f6b14100250 2023-09-10T01:17:37.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.000+0000 7f6b12d9d700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14100250 0x7f6b141a4c20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00b9e0 tx=0x7f6b0c00bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.000+0000 7f6b0bfff700 1 -- 172.21.15.3:0/3222598851 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c00d820 con 0x7f6b14100250 2023-09-10T01:17:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.000+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b141a58c0 con 0x7f6b14100250 2023-09-10T01:17:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.000+0000 7f6b0bfff700 1 -- 172.21.15.3:0/3222598851 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b0c00de40 con 0x7f6b14100250 2023-09-10T01:17:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.000+0000 7f6b0bfff700 1 -- 172.21.15.3:0/3222598851 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c0143d0 con 0x7f6b14100250 2023-09-10T01:17:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.000+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b141a5ea0 con 0x7f6b14100250 2023-09-10T01:17:37.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.002+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14066bf0 con 0x7f6b14100250 2023-09-10T01:17:37.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.005+0000 7f6b0bfff700 1 -- 172.21.15.3:0/3222598851 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6b0c0155b0 con 0x7f6b14100250 2023-09-10T01:17:37.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.006+0000 7f6b0bfff700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6b000a07f0 0x7f6b000a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.007+0000 7f6b0bfff700 1 -- 172.21.15.3:0/3222598851 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f6b0c011070 con 0x7f6b14100250 2023-09-10T01:17:37.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.007+0000 7f6b1259c700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6b000a07f0 0x7f6b000a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.008+0000 7f6b1259c700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6b000a07f0 0x7f6b000a2cb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6b04006010 tx=0x7f6b04000bc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.011+0000 7f6b0bfff700 1 -- 172.21.15.3:0/3222598851 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6b0c095ae0 con 0x7f6b14100250 2023-09-10T01:17:37.177 INFO:tasks.workunit.client.0.smithi003.stdout: 10 43384 32.16 MB/sec warmup 78 sec latency 33.609 ms 2023-09-10T01:17:37.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.408+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b14110800 con 0x7f6b14100250 2023-09-10T01:17:37.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.409+0000 7f6b0bfff700 1 -- 172.21.15.3:0/3222598851 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+11299 (secure 0 0 0) 0x7f6b0c099100 con 0x7f6b14100250 2023-09-10T01:17:37.410 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:17:37.410 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:16:25.285703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:17:37.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6b000a07f0 msgr2=0x7f6b000a2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6b000a07f0 0x7f6b000a2cb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6b04006010 tx=0x7f6b04000bc0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14100250 msgr2=0x7f6b141a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14100250 0x7f6b141a4c20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00b9e0 tx=0x7f6b0c00bcf0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 shutdown_connections 2023-09-10T01:17:37.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f6b000a07f0 0x7f6b000a2cb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b14100250 0x7f6b141a4c20 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b14068610 0x7f6b140ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 --2- 172.21.15.3:0/3222598851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b14068fd0 0x7f6b140ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 >> 172.21.15.3:0/3222598851 conn(0x7f6b14077480 msgr2=0x7f6b140fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:37.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 shutdown_connections 2023-09-10T01:17:37.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.412+0000 7f6b18fff700 1 -- 172.21.15.3:0/3222598851 wait complete. 2023-09-10T01:17:37.416 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 38 2023-09-10T01:17:37.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:17:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.782+0000 7faa8f3ff700 1 -- 172.21.15.3:0/2687913911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa88112550 msgr2=0x7faa88114940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.782+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/2687913911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa88112550 0x7faa88114940 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7faa78009990 tx=0x7faa78009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.783+0000 7faa8f3ff700 1 -- 172.21.15.3:0/2687913911 shutdown_connections 2023-09-10T01:17:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.783+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/2687913911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa88108fa0 0x7faa88109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.783+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/2687913911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa88112550 0x7faa88114940 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.783+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/2687913911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa881098c0 0x7faa88112010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.783+0000 7faa8f3ff700 1 -- 172.21.15.3:0/2687913911 >> 172.21.15.3:0/2687913911 conn(0x7faa88072090 msgr2=0x7faa8806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.789+0000 7faa8f3ff700 1 -- 172.21.15.3:0/2687913911 shutdown_connections 2023-09-10T01:17:37.790 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.790+0000 7faa8f3ff700 1 -- 172.21.15.3:0/2687913911 wait complete. 2023-09-10T01:17:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.791+0000 7faa8f3ff700 1 Processor -- start 2023-09-10T01:17:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.791+0000 7faa8f3ff700 1 -- start start 2023-09-10T01:17:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.792+0000 7faa8f3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa88108fa0 0x7faa8811ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.792+0000 7faa8f3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa881098c0 0x7faa8811d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.792+0000 7faa8f3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa88112550 0x7faa88117f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.792+0000 7faa8f3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa88118720 con 0x7faa88112550 2023-09-10T01:17:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.792+0000 7faa8f3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa88118890 con 0x7faa881098c0 2023-09-10T01:17:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.792+0000 7faa8f3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa88118a50 con 0x7faa88108fa0 2023-09-10T01:17:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.794+0000 7faa8d19b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa88108fa0 0x7faa8811ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.794+0000 7faa8d19b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa88108fa0 0x7faa8811ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:36908/0 (socket says 172.21.15.3:36908) 2023-09-10T01:17:37.796 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.794+0000 7faa8d19b700 1 -- 172.21.15.3:0/901056298 learned_addr learned my addr 172.21.15.3:0/901056298 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.794+0000 7faa8c99a700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa881098c0 0x7faa8811d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.795+0000 7faa8c99a700 1 -- 172.21.15.3:0/901056298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa88108fa0 msgr2=0x7faa8811ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.795+0000 7faa8c99a700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa88108fa0 0x7faa8811ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.797 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.795+0000 7faa8c99a700 1 -- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa88112550 msgr2=0x7faa88117f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.795+0000 7faa8c99a700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa88112550 0x7faa88117f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.795+0000 7faa8c99a700 1 -- 172.21.15.3:0/901056298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780095f0 con 0x7faa881098c0 2023-09-10T01:17:37.798 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.795+0000 7faa8c99a700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa881098c0 0x7faa8811d3f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7faa8400b450 tx=0x7faa8400b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.799 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.795+0000 7faa7e7fc700 1 -- 172.21.15.3:0/901056298 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84011830 con 0x7faa881098c0 2023-09-10T01:17:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.796+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa88118d30 con 0x7faa881098c0 2023-09-10T01:17:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.796+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa88122f80 con 0x7faa881098c0 2023-09-10T01:17:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.799+0000 7faa7e7fc700 1 -- 172.21.15.3:0/901056298 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa84011e50 con 0x7faa881098c0 2023-09-10T01:17:37.800 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.799+0000 7faa7e7fc700 1 -- 172.21.15.3:0/901056298 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84010db0 con 0x7faa881098c0 2023-09-10T01:17:37.802 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 Processor -- start 2023-09-10T01:17:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 -- start start 2023-09-10T01:17:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578913af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.804 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578920230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57886c730 con 0x55a57891de40 2023-09-10T01:17:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57886a050 con 0x55a578914030 2023-09-10T01:17:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.802+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a578865940 con 0x55a578913710 2023-09-10T01:17:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.804+0000 7faa8d99c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578913af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.805+0000 7faa7e7fc700 1 -- 172.21.15.3:0/901056298 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7faa84011990 con 0x7faa881098c0 2023-09-10T01:17:37.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.805+0000 7faa8d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.805+0000 7faa8d19b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53688/0 (socket says 172.21.15.3:53688) 2023-09-10T01:17:37.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.805+0000 7faa8d19b700 1 -- 172.21.15.3:0/1110993489 learned_addr learned my addr 172.21.15.3:0/1110993489 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:37.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.807+0000 7faa7e7fc700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7faa740a0870 0x7faa740a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.807+0000 7faa7e7fc700 1 -- 172.21.15.3:0/901056298 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7faa840cba70 con 0x7faa881098c0 2023-09-10T01:17:37.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.809+0000 7faa8d99c700 1 -- 172.21.15.3:0/1110993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 msgr2=0x55a57891d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.809+0000 7faa8d99c700 1 --2- 172.21.15.3:0/1110993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.810 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.809+0000 7faa8d99c700 1 -- 172.21.15.3:0/1110993489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 msgr2=0x55a578920230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.809+0000 7faa8d99c700 1 --2- 172.21.15.3:0/1110993489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578920230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.811 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.809+0000 7faa8d99c700 1 -- 172.21.15.3:0/1110993489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a578691de0 con 0x55a578913710 2023-09-10T01:17:37.812 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.810+0000 7faa8d99c700 1 --2- 172.21.15.3:0/1110993489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578913af0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7faa780098e0 tx=0x7faa78003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afe52156984da81e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.811+0000 7faa8d99c700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7faa740a0870 0x7faa740a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.812+0000 7faa737fe700 1 -- 172.21.15.3:0/1110993489 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa78028070 con 0x55a578913710 2023-09-10T01:17:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.812+0000 7faa737fe700 1 -- 172.21.15.3:0/1110993489 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa7800be00 con 0x55a578913710 2023-09-10T01:17:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.812+0000 7faa737fe700 1 -- 172.21.15.3:0/1110993489 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7801d070 con 0x55a578913710 2023-09-10T01:17:37.814 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.813+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1110993489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 msgr2=0x55a578913af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.813+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1110993489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578913af0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7faa780098e0 tx=0x7faa78003680 comp rx=0 tx=0).stop 2023-09-10T01:17:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.813+0000 7faa8d99c700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7faa740a0870 0x7faa740a2d30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa78003fd0 tx=0x7faa78005390 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.815 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.814+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1110993489 shutdown_connections 2023-09-10T01:17:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.814+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1110993489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578913af0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.814+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1110993489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578920230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.814+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1110993489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.814+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1110993489 >> 172.21.15.3:0/1110993489 conn(0x55a578888090 msgr2=0x55a57890ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:37.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.814+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1110993489 shutdown_connections 2023-09-10T01:17:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.814+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1110993489 wait complete. 2023-09-10T01:17:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.815+0000 7faaa2c8ab80 1 -- start start 2023-09-10T01:17:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.815+0000 7faaa2c8ab80 1 Processor -- start 2023-09-10T01:17:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.816+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578913710 0x55a57891d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.816+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.816+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a57891afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.816+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a578803b70 con 0x55a578913710 2023-09-10T01:17:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.816+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a578803870 con 0x55a578914030 2023-09-10T01:17:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.816+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5786e7920 con 0x55a57891de40 2023-09-10T01:17:37.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.816+0000 7faaa2c8ab80 10 client.0 _ll_register_callbacks cb 0x55a57874da80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:37.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.817+0000 7faa8c99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.817+0000 7faa8c99a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53704/0 (socket says 172.21.15.3:53704) 2023-09-10T01:17:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.817+0000 7faa8c99a700 1 -- 172.21.15.3:0/1409556485 learned_addr learned my addr 172.21.15.3:0/1409556485 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.818+0000 7faa8d19b700 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578913710 0x55a57891d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.818+0000 7faa8d99c700 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a57891afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.818+0000 7faa8c99a700 1 -- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 msgr2=0x55a57891afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.818+0000 7faa8c99a700 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a57891afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.818+0000 7faa8c99a700 1 -- 172.21.15.3:0/1409556485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578913710 msgr2=0x55a57891d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.818+0000 7faa8c99a700 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578913710 0x55a57891d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.822 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.818+0000 7faa8c99a700 1 -- 172.21.15.3:0/1409556485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa88108210 con 0x55a578914030 2023-09-10T01:17:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.819+0000 7faa8c99a700 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891aa00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7faa84098970 tx=0x7faa84098700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce874146e98397dc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.819+0000 7faa70ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.821+0000 7faa70ff9700 1 -- 172.21.15.3:0/1409556485 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840990d0 con 0x55a578914030 2023-09-10T01:17:37.823 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.821+0000 7faa70ff9700 1 -- 172.21.15.3:0/1409556485 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7faa84098c70 con 0x55a578914030 2023-09-10T01:17:37.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.821+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5787bd4f0 con 0x55a578914030 2023-09-10T01:17:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.822+0000 7faa70ff9700 1 -- 172.21.15.3:0/1409556485 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa8409caa0 con 0x55a578914030 2023-09-10T01:17:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.822+0000 7faaa2c8ab80 10 client.28925 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.822+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55a57867ac60 con 0x55a578914030 2023-09-10T01:17:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faa70ff9700 1 -- 172.21.15.3:0/1409556485 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a5787bd4f0 con 0x55a578914030 2023-09-10T01:17:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faa70ff9700 1 -- 172.21.15.3:0/1409556485 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7faa840a0970 con 0x55a578914030 2023-09-10T01:17:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faaa2c8ab80 10 client.28925 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faaa2c8ab80 10 client.28925 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:17:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faaa2c8ab80 20 client.28925 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a57892d920 0x55a57892fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faaa2c8ab80 4 client.28925 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.823+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a57887a8a0 con 0x55a57892d920 2023-09-10T01:17:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.824+0000 7faa8d19b700 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a57892d920 0x55a57892fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.844+0000 7faa8d19b700 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a57892d920 0x55a57892fd40 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b2d13b12c1f180 server_cookie=89be90e2553be029 in_seq=0 out_seq=0 2023-09-10T01:17:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.844+0000 7faa70ff9700 10 client.28925 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:37.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.850+0000 7faa70ff9700 1 -- 172.21.15.3:0/1409556485 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a57887a8a0 con 0x55a57892d920 2023-09-10T01:17:37.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.850+0000 7faa70ff9700 10 client.28925 handle_command_reply: tid=0 2023-09-10T01:17:37.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.850+0000 7faaa2c8ab80 1 client.28925 shutdown 2023-09-10T01:17:37.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.851+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 msgr2=0x55a57891aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.851+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891aa00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7faa84098970 tx=0x7faa84098700 comp rx=0 tx=0).stop 2023-09-10T01:17:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.853+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 shutdown_connections 2023-09-10T01:17:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.853+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a57892d920 0x55a57892fd40 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.853+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a57891afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.853+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578913710 0x55a57891d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.853+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/1409556485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a57891aa00 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.853+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 >> 172.21.15.3:0/1409556485 conn(0x55a578888090 msgr2=0x55a57890e780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:37.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.854+0000 7faa70ff9700 0 client.28925 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:37.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.855+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 shutdown_connections 2023-09-10T01:17:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.860+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/1409556485 wait complete. 2023-09-10T01:17:37.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.862+0000 7faaa2c8ab80 20 client.28925 trim_cache size 0 max 16384 2023-09-10T01:17:37.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.885+0000 7faaa2c8ab80 1 Processor -- start 2023-09-10T01:17:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.886+0000 7faaa2c8ab80 1 -- start start 2023-09-10T01:17:37.887 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.886+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578924a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.886+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a5789291e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.886+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578915d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.886+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a578925000 con 0x55a57891de40 2023-09-10T01:17:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.886+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a578914410 con 0x55a578914030 2023-09-10T01:17:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.886+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57892a280 con 0x55a578913710 2023-09-10T01:17:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.888+0000 7faa8d99c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578915d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.888+0000 7faa8d99c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578915d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57974/0 (socket says 172.21.15.3:57974) 2023-09-10T01:17:37.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8d99c700 1 -- 172.21.15.3:0/381509777 learned_addr learned my addr 172.21.15.3:0/381509777 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:37.890 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8d19b700 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578924a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8c99a700 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a5789291e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8d19b700 1 -- 172.21.15.3:0/381509777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 msgr2=0x55a5789291e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8d19b700 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a5789291e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.891 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8d19b700 1 -- 172.21.15.3:0/381509777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 msgr2=0x55a578915d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8d19b700 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578915d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.889+0000 7faa8d19b700 1 -- 172.21.15.3:0/381509777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5786b8250 con 0x55a578913710 2023-09-10T01:17:37.892 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.890+0000 7faa8d19b700 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578924a90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55a578930620 tx=0x7faa80004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=541471f244396471 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.890+0000 7faa73fff700 1 -- 172.21.15.3:0/381509777 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8001f4c0 con 0x55a578913710 2023-09-10T01:17:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.891+0000 7faa73fff700 1 -- 172.21.15.3:0/381509777 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa8000f040 con 0x55a578913710 2023-09-10T01:17:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.891+0000 7faa73fff700 1 -- 172.21.15.3:0/381509777 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80004030 con 0x55a578913710 2023-09-10T01:17:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/381509777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 msgr2=0x55a578924a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578924a90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55a578930620 tx=0x7faa80004ad0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/381509777 shutdown_connections 2023-09-10T01:17:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a578913710 0x55a578924a90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a57891de40 0x55a578915d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/381509777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578914030 0x55a5789291e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.894 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/381509777 >> 172.21.15.3:0/381509777 conn(0x55a578888090 msgr2=0x55a57892f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.892+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/381509777 shutdown_connections 2023-09-10T01:17:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.893+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/381509777 wait complete. 2023-09-10T01:17:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.893+0000 7faaa2c8ab80 1 -- start start 2023-09-10T01:17:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.895+0000 7faaa2c8ab80 1 Processor -- start 2023-09-10T01:17:37.896 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578913710 0x55a578924920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578914030 0x55a578922c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faaa2c8ab80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a578923150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a578923840 con 0x55a578914030 2023-09-10T01:17:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5789239b0 con 0x55a578913710 2023-09-10T01:17:37.897 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faaa2c8ab80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a578981760 con 0x55a57891de40 2023-09-10T01:17:37.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faaa2c8ab80 10 client.0 _ll_register_callbacks cb 0x55a57874da80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:37.898 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.896+0000 7faa8d99c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a578923150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.897+0000 7faa8d99c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a578923150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:36950/0 (socket says 172.21.15.3:36950) 2023-09-10T01:17:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.897+0000 7faa8d99c700 1 -- 172.21.15.3:0/2561118351 learned_addr learned my addr 172.21.15.3:0/2561118351 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.898+0000 7faa8d19b700 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578913710 0x55a578924920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.899 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.898+0000 7faa8c99a700 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578914030 0x55a578922c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.899+0000 7faa8c99a700 1 -- 172.21.15.3:0/2561118351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 msgr2=0x55a578923150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.899+0000 7faa8c99a700 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a578923150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.899+0000 7faa8c99a700 1 -- 172.21.15.3:0/2561118351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578913710 msgr2=0x55a578924920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.899+0000 7faa8c99a700 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578913710 0x55a578924920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.899+0000 7faa8c99a700 1 -- 172.21.15.3:0/2561118351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8001f880 con 0x55a578914030 2023-09-10T01:17:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.899+0000 7faa8c99a700 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578914030 0x55a578922c10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faa84011040 tx=0x7faa84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4660fa8ffb99ef server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.900+0000 7faa70ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.901+0000 7faa70ff9700 1 -- 172.21.15.3:0/2561118351 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840b8310 con 0x55a578914030 2023-09-10T01:17:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.901+0000 7faa70ff9700 1 -- 172.21.15.3:0/2561118351 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7faa840b0020 con 0x55a578914030 2023-09-10T01:17:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.901+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa7800be00 con 0x55a578914030 2023-09-10T01:17:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.901+0000 7faa70ff9700 1 -- 172.21.15.3:0/2561118351 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa84098c70 con 0x55a578914030 2023-09-10T01:17:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.902+0000 7faaa2c8ab80 10 client.18801 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:37.903 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.902+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55a5786902b0 con 0x55a578914030 2023-09-10T01:17:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.902+0000 7faa70ff9700 1 -- 172.21.15.3:0/2561118351 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa7800be00 con 0x55a578914030 2023-09-10T01:17:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.903+0000 7faa70ff9700 1 -- 172.21.15.3:0/2561118351 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7faa84077b40 con 0x55a578914030 2023-09-10T01:17:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.904+0000 7faaa2c8ab80 10 client.18801 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.904+0000 7faaa2c8ab80 10 client.18801 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:17:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.904+0000 7faaa2c8ab80 20 client.18801 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.904+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a578982aa0 0x55a578984ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.904+0000 7faaa2c8ab80 4 client.18801 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:17:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.904+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a57885e920 con 0x55a578982aa0 2023-09-10T01:17:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.906+0000 7faa8d19b700 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a578982aa0 0x55a578984ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:37.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.906+0000 7faa8d19b700 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a578982aa0 0x55a578984ec0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=301558ca529bfea8 server_cookie=980619a690e6051c in_seq=0 out_seq=0 2023-09-10T01:17:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.907+0000 7faa70ff9700 10 client.18801 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.907+0000 7faa70ff9700 1 -- 172.21.15.3:0/2561118351 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a57885e920 con 0x55a578982aa0 2023-09-10T01:17:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.907+0000 7faa70ff9700 10 client.18801 handle_command_reply: tid=0 2023-09-10T01:17:37.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.907+0000 7faaa2c8ab80 1 client.18801 shutdown 2023-09-10T01:17:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578914030 msgr2=0x55a578922c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578914030 0x55a578922c10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faa84011040 tx=0x7faa84004750 comp rx=0 tx=0).stop 2023-09-10T01:17:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 shutdown_connections 2023-09-10T01:17:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x55a578982aa0 0x55a578984ec0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a57891de40 0x55a578923150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a578913710 0x55a578924920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 --2- 172.21.15.3:0/2561118351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a578914030 0x55a578922c10 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 >> 172.21.15.3:0/2561118351 conn(0x55a578888090 msgr2=0x55a578915d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.908+0000 7faa70ff9700 0 client.18801 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:17:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.909+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 shutdown_connections 2023-09-10T01:17:37.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.912+0000 7faaa2c8ab80 1 -- 172.21.15.3:0/2561118351 wait complete. 2023-09-10T01:17:37.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.912+0000 7faaa2c8ab80 20 client.18801 trim_cache size 0 max 16384 2023-09-10T01:17:37.921 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:17:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.925+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7faa740a0870 msgr2=0x7faa740a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.925+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7faa740a0870 0x7faa740a2d30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa78003fd0 tx=0x7faa78005390 comp rx=0 tx=0).stop 2023-09-10T01:17:37.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.925+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa881098c0 msgr2=0x7faa8811d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:37.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.925+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa881098c0 0x7faa8811d3f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7faa8400b450 tx=0x7faa8400b760 comp rx=0 tx=0).stop 2023-09-10T01:17:37.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.927+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 shutdown_connections 2023-09-10T01:17:37.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.927+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7faa740a0870 0x7faa740a2d30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.927+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa88108fa0 0x7faa8811ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.927+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa88112550 0x7faa88117f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.927+0000 7faa8f3ff700 1 --2- 172.21.15.3:0/901056298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa881098c0 0x7faa8811d3f0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:37.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.927+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 >> 172.21.15.3:0/901056298 conn(0x7faa88072090 msgr2=0x7faa8806b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:37.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.928+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 shutdown_connections 2023-09-10T01:17:37.929 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:37.928+0000 7faa8f3ff700 1 -- 172.21.15.3:0/901056298 wait complete. 2023-09-10T01:17:37.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:17:38.179 INFO:tasks.workunit.client.0.smithi003.stdout: 10 43971 32.19 MB/sec warmup 79 sec latency 119.002 ms 2023-09-10T01:17:38.340 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 -- 172.21.15.3:0/705939218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb810ce80 msgr2=0x7efcb810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 --2- 172.21.15.3:0/705939218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb810ce80 0x7efcb810f2f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7efcb000b0a0 tx=0x7efcb000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 -- 172.21.15.3:0/705939218 shutdown_connections 2023-09-10T01:17:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 --2- 172.21.15.3:0/705939218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb810ce80 0x7efcb810f2f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 --2- 172.21.15.3:0/705939218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 0x7efcb8072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 --2- 172.21.15.3:0/705939218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb8073350 0x7efcb810c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 -- 172.21.15.3:0/705939218 >> 172.21.15.3:0/705939218 conn(0x7efcb80fc170 msgr2=0x7efcb80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 -- 172.21.15.3:0/705939218 shutdown_connections 2023-09-10T01:17:38.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.339+0000 7efcbdfff700 1 -- 172.21.15.3:0/705939218 wait complete. 2023-09-10T01:17:38.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.340+0000 7efcbdfff700 1 Processor -- start 2023-09-10T01:17:38.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.340+0000 7efcbdfff700 1 -- start start 2023-09-10T01:17:38.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcbdfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 0x7efcb819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcbdfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb8073350 0x7efcb819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcbdfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb810ce80 0x7efcb8198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcbdfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8198f00 con 0x7efcb8073350 2023-09-10T01:17:38.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcbdfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8199070 con 0x7efcb8072a30 2023-09-10T01:17:38.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcbdfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8199230 con 0x7efcb810ce80 2023-09-10T01:17:38.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 0x7efcb819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb8073350 0x7efcb819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 0x7efcb819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53736/0 (socket says 172.21.15.3:53736) 2023-09-10T01:17:38.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb77fe700 1 -- 172.21.15.3:0/3626080462 learned_addr learned my addr 172.21.15.3:0/3626080462 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb7fff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb810ce80 0x7efcb8198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb6ffd700 1 -- 172.21.15.3:0/3626080462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb810ce80 msgr2=0x7efcb8198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb810ce80 0x7efcb8198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb6ffd700 1 -- 172.21.15.3:0/3626080462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 msgr2=0x7efcb819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.341+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 0x7efcb819e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcb6ffd700 1 -- 172.21.15.3:0/3626080462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0009d00 con 0x7efcb8073350 2023-09-10T01:17:38.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcb77fe700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 0x7efcb819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:38.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb8073350 0x7efcb819ed60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7efcac00e9f0 tx=0x7efcac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcb4ff9700 1 -- 172.21.15.3:0/3626080462 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac00eee0 con 0x7efcb8073350 2023-09-10T01:17:38.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcb4ff9700 1 -- 172.21.15.3:0/3626080462 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcac00cbd0 con 0x7efcb8073350 2023-09-10T01:17:38.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcb4ff9700 1 -- 172.21.15.3:0/3626080462 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac018680 con 0x7efcb8073350 2023-09-10T01:17:38.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb8199510 con 0x7efcb8073350 2023-09-10T01:17:38.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.342+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb81a5810 con 0x7efcb8073350 2023-09-10T01:17:38.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.345+0000 7efcb4ff9700 1 -- 172.21.15.3:0/3626080462 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7efcac00cd40 con 0x7efcb8073350 2023-09-10T01:17:38.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 Processor -- start 2023-09-10T01:17:38.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcb4ff9700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efca00a07b0 0x7efca00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 -- start start 2023-09-10T01:17:38.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcb4ff9700 1 -- 172.21.15.3:0/3626080462 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7efcac014070 con 0x7efcb8073350 2023-09-10T01:17:38.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331325af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331326030 0x55933132f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55933132fe40 0x559331332230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933127e730 con 0x559331325710 2023-09-10T01:17:38.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933127c050 con 0x559331326030 2023-09-10T01:17:38.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559331277940 con 0x55933132fe40 2023-09-10T01:17:38.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcb7fff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efca00a07b0 0x7efca00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331325af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcb6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331326030 0x55933132f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcb6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331326030 0x55933132f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53740/0 (socket says 172.21.15.3:53740) 2023-09-10T01:17:38.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.347+0000 7efcb6ffd700 1 -- 172.21.15.3:0/3203990608 learned_addr learned my addr 172.21.15.3:0/3203990608 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb7fff700 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55933132fe40 0x559331332230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb6ffd700 1 -- 172.21.15.3:0/3203990608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55933132fe40 msgr2=0x559331332230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55933132fe40 0x559331332230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb6ffd700 1 -- 172.21.15.3:0/3203990608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 msgr2=0x559331325af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331325af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb6ffd700 1 -- 172.21.15.3:0/3203990608 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593310a3de0 con 0x559331326030 2023-09-10T01:17:38.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb77fe700 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331325af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:38.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb7fff700 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55933132fe40 0x559331332230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:38.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331326030 0x55933132f900 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7efcac009800 tx=0x7efcac098f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d952f65c478ccc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efcb7fff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efca00a07b0 0x7efca00a2c70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009fd0 tx=0x7efcb0009480 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efc9dffb700 1 -- 172.21.15.3:0/3203990608 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac09d3e0 con 0x559331326030 2023-09-10T01:17:38.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efc9dffb700 1 -- 172.21.15.3:0/3203990608 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcac0a1620 con 0x559331326030 2023-09-10T01:17:38.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.348+0000 7efc9dffb700 1 -- 172.21.15.3:0/3203990608 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac099810 con 0x559331326030 2023-09-10T01:17:38.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 -- 172.21.15.3:0/3203990608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331326030 msgr2=0x55933132f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331326030 0x55933132f900 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7efcac009800 tx=0x7efcac098f90 comp rx=0 tx=0).stop 2023-09-10T01:17:38.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 -- 172.21.15.3:0/3203990608 shutdown_connections 2023-09-10T01:17:38.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55933132fe40 0x559331332230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331325af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/3203990608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331326030 0x55933132f900 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 -- 172.21.15.3:0/3203990608 >> 172.21.15.3:0/3203990608 conn(0x55933129a090 msgr2=0x559331320bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 -- 172.21.15.3:0/3203990608 shutdown_connections 2023-09-10T01:17:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.349+0000 7efcd18d3b80 1 -- 172.21.15.3:0/3203990608 wait complete. 2023-09-10T01:17:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.350+0000 7efcd18d3b80 1 -- start start 2023-09-10T01:17:38.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.350+0000 7efcd18d3b80 1 Processor -- start 2023-09-10T01:17:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x55933132f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x55933132d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559331215b70 con 0x559331325710 2023-09-10T01:17:38.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559331215870 con 0x55933132fe40 2023-09-10T01:17:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593310f9920 con 0x559331326030 2023-09-10T01:17:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcd18d3b80 10 client.0 _ll_register_callbacks cb 0x55933115fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x55933132f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x55933132f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58002/0 (socket says 172.21.15.3:58002) 2023-09-10T01:17:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 -- 172.21.15.3:0/2912077725 learned_addr learned my addr 172.21.15.3:0/2912077725 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb7fff700 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x55933132d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 -- 172.21.15.3:0/2912077725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 msgr2=0x55933132ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 -- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 msgr2=0x55933132d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x55933132d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.351+0000 7efcb77fe700 1 -- 172.21.15.3:0/2912077725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb81a5810 con 0x559331325710 2023-09-10T01:17:38.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efcb77fe700 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x55933132f7a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efca8009880 tx=0x7efca8009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f55bd2e4ed4516 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efc96ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:38.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efc96ffd700 1 -- 172.21.15.3:0/2912077725 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8011ae0 con 0x559331325710 2023-09-10T01:17:38.365 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593311cf4f0 con 0x559331325710 2023-09-10T01:17:38.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efc96ffd700 1 -- 172.21.15.3:0/2912077725 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7efca8011c40 con 0x559331325710 2023-09-10T01:17:38.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efc96ffd700 1 -- 172.21.15.3:0/2912077725 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7efca8011e00 con 0x559331325710 2023-09-10T01:17:38.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efcd18d3b80 10 client.18810 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:38.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55933108cc60 con 0x559331325710 2023-09-10T01:17:38.366 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.352+0000 7efc96ffd700 1 -- 172.21.15.3:0/2912077725 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5593311cf4f0 con 0x559331325710 2023-09-10T01:17:38.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efc96ffd700 1 -- 172.21.15.3:0/2912077725 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7efca801a4d0 con 0x559331325710 2023-09-10T01:17:38.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcd18d3b80 10 client.18810 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:38.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcd18d3b80 10 client.18810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:17:38.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcd18d3b80 20 client.18810 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:38.367 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55933133f7b0 0x559331341bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcd18d3b80 4 client.18810 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:38.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55933128c8a0 con 0x55933133f7b0 2023-09-10T01:17:38.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55933133f7b0 0x559331341bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55933133f7b0 0x559331341bd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0585fd3729853d6 server_cookie=4db7c073fe9a53d9 in_seq=0 out_seq=0 2023-09-10T01:17:38.368 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.353+0000 7efc96ffd700 10 client.18810 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efc96ffd700 1 -- 172.21.15.3:0/2912077725 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55933128c8a0 con 0x55933133f7b0 2023-09-10T01:17:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efc96ffd700 10 client.18810 handle_command_reply: tid=0 2023-09-10T01:17:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 client.18810 shutdown 2023-09-10T01:17:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 msgr2=0x55933132f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x55933132f7a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efca8009880 tx=0x7efca8009e70 comp rx=0 tx=0).stop 2023-09-10T01:17:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 shutdown_connections 2023-09-10T01:17:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x55933132d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x55933132f7a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x55933133f7b0 0x559331341bd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/2912077725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 >> 172.21.15.3:0/2912077725 conn(0x55933129a090 msgr2=0x559331320780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.354+0000 7efc96ffd700 0 client.18810 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.355+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 shutdown_connections 2023-09-10T01:17:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.355+0000 7efcd18d3b80 1 -- 172.21.15.3:0/2912077725 wait complete. 2023-09-10T01:17:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.355+0000 7efcd18d3b80 20 client.18810 trim_cache size 0 max 16384 2023-09-10T01:17:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.368+0000 7efcd18d3b80 1 Processor -- start 2023-09-10T01:17:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcd18d3b80 1 -- start start 2023-09-10T01:17:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331332230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x559331327d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593313269f0 con 0x559331325710 2023-09-10T01:17:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55933132dd70 con 0x55933132fe40 2023-09-10T01:17:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559331341ba0 con 0x559331326030 2023-09-10T01:17:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331332230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331332230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58012/0 (socket says 172.21.15.3:58012) 2023-09-10T01:17:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcb77fe700 1 -- 172.21.15.3:0/261933807 learned_addr learned my addr 172.21.15.3:0/261933807 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcb7fff700 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x559331327d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.369+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efcb77fe700 1 -- 172.21.15.3:0/261933807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 msgr2=0x55933132f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efcb77fe700 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efcb77fe700 1 -- 172.21.15.3:0/261933807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 msgr2=0x559331327d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efcb77fe700 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x559331327d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efcb77fe700 1 -- 172.21.15.3:0/261933807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593310ca250 con 0x559331325710 2023-09-10T01:17:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efcb7fff700 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x559331327d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:17:38.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efcb77fe700 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331332230 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7efca80096b0 tx=0x7efca80117a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fb43e84262576e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efc9e7fc700 1 -- 172.21.15.3:0/261933807 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca801ac20 con 0x559331325710 2023-09-10T01:17:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efc9e7fc700 1 -- 172.21.15.3:0/261933807 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efca801a3b0 con 0x559331325710 2023-09-10T01:17:38.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.370+0000 7efc9e7fc700 1 -- 172.21.15.3:0/261933807 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8005660 con 0x559331325710 2023-09-10T01:17:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 -- 172.21.15.3:0/261933807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 msgr2=0x559331332230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331332230 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7efca80096b0 tx=0x7efca80117a0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 -- 172.21.15.3:0/261933807 shutdown_connections 2023-09-10T01:17:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55933132fe40 0x559331327d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559331325710 0x559331332230 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/261933807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x55933132f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 -- 172.21.15.3:0/261933807 >> 172.21.15.3:0/261933807 conn(0x55933129a090 msgr2=0x55933133bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 -- 172.21.15.3:0/261933807 shutdown_connections 2023-09-10T01:17:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 -- 172.21.15.3:0/261933807 wait complete. 2023-09-10T01:17:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 -- start start 2023-09-10T01:17:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.371+0000 7efcd18d3b80 1 Processor -- start 2023-09-10T01:17:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331325710 0x5593313366f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x559331334970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcd18d3b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55933132fe40 0x559331334eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593313353f0 con 0x55933132fe40 2023-09-10T01:17:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559331335560 con 0x559331325710 2023-09-10T01:17:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcd18d3b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559331335720 con 0x559331326030 2023-09-10T01:17:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcd18d3b80 10 client.0 _ll_register_callbacks cb 0x7efca8013800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331325710 0x5593313366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331325710 0x5593313366f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53780/0 (socket says 172.21.15.3:53780) 2023-09-10T01:17:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcb77fe700 1 -- 172.21.15.3:0/1882057522 learned_addr learned my addr 172.21.15.3:0/1882057522 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcb7fff700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55933132fe40 0x559331334eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.372+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x559331334970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb77fe700 1 -- 172.21.15.3:0/1882057522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 msgr2=0x559331334970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb77fe700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x559331334970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb77fe700 1 -- 172.21.15.3:0/1882057522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55933132fe40 msgr2=0x559331334eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb77fe700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55933132fe40 0x559331334eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb77fe700 1 -- 172.21.15.3:0/1882057522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55933108cc60 con 0x559331325710 2023-09-10T01:17:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x559331334970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb7fff700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55933132fe40 0x559331334eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:17:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcb77fe700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331325710 0x5593313366f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7efca801a7b0 tx=0x7efca8009e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1dcdbb6ce49a011 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efc96ffd700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:17:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efc96ffd700 1 -- 172.21.15.3:0/1882057522 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8005180 con 0x559331325710 2023-09-10T01:17:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efcac0a1620 con 0x559331325710 2023-09-10T01:17:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efc96ffd700 1 -- 172.21.15.3:0/1882057522 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7efca801ac30 con 0x559331325710 2023-09-10T01:17:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efc96ffd700 1 -- 172.21.15.3:0/1882057522 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7efca8005380 con 0x559331325710 2023-09-10T01:17:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcd18d3b80 10 client.28949 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.373+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5593310a22b0 con 0x559331325710 2023-09-10T01:17:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efc96ffd700 1 -- 172.21.15.3:0/1882057522 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcac0a1620 con 0x559331325710 2023-09-10T01:17:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efc96ffd700 1 -- 172.21.15.3:0/1882057522 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7efca8023040 con 0x559331325710 2023-09-10T01:17:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcd18d3b80 10 client.28949 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcd18d3b80 10 client.28949 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:17:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcd18d3b80 20 client.28949 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5593313945a0 0x5593313969c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcd18d3b80 4 client.28949 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:17:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559331270920 con 0x5593313945a0 2023-09-10T01:17:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5593313945a0 0x5593313969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efcb6ffd700 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5593313945a0 0x5593313969c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=43c28a4122fc85e4 server_cookie=6524c2038be98f81 in_seq=0 out_seq=0 2023-09-10T01:17:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.374+0000 7efc96ffd700 10 client.28949 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efc96ffd700 1 -- 172.21.15.3:0/1882057522 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559331270920 con 0x5593313945a0 2023-09-10T01:17:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efc96ffd700 10 client.28949 handle_command_reply: tid=0 2023-09-10T01:17:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 client.28949 shutdown 2023-09-10T01:17:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331325710 msgr2=0x5593313366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331325710 0x5593313366f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7efca801a7b0 tx=0x7efca8009e40 comp rx=0 tx=0).stop 2023-09-10T01:17:38.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 shutdown_connections 2023-09-10T01:17:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55933132fe40 0x559331334eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559331325710 0x5593313366f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x5593313945a0 0x5593313969c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 --2- 172.21.15.3:0/1882057522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559331326030 0x559331334970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efc96ffd700 0 client.28949 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:17:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 >> 172.21.15.3:0/1882057522 conn(0x55933129a090 msgr2=0x559331327d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 shutdown_connections 2023-09-10T01:17:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 1 -- 172.21.15.3:0/1882057522 wait complete. 2023-09-10T01:17:38.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.375+0000 7efcd18d3b80 20 client.28949 trim_cache size 0 max 16384 2023-09-10T01:17:38.395 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:17:38.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efca00a07b0 msgr2=0x7efca00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efca00a07b0 0x7efca00a2c70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009fd0 tx=0x7efcb0009480 comp rx=0 tx=0).stop 2023-09-10T01:17:38.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb8073350 msgr2=0x7efcb819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb8073350 0x7efcb819ed60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7efcac00e9f0 tx=0x7efcac00ed00 comp rx=0 tx=0).stop 2023-09-10T01:17:38.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 shutdown_connections 2023-09-10T01:17:38.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efca00a07b0 0x7efca00a2c70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb810ce80 0x7efcb8198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efcb8072a30 0x7efcb819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 --2- 172.21.15.3:0/3626080462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efcb8073350 0x7efcb819ed60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 >> 172.21.15.3:0/3626080462 conn(0x7efcb80fc170 msgr2=0x7efcb81048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 shutdown_connections 2023-09-10T01:17:38.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.376+0000 7efcbdfff700 1 -- 172.21.15.3:0/3626080462 wait complete. 2023-09-10T01:17:38.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:17:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:38 smithi003 ceph-mon[97654]: pgmap v452: 129 pgs: 129 active+clean; 726 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.0 MiB/s wr, 569 op/s 2023-09-10T01:17:38.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:38 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3222598851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:17:38.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.604+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1365917256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 msgr2=0x7f0d54072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.604+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1365917256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 0x7f0d54072e10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009990 tx=0x7f0d50009ca0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.605+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1365917256 shutdown_connections 2023-09-10T01:17:38.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.605+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1365917256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5410ce80 0x7f0d5410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.606 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.605+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1365917256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 0x7f0d54072e10 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.605+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1365917256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d54073350 0x7f0d5410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.605+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1365917256 >> 172.21.15.3:0/1365917256 conn(0x7f0d540fc170 msgr2=0x7f0d540fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.605+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1365917256 shutdown_connections 2023-09-10T01:17:38.607 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.605+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1365917256 wait complete. 2023-09-10T01:17:38.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.606+0000 7f0d5b5ff700 1 Processor -- start 2023-09-10T01:17:38.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.606+0000 7f0d5b5ff700 1 -- start start 2023-09-10T01:17:38.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5b5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 0x7f0d54198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5b5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d54073350 0x7f0d54198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5b5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d5410ce80 0x7f0d5419d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5b5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5419d920 con 0x7f0d54073350 2023-09-10T01:17:38.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5b5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5419da90 con 0x7f0d54072a30 2023-09-10T01:17:38.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5b5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5419dc50 con 0x7f0d5410ce80 2023-09-10T01:17:38.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d58b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d54073350 0x7f0d54198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 0x7f0d54198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d58b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d54073350 0x7f0d54198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58046/0 (socket says 172.21.15.3:58046) 2023-09-10T01:17:38.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d58b9a700 1 -- 172.21.15.3:0/1328901191 learned_addr learned my addr 172.21.15.3:0/1328901191 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 0x7f0d54198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53796/0 (socket says 172.21.15.3:53796) 2023-09-10T01:17:38.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.607+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d5410ce80 0x7f0d5419d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d58b9a700 1 -- 172.21.15.3:0/1328901191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d5410ce80 msgr2=0x7f0d5419d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d5410ce80 0x7f0d5419d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d58b9a700 1 -- 172.21.15.3:0/1328901191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 msgr2=0x7f0d54198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 0x7f0d54198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d58b9a700 1 -- 172.21.15.3:0/1328901191 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d500095f0 con 0x7f0d54073350 2023-09-10T01:17:38.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d54073350 0x7f0d54198ec0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400e9a0 tx=0x7f0d4400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d4a7fc700 1 -- 172.21.15.3:0/1328901191 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44009790 con 0x7f0d54073350 2023-09-10T01:17:38.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5419df30 con 0x7f0d54073350 2023-09-10T01:17:38.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d4a7fc700 1 -- 172.21.15.3:0/1328901191 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d44010430 con 0x7f0d54073350 2023-09-10T01:17:38.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d4a7fc700 1 -- 172.21.15.3:0/1328901191 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44018680 con 0x7f0d54073350 2023-09-10T01:17:38.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.608+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d541a57e0 con 0x7f0d54073350 2023-09-10T01:17:38.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.611+0000 7f0d4a7fc700 1 -- 172.21.15.3:0/1328901191 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f0d440105a0 con 0x7f0d54073350 2023-09-10T01:17:38.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.613+0000 7f0d6ee45b80 1 Processor -- start 2023-09-10T01:17:38.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.613+0000 7f0d6ee45b80 1 -- start start 2023-09-10T01:17:38.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d4a7fc700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d400a07b0 0x7f0d400a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d4a7fc700 1 -- 172.21.15.3:0/1328901191 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f0d44014070 con 0x7f0d54073350 2023-09-10T01:17:38.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563dfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563e9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 0x5610563ec230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561056338730 con 0x5610563df710 2023-09-10T01:17:38.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x561056336050 con 0x5610563e9e40 2023-09-10T01:17:38.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x561056331940 con 0x5610563e0030 2023-09-10T01:17:38.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d400a07b0 0x7f0d400a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563dfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563dfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58048/0 (socket says 172.21.15.3:58048) 2023-09-10T01:17:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d5939b700 1 -- 172.21.15.3:0/3131147455 learned_addr learned my addr 172.21.15.3:0/3131147455 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.614+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 0x5610563ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d58b9a700 1 -- 172.21.15.3:0/3131147455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 msgr2=0x5610563ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 0x5610563ec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d58b9a700 1 -- 172.21.15.3:0/3131147455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 msgr2=0x5610563dfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563dfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d58b9a700 1 -- 172.21.15.3:0/3131147455 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56105615dde0 con 0x5610563e0030 2023-09-10T01:17:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d5939b700 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563dfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d400a07b0 0x7f0d400a2c70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00f870 tx=0x7f0d4c00fc30 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563e9900 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0d440c98a0 tx=0x7f0d440994e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a76fa11705b60e96 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d3f7fe700 1 -- 172.21.15.3:0/3131147455 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4409d430 con 0x5610563e0030 2023-09-10T01:17:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d3f7fe700 1 -- 172.21.15.3:0/3131147455 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d440a1670 con 0x5610563e0030 2023-09-10T01:17:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.615+0000 7f0d3f7fe700 1 -- 172.21.15.3:0/3131147455 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44084070 con 0x5610563e0030 2023-09-10T01:17:38.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.616+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/3131147455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 msgr2=0x5610563e9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.616+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563e9900 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0d440c98a0 tx=0x7f0d440994e0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.616+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/3131147455 shutdown_connections 2023-09-10T01:17:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.616+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 0x5610563ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.616+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563dfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.616+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/3131147455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563e9900 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.616+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/3131147455 >> 172.21.15.3:0/3131147455 conn(0x561056354090 msgr2=0x5610563dabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.620+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/3131147455 shutdown_connections 2023-09-10T01:17:38.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.620+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/3131147455 wait complete. 2023-09-10T01:17:38.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.621+0000 7f0d6ee45b80 1 -- start start 2023-09-10T01:17:38.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.621+0000 7f0d6ee45b80 1 Processor -- start 2023-09-10T01:17:38.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563df710 0x5610563e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x5610563e6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563e9e40 0x5610563e7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610562cfb70 con 0x5610563e9e40 2023-09-10T01:17:38.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610562cf870 con 0x5610563e0030 2023-09-10T01:17:38.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610561b3920 con 0x5610563df710 2023-09-10T01:17:38.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d6ee45b80 10 client.0 _ll_register_callbacks cb 0x561056219a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:38.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563df710 0x5610563e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563df710 0x5610563e98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37044/0 (socket says 172.21.15.3:37044) 2023-09-10T01:17:38.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d5939b700 1 -- 172.21.15.3:0/998493926 learned_addr learned my addr 172.21.15.3:0/998493926 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.622+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563e9e40 0x5610563e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.623+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x5610563e6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.623+0000 7f0d59b9c700 1 -- 172.21.15.3:0/998493926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563df710 msgr2=0x5610563e98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.623+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563df710 0x5610563e98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.623+0000 7f0d59b9c700 1 -- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 msgr2=0x5610563e6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.623+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x5610563e6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.623+0000 7f0d59b9c700 1 -- 172.21.15.3:0/998493926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5410ffb0 con 0x5610563e9e40 2023-09-10T01:17:38.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.624+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563e9e40 0x5610563e7240 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c018ad0 tx=0x7f0d4c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c681992932d99cf5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.628+0000 7f0d3cff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.628+0000 7f0d3cff9700 1 -- 172.21.15.3:0/998493926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00e040 con 0x5610563e9e40 2023-09-10T01:17:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.628+0000 7f0d3cff9700 1 -- 172.21.15.3:0/998493926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f0d4c008920 con 0x5610563e9e40 2023-09-10T01:17:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.628+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610562894f0 con 0x5610563e9e40 2023-09-10T01:17:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.630+0000 7f0d3cff9700 1 -- 172.21.15.3:0/998493926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d4c008ae0 con 0x5610563e9e40 2023-09-10T01:17:38.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.631+0000 7f0d6ee45b80 10 client.18828 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:38.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.631+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x7f0d541a57e0 con 0x5610563e9e40 2023-09-10T01:17:38.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.632+0000 7f0d3cff9700 1 -- 172.21.15.3:0/998493926 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5610562894f0 con 0x5610563e9e40 2023-09-10T01:17:38.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d3cff9700 1 -- 172.21.15.3:0/998493926 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0d4c024b40 con 0x5610563e9e40 2023-09-10T01:17:38.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d6ee45b80 10 client.18828 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:38.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d6ee45b80 10 client.18828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:38.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d6ee45b80 20 client.18828 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:38.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610563f79b0 0x5610563f9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d6ee45b80 4 client.18828 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:38.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610563468a0 con 0x5610563f79b0 2023-09-10T01:17:38.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.633+0000 7f0d5939b700 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610563f79b0 0x5610563f9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.641+0000 7f0d3cff9700 10 client.18828 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:38.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.641+0000 7f0d5939b700 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610563f79b0 0x5610563f9dd0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=161ad7281fdc01e3 server_cookie=a77ecc664d45145 in_seq=0 out_seq=0 2023-09-10T01:17:38.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.641+0000 7f0d3cff9700 1 -- 172.21.15.3:0/998493926 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5610563468a0 con 0x5610563f79b0 2023-09-10T01:17:38.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.641+0000 7f0d3cff9700 10 client.18828 handle_command_reply: tid=0 2023-09-10T01:17:38.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.642+0000 7f0d6ee45b80 1 client.18828 shutdown 2023-09-10T01:17:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.646+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563e9e40 msgr2=0x5610563e7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.646+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563e9e40 0x5610563e7240 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c018ad0 tx=0x7f0d4c004b40 comp rx=0 tx=0).stop 2023-09-10T01:17:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 shutdown_connections 2023-09-10T01:17:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610563f79b0 0x5610563f9dd0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563df710 0x5610563e98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563e9e40 0x5610563e7240 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/998493926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x5610563e6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d3cff9700 0 client.18828 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:38.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 >> 172.21.15.3:0/998493926 conn(0x561056354090 msgr2=0x5610563da7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 shutdown_connections 2023-09-10T01:17:38.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.647+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/998493926 wait complete. 2023-09-10T01:17:38.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.648+0000 7f0d6ee45b80 20 client.18828 trim_cache size 0 max 16384 2023-09-10T01:17:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:38 smithi136 ceph-mon[133708]: pgmap v452: 129 pgs: 129 active+clean; 726 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.0 MiB/s wr, 569 op/s 2023-09-10T01:17:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:38 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3222598851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:17:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.672+0000 7f0d6ee45b80 1 Processor -- start 2023-09-10T01:17:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.675+0000 7f0d6ee45b80 1 -- start start 2023-09-10T01:17:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.675+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563f0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.675+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.675+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 0x5610563e9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.675+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610563d7410 con 0x5610563df710 2023-09-10T01:17:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.675+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610563f0ed0 con 0x5610563e9e40 2023-09-10T01:17:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.675+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610563da4d0 con 0x5610563e0030 2023-09-10T01:17:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.676+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.676+0000 7f0d5939b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563f0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58068/0 (socket says 172.21.15.3:58068) 2023-09-10T01:17:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.676+0000 7f0d5939b700 1 -- 172.21.15.3:0/2477773120 learned_addr learned my addr 172.21.15.3:0/2477773120 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.677+0000 7f0d58b9a700 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.677+0000 7f0d5939b700 1 -- 172.21.15.3:0/2477773120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 msgr2=0x5610563f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.677+0000 7f0d5939b700 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563f30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.677+0000 7f0d5939b700 1 -- 172.21.15.3:0/2477773120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 msgr2=0x5610563e9420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.677+0000 7f0d5939b700 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 0x5610563e9420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.677+0000 7f0d5939b700 1 -- 172.21.15.3:0/2477773120 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561056146c60 con 0x5610563df710 2023-09-10T01:17:38.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.682+0000 7f0d5939b700 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563f0990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0d50000c00 tx=0x7f0d50005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfaa2ba6bbae700 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.682+0000 7f0d3ffff700 1 -- 172.21.15.3:0/2477773120 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5001d040 con 0x5610563df710 2023-09-10T01:17:38.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.682+0000 7f0d3ffff700 1 -- 172.21.15.3:0/2477773120 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d5001e040 con 0x5610563df710 2023-09-10T01:17:38.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.682+0000 7f0d3ffff700 1 -- 172.21.15.3:0/2477773120 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50005330 con 0x5610563df710 2023-09-10T01:17:38.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.683+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/2477773120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 msgr2=0x5610563f0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.683+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563f0990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0d50000c00 tx=0x7f0d50005fb0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/2477773120 shutdown_connections 2023-09-10T01:17:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e9e40 0x5610563e9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x5610563f0990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/2477773120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e0030 0x5610563f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/2477773120 >> 172.21.15.3:0/2477773120 conn(0x561056354090 msgr2=0x5610563d9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/2477773120 shutdown_connections 2023-09-10T01:17:38.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/2477773120 wait complete. 2023-09-10T01:17:38.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.684+0000 7f0d6ee45b80 1 -- start start 2023-09-10T01:17:38.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.685+0000 7f0d6ee45b80 1 Processor -- start 2023-09-10T01:17:38.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.686+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x56105648d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.686+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x56105648d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.686+0000 7f0d6ee45b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e9e40 0x56105648de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.686+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610563edff0 con 0x5610563df710 2023-09-10T01:17:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.686+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610563ee130 con 0x5610563e0030 2023-09-10T01:17:38.691 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.686+0000 7f0d6ee45b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610563ee2f0 con 0x5610563e9e40 2023-09-10T01:17:38.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.686+0000 7f0d6ee45b80 10 client.0 _ll_register_callbacks cb 0x561056219a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:38.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.687+0000 7f0d58b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x56105648d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.687+0000 7f0d58b9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x56105648d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53828/0 (socket says 172.21.15.3:53828) 2023-09-10T01:17:38.692 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.687+0000 7f0d58b9a700 1 -- 172.21.15.3:0/865179603 learned_addr learned my addr 172.21.15.3:0/865179603 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:38.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.687+0000 7f0d5939b700 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x56105648d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.688+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e9e40 0x56105648de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.688+0000 7f0d59b9c700 1 -- 172.21.15.3:0/865179603 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 msgr2=0x56105648d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.688+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x56105648d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.688+0000 7f0d59b9c700 1 -- 172.21.15.3:0/865179603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 msgr2=0x56105648d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:38.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.688+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x56105648d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.688+0000 7f0d59b9c700 1 -- 172.21.15.3:0/865179603 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561056184250 con 0x5610563e9e40 2023-09-10T01:17:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.689+0000 7f0d59b9c700 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e9e40 0x56105648de00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c018de0 tx=0x7f0d4c00f530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc3bd2cb062b9df3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.690+0000 7f0d3cff9700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.691+0000 7f0d3cff9700 1 -- 172.21.15.3:0/865179603 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00e030 con 0x5610563e9e40 2023-09-10T01:17:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.691+0000 7f0d3cff9700 1 -- 172.21.15.3:0/865179603 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f0d4c004db0 con 0x5610563e9e40 2023-09-10T01:17:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.691+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d440a1670 con 0x5610563e9e40 2023-09-10T01:17:38.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.692+0000 7f0d3cff9700 1 -- 172.21.15.3:0/865179603 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d4c018640 con 0x5610563e9e40 2023-09-10T01:17:38.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.692+0000 7f0d6ee45b80 10 client.28825 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:38.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.692+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56105615c2b0 con 0x5610563e9e40 2023-09-10T01:17:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d3cff9700 1 -- 172.21.15.3:0/865179603 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d4c044030 con 0x5610563e9e40 2023-09-10T01:17:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d3cff9700 1 -- 172.21.15.3:0/865179603 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0d4c03e060 con 0x5610563e9e40 2023-09-10T01:17:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d6ee45b80 10 client.28825 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d6ee45b80 10 client.28825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:38.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d6ee45b80 20 client.28825 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:38.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610564a3d50 0x5610564a6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:38.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d6ee45b80 4 client.28825 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:17:38.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.695+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56105632a920 con 0x5610564a3d50 2023-09-10T01:17:38.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.696+0000 7f0d5939b700 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610564a3d50 0x5610564a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:38.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.697+0000 7f0d5939b700 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610564a3d50 0x5610564a6170 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=557039cca005b8d9 server_cookie=9ce75fd6c0ef25a3 in_seq=0 out_seq=0 2023-09-10T01:17:38.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.697+0000 7f0d3cff9700 10 client.28825 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.704+0000 7f0d3cff9700 1 -- 172.21.15.3:0/865179603 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56105632a920 con 0x5610564a3d50 2023-09-10T01:17:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.704+0000 7f0d3cff9700 10 client.28825 handle_command_reply: tid=0 2023-09-10T01:17:38.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.704+0000 7f0d6ee45b80 1 client.28825 shutdown 2023-09-10T01:17:38.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e9e40 msgr2=0x56105648de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e9e40 0x56105648de00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c018de0 tx=0x7f0d4c00f530 comp rx=0 tx=0).stop 2023-09-10T01:17:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 shutdown_connections 2023-09-10T01:17:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5610564a3d50 0x5610564a6170 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5610563e9e40 0x56105648de00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5610563df710 0x56105648d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 --2- 172.21.15.3:0/865179603 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5610563e0030 0x56105648d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 >> 172.21.15.3:0/865179603 conn(0x561056354090 msgr2=0x5610563f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.705+0000 7f0d3cff9700 0 client.28825 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:38.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.706+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 shutdown_connections 2023-09-10T01:17:38.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.706+0000 7f0d6ee45b80 1 -- 172.21.15.3:0/865179603 wait complete. 2023-09-10T01:17:38.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.706+0000 7f0d6ee45b80 20 client.28825 trim_cache size 0 max 16384 2023-09-10T01:17:38.710 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:17:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:38 smithi092 ceph-mon[134067]: pgmap v452: 129 pgs: 129 active+clean; 726 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.0 MiB/s wr, 569 op/s 2023-09-10T01:17:38.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:38 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3222598851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:17:38.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d400a07b0 msgr2=0x7f0d400a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d400a07b0 0x7f0d400a2c70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00f870 tx=0x7f0d4c00fc30 comp rx=0 tx=0).stop 2023-09-10T01:17:38.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d54073350 msgr2=0x7f0d54198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:38.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d54073350 0x7f0d54198ec0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400e9a0 tx=0x7f0d4400ecb0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 shutdown_connections 2023-09-10T01:17:38.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0d400a07b0 0x7f0d400a2c70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d5410ce80 0x7f0d5419d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d54072a30 0x7f0d54198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 --2- 172.21.15.3:0/1328901191 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d54073350 0x7f0d54198ec0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:38.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 >> 172.21.15.3:0/1328901191 conn(0x7f0d540fc170 msgr2=0x7f0d5410efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:38.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 shutdown_connections 2023-09-10T01:17:38.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:38.724+0000 7f0d5b5ff700 1 -- 172.21.15.3:0/1328901191 wait complete. 2023-09-10T01:17:39.179 INFO:tasks.workunit.client.0.smithi003.stdout: 10 44279 32.04 MB/sec warmup 80 sec latency 361.166 ms 2023-09-10T01:17:39.768 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:17:39.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:17:40.083 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.083+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/1325784328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c88108220 msgr2=0x7f8c8810bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.084 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.083+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/1325784328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c88108220 0x7f8c8810bce0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0099c0 tx=0x7f8c7c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.087 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.085+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/1325784328 shutdown_connections 2023-09-10T01:17:40.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.085+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/1325784328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8810c410 0x7f8c88113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.085+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/1325784328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c88107870 0x7f8c88107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.085+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/1325784328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c88108220 0x7f8c8810bce0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.088 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.085+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/1325784328 >> 172.21.15.3:0/1325784328 conn(0x7f8c88072090 msgr2=0x7f8c8806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.085+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/1325784328 shutdown_connections 2023-09-10T01:17:40.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.085+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/1325784328 wait complete. 2023-09-10T01:17:40.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.086+0000 7f8c8d9ff700 1 Processor -- start 2023-09-10T01:17:40.089 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.087+0000 7f8c8d9ff700 1 -- start start 2023-09-10T01:17:40.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.087+0000 7f8c8d9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c88107870 0x7f8c8807a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.090 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.087+0000 7f8c8d9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c8810c410 0x7f8c8807ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.087+0000 7f8c8d9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c881133a0 0x7f8c88113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.087+0000 7f8c8d9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c88113e20 con 0x7f8c88107870 2023-09-10T01:17:40.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.087+0000 7f8c8d9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c88113f90 con 0x7f8c8810c410 2023-09-10T01:17:40.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.087+0000 7f8c8d9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c88114150 con 0x7f8c881133a0 2023-09-10T01:17:40.091 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c881133a0 0x7f8c88113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c881133a0 0x7f8c88113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37088/0 (socket says 172.21.15.3:37088) 2023-09-10T01:17:40.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 -- 172.21.15.3:0/3787114033 learned_addr learned my addr 172.21.15.3:0/3787114033 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c867fc700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c8810c410 0x7f8c8807ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c88107870 0x7f8c8807a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.092 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 -- 172.21.15.3:0/3787114033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c8810c410 msgr2=0x7f8c8807ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c8810c410 0x7f8c8807ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 -- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c88107870 msgr2=0x7f8c8807a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c88107870 0x7f8c8807a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.093 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.088+0000 7f8c877fe700 1 -- 172.21.15.3:0/3787114033 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0095f0 con 0x7f8c881133a0 2023-09-10T01:17:40.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.089+0000 7f8c877fe700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c881133a0 0x7f8c88113850 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000b780 tx=0x7f8c8000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.089+0000 7f8c8c9fd700 1 -- 172.21.15.3:0/3787114033 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80010860 con 0x7f8c881133a0 2023-09-10T01:17:40.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.089+0000 7f8c8c9fd700 1 -- 172.21.15.3:0/3787114033 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c80010e80 con 0x7f8c881133a0 2023-09-10T01:17:40.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.089+0000 7f8c8c9fd700 1 -- 172.21.15.3:0/3787114033 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80007370 con 0x7f8c881133a0 2023-09-10T01:17:40.094 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.090+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c88081550 con 0x7f8c881133a0 2023-09-10T01:17:40.095 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.091+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c88081a60 con 0x7f8c881133a0 2023-09-10T01:17:40.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.100+0000 7f8ca1397b80 1 Processor -- start 2023-09-10T01:17:40.103 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.101+0000 7f8c8c9fd700 1 -- 172.21.15.3:0/3787114033 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f8c8000f640 con 0x7f8c881133a0 2023-09-10T01:17:40.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.101+0000 7f8ca1397b80 1 -- start start 2023-09-10T01:17:40.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.102+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699eb9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.102+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 0x564699ec3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.102+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.104 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.102+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699e127b0 con 0x564699ec3ec0 2023-09-10T01:17:40.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.102+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699e100d0 con 0x564699eba0b0 2023-09-10T01:17:40.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.102+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699e0b9c0 con 0x564699eb9790 2023-09-10T01:17:40.105 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.103+0000 7f8c8c9fd700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c700a0830 0x7f8c700a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.104+0000 7f8c8c9fd700 1 -- 172.21.15.3:0/3787114033 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f8c800cc160 con 0x7f8c881133a0 2023-09-10T01:17:40.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.104+0000 7f8c877fe700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c700a0830 0x7f8c700a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.105+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699eb9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.105+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699eb9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37094/0 (socket says 172.21.15.3:37094) 2023-09-10T01:17:40.106 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.105+0000 7f8c867fc700 1 -- 172.21.15.3:0/791043705 learned_addr learned my addr 172.21.15.3:0/791043705 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.105+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 0x564699ec3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.105+0000 7f8c877fe700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c700a0830 0x7f8c700a2cf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8c800c94f0 tx=0x7f8c8000f940 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.106+0000 7f8c867fc700 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.106+0000 7f8c867fc700 1 -- 172.21.15.3:0/791043705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 msgr2=0x564699ec3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.106+0000 7f8c867fc700 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 0x564699ec3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.106+0000 7f8c867fc700 1 -- 172.21.15.3:0/791043705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 msgr2=0x564699ec62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.106+0000 7f8c867fc700 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.106+0000 7f8c867fc700 1 -- 172.21.15.3:0/791043705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564699c37e60 con 0x564699eb9790 2023-09-10T01:17:40.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.106+0000 7f8c867fc700 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699eb9b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0099c0 tx=0x7f8c7c01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2691a51badc03d7 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.107+0000 7f8c6dffb700 1 -- 172.21.15.3:0/791043705 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c005050 con 0x564699eb9790 2023-09-10T01:17:40.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.108+0000 7f8c6dffb700 1 -- 172.21.15.3:0/791043705 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c7c00bd80 con 0x564699eb9790 2023-09-10T01:17:40.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.108+0000 7f8c6dffb700 1 -- 172.21.15.3:0/791043705 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c003ec0 con 0x564699eb9790 2023-09-10T01:17:40.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.109+0000 7f8ca1397b80 1 -- 172.21.15.3:0/791043705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 msgr2=0x564699eb9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.109+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699eb9b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0099c0 tx=0x7f8c7c01a040 comp rx=0 tx=0).stop 2023-09-10T01:17:40.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.109+0000 7f8ca1397b80 1 -- 172.21.15.3:0/791043705 shutdown_connections 2023-09-10T01:17:40.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.109+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699eb9b70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.109+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.111 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.109+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/791043705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 0x564699ec3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.109+0000 7f8ca1397b80 1 -- 172.21.15.3:0/791043705 >> 172.21.15.3:0/791043705 conn(0x564699e2e110 msgr2=0x564699eb4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.112+0000 7f8ca1397b80 1 -- 172.21.15.3:0/791043705 shutdown_connections 2023-09-10T01:17:40.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.112+0000 7f8ca1397b80 1 -- 172.21.15.3:0/791043705 wait complete. 2023-09-10T01:17:40.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.113+0000 7f8ca1397b80 1 -- start start 2023-09-10T01:17:40.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.113+0000 7f8ca1397b80 1 Processor -- start 2023-09-10T01:17:40.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.115+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699ec0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.115+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 0x564699ec1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.115+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.115+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699da9bf0 con 0x564699ec3ec0 2023-09-10T01:17:40.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.115+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699da98f0 con 0x564699eba0b0 2023-09-10T01:17:40.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.115+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699c8d9a0 con 0x564699eb9790 2023-09-10T01:17:40.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.115+0000 7f8ca1397b80 10 client.0 _ll_register_callbacks cb 0x564699cf3b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:40.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.116+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699ec0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.117+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.117+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58104/0 (socket says 172.21.15.3:58104) 2023-09-10T01:17:40.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.117+0000 7f8c867fc700 1 -- 172.21.15.3:0/66195318 learned_addr learned my addr 172.21.15.3:0/66195318 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.117+0000 7f8c867fc700 1 -- 172.21.15.3:0/66195318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 msgr2=0x564699ec1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.119 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.117+0000 7f8c867fc700 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 0x564699ec1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.117+0000 7f8c867fc700 1 -- 172.21.15.3:0/66195318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 msgr2=0x564699ec1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.121 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.117+0000 7f8c867fc700 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.118+0000 7f8c867fc700 1 -- 172.21.15.3:0/66195318 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c020040 con 0x564699eb9790 2023-09-10T01:17:40.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.118+0000 7f8c867fc700 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699ec0c00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c000c00 tx=0x7f8c7c003aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33d3441116e7185f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.118+0000 7f8c66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:40.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.118+0000 7f8c66ffd700 1 -- 172.21.15.3:0/66195318 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c005220 con 0x564699eb9790 2023-09-10T01:17:40.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.118+0000 7f8c66ffd700 1 -- 172.21.15.3:0/66195318 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f8c7c005380 con 0x564699eb9790 2023-09-10T01:17:40.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.118+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564699d63570 con 0x564699eb9790 2023-09-10T01:17:40.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.119+0000 7f8c66ffd700 1 -- 172.21.15.3:0/66195318 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c7c00bb90 con 0x564699eb9790 2023-09-10T01:17:40.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.120+0000 7f8ca1397b80 10 client.28834 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:40.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.120+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x564699c20ce0 con 0x564699eb9790 2023-09-10T01:17:40.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.120+0000 7f8c66ffd700 1 -- 172.21.15.3:0/66195318 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564699d63570 con 0x564699eb9790 2023-09-10T01:17:40.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8c66ffd700 1 -- 172.21.15.3:0/66195318 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f8c7c004b60 con 0x564699eb9790 2023-09-10T01:17:40.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8ca1397b80 10 client.28834 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:40.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8ca1397b80 10 client.28834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:40.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8ca1397b80 20 client.28834 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:40.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699ed35f0 0x564699ed5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8ca1397b80 4 client.28834 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:40.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564699e20920 con 0x564699ed35f0 2023-09-10T01:17:40.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.121+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699ed35f0 0x564699ed5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.122+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699ed35f0 0x564699ed5a10 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf91bccef3c3e31 server_cookie=896e3ae2220103d7 in_seq=0 out_seq=0 2023-09-10T01:17:40.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.122+0000 7f8c66ffd700 10 client.28834 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.123+0000 7f8c66ffd700 1 -- 172.21.15.3:0/66195318 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564699e20920 con 0x564699ed35f0 2023-09-10T01:17:40.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.123+0000 7f8c66ffd700 10 client.28834 handle_command_reply: tid=0 2023-09-10T01:17:40.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.123+0000 7f8ca1397b80 1 client.28834 shutdown 2023-09-10T01:17:40.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.123+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 msgr2=0x564699ec0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.123+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699ec0c00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c000c00 tx=0x7f8c7c003aa0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 shutdown_connections 2023-09-10T01:17:40.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699ed35f0 0x564699ed5a10 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699ec0c00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ec1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/66195318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eba0b0 0x564699ec1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 >> 172.21.15.3:0/66195318 conn(0x564699e2e110 msgr2=0x564699eb4860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8c66ffd700 0 client.28834 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.124+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 shutdown_connections 2023-09-10T01:17:40.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.125+0000 7f8ca1397b80 1 -- 172.21.15.3:0/66195318 wait complete. 2023-09-10T01:17:40.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.125+0000 7f8ca1397b80 20 client.28834 trim_cache size 0 max 16384 2023-09-10T01:17:40.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 Processor -- start 2023-09-10T01:17:40.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 -- start start 2023-09-10T01:17:40.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eb9790 0x564699ec14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eba0b0 0x564699ebde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ebe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699ebaa90 con 0x564699ec3ec0 2023-09-10T01:17:40.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699eaedf0 con 0x564699eb9790 2023-09-10T01:17:40.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.154+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699ecff70 con 0x564699eba0b0 2023-09-10T01:17:40.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.155+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eb9790 0x564699ec14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.155+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eb9790 0x564699ec14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53878/0 (socket says 172.21.15.3:53878) 2023-09-10T01:17:40.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.155+0000 7f8c867fc700 1 -- 172.21.15.3:0/1733375941 learned_addr learned my addr 172.21.15.3:0/1733375941 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.155+0000 7f8c867fc700 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ebe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.155+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eba0b0 0x564699ebde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.156+0000 7f8c86ffd700 1 -- 172.21.15.3:0/1733375941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eb9790 msgr2=0x564699ec14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.156+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eb9790 0x564699ec14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.156+0000 7f8c86ffd700 1 -- 172.21.15.3:0/1733375941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 msgr2=0x564699ebe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.156+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ebe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.156+0000 7f8c86ffd700 1 -- 172.21.15.3:0/1733375941 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564699c5e2d0 con 0x564699eba0b0 2023-09-10T01:17:40.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.156+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eba0b0 0x564699ebde70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x564699ed62c0 tx=0x7f8c78018de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fbad9c24972e284 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.156+0000 7f8c6e7fc700 1 -- 172.21.15.3:0/1733375941 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7800bda0 con 0x564699eba0b0 2023-09-10T01:17:40.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.157+0000 7f8c6e7fc700 1 -- 172.21.15.3:0/1733375941 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c78018570 con 0x564699eba0b0 2023-09-10T01:17:40.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.157+0000 7f8c6e7fc700 1 -- 172.21.15.3:0/1733375941 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78015dc0 con 0x564699eba0b0 2023-09-10T01:17:40.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.157+0000 7f8ca1397b80 1 -- 172.21.15.3:0/1733375941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eba0b0 msgr2=0x564699ebde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.158+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eba0b0 0x564699ebde70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x564699ed62c0 tx=0x7f8c78018de0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.158+0000 7f8ca1397b80 1 -- 172.21.15.3:0/1733375941 shutdown_connections 2023-09-10T01:17:40.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.158+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699ec3ec0 0x564699ebe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.158+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699eb9790 0x564699ec14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.158+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/1733375941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eba0b0 0x564699ebde70 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.158+0000 7f8ca1397b80 1 -- 172.21.15.3:0/1733375941 >> 172.21.15.3:0/1733375941 conn(0x564699e2e110 msgr2=0x564699eb9010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.158+0000 7f8ca1397b80 1 -- 172.21.15.3:0/1733375941 shutdown_connections 2023-09-10T01:17:40.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.159+0000 7f8ca1397b80 1 -- 172.21.15.3:0/1733375941 wait complete. 2023-09-10T01:17:40.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.160+0000 7f8ca1397b80 1 -- start start 2023-09-10T01:17:40.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.160+0000 7f8ca1397b80 1 Processor -- start 2023-09-10T01:17:40.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699f219c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699eba0b0 0x564699f21f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8ca1397b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699ec3ec0 0x564699ec9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699eca020 con 0x564699eba0b0 2023-09-10T01:17:40.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699eca190 con 0x564699ec3ec0 2023-09-10T01:17:40.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8ca1397b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x564699eca350 con 0x564699eb9790 2023-09-10T01:17:40.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8ca1397b80 10 client.0 _ll_register_callbacks cb 0x7f8c7c00bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:40.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699f219c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8c867fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699f219c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37132/0 (socket says 172.21.15.3:37132) 2023-09-10T01:17:40.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8c867fc700 1 -- 172.21.15.3:0/4022694854 learned_addr learned my addr 172.21.15.3:0/4022694854 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.161+0000 7f8c867fc700 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699ec3ec0 0x564699ec9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.162+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699eba0b0 0x564699f21f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.162+0000 7f8c867fc700 1 -- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699ec3ec0 msgr2=0x564699ec9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.162+0000 7f8c867fc700 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699ec3ec0 0x564699ec9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.162+0000 7f8c867fc700 1 -- 172.21.15.3:0/4022694854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699eba0b0 msgr2=0x564699f21f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.162+0000 7f8c867fc700 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699eba0b0 0x564699f21f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.162+0000 7f8c867fc700 1 -- 172.21.15.3:0/4022694854 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7800e040 con 0x564699eb9790 2023-09-10T01:17:40.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.165+0000 7f8c867fc700 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699f219c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c005b10 tx=0x7f8c7c003a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17156b5ba2213447 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.165+0000 7f8c66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:40.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.166+0000 7f8c66ffd700 1 -- 172.21.15.3:0/4022694854 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c004000 con 0x564699eb9790 2023-09-10T01:17:40.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.166+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c7c00bd80 con 0x564699eb9790 2023-09-10T01:17:40.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.166+0000 7f8c66ffd700 1 -- 172.21.15.3:0/4022694854 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f8c7c004160 con 0x564699eb9790 2023-09-10T01:17:40.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.166+0000 7f8c66ffd700 1 -- 172.21.15.3:0/4022694854 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c7c004ff0 con 0x564699eb9790 2023-09-10T01:17:40.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.166+0000 7f8ca1397b80 10 client.28840 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:40.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.166+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x564699c36330 con 0x564699eb9790 2023-09-10T01:17:40.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.166+0000 7f8c66ffd700 1 -- 172.21.15.3:0/4022694854 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c7c049030 con 0x564699eb9790 2023-09-10T01:17:40.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8c66ffd700 1 -- 172.21.15.3:0/4022694854 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f8c7c0377a0 con 0x564699eb9790 2023-09-10T01:17:40.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8ca1397b80 10 client.28840 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:40.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8ca1397b80 10 client.28840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:40.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8ca1397b80 20 client.28840 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:40.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699f29300 0x564699f2b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8ca1397b80 4 client.28840 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:17:40.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564699e049a0 con 0x564699f29300 2023-09-10T01:17:40.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.167+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699f29300 0x564699f2b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.168+0000 7f8c86ffd700 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699f29300 0x564699f2b6f0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=537f9f4a71688748 server_cookie=1fe87fe2b2a68300 in_seq=0 out_seq=0 2023-09-10T01:17:40.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.168+0000 7f8c66ffd700 10 client.28840 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.171+0000 7f8c66ffd700 1 -- 172.21.15.3:0/4022694854 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564699e049a0 con 0x564699f29300 2023-09-10T01:17:40.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.171+0000 7f8c66ffd700 10 client.28840 handle_command_reply: tid=0 2023-09-10T01:17:40.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.171+0000 7f8ca1397b80 1 client.28840 shutdown 2023-09-10T01:17:40.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 msgr2=0x564699f219c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699f219c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c005b10 tx=0x7f8c7c003a00 comp rx=0 tx=0).stop 2023-09-10T01:17:40.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 shutdown_connections 2023-09-10T01:17:40.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x564699f29300 0x564699f2b6f0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564699eb9790 0x564699f219c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564699ec3ec0 0x564699ec9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 --2- 172.21.15.3:0/4022694854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564699eba0b0 0x564699f21f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 >> 172.21.15.3:0/4022694854 conn(0x564699e2e110 msgr2=0x564699ebf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8c66ffd700 0 client.28840 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 shutdown_connections 2023-09-10T01:17:40.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.172+0000 7f8ca1397b80 1 -- 172.21.15.3:0/4022694854 wait complete. 2023-09-10T01:17:40.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.173+0000 7f8ca1397b80 20 client.28840 trim_cache size 0 max 16384 2023-09-10T01:17:40.177 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:17:40.177 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:17:40.178 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "237c374f-6c41-493f-8151-f182b1ebdea0", 2023-09-10T01:17:40.178 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:17:40.178 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:17:40.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c700a0830 msgr2=0x7f8c700a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c700a0830 0x7f8c700a2cf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8c800c94f0 tx=0x7f8c8000f940 comp rx=0 tx=0).stop 2023-09-10T01:17:40.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c881133a0 msgr2=0x7f8c88113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c881133a0 0x7f8c88113850 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000b780 tx=0x7f8c8000ba90 comp rx=0 tx=0).stop 2023-09-10T01:17:40.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 shutdown_connections 2023-09-10T01:17:40.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8c700a0830 0x7f8c700a2cf0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c881133a0 0x7f8c88113850 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8c88107870 0x7f8c8807a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 --2- 172.21.15.3:0/3787114033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c8810c410 0x7f8c8807ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.176+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 >> 172.21.15.3:0/3787114033 conn(0x7f8c88072090 msgr2=0x7f8c8806aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.177+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 shutdown_connections 2023-09-10T01:17:40.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.177+0000 7f8c8d9ff700 1 -- 172.21.15.3:0/3787114033 wait complete. 2023-09-10T01:17:40.182 INFO:tasks.workunit.client.0.smithi003.stdout: 10 45033 31.98 MB/sec warmup 81 sec latency 283.402 ms 2023-09-10T01:17:40.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:17:40.565 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:40 smithi003 ceph-mon[97654]: pgmap v453: 129 pgs: 129 active+clean; 728 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 5.2 MiB/s wr, 587 op/s 2023-09-10T01:17:40.565 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:40 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:40.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.582+0000 7ff3059ff700 1 -- 172.21.15.3:0/2174358376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300108850 msgr2=0x7ff300108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.583 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.582+0000 7ff3059ff700 1 --2- 172.21.15.3:0/2174358376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300108850 0x7ff300108c30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000b0a0 tx=0x7ff2f000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.586+0000 7ff3059ff700 1 -- 172.21.15.3:0/2174358376 shutdown_connections 2023-09-10T01:17:40.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.586+0000 7ff3059ff700 1 --2- 172.21.15.3:0/2174358376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300108850 0x7ff300108c30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.586+0000 7ff3059ff700 1 --2- 172.21.15.3:0/2174358376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff300103200 0x7ff30010f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.586+0000 7ff3059ff700 1 --2- 172.21.15.3:0/2174358376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff300102840 0x7ff300102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.590 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.586+0000 7ff3059ff700 1 -- 172.21.15.3:0/2174358376 >> 172.21.15.3:0/2174358376 conn(0x7ff3000fe330 msgr2=0x7ff300100750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.594+0000 7ff3059ff700 1 -- 172.21.15.3:0/2174358376 shutdown_connections 2023-09-10T01:17:40.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.600+0000 7ff3059ff700 1 -- 172.21.15.3:0/2174358376 wait complete. 2023-09-10T01:17:40.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 Processor -- start 2023-09-10T01:17:40.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 -- start start 2023-09-10T01:17:40.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300102840 0x7ff300074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff300103200 0x7ff300075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff300108850 0x7ff300075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300070a40 con 0x7ff300108850 2023-09-10T01:17:40.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300070bb0 con 0x7ff300103200 2023-09-10T01:17:40.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.602+0000 7ff3059ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300070d70 con 0x7ff300102840 2023-09-10T01:17:40.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.603+0000 7ff2ff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff300108850 0x7ff300075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.603+0000 7ff2ff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff300108850 0x7ff300075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58136/0 (socket says 172.21.15.3:58136) 2023-09-10T01:17:40.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.603+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/1618641243 learned_addr learned my addr 172.21.15.3:0/1618641243 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.609 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.604+0000 7ff2f67fc700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff300103200 0x7ff300075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.605+0000 7ff2feffd700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300102840 0x7ff300074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.605+0000 7ff2feffd700 1 -- 172.21.15.3:0/1618641243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff300103200 msgr2=0x7ff300075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.605+0000 7ff2feffd700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff300103200 0x7ff300075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.605+0000 7ff2feffd700 1 -- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff300108850 msgr2=0x7ff300075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.610 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.605+0000 7ff2feffd700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff300108850 0x7ff300075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.605+0000 7ff2feffd700 1 -- 172.21.15.3:0/1618641243 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0009d00 con 0x7ff300102840 2023-09-10T01:17:40.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.606+0000 7ff2feffd700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300102840 0x7ff300074b10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000b070 tx=0x7ff2f0007b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.606+0000 7ff3049fd700 1 -- 172.21.15.3:0/1618641243 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000e040 con 0x7ff300102840 2023-09-10T01:17:40.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.606+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff300070ff0 con 0x7ff300102840 2023-09-10T01:17:40.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.607+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3000714f0 con 0x7ff300102840 2023-09-10T01:17:40.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.611+0000 7ff3049fd700 1 -- 172.21.15.3:0/1618641243 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2f0003e10 con 0x7ff300102840 2023-09-10T01:17:40.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.611+0000 7ff3049fd700 1 -- 172.21.15.3:0/1618641243 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0005630 con 0x7ff300102840 2023-09-10T01:17:40.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.614+0000 7ff319283b80 1 Processor -- start 2023-09-10T01:17:40.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.616+0000 7ff319283b80 1 -- start start 2023-09-10T01:17:40.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.617+0000 7ff319283b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.617+0000 7ff319283b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.617+0000 7ff319283b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.617+0000 7ff319283b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f64fa730 con 0x55e4f65a2030 2023-09-10T01:17:40.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.617+0000 7ff319283b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f64f8050 con 0x55e4f65abe40 2023-09-10T01:17:40.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.617+0000 7ff319283b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f64f3940 con 0x55e4f65a1710 2023-09-10T01:17:40.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.617+0000 7ff2ff7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2f67fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2ff7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37154/0 (socket says 172.21.15.3:37154) 2023-09-10T01:17:40.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/3883060599 learned_addr learned my addr 172.21.15.3:0/3883060599 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2f67fc700 1 -- 172.21.15.3:0/3883060599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 msgr2=0x55e4f65a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2f67fc700 1 --2- 172.21.15.3:0/3883060599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2f67fc700 1 -- 172.21.15.3:0/3883060599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 msgr2=0x55e4f65ab900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:40.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2f67fc700 1 --2- 172.21.15.3:0/3883060599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65ab900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.618+0000 7ff2f67fc700 1 -- 172.21.15.3:0/3883060599 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4f631fde0 con 0x55e4f65abe40 2023-09-10T01:17:40.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.619+0000 7ff2f67fc700 1 --2- 172.21.15.3:0/3883060599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65ae230 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff300100c30 tx=0x7ff2e8007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=877a4ce5beb60159 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.620+0000 7ff3049fd700 1 -- 172.21.15.3:0/1618641243 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7ff2f0005790 con 0x7ff300102840 2023-09-10T01:17:40.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.623+0000 7ff3049fd700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff2ec0a08f0 0x7ff2ec0a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.623+0000 7ff3049fd700 1 -- 172.21.15.3:0/1618641243 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7ff2f00cd330 con 0x7ff300102840 2023-09-10T01:17:40.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.625+0000 7ff2f5ffb700 1 -- 172.21.15.3:0/3883060599 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8011590 con 0x55e4f65abe40 2023-09-10T01:17:40.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.625+0000 7ff2f5ffb700 1 -- 172.21.15.3:0/3883060599 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2e8011bb0 con 0x55e4f65abe40 2023-09-10T01:17:40.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.625+0000 7ff2f5ffb700 1 -- 172.21.15.3:0/3883060599 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8012c10 con 0x55e4f65abe40 2023-09-10T01:17:40.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.626+0000 7ff319283b80 1 -- 172.21.15.3:0/3883060599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 msgr2=0x55e4f65ae230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.626+0000 7ff319283b80 1 --2- 172.21.15.3:0/3883060599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65ae230 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff300100c30 tx=0x7ff2e8007a00 comp rx=0 tx=0).stop 2023-09-10T01:17:40.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.627+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff2ec0a08f0 0x7ff2ec0a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.627+0000 7ff319283b80 1 -- 172.21.15.3:0/3883060599 shutdown_connections 2023-09-10T01:17:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.627+0000 7ff319283b80 1 --2- 172.21.15.3:0/3883060599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.627+0000 7ff319283b80 1 --2- 172.21.15.3:0/3883060599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65ae230 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.627+0000 7ff319283b80 1 --2- 172.21.15.3:0/3883060599 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.627+0000 7ff319283b80 1 -- 172.21.15.3:0/3883060599 >> 172.21.15.3:0/3883060599 conn(0x55e4f6516090 msgr2=0x55e4f659cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.628+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff2ec0a08f0 0x7ff2ec0a2db0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8007ab0 tx=0x7ff2f8008040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.629+0000 7ff319283b80 1 -- 172.21.15.3:0/3883060599 shutdown_connections 2023-09-10T01:17:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.629+0000 7ff319283b80 1 -- 172.21.15.3:0/3883060599 wait complete. 2023-09-10T01:17:40.633 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.630+0000 7ff319283b80 1 -- start start 2023-09-10T01:17:40.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.630+0000 7ff319283b80 1 Processor -- start 2023-09-10T01:17:40.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.631+0000 7ff319283b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65a1710 0x55e4f65a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.631+0000 7ff319283b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65a90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.631+0000 7ff319283b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65abe40 0x55e4f65a9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.631+0000 7ff319283b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f6491b70 con 0x55e4f65abe40 2023-09-10T01:17:40.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.631+0000 7ff319283b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f6491870 con 0x55e4f65a1710 2023-09-10T01:17:40.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.631+0000 7ff319283b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f6375920 con 0x55e4f65a2030 2023-09-10T01:17:40.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.631+0000 7ff319283b80 10 client.0 _ll_register_callbacks cb 0x55e4f63dba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:40.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.632+0000 7ff2feffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65a1710 0x55e4f65a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.632+0000 7ff2feffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65a1710 0x55e4f65a8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53916/0 (socket says 172.21.15.3:53916) 2023-09-10T01:17:40.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.632+0000 7ff2feffd700 1 -- 172.21.15.3:0/3803483724 learned_addr learned my addr 172.21.15.3:0/3803483724 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.632+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65abe40 0x55e4f65a9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.633+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/3803483724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 msgr2=0x55e4f65a90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.633+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65a90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.633+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/3803483724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65a1710 msgr2=0x55e4f65a8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.633+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65a1710 0x55e4f65a8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.633+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/3803483724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e800e040 con 0x55e4f65abe40 2023-09-10T01:17:40.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.638+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65abe40 0x55e4f65a9620 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8010530 tx=0x7ff2f8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c188146b93cc4094 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.638+0000 7ff2deffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:40.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.640+0000 7ff2deffd700 1 -- 172.21.15.3:0/3803483724 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8014640 con 0x55e4f65abe40 2023-09-10T01:17:40.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.640+0000 7ff2deffd700 1 -- 172.21.15.3:0/3803483724 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7ff2f80147a0 con 0x55e4f65abe40 2023-09-10T01:17:40.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.640+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e4f644b4f0 con 0x55e4f65abe40 2023-09-10T01:17:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.641+0000 7ff2deffd700 1 -- 172.21.15.3:0/3803483724 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2f8005a00 con 0x55e4f65abe40 2023-09-10T01:17:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.641+0000 7ff319283b80 10 client.18855 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.641+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55e4f6308c60 con 0x55e4f65abe40 2023-09-10T01:17:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff2deffd700 1 -- 172.21.15.3:0/3803483724 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e4f644b4f0 con 0x55e4f65abe40 2023-09-10T01:17:40.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff2deffd700 1 -- 172.21.15.3:0/3803483724 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7ff2f80061a0 con 0x55e4f65abe40 2023-09-10T01:17:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff319283b80 10 client.18855 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff319283b80 10 client.18855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff319283b80 20 client.18855 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff319283b80 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f65bbbb0 0x55e4f65bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff319283b80 4 client.18855 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.642+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4f65088a0 con 0x55e4f65bbbb0 2023-09-10T01:17:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.643+0000 7ff2feffd700 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f65bbbb0 0x55e4f65bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.644+0000 7ff2feffd700 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f65bbbb0 0x55e4f65bdfd0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74f41dbe39cad4c7 server_cookie=95bdeb3af8479532 in_seq=0 out_seq=0 2023-09-10T01:17:40.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.644+0000 7ff2deffd700 10 client.18855 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.647+0000 7ff2deffd700 1 -- 172.21.15.3:0/3803483724 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e4f65088a0 con 0x55e4f65bbbb0 2023-09-10T01:17:40.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.647+0000 7ff2deffd700 10 client.18855 handle_command_reply: tid=0 2023-09-10T01:17:40.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.647+0000 7ff319283b80 1 client.18855 shutdown 2023-09-10T01:17:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65abe40 msgr2=0x55e4f65a9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65abe40 0x55e4f65a9620 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8010530 tx=0x7ff2f8004770 comp rx=0 tx=0).stop 2023-09-10T01:17:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 shutdown_connections 2023-09-10T01:17:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f65bbbb0 0x55e4f65bdfd0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65abe40 0x55e4f65a9620 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65a1710 0x55e4f65a8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 --2- 172.21.15.3:0/3803483724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65a90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 >> 172.21.15.3:0/3803483724 conn(0x55e4f6516090 msgr2=0x55e4f659c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff2deffd700 0 client.18855 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 shutdown_connections 2023-09-10T01:17:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.648+0000 7ff319283b80 1 -- 172.21.15.3:0/3803483724 wait complete. 2023-09-10T01:17:40.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.649+0000 7ff319283b80 20 client.18855 trim_cache size 0 max 16384 2023-09-10T01:17:40.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:40 smithi136 ceph-mon[133708]: pgmap v453: 129 pgs: 129 active+clean; 728 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 5.2 MiB/s wr, 587 op/s 2023-09-10T01:17:40.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:40 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:40.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.684+0000 7ff319283b80 1 Processor -- start 2023-09-10T01:17:40.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.684+0000 7ff319283b80 1 -- start start 2023-09-10T01:17:40.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff319283b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff319283b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff319283b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff319283b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f65ae6d0 con 0x55e4f65a2030 2023-09-10T01:17:40.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff319283b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f65a2770 con 0x55e4f65abe40 2023-09-10T01:17:40.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff319283b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f6594400 con 0x55e4f65a1710 2023-09-10T01:17:40.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff2f67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff2f67fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58152/0 (socket says 172.21.15.3:58152) 2023-09-10T01:17:40.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff2f67fc700 1 -- 172.21.15.3:0/1497370650 learned_addr learned my addr 172.21.15.3:0/1497370650 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.685+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2feffd700 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/1497370650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 msgr2=0x55e4f65b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/1497370650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 msgr2=0x55e4f65a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2ff7fe700 1 -- 172.21.15.3:0/1497370650 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4f6346250 con 0x55e4f65abe40 2023-09-10T01:17:40.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2ff7fe700 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65a5e10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8006500 tx=0x7ff2f8007340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64c076689c3b634c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.686+0000 7ff2f6ffd700 1 -- 172.21.15.3:0/1497370650 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8010560 con 0x55e4f65abe40 2023-09-10T01:17:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff2f6ffd700 1 -- 172.21.15.3:0/1497370650 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2f801a070 con 0x55e4f65abe40 2023-09-10T01:17:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff2f6ffd700 1 -- 172.21.15.3:0/1497370650 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8003af0 con 0x55e4f65abe40 2023-09-10T01:17:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff319283b80 1 -- 172.21.15.3:0/1497370650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 msgr2=0x55e4f65a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff319283b80 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65a5e10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8006500 tx=0x7ff2f8007340 comp rx=0 tx=0).stop 2023-09-10T01:17:40.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff319283b80 1 -- 172.21.15.3:0/1497370650 shutdown_connections 2023-09-10T01:17:40.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff319283b80 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a1710 0x55e4f65b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff319283b80 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65a5e10 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff319283b80 1 --2- 172.21.15.3:0/1497370650 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65a2030 0x55e4f65a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.687+0000 7ff319283b80 1 -- 172.21.15.3:0/1497370650 >> 172.21.15.3:0/1497370650 conn(0x55e4f6516090 msgr2=0x55e4f65a3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.693+0000 7ff319283b80 1 -- 172.21.15.3:0/1497370650 shutdown_connections 2023-09-10T01:17:40.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.693+0000 7ff319283b80 1 -- 172.21.15.3:0/1497370650 wait complete. 2023-09-10T01:17:40.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.693+0000 7ff319283b80 1 -- start start 2023-09-10T01:17:40.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.694+0000 7ff319283b80 1 Processor -- start 2023-09-10T01:17:40.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff319283b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65b2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff319283b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff319283b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65b0950 0x55e4f65b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff319283b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f65b14e0 con 0x55e4f65b0950 2023-09-10T01:17:40.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff319283b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f65b1650 con 0x55e4f65abe40 2023-09-10T01:17:40.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff319283b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f660fa10 con 0x55e4f65a2030 2023-09-10T01:17:40.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff319283b80 10 client.0 _ll_register_callbacks cb 0x7ff2f8010ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:40.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.695+0000 7ff2feffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.696+0000 7ff2feffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65b2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:37178/0 (socket says 172.21.15.3:37178) 2023-09-10T01:17:40.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.696+0000 7ff2feffd700 1 -- 172.21.15.3:0/368981434 learned_addr learned my addr 172.21.15.3:0/368981434 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:40.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.696+0000 7ff2feffd700 1 -- 172.21.15.3:0/368981434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 msgr2=0x55e4f65b2a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:40.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.696+0000 7ff2feffd700 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65b2a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.696+0000 7ff2feffd700 1 -- 172.21.15.3:0/368981434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65b0950 msgr2=0x55e4f65b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:40.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.696+0000 7ff2feffd700 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65b0950 0x55e4f65b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.696+0000 7ff2feffd700 1 -- 172.21.15.3:0/368981434 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4f6308c60 con 0x55e4f65a2030 2023-09-10T01:17:40.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.697+0000 7ff2feffd700 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65b2530 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0000f80 tx=0x7ff2f009a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=682f429f4cc25212 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:40.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.697+0000 7ff2deffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:40.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.697+0000 7ff2deffd700 1 -- 172.21.15.3:0/368981434 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f009abe0 con 0x55e4f65a2030 2023-09-10T01:17:40.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.697+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff2e8011bb0 con 0x55e4f65a2030 2023-09-10T01:17:40.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.698+0000 7ff2deffd700 1 -- 172.21.15.3:0/368981434 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7ff2f009a4e0 con 0x55e4f65a2030 2023-09-10T01:17:40.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff2deffd700 1 -- 172.21.15.3:0/368981434 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2f009e310 con 0x55e4f65a2030 2023-09-10T01:17:40.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 10 client.28855 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:40.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55e4f631e2b0 con 0x55e4f65a2030 2023-09-10T01:17:40.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff2deffd700 1 -- 172.21.15.3:0/368981434 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2e8011bb0 con 0x55e4f65a2030 2023-09-10T01:17:40.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff2deffd700 1 -- 172.21.15.3:0/368981434 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7ff2f009a740 con 0x55e4f65a2030 2023-09-10T01:17:40.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 10 client.28855 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:40.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 10 client.28855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:40.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 20 client.28855 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:40.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f6610fb0 0x55e4f66133d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:40.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 4 client.28855 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:17:40.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.699+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e4f64ec920 con 0x55e4f6610fb0 2023-09-10T01:17:40.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.700+0000 7ff2f67fc700 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f6610fb0 0x55e4f66133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:40.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.701+0000 7ff2f67fc700 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f6610fb0 0x55e4f66133d0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a42a8221b4e377c4 server_cookie=92dd362126a6eaba in_seq=0 out_seq=0 2023-09-10T01:17:40.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.701+0000 7ff2deffd700 10 client.28855 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.704+0000 7ff2deffd700 1 -- 172.21.15.3:0/368981434 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e4f64ec920 con 0x55e4f6610fb0 2023-09-10T01:17:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.704+0000 7ff2deffd700 10 client.28855 handle_command_reply: tid=0 2023-09-10T01:17:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.704+0000 7ff319283b80 1 client.28855 shutdown 2023-09-10T01:17:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 msgr2=0x55e4f65b2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65b2530 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0000f80 tx=0x7ff2f009a150 comp rx=0 tx=0).stop 2023-09-10T01:17:40.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 shutdown_connections 2023-09-10T01:17:40.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55e4f6610fb0 0x55e4f66133d0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e4f65a2030 0x55e4f65b2530 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e4f65b0950 0x55e4f65b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff2deffd700 0 client.28855 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:40.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 --2- 172.21.15.3:0/368981434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f65abe40 0x55e4f65b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 >> 172.21.15.3:0/368981434 conn(0x55e4f6516090 msgr2=0x55e4f65a0700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 shutdown_connections 2023-09-10T01:17:40.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 1 -- 172.21.15.3:0/368981434 wait complete. 2023-09-10T01:17:40.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.705+0000 7ff319283b80 20 client.28855 trim_cache size 0 max 16384 2023-09-10T01:17:40.712 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:17:40.712 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:17:40.712 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:17:40.713 INFO:teuthology.orchestra.run.smithi003.stdout: "237c374f-6c41-493f-8151-f182b1ebdea0": { 2023-09-10T01:17:40.713 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:17:40.713 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "237c374f-6c41-493f-8151-f182b1ebdea0", 2023-09-10T01:17:40.713 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:17:40.713 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:17:40.714 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:17:40.714 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:17:40.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.706+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff2ec0a08f0 msgr2=0x7ff2ec0a2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.706+0000 7ff3059ff700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff2ec0a08f0 0x7ff2ec0a2db0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8007ab0 tx=0x7ff2f8008040 comp rx=0 tx=0).stop 2023-09-10T01:17:40.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.706+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300102840 msgr2=0x7ff300074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:40.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.706+0000 7ff3059ff700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300102840 0x7ff300074b10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000b070 tx=0x7ff2f0007b60 comp rx=0 tx=0).stop 2023-09-10T01:17:40.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.707+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 shutdown_connections 2023-09-10T01:17:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.707+0000 7ff3059ff700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7ff2ec0a08f0 0x7ff2ec0a2db0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.707+0000 7ff3059ff700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff300102840 0x7ff300074b10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.707+0000 7ff3059ff700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff300108850 0x7ff300075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.707+0000 7ff3059ff700 1 --2- 172.21.15.3:0/1618641243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff300103200 0x7ff300075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:40.716 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.707+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 >> 172.21.15.3:0/1618641243 conn(0x7ff3000fe330 msgr2=0x7ff3000fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:40.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.709+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 shutdown_connections 2023-09-10T01:17:40.717 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:17:40.712+0000 7ff3059ff700 1 -- 172.21.15.3:0/1618641243 wait complete. 2023-09-10T01:17:40.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:40 smithi092 ceph-mon[134067]: pgmap v453: 129 pgs: 129 active+clean; 728 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 5.2 MiB/s wr, 587 op/s 2023-09-10T01:17:40.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:40 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:40.753 INFO:tasks.cephfs.filesystem:scrub status for tag:237c374f-6c41-493f-8151-f182b1ebdea0 - {'path': '/', 'tag': '237c374f-6c41-493f-8151-f182b1ebdea0', 'options': 'recursive,force'} 2023-09-10T01:17:41.181 INFO:tasks.workunit.client.0.smithi003.stdout: 10 45479 32.00 MB/sec warmup 82 sec latency 155.464 ms 2023-09-10T01:17:41.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:17:41.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:41.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:41.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:17:41.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:41 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:17:41.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:17:41.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:41.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:41.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:17:41.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.721 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.721 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.721 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:41 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:17:41.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:17:41.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:41.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:41.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:17:41.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:17:41.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:41 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:17:42.185 INFO:tasks.workunit.client.0.smithi003.stdout: 10 45531 31.64 MB/sec warmup 83 sec latency 898.101 ms 2023-09-10T01:17:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:42 smithi136 ceph-mon[133708]: pgmap v454: 129 pgs: 129 active+clean; 749 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.6 MiB/s wr, 701 op/s 2023-09-10T01:17:42.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:42 smithi092 ceph-mon[134067]: pgmap v454: 129 pgs: 129 active+clean; 749 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.6 MiB/s wr, 701 op/s 2023-09-10T01:17:42.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:42 smithi003 ceph-mon[97654]: pgmap v454: 129 pgs: 129 active+clean; 749 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.6 MiB/s wr, 701 op/s 2023-09-10T01:17:43.183 INFO:tasks.workunit.client.0.smithi003.stdout: 10 45679 31.36 MB/sec warmup 84 sec latency 1778.143 ms 2023-09-10T01:17:43.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:43 smithi092 ceph-mon[134067]: pgmap v455: 129 pgs: 129 active+clean; 749 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2023-09-10T01:17:43.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:43 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:17:43.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:43 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:17:43.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:43 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:17:43.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:43 smithi003 ceph-mon[97654]: pgmap v455: 129 pgs: 129 active+clean; 749 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2023-09-10T01:17:43.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:43 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:17:43.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:43 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:17:43.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:43 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:17:43.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:43 smithi136 ceph-mon[133708]: pgmap v455: 129 pgs: 129 active+clean; 749 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2023-09-10T01:17:43.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:43 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:17:43.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:43 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:17:43.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:43 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:17:44.188 INFO:tasks.workunit.client.0.smithi003.stdout: 10 46060 31.20 MB/sec warmup 85 sec latency 898.722 ms 2023-09-10T01:17:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.622+0000 7f9aea1ff700 1 -- 172.21.15.3:0/3599582622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ae410b970 msgr2=0x7f9ae4117e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.622+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/3599582622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ae410b970 0x7f9ae4117e10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009f90 tx=0x7f9adc00b330 comp rx=0 tx=0).stop 2023-09-10T01:17:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.625+0000 7f9aea1ff700 1 -- 172.21.15.3:0/3599582622 shutdown_connections 2023-09-10T01:17:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.625+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/3599582622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ae410b970 0x7f9ae4117e10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.625+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/3599582622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae4111070 0x7f9ae4111450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.625+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/3599582622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae410afb0 0x7f9ae410b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.625+0000 7f9aea1ff700 1 -- 172.21.15.3:0/3599582622 >> 172.21.15.3:0/3599582622 conn(0x7f9ae406b590 msgr2=0x7f9ae4109610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.631+0000 7f9ae2ffd700 1 --2- 172.21.15.3:0/3599582622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae410afb0 0x7f9ae410b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:44.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.639+0000 7f9aea1ff700 1 -- 172.21.15.3:0/3599582622 shutdown_connections 2023-09-10T01:17:44.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.646+0000 7f9aea1ff700 1 -- 172.21.15.3:0/3599582622 wait complete. 2023-09-10T01:17:44.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.647+0000 7f9aea1ff700 1 Processor -- start 2023-09-10T01:17:44.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.648+0000 7f9aea1ff700 1 -- start start 2023-09-10T01:17:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.648+0000 7f9aea1ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae410b970 0x7f9ae41b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.648+0000 7f9aea1ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ae4111070 0x7f9ae41b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.648+0000 7f9aea1ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae41b26b0 0x7f9ae41b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.648+0000 7f9aea1ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41b30a0 con 0x7f9ae410b970 2023-09-10T01:17:44.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.648+0000 7f9aea1ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41b3210 con 0x7f9ae41b26b0 2023-09-10T01:17:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.648+0000 7f9aea1ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41b33d0 con 0x7f9ae4111070 2023-09-10T01:17:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae41b26b0 0x7f9ae41b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae41b26b0 0x7f9ae41b2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:53956/0 (socket says 172.21.15.3:53956) 2023-09-10T01:17:44.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 -- 172.21.15.3:0/1794258634 learned_addr learned my addr 172.21.15.3:0/1794258634 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 -- 172.21.15.3:0/1794258634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ae4111070 msgr2=0x7f9ae41b3e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:17:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ae4111070 0x7f9ae41b3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 -- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae410b970 msgr2=0x7f9ae41b3930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:17:44.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae410b970 0x7f9ae41b3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.653 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.649+0000 7f9ae3fff700 1 -- 172.21.15.3:0/1794258634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40095f0 con 0x7f9ae41b26b0 2023-09-10T01:17:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.651+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae41b26b0 0x7f9ae41b2b60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9adc01e040 tx=0x7f9adc004da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.652+0000 7f9ae0ff9700 1 -- 172.21.15.3:0/1794258634 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc014e80 con 0x7f9ae41b26b0 2023-09-10T01:17:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.652+0000 7f9ae0ff9700 1 -- 172.21.15.3:0/1794258634 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9adc007870 con 0x7f9ae41b26b0 2023-09-10T01:17:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.652+0000 7f9ae0ff9700 1 -- 172.21.15.3:0/1794258634 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc014530 con 0x7f9ae41b26b0 2023-09-10T01:17:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.652+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc009c40 con 0x7f9ae41b26b0 2023-09-10T01:17:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.652+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae41baba0 con 0x7f9ae41b26b0 2023-09-10T01:17:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.659+0000 7f9afdaccb80 1 Processor -- start 2023-09-10T01:17:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.663+0000 7f9afdaccb80 1 -- start start 2023-09-10T01:17:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.664+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.664+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 0x55dd1df56780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.664+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56cc0 0x55dd1df62f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.664+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1deaf6d0 con 0x55dd1df5c1a0 2023-09-10T01:17:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.664+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1deacff0 con 0x55dd1df56300 2023-09-10T01:17:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.664+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1dea88e0 con 0x55dd1df56cc0 2023-09-10T01:17:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.666+0000 7f9ae2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56cc0 0x55dd1df62f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.666+0000 7f9ae2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56cc0 0x55dd1df62f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51618/0 (socket says 172.21.15.3:51618) 2023-09-10T01:17:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.666+0000 7f9ae2ffd700 1 -- 172.21.15.3:0/4009283170 learned_addr learned my addr 172.21.15.3:0/4009283170 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.666+0000 7f9ae2ffd700 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.668+0000 7f9ae2ffd700 1 -- 172.21.15.3:0/4009283170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56cc0 msgr2=0x55dd1df62f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.668+0000 7f9ae2ffd700 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56cc0 0x55dd1df62f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.668+0000 7f9ae2ffd700 1 -- 172.21.15.3:0/4009283170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 msgr2=0x55dd1df56780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.668+0000 7f9ae2ffd700 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 0x55dd1df56780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.668+0000 7f9ae2ffd700 1 -- 172.21.15.3:0/4009283170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd1dcd4d80 con 0x55dd1df5c1a0 2023-09-10T01:17:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.669+0000 7f9ae2ffd700 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5c580 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00ef90 tx=0x7f9acc005ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae58eafee5f72abd server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.670+0000 7f9ad9ffb700 1 -- 172.21.15.3:0/4009283170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc011460 con 0x55dd1df5c1a0 2023-09-10T01:17:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.670+0000 7f9ad9ffb700 1 -- 172.21.15.3:0/4009283170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9acc017070 con 0x55dd1df5c1a0 2023-09-10T01:17:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.670+0000 7f9ad9ffb700 1 -- 172.21.15.3:0/4009283170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc010740 con 0x55dd1df5c1a0 2023-09-10T01:17:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.671+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4009283170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 msgr2=0x55dd1df5c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.671+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5c580 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00ef90 tx=0x7f9acc005ba0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.680+0000 7f9ae0ff9700 1 -- 172.21.15.3:0/1794258634 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9adc003a00 con 0x7f9ae41b26b0 2023-09-10T01:17:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.683+0000 7f9ae0ff9700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ad00a0860 0x7f9ad00a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.684+0000 7f9ae0ff9700 1 -- 172.21.15.3:0/1794258634 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f9adc0cd7a0 con 0x7f9ae41b26b0 2023-09-10T01:17:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.687+0000 7f9ae2ffd700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ad00a0860 0x7f9ad00a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.690+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4009283170 shutdown_connections 2023-09-10T01:17:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.690+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56cc0 0x55dd1df62f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.690+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5c580 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.690+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4009283170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 0x55dd1df56780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.690+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4009283170 >> 172.21.15.3:0/4009283170 conn(0x55dd1decb030 msgr2=0x55dd1df51560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.692+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4009283170 shutdown_connections 2023-09-10T01:17:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.692+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4009283170 wait complete. 2023-09-10T01:17:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.693+0000 7f9afdaccb80 1 -- start start 2023-09-10T01:17:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.693+0000 7f9afdaccb80 1 Processor -- start 2023-09-10T01:17:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.693+0000 7f9ae2ffd700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ad00a0860 0x7f9ad00a2d20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00ef90 tx=0x7f9acc005f30 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.694+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.694+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56cc0 0x55dd1df60070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.694+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.694+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1de46b10 con 0x55dd1df5c1a0 2023-09-10T01:17:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.694+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1de46810 con 0x55dd1df56cc0 2023-09-10T01:17:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.694+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1dd2a8c0 con 0x55dd1df56300 2023-09-10T01:17:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.694+0000 7f9afdaccb80 10 client.0 _ll_register_callbacks cb 0x55dd1dd90a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.695+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.695+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57154/0 (socket says 172.21.15.3:57154) 2023-09-10T01:17:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.695+0000 7f9ae3fff700 1 -- 172.21.15.3:0/4091691310 learned_addr learned my addr 172.21.15.3:0/4091691310 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.696+0000 7f9ae3fff700 1 -- 172.21.15.3:0/4091691310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 msgr2=0x55dd1df62df0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.696+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.696+0000 7f9ae3fff700 1 -- 172.21.15.3:0/4091691310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56cc0 msgr2=0x55dd1df60070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.696+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56cc0 0x55dd1df60070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.696+0000 7f9ae3fff700 1 -- 172.21.15.3:0/4091691310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc013400 con 0x55dd1df5c1a0 2023-09-10T01:17:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.696+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df606a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9adc096ed0 tx=0x7f9adc09b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c39a214a6a70bc server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.697+0000 7f9ac2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.697+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/4091691310 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc09e800 con 0x55dd1df5c1a0 2023-09-10T01:17:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.697+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/4091691310 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f9adc0a29b0 con 0x55dd1df5c1a0 2023-09-10T01:17:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.698+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd1de00490 con 0x55dd1df5c1a0 2023-09-10T01:17:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.698+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/4091691310 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9adc003cb0 con 0x55dd1df5c1a0 2023-09-10T01:17:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.698+0000 7f9afdaccb80 10 client.18867 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.698+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55dd1dcbdc00 con 0x55dd1df5c1a0 2023-09-10T01:17:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/4091691310 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dd1de00490 con 0x55dd1df5c1a0 2023-09-10T01:17:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/4091691310 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9adc0cd360 con 0x55dd1df5c1a0 2023-09-10T01:17:44.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9afdaccb80 10 client.18867 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:44.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9afdaccb80 10 client.18867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:44.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9afdaccb80 20 client.18867 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:44.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1df70270 0x55dd1df72690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9afdaccb80 4 client.18867 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:44.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd1debd840 con 0x55dd1df70270 2023-09-10T01:17:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.699+0000 7f9ae37fe700 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1df70270 0x55dd1df72690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.706+0000 7f9ae37fe700 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1df70270 0x55dd1df72690 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74233095140a0971 server_cookie=1344a608b2cead5f in_seq=0 out_seq=0 2023-09-10T01:17:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.708+0000 7f9ac2ffd700 10 client.18867 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.708+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/4091691310 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dd1debd840 con 0x55dd1df70270 2023-09-10T01:17:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.708+0000 7f9ac2ffd700 10 client.18867 handle_command_reply: tid=0 2023-09-10T01:17:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.709+0000 7f9afdaccb80 1 client.18867 shutdown 2023-09-10T01:17:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.719+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 msgr2=0x55dd1df606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:44.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.719+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df606a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9adc096ed0 tx=0x7f9adc09b0d0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.720+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 shutdown_connections 2023-09-10T01:17:44.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.720+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1df70270 0x55dd1df72690 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.720+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.720+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df5c1a0 0x55dd1df606a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.720+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/4091691310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56cc0 0x55dd1df60070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.720+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 >> 172.21.15.3:0/4091691310 conn(0x55dd1decb030 msgr2=0x55dd1df507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:44.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.720+0000 7f9ac2ffd700 0 client.18867 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.731+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 shutdown_connections 2023-09-10T01:17:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.732+0000 7f9afdaccb80 1 -- 172.21.15.3:0/4091691310 wait complete. 2023-09-10T01:17:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.744+0000 7f9afdaccb80 20 client.18867 trim_cache size 0 max 16384 2023-09-10T01:17:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.774+0000 7f9afdaccb80 1 Processor -- start 2023-09-10T01:17:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.775+0000 7f9afdaccb80 1 -- start start 2023-09-10T01:17:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.775+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.775+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df56cc0 0x55dd1df58b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.775+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.775+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1debd840 con 0x55dd1df56cc0 2023-09-10T01:17:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.775+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1de00490 con 0x55dd1df5c1a0 2023-09-10T01:17:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.775+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1df570a0 con 0x55dd1df56300 2023-09-10T01:17:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.777+0000 7f9ae37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.777+0000 7f9ae37fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51634/0 (socket says 172.21.15.3:51634) 2023-09-10T01:17:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.777+0000 7f9ae37fe700 1 -- 172.21.15.3:0/2895563058 learned_addr learned my addr 172.21.15.3:0/2895563058 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.778+0000 7f9ae37fe700 1 -- 172.21.15.3:0/2895563058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df5c1a0 msgr2=0x55dd1df5b0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:17:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.778+0000 7f9ae37fe700 1 --2- 172.21.15.3:0/2895563058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.778+0000 7f9ae37fe700 1 -- 172.21.15.3:0/2895563058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df56cc0 msgr2=0x55dd1df58b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.778+0000 7f9ae37fe700 1 --2- 172.21.15.3:0/2895563058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df56cc0 0x55dd1df58b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.778+0000 7f9ae37fe700 1 -- 172.21.15.3:0/2895563058 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd1dcfb1f0 con 0x55dd1df56300 2023-09-10T01:17:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.779+0000 7f9ae37fe700 1 --2- 172.21.15.3:0/2895563058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55dd1df72fb0 tx=0x7f9ad4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22fc7e7c51b6ab07 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.779+0000 7f9ada7fc700 1 -- 172.21.15.3:0/2895563058 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4015070 con 0x55dd1df56300 2023-09-10T01:17:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.779+0000 7f9ada7fc700 1 -- 172.21.15.3:0/2895563058 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ad4004510 con 0x55dd1df56300 2023-09-10T01:17:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.779+0000 7f9ada7fc700 1 -- 172.21.15.3:0/2895563058 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4003d20 con 0x55dd1df56300 2023-09-10T01:17:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.781+0000 7f9afdaccb80 1 -- 172.21.15.3:0/2895563058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 msgr2=0x55dd1df62990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.781+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/2895563058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55dd1df72fb0 tx=0x7f9ad4009310 comp rx=0 tx=0).stop 2023-09-10T01:17:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.781+0000 7f9afdaccb80 1 -- 172.21.15.3:0/2895563058 shutdown_connections 2023-09-10T01:17:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.781+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/2895563058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df56300 0x55dd1df62990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.781+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/2895563058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df5c1a0 0x55dd1df5b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.781+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/2895563058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df56cc0 0x55dd1df58b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.781+0000 7f9afdaccb80 1 -- 172.21.15.3:0/2895563058 >> 172.21.15.3:0/2895563058 conn(0x55dd1decb030 msgr2=0x55dd1df5a120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.783+0000 7f9afdaccb80 1 -- 172.21.15.3:0/2895563058 shutdown_connections 2023-09-10T01:17:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.789+0000 7f9afdaccb80 1 -- 172.21.15.3:0/2895563058 wait complete. 2023-09-10T01:17:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.790+0000 7f9afdaccb80 1 -- start start 2023-09-10T01:17:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.793+0000 7f9afdaccb80 1 Processor -- start 2023-09-10T01:17:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.795+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 0x55dd1df676e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.795+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df5c1a0 0x55dd1df65b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.795+0000 7f9afdaccb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df661b0 0x55dd1dfc40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.795+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1df66800 con 0x55dd1df661b0 2023-09-10T01:17:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.795+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1df66970 con 0x55dd1df56300 2023-09-10T01:17:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.795+0000 7f9afdaccb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd1dfc4630 con 0x55dd1df5c1a0 2023-09-10T01:17:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.795+0000 7f9afdaccb80 10 client.0 _ll_register_callbacks cb 0x7f9adc09a5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.796+0000 7f9ae2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df5c1a0 0x55dd1df65b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.796+0000 7f9ae2ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df5c1a0 0x55dd1df65b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:51646/0 (socket says 172.21.15.3:51646) 2023-09-10T01:17:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.796+0000 7f9ae2ffd700 1 -- 172.21.15.3:0/1372493136 learned_addr learned my addr 172.21.15.3:0/1372493136 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.797+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df661b0 0x55dd1dfc40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.799+0000 7f9ae3fff700 1 -- 172.21.15.3:0/1372493136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df5c1a0 msgr2=0x55dd1df65b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.799+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df5c1a0 0x55dd1df65b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.799+0000 7f9ae3fff700 1 -- 172.21.15.3:0/1372493136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 msgr2=0x55dd1df676e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.799+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 0x55dd1df676e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.799+0000 7f9ae3fff700 1 -- 172.21.15.3:0/1372493136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad4009c60 con 0x55dd1df661b0 2023-09-10T01:17:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.800+0000 7f9ae3fff700 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df661b0 0x55dd1dfc40f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0cd690 tx=0x7f9adc003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f916cbbeaaefa8f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.801+0000 7f9ac2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:17:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.801+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/1372493136 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc09aef0 con 0x55dd1df661b0 2023-09-10T01:17:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.801+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/1372493136 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f9adc0a29b0 con 0x55dd1df661b0 2023-09-10T01:17:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.801+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9acc017070 con 0x55dd1df661b0 2023-09-10T01:17:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.802+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/1372493136 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9adc09e800 con 0x55dd1df661b0 2023-09-10T01:17:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.804+0000 7f9afdaccb80 10 client.18870 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.804+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55dd1dcd3250 con 0x55dd1df661b0 2023-09-10T01:17:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/1372493136 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9acc017070 con 0x55dd1df661b0 2023-09-10T01:17:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/1372493136 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9adc0a2520 con 0x55dd1df661b0 2023-09-10T01:17:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9afdaccb80 10 client.18870 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9afdaccb80 10 client.18870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9afdaccb80 20 client.18870 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1dfc58c0 0x55dd1dfc7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9afdaccb80 4 client.18870 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:17:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.805+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55dd1dea18c0 con 0x55dd1dfc58c0 2023-09-10T01:17:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.806+0000 7f9ae37fe700 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1dfc58c0 0x55dd1dfc7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.808+0000 7f9ae37fe700 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1dfc58c0 0x55dd1dfc7cb0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf3450778a4f7c02 server_cookie=53bcec4f618a59f9 in_seq=0 out_seq=0 2023-09-10T01:17:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:44.808+0000 7f9ac2ffd700 10 client.18870 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:45.187 INFO:tasks.workunit.client.0.smithi003.stdout: 10 46348 31.03 MB/sec warmup 86 sec latency 318.230 ms 2023-09-10T01:17:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.441+0000 7f9ac2ffd700 1 -- 172.21.15.3:0/1372493136 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55dd1dea18c0 con 0x55dd1dfc58c0 2023-09-10T01:17:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.441+0000 7f9ac2ffd700 10 client.18870 handle_command_reply: tid=0 2023-09-10T01:17:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.441+0000 7f9afdaccb80 1 client.18870 shutdown 2023-09-10T01:17:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.442+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df661b0 msgr2=0x55dd1dfc40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.442+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df661b0 0x55dd1dfc40f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0cd690 tx=0x7f9adc003f50 comp rx=0 tx=0).stop 2023-09-10T01:17:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 shutdown_connections 2023-09-10T01:17:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x55dd1dfc58c0 0x55dd1dfc7cb0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd1df661b0 0x55dd1dfc40f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.444 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd1df56300 0x55dd1df676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 --2- 172.21.15.3:0/1372493136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd1df5c1a0 0x55dd1df65b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 >> 172.21.15.3:0/1372493136 conn(0x55dd1decb030 msgr2=0x55dd1df5b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9ac2ffd700 0 client.18870 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 shutdown_connections 2023-09-10T01:17:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.443+0000 7f9afdaccb80 1 -- 172.21.15.3:0/1372493136 wait complete. 2023-09-10T01:17:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.444+0000 7f9afdaccb80 20 client.18870 trim_cache size 0 max 16384 2023-09-10T01:17:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:17:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:17:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:17:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:17:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ad00a0860 msgr2=0x7f9ad00a2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ad00a0860 0x7f9ad00a2d20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00ef90 tx=0x7f9acc005f30 comp rx=0 tx=0).stop 2023-09-10T01:17:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae41b26b0 msgr2=0x7f9ae41b2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae41b26b0 0x7f9ae41b2b60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9adc01e040 tx=0x7f9adc004da0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 shutdown_connections 2023-09-10T01:17:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9ad00a0860 0x7f9ad00a2d20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ae41b26b0 0x7f9ae41b2b60 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae410b970 0x7f9ae41b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 --2- 172.21.15.3:0/1794258634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ae4111070 0x7f9ae41b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.445+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 >> 172.21.15.3:0/1794258634 conn(0x7f9ae406b590 msgr2=0x7f9ae41098c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.446+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 shutdown_connections 2023-09-10T01:17:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:45.446+0000 7f9aea1ff700 1 -- 172.21.15.3:0/1794258634 wait complete. 2023-09-10T01:17:46.185 INFO:tasks.workunit.client.0.smithi003.stdout: 10 46556 30.73 MB/sec warmup 87 sec latency 481.316 ms 2023-09-10T01:17:46.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:46 smithi003 ceph-mon[97654]: pgmap v456: 129 pgs: 129 active+clean; 862 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 15 MiB/s wr, 867 op/s 2023-09-10T01:17:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:46 smithi136 ceph-mon[133708]: pgmap v456: 129 pgs: 129 active+clean; 862 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 15 MiB/s wr, 867 op/s 2023-09-10T01:17:46.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:46 smithi092 ceph-mon[134067]: pgmap v456: 129 pgs: 129 active+clean; 862 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 15 MiB/s wr, 867 op/s 2023-09-10T01:17:47.187 INFO:tasks.workunit.client.0.smithi003.stdout: 10 46783 30.53 MB/sec warmup 88 sec latency 471.209 ms 2023-09-10T01:17:48.187 INFO:tasks.workunit.client.0.smithi003.stdout: 10 46973 30.34 MB/sec warmup 89 sec latency 847.982 ms 2023-09-10T01:17:48.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:48 smithi003 ceph-mon[97654]: pgmap v457: 129 pgs: 129 active+clean; 862 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 511 op/s 2023-09-10T01:17:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:48 smithi136 ceph-mon[133708]: pgmap v457: 129 pgs: 129 active+clean; 862 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 511 op/s 2023-09-10T01:17:48.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:48 smithi092 ceph-mon[134067]: pgmap v457: 129 pgs: 129 active+clean; 862 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 511 op/s 2023-09-10T01:17:49.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 47291 30.21 MB/sec warmup 90 sec latency 170.172 ms 2023-09-10T01:17:49.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:49 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:17:49.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:49 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:17:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:49 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:17:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:49 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:17:49.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:49 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:17:49.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:49 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:17:50.188 INFO:tasks.workunit.client.0.smithi003.stdout: 10 47977 30.31 MB/sec warmup 91 sec latency 139.737 ms 2023-09-10T01:17:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:50 smithi136 ceph-mon[133708]: pgmap v458: 129 pgs: 129 active+clean; 851 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 15 MiB/s wr, 682 op/s 2023-09-10T01:17:50.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:50 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:17:50.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:50 smithi092 ceph-mon[134067]: pgmap v458: 129 pgs: 129 active+clean; 851 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 15 MiB/s wr, 682 op/s 2023-09-10T01:17:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:50 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:17:50.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:50 smithi003 ceph-mon[97654]: pgmap v458: 129 pgs: 129 active+clean; 851 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 15 MiB/s wr, 682 op/s 2023-09-10T01:17:50.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:50 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:17:51.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 48745 30.43 MB/sec warmup 92 sec latency 40.972 ms 2023-09-10T01:17:52.190 INFO:tasks.workunit.client.0.smithi003.stdout: 10 49363 30.53 MB/sec warmup 93 sec latency 85.774 ms 2023-09-10T01:17:52.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:52 smithi136 ceph-mon[133708]: pgmap v459: 129 pgs: 129 active+clean; 799 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2023-09-10T01:17:52.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:52 smithi092 ceph-mon[134067]: pgmap v459: 129 pgs: 129 active+clean; 799 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2023-09-10T01:17:52.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:52 smithi003 ceph-mon[97654]: pgmap v459: 129 pgs: 129 active+clean; 799 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2023-09-10T01:17:53.190 INFO:tasks.workunit.client.0.smithi003.stdout: 10 50116 30.56 MB/sec warmup 94 sec latency 44.368 ms 2023-09-10T01:17:54.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 50877 30.67 MB/sec warmup 95 sec latency 68.147 ms 2023-09-10T01:17:54.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:54 smithi136 ceph-mon[133708]: pgmap v460: 129 pgs: 129 active+clean; 799 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 18 MiB/s wr, 873 op/s 2023-09-10T01:17:54.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:54 smithi092 ceph-mon[134067]: pgmap v460: 129 pgs: 129 active+clean; 799 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 18 MiB/s wr, 873 op/s 2023-09-10T01:17:54.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:54 smithi003 ceph-mon[97654]: pgmap v460: 129 pgs: 129 active+clean; 799 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 18 MiB/s wr, 873 op/s 2023-09-10T01:17:55.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 51581 30.71 MB/sec warmup 96 sec latency 74.694 ms 2023-09-10T01:17:56.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 52298 30.86 MB/sec warmup 97 sec latency 71.127 ms 2023-09-10T01:17:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:56 smithi136 ceph-mon[133708]: pgmap v461: 129 pgs: 129 active+clean; 767 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 20 MiB/s wr, 1.13k op/s 2023-09-10T01:17:56.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:56 smithi092 ceph-mon[134067]: pgmap v461: 129 pgs: 129 active+clean; 767 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 20 MiB/s wr, 1.13k op/s 2023-09-10T01:17:56.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:56 smithi003 ceph-mon[97654]: pgmap v461: 129 pgs: 129 active+clean; 767 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 20 MiB/s wr, 1.13k op/s 2023-09-10T01:17:57.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 53200 31.00 MB/sec warmup 98 sec latency 92.760 ms 2023-09-10T01:17:58.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 53876 31.12 MB/sec warmup 99 sec latency 61.648 ms 2023-09-10T01:17:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:17:58 smithi136 ceph-mon[133708]: pgmap v462: 129 pgs: 129 active+clean; 767 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.2 MiB/s wr, 804 op/s 2023-09-10T01:17:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.696+0000 7f3d1dfff700 1 -- 172.21.15.3:0/2412182436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d1810cde0 msgr2=0x7f3d1810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.696+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/2412182436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d1810cde0 0x7f3d1810f250 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000b0a0 tx=0x7f3d1000b3b0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.696+0000 7f3d1dfff700 1 -- 172.21.15.3:0/2412182436 shutdown_connections 2023-09-10T01:17:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.696+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/2412182436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d1810cde0 0x7f3d1810f250 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.696+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/2412182436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d18072a30 0x7f3d18072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.696+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/2412182436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d18073350 0x7f3d1810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.696+0000 7f3d1dfff700 1 -- 172.21.15.3:0/2412182436 >> 172.21.15.3:0/2412182436 conn(0x7f3d180fc0b0 msgr2=0x7f3d180fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.698+0000 7f3d1dfff700 1 -- 172.21.15.3:0/2412182436 shutdown_connections 2023-09-10T01:17:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.698+0000 7f3d1dfff700 1 -- 172.21.15.3:0/2412182436 wait complete. 2023-09-10T01:17:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.699+0000 7f3d1dfff700 1 Processor -- start 2023-09-10T01:17:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.699+0000 7f3d1dfff700 1 -- start start 2023-09-10T01:17:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d1dfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d18072a30 0x7f3d18106ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d1dfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d18073350 0x7f3d1810c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d1dfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d1810cde0 0x7f3d18107210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d1dfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18101dc0 con 0x7f3d18072a30 2023-09-10T01:17:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d1dfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18101f30 con 0x7f3d18073350 2023-09-10T01:17:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d1dfff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d181020f0 con 0x7f3d1810cde0 2023-09-10T01:17:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d17fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d1810cde0 0x7f3d18107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d17fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d1810cde0 0x7f3d18107210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:34680/0 (socket says 172.21.15.3:34680) 2023-09-10T01:17:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.700+0000 7f3d17fff700 1 -- 172.21.15.3:0/1031144675 learned_addr learned my addr 172.21.15.3:0/1031144675 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d0effd700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d18073350 0x7f3d1810c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d17fff700 1 -- 172.21.15.3:0/1031144675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d18073350 msgr2=0x7f3d1810c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d17fff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d18073350 0x7f3d1810c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d17fff700 1 -- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d18072a30 msgr2=0x7f3d18106ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-10T01:17:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d17fff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d18072a30 0x7f3d18106ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d17fff700 1 -- 172.21.15.3:0/1031144675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d10009d00 con 0x7f3d1810cde0 2023-09-10T01:17:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d0effd700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d18073350 0x7f3d1810c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.701+0000 7f3d17fff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d1810cde0 0x7f3d18107210 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009450 tx=0x7f3d10009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.703+0000 7f3d157fa700 1 -- 172.21.15.3:0/1031144675 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000e050 con 0x7f3d1810cde0 2023-09-10T01:17:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.703+0000 7f3d157fa700 1 -- 172.21.15.3:0/1031144675 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d10005570 con 0x7f3d1810cde0 2023-09-10T01:17:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.703+0000 7f3d157fa700 1 -- 172.21.15.3:0/1031144675 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1001ba90 con 0x7f3d1810cde0 2023-09-10T01:17:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.705+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d18102370 con 0x7f3d1810cde0 2023-09-10T01:17:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.705+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d18102800 con 0x7f3d1810cde0 2023-09-10T01:17:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.711+0000 7f3d157fa700 1 -- 172.21.15.3:0/1031144675 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3d100056e0 con 0x7f3d1810cde0 2023-09-10T01:17:58.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.712+0000 7f3d31908b80 1 Processor -- start 2023-09-10T01:17:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d31908b80 1 -- start start 2023-09-10T01:17:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d157fa700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d040a0830 0x7f3d040a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc260d1a0 0x562bc260d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc2607300 0x562bc2607780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2613f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d31908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc25606d0 con 0x562bc2607cc0 2023-09-10T01:17:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d31908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc255dff0 con 0x562bc2607300 2023-09-10T01:17:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.713+0000 7f3d31908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc25598e0 con 0x562bc260d1a0 2023-09-10T01:17:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d157fa700 1 -- 172.21.15.3:0/1031144675 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f3d100cd7a0 con 0x7f3d1810cde0 2023-09-10T01:17:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d177fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc260d1a0 0x562bc260d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc2607300 0x562bc2607780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc2607300 0x562bc2607780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:51690/0 (socket says 172.21.15.3:51690) 2023-09-10T01:17:58.716 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 -- 172.21.15.3:0/2650818219 learned_addr learned my addr 172.21.15.3:0/2650818219 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:58.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:17:58 smithi092 ceph-mon[134067]: pgmap v462: 129 pgs: 129 active+clean; 767 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.2 MiB/s wr, 804 op/s 2023-09-10T01:17:58.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d17fff700 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2613f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 -- 172.21.15.3:0/2650818219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc260d1a0 msgr2=0x562bc260d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc260d1a0 0x562bc260d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 -- 172.21.15.3:0/2650818219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 msgr2=0x562bc2613f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2613f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.714+0000 7f3d0effd700 1 -- 172.21.15.3:0/2650818219 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bc2385d80 con 0x562bc2607300 2023-09-10T01:17:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.715+0000 7f3d0effd700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d040a0830 0x7f3d040a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.715+0000 7f3d0effd700 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc2607300 0x562bc2607780 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000dc70 tx=0x7f3d0000df80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df4ce7f266ef9923 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.715+0000 7f3d0dffb700 1 -- 172.21.15.3:0/2650818219 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00011c90 con 0x562bc2607300 2023-09-10T01:17:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.715+0000 7f3d0dffb700 1 -- 172.21.15.3:0/2650818219 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d0000d720 con 0x562bc2607300 2023-09-10T01:17:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.715+0000 7f3d0dffb700 1 -- 172.21.15.3:0/2650818219 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00010610 con 0x562bc2607300 2023-09-10T01:17:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d31908b80 1 -- 172.21.15.3:0/2650818219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc2607300 msgr2=0x562bc2607780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc2607300 0x562bc2607780 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000dc70 tx=0x7f3d0000df80 comp rx=0 tx=0).stop 2023-09-10T01:17:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d0effd700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d040a0830 0x7f3d040a2cf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d0001f000 tx=0x7f3d00020000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d31908b80 1 -- 172.21.15.3:0/2650818219 shutdown_connections 2023-09-10T01:17:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc260d1a0 0x562bc260d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2613f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2650818219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc2607300 0x562bc2607780 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.716+0000 7f3d31908b80 1 -- 172.21.15.3:0/2650818219 >> 172.21.15.3:0/2650818219 conn(0x562bc257c030 msgr2=0x562bc2602560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.717+0000 7f3d31908b80 1 -- 172.21.15.3:0/2650818219 shutdown_connections 2023-09-10T01:17:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.717+0000 7f3d31908b80 1 -- 172.21.15.3:0/2650818219 wait complete. 2023-09-10T01:17:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.717+0000 7f3d31908b80 1 -- start start 2023-09-10T01:17:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.717+0000 7f3d31908b80 1 Processor -- start 2023-09-10T01:17:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2613ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2611120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2611860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d31908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc24f7b10 con 0x562bc2607cc0 2023-09-10T01:17:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d31908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc24f7810 con 0x562bc260d1a0 2023-09-10T01:17:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d31908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc23db8c0 con 0x562bc2607300 2023-09-10T01:17:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d31908b80 10 client.0 _ll_register_callbacks cb 0x562bc2441a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.718+0000 7f3d177fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2613ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d177fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2613ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:34700/0 (socket says 172.21.15.3:34700) 2023-09-10T01:17:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d177fe700 1 -- 172.21.15.3:0/2392037621 learned_addr learned my addr 172.21.15.3:0/2392037621 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d0effd700 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2611120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d177fe700 1 -- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 msgr2=0x562bc2611860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d17fff700 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2611860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d177fe700 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2611860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d177fe700 1 -- 172.21.15.3:0/2392037621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 msgr2=0x562bc2611120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d177fe700 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2611120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.719+0000 7f3d177fe700 1 -- 172.21.15.3:0/2392037621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0000d8a0 con 0x562bc2607300 2023-09-10T01:17:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.722+0000 7f3d177fe700 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2613ea0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3d08011660 tx=0x7f3d08011a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51fa5cde4fe60e81 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.722+0000 7f3cf6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.722+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/2392037621 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08009840 con 0x562bc2607300 2023-09-10T01:17:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.722+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/2392037621 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f3d08012980 con 0x562bc2607300 2023-09-10T01:17:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.723+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562bc256e840 con 0x562bc2607300 2023-09-10T01:17:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.724+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/2392037621 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d08012b40 con 0x562bc2607300 2023-09-10T01:17:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.724+0000 7f3d31908b80 10 client.28876 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.724+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x562bc236ec00 con 0x562bc2607300 2023-09-10T01:17:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.724+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/2392037621 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562bc256e840 con 0x562bc2607300 2023-09-10T01:17:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.724+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/2392037621 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3d0801a3f0 con 0x562bc2607300 2023-09-10T01:17:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.725+0000 7f3d31908b80 10 client.28876 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.725+0000 7f3d31908b80 10 client.28876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.725+0000 7f3d31908b80 20 client.28876 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.725+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc2621550 0x562bc2623970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.725+0000 7f3d31908b80 4 client.28876 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:17:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.725+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562bc25528c0 con 0x562bc2621550 2023-09-10T01:17:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.725+0000 7f3d0effd700 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc2621550 0x562bc2623970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.726+0000 7f3d0effd700 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc2621550 0x562bc2623970 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beb1863e77059a5c server_cookie=67736d7d78669123 in_seq=0 out_seq=0 2023-09-10T01:17:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.727+0000 7f3cf6ffd700 10 client.28876 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.729+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/2392037621 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562bc25528c0 con 0x562bc2621550 2023-09-10T01:17:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.729+0000 7f3cf6ffd700 10 client.28876 handle_command_reply: tid=0 2023-09-10T01:17:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.729+0000 7f3d31908b80 1 client.28876 shutdown 2023-09-10T01:17:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 msgr2=0x562bc2613ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2613ea0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3d08011660 tx=0x7f3d08011a20 comp rx=0 tx=0).stop 2023-09-10T01:17:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 shutdown_connections 2023-09-10T01:17:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc2621550 0x562bc2623970 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2613ea0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2611860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3cf6ffd700 0 client.28876 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2392037621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2611120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.730+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 >> 172.21.15.3:0/2392037621 conn(0x562bc257c030 msgr2=0x562bc26017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.732+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 shutdown_connections 2023-09-10T01:17:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.733+0000 7f3d31908b80 1 -- 172.21.15.3:0/2392037621 wait complete. 2023-09-10T01:17:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.733+0000 7f3d31908b80 20 client.28876 trim_cache size 0 max 16384 2023-09-10T01:17:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.756+0000 7f3d31908b80 1 Processor -- start 2023-09-10T01:17:58.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.756+0000 7f3d31908b80 1 -- start start 2023-09-10T01:17:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc260ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2618520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc26139d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d31908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc2608140 con 0x562bc2607cc0 2023-09-10T01:17:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d31908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc2603d10 con 0x562bc260d1a0 2023-09-10T01:17:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d31908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc2612100 con 0x562bc2607300 2023-09-10T01:17:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d177fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc260ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d177fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc260ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:34710/0 (socket says 172.21.15.3:34710) 2023-09-10T01:17:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d177fe700 1 -- 172.21.15.3:0/2258143079 learned_addr learned my addr 172.21.15.3:0/2258143079 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d17fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc26139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.757+0000 7f3d0effd700 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2618520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.758+0000 7f3d177fe700 1 -- 172.21.15.3:0/2258143079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 msgr2=0x562bc26139d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.758+0000 7f3d177fe700 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc26139d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.758+0000 7f3d177fe700 1 -- 172.21.15.3:0/2258143079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 msgr2=0x562bc2618520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.758+0000 7f3d177fe700 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2618520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.758+0000 7f3d177fe700 1 -- 172.21.15.3:0/2258143079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bc23ac1f0 con 0x562bc2607300 2023-09-10T01:17:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.758+0000 7f3d17fff700 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc26139d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:17:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.758+0000 7f3d177fe700 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc260ed80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d08005d90 tx=0x7f3d08009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe5e4899469cde2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.759+0000 7f3d0e7fc700 1 -- 172.21.15.3:0/2258143079 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0801a3f0 con 0x562bc2607300 2023-09-10T01:17:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.759+0000 7f3d0e7fc700 1 -- 172.21.15.3:0/2258143079 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d0801a550 con 0x562bc2607300 2023-09-10T01:17:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.759+0000 7f3d0e7fc700 1 -- 172.21.15.3:0/2258143079 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08012ba0 con 0x562bc2607300 2023-09-10T01:17:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.760+0000 7f3d31908b80 1 -- 172.21.15.3:0/2258143079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 msgr2=0x562bc260ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.760+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc260ed80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d08005d90 tx=0x7f3d08009820 comp rx=0 tx=0).stop 2023-09-10T01:17:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 -- 172.21.15.3:0/2258143079 shutdown_connections 2023-09-10T01:17:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc260ed80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc26139d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 --2- 172.21.15.3:0/2258143079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2618520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 -- 172.21.15.3:0/2258143079 >> 172.21.15.3:0/2258143079 conn(0x562bc257c030 msgr2=0x562bc260b190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 -- 172.21.15.3:0/2258143079 shutdown_connections 2023-09-10T01:17:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 -- 172.21.15.3:0/2258143079 wait complete. 2023-09-10T01:17:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.761+0000 7f3d31908b80 1 -- start start 2023-09-10T01:17:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.762+0000 7f3d31908b80 1 Processor -- start 2023-09-10T01:17:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.762+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2609f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.762+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2608b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d31908b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2609090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d31908b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc2609750 con 0x562bc2607cc0 2023-09-10T01:17:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d31908b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc26098c0 con 0x562bc260d1a0 2023-09-10T01:17:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d31908b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc2675360 con 0x562bc2607300 2023-09-10T01:17:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d31908b80 10 client.0 _ll_register_callbacks cb 0x7f3d08009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:17:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d177fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2609f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d177fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2609f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:34726/0 (socket says 172.21.15.3:34726) 2023-09-10T01:17:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d177fe700 1 -- 172.21.15.3:0/3228663079 learned_addr learned my addr 172.21.15.3:0/3228663079 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:17:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d17fff700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2609090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d177fe700 1 -- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 msgr2=0x562bc2609090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:17:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d177fe700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2609090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.763+0000 7f3d177fe700 1 -- 172.21.15.3:0/3228663079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 msgr2=0x562bc2608b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:17:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.764+0000 7f3d0effd700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2608b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.764+0000 7f3d177fe700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2608b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.764+0000 7f3d177fe700 1 -- 172.21.15.3:0/3228663079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d18102800 con 0x562bc2607300 2023-09-10T01:17:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.764+0000 7f3d0effd700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2608b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:17:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.765+0000 7f3d177fe700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2609f90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3d08011660 tx=0x7f3d08012e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee381d95135f928f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:17:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.765+0000 7f3cf6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:17:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.765+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/3228663079 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0801a3f0 con 0x562bc2607300 2023-09-10T01:17:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.765+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/3228663079 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f3d08000e40 con 0x562bc2607300 2023-09-10T01:17:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.765+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562bc24b1490 con 0x562bc2607300 2023-09-10T01:17:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.765+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/3228663079 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d08009260 con 0x562bc2607300 2023-09-10T01:17:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.766+0000 7f3d31908b80 10 client.28882 fetch_fsmap learned FSMap version 38 2023-09-10T01:17:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.766+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x562bc236ec00 con 0x562bc2607300 2023-09-10T01:17:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.766+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/3228663079 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d0803d030 con 0x562bc2607300 2023-09-10T01:17:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/3228663079 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3d0801ad20 con 0x562bc2607300 2023-09-10T01:17:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3d31908b80 10 client.28882 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:17:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3d31908b80 10 client.28882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:17:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3d31908b80 20 client.28882 populate_metadata read hostname 'smithi003' 2023-09-10T01:17:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3d31908b80 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc26766d0 0x562bc2678af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:17:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3d31908b80 4 client.28882 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:17:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562bc2513050 con 0x562bc26766d0 2023-09-10T01:17:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.767+0000 7f3d0effd700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc26766d0 0x562bc2678af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:17:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.768+0000 7f3d0effd700 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc26766d0 0x562bc2678af0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63cc8b2a8458a866 server_cookie=9314d6552db5d5fb in_seq=0 out_seq=0 2023-09-10T01:17:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:58.768+0000 7f3cf6ffd700 10 client.28882 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:58.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:17:58 smithi003 ceph-mon[97654]: pgmap v462: 129 pgs: 129 active+clean; 767 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.2 MiB/s wr, 804 op/s 2023-09-10T01:17:59.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 54332 31.01 MB/sec warmup 100 sec latency 366.013 ms 2023-09-10T01:17:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3cf6ffd700 1 -- 172.21.15.3:0/3228663079 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562bc2513050 con 0x562bc26766d0 2023-09-10T01:17:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3cf6ffd700 10 client.28882 handle_command_reply: tid=0 2023-09-10T01:17:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 client.28882 shutdown 2023-09-10T01:17:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 msgr2=0x562bc2609f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2609f90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3d08011660 tx=0x7f3d08012e40 comp rx=0 tx=0).stop 2023-09-10T01:17:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 shutdown_connections 2023-09-10T01:17:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x562bc26766d0 0x562bc2678af0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562bc2607300 0x562bc2609f90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562bc260d1a0 0x562bc2609090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 --2- 172.21.15.3:0/3228663079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc2607cc0 0x562bc2608b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.336+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 >> 172.21.15.3:0/3228663079 conn(0x562bc257c030 msgr2=0x562bc2602090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:59.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.337+0000 7f3cf6ffd700 0 client.28882 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:17:59.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.337+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 shutdown_connections 2023-09-10T01:17:59.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.337+0000 7f3d31908b80 1 -- 172.21.15.3:0/3228663079 wait complete. 2023-09-10T01:17:59.340 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.337+0000 7f3d31908b80 20 client.28882 trim_cache size 0 max 16384 2023-09-10T01:17:59.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2023-09-10T01:17:59.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2023-09-10T01:17:59.341 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2023-09-10T01:17:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2023-09-10T01:17:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d040a0830 msgr2=0x7f3d040a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:59.342 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d040a0830 0x7f3d040a2cf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d0001f000 tx=0x7f3d00020000 comp rx=0 tx=0).stop 2023-09-10T01:17:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d1810cde0 msgr2=0x7f3d18107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:17:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d1810cde0 0x7f3d18107210 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009450 tx=0x7f3d10009530 comp rx=0 tx=0).stop 2023-09-10T01:17:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 shutdown_connections 2023-09-10T01:17:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3d040a0830 0x7f3d040a2cf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d1810cde0 0x7f3d18107210 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d18072a30 0x7f3d18106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 --2- 172.21.15.3:0/1031144675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3d18073350 0x7f3d1810c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:17:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 >> 172.21.15.3:0/1031144675 conn(0x7f3d180fc0b0 msgr2=0x7f3d18111fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:17:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 shutdown_connections 2023-09-10T01:17:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:17:59.338+0000 7f3d1dfff700 1 -- 172.21.15.3:0/1031144675 wait complete. 2023-09-10T01:18:00.194 INFO:tasks.workunit.client.0.smithi003.stdout: 10 54841 30.92 MB/sec warmup 101 sec latency 501.934 ms 2023-09-10T01:18:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:00 smithi136 ceph-mon[133708]: pgmap v463: 129 pgs: 129 active+clean; 776 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 871 op/s 2023-09-10T01:18:00.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:00 smithi092 ceph-mon[134067]: pgmap v463: 129 pgs: 129 active+clean; 776 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 871 op/s 2023-09-10T01:18:00.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:00 smithi003 ceph-mon[97654]: pgmap v463: 129 pgs: 129 active+clean; 776 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 871 op/s 2023-09-10T01:18:01.191 INFO:tasks.workunit.client.0.smithi003.stdout: 10 55602 31.00 MB/sec warmup 102 sec latency 229.139 ms 2023-09-10T01:18:02.193 INFO:tasks.workunit.client.0.smithi003.stdout: 10 55788 30.77 MB/sec warmup 103 sec latency 313.370 ms 2023-09-10T01:18:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:02 smithi136 ceph-mon[133708]: pgmap v464: 129 pgs: 129 active+clean; 793 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.7 MiB/s wr, 833 op/s 2023-09-10T01:18:02.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:02 smithi092 ceph-mon[134067]: pgmap v464: 129 pgs: 129 active+clean; 793 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.7 MiB/s wr, 833 op/s 2023-09-10T01:18:02.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:02 smithi003 ceph-mon[97654]: pgmap v464: 129 pgs: 129 active+clean; 793 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.7 MiB/s wr, 833 op/s 2023-09-10T01:18:03.192 INFO:tasks.workunit.client.0.smithi003.stdout: 10 55798 30.50 MB/sec warmup 104 sec latency 483.052 ms 2023-09-10T01:18:04.193 INFO:tasks.workunit.client.0.smithi003.stdout: 10 55805 30.23 MB/sec warmup 105 sec latency 317.201 ms 2023-09-10T01:18:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:04 smithi136 ceph-mon[133708]: pgmap v465: 129 pgs: 129 active+clean; 793 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.6 MiB/s wr, 459 op/s 2023-09-10T01:18:04.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:04 smithi092 ceph-mon[134067]: pgmap v465: 129 pgs: 129 active+clean; 793 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.6 MiB/s wr, 459 op/s 2023-09-10T01:18:04.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:04 smithi003 ceph-mon[97654]: pgmap v465: 129 pgs: 129 active+clean; 793 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.6 MiB/s wr, 459 op/s 2023-09-10T01:18:05.193 INFO:tasks.workunit.client.0.smithi003.stdout: 10 55815 29.96 MB/sec warmup 106 sec latency 827.313 ms 2023-09-10T01:18:05.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:05 smithi136 ceph-mon[133708]: osdmap e109: 12 total, 12 up, 12 in 2023-09-10T01:18:05.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:05 smithi092 ceph-mon[134067]: osdmap e109: 12 total, 12 up, 12 in 2023-09-10T01:18:05.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:05 smithi003 ceph-mon[97654]: osdmap e109: 12 total, 12 up, 12 in 2023-09-10T01:18:06.194 INFO:tasks.workunit.client.0.smithi003.stdout: 10 55892 29.72 MB/sec warmup 107 sec latency 889.981 ms 2023-09-10T01:18:06.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:06 smithi092 ceph-mon[134067]: pgmap v467: 129 pgs: 129 active+clean; 795 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.0 MiB/s wr, 707 op/s 2023-09-10T01:18:06.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:06 smithi003 ceph-mon[97654]: pgmap v467: 129 pgs: 129 active+clean; 795 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.0 MiB/s wr, 707 op/s 2023-09-10T01:18:06.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:06 smithi136 ceph-mon[133708]: pgmap v467: 129 pgs: 129 active+clean; 795 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.0 MiB/s wr, 707 op/s 2023-09-10T01:18:07.200 INFO:tasks.workunit.client.0.smithi003.stdout: 10 56087 29.57 MB/sec warmup 108 sec latency 874.472 ms 2023-09-10T01:18:07.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:07 smithi092 ceph-mon[134067]: pgmap v468: 129 pgs: 129 active+clean; 795 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.0 MiB/s wr, 707 op/s 2023-09-10T01:18:07.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:07 smithi003 ceph-mon[97654]: pgmap v468: 129 pgs: 129 active+clean; 795 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.0 MiB/s wr, 707 op/s 2023-09-10T01:18:07.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:07 smithi136 ceph-mon[133708]: pgmap v468: 129 pgs: 129 active+clean; 795 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.0 MiB/s wr, 707 op/s 2023-09-10T01:18:08.200 INFO:tasks.workunit.client.0.smithi003.stdout: 10 56497 29.55 MB/sec warmup 109 sec latency 630.426 ms 2023-09-10T01:18:09.200 INFO:tasks.workunit.client.0.smithi003.stdout: 10 56959 29.53 MB/sec warmup 110 sec latency 76.003 ms 2023-09-10T01:18:09.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:09 smithi136 ceph-mon[133708]: pgmap v469: 129 pgs: 129 active+clean; 799 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 690 op/s 2023-09-10T01:18:09.965 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:09 smithi092 ceph-mon[134067]: pgmap v469: 129 pgs: 129 active+clean; 799 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 690 op/s 2023-09-10T01:18:10.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:09 smithi003 ceph-mon[97654]: pgmap v469: 129 pgs: 129 active+clean; 799 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 690 op/s 2023-09-10T01:18:10.200 INFO:tasks.workunit.client.0.smithi003.stdout: 10 57528 29.51 MB/sec warmup 111 sec latency 60.605 ms 2023-09-10T01:18:10.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:18:11.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.151+0000 7f05209ff700 1 -- 172.21.15.3:0/3107176317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 msgr2=0x7f051c105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.151+0000 7f05209ff700 1 --2- 172.21.15.3:0/3107176317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 0x7f051c105a10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f050c0099c0 tx=0x7f050c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.151+0000 7f05209ff700 1 -- 172.21.15.3:0/3107176317 shutdown_connections 2023-09-10T01:18:11.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.151+0000 7f05209ff700 1 --2- 172.21.15.3:0/3107176317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 0x7f051c105a10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.151+0000 7f05209ff700 1 --2- 172.21.15.3:0/3107176317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0ff060 0x7f051c10d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.151+0000 7f05209ff700 1 --2- 172.21.15.3:0/3107176317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f051c0fe6a0 0x7f051c0feb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.151+0000 7f05209ff700 1 -- 172.21.15.3:0/3107176317 >> 172.21.15.3:0/3107176317 conn(0x7f051c0fa2e0 msgr2=0x7f051c0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.152+0000 7f05209ff700 1 -- 172.21.15.3:0/3107176317 shutdown_connections 2023-09-10T01:18:11.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.152+0000 7f05209ff700 1 -- 172.21.15.3:0/3107176317 wait complete. 2023-09-10T01:18:11.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.153+0000 7f05209ff700 1 Processor -- start 2023-09-10T01:18:11.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.153+0000 7f05209ff700 1 -- start start 2023-09-10T01:18:11.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f05209ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0fe6a0 0x7f051c114590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f05209ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f051c0ff060 0x7f051c114af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f05209ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 0x7f051c10f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f05209ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c10fe70 con 0x7f051c0fe6a0 2023-09-10T01:18:11.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f05209ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c10ffe0 con 0x7f051c0ff060 2023-09-10T01:18:11.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f05209ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c1101a0 con 0x7f051c105630 2023-09-10T01:18:11.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f051a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0fe6a0 0x7f051c114590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f0519d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f051c0ff060 0x7f051c114af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f051a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0fe6a0 0x7f051c114590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55568/0 (socket says 172.21.15.3:55568) 2023-09-10T01:18:11.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f051a59c700 1 -- 172.21.15.3:0/3366198832 learned_addr learned my addr 172.21.15.3:0/3366198832 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.154+0000 7f051ad9d700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 0x7f051c10f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.155+0000 7f051a59c700 1 -- 172.21.15.3:0/3366198832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 msgr2=0x7f051c10f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.155+0000 7f051a59c700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 0x7f051c10f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.155+0000 7f051a59c700 1 -- 172.21.15.3:0/3366198832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f051c0ff060 msgr2=0x7f051c114af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.155+0000 7f051a59c700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f051c0ff060 0x7f051c114af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.155+0000 7f051a59c700 1 -- 172.21.15.3:0/3366198832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c0095f0 con 0x7f051c0fe6a0 2023-09-10T01:18:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.155+0000 7f051a59c700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0fe6a0 0x7f051c114590 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f050c006010 tx=0x7f050c005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.155+0000 7f050b7fe700 1 -- 172.21.15.3:0/3366198832 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c01d070 con 0x7f051c0fe6a0 2023-09-10T01:18:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.156+0000 7f050b7fe700 1 -- 172.21.15.3:0/3366198832 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f050c0056d0 con 0x7f051c0fe6a0 2023-09-10T01:18:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.156+0000 7f050b7fe700 1 -- 172.21.15.3:0/3366198832 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c022ce0 con 0x7f051c0fe6a0 2023-09-10T01:18:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.158+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c110420 con 0x7f051c0fe6a0 2023-09-10T01:18:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.159+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c11a4f0 con 0x7f051c0fe6a0 2023-09-10T01:18:11.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.167+0000 7f053434eb80 1 Processor -- start 2023-09-10T01:18:11.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.168+0000 7f053434eb80 1 -- start start 2023-09-10T01:18:11.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.168+0000 7f053434eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e22af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.168+0000 7f053434eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e23030 0x555e03e2c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.168+0000 7f053434eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e2f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.168+0000 7f053434eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03d7b730 con 0x555e03e2ce40 2023-09-10T01:18:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.168+0000 7f053434eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03d79050 con 0x555e03e23030 2023-09-10T01:18:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.168+0000 7f053434eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03d74940 con 0x555e03e22710 2023-09-10T01:18:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e22af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e22af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42798/0 (socket says 172.21.15.3:42798) 2023-09-10T01:18:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 -- 172.21.15.3:0/1355441850 learned_addr learned my addr 172.21.15.3:0/1355441850 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f0519d9b700 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e2f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 -- 172.21.15.3:0/1355441850 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e23030 msgr2=0x555e03e2c900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:18:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e23030 0x555e03e2c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 -- 172.21.15.3:0/1355441850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 msgr2=0x555e03e2f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e2f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 -- 172.21.15.3:0/1355441850 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e03ba0de0 con 0x555e03e22710 2023-09-10T01:18:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.169+0000 7f051ad9d700 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e22af0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0514003fc0 tx=0x7f0514009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e076893f8aa646c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.170+0000 7f0508ff9700 1 -- 172.21.15.3:0/1355441850 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514019070 con 0x555e03e22710 2023-09-10T01:18:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.170+0000 7f0508ff9700 1 -- 172.21.15.3:0/1355441850 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f051400f040 con 0x555e03e22710 2023-09-10T01:18:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.170+0000 7f0508ff9700 1 -- 172.21.15.3:0/1355441850 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514007b20 con 0x555e03e22710 2023-09-10T01:18:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.171+0000 7f053434eb80 1 -- 172.21.15.3:0/1355441850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 msgr2=0x555e03e22af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.171+0000 7f053434eb80 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e22af0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0514003fc0 tx=0x7f0514009ba0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.171+0000 7f053434eb80 1 -- 172.21.15.3:0/1355441850 shutdown_connections 2023-09-10T01:18:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.171+0000 7f053434eb80 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e22af0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.171+0000 7f053434eb80 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e2f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.171+0000 7f053434eb80 1 --2- 172.21.15.3:0/1355441850 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e23030 0x555e03e2c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.171+0000 7f053434eb80 1 -- 172.21.15.3:0/1355441850 >> 172.21.15.3:0/1355441850 conn(0x555e03d97090 msgr2=0x555e03e1dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.172+0000 7f050b7fe700 1 -- 172.21.15.3:0/3366198832 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f050c022470 con 0x7f051c0fe6a0 2023-09-10T01:18:11.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.173+0000 7f053434eb80 1 -- 172.21.15.3:0/1355441850 shutdown_connections 2023-09-10T01:18:11.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.174+0000 7f053434eb80 1 -- 172.21.15.3:0/1355441850 wait complete. 2023-09-10T01:18:11.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.174+0000 7f053434eb80 1 -- start start 2023-09-10T01:18:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.175+0000 7f053434eb80 1 Processor -- start 2023-09-10T01:18:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f050b7fe700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05040a07c0 0x7f05040a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f050b7fe700 1 -- 172.21.15.3:0/3366198832 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f050c022e40 con 0x7f051c0fe6a0 2023-09-10T01:18:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f053434eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e22710 0x555e03e29c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f053434eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e2ce40 0x555e03e2a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f053434eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2a6c0 0x555e03e38ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f053434eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03d12b70 con 0x555e03e2a6c0 2023-09-10T01:18:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f053434eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03d12870 con 0x555e03e22710 2023-09-10T01:18:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.176+0000 7f053434eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03bf6920 con 0x555e03e2ce40 2023-09-10T01:18:11.182 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.177+0000 7f053434eb80 10 client.0 _ll_register_callbacks cb 0x555e03c5ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:11.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.177+0000 7f051ad9d700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05040a07c0 0x7f05040a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.177+0000 7f051ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2a6c0 0x555e03e38ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.177+0000 7f051ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2a6c0 0x555e03e38ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55586/0 (socket says 172.21.15.3:55586) 2023-09-10T01:18:11.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.177+0000 7f051ad9d700 1 -- 172.21.15.3:0/2592887627 learned_addr learned my addr 172.21.15.3:0/2592887627 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:11.183 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.178+0000 7f0519d9b700 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e2ce40 0x555e03e2a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.178+0000 7f051ad9d700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05040a07c0 0x7f05040a2c80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0514009580 tx=0x7f05140045d0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.178+0000 7f051ad9d700 1 -- 172.21.15.3:0/2592887627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e2ce40 msgr2=0x555e03e2a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:11.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.178+0000 7f051ad9d700 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e2ce40 0x555e03e2a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.178+0000 7f051ad9d700 1 -- 172.21.15.3:0/2592887627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e22710 msgr2=0x555e03e29c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:18:11.184 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.178+0000 7f051ad9d700 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e22710 0x555e03e29c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.178+0000 7f051ad9d700 1 -- 172.21.15.3:0/2592887627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0514003bf0 con 0x555e03e2a6c0 2023-09-10T01:18:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.179+0000 7f0519d9b700 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e2ce40 0x555e03e2a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:18:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.179+0000 7f051ad9d700 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2a6c0 0x555e03e38ae0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0514007950 tx=0x7f0514023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad01ceac6f4ac204 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.179+0000 7f0501ffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:11.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.180+0000 7f0501ffb700 1 -- 172.21.15.3:0/2592887627 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514019050 con 0x555e03e2a6c0 2023-09-10T01:18:11.186 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.180+0000 7f0501ffb700 1 -- 172.21.15.3:0/2592887627 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f0514027070 con 0x555e03e2a6c0 2023-09-10T01:18:11.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.180+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e03ccc4f0 con 0x555e03e2a6c0 2023-09-10T01:18:11.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.182+0000 7f0501ffb700 1 -- 172.21.15.3:0/2592887627 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0514021e40 con 0x555e03e2a6c0 2023-09-10T01:18:11.187 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.184+0000 7f053434eb80 10 client.18894 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:11.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.184+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555e03b89c60 con 0x555e03e2a6c0 2023-09-10T01:18:11.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.185+0000 7f0501ffb700 1 -- 172.21.15.3:0/2592887627 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e03ccc4f0 con 0x555e03e2a6c0 2023-09-10T01:18:11.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.185+0000 7f0501ffb700 1 -- 172.21.15.3:0/2592887627 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0514015a20 con 0x555e03e2a6c0 2023-09-10T01:18:11.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.186+0000 7f053434eb80 10 client.18894 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:11.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.186+0000 7f053434eb80 10 client.18894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.186+0000 7f053434eb80 20 client.18894 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.186+0000 7f053434eb80 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e3cf50 0x555e03e3f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.186+0000 7f053434eb80 4 client.18894 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:11.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.186+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e03d898a0 con 0x555e03e3cf50 2023-09-10T01:18:11.191 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.188+0000 7f051a59c700 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e3cf50 0x555e03e3f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.189+0000 7f051a59c700 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e3cf50 0x555e03e3f370 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe95b10130dcc636 server_cookie=26b9488a01ae40e9 in_seq=0 out_seq=0 2023-09-10T01:18:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.189+0000 7f0501ffb700 10 client.18894 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.190+0000 7f0501ffb700 1 -- 172.21.15.3:0/2592887627 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555e03d898a0 con 0x555e03e3cf50 2023-09-10T01:18:11.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.190+0000 7f0501ffb700 10 client.18894 handle_command_reply: tid=0 2023-09-10T01:18:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.190+0000 7f053434eb80 1 client.18894 shutdown 2023-09-10T01:18:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.191+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2a6c0 msgr2=0x555e03e38ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.191+0000 7f053434eb80 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2a6c0 0x555e03e38ae0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0514007950 tx=0x7f0514023000 comp rx=0 tx=0).stop 2023-09-10T01:18:11.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 shutdown_connections 2023-09-10T01:18:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e3cf50 0x555e03e3f370 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2a6c0 0x555e03e38ae0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e22710 0x555e03e29c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 --2- 172.21.15.3:0/2592887627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e2ce40 0x555e03e2a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 >> 172.21.15.3:0/2592887627 conn(0x555e03d97090 msgr2=0x555e03e1d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f0501ffb700 0 client.18894 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 shutdown_connections 2023-09-10T01:18:11.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 1 -- 172.21.15.3:0/2592887627 wait complete. 2023-09-10T01:18:11.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.192+0000 7f053434eb80 20 client.18894 trim_cache size 0 max 16384 2023-09-10T01:18:11.202 INFO:tasks.workunit.client.0.smithi003.stdout: 10 58036 29.46 MB/sec warmup 112 sec latency 80.596 ms 2023-09-10T01:18:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.218+0000 7f053434eb80 1 Processor -- start 2023-09-10T01:18:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f053434eb80 1 -- start start 2023-09-10T01:18:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f053434eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e33a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.220 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f053434eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e24ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f053434eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e253e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f053434eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03e1f270 con 0x555e03e2ce40 2023-09-10T01:18:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f053434eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03e1a400 con 0x555e03e2a6c0 2023-09-10T01:18:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f053434eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03e33f70 con 0x555e03e22710 2023-09-10T01:18:11.221 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f051a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e33a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f051a59c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e33a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42814/0 (socket says 172.21.15.3:42814) 2023-09-10T01:18:11.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f051a59c700 1 -- 172.21.15.3:0/3276228723 learned_addr learned my addr 172.21.15.3:0/3276228723 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:11.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f0519d9b700 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e24ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.219+0000 7f051ad9d700 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e253e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.220+0000 7f051a59c700 1 -- 172.21.15.3:0/3276228723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 msgr2=0x555e03e24ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.220+0000 7f051a59c700 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e24ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.220+0000 7f051a59c700 1 -- 172.21.15.3:0/3276228723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 msgr2=0x555e03e253e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:11.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.220+0000 7f051a59c700 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e253e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.220+0000 7f051a59c700 1 -- 172.21.15.3:0/3276228723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e03bc7250 con 0x555e03e22710 2023-09-10T01:18:11.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.220+0000 7f051a59c700 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e33a30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f050c096fb0 tx=0x7f050c009cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f9aa45704ce12ee server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.225 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f05097fa700 1 -- 172.21.15.3:0/3276228723 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c0caea0 con 0x555e03e22710 2023-09-10T01:18:11.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f05097fa700 1 -- 172.21.15.3:0/3276228723 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f050c09e170 con 0x555e03e22710 2023-09-10T01:18:11.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f05097fa700 1 -- 172.21.15.3:0/3276228723 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c098a00 con 0x555e03e22710 2023-09-10T01:18:11.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f053434eb80 1 -- 172.21.15.3:0/3276228723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 msgr2=0x555e03e33a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.226 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f053434eb80 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e33a30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f050c096fb0 tx=0x7f050c009cd0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f053434eb80 1 -- 172.21.15.3:0/3276228723 shutdown_connections 2023-09-10T01:18:11.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f053434eb80 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e33a30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f053434eb80 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e253e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f053434eb80 1 --2- 172.21.15.3:0/3276228723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e24ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.221+0000 7f053434eb80 1 -- 172.21.15.3:0/3276228723 >> 172.21.15.3:0/3276228723 conn(0x555e03d97090 msgr2=0x555e03e1c320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.222+0000 7f053434eb80 1 -- 172.21.15.3:0/3276228723 shutdown_connections 2023-09-10T01:18:11.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.222+0000 7f053434eb80 1 -- 172.21.15.3:0/3276228723 wait complete. 2023-09-10T01:18:11.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.222+0000 7f053434eb80 1 -- start start 2023-09-10T01:18:11.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 1 Processor -- start 2023-09-10T01:18:11.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e8a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e8aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e84d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03e85320 con 0x555e03e2ce40 2023-09-10T01:18:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03e85490 con 0x555e03e2a6c0 2023-09-10T01:18:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e03e85650 con 0x555e03e22710 2023-09-10T01:18:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.223+0000 7f053434eb80 10 client.0 _ll_register_callbacks cb 0x7f0514007e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e8aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.232 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e8aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47820/0 (socket says 172.21.15.3:47820) 2023-09-10T01:18:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 -- 172.21.15.3:0/1106767385 learned_addr learned my addr 172.21.15.3:0/1106767385 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f051a59c700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e8a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f051ad9d700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e84d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 -- 172.21.15.3:0/1106767385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 msgr2=0x555e03e8a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:11.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e8a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 -- 172.21.15.3:0/1106767385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 msgr2=0x555e03e84d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e84d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f0519d9b700 1 -- 172.21.15.3:0/1106767385 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c11a4f0 con 0x555e03e2a6c0 2023-09-10T01:18:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f051a59c700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e8a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.224+0000 7f051ad9d700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e84d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:18:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f0519d9b700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e8aa10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f051c0fcc90 tx=0x7f0510015920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6482e3da0899e71 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f0501ffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:18:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f0501ffb700 1 -- 172.21.15.3:0/1106767385 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510016840 con 0x555e03e2a6c0 2023-09-10T01:18:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f051400f040 con 0x555e03e2a6c0 2023-09-10T01:18:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f0501ffb700 1 -- 172.21.15.3:0/1106767385 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f05100169a0 con 0x555e03e2a6c0 2023-09-10T01:18:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f0501ffb700 1 -- 172.21.15.3:0/1106767385 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0510016b60 con 0x555e03e2a6c0 2023-09-10T01:18:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f053434eb80 10 client.29015 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.225+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555e03b9f2b0 con 0x555e03e2a6c0 2023-09-10T01:18:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.226+0000 7f0501ffb700 1 -- 172.21.15.3:0/1106767385 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f051400f040 con 0x555e03e2a6c0 2023-09-10T01:18:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f0501ffb700 1 -- 172.21.15.3:0/1106767385 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f051001d3a0 con 0x555e03e2a6c0 2023-09-10T01:18:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f053434eb80 10 client.29015 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f053434eb80 10 client.29015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f053434eb80 20 client.29015 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f053434eb80 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e924c0 0x555e03e948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f053434eb80 4 client.29015 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:18:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555e03d6d920 con 0x555e03e924c0 2023-09-10T01:18:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.227+0000 7f051a59c700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e924c0 0x555e03e948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.228+0000 7f0501ffb700 10 client.29015 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.228+0000 7f051a59c700 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e924c0 0x555e03e948e0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=225876583221e3f7 server_cookie=286b696aa5fea50b in_seq=0 out_seq=0 2023-09-10T01:18:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.229+0000 7f0501ffb700 1 -- 172.21.15.3:0/1106767385 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555e03d6d920 con 0x555e03e924c0 2023-09-10T01:18:11.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.229+0000 7f0501ffb700 10 client.29015 handle_command_reply: tid=0 2023-09-10T01:18:11.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.229+0000 7f053434eb80 1 client.29015 shutdown 2023-09-10T01:18:11.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.229+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 msgr2=0x555e03e8aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.229+0000 7f053434eb80 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e8aa10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f051c0fcc90 tx=0x7f0510015920 comp rx=0 tx=0).stop 2023-09-10T01:18:11.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 shutdown_connections 2023-09-10T01:18:11.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x555e03e924c0 0x555e03e948e0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555e03e22710 0x555e03e8a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e03e2ce40 0x555e03e84d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 --2- 172.21.15.3:0/1106767385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555e03e2a6c0 0x555e03e8aa10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 >> 172.21.15.3:0/1106767385 conn(0x555e03d97090 msgr2=0x555e03e38180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f0501ffb700 0 client.29015 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:11.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 shutdown_connections 2023-09-10T01:18:11.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 1 -- 172.21.15.3:0/1106767385 wait complete. 2023-09-10T01:18:11.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.230+0000 7f053434eb80 20 client.29015 trim_cache size 0 max 16384 2023-09-10T01:18:11.241 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:18:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:18:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:18:11.242 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:18:11.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05040a07c0 msgr2=0x7f05040a2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05040a07c0 0x7f05040a2c80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0514009580 tx=0x7f05140045d0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0fe6a0 msgr2=0x7f051c114590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0fe6a0 0x7f051c114590 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f050c006010 tx=0x7f050c005e70 comp rx=0 tx=0).stop 2023-09-10T01:18:11.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 shutdown_connections 2023-09-10T01:18:11.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f05040a07c0 0x7f05040a2c80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f051c105630 0x7f051c10f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f051c0fe6a0 0x7f051c114590 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 --2- 172.21.15.3:0/3366198832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f051c0ff060 0x7f051c114af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.231+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 >> 172.21.15.3:0/3366198832 conn(0x7f051c0fa2e0 msgr2=0x7f051c0faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.232+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 shutdown_connections 2023-09-10T01:18:11.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.232+0000 7f05209ff700 1 -- 172.21.15.3:0/3366198832 wait complete. 2023-09-10T01:18:11.258 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:18:11.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:18:11.301 INFO:tasks.workunit.client.0.smithi003.stdout:[58305] write failed on handle 13895 (Resource temporarily unavailable) 2023-09-10T01:18:11.301 INFO:tasks.workunit.client.0.smithi003.stdout:Child failed with status 1 2023-09-10T01:18:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T01:18:11.312 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2023-09-10T01:18:11.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-09-10T01:18:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.486+0000 7f118d9ff700 1 -- 172.21.15.3:0/1428248742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11880729a0 msgr2=0x7f1188072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.486+0000 7f118d9ff700 1 --2- 172.21.15.3:0/1428248742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11880729a0 0x7f1188072d80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1178009990 tx=0x7f1178009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.486+0000 7f118d9ff700 1 -- 172.21.15.3:0/1428248742 shutdown_connections 2023-09-10T01:18:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.486+0000 7f118d9ff700 1 --2- 172.21.15.3:0/1428248742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11880729a0 0x7f1188072d80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.486+0000 7f118d9ff700 1 --2- 172.21.15.3:0/1428248742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1188111610 0x7f1188113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.486+0000 7f118d9ff700 1 --2- 172.21.15.3:0/1428248742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11880732c0 0x7f11881110d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.486+0000 7f118d9ff700 1 -- 172.21.15.3:0/1428248742 >> 172.21.15.3:0/1428248742 conn(0x7f11880fc0d0 msgr2=0x7f11880fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.487+0000 7f118d9ff700 1 -- 172.21.15.3:0/1428248742 shutdown_connections 2023-09-10T01:18:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.487+0000 7f118d9ff700 1 -- 172.21.15.3:0/1428248742 wait complete. 2023-09-10T01:18:11.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.487+0000 7f118d9ff700 1 Processor -- start 2023-09-10T01:18:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f118d9ff700 1 -- start start 2023-09-10T01:18:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f118d9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11880729a0 0x7f118819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f118d9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11880732c0 0x7f118819d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f118d9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1188111610 0x7f11881a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f118d9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881a1df0 con 0x7f11880732c0 2023-09-10T01:18:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f118d9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881a1f60 con 0x7f11880729a0 2023-09-10T01:18:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f118d9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881a2120 con 0x7f1188111610 2023-09-10T01:18:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f11877fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1188111610 0x7f11881a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f1186ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11880729a0 0x7f118819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f11867fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11880732c0 0x7f118819d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f11877fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1188111610 0x7f11881a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42844/0 (socket says 172.21.15.3:42844) 2023-09-10T01:18:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.488+0000 7f11877fe700 1 -- 172.21.15.3:0/1254825442 learned_addr learned my addr 172.21.15.3:0/1254825442 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f1186ffd700 1 -- 172.21.15.3:0/1254825442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1188111610 msgr2=0x7f11881a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f1186ffd700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1188111610 0x7f11881a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f1186ffd700 1 -- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11880732c0 msgr2=0x7f118819d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f1186ffd700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11880732c0 0x7f118819d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f1186ffd700 1 -- 172.21.15.3:0/1254825442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1180009c40 con 0x7f11880729a0 2023-09-10T01:18:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f1186ffd700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11880729a0 0x7f118819ce10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1178000c00 tx=0x7f1178005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f118c9fd700 1 -- 172.21.15.3:0/1254825442 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117801d070 con 0x7f11880729a0 2023-09-10T01:18:11.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.489+0000 7f118d9ff700 1 -- 172.21.15.3:0/1254825442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11780095f0 con 0x7f11880729a0 2023-09-10T01:18:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.490+0000 7f118d9ff700 1 -- 172.21.15.3:0/1254825442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11881a9a60 con 0x7f11880729a0 2023-09-10T01:18:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.490+0000 7f118c9fd700 1 -- 172.21.15.3:0/1254825442 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1178021410 con 0x7f11880729a0 2023-09-10T01:18:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.490+0000 7f118c9fd700 1 -- 172.21.15.3:0/1254825442 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178017650 con 0x7f11880729a0 2023-09-10T01:18:11.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.491+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11680052f0 con 0x7f11880729a0 2023-09-10T01:18:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.493+0000 7f118c9fd700 1 -- 172.21.15.3:0/1254825442 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f11780217d0 con 0x7f11880729a0 2023-09-10T01:18:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.494+0000 7f118c9fd700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f11700a07b0 0x7f11700a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.494+0000 7f118c9fd700 1 -- 172.21.15.3:0/1254825442 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f11780cd910 con 0x7f11880729a0 2023-09-10T01:18:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.494+0000 7f11867fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f11700a07b0 0x7f11700a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.495+0000 7f11867fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f11700a07b0 0x7f11700a2c70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f118819e420 tx=0x7f117c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.496+0000 7f118c9fd700 1 -- 172.21.15.3:0/1254825442 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1178096cf0 con 0x7f11880729a0 2023-09-10T01:18:11.759 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.759+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1168005160 con 0x7f11880729a0 2023-09-10T01:18:11.760 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.760+0000 7f118c9fd700 1 -- 172.21.15.3:0/1254825442 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+11299 (secure 0 0 0) 0x7f117809a310 con 0x7f11880729a0 2023-09-10T01:18:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:18:11.761 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:16:25.285703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:18:11.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f11700a07b0 msgr2=0x7f11700a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f11700a07b0 0x7f11700a2c70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f118819e420 tx=0x7f117c009260 comp rx=0 tx=0).stop 2023-09-10T01:18:11.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11880729a0 msgr2=0x7f118819ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11880729a0 0x7f118819ce10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1178000c00 tx=0x7f1178005740 comp rx=0 tx=0).stop 2023-09-10T01:18:11.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 shutdown_connections 2023-09-10T01:18:11.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f11700a07b0 0x7f11700a2c70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.764 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1188111610 0x7f11881a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11880729a0 0x7f118819ce10 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 --2- 172.21.15.3:0/1254825442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11880732c0 0x7f118819d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.762+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 >> 172.21.15.3:0/1254825442 conn(0x7f11880fc0d0 msgr2=0x7f11880fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.763+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 shutdown_connections 2023-09-10T01:18:11.765 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.763+0000 7f116e7fc700 1 -- 172.21.15.3:0/1254825442 wait complete. 2023-09-10T01:18:11.766 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 38 2023-09-10T01:18:11.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:18:11.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 -- 172.21.15.3:0/2915883933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480100640 msgr2=0x7f2480100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:11.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 --2- 172.21.15.3:0/2915883933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480100640 0x7f2480100ac0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2470009990 tx=0x7f2470009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 -- 172.21.15.3:0/2915883933 shutdown_connections 2023-09-10T01:18:11.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 --2- 172.21.15.3:0/2915883933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2480101000 0x7f248010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 --2- 172.21.15.3:0/2915883933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2480106650 0x7f2480106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 --2- 172.21.15.3:0/2915883933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480100640 0x7f2480100ac0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 -- 172.21.15.3:0/2915883933 >> 172.21.15.3:0/2915883933 conn(0x7f24800fc170 msgr2=0x7f24800fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 -- 172.21.15.3:0/2915883933 shutdown_connections 2023-09-10T01:18:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.993+0000 7f24879ff700 1 -- 172.21.15.3:0/2915883933 wait complete. 2023-09-10T01:18:11.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.995+0000 7f24879ff700 1 Processor -- start 2023-09-10T01:18:11.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.995+0000 7f24879ff700 1 -- start start 2023-09-10T01:18:11.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f24879ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2480100640 0x7f248019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f24879ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480101000 0x7f248019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f24879ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2480106650 0x7f2480198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:11.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f24879ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2480198d80 con 0x7f2480101000 2023-09-10T01:18:11.999 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f24879ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2480198ef0 con 0x7f2480100640 2023-09-10T01:18:12.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f24879ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24801990b0 con 0x7f2480106650 2023-09-10T01:18:12.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480101000 0x7f248019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480101000 0x7f248019ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55628/0 (socket says 172.21.15.3:55628) 2023-09-10T01:18:12.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 -- 172.21.15.3:0/882550743 learned_addr learned my addr 172.21.15.3:0/882550743 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f248579b700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2480100640 0x7f248019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2485f9c700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2480106650 0x7f2480198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 -- 172.21.15.3:0/882550743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2480106650 msgr2=0x7f2480198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2480106650 0x7f2480198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 -- 172.21.15.3:0/882550743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2480100640 msgr2=0x7f248019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2480100640 0x7f248019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2484f9a700 1 -- 172.21.15.3:0/882550743 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700095f0 con 0x7f2480101000 2023-09-10T01:18:12.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.996+0000 7f2485f9c700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2480106650 0x7f2480198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:18:12.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.997+0000 7f2484f9a700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480101000 0x7f248019ebe0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f24700095c0 tx=0x7f247000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.997+0000 7f24767fc700 1 -- 172.21.15.3:0/882550743 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247001d070 con 0x7f2480101000 2023-09-10T01:18:12.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.997+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2480199330 con 0x7f2480101000 2023-09-10T01:18:12.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.997+0000 7f24767fc700 1 -- 172.21.15.3:0/882550743 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2470005050 con 0x7f2480101000 2023-09-10T01:18:12.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.997+0000 7f24767fc700 1 -- 172.21.15.3:0/882550743 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470004b30 con 0x7f2480101000 2023-09-10T01:18:12.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:11.997+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24801a5690 con 0x7f2480101000 2023-09-10T01:18:12.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.000+0000 7f24767fc700 1 -- 172.21.15.3:0/882550743 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f2470004c90 con 0x7f2480101000 2023-09-10T01:18:12.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.003+0000 7f24767fc700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f246c0a07b0 0x7f246c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.003+0000 7f24767fc700 1 -- 172.21.15.3:0/882550743 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f24700cd330 con 0x7f2480101000 2023-09-10T01:18:12.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.003+0000 7f248579b700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f246c0a07b0 0x7f246c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.003+0000 7f249b3b7b80 1 Processor -- start 2023-09-10T01:18:12.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f249b3b7b80 1 -- start start 2023-09-10T01:18:12.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f248579b700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f246c0a07b0 0x7f246c0a2c70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f247c007b60 tx=0x7f247c0073d0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58026af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58030900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58033230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57f7f730 con 0x560f58030e40 2023-09-10T01:18:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57f7d050 con 0x560f58027030 2023-09-10T01:18:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57f78940 con 0x560f58026710 2023-09-10T01:18:12.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.004+0000 7f2484f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58030900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f248579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58026af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58030900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47860/0 (socket says 172.21.15.3:47860) 2023-09-10T01:18:12.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 -- 172.21.15.3:0/2822528046 learned_addr learned my addr 172.21.15.3:0/2822528046 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2485f9c700 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58033230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 -- 172.21.15.3:0/2822528046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 msgr2=0x560f58026af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58026af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 -- 172.21.15.3:0/2822528046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 msgr2=0x560f58033230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58033230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 -- 172.21.15.3:0/2822528046 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f57da4de0 con 0x560f58027030 2023-09-10T01:18:12.014 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f2484f9a700 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58030900 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2470096e40 tx=0x7f2470099d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaba43f7aaa30ef3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f246b7fe700 1 -- 172.21.15.3:0/2822528046 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24700cd120 con 0x560f58027030 2023-09-10T01:18:12.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f246b7fe700 1 -- 172.21.15.3:0/2822528046 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f247009a3d0 con 0x560f58027030 2023-09-10T01:18:12.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.005+0000 7f246b7fe700 1 -- 172.21.15.3:0/2822528046 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24700a00c0 con 0x560f58027030 2023-09-10T01:18:12.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2822528046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 msgr2=0x560f58030900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58030900 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2470096e40 tx=0x7f2470099d30 comp rx=0 tx=0).stop 2023-09-10T01:18:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2822528046 shutdown_connections 2023-09-10T01:18:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58026af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58033230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2822528046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58030900 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2822528046 >> 172.21.15.3:0/2822528046 conn(0x560f57f9b090 msgr2=0x560f58021bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2822528046 shutdown_connections 2023-09-10T01:18:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.006+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2822528046 wait complete. 2023-09-10T01:18:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.007+0000 7f249b3b7b80 1 -- start start 2023-09-10T01:18:12.017 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.007+0000 7f249b3b7b80 1 Processor -- start 2023-09-10T01:18:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58031220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58031760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f5803c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57f16b70 con 0x560f58027030 2023-09-10T01:18:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57f16870 con 0x560f58030e40 2023-09-10T01:18:12.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57dfa920 con 0x560f58026710 2023-09-10T01:18:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f249b3b7b80 10 client.0 _ll_register_callbacks cb 0x560f57e60a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f2484f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58031760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f2485f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f5803c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f2485f9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f5803c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47874/0 (socket says 172.21.15.3:47874) 2023-09-10T01:18:12.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f2485f9c700 1 -- 172.21.15.3:0/2295049347 learned_addr learned my addr 172.21.15.3:0/2295049347 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.008+0000 7f248579b700 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58031220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2485f9c700 1 -- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 msgr2=0x560f58031220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2485f9c700 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58031220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.020 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2485f9c700 1 -- 172.21.15.3:0/2295049347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 msgr2=0x560f58031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2485f9c700 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58031760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2485f9c700 1 -- 172.21.15.3:0/2295049347 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24801a5690 con 0x560f58030e40 2023-09-10T01:18:12.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2485f9c700 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f5803c2a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2478007b40 tx=0x7f2478007e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a8ad306161f6b63 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2468ff9700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:18:12.021 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2468ff9700 1 -- 172.21.15.3:0/2295049347 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478012b40 con 0x560f58030e40 2023-09-10T01:18:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f57ed04f0 con 0x560f58030e40 2023-09-10T01:18:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2468ff9700 1 -- 172.21.15.3:0/2295049347 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f2478012ca0 con 0x560f58030e40 2023-09-10T01:18:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.009+0000 7f2468ff9700 1 -- 172.21.15.3:0/2295049347 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2478012e60 con 0x560f58030e40 2023-09-10T01:18:12.022 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 10 client.29033 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x560f57d8dc60 con 0x560f58030e40 2023-09-10T01:18:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f2468ff9700 1 -- 172.21.15.3:0/2295049347 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560f57ed04f0 con 0x560f58030e40 2023-09-10T01:18:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f2468ff9700 1 -- 172.21.15.3:0/2295049347 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f2478014040 con 0x560f58030e40 2023-09-10T01:18:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 10 client.29033 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:12.023 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 10 client.29033 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:18:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 20 client.29033 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58040680 0x560f58042aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 4 client.29033 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:12.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.010+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f57f8d8a0 con 0x560f58040680 2023-09-10T01:18:12.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.011+0000 7f248579b700 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58040680 0x560f58042aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.011+0000 7f248579b700 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58040680 0x560f58042aa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a0a83c9607a4f758 server_cookie=c1ceb256cb0527ca in_seq=0 out_seq=0 2023-09-10T01:18:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.011+0000 7f2468ff9700 10 client.29033 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f2468ff9700 1 -- 172.21.15.3:0/2295049347 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560f57f8d8a0 con 0x560f58040680 2023-09-10T01:18:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f2468ff9700 10 client.29033 handle_command_reply: tid=0 2023-09-10T01:18:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 client.29033 shutdown 2023-09-10T01:18:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 msgr2=0x560f5803c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f5803c2a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2478007b40 tx=0x7f2478007e50 comp rx=0 tx=0).stop 2023-09-10T01:18:12.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 shutdown_connections 2023-09-10T01:18:12.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58040680 0x560f58042aa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58031220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f5803c2a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.028 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f2468ff9700 0 client.29033 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2295049347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58031760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 >> 172.21.15.3:0/2295049347 conn(0x560f57f9b090 msgr2=0x560f580217e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 shutdown_connections 2023-09-10T01:18:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.012+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2295049347 wait complete. 2023-09-10T01:18:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.013+0000 7f249b3b7b80 20 client.29033 trim_cache size 0 max 16384 2023-09-10T01:18:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.026+0000 7f249b3b7b80 1 Processor -- start 2023-09-10T01:18:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.026+0000 7f249b3b7b80 1 -- start start 2023-09-10T01:18:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58036250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58031220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58031760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f58036790 con 0x560f58030e40 2023-09-10T01:18:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f58027740 con 0x560f58027030 2023-09-10T01:18:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f58030200 con 0x560f58026710 2023-09-10T01:18:12.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f248579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58036250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f248579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58036250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42894/0 (socket says 172.21.15.3:42894) 2023-09-10T01:18:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f248579b700 1 -- 172.21.15.3:0/3709759172 learned_addr learned my addr 172.21.15.3:0/3709759172 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f2484f9a700 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58031220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f2485f9c700 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58031760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.027+0000 7f248579b700 1 -- 172.21.15.3:0/3709759172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 msgr2=0x560f58031220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f248579b700 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58031220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f248579b700 1 -- 172.21.15.3:0/3709759172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 msgr2=0x560f58031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f248579b700 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58031760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f248579b700 1 -- 172.21.15.3:0/3709759172 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f57dcb250 con 0x560f58026710 2023-09-10T01:18:12.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f248579b700 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58036250 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x560f580433b0 tx=0x7f247c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efe00d03fd709758 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f246bfff700 1 -- 172.21.15.3:0/3709759172 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c024040 con 0x560f58026710 2023-09-10T01:18:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f246bfff700 1 -- 172.21.15.3:0/3709759172 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f247c015070 con 0x560f58026710 2023-09-10T01:18:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.028+0000 7f246bfff700 1 -- 172.21.15.3:0/3709759172 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c01f690 con 0x560f58026710 2023-09-10T01:18:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 -- 172.21.15.3:0/3709759172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 msgr2=0x560f58036250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.038 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58036250 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x560f580433b0 tx=0x7f247c013000 comp rx=0 tx=0).stop 2023-09-10T01:18:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 -- 172.21.15.3:0/3709759172 shutdown_connections 2023-09-10T01:18:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58036250 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58030e40 0x560f58031760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/3709759172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58027030 0x560f58031220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 -- 172.21.15.3:0/3709759172 >> 172.21.15.3:0/3709759172 conn(0x560f57f9b090 msgr2=0x560f5802b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 -- 172.21.15.3:0/3709759172 shutdown_connections 2023-09-10T01:18:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.029+0000 7f249b3b7b80 1 -- 172.21.15.3:0/3709759172 wait complete. 2023-09-10T01:18:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.030+0000 7f249b3b7b80 1 -- start start 2023-09-10T01:18:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.030+0000 7f249b3b7b80 1 Processor -- start 2023-09-10T01:18:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58035ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58032300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f249b3b7b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f58032840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f58032f50 con 0x560f58027030 2023-09-10T01:18:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f580330c0 con 0x560f58030e40 2023-09-10T01:18:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f249b3b7b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f580944b0 con 0x560f58026710 2023-09-10T01:18:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f249b3b7b80 10 client.0 _ll_register_callbacks cb 0x7f2478009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58032300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f248579b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58035ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58032300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55662/0 (socket says 172.21.15.3:55662) 2023-09-10T01:18:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 -- 172.21.15.3:0/2390967814 learned_addr learned my addr 172.21.15.3:0/2390967814 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2485f9c700 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f58032840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 -- 172.21.15.3:0/2390967814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 msgr2=0x560f58035ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58035ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 -- 172.21.15.3:0/2390967814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 msgr2=0x560f58032840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f58032840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.031+0000 7f2484f9a700 1 -- 172.21.15.3:0/2390967814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c010b60 con 0x560f58027030 2023-09-10T01:18:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f2484f9a700 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58032300 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f247009a0d0 tx=0x7f247009ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3fbd21f436d778 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f2468ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f2468ff9700 1 -- 172.21.15.3:0/2390967814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247009a830 con 0x560f58027030 2023-09-10T01:18:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f2468ff9700 1 -- 172.21.15.3:0/2390967814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f24700a23b0 con 0x560f58027030 2023-09-10T01:18:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f247009a3d0 con 0x560f58027030 2023-09-10T01:18:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f2468ff9700 1 -- 172.21.15.3:0/2390967814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24700850b0 con 0x560f58027030 2023-09-10T01:18:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f249b3b7b80 10 client.18921 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.032+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x560f57da32b0 con 0x560f58027030 2023-09-10T01:18:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f2468ff9700 1 -- 172.21.15.3:0/2390967814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2470066080 con 0x560f58027030 2023-09-10T01:18:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f2468ff9700 1 -- 172.21.15.3:0/2390967814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f2470099950 con 0x560f58027030 2023-09-10T01:18:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f249b3b7b80 10 client.18921 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f249b3b7b80 10 client.18921 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:18:12.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f249b3b7b80 20 client.18921 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58095660 0x560f58097a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f249b3b7b80 4 client.18921 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:18:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560f57f71920 con 0x560f58095660 2023-09-10T01:18:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.033+0000 7f248579b700 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58095660 0x560f58097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f248579b700 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58095660 0x560f58097a80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de19873ac397bc5d server_cookie=c9e44a882e2482e4 in_seq=0 out_seq=0 2023-09-10T01:18:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f2468ff9700 10 client.18921 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f2468ff9700 1 -- 172.21.15.3:0/2390967814 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560f57f71920 con 0x560f58095660 2023-09-10T01:18:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f2468ff9700 10 client.18921 handle_command_reply: tid=0 2023-09-10T01:18:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 client.18921 shutdown 2023-09-10T01:18:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 msgr2=0x560f58032300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58032300 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f247009a0d0 tx=0x7f247009ac20 comp rx=0 tx=0).stop 2023-09-10T01:18:12.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 shutdown_connections 2023-09-10T01:18:12.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x560f58095660 0x560f58097a80 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560f58026710 0x560f58035ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.050 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f58030e40 0x560f58032840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 --2- 172.21.15.3:0/2390967814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560f58027030 0x560f58032300 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.034+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 >> 172.21.15.3:0/2390967814 conn(0x560f57f9b090 msgr2=0x560f5802e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f2468ff9700 0 client.18921 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 shutdown_connections 2023-09-10T01:18:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f249b3b7b80 1 -- 172.21.15.3:0/2390967814 wait complete. 2023-09-10T01:18:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f249b3b7b80 20 client.18921 trim_cache size 0 max 16384 2023-09-10T01:18:12.052 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:18:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f246c0a07b0 msgr2=0x7f246c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f24879ff700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f246c0a07b0 0x7f246c0a2c70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f247c007b60 tx=0x7f247c0073d0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480101000 msgr2=0x7f248019ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.035+0000 7f24879ff700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480101000 0x7f248019ebe0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f24700095c0 tx=0x7f247000f690 comp rx=0 tx=0).stop 2023-09-10T01:18:12.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 shutdown_connections 2023-09-10T01:18:12.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f246c0a07b0 0x7f246c0a2c70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2480106650 0x7f2480198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.054 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2480100640 0x7f248019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 --2- 172.21.15.3:0/882550743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2480101000 0x7f248019ebe0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 >> 172.21.15.3:0/882550743 conn(0x7f24800fc170 msgr2=0x7f24800fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 shutdown_connections 2023-09-10T01:18:12.055 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.036+0000 7f24879ff700 1 -- 172.21.15.3:0/882550743 wait complete. 2023-09-10T01:18:12.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:18:12.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 -- 172.21.15.3:0/3200440070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e801021c0 msgr2=0x7f7e8010a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 --2- 172.21.15.3:0/3200440070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e801021c0 0x7f7e8010a6a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009990 tx=0x7f7e70009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 -- 172.21.15.3:0/3200440070 shutdown_connections 2023-09-10T01:18:12.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 --2- 172.21.15.3:0/3200440070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e8010ac70 0x7f7e8010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 --2- 172.21.15.3:0/3200440070 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e801018a0 0x7f7e80101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 --2- 172.21.15.3:0/3200440070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e801021c0 0x7f7e8010a6a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 -- 172.21.15.3:0/3200440070 >> 172.21.15.3:0/3200440070 conn(0x7f7e80076710 msgr2=0x7f7e80076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 -- 172.21.15.3:0/3200440070 shutdown_connections 2023-09-10T01:18:12.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.274+0000 7f7e871ff700 1 -- 172.21.15.3:0/3200440070 wait complete. 2023-09-10T01:18:12.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.275+0000 7f7e871ff700 1 Processor -- start 2023-09-10T01:18:12.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e871ff700 1 -- start start 2023-09-10T01:18:12.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e871ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e801018a0 0x7f7e801989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e871ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e801021c0 0x7f7e80198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e871ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8010ac70 0x7f7e8019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e871ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8019d940 con 0x7f7e8010ac70 2023-09-10T01:18:12.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e871ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8019dab0 con 0x7f7e801021c0 2023-09-10T01:18:12.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e871ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8019dc70 con 0x7f7e801018a0 2023-09-10T01:18:12.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e84f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e801018a0 0x7f7e801989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e8579c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8010ac70 0x7f7e8019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e84f9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e801018a0 0x7f7e801989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42924/0 (socket says 172.21.15.3:42924) 2023-09-10T01:18:12.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.276+0000 7f7e84f9b700 1 -- 172.21.15.3:0/860080210 learned_addr learned my addr 172.21.15.3:0/860080210 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e77fff700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e801021c0 0x7f7e80198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e84f9b700 1 -- 172.21.15.3:0/860080210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e801021c0 msgr2=0x7f7e80198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e801021c0 0x7f7e80198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e84f9b700 1 -- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8010ac70 msgr2=0x7f7e8019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8010ac70 0x7f7e8019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e84f9b700 1 -- 172.21.15.3:0/860080210 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c009520 con 0x7f7e801018a0 2023-09-10T01:18:12.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e8579c700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8010ac70 0x7f7e8019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:12.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e801018a0 0x7f7e801989a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00eb30 tx=0x7f7e7c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e75ffb700 1 -- 172.21.15.3:0/860080210 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c009880 con 0x7f7e801018a0 2023-09-10T01:18:12.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.277+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e700095f0 con 0x7f7e801018a0 2023-09-10T01:18:12.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.278+0000 7f7e75ffb700 1 -- 172.21.15.3:0/860080210 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e7c0044e0 con 0x7f7e801018a0 2023-09-10T01:18:12.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.278+0000 7f7e75ffb700 1 -- 172.21.15.3:0/860080210 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c003d80 con 0x7f7e801018a0 2023-09-10T01:18:12.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.278+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e801a5620 con 0x7f7e801018a0 2023-09-10T01:18:12.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.281+0000 7f7e75ffb700 1 -- 172.21.15.3:0/860080210 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7e7c003f20 con 0x7f7e801018a0 2023-09-10T01:18:12.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.282+0000 7f7e9aa30b80 1 Processor -- start 2023-09-10T01:18:12.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e9aa30b80 1 -- start start 2023-09-10T01:18:12.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e75ffb700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7e680a07b0 0x7f7e680a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e75ffb700 1 -- 172.21.15.3:0/860080210 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f7e7c014070 con 0x7f7e801018a0 2023-09-10T01:18:12.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2dae4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e8579c700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7e680a07b0 0x7f7e680a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2da3d730 con 0x563a2daeee40 2023-09-10T01:18:12.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2da3b050 con 0x563a2dae4710 2023-09-10T01:18:12.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.283+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2da36940 con 0x563a2dae5030 2023-09-10T01:18:12.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2dae4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2dae4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47930/0 (socket says 172.21.15.3:47930) 2023-09-10T01:18:12.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1975571485 learned_addr learned my addr 172.21.15.3:0/1975571485 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e77fff700 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e8579c700 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1975571485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 msgr2=0x563a2daee900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1975571485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 msgr2=0x563a2daf1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1975571485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a2d862de0 con 0x563a2dae4710 2023-09-10T01:18:12.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e77fff700 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:18:12.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2dae4af0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c014040 tx=0x7f7e7c099380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f591aee088aeac9 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e8579c700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7e680a07b0 0x7f7e680a2c70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7e78004230 tx=0x7f7e780045f0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e66ffd700 1 -- 172.21.15.3:0/1975571485 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0b9070 con 0x563a2dae4710 2023-09-10T01:18:12.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e66ffd700 1 -- 172.21.15.3:0/1975571485 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e7c0999a0 con 0x563a2dae4710 2023-09-10T01:18:12.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.284+0000 7f7e66ffd700 1 -- 172.21.15.3:0/1975571485 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c099c00 con 0x563a2dae4710 2023-09-10T01:18:12.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.285+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1975571485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 msgr2=0x563a2dae4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.285+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2dae4af0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c014040 tx=0x7f7e7c099380 comp rx=0 tx=0).stop 2023-09-10T01:18:12.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.286+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1975571485 shutdown_connections 2023-09-10T01:18:12.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.286+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.286+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2dae4af0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.286+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1975571485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.286+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1975571485 >> 172.21.15.3:0/1975571485 conn(0x563a2da59090 msgr2=0x563a2dadfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.286+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1975571485 shutdown_connections 2023-09-10T01:18:12.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.286+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1975571485 wait complete. 2023-09-10T01:18:12.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.287+0000 7f7e9aa30b80 1 -- start start 2023-09-10T01:18:12.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.287+0000 7f7e9aa30b80 1 Processor -- start 2023-09-10T01:18:12.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2dae4710 0x563a2daee890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2daeee40 0x563a2daec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2d9d4b70 con 0x563a2dae4710 2023-09-10T01:18:12.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2d9d4870 con 0x563a2dae5030 2023-09-10T01:18:12.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2d8b8920 con 0x563a2daeee40 2023-09-10T01:18:12.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e9aa30b80 10 client.0 _ll_register_callbacks cb 0x563a2d91ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:12.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e84f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2dae4710 0x563a2daee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e8579c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2daeee40 0x563a2daec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e84f9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2dae4710 0x563a2daee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55712/0 (socket says 172.21.15.3:55712) 2023-09-10T01:18:12.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1856007299 learned_addr learned my addr 172.21.15.3:0/1856007299 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e77fff700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.288+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1856007299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2daeee40 msgr2=0x563a2daec1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2daeee40 0x563a2daec1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 msgr2=0x563a2daebaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daebaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e84f9b700 1 -- 172.21.15.3:0/1856007299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e801a5620 con 0x563a2dae4710 2023-09-10T01:18:12.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e77fff700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daebaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:12.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e8579c700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2daeee40 0x563a2daec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:12.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2dae4710 0x563a2daee890 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0996a0 tx=0x7f7e7c01fff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518d292ecee24ed6 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e5ffff700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:12.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e5ffff700 1 -- 172.21.15.3:0/1856007299 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c09d7d0 con 0x563a2dae4710 2023-09-10T01:18:12.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563a2d98e4f0 con 0x563a2dae4710 2023-09-10T01:18:12.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e5ffff700 1 -- 172.21.15.3:0/1856007299 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f7e7c00cdd0 con 0x563a2dae4710 2023-09-10T01:18:12.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e5ffff700 1 -- 172.21.15.3:0/1856007299 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e7c098e90 con 0x563a2dae4710 2023-09-10T01:18:12.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e9aa30b80 10 client.18933 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:12.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.289+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x563a2d84bc60 con 0x563a2dae4710 2023-09-10T01:18:12.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e5ffff700 1 -- 172.21.15.3:0/1856007299 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e7c07c020 con 0x563a2dae4710 2023-09-10T01:18:12.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e5ffff700 1 -- 172.21.15.3:0/1856007299 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7e7c0a1580 con 0x563a2dae4710 2023-09-10T01:18:12.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e9aa30b80 10 client.18933 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:12.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e9aa30b80 10 client.18933 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:18:12.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e9aa30b80 20 client.18933 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:12.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2dafe430 0x563a2db00850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e9aa30b80 4 client.18933 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:12.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.290+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a2da4b8a0 con 0x563a2dafe430 2023-09-10T01:18:12.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.291+0000 7f7e77fff700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2dafe430 0x563a2db00850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.291+0000 7f7e77fff700 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2dafe430 0x563a2db00850 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a29c1dcae46dfd8 server_cookie=ed3b33b5de96da16 in_seq=0 out_seq=0 2023-09-10T01:18:12.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.291+0000 7f7e5ffff700 10 client.18933 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:12.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e5ffff700 1 -- 172.21.15.3:0/1856007299 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563a2da4b8a0 con 0x563a2dafe430 2023-09-10T01:18:12.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e5ffff700 10 client.18933 handle_command_reply: tid=0 2023-09-10T01:18:12.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 client.18933 shutdown 2023-09-10T01:18:12.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2dae4710 msgr2=0x563a2daee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2dae4710 0x563a2daee890 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0996a0 tx=0x7f7e7c01fff0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 shutdown_connections 2023-09-10T01:18:12.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2dafe430 0x563a2db00850 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2daeee40 0x563a2daec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2dae4710 0x563a2daee890 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/1856007299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daebaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e5ffff700 0 client.18933 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:12.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 >> 172.21.15.3:0/1856007299 conn(0x563a2da59090 msgr2=0x563a2dadf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 shutdown_connections 2023-09-10T01:18:12.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.292+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/1856007299 wait complete. 2023-09-10T01:18:12.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.293+0000 7f7e9aa30b80 20 client.18933 trim_cache size 0 max 16384 2023-09-10T01:18:12.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 Processor -- start 2023-09-10T01:18:12.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 -- start start 2023-09-10T01:18:12.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2daf59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2dadf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2daf5f60 con 0x563a2daeee40 2023-09-10T01:18:12.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2dae5990 con 0x563a2dae4710 2023-09-10T01:18:12.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.307+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2dae06e0 con 0x563a2dae5030 2023-09-10T01:18:12.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:12 smithi003 ceph-mon[97654]: pgmap v470: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 800 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 628 op/s 2023-09-10T01:18:12.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:12 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1254825442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:18:12.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e8579c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2dadf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.317 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e84f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2daf59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42952/0 (socket says 172.21.15.3:42952) 2023-09-10T01:18:12.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 -- 172.21.15.3:0/673329299 learned_addr learned my addr 172.21.15.3:0/673329299 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 -- 172.21.15.3:0/673329299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 msgr2=0x563a2daf59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.318 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 --2- 172.21.15.3:0/673329299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2daf59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 -- 172.21.15.3:0/673329299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 msgr2=0x563a2dadf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 --2- 172.21.15.3:0/673329299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2dadf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.308+0000 7f7e77fff700 1 -- 172.21.15.3:0/673329299 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a2d889250 con 0x563a2dae5030 2023-09-10T01:18:12.319 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.309+0000 7f7e77fff700 1 --2- 172.21.15.3:0/673329299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee3c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009990 tx=0x7f7e70005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fa6c134fb136108 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.309+0000 7f7e677fe700 1 -- 172.21.15.3:0/673329299 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7001d040 con 0x563a2dae5030 2023-09-10T01:18:12.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.309+0000 7f7e677fe700 1 -- 172.21.15.3:0/673329299 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e7001e040 con 0x563a2dae5030 2023-09-10T01:18:12.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.309+0000 7f7e677fe700 1 -- 172.21.15.3:0/673329299 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e700052b0 con 0x563a2dae5030 2023-09-10T01:18:12.320 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/673329299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 msgr2=0x563a2daee3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/673329299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee3c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7e70009990 tx=0x7f7e70005fb0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/673329299 shutdown_connections 2023-09-10T01:18:12.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/673329299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2dadf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.321 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/673329299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae4710 0x563a2daf59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/673329299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae5030 0x563a2daee3c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/673329299 >> 172.21.15.3:0/673329299 conn(0x563a2da59090 msgr2=0x563a2dafa140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.322 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/673329299 shutdown_connections 2023-09-10T01:18:12.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/673329299 wait complete. 2023-09-10T01:18:12.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.310+0000 7f7e9aa30b80 1 -- start start 2023-09-10T01:18:12.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.311+0000 7f7e9aa30b80 1 Processor -- start 2023-09-10T01:18:12.323 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae4710 0x563a2daf5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daf3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e9aa30b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2daf4bd0 con 0x563a2daeee40 2023-09-10T01:18:12.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2db52230 con 0x563a2dae5030 2023-09-10T01:18:12.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e9aa30b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2db52370 con 0x563a2dae4710 2023-09-10T01:18:12.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e9aa30b80 10 client.0 _ll_register_callbacks cb 0x7f7e7c0994e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:12.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daf3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daf3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:47970/0 (socket says 172.21.15.3:47970) 2023-09-10T01:18:12.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 -- 172.21.15.3:0/2916707234 learned_addr learned my addr 172.21.15.3:0/2916707234 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e8579c700 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 -- 172.21.15.3:0/2916707234 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae4710 msgr2=0x563a2daf5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae4710 0x563a2daf5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 -- 172.21.15.3:0/2916707234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 msgr2=0x563a2daf44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.312+0000 7f7e77fff700 1 -- 172.21.15.3:0/2916707234 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8010ddb0 con 0x563a2dae5030 2023-09-10T01:18:12.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e77fff700 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daf3e20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7e700098e0 tx=0x7f7e700281b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0b6287a4b7774d server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e5ffff700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:18:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e5ffff700 1 -- 172.21.15.3:0/2916707234 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7001d040 con 0x563a2dae5030 2023-09-10T01:18:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e7c0999a0 con 0x563a2dae5030 2023-09-10T01:18:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e5ffff700 1 -- 172.21.15.3:0/2916707234 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f7e70028c80 con 0x563a2dae5030 2023-09-10T01:18:12.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e5ffff700 1 -- 172.21.15.3:0/2916707234 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e7000bb60 con 0x563a2dae5030 2023-09-10T01:18:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e9aa30b80 10 client.29057 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.313+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x563a2d8612b0 con 0x563a2dae5030 2023-09-10T01:18:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e5ffff700 1 -- 172.21.15.3:0/2916707234 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e7c0999a0 con 0x563a2dae5030 2023-09-10T01:18:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e5ffff700 1 -- 172.21.15.3:0/2916707234 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7e70004730 con 0x563a2dae5030 2023-09-10T01:18:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e9aa30b80 10 client.29057 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:12.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e9aa30b80 10 client.29057 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:18:12.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e9aa30b80 20 client.29057 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:12.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2db536f0 0x563a2db55b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e9aa30b80 4 client.29057 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:18:12.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563a2da2f920 con 0x563a2db536f0 2023-09-10T01:18:12.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2db536f0 0x563a2db55b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e84f9b700 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2db536f0 0x563a2db55b10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=905d2747e6099270 server_cookie=5c3a25b86a1db608 in_seq=0 out_seq=0 2023-09-10T01:18:12.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.314+0000 7f7e5ffff700 10 client.29057 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:12.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e5ffff700 1 -- 172.21.15.3:0/2916707234 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563a2da2f920 con 0x563a2db536f0 2023-09-10T01:18:12.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e5ffff700 10 client.29057 handle_command_reply: tid=0 2023-09-10T01:18:12.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 client.29057 shutdown 2023-09-10T01:18:12.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 msgr2=0x563a2daf3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daf3e20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7e700098e0 tx=0x7f7e700281b0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 shutdown_connections 2023-09-10T01:18:12.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x563a2db536f0 0x563a2db55b10 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a2dae4710 0x563a2daf5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e5ffff700 0 client.29057 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:12.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563a2daeee40 0x563a2daf44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 --2- 172.21.15.3:0/2916707234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a2dae5030 0x563a2daf3e20 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 >> 172.21.15.3:0/2916707234 conn(0x563a2da59090 msgr2=0x563a2dadf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 shutdown_connections 2023-09-10T01:18:12.332 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 1 -- 172.21.15.3:0/2916707234 wait complete. 2023-09-10T01:18:12.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.315+0000 7f7e9aa30b80 20 client.29057 trim_cache size 0 max 16384 2023-09-10T01:18:12.333 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:18:12.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7e680a07b0 msgr2=0x7f7e680a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7e680a07b0 0x7f7e680a2c70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7e78004230 tx=0x7f7e780045f0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e801018a0 msgr2=0x7f7e801989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e801018a0 0x7f7e801989a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00eb30 tx=0x7f7e7c00ee40 comp rx=0 tx=0).stop 2023-09-10T01:18:12.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 shutdown_connections 2023-09-10T01:18:12.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7e680a07b0 0x7f7e680a2c70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e801018a0 0x7f7e801989a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8010ac70 0x7f7e8019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 --2- 172.21.15.3:0/860080210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e801021c0 0x7f7e80198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 >> 172.21.15.3:0/860080210 conn(0x7f7e80076710 msgr2=0x7f7e800fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 shutdown_connections 2023-09-10T01:18:12.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:12.316+0000 7f7e871ff700 1 -- 172.21.15.3:0/860080210 wait complete. 2023-09-10T01:18:12.337 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9670cca5bc8e17916865dc84939edfc10b1f2c00 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-09-10T01:18:12.679 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f1e42f9c98ab4631ab1218a59bf882dc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_9670cca5bc8e17916865dc84939edfc10b1f2c00/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9670cca5bc8e17916865dc84939edfc10b1f2c00 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-09-10T01:18:12.684 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-09-10T01:18:12.695 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:18:12.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:18:12.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:12 smithi136 ceph-mon[133708]: pgmap v470: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 800 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 628 op/s 2023-09-10T01:18:12.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:12 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1254825442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:18:12.727 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:12 smithi092 ceph-mon[134067]: pgmap v470: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 800 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 628 op/s 2023-09-10T01:18:12.727 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:12 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1254825442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:18:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 -- 172.21.15.3:0/3396848121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c0072a30 msgr2=0x7f16c0072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 --2- 172.21.15.3:0/3396848121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c0072a30 0x7f16c0072e10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009990 tx=0x7f16b0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 -- 172.21.15.3:0/3396848121 shutdown_connections 2023-09-10T01:18:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 --2- 172.21.15.3:0/3396848121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16c010cde0 0x7f16c010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 --2- 172.21.15.3:0/3396848121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c0072a30 0x7f16c0072e10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 --2- 172.21.15.3:0/3396848121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c0073350 0x7f16c010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 -- 172.21.15.3:0/3396848121 >> 172.21.15.3:0/3396848121 conn(0x7f16c00fc0b0 msgr2=0x7f16c00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 -- 172.21.15.3:0/3396848121 shutdown_connections 2023-09-10T01:18:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.492+0000 7f16c65ff700 1 -- 172.21.15.3:0/3396848121 wait complete. 2023-09-10T01:18:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.493+0000 7f16c65ff700 1 Processor -- start 2023-09-10T01:18:12.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.493+0000 7f16c65ff700 1 -- start start 2023-09-10T01:18:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16c65ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c0072a30 0x7f16c019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16c65ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16c0073350 0x7f16c019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16c65ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c010cde0 0x7f16c01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16c65ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0198d60 con 0x7f16c0072a30 2023-09-10T01:18:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16c65ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0198ed0 con 0x7f16c010cde0 2023-09-10T01:18:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16c65ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0199090 con 0x7f16c0073350 2023-09-10T01:18:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16c0073350 0x7f16c019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16c0073350 0x7f16c019ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:42980/0 (socket says 172.21.15.3:42980) 2023-09-10T01:18:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 -- 172.21.15.3:0/4017846643 learned_addr learned my addr 172.21.15.3:0/4017846643 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bffff700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c0072a30 0x7f16c019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16c4b9c700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c010cde0 0x7f16c01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 -- 172.21.15.3:0/4017846643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c010cde0 msgr2=0x7f16c01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c010cde0 0x7f16c01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 -- 172.21.15.3:0/4017846643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c0072a30 msgr2=0x7f16c019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c0072a30 0x7f16c019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bf7fe700 1 -- 172.21.15.3:0/4017846643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b00095f0 con 0x7f16c0073350 2023-09-10T01:18:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.494+0000 7f16bffff700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c0072a30 0x7f16c019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.495+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16c0073350 0x7f16c019ebc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f16b400e9a0 tx=0x7f16b400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.495+0000 7f16bd7fa700 1 -- 172.21.15.3:0/4017846643 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4009790 con 0x7f16c0073350 2023-09-10T01:18:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.495+0000 7f16bd7fa700 1 -- 172.21.15.3:0/4017846643 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16b400cbd0 con 0x7f16c0073350 2023-09-10T01:18:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.495+0000 7f16c65ff700 1 -- 172.21.15.3:0/4017846643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c0199370 con 0x7f16c0073350 2023-09-10T01:18:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.495+0000 7f16bd7fa700 1 -- 172.21.15.3:0/4017846643 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4018680 con 0x7f16c0073350 2023-09-10T01:18:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.495+0000 7f16c65ff700 1 -- 172.21.15.3:0/4017846643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c01a5670 con 0x7f16c0073350 2023-09-10T01:18:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.499+0000 7f16bd7fa700 1 -- 172.21.15.3:0/4017846643 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f16b400cd40 con 0x7f16c0073350 2023-09-10T01:18:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.500+0000 7f16d9ffdb80 1 Processor -- start 2023-09-10T01:18:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16d9ffdb80 1 -- start start 2023-09-10T01:18:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bd3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bcd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bd9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058b266d0 con 0x565058bd31a0 2023-09-10T01:18:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058b23ff0 con 0x565058bcd300 2023-09-10T01:18:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058b1f8e0 con 0x565058bcdcc0 2023-09-10T01:18:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16c4b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bd3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16c4b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bd3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55742/0 (socket says 172.21.15.3:55742) 2023-09-10T01:18:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16c4b9c700 1 -- 172.21.15.3:0/1344973684 learned_addr learned my addr 172.21.15.3:0/1344973684 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16bffff700 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bcd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.501+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bd9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bd7fa700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f16a80a07b0 0x7f16a80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bffff700 1 -- 172.21.15.3:0/1344973684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 msgr2=0x565058bd9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bffff700 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bd9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bffff700 1 -- 172.21.15.3:0/1344973684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 msgr2=0x565058bd3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bffff700 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bd3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bffff700 1 -- 172.21.15.3:0/1344973684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56505894bd80 con 0x565058bcd300 2023-09-10T01:18:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bd7fa700 1 -- 172.21.15.3:0/4017846643 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f16b4014070 con 0x7f16c0073350 2023-09-10T01:18:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16c4b9c700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f16a80a07b0 0x7f16a80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.502+0000 7f16bffff700 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bcd780 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f16b00059d0 tx=0x7f16b0005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=651d15679c5dd661 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.503+0000 7f16a67fc700 1 -- 172.21.15.3:0/1344973684 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b001d070 con 0x565058bcd300 2023-09-10T01:18:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.503+0000 7f16c4b9c700 1 --2- 172.21.15.3:0/4017846643 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f16a80a07b0 0x7f16a80a2c70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009e70 tx=0x7f16b8004ad0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.503+0000 7f16a67fc700 1 -- 172.21.15.3:0/1344973684 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16b001e040 con 0x565058bcd300 2023-09-10T01:18:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.503+0000 7f16a67fc700 1 -- 172.21.15.3:0/1344973684 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0004e90 con 0x565058bcd300 2023-09-10T01:18:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.503+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/1344973684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 msgr2=0x565058bcd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.503+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bcd780 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f16b00059d0 tx=0x7f16b0005bc0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/1344973684 shutdown_connections 2023-09-10T01:18:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bd9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bd3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/1344973684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bcd780 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/1344973684 >> 172.21.15.3:0/1344973684 conn(0x565058b42030 msgr2=0x565058bc8560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/1344973684 shutdown_connections 2023-09-10T01:18:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/1344973684 wait complete. 2023-09-10T01:18:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.504+0000 7f16d9ffdb80 1 -- start start 2023-09-10T01:18:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.505+0000 7f16d9ffdb80 1 Processor -- start 2023-09-10T01:18:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.505+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bd9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.505+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcdcc0 0x565058bd7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.505+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bd31a0 0x565058bd7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.505+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058abdb10 con 0x565058bcdcc0 2023-09-10T01:18:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.505+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058abd810 con 0x565058bcd300 2023-09-10T01:18:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.505+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650589a18c0 con 0x565058bd31a0 2023-09-10T01:18:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16d9ffdb80 10 client.0 _ll_register_callbacks cb 0x565058a07a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bf7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcdcc0 0x565058bd7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bd9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16c4b9c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bd31a0 0x565058bd7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bd9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48012/0 (socket says 172.21.15.3:48012) 2023-09-10T01:18:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 -- 172.21.15.3:0/567335755 learned_addr learned my addr 172.21.15.3:0/567335755 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 -- 172.21.15.3:0/567335755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bd31a0 msgr2=0x565058bd7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bd31a0 0x565058bd7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 -- 172.21.15.3:0/567335755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcdcc0 msgr2=0x565058bd7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcdcc0 0x565058bd7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 -- 172.21.15.3:0/567335755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c010ff10 con 0x565058bcd300 2023-09-10T01:18:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcdcc0 0x565058bd7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f16bffff700 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bd9c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f16b0000c00 tx=0x7f16b00226b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b194c5aa762a2621 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.506+0000 7f169f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:18:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f169f7fe700 1 -- 172.21.15.3:0/567335755 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b001d030 con 0x565058bcd300 2023-09-10T01:18:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565058a77490 con 0x565058bcd300 2023-09-10T01:18:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f169f7fe700 1 -- 172.21.15.3:0/567335755 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f16b0022d60 con 0x565058bcd300 2023-09-10T01:18:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f169f7fe700 1 -- 172.21.15.3:0/567335755 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16b000f460 con 0x565058bcd300 2023-09-10T01:18:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f16d9ffdb80 10 client.29069 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:12.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x565058934c00 con 0x565058bcd300 2023-09-10T01:18:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f169f7fe700 1 -- 172.21.15.3:0/567335755 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565058a77490 con 0x565058bcd300 2023-09-10T01:18:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.507+0000 7f169f7fe700 1 -- 172.21.15.3:0/567335755 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f16b002c4b0 con 0x565058bcd300 2023-09-10T01:18:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16d9ffdb80 10 client.29069 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16d9ffdb80 10 client.29069 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16d9ffdb80 20 client.29069 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x565058be7070 0x565058be9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16d9ffdb80 4 client.29069 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565058b34840 con 0x565058be7070 2023-09-10T01:18:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x565058be7070 0x565058be9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x565058be7070 0x565058be9490 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3514e18f87765bba server_cookie=3843a5546128c4f0 in_seq=0 out_seq=0 2023-09-10T01:18:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.508+0000 7f169f7fe700 10 client.29069 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.509+0000 7f169f7fe700 1 -- 172.21.15.3:0/567335755 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x565058b34840 con 0x565058be7070 2023-09-10T01:18:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.509+0000 7f169f7fe700 10 client.29069 handle_command_reply: tid=0 2023-09-10T01:18:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 client.29069 shutdown 2023-09-10T01:18:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 msgr2=0x565058bd9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bd9c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f16b0000c00 tx=0x7f16b00226b0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 shutdown_connections 2023-09-10T01:18:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x565058be7070 0x565058be9490 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bd31a0 0x565058bd7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f169f7fe700 0 client.29069 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcd300 0x565058bd9c80 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/567335755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcdcc0 0x565058bd7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 >> 172.21.15.3:0/567335755 conn(0x565058b42030 msgr2=0x565058bc77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 shutdown_connections 2023-09-10T01:18:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/567335755 wait complete. 2023-09-10T01:18:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.510+0000 7f16d9ffdb80 20 client.29069 trim_cache size 0 max 16384 2023-09-10T01:18:12.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 Processor -- start 2023-09-10T01:18:12.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 -- start start 2023-09-10T01:18:12.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcd300 0x565058bde520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bcfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bd31a0 0x565058bd97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058bdea60 con 0x565058bcd300 2023-09-10T01:18:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058bd5640 con 0x565058bd31a0 2023-09-10T01:18:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058bd1c90 con 0x565058bcdcc0 2023-09-10T01:18:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16bffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcd300 0x565058bde520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16bffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcd300 0x565058bde520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55762/0 (socket says 172.21.15.3:55762) 2023-09-10T01:18:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.526+0000 7f16bffff700 1 -- 172.21.15.3:0/3040633918 learned_addr learned my addr 172.21.15.3:0/3040633918 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16c4b9c700 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bd31a0 0x565058bd97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bcfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16bffff700 1 -- 172.21.15.3:0/3040633918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 msgr2=0x565058bcfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16bffff700 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bcfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16bffff700 1 -- 172.21.15.3:0/3040633918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bd31a0 msgr2=0x565058bd97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16bffff700 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bd31a0 0x565058bd97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16bffff700 1 -- 172.21.15.3:0/3040633918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650589721f0 con 0x565058bcd300 2023-09-10T01:18:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16c4b9c700 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bd31a0 0x565058bd97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:18:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16bffff700 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcd300 0x565058bde520 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f16b00098e0 tx=0x7f16b0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8a170fffb92eb5 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16a6ffd700 1 -- 172.21.15.3:0/3040633918 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b001d040 con 0x565058bcd300 2023-09-10T01:18:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16a6ffd700 1 -- 172.21.15.3:0/3040633918 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16b00227d0 con 0x565058bcd300 2023-09-10T01:18:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.527+0000 7f16a6ffd700 1 -- 172.21.15.3:0/3040633918 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b000f920 con 0x565058bcd300 2023-09-10T01:18:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/3040633918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcd300 msgr2=0x565058bde520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcd300 0x565058bde520 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f16b00098e0 tx=0x7f16b0009310 comp rx=0 tx=0).stop 2023-09-10T01:18:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/3040633918 shutdown_connections 2023-09-10T01:18:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bd31a0 0x565058bd97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bcd300 0x565058bde520 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/3040633918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcdcc0 0x565058bcfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/3040633918 >> 172.21.15.3:0/3040633918 conn(0x565058b42030 msgr2=0x565058bd20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:12.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/3040633918 shutdown_connections 2023-09-10T01:18:12.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.528+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/3040633918 wait complete. 2023-09-10T01:18:12.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.529+0000 7f16d9ffdb80 1 -- start start 2023-09-10T01:18:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.529+0000 7f16d9ffdb80 1 Processor -- start 2023-09-10T01:18:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcd300 0x565058bdc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcdcc0 0x565058bdca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16d9ffdb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bdcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058c39ee0 con 0x565058bd31a0 2023-09-10T01:18:12.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058c3a020 con 0x565058bcdcc0 2023-09-10T01:18:12.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16d9ffdb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x565058c3a1e0 con 0x565058bcd300 2023-09-10T01:18:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16d9ffdb80 10 client.0 _ll_register_callbacks cb 0x7f16b80091d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcd300 0x565058bdc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcdcc0 0x565058bdca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16c4b9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bdcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcd300 0x565058bdc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43018/0 (socket says 172.21.15.3:43018) 2023-09-10T01:18:12.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bffff700 1 -- 172.21.15.3:0/4243165056 learned_addr learned my addr 172.21.15.3:0/4243165056 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:12.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bf7fe700 1 -- 172.21.15.3:0/4243165056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcd300 msgr2=0x565058bdc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/4243165056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565058bcd300 0x565058bdc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bf7fe700 1 -- 172.21.15.3:0/4243165056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 msgr2=0x565058bdcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/4243165056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565058bd31a0 0x565058bdcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.530+0000 7f16bf7fe700 1 -- 172.21.15.3:0/4243165056 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565058934c00 con 0x565058bcdcc0 2023-09-10T01:18:12.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f16bf7fe700 1 --2- 172.21.15.3:0/4243165056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565058bcdcc0 0x565058bdca50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f16b40c99c0 tx=0x7f16b40d2060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2367aeefca7dc758 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:12.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f169f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:18:12.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f169f7fe700 1 -- 172.21.15.3:0/4243165056 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b409d330 con 0x565058bcdcc0 2023-09-10T01:18:12.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/4243165056 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565058b188c0 con 0x565058bcdcc0 2023-09-10T01:18:12.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f169f7fe700 1 -- 172.21.15.3:0/4243165056 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f16b4099e20 con 0x565058bcdcc0 2023-09-10T01:18:12.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f169f7fe700 1 -- 172.21.15.3:0/4243165056 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16b4099260 con 0x565058bcdcc0 2023-09-10T01:18:12.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f16d9ffdb80 10 client.29075 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:12.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/4243165056 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56505894a250 con 0x565058bcdcc0 2023-09-10T01:18:12.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f169f7fe700 1 -- 172.21.15.3:0/4243165056 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565058b188c0 con 0x565058bcdcc0 2023-09-10T01:18:12.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.531+0000 7f169f7fe700 1 -- 172.21.15.3:0/4243165056 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f16b40a1300 con 0x565058bcdcc0 2023-09-10T01:18:12.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16d9ffdb80 10 client.29075 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:12.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16d9ffdb80 10 client.29075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:12.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16d9ffdb80 20 client.29075 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:12.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16d9ffdb80 1 --2- 172.21.15.3:0/4243165056 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x565058c3b930 0x565058c3dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:12.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16d9ffdb80 4 client.29075 mds_command: new command op to 24698 tid=0[{"prefix": "flush journal"}] 2023-09-10T01:18:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16d9ffdb80 1 -- 172.21.15.3:0/4243165056 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565058ad9050 con 0x565058c3b930 2023-09-10T01:18:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16bffff700 1 --2- 172.21.15.3:0/4243165056 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x565058c3b930 0x565058c3dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f16bffff700 1 --2- 172.21.15.3:0/4243165056 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x565058c3b930 0x565058c3dd50 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee7d9aab3677fd32 server_cookie=8f0e9025cdfb7b45 in_seq=0 out_seq=0 2023-09-10T01:18:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2023-09-10T01:18:12.532+0000 7f169f7fe700 10 client.29075 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:12.789 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2023-09-10T01:18:12.801 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2023-09-10T01:18:12.802 INFO:teuthology.task.exec:Executing custom commands... 2023-09-10T01:18:12.802 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:18:12.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2023-09-10T01:18:13.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2023-09-10T01:18:13.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 -- 172.21.15.3:0/4136831180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c103240 msgr2=0x7f3f9c10f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/4136831180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c103240 0x7f3f9c10f750 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400b0a0 tx=0x7f3f9400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 -- 172.21.15.3:0/4136831180 shutdown_connections 2023-09-10T01:18:13.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/4136831180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c103240 0x7f3f9c10f750 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/4136831180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9c108890 0x7f3f9c108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/4136831180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 0x7f3f9c102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 -- 172.21.15.3:0/4136831180 >> 172.21.15.3:0/4136831180 conn(0x7f3f9c0fe330 msgr2=0x7f3f9c100750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:13.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.245+0000 7f3fa33ff700 1 -- 172.21.15.3:0/4136831180 shutdown_connections 2023-09-10T01:18:13.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.246+0000 7f3fa33ff700 1 -- 172.21.15.3:0/4136831180 wait complete. 2023-09-10T01:18:13.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.246+0000 7f3fa33ff700 1 Processor -- start 2023-09-10T01:18:13.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa33ff700 1 -- start start 2023-09-10T01:18:13.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa33ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 0x7f3f9c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa33ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9c103240 0x7f3f9c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa33ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c108890 0x7f3f9c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa33ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c198db0 con 0x7f3f9c108890 2023-09-10T01:18:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa33ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c198f20 con 0x7f3f9c103240 2023-09-10T01:18:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa33ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c1990e0 con 0x7f3f9c102880 2023-09-10T01:18:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa199c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c108890 0x7f3f9c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa119b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 0x7f3f9c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa119b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 0x7f3f9c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43030/0 (socket says 172.21.15.3:43030) 2023-09-10T01:18:13.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa119b700 1 -- 172.21.15.3:0/2880038082 learned_addr learned my addr 172.21.15.3:0/2880038082 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:13.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.247+0000 7f3fa099a700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9c103240 0x7f3f9c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa119b700 1 -- 172.21.15.3:0/2880038082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9c103240 msgr2=0x7f3f9c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa119b700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9c103240 0x7f3f9c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa119b700 1 -- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c108890 msgr2=0x7f3f9c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa119b700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c108890 0x7f3f9c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa119b700 1 -- 172.21.15.3:0/2880038082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94009d00 con 0x7f3f9c102880 2023-09-10T01:18:13.255 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa199c700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c108890 0x7f3f9c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:13.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa119b700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 0x7f3f9c19e6d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800e9a0 tx=0x7f3f9800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3f927fc700 1 -- 172.21.15.3:0/2880038082 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98009790 con 0x7f3f9c102880 2023-09-10T01:18:13.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3f927fc700 1 -- 172.21.15.3:0/2880038082 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f98010430 con 0x7f3f9c102880 2023-09-10T01:18:13.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c1993c0 con 0x7f3f9c102880 2023-09-10T01:18:13.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3f927fc700 1 -- 172.21.15.3:0/2880038082 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98018680 con 0x7f3f9c102880 2023-09-10T01:18:13.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.248+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c1a56c0 con 0x7f3f9c102880 2023-09-10T01:18:13.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.251+0000 7f3f927fc700 1 -- 172.21.15.3:0/2880038082 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3f980105a0 con 0x7f3f9c102880 2023-09-10T01:18:13.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 Processor -- start 2023-09-10T01:18:13.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 -- start start 2023-09-10T01:18:13.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3f927fc700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f880a0830 0x7f3f880a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3f927fc700 1 -- 172.21.15.3:0/2880038082 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f3f98014070 con 0x7f3f9c102880 2023-09-10T01:18:13.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fa199c700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f880a0830 0x7f3f880a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c482af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c483030 0x56338c48c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c48f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c3db730 con 0x56338c483030 2023-09-10T01:18:13.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c3d9050 con 0x56338c482710 2023-09-10T01:18:13.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.254+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c3d4940 con 0x56338c48ce40 2023-09-10T01:18:13.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c482af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c482af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48044/0 (socket says 172.21.15.3:48044) 2023-09-10T01:18:13.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 -- 172.21.15.3:0/2602230436 learned_addr learned my addr 172.21.15.3:0/2602230436 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:13.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa099a700 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c483030 0x56338c48c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa199c700 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c48f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 -- 172.21.15.3:0/2602230436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 msgr2=0x56338c48f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c48f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 -- 172.21.15.3:0/2602230436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c483030 msgr2=0x56338c48c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c483030 0x56338c48c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 -- 172.21.15.3:0/2602230436 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56338c200de0 con 0x56338c482710 2023-09-10T01:18:13.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa199c700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f880a0830 0x7f3f880a2cf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009610 tx=0x7f3f9400bbc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.268 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3fa119b700 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c482af0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3f98095fd0 tx=0x7f3f98098fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=407c147822589b6e server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.255+0000 7f3f877fe700 1 -- 172.21.15.3:0/2602230436 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9809d170 con 0x56338c482710 2023-09-10T01:18:13.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3f877fe700 1 -- 172.21.15.3:0/2602230436 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f9809d390 con 0x56338c482710 2023-09-10T01:18:13.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3f877fe700 1 -- 172.21.15.3:0/2602230436 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f980997c0 con 0x56338c482710 2023-09-10T01:18:13.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2602230436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 msgr2=0x56338c482af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c482af0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3f98095fd0 tx=0x7f3f98098fb0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2602230436 shutdown_connections 2023-09-10T01:18:13.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c48f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c482af0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2602230436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c483030 0x56338c48c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2602230436 >> 172.21.15.3:0/2602230436 conn(0x56338c3f7090 msgr2=0x56338c47dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:13.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2602230436 shutdown_connections 2023-09-10T01:18:13.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.256+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2602230436 wait complete. 2023-09-10T01:18:13.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.257+0000 7f3fb6d11b80 1 -- start start 2023-09-10T01:18:13.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.257+0000 7f3fb6d11b80 1 Processor -- start 2023-09-10T01:18:13.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c489b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c483030 0x56338c48a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c372b70 con 0x56338c48ce40 2023-09-10T01:18:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c372870 con 0x56338c482710 2023-09-10T01:18:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c256920 con 0x56338c483030 2023-09-10T01:18:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fb6d11b80 10 client.0 _ll_register_callbacks cb 0x56338c2bca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa099a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c483030 0x56338c48a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55812/0 (socket says 172.21.15.3:55812) 2023-09-10T01:18:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 -- 172.21.15.3:0/3763864159 learned_addr learned my addr 172.21.15.3:0/3763864159 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa119b700 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c489b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 -- 172.21.15.3:0/3763864159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c483030 msgr2=0x56338c48a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c483030 0x56338c48a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 -- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 msgr2=0x56338c489b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c489b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa199c700 1 -- 172.21.15.3:0/3763864159 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c1a56c0 con 0x56338c48ce40 2023-09-10T01:18:13.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.258+0000 7f3fa119b700 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c489b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:13.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3fa199c700 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3f94015000 tx=0x7f3f9400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67652a12a800412 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3f84ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:13.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3f84ff9700 1 -- 172.21.15.3:0/3763864159 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400e040 con 0x56338c48ce40 2023-09-10T01:18:13.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56338c32c4f0 con 0x56338c48ce40 2023-09-10T01:18:13.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3f84ff9700 1 -- 172.21.15.3:0/3763864159 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f3f94004030 con 0x56338c48ce40 2023-09-10T01:18:13.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3f84ff9700 1 -- 172.21.15.3:0/3763864159 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f940041f0 con 0x56338c48ce40 2023-09-10T01:18:13.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3fb6d11b80 10 client.18969 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:13.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56338c1e9c60 con 0x56338c48ce40 2023-09-10T01:18:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.259+0000 7f3f84ff9700 1 -- 172.21.15.3:0/3763864159 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56338c32c4f0 con 0x56338c48ce40 2023-09-10T01:18:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3f84ff9700 1 -- 172.21.15.3:0/3763864159 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3f9402f030 con 0x56338c48ce40 2023-09-10T01:18:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3fb6d11b80 10 client.18969 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3fb6d11b80 10 client.18969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:13.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3fb6d11b80 20 client.18969 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:13.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c49c9e0 0x56338c49ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3fb6d11b80 4 client.18969 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:13.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56338c3e98a0 con 0x56338c49c9e0 2023-09-10T01:18:13.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.260+0000 7f3fa119b700 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c49c9e0 0x56338c49ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.261+0000 7f3fa119b700 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c49c9e0 0x56338c49ee00 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb5a7bcc5585322a server_cookie=d5f6c9472258ef7c in_seq=0 out_seq=0 2023-09-10T01:18:13.282 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.261+0000 7f3f84ff9700 10 client.18969 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:13.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-09-10T01:18:13.542 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.541+0000 7f3f84ff9700 1 -- 172.21.15.3:0/3763864159 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56338c3e98a0 con 0x56338c49c9e0 2023-09-10T01:18:13.543 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.541+0000 7f3f84ff9700 10 client.18969 handle_command_reply: tid=0 2023-09-10T01:18:13.546 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.546+0000 7f3fb6d11b80 1 client.18969 shutdown 2023-09-10T01:18:13.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.555+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 msgr2=0x56338c48a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.555+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3f94015000 tx=0x7f3f9400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 shutdown_connections 2023-09-10T01:18:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c49c9e0 0x56338c49ee00 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5a0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.557 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c482710 0x56338c489b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/3763864159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c483030 0x56338c48a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 >> 172.21.15.3:0/3763864159 conn(0x56338c3f7090 msgr2=0x56338c47d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:13.558 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3f84ff9700 0 client.18969 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:13.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.556+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 shutdown_connections 2023-09-10T01:18:13.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.557+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/3763864159 wait complete. 2023-09-10T01:18:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.557+0000 7f3fb6d11b80 20 client.18969 trim_cache size 0 max 16384 2023-09-10T01:18:13.598 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.597+0000 7f3fb6d11b80 1 Processor -- start 2023-09-10T01:18:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.598+0000 7f3fb6d11b80 1 -- start start 2023-09-10T01:18:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.598+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c482710 0x56338c493930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.600 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.598+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c48a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.598+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.598+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c493e70 con 0x56338c48ce40 2023-09-10T01:18:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.598+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c47e6e0 con 0x56338c483030 2023-09-10T01:18:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.598+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c475400 con 0x56338c482710 2023-09-10T01:18:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.599+0000 7f3fa099a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c48a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.599+0000 7f3fa119b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c482710 0x56338c493930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.602 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.599+0000 7f3fa099a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c48a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:48070/0 (socket says 172.21.15.3:48070) 2023-09-10T01:18:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.599+0000 7f3fa099a700 1 -- 172.21.15.3:0/2679789511 learned_addr learned my addr 172.21.15.3:0/2679789511 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.600+0000 7f3fa099a700 1 -- 172.21.15.3:0/2679789511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c482710 msgr2=0x56338c493930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.600+0000 7f3fa099a700 1 --2- 172.21.15.3:0/2679789511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c482710 0x56338c493930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.600+0000 7f3fa099a700 1 -- 172.21.15.3:0/2679789511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 msgr2=0x56338c48a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.604 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.600+0000 7f3fa099a700 1 --2- 172.21.15.3:0/2679789511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.600+0000 7f3fa099a700 1 -- 172.21.15.3:0/2679789511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56338c227250 con 0x56338c483030 2023-09-10T01:18:13.605 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.601+0000 7f3fa099a700 1 --2- 172.21.15.3:0/2679789511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c48a090 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009f00 tx=0x7f3f8c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265cb17c859eae23 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.602+0000 7f3f87fff700 1 -- 172.21.15.3:0/2679789511 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c017050 con 0x56338c483030 2023-09-10T01:18:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.602+0000 7f3f87fff700 1 -- 172.21.15.3:0/2679789511 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f8c018040 con 0x56338c483030 2023-09-10T01:18:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.602+0000 7f3f87fff700 1 -- 172.21.15.3:0/2679789511 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c003c60 con 0x56338c483030 2023-09-10T01:18:13.611 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.603+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2679789511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 msgr2=0x56338c48a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.603+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2679789511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c48a090 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009f00 tx=0x7f3f8c009260 comp rx=0 tx=0).stop 2023-09-10T01:18:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.606+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2679789511 shutdown_connections 2023-09-10T01:18:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.606+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2679789511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c482710 0x56338c493930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.606+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2679789511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c48ce40 0x56338c48a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.606+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/2679789511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c48a090 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.606+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2679789511 >> 172.21.15.3:0/2679789511 conn(0x56338c3f7090 msgr2=0x56338c484e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:13.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.618+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2679789511 shutdown_connections 2023-09-10T01:18:13.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.618+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/2679789511 wait complete. 2023-09-10T01:18:13.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.618+0000 7f3fb6d11b80 1 -- start start 2023-09-10T01:18:13.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 1 Processor -- start 2023-09-10T01:18:13.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c482710 0x56338c4f3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c4f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c4f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c4f1e70 con 0x56338c482710 2023-09-10T01:18:13.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c4f1fe0 con 0x56338c483030 2023-09-10T01:18:13.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338c4ecf70 con 0x56338c48ce40 2023-09-10T01:18:13.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.619+0000 7f3fb6d11b80 10 client.0 _ll_register_callbacks cb 0x7f3f94004d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.623+0000 7f3fa199c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c4f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.623+0000 7f3fa199c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c4f17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43058/0 (socket says 172.21.15.3:43058) 2023-09-10T01:18:13.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.623+0000 7f3fa199c700 1 -- 172.21.15.3:0/634691005 learned_addr learned my addr 172.21.15.3:0/634691005 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:13.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.623+0000 7f3fa099a700 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c4f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.624+0000 7f3fa119b700 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c482710 0x56338c4f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.626+0000 7f3fa119b700 1 -- 172.21.15.3:0/634691005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 msgr2=0x56338c4f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.626+0000 7f3fa119b700 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c4f17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.626+0000 7f3fa119b700 1 -- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 msgr2=0x56338c4f12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.626+0000 7f3fa119b700 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c4f12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.626+0000 7f3fa119b700 1 -- 172.21.15.3:0/634691005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c009bb0 con 0x56338c482710 2023-09-10T01:18:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.635+0000 7f3fa119b700 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c482710 0x56338c4f3750 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x56338c483560 tx=0x7f3f98010f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99fe5d05ad4daf4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.641+0000 7f3f84ff9700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.641+0000 7f3f84ff9700 1 -- 172.21.15.3:0/634691005 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f980a15d0 con 0x56338c482710 2023-09-10T01:18:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.641+0000 7f3f84ff9700 1 -- 172.21.15.3:0/634691005 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7f3f980840b0 con 0x56338c482710 2023-09-10T01:18:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.641+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f9809d390 con 0x56338c482710 2023-09-10T01:18:13.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.644+0000 7f3f84ff9700 1 -- 172.21.15.3:0/634691005 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f98098a50 con 0x56338c482710 2023-09-10T01:18:13.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.644+0000 7f3fb6d11b80 10 client.18972 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:13.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.644+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x56338c1ff2b0 con 0x56338c482710 2023-09-10T01:18:13.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3f84ff9700 1 -- 172.21.15.3:0/634691005 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f9809d390 con 0x56338c482710 2023-09-10T01:18:13.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3f84ff9700 1 -- 172.21.15.3:0/634691005 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3f980997c0 con 0x56338c482710 2023-09-10T01:18:13.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3fb6d11b80 10 client.18972 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:13.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3fb6d11b80 10 client.18972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:13.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3fb6d11b80 20 client.18972 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:13.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c50b250 0x56338c50d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3fb6d11b80 4 client.18972 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:18:13.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.646+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56338c3cd920 con 0x56338c50b250 2023-09-10T01:18:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.654+0000 7f3fa099a700 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c50b250 0x56338c50d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.659+0000 7f3fa099a700 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c50b250 0x56338c50d670 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92e6ddc51a3a4c2c server_cookie=7ac1c2bea2abce60 in_seq=0 out_seq=0 2023-09-10T01:18:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.659+0000 7f3f84ff9700 10 client.18972 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:13.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.660+0000 7f3f84ff9700 1 -- 172.21.15.3:0/634691005 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56338c3cd920 con 0x56338c50b250 2023-09-10T01:18:13.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.660+0000 7f3f84ff9700 10 client.18972 handle_command_reply: tid=0 2023-09-10T01:18:13.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.660+0000 7f3fb6d11b80 1 client.18972 shutdown 2023-09-10T01:18:13.664 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c482710 msgr2=0x56338c4f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c482710 0x56338c4f3750 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x56338c483560 tx=0x7f3f98010f80 comp rx=0 tx=0).stop 2023-09-10T01:18:13.665 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 shutdown_connections 2023-09-10T01:18:13.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56338c50b250 0x56338c50d670 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56338c48ce40 0x56338c4f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56338c482710 0x56338c4f3750 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 --2- 172.21.15.3:0/634691005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56338c483030 0x56338c4f12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.667 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 >> 172.21.15.3:0/634691005 conn(0x56338c3f7090 msgr2=0x56338c484e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:13.668 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.664+0000 7f3f84ff9700 0 client.18972 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:13.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.672+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 shutdown_connections 2023-09-10T01:18:13.673 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.673+0000 7f3fb6d11b80 1 -- 172.21.15.3:0/634691005 wait complete. 2023-09-10T01:18:13.676 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.676+0000 7f3fb6d11b80 20 client.18972 trim_cache size 0 max 16384 2023-09-10T01:18:13.677 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:18:13.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.687+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f880a0830 msgr2=0x7f3f880a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.687+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f880a0830 0x7f3f880a2cf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009610 tx=0x7f3f9400bbc0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.687+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 msgr2=0x7f3f9c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.687+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 0x7f3f9c19e6d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800e9a0 tx=0x7f3f9800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.689+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 shutdown_connections 2023-09-10T01:18:13.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.689+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3f880a0830 0x7f3f880a2cf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.693 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.689+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c102880 0x7f3f9c19e6d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.689+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f9c108890 0x7f3f9c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.689+0000 7f3fa33ff700 1 --2- 172.21.15.3:0/2880038082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f9c103240 0x7f3f9c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.694 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.689+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 >> 172.21.15.3:0/2880038082 conn(0x7f3f9c0fe330 msgr2=0x7f3f9c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:13.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.693+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 shutdown_connections 2023-09-10T01:18:13.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.696+0000 7f3fa33ff700 1 -- 172.21.15.3:0/2880038082 wait complete. 2023-09-10T01:18:13.904 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.902+0000 7fd1b65ff700 1 -- 172.21.15.3:0/547070840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fe610 msgr2=0x7fd1b00fe9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:13.905 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.902+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/547070840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fe610 0x7fd1b00fe9f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009990 tx=0x7fd1a0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.905+0000 7fd1b65ff700 1 -- 172.21.15.3:0/547070840 shutdown_connections 2023-09-10T01:18:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.905+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/547070840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1b01112a0 0x7fd1b0113690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.908 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.905+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/547070840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fe610 0x7fd1b00fe9f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.905+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/547070840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b00fef30 0x7fd1b0110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.909 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.905+0000 7fd1b65ff700 1 -- 172.21.15.3:0/547070840 >> 172.21.15.3:0/547070840 conn(0x7fd1b0076590 msgr2=0x7fd1b00fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.911+0000 7fd1b65ff700 1 -- 172.21.15.3:0/547070840 shutdown_connections 2023-09-10T01:18:13.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.913+0000 7fd1b65ff700 1 -- 172.21.15.3:0/547070840 wait complete. 2023-09-10T01:18:13.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.914+0000 7fd1b65ff700 1 Processor -- start 2023-09-10T01:18:13.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.915+0000 7fd1b65ff700 1 -- start start 2023-09-10T01:18:13.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.915+0000 7fd1b65ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1b00fe610 0x7fd1b01104f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.919 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.915+0000 7fd1b65ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fef30 0x7fd1b0110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.915+0000 7fd1b65ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b01112a0 0x7fd1b0072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.915+0000 7fd1b65ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b010c5c0 con 0x7fd1b00fef30 2023-09-10T01:18:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.915+0000 7fd1b65ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b010c730 con 0x7fd1b00fe610 2023-09-10T01:18:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.915+0000 7fd1b65ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b010c8f0 con 0x7fd1b01112a0 2023-09-10T01:18:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.916+0000 7fd1af7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fef30 0x7fd1b0110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.916+0000 7fd1af7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fef30 0x7fd1b0110a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55842/0 (socket says 172.21.15.3:55842) 2023-09-10T01:18:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.916+0000 7fd1af7fe700 1 -- 172.21.15.3:0/1167545759 learned_addr learned my addr 172.21.15.3:0/1167545759 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.917+0000 7fd1af7fe700 1 -- 172.21.15.3:0/1167545759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b01112a0 msgr2=0x7fd1b0072070 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-10T01:18:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.917+0000 7fd1af7fe700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b01112a0 0x7fd1b0072070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.917+0000 7fd1af7fe700 1 -- 172.21.15.3:0/1167545759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1b00fe610 msgr2=0x7fd1b01104f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-10T01:18:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.917+0000 7fd1af7fe700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1b00fe610 0x7fd1b01104f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.917+0000 7fd1af7fe700 1 -- 172.21.15.3:0/1167545759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a00095f0 con 0x7fd1b00fef30 2023-09-10T01:18:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.918+0000 7fd1af7fe700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fef30 0x7fd1b0110a30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400b550 tx=0x7fd1a400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.922+0000 7fd1ad7fa700 1 -- 172.21.15.3:0/1167545759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4011830 con 0x7fd1b00fef30 2023-09-10T01:18:13.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.922+0000 7fd1ad7fa700 1 -- 172.21.15.3:0/1167545759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1a4011e50 con 0x7fd1b00fef30 2023-09-10T01:18:13.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.922+0000 7fd1ad7fa700 1 -- 172.21.15.3:0/1167545759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a400f2e0 con 0x7fd1b00fef30 2023-09-10T01:18:13.927 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.926+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b010cbd0 con 0x7fd1b00fef30 2023-09-10T01:18:13.928 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.926+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b010d060 con 0x7fd1b00fef30 2023-09-10T01:18:13.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.931+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b004e300 con 0x7fd1b00fef30 2023-09-10T01:18:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.934+0000 7fd1ad7fa700 1 -- 172.21.15.3:0/1167545759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fd1a4011990 con 0x7fd1b00fef30 2023-09-10T01:18:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.936+0000 7fd1ad7fa700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd1980a09c0 0x7fd1980a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.936+0000 7fd1ad7fa700 1 -- 172.21.15.3:0/1167545759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fd1a40cbd30 con 0x7fd1b00fef30 2023-09-10T01:18:13.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.945+0000 7fd1affff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd1980a09c0 0x7fd1980a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:13.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.947+0000 7fd1affff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd1980a09c0 0x7fd1980a2e80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0005cb0 tx=0x7fd1a0005bc0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:13.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:13.947+0000 7fd1ad7fa700 1 -- 172.21.15.3:0/1167545759 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fd1a4095190 con 0x7fd1b00fef30 2023-09-10T01:18:14.192 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.191+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 --> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1b00009e0 con 0x7fd1980a09c0 2023-09-10T01:18:14.538 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.538+0000 7fd1ad7fa700 1 -- 172.21.15.3:0/1167545759 <== mgr.25667 v2:172.21.15.3:6800/3336040255 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7fd1a0003680 con 0x7fd1980a09c0 2023-09-10T01:18:14.542 INFO:teuthology.orchestra.run.smithi003.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-09-10T00:00:00", "created": "2023-09-10T01:03:13", "first": "2023-09-10T01:04:00", "last": "2023-09-10T01:18:00", "last_pruned": "2023-09-10T01:18:00", "created_count": 15, "pruned_count": 9, "active": true} 2023-09-10T01:18:14.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.545+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd1980a09c0 msgr2=0x7fd1980a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:14.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.545+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd1980a09c0 0x7fd1980a2e80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0005cb0 tx=0x7fd1a0005bc0 comp rx=0 tx=0).stop 2023-09-10T01:18:14.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.545+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fef30 msgr2=0x7fd1b0110a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:14.549 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.545+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fef30 0x7fd1b0110a30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400b550 tx=0x7fd1a400b910 comp rx=0 tx=0).stop 2023-09-10T01:18:14.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.548+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 shutdown_connections 2023-09-10T01:18:14.553 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.548+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fd1980a09c0 0x7fd1980a2e80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:14.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.548+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b01112a0 0x7fd1b0072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:14.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.548+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd1b00fe610 0x7fd1b01104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:14.554 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.548+0000 7fd1b65ff700 1 --2- 172.21.15.3:0/1167545759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1b00fef30 0x7fd1b0110a30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:14.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.548+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 >> 172.21.15.3:0/1167545759 conn(0x7fd1b0076590 msgr2=0x7fd1b0112600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:14.555 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.554+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 shutdown_connections 2023-09-10T01:18:14.559 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:14.555+0000 7fd1b65ff700 1 -- 172.21.15.3:0/1167545759 wait complete. 2023-09-10T01:18:14.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:14 smithi003 ceph-mon[97654]: pgmap v471: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 800 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 628 op/s 2023-09-10T01:18:14.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-09-10T01:18:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:14 smithi136 ceph-mon[133708]: pgmap v471: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 800 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 628 op/s 2023-09-10T01:18:14.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:14 smithi092 ceph-mon[134067]: pgmap v471: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 800 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 628 op/s 2023-09-10T01:18:14.753 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-09-10T01:18:14.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-09-10T01:18:15.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.127+0000 7fa4bddff700 1 -- 172.21.15.3:0/1812906952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b81115b0 msgr2=0x7fa4b81139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.127+0000 7fa4bddff700 1 --2- 172.21.15.3:0/1812906952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b81115b0 0x7fa4b81139a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009990 tx=0x7fa4ac009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.127+0000 7fa4bddff700 1 -- 172.21.15.3:0/1812906952 shutdown_connections 2023-09-10T01:18:15.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.127+0000 7fa4bddff700 1 --2- 172.21.15.3:0/1812906952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4b80729a0 0x7fa4b8072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.127+0000 7fa4bddff700 1 --2- 172.21.15.3:0/1812906952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b81115b0 0x7fa4b81139a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.127+0000 7fa4bddff700 1 --2- 172.21.15.3:0/1812906952 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4b80732c0 0x7fa4b8111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.127+0000 7fa4bddff700 1 -- 172.21.15.3:0/1812906952 >> 172.21.15.3:0/1812906952 conn(0x7fa4b80fc0b0 msgr2=0x7fa4b80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.128+0000 7fa4bddff700 1 -- 172.21.15.3:0/1812906952 shutdown_connections 2023-09-10T01:18:15.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.128+0000 7fa4bddff700 1 -- 172.21.15.3:0/1812906952 wait complete. 2023-09-10T01:18:15.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.129+0000 7fa4bddff700 1 Processor -- start 2023-09-10T01:18:15.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.130+0000 7fa4bddff700 1 -- start start 2023-09-10T01:18:15.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.130+0000 7fa4bddff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4b80729a0 0x7fa4b810d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.130+0000 7fa4bddff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b80732c0 0x7fa4b810d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.130+0000 7fa4bddff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4b81115b0 0x7fa4b8109750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.130+0000 7fa4bddff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8109d50 con 0x7fa4b81115b0 2023-09-10T01:18:15.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.130+0000 7fa4bddff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8109ec0 con 0x7fa4b80732c0 2023-09-10T01:18:15.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.130+0000 7fa4bddff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b810a080 con 0x7fa4b80729a0 2023-09-10T01:18:15.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.133+0000 7fa4b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4b81115b0 0x7fa4b8109750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.133+0000 7fa4b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4b81115b0 0x7fa4b8109750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44966/0 (socket says 172.21.15.3:44966) 2023-09-10T01:18:15.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.133+0000 7fa4b7fff700 1 -- 172.21.15.3:0/2810540001 learned_addr learned my addr 172.21.15.3:0/2810540001 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.133+0000 7fa4b77fe700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4b80729a0 0x7fa4b810d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.133+0000 7fa4b6ffd700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b80732c0 0x7fa4b810d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.134+0000 7fa4b77fe700 1 -- 172.21.15.3:0/2810540001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b80732c0 msgr2=0x7fa4b810d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.134+0000 7fa4b77fe700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b80732c0 0x7fa4b810d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.134+0000 7fa4b77fe700 1 -- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4b81115b0 msgr2=0x7fa4b8109750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.134+0000 7fa4b77fe700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4b81115b0 0x7fa4b8109750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.134+0000 7fa4b77fe700 1 -- 172.21.15.3:0/2810540001 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0095f0 con 0x7fa4b80729a0 2023-09-10T01:18:15.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.139+0000 7fa4b77fe700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4b80729a0 0x7fa4b810d4a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800c970 tx=0x7fa4a800cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.140+0000 7fa4b4ff9700 1 -- 172.21.15.3:0/2810540001 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8007b00 con 0x7fa4b80729a0 2023-09-10T01:18:15.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.140+0000 7fa4b4ff9700 1 -- 172.21.15.3:0/2810540001 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4a800ce60 con 0x7fa4b80729a0 2023-09-10T01:18:15.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.140+0000 7fa4b4ff9700 1 -- 172.21.15.3:0/2810540001 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8005440 con 0x7fa4b80729a0 2023-09-10T01:18:15.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.141+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b810a360 con 0x7fa4b80729a0 2023-09-10T01:18:15.143 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.142+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b810e540 con 0x7fa4b80729a0 2023-09-10T01:18:15.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.149+0000 7fa4b4ff9700 1 -- 172.21.15.3:0/2810540001 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa4a80055a0 con 0x7fa4b80729a0 2023-09-10T01:18:15.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.152+0000 7fa4b4ff9700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a00a09c0 0x7fa4a00a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.152+0000 7fa4b4ff9700 1 -- 172.21.15.3:0/2810540001 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fa4a80cc150 con 0x7fa4b80729a0 2023-09-10T01:18:15.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.154+0000 7fa4b6ffd700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a00a09c0 0x7fa4a00a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.158+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b804e300 con 0x7fa4b80729a0 2023-09-10T01:18:15.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.167+0000 7fa4b6ffd700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a00a09c0 0x7fa4a00a2e80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa4b810af40 tx=0x7fa4b000d040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.171+0000 7fa4b4ff9700 1 -- 172.21.15.3:0/2810540001 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa4a8095520 con 0x7fa4b80729a0 2023-09-10T01:18:15.233 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.231+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1813257988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740ff690 msgr2=0x7fa3740ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.231+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1813257988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740ff690 0x7fa3740ffa70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009f90 tx=0x7fa36c00b330 comp rx=0 tx=0).stop 2023-09-10T01:18:15.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.232+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1813257988 shutdown_connections 2023-09-10T01:18:15.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.232+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1813257988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa374104a90 0x7fa374104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.232+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1813257988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740ff690 0x7fa3740ffa70 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.232+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1813257988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740fffb0 0x7fa374104470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.232+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1813257988 >> 172.21.15.3:0/1813257988 conn(0x7fa374076710 msgr2=0x7fa374076b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.239 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.233+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1813257988 shutdown_connections 2023-09-10T01:18:15.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.233+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1813257988 wait complete. 2023-09-10T01:18:15.240 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.234+0000 7fa37a7ff700 1 Processor -- start 2023-09-10T01:18:15.241 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.234+0000 7fa37a7ff700 1 -- start start 2023-09-10T01:18:15.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.235+0000 7fa37a7ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740ff690 0x7fa37410f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.242 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.235+0000 7fa373fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740ff690 0x7fa37410f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.243 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.235+0000 7fa373fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740ff690 0x7fa37410f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35062/0 (socket says 172.21.15.3:35062) 2023-09-10T01:18:15.244 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.241+0000 7fa37a7ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740fffb0 0x7fa37410fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.241+0000 7fa37a7ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa374104a90 0x7fa3741c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.245 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.241+0000 7fa37a7ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741c28e0 con 0x7fa3740fffb0 2023-09-10T01:18:15.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.241+0000 7fa37a7ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741c2a20 con 0x7fa3740ff690 2023-09-10T01:18:15.246 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.241+0000 7fa37a7ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741c2b60 con 0x7fa374104a90 2023-09-10T01:18:15.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.242+0000 7fa373fff700 1 -- 172.21.15.3:0/1528634194 learned_addr learned my addr 172.21.15.3:0/1528634194 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.247 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.243+0000 7fa378d9c700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa374104a90 0x7fa3741c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.248 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.243+0000 7fa3737fe700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740fffb0 0x7fa37410fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.243+0000 7fa373fff700 1 -- 172.21.15.3:0/1528634194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa374104a90 msgr2=0x7fa3741c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.249 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.243+0000 7fa373fff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa374104a90 0x7fa3741c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.250 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.243+0000 7fa373fff700 1 -- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740fffb0 msgr2=0x7fa37410fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.243+0000 7fa373fff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740fffb0 0x7fa37410fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.251 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.243+0000 7fa373fff700 1 -- 172.21.15.3:0/1528634194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c009c40 con 0x7fa3740ff690 2023-09-10T01:18:15.252 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.244+0000 7fa373fff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740ff690 0x7fa37410f4d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00b6c0 tx=0x7fa36c004d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.249+0000 7fa3717fa700 1 -- 172.21.15.3:0/1528634194 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c00e040 con 0x7fa3740ff690 2023-09-10T01:18:15.253 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.250+0000 7fa3717fa700 1 -- 172.21.15.3:0/1528634194 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa36c00bc00 con 0x7fa3740ff690 2023-09-10T01:18:15.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.250+0000 7fa3717fa700 1 -- 172.21.15.3:0/1528634194 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c01b6c0 con 0x7fa3740ff690 2023-09-10T01:18:15.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.255+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3741c2de0 con 0x7fa3740ff690 2023-09-10T01:18:15.256 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.255+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3741c31e0 con 0x7fa3740ff690 2023-09-10T01:18:15.262 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.261+0000 7fa38e0a6b80 1 Processor -- start 2023-09-10T01:18:15.263 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.261+0000 7fa38e0a6b80 1 -- start start 2023-09-10T01:18:15.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.262+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17479b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.264 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.262+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b17483980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.262+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174862b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.265 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.262+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b173d27b0 con 0x559b17479790 2023-09-10T01:18:15.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.262+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b173d00d0 con 0x559b17483ec0 2023-09-10T01:18:15.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.262+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b173cb9c0 con 0x559b1747a0b0 2023-09-10T01:18:15.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.262+0000 7fa3737fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174862b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.263+0000 7fa3737fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174862b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35078/0 (socket says 172.21.15.3:35078) 2023-09-10T01:18:15.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.264+0000 7fa3737fe700 1 -- 172.21.15.3:0/1239203570 learned_addr learned my addr 172.21.15.3:0/1239203570 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.264+0000 7fa378d9c700 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17479b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.264+0000 7fa3737fe700 1 -- 172.21.15.3:0/1239203570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 msgr2=0x559b17483980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:18:15.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.264+0000 7fa3737fe700 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b17483980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.264+0000 7fa3737fe700 1 -- 172.21.15.3:0/1239203570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 msgr2=0x559b17479b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.264+0000 7fa3737fe700 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17479b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.264+0000 7fa3737fe700 1 -- 172.21.15.3:0/1239203570 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b171f7e60 con 0x559b17483ec0 2023-09-10T01:18:15.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.265+0000 7fa3737fe700 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174862b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa36801ad00 tx=0x7fa36800e630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=595d185ff8638e03 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.274 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.265+0000 7fa35a7fc700 1 -- 172.21.15.3:0/1239203570 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36800ee40 con 0x559b17483ec0 2023-09-10T01:18:15.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.265+0000 7fa35a7fc700 1 -- 172.21.15.3:0/1239203570 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa368004510 con 0x559b17483ec0 2023-09-10T01:18:15.275 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.265+0000 7fa35a7fc700 1 -- 172.21.15.3:0/1239203570 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368005280 con 0x559b17483ec0 2023-09-10T01:18:15.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.266+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/1239203570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 msgr2=0x559b174862b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.266+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174862b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa36801ad00 tx=0x7fa36800e630 comp rx=0 tx=0).stop 2023-09-10T01:18:15.276 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.267+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/1239203570 shutdown_connections 2023-09-10T01:18:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.267+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174862b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.267+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17479b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.267+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/1239203570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b17483980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.267+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/1239203570 >> 172.21.15.3:0/1239203570 conn(0x559b173ee110 msgr2=0x559b17474c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.267+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/1239203570 shutdown_connections 2023-09-10T01:18:15.279 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.267+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/1239203570 wait complete. 2023-09-10T01:18:15.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.268+0000 7fa38e0a6b80 1 -- start start 2023-09-10T01:18:15.280 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.268+0000 7fa3717fa700 1 -- 172.21.15.3:0/1528634194 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa36c01d470 con 0x7fa3740ff690 2023-09-10T01:18:15.281 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.268+0000 7fa38e0a6b80 1 Processor -- start 2023-09-10T01:18:15.284 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.269+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17483980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.269+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b17480b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.285 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.269+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174812d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.269+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b17369bf0 con 0x559b17479790 2023-09-10T01:18:15.286 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.269+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b173698f0 con 0x559b17483ec0 2023-09-10T01:18:15.287 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.269+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b1724d9a0 con 0x559b1747a0b0 2023-09-10T01:18:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.269+0000 7fa38e0a6b80 10 client.0 _ll_register_callbacks cb 0x559b172b3b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.270+0000 7fa3737fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174812d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17483980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17483980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45004/0 (socket says 172.21.15.3:45004) 2023-09-10T01:18:15.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 -- 172.21.15.3:0/3693152287 learned_addr learned my addr 172.21.15.3:0/3693152287 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.291 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 -- 172.21.15.3:0/3693152287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 msgr2=0x559b17480b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b17480b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 -- 172.21.15.3:0/3693152287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 msgr2=0x559b174812d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174812d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.271+0000 7fa378d9c700 1 -- 172.21.15.3:0/3693152287 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36801a930 con 0x559b17479790 2023-09-10T01:18:15.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.272+0000 7fa378d9c700 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17483980 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa364003f50 tx=0x7fa3640098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4ba88f3b20960b server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.272+0000 7fa3537fe700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.272+0000 7fa3717fa700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa35c0a08d0 0x7fa35c0a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.273+0000 7fa3717fa700 1 -- 172.21.15.3:0/1528634194 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fa36c0cd000 con 0x7fa3740ff690 2023-09-10T01:18:15.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.273+0000 7fa3537fe700 1 -- 172.21.15.3:0/3693152287 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364004360 con 0x559b17479790 2023-09-10T01:18:15.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.273+0000 7fa3537fe700 1 -- 172.21.15.3:0/3693152287 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9615+0+0 (secure 0 0 0) 0x7fa364009e30 con 0x559b17479790 2023-09-10T01:18:15.296 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.273+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b17323570 con 0x559b17479790 2023-09-10T01:18:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.273+0000 7fa3537fe700 1 -- 172.21.15.3:0/3693152287 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa364004a30 con 0x559b17479790 2023-09-10T01:18:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.275+0000 7fa38e0a6b80 10 client.18993 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.275+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x559b171e0ce0 con 0x559b17479790 2023-09-10T01:18:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.275+0000 7fa373fff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa35c0a08d0 0x7fa35c0a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa3537fe700 1 -- 172.21.15.3:0/3693152287 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b17323570 con 0x559b17479790 2023-09-10T01:18:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa3537fe700 1 -- 172.21.15.3:0/3693152287 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fa364011890 con 0x559b17479790 2023-09-10T01:18:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa38e0a6b80 10 client.18993 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:15.298 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa38e0a6b80 10 client.18993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa38e0a6b80 20 client.18993 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174a4ab0 0x559b174a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.299 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa38e0a6b80 4 client.18993 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.276+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b173e0920 con 0x559b174a4ab0 2023-09-10T01:18:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.277+0000 7fa373fff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa35c0a08d0 0x7fa35c0a2d90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559b1746f300 tx=0x7fa36c01df90 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.300 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.279+0000 7fa3737fe700 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174a4ab0 0x559b174a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.281+0000 7fa3737fe700 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174a4ab0 0x559b174a6ed0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ddf911fc9e41604 server_cookie=e22c10d96d5fa3db in_seq=0 out_seq=0 2023-09-10T01:18:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.282+0000 7fa3537fe700 10 client.18993 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.283+0000 7fa3537fe700 1 -- 172.21.15.3:0/3693152287 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559b173e0920 con 0x559b174a4ab0 2023-09-10T01:18:15.301 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.283+0000 7fa3537fe700 10 client.18993 handle_command_reply: tid=0 2023-09-10T01:18:15.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.283+0000 7fa38e0a6b80 1 client.18993 shutdown 2023-09-10T01:18:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.287+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 msgr2=0x559b17483980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.287+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17483980 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa364003f50 tx=0x7fa3640098a0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.288+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 shutdown_connections 2023-09-10T01:18:15.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.288+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174a4ab0 0x559b174a6ed0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.288+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174812d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.288+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b17483980 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.288+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/3693152287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b17480b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.288+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 >> 172.21.15.3:0/3693152287 conn(0x559b173ee110 msgr2=0x559b17474860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.289+0000 7fa3537fe700 0 client.18993 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:15.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.289+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 shutdown_connections 2023-09-10T01:18:15.305 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.290+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/3693152287 wait complete. 2023-09-10T01:18:15.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.290+0000 7fa38e0a6b80 20 client.18993 trim_cache size 0 max 16384 2023-09-10T01:18:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.323+0000 7fa38e0a6b80 1 Processor -- start 2023-09-10T01:18:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.323+0000 7fa38e0a6b80 1 -- start start 2023-09-10T01:18:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.324+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b17479790 0x559b174834b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.324+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b1747a0b0 0x559b1747de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.326 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.324+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b1747e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.324+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b17323570 con 0x559b1747a0b0 2023-09-10T01:18:15.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.324+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b1747a510 con 0x559b17483ec0 2023-09-10T01:18:15.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.324+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b1746c550 con 0x559b17479790 2023-09-10T01:18:15.327 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b1747e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b1747e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35102/0 (socket says 172.21.15.3:35102) 2023-09-10T01:18:15.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 -- 172.21.15.3:0/478348081 learned_addr learned my addr 172.21.15.3:0/478348081 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 -- 172.21.15.3:0/478348081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b17479790 msgr2=0x559b174834b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 --2- 172.21.15.3:0/478348081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b17479790 0x559b174834b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 -- 172.21.15.3:0/478348081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b1747a0b0 msgr2=0x559b1747de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 --2- 172.21.15.3:0/478348081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b1747a0b0 0x559b1747de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 -- 172.21.15.3:0/478348081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b1721e2d0 con 0x559b17483ec0 2023-09-10T01:18:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.325+0000 7fa378d9c700 1 --2- 172.21.15.3:0/478348081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b1747e360 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa364005a80 tx=0x7fa364009d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd812ee4625a51b8 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.329+0000 7fa35affd700 1 -- 172.21.15.3:0/478348081 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364011850 con 0x559b17483ec0 2023-09-10T01:18:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.329+0000 7fa35affd700 1 -- 172.21.15.3:0/478348081 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa364009e30 con 0x559b17483ec0 2023-09-10T01:18:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.329+0000 7fa35affd700 1 -- 172.21.15.3:0/478348081 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364004c10 con 0x559b17483ec0 2023-09-10T01:18:15.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.334+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/478348081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 msgr2=0x559b1747e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.334+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/478348081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b1747e360 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa364005a80 tx=0x7fa364009d20 comp rx=0 tx=0).stop 2023-09-10T01:18:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.334+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/478348081 shutdown_connections 2023-09-10T01:18:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.334+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/478348081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b17479790 0x559b174834b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.334+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/478348081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b1747e360 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.334+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/478348081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b1747a0b0 0x559b1747de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.334+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/478348081 >> 172.21.15.3:0/478348081 conn(0x559b173ee110 msgr2=0x559b1747bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.341 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.341+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/478348081 shutdown_connections 2023-09-10T01:18:15.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.341+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/478348081 wait complete. 2023-09-10T01:18:15.342 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.341+0000 7fa38e0a6b80 1 -- start start 2023-09-10T01:18:15.343 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.342+0000 7fa38e0a6b80 1 Processor -- start 2023-09-10T01:18:15.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.344+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b1749a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.344+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b1749ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.344+0000 7fa38e0a6b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174f8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.344+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b174f8e50 con 0x559b17479790 2023-09-10T01:18:15.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.344+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b174f8f90 con 0x559b17483ec0 2023-09-10T01:18:15.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.344+0000 7fa38e0a6b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b174f90d0 con 0x559b1747a0b0 2023-09-10T01:18:15.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.344+0000 7fa38e0a6b80 10 client.0 _ll_register_callbacks cb 0x7fa364003a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:15.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.345+0000 7fa378d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.345+0000 7fa378d9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174f8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35118/0 (socket says 172.21.15.3:35118) 2023-09-10T01:18:15.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.345+0000 7fa378d9c700 1 -- 172.21.15.3:0/664276029 learned_addr learned my addr 172.21.15.3:0/664276029 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.346+0000 7fa378d9c700 1 -- 172.21.15.3:0/664276029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 msgr2=0x559b1749ad10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:18:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.346+0000 7fa378d9c700 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b1749ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.346+0000 7fa378d9c700 1 -- 172.21.15.3:0/664276029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 msgr2=0x559b1749a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.346+0000 7fa378d9c700 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b1749a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.346+0000 7fa378d9c700 1 -- 172.21.15.3:0/664276029 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b171e0ce0 con 0x559b17483ec0 2023-09-10T01:18:15.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.349+0000 7fa378d9c700 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174f8910 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa364009fe0 tx=0x7fa364005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8571b405c1ff2e83 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.349+0000 7fa3537fe700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:18:15.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.352+0000 7fa3537fe700 1 -- 172.21.15.3:0/664276029 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364005030 con 0x559b17483ec0 2023-09-10T01:18:15.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.353+0000 7fa3537fe700 1 -- 172.21.15.3:0/664276029 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 9539+0+0 (secure 0 0 0) 0x7fa364000e40 con 0x559b17483ec0 2023-09-10T01:18:15.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.353+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa368004510 con 0x559b17483ec0 2023-09-10T01:18:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.355+0000 7fa3537fe700 1 -- 172.21.15.3:0/664276029 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa364005190 con 0x559b17483ec0 2023-09-10T01:18:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.357+0000 7fa38e0a6b80 10 client.29114 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.357+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x559b171f6330 con 0x559b17483ec0 2023-09-10T01:18:15.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa3537fe700 1 -- 172.21.15.3:0/664276029 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa368004510 con 0x559b17483ec0 2023-09-10T01:18:15.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa3537fe700 1 -- 172.21.15.3:0/664276029 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fa36402fa20 con 0x559b17483ec0 2023-09-10T01:18:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa38e0a6b80 10 client.29114 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa38e0a6b80 10 client.29114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa38e0a6b80 20 client.29114 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:15.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174fa3b0 0x559b174fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa38e0a6b80 4 client.29114 mds_command: new command op to 24698 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-09-10T01:18:15.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.359+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559b173c49a0 con 0x559b174fa3b0 2023-09-10T01:18:15.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.360+0000 7fa3737fe700 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174fa3b0 0x559b174fc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.375+0000 7fa3737fe700 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174fa3b0 0x559b174fc7a0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=755ba6fd52f3be67 server_cookie=3ca2e3abe1979537 in_seq=0 out_seq=0 2023-09-10T01:18:15.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.377+0000 7fa3537fe700 10 client.29114 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.399+0000 7fa3537fe700 1 -- 172.21.15.3:0/664276029 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559b173c49a0 con 0x559b174fa3b0 2023-09-10T01:18:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.399+0000 7fa3537fe700 10 client.29114 handle_command_reply: tid=0 2023-09-10T01:18:15.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.399+0000 7fa38e0a6b80 1 client.29114 shutdown 2023-09-10T01:18:15.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 msgr2=0x559b174f8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174f8910 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa364009fe0 tx=0x7fa364005dc0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 shutdown_connections 2023-09-10T01:18:15.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x559b174fa3b0 0x559b174fc7a0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b17483ec0 0x559b174f8910 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b17479790 0x559b1749a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa3537fe700 0 client.29114 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 --2- 172.21.15.3:0/664276029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559b1747a0b0 0x559b1749ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 >> 172.21.15.3:0/664276029 conn(0x559b173ee110 msgr2=0x559b17478780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 shutdown_connections 2023-09-10T01:18:15.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 1 -- 172.21.15.3:0/664276029 wait complete. 2023-09-10T01:18:15.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.400+0000 7fa38e0a6b80 20 client.29114 trim_cache size 0 max 16384 2023-09-10T01:18:15.405 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:18:15.405 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2023-09-10T01:18:15.406 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "fb82e4cb-5a53-4ccc-81c3-9d3fb173da89", 2023-09-10T01:18:15.406 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2023-09-10T01:18:15.406 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:18:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa35c0a08d0 msgr2=0x7fa35c0a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa35c0a08d0 0x7fa35c0a2d90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559b1746f300 tx=0x7fa36c01df90 comp rx=0 tx=0).stop 2023-09-10T01:18:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740ff690 msgr2=0x7fa37410f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740ff690 0x7fa37410f4d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00b6c0 tx=0x7fa36c004d30 comp rx=0 tx=0).stop 2023-09-10T01:18:15.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 shutdown_connections 2023-09-10T01:18:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa35c0a08d0 0x7fa35c0a2d90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa374104a90 0x7fa3741c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3740ff690 0x7fa37410f4d0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 --2- 172.21.15.3:0/1528634194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3740fffb0 0x7fa37410fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 >> 172.21.15.3:0/1528634194 conn(0x7fa374076710 msgr2=0x7fa3741034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 shutdown_connections 2023-09-10T01:18:15.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.401+0000 7fa37a7ff700 1 -- 172.21.15.3:0/1528634194 wait complete. 2023-09-10T01:18:15.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:18:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.616+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 --> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa4b8116be0 con 0x7fa4a00a09c0 2023-09-10T01:18:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:15 smithi136 ceph-mon[133708]: from='client.18978 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:18:15.666 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.666+0000 7fa4b4ff9700 1 -- 172.21.15.3:0/2810540001 <== mgr.25667 v2:172.21.15.3:6800/3336040255 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fa4b8116be0 con 0x7fa4a00a09c0 2023-09-10T01:18:15.668 INFO:teuthology.orchestra.run.smithi003.stdout:/ 1M 6M3h 2023-09-10T01:18:15.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.674+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a00a09c0 msgr2=0x7fa4a00a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.674+0000 7fa4bddff700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a00a09c0 0x7fa4a00a2e80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa4b810af40 tx=0x7fa4b000d040 comp rx=0 tx=0).stop 2023-09-10T01:18:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.674+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4b80729a0 msgr2=0x7fa4b810d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.674+0000 7fa4bddff700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4b80729a0 0x7fa4b810d4a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa4a800c970 tx=0x7fa4a800cc80 comp rx=0 tx=0).stop 2023-09-10T01:18:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.684+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 shutdown_connections 2023-09-10T01:18:15.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.684+0000 7fa4bddff700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa4a00a09c0 0x7fa4a00a2e80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.684+0000 7fa4bddff700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4b80729a0 0x7fa4b810d4a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.684+0000 7fa4bddff700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4b81115b0 0x7fa4b8109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.684+0000 7fa4bddff700 1 --2- 172.21.15.3:0/2810540001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa4b80732c0 0x7fa4b810d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.684+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 >> 172.21.15.3:0/2810540001 conn(0x7fa4b80fc0b0 msgr2=0x7fa4b80fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.686+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 shutdown_connections 2023-09-10T01:18:15.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.688+0000 7fa4bddff700 1 -- 172.21.15.3:0/2810540001 wait complete. 2023-09-10T01:18:15.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:15 smithi092 ceph-mon[134067]: from='client.18978 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:18:15.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:15 smithi003 ceph-mon[97654]: from='client.18978 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:18:15.726 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2023-09-10T01:18:15.737 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2023-09-10T01:18:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.792+0000 7f7a115ff700 1 -- 172.21.15.3:0/1892610418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c10c000 msgr2=0x7f7a0c113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.793 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.792+0000 7f7a115ff700 1 --2- 172.21.15.3:0/1892610418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c10c000 0x7f7a0c113b80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f79fc009990 tx=0x7f79fc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.806 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.800+0000 7f7a115ff700 1 -- 172.21.15.3:0/1892610418 shutdown_connections 2023-09-10T01:18:15.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.800+0000 7f7a115ff700 1 --2- 172.21.15.3:0/1892610418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c10c000 0x7f7a0c113b80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.807 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.800+0000 7f7a115ff700 1 --2- 172.21.15.3:0/1892610418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a0c106ee0 0x7f7a0c1072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.800+0000 7f7a115ff700 1 --2- 172.21.15.3:0/1892610418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a0c107890 0x7f7a0c10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.808 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.800+0000 7f7a115ff700 1 -- 172.21.15.3:0/1892610418 >> 172.21.15.3:0/1892610418 conn(0x7f7a0c074ba0 msgr2=0x7f7a0c076fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.807+0000 7f7a115ff700 1 -- 172.21.15.3:0/1892610418 shutdown_connections 2023-09-10T01:18:15.809 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.808+0000 7f7a115ff700 1 -- 172.21.15.3:0/1892610418 wait complete. 2023-09-10T01:18:15.813 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.810+0000 7f7a115ff700 1 Processor -- start 2023-09-10T01:18:15.817 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.816+0000 7f7a115ff700 1 -- start start 2023-09-10T01:18:15.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.816+0000 7f7a115ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a0c106ee0 0x7f7a0c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.816+0000 7f7a115ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a0c10c000 0x7f7a0c19d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.816+0000 7f7a115ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c1a1760 0x7f7a0c1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.816+0000 7f7a115ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c1a21e0 con 0x7f7a0c106ee0 2023-09-10T01:18:15.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.816+0000 7f7a115ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c1a2320 con 0x7f7a0c1a1760 2023-09-10T01:18:15.819 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.816+0000 7f7a115ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0c1a2490 con 0x7f7a0c10c000 2023-09-10T01:18:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.817+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c1a1760 0x7f7a0c1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.817+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c1a1760 0x7f7a0c1a1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:35136/0 (socket says 172.21.15.3:35136) 2023-09-10T01:18:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.817+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/2371019536 learned_addr learned my addr 172.21.15.3:0/2371019536 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.817+0000 7f7a0a7fc700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a0c10c000 0x7f7a0c19d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.821 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.820+0000 7f7a0affd700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a0c106ee0 0x7f7a0c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.824 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.822+0000 7f7a0affd700 1 -- 172.21.15.3:0/2371019536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a0c10c000 msgr2=0x7f7a0c19d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.822+0000 7f7a0affd700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a0c10c000 0x7f7a0c19d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.822+0000 7f7a0affd700 1 -- 172.21.15.3:0/2371019536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c1a1760 msgr2=0x7f7a0c1a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.825 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.822+0000 7f7a0affd700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c1a1760 0x7f7a0c1a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.822+0000 7f7a0affd700 1 -- 172.21.15.3:0/2371019536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79fc0095f0 con 0x7f7a0c106ee0 2023-09-10T01:18:15.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.823+0000 7f7a0affd700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a0c106ee0 0x7f7a0c19cdc0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400bd70 tx=0x7f7a0400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.826 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.824+0000 7f79f3fff700 1 -- 172.21.15.3:0/2371019536 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0400cc90 con 0x7f7a0c106ee0 2023-09-10T01:18:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.824+0000 7f79f3fff700 1 -- 172.21.15.3:0/2371019536 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a0400cdf0 con 0x7f7a0c106ee0 2023-09-10T01:18:15.827 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.824+0000 7f79f3fff700 1 -- 172.21.15.3:0/2371019536 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04012740 con 0x7f7a0c106ee0 2023-09-10T01:18:15.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.825+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0c06a020 con 0x7f7a0c106ee0 2023-09-10T01:18:15.830 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.825+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0c06a4e0 con 0x7f7a0c106ee0 2023-09-10T01:18:15.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.832+0000 7f7a25005b80 1 Processor -- start 2023-09-10T01:18:15.838 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.836+0000 7f79f3fff700 1 -- 172.21.15.3:0/2371019536 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7a040128c0 con 0x7f7a0c106ee0 2023-09-10T01:18:15.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.838+0000 7f79f3fff700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f40a09c0 0x7f79f40a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.838+0000 7f79f3fff700 1 -- 172.21.15.3:0/2371019536 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f7a04006e40 con 0x7f7a0c106ee0 2023-09-10T01:18:15.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a25005b80 1 -- start start 2023-09-10T01:18:15.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f40a09c0 0x7f79f40a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b55af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b56030 0x5635f4b5f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b62230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a25005b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4aae730 con 0x5635f4b55710 2023-09-10T01:18:15.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a25005b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4aac050 con 0x5635f4b56030 2023-09-10T01:18:15.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.839+0000 7f7a25005b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4aa7940 con 0x5635f4b5fe40 2023-09-10T01:18:15.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.840+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b62230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.840+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b62230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41580/0 (socket says 172.21.15.3:41580) 2023-09-10T01:18:15.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.840+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/1398742564 learned_addr learned my addr 172.21.15.3:0/1398742564 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.841+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/1398742564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b56030 msgr2=0x5635f4b5f900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-10T01:18:15.844 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.841+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/1398742564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b56030 0x5635f4b5f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.841+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/1398742564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 msgr2=0x5635f4b55af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.841+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/1398742564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b55af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.845 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.841+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/1398742564 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635f48d3de0 con 0x5635f4b5fe40 2023-09-10T01:18:15.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.842+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/1398742564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b62230 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79fc005cb0 tx=0x7f79fc029000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55af8b999931944 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.842+0000 7f79f17fa700 1 -- 172.21.15.3:0/1398742564 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc004ba0 con 0x5635f4b5fe40 2023-09-10T01:18:15.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.842+0000 7f79f17fa700 1 -- 172.21.15.3:0/1398742564 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79fc01e040 con 0x5635f4b5fe40 2023-09-10T01:18:15.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.842+0000 7f79f17fa700 1 -- 172.21.15.3:0/1398742564 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc00f460 con 0x5635f4b5fe40 2023-09-10T01:18:15.847 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.843+0000 7f7a25005b80 1 -- 172.21.15.3:0/1398742564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 msgr2=0x5635f4b62230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.843+0000 7f7a25005b80 1 --2- 172.21.15.3:0/1398742564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b62230 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79fc005cb0 tx=0x7f79fc029000 comp rx=0 tx=0).stop 2023-09-10T01:18:15.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.843+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f40a09c0 0x7f79f40a2e80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00b7e0 tx=0x7f79fc005550 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.848 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.844+0000 7f7a25005b80 1 -- 172.21.15.3:0/1398742564 shutdown_connections 2023-09-10T01:18:15.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.844+0000 7f7a25005b80 1 --2- 172.21.15.3:0/1398742564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b62230 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.844+0000 7f7a25005b80 1 --2- 172.21.15.3:0/1398742564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b55af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.844+0000 7f7a25005b80 1 --2- 172.21.15.3:0/1398742564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b56030 0x5635f4b5f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.844+0000 7f7a25005b80 1 -- 172.21.15.3:0/1398742564 >> 172.21.15.3:0/1398742564 conn(0x5635f4aca090 msgr2=0x5635f4b50bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.848+0000 7f7a25005b80 1 -- 172.21.15.3:0/1398742564 shutdown_connections 2023-09-10T01:18:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.849+0000 7f7a25005b80 1 -- 172.21.15.3:0/1398742564 wait complete. 2023-09-10T01:18:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.849+0000 7f7a25005b80 1 -- start start 2023-09-10T01:18:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.850+0000 7f7a25005b80 1 Processor -- start 2023-09-10T01:18:15.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b5cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.853 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b5d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4b6bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a25005b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4a45b70 con 0x5635f4b55710 2023-09-10T01:18:15.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a25005b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4a45870 con 0x5635f4b5d640 2023-09-10T01:18:15.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a25005b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4929920 con 0x5635f4b5fe40 2023-09-10T01:18:15.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a25005b80 10 client.0 _ll_register_callbacks cb 0x5635f498fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:15.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.851+0000 7f7a0a7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b5d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.852+0000 7f7a0a7fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b5d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41582/0 (socket says 172.21.15.3:41582) 2023-09-10T01:18:15.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.852+0000 7f7a0a7fc700 1 -- 172.21.15.3:0/741787949 learned_addr learned my addr 172.21.15.3:0/741787949 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.853+0000 7f7a0a7fc700 1 -- 172.21.15.3:0/741787949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 msgr2=0x5635f4b6bae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:18:15.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.853+0000 7f7a0a7fc700 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4b6bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.853+0000 7f7a0a7fc700 1 -- 172.21.15.3:0/741787949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 msgr2=0x5635f4b5cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.853+0000 7f7a0a7fc700 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b5cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.853+0000 7f7a0a7fc700 1 -- 172.21.15.3:0/741787949 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0c1143f0 con 0x5635f4b5fe40 2023-09-10T01:18:15.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.858+0000 7f7a0a7fc700 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b5d100 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7a00011610 tx=0x7f7a000119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1325bd1ad8add019 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.858+0000 7f79eeff5700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:18:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.860+0000 7f79eeff5700 1 -- 172.21.15.3:0/741787949 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00004440 con 0x5635f4b5fe40 2023-09-10T01:18:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.860+0000 7f79eeff5700 1 -- 172.21.15.3:0/741787949 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f7a00003680 con 0x5635f4b5fe40 2023-09-10T01:18:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.860+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635f49ff4f0 con 0x5635f4b5fe40 2023-09-10T01:18:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.861+0000 7f79eeff5700 1 -- 172.21.15.3:0/741787949 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a000045a0 con 0x5635f4b5fe40 2023-09-10T01:18:15.865 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.865+0000 7f7a25005b80 10 client.28993 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:15.866 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.865+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5635f48bcc60 con 0x5635f4b5fe40 2023-09-10T01:18:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f79eeff5700 1 -- 172.21.15.3:0/741787949 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5635f49ff4f0 con 0x5635f4b5fe40 2023-09-10T01:18:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f79eeff5700 1 -- 172.21.15.3:0/741787949 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7a0001ba70 con 0x5635f4b5fe40 2023-09-10T01:18:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f7a25005b80 10 client.28993 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:15.874 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f7a25005b80 10 client.28993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f7a25005b80 20 client.28993 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f7a25005b80 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4b6ff70 0x5635f4b72390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f7a25005b80 4 client.28993 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:15.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.873+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635f4abc8a0 con 0x5635f4b6ff70 2023-09-10T01:18:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.890+0000 7f7a0affd700 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4b6ff70 0x5635f4b72390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.906+0000 7f7a0affd700 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4b6ff70 0x5635f4b72390 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e245c11f1a3166 server_cookie=e587672322ee76b1 in_seq=0 out_seq=0 2023-09-10T01:18:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.907+0000 7f79eeff5700 10 client.28993 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:15.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.910+0000 7f79eeff5700 1 -- 172.21.15.3:0/741787949 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5635f4abc8a0 con 0x5635f4b6ff70 2023-09-10T01:18:15.911 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.910+0000 7f79eeff5700 10 client.28993 handle_command_reply: tid=0 2023-09-10T01:18:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.911+0000 7f7a25005b80 1 client.28993 shutdown 2023-09-10T01:18:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.911+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 msgr2=0x5635f4b5d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.912 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.911+0000 7f7a25005b80 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b5d100 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7a00011610 tx=0x7f7a000119d0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.912+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 shutdown_connections 2023-09-10T01:18:15.913 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.912+0000 7f7a25005b80 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4b6ff70 0x5635f4b72390 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.912+0000 7f7a25005b80 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4b6bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.912+0000 7f7a25005b80 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b5cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.912+0000 7f7a25005b80 1 --2- 172.21.15.3:0/741787949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b5d100 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.914 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.912+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 >> 172.21.15.3:0/741787949 conn(0x5635f4aca090 msgr2=0x5635f4b50780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.912+0000 7f79eeff5700 0 client.28993 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:15.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.914+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 shutdown_connections 2023-09-10T01:18:15.916 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.914+0000 7f7a25005b80 1 -- 172.21.15.3:0/741787949 wait complete. 2023-09-10T01:18:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.920+0000 7f7a25005b80 20 client.28993 trim_cache size 0 max 16384 2023-09-10T01:18:15.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.953+0000 7f7a25005b80 1 Processor -- start 2023-09-10T01:18:15.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.955+0000 7f7a25005b80 1 -- start start 2023-09-10T01:18:15.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.955+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b66a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.955+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4b57e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.955+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b58340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.955+0000 7f7a25005b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4b569f0 con 0x5635f4b55710 2023-09-10T01:18:15.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.955+0000 7f7a25005b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4b52090 con 0x5635f4b5d640 2023-09-10T01:18:15.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.955+0000 7f7a25005b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4b4d3d0 con 0x5635f4b5fe40 2023-09-10T01:18:15.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.957+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b58340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.957+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b58340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41590/0 (socket says 172.21.15.3:41590) 2023-09-10T01:18:15.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.957+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/4064927414 learned_addr learned my addr 172.21.15.3:0/4064927414 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/4064927414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 msgr2=0x5635f4b57e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0a7fc700 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4b57e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4b57e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0affd700 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b66a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/4064927414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 msgr2=0x5635f4b66a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b66a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/4064927414 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635f48fa250 con 0x5635f4b5fe40 2023-09-10T01:18:15.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0affd700 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b66a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:18:15.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.959+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b58340 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0098e0 tx=0x7f79fc00b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1468e52313937bb4 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.965+0000 7f79f1ffb700 1 -- 172.21.15.3:0/4064927414 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc005160 con 0x5635f4b5fe40 2023-09-10T01:18:15.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.966+0000 7f79f1ffb700 1 -- 172.21.15.3:0/4064927414 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79fc0052c0 con 0x5635f4b5fe40 2023-09-10T01:18:15.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.966+0000 7f79f1ffb700 1 -- 172.21.15.3:0/4064927414 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc0296d0 con 0x5635f4b5fe40 2023-09-10T01:18:15.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.967+0000 7f7a25005b80 1 -- 172.21.15.3:0/4064927414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 msgr2=0x5635f4b58340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:15.970 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.967+0000 7f7a25005b80 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b58340 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0098e0 tx=0x7f79fc00b410 comp rx=0 tx=0).stop 2023-09-10T01:18:15.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.978+0000 7f7a25005b80 1 -- 172.21.15.3:0/4064927414 shutdown_connections 2023-09-10T01:18:15.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.978+0000 7f7a25005b80 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b5fe40 0x5635f4b58340 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.978+0000 7f7a25005b80 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b55710 0x5635f4b66a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.978+0000 7f7a25005b80 1 --2- 172.21.15.3:0/4064927414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4b57e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.978+0000 7f7a25005b80 1 -- 172.21.15.3:0/4064927414 >> 172.21.15.3:0/4064927414 conn(0x5635f4aca090 msgr2=0x5635f4b4fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:15.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.978+0000 7f7a25005b80 1 -- 172.21.15.3:0/4064927414 shutdown_connections 2023-09-10T01:18:15.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.979+0000 7f7a25005b80 1 -- 172.21.15.3:0/4064927414 wait complete. 2023-09-10T01:18:15.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.979+0000 7f7a25005b80 1 -- start start 2023-09-10T01:18:15.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.979+0000 7f7a25005b80 1 Processor -- start 2023-09-10T01:18:15.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.980+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b55710 0x5635f4bbe2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.982 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.980+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4bbe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.980+0000 7f7a25005b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b5fe40 0x5635f4bb7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:15.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.980+0000 7f7a25005b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4bb85a0 con 0x5635f4b5fe40 2023-09-10T01:18:15.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.980+0000 7f7a25005b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4bb8710 con 0x5635f4b5d640 2023-09-10T01:18:15.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.980+0000 7f7a25005b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635f4bb88d0 con 0x5635f4b55710 2023-09-10T01:18:15.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.980+0000 7f7a25005b80 10 client.0 _ll_register_callbacks cb 0x7f7a00009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:15.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.983+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b5fe40 0x5635f4bb7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.983+0000 7f7a0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b5fe40 0x5635f4bb7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45092/0 (socket says 172.21.15.3:45092) 2023-09-10T01:18:15.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.983+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/2861461693 learned_addr learned my addr 172.21.15.3:0/2861461693 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:15.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.983+0000 7f7a0affd700 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b55710 0x5635f4bbe2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:15.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.984+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/2861461693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b55710 msgr2=0x5635f4bbe2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:15.986 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.984+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b55710 0x5635f4bbe2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.984+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/2861461693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 msgr2=0x5635f4bbe7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-10T01:18:15.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.984+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4bbe7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:15.987 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.984+0000 7f7a0b7fe700 1 -- 172.21.15.3:0/2861461693 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635f48d3de0 con 0x5635f4b5fe40 2023-09-10T01:18:15.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.984+0000 7f7a0b7fe700 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b5fe40 0x5635f4bb7e90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00bd10 tx=0x7f79fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6048a2c503868c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:15.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.992+0000 7f79eeff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:15.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.992+0000 7f79eeff5700 1 -- 172.21.15.3:0/2861461693 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc00bea0 con 0x5635f4b5fe40 2023-09-10T01:18:15.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.992+0000 7f79eeff5700 1 -- 172.21.15.3:0/2861461693 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f79fc004b70 con 0x5635f4b5fe40 2023-09-10T01:18:15.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.992+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79fc01e040 con 0x5635f4b5fe40 2023-09-10T01:18:15.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:15.994+0000 7f79eeff5700 1 -- 172.21.15.3:0/2861461693 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79fc004d30 con 0x5635f4b5fe40 2023-09-10T01:18:16.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.002+0000 7f7a25005b80 10 client.18999 fetch_fsmap learned FSMap version 38 2023-09-10T01:18:16.004 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.003+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5635f48d22b0 con 0x5635f4b5fe40 2023-09-10T01:18:16.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.008+0000 7f79eeff5700 1 -- 172.21.15.3:0/2861461693 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79fc01e040 con 0x5635f4b5fe40 2023-09-10T01:18:16.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.008+0000 7f79eeff5700 1 -- 172.21.15.3:0/2861461693 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f79fc03d030 con 0x5635f4b5fe40 2023-09-10T01:18:16.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.009+0000 7f7a25005b80 10 client.18999 fetch_fsmap finished waiting for FSMap version 38 2023-09-10T01:18:16.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.009+0000 7f7a25005b80 10 client.18999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:16.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.009+0000 7f7a25005b80 20 client.18999 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:16.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.009+0000 7f7a25005b80 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4bd5db0 0x5635f4bd81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:16.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.009+0000 7f7a25005b80 4 client.18999 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:18:16.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.009+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5635f4aa0920 con 0x5635f4bd5db0 2023-09-10T01:18:16.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.010+0000 7f7a0affd700 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4bd5db0 0x5635f4bd81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:16.015 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.014+0000 7f7a0affd700 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4bd5db0 0x5635f4bd81a0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a82b7ab65c5f544c server_cookie=a47812377c0a6070 in_seq=0 out_seq=0 2023-09-10T01:18:16.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.017+0000 7f79eeff5700 10 client.18999 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:16.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.017+0000 7f79eeff5700 1 -- 172.21.15.3:0/2861461693 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5635f4aa0920 con 0x5635f4bd5db0 2023-09-10T01:18:16.018 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.017+0000 7f79eeff5700 10 client.18999 handle_command_reply: tid=0 2023-09-10T01:18:16.019 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.017+0000 7f7a25005b80 1 client.18999 shutdown 2023-09-10T01:18:16.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b5fe40 msgr2=0x5635f4bb7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:16.024 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b5fe40 0x5635f4bb7e90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00bd10 tx=0x7f79fc0057f0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 shutdown_connections 2023-09-10T01:18:16.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5635f4bd5db0 0x5635f4bd81a0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5635f4b55710 0x5635f4bbe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.025 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5635f4b5fe40 0x5635f4bb7e90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 --2- 172.21.15.3:0/2861461693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5635f4b5d640 0x5635f4bbe7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.026 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.023+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 >> 172.21.15.3:0/2861461693 conn(0x5635f4aca090 msgr2=0x5635f4b57e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:16.027 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.025+0000 7f79eeff5700 0 client.18999 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:16.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.028+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 shutdown_connections 2023-09-10T01:18:16.031 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.028+0000 7f7a25005b80 1 -- 172.21.15.3:0/2861461693 wait complete. 2023-09-10T01:18:16.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.030+0000 7f7a25005b80 20 client.18999 trim_cache size 0 max 16384 2023-09-10T01:18:16.042 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:18:16.042 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2023-09-10T01:18:16.043 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2023-09-10T01:18:16.043 INFO:teuthology.orchestra.run.smithi003.stdout: "fb82e4cb-5a53-4ccc-81c3-9d3fb173da89": { 2023-09-10T01:18:16.043 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2023-09-10T01:18:16.043 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "fb82e4cb-5a53-4ccc-81c3-9d3fb173da89", 2023-09-10T01:18:16.044 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2023-09-10T01:18:16.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:16.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:16.045 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:18:16.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.035+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f40a09c0 msgr2=0x7f79f40a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:16.045 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.035+0000 7f7a115ff700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f40a09c0 0x7f79f40a2e80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00b7e0 tx=0x7f79fc005550 comp rx=0 tx=0).stop 2023-09-10T01:18:16.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.035+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a0c106ee0 msgr2=0x7f7a0c19cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:16.046 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.035+0000 7f7a115ff700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a0c106ee0 0x7f7a0c19cdc0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400bd70 tx=0x7f7a0400bda0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.043+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 shutdown_connections 2023-09-10T01:18:16.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.043+0000 7f7a115ff700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f79f40a09c0 0x7f79f40a2e80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.047 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.043+0000 7f7a115ff700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a0c1a1760 0x7f7a0c1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.043+0000 7f7a115ff700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a0c106ee0 0x7f7a0c19cdc0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.043+0000 7f7a115ff700 1 --2- 172.21.15.3:0/2371019536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a0c10c000 0x7f7a0c19d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:16.048 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.043+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 >> 172.21.15.3:0/2371019536 conn(0x7f7a0c074ba0 msgr2=0x7f7a0c0755f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:16.053 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.053+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 shutdown_connections 2023-09-10T01:18:16.065 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:16.065+0000 7f7a115ff700 1 -- 172.21.15.3:0/2371019536 wait complete. 2023-09-10T01:18:16.109 INFO:tasks.cephfs.filesystem:scrub status for tag:fb82e4cb-5a53-4ccc-81c3-9d3fb173da89 - {'path': '/', 'tag': 'fb82e4cb-5a53-4ccc-81c3-9d3fb173da89', 'options': 'recursive,force'} 2023-09-10T01:18:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: pgmap v472: 129 pgs: 129 active+clean; 842 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 415 op/s 2023-09-10T01:18:16.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: osdmap e110: 12 total, 12 up, 12 in 2023-09-10T01:18:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: scrub queued for path: / 2023-09-10T01:18:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:18:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:18:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: scrub summary: active paths [/] 2023-09-10T01:18:16.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:16 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: pgmap v472: 129 pgs: 129 active+clean; 842 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 415 op/s 2023-09-10T01:18:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: osdmap e110: 12 total, 12 up, 12 in 2023-09-10T01:18:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: scrub queued for path: / 2023-09-10T01:18:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:18:16.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:18:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: scrub summary: active paths [/] 2023-09-10T01:18:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:16 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: pgmap v472: 129 pgs: 129 active+clean; 842 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 415 op/s 2023-09-10T01:18:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: osdmap e110: 12 total, 12 up, 12 in 2023-09-10T01:18:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: scrub queued for path: / 2023-09-10T01:18:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:18:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:18:16.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: scrub summary: active paths [/] 2023-09-10T01:18:16.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:16.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:16 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:17.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:17 smithi136 ceph-mon[133708]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:17.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:17 smithi136 ceph-mon[133708]: from='client.28975 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:18:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:17 smithi092 ceph-mon[134067]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:17.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:17 smithi092 ceph-mon[134067]: from='client.28975 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:18:17.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:17 smithi003 ceph-mon[97654]: scrub summary: idle+waiting paths [/] 2023-09-10T01:18:17.843 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:17 smithi003 ceph-mon[97654]: from='client.28975 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-09-10T01:18:18.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:18 smithi136 ceph-mon[133708]: pgmap v474: 129 pgs: 129 active+clean; 842 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 419 op/s 2023-09-10T01:18:18.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:18 smithi092 ceph-mon[134067]: pgmap v474: 129 pgs: 129 active+clean; 842 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 419 op/s 2023-09-10T01:18:18.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:18 smithi003 ceph-mon[97654]: pgmap v474: 129 pgs: 129 active+clean; 842 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 419 op/s 2023-09-10T01:18:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:20 smithi136 ceph-mon[133708]: pgmap v475: 129 pgs: 129 active+clean; 831 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 391 op/s 2023-09-10T01:18:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:20 smithi136 ceph-mon[133708]: osdmap e111: 12 total, 12 up, 12 in 2023-09-10T01:18:20.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:20 smithi092 ceph-mon[134067]: pgmap v475: 129 pgs: 129 active+clean; 831 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 391 op/s 2023-09-10T01:18:20.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:20 smithi092 ceph-mon[134067]: osdmap e111: 12 total, 12 up, 12 in 2023-09-10T01:18:20.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:20 smithi003 ceph-mon[97654]: pgmap v475: 129 pgs: 129 active+clean; 831 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 391 op/s 2023-09-10T01:18:20.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:20 smithi003 ceph-mon[97654]: osdmap e111: 12 total, 12 up, 12 in 2023-09-10T01:18:22.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:22 smithi136 ceph-mon[133708]: pgmap v477: 129 pgs: 129 active+clean; 812 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 703 op/s 2023-09-10T01:18:22.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:22 smithi092 ceph-mon[134067]: pgmap v477: 129 pgs: 129 active+clean; 812 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 703 op/s 2023-09-10T01:18:22.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:22 smithi003 ceph-mon[97654]: pgmap v477: 129 pgs: 129 active+clean; 812 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 703 op/s 2023-09-10T01:18:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:24 smithi136 ceph-mon[133708]: pgmap v478: 129 pgs: 129 active+clean; 812 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.3 MiB/s wr, 379 op/s 2023-09-10T01:18:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:24 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:18:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:24 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:18:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:24 smithi092 ceph-mon[134067]: pgmap v478: 129 pgs: 129 active+clean; 812 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.3 MiB/s wr, 379 op/s 2023-09-10T01:18:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:24 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:18:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:24 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:18:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:24 smithi003 ceph-mon[97654]: pgmap v478: 129 pgs: 129 active+clean; 812 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.3 MiB/s wr, 379 op/s 2023-09-10T01:18:24.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:24 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:18:24.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:24 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:18:25.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:25 smithi136 ceph-mon[133708]: scrub summary: idle 2023-09-10T01:18:25.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:25 smithi092 ceph-mon[134067]: scrub summary: idle 2023-09-10T01:18:25.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:25 smithi003 ceph-mon[97654]: scrub summary: idle 2023-09-10T01:18:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:26 smithi136 ceph-mon[133708]: pgmap v479: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.7 MiB/s wr, 484 op/s 2023-09-10T01:18:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:26 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:26.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:26 smithi092 ceph-mon[134067]: pgmap v479: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.7 MiB/s wr, 484 op/s 2023-09-10T01:18:26.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:26 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:26.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:26 smithi003 ceph-mon[97654]: pgmap v479: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.7 MiB/s wr, 484 op/s 2023-09-10T01:18:26.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:26 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:28.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:28 smithi136 ceph-mon[133708]: pgmap v480: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.0 MiB/s wr, 445 op/s 2023-09-10T01:18:28.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:28 smithi092 ceph-mon[134067]: pgmap v480: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.0 MiB/s wr, 445 op/s 2023-09-10T01:18:28.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:28 smithi003 ceph-mon[97654]: pgmap v480: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.0 MiB/s wr, 445 op/s 2023-09-10T01:18:30.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:30 smithi136 ceph-mon[133708]: pgmap v481: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.7 MiB/s wr, 419 op/s 2023-09-10T01:18:30.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:30 smithi092 ceph-mon[134067]: pgmap v481: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.7 MiB/s wr, 419 op/s 2023-09-10T01:18:30.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:30 smithi003 ceph-mon[97654]: pgmap v481: 129 pgs: 129 active+clean; 820 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 8.7 MiB/s wr, 419 op/s 2023-09-10T01:18:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:32 smithi136 ceph-mon[133708]: pgmap v482: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 4.0 MiB/s wr, 186 op/s 2023-09-10T01:18:32.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:32 smithi092 ceph-mon[134067]: pgmap v482: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 4.0 MiB/s wr, 186 op/s 2023-09-10T01:18:32.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:32 smithi003 ceph-mon[97654]: pgmap v482: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 4.0 MiB/s wr, 186 op/s 2023-09-10T01:18:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:34 smithi136 ceph-mon[133708]: pgmap v483: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 3.4 MiB/s wr, 158 op/s 2023-09-10T01:18:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:34 smithi092 ceph-mon[134067]: pgmap v483: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 3.4 MiB/s wr, 158 op/s 2023-09-10T01:18:34.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:34 smithi003 ceph-mon[97654]: pgmap v483: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 3.4 MiB/s wr, 158 op/s 2023-09-10T01:18:36.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:36 smithi136 ceph-mon[133708]: pgmap v484: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 3.4 MiB/s wr, 164 op/s 2023-09-10T01:18:36.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:36 smithi092 ceph-mon[134067]: pgmap v484: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 3.4 MiB/s wr, 164 op/s 2023-09-10T01:18:36.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:36 smithi003 ceph-mon[97654]: pgmap v484: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 3.4 MiB/s wr, 164 op/s 2023-09-10T01:18:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:38 smithi136 ceph-mon[133708]: pgmap v485: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 141 KiB/s wr, 46 op/s 2023-09-10T01:18:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:38 smithi092 ceph-mon[134067]: pgmap v485: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 141 KiB/s wr, 46 op/s 2023-09-10T01:18:38.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:38 smithi003 ceph-mon[97654]: pgmap v485: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 141 KiB/s wr, 46 op/s 2023-09-10T01:18:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:40 smithi136 ceph-mon[133708]: pgmap v486: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 153 KiB/s wr, 48 op/s 2023-09-10T01:18:40.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:40 smithi092 ceph-mon[134067]: pgmap v486: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 153 KiB/s wr, 48 op/s 2023-09-10T01:18:40.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:40 smithi003 ceph-mon[97654]: pgmap v486: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 153 KiB/s wr, 48 op/s 2023-09-10T01:18:42.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:42 smithi136 ceph-mon[133708]: pgmap v487: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 86 KiB/s wr, 41 op/s 2023-09-10T01:18:42.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:42 smithi092 ceph-mon[134067]: pgmap v487: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 86 KiB/s wr, 41 op/s 2023-09-10T01:18:42.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:42 smithi003 ceph-mon[97654]: pgmap v487: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 86 KiB/s wr, 41 op/s 2023-09-10T01:18:43.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:43 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:18:43.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:43 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:18:43.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:43 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-09-10T01:18:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:44 smithi136 ceph-mon[133708]: pgmap v488: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-09-10T01:18:44.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:44 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:18:44.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:44 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:18:44.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:44 smithi136 ceph-mon[133708]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:18:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:44 smithi092 ceph-mon[134067]: pgmap v488: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-09-10T01:18:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:44 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:18:44.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:44 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:18:44.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:44 smithi092 ceph-mon[134067]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:18:44.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:44 smithi003 ceph-mon[97654]: pgmap v488: 129 pgs: 129 active+clean; 808 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-09-10T01:18:44.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:44 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-09-10T01:18:44.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:44 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-09-10T01:18:44.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:44 smithi003 ceph-mon[97654]: from='mgr.25667 ' entity='mgr.x' 2023-09-10T01:18:46.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-09-10T01:18:46.328 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 -- 172.21.15.3:0/1428442356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c072a30 msgr2=0x7f364c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 --2- 172.21.15.3:0/1428442356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c072a30 0x7f364c072e10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3640009990 tx=0x7f3640009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 -- 172.21.15.3:0/1428442356 shutdown_connections 2023-09-10T01:18:46.329 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 --2- 172.21.15.3:0/1428442356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c10ce80 0x7f364c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 --2- 172.21.15.3:0/1428442356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c072a30 0x7f364c072e10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 --2- 172.21.15.3:0/1428442356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c073350 0x7f364c10c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.330 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 -- 172.21.15.3:0/1428442356 >> 172.21.15.3:0/1428442356 conn(0x7f364c0fc170 msgr2=0x7f364c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 -- 172.21.15.3:0/1428442356 shutdown_connections 2023-09-10T01:18:46.331 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.327+0000 7f36531ff700 1 -- 172.21.15.3:0/1428442356 wait complete. 2023-09-10T01:18:46.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.328+0000 7f36531ff700 1 Processor -- start 2023-09-10T01:18:46.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f36531ff700 1 -- start start 2023-09-10T01:18:46.333 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f36531ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c072a30 0x7f364c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f36531ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c073350 0x7f364c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f36531ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c10ce80 0x7f364c198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f36531ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c198ea0 con 0x7f364c10ce80 2023-09-10T01:18:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f36531ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c199010 con 0x7f364c072a30 2023-09-10T01:18:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f36531ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c1991d0 con 0x7f364c073350 2023-09-10T01:18:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f365179c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c10ce80 0x7f364c198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f365179c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c10ce80 0x7f364c198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34658/0 (socket says 172.21.15.3:34658) 2023-09-10T01:18:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f365179c700 1 -- 172.21.15.3:0/1808104901 learned_addr learned my addr 172.21.15.3:0/1808104901 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c072a30 0x7f364c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.329+0000 7f364bfff700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c073350 0x7f364c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f365179c700 1 -- 172.21.15.3:0/1808104901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c073350 msgr2=0x7f364c19ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f365179c700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c073350 0x7f364c19ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f365179c700 1 -- 172.21.15.3:0/1808104901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c072a30 msgr2=0x7f364c19e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f365179c700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c072a30 0x7f364c19e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f365179c700 1 -- 172.21.15.3:0/1808104901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400095f0 con 0x7f364c10ce80 2023-09-10T01:18:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f365179c700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c10ce80 0x7f364c198830 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3644007ee0 tx=0x7f364400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f3649ffb700 1 -- 172.21.15.3:0/1808104901 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364400dd20 con 0x7f364c10ce80 2023-09-10T01:18:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f3649ffb700 1 -- 172.21.15.3:0/1808104901 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f364400f040 con 0x7f364c10ce80 2023-09-10T01:18:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c1994b0 con 0x7f364c10ce80 2023-09-10T01:18:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f3649ffb700 1 -- 172.21.15.3:0/1808104901 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440048a0 con 0x7f364c10ce80 2023-09-10T01:18:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.330+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c1a57b0 con 0x7f364c10ce80 2023-09-10T01:18:46.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.335+0000 7f3649ffb700 1 -- 172.21.15.3:0/1808104901 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3644003a00 con 0x7f364c10ce80 2023-09-10T01:18:46.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 Processor -- start 2023-09-10T01:18:46.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 -- start start 2023-09-10T01:18:46.344 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b45ffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4600030 0x5594b4609900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4609e40 0x5594b460c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.345 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b4558730 con 0x5594b45ff710 2023-09-10T01:18:46.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b4556050 con 0x5594b4600030 2023-09-10T01:18:46.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3666a60b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b4551940 con 0x5594b4609e40 2023-09-10T01:18:46.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.336+0000 7f3649ffb700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f363c0a07b0 0x7f363c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.346 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3649ffb700 1 -- 172.21.15.3:0/1808104901 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f36440cbba0 con 0x7f364c10ce80 2023-09-10T01:18:46.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f364bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b45ffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4600030 0x5594b4609900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f365179c700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f363c0a07b0 0x7f363c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.347 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4600030 0x5594b4609900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36490/0 (socket says 172.21.15.3:36490) 2023-09-10T01:18:46.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 -- 172.21.15.3:0/4217535047 learned_addr learned my addr 172.21.15.3:0/4217535047 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:46.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f364bfff700 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4609e40 0x5594b460c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.348 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 -- 172.21.15.3:0/4217535047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4609e40 msgr2=0x5594b460c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4609e40 0x5594b460c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 -- 172.21.15.3:0/4217535047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 msgr2=0x5594b45ffaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b45ffaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.349 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f3650f9b700 1 -- 172.21.15.3:0/4217535047 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594b437dde0 con 0x5594b4600030 2023-09-10T01:18:46.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f364bfff700 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b45ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:18:46.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.337+0000 7f365179c700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f363c0a07b0 0x7f363c0a2c70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3644004a20 tx=0x7f3644021000 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.350 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.338+0000 7f3650f9b700 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4600030 0x5594b4609900 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3640005f50 tx=0x7f364000b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=150e8c85081d04b2 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.338+0000 7f3636ffd700 1 -- 172.21.15.3:0/4217535047 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364001d070 con 0x5594b4600030 2023-09-10T01:18:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.338+0000 7f3636ffd700 1 -- 172.21.15.3:0/4217535047 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f364001e040 con 0x5594b4600030 2023-09-10T01:18:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.338+0000 7f3636ffd700 1 -- 172.21.15.3:0/4217535047 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640004af0 con 0x5594b4600030 2023-09-10T01:18:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.338+0000 7f3666a60b80 1 -- 172.21.15.3:0/4217535047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4600030 msgr2=0x5594b4609900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.338+0000 7f3666a60b80 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4600030 0x5594b4609900 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3640005f50 tx=0x7f364000b610 comp rx=0 tx=0).stop 2023-09-10T01:18:46.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 -- 172.21.15.3:0/4217535047 shutdown_connections 2023-09-10T01:18:46.352 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4609e40 0x5594b460c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b45ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 --2- 172.21.15.3:0/4217535047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4600030 0x5594b4609900 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 -- 172.21.15.3:0/4217535047 >> 172.21.15.3:0/4217535047 conn(0x5594b4574090 msgr2=0x5594b45fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.353 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 -- 172.21.15.3:0/4217535047 shutdown_connections 2023-09-10T01:18:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 -- 172.21.15.3:0/4217535047 wait complete. 2023-09-10T01:18:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.339+0000 7f3666a60b80 1 -- start start 2023-09-10T01:18:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.340+0000 7f3666a60b80 1 Processor -- start 2023-09-10T01:18:46.354 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.340+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b4609810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b4606a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b4607030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.355 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3666a60b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b44efb70 con 0x5594b45ff710 2023-09-10T01:18:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3666a60b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b44ef870 con 0x5594b4609e40 2023-09-10T01:18:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3666a60b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b43d3920 con 0x5594b4600030 2023-09-10T01:18:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3666a60b80 10 client.0 _ll_register_callbacks cb 0x5594b4439a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f364bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b4609810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f364bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b4609810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34680/0 (socket says 172.21.15.3:34680) 2023-09-10T01:18:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f364bfff700 1 -- 172.21.15.3:0/1697507747 learned_addr learned my addr 172.21.15.3:0/1697507747 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b4606a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f364bfff700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b4607030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3650f9b700 1 -- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 msgr2=0x5594b4607030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b4607030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.358 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3650f9b700 1 -- 172.21.15.3:0/1697507747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 msgr2=0x5594b4609810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b4609810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f3650f9b700 1 -- 172.21.15.3:0/1697507747 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c10ffb0 con 0x5594b4600030 2023-09-10T01:18:46.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.341+0000 7f364bfff700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b4609810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:46.359 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b4606a20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f36400098e0 tx=0x7f364000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d5de2ca2832a4d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f362bfff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:18:46.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f362bfff700 1 -- 172.21.15.3:0/1697507747 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364001d040 con 0x5594b4600030 2023-09-10T01:18:46.360 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594b44a94f0 con 0x5594b4600030 2023-09-10T01:18:46.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f362bfff700 1 -- 172.21.15.3:0/1697507747 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f364000fc90 con 0x5594b4600030 2023-09-10T01:18:46.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f362bfff700 1 -- 172.21.15.3:0/1697507747 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f364000fe90 con 0x5594b4600030 2023-09-10T01:18:46.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f3666a60b80 10 client.29008 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:46.361 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.342+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5594b4366c60 con 0x5594b4600030 2023-09-10T01:18:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f362bfff700 1 -- 172.21.15.3:0/1697507747 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5594b44a94f0 con 0x5594b4600030 2023-09-10T01:18:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f362bfff700 1 -- 172.21.15.3:0/1697507747 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f364002b3f0 con 0x5594b4600030 2023-09-10T01:18:46.362 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f3666a60b80 10 client.29008 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f3666a60b80 10 client.29008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f3666a60b80 20 client.29008 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b46194c0 0x5594b461b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.363 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f3666a60b80 4 client.29008 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:46.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594b45668a0 con 0x5594b46194c0 2023-09-10T01:18:46.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.343+0000 7f364bfff700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b46194c0 0x5594b461b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.344+0000 7f364bfff700 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b46194c0 0x5594b461b8e0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=202d6b2a0af81c72 server_cookie=6ad5fddbe2c4ab6d in_seq=0 out_seq=0 2023-09-10T01:18:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.344+0000 7f362bfff700 10 client.29008 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f362bfff700 1 -- 172.21.15.3:0/1697507747 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5594b45668a0 con 0x5594b46194c0 2023-09-10T01:18:46.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f362bfff700 10 client.29008 handle_command_reply: tid=0 2023-09-10T01:18:46.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 client.29008 shutdown 2023-09-10T01:18:46.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 msgr2=0x5594b4606a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b4606a20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f36400098e0 tx=0x7f364000f690 comp rx=0 tx=0).stop 2023-09-10T01:18:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 shutdown_connections 2023-09-10T01:18:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b46194c0 0x5594b461b8e0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b4607030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.373 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b4609810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1697507747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b4606a20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f362bfff700 0 client.29008 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 >> 172.21.15.3:0/1697507747 conn(0x5594b4574090 msgr2=0x5594b45fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 shutdown_connections 2023-09-10T01:18:46.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 1 -- 172.21.15.3:0/1697507747 wait complete. 2023-09-10T01:18:46.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.345+0000 7f3666a60b80 20 client.29008 trim_cache size 0 max 16384 2023-09-10T01:18:46.375 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.362+0000 7f3666a60b80 1 Processor -- start 2023-09-10T01:18:46.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3666a60b80 1 -- start start 2023-09-10T01:18:46.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b461b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.376 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b46062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b45f9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3666a60b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b4600470 con 0x5594b45ff710 2023-09-10T01:18:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3666a60b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b45f9860 con 0x5594b4609e40 2023-09-10T01:18:46.377 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3666a60b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b45f99f0 con 0x5594b4600030 2023-09-10T01:18:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f364bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b461b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f364bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b461b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34694/0 (socket says 172.21.15.3:34694) 2023-09-10T01:18:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f364bfff700 1 -- 172.21.15.3:0/1169190450 learned_addr learned my addr 172.21.15.3:0/1169190450 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:46.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.363+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b46062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f364bfff700 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b45f9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f364bfff700 1 -- 172.21.15.3:0/1169190450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 msgr2=0x5594b46062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f364bfff700 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b46062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f364bfff700 1 -- 172.21.15.3:0/1169190450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 msgr2=0x5594b45f9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f364bfff700 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b45f9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f364bfff700 1 -- 172.21.15.3:0/1169190450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594b43a4250 con 0x5594b45ff710 2023-09-10T01:18:46.380 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f364bfff700 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b461b8e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3638004790 tx=0x7f36380058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecc0027ab0d9957 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f36377fe700 1 -- 172.21.15.3:0/1169190450 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638024500 con 0x5594b45ff710 2023-09-10T01:18:46.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f36377fe700 1 -- 172.21.15.3:0/1169190450 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3638019030 con 0x5594b45ff710 2023-09-10T01:18:46.381 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.364+0000 7f36377fe700 1 -- 172.21.15.3:0/1169190450 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3638005280 con 0x5594b45ff710 2023-09-10T01:18:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 -- 172.21.15.3:0/1169190450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 msgr2=0x5594b461b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b461b8e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3638004790 tx=0x7f36380058e0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 -- 172.21.15.3:0/1169190450 shutdown_connections 2023-09-10T01:18:46.382 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b4609e40 0x5594b45f9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b45ff710 0x5594b461b8e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1169190450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b46062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 -- 172.21.15.3:0/1169190450 >> 172.21.15.3:0/1169190450 conn(0x5594b4574090 msgr2=0x5594b4620240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 -- 172.21.15.3:0/1169190450 shutdown_connections 2023-09-10T01:18:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 -- 172.21.15.3:0/1169190450 wait complete. 2023-09-10T01:18:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 -- start start 2023-09-10T01:18:46.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.365+0000 7f3666a60b80 1 Processor -- start 2023-09-10T01:18:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b45ff710 0x5594b460e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b460ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3666a60b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b4609e40 0x5594b4666330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3666a60b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b4666870 con 0x5594b4609e40 2023-09-10T01:18:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3666a60b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b46669e0 con 0x5594b45ff710 2023-09-10T01:18:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3666a60b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594b4666ba0 con 0x5594b4600030 2023-09-10T01:18:46.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3666a60b80 10 client.0 _ll_register_callbacks cb 0x7f3640005ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f364bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b4609e40 0x5594b4666330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f364bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b4609e40 0x5594b4666330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34708/0 (socket says 172.21.15.3:34708) 2023-09-10T01:18:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f364bfff700 1 -- 172.21.15.3:0/1887977407 learned_addr learned my addr 172.21.15.3:0/1887977407 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:46.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b460ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.366+0000 7f364bfff700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b45ff710 0x5594b460e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f3650f9b700 1 -- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b45ff710 msgr2=0x5594b460e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b45ff710 0x5594b460e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f3650f9b700 1 -- 172.21.15.3:0/1887977407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b4609e40 msgr2=0x5594b4666330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:46.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b4609e40 0x5594b4666330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f3650f9b700 1 -- 172.21.15.3:0/1887977407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363801b040 con 0x5594b4600030 2023-09-10T01:18:46.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f364bfff700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b4609e40 0x5594b4666330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:46.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f3650f9b700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b460ed90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3640005f50 tx=0x7f3640005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e19e43feb48e9ca1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f362bfff700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:18:46.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f362bfff700 1 -- 172.21.15.3:0/1887977407 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364002bbc0 con 0x5594b4600030 2023-09-10T01:18:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f364001e040 con 0x5594b4600030 2023-09-10T01:18:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.367+0000 7f362bfff700 1 -- 172.21.15.3:0/1887977407 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f3640000e40 con 0x5594b4600030 2023-09-10T01:18:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f362bfff700 1 -- 172.21.15.3:0/1887977407 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f364000f700 con 0x5594b4600030 2023-09-10T01:18:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 10 client.29014 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5594b437c2b0 con 0x5594b4600030 2023-09-10T01:18:46.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f362bfff700 1 -- 172.21.15.3:0/1887977407 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f364001e040 con 0x5594b4600030 2023-09-10T01:18:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f362bfff700 1 -- 172.21.15.3:0/1887977407 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f364002bd20 con 0x5594b4600030 2023-09-10T01:18:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 10 client.29014 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 10 client.29014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:46.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 20 client.29014 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b466df30 0x5594b4670350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 4 client.29014 mds_command: new command op to 24698 tid=0[{"prefix": "scrub status"}] 2023-09-10T01:18:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.368+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5594b454a920 con 0x5594b466df30 2023-09-10T01:18:46.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f364bfff700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b466df30 0x5594b4670350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f364bfff700 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b466df30 0x5594b4670350 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a156edd41ed6edf7 server_cookie=e3a6a66a1558144a in_seq=0 out_seq=0 2023-09-10T01:18:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f362bfff700 10 client.29014 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:46.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f362bfff700 1 -- 172.21.15.3:0/1887977407 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5594b454a920 con 0x5594b466df30 2023-09-10T01:18:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f362bfff700 10 client.29014 handle_command_reply: tid=0 2023-09-10T01:18:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 client.29014 shutdown 2023-09-10T01:18:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 msgr2=0x5594b460ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b460ed90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3640005f50 tx=0x7f3640005750 comp rx=0 tx=0).stop 2023-09-10T01:18:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 shutdown_connections 2023-09-10T01:18:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x5594b466df30 0x5594b4670350 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594b4609e40 0x5594b4666330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594b45ff710 0x5594b460e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 --2- 172.21.15.3:0/1887977407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5594b4600030 0x5594b460ed90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f362bfff700 0 client.29014 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 >> 172.21.15.3:0/1887977407 conn(0x5594b4574090 msgr2=0x5594b4601d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.369+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 shutdown_connections 2023-09-10T01:18:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f3666a60b80 1 -- 172.21.15.3:0/1887977407 wait complete. 2023-09-10T01:18:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f3666a60b80 20 client.29014 trim_cache size 0 max 16384 2023-09-10T01:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2023-09-10T01:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2023-09-10T01:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:18:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f363c0a07b0 msgr2=0x7f363c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f363c0a07b0 0x7f363c0a2c70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3644004a20 tx=0x7f3644021000 comp rx=0 tx=0).stop 2023-09-10T01:18:46.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c10ce80 msgr2=0x7f364c198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c10ce80 0x7f364c198830 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3644007ee0 tx=0x7f364400d3b0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 shutdown_connections 2023-09-10T01:18:46.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f363c0a07b0 0x7f363c0a2c70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364c10ce80 0x7f364c198830 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f364c072a30 0x7f364c19e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 --2- 172.21.15.3:0/1808104901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c073350 0x7f364c19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 >> 172.21.15.3:0/1808104901 conn(0x7f364c0fc170 msgr2=0x7f364c10f040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 shutdown_connections 2023-09-10T01:18:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.370+0000 7f36531ff700 1 -- 172.21.15.3:0/1808104901 wait complete. 2023-09-10T01:18:46.407 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-09-10T01:18:46.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:18:46.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:46 smithi003 ceph-mon[97654]: pgmap v489: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 88 KiB/s wr, 14 op/s 2023-09-10T01:18:46.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.612+0000 7fe70d5ff700 1 -- 172.21.15.3:0/613585800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 msgr2=0x7fe7081110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/613585800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 0x7fe7081110f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009990 tx=0x7fe6fc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.614 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 -- 172.21.15.3:0/613585800 shutdown_connections 2023-09-10T01:18:46.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/613585800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe708111630 0x7fe708113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/613585800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7080729a0 0x7fe708072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/613585800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 0x7fe7081110f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 -- 172.21.15.3:0/613585800 >> 172.21.15.3:0/613585800 conn(0x7fe7080fc110 msgr2=0x7fe7080fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 -- 172.21.15.3:0/613585800 shutdown_connections 2023-09-10T01:18:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.613+0000 7fe70d5ff700 1 -- 172.21.15.3:0/613585800 wait complete. 2023-09-10T01:18:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 Processor -- start 2023-09-10T01:18:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 -- start start 2023-09-10T01:18:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7080729a0 0x7fe70819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 0x7fe70819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe708111630 0x7fe7081a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7081a1d80 con 0x7fe7080729a0 2023-09-10T01:18:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7081a1ef0 con 0x7fe708111630 2023-09-10T01:18:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.614+0000 7fe70d5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7081a20b0 con 0x7fe7080732c0 2023-09-10T01:18:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7080729a0 0x7fe70819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe7077fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe708111630 0x7fe7081a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7080729a0 0x7fe70819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34720/0 (socket says 172.21.15.3:34720) 2023-09-10T01:18:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 -- 172.21.15.3:0/2845996968 learned_addr learned my addr 172.21.15.3:0/2845996968 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe7067fc700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 0x7fe70819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 -- 172.21.15.3:0/2845996968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 msgr2=0x7fe70819d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 0x7fe70819d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 -- 172.21.15.3:0/2845996968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe708111630 msgr2=0x7fe7081a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe708111630 0x7fe7081a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 -- 172.21.15.3:0/2845996968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0095f0 con 0x7fe7080729a0 2023-09-10T01:18:46.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe7077fe700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe708111630 0x7fe7081a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:46.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe706ffd700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7080729a0 0x7fe70819cda0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800e9a0 tx=0x7fe6f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.615+0000 7fe6effff700 1 -- 172.21.15.3:0/2845996968 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8009790 con 0x7fe7080729a0 2023-09-10T01:18:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.616+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7081a2390 con 0x7fe7080729a0 2023-09-10T01:18:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.616+0000 7fe6effff700 1 -- 172.21.15.3:0/2845996968 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6f8010430 con 0x7fe7080729a0 2023-09-10T01:18:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.616+0000 7fe6effff700 1 -- 172.21.15.3:0/2845996968 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8018680 con 0x7fe7080729a0 2023-09-10T01:18:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.616+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7081a9bd0 con 0x7fe7080729a0 2023-09-10T01:18:46.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.617+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe708103b10 con 0x7fe7080729a0 2023-09-10T01:18:46.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.619+0000 7fe6effff700 1 -- 172.21.15.3:0/2845996968 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fe6f80105a0 con 0x7fe7080729a0 2023-09-10T01:18:46.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.620+0000 7fe6effff700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6f00a0830 0x7fe6f00a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:46.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.620+0000 7fe6effff700 1 -- 172.21.15.3:0/2845996968 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fe6f8014070 con 0x7fe7080729a0 2023-09-10T01:18:46.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.620+0000 7fe7067fc700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6f00a0830 0x7fe6f00a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:46.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.621+0000 7fe7067fc700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6f00a0830 0x7fe6f00a2cf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe70819e3b0 tx=0x7fe6fc005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:46.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.624+0000 7fe6effff700 1 -- 172.21.15.3:0/2845996968 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fe6f8051e10 con 0x7fe7080729a0 2023-09-10T01:18:46.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:46 smithi136 ceph-mon[133708]: pgmap v489: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 88 KiB/s wr, 14 op/s 2023-09-10T01:18:46.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:46 smithi092 ceph-mon[134067]: pgmap v489: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 88 KiB/s wr, 14 op/s 2023-09-10T01:18:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.879+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe708067480 con 0x7fe7080729a0 2023-09-10T01:18:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.879+0000 7fe6effff700 1 -- 172.21.15.3:0/2845996968 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+11297 (secure 0 0 0) 0x7fe6f80958e0 con 0x7fe7080729a0 2023-09-10T01:18:46.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:18:46.881 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:18:25.307898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:18:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6f00a0830 msgr2=0x7fe6f00a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6f00a0830 0x7fe6f00a2cf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe70819e3b0 tx=0x7fe6fc005fb0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7080729a0 msgr2=0x7fe70819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:46.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7080729a0 0x7fe70819cda0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800e9a0 tx=0x7fe6f800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 shutdown_connections 2023-09-10T01:18:46.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fe6f00a0830 0x7fe6f00a2cf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.884 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe708111630 0x7fe7081a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7080729a0 0x7fe70819cda0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 --2- 172.21.15.3:0/2845996968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7080732c0 0x7fe70819d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:46.885 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 >> 172.21.15.3:0/2845996968 conn(0x7fe7080fc110 msgr2=0x7fe7080fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:46.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 shutdown_connections 2023-09-10T01:18:46.886 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:46.882+0000 7fe70d5ff700 1 -- 172.21.15.3:0/2845996968 wait complete. 2023-09-10T01:18:46.886 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 39 2023-09-10T01:18:46.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-09-10T01:18:47.107 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.106+0000 7fb7497ff700 1 -- 172.21.15.3:0/3830773993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 msgr2=0x7fb744106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.106+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3830773993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 0x7fb744106a90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb734009990 tx=0x7fb734009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.106+0000 7fb7497ff700 1 -- 172.21.15.3:0/3830773993 shutdown_connections 2023-09-10T01:18:47.108 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.106+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3830773993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 0x7fb74410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.106+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3830773993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 0x7fb744106a90 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.106+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3830773993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb744100610 0x7fb744100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.109 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.106+0000 7fb7497ff700 1 -- 172.21.15.3:0/3830773993 >> 172.21.15.3:0/3830773993 conn(0x7fb7440fc100 msgr2=0x7fb7440fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.107+0000 7fb7497ff700 1 -- 172.21.15.3:0/3830773993 shutdown_connections 2023-09-10T01:18:47.110 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.107+0000 7fb7497ff700 1 -- 172.21.15.3:0/3830773993 wait complete. 2023-09-10T01:18:47.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.108+0000 7fb7497ff700 1 Processor -- start 2023-09-10T01:18:47.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.108+0000 7fb7497ff700 1 -- start start 2023-09-10T01:18:47.112 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7497ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb744100610 0x7fb744072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7497ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 0x7fb744072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7497ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 0x7fb7441a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7497ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744073670 con 0x7fb744100fd0 2023-09-10T01:18:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7497ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7441a4e10 con 0x7fb7441066b0 2023-09-10T01:18:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7497ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7441a4f50 con 0x7fb744100610 2023-09-10T01:18:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 0x7fb744072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 0x7fb744072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34734/0 (socket says 172.21.15.3:34734) 2023-09-10T01:18:47.114 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 -- 172.21.15.3:0/3610061198 learned_addr learned my addr 172.21.15.3:0/3610061198 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb742ffd700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb744100610 0x7fb744072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7437fe700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 0x7fb7441a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 -- 172.21.15.3:0/3610061198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb744100610 msgr2=0x7fb744072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb744100610 0x7fb744072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 -- 172.21.15.3:0/3610061198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 msgr2=0x7fb7441a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 0x7fb7441a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.116 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.109+0000 7fb7427fc700 1 -- 172.21.15.3:0/3610061198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7340095f0 con 0x7fb744100fd0 2023-09-10T01:18:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.110+0000 7fb7427fc700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 0x7fb744072e40 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb73800e9a0 tx=0x7fb73800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.110+0000 7fb72bfff700 1 -- 172.21.15.3:0/3610061198 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738009790 con 0x7fb744100fd0 2023-09-10T01:18:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.110+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7441a51b0 con 0x7fb744100fd0 2023-09-10T01:18:47.117 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.110+0000 7fb72bfff700 1 -- 172.21.15.3:0/3610061198 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb738010430 con 0x7fb744100fd0 2023-09-10T01:18:47.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.110+0000 7fb72bfff700 1 -- 172.21.15.3:0/3610061198 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738018680 con 0x7fb744100fd0 2023-09-10T01:18:47.118 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.110+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7441a5690 con 0x7fb744100fd0 2023-09-10T01:18:47.122 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.113+0000 7fb72bfff700 1 -- 172.21.15.3:0/3610061198 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fb7380105a0 con 0x7fb744100fd0 2023-09-10T01:18:47.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.115+0000 7fb72bfff700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb72c0a07b0 0x7fb72c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.115+0000 7fb75d053b80 1 Processor -- start 2023-09-10T01:18:47.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.115+0000 7fb72bfff700 1 -- 172.21.15.3:0/3610061198 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fb738014070 con 0x7fb744100fd0 2023-09-10T01:18:47.123 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.115+0000 7fb742ffd700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb72c0a07b0 0x7fb72c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.115+0000 7fb75d053b80 1 -- start start 2023-09-10T01:18:47.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d9914af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9915030 0x5564d9927110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.124 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9929a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb75d053b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d986d730 con 0x5564d9927650 2023-09-10T01:18:47.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb75d053b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d986b050 con 0x5564d9915030 2023-09-10T01:18:47.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb75d053b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9866940 con 0x5564d9914710 2023-09-10T01:18:47.125 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb742ffd700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb72c0a07b0 0x7fb72c0a2c70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb73400b2e0 tx=0x7fb734005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9929a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9915030 0x5564d9927110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.126 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9915030 0x5564d9927110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36580/0 (socket says 172.21.15.3:36580) 2023-09-10T01:18:47.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9929a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34736/0 (socket says 172.21.15.3:34736) 2023-09-10T01:18:47.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb742ffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d9914af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7427fc700 1 -- 172.21.15.3:0/841523137 learned_addr learned my addr 172.21.15.3:0/841523137 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.127 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7437fe700 1 -- 172.21.15.3:0/841523137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 msgr2=0x5564d9914af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7437fe700 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d9914af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7437fe700 1 -- 172.21.15.3:0/841523137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9915030 msgr2=0x5564d9927110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7437fe700 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9915030 0x5564d9927110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.128 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb7437fe700 1 -- 172.21.15.3:0/841523137 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564d9692de0 con 0x5564d9927650 2023-09-10T01:18:47.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.116+0000 7fb742ffd700 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d9914af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:18:47.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.117+0000 7fb7437fe700 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9929a40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb73c011550 tx=0x7fb73c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0612d25364f2e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.117+0000 7fb7297fa700 1 -- 172.21.15.3:0/841523137 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00e040 con 0x5564d9927650 2023-09-10T01:18:47.129 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.117+0000 7fb7297fa700 1 -- 172.21.15.3:0/841523137 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb73c00f040 con 0x5564d9927650 2023-09-10T01:18:47.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.117+0000 7fb7297fa700 1 -- 172.21.15.3:0/841523137 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c004390 con 0x5564d9927650 2023-09-10T01:18:47.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 -- 172.21.15.3:0/841523137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 msgr2=0x5564d9929a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9929a40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb73c011550 tx=0x7fb73c009920 comp rx=0 tx=0).stop 2023-09-10T01:18:47.130 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 -- 172.21.15.3:0/841523137 shutdown_connections 2023-09-10T01:18:47.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d9914af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9929a40 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.131 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 --2- 172.21.15.3:0/841523137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9915030 0x5564d9927110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 -- 172.21.15.3:0/841523137 >> 172.21.15.3:0/841523137 conn(0x5564d9889090 msgr2=0x5564d990fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 -- 172.21.15.3:0/841523137 shutdown_connections 2023-09-10T01:18:47.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 -- 172.21.15.3:0/841523137 wait complete. 2023-09-10T01:18:47.132 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.118+0000 7fb75d053b80 1 -- start start 2023-09-10T01:18:47.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 1 Processor -- start 2023-09-10T01:18:47.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9914710 0x5564d9926fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d9924250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.133 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9924880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9804b70 con 0x5564d9927650 2023-09-10T01:18:47.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9804870 con 0x5564d9914710 2023-09-10T01:18:47.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d96e8920 con 0x5564d9915030 2023-09-10T01:18:47.134 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.119+0000 7fb75d053b80 10 client.0 _ll_register_callbacks cb 0x5564d974ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:47.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9924880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d9924250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.135 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9924880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34740/0 (socket says 172.21.15.3:34740) 2023-09-10T01:18:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 -- 172.21.15.3:0/3770555108 learned_addr learned my addr 172.21.15.3:0/3770555108 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb742ffd700 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9914710 0x5564d9926fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 -- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 msgr2=0x5564d9924250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d9924250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 -- 172.21.15.3:0/3770555108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9914710 msgr2=0x5564d9926fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9914710 0x5564d9926fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 -- 172.21.15.3:0/3770555108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c011230 con 0x5564d9927650 2023-09-10T01:18:47.137 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7427fc700 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d9924250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb7437fe700 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9924880 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb73c004f30 tx=0x7fb73c009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19639ec5510c57f server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb726ff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:47.138 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb726ff5700 1 -- 172.21.15.3:0/3770555108 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00e040 con 0x5564d9927650 2023-09-10T01:18:47.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564d97be4f0 con 0x5564d9927650 2023-09-10T01:18:47.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.120+0000 7fb726ff5700 1 -- 172.21.15.3:0/3770555108 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fb73c004240 con 0x5564d9927650 2023-09-10T01:18:47.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb726ff5700 1 -- 172.21.15.3:0/3770555108 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb73c004400 con 0x5564d9927650 2023-09-10T01:18:47.139 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 10 client.19035 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:47.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5564d967bc60 con 0x5564d9927650 2023-09-10T01:18:47.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb726ff5700 1 -- 172.21.15.3:0/3770555108 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5564d97be4f0 con 0x5564d9927650 2023-09-10T01:18:47.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb726ff5700 1 -- 172.21.15.3:0/3770555108 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fb73c01b020 con 0x5564d9927650 2023-09-10T01:18:47.140 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 10 client.19035 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:47.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 10 client.19035 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:18:47.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 20 client.19035 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:47.141 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d99370f0 0x5564d9939510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 4 client.19035 mds_command: new command op to 24634 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:47.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.121+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564d987b8a0 con 0x5564d99370f0 2023-09-10T01:18:47.142 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.122+0000 7fb742ffd700 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d99370f0 0x5564d9939510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.122+0000 7fb742ffd700 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d99370f0 0x5564d9939510 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29841ad919ed7b9c server_cookie=7f1da730eb2480b in_seq=0 out_seq=0 2023-09-10T01:18:47.148 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.122+0000 7fb726ff5700 10 client.19035 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:47.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.123+0000 7fb726ff5700 1 -- 172.21.15.3:0/3770555108 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5564d987b8a0 con 0x5564d99370f0 2023-09-10T01:18:47.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.123+0000 7fb726ff5700 10 client.19035 handle_command_reply: tid=0 2023-09-10T01:18:47.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.123+0000 7fb75d053b80 1 client.19035 shutdown 2023-09-10T01:18:47.149 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 msgr2=0x5564d9924880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9924880 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb73c004f30 tx=0x7fb73c009510 comp rx=0 tx=0).stop 2023-09-10T01:18:47.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 shutdown_connections 2023-09-10T01:18:47.150 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d99370f0 0x5564d9939510 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9927650 0x5564d9924880 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9914710 0x5564d9926fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 --2- 172.21.15.3:0/3770555108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d9924250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.151 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 >> 172.21.15.3:0/3770555108 conn(0x5564d9889090 msgr2=0x5564d990f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb726ff5700 0 client.19035 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:47.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 shutdown_connections 2023-09-10T01:18:47.152 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 1 -- 172.21.15.3:0/3770555108 wait complete. 2023-09-10T01:18:47.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.124+0000 7fb75d053b80 20 client.19035 trim_cache size 0 max 16384 2023-09-10T01:18:47.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 Processor -- start 2023-09-10T01:18:47.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 -- start start 2023-09-10T01:18:47.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d992e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9915030 0x5564d991f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d991faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9925860 con 0x5564d9915030 2023-09-10T01:18:47.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d99159e0 con 0x5564d9927650 2023-09-10T01:18:47.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb75d053b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9929ee0 con 0x5564d9914710 2023-09-10T01:18:47.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9915030 0x5564d991f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb7427fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9915030 0x5564d991f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34744/0 (socket says 172.21.15.3:34744) 2023-09-10T01:18:47.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb7427fc700 1 -- 172.21.15.3:0/4177351876 learned_addr learned my addr 172.21.15.3:0/4177351876 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb7437fe700 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d991faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.138+0000 7fb742ffd700 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d992e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb7427fc700 1 -- 172.21.15.3:0/4177351876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 msgr2=0x5564d992e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb7427fc700 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d992e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb7427fc700 1 -- 172.21.15.3:0/4177351876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 msgr2=0x5564d991faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb7427fc700 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d991faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb7427fc700 1 -- 172.21.15.3:0/4177351876 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564d96b9250 con 0x5564d9915030 2023-09-10T01:18:47.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb7437fe700 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d991faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb7427fc700 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9915030 0x5564d991f5b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb738095ab0 tx=0x7fb738084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4264108b656533e0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb729ffb700 1 -- 172.21.15.3:0/4177351876 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738099600 con 0x5564d9915030 2023-09-10T01:18:47.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb729ffb700 1 -- 172.21.15.3:0/4177351876 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb73809d430 con 0x5564d9915030 2023-09-10T01:18:47.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.139+0000 7fb729ffb700 1 -- 172.21.15.3:0/4177351876 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738097ac0 con 0x5564d9915030 2023-09-10T01:18:47.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 -- 172.21.15.3:0/4177351876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9915030 msgr2=0x5564d991f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9915030 0x5564d991f5b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb738095ab0 tx=0x7fb738084040 comp rx=0 tx=0).stop 2023-09-10T01:18:47.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 -- 172.21.15.3:0/4177351876 shutdown_connections 2023-09-10T01:18:47.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9914710 0x5564d992e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d991faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 --2- 172.21.15.3:0/4177351876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9915030 0x5564d991f5b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 -- 172.21.15.3:0/4177351876 >> 172.21.15.3:0/4177351876 conn(0x5564d9889090 msgr2=0x5564d9932950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 -- 172.21.15.3:0/4177351876 shutdown_connections 2023-09-10T01:18:47.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 -- 172.21.15.3:0/4177351876 wait complete. 2023-09-10T01:18:47.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.140+0000 7fb75d053b80 1 -- start start 2023-09-10T01:18:47.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 1 Processor -- start 2023-09-10T01:18:47.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9914710 0x5564d997f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d997f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d9983f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9984440 con 0x5564d9914710 2023-09-10T01:18:47.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9984580 con 0x5564d9927650 2023-09-10T01:18:47.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564d9984740 con 0x5564d9915030 2023-09-10T01:18:47.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb75d053b80 10 client.0 _ll_register_callbacks cb 0x7fb738095220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:47.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.141+0000 7fb742ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9914710 0x5564d997f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9914710 0x5564d997f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34750/0 (socket says 172.21.15.3:34750) 2023-09-10T01:18:47.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 -- 172.21.15.3:0/164842505 learned_addr learned my addr 172.21.15.3:0/164842505 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb7427fc700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d997f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb7437fe700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d9983f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 -- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 msgr2=0x5564d997f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d997f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 -- 172.21.15.3:0/164842505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 msgr2=0x5564d9983f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d9983f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 -- 172.21.15.3:0/164842505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564d967bc60 con 0x5564d9914710 2023-09-10T01:18:47.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb7437fe700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d9983f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:18:47.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb742ffd700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9914710 0x5564d997f0a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb734009990 tx=0x7fb734009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d3997336b0de57 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb726ff5700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:47.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb726ff5700 1 -- 172.21.15.3:0/164842505 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7340052a0 con 0x5564d9914710 2023-09-10T01:18:47.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.142+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb73c00f040 con 0x5564d9914710 2023-09-10T01:18:47.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb726ff5700 1 -- 172.21.15.3:0/164842505 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fb734005400 con 0x5564d9914710 2023-09-10T01:18:47.168 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb726ff5700 1 -- 172.21.15.3:0/164842505 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7340055c0 con 0x5564d9914710 2023-09-10T01:18:47.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 10 client.19041 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:47.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x7fb7441a5690 con 0x5564d9914710 2023-09-10T01:18:47.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb726ff5700 1 -- 172.21.15.3:0/164842505 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb73c00f040 con 0x5564d9914710 2023-09-10T01:18:47.169 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb726ff5700 1 -- 172.21.15.3:0/164842505 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fb73402f070 con 0x5564d9914710 2023-09-10T01:18:47.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 10 client.19041 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:47.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 10 client.19041 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-09-10T01:18:47.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 20 client.19041 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:47.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d998bfa0 0x5564d998e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 4 client.19041 mds_command: new command op to 24634 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:18:47.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.143+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 --> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5564d985f920 con 0x5564d998bfa0 2023-09-10T01:18:47.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb7427fc700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d998bfa0 0x5564d998e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb7427fc700 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d998bfa0 0x5564d998e3c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=321c71f08a87b7b8 server_cookie=b9591039270ee86f in_seq=0 out_seq=0 2023-09-10T01:18:47.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb726ff5700 10 client.19041 ms_handle_connect on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:47.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb726ff5700 1 -- 172.21.15.3:0/164842505 <== mds.2 v2:172.21.15.136:6838/1259564848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5564d985f920 con 0x5564d998bfa0 2023-09-10T01:18:47.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb726ff5700 10 client.19041 handle_command_reply: tid=0 2023-09-10T01:18:47.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 client.19041 shutdown 2023-09-10T01:18:47.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9914710 msgr2=0x5564d997f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9914710 0x5564d997f0a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb734009990 tx=0x7fb734009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 shutdown_connections 2023-09-10T01:18:47.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:6838/1259564848,v1:172.21.15.136:6839/1259564848] conn(0x5564d998bfa0 0x5564d998e3c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5564d9927650 0x5564d9983f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564d9914710 0x5564d997f0a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb75d053b80 1 --2- 172.21.15.3:0/164842505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564d9915030 0x5564d997f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.144+0000 7fb726ff5700 0 client.19041 ms_handle_reset on v2:172.21.15.136:6838/1259564848 2023-09-10T01:18:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 >> 172.21.15.3:0/164842505 conn(0x5564d9889090 msgr2=0x5564d991f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 shutdown_connections 2023-09-10T01:18:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb75d053b80 1 -- 172.21.15.3:0/164842505 wait complete. 2023-09-10T01:18:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb75d053b80 20 client.19041 trim_cache size 0 max 16384 2023-09-10T01:18:47.176 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:18:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb72c0a07b0 msgr2=0x7fb72c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb72c0a07b0 0x7fb72c0a2c70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb73400b2e0 tx=0x7fb734005fd0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 msgr2=0x7fb744072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 0x7fb744072e40 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb73800e9a0 tx=0x7fb73800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 shutdown_connections 2023-09-10T01:18:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.145+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fb72c0a07b0 0x7fb72c0a2c70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.179 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.146+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb744100610 0x7fb744072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.146+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7441066b0 0x7fb7441a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.146+0000 7fb7497ff700 1 --2- 172.21.15.3:0/3610061198 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb744100fd0 0x7fb744072e40 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.146+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 >> 172.21.15.3:0/3610061198 conn(0x7fb7440fc100 msgr2=0x7fb7440fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.146+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 shutdown_connections 2023-09-10T01:18:47.181 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.146+0000 7fb7497ff700 1 -- 172.21.15.3:0/3610061198 wait complete. 2023-09-10T01:18:47.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-09-10T01:18:47.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 -- 172.21.15.3:0/4035641218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc106600 msgr2=0x7fcdbc1069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/4035641218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc106600 0x7fcdbc1069e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0099c0 tx=0x7fcdac009cd0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 -- 172.21.15.3:0/4035641218 shutdown_connections 2023-09-10T01:18:47.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/4035641218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100f00 0x7fcdbc10d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/4035641218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc106600 0x7fcdbc1069e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/4035641218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdbc100540 0x7fcdbc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 -- 172.21.15.3:0/4035641218 >> 172.21.15.3:0/4035641218 conn(0x7fcdbc0fc030 msgr2=0x7fcdbc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 -- 172.21.15.3:0/4035641218 shutdown_connections 2023-09-10T01:18:47.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.388+0000 7fcdc19ff700 1 -- 172.21.15.3:0/4035641218 wait complete. 2023-09-10T01:18:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.389+0000 7fcdc19ff700 1 Processor -- start 2023-09-10T01:18:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.389+0000 7fcdc19ff700 1 -- start start 2023-09-10T01:18:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdc19ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100540 0x7fcdbc196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdc19ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc100f00 0x7fcdbc196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdc19ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdbc106600 0x7fcdbc19b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdc19ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc19b640 con 0x7fcdbc100f00 2023-09-10T01:18:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdc19ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc19b7b0 con 0x7fcdbc100540 2023-09-10T01:18:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdc19ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc19b970 con 0x7fcdbc106600 2023-09-10T01:18:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100540 0x7fcdbc196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdba7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc100f00 0x7fcdbc196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100540 0x7fcdbc196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36622/0 (socket says 172.21.15.3:36622) 2023-09-10T01:18:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3236977081 learned_addr learned my addr 172.21.15.3:0/3236977081 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbb7fe700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdbc106600 0x7fcdbc19b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3236977081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdbc106600 msgr2=0x7fcdbc19b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdbc106600 0x7fcdbc19b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc100f00 msgr2=0x7fcdbc196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc100f00 0x7fcdbc196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.390+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3236977081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4009c40 con 0x7fcdbc100540 2023-09-10T01:18:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.391+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100540 0x7fcdbc196660 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00b580 tx=0x7fcdac00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.391+0000 7fcdc09fd700 1 -- 172.21.15.3:0/3236977081 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac01d070 con 0x7fcdbc100540 2023-09-10T01:18:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.391+0000 7fcdc09fd700 1 -- 172.21.15.3:0/3236977081 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdac005050 con 0x7fcdbc100540 2023-09-10T01:18:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.391+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac0095f0 con 0x7fcdbc100540 2023-09-10T01:18:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.391+0000 7fcdc09fd700 1 -- 172.21.15.3:0/3236977081 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac004a40 con 0x7fcdbc100540 2023-09-10T01:18:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.391+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc19be80 con 0x7fcdbc100540 2023-09-10T01:18:47.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.394+0000 7fcdc09fd700 1 -- 172.21.15.3:0/3236977081 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fcdac0051c0 con 0x7fcdbc100540 2023-09-10T01:18:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdd538bb80 1 Processor -- start 2023-09-10T01:18:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdc09fd700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcda40a07b0 0x7fcda40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdd538bb80 1 -- start start 2023-09-10T01:18:47.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdc09fd700 1 -- 172.21.15.3:0/3236977081 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fcdac0cd250 con 0x7fcdbc100540 2023-09-10T01:18:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdbb7fe700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcda40a07b0 0x7fcda40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8d6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.406 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d7030 0x56385d8e0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8e3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d82f730 con 0x56385d8e0e40 2023-09-10T01:18:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.396+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d82d050 con 0x56385d8d7030 2023-09-10T01:18:47.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d828940 con 0x56385d8d6710 2023-09-10T01:18:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d7030 0x56385d8e0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8d6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d7030 0x56385d8e0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36626/0 (socket says 172.21.15.3:36626) 2023-09-10T01:18:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8d6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43700/0 (socket says 172.21.15.3:43700) 2023-09-10T01:18:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 -- 172.21.15.3:0/2295467700 learned_addr learned my addr 172.21.15.3:0/2295467700 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdbb7fe700 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdbb7fe700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcda40a07b0 0x7fcda40a2c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009c00 tx=0x7fcdb4009920 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 -- 172.21.15.3:0/2295467700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 msgr2=0x56385d8d6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8d6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 -- 172.21.15.3:0/2295467700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 msgr2=0x56385d8e3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8e3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdba7fc700 1 -- 172.21.15.3:0/2295467700 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56385d654de0 con 0x56385d8d7030 2023-09-10T01:18:47.413 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.397+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8d6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.398+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d7030 0x56385d8e0900 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009d20 tx=0x7fcdb0011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e550287c4deb414 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.398+0000 7fcda1ffb700 1 -- 172.21.15.3:0/2295467700 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0019630 con 0x56385d8d7030 2023-09-10T01:18:47.414 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.398+0000 7fcda1ffb700 1 -- 172.21.15.3:0/2295467700 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdb0019c50 con 0x56385d8d7030 2023-09-10T01:18:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.398+0000 7fcda1ffb700 1 -- 172.21.15.3:0/2295467700 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000ec40 con 0x56385d8d7030 2023-09-10T01:18:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 -- 172.21.15.3:0/2295467700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d7030 msgr2=0x56385d8e0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d7030 0x56385d8e0900 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009d20 tx=0x7fcdb0011720 comp rx=0 tx=0).stop 2023-09-10T01:18:47.415 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 -- 172.21.15.3:0/2295467700 shutdown_connections 2023-09-10T01:18:47.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8d6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.416 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8e3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/2295467700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d7030 0x56385d8e0900 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 -- 172.21.15.3:0/2295467700 >> 172.21.15.3:0/2295467700 conn(0x56385d84b090 msgr2=0x56385d8d1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.417 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 -- 172.21.15.3:0/2295467700 shutdown_connections 2023-09-10T01:18:47.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.399+0000 7fcdd538bb80 1 -- 172.21.15.3:0/2295467700 wait complete. 2023-09-10T01:18:47.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.400+0000 7fcdd538bb80 1 -- start start 2023-09-10T01:18:47.418 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.400+0000 7fcdd538bb80 1 Processor -- start 2023-09-10T01:18:47.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8e0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d7030 0x56385d8ddb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.419 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8de250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d7c6b70 con 0x56385d8d7030 2023-09-10T01:18:47.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d7c6870 con 0x56385d8e0e40 2023-09-10T01:18:47.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d6aa920 con 0x56385d8d6710 2023-09-10T01:18:47.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8e0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdd538bb80 10 client.0 _ll_register_callbacks cb 0x56385d710a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8e0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43708/0 (socket says 172.21.15.3:43708) 2023-09-10T01:18:47.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdbaffd700 1 -- 172.21.15.3:0/1578686806 learned_addr learned my addr 172.21.15.3:0/1578686806 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d7030 0x56385d8ddb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.401+0000 7fcdbb7fe700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8de250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdbaffd700 1 -- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 msgr2=0x56385d8de250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8de250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdbaffd700 1 -- 172.21.15.3:0/1578686806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d7030 msgr2=0x56385d8ddb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d7030 0x56385d8ddb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.423 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdbaffd700 1 -- 172.21.15.3:0/1578686806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc19be80 con 0x56385d8d6710 2023-09-10T01:18:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d7030 0x56385d8ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdbb7fe700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8de250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8e0890 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009fd0 tx=0x7fcdac09a350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0558fbe014d2d1c server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.402+0000 7fcd9affd700 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2023-09-10T01:18:47.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.404+0000 7fcd9affd700 1 -- 172.21.15.3:0/1578686806 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac004e00 con 0x56385d8d6710 2023-09-10T01:18:47.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.404+0000 7fcd9affd700 1 -- 172.21.15.3:0/1578686806 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fcdac0a2330 con 0x56385d8d6710 2023-09-10T01:18:47.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.404+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56385d7804f0 con 0x56385d8d6710 2023-09-10T01:18:47.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.409+0000 7fcd9affd700 1 -- 172.21.15.3:0/1578686806 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdac0b8170 con 0x56385d8d6710 2023-09-10T01:18:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.409+0000 7fcdd538bb80 10 client.29050 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.409+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x7fcdb0009980 con 0x56385d8d6710 2023-09-10T01:18:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.409+0000 7fcd9affd700 1 -- 172.21.15.3:0/1578686806 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdac064080 con 0x56385d8d6710 2023-09-10T01:18:47.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.409+0000 7fcd9affd700 1 -- 172.21.15.3:0/1578686806 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fcdac09a640 con 0x56385d8d6710 2023-09-10T01:18:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.409+0000 7fcdd538bb80 10 client.29050 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcdd538bb80 10 client.29050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:18:47.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcdd538bb80 20 client.29050 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d8f0660 0x56385d8f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcdd538bb80 4 client.29050 mds_command: new command op to 24647 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56385d83d8a0 con 0x56385d8f0660 2023-09-10T01:18:47.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d8f0660 0x56385d8f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d8f0660 0x56385d8f2a80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a2e7cb54b40fb837 server_cookie=f1f8834b1d091f04 in_seq=0 out_seq=0 2023-09-10T01:18:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.410+0000 7fcd9affd700 10 client.29050 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.411+0000 7fcd9affd700 1 -- 172.21.15.3:0/1578686806 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56385d83d8a0 con 0x56385d8f0660 2023-09-10T01:18:47.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.411+0000 7fcd9affd700 10 client.29050 handle_command_reply: tid=0 2023-09-10T01:18:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.411+0000 7fcdd538bb80 1 client.29050 shutdown 2023-09-10T01:18:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.411+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 msgr2=0x56385d8e0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.434 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.411+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8e0890 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009fd0 tx=0x7fcdac09a350 comp rx=0 tx=0).stop 2023-09-10T01:18:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.411+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 shutdown_connections 2023-09-10T01:18:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d8f0660 0x56385d8f2a80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d6710 0x56385d8e0890 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcd9affd700 0 client.29050 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8de250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/1578686806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d7030 0x56385d8ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.436 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 >> 172.21.15.3:0/1578686806 conn(0x56385d84b090 msgr2=0x56385d8d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 shutdown_connections 2023-09-10T01:18:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 1 -- 172.21.15.3:0/1578686806 wait complete. 2023-09-10T01:18:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.412+0000 7fcdd538bb80 20 client.29050 trim_cache size 0 max 16384 2023-09-10T01:18:47.437 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.425+0000 7fcdd538bb80 1 Processor -- start 2023-09-10T01:18:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdd538bb80 1 -- start start 2023-09-10T01:18:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d6710 0x56385d8e3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.438 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8ec140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d8d76d0 con 0x56385d8e0e40 2023-09-10T01:18:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d8ce3d0 con 0x56385d8d6710 2023-09-10T01:18:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d8e7f30 con 0x56385d8d7030 2023-09-10T01:18:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdbb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8ec140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.440 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d6710 0x56385d8e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d6710 0x56385d8e3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36632/0 (socket says 172.21.15.3:36632) 2023-09-10T01:18:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.426+0000 7fcdbaffd700 1 -- 172.21.15.3:0/914531576 learned_addr learned my addr 172.21.15.3:0/914531576 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.441 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcdbaffd700 1 -- 172.21.15.3:0/914531576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 msgr2=0x56385d8e79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcdbaffd700 1 -- 172.21.15.3:0/914531576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 msgr2=0x56385d8ec140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8ec140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcdbaffd700 1 -- 172.21.15.3:0/914531576 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56385d63dc60 con 0x56385d8d6710 2023-09-10T01:18:47.443 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d6710 0x56385d8e3230 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcdac096dc0 tx=0x7fcdac066310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfcfc46183906065 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcda27fc700 1 -- 172.21.15.3:0/914531576 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0a2330 con 0x56385d8d6710 2023-09-10T01:18:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcda27fc700 1 -- 172.21.15.3:0/914531576 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56385d7804f0 con 0x56385d8d6710 2023-09-10T01:18:47.444 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.427+0000 7fcda27fc700 1 -- 172.21.15.3:0/914531576 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0a1ea0 con 0x56385d8d6710 2023-09-10T01:18:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 -- 172.21.15.3:0/914531576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d6710 msgr2=0x56385d8e3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d6710 0x56385d8e3230 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcdac096dc0 tx=0x7fcdac066310 comp rx=0 tx=0).stop 2023-09-10T01:18:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 -- 172.21.15.3:0/914531576 shutdown_connections 2023-09-10T01:18:47.445 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8e0e40 0x56385d8ec140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8d6710 0x56385d8e3230 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/914531576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.446 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 -- 172.21.15.3:0/914531576 >> 172.21.15.3:0/914531576 conn(0x56385d84b090 msgr2=0x56385d8d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 -- 172.21.15.3:0/914531576 shutdown_connections 2023-09-10T01:18:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.428+0000 7fcdd538bb80 1 -- 172.21.15.3:0/914531576 wait complete. 2023-09-10T01:18:47.447 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.429+0000 7fcdd538bb80 1 -- start start 2023-09-10T01:18:47.448 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.429+0000 7fcdd538bb80 1 Processor -- start 2023-09-10T01:18:47.451 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d6710 0x56385d8e50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdd538bb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8e5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d943640 con 0x56385d8d6710 2023-09-10T01:18:47.452 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d9437b0 con 0x56385d8e0e40 2023-09-10T01:18:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdd538bb80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56385d943970 con 0x56385d8d7030 2023-09-10T01:18:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdd538bb80 10 client.0 _ll_register_callbacks cb 0x7fcdac09aa50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:47.453 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d6710 0x56385d8e50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d6710 0x56385d8e50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34798/0 (socket says 172.21.15.3:34798) 2023-09-10T01:18:47.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3976654951 learned_addr learned my addr 172.21.15.3:0/3976654951 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbb7fe700 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8e5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.454 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3976654951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 msgr2=0x56385d8e5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 msgr2=0x56385d8e5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8e5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.430+0000 7fcdbaffd700 1 -- 172.21.15.3:0/3976654951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb0009980 con 0x56385d8d6710 2023-09-10T01:18:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcdbaffd700 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d6710 0x56385d8e50c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcdac09a9b0 tx=0x7fcdac099b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56fc95b4e5a04525 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcd9affd700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcd9affd700 1 -- 172.21.15.3:0/3976654951 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdac09a7b0 con 0x56385d8d6710 2023-09-10T01:18:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcd9affd700 1 -- 172.21.15.3:0/3976654951 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7fcdac0a2330 con 0x56385d8d6710 2023-09-10T01:18:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcdb0019c50 con 0x56385d8d6710 2023-09-10T01:18:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcd9affd700 1 -- 172.21.15.3:0/3976654951 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdac0adbe0 con 0x56385d8d6710 2023-09-10T01:18:47.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcdd538bb80 10 client.19059 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:47.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x56385d67b250 con 0x56385d8d6710 2023-09-10T01:18:47.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcd9affd700 1 -- 172.21.15.3:0/3976654951 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdb0019c50 con 0x56385d8d6710 2023-09-10T01:18:47.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcd9affd700 1 -- 172.21.15.3:0/3976654951 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fcdac004c00 con 0x56385d8d6710 2023-09-10T01:18:47.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcdd538bb80 10 client.19059 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:47.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcdd538bb80 10 client.19059 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-09-10T01:18:47.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.431+0000 7fcdd538bb80 20 client.19059 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:47.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.432+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d944c10 0x56385d947000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.432+0000 7fcdd538bb80 4 client.19059 mds_command: new command op to 24647 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:18:47.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.432+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 --> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56385d821920 con 0x56385d944c10 2023-09-10T01:18:47.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.432+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d944c10 0x56385d947000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.432+0000 7fcdba7fc700 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d944c10 0x56385d947000 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f818673233ebc54 server_cookie=19427fc5549b5b27 in_seq=0 out_seq=0 2023-09-10T01:18:47.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.432+0000 7fcd9affd700 10 client.19059 ms_handle_connect on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:47.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcd9affd700 1 -- 172.21.15.3:0/3976654951 <== mds.1 v2:172.21.15.92:6836/12010426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56385d821920 con 0x56385d944c10 2023-09-10T01:18:47.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcd9affd700 10 client.19059 handle_command_reply: tid=0 2023-09-10T01:18:47.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 client.19059 shutdown 2023-09-10T01:18:47.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d6710 msgr2=0x56385d8e50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d6710 0x56385d8e50c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcdac09a9b0 tx=0x7fcdac099b30 comp rx=0 tx=0).stop 2023-09-10T01:18:47.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 shutdown_connections 2023-09-10T01:18:47.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385d8e0e40 0x56385d8e5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56385d8d6710 0x56385d8e50c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.463 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.92:6836/12010426,v1:172.21.15.92:6837/12010426] conn(0x56385d944c10 0x56385d947000 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 --2- 172.21.15.3:0/3976654951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56385d8d7030 0x56385d8e5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 >> 172.21.15.3:0/3976654951 conn(0x56385d84b090 msgr2=0x56385d8d8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcd9affd700 0 client.19059 ms_handle_reset on v2:172.21.15.92:6836/12010426 2023-09-10T01:18:47.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 shutdown_connections 2023-09-10T01:18:47.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 1 -- 172.21.15.3:0/3976654951 wait complete. 2023-09-10T01:18:47.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.433+0000 7fcdd538bb80 20 client.19059 trim_cache size 0 max 16384 2023-09-10T01:18:47.466 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:18:47.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcda40a07b0 msgr2=0x7fcda40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcda40a07b0 0x7fcda40a2c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009c00 tx=0x7fcdb4009920 comp rx=0 tx=0).stop 2023-09-10T01:18:47.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100540 msgr2=0x7fcdbc196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100540 0x7fcdbc196660 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00b580 tx=0x7fcdac00bfd0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 shutdown_connections 2023-09-10T01:18:47.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fcda40a07b0 0x7fcda40a2c70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdbc106600 0x7fcdbc19b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdbc100540 0x7fcdbc196660 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 --2- 172.21.15.3:0/3236977081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdbc100f00 0x7fcdbc196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 >> 172.21.15.3:0/3236977081 conn(0x7fcdbc0fc030 msgr2=0x7fcdbc0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 shutdown_connections 2023-09-10T01:18:47.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.434+0000 7fcdc19ff700 1 -- 172.21.15.3:0/3236977081 wait complete. 2023-09-10T01:18:47.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:47 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2845996968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:18:47.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-09-10T01:18:47.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:47 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2845996968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:18:47.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/791113115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 msgr2=0x7f8fc8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.679 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/791113115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 0x7f8fc8100ac0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009990 tx=0x7f8fbc009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/791113115 shutdown_connections 2023-09-10T01:18:47.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/791113115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc8106650 0x7f8fc8106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.680 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/791113115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc8101000 0x7f8fc810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/791113115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 0x7f8fc8100ac0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/791113115 >> 172.21.15.3:0/791113115 conn(0x7f8fc80fc170 msgr2=0x7f8fc80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/791113115 shutdown_connections 2023-09-10T01:18:47.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.678+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/791113115 wait complete. 2023-09-10T01:18:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.679+0000 7f8fcc9ff700 1 Processor -- start 2023-09-10T01:18:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fcc9ff700 1 -- start start 2023-09-10T01:18:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fcc9ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 0x7f8fc819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fcc9ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc8101000 0x7f8fc819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fcc9ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc8106650 0x7f8fc8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fcc9ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8198db0 con 0x7f8fc8106650 2023-09-10T01:18:47.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fcc9ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8198f20 con 0x7f8fc8100640 2023-09-10T01:18:47.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fcc9ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc81990e0 con 0x7f8fc8101000 2023-09-10T01:18:47.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fc5d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc8101000 0x7f8fc819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fc6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc8106650 0x7f8fc8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 0x7f8fc819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fc5d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc8101000 0x7f8fc819ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43732/0 (socket says 172.21.15.3:43732) 2023-09-10T01:18:47.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.680+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/707428759 learned_addr learned my addr 172.21.15.3:0/707428759 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fc6d9d700 1 -- 172.21.15.3:0/707428759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc8101000 msgr2=0x7f8fc819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc8101000 0x7f8fc819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fc6d9d700 1 -- 172.21.15.3:0/707428759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 msgr2=0x7f8fc819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 0x7f8fc819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fc6d9d700 1 -- 172.21.15.3:0/707428759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc0095f0 con 0x7f8fc8106650 2023-09-10T01:18:47.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc8106650 0x7f8fc8198740 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000efc0 tx=0x7f8fc000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fb77fe700 1 -- 172.21.15.3:0/707428759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0018070 con 0x7f8fc8106650 2023-09-10T01:18:47.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc8199360 con 0x7f8fc8106650 2023-09-10T01:18:47.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fb77fe700 1 -- 172.21.15.3:0/707428759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fc000f040 con 0x7f8fc8106650 2023-09-10T01:18:47.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fb77fe700 1 -- 172.21.15.3:0/707428759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc000cd40 con 0x7f8fc8106650 2023-09-10T01:18:47.690 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.681+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc81a56c0 con 0x7f8fc8106650 2023-09-10T01:18:47.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.685+0000 7f8fb77fe700 1 -- 172.21.15.3:0/707428759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f8fc0003a00 con 0x7f8fc8106650 2023-09-10T01:18:47.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.687+0000 7f8fe0362b80 1 Processor -- start 2023-09-10T01:18:47.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.687+0000 7f8fe0362b80 1 -- start start 2023-09-10T01:18:47.695 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.687+0000 7f8fb77fe700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8fb00a07b0 0x7f8fb00a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.687+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c635af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.687+0000 7f8fb77fe700 1 -- 172.21.15.3:0/707428759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f8fc00cbf50 con 0x7f8fc8106650 2023-09-10T01:18:47.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.687+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.687+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c642230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c58e730 con 0x56125c63fe40 2023-09-10T01:18:47.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c58c050 con 0x56125c635710 2023-09-10T01:18:47.697 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c587940 con 0x56125c636030 2023-09-10T01:18:47.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8fb00a07b0 0x7f8fb00a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c635af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c635af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36680/0 (socket says 172.21.15.3:36680) 2023-09-10T01:18:47.698 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 -- 172.21.15.3:0/3482777978 learned_addr learned my addr 172.21.15.3:0/3482777978 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c642230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.699 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 -- 172.21.15.3:0/3482777978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 msgr2=0x56125c63f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 -- 172.21.15.3:0/3482777978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 msgr2=0x56125c642230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c642230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.700 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc659c700 1 -- 172.21.15.3:0/3482777978 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56125c3b3de0 con 0x56125c635710 2023-09-10T01:18:47.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.688+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8fb00a07b0 0x7f8fb00a2c70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8199bb0 tx=0x7f8fbc00bb70 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.689+0000 7f8fc659c700 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c635af0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009d00 tx=0x7f8fb8011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17fee3ae5e307635 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.689+0000 7f8fb4ff9700 1 -- 172.21.15.3:0/3482777978 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8019630 con 0x56125c635710 2023-09-10T01:18:47.701 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.689+0000 7f8fb4ff9700 1 -- 172.21.15.3:0/3482777978 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fb8019c50 con 0x56125c635710 2023-09-10T01:18:47.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.689+0000 7f8fb4ff9700 1 -- 172.21.15.3:0/3482777978 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb800eae0 con 0x56125c635710 2023-09-10T01:18:47.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3482777978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 msgr2=0x56125c635af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c635af0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009d00 tx=0x7f8fb8011720 comp rx=0 tx=0).stop 2023-09-10T01:18:47.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3482777978 shutdown_connections 2023-09-10T01:18:47.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c642230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c635af0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.703 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3482777978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3482777978 >> 172.21.15.3:0/3482777978 conn(0x56125c5aa090 msgr2=0x56125c630bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3482777978 shutdown_connections 2023-09-10T01:18:47.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3482777978 wait complete. 2023-09-10T01:18:47.704 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 -- start start 2023-09-10T01:18:47.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.690+0000 7f8fe0362b80 1 Processor -- start 2023-09-10T01:18:47.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c63f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c63d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c525b70 con 0x56125c63fe40 2023-09-10T01:18:47.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c525870 con 0x56125c635710 2023-09-10T01:18:47.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c409920 con 0x56125c636030 2023-09-10T01:18:47.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fe0362b80 10 client.0 _ll_register_callbacks cb 0x56125c46fa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:47.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c63f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c63f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36690/0 (socket says 172.21.15.3:36690) 2023-09-10T01:18:47.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fc659c700 1 -- 172.21.15.3:0/3579478173 learned_addr learned my addr 172.21.15.3:0/3579478173 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c63d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.691+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fc6d9d700 1 -- 172.21.15.3:0/3579478173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 msgr2=0x56125c63cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fc6d9d700 1 -- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 msgr2=0x56125c63f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c63f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fc6d9d700 1 -- 172.21.15.3:0/3579478173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb8009960 con 0x56125c63fe40 2023-09-10T01:18:47.709 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fc659c700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c63f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c63d110 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00c9260 tx=0x7f8fc0098c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6cc02093c1d0eb server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fadffb700 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2023-09-10T01:18:47.710 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fadffb700 1 -- 172.21.15.3:0/3579478173 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00994b0 con 0x56125c63fe40 2023-09-10T01:18:47.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56125c4df4f0 con 0x56125c63fe40 2023-09-10T01:18:47.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fadffb700 1 -- 172.21.15.3:0/3579478173 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f8fc001b530 con 0x56125c63fe40 2023-09-10T01:18:47.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.692+0000 7f8fadffb700 1 -- 172.21.15.3:0/3579478173 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fc0099750 con 0x56125c63fe40 2023-09-10T01:18:47.711 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 10 client.19071 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:47.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x56125c39cc60 con 0x56125c63fe40 2023-09-10T01:18:47.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fadffb700 1 -- 172.21.15.3:0/3579478173 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56125c4df4f0 con 0x56125c63fe40 2023-09-10T01:18:47.712 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fadffb700 1 -- 172.21.15.3:0/3579478173 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f8fc0098540 con 0x56125c63fe40 2023-09-10T01:18:47.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 10 client.19071 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:47.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 10 client.19071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:47.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 20 client.19071 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:47.713 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c64fa60 0x56125c651e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 4 client.19071 mds_command: new command op to 24698 tid=0[{"prefix": "get_command_descriptions"}] 2023-09-10T01:18:47.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56125c59c8a0 con 0x56125c64fa60 2023-09-10T01:18:47.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.693+0000 7f8fc659c700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c64fa60 0x56125c651e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.714 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.694+0000 7f8fc659c700 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c64fa60 0x56125c651e80 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5ff17c2c4e9030 server_cookie=25ce2702eb8e79f2 in_seq=0 out_seq=0 2023-09-10T01:18:47.715 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.694+0000 7f8fadffb700 10 client.19071 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:47.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:47 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2845996968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:18:47.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fadffb700 1 -- 172.21.15.3:0/3579478173 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56125c59c8a0 con 0x56125c64fa60 2023-09-10T01:18:47.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fadffb700 10 client.19071 handle_command_reply: tid=0 2023-09-10T01:18:47.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 client.19071 shutdown 2023-09-10T01:18:47.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 msgr2=0x56125c63d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.722 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c63d110 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00c9260 tx=0x7f8fc0098c80 comp rx=0 tx=0).stop 2023-09-10T01:18:47.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 shutdown_connections 2023-09-10T01:18:47.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c64fa60 0x56125c651e80 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c63d110 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c635710 0x56125c63f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fadffb700 0 client.19071 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:47.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/3579478173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c636030 0x56125c63cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.724 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 >> 172.21.15.3:0/3579478173 conn(0x56125c5aa090 msgr2=0x56125c6307e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 shutdown_connections 2023-09-10T01:18:47.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 1 -- 172.21.15.3:0/3579478173 wait complete. 2023-09-10T01:18:47.725 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.695+0000 7f8fe0362b80 20 client.19071 trim_cache size 0 max 16384 2023-09-10T01:18:47.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 Processor -- start 2023-09-10T01:18:47.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 -- start start 2023-09-10T01:18:47.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c635710 0x56125c634e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c63f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c639d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c64cfa0 con 0x56125c63fe40 2023-09-10T01:18:47.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c6316e0 con 0x56125c636030 2023-09-10T01:18:47.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c6366d0 con 0x56125c635710 2023-09-10T01:18:47.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fc5d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c63f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fc5d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c63f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36698/0 (socket says 172.21.15.3:36698) 2023-09-10T01:18:47.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/4102803228 learned_addr learned my addr 172.21.15.3:0/4102803228 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c639d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.709+0000 7f8fc659c700 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c635710 0x56125c634e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/4102803228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c635710 msgr2=0x56125c634e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c635710 0x56125c634e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/4102803228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 msgr2=0x56125c639d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c639d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.730 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/4102803228 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56125c3da250 con 0x56125c636030 2023-09-10T01:18:47.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fc659c700 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c635710 0x56125c634e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c63f390 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc005910 tx=0x7f8fbc027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5515557502e6f0b1 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fb57fa700 1 -- 172.21.15.3:0/4102803228 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc01d070 con 0x56125c636030 2023-09-10T01:18:47.731 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fb57fa700 1 -- 172.21.15.3:0/4102803228 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fbc0049c0 con 0x56125c636030 2023-09-10T01:18:47.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.710+0000 7f8fb57fa700 1 -- 172.21.15.3:0/4102803228 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc00f9f0 con 0x56125c636030 2023-09-10T01:18:47.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 -- 172.21.15.3:0/4102803228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 msgr2=0x56125c63f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c63f390 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc005910 tx=0x7f8fbc027000 comp rx=0 tx=0).stop 2023-09-10T01:18:47.732 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 -- 172.21.15.3:0/4102803228 shutdown_connections 2023-09-10T01:18:47.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c635710 0x56125c634e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c63fe40 0x56125c639d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.733 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/4102803228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c63f390 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 -- 172.21.15.3:0/4102803228 >> 172.21.15.3:0/4102803228 conn(0x56125c5aa090 msgr2=0x56125c634700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 -- 172.21.15.3:0/4102803228 shutdown_connections 2023-09-10T01:18:47.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 -- 172.21.15.3:0/4102803228 wait complete. 2023-09-10T01:18:47.734 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.711+0000 7f8fe0362b80 1 -- start start 2023-09-10T01:18:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 1 Processor -- start 2023-09-10T01:18:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c635710 0x56125c646780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c644a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c63fe40 0x56125c644fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c6454f0 con 0x56125c635710 2023-09-10T01:18:47.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c645660 con 0x56125c636030 2023-09-10T01:18:47.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x56125c645820 con 0x56125c63fe40 2023-09-10T01:18:47.736 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fe0362b80 10 client.0 _ll_register_callbacks cb 0x7f8fc0095480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-09-10T01:18:47.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c635710 0x56125c646780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fc5d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c644a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fc659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c635710 0x56125c646780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34850/0 (socket says 172.21.15.3:34850) 2023-09-10T01:18:47.737 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fc659c700 1 -- 172.21.15.3:0/1517251143 learned_addr learned my addr 172.21.15.3:0/1517251143 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fc5d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c644a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36700/0 (socket says 172.21.15.3:36700) 2023-09-10T01:18:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.712+0000 7f8fc6d9d700 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c63fe40 0x56125c644fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/1517251143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c63fe40 msgr2=0x56125c644fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c63fe40 0x56125c644fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/1517251143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c635710 msgr2=0x56125c646780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-10T01:18:47.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c635710 0x56125c646780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fc5d9b700 1 -- 172.21.15.3:0/1517251143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56125c3b3de0 con 0x56125c636030 2023-09-10T01:18:47.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fc659c700 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c635710 0x56125c646780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:47.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fc5d9b700 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c644a70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x56125c5fa900 tx=0x7f8fbc005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=badd5411c6025012 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:47.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fadffb700 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2023-09-10T01:18:47.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fadffb700 1 -- 172.21.15.3:0/1517251143 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc01d030 con 0x56125c636030 2023-09-10T01:18:47.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fb8019c50 con 0x56125c636030 2023-09-10T01:18:47.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.713+0000 7f8fadffb700 1 -- 172.21.15.3:0/1517251143 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f8fbc0045c0 con 0x56125c636030 2023-09-10T01:18:47.741 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fadffb700 1 -- 172.21.15.3:0/1517251143 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fbc004780 con 0x56125c636030 2023-09-10T01:18:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 10 client.29195 fetch_fsmap learned FSMap version 39 2023-09-10T01:18:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x7f8fbc020040 con 0x56125c636030 2023-09-10T01:18:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fadffb700 1 -- 172.21.15.3:0/1517251143 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fb8019c50 con 0x56125c636030 2023-09-10T01:18:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fadffb700 1 -- 172.21.15.3:0/1517251143 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 39) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f8fbc00fcb0 con 0x56125c636030 2023-09-10T01:18:47.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 10 client.29195 fetch_fsmap finished waiting for FSMap version 39 2023-09-10T01:18:47.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 10 client.29195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-09-10T01:18:47.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 20 client.29195 populate_metadata read hostname 'smithi003' 2023-09-10T01:18:47.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c6a4940 0x56125c6a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:47.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 4 client.29195 mds_command: new command op to 24698 tid=0[{"prefix": "damage ls"}] 2023-09-10T01:18:47.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.714+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 --> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56125c580920 con 0x56125c6a4940 2023-09-10T01:18:47.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.715+0000 7f8fc659c700 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c6a4940 0x56125c6a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:47.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.715+0000 7f8fc659c700 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c6a4940 0x56125c6a6d60 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f078809f2cb246e server_cookie=ce629f69a018752f in_seq=0 out_seq=0 2023-09-10T01:18:47.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.715+0000 7f8fadffb700 10 client.29195 ms_handle_connect on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:47.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fadffb700 1 -- 172.21.15.3:0/1517251143 <== mds.0 v2:172.21.15.92:6838/149607042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56125c580920 con 0x56125c6a4940 2023-09-10T01:18:47.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fadffb700 10 client.29195 handle_command_reply: tid=0 2023-09-10T01:18:47.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 client.29195 shutdown 2023-09-10T01:18:47.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 msgr2=0x56125c644a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c644a70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x56125c5fa900 tx=0x7f8fbc005a00 comp rx=0 tx=0).stop 2023-09-10T01:18:47.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 shutdown_connections 2023-09-10T01:18:47.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:6838/149607042,v1:172.21.15.92:6839/149607042] conn(0x56125c6a4940 0x56125c6a6d60 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56125c63fe40 0x56125c644fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.747 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56125c635710 0x56125c646780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fadffb700 0 client.29195 ms_handle_reset on v2:172.21.15.92:6838/149607042 2023-09-10T01:18:47.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 --2- 172.21.15.3:0/1517251143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56125c636030 0x56125c644a70 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 >> 172.21.15.3:0/1517251143 conn(0x56125c5aa090 msgr2=0x56125c634f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.748 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 shutdown_connections 2023-09-10T01:18:47.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 1 -- 172.21.15.3:0/1517251143 wait complete. 2023-09-10T01:18:47.749 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.716+0000 7f8fe0362b80 20 client.29195 trim_cache size 0 max 16384 2023-09-10T01:18:47.749 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2023-09-10T01:18:47.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8fb00a07b0 msgr2=0x7f8fb00a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8fb00a07b0 0x7f8fb00a2c70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8199bb0 tx=0x7f8fbc00bb70 comp rx=0 tx=0).stop 2023-09-10T01:18:47.751 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc8106650 msgr2=0x7f8fc8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:47.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc8106650 0x7f8fc8198740 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000efc0 tx=0x7f8fc000c3a0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 shutdown_connections 2023-09-10T01:18:47.752 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f8fb00a07b0 0x7f8fb00a2c70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8fc8106650 0x7f8fc8198740 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fc8100640 0x7f8fc819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 --2- 172.21.15.3:0/707428759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc8101000 0x7f8fc819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:47.753 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 >> 172.21.15.3:0/707428759 conn(0x7f8fc80fc170 msgr2=0x7f8fc810bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:47.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 shutdown_connections 2023-09-10T01:18:47.754 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:47.717+0000 7f8fcc9ff700 1 -- 172.21.15.3:0/707428759 wait complete. 2023-09-10T01:18:48.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:48 smithi136 ceph-mon[133708]: pgmap v490: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-09-10T01:18:48.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:48 smithi092 ceph-mon[134067]: pgmap v490: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-09-10T01:18:48.755 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2023-09-10T01:18:48.756 INFO:tasks.fwd_scrub:done joining 2023-09-10T01:18:48.756 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-09-10T01:18:48.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:18:48.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:48 smithi003 ceph-mon[97654]: pgmap v490: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-09-10T01:18:48.983 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 -- 172.21.15.3:0/371405446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c106630 msgr2=0x7f3c9c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/371405446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c106630 0x7f3c9c106a10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009990 tx=0x7f3c98009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 -- 172.21.15.3:0/371405446 shutdown_connections 2023-09-10T01:18:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/371405446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c106630 0x7f3c9c106a10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:48.984 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/371405446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 0x7f3c9c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/371405446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c9c100620 0x7f3c9c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 -- 172.21.15.3:0/371405446 >> 172.21.15.3:0/371405446 conn(0x7f3c9c0fc110 msgr2=0x7f3c9c0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 -- 172.21.15.3:0/371405446 shutdown_connections 2023-09-10T01:18:48.985 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.982+0000 7f3ca43ff700 1 -- 172.21.15.3:0/371405446 wait complete. 2023-09-10T01:18:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.983+0000 7f3ca43ff700 1 Processor -- start 2023-09-10T01:18:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.983+0000 7f3ca43ff700 1 -- start start 2023-09-10T01:18:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca43ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c100620 0x7f3c9c072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:48.988 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca43ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 0x7f3c9c072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca43ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c9c106630 0x7f3c9c1a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca43ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c073670 con 0x7f3c9c100fe0 2023-09-10T01:18:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca43ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c1a4e10 con 0x7f3c9c106630 2023-09-10T01:18:48.989 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca43ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c1a4f50 con 0x7f3c9c100620 2023-09-10T01:18:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca199a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 0x7f3c9c072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca219b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c100620 0x7f3c9c072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:48.990 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca199a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 0x7f3c9c072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34874/0 (socket says 172.21.15.3:34874) 2023-09-10T01:18:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca219b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c100620 0x7f3c9c072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43778/0 (socket says 172.21.15.3:43778) 2023-09-10T01:18:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca199a700 1 -- 172.21.15.3:0/1909912084 learned_addr learned my addr 172.21.15.3:0/1909912084 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.984+0000 7f3ca299c700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c9c106630 0x7f3c9c1a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:48.991 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca199a700 1 -- 172.21.15.3:0/1909912084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c100620 msgr2=0x7f3c9c072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca199a700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c100620 0x7f3c9c072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca199a700 1 -- 172.21.15.3:0/1909912084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c9c106630 msgr2=0x7f3c9c1a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:48.992 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca199a700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c9c106630 0x7f3c9c1a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca199a700 1 -- 172.21.15.3:0/1909912084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c009520 con 0x7f3c9c100fe0 2023-09-10T01:18:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca299c700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c9c106630 0x7f3c9c1a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:48.993 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca199a700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 0x7f3c9c072e40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00eb30 tx=0x7f3c8c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3c937fe700 1 -- 172.21.15.3:0/1909912084 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c009880 con 0x7f3c9c100fe0 2023-09-10T01:18:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c94009c40 con 0x7f3c9c100fe0 2023-09-10T01:18:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3c937fe700 1 -- 172.21.15.3:0/1909912084 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c8c0044e0 con 0x7f3c9c100fe0 2023-09-10T01:18:48.994 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3c937fe700 1 -- 172.21.15.3:0/1909912084 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c003d80 con 0x7f3c9c100fe0 2023-09-10T01:18:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.985+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c980095f0 con 0x7f3c9c100fe0 2023-09-10T01:18:48.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.986+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c067480 con 0x7f3c9c100fe0 2023-09-10T01:18:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.988+0000 7f3c937fe700 1 -- 172.21.15.3:0/1909912084 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f3c8c003ee0 con 0x7f3c9c100fe0 2023-09-10T01:18:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.990+0000 7f3c937fe700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c880a07b0 0x7f3c880a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.990+0000 7f3c937fe700 1 -- 172.21.15.3:0/1909912084 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f3c8c014070 con 0x7f3c9c100fe0 2023-09-10T01:18:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.990+0000 7f3ca219b700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c880a07b0 0x7f3c880a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.990+0000 7f3ca219b700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c880a07b0 0x7f3c880a2c70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800b2e0 tx=0x7f3c98005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:48.997 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:48.994+0000 7f3c937fe700 1 -- 172.21.15.3:0/1909912084 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3c8c095b90 con 0x7f3c9c100fe0 2023-09-10T01:18:49.207 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.206+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c9c1105b0 con 0x7f3c9c100fe0 2023-09-10T01:18:49.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.208+0000 7f3c937fe700 1 -- 172.21.15.3:0/1909912084 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+25123 (secure 0 0 0) 0x7f3c8c004190 con 0x7f3c9c100fe0 2023-09-10T01:18:49.209 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:18:49.210 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":111,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:18:20.300292+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/3516755706":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/1056299694":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6801/91094391":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6800/91094391":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/2575312228":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/2296857090":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":25,"length":1}]},{"pool":4,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:18:49.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c880a07b0 msgr2=0x7f3c880a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c880a07b0 0x7f3c880a2c70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800b2e0 tx=0x7f3c98005fd0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 msgr2=0x7f3c9c072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.212 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 0x7f3c9c072e40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00eb30 tx=0x7f3c8c00ee40 comp rx=0 tx=0).stop 2023-09-10T01:18:49.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 shutdown_connections 2023-09-10T01:18:49.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f3c880a07b0 0x7f3c880a2c70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c9c100620 0x7f3c9c072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.213 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c9c106630 0x7f3c9c1a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 --2- 172.21.15.3:0/1909912084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c9c100fe0 0x7f3c9c072e40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 >> 172.21.15.3:0/1909912084 conn(0x7f3c9c0fc110 msgr2=0x7f3c9c0fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:49.214 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 shutdown_connections 2023-09-10T01:18:49.215 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.211+0000 7f3ca43ff700 1 -- 172.21.15.3:0/1909912084 wait complete. 2023-09-10T01:18:49.231 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:18:49.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:18:49.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 -- 172.21.15.3:0/2137385326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc100640 msgr2=0x7f53cc100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 --2- 172.21.15.3:0/2137385326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc100640 0x7f53cc100ac0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009990 tx=0x7f53c0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 -- 172.21.15.3:0/2137385326 shutdown_connections 2023-09-10T01:18:49.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 --2- 172.21.15.3:0/2137385326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc101000 0x7f53cc10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 --2- 172.21.15.3:0/2137385326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53cc106650 0x7f53cc106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 --2- 172.21.15.3:0/2137385326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc100640 0x7f53cc100ac0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 -- 172.21.15.3:0/2137385326 >> 172.21.15.3:0/2137385326 conn(0x7f53cc0fc170 msgr2=0x7f53cc0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:49.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 -- 172.21.15.3:0/2137385326 shutdown_connections 2023-09-10T01:18:49.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.480+0000 7f53d1bff700 1 -- 172.21.15.3:0/2137385326 wait complete. 2023-09-10T01:18:49.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.481+0000 7f53d1bff700 1 Processor -- start 2023-09-10T01:18:49.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.481+0000 7f53d1bff700 1 -- start start 2023-09-10T01:18:49.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53d1bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc100640 0x7f53cc19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53d1bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53cc101000 0x7f53cc19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53d1bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc106650 0x7f53cc198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53d1bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc198de0 con 0x7f53cc100640 2023-09-10T01:18:49.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53d1bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc198f50 con 0x7f53cc101000 2023-09-10T01:18:49.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53d1bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc199110 con 0x7f53cc106650 2023-09-10T01:18:49.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc100640 0x7f53cc19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53caffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53cc101000 0x7f53cc19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc100640 0x7f53cc19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34878/0 (socket says 172.21.15.3:34878) 2023-09-10T01:18:49.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 -- 172.21.15.3:0/1375141505 learned_addr learned my addr 172.21.15.3:0/1375141505 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:49.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cbfff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc106650 0x7f53cc198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 -- 172.21.15.3:0/1375141505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc106650 msgr2=0x7f53cc198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc106650 0x7f53cc198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 -- 172.21.15.3:0/1375141505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53cc101000 msgr2=0x7f53cc19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.493 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53cc101000 0x7f53cc19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cb7fe700 1 -- 172.21.15.3:0/1375141505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00095f0 con 0x7f53cc100640 2023-09-10T01:18:49.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53caffd700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53cc101000 0x7f53cc19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:49.494 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.482+0000 7f53cbfff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc106650 0x7f53cc198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-10T01:18:49.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.483+0000 7f53cb7fe700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc100640 0x7f53cc19e700 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00e9a0 tx=0x7f53bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:49.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.483+0000 7f53c8ff9700 1 -- 172.21.15.3:0/1375141505 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc009790 con 0x7f53cc100640 2023-09-10T01:18:49.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.483+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc1993f0 con 0x7f53cc100640 2023-09-10T01:18:49.495 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.483+0000 7f53c8ff9700 1 -- 172.21.15.3:0/1375141505 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53bc00cbd0 con 0x7f53cc100640 2023-09-10T01:18:49.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.483+0000 7f53c8ff9700 1 -- 172.21.15.3:0/1375141505 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc018680 con 0x7f53cc100640 2023-09-10T01:18:49.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.483+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc1a56f0 con 0x7f53cc100640 2023-09-10T01:18:49.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.484+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc067480 con 0x7f53cc100640 2023-09-10T01:18:49.496 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.486+0000 7f53c8ff9700 1 -- 172.21.15.3:0/1375141505 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f53bc00cd40 con 0x7f53cc100640 2023-09-10T01:18:49.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.487+0000 7f53c8ff9700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f53b40a07b0 0x7f53b40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.487+0000 7f53c8ff9700 1 -- 172.21.15.3:0/1375141505 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f53bc014070 con 0x7f53cc100640 2023-09-10T01:18:49.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.488+0000 7f53caffd700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f53b40a07b0 0x7f53b40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.488+0000 7f53caffd700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f53b40a07b0 0x7f53b40a2c70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f53cc199be0 tx=0x7f53c0005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:49.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.490+0000 7f53c8ff9700 1 -- 172.21.15.3:0/1375141505 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f53bc095c70 con 0x7f53cc100640 2023-09-10T01:18:49.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:49 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:49.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:49 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1909912084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:18:49.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.700+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53cc199c80 con 0x7f53cc100640 2023-09-10T01:18:49.702 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.701+0000 7f53c8ff9700 1 -- 172.21.15.3:0/1375141505 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53bc099290 con 0x7f53cc100640 2023-09-10T01:18:49.703 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:18:49.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.703+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f53b40a07b0 msgr2=0x7f53b40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.705 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.703+0000 7f53d1bff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f53b40a07b0 0x7f53b40a2c70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f53cc199be0 tx=0x7f53c0005fd0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.703+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc100640 msgr2=0x7f53cc19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.703+0000 7f53d1bff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc100640 0x7f53cc19e700 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00e9a0 tx=0x7f53bc00ecb0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.706 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.703+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 shutdown_connections 2023-09-10T01:18:49.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.703+0000 7f53d1bff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f53b40a07b0 0x7f53b40a2c70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.704+0000 7f53d1bff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53cc106650 0x7f53cc198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.704+0000 7f53d1bff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53cc100640 0x7f53cc19e700 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.707 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.704+0000 7f53d1bff700 1 --2- 172.21.15.3:0/1375141505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53cc101000 0x7f53cc19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.704+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 >> 172.21.15.3:0/1375141505 conn(0x7f53cc0fc170 msgr2=0x7f53cc0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:49.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.704+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 shutdown_connections 2023-09-10T01:18:49.708 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.704+0000 7f53d1bff700 1 -- 172.21.15.3:0/1375141505 wait complete. 2023-09-10T01:18:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:49 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:49.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:49 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1909912084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:18:49.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:18:49.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:49 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:49.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:49 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1909912084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:18:49.940 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 -- 172.21.15.3:0/3131355291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 msgr2=0x7f077c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 --2- 172.21.15.3:0/3131355291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c106a90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f076c009990 tx=0x7f076c009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.941 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 -- 172.21.15.3:0/3131355291 shutdown_connections 2023-09-10T01:18:49.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 --2- 172.21.15.3:0/3131355291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f077c100fd0 0x7f077c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 --2- 172.21.15.3:0/3131355291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c106a90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.942 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 --2- 172.21.15.3:0/3131355291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f077c100610 0x7f077c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 -- 172.21.15.3:0/3131355291 >> 172.21.15.3:0/3131355291 conn(0x7f077c0fc100 msgr2=0x7f077c0fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 -- 172.21.15.3:0/3131355291 shutdown_connections 2023-09-10T01:18:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.939+0000 7f07815ff700 1 -- 172.21.15.3:0/3131355291 wait complete. 2023-09-10T01:18:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.940+0000 7f07815ff700 1 Processor -- start 2023-09-10T01:18:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f07815ff700 1 -- start start 2023-09-10T01:18:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f07815ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f077c100610 0x7f077c072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f07815ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f077c100fd0 0x7f077c072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f07815ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c1a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f07815ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c073670 con 0x7f077c1066b0 2023-09-10T01:18:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f07815ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c1a4df0 con 0x7f077c100fd0 2023-09-10T01:18:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f07815ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c1a4f30 con 0x7f077c100610 2023-09-10T01:18:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f077b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c1a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f077b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c1a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34896/0 (socket says 172.21.15.3:34896) 2023-09-10T01:18:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.941+0000 7f077b7fe700 1 -- 172.21.15.3:0/2687615488 learned_addr learned my addr 172.21.15.3:0/2687615488 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:49.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077a7fc700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f077c100fd0 0x7f077c072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077affd700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f077c100610 0x7f077c072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077b7fe700 1 -- 172.21.15.3:0/2687615488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f077c100610 msgr2=0x7f077c072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.949 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077b7fe700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f077c100610 0x7f077c072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077b7fe700 1 -- 172.21.15.3:0/2687615488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f077c100fd0 msgr2=0x7f077c072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:49.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077b7fe700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f077c100fd0 0x7f077c072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:49.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077b7fe700 1 -- 172.21.15.3:0/2687615488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0095f0 con 0x7f077c1066b0 2023-09-10T01:18:49.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f077b7fe700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c1a48b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f077400ef30 tx=0x7f077400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:49.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f0763fff700 1 -- 172.21.15.3:0/2687615488 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0774018070 con 0x7f077c1066b0 2023-09-10T01:18:49.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f0763fff700 1 -- 172.21.15.3:0/2687615488 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f077400f040 con 0x7f077c1066b0 2023-09-10T01:18:49.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077c1a5130 con 0x7f077c1066b0 2023-09-10T01:18:49.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.942+0000 7f0763fff700 1 -- 172.21.15.3:0/2687615488 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077400cd00 con 0x7f077c1066b0 2023-09-10T01:18:49.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.943+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077c1a5660 con 0x7f077c1066b0 2023-09-10T01:18:49.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.943+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077c0678d0 con 0x7f077c1066b0 2023-09-10T01:18:49.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.945+0000 7f0763fff700 1 -- 172.21.15.3:0/2687615488 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f0774003a00 con 0x7f077c1066b0 2023-09-10T01:18:49.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.946+0000 7f0763fff700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f07640a0830 0x7f07640a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:49.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.947+0000 7f0763fff700 1 -- 172.21.15.3:0/2687615488 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f07740cbe90 con 0x7f077c1066b0 2023-09-10T01:18:49.954 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.947+0000 7f077affd700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f07640a0830 0x7f07640a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:49.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.947+0000 7f077affd700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f07640a0830 0x7f07640a2cf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f076c00b2e0 tx=0x7f076c005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:49.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:49.951+0000 7f0763fff700 1 -- 172.21.15.3:0/2687615488 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f07740952d0 con 0x7f077c1066b0 2023-09-10T01:18:50.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.164+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f077c1105f0 con 0x7f077c1066b0 2023-09-10T01:18:50.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.165+0000 7f0763fff700 1 -- 172.21.15.3:0/2687615488 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07740988f0 con 0x7f077c1066b0 2023-09-10T01:18:50.168 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:18:50.170 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f07640a0830 msgr2=0x7f07640a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f07640a0830 0x7f07640a2cf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f076c00b2e0 tx=0x7f076c005fb0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.171 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 msgr2=0x7f077c1a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c1a48b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f077400ef30 tx=0x7f077400c3a0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.172 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 shutdown_connections 2023-09-10T01:18:50.173 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f07640a0830 0x7f07640a2cf0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.174 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f077c100610 0x7f077c072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f077c1066b0 0x7f077c1a48b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.175 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 --2- 172.21.15.3:0/2687615488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f077c100fd0 0x7f077c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.176 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.167+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 >> 172.21.15.3:0/2687615488 conn(0x7f077c0fc100 msgr2=0x7f077c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:50.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.168+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 shutdown_connections 2023-09-10T01:18:50.177 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.168+0000 7f07815ff700 1 -- 172.21.15.3:0/2687615488 wait complete. 2023-09-10T01:18:50.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:18:50.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.395+0000 7f24d13ff700 1 -- 172.21.15.3:0/4205074932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 msgr2=0x7f24cc108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.395+0000 7f24d13ff700 1 --2- 172.21.15.3:0/4205074932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 0x7f24cc108c20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0098f0 tx=0x7f24bc009c00 comp rx=0 tx=0).stop 2023-09-10T01:18:50.397 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.396+0000 7f24d13ff700 1 -- 172.21.15.3:0/4205074932 shutdown_connections 2023-09-10T01:18:50.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.396+0000 7f24d13ff700 1 --2- 172.21.15.3:0/4205074932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 0x7f24cc108c20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.396+0000 7f24d13ff700 1 --2- 172.21.15.3:0/4205074932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 0x7f24cc10f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.396+0000 7f24d13ff700 1 --2- 172.21.15.3:0/4205074932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24cc102830 0x7f24cc102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.398 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.396+0000 7f24d13ff700 1 -- 172.21.15.3:0/4205074932 >> 172.21.15.3:0/4205074932 conn(0x7f24cc0fe2e0 msgr2=0x7f24cc100700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:50.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.396+0000 7f24d13ff700 1 -- 172.21.15.3:0/4205074932 shutdown_connections 2023-09-10T01:18:50.399 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.396+0000 7f24d13ff700 1 -- 172.21.15.3:0/4205074932 wait complete. 2023-09-10T01:18:50.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 Processor -- start 2023-09-10T01:18:50.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 -- start start 2023-09-10T01:18:50.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24cc102830 0x7f24cc19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.401 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 0x7f24cc19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 0x7f24cc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc198db0 con 0x7f24cc1031f0 2023-09-10T01:18:50.402 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc198f20 con 0x7f24cc102830 2023-09-10T01:18:50.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.397+0000 7f24d13ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc1990e0 con 0x7f24cc108840 2023-09-10T01:18:50.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 0x7f24cc19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24caffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24cc102830 0x7f24cc19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.403 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 0x7f24cc19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34912/0 (socket says 172.21.15.3:34912) 2023-09-10T01:18:50.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 -- 172.21.15.3:0/1773168980 learned_addr learned my addr 172.21.15.3:0/1773168980 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:50.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24cb7fe700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 0x7f24cc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 -- 172.21.15.3:0/1773168980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 msgr2=0x7f24cc198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.404 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 0x7f24cc198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 -- 172.21.15.3:0/1773168980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24cc102830 msgr2=0x7f24cc19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.405 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24cc102830 0x7f24cc19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 -- 172.21.15.3:0/1773168980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c4009d00 con 0x7f24cc1031f0 2023-09-10T01:18:50.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24caffd700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24cc102830 0x7f24cc19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:50.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.398+0000 7f24ca7fc700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 0x7f24cc19ec10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f24c000e9a0 tx=0x7f24c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:50.407 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.399+0000 7f24b3fff700 1 -- 172.21.15.3:0/1773168980 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0009790 con 0x7f24cc1031f0 2023-09-10T01:18:50.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.399+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc009520 con 0x7f24cc1031f0 2023-09-10T01:18:50.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.399+0000 7f24b3fff700 1 -- 172.21.15.3:0/1773168980 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24c0010430 con 0x7f24cc1031f0 2023-09-10T01:18:50.408 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.399+0000 7f24b3fff700 1 -- 172.21.15.3:0/1773168980 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0018680 con 0x7f24cc1031f0 2023-09-10T01:18:50.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.399+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc1a54e0 con 0x7f24cc1031f0 2023-09-10T01:18:50.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.400+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc0678d0 con 0x7f24cc1031f0 2023-09-10T01:18:50.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.402+0000 7f24b3fff700 1 -- 172.21.15.3:0/1773168980 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f24c00105a0 con 0x7f24cc1031f0 2023-09-10T01:18:50.409 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.403+0000 7f24b3fff700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24b40a07b0 0x7f24b40a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.403+0000 7f24b3fff700 1 -- 172.21.15.3:0/1773168980 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f24c0014070 con 0x7f24cc1031f0 2023-09-10T01:18:50.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.403+0000 7f24caffd700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24b40a07b0 0x7f24b40a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.410 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.404+0000 7f24caffd700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24b40a07b0 0x7f24b40a2c70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009fd0 tx=0x7f24bc019040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:50.411 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.407+0000 7f24b3fff700 1 -- 172.21.15.3:0/1773168980 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f24c0051e10 con 0x7f24cc1031f0 2023-09-10T01:18:50.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:50 smithi003 ceph-mon[97654]: pgmap v491: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-09-10T01:18:50.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1375141505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:18:50.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:50 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2687615488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:18:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.619+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24cc112850 con 0x7f24cc1031f0 2023-09-10T01:18:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.620+0000 7f24b3fff700 1 -- 172.21.15.3:0/1773168980 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24c0095940 con 0x7f24cc1031f0 2023-09-10T01:18:50.621 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:18:50.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24b40a07b0 msgr2=0x7f24b40a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24b40a07b0 0x7f24b40a2c70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009fd0 tx=0x7f24bc019040 comp rx=0 tx=0).stop 2023-09-10T01:18:50.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 msgr2=0x7f24cc19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 0x7f24cc19ec10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f24c000e9a0 tx=0x7f24c000ecb0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 shutdown_connections 2023-09-10T01:18:50.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f24b40a07b0 0x7f24b40a2c70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24cc108840 0x7f24cc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24cc102830 0x7f24cc19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 --2- 172.21.15.3:0/1773168980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24cc1031f0 0x7f24cc19ec10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 >> 172.21.15.3:0/1773168980 conn(0x7f24cc0fe2e0 msgr2=0x7f24cc0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:50.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 shutdown_connections 2023-09-10T01:18:50.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.622+0000 7f24d13ff700 1 -- 172.21.15.3:0/1773168980 wait complete. 2023-09-10T01:18:50.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:18:50.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:50 smithi092 ceph-mon[134067]: pgmap v491: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-09-10T01:18:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:50 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1375141505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:18:50.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:50 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2687615488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:18:50.849 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.848+0000 7f30569ff700 1 -- 172.21.15.3:0/2297231927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 msgr2=0x7f3050106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.848+0000 7f30569ff700 1 --2- 172.21.15.3:0/2297231927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 0x7f3050106a30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3040009990 tx=0x7f3040009ca0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.850 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.849+0000 7f30569ff700 1 -- 172.21.15.3:0/2297231927 shutdown_connections 2023-09-10T01:18:50.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.849+0000 7f30569ff700 1 --2- 172.21.15.3:0/2297231927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050101000 0x7f305010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.849+0000 7f30569ff700 1 --2- 172.21.15.3:0/2297231927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 0x7f3050106a30 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.849+0000 7f30569ff700 1 --2- 172.21.15.3:0/2297231927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3050100640 0x7f3050100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.851 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.849+0000 7f30569ff700 1 -- 172.21.15.3:0/2297231927 >> 172.21.15.3:0/2297231927 conn(0x7f30500fc170 msgr2=0x7f30500fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:50.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.849+0000 7f30569ff700 1 -- 172.21.15.3:0/2297231927 shutdown_connections 2023-09-10T01:18:50.852 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.849+0000 7f30569ff700 1 -- 172.21.15.3:0/2297231927 wait complete. 2023-09-10T01:18:50.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 Processor -- start 2023-09-10T01:18:50.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 -- start start 2023-09-10T01:18:50.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050100640 0x7f305019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3050101000 0x7f305019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 0x7f3050198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.855 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3050198db0 con 0x7f3050106650 2023-09-10T01:18:50.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3050198f20 con 0x7f3050101000 2023-09-10T01:18:50.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.850+0000 7f30569ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30501990e0 con 0x7f3050100640 2023-09-10T01:18:50.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050100640 0x7f305019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.856 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f3054f9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 0x7f3050198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050100640 0x7f305019e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:43870/0 (socket says 172.21.15.3:43870) 2023-09-10T01:18:50.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 -- 172.21.15.3:0/3820539935 learned_addr learned my addr 172.21.15.3:0/3820539935 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:50.857 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304f7fe700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3050101000 0x7f305019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 -- 172.21.15.3:0/3820539935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3050101000 msgr2=0x7f305019ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3050101000 0x7f305019ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 -- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 msgr2=0x7f3050198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:50.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 0x7f3050198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:50.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 -- 172.21.15.3:0/3820539935 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30400095f0 con 0x7f3050100640 2023-09-10T01:18:50.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f3054f9c700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 0x7f3050198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:50.859 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.851+0000 7f304ffff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050100640 0x7f305019e6d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3040005910 tx=0x7f3040005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:50.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.852+0000 7f304d7fa700 1 -- 172.21.15.3:0/3820539935 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304001d070 con 0x7f3050100640 2023-09-10T01:18:50.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.852+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3050199360 con 0x7f3050100640 2023-09-10T01:18:50.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.852+0000 7f304d7fa700 1 -- 172.21.15.3:0/3820539935 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3040022470 con 0x7f3050100640 2023-09-10T01:18:50.860 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.852+0000 7f304d7fa700 1 -- 172.21.15.3:0/3820539935 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040017650 con 0x7f3050100640 2023-09-10T01:18:50.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.852+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30501a56c0 con 0x7f3050100640 2023-09-10T01:18:50.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.853+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3050067480 con 0x7f3050100640 2023-09-10T01:18:50.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.855+0000 7f304d7fa700 1 -- 172.21.15.3:0/3820539935 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f304000fad0 con 0x7f3050100640 2023-09-10T01:18:50.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.856+0000 7f304d7fa700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f30380a07b0 0x7f30380a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.856+0000 7f304d7fa700 1 -- 172.21.15.3:0/3820539935 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f30400ce230 con 0x7f3050100640 2023-09-10T01:18:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.856+0000 7f304f7fe700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f30380a07b0 0x7f30380a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.857+0000 7f304f7fe700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f30380a07b0 0x7f30380a2c70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3050199bb0 tx=0x7f3044009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:50.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:50.859+0000 7f304d7fa700 1 -- 172.21.15.3:0/3820539935 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f3040097670 con 0x7f3050100640 2023-09-10T01:18:50.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:50 smithi136 ceph-mon[133708]: pgmap v491: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-09-10T01:18:50.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1375141505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:18:50.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:50 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2687615488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:18:51.070 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.070+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f30501105b0 con 0x7f3050100640 2023-09-10T01:18:51.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.071+0000 7f304d7fa700 1 -- 172.21.15.3:0/3820539935 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v111) v1 ==== 103+0+11 (secure 0 0 0) 0x7f304009ac90 con 0x7f3050100640 2023-09-10T01:18:51.072 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:18:51.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.073+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f30380a07b0 msgr2=0x7f30380a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:51.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.073+0000 7f30569ff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f30380a07b0 0x7f30380a2c70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3050199bb0 tx=0x7f3044009260 comp rx=0 tx=0).stop 2023-09-10T01:18:51.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.073+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050100640 msgr2=0x7f305019e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:51.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.073+0000 7f30569ff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050100640 0x7f305019e6d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3040005910 tx=0x7f3040005740 comp rx=0 tx=0).stop 2023-09-10T01:18:51.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 shutdown_connections 2023-09-10T01:18:51.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f30380a07b0 0x7f30380a2c70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3050100640 0x7f305019e6d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3050106650 0x7f3050198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 --2- 172.21.15.3:0/3820539935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3050101000 0x7f305019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 >> 172.21.15.3:0/3820539935 conn(0x7f30500fc170 msgr2=0x7f305010b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:51.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 shutdown_connections 2023-09-10T01:18:51.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.074+0000 7f30569ff700 1 -- 172.21.15.3:0/3820539935 wait complete. 2023-09-10T01:18:51.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-09-10T01:18:51.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 -- 172.21.15.3:0/4247924456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 msgr2=0x7f128010f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:51.302 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 --2- 172.21.15.3:0/4247924456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 0x7f128010f2d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1278009f90 tx=0x7f127800b330 comp rx=0 tx=0).stop 2023-09-10T01:18:51.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 -- 172.21.15.3:0/4247924456 shutdown_connections 2023-09-10T01:18:51.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 --2- 172.21.15.3:0/4247924456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1280072a30 0x7f1280072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 --2- 172.21.15.3:0/4247924456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 0x7f128010f2d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.303 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 --2- 172.21.15.3:0/4247924456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 0x7f128010c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 -- 172.21.15.3:0/4247924456 >> 172.21.15.3:0/4247924456 conn(0x7f12800fc110 msgr2=0x7f12800fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:51.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 -- 172.21.15.3:0/4247924456 shutdown_connections 2023-09-10T01:18:51.304 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.301+0000 7f12855ff700 1 -- 172.21.15.3:0/4247924456 wait complete. 2023-09-10T01:18:51.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 Processor -- start 2023-09-10T01:18:51.306 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 -- start start 2023-09-10T01:18:51.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1280072a30 0x7f128019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:51.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 0x7f128019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:51.307 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 0x7f12801988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:51.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280198f30 con 0x7f128010ce60 2023-09-10T01:18:51.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12801990a0 con 0x7f1280073350 2023-09-10T01:18:51.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.302+0000 7f12855ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280199260 con 0x7f1280072a30 2023-09-10T01:18:51.308 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 0x7f128019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:51.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f127f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 0x7f12801988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:51.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 0x7f128019ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:36794/0 (socket says 172.21.15.3:36794) 2023-09-10T01:18:51.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 -- 172.21.15.3:0/4132580390 learned_addr learned my addr 172.21.15.3:0/4132580390 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:18:51.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f127effd700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1280072a30 0x7f128019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:51.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 -- 172.21.15.3:0/4132580390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1280072a30 msgr2=0x7f128019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:51.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1280072a30 0x7f128019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.310 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 -- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 msgr2=0x7f12801988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:51.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 0x7f12801988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:51.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 -- 172.21.15.3:0/4132580390 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700095f0 con 0x7f1280073350 2023-09-10T01:18:51.311 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f127f7fe700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 0x7f12801988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:18:51.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.303+0000 7f1277fff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 0x7f128019ed90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f126800e9a0 tx=0x7f126800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:51.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.304+0000 7f127cff9700 1 -- 172.21.15.3:0/4132580390 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268009790 con 0x7f1280073350 2023-09-10T01:18:51.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.304+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1278009c40 con 0x7f1280073350 2023-09-10T01:18:51.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.304+0000 7f127cff9700 1 -- 172.21.15.3:0/4132580390 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1268010430 con 0x7f1280073350 2023-09-10T01:18:51.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.304+0000 7f127cff9700 1 -- 172.21.15.3:0/4132580390 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268018680 con 0x7f1280073350 2023-09-10T01:18:51.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.304+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12801a5660 con 0x7f1280073350 2023-09-10T01:18:51.313 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.305+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1280067480 con 0x7f1280073350 2023-09-10T01:18:51.314 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.307+0000 7f127cff9700 1 -- 172.21.15.3:0/4132580390 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f12680105a0 con 0x7f1280073350 2023-09-10T01:18:51.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.308+0000 7f127cff9700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f126c0a07b0 0x7f126c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:18:51.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.309+0000 7f127cff9700 1 -- 172.21.15.3:0/4132580390 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 9511+0+0 (secure 0 0 0) 0x7f1268014070 con 0x7f1280073350 2023-09-10T01:18:51.315 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.309+0000 7f127effd700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f126c0a07b0 0x7f126c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:18:51.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.309+0000 7f127effd700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f126c0a07b0 0x7f126c0a2c70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1270005fd0 tx=0x7f1270009310 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:18:51.316 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.311+0000 7f127cff9700 1 -- 172.21.15.3:0/4132580390 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1268095b20 con 0x7f1280073350 2023-09-10T01:18:51.585 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.584+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f1280199dd0 con 0x7f1280073350 2023-09-10T01:18:51.594 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.593+0000 7f127cff9700 1 -- 172.21.15.3:0/4132580390 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+730417 (secure 0 0 0) 0x7f1268026090 con 0x7f1280073350 2023-09-10T01:18:51.604 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:18:51.604 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:18:51.605 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 27, 2023-09-10T01:18:51.605 INFO:teuthology.orchestra.run.smithi003.stdout: "active_gid": 25667, 2023-09-10T01:18:51.605 INFO:teuthology.orchestra.run.smithi003.stdout: "active_name": "x", 2023-09-10T01:18:51.605 INFO:teuthology.orchestra.run.smithi003.stdout: "active_addrs": { 2023-09-10T01:18:51.606 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2023-09-10T01:18:51.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.606 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2023-09-10T01:18:51.607 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6800", 2023-09-10T01:18:51.607 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3336040255 2023-09-10T01:18:51.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.608 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2023-09-10T01:18:51.608 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2023-09-10T01:18:51.608 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3336040255 2023-09-10T01:18:51.608 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.609 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:51.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.609 INFO:teuthology.orchestra.run.smithi003.stdout: "active_addr": "172.21.15.3:6801/3336040255", 2023-09-10T01:18:51.610 INFO:teuthology.orchestra.run.smithi003.stdout: "active_change": "2023-09-10T01:03:12.190272+0000", 2023-09-10T01:18:51.610 INFO:teuthology.orchestra.run.smithi003.stdout: "active_mgr_features": 4540138322906710015, 2023-09-10T01:18:51.610 INFO:teuthology.orchestra.run.smithi003.stdout: "available": true, 2023-09-10T01:18:51.610 INFO:teuthology.orchestra.run.smithi003.stdout: "standbys": [ 2023-09-10T01:18:51.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.611 INFO:teuthology.orchestra.run.smithi003.stdout: "gid": 25606, 2023-09-10T01:18:51.611 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "z", 2023-09-10T01:18:51.611 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_features": 4540138322906710015, 2023-09-10T01:18:51.612 INFO:teuthology.orchestra.run.smithi003.stdout: "available_modules": [ 2023-09-10T01:18:51.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.612 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "alerts", 2023-09-10T01:18:51.612 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:51.613 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:51.613 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:51.613 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:51.613 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:51.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.614 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.615 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:51.615 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.615 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.615 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.616 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How frequently to reexamine health status", 2023-09-10T01:18:51.616 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.616 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.616 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.617 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:51.617 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:51.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.618 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.618 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.618 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.619 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.619 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.619 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.619 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.620 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.620 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.620 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.620 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.621 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.621 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.621 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.621 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_destination": { 2023-09-10T01:18:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_destination", 2023-09-10T01:18:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email address to send alerts to", 2023-09-10T01:18:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_from_name": { 2023-09-10T01:18:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_from_name", 2023-09-10T01:18:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "Ceph", 2023-09-10T01:18:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email From: name", 2023-09-10T01:18:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_host": { 2023-09-10T01:18:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_host", 2023-09-10T01:18:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP server", 2023-09-10T01:18:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_password": { 2023-09-10T01:18:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_password", 2023-09-10T01:18:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Password to authenticate with", 2023-09-10T01:18:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_port": { 2023-09-10T01:18:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_port", 2023-09-10T01:18:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "465", 2023-09-10T01:18:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP port", 2023-09-10T01:18:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_sender": { 2023-09-10T01:18:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_sender", 2023-09-10T01:18:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP envelope sender", 2023-09-10T01:18:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_ssl": { 2023-09-10T01:18:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_ssl", 2023-09-10T01:18:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use SSL to connect to SMTP server", 2023-09-10T01:18:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_user": { 2023-09-10T01:18:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_user", 2023-09-10T01:18:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "User to authenticate as", 2023-09-10T01:18:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "balancer", 2023-09-10T01:18:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "active": { 2023-09-10T01:18:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "active", 2023-09-10T01:18:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically balance PGs across cluster", 2023-09-10T01:18:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_time": { 2023-09-10T01:18:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_time", 2023-09-10T01:18:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0000", 2023-09-10T01:18:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "beginning time of day to automatically balance", 2023-09-10T01:18:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-09-10T01:18:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_weekday": { 2023-09-10T01:18:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_weekday", 2023-09-10T01:18:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2023-09-10T01:18:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2023-09-10T01:18:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-09-10T01:18:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-09-10T01:18:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_max_iterations": { 2023-09-10T01:18:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_max_iterations", 2023-09-10T01:18:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "25", 2023-09-10T01:18:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "250", 2023-09-10T01:18:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-09-10T01:18:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_metrics": { 2023-09-10T01:18:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_metrics", 2023-09-10T01:18:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "pgs,objects,bytes", 2023-09-10T01:18:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-09-10T01:18:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-09-10T01:18:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_step": { 2023-09-10T01:18:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_step", 2023-09-10T01:18:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.5", 2023-09-10T01:18:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0.001", 2023-09-10T01:18:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "0.999", 2023-09-10T01:18:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "aggressiveness of optimization", 2023-09-10T01:18:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-09-10T01:18:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "end_time": { 2023-09-10T01:18:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_time", 2023-09-10T01:18:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2359", 2023-09-10T01:18:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ending time of day to automatically balance", 2023-09-10T01:18:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-09-10T01:18:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "end_weekday": { 2023-09-10T01:18:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_weekday", 2023-09-10T01:18:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2023-09-10T01:18:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2023-09-10T01:18:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-09-10T01:18:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-09-10T01:18:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.717 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.717 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.717 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.717 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.718 INFO:teuthology.orchestra.run.smithi003.stdout: "min_score": { 2023-09-10T01:18:51.718 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_score", 2023-09-10T01:18:51.718 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:51.719 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.719 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.719 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:51.720 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.720 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.720 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.720 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-09-10T01:18:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2023-09-10T01:18:51.722 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2023-09-10T01:18:51.722 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "upmap", 2023-09-10T01:18:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-compat", 2023-09-10T01:18:51.724 INFO:teuthology.orchestra.run.smithi003.stdout: "none", 2023-09-10T01:18:51.725 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap" 2023-09-10T01:18:51.725 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.725 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Balancer mode", 2023-09-10T01:18:51.725 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.726 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.726 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_ids": { 2023-09-10T01:18:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_ids", 2023-09-10T01:18:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.727 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.728 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.728 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.728 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.729 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.729 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.729 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-09-10T01:18:51.729 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.730 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.732 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:51.732 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.732 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-09-10T01:18:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.733 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_deviation": { 2023-09-10T01:18:51.734 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_deviation", 2023-09-10T01:18:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.735 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:51.736 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:51.736 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.736 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "deviation below which no optimization is attempted", 2023-09-10T01:18:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-09-10T01:18:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.737 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_optimizations": { 2023-09-10T01:18:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_optimizations", 2023-09-10T01:18:51.738 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:51.739 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.739 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.739 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.740 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-09-10T01:18:51.741 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.741 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.741 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.741 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.742 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm", 2023-09-10T01:18:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:51.743 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_down_multiplier": { 2023-09-10T01:18:51.744 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_down_multiplier", 2023-09-10T01:18:51.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:51.744 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2023-09-10T01:18:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.745 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-09-10T01:18:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.747 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.747 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_refresh_rate": { 2023-09-10T01:18:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_refresh_rate", 2023-09-10T01:18:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.748 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "20", 2023-09-10T01:18:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.749 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.750 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-09-10T01:18:51.750 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.750 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_starting_port": { 2023-09-10T01:18:51.751 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_starting_port", 2023-09-10T01:18:51.752 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.752 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.752 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.753 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4721", 2023-09-10T01:18:51.753 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-09-10T01:18:51.754 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.755 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.755 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.756 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_ptrace": { 2023-09-10T01:18:51.756 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_ptrace", 2023-09-10T01:18:51.756 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.757 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-09-10T01:18:51.758 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-09-10T01:18:51.759 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.759 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_interval": { 2023-09-10T01:18:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_interval", 2023-09-10T01:18:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.760 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to autotune daemon memory", 2023-09-10T01:18:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.762 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.763 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.763 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_memory_target_ratio": { 2023-09-10T01:18:51.764 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_memory_target_ratio", 2023-09-10T01:18:51.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:51.764 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.7", 2023-09-10T01:18:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.765 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.766 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-09-10T01:18:51.766 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.766 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.767 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.767 INFO:teuthology.orchestra.run.smithi003.stdout: "cgroups_split": { 2023-09-10T01:18:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cgroups_split", 2023-09-10T01:18:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.768 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.769 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.770 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-09-10T01:18:51.770 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.770 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: "config_checks_enabled": { 2023-09-10T01:18:51.771 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_checks_enabled", 2023-09-10T01:18:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.772 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.773 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-09-10T01:18:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: "config_dashboard": { 2023-09-10T01:18:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_dashboard", 2023-09-10T01:18:51.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.776 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-09-10T01:18:51.777 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.778 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.778 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_alertmanager": { 2023-09-10T01:18:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_alertmanager", 2023-09-10T01:18:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.779 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-09-10T01:18:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.780 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.781 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.781 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:51.781 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.782 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_base": { 2023-09-10T01:18:51.783 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_base", 2023-09-10T01:18:51.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.783 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.783 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.784 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph", 2023-09-10T01:18:51.784 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.784 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Container image name, without the tag", 2023-09-10T01:18:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.785 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_elasticsearch": { 2023-09-10T01:18:51.786 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_elasticsearch", 2023-09-10T01:18:51.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.787 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.787 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-09-10T01:18:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "elasticsearch container image", 2023-09-10T01:18:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.789 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_grafana": { 2023-09-10T01:18:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_grafana", 2023-09-10T01:18:51.790 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.791 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.791 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.791 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-09-10T01:18:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.792 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.794 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_haproxy": { 2023-09-10T01:18:51.794 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_haproxy", 2023-09-10T01:18:51.794 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-09-10T01:18:51.795 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "HAproxy container image", 2023-09-10T01:18:51.796 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_agent": { 2023-09-10T01:18:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_agent", 2023-09-10T01:18:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-09-10T01:18:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.800 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.800 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger agent container image", 2023-09-10T01:18:51.800 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.801 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.801 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_collector": { 2023-09-10T01:18:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_collector", 2023-09-10T01:18:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.802 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.803 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.803 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-09-10T01:18:51.803 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger collector container image", 2023-09-10T01:18:51.804 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.805 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_query": { 2023-09-10T01:18:51.806 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_query", 2023-09-10T01:18:51.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.806 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-09-10T01:18:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.807 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger query container image", 2023-09-10T01:18:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.808 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.809 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.809 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_keepalived": { 2023-09-10T01:18:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_keepalived", 2023-09-10T01:18:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.810 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-09-10T01:18:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.811 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.812 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Keepalived container image", 2023-09-10T01:18:51.812 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.812 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.812 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.813 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_loki": { 2023-09-10T01:18:51.813 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_loki", 2023-09-10T01:18:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.814 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-09-10T01:18:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.815 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Loki container image", 2023-09-10T01:18:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.816 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_node_exporter": { 2023-09-10T01:18:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_node_exporter", 2023-09-10T01:18:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-09-10T01:18:51.818 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.819 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.819 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.819 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_nvmeof": { 2023-09-10T01:18:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_nvmeof", 2023-09-10T01:18:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.821 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.822 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.822 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.3", 2023-09-10T01:18:51.822 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Nvme-of container image", 2023-09-10T01:18:51.823 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.824 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_prometheus": { 2023-09-10T01:18:51.825 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_prometheus", 2023-09-10T01:18:51.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.825 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-09-10T01:18:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.826 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.827 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.827 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:51.827 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.827 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.828 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.828 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_promtail": { 2023-09-10T01:18:51.829 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_promtail", 2023-09-10T01:18:51.829 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.829 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-09-10T01:18:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.830 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Promtail container image", 2023-09-10T01:18:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.831 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_snmp_gateway": { 2023-09-10T01:18:51.832 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_snmp_gateway", 2023-09-10T01:18:51.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.833 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.833 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-09-10T01:18:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.834 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SNMP Gateway container image", 2023-09-10T01:18:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.835 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: "container_init": { 2023-09-10T01:18:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_init", 2023-09-10T01:18:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.838 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Run podman/docker with `--init`", 2023-09-10T01:18:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.840 INFO:teuthology.orchestra.run.smithi003.stdout: "daemon_cache_timeout": { 2023-09-10T01:18:51.840 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "daemon_cache_timeout", 2023-09-10T01:18:51.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:51.841 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.842 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.842 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.842 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-09-10T01:18:51.842 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.843 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.843 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: "default_cephadm_command_timeout": { 2023-09-10T01:18:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_cephadm_command_timeout", 2023-09-10T01:18:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.844 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "900", 2023-09-10T01:18:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.845 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.846 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.846 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-09-10T01:18:51.846 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.847 INFO:teuthology.orchestra.run.smithi003.stdout: "default_registry": { 2023-09-10T01:18:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_registry", 2023-09-10T01:18:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.848 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io", 2023-09-10T01:18:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.849 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.850 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-09-10T01:18:51.850 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.850 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: "device_cache_timeout": { 2023-09-10T01:18:51.851 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_cache_timeout", 2023-09-10T01:18:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.852 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1800", 2023-09-10T01:18:51.853 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.853 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.853 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache device inventory", 2023-09-10T01:18:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: "device_enhanced_scan": { 2023-09-10T01:18:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_enhanced_scan", 2023-09-10T01:18:51.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.856 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.856 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.856 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.857 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use libstoragemgmt during device scans", 2023-09-10T01:18:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.859 INFO:teuthology.orchestra.run.smithi003.stdout: "facts_cache_timeout": { 2023-09-10T01:18:51.859 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "facts_cache_timeout", 2023-09-10T01:18:51.859 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:51.860 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache host facts data", 2023-09-10T01:18:51.861 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.862 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.862 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.862 INFO:teuthology.orchestra.run.smithi003.stdout: "host_check_interval": { 2023-09-10T01:18:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "host_check_interval", 2023-09-10T01:18:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.863 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.864 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:51.864 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.864 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to perform a host check", 2023-09-10T01:18:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.865 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: "inventory_list_all": { 2023-09-10T01:18:51.866 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "inventory_list_all", 2023-09-10T01:18:51.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.867 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.867 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.868 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.868 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.868 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2023-09-10T01:18:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.869 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:51.870 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.872 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.872 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.872 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.873 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.874 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.875 INFO:teuthology.orchestra.run.smithi003.stdout: "log_refresh_metadata": { 2023-09-10T01:18:51.876 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_refresh_metadata", 2023-09-10T01:18:51.876 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.876 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.877 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-09-10T01:18:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.878 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.879 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.879 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:51.879 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.880 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.881 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-09-10T01:18:51.882 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.882 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.882 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:51.883 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:51.884 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.885 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.885 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.885 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.885 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.886 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.887 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.887 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.887 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.888 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.889 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.890 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.891 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf": { 2023-09-10T01:18:51.892 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-09-10T01:18:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.893 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.894 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-09-10T01:18:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.895 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-09-10T01:18:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-09-10T01:18:51.896 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "*", 2023-09-10T01:18:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.898 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.898 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.898 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-09-10T01:18:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: "max_count_per_host": { 2023-09-10T01:18:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_count_per_host", 2023-09-10T01:18:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.900 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.901 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.902 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.902 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of daemons per service per host", 2023-09-10T01:18:51.902 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.903 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd_draining_count": { 2023-09-10T01:18:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_osd_draining_count", 2023-09-10T01:18:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.905 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:51.905 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.905 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.906 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.906 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-09-10T01:18:51.906 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.906 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: "migration_current": { 2023-09-10T01:18:51.907 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "migration_current", 2023-09-10T01:18:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.908 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.909 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "internal - do not modify", 2023-09-10T01:18:51.910 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.910 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.910 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2023-09-10T01:18:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2023-09-10T01:18:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "root", 2023-09-10T01:18:51.912 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm-package", 2023-09-10T01:18:51.913 INFO:teuthology.orchestra.run.smithi003.stdout: "root" 2023-09-10T01:18:51.914 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.914 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "mode for remote execution of cephadm", 2023-09-10T01:18:51.914 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.915 INFO:teuthology.orchestra.run.smithi003.stdout: "prometheus_alerts_path": { 2023-09-10T01:18:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus_alerts_path", 2023-09-10T01:18:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.916 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.917 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-09-10T01:18:51.917 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.917 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-09-10T01:18:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.918 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_insecure": { 2023-09-10T01:18:51.919 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_insecure", 2023-09-10T01:18:51.920 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.920 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.920 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.921 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.922 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-09-10T01:18:51.922 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.922 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_password": { 2023-09-10T01:18:51.923 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_password", 2023-09-10T01:18:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.924 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.925 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.925 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.925 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.926 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-09-10T01:18:51.926 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.926 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.927 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.927 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_url": { 2023-09-10T01:18:51.928 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_url", 2023-09-10T01:18:51.928 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.928 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.929 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.929 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.929 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-09-10T01:18:51.930 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.931 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_username": { 2023-09-10T01:18:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_username", 2023-09-10T01:18:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.932 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.933 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-09-10T01:18:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.934 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: "secure_monitoring_stack": { 2023-09-10T01:18:51.935 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "secure_monitoring_stack", 2023-09-10T01:18:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-09-10T01:18:51.938 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.938 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.938 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: "service_discovery_port": { 2023-09-10T01:18:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "service_discovery_port", 2023-09-10T01:18:51.939 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8765", 2023-09-10T01:18:51.940 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.941 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.941 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.941 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "cephadm service discovery port", 2023-09-10T01:18:51.942 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.942 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.942 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.943 INFO:teuthology.orchestra.run.smithi003.stdout: "ssh_config_file": { 2023-09-10T01:18:51.943 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssh_config_file", 2023-09-10T01:18:51.943 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.943 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.944 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.944 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.944 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.944 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.945 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.945 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-09-10T01:18:51.945 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.946 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.946 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.946 INFO:teuthology.orchestra.run.smithi003.stdout: "use_agent": { 2023-09-10T01:18:51.947 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_agent", 2023-09-10T01:18:51.947 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.947 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.947 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.948 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.948 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.948 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.949 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.949 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-09-10T01:18:51.949 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.949 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.950 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.950 INFO:teuthology.orchestra.run.smithi003.stdout: "use_repo_digest": { 2023-09-10T01:18:51.950 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_repo_digest", 2023-09-10T01:18:51.951 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.951 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.951 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.952 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.952 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.952 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.952 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.953 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-09-10T01:18:51.953 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.953 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.953 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.954 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_failed_host_check": { 2023-09-10T01:18:51.954 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_failed_host_check", 2023-09-10T01:18:51.955 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.955 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.955 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.955 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.956 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.956 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.956 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.956 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if the host check fails", 2023-09-10T01:18:51.957 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.957 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.957 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.958 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_daemons": { 2023-09-10T01:18:51.958 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_daemons", 2023-09-10T01:18:51.958 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.959 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.959 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.959 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.959 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-09-10T01:18:51.960 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.961 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.961 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.961 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_hosts": { 2023-09-10T01:18:51.962 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_hosts", 2023-09-10T01:18:51.962 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.962 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.963 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.963 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:51.963 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.963 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.964 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.964 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-09-10T01:18:51.964 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.964 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.965 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.965 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.965 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.966 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crash", 2023-09-10T01:18:51.966 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:51.967 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:51.967 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:51.967 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:51.968 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:51.968 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.968 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.968 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.969 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:51.969 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.969 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.969 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.970 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.970 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.970 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.971 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.971 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.971 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.971 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.972 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.972 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.972 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.972 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.973 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:51.973 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:51.973 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.974 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.974 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.974 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.975 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.975 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.975 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.975 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.976 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.976 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.976 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.977 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:51.977 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:51.977 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:51.977 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.978 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.978 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:51.978 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.979 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.979 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:51.979 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:51.979 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:51.980 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:51.980 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:51.980 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:51.980 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:51.981 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:51.981 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.981 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.981 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.982 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.982 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:51.982 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:51.983 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:51.983 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.983 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.983 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:51.984 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.984 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.984 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.984 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.985 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.985 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.985 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.986 INFO:teuthology.orchestra.run.smithi003.stdout: "retain_interval": { 2023-09-10T01:18:51.986 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retain_interval", 2023-09-10T01:18:51.986 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.987 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.987 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.987 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "31536000", 2023-09-10T01:18:51.987 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.988 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.988 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.988 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain crashes before pruning them", 2023-09-10T01:18:51.988 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.989 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.989 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.990 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_recent_interval": { 2023-09-10T01:18:51.990 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_recent_interval", 2023-09-10T01:18:51.990 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:51.990 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.991 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:51.991 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1209600", 2023-09-10T01:18:51.991 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.991 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.992 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.992 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "time interval in which to warn about recent crashes", 2023-09-10T01:18:51.992 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.992 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.993 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:51.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:51.994 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dashboard", 2023-09-10T01:18:51.994 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:51.995 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:51.995 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:51.995 INFO:teuthology.orchestra.run.smithi003.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-09-10T01:18:51.995 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-09-10T01:18:51.996 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:51.996 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:51.996 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:51.996 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:51.997 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:51.997 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:51.997 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:51.998 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:51.998 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:51.998 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:51.998 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:51.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:51.999 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_HOST": { 2023-09-10T01:18:51.999 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_HOST", 2023-09-10T01:18:51.999 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.000 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.000 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.000 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.000 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.001 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.001 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.001 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.002 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.002 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.002 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.003 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-09-10T01:18:52.003 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-09-10T01:18:52.003 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.004 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.004 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.004 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.005 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.005 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.005 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.005 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.006 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.006 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.006 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.007 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_ENABLED": { 2023-09-10T01:18:52.007 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_ENABLED", 2023-09-10T01:18:52.007 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.008 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.008 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.008 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.008 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.009 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.009 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.009 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.009 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.010 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.010 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.011 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-09-10T01:18:52.011 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-09-10T01:18:52.011 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.011 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.012 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.012 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.012 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.012 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.013 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.013 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.013 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.013 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.014 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.014 INFO:teuthology.orchestra.run.smithi003.stdout: "ENABLE_BROWSABLE_API": { 2023-09-10T01:18:52.015 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ENABLE_BROWSABLE_API", 2023-09-10T01:18:52.015 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.015 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.015 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.016 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.016 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.016 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.016 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.017 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.017 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.017 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.017 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.018 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-09-10T01:18:52.018 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-09-10T01:18:52.019 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.019 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.019 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.019 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.020 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.020 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.020 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.020 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.021 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.021 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.021 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.022 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-09-10T01:18:52.022 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-09-10T01:18:52.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.023 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.023 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.023 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.023 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.024 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.024 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.024 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.025 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.025 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.025 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.026 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-09-10T01:18:52.026 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-09-10T01:18:52.026 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.026 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.027 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.027 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.027 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.028 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.028 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.028 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.028 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.029 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.029 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.030 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-09-10T01:18:52.030 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-09-10T01:18:52.030 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.031 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.031 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.031 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.032 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.032 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.032 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.032 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.033 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.033 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.033 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.034 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_NFS": { 2023-09-10T01:18:52.034 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-09-10T01:18:52.034 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.035 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.035 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.035 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.035 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.036 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.036 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.036 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.036 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.037 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.037 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.037 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RBD": { 2023-09-10T01:18:52.038 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-09-10T01:18:52.038 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.038 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.039 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.039 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.039 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.040 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.040 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.040 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.040 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.041 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.041 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RGW": { 2023-09-10T01:18:52.042 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-09-10T01:18:52.042 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.042 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.042 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.043 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.043 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.043 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.043 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.044 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.044 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.044 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.045 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.045 INFO:teuthology.orchestra.run.smithi003.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-09-10T01:18:52.045 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-09-10T01:18:52.046 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.046 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.046 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.046 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.047 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.047 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.047 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.047 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.048 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.048 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.048 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.049 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_PASSWORD": { 2023-09-10T01:18:52.049 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_PASSWORD", 2023-09-10T01:18:52.049 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.050 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.050 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.050 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:52.050 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.051 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.051 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.051 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.052 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.052 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.052 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.053 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-09-10T01:18:52.053 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-09-10T01:18:52.053 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.053 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.054 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.054 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.054 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.055 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.055 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.055 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.055 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.056 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.056 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.056 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_URL": { 2023-09-10T01:18:52.057 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_URL", 2023-09-10T01:18:52.057 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.057 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.058 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.058 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.058 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.058 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.059 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.059 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.059 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.062 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.062 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.062 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_USERNAME": { 2023-09-10T01:18:52.063 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_USERNAME", 2023-09-10T01:18:52.063 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.063 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.063 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.064 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:52.064 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.064 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.065 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.065 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.065 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.065 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.066 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.066 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-09-10T01:18:52.066 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-09-10T01:18:52.067 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.067 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.067 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.067 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.068 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.068 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.068 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.068 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.069 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.069 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.069 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.070 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-09-10T01:18:52.070 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-09-10T01:18:52.070 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.071 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.071 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.071 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.071 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.072 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.072 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.072 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.073 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.073 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.073 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.074 INFO:teuthology.orchestra.run.smithi003.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-09-10T01:18:52.074 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-09-10T01:18:52.074 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.074 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.075 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.075 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.075 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.075 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.076 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.076 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.076 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.076 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.077 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.077 INFO:teuthology.orchestra.run.smithi003.stdout: "ISSUE_TRACKER_API_KEY": { 2023-09-10T01:18:52.077 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-09-10T01:18:52.078 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.078 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.078 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.079 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.079 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.079 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.079 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.080 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.080 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.080 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.080 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.081 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_HOST": { 2023-09-10T01:18:52.081 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_HOST", 2023-09-10T01:18:52.081 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.082 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.082 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.082 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.082 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.083 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.083 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.083 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.083 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.084 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.084 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.084 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-09-10T01:18:52.085 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-09-10T01:18:52.085 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.085 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.086 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.086 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.086 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.086 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.087 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.087 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.087 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.087 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.088 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.088 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-09-10T01:18:52.089 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-09-10T01:18:52.089 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.089 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.089 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.090 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.090 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.090 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.090 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.091 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.091 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.091 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.092 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.092 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-09-10T01:18:52.092 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-09-10T01:18:52.093 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.093 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.093 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.093 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.094 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.094 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.094 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.095 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.095 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.095 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.095 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.096 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-09-10T01:18:52.096 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-09-10T01:18:52.096 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.097 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.097 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.097 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.097 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.098 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.098 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.098 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.098 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.099 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.099 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.100 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-09-10T01:18:52.100 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-09-10T01:18:52.100 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.100 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.101 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.101 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.101 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.101 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.102 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.102 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.102 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.102 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.103 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.104 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-09-10T01:18:52.104 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-09-10T01:18:52.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.105 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.105 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.105 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.105 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.106 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.106 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.106 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.106 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.107 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.107 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.108 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-09-10T01:18:52.108 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-09-10T01:18:52.108 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.108 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.109 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.109 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.109 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.109 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.110 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.110 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.110 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.110 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.111 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.111 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-09-10T01:18:52.112 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-09-10T01:18:52.112 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.112 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.112 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.113 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.113 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.113 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.113 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.114 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.114 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.114 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.115 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.115 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_ENABLED": { 2023-09-10T01:18:52.115 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_ENABLED", 2023-09-10T01:18:52.116 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.116 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.116 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.116 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.117 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.117 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.117 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.117 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.118 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.118 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.118 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.119 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-09-10T01:18:52.119 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-09-10T01:18:52.119 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.119 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.120 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.120 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-09-10T01:18:52.120 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.120 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.121 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.121 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.121 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.122 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.122 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.122 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-09-10T01:18:52.123 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-09-10T01:18:52.123 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.123 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.123 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.124 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:52.124 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.124 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.124 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.125 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.125 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.125 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.126 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.126 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-09-10T01:18:52.126 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-09-10T01:18:52.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.127 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.127 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.128 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8", 2023-09-10T01:18:52.128 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.128 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.128 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.129 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.129 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.129 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.129 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.130 INFO:teuthology.orchestra.run.smithi003.stdout: "REST_REQUESTS_TIMEOUT": { 2023-09-10T01:18:52.130 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-09-10T01:18:52.130 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.131 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.131 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.131 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "45", 2023-09-10T01:18:52.132 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.132 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.132 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.132 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.133 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.133 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.133 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.134 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ACCESS_KEY": { 2023-09-10T01:18:52.134 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ACCESS_KEY", 2023-09-10T01:18:52.134 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.134 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.135 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.135 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.135 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.136 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.136 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.136 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.136 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.137 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.137 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.137 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-09-10T01:18:52.138 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-09-10T01:18:52.138 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.138 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.138 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.139 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:52.139 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.139 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.139 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.140 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.140 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.140 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.141 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.141 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SECRET_KEY": { 2023-09-10T01:18:52.141 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SECRET_KEY", 2023-09-10T01:18:52.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.142 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.142 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.142 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.143 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.143 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.143 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.144 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.144 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.144 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.144 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.145 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SSL_VERIFY": { 2023-09-10T01:18:52.145 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SSL_VERIFY", 2023-09-10T01:18:52.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.146 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.146 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.146 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.146 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.147 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.147 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.147 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.147 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.148 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.148 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.148 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-09-10T01:18:52.149 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-09-10T01:18:52.149 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.149 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.150 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.150 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:52.150 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.150 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.151 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.151 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.151 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.151 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.152 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.152 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-09-10T01:18:52.152 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-09-10T01:18:52.153 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.153 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.153 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.153 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:52.154 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.154 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.154 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.155 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.155 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.155 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.155 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.156 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-09-10T01:18:52.156 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-09-10T01:18:52.156 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.157 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.157 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.157 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:52.157 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.158 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.158 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.158 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.158 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.159 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.159 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.159 INFO:teuthology.orchestra.run.smithi003.stdout: "cross_origin_url": { 2023-09-10T01:18:52.160 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cross_origin_url", 2023-09-10T01:18:52.160 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.160 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.160 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.161 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.161 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.161 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.161 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.162 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.162 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.162 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.163 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.163 INFO:teuthology.orchestra.run.smithi003.stdout: "crt_file": { 2023-09-10T01:18:52.163 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crt_file", 2023-09-10T01:18:52.164 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.164 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.164 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.164 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.165 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.165 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.165 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.165 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.166 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.166 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.166 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.167 INFO:teuthology.orchestra.run.smithi003.stdout: "debug": { 2023-09-10T01:18:52.167 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "debug", 2023-09-10T01:18:52.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.168 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.168 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.169 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.169 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.169 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.169 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.170 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable/disable debug options", 2023-09-10T01:18:52.170 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.170 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.171 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.171 INFO:teuthology.orchestra.run.smithi003.stdout: "jwt_token_ttl": { 2023-09-10T01:18:52.171 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "jwt_token_ttl", 2023-09-10T01:18:52.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.172 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.172 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.172 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "28800", 2023-09-10T01:18:52.173 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.173 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.173 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.173 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.174 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.174 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.174 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.175 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2023-09-10T01:18:52.175 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2023-09-10T01:18:52.175 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.175 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.176 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.176 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.176 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.177 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.177 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.177 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.177 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.178 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.178 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.178 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.179 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.179 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.179 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "motd": { 2023-09-10T01:18:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "motd", 2023-09-10T01:18:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "The message of the day", 2023-09-10T01:18:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect_resolve_ip_addr": { 2023-09-10T01:18:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "redirect_resolve_ip_addr", 2023-09-10T01:18:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2023-09-10T01:18:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8080", 2023-09-10T01:18:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2023-09-10T01:18:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2023-09-10T01:18:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl_server_port": { 2023-09-10T01:18:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl_server_port", 2023-09-10T01:18:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8443", 2023-09-10T01:18:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2023-09-10T01:18:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2023-09-10T01:18:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "redirect", 2023-09-10T01:18:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect" 2023-09-10T01:18:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2023-09-10T01:18:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2023-09-10T01:18:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2023-09-10T01:18:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2023-09-10T01:18:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2023-09-10T01:18:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "url_prefix": { 2023-09-10T01:18:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url_prefix", 2023-09-10T01:18:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2023-09-10T01:18:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_monitoring": { 2023-09-10T01:18:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_monitoring", 2023-09-10T01:18:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "monitor device health metrics", 2023-09-10T01:18:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "mark_out_threshold": { 2023-09-10T01:18:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mark_out_threshold", 2023-09-10T01:18:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2419200", 2023-09-10T01:18:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-09-10T01:18:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": { 2023-09-10T01:18:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_name", 2023-09-10T01:18:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "device_health_metrics", 2023-09-10T01:18:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name of pool in which to store device health metrics", 2023-09-10T01:18:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "retention_period": { 2023-09-10T01:18:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retention_period", 2023-09-10T01:18:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15552000", 2023-09-10T01:18:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain device health metrics", 2023-09-10T01:18:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_frequency": { 2023-09-10T01:18:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_frequency", 2023-09-10T01:18:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2023-09-10T01:18:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to scrape device health metrics", 2023-09-10T01:18:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "self_heal": { 2023-09-10T01:18:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "self_heal", 2023-09-10T01:18:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-09-10T01:18:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and check device health", 2023-09-10T01:18:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_threshold": { 2023-09-10T01:18:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_threshold", 2023-09-10T01:18:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7257600", 2023-09-10T01:18:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-09-10T01:18:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "diskprediction_local", 2023-09-10T01:18:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.284 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.284 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.284 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.285 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.285 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.285 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.285 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.286 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.286 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.286 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.286 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.287 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.287 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.287 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.288 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.288 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.288 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.288 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.289 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.289 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.289 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.290 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.290 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.290 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.290 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.291 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.291 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.291 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.292 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.292 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.292 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.292 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.293 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.293 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.293 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.294 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.294 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.294 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.294 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.295 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.295 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.295 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.295 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.296 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.296 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.296 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.297 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.297 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.297 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.297 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.298 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.298 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.298 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.298 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.299 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.299 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.299 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.299 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.300 INFO:teuthology.orchestra.run.smithi003.stdout: "predict_interval": { 2023-09-10T01:18:52.300 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predict_interval", 2023-09-10T01:18:52.301 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.301 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.301 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.301 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2023-09-10T01:18:52.302 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.302 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.302 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.302 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.303 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.303 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.303 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.304 INFO:teuthology.orchestra.run.smithi003.stdout: "predictor_model": { 2023-09-10T01:18:52.304 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predictor_model", 2023-09-10T01:18:52.304 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.305 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.305 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.305 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "prophetstor", 2023-09-10T01:18:52.305 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.306 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.306 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.306 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.306 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.307 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.307 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.307 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:52.308 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:52.308 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.308 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.308 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.309 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:52.309 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.309 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.309 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.310 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.310 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.310 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.311 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.311 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.311 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.312 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "influx", 2023-09-10T01:18:52.312 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": false, 2023-09-10T01:18:52.312 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "influxdb python module not found", 2023-09-10T01:18:52.313 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.313 INFO:teuthology.orchestra.run.smithi003.stdout: "batch_size": { 2023-09-10T01:18:52.313 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "batch_size", 2023-09-10T01:18:52.313 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.314 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.314 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.314 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5000", 2023-09-10T01:18:52.315 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.315 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.315 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.315 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-09-10T01:18:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.317 INFO:teuthology.orchestra.run.smithi003.stdout: "database": { 2023-09-10T01:18:52.317 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "database", 2023-09-10T01:18:52.317 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.317 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.318 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.318 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "ceph", 2023-09-10T01:18:52.318 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.318 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.319 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.319 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-09-10T01:18:52.319 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.320 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.320 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.320 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": { 2023-09-10T01:18:52.321 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hostname", 2023-09-10T01:18:52.321 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.321 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.321 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.322 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.322 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.322 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.322 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.323 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server hostname", 2023-09-10T01:18:52.323 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.323 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.323 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.324 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:52.324 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:52.324 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.325 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.325 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.325 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "30", 2023-09-10T01:18:52.325 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "5", 2023-09-10T01:18:52.326 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.326 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.326 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-09-10T01:18:52.327 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.327 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.327 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.328 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.328 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.328 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.328 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.329 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.329 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.329 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.329 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.330 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.330 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.330 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.330 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.331 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.331 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.331 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.331 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.332 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.332 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.332 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.332 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.333 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.333 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.334 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.334 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.334 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.334 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.335 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.335 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.335 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.335 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.336 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.336 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.336 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.337 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.337 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.337 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.338 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.338 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.338 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.338 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.339 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.339 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.339 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.339 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.340 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.340 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.340 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.340 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.341 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.341 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.341 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.341 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.342 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.342 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.343 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.343 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.343 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.343 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.344 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.344 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.344 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.344 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.345 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.345 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.345 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.346 INFO:teuthology.orchestra.run.smithi003.stdout: "password": { 2023-09-10T01:18:52.346 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "password", 2023-09-10T01:18:52.346 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.347 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.347 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.347 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.347 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.348 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.348 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.348 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "password of InfluxDB server user", 2023-09-10T01:18:52.348 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.349 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.349 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.350 INFO:teuthology.orchestra.run.smithi003.stdout: "port": { 2023-09-10T01:18:52.350 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "port", 2023-09-10T01:18:52.350 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.350 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.351 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.351 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8086", 2023-09-10T01:18:52.351 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.351 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.352 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.352 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server port", 2023-09-10T01:18:52.352 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.353 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.353 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.353 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2023-09-10T01:18:52.354 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2023-09-10T01:18:52.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.354 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "false", 2023-09-10T01:18:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.355 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-09-10T01:18:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.356 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "threads": { 2023-09-10T01:18:52.357 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threads", 2023-09-10T01:18:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.358 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "32", 2023-09-10T01:18:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.359 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-09-10T01:18:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "username": { 2023-09-10T01:18:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "username", 2023-09-10T01:18:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.362 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "username of InfluxDB server user", 2023-09-10T01:18:52.363 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.364 INFO:teuthology.orchestra.run.smithi003.stdout: "verify_ssl": { 2023-09-10T01:18:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "verify_ssl", 2023-09-10T01:18:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.365 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "true", 2023-09-10T01:18:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.366 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-09-10T01:18:52.367 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "insights", 2023-09-10T01:18:52.369 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.370 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.371 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.373 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.374 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.376 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.377 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.378 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.381 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.382 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.383 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.384 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.385 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.386 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.387 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.388 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "iostat", 2023-09-10T01:18:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.390 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.392 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.393 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.394 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.395 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.397 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.398 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.399 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.400 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.401 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.402 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.403 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.404 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.405 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.406 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.408 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.409 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.410 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.411 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "k8sevents", 2023-09-10T01:18:52.411 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_event_retention_days": { 2023-09-10T01:18:52.412 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ceph_event_retention_days", 2023-09-10T01:18:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.413 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7", 2023-09-10T01:18:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.414 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Days to hold ceph event information within local cache", 2023-09-10T01:18:52.415 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.415 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.415 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: "config_check_secs": { 2023-09-10T01:18:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_check_secs", 2023-09-10T01:18:52.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "10", 2023-09-10T01:18:52.418 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.418 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.418 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-09-10T01:18:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.420 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.421 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.422 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.422 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.422 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.423 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.424 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.424 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.424 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.425 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.425 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.425 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.426 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.427 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.428 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.428 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.428 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.428 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.429 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.429 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.430 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.430 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.430 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.430 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.431 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.432 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.433 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.434 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.434 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.436 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.436 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.436 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.436 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.437 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.439 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "localpool", 2023-09-10T01:18:52.439 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.439 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.439 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.440 INFO:teuthology.orchestra.run.smithi003.stdout: "failure_domain": { 2023-09-10T01:18:52.440 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "failure_domain", 2023-09-10T01:18:52.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "host", 2023-09-10T01:18:52.441 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.442 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.442 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.442 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "failure domain for any created local pool", 2023-09-10T01:18:52.443 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-09-10T01:18:52.443 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.443 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.444 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.445 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.445 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.445 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.447 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.448 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.449 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.451 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.452 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.453 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.454 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.454 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.454 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.455 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.456 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.457 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.457 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.457 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.457 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.458 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.458 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.459 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.460 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.461 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.462 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.462 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": { 2023-09-10T01:18:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_size", 2023-09-10T01:18:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.463 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.464 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default min_size for any created local pool", 2023-09-10T01:18:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-09-10T01:18:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.465 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: "num_rep": { 2023-09-10T01:18:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "num_rep", 2023-09-10T01:18:52.466 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3", 2023-09-10T01:18:52.467 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default replica count for any created local pool", 2023-09-10T01:18:52.468 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.469 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": { 2023-09-10T01:18:52.470 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_num", 2023-09-10T01:18:52.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.470 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "128", 2023-09-10T01:18:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.471 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default pg_num for any created local pool", 2023-09-10T01:18:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.473 INFO:teuthology.orchestra.run.smithi003.stdout: "prefix": { 2023-09-10T01:18:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prefix", 2023-09-10T01:18:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.474 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.475 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name prefix for any created local pool", 2023-09-10T01:18:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: "subtree": { 2023-09-10T01:18:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "subtree", 2023-09-10T01:18:52.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "rack", 2023-09-10T01:18:52.478 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "CRUSH level for which to create a local pool", 2023-09-10T01:18:52.479 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-09-10T01:18:52.480 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.480 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.481 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mds_autoscaler", 2023-09-10T01:18:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.482 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.483 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.484 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.484 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.484 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.485 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.486 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.487 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.488 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.489 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.490 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.491 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.494 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.495 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.497 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.498 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.499 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.500 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirroring", 2023-09-10T01:18:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.502 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.504 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.505 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.506 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.507 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.509 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.510 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.511 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.512 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.513 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.514 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.515 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.516 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.517 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.518 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.519 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.520 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.521 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "nfs", 2023-09-10T01:18:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.523 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.524 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.525 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.526 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.527 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.528 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.529 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.530 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.531 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.536 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.537 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.538 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.539 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.540 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.541 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2023-09-10T01:18:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.543 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "fail_fs": { 2023-09-10T01:18:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "fail_fs", 2023-09-10T01:18:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.544 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.545 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-09-10T01:18:52.546 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.547 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.548 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.549 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.550 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.551 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.552 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.553 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.554 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.555 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.558 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.559 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.560 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.561 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.562 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.563 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.564 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.565 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator": { 2023-09-10T01:18:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2023-09-10T01:18:52.566 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.567 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.568 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2023-09-10T01:18:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "rook", 2023-09-10T01:18:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "test_orchestrator" 2023-09-10T01:18:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.569 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Orchestrator backend", 2023-09-10T01:18:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_perf_query", 2023-09-10T01:18:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.572 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.573 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.574 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.575 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.576 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.577 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.579 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.580 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.582 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.583 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.584 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.585 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.586 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.587 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.588 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.589 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.590 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.591 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_support", 2023-09-10T01:18:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.592 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.593 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.594 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.595 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.596 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.597 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.598 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.599 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.600 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.602 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.603 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.604 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.605 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.606 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.607 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.608 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.609 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.610 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_autoscaler", 2023-09-10T01:18:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.612 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.613 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.614 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.615 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.616 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.617 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.618 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.619 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.620 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.622 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.622 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.623 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.624 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.625 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.626 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.627 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.628 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.629 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.630 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: "noautoscale": { 2023-09-10T01:18:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "noautoscale", 2023-09-10T01:18:52.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.632 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "global autoscale flag", 2023-09-10T01:18:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-09-10T01:18:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: "threshold": { 2023-09-10T01:18:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threshold", 2023-09-10T01:18:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2023-09-10T01:18:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1.0", 2023-09-10T01:18:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.640 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "scaling threshold", 2023-09-10T01:18:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-09-10T01:18:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.642 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "progress", 2023-09-10T01:18:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.643 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_pg_recovery_event": { 2023-09-10T01:18:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_pg_recovery_event", 2023-09-10T01:18:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.644 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.645 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.645 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.645 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow the module to show pg recovery progress", 2023-09-10T01:18:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2023-09-10T01:18:52.647 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2023-09-10T01:18:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.648 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.649 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.652 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.653 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.654 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.655 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.656 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.657 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.658 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.658 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.658 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.659 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.660 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.661 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.661 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.662 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.663 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.664 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.665 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.666 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.667 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: "max_completed_events": { 2023-09-10T01:18:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_completed_events", 2023-09-10T01:18:52.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "50", 2023-09-10T01:18:52.670 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.671 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "number of past completed events to remember", 2023-09-10T01:18:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.673 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.674 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.674 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:52.674 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long the module is going to sleep", 2023-09-10T01:18:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.675 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.676 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus", 2023-09-10T01:18:52.677 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2023-09-10T01:18:52.678 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cache", 2023-09-10T01:18:52.679 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.679 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.679 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.679 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.680 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.681 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: "exclude_perf_counters": { 2023-09-10T01:18:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "exclude_perf_counters", 2023-09-10T01:18:52.682 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.683 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-09-10T01:18:52.684 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-09-10T01:18:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.686 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.686 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.687 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.688 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.689 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.690 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.691 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.692 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.692 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.693 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.695 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.695 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.695 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.696 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.696 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.696 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.696 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.697 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.697 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.697 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.698 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.698 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.698 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.698 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.701 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.701 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.702 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.702 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.702 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.703 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools": { 2023-09-10T01:18:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools", 2023-09-10T01:18:52.704 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.705 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.706 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.706 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.706 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.708 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools_refresh_interval": { 2023-09-10T01:18:52.708 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-09-10T01:18:52.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300", 2023-09-10T01:18:52.709 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.710 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.711 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_interval": { 2023-09-10T01:18:52.712 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_interval", 2023-09-10T01:18:52.712 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:52.712 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15.0", 2023-09-10T01:18:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.713 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.714 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.714 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.714 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.714 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:52.715 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.716 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2023-09-10T01:18:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.717 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-09-10T01:18:52.718 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.718 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.718 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.719 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:52.719 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:52.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.720 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.720 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.720 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "9283", 2023-09-10T01:18:52.720 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.721 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.721 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.721 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-09-10T01:18:52.721 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.722 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.722 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: "stale_cache_strategy": { 2023-09-10T01:18:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stale_cache_strategy", 2023-09-10T01:18:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "log", 2023-09-10T01:18:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.725 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.725 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.725 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.725 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.726 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.726 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2023-09-10T01:18:52.727 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2023-09-10T01:18:52.727 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.727 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.727 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "default", 2023-09-10T01:18:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.728 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.729 INFO:teuthology.orchestra.run.smithi003.stdout: "default", 2023-09-10T01:18:52.729 INFO:teuthology.orchestra.run.smithi003.stdout: "error" 2023-09-10T01:18:52.729 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.730 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.730 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.730 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.731 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.731 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2023-09-10T01:18:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2023-09-10T01:18:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.732 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.733 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2023-09-10T01:18:52.733 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2023-09-10T01:18:52.733 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2023-09-10T01:18:52.733 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.734 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.734 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.734 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.735 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.736 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2023-09-10T01:18:52.736 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.736 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.737 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.737 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.737 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.737 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.738 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.738 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.738 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.739 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.739 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.739 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.739 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.740 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.741 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.741 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.741 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.741 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.742 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.743 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.743 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.743 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.743 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.744 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.744 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.744 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.744 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.745 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.745 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.745 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.746 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.746 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.747 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.748 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.748 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.748 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.749 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.749 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.749 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.749 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.750 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.751 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.751 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.751 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.752 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.752 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.752 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.753 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.753 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.753 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.754 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.754 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.754 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.754 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.755 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.755 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.755 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_snap_create": { 2023-09-10T01:18:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_snap_create", 2023-09-10T01:18:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.757 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.757 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.757 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:52.757 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.758 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.758 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.758 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.759 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.759 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.759 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.760 INFO:teuthology.orchestra.run.smithi003.stdout: "mirror_snapshot_schedule": { 2023-09-10T01:18:52.760 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirror_snapshot_schedule", 2023-09-10T01:18:52.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.760 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.761 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.761 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.761 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.761 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.762 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.762 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.762 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.763 INFO:teuthology.orchestra.run.smithi003.stdout: "trash_purge_schedule": { 2023-09-10T01:18:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "trash_purge_schedule", 2023-09-10T01:18:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.766 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.766 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.766 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.767 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "restful", 2023-09-10T01:18:52.768 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_auth": { 2023-09-10T01:18:52.769 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_auth", 2023-09-10T01:18:52.770 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.770 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.770 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.771 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.771 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.771 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.771 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.772 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.772 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.772 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.772 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.773 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2023-09-10T01:18:52.773 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2023-09-10T01:18:52.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.774 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.774 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.774 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.775 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.775 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.775 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.775 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.777 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.777 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.777 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.778 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.778 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.778 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.778 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.779 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.779 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.779 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.779 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.780 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.780 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.780 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.780 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.781 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.781 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.781 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.782 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.782 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.782 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.783 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.783 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.783 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.784 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.784 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.784 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.785 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.785 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.785 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.785 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.786 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.786 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.786 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.787 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.787 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.787 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.788 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.788 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.788 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.788 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.789 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.789 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.789 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.790 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.790 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.790 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.790 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.791 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.791 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.791 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.792 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.792 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.792 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.792 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.794 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.794 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.794 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.795 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.795 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.796 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:52.796 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:52.796 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.796 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.797 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.797 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.797 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.798 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.798 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.798 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.798 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.799 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.799 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.799 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:52.800 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:52.800 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.800 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.800 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.801 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.801 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.801 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.802 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.802 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.802 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.802 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.803 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.804 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rgw", 2023-09-10T01:18:52.804 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.806 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.806 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.807 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.807 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.807 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.807 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.808 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.809 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.810 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.810 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.810 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.811 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.812 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.812 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.812 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.812 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.813 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.814 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.814 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.815 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.816 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.816 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.816 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.816 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.817 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.817 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.817 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.817 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.818 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.818 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.818 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.818 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.820 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.820 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.820 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.821 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.822 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.822 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.823 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.823 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.823 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.823 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.824 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.826 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rook", 2023-09-10T01:18:52.826 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.826 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.827 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.827 INFO:teuthology.orchestra.run.smithi003.stdout: "drive_group_interval": { 2023-09-10T01:18:52.827 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "drive_group_interval", 2023-09-10T01:18:52.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:52.828 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.828 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.828 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300.0", 2023-09-10T01:18:52.829 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.829 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.829 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.829 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-09-10T01:18:52.830 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.830 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.830 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.831 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.831 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.831 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.832 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.832 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.832 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.832 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.833 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.833 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.833 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.833 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.834 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.834 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.834 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.835 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.835 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.835 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.835 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.836 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.836 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.836 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.837 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.837 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.837 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.838 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.838 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.838 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.839 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.839 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.839 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.839 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.840 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.840 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.840 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.841 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.841 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.841 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.842 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.842 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.842 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.842 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.843 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.843 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.843 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.843 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.844 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.844 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.844 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.845 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.845 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.845 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.845 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.846 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.846 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.846 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.847 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.847 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.847 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.848 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.848 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.848 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.848 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.849 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.849 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.849 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.850 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_class": { 2023-09-10T01:18:52.850 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "storage_class", 2023-09-10T01:18:52.850 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.851 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.851 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.851 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "local", 2023-09-10T01:18:52.851 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.852 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.852 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.852 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-09-10T01:18:52.853 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.853 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.853 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.854 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.854 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "selftest", 2023-09-10T01:18:52.855 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.855 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.855 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.855 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.856 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.856 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.856 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.856 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.857 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.857 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.857 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.857 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.858 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.858 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.858 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.859 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.859 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.859 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.859 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.860 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.860 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.860 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.860 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.861 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.861 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.862 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.862 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.862 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.862 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.863 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.863 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.863 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.863 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.864 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.864 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.864 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.865 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.865 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.865 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.866 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.866 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.866 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.867 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.867 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.867 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.867 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.868 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.868 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.868 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.868 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.869 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.869 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.869 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.869 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.870 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.870 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.870 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.871 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.871 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.872 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.872 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.872 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.872 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.873 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.873 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.873 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.873 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.874 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.874 INFO:teuthology.orchestra.run.smithi003.stdout: "roption1": { 2023-09-10T01:18:52.874 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption1", 2023-09-10T01:18:52.875 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.875 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.875 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.875 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.876 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.876 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.876 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.877 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.877 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.877 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.877 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.878 INFO:teuthology.orchestra.run.smithi003.stdout: "roption2": { 2023-09-10T01:18:52.878 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption2", 2023-09-10T01:18:52.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.879 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.879 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.879 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "xyz", 2023-09-10T01:18:52.880 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.880 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.880 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.880 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.881 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.881 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.881 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.882 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption1": { 2023-09-10T01:18:52.882 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption1", 2023-09-10T01:18:52.882 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.883 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.883 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.883 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.883 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.884 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.884 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.884 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.884 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.885 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.885 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.886 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption2": { 2023-09-10T01:18:52.886 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption2", 2023-09-10T01:18:52.886 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.886 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.887 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.887 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.887 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.888 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.888 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.888 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.888 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.889 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.889 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.889 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption3": { 2023-09-10T01:18:52.890 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption3", 2023-09-10T01:18:52.890 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:52.890 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.891 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.891 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.891 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.891 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.892 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.892 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.892 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.892 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.893 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.893 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption4": { 2023-09-10T01:18:52.893 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption4", 2023-09-10T01:18:52.894 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.894 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.894 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.895 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.895 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.895 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.895 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption5": { 2023-09-10T01:18:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption5", 2023-09-10T01:18:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption6": { 2023-09-10T01:18:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption6", 2023-09-10T01:18:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption7": { 2023-09-10T01:18:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption7", 2023-09-10T01:18:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "42", 2023-09-10T01:18:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "testkey": { 2023-09-10T01:18:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testkey", 2023-09-10T01:18:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "testlkey": { 2023-09-10T01:18:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testlkey", 2023-09-10T01:18:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "testnewline": { 2023-09-10T01:18:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testnewline", 2023-09-10T01:18:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2023-09-10T01:18:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_m_granularity": { 2023-09-10T01:18:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_m_granularity", 2023-09-10T01:18:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow minute scheduled snapshots", 2023-09-10T01:18:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "dump_on_update": { 2023-09-10T01:18:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dump_on_update", 2023-09-10T01:18:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "dump database to debug log on update", 2023-09-10T01:18:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stats", 2023-09-10T01:18:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.954 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.954 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.954 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.954 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.970 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "status", 2023-09-10T01:18:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telegraf", 2023-09-10T01:18:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "address": { 2023-09-10T01:18:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "address", 2023-09-10T01:18:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-09-10T01:18:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15", 2023-09-10T01:18:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telemetry", 2023-09-10T01:18:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_basic": { 2023-09-10T01:18:53.022 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_basic", 2023-09-10T01:18:53.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.022 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.023 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.023 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.023 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.023 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.024 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.024 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share basic cluster information (size, version)", 2023-09-10T01:18:53.024 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.024 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.025 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.025 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_crash": { 2023-09-10T01:18:53.025 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_crash", 2023-09-10T01:18:53.026 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.026 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.026 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.026 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.027 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.027 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.027 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.027 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-09-10T01:18:53.028 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.028 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.028 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.029 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_device": { 2023-09-10T01:18:53.029 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_device", 2023-09-10T01:18:53.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.030 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.030 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.030 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.030 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.031 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.031 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.031 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-09-10T01:18:53.031 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.032 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.032 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.032 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_ident": { 2023-09-10T01:18:53.033 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_ident", 2023-09-10T01:18:53.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.033 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.033 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.034 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.034 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.034 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.034 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.035 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-09-10T01:18:53.035 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.035 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.035 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.036 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_perf": { 2023-09-10T01:18:53.036 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_perf", 2023-09-10T01:18:53.036 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.037 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.037 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.037 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.038 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.038 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.038 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.038 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share various performance metrics of a cluster", 2023-09-10T01:18:53.039 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.039 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.039 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.040 INFO:teuthology.orchestra.run.smithi003.stdout: "contact": { 2023-09-10T01:18:53.040 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "contact", 2023-09-10T01:18:53.040 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.040 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.041 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.041 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.041 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.041 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.042 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.042 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.042 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.042 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.043 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.043 INFO:teuthology.orchestra.run.smithi003.stdout: "description": { 2023-09-10T01:18:53.043 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "description", 2023-09-10T01:18:53.044 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.044 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.044 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.045 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.045 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.045 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.045 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.046 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.046 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.046 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.046 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.047 INFO:teuthology.orchestra.run.smithi003.stdout: "device_url": { 2023-09-10T01:18:53.047 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_url", 2023-09-10T01:18:53.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.048 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.048 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.048 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-09-10T01:18:53.049 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.049 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.049 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.049 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.050 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.050 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.050 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.051 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2023-09-10T01:18:53.051 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2023-09-10T01:18:53.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.052 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.052 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.052 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.052 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.053 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.053 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.053 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.053 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.054 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.054 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.054 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:53.055 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:53.055 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.055 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.055 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.056 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "24", 2023-09-10T01:18:53.056 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "8", 2023-09-10T01:18:53.056 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.056 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.057 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.057 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.057 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.057 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.058 INFO:teuthology.orchestra.run.smithi003.stdout: "last_opt_revision": { 2023-09-10T01:18:53.058 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "last_opt_revision", 2023-09-10T01:18:53.059 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.059 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.059 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.059 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1", 2023-09-10T01:18:53.060 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.060 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.060 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.060 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.061 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.061 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.061 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.062 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard": { 2023-09-10T01:18:53.062 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard", 2023-09-10T01:18:53.062 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.063 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.063 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.063 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.063 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.064 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.064 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.064 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.064 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.065 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.065 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.066 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard_description": { 2023-09-10T01:18:53.066 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard_description", 2023-09-10T01:18:53.066 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.066 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.067 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.067 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.067 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.067 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.068 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.068 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.068 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.068 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.069 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.069 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.069 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.070 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.070 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.070 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.071 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.071 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.071 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.071 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.072 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.072 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.072 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.072 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.073 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.073 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.073 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.073 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.074 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.074 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.074 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.075 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.075 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.075 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.076 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.076 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.076 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.076 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.077 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.077 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.077 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.077 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.078 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.078 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.078 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.079 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.079 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.079 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.079 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.080 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.080 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.080 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.081 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.081 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.081 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.081 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.082 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.082 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.082 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.083 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.083 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.084 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.084 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.084 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.085 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.085 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.085 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.085 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.086 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.086 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.086 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.087 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.087 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.087 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.087 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.088 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.088 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.088 INFO:teuthology.orchestra.run.smithi003.stdout: "organization": { 2023-09-10T01:18:53.089 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "organization", 2023-09-10T01:18:53.089 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.089 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.090 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.090 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.090 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.090 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.091 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.091 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.091 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.091 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.092 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.092 INFO:teuthology.orchestra.run.smithi003.stdout: "proxy": { 2023-09-10T01:18:53.092 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "proxy", 2023-09-10T01:18:53.093 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.093 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.093 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.093 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.094 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.094 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.094 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.095 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.095 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.095 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.095 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.096 INFO:teuthology.orchestra.run.smithi003.stdout: "url": { 2023-09-10T01:18:53.096 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url", 2023-09-10T01:18:53.096 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.097 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.097 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.097 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-09-10T01:18:53.097 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.098 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.098 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.098 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.098 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.099 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.099 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.099 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.100 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.100 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "test_orchestrator", 2023-09-10T01:18:53.101 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.101 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.101 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.101 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.102 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.102 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.102 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.103 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.103 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.103 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.103 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.104 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.104 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.104 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.104 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.105 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.105 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.105 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.105 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.106 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.106 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.106 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.106 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.107 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.107 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.107 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.108 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.108 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.108 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.109 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.109 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.109 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.110 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.110 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.110 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.110 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.111 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.111 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.112 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.112 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.112 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.112 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.113 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.113 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.113 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.113 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.114 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.114 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.114 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.115 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.115 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.115 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.115 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.116 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.116 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.116 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.117 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.117 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.117 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.118 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.118 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.118 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.119 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.119 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.119 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.119 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.120 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.120 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.120 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.120 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.121 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.122 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2023-09-10T01:18:53.122 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.122 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.122 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.123 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.123 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.123 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.123 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.124 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.124 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.124 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.125 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.125 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.125 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.125 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.126 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.126 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.126 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.126 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.127 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.127 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.127 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.127 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.128 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.128 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.129 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.129 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.129 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.129 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.130 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.130 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.130 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.130 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.131 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.131 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.131 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.132 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.132 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.132 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.133 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.133 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.133 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.133 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.134 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.134 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.134 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.135 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.135 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.135 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.135 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.136 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.136 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.136 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.136 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.137 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.137 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.137 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.138 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.138 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.138 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.139 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.139 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.139 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.139 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.140 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.140 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.140 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.140 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.141 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.141 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.141 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_clones": { 2023-09-10T01:18:53.142 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_clones", 2023-09-10T01:18:53.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.142 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.143 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.143 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4", 2023-09-10T01:18:53.143 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.143 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.145 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.145 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Number of asynchronous cloner threads", 2023-09-10T01:18:53.146 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.146 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.146 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.147 INFO:teuthology.orchestra.run.smithi003.stdout: "snapshot_clone_delay": { 2023-09-10T01:18:53.147 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snapshot_clone_delay", 2023-09-10T01:18:53.147 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.147 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.148 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.148 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:53.148 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.149 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.149 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.149 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-09-10T01:18:53.149 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.150 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.150 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.150 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.150 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.151 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix", 2023-09-10T01:18:53.151 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.152 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.152 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.152 INFO:teuthology.orchestra.run.smithi003.stdout: "discovery_interval": { 2023-09-10T01:18:53.152 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "discovery_interval", 2023-09-10T01:18:53.153 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:53.153 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.153 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.153 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "100", 2023-09-10T01:18:53.154 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.154 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.154 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.155 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.155 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.155 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.155 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.156 INFO:teuthology.orchestra.run.smithi003.stdout: "identifier": { 2023-09-10T01:18:53.156 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "identifier", 2023-09-10T01:18:53.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.157 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.157 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.157 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.158 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.158 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.158 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.158 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.159 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.159 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.159 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.160 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:53.160 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:53.160 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.161 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.161 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.161 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:53.162 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.162 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.162 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.162 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.163 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.163 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.163 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.164 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.164 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.164 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.165 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.165 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.165 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.165 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.166 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.166 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.166 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.167 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.167 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.167 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.168 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.168 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.168 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.168 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.169 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.169 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.169 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.170 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.170 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.170 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.171 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.171 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.171 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.171 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.172 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.172 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.172 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.173 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.173 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.173 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.174 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.174 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.174 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.175 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.175 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.175 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.176 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.176 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.176 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.176 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.177 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.177 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.177 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.177 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.178 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.178 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.178 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.179 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.179 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.179 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.180 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.180 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.180 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.181 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.181 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.181 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.182 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.182 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.182 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.182 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.183 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_host": { 2023-09-10T01:18:53.183 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_host", 2023-09-10T01:18:53.183 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.184 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.184 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.184 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.185 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.185 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.185 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.185 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.186 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.186 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.186 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.187 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_port": { 2023-09-10T01:18:53.187 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_port", 2023-09-10T01:18:53.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.188 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.188 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.188 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10051", 2023-09-10T01:18:53.188 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.189 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.189 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.189 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.190 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.190 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.190 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.191 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_sender": { 2023-09-10T01:18:53.191 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_sender", 2023-09-10T01:18:53.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.191 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.192 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.192 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-09-10T01:18:53.192 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.193 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.193 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.193 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.193 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.194 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.194 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.195 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:53.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.196 INFO:teuthology.orchestra.run.smithi003.stdout: "gid": 25664, 2023-09-10T01:18:53.196 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "y", 2023-09-10T01:18:53.196 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_features": 4540138322906710015, 2023-09-10T01:18:53.196 INFO:teuthology.orchestra.run.smithi003.stdout: "available_modules": [ 2023-09-10T01:18:53.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.197 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "alerts", 2023-09-10T01:18:53.197 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.197 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.198 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.198 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:53.198 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:53.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.199 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.199 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.199 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:53.200 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.200 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.200 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.200 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How frequently to reexamine health status", 2023-09-10T01:18:53.201 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.201 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.201 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.202 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.202 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.202 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.203 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.203 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.203 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.203 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.204 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.204 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.204 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.204 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.205 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.205 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.205 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.205 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.206 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.206 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.206 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.206 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.207 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.207 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.207 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.208 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.208 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.208 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.209 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.209 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.209 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.210 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.210 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.210 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.210 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.211 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.211 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.211 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.212 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.212 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.213 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.213 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.213 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.213 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.214 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.214 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.214 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.215 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.215 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.215 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.215 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.216 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.216 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.216 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.216 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.217 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.217 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.217 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.218 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.218 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.218 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.219 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.219 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.219 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.219 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.220 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.220 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.220 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.221 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_destination": { 2023-09-10T01:18:53.221 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_destination", 2023-09-10T01:18:53.221 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.222 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.222 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.222 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.222 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.223 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.223 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.223 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email address to send alerts to", 2023-09-10T01:18:53.223 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.224 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.224 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.225 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_from_name": { 2023-09-10T01:18:53.225 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_from_name", 2023-09-10T01:18:53.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.225 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.226 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.226 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "Ceph", 2023-09-10T01:18:53.226 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.227 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.227 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.227 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email From: name", 2023-09-10T01:18:53.227 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.228 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.228 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.228 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_host": { 2023-09-10T01:18:53.229 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_host", 2023-09-10T01:18:53.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.229 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.229 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.230 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.230 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.230 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.231 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.231 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP server", 2023-09-10T01:18:53.231 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.231 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.232 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.232 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_password": { 2023-09-10T01:18:53.232 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_password", 2023-09-10T01:18:53.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.233 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.233 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.233 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.234 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.234 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.235 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Password to authenticate with", 2023-09-10T01:18:53.235 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.235 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.235 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.236 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_port": { 2023-09-10T01:18:53.236 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_port", 2023-09-10T01:18:53.236 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.237 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.237 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "465", 2023-09-10T01:18:53.237 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.238 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.238 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.238 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP port", 2023-09-10T01:18:53.239 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.239 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.239 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.240 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_sender": { 2023-09-10T01:18:53.240 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_sender", 2023-09-10T01:18:53.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.240 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.241 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.241 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.241 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.241 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.242 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.242 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP envelope sender", 2023-09-10T01:18:53.242 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.242 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.243 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.243 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_ssl": { 2023-09-10T01:18:53.243 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_ssl", 2023-09-10T01:18:53.244 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.244 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.244 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.244 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.245 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.245 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.245 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.246 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use SSL to connect to SMTP server", 2023-09-10T01:18:53.246 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.246 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.246 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.247 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_user": { 2023-09-10T01:18:53.247 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_user", 2023-09-10T01:18:53.247 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.248 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.248 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.248 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.248 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.249 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.249 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.249 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "User to authenticate as", 2023-09-10T01:18:53.249 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.250 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.250 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.250 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.251 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.251 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "balancer", 2023-09-10T01:18:53.252 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.252 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.252 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.252 INFO:teuthology.orchestra.run.smithi003.stdout: "active": { 2023-09-10T01:18:53.253 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "active", 2023-09-10T01:18:53.253 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.253 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.253 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.254 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.254 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.254 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.255 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.255 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically balance PGs across cluster", 2023-09-10T01:18:53.255 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.255 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.256 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.256 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_time": { 2023-09-10T01:18:53.256 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_time", 2023-09-10T01:18:53.257 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.257 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.257 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.257 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0000", 2023-09-10T01:18:53.258 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.258 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.258 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.259 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "beginning time of day to automatically balance", 2023-09-10T01:18:53.259 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-09-10T01:18:53.259 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.259 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.260 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_weekday": { 2023-09-10T01:18:53.260 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_weekday", 2023-09-10T01:18:53.260 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:53.261 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.261 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.261 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:53.262 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2023-09-10T01:18:53.262 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2023-09-10T01:18:53.262 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.262 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-09-10T01:18:53.263 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-09-10T01:18:53.263 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.263 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.264 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_max_iterations": { 2023-09-10T01:18:53.264 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_max_iterations", 2023-09-10T01:18:53.264 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:53.264 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.265 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.265 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "25", 2023-09-10T01:18:53.265 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:53.266 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "250", 2023-09-10T01:18:53.266 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.266 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-09-10T01:18:53.266 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.267 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.267 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.268 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_metrics": { 2023-09-10T01:18:53.268 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_metrics", 2023-09-10T01:18:53.268 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.268 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.269 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.269 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "pgs,objects,bytes", 2023-09-10T01:18:53.269 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.269 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.270 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.270 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-09-10T01:18:53.270 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-09-10T01:18:53.270 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.271 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.271 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_step": { 2023-09-10T01:18:53.271 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_step", 2023-09-10T01:18:53.272 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:53.272 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.272 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.272 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.5", 2023-09-10T01:18:53.273 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0.001", 2023-09-10T01:18:53.273 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "0.999", 2023-09-10T01:18:53.273 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.274 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "aggressiveness of optimization", 2023-09-10T01:18:53.274 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-09-10T01:18:53.274 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.274 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.275 INFO:teuthology.orchestra.run.smithi003.stdout: "end_time": { 2023-09-10T01:18:53.275 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_time", 2023-09-10T01:18:53.276 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.276 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.276 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.276 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2359", 2023-09-10T01:18:53.277 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.277 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.277 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.277 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ending time of day to automatically balance", 2023-09-10T01:18:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-09-10T01:18:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.279 INFO:teuthology.orchestra.run.smithi003.stdout: "end_weekday": { 2023-09-10T01:18:53.279 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_weekday", 2023-09-10T01:18:53.279 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:53.279 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.280 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.280 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:53.280 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2023-09-10T01:18:53.280 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2023-09-10T01:18:53.281 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.281 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-09-10T01:18:53.281 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-09-10T01:18:53.281 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.282 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.282 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.285 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.285 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.285 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.285 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.286 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.286 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.286 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.287 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.287 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.287 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.287 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.288 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.288 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.288 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.289 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.289 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.289 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.289 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.290 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.290 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.290 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.290 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.291 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.291 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.291 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.292 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.292 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.292 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.293 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.293 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.293 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.293 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.294 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.294 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.294 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.294 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.295 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.295 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.295 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.295 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.296 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.296 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.296 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.296 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.297 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.297 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.297 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.298 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.298 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.298 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.298 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.299 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.299 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.299 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.299 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.300 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.300 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.300 INFO:teuthology.orchestra.run.smithi003.stdout: "min_score": { 2023-09-10T01:18:53.301 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_score", 2023-09-10T01:18:53.301 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:53.301 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.301 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.302 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:53.302 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.302 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.303 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.303 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-09-10T01:18:53.303 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.303 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.304 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.304 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2023-09-10T01:18:53.304 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2023-09-10T01:18:53.305 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.305 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.305 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.305 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "upmap", 2023-09-10T01:18:53.306 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.306 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.306 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.307 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-compat", 2023-09-10T01:18:53.307 INFO:teuthology.orchestra.run.smithi003.stdout: "none", 2023-09-10T01:18:53.307 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap" 2023-09-10T01:18:53.307 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.308 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Balancer mode", 2023-09-10T01:18:53.308 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.308 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.308 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.309 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_ids": { 2023-09-10T01:18:53.309 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_ids", 2023-09-10T01:18:53.310 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.310 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.310 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.310 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.311 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-09-10T01:18:53.312 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.312 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.312 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.313 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:53.313 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:53.313 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.314 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.314 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.314 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:53.315 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.315 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.315 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.315 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-09-10T01:18:53.316 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.316 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.316 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.317 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_deviation": { 2023-09-10T01:18:53.317 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_deviation", 2023-09-10T01:18:53.317 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.317 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.318 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.318 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:53.318 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:53.319 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.319 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.319 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "deviation below which no optimization is attempted", 2023-09-10T01:18:53.319 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-09-10T01:18:53.320 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.320 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.321 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_optimizations": { 2023-09-10T01:18:53.321 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_optimizations", 2023-09-10T01:18:53.321 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:53.321 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.322 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.322 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:53.322 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.322 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.323 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.323 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-09-10T01:18:53.323 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.323 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.324 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.324 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.324 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.325 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm", 2023-09-10T01:18:53.325 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.326 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.326 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.326 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_down_multiplier": { 2023-09-10T01:18:53.326 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_down_multiplier", 2023-09-10T01:18:53.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:53.327 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.327 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.327 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2023-09-10T01:18:53.328 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.328 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.328 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.328 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-09-10T01:18:53.329 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.329 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.329 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.330 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_refresh_rate": { 2023-09-10T01:18:53.330 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_refresh_rate", 2023-09-10T01:18:53.330 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.331 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.331 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.331 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "20", 2023-09-10T01:18:53.331 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.332 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.332 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.332 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-09-10T01:18:53.332 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.333 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.333 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.333 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_starting_port": { 2023-09-10T01:18:53.334 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_starting_port", 2023-09-10T01:18:53.334 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.334 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.334 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.335 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4721", 2023-09-10T01:18:53.335 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.335 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.335 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.336 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-09-10T01:18:53.336 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.336 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.336 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.337 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_ptrace": { 2023-09-10T01:18:53.337 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_ptrace", 2023-09-10T01:18:53.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.338 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.338 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.338 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.339 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.339 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.339 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.339 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-09-10T01:18:53.340 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-09-10T01:18:53.340 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.340 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.341 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_interval": { 2023-09-10T01:18:53.341 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_interval", 2023-09-10T01:18:53.341 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.342 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.342 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.342 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:53.342 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.343 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.343 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.343 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to autotune daemon memory", 2023-09-10T01:18:53.343 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.344 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.344 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.344 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_memory_target_ratio": { 2023-09-10T01:18:53.345 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_memory_target_ratio", 2023-09-10T01:18:53.345 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:53.345 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.345 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.346 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.7", 2023-09-10T01:18:53.346 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.346 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.346 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.347 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-09-10T01:18:53.347 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.347 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.348 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.348 INFO:teuthology.orchestra.run.smithi003.stdout: "cgroups_split": { 2023-09-10T01:18:53.348 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cgroups_split", 2023-09-10T01:18:53.349 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.349 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.349 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.350 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.350 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.350 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.350 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.351 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-09-10T01:18:53.351 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.351 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.351 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.352 INFO:teuthology.orchestra.run.smithi003.stdout: "config_checks_enabled": { 2023-09-10T01:18:53.352 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_checks_enabled", 2023-09-10T01:18:53.352 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.353 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.353 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.353 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.354 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.354 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.354 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.354 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-09-10T01:18:53.355 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.355 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.355 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.356 INFO:teuthology.orchestra.run.smithi003.stdout: "config_dashboard": { 2023-09-10T01:18:53.356 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_dashboard", 2023-09-10T01:18:53.356 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.357 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.357 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.357 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.357 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.358 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.358 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.358 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-09-10T01:18:53.358 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.359 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.359 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.360 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_alertmanager": { 2023-09-10T01:18:53.360 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_alertmanager", 2023-09-10T01:18:53.360 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.360 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.361 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.361 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-09-10T01:18:53.361 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.361 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.362 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.362 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:53.362 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.363 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.363 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.363 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_base": { 2023-09-10T01:18:53.364 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_base", 2023-09-10T01:18:53.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.364 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.364 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.365 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph", 2023-09-10T01:18:53.365 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.365 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.365 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.366 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Container image name, without the tag", 2023-09-10T01:18:53.366 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.366 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.367 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.367 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_elasticsearch": { 2023-09-10T01:18:53.368 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_elasticsearch", 2023-09-10T01:18:53.368 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.368 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.368 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.369 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-09-10T01:18:53.369 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.369 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.369 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.370 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "elasticsearch container image", 2023-09-10T01:18:53.370 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.370 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.370 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.371 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_grafana": { 2023-09-10T01:18:53.371 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_grafana", 2023-09-10T01:18:53.372 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.372 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.372 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-09-10T01:18:53.373 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.373 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.373 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.373 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:53.374 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.374 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.374 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.375 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_haproxy": { 2023-09-10T01:18:53.375 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_haproxy", 2023-09-10T01:18:53.375 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.376 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.376 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.376 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-09-10T01:18:53.376 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.377 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.377 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.377 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "HAproxy container image", 2023-09-10T01:18:53.378 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.378 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.378 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.379 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_agent": { 2023-09-10T01:18:53.379 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_agent", 2023-09-10T01:18:53.379 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.379 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.380 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.380 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-09-10T01:18:53.380 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.380 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.381 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.381 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger agent container image", 2023-09-10T01:18:53.381 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.382 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.382 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_collector": { 2023-09-10T01:18:53.383 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_collector", 2023-09-10T01:18:53.383 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.383 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.383 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.384 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-09-10T01:18:53.384 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.384 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.385 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.385 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger collector container image", 2023-09-10T01:18:53.385 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.385 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.386 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.386 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_query": { 2023-09-10T01:18:53.386 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_query", 2023-09-10T01:18:53.387 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.387 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.387 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.387 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-09-10T01:18:53.388 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.388 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.388 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.388 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger query container image", 2023-09-10T01:18:53.389 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.389 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.389 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.390 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_keepalived": { 2023-09-10T01:18:53.390 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_keepalived", 2023-09-10T01:18:53.390 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.391 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.391 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-09-10T01:18:53.392 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.392 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.392 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.392 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Keepalived container image", 2023-09-10T01:18:53.393 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.393 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.393 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.394 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_loki": { 2023-09-10T01:18:53.394 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_loki", 2023-09-10T01:18:53.394 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.394 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.395 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.395 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-09-10T01:18:53.395 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.395 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.396 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.396 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Loki container image", 2023-09-10T01:18:53.396 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.397 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.397 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.397 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_node_exporter": { 2023-09-10T01:18:53.398 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_node_exporter", 2023-09-10T01:18:53.398 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.398 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.399 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-09-10T01:18:53.399 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.399 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.399 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.400 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:53.400 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.400 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.401 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.401 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_nvmeof": { 2023-09-10T01:18:53.401 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_nvmeof", 2023-09-10T01:18:53.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.402 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.402 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.403 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.3", 2023-09-10T01:18:53.403 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.403 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.403 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.404 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Nvme-of container image", 2023-09-10T01:18:53.404 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.404 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.405 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.405 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_prometheus": { 2023-09-10T01:18:53.405 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_prometheus", 2023-09-10T01:18:53.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.406 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.406 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.406 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-09-10T01:18:53.407 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.407 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.407 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.407 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:53.408 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.408 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.408 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.409 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_promtail": { 2023-09-10T01:18:53.409 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_promtail", 2023-09-10T01:18:53.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.410 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.410 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.410 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-09-10T01:18:53.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.411 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.411 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.412 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Promtail container image", 2023-09-10T01:18:53.412 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.412 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.412 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.413 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_snmp_gateway": { 2023-09-10T01:18:53.413 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_snmp_gateway", 2023-09-10T01:18:53.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.414 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.414 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.414 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-09-10T01:18:53.415 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.415 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.415 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.415 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SNMP Gateway container image", 2023-09-10T01:18:53.416 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.416 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.416 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.417 INFO:teuthology.orchestra.run.smithi003.stdout: "container_init": { 2023-09-10T01:18:53.417 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_init", 2023-09-10T01:18:53.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.418 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.418 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.418 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.418 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.419 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.419 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.419 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Run podman/docker with `--init`", 2023-09-10T01:18:53.419 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.420 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.420 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.421 INFO:teuthology.orchestra.run.smithi003.stdout: "daemon_cache_timeout": { 2023-09-10T01:18:53.421 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "daemon_cache_timeout", 2023-09-10T01:18:53.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.421 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.422 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.422 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:53.422 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.422 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.423 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.423 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-09-10T01:18:53.423 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.424 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.424 INFO:teuthology.orchestra.run.smithi003.stdout: "default_cephadm_command_timeout": { 2023-09-10T01:18:53.424 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_cephadm_command_timeout", 2023-09-10T01:18:53.425 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.425 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.425 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.426 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "900", 2023-09-10T01:18:53.426 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.426 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.426 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.427 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-09-10T01:18:53.427 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.427 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.427 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.428 INFO:teuthology.orchestra.run.smithi003.stdout: "default_registry": { 2023-09-10T01:18:53.428 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_registry", 2023-09-10T01:18:53.428 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.429 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.429 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io", 2023-09-10T01:18:53.430 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.430 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.430 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.430 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-09-10T01:18:53.431 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.431 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.431 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.432 INFO:teuthology.orchestra.run.smithi003.stdout: "device_cache_timeout": { 2023-09-10T01:18:53.432 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_cache_timeout", 2023-09-10T01:18:53.432 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.433 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.433 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.433 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1800", 2023-09-10T01:18:53.433 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.434 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.434 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.434 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache device inventory", 2023-09-10T01:18:53.435 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.435 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.435 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.436 INFO:teuthology.orchestra.run.smithi003.stdout: "device_enhanced_scan": { 2023-09-10T01:18:53.436 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_enhanced_scan", 2023-09-10T01:18:53.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.437 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.437 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.437 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.437 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.438 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.438 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.438 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use libstoragemgmt during device scans", 2023-09-10T01:18:53.439 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.439 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.439 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.440 INFO:teuthology.orchestra.run.smithi003.stdout: "facts_cache_timeout": { 2023-09-10T01:18:53.440 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "facts_cache_timeout", 2023-09-10T01:18:53.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.440 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.441 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.441 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:53.441 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.441 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.442 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.442 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache host facts data", 2023-09-10T01:18:53.442 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.443 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.443 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.443 INFO:teuthology.orchestra.run.smithi003.stdout: "host_check_interval": { 2023-09-10T01:18:53.444 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "host_check_interval", 2023-09-10T01:18:53.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.444 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.444 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.445 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:53.445 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.445 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.446 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.446 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to perform a host check", 2023-09-10T01:18:53.446 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.446 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.447 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.447 INFO:teuthology.orchestra.run.smithi003.stdout: "inventory_list_all": { 2023-09-10T01:18:53.447 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "inventory_list_all", 2023-09-10T01:18:53.448 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.448 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.448 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.448 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.449 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.449 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.449 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.450 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2023-09-10T01:18:53.450 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.450 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.450 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.451 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.451 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.451 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.452 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.452 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.452 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.453 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.453 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.453 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.453 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.454 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.454 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.454 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.454 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.455 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.455 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.455 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.456 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.456 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.456 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.457 INFO:teuthology.orchestra.run.smithi003.stdout: "log_refresh_metadata": { 2023-09-10T01:18:53.457 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_refresh_metadata", 2023-09-10T01:18:53.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.457 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.458 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.458 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.458 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.459 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.459 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.459 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-09-10T01:18:53.459 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.460 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.460 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.460 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.461 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.461 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.462 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.462 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.462 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.462 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.463 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.463 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-09-10T01:18:53.463 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.464 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.464 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.465 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.465 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.465 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.466 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.466 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.466 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.466 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.467 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.467 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.467 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.468 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.468 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.468 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.468 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.469 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.469 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.469 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.469 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.470 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.470 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.471 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.471 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.471 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.471 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.472 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.472 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.472 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.473 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.473 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.473 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.474 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf": { 2023-09-10T01:18:53.474 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-09-10T01:18:53.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.474 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.475 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.475 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.475 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.475 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.476 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.476 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-09-10T01:18:53.476 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.477 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.477 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.477 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-09-10T01:18:53.478 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-09-10T01:18:53.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.478 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.478 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.479 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "*", 2023-09-10T01:18:53.479 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.479 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.479 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.480 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-09-10T01:18:53.480 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.480 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.481 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.481 INFO:teuthology.orchestra.run.smithi003.stdout: "max_count_per_host": { 2023-09-10T01:18:53.481 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_count_per_host", 2023-09-10T01:18:53.482 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.482 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.482 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.482 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:53.483 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.483 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.483 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.483 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of daemons per service per host", 2023-09-10T01:18:53.484 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.484 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.484 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.485 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd_draining_count": { 2023-09-10T01:18:53.485 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_osd_draining_count", 2023-09-10T01:18:53.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.486 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.486 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.486 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:53.486 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.487 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.487 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.487 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-09-10T01:18:53.487 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.488 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.488 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.489 INFO:teuthology.orchestra.run.smithi003.stdout: "migration_current": { 2023-09-10T01:18:53.489 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "migration_current", 2023-09-10T01:18:53.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.489 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.490 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.490 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.490 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.490 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.491 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.491 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "internal - do not modify", 2023-09-10T01:18:53.491 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.491 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.492 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.492 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2023-09-10T01:18:53.493 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2023-09-10T01:18:53.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.493 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.493 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.494 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "root", 2023-09-10T01:18:53.494 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.494 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.494 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.495 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm-package", 2023-09-10T01:18:53.495 INFO:teuthology.orchestra.run.smithi003.stdout: "root" 2023-09-10T01:18:53.495 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.496 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "mode for remote execution of cephadm", 2023-09-10T01:18:53.496 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.496 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.496 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.497 INFO:teuthology.orchestra.run.smithi003.stdout: "prometheus_alerts_path": { 2023-09-10T01:18:53.497 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus_alerts_path", 2023-09-10T01:18:53.497 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-09-10T01:18:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.499 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.499 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.499 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-09-10T01:18:53.500 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.500 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.500 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.501 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_insecure": { 2023-09-10T01:18:53.501 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_insecure", 2023-09-10T01:18:53.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.501 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.502 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.502 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.502 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.503 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.503 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.503 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-09-10T01:18:53.503 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.504 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.504 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.504 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_password": { 2023-09-10T01:18:53.505 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_password", 2023-09-10T01:18:53.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.505 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.505 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.506 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.506 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.506 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.507 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.507 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-09-10T01:18:53.507 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.507 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.508 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.508 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_url": { 2023-09-10T01:18:53.508 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_url", 2023-09-10T01:18:53.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.509 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.509 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.510 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.510 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.510 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.511 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.511 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-09-10T01:18:53.511 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.511 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.512 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.512 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_username": { 2023-09-10T01:18:53.512 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_username", 2023-09-10T01:18:53.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.513 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.513 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.513 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.514 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.514 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.514 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.515 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-09-10T01:18:53.515 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.515 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.515 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.516 INFO:teuthology.orchestra.run.smithi003.stdout: "secure_monitoring_stack": { 2023-09-10T01:18:53.516 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "secure_monitoring_stack", 2023-09-10T01:18:53.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.517 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.517 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.517 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.518 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.518 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.518 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.518 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-09-10T01:18:53.519 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.519 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.519 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.520 INFO:teuthology.orchestra.run.smithi003.stdout: "service_discovery_port": { 2023-09-10T01:18:53.520 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "service_discovery_port", 2023-09-10T01:18:53.520 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.520 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.521 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.521 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8765", 2023-09-10T01:18:53.521 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.522 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.522 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.522 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "cephadm service discovery port", 2023-09-10T01:18:53.522 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.523 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.523 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.523 INFO:teuthology.orchestra.run.smithi003.stdout: "ssh_config_file": { 2023-09-10T01:18:53.524 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssh_config_file", 2023-09-10T01:18:53.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.524 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.524 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.525 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.525 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.525 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.526 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.526 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-09-10T01:18:53.526 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.526 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.527 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.527 INFO:teuthology.orchestra.run.smithi003.stdout: "use_agent": { 2023-09-10T01:18:53.527 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_agent", 2023-09-10T01:18:53.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.528 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.528 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.529 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.529 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.529 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.529 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.530 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-09-10T01:18:53.530 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.530 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.530 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.531 INFO:teuthology.orchestra.run.smithi003.stdout: "use_repo_digest": { 2023-09-10T01:18:53.531 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_repo_digest", 2023-09-10T01:18:53.531 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.532 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.532 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.532 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.533 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.533 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.533 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.533 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-09-10T01:18:53.534 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.534 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.534 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.535 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_failed_host_check": { 2023-09-10T01:18:53.535 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_failed_host_check", 2023-09-10T01:18:53.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.536 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.536 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.536 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.536 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.537 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.537 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.537 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if the host check fails", 2023-09-10T01:18:53.537 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.538 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.538 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.539 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_daemons": { 2023-09-10T01:18:53.539 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_daemons", 2023-09-10T01:18:53.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.539 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.540 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.540 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.540 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.540 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.541 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.541 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-09-10T01:18:53.541 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.542 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.542 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.542 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_hosts": { 2023-09-10T01:18:53.543 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_hosts", 2023-09-10T01:18:53.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.543 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.543 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.544 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.544 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.544 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.545 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.545 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-09-10T01:18:53.545 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.545 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.546 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.546 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.546 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.547 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crash", 2023-09-10T01:18:53.547 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.548 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.548 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.548 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.548 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.549 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.549 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.549 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.550 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.550 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.550 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.550 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.551 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.551 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.551 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.551 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.552 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.552 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.552 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.553 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.553 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.553 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.553 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.554 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.554 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.554 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.555 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.555 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.555 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.555 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.556 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.556 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.556 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.557 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.557 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.557 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.558 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.558 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.558 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.559 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.559 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.559 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.560 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.560 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.560 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.560 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.561 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.561 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.561 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.561 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.562 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.562 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.562 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.562 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.563 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.563 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.564 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.564 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.564 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.564 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.565 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.565 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.565 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.565 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.566 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.566 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.566 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.567 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.567 INFO:teuthology.orchestra.run.smithi003.stdout: "retain_interval": { 2023-09-10T01:18:53.568 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retain_interval", 2023-09-10T01:18:53.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.568 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.568 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.569 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "31536000", 2023-09-10T01:18:53.569 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.569 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.569 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.570 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain crashes before pruning them", 2023-09-10T01:18:53.570 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.570 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.571 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.571 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_recent_interval": { 2023-09-10T01:18:53.571 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_recent_interval", 2023-09-10T01:18:53.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.572 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.572 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1209600", 2023-09-10T01:18:53.573 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.573 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.573 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.574 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "time interval in which to warn about recent crashes", 2023-09-10T01:18:53.574 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.574 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.574 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.576 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dashboard", 2023-09-10T01:18:53.576 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.576 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.577 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.577 INFO:teuthology.orchestra.run.smithi003.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-09-10T01:18:53.577 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-09-10T01:18:53.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.578 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.578 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.578 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:53.579 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.579 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.579 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.579 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.580 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.580 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.580 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.581 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_HOST": { 2023-09-10T01:18:53.581 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_HOST", 2023-09-10T01:18:53.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.582 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.582 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.582 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.582 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.583 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.583 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.583 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.583 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.584 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.584 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.584 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-09-10T01:18:53.585 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-09-10T01:18:53.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.585 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.586 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.586 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.586 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.586 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.587 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.587 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.587 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.587 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.588 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.588 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_ENABLED": { 2023-09-10T01:18:53.588 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_ENABLED", 2023-09-10T01:18:53.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.589 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.589 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.590 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.590 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.590 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.590 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.591 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.591 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.591 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.591 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.592 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-09-10T01:18:53.592 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-09-10T01:18:53.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.593 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.593 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.593 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.594 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.594 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.594 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.594 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.595 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.595 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.595 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.596 INFO:teuthology.orchestra.run.smithi003.stdout: "ENABLE_BROWSABLE_API": { 2023-09-10T01:18:53.596 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ENABLE_BROWSABLE_API", 2023-09-10T01:18:53.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.597 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.597 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.597 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.597 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.598 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.598 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.598 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.598 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.599 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.599 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.599 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-09-10T01:18:53.600 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-09-10T01:18:53.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.600 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.600 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.601 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.601 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.601 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.602 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.602 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.602 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.603 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.603 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-09-10T01:18:53.603 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-09-10T01:18:53.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.604 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.604 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.605 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.605 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.605 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.605 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.606 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.606 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.606 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.606 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.607 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-09-10T01:18:53.607 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-09-10T01:18:53.607 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.608 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.608 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.608 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.610 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.611 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-09-10T01:18:53.611 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-09-10T01:18:53.611 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.612 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.612 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.612 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.612 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.613 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.613 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.613 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.614 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.614 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.614 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.615 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_NFS": { 2023-09-10T01:18:53.615 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-09-10T01:18:53.615 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.616 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.616 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.616 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.616 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.617 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.617 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.617 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.617 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.618 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.618 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RBD": { 2023-09-10T01:18:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-09-10T01:18:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RGW": { 2023-09-10T01:18:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-09-10T01:18:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-09-10T01:18:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-09-10T01:18:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_PASSWORD": { 2023-09-10T01:18:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_PASSWORD", 2023-09-10T01:18:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-09-10T01:18:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-09-10T01:18:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_URL": { 2023-09-10T01:18:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_URL", 2023-09-10T01:18:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_USERNAME": { 2023-09-10T01:18:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_USERNAME", 2023-09-10T01:18:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-09-10T01:18:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-09-10T01:18:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-09-10T01:18:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-09-10T01:18:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-09-10T01:18:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-09-10T01:18:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "ISSUE_TRACKER_API_KEY": { 2023-09-10T01:18:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-09-10T01:18:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_HOST": { 2023-09-10T01:18:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_HOST", 2023-09-10T01:18:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.664 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-09-10T01:18:53.664 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-09-10T01:18:53.664 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-09-10T01:18:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-09-10T01:18:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-09-10T01:18:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-09-10T01:18:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-09-10T01:18:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-09-10T01:18:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-09-10T01:18:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-09-10T01:18:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-09-10T01:18:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-09-10T01:18:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-09-10T01:18:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-09-10T01:18:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-09-10T01:18:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-09-10T01:18:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_ENABLED": { 2023-09-10T01:18:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_ENABLED", 2023-09-10T01:18:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-09-10T01:18:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-09-10T01:18:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-09-10T01:18:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-09-10T01:18:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-09-10T01:18:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-09-10T01:18:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-09-10T01:18:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8", 2023-09-10T01:18:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "REST_REQUESTS_TIMEOUT": { 2023-09-10T01:18:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-09-10T01:18:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "45", 2023-09-10T01:18:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ACCESS_KEY": { 2023-09-10T01:18:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ACCESS_KEY", 2023-09-10T01:18:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-09-10T01:18:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-09-10T01:18:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SECRET_KEY": { 2023-09-10T01:18:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SECRET_KEY", 2023-09-10T01:18:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SSL_VERIFY": { 2023-09-10T01:18:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SSL_VERIFY", 2023-09-10T01:18:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-09-10T01:18:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-09-10T01:18:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-09-10T01:18:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-09-10T01:18:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-09-10T01:18:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-09-10T01:18:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.737 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:53.737 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.737 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.737 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.738 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.738 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.738 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.739 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.739 INFO:teuthology.orchestra.run.smithi003.stdout: "cross_origin_url": { 2023-09-10T01:18:53.739 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cross_origin_url", 2023-09-10T01:18:53.740 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.740 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.740 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.740 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.741 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.741 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.741 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.741 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.742 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.742 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.743 INFO:teuthology.orchestra.run.smithi003.stdout: "crt_file": { 2023-09-10T01:18:53.743 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crt_file", 2023-09-10T01:18:53.743 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.744 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.744 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.744 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.745 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.745 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.745 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.745 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.746 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.746 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.747 INFO:teuthology.orchestra.run.smithi003.stdout: "debug": { 2023-09-10T01:18:53.747 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "debug", 2023-09-10T01:18:53.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.747 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.748 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.748 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.748 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.748 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.749 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.749 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable/disable debug options", 2023-09-10T01:18:53.749 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.750 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.750 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.750 INFO:teuthology.orchestra.run.smithi003.stdout: "jwt_token_ttl": { 2023-09-10T01:18:53.751 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "jwt_token_ttl", 2023-09-10T01:18:53.751 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.751 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.752 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.752 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "28800", 2023-09-10T01:18:53.752 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.752 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.753 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.753 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.753 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.753 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.754 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.754 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2023-09-10T01:18:53.755 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2023-09-10T01:18:53.755 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.755 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.755 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.756 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.756 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.756 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.756 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.757 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.757 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.757 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.757 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.758 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.758 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.759 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.759 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.759 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.760 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.760 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.760 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.760 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.761 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.761 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.761 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.762 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.762 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.762 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.762 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.763 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.763 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.763 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.764 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.764 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.765 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.765 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.765 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.765 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.766 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.766 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.766 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.767 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.767 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.767 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.768 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.769 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.769 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.769 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.769 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.770 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.770 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.770 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.771 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.771 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.771 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.771 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.772 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.772 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.772 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.773 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.773 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.773 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.774 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.774 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.774 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.775 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.775 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.775 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.775 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.776 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.776 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.776 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.777 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.777 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.777 INFO:teuthology.orchestra.run.smithi003.stdout: "motd": { 2023-09-10T01:18:53.778 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "motd", 2023-09-10T01:18:53.778 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.778 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.778 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.779 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.779 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.779 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.779 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.780 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "The message of the day", 2023-09-10T01:18:53.780 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.780 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.781 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.781 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect_resolve_ip_addr": { 2023-09-10T01:18:53.781 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "redirect_resolve_ip_addr", 2023-09-10T01:18:53.782 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.782 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.782 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.782 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.783 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.783 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.783 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.783 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.784 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.784 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.784 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.785 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:53.785 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:53.786 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.786 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.786 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.787 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2023-09-10T01:18:53.787 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.787 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.787 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.788 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.788 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.788 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.788 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.789 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:53.789 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:53.789 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.790 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.790 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.790 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8080", 2023-09-10T01:18:53.791 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.791 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.791 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.791 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.792 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.792 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.792 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.793 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2023-09-10T01:18:53.793 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2023-09-10T01:18:53.793 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.793 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.794 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.794 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.794 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.795 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.795 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.795 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.795 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.796 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.796 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.796 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl_server_port": { 2023-09-10T01:18:53.797 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl_server_port", 2023-09-10T01:18:53.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.797 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.798 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.798 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8443", 2023-09-10T01:18:53.798 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.798 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.799 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.799 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.799 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.799 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.800 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.800 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2023-09-10T01:18:53.800 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2023-09-10T01:18:53.801 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.801 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.801 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.802 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "redirect", 2023-09-10T01:18:53.802 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.802 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.802 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.803 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.803 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect" 2023-09-10T01:18:53.803 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.803 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.804 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.804 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.804 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.805 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2023-09-10T01:18:53.805 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2023-09-10T01:18:53.805 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.806 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.806 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.806 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2023-09-10T01:18:53.806 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2023-09-10T01:18:53.807 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2023-09-10T01:18:53.807 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.807 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.808 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.808 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.808 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.809 INFO:teuthology.orchestra.run.smithi003.stdout: "url_prefix": { 2023-09-10T01:18:53.809 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url_prefix", 2023-09-10T01:18:53.809 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.809 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.810 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.810 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.810 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.811 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.811 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.811 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.812 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.812 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.812 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.812 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.813 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2023-09-10T01:18:53.814 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.814 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.814 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.815 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_monitoring": { 2023-09-10T01:18:53.815 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_monitoring", 2023-09-10T01:18:53.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.815 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.816 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.816 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.816 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.816 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.817 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.817 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "monitor device health metrics", 2023-09-10T01:18:53.817 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.818 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.818 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.818 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.819 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.819 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.820 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.820 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.820 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.820 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.821 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.821 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.821 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.821 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.822 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.822 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.822 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.823 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.823 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.823 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.823 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.824 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.824 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.824 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.825 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.825 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.826 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.826 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.826 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.826 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.827 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.827 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.827 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.827 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.828 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.828 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.829 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.829 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.829 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.829 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.830 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.830 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.830 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.830 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.831 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.832 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.832 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.832 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.834 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.834 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.834 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.835 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.835 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.835 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.835 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.836 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.836 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.836 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.837 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.837 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.837 INFO:teuthology.orchestra.run.smithi003.stdout: "mark_out_threshold": { 2023-09-10T01:18:53.838 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mark_out_threshold", 2023-09-10T01:18:53.838 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.838 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.839 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.839 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2419200", 2023-09-10T01:18:53.839 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.839 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.840 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.840 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-09-10T01:18:53.840 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.840 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.841 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.841 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": { 2023-09-10T01:18:53.841 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_name", 2023-09-10T01:18:53.842 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.842 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.842 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.843 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "device_health_metrics", 2023-09-10T01:18:53.843 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.843 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.843 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.844 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name of pool in which to store device health metrics", 2023-09-10T01:18:53.844 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.844 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.844 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.845 INFO:teuthology.orchestra.run.smithi003.stdout: "retention_period": { 2023-09-10T01:18:53.845 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retention_period", 2023-09-10T01:18:53.846 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.846 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.846 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.846 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15552000", 2023-09-10T01:18:53.847 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.847 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.847 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.847 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain device health metrics", 2023-09-10T01:18:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.849 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_frequency": { 2023-09-10T01:18:53.849 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_frequency", 2023-09-10T01:18:53.849 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.850 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.850 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.850 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2023-09-10T01:18:53.850 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to scrape device health metrics", 2023-09-10T01:18:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.853 INFO:teuthology.orchestra.run.smithi003.stdout: "self_heal": { 2023-09-10T01:18:53.853 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "self_heal", 2023-09-10T01:18:53.853 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.853 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.854 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.854 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:53.854 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.855 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.855 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.855 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-09-10T01:18:53.855 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.856 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.856 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.856 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:53.857 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:53.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.857 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.857 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.858 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:53.858 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.858 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.859 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.859 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and check device health", 2023-09-10T01:18:53.859 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.859 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.860 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.860 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_threshold": { 2023-09-10T01:18:53.860 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_threshold", 2023-09-10T01:18:53.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.861 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.861 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.862 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7257600", 2023-09-10T01:18:53.862 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.862 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.862 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.863 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-09-10T01:18:53.863 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.863 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.863 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.865 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "diskprediction_local", 2023-09-10T01:18:53.865 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.865 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.866 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.866 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.866 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.867 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.867 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.869 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.869 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.869 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.869 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.870 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.870 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.870 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.870 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.871 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.871 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.871 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.872 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.872 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.872 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.873 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.873 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.873 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.873 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.874 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.874 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.874 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.875 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.875 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.875 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.876 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.876 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.876 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.876 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.877 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.877 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.877 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.877 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.878 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.878 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.878 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.879 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.879 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.879 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.879 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.880 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.880 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.880 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.880 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.881 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.881 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.882 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.882 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.882 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.882 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.883 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.883 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.883 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.884 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.884 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.884 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.884 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.885 INFO:teuthology.orchestra.run.smithi003.stdout: "predict_interval": { 2023-09-10T01:18:53.885 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predict_interval", 2023-09-10T01:18:53.885 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.886 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.886 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.886 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2023-09-10T01:18:53.887 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.887 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.887 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.887 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.888 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.888 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.888 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.889 INFO:teuthology.orchestra.run.smithi003.stdout: "predictor_model": { 2023-09-10T01:18:53.889 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predictor_model", 2023-09-10T01:18:53.889 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.890 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.890 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.890 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "prophetstor", 2023-09-10T01:18:53.890 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.891 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.891 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.891 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.892 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.892 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.892 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.893 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:53.893 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:53.893 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.893 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.894 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.894 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:53.894 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.895 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.895 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.895 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.895 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.896 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.896 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.896 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.896 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.897 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "influx", 2023-09-10T01:18:53.897 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": false, 2023-09-10T01:18:53.898 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "influxdb python module not found", 2023-09-10T01:18:53.898 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.898 INFO:teuthology.orchestra.run.smithi003.stdout: "batch_size": { 2023-09-10T01:18:53.899 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "batch_size", 2023-09-10T01:18:53.899 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.899 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.899 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.900 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5000", 2023-09-10T01:18:53.900 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.900 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.901 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.901 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-09-10T01:18:53.901 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.901 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.902 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.902 INFO:teuthology.orchestra.run.smithi003.stdout: "database": { 2023-09-10T01:18:53.903 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "database", 2023-09-10T01:18:53.903 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.903 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.903 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.904 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "ceph", 2023-09-10T01:18:53.904 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.904 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.904 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.905 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-09-10T01:18:53.905 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.905 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.905 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.906 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": { 2023-09-10T01:18:53.906 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hostname", 2023-09-10T01:18:53.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.907 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.907 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.907 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.908 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.908 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.908 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.908 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server hostname", 2023-09-10T01:18:53.909 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.909 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.909 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.910 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:53.910 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:53.910 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:53.911 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.911 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.911 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "30", 2023-09-10T01:18:53.912 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "5", 2023-09-10T01:18:53.912 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.912 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.912 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-09-10T01:18:53.913 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.913 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.913 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.914 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.914 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.915 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.915 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.915 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.915 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.916 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.916 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.916 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.916 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.917 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.917 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.917 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.918 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.918 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.918 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.918 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.919 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.919 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.919 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.920 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.920 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.920 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.920 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.921 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.921 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.921 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.922 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.922 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.922 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.922 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.923 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.923 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.924 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.924 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.924 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.924 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.925 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.925 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.925 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.925 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.926 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.926 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.926 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.927 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.927 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.927 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.927 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.928 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.928 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.928 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.928 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.929 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.929 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.930 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.930 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.930 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.930 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.931 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.931 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.931 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.931 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.932 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.932 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.932 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.933 INFO:teuthology.orchestra.run.smithi003.stdout: "password": { 2023-09-10T01:18:53.933 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "password", 2023-09-10T01:18:53.933 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.934 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.934 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.934 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.934 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.935 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.935 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.935 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "password of InfluxDB server user", 2023-09-10T01:18:53.935 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.936 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.936 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.936 INFO:teuthology.orchestra.run.smithi003.stdout: "port": { 2023-09-10T01:18:53.937 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "port", 2023-09-10T01:18:53.937 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.937 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.937 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.938 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8086", 2023-09-10T01:18:53.938 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.938 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.939 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.939 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server port", 2023-09-10T01:18:53.939 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.939 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.940 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.940 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2023-09-10T01:18:53.940 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2023-09-10T01:18:53.941 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.941 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.941 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.941 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "false", 2023-09-10T01:18:53.942 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.942 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.942 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.943 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-09-10T01:18:53.943 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.943 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.943 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.944 INFO:teuthology.orchestra.run.smithi003.stdout: "threads": { 2023-09-10T01:18:53.944 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threads", 2023-09-10T01:18:53.944 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:53.945 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.945 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:53.946 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:53.946 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "32", 2023-09-10T01:18:53.946 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.946 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-09-10T01:18:53.947 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.947 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.947 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.948 INFO:teuthology.orchestra.run.smithi003.stdout: "username": { 2023-09-10T01:18:53.948 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "username", 2023-09-10T01:18:53.948 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.948 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.949 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.949 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.949 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.950 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.950 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.950 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "username of InfluxDB server user", 2023-09-10T01:18:53.950 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.951 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.951 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.952 INFO:teuthology.orchestra.run.smithi003.stdout: "verify_ssl": { 2023-09-10T01:18:53.952 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "verify_ssl", 2023-09-10T01:18:53.952 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.952 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.953 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:53.953 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "true", 2023-09-10T01:18:53.953 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.954 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.954 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.954 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-09-10T01:18:53.954 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.955 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.955 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.955 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.956 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.956 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "insights", 2023-09-10T01:18:53.957 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.957 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.957 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.957 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.958 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.958 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.958 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.959 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.959 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.959 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.959 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.960 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.960 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.960 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.961 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.961 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.961 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.961 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.962 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.962 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.962 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.962 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.963 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.963 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.964 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.964 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.964 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.964 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.965 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.965 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.965 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.965 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.966 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.966 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.966 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.967 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.967 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.968 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.969 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.970 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.970 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.970 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.971 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.972 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.973 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.973 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.974 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.975 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.975 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.975 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.976 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.977 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.977 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.978 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "iostat", 2023-09-10T01:18:53.978 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.978 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.978 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.979 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:53.979 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:53.979 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:53.980 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.981 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.982 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.983 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.984 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.985 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.987 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:53.988 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.989 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:53.990 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:53.991 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:53.991 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:53.991 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:53.992 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.994 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:53.994 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:53.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:53.995 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:53.996 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.997 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:53.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:53.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:53.998 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "k8sevents", 2023-09-10T01:18:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:53.999 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_event_retention_days": { 2023-09-10T01:18:54.000 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ceph_event_retention_days", 2023-09-10T01:18:54.000 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.000 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7", 2023-09-10T01:18:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.001 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Days to hold ceph event information within local cache", 2023-09-10T01:18:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.002 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.003 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.003 INFO:teuthology.orchestra.run.smithi003.stdout: "config_check_secs": { 2023-09-10T01:18:54.004 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_check_secs", 2023-09-10T01:18:54.004 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.004 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "10", 2023-09-10T01:18:54.005 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.006 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.006 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-09-10T01:18:54.007 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.007 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.007 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.008 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.008 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.008 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.008 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.009 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.009 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.009 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.010 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.010 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.010 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.010 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.011 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.011 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.011 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.012 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.012 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.012 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.013 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.013 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.013 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.014 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.014 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.014 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.014 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.015 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.015 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.015 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.015 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.016 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.016 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.016 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.017 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.017 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.017 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.018 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.018 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.018 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.019 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.019 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.019 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.019 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.020 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.020 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.020 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.020 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.021 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.021 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.021 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.022 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.022 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.022 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.022 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.023 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.023 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.023 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "localpool", 2023-09-10T01:18:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.028 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "failure_domain": { 2023-09-10T01:18:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "failure_domain", 2023-09-10T01:18:54.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.030 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.030 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.030 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "host", 2023-09-10T01:18:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.031 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "failure domain for any created local pool", 2023-09-10T01:18:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-09-10T01:18:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.033 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.033 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.034 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.034 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.034 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.034 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.035 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.035 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.035 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.035 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.036 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.036 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.036 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.037 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.037 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.037 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.037 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.038 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.038 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.038 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.039 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.039 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.039 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.040 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.040 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.040 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.040 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.041 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.041 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.041 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.041 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.042 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.042 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.042 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.043 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.043 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.043 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.044 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.044 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.044 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.044 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.045 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.045 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.045 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.045 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.046 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.046 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.046 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.046 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.047 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.047 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.047 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.048 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.048 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.048 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.049 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.050 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.050 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.050 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.050 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.051 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.051 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": { 2023-09-10T01:18:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_size", 2023-09-10T01:18:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.052 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.053 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default min_size for any created local pool", 2023-09-10T01:18:54.054 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-09-10T01:18:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.055 INFO:teuthology.orchestra.run.smithi003.stdout: "num_rep": { 2023-09-10T01:18:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "num_rep", 2023-09-10T01:18:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.056 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3", 2023-09-10T01:18:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.057 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default replica count for any created local pool", 2023-09-10T01:18:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": { 2023-09-10T01:18:54.059 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_num", 2023-09-10T01:18:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.060 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "128", 2023-09-10T01:18:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.061 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default pg_num for any created local pool", 2023-09-10T01:18:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.062 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: "prefix": { 2023-09-10T01:18:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prefix", 2023-09-10T01:18:54.063 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.064 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.064 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.064 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.065 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name prefix for any created local pool", 2023-09-10T01:18:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.067 INFO:teuthology.orchestra.run.smithi003.stdout: "subtree": { 2023-09-10T01:18:54.067 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "subtree", 2023-09-10T01:18:54.067 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "rack", 2023-09-10T01:18:54.068 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "CRUSH level for which to create a local pool", 2023-09-10T01:18:54.069 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-09-10T01:18:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mds_autoscaler", 2023-09-10T01:18:54.071 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.072 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.073 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.073 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.073 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.073 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.074 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.074 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.074 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.077 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.078 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.078 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.078 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.079 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.080 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.081 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.082 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.082 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.082 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.083 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.084 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.085 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.086 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.087 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.087 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.087 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.088 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.090 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.090 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.090 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.091 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.092 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirroring", 2023-09-10T01:18:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.095 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.096 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.097 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.098 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.098 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.098 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.099 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.100 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.101 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.102 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.103 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.104 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.104 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.104 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.105 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.106 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.107 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.108 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.108 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.108 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.109 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.109 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.109 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.110 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.110 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.110 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.111 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.111 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.111 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.112 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.112 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.112 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.113 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.114 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.114 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.114 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.116 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "nfs", 2023-09-10T01:18:54.116 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.116 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.116 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.117 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.117 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.117 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.117 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.118 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.118 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.118 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.119 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.121 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.121 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.121 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.122 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.122 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.122 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.123 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.124 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.124 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.124 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.125 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.126 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.127 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.127 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.127 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.128 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.128 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.128 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.128 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.129 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.129 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.129 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.130 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.131 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.131 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.131 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.132 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.132 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.132 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.133 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.133 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.133 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.133 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.134 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.134 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.134 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.135 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.135 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.135 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.136 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.136 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2023-09-10T01:18:54.137 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.137 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.137 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.138 INFO:teuthology.orchestra.run.smithi003.stdout: "fail_fs": { 2023-09-10T01:18:54.138 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "fail_fs", 2023-09-10T01:18:54.138 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.139 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.139 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.139 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.140 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.140 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.140 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.141 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-09-10T01:18:54.141 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.141 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.141 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.142 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.142 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.143 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.144 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.144 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.144 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.145 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.145 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.145 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.145 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.146 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.146 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.146 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.146 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.147 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.147 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.148 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.148 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.148 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.149 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.149 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.149 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.149 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.150 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.150 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.150 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.151 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.151 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.151 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.152 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.152 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.152 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.153 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.153 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.153 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.154 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.154 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.154 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.154 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.155 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.155 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.155 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.155 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.156 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.156 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.156 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.157 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.157 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.157 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.158 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.158 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.158 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.158 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.159 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.159 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.159 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.159 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.160 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.160 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.160 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator": { 2023-09-10T01:18:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2023-09-10T01:18:54.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.162 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.162 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.162 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.163 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2023-09-10T01:18:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: "rook", 2023-09-10T01:18:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: "test_orchestrator" 2023-09-10T01:18:54.164 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Orchestrator backend", 2023-09-10T01:18:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.165 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.167 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_perf_query", 2023-09-10T01:18:54.167 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.168 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.169 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.170 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.171 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.172 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.174 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.174 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.175 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.176 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.177 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.178 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.179 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.179 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.179 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.179 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.180 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.181 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.182 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.183 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.184 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.185 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_support", 2023-09-10T01:18:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.188 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.189 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.190 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.191 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.191 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.191 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.192 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.194 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.195 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.195 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.196 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.197 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.198 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.198 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.199 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.199 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.199 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.200 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.201 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.201 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.201 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.202 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.204 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.204 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.205 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.206 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.207 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.208 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_autoscaler", 2023-09-10T01:18:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.209 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.211 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.211 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.211 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.212 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.212 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.212 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.213 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.214 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.214 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.214 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.215 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.216 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.217 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.217 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.217 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.217 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.218 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.218 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.218 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.219 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.221 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.222 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.223 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.224 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.226 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.227 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: "noautoscale": { 2023-09-10T01:18:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "noautoscale", 2023-09-10T01:18:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.229 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.230 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "global autoscale flag", 2023-09-10T01:18:54.231 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-09-10T01:18:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.232 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.233 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.234 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:54.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.234 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.235 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: "threshold": { 2023-09-10T01:18:54.236 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threshold", 2023-09-10T01:18:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.237 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2023-09-10T01:18:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1.0", 2023-09-10T01:18:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.238 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "scaling threshold", 2023-09-10T01:18:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-09-10T01:18:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.239 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "progress", 2023-09-10T01:18:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_pg_recovery_event": { 2023-09-10T01:18:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_pg_recovery_event", 2023-09-10T01:18:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.243 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.244 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.244 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.244 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow the module to show pg recovery progress", 2023-09-10T01:18:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2023-09-10T01:18:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2023-09-10T01:18:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.246 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.247 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.248 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.248 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.248 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.249 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.249 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.249 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.250 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.251 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.251 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.251 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.252 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.252 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.252 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.252 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.254 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.255 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.255 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.256 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.256 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.256 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.256 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.257 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.257 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.257 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.257 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.258 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.258 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.258 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.259 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.259 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.259 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.260 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.260 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.260 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.260 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.261 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.261 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.261 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.262 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.262 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.262 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.262 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.263 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.263 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.263 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.263 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.264 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.264 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.264 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.265 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.265 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.265 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.266 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.266 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.266 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.266 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.267 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.267 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.267 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.267 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.268 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.268 INFO:teuthology.orchestra.run.smithi003.stdout: "max_completed_events": { 2023-09-10T01:18:54.269 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_completed_events", 2023-09-10T01:18:54.269 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.269 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.269 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.270 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "50", 2023-09-10T01:18:54.270 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.270 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.270 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.271 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "number of past completed events to remember", 2023-09-10T01:18:54.271 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.271 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.272 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.272 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:54.272 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:54.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.273 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.273 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.273 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:54.274 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.274 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.274 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.275 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long the module is going to sleep", 2023-09-10T01:18:54.275 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.275 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.275 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.276 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.276 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.277 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus", 2023-09-10T01:18:54.277 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.277 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.277 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.278 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2023-09-10T01:18:54.278 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cache", 2023-09-10T01:18:54.278 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.278 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.279 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.279 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.279 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.280 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.280 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.280 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.280 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.281 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.281 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.281 INFO:teuthology.orchestra.run.smithi003.stdout: "exclude_perf_counters": { 2023-09-10T01:18:54.282 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "exclude_perf_counters", 2023-09-10T01:18:54.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.282 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.282 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.283 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.283 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.283 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.283 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.284 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-09-10T01:18:54.284 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-09-10T01:18:54.284 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.285 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.285 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.285 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.286 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.286 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.286 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.286 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.287 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.287 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.287 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.287 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.288 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.288 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.288 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.289 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.289 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.289 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.289 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.290 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.290 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.290 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.291 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.291 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.291 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.291 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.292 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.292 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.292 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.293 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.293 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.293 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.293 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.294 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.294 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.294 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.295 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.295 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.295 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.295 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.296 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.296 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.296 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.297 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.297 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.297 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.297 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.298 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.298 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.298 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.298 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.299 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.299 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.299 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.299 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.300 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.300 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.301 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.301 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.301 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.301 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.302 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.302 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.302 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.303 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.303 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.303 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.303 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.304 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools": { 2023-09-10T01:18:54.304 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools", 2023-09-10T01:18:54.304 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.305 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.305 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.305 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.305 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.306 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.306 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.306 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.306 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.307 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.307 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.308 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools_refresh_interval": { 2023-09-10T01:18:54.308 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-09-10T01:18:54.308 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.308 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.309 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.309 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300", 2023-09-10T01:18:54.309 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.309 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.310 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.310 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.310 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.310 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.311 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.311 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_interval": { 2023-09-10T01:18:54.312 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_interval", 2023-09-10T01:18:54.312 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.312 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.312 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.313 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15.0", 2023-09-10T01:18:54.313 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.313 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.314 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.314 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.314 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.314 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.315 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.315 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:54.315 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:54.316 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.316 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.316 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.317 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2023-09-10T01:18:54.317 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.317 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.317 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.318 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-09-10T01:18:54.318 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.318 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.319 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.319 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:54.319 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:54.320 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.320 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.320 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.320 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "9283", 2023-09-10T01:18:54.321 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.321 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.321 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.322 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-09-10T01:18:54.322 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.322 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.322 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.323 INFO:teuthology.orchestra.run.smithi003.stdout: "stale_cache_strategy": { 2023-09-10T01:18:54.323 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stale_cache_strategy", 2023-09-10T01:18:54.323 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.324 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.324 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.324 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "log", 2023-09-10T01:18:54.324 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.325 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.325 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.325 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.326 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.326 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.326 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.327 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2023-09-10T01:18:54.327 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2023-09-10T01:18:54.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.327 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.328 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.328 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "default", 2023-09-10T01:18:54.328 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.328 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.329 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.329 INFO:teuthology.orchestra.run.smithi003.stdout: "default", 2023-09-10T01:18:54.329 INFO:teuthology.orchestra.run.smithi003.stdout: "error" 2023-09-10T01:18:54.329 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.330 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.330 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.330 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.331 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.331 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2023-09-10T01:18:54.331 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2023-09-10T01:18:54.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.332 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.332 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.332 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2023-09-10T01:18:54.333 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2023-09-10T01:18:54.333 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2023-09-10T01:18:54.333 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.334 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.334 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.334 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.334 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.335 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.335 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.336 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2023-09-10T01:18:54.336 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.336 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.336 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.337 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.337 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.337 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.338 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.338 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.338 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.339 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.339 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.339 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.339 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.340 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.340 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.340 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.340 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.341 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.341 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.341 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.341 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.342 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.342 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.343 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.343 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.343 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.343 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.344 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.344 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.344 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.344 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.345 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.345 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.345 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.345 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.346 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.346 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.347 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.347 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.347 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.347 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.348 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.348 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.348 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.348 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.349 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.349 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.349 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.349 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.350 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.350 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.350 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.351 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.351 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.351 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.352 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.352 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.352 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.352 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.353 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.353 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.353 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.353 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.354 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.354 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.354 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.354 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.355 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.355 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_snap_create": { 2023-09-10T01:18:54.356 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_snap_create", 2023-09-10T01:18:54.356 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.356 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.356 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.357 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:54.357 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.357 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.357 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.358 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.358 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.358 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.358 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.359 INFO:teuthology.orchestra.run.smithi003.stdout: "mirror_snapshot_schedule": { 2023-09-10T01:18:54.359 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirror_snapshot_schedule", 2023-09-10T01:18:54.360 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.360 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.360 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.360 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.361 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.361 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.361 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.362 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.362 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.362 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.363 INFO:teuthology.orchestra.run.smithi003.stdout: "trash_purge_schedule": { 2023-09-10T01:18:54.363 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "trash_purge_schedule", 2023-09-10T01:18:54.363 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.364 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.364 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.364 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.364 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.365 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.365 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.365 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.366 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.366 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.366 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.369 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "restful", 2023-09-10T01:18:54.369 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.369 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.370 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.370 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_auth": { 2023-09-10T01:18:54.370 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_auth", 2023-09-10T01:18:54.371 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.371 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.371 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.371 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.372 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.372 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.372 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.372 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.373 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.373 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.373 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.374 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2023-09-10T01:18:54.374 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2023-09-10T01:18:54.374 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.376 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.376 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.376 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.376 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.377 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.377 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.377 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.378 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.378 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.378 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.379 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.379 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.379 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.379 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.380 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.380 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.380 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.380 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.381 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.381 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.381 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.381 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.382 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.382 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.382 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.383 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.383 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.383 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.384 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.384 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.384 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.384 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.385 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.385 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.385 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.385 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.386 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.386 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.386 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.387 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.387 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.387 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.387 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.388 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.388 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.388 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.388 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.389 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.389 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.389 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.389 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.390 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.390 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.390 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.391 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.391 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.391 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.391 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.392 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.392 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.393 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.393 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.393 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.393 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.393 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.394 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.394 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.394 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.395 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.395 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.395 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:54.396 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:54.396 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.396 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.396 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.397 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.397 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.397 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.397 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.398 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.398 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.398 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.399 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.399 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:54.399 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:54.400 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.400 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.400 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.400 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.401 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.401 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.401 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.401 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.402 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.402 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.402 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.403 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rgw", 2023-09-10T01:18:54.404 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.404 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.404 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.404 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.405 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.405 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.405 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.406 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.406 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.406 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.407 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.407 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.407 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.407 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.408 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.408 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.408 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.408 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.409 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.409 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.409 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.409 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.411 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.411 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.412 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.412 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.412 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.412 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.413 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.413 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.413 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.414 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.414 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.415 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.415 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.415 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.416 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.416 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.416 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.416 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.417 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.417 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.417 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.417 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.418 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.418 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.418 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.418 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.419 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.419 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.420 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.420 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.420 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.421 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.421 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.421 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.422 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.422 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.422 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.423 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.424 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rook", 2023-09-10T01:18:54.424 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.425 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.425 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.425 INFO:teuthology.orchestra.run.smithi003.stdout: "drive_group_interval": { 2023-09-10T01:18:54.425 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "drive_group_interval", 2023-09-10T01:18:54.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.426 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.426 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300.0", 2023-09-10T01:18:54.427 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.427 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.427 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.427 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-09-10T01:18:54.428 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.428 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.428 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.429 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.429 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.429 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.430 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.430 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.430 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.430 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.431 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.431 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.431 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.432 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.432 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.432 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.432 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.433 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.433 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.433 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.433 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.434 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.434 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.434 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.435 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.435 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.436 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.436 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.436 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.436 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.437 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.437 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.438 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.438 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.438 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.439 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.439 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.439 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.439 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.440 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.440 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.440 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.440 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.441 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.441 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.441 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.442 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.442 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.442 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.442 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.443 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.443 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.443 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.444 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.444 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.444 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.445 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.445 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.445 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.445 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.446 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.446 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.446 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.447 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_class": { 2023-09-10T01:18:54.447 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "storage_class", 2023-09-10T01:18:54.447 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.448 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.448 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.448 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "local", 2023-09-10T01:18:54.448 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.449 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.449 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.449 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-09-10T01:18:54.449 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.450 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.450 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.451 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "selftest", 2023-09-10T01:18:54.452 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.452 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.452 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.452 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.453 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.453 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.453 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.454 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.454 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.454 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.455 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.455 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.455 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.455 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.456 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.456 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.456 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.456 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.457 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.457 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.457 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.458 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.458 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.458 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.459 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.459 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.459 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.460 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.460 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.460 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.460 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.461 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.461 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.461 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.462 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.462 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.463 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.463 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.463 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.463 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.464 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.464 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.464 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.465 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.465 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.465 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.465 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.466 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.466 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.466 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.466 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.467 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.467 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.468 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.468 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.468 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.469 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.469 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.469 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.469 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.470 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.470 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.470 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.471 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.471 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.471 INFO:teuthology.orchestra.run.smithi003.stdout: "roption1": { 2023-09-10T01:18:54.472 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption1", 2023-09-10T01:18:54.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.472 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.473 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.473 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.473 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.473 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.474 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.474 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.474 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.475 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.475 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.475 INFO:teuthology.orchestra.run.smithi003.stdout: "roption2": { 2023-09-10T01:18:54.476 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption2", 2023-09-10T01:18:54.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.476 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.477 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "xyz", 2023-09-10T01:18:54.477 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.477 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.478 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.478 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.478 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.478 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.479 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.479 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption1": { 2023-09-10T01:18:54.479 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption1", 2023-09-10T01:18:54.480 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.480 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.480 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.480 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.481 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.481 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.481 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.482 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.482 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.482 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.482 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption2": { 2023-09-10T01:18:54.483 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption2", 2023-09-10T01:18:54.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.484 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.484 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.484 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.485 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.485 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.485 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.486 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.486 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.486 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.487 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption3": { 2023-09-10T01:18:54.487 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption3", 2023-09-10T01:18:54.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.488 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.488 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.488 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.489 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.489 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.489 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.489 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.490 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.490 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.490 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption4": { 2023-09-10T01:18:54.491 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption4", 2023-09-10T01:18:54.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.492 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.492 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.492 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.492 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.493 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.493 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.493 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.494 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.494 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.494 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption5": { 2023-09-10T01:18:54.495 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption5", 2023-09-10T01:18:54.495 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.495 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.496 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.496 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.496 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.496 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.497 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.497 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.497 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.498 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.498 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption6": { 2023-09-10T01:18:54.499 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption6", 2023-09-10T01:18:54.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.499 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.500 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.500 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.500 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.501 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.501 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.501 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.501 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.502 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.502 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption7": { 2023-09-10T01:18:54.503 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption7", 2023-09-10T01:18:54.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.503 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.504 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.504 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.504 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:54.504 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "42", 2023-09-10T01:18:54.505 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.505 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.505 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.506 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.506 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.506 INFO:teuthology.orchestra.run.smithi003.stdout: "testkey": { 2023-09-10T01:18:54.507 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testkey", 2023-09-10T01:18:54.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.507 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.508 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.508 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.508 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.508 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.509 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.509 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.509 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.509 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.510 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.510 INFO:teuthology.orchestra.run.smithi003.stdout: "testlkey": { 2023-09-10T01:18:54.511 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testlkey", 2023-09-10T01:18:54.511 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.511 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.511 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.512 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.512 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.512 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.512 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.513 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.513 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.513 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.513 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.514 INFO:teuthology.orchestra.run.smithi003.stdout: "testnewline": { 2023-09-10T01:18:54.514 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testnewline", 2023-09-10T01:18:54.515 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.515 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.515 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.516 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.516 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.516 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.516 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.517 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.517 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.517 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.518 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.518 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.519 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2023-09-10T01:18:54.519 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.519 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.520 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.520 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_m_granularity": { 2023-09-10T01:18:54.520 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_m_granularity", 2023-09-10T01:18:54.520 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.521 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.521 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.521 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.521 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.522 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.522 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.522 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow minute scheduled snapshots", 2023-09-10T01:18:54.523 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.523 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.523 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.524 INFO:teuthology.orchestra.run.smithi003.stdout: "dump_on_update": { 2023-09-10T01:18:54.524 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dump_on_update", 2023-09-10T01:18:54.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.525 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.525 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.525 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.525 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.526 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.526 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.526 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "dump database to debug log on update", 2023-09-10T01:18:54.527 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.527 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.528 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.528 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.528 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.529 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.529 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.529 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.530 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.530 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.530 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.530 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.531 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.531 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.531 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.531 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.532 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.532 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.532 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.533 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.533 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.534 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.534 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.534 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.535 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.535 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.535 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.535 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.536 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.536 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.536 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.536 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.537 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.537 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.537 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.538 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.538 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.539 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.539 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.539 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.539 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.540 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.540 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.540 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.540 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.541 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.541 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.541 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.541 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.542 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.542 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.542 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.543 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.543 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.544 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.544 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.544 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.544 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.545 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.545 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.545 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.545 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.546 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.546 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.546 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.546 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.547 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stats", 2023-09-10T01:18:54.547 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.548 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.548 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.548 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.549 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.549 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.549 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.549 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.550 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.550 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.550 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.550 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.551 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.551 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.551 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.552 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.552 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.552 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.552 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.553 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.553 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.553 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.553 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.554 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.554 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.555 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.555 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.555 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.555 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.556 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.556 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.556 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.556 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.557 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.557 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.557 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.558 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.558 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.559 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.559 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.559 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.559 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.560 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.560 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.560 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.560 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.561 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.561 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.561 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.561 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.562 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.562 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.562 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.563 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.563 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.563 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.564 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.564 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.564 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.564 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.565 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.565 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.565 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.565 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.566 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.566 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.566 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.567 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.568 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "status", 2023-09-10T01:18:54.568 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.569 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.569 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.569 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.569 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.570 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.570 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.572 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.573 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.573 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.573 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.574 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.574 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.574 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.574 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.575 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.575 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.575 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.576 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.576 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.576 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.576 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.577 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.577 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.578 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.578 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.578 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.579 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.579 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.579 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.580 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.580 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.580 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.580 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.581 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.581 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.582 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.582 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.582 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.583 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.583 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.583 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.583 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.584 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.584 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.584 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.585 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.585 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.585 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.585 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.586 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.586 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.586 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.587 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.587 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.587 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.588 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.588 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.588 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.588 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.589 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.589 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.589 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.589 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.590 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.590 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.590 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.590 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.591 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telegraf", 2023-09-10T01:18:54.592 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.592 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.592 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.592 INFO:teuthology.orchestra.run.smithi003.stdout: "address": { 2023-09-10T01:18:54.593 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "address", 2023-09-10T01:18:54.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.593 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.593 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.594 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-09-10T01:18:54.594 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.594 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.595 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.595 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.595 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.595 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.596 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.596 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:54.596 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:54.597 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.597 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.597 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.597 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15", 2023-09-10T01:18:54.598 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.598 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.598 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.599 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.599 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.599 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.599 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.600 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.600 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.600 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.601 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.601 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.602 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.602 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.602 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.602 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.603 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.603 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.603 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.603 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.604 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.604 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.604 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.604 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.605 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.605 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.606 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.606 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.606 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.606 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.607 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.607 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.607 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.607 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.608 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.608 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.608 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.608 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.609 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.609 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.610 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.610 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.610 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.611 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.611 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.611 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.611 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.612 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.612 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.612 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.613 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.613 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.613 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.613 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.614 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.614 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.614 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.615 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.615 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.615 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.616 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.616 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.616 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.616 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.617 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.617 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.617 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.617 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.618 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.618 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.619 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telemetry", 2023-09-10T01:18:54.620 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.620 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.620 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.620 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_basic": { 2023-09-10T01:18:54.621 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_basic", 2023-09-10T01:18:54.621 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.621 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.622 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.622 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.622 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.622 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.623 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.623 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share basic cluster information (size, version)", 2023-09-10T01:18:54.623 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.624 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.624 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.625 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_crash": { 2023-09-10T01:18:54.625 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_crash", 2023-09-10T01:18:54.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.625 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.626 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.626 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.626 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.626 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.627 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.627 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-09-10T01:18:54.627 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.627 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.628 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.628 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_device": { 2023-09-10T01:18:54.629 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_device", 2023-09-10T01:18:54.629 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.629 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.629 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.630 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.630 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.630 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.630 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.631 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-09-10T01:18:54.631 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.631 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.631 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.632 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_ident": { 2023-09-10T01:18:54.632 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_ident", 2023-09-10T01:18:54.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.633 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.633 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.633 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.634 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.634 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.634 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.635 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-09-10T01:18:54.635 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.635 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.635 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.636 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_perf": { 2023-09-10T01:18:54.636 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_perf", 2023-09-10T01:18:54.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.637 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.637 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.637 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.637 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.638 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.638 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.638 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share various performance metrics of a cluster", 2023-09-10T01:18:54.639 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.639 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.639 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.640 INFO:teuthology.orchestra.run.smithi003.stdout: "contact": { 2023-09-10T01:18:54.640 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "contact", 2023-09-10T01:18:54.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.640 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.641 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.641 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.641 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.642 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.642 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.642 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.642 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.643 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.643 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.644 INFO:teuthology.orchestra.run.smithi003.stdout: "description": { 2023-09-10T01:18:54.644 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "description", 2023-09-10T01:18:54.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.644 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.645 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.645 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.645 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.645 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.646 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.646 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.646 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.647 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.647 INFO:teuthology.orchestra.run.smithi003.stdout: "device_url": { 2023-09-10T01:18:54.647 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_url", 2023-09-10T01:18:54.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.648 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.648 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-09-10T01:18:54.649 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.649 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.649 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.649 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.650 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.650 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.650 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.651 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2023-09-10T01:18:54.651 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2023-09-10T01:18:54.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.652 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.652 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.652 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.652 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.653 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.653 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.653 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.653 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.654 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.654 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.654 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:54.655 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:54.655 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.655 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.656 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "24", 2023-09-10T01:18:54.656 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "8", 2023-09-10T01:18:54.656 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.657 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.657 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.657 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.657 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.658 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.658 INFO:teuthology.orchestra.run.smithi003.stdout: "last_opt_revision": { 2023-09-10T01:18:54.658 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "last_opt_revision", 2023-09-10T01:18:54.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.659 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.659 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.659 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1", 2023-09-10T01:18:54.660 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.660 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.660 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.661 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.661 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.661 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.662 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard": { 2023-09-10T01:18:54.662 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard", 2023-09-10T01:18:54.662 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.663 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.663 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.663 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.664 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.664 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.664 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.664 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.665 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.666 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard_description": { 2023-09-10T01:18:54.666 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard_description", 2023-09-10T01:18:54.666 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.666 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.667 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.667 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.667 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.668 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.668 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.668 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.668 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.669 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.669 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.669 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.670 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.670 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.671 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.671 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.671 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.671 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.672 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.672 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.672 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.672 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.673 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.673 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.673 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.673 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.674 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.674 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.675 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.675 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.675 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.676 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.676 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.676 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.676 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.677 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.677 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.677 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.677 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.678 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.678 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.678 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.679 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.679 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.679 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.679 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.680 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.680 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.680 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.681 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.681 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.681 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.681 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.682 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.682 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.682 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.682 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.683 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.683 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.683 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.683 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.684 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.684 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.684 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.685 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.685 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.685 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.685 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.686 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.686 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.686 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.687 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.687 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.687 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.688 INFO:teuthology.orchestra.run.smithi003.stdout: "organization": { 2023-09-10T01:18:54.688 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "organization", 2023-09-10T01:18:54.688 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.688 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.689 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.689 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.689 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.689 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.690 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.690 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.690 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.690 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.691 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.691 INFO:teuthology.orchestra.run.smithi003.stdout: "proxy": { 2023-09-10T01:18:54.692 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "proxy", 2023-09-10T01:18:54.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.692 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.693 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.693 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.693 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.693 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.694 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.694 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.694 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.694 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.695 INFO:teuthology.orchestra.run.smithi003.stdout: "url": { 2023-09-10T01:18:54.695 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url", 2023-09-10T01:18:54.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.696 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.696 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.696 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-09-10T01:18:54.696 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.697 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.697 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.697 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.697 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.698 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.698 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.699 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "test_orchestrator", 2023-09-10T01:18:54.700 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.700 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.700 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.700 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.701 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.701 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.702 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.702 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.702 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.703 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.703 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.703 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.704 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.704 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.704 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.704 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.705 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.705 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.705 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.705 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.706 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.706 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.707 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.707 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.707 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.708 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.708 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.708 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.709 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.709 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.709 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.709 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.710 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.710 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.711 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.711 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.711 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.712 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.712 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.712 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.712 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.713 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.713 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.713 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.714 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.714 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.714 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.715 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.715 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.715 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.716 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.716 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.716 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.717 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.717 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.718 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.718 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.718 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.719 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.719 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.719 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.719 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.720 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.720 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.720 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.721 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.721 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.722 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2023-09-10T01:18:54.722 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.722 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.723 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.723 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.723 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.724 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.724 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.724 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.724 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.725 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.725 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.725 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.725 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.726 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.726 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.726 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.727 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.727 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.727 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.727 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.728 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.728 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.728 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.729 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.729 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.729 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.729 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.730 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.730 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.730 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.731 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.731 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.731 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.731 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.732 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.732 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.732 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.733 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.733 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.733 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.733 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.734 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.734 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.734 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.734 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.735 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.735 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.735 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.735 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.736 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.736 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.736 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.736 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.737 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.737 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.738 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.738 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.738 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.738 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.739 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.739 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.739 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.739 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.740 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.740 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.740 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.741 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.741 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.741 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_clones": { 2023-09-10T01:18:54.742 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_clones", 2023-09-10T01:18:54.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.742 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.742 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.743 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4", 2023-09-10T01:18:54.743 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.743 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.744 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.744 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Number of asynchronous cloner threads", 2023-09-10T01:18:54.744 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.744 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.745 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.745 INFO:teuthology.orchestra.run.smithi003.stdout: "snapshot_clone_delay": { 2023-09-10T01:18:54.745 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snapshot_clone_delay", 2023-09-10T01:18:54.745 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.746 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.746 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.746 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:54.747 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.747 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.747 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.747 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-09-10T01:18:54.748 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.748 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.748 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.748 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.749 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix", 2023-09-10T01:18:54.750 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.750 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.750 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.751 INFO:teuthology.orchestra.run.smithi003.stdout: "discovery_interval": { 2023-09-10T01:18:54.751 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "discovery_interval", 2023-09-10T01:18:54.751 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:54.751 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.752 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.752 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "100", 2023-09-10T01:18:54.752 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.752 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.753 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.753 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.753 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.753 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.754 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.754 INFO:teuthology.orchestra.run.smithi003.stdout: "identifier": { 2023-09-10T01:18:54.754 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "identifier", 2023-09-10T01:18:54.755 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.755 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.755 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.755 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.756 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.756 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.756 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.756 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.757 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.757 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.757 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.759 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.759 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.759 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:54.759 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.760 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.760 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.760 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.760 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.761 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.762 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.762 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.762 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.762 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.763 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.763 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.763 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.763 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.764 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.764 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.764 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.765 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.765 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.765 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.765 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.766 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.766 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.766 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.767 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.767 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.768 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.769 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.769 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.769 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.769 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.770 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.770 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.770 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.770 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.771 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.771 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.771 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.772 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.772 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.772 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.773 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.773 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.773 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.774 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.774 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.774 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.774 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.777 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.777 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.778 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.778 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.778 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.778 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.779 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.779 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.779 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.780 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.780 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.781 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.781 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.781 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.781 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.782 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.782 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.782 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.782 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.783 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.783 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_host": { 2023-09-10T01:18:54.783 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_host", 2023-09-10T01:18:54.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.784 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.784 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.784 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.785 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.785 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.785 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.786 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.786 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.786 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.786 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.787 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_port": { 2023-09-10T01:18:54.787 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_port", 2023-09-10T01:18:54.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.788 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.788 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.788 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10051", 2023-09-10T01:18:54.788 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.789 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.789 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.789 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.790 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.790 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.790 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.791 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_sender": { 2023-09-10T01:18:54.791 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_sender", 2023-09-10T01:18:54.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.792 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.792 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.792 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-09-10T01:18:54.792 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.793 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.793 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.793 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.793 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.794 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.794 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.794 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.795 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:54.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.795 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.796 INFO:teuthology.orchestra.run.smithi003.stdout: "modules": [ 2023-09-10T01:18:54.796 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2023-09-10T01:18:54.796 INFO:teuthology.orchestra.run.smithi003.stdout: "dashboard", 2023-09-10T01:18:54.796 INFO:teuthology.orchestra.run.smithi003.stdout: "iostat", 2023-09-10T01:18:54.797 INFO:teuthology.orchestra.run.smithi003.stdout: "nfs", 2023-09-10T01:18:54.797 INFO:teuthology.orchestra.run.smithi003.stdout: "restful", 2023-09-10T01:18:54.797 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_schedule" 2023-09-10T01:18:54.798 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.798 INFO:teuthology.orchestra.run.smithi003.stdout: "available_modules": [ 2023-09-10T01:18:54.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.798 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "alerts", 2023-09-10T01:18:54.799 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.799 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.799 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.799 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:54.800 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:54.800 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.800 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.801 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.801 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:54.801 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.801 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.802 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.802 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How frequently to reexamine health status", 2023-09-10T01:18:54.802 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.802 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.803 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.803 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.804 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.804 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.804 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.804 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.805 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.805 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.805 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.805 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.806 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.806 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.806 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.806 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.807 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.807 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.807 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.808 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.808 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.808 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.808 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.809 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.809 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.810 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.810 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.810 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.811 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.811 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.811 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.811 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.812 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.812 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.812 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.813 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.813 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.813 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.814 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.814 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.814 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.814 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.815 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.815 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.815 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.816 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.816 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.816 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.816 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.817 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.817 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.817 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.817 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.818 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.818 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.818 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.819 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.819 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.819 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.820 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.820 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.820 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.821 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.821 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.821 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.821 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.822 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.822 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_destination": { 2023-09-10T01:18:54.822 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_destination", 2023-09-10T01:18:54.823 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.823 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.823 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.823 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.824 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.824 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.824 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.824 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email address to send alerts to", 2023-09-10T01:18:54.825 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.825 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.825 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.826 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_from_name": { 2023-09-10T01:18:54.826 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_from_name", 2023-09-10T01:18:54.826 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.827 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.827 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.827 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "Ceph", 2023-09-10T01:18:54.827 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.828 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.828 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.828 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email From: name", 2023-09-10T01:18:54.828 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.829 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.829 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.829 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_host": { 2023-09-10T01:18:54.830 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_host", 2023-09-10T01:18:54.830 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.830 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.831 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.831 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.832 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.832 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.832 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.832 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP server", 2023-09-10T01:18:54.833 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.833 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.833 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.834 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_password": { 2023-09-10T01:18:54.834 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_password", 2023-09-10T01:18:54.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.835 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.835 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.835 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.835 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.836 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.836 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.836 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Password to authenticate with", 2023-09-10T01:18:54.836 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.837 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.837 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.837 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_port": { 2023-09-10T01:18:54.838 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_port", 2023-09-10T01:18:54.838 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.838 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.839 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.839 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "465", 2023-09-10T01:18:54.839 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.839 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.840 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.840 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP port", 2023-09-10T01:18:54.840 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.840 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.841 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.841 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_sender": { 2023-09-10T01:18:54.841 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_sender", 2023-09-10T01:18:54.842 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.842 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.842 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.843 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.843 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.843 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.843 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.844 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP envelope sender", 2023-09-10T01:18:54.844 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.844 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.844 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.845 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_ssl": { 2023-09-10T01:18:54.845 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_ssl", 2023-09-10T01:18:54.846 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.846 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.846 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.846 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.847 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.847 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.847 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.847 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use SSL to connect to SMTP server", 2023-09-10T01:18:54.848 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.848 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.848 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.849 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_user": { 2023-09-10T01:18:54.849 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_user", 2023-09-10T01:18:54.849 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.850 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.850 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.850 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.850 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.851 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.851 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.851 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "User to authenticate as", 2023-09-10T01:18:54.852 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.852 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.852 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.852 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.853 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "balancer", 2023-09-10T01:18:54.854 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.854 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.854 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.855 INFO:teuthology.orchestra.run.smithi003.stdout: "active": { 2023-09-10T01:18:54.855 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "active", 2023-09-10T01:18:54.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.855 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.856 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically balance PGs across cluster", 2023-09-10T01:18:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_time": { 2023-09-10T01:18:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_time", 2023-09-10T01:18:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0000", 2023-09-10T01:18:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "beginning time of day to automatically balance", 2023-09-10T01:18:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-09-10T01:18:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_weekday": { 2023-09-10T01:18:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_weekday", 2023-09-10T01:18:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2023-09-10T01:18:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2023-09-10T01:18:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-09-10T01:18:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-09-10T01:18:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_max_iterations": { 2023-09-10T01:18:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_max_iterations", 2023-09-10T01:18:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "25", 2023-09-10T01:18:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "250", 2023-09-10T01:18:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-09-10T01:18:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_metrics": { 2023-09-10T01:18:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_metrics", 2023-09-10T01:18:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "pgs,objects,bytes", 2023-09-10T01:18:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-09-10T01:18:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-09-10T01:18:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_step": { 2023-09-10T01:18:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_step", 2023-09-10T01:18:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.5", 2023-09-10T01:18:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0.001", 2023-09-10T01:18:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "0.999", 2023-09-10T01:18:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "aggressiveness of optimization", 2023-09-10T01:18:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-09-10T01:18:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "end_time": { 2023-09-10T01:18:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_time", 2023-09-10T01:18:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2359", 2023-09-10T01:18:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ending time of day to automatically balance", 2023-09-10T01:18:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-09-10T01:18:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "end_weekday": { 2023-09-10T01:18:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_weekday", 2023-09-10T01:18:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2023-09-10T01:18:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2023-09-10T01:18:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-09-10T01:18:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-09-10T01:18:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "min_score": { 2023-09-10T01:18:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_score", 2023-09-10T01:18:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-09-10T01:18:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2023-09-10T01:18:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2023-09-10T01:18:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "upmap", 2023-09-10T01:18:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-compat", 2023-09-10T01:18:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "none", 2023-09-10T01:18:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap" 2023-09-10T01:18:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Balancer mode", 2023-09-10T01:18:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_ids": { 2023-09-10T01:18:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_ids", 2023-09-10T01:18:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-09-10T01:18:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-09-10T01:18:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_deviation": { 2023-09-10T01:18:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_deviation", 2023-09-10T01:18:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "deviation below which no optimization is attempted", 2023-09-10T01:18:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-09-10T01:18:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_optimizations": { 2023-09-10T01:18:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_optimizations", 2023-09-10T01:18:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-09-10T01:18:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm", 2023-09-10T01:18:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_down_multiplier": { 2023-09-10T01:18:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_down_multiplier", 2023-09-10T01:18:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2023-09-10T01:18:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-09-10T01:18:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_refresh_rate": { 2023-09-10T01:18:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_refresh_rate", 2023-09-10T01:18:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "20", 2023-09-10T01:18:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-09-10T01:18:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_starting_port": { 2023-09-10T01:18:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_starting_port", 2023-09-10T01:18:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4721", 2023-09-10T01:18:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-09-10T01:18:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_ptrace": { 2023-09-10T01:18:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_ptrace", 2023-09-10T01:18:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-09-10T01:18:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-09-10T01:18:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_interval": { 2023-09-10T01:18:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_interval", 2023-09-10T01:18:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to autotune daemon memory", 2023-09-10T01:18:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_memory_target_ratio": { 2023-09-10T01:18:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_memory_target_ratio", 2023-09-10T01:18:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.7", 2023-09-10T01:18:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-09-10T01:18:54.951 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "cgroups_split": { 2023-09-10T01:18:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cgroups_split", 2023-09-10T01:18:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.953 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.954 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-09-10T01:18:54.955 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.956 INFO:teuthology.orchestra.run.smithi003.stdout: "config_checks_enabled": { 2023-09-10T01:18:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_checks_enabled", 2023-09-10T01:18:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.957 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.958 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-09-10T01:18:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.959 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: "config_dashboard": { 2023-09-10T01:18:54.960 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_dashboard", 2023-09-10T01:18:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.961 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.962 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-09-10T01:18:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_alertmanager": { 2023-09-10T01:18:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_alertmanager", 2023-09-10T01:18:54.964 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.965 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-09-10T01:18:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.966 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.967 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_base": { 2023-09-10T01:18:54.968 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_base", 2023-09-10T01:18:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.969 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph", 2023-09-10T01:18:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.970 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Container image name, without the tag", 2023-09-10T01:18:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.971 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_elasticsearch": { 2023-09-10T01:18:54.972 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_elasticsearch", 2023-09-10T01:18:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.973 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-09-10T01:18:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.974 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "elasticsearch container image", 2023-09-10T01:18:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.975 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_grafana": { 2023-09-10T01:18:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_grafana", 2023-09-10T01:18:54.976 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-09-10T01:18:54.977 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.978 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:54.981 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_haproxy": { 2023-09-10T01:18:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_haproxy", 2023-09-10T01:18:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-09-10T01:18:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.984 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "HAproxy container image", 2023-09-10T01:18:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.985 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.986 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_agent": { 2023-09-10T01:18:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_agent", 2023-09-10T01:18:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.987 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-09-10T01:18:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger agent container image", 2023-09-10T01:18:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.989 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_collector": { 2023-09-10T01:18:54.990 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_collector", 2023-09-10T01:18:54.991 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.991 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.991 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.991 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-09-10T01:18:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.992 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger collector container image", 2023-09-10T01:18:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.993 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_query": { 2023-09-10T01:18:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_query", 2023-09-10T01:18:54.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-09-10T01:18:54.995 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger query container image", 2023-09-10T01:18:54.996 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:54.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_keepalived": { 2023-09-10T01:18:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_keepalived", 2023-09-10T01:18:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:54.998 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-09-10T01:18:54.999 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Keepalived container image", 2023-09-10T01:18:55.000 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.001 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_loki": { 2023-09-10T01:18:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_loki", 2023-09-10T01:18:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.002 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-09-10T01:18:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.003 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Loki container image", 2023-09-10T01:18:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.004 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_node_exporter": { 2023-09-10T01:18:55.005 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_node_exporter", 2023-09-10T01:18:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.006 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-09-10T01:18:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.007 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.008 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_nvmeof": { 2023-09-10T01:18:55.009 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_nvmeof", 2023-09-10T01:18:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.3", 2023-09-10T01:18:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Nvme-of container image", 2023-09-10T01:18:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.012 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_prometheus": { 2023-09-10T01:18:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_prometheus", 2023-09-10T01:18:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.014 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-09-10T01:18:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.015 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2023-09-10T01:18:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.016 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_promtail": { 2023-09-10T01:18:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_promtail", 2023-09-10T01:18:55.017 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-09-10T01:18:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.019 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Promtail container image", 2023-09-10T01:18:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.020 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_snmp_gateway": { 2023-09-10T01:18:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_snmp_gateway", 2023-09-10T01:18:55.021 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-09-10T01:18:55.022 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SNMP Gateway container image", 2023-09-10T01:18:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "container_init": { 2023-09-10T01:18:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_init", 2023-09-10T01:18:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.025 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.026 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Run podman/docker with `--init`", 2023-09-10T01:18:55.027 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.028 INFO:teuthology.orchestra.run.smithi003.stdout: "daemon_cache_timeout": { 2023-09-10T01:18:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "daemon_cache_timeout", 2023-09-10T01:18:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.029 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.030 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-09-10T01:18:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.031 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.032 INFO:teuthology.orchestra.run.smithi003.stdout: "default_cephadm_command_timeout": { 2023-09-10T01:18:55.033 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_cephadm_command_timeout", 2023-09-10T01:18:55.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.033 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.033 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.034 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "900", 2023-09-10T01:18:55.034 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.034 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.035 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.035 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-09-10T01:18:55.035 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.035 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.036 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.036 INFO:teuthology.orchestra.run.smithi003.stdout: "default_registry": { 2023-09-10T01:18:55.036 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_registry", 2023-09-10T01:18:55.037 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.037 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.037 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.038 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io", 2023-09-10T01:18:55.038 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.038 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.038 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.039 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-09-10T01:18:55.039 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.039 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.039 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.040 INFO:teuthology.orchestra.run.smithi003.stdout: "device_cache_timeout": { 2023-09-10T01:18:55.040 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_cache_timeout", 2023-09-10T01:18:55.040 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.041 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.041 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.041 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1800", 2023-09-10T01:18:55.042 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.042 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.042 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.042 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache device inventory", 2023-09-10T01:18:55.043 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.043 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.043 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.044 INFO:teuthology.orchestra.run.smithi003.stdout: "device_enhanced_scan": { 2023-09-10T01:18:55.044 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_enhanced_scan", 2023-09-10T01:18:55.044 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.045 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.045 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.045 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.045 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.046 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.046 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.046 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use libstoragemgmt during device scans", 2023-09-10T01:18:55.047 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.047 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.047 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "facts_cache_timeout": { 2023-09-10T01:18:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "facts_cache_timeout", 2023-09-10T01:18:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.049 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.049 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:55.049 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.050 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.050 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.050 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache host facts data", 2023-09-10T01:18:55.050 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.051 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.051 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.052 INFO:teuthology.orchestra.run.smithi003.stdout: "host_check_interval": { 2023-09-10T01:18:55.052 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "host_check_interval", 2023-09-10T01:18:55.052 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.052 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.053 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.053 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:55.053 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.053 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.054 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.054 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to perform a host check", 2023-09-10T01:18:55.054 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.055 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.055 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.055 INFO:teuthology.orchestra.run.smithi003.stdout: "inventory_list_all": { 2023-09-10T01:18:55.056 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "inventory_list_all", 2023-09-10T01:18:55.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.056 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.056 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.057 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.057 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.057 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.057 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.058 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2023-09-10T01:18:55.058 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.058 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.059 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.059 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.059 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.060 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.060 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.060 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.061 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.061 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.061 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.062 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.062 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.062 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.062 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.063 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.063 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.063 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.063 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.064 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.064 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.064 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.065 INFO:teuthology.orchestra.run.smithi003.stdout: "log_refresh_metadata": { 2023-09-10T01:18:55.065 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_refresh_metadata", 2023-09-10T01:18:55.065 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.066 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.066 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.066 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.066 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.067 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.067 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.067 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-09-10T01:18:55.067 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.068 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.068 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.069 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.069 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.069 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.069 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.070 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.070 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.070 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.070 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.071 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.071 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-09-10T01:18:55.071 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.071 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.072 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.072 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.073 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.073 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.073 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.073 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.074 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.074 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.074 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.074 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.075 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.075 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.075 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.075 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.076 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.076 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.076 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.076 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.077 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.077 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.077 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.078 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.078 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.078 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.079 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.079 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.079 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.079 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.080 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.080 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.080 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.080 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.081 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.081 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.082 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf": { 2023-09-10T01:18:55.082 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-09-10T01:18:55.082 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.082 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.083 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.083 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.083 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.083 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.084 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.084 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-09-10T01:18:55.084 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.085 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.085 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.085 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-09-10T01:18:55.086 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-09-10T01:18:55.086 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.086 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.087 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.087 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "*", 2023-09-10T01:18:55.087 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.087 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.088 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.088 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-09-10T01:18:55.088 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.088 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.089 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.089 INFO:teuthology.orchestra.run.smithi003.stdout: "max_count_per_host": { 2023-09-10T01:18:55.090 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_count_per_host", 2023-09-10T01:18:55.090 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.090 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.090 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.091 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:55.091 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.091 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.092 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.092 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of daemons per service per host", 2023-09-10T01:18:55.092 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.092 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.093 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.093 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd_draining_count": { 2023-09-10T01:18:55.093 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_osd_draining_count", 2023-09-10T01:18:55.094 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.094 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.094 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.095 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:55.095 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.095 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.095 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.096 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-09-10T01:18:55.096 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.096 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.096 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.097 INFO:teuthology.orchestra.run.smithi003.stdout: "migration_current": { 2023-09-10T01:18:55.097 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "migration_current", 2023-09-10T01:18:55.097 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.098 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.098 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.098 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.099 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.099 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.099 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.099 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "internal - do not modify", 2023-09-10T01:18:55.100 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.100 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.100 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.101 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2023-09-10T01:18:55.101 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2023-09-10T01:18:55.101 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.102 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.102 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.102 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "root", 2023-09-10T01:18:55.102 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.103 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.103 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.103 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm-package", 2023-09-10T01:18:55.103 INFO:teuthology.orchestra.run.smithi003.stdout: "root" 2023-09-10T01:18:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "mode for remote execution of cephadm", 2023-09-10T01:18:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "prometheus_alerts_path": { 2023-09-10T01:18:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus_alerts_path", 2023-09-10T01:18:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-09-10T01:18:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-09-10T01:18:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_insecure": { 2023-09-10T01:18:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_insecure", 2023-09-10T01:18:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-09-10T01:18:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_password": { 2023-09-10T01:18:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_password", 2023-09-10T01:18:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-09-10T01:18:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_url": { 2023-09-10T01:18:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_url", 2023-09-10T01:18:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-09-10T01:18:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_username": { 2023-09-10T01:18:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_username", 2023-09-10T01:18:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-09-10T01:18:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "secure_monitoring_stack": { 2023-09-10T01:18:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "secure_monitoring_stack", 2023-09-10T01:18:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-09-10T01:18:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "service_discovery_port": { 2023-09-10T01:18:55.129 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "service_discovery_port", 2023-09-10T01:18:55.129 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.129 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.129 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.130 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8765", 2023-09-10T01:18:55.130 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.130 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.131 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.131 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "cephadm service discovery port", 2023-09-10T01:18:55.131 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.131 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.132 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.132 INFO:teuthology.orchestra.run.smithi003.stdout: "ssh_config_file": { 2023-09-10T01:18:55.132 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssh_config_file", 2023-09-10T01:18:55.133 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.133 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.133 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.133 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.134 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.134 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.134 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.135 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-09-10T01:18:55.135 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.135 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.135 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.136 INFO:teuthology.orchestra.run.smithi003.stdout: "use_agent": { 2023-09-10T01:18:55.136 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_agent", 2023-09-10T01:18:55.136 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.137 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.137 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.137 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.137 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.138 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.138 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.138 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-09-10T01:18:55.139 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.139 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.139 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.140 INFO:teuthology.orchestra.run.smithi003.stdout: "use_repo_digest": { 2023-09-10T01:18:55.140 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_repo_digest", 2023-09-10T01:18:55.140 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.141 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.141 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.141 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.141 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.142 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.142 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.142 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-09-10T01:18:55.142 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.143 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.143 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.143 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_failed_host_check": { 2023-09-10T01:18:55.144 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_failed_host_check", 2023-09-10T01:18:55.144 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.144 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.145 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.145 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.145 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.145 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.146 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.146 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if the host check fails", 2023-09-10T01:18:55.146 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.146 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.147 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.147 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_daemons": { 2023-09-10T01:18:55.147 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_daemons", 2023-09-10T01:18:55.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.148 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.148 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.149 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.149 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.149 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.149 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.150 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-09-10T01:18:55.150 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.150 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.150 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.151 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_hosts": { 2023-09-10T01:18:55.151 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_hosts", 2023-09-10T01:18:55.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.152 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.152 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.152 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.153 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.153 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.153 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.153 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-09-10T01:18:55.154 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.154 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.154 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.155 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.155 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.156 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crash", 2023-09-10T01:18:55.156 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.156 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.156 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.157 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.157 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.157 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.158 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.158 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.158 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.159 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.159 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.159 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.159 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.160 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.160 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.160 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.160 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.161 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.161 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.161 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.161 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.162 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.162 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.163 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.163 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.164 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.164 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.164 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.164 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.165 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.165 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.165 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.165 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.166 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.166 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.167 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.167 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.167 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.168 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.168 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.168 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.169 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.169 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.169 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.169 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.170 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.170 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.170 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.170 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.171 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.171 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.171 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.171 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.172 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.172 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.173 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.173 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.173 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.173 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.174 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.174 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.174 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.174 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.175 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.175 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.175 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.176 INFO:teuthology.orchestra.run.smithi003.stdout: "retain_interval": { 2023-09-10T01:18:55.176 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retain_interval", 2023-09-10T01:18:55.176 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.177 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.177 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.177 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "31536000", 2023-09-10T01:18:55.177 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.178 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.178 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.178 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain crashes before pruning them", 2023-09-10T01:18:55.179 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.179 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.179 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.180 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_recent_interval": { 2023-09-10T01:18:55.180 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_recent_interval", 2023-09-10T01:18:55.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.181 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.181 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.181 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1209600", 2023-09-10T01:18:55.182 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.182 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.182 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.182 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "time interval in which to warn about recent crashes", 2023-09-10T01:18:55.183 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.183 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.183 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.186 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dashboard", 2023-09-10T01:18:55.186 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.186 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.187 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.187 INFO:teuthology.orchestra.run.smithi003.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-09-10T01:18:55.187 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-09-10T01:18:55.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.188 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.188 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.188 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:55.188 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.189 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.189 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.189 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.190 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.190 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.190 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.191 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_HOST": { 2023-09-10T01:18:55.191 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_HOST", 2023-09-10T01:18:55.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.191 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.194 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.194 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-09-10T01:18:55.195 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-09-10T01:18:55.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.195 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.195 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.196 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.196 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.196 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.196 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.197 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.197 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.197 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.198 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.198 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_ENABLED": { 2023-09-10T01:18:55.198 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_ENABLED", 2023-09-10T01:18:55.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.199 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.199 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.199 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.200 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.200 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.200 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.200 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.201 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.201 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.201 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.202 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-09-10T01:18:55.202 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-09-10T01:18:55.202 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.203 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.203 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.203 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.203 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.204 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.204 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.204 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.205 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.205 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.205 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.206 INFO:teuthology.orchestra.run.smithi003.stdout: "ENABLE_BROWSABLE_API": { 2023-09-10T01:18:55.206 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ENABLE_BROWSABLE_API", 2023-09-10T01:18:55.206 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.206 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.207 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.207 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.207 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.208 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.208 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.208 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.208 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.209 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.209 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.209 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-09-10T01:18:55.210 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-09-10T01:18:55.210 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.210 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.210 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.211 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.211 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.211 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.212 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.212 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.212 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.212 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.213 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.213 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-09-10T01:18:55.213 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-09-10T01:18:55.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.214 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.215 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.215 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.215 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.215 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.216 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.216 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.216 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.216 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.217 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-09-10T01:18:55.217 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-09-10T01:18:55.218 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.218 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.218 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.218 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.219 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.219 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.219 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.220 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.220 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.220 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.220 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.221 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-09-10T01:18:55.221 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-09-10T01:18:55.221 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.222 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.222 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.222 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.222 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.223 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.223 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.223 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.224 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.224 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.224 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.225 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_NFS": { 2023-09-10T01:18:55.225 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-09-10T01:18:55.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.225 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.226 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.226 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.226 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.226 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.227 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.227 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.227 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.228 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.228 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.228 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RBD": { 2023-09-10T01:18:55.229 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-09-10T01:18:55.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.229 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.229 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.230 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.230 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.230 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.230 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.231 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.231 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.231 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.232 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.232 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RGW": { 2023-09-10T01:18:55.232 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-09-10T01:18:55.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.233 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.233 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.233 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.234 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.234 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.235 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.235 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.235 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.235 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.236 INFO:teuthology.orchestra.run.smithi003.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-09-10T01:18:55.236 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-09-10T01:18:55.236 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.237 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.237 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.237 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.238 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.238 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.238 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.238 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.239 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.239 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.240 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_PASSWORD": { 2023-09-10T01:18:55.240 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_PASSWORD", 2023-09-10T01:18:55.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.240 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.241 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.241 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:55.241 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.241 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.242 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.242 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.242 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.242 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.243 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.243 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-09-10T01:18:55.244 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-09-10T01:18:55.244 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.244 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.244 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.245 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.245 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.245 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.245 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.246 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.246 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.246 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.246 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.247 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_URL": { 2023-09-10T01:18:55.247 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_URL", 2023-09-10T01:18:55.248 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.248 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.248 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.248 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.249 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.249 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.249 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.249 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.250 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.250 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.250 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.251 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_USERNAME": { 2023-09-10T01:18:55.251 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_USERNAME", 2023-09-10T01:18:55.251 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.252 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.252 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.252 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:55.252 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.253 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.253 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.253 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.253 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.254 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.254 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.255 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-09-10T01:18:55.255 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-09-10T01:18:55.255 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.255 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.256 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.256 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.256 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.257 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.257 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.257 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.257 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.258 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.258 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.258 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-09-10T01:18:55.259 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-09-10T01:18:55.259 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.259 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.260 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.260 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.260 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.260 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.261 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.261 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.261 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.261 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.262 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.262 INFO:teuthology.orchestra.run.smithi003.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-09-10T01:18:55.262 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-09-10T01:18:55.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.263 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.263 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.264 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.264 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.264 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.264 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.265 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.265 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.265 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.266 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.266 INFO:teuthology.orchestra.run.smithi003.stdout: "ISSUE_TRACKER_API_KEY": { 2023-09-10T01:18:55.266 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-09-10T01:18:55.267 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.267 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.267 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.267 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.268 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.268 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.268 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.269 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.269 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.269 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.269 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.270 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_HOST": { 2023-09-10T01:18:55.270 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_HOST", 2023-09-10T01:18:55.270 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.271 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.271 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.271 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.271 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.272 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.272 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.272 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.273 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.273 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.273 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.274 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-09-10T01:18:55.274 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-09-10T01:18:55.274 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.274 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.275 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.275 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.275 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.275 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.276 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.276 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.276 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.277 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.277 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.277 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-09-10T01:18:55.278 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-09-10T01:18:55.278 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.278 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.278 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.279 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.279 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.279 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.279 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.280 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.280 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.280 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.280 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.281 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-09-10T01:18:55.281 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-09-10T01:18:55.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.282 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.282 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.282 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.283 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.283 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.283 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.283 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.284 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.284 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.284 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.285 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-09-10T01:18:55.285 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-09-10T01:18:55.285 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.286 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.286 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.286 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.286 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.287 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.287 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.287 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.287 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.288 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.288 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.288 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-09-10T01:18:55.289 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-09-10T01:18:55.289 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.289 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.290 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.290 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.290 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.290 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.291 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.291 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.291 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.291 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.292 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.292 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-09-10T01:18:55.293 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-09-10T01:18:55.293 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.293 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.293 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.294 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.294 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.294 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.294 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.295 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.295 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.295 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.295 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.296 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-09-10T01:18:55.296 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-09-10T01:18:55.297 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.297 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.297 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.297 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.298 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.298 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.298 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.298 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.299 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.299 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.299 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.300 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-09-10T01:18:55.300 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-09-10T01:18:55.300 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.301 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.301 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.301 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.301 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.302 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.302 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.302 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.303 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.303 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.303 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.304 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_ENABLED": { 2023-09-10T01:18:55.304 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_ENABLED", 2023-09-10T01:18:55.304 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.304 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.305 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.305 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.305 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.305 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.306 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.306 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.306 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.307 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.307 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.307 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-09-10T01:18:55.308 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-09-10T01:18:55.308 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.308 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.308 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.309 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-09-10T01:18:55.309 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.309 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.309 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.310 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.310 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.310 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.311 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.311 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-09-10T01:18:55.311 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-09-10T01:18:55.312 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.312 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.312 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.312 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:55.313 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.313 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.313 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.313 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.314 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.314 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.314 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.315 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-09-10T01:18:55.315 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-09-10T01:18:55.315 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.316 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.316 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.316 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8", 2023-09-10T01:18:55.316 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.317 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.317 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.317 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.318 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.318 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.318 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.319 INFO:teuthology.orchestra.run.smithi003.stdout: "REST_REQUESTS_TIMEOUT": { 2023-09-10T01:18:55.319 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-09-10T01:18:55.319 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.320 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.320 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.320 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "45", 2023-09-10T01:18:55.320 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.321 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.321 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.321 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.321 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.322 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.322 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.322 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ACCESS_KEY": { 2023-09-10T01:18:55.323 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ACCESS_KEY", 2023-09-10T01:18:55.323 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.323 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.324 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.324 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.324 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.324 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.325 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.325 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.325 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.326 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.326 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.326 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-09-10T01:18:55.327 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-09-10T01:18:55.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.327 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.327 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.328 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2023-09-10T01:18:55.328 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.328 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.329 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.329 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.329 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.329 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.330 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.330 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SECRET_KEY": { 2023-09-10T01:18:55.330 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SECRET_KEY", 2023-09-10T01:18:55.331 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.331 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.331 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.331 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.332 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.332 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.332 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.333 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.333 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.333 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.333 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.334 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SSL_VERIFY": { 2023-09-10T01:18:55.334 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SSL_VERIFY", 2023-09-10T01:18:55.334 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.335 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.335 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.335 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.336 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.336 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.336 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.336 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.337 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.337 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.337 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-09-10T01:18:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-09-10T01:18:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.339 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.339 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.339 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:55.339 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.340 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.340 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.340 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.340 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.341 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.341 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.341 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-09-10T01:18:55.342 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-09-10T01:18:55.342 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.342 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.343 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.343 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:55.343 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.343 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.344 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.344 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.344 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.344 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.345 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.345 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-09-10T01:18:55.345 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-09-10T01:18:55.346 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.346 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.346 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.347 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:55.347 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.347 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.347 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.348 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.348 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.348 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.348 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.349 INFO:teuthology.orchestra.run.smithi003.stdout: "cross_origin_url": { 2023-09-10T01:18:55.349 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cross_origin_url", 2023-09-10T01:18:55.349 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.350 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.350 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.350 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.351 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.351 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.351 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.351 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.352 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.352 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.352 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.353 INFO:teuthology.orchestra.run.smithi003.stdout: "crt_file": { 2023-09-10T01:18:55.353 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crt_file", 2023-09-10T01:18:55.353 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.353 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.354 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.354 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.355 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.355 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.355 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.355 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.356 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.356 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.356 INFO:teuthology.orchestra.run.smithi003.stdout: "debug": { 2023-09-10T01:18:55.357 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "debug", 2023-09-10T01:18:55.357 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.357 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.357 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.358 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.358 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.358 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.359 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.359 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable/disable debug options", 2023-09-10T01:18:55.359 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.359 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.360 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.360 INFO:teuthology.orchestra.run.smithi003.stdout: "jwt_token_ttl": { 2023-09-10T01:18:55.360 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "jwt_token_ttl", 2023-09-10T01:18:55.361 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.361 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.361 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.361 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "28800", 2023-09-10T01:18:55.362 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.362 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.362 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.363 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.363 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.363 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.363 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.364 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2023-09-10T01:18:55.364 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2023-09-10T01:18:55.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.365 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.365 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.365 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.365 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.366 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.366 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.366 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.367 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.367 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.367 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.368 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.368 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.368 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.369 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.369 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.369 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.370 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.370 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.370 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.371 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.371 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.371 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.371 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.372 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.372 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.372 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.372 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.373 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.373 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.374 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.374 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.374 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.374 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.375 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.375 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.375 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.376 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.376 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.376 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.377 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.377 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.377 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.378 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.378 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.378 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.378 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.379 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.379 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.379 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.379 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.380 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.380 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.380 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.381 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.381 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.381 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.381 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.382 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.382 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.382 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.383 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.383 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.384 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.384 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.384 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.385 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.385 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.385 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.385 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.386 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.386 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.386 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.387 INFO:teuthology.orchestra.run.smithi003.stdout: "motd": { 2023-09-10T01:18:55.387 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "motd", 2023-09-10T01:18:55.387 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.388 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.388 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.388 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.388 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.389 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.389 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.389 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "The message of the day", 2023-09-10T01:18:55.389 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.390 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.390 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.391 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect_resolve_ip_addr": { 2023-09-10T01:18:55.391 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "redirect_resolve_ip_addr", 2023-09-10T01:18:55.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.391 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.392 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.392 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.392 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.393 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.393 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.393 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.393 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.394 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.394 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:55.395 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:55.395 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.395 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.395 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.396 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2023-09-10T01:18:55.396 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.396 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.396 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.397 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.397 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.397 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.398 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.398 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:55.398 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:55.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.399 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.399 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.399 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8080", 2023-09-10T01:18:55.400 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.400 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.400 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.401 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.401 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.401 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.401 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.402 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2023-09-10T01:18:55.402 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2023-09-10T01:18:55.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.403 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.403 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.404 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.404 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.404 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.404 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.405 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.405 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.405 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.406 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl_server_port": { 2023-09-10T01:18:55.406 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl_server_port", 2023-09-10T01:18:55.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.406 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.407 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8443", 2023-09-10T01:18:55.407 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.407 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.408 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.408 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.408 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.409 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.409 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.409 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2023-09-10T01:18:55.410 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2023-09-10T01:18:55.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.410 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.410 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "redirect", 2023-09-10T01:18:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.411 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.412 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.412 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect" 2023-09-10T01:18:55.412 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.413 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2023-09-10T01:18:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2023-09-10T01:18:55.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.415 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.415 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.415 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2023-09-10T01:18:55.416 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2023-09-10T01:18:55.416 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2023-09-10T01:18:55.416 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.416 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.417 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.417 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.417 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.418 INFO:teuthology.orchestra.run.smithi003.stdout: "url_prefix": { 2023-09-10T01:18:55.418 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url_prefix", 2023-09-10T01:18:55.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.419 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.419 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.419 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.420 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.420 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.420 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.420 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.421 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.421 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.421 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.421 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2023-09-10T01:18:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_monitoring": { 2023-09-10T01:18:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_monitoring", 2023-09-10T01:18:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "monitor device health metrics", 2023-09-10T01:18:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mark_out_threshold": { 2023-09-10T01:18:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mark_out_threshold", 2023-09-10T01:18:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2419200", 2023-09-10T01:18:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-09-10T01:18:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": { 2023-09-10T01:18:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_name", 2023-09-10T01:18:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "device_health_metrics", 2023-09-10T01:18:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name of pool in which to store device health metrics", 2023-09-10T01:18:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "retention_period": { 2023-09-10T01:18:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retention_period", 2023-09-10T01:18:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15552000", 2023-09-10T01:18:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain device health metrics", 2023-09-10T01:18:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_frequency": { 2023-09-10T01:18:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_frequency", 2023-09-10T01:18:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2023-09-10T01:18:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to scrape device health metrics", 2023-09-10T01:18:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "self_heal": { 2023-09-10T01:18:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "self_heal", 2023-09-10T01:18:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-09-10T01:18:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and check device health", 2023-09-10T01:18:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_threshold": { 2023-09-10T01:18:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_threshold", 2023-09-10T01:18:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7257600", 2023-09-10T01:18:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-09-10T01:18:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "diskprediction_local", 2023-09-10T01:18:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.492 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.492 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.492 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "predict_interval": { 2023-09-10T01:18:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predict_interval", 2023-09-10T01:18:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2023-09-10T01:18:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "predictor_model": { 2023-09-10T01:18:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predictor_model", 2023-09-10T01:18:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "prophetstor", 2023-09-10T01:18:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2023-09-10T01:18:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "influx", 2023-09-10T01:18:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": false, 2023-09-10T01:18:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "influxdb python module not found", 2023-09-10T01:18:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "batch_size": { 2023-09-10T01:18:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "batch_size", 2023-09-10T01:18:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5000", 2023-09-10T01:18:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-09-10T01:18:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "database": { 2023-09-10T01:18:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "database", 2023-09-10T01:18:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "ceph", 2023-09-10T01:18:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-09-10T01:18:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": { 2023-09-10T01:18:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hostname", 2023-09-10T01:18:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server hostname", 2023-09-10T01:18:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "30", 2023-09-10T01:18:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "5", 2023-09-10T01:18:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-09-10T01:18:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.538 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.538 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.538 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.539 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.539 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.539 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.539 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.540 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.540 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.540 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.541 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.541 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.541 INFO:teuthology.orchestra.run.smithi003.stdout: "password": { 2023-09-10T01:18:55.542 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "password", 2023-09-10T01:18:55.542 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.542 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.543 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.543 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.543 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.543 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.544 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.544 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "password of InfluxDB server user", 2023-09-10T01:18:55.544 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.544 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.545 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.545 INFO:teuthology.orchestra.run.smithi003.stdout: "port": { 2023-09-10T01:18:55.545 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "port", 2023-09-10T01:18:55.546 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.546 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.546 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.547 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8086", 2023-09-10T01:18:55.547 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.547 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.547 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.548 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server port", 2023-09-10T01:18:55.548 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.548 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.549 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.549 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2023-09-10T01:18:55.549 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2023-09-10T01:18:55.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.550 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.550 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.550 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "false", 2023-09-10T01:18:55.551 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.551 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.551 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.552 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-09-10T01:18:55.552 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.552 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.552 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.553 INFO:teuthology.orchestra.run.smithi003.stdout: "threads": { 2023-09-10T01:18:55.553 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threads", 2023-09-10T01:18:55.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.554 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.554 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.554 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:55.555 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:55.555 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "32", 2023-09-10T01:18:55.555 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.555 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-09-10T01:18:55.556 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.556 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.556 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.557 INFO:teuthology.orchestra.run.smithi003.stdout: "username": { 2023-09-10T01:18:55.557 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "username", 2023-09-10T01:18:55.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.558 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.558 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.558 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.558 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.559 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.559 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.559 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "username of InfluxDB server user", 2023-09-10T01:18:55.559 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.560 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.560 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.560 INFO:teuthology.orchestra.run.smithi003.stdout: "verify_ssl": { 2023-09-10T01:18:55.561 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "verify_ssl", 2023-09-10T01:18:55.561 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.561 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.562 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.562 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "true", 2023-09-10T01:18:55.562 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.562 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.563 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.563 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-09-10T01:18:55.563 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.563 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.564 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.564 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.564 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.565 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "insights", 2023-09-10T01:18:55.565 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.566 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.566 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.566 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.567 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.567 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.568 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.568 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.568 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.569 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.569 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.569 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.570 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.570 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.570 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.570 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.571 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.571 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.571 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.571 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.572 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.572 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.573 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.573 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.573 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.574 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.574 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.574 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.574 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.575 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.575 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.575 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.576 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.576 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.576 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.577 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.577 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.577 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.578 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.578 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.578 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.579 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.579 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.579 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.579 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.580 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.580 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.580 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.580 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.581 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.581 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.581 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.581 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.582 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.582 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.583 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.583 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.583 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.584 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.584 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.584 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.585 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.585 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.585 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.585 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.586 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.587 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "iostat", 2023-09-10T01:18:55.587 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.587 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.588 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.588 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.588 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.589 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.589 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.589 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.590 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.590 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.593 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.593 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.594 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.594 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.595 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.595 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.596 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.596 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.596 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.597 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.597 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.598 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.598 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.599 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.599 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.600 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.600 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.600 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.601 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.601 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.601 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.602 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.602 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.602 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.603 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.603 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.604 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.604 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.604 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.605 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.605 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.605 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.605 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.606 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.606 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.606 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.606 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.607 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.607 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.607 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.608 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.608 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.608 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.609 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.609 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.609 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.610 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.610 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.610 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.611 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.611 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.611 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.611 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.612 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.612 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.613 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "k8sevents", 2023-09-10T01:18:55.613 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.614 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.614 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.614 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_event_retention_days": { 2023-09-10T01:18:55.615 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ceph_event_retention_days", 2023-09-10T01:18:55.615 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.615 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.615 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.616 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7", 2023-09-10T01:18:55.616 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.616 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.617 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.617 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Days to hold ceph event information within local cache", 2023-09-10T01:18:55.617 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.617 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.618 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.618 INFO:teuthology.orchestra.run.smithi003.stdout: "config_check_secs": { 2023-09-10T01:18:55.619 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_check_secs", 2023-09-10T01:18:55.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.619 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.620 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.620 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:55.620 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "10", 2023-09-10T01:18:55.620 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.621 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.621 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-09-10T01:18:55.621 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.621 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.622 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.622 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.623 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.623 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.623 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.624 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.624 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.624 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.625 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.625 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.625 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.625 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.626 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.626 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.626 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.627 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.627 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.627 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.627 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.628 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.628 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.629 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.629 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.629 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.629 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.630 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.630 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.630 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.630 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.631 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.631 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.631 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.632 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.632 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.632 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.633 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.633 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.633 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.634 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.634 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.634 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.635 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.635 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.635 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.635 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.636 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.636 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.636 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.636 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.637 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.637 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.638 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.638 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.638 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.639 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.639 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.639 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.639 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.640 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.640 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.640 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.640 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.641 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.641 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.642 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "localpool", 2023-09-10T01:18:55.643 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.643 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.643 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.643 INFO:teuthology.orchestra.run.smithi003.stdout: "failure_domain": { 2023-09-10T01:18:55.644 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "failure_domain", 2023-09-10T01:18:55.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.644 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.644 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.645 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "host", 2023-09-10T01:18:55.645 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.645 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.646 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.646 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "failure domain for any created local pool", 2023-09-10T01:18:55.646 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-09-10T01:18:55.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.647 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.647 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.647 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.648 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.649 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.649 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.649 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.649 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.650 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.650 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.650 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.650 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.651 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.651 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.651 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.652 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.652 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.652 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.652 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.653 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.653 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.654 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.654 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.654 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.655 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.655 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.655 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.656 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.656 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.656 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.657 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.657 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.657 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.658 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.658 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.658 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.658 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.659 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.659 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.659 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.660 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.660 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.660 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.660 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.661 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.661 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.661 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.662 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.662 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.662 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.663 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.664 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.664 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.664 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.664 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.665 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.665 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.665 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.666 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": { 2023-09-10T01:18:55.666 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_size", 2023-09-10T01:18:55.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.667 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.667 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.668 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.668 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.668 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.668 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default min_size for any created local pool", 2023-09-10T01:18:55.669 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-09-10T01:18:55.669 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.669 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.670 INFO:teuthology.orchestra.run.smithi003.stdout: "num_rep": { 2023-09-10T01:18:55.670 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "num_rep", 2023-09-10T01:18:55.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.671 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.671 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.671 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3", 2023-09-10T01:18:55.671 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.672 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.672 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.672 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default replica count for any created local pool", 2023-09-10T01:18:55.673 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.673 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.673 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.674 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": { 2023-09-10T01:18:55.674 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_num", 2023-09-10T01:18:55.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.674 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.675 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.675 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "128", 2023-09-10T01:18:55.675 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.675 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.676 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.676 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default pg_num for any created local pool", 2023-09-10T01:18:55.676 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.676 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.677 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.677 INFO:teuthology.orchestra.run.smithi003.stdout: "prefix": { 2023-09-10T01:18:55.678 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prefix", 2023-09-10T01:18:55.678 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.678 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.678 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.679 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.679 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.679 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.679 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.680 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name prefix for any created local pool", 2023-09-10T01:18:55.680 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.680 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.680 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.681 INFO:teuthology.orchestra.run.smithi003.stdout: "subtree": { 2023-09-10T01:18:55.681 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "subtree", 2023-09-10T01:18:55.681 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.682 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.682 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.682 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "rack", 2023-09-10T01:18:55.683 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.683 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.683 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.683 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "CRUSH level for which to create a local pool", 2023-09-10T01:18:55.684 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-09-10T01:18:55.684 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.684 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.684 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.685 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mds_autoscaler", 2023-09-10T01:18:55.686 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.686 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.686 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.687 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.687 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.687 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.688 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.688 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.688 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.688 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.689 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.689 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.689 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.689 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.690 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.690 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.690 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.691 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.691 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.691 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.691 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.692 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.692 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.692 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.693 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.693 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.693 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.693 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.694 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.694 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.694 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.695 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.695 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.695 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.695 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.696 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.696 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.697 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.697 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.697 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.697 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.698 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.698 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.698 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.699 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.699 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.699 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.699 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.700 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.700 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.700 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.700 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.702 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.703 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.704 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.705 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.706 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirroring", 2023-09-10T01:18:55.706 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.707 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.707 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.707 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.707 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.708 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.708 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.709 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.709 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.709 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.709 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.710 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.710 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.710 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.710 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.711 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.711 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.711 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.712 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.712 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.712 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.712 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.713 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.713 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.714 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.714 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.714 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.715 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.717 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.718 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.719 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.720 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.721 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.723 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.724 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.725 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "nfs", 2023-09-10T01:18:55.727 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.728 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.729 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.730 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.731 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.732 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.733 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.735 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.735 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.735 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.735 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.736 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.736 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.736 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.736 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.737 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.737 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.738 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.738 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.738 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.738 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.739 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.739 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.739 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.739 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.740 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.740 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.740 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.740 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.741 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.741 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.741 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.742 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.742 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.742 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.743 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.743 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.743 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.744 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.744 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.745 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.745 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.745 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.745 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.746 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.746 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.746 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.748 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2023-09-10T01:18:55.748 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.748 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.749 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.749 INFO:teuthology.orchestra.run.smithi003.stdout: "fail_fs": { 2023-09-10T01:18:55.749 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "fail_fs", 2023-09-10T01:18:55.749 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.750 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.750 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.750 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.750 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.751 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.751 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.751 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-09-10T01:18:55.752 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.752 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.752 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.753 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.753 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.753 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.754 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.754 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.754 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.754 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.755 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.755 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.755 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.756 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.756 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.756 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.756 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.757 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.757 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.757 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.757 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.758 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.758 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.759 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.759 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.759 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.760 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.760 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.760 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.760 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.761 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.761 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.761 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.762 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.762 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.763 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.763 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.763 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.763 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.764 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.764 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.764 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.764 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.765 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.765 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.765 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.766 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.766 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.766 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.766 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.767 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.767 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.767 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.768 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.769 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.769 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.769 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.769 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.770 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.770 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.770 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.770 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.771 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.771 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.771 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.772 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator": { 2023-09-10T01:18:55.772 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2023-09-10T01:18:55.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.773 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.773 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.773 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.773 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.774 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.774 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.774 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2023-09-10T01:18:55.775 INFO:teuthology.orchestra.run.smithi003.stdout: "rook", 2023-09-10T01:18:55.775 INFO:teuthology.orchestra.run.smithi003.stdout: "test_orchestrator" 2023-09-10T01:18:55.775 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.775 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Orchestrator backend", 2023-09-10T01:18:55.776 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.776 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.776 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.777 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.777 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_perf_query", 2023-09-10T01:18:55.778 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.778 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.778 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.779 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.779 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.779 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.779 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.780 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.780 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.780 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.780 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.781 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.781 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.781 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.782 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.782 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.782 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.782 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.783 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.783 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.783 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.783 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.784 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.784 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.785 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.785 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.785 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.785 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.786 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.786 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.786 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.788 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.788 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.788 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.789 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.789 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.789 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.790 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.790 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.790 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.790 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.791 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.791 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.791 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.792 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.792 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.792 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.792 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.793 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.793 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.793 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.793 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.794 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.794 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.795 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.795 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.795 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.795 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.796 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.796 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.796 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.796 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.797 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.797 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.797 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.798 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.799 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_support", 2023-09-10T01:18:55.799 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.799 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.800 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.800 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.800 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.801 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.801 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.801 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.801 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.802 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.802 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.802 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.802 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.803 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.803 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.803 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.804 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.804 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.804 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.804 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.805 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.805 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.805 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.806 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.806 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.807 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.807 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.807 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.807 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.808 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.808 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.808 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.808 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.809 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.809 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.809 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.810 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.810 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.811 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.811 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.811 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.811 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.812 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.812 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.812 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.812 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.813 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.813 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.813 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.814 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.814 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.814 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.814 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.815 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.815 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.815 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.816 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.816 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.816 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.817 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.817 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.817 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.817 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.818 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.818 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.818 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.819 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.820 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_autoscaler", 2023-09-10T01:18:55.820 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.821 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.821 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.821 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.821 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.822 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.822 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.823 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.823 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.823 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.823 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.824 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.824 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.824 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.824 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.825 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.825 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.825 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.826 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.826 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.826 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.827 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.827 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.827 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.828 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.828 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.828 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.829 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.829 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.829 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.829 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.830 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.830 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.830 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.830 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.831 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.831 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.832 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.832 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.832 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.832 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.833 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.833 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.833 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.833 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.834 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.834 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.834 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.835 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.835 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.835 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.835 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.836 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.836 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.836 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.837 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.837 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.838 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.838 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.838 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.838 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.839 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.839 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.839 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.839 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.840 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.840 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.841 INFO:teuthology.orchestra.run.smithi003.stdout: "noautoscale": { 2023-09-10T01:18:55.841 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "noautoscale", 2023-09-10T01:18:55.841 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.841 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.842 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.842 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.842 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.843 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.843 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.843 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "global autoscale flag", 2023-09-10T01:18:55.843 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-09-10T01:18:55.844 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.844 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.844 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:55.845 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:55.845 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.845 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.846 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.846 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:55.846 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.846 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.847 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.847 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.847 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.847 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.848 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.848 INFO:teuthology.orchestra.run.smithi003.stdout: "threshold": { 2023-09-10T01:18:55.849 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threshold", 2023-09-10T01:18:55.849 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:55.849 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.849 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.850 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2023-09-10T01:18:55.850 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1.0", 2023-09-10T01:18:55.850 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.850 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.851 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "scaling threshold", 2023-09-10T01:18:55.851 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-09-10T01:18:55.851 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.852 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.852 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.852 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.853 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "progress", 2023-09-10T01:18:55.853 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.853 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.854 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.854 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_pg_recovery_event": { 2023-09-10T01:18:55.854 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_pg_recovery_event", 2023-09-10T01:18:55.854 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.855 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.855 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.855 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.856 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.856 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.856 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.856 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow the module to show pg recovery progress", 2023-09-10T01:18:55.857 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.857 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.857 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.858 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2023-09-10T01:18:55.858 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2023-09-10T01:18:55.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.858 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.859 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.859 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.859 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.860 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.860 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.860 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.860 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.861 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.861 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.861 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.862 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.862 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.862 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.862 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.863 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.863 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.863 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.864 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.864 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.864 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.864 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.865 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.865 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.865 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.865 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.866 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.866 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.866 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.867 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.867 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.867 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.868 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.868 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.868 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.868 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.869 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.869 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.869 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.869 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.870 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.870 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.870 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.871 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.871 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.872 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.872 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.872 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.872 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.873 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.873 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.873 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.874 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.874 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.874 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.874 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.875 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.875 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.875 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.875 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.876 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.876 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.876 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.877 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.877 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.877 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.878 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.878 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.878 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.879 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.879 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.879 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.879 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.880 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.880 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.880 INFO:teuthology.orchestra.run.smithi003.stdout: "max_completed_events": { 2023-09-10T01:18:55.881 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_completed_events", 2023-09-10T01:18:55.881 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.881 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.881 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.882 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "50", 2023-09-10T01:18:55.882 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.882 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.883 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.883 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "number of past completed events to remember", 2023-09-10T01:18:55.884 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.884 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.884 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.885 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2023-09-10T01:18:55.885 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2023-09-10T01:18:55.886 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:55.886 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.887 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.887 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2023-09-10T01:18:55.887 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.887 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.888 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.888 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long the module is going to sleep", 2023-09-10T01:18:55.889 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.889 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.889 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.890 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.890 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus", 2023-09-10T01:18:55.891 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.891 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.891 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.892 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2023-09-10T01:18:55.892 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cache", 2023-09-10T01:18:55.892 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.892 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.893 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.893 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.893 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.893 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.894 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.894 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.894 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.894 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.895 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.895 INFO:teuthology.orchestra.run.smithi003.stdout: "exclude_perf_counters": { 2023-09-10T01:18:55.896 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "exclude_perf_counters", 2023-09-10T01:18:55.896 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.896 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.896 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.897 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.897 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.897 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.897 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.898 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-09-10T01:18:55.898 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-09-10T01:18:55.898 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.899 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.899 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.899 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.900 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.900 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.900 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.900 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.901 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.901 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.901 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.901 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.902 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.902 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.902 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.903 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.903 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.903 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.903 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.904 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.904 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.904 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.905 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.905 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.905 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.906 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.906 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.906 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.907 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.907 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.907 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.907 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.908 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.908 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.908 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.909 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.909 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.909 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.909 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.910 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.910 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.910 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.911 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.911 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.911 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.911 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.912 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.912 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.912 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.912 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.913 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.913 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.913 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.914 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.914 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.915 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.915 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.915 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.916 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.916 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.916 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.916 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.917 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.917 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.917 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.918 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.918 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.918 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools": { 2023-09-10T01:18:55.919 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools", 2023-09-10T01:18:55.919 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.919 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.919 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.920 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.920 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.920 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.921 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.921 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.921 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.922 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.922 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.922 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools_refresh_interval": { 2023-09-10T01:18:55.923 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-09-10T01:18:55.923 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.923 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.923 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.924 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300", 2023-09-10T01:18:55.924 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.924 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.925 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.925 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.925 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.925 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.926 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.926 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_interval": { 2023-09-10T01:18:55.927 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_interval", 2023-09-10T01:18:55.927 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:55.927 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.927 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.928 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15.0", 2023-09-10T01:18:55.928 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.928 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.928 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.929 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.929 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.929 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.930 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.930 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:55.930 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:55.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.931 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.931 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2023-09-10T01:18:55.932 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.932 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.932 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.933 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-09-10T01:18:55.933 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.933 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.933 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.934 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:55.934 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:55.934 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.935 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.935 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.935 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "9283", 2023-09-10T01:18:55.936 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.936 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.936 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.936 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-09-10T01:18:55.937 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.937 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.937 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.938 INFO:teuthology.orchestra.run.smithi003.stdout: "stale_cache_strategy": { 2023-09-10T01:18:55.938 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stale_cache_strategy", 2023-09-10T01:18:55.938 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.939 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.939 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.939 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "log", 2023-09-10T01:18:55.939 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.940 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.940 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.940 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.941 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.941 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.941 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.942 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2023-09-10T01:18:55.942 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2023-09-10T01:18:55.942 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.943 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.943 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.943 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "default", 2023-09-10T01:18:55.943 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.944 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.944 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.944 INFO:teuthology.orchestra.run.smithi003.stdout: "default", 2023-09-10T01:18:55.944 INFO:teuthology.orchestra.run.smithi003.stdout: "error" 2023-09-10T01:18:55.945 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.945 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.945 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.946 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.946 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.946 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2023-09-10T01:18:55.947 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2023-09-10T01:18:55.947 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.947 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.947 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.948 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2023-09-10T01:18:55.948 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2023-09-10T01:18:55.949 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2023-09-10T01:18:55.949 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.949 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.949 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.950 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.950 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.951 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.952 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2023-09-10T01:18:55.952 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.952 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.952 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.953 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.953 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.953 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.954 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.954 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.955 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.955 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.955 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.955 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.956 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.956 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.956 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.957 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.957 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.957 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.957 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.958 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.958 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.958 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.959 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:55.959 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:55.959 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.960 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.960 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.960 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.960 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.961 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.961 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.961 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.962 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.962 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.962 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.963 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:55.963 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:55.963 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.963 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.964 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.964 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:55.964 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.965 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.965 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.965 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.965 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:55.966 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:55.966 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:55.966 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:55.967 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:55.967 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:55.967 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.968 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.968 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.968 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.969 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:55.969 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:55.969 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.970 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.970 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.970 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:55.970 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.971 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.971 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.971 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.971 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.972 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.972 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.973 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_snap_create": { 2023-09-10T01:18:55.973 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_snap_create", 2023-09-10T01:18:55.973 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:55.973 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.974 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.974 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2023-09-10T01:18:55.974 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.975 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.975 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.975 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.975 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.976 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.976 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.977 INFO:teuthology.orchestra.run.smithi003.stdout: "mirror_snapshot_schedule": { 2023-09-10T01:18:55.977 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirror_snapshot_schedule", 2023-09-10T01:18:55.977 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.977 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.978 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.978 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.978 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.979 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.979 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.979 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.979 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.980 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.980 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.980 INFO:teuthology.orchestra.run.smithi003.stdout: "trash_purge_schedule": { 2023-09-10T01:18:55.981 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "trash_purge_schedule", 2023-09-10T01:18:55.981 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.981 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.982 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.982 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.982 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.982 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.983 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.983 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.983 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.984 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.984 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.985 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.985 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:55.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:55.986 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "restful", 2023-09-10T01:18:55.987 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:55.987 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:55.988 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:55.988 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_auth": { 2023-09-10T01:18:55.988 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_auth", 2023-09-10T01:18:55.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:55.989 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.990 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.990 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:55.990 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.991 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.991 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.991 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.991 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.992 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.992 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.992 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2023-09-10T01:18:55.993 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2023-09-10T01:18:55.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.993 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.994 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:55.994 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.994 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.994 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.995 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:55.995 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:55.997 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:55.997 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:55.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:55.997 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:55.998 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:55.998 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:55.998 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:55.999 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:55.999 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:55.999 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:55.999 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.000 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.000 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.000 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.001 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.001 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.001 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.002 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.002 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.002 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.003 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.003 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.003 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.004 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.004 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.004 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.004 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.005 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.005 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.005 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.005 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.006 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.006 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.007 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.007 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.007 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.007 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.008 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.008 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.008 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.009 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.009 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.009 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.009 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.010 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.010 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.010 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.011 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.011 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.011 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.011 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.012 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.012 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.013 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.013 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.013 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.014 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.014 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.014 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.014 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.015 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.015 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.015 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.016 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.016 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.017 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2023-09-10T01:18:56.017 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2023-09-10T01:18:56.018 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.018 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.019 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.019 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.019 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.020 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.020 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.021 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.021 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.021 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.022 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.022 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2023-09-10T01:18:56.023 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2023-09-10T01:18:56.023 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.023 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.023 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.024 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.024 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.024 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.025 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.025 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.025 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.025 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.026 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.027 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rgw", 2023-09-10T01:18:56.027 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.028 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.028 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.028 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.028 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.029 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.029 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.030 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.030 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.030 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.030 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.031 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.031 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.031 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.032 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.032 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.032 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.032 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.033 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.033 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.033 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.033 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.034 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.034 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.035 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.035 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.035 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.035 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.036 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.036 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.036 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.036 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.037 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.037 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.037 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.038 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.038 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.038 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.039 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.039 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.040 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.040 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.040 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.040 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.041 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.041 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.041 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.041 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.042 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.042 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.042 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.043 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.043 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.043 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.044 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.044 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.044 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.045 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.046 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.047 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.047 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.047 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.048 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.048 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rook", 2023-09-10T01:18:56.049 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.049 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.049 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.049 INFO:teuthology.orchestra.run.smithi003.stdout: "drive_group_interval": { 2023-09-10T01:18:56.050 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "drive_group_interval", 2023-09-10T01:18:56.050 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:56.050 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.051 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.051 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300.0", 2023-09-10T01:18:56.051 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.052 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.052 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.052 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-09-10T01:18:56.052 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.053 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.053 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.053 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.054 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.054 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.054 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.055 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.055 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.055 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.055 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.056 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.056 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.056 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.056 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.057 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.057 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.057 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.058 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.058 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.058 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.058 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.059 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.059 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.059 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.060 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.060 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.061 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.061 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.061 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.062 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.062 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.062 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.062 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.063 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.063 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.064 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.065 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.066 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.066 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.066 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.067 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.067 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.067 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.067 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.068 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.070 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.070 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.070 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.070 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.071 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.071 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.071 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.071 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.072 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.072 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.072 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.073 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_class": { 2023-09-10T01:18:56.073 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "storage_class", 2023-09-10T01:18:56.073 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "local", 2023-09-10T01:18:56.074 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.075 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.075 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.075 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-09-10T01:18:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.077 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "selftest", 2023-09-10T01:18:56.078 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.078 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.078 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.079 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.079 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.079 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.079 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.080 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.080 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.080 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.080 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.081 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.081 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.081 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.082 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.083 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.083 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.083 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.083 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.084 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.085 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.085 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.085 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.086 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.086 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.086 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.086 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.088 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.089 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.090 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.091 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.092 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.094 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.096 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.097 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.098 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "roption1": { 2023-09-10T01:18:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption1", 2023-09-10T01:18:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.100 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.101 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.102 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.103 INFO:teuthology.orchestra.run.smithi003.stdout: "roption2": { 2023-09-10T01:18:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption2", 2023-09-10T01:18:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.104 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "xyz", 2023-09-10T01:18:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.105 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.106 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.107 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption1": { 2023-09-10T01:18:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption1", 2023-09-10T01:18:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.108 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.109 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.110 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption2": { 2023-09-10T01:18:56.111 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption2", 2023-09-10T01:18:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.112 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.113 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.114 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption3": { 2023-09-10T01:18:56.115 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption3", 2023-09-10T01:18:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2023-09-10T01:18:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.116 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.117 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.118 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption4": { 2023-09-10T01:18:56.119 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption4", 2023-09-10T01:18:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.120 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.121 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.122 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption5": { 2023-09-10T01:18:56.123 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption5", 2023-09-10T01:18:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.124 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.125 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.126 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption6": { 2023-09-10T01:18:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption6", 2023-09-10T01:18:56.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.128 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.128 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.128 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.129 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.130 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.130 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.130 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.131 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption7": { 2023-09-10T01:18:56.131 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption7", 2023-09-10T01:18:56.131 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:56.132 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.132 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.132 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.132 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2023-09-10T01:18:56.133 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "42", 2023-09-10T01:18:56.133 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.133 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.135 INFO:teuthology.orchestra.run.smithi003.stdout: "testkey": { 2023-09-10T01:18:56.135 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testkey", 2023-09-10T01:18:56.135 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.136 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.137 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.137 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.137 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "testlkey": { 2023-09-10T01:18:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testlkey", 2023-09-10T01:18:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.139 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.140 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.140 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.140 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.141 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.141 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.141 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.141 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.142 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.142 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.142 INFO:teuthology.orchestra.run.smithi003.stdout: "testnewline": { 2023-09-10T01:18:56.143 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testnewline", 2023-09-10T01:18:56.143 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.143 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.144 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.145 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.145 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.145 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.145 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.146 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.146 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.146 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2023-09-10T01:18:56.147 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.148 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.148 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.148 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_m_granularity": { 2023-09-10T01:18:56.148 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_m_granularity", 2023-09-10T01:18:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.149 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.150 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.150 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.150 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.151 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow minute scheduled snapshots", 2023-09-10T01:18:56.151 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.151 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: "dump_on_update": { 2023-09-10T01:18:56.152 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dump_on_update", 2023-09-10T01:18:56.153 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.153 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.153 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.154 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.155 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "dump database to debug log on update", 2023-09-10T01:18:56.155 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.155 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.156 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.156 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.156 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.157 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.158 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.158 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.158 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.159 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.160 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.161 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.161 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.161 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.162 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.162 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.162 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.163 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.163 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.164 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.164 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.164 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.164 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.167 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.167 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.167 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.167 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.168 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.169 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.169 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.169 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.170 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.170 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.170 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.171 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.171 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.171 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.172 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.172 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.172 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.173 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.173 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.173 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.174 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.174 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.174 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.174 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.175 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.175 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.175 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.176 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stats", 2023-09-10T01:18:56.177 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.177 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.177 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.177 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.178 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.178 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.178 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.179 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.179 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.179 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.180 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.181 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.182 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.182 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.182 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.183 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.184 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.185 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.185 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.185 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.186 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.188 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.188 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.188 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.189 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.190 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.190 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.190 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.191 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.192 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.192 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.192 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.193 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.193 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.193 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.194 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.194 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.194 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.195 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.197 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "status", 2023-09-10T01:18:56.198 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.198 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.198 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.199 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.200 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.200 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.200 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.200 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.201 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.202 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.202 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.202 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.202 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.203 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.203 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.203 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.204 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.205 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.205 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.206 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.206 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.207 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.207 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.207 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.208 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.208 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.209 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.209 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.210 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.211 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.211 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.212 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.212 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.212 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.213 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.213 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.214 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.214 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.215 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.215 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.216 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.216 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.216 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.217 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.217 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.217 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.218 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.219 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.219 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.219 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.220 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.221 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.222 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.223 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telegraf", 2023-09-10T01:18:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.224 INFO:teuthology.orchestra.run.smithi003.stdout: "address": { 2023-09-10T01:18:56.225 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "address", 2023-09-10T01:18:56.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.225 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.225 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.226 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-09-10T01:18:56.226 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.226 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.227 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.227 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.227 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.227 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.228 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.228 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.229 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.230 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15", 2023-09-10T01:18:56.230 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.230 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.231 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.232 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.232 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.233 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.233 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.234 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.235 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.235 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.235 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.236 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.237 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.238 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.238 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.239 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.239 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.239 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.240 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.240 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.240 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.241 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.242 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.243 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.244 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.244 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.244 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.244 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.245 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.245 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.245 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.246 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.246 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.246 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.247 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.248 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.249 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.249 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.249 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.250 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.251 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.251 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.251 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.252 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.253 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telemetry", 2023-09-10T01:18:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.254 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_basic": { 2023-09-10T01:18:56.255 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_basic", 2023-09-10T01:18:56.255 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.255 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.256 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.256 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:56.256 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.256 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share basic cluster information (size, version)", 2023-09-10T01:18:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.258 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.258 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_crash": { 2023-09-10T01:18:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_crash", 2023-09-10T01:18:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.259 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.260 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:56.260 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.260 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.261 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.261 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-09-10T01:18:56.261 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.261 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.262 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.262 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_device": { 2023-09-10T01:18:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_device", 2023-09-10T01:18:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.263 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.264 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2023-09-10T01:18:56.264 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.264 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.264 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.265 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-09-10T01:18:56.265 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.265 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.266 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.266 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_ident": { 2023-09-10T01:18:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_ident", 2023-09-10T01:18:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.267 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.268 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.268 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.268 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-09-10T01:18:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.269 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.270 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.270 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_perf": { 2023-09-10T01:18:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_perf", 2023-09-10T01:18:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.271 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.272 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.273 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share various performance metrics of a cluster", 2023-09-10T01:18:56.273 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.273 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.273 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.274 INFO:teuthology.orchestra.run.smithi003.stdout: "contact": { 2023-09-10T01:18:56.274 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "contact", 2023-09-10T01:18:56.275 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.275 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.275 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.276 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.277 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.277 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.277 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.278 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.278 INFO:teuthology.orchestra.run.smithi003.stdout: "description": { 2023-09-10T01:18:56.279 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "description", 2023-09-10T01:18:56.279 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.279 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.280 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.280 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.280 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.280 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.281 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.282 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.282 INFO:teuthology.orchestra.run.smithi003.stdout: "device_url": { 2023-09-10T01:18:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_url", 2023-09-10T01:18:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.283 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.284 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-09-10T01:18:56.284 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.284 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.284 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.285 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.285 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.285 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.286 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.286 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2023-09-10T01:18:56.286 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2023-09-10T01:18:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.287 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.288 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.288 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.288 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.289 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.290 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:56.290 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:56.290 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:56.291 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.291 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.291 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "24", 2023-09-10T01:18:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "8", 2023-09-10T01:18:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.292 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: "last_opt_revision": { 2023-09-10T01:18:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "last_opt_revision", 2023-09-10T01:18:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:56.294 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.295 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.295 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1", 2023-09-10T01:18:56.295 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.296 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.297 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.297 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.298 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard": { 2023-09-10T01:18:56.298 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard", 2023-09-10T01:18:56.299 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.299 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.300 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.300 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.301 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.301 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.302 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.302 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.303 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.303 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.303 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.304 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard_description": { 2023-09-10T01:18:56.305 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard_description", 2023-09-10T01:18:56.305 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.306 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.307 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.308 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.308 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.308 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.308 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.309 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.310 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.310 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.311 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.311 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.312 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.312 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.313 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.313 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.314 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.314 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.314 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.315 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.315 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.316 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.316 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.317 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.317 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.318 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.318 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.319 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.319 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.319 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.320 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.320 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.320 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.321 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.321 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.321 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.322 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.322 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.323 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.324 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.324 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.324 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.325 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.326 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.326 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.326 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.326 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.327 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.328 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.328 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.330 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.331 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.332 INFO:teuthology.orchestra.run.smithi003.stdout: "organization": { 2023-09-10T01:18:56.332 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "organization", 2023-09-10T01:18:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.333 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.334 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.334 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.334 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.335 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.335 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.336 INFO:teuthology.orchestra.run.smithi003.stdout: "proxy": { 2023-09-10T01:18:56.337 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "proxy", 2023-09-10T01:18:56.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.337 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.338 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.338 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.338 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.339 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.339 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.339 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.340 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.340 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.340 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.341 INFO:teuthology.orchestra.run.smithi003.stdout: "url": { 2023-09-10T01:18:56.341 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url", 2023-09-10T01:18:56.342 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.342 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.342 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.343 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-09-10T01:18:56.343 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.343 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.344 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.344 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.344 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.345 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.345 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.345 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.346 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "test_orchestrator", 2023-09-10T01:18:56.347 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.347 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.347 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.348 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.348 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.348 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.348 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.349 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.349 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.349 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.349 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.350 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.350 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.350 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.351 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.351 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.351 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.352 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.352 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.352 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.352 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.353 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.353 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.353 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.354 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.354 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.355 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.355 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.355 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.355 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.356 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.356 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.356 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.357 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.357 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.357 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.358 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.358 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.358 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.358 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.359 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.359 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.359 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.360 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.360 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.360 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.360 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.361 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.361 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.361 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.361 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.362 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.362 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.363 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.363 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.363 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.364 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.364 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.364 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.365 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.365 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.365 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.365 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.366 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.366 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.366 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.368 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2023-09-10T01:18:56.368 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.368 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.368 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.369 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.369 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.369 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.370 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.370 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.370 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.370 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.371 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.371 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.372 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.372 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.373 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.373 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.374 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.374 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.375 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.375 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.376 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.376 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.377 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.377 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.378 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.378 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.379 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.379 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.380 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.380 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.381 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.381 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.382 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.382 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.383 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.383 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.383 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.384 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.384 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.384 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.385 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.385 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.385 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.385 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.386 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.386 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.386 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.387 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.387 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.387 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.387 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.388 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.388 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.388 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.389 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.389 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.389 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.390 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.390 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.390 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.391 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.391 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.391 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.391 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.392 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.392 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_clones": { 2023-09-10T01:18:56.393 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_clones", 2023-09-10T01:18:56.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:56.393 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.394 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.394 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4", 2023-09-10T01:18:56.394 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.394 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.395 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.395 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Number of asynchronous cloner threads", 2023-09-10T01:18:56.395 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.396 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.396 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.396 INFO:teuthology.orchestra.run.smithi003.stdout: "snapshot_clone_delay": { 2023-09-10T01:18:56.397 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snapshot_clone_delay", 2023-09-10T01:18:56.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:56.397 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.397 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.398 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2023-09-10T01:18:56.398 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.398 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.399 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.399 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-09-10T01:18:56.399 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.399 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.400 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.400 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.400 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.401 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix", 2023-09-10T01:18:56.401 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2023-09-10T01:18:56.401 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2023-09-10T01:18:56.402 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2023-09-10T01:18:56.402 INFO:teuthology.orchestra.run.smithi003.stdout: "discovery_interval": { 2023-09-10T01:18:56.402 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "discovery_interval", 2023-09-10T01:18:56.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2023-09-10T01:18:56.403 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.403 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "100", 2023-09-10T01:18:56.404 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.404 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.404 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.404 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.405 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.405 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.405 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.406 INFO:teuthology.orchestra.run.smithi003.stdout: "identifier": { 2023-09-10T01:18:56.406 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "identifier", 2023-09-10T01:18:56.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.407 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.407 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.407 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.408 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.408 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.408 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.409 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.409 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.409 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.410 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2023-09-10T01:18:56.410 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2023-09-10T01:18:56.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2023-09-10T01:18:56.411 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.411 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2023-09-10T01:18:56.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.412 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.412 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.412 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.413 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.413 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.413 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.414 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2023-09-10T01:18:56.414 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2023-09-10T01:18:56.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.415 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.415 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.415 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.415 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.416 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.416 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.416 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.417 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.417 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.417 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.417 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.418 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.418 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.418 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.418 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.419 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.419 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.420 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2023-09-10T01:18:56.420 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2023-09-10T01:18:56.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.420 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.421 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.421 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.422 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.422 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.422 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.423 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.423 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.424 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2023-09-10T01:18:56.424 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2023-09-10T01:18:56.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.424 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.425 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.425 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2023-09-10T01:18:56.425 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.426 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.426 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2023-09-10T01:18:56.426 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2023-09-10T01:18:56.426 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2023-09-10T01:18:56.427 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2023-09-10T01:18:56.427 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2023-09-10T01:18:56.427 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2023-09-10T01:18:56.427 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2023-09-10T01:18:56.428 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.428 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.428 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.428 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2023-09-10T01:18:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2023-09-10T01:18:56.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2023-09-10T01:18:56.430 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.430 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2023-09-10T01:18:56.431 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2023-09-10T01:18:56.431 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.431 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.431 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.432 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.432 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.432 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.432 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.433 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_host": { 2023-09-10T01:18:56.434 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_host", 2023-09-10T01:18:56.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.434 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.435 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2023-09-10T01:18:56.435 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.435 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.436 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.436 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.436 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.437 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.437 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_port": { 2023-09-10T01:18:56.437 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_port", 2023-09-10T01:18:56.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2023-09-10T01:18:56.438 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.438 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.439 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10051", 2023-09-10T01:18:56.439 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.439 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.439 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.440 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.440 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.440 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.440 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.441 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_sender": { 2023-09-10T01:18:56.441 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_sender", 2023-09-10T01:18:56.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2023-09-10T01:18:56.442 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2023-09-10T01:18:56.442 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2023-09-10T01:18:56.442 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-09-10T01:18:56.443 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2023-09-10T01:18:56.443 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2023-09-10T01:18:56.443 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2023-09-10T01:18:56.444 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2023-09-10T01:18:56.444 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2023-09-10T01:18:56.444 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2023-09-10T01:18:56.444 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2023-09-10T01:18:56.445 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.445 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.445 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.445 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.446 INFO:teuthology.orchestra.run.smithi003.stdout: "services": { 2023-09-10T01:18:56.446 INFO:teuthology.orchestra.run.smithi003.stdout: "dashboard": "https://172.21.15.3:8443/" 2023-09-10T01:18:56.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.447 INFO:teuthology.orchestra.run.smithi003.stdout: "always_on_modules": { 2023-09-10T01:18:56.447 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus": [ 2023-09-10T01:18:56.447 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2023-09-10T01:18:56.447 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2023-09-10T01:18:56.448 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2023-09-10T01:18:56.448 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2023-09-10T01:18:56.448 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2023-09-10T01:18:56.448 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2023-09-10T01:18:56.449 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2023-09-10T01:18:56.449 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2023-09-10T01:18:56.449 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2023-09-10T01:18:56.449 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2023-09-10T01:18:56.450 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.450 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific": [ 2023-09-10T01:18:56.450 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2023-09-10T01:18:56.451 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2023-09-10T01:18:56.451 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2023-09-10T01:18:56.451 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2023-09-10T01:18:56.451 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2023-09-10T01:18:56.452 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2023-09-10T01:18:56.452 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2023-09-10T01:18:56.452 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2023-09-10T01:18:56.453 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2023-09-10T01:18:56.453 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2023-09-10T01:18:56.453 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.454 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy": [ 2023-09-10T01:18:56.454 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2023-09-10T01:18:56.454 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2023-09-10T01:18:56.454 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2023-09-10T01:18:56.455 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2023-09-10T01:18:56.455 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2023-09-10T01:18:56.455 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2023-09-10T01:18:56.455 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2023-09-10T01:18:56.456 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2023-09-10T01:18:56.456 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2023-09-10T01:18:56.456 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2023-09-10T01:18:56.457 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2023-09-10T01:18:56.457 INFO:teuthology.orchestra.run.smithi003.stdout: "reef": [ 2023-09-10T01:18:56.457 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2023-09-10T01:18:56.457 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2023-09-10T01:18:56.458 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2023-09-10T01:18:56.458 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2023-09-10T01:18:56.458 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2023-09-10T01:18:56.459 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2023-09-10T01:18:56.459 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2023-09-10T01:18:56.459 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2023-09-10T01:18:56.459 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2023-09-10T01:18:56.460 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2023-09-10T01:18:56.460 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.460 INFO:teuthology.orchestra.run.smithi003.stdout: "last_failure_osd_epoch": 91, 2023-09-10T01:18:56.461 INFO:teuthology.orchestra.run.smithi003.stdout: "active_clients": [ 2023-09-10T01:18:56.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.461 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2023-09-10T01:18:56.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2023-09-10T01:18:56.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2023-09-10T01:18:56.462 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:18:56.463 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3317768911 2023-09-10T01:18:56.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.464 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "libcephsqlite", 2023-09-10T01:18:56.464 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2023-09-10T01:18:56.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2023-09-10T01:18:56.465 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:18:56.465 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2972125950 2023-09-10T01:18:56.466 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.466 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.467 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2023-09-10T01:18:56.467 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2023-09-10T01:18:56.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2023-09-10T01:18:56.468 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:18:56.468 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2621396580 2023-09-10T01:18:56.468 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.469 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.469 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2023-09-10T01:18:56.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2023-09-10T01:18:56.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2023-09-10T01:18:56.470 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:18:56.471 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1966345563 2023-09-10T01:18:56.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.471 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.472 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2023-09-10T01:18:56.472 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2023-09-10T01:18:56.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2023-09-10T01:18:56.473 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:18:56.473 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3772733119 2023-09-10T01:18:56.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.474 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:56.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.475 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2023-09-10T01:18:56.475 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2023-09-10T01:18:56.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2023-09-10T01:18:56.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2023-09-10T01:18:56.476 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2023-09-10T01:18:56.476 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3825715875 2023-09-10T01:18:56.476 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.476 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.477 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:56.477 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2023-09-10T01:18:56.477 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:18:56.479 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:51 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1773168980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:18:56.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:51 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3820539935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:18:56.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:52 smithi136 ceph-mon[133708]: pgmap v492: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-09-10T01:18:56.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:52 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/4132580390' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-09-10T01:18:56.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:53 smithi136 ceph-mon[133708]: pgmap v493: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-09-10T01:18:56.481 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:51 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1773168980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:18:56.481 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:51 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3820539935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:18:56.481 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:52 smithi092 ceph-mon[134067]: pgmap v492: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-09-10T01:18:56.482 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:52 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/4132580390' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-09-10T01:18:56.482 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:53 smithi092 ceph-mon[134067]: pgmap v493: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-09-10T01:18:56.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f126c0a07b0 msgr2=0x7f126c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f126c0a07b0 0x7f126c0a2c70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1270005fd0 tx=0x7f1270009310 comp rx=0 tx=0).stop 2023-09-10T01:18:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 msgr2=0x7f128019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:18:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 0x7f128019ed90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f126800e9a0 tx=0x7f126800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:18:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 shutdown_connections 2023-09-10T01:18:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f126c0a07b0 0x7f126c0a2c70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1280072a30 0x7f128019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:56.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f128010ce60 0x7f12801988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:56.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 --2- 172.21.15.3:0/4132580390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1280073350 0x7f128019ed90 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:18:56.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.600+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 >> 172.21.15.3:0/4132580390 conn(0x7f12800fc110 msgr2=0x7f128010efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:18:56.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.601+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 shutdown_connections 2023-09-10T01:18:56.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:18:51.601+0000 7f12855ff700 1 -- 172.21.15.3:0/4132580390 wait complete. 2023-09-10T01:18:56.492 DEBUG:tasks.check_counter:Getting stats from a 2023-09-10T01:18:56.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.a perf dump 2023-09-10T01:18:56.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:51 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1773168980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:18:56.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:51 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3820539935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:18:56.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:52 smithi003 ceph-mon[97654]: pgmap v492: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-09-10T01:18:56.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:52 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/4132580390' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-09-10T01:18:56.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:53 smithi003 ceph-mon[97654]: pgmap v493: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-09-10T01:18:56.566 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:56 smithi003 ceph-mon[97654]: pgmap v494: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 13 op/s 2023-09-10T01:18:56.566 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:56 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:56 smithi136 ceph-mon[133708]: pgmap v494: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 13 op/s 2023-09-10T01:18:56.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:56 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:56.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:56 smithi092 ceph-mon[134067]: pgmap v494: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 13 op/s 2023-09-10T01:18:56.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:56 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:18:57.235 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:18:58.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:18:58 smithi003 ceph-mon[97654]: pgmap v495: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-09-10T01:18:58.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:18:58 smithi136 ceph-mon[133708]: pgmap v495: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-09-10T01:18:58.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:18:58 smithi092 ceph-mon[134067]: pgmap v495: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-09-10T01:18:59.018 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:18:59.018 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:18:59.018 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 0, 2023-09-10T01:18:59.019 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 2, 2023-09-10T01:18:59.019 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 0, 2023-09-10T01:18:59.019 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 0, 2023-09-10T01:18:59.019 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 4, 2023-09-10T01:18:59.020 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 0, 2023-09-10T01:18:59.020 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.003796249, 2023-09-10T01:18:59.020 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.000000000, 2023-09-10T01:18:59.021 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.000681258, 2023-09-10T01:18:59.021 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:18:59.021 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:18:59.021 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:18:59.022 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:18:59.022 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:18:59.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.023 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:18:59.023 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:18:59.023 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:18:59.023 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:18:59.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.024 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 0, 2023-09-10T01:18:59.024 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 0 2023-09-10T01:18:59.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.025 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:18:59.025 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 307, 2023-09-10T01:18:59.025 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 295, 2023-09-10T01:18:59.025 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 1712048, 2023-09-10T01:18:59.026 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 177248, 2023-09-10T01:18:59.026 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 2, 2023-09-10T01:18:59.026 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 1, 2023-09-10T01:18:59.027 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.060772025, 2023-09-10T01:18:59.027 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.022142573, 2023-09-10T01:18:59.027 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.042500149, 2023-09-10T01:18:59.027 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.006454606, 2023-09-10T01:18:59.028 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:18:59.028 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 294, 2023-09-10T01:18:59.028 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.032771520, 2023-09-10T01:18:59.028 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000111467 2023-09-10T01:18:59.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.029 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:18:59.029 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:18:59.030 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:18:59.030 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:18:59.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.030 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1712048, 2023-09-10T01:18:59.031 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 177248 2023-09-10T01:18:59.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.031 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:18:59.031 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 7, 2023-09-10T01:18:59.032 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 228, 2023-09-10T01:18:59.032 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 3424, 2023-09-10T01:18:59.032 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 269328, 2023-09-10T01:18:59.032 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 6, 2023-09-10T01:18:59.033 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2023-09-10T01:18:59.033 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.022847862, 2023-09-10T01:18:59.033 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.014630463, 2023-09-10T01:18:59.034 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.004524316, 2023-09-10T01:18:59.034 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:18:59.034 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:18:59.034 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 226, 2023-09-10T01:18:59.035 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.023750851, 2023-09-10T01:18:59.035 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000105092 2023-09-10T01:18:59.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.035 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:18:59.036 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:18:59.036 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:18:59.036 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:18:59.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.037 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-09-10T01:18:59.037 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 269328 2023-09-10T01:18:59.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.038 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2023-09-10T01:18:59.038 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 0, 2023-09-10T01:18:59.038 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2023-09-10T01:18:59.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.039 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2023-09-10T01:18:59.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:18:59.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2023-09-10T01:18:59.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:18:59.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:18:59.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:18:59.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:18:59.041 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:18:59.041 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:18:59.041 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:18:59.041 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:18:59.042 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:18:59.042 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:18:59.042 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:18:59.042 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:18:59.043 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:18:59.043 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2023-09-10T01:18:59.043 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:18:59.044 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2023-09-10T01:18:59.044 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:18:59.044 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:18:59.044 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:18:59.045 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:18:59.045 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:18:59.045 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:18:59.045 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:18:59.046 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2023-09-10T01:18:59.046 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:18:59.046 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:18:59.047 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:18:59.047 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2023-09-10T01:18:59.047 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2023-09-10T01:18:59.047 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2023-09-10T01:18:59.048 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:18:59.048 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:18:59.048 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:18:59.048 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:18:59.049 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 150776, 2023-09-10T01:18:59.049 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 32, 2023-09-10T01:18:59.049 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:18:59.050 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2023-09-10T01:18:59.050 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2023-09-10T01:18:59.050 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2023-09-10T01:18:59.051 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:18:59.051 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2023-09-10T01:18:59.051 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:18:59.051 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2023-09-10T01:18:59.052 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 0, 2023-09-10T01:18:59.052 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 0, 2023-09-10T01:18:59.052 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:18:59.052 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:18:59.053 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2023-09-10T01:18:59.053 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2023-09-10T01:18:59.053 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 928, 2023-09-10T01:18:59.054 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 2, 2023-09-10T01:18:59.054 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2023-09-10T01:18:59.054 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2023-09-10T01:18:59.054 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2023-09-10T01:18:59.055 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2023-09-10T01:18:59.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:18:59.055 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:18:59.055 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2023-09-10T01:18:59.056 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2023-09-10T01:18:59.056 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2023-09-10T01:18:59.056 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 310, 2023-09-10T01:18:59.057 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 1668771, 2023-09-10T01:18:59.057 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2023-09-10T01:18:59.057 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 310, 2023-09-10T01:18:59.057 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2023-09-10T01:18:59.058 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2023-09-10T01:18:59.058 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 310, 2023-09-10T01:18:59.058 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 1668771, 2023-09-10T01:18:59.058 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2023-09-10T01:18:59.059 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:18:59.059 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:18:59.059 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:18:59.060 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:59.060 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:18:59.060 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:18:59.633 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-09-10T01:18:59.633 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-09-10T01:18:59.634 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2023-09-10T01:18:59.634 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2023-09-10T01:18:59.635 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2023-09-10T01:18:59.635 DEBUG:tasks.check_counter:Getting stats from d 2023-09-10T01:18:59.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.d perf dump 2023-09-10T01:19:00.372 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:19:00.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:00 smithi136 ceph-mon[133708]: pgmap v496: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-09-10T01:19:00.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:00 smithi003 ceph-mon[97654]: pgmap v496: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-09-10T01:19:00.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:00 smithi092 ceph-mon[134067]: pgmap v496: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-09-10T01:19:02.143 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:19:02.144 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:02.144 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 0, 2023-09-10T01:19:02.144 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 2, 2023-09-10T01:19:02.145 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 0, 2023-09-10T01:19:02.145 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 0, 2023-09-10T01:19:02.145 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 4, 2023-09-10T01:19:02.146 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 0, 2023-09-10T01:19:02.146 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.003453968, 2023-09-10T01:19:02.146 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.000000000, 2023-09-10T01:19:02.146 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.000646638, 2023-09-10T01:19:02.147 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:02.147 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:02.147 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:02.148 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:02.148 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:02.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.148 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 0, 2023-09-10T01:19:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 0 2023-09-10T01:19:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.151 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:02.151 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 316, 2023-09-10T01:19:02.151 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 302, 2023-09-10T01:19:02.151 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 1716352, 2023-09-10T01:19:02.152 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 180816, 2023-09-10T01:19:02.152 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 2, 2023-09-10T01:19:02.152 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 1, 2023-09-10T01:19:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.063444137, 2023-09-10T01:19:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.023334375, 2023-09-10T01:19:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.044565958, 2023-09-10T01:19:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.006461047, 2023-09-10T01:19:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 302, 2023-09-10T01:19:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.034104132, 2023-09-10T01:19:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000112927 2023-09-10T01:19:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1716352, 2023-09-10T01:19:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 180816 2023-09-10T01:19:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 3, 2023-09-10T01:19:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 229, 2023-09-10T01:19:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 480, 2023-09-10T01:19:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 269408, 2023-09-10T01:19:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 6, 2023-09-10T01:19:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.023291649, 2023-09-10T01:19:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.015553407, 2023-09-10T01:19:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.003587183, 2023-09-10T01:19:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 226, 2023-09-10T01:19:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.025943133, 2023-09-10T01:19:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000114792 2023-09-10T01:19:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 480, 2023-09-10T01:19:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 269408 2023-09-10T01:19:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2023-09-10T01:19:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 0, 2023-09-10T01:19:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2023-09-10T01:19:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2023-09-10T01:19:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2023-09-10T01:19:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 148714, 2023-09-10T01:19:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 30, 2023-09-10T01:19:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2023-09-10T01:19:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2023-09-10T01:19:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 0, 2023-09-10T01:19:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 0, 2023-09-10T01:19:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2023-09-10T01:19:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 928, 2023-09-10T01:19:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 2, 2023-09-10T01:19:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2023-09-10T01:19:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2023-09-10T01:19:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2023-09-10T01:19:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2023-09-10T01:19:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2023-09-10T01:19:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 315, 2023-09-10T01:19:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 1669421, 2023-09-10T01:19:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 315, 2023-09-10T01:19:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2023-09-10T01:19:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2023-09-10T01:19:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 315, 2023-09-10T01:19:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 1669421, 2023-09-10T01:19:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2023-09-10T01:19:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:19:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:19:02.186 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:19:02.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:02 smithi003 ceph-mon[97654]: pgmap v497: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-09-10T01:19:02.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:02 smithi136 ceph-mon[133708]: pgmap v497: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-09-10T01:19:02.663 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-09-10T01:19:02.664 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-09-10T01:19:02.664 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2023-09-10T01:19:02.664 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2023-09-10T01:19:02.665 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2023-09-10T01:19:02.665 DEBUG:tasks.check_counter:Getting stats from g 2023-09-10T01:19:02.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.g perf dump 2023-09-10T01:19:02.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:02 smithi092 ceph-mon[134067]: pgmap v497: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-09-10T01:19:03.404 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:19:04.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:04 smithi003 ceph-mon[97654]: pgmap v498: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-09-10T01:19:04.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:04 smithi136 ceph-mon[133708]: pgmap v498: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-09-10T01:19:04.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:04 smithi092 ceph-mon[134067]: pgmap v498: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-09-10T01:19:04.862 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:19:04.863 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:04.863 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 0, 2023-09-10T01:19:04.864 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 2, 2023-09-10T01:19:04.864 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 0, 2023-09-10T01:19:04.864 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 0, 2023-09-10T01:19:04.864 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 4, 2023-09-10T01:19:04.865 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 0, 2023-09-10T01:19:04.865 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.003608048, 2023-09-10T01:19:04.865 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.000000000, 2023-09-10T01:19:04.865 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.000720545, 2023-09-10T01:19:04.866 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:04.866 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:04.866 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:04.867 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:04.867 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:04.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.867 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:04.868 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:04.868 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:04.868 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:04.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.869 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 0, 2023-09-10T01:19:04.869 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 0 2023-09-10T01:19:04.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.870 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:04.870 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 3, 2023-09-10T01:19:04.870 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 227, 2023-09-10T01:19:04.871 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 480, 2023-09-10T01:19:04.871 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 269056, 2023-09-10T01:19:04.871 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 6, 2023-09-10T01:19:04.871 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:04.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.022549251, 2023-09-10T01:19:04.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.015423976, 2023-09-10T01:19:04.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.003316285, 2023-09-10T01:19:04.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:04.873 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:04.873 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 224, 2023-09-10T01:19:04.873 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.033257255, 2023-09-10T01:19:04.874 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000148469 2023-09-10T01:19:04.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.874 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:04.874 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:04.875 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:04.875 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:04.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.876 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 480, 2023-09-10T01:19:04.876 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 269056 2023-09-10T01:19:04.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.876 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:04.877 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 313, 2023-09-10T01:19:04.877 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 299, 2023-09-10T01:19:04.877 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 1715536, 2023-09-10T01:19:04.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 178944, 2023-09-10T01:19:04.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 2, 2023-09-10T01:19:04.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 1, 2023-09-10T01:19:04.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.064239251, 2023-09-10T01:19:04.879 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.023121526, 2023-09-10T01:19:04.879 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.046553311, 2023-09-10T01:19:04.879 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.006439445, 2023-09-10T01:19:04.880 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:04.880 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 299, 2023-09-10T01:19:04.880 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.041777804, 2023-09-10T01:19:04.880 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000139725 2023-09-10T01:19:04.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.881 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:04.881 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:04.882 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:04.882 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:04.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.883 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1715536, 2023-09-10T01:19:04.883 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 178944 2023-09-10T01:19:04.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.883 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2023-09-10T01:19:04.884 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 0, 2023-09-10T01:19:04.884 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2023-09-10T01:19:04.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.884 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2023-09-10T01:19:04.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:04.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:04.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:04.886 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:04.886 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:04.886 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:04.886 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:04.887 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:04.887 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:04.887 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:04.888 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:04.888 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:04.888 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:04.888 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:04.889 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:04.889 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:04.889 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:04.890 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:04.890 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:04.890 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:04.890 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:04.891 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:04.891 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:04.891 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:04.891 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:04.892 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:04.892 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:04.892 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:04.893 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:04.893 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:04.893 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:04.893 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2023-09-10T01:19:04.894 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:04.894 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:04.894 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:04.894 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:04.895 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 148714, 2023-09-10T01:19:04.895 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 30, 2023-09-10T01:19:04.895 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:04.895 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:04.896 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2023-09-10T01:19:04.896 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2023-09-10T01:19:04.896 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:04.897 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:04.897 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:04.897 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:04.897 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 0, 2023-09-10T01:19:04.898 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 0, 2023-09-10T01:19:04.898 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:04.898 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:04.898 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:04.899 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2023-09-10T01:19:04.899 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 928, 2023-09-10T01:19:04.899 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 2, 2023-09-10T01:19:04.900 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:04.900 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2023-09-10T01:19:04.900 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:04.900 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2023-09-10T01:19:04.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:04.901 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:04.901 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2023-09-10T01:19:04.902 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2023-09-10T01:19:04.902 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2023-09-10T01:19:04.902 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 312, 2023-09-10T01:19:04.903 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 1669031, 2023-09-10T01:19:04.903 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:04.903 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 312, 2023-09-10T01:19:04.903 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2023-09-10T01:19:04.904 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2023-09-10T01:19:04.904 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 312, 2023-09-10T01:19:04.904 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 1669031, 2023-09-10T01:19:04.904 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2023-09-10T01:19:04.905 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:04.905 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:04.905 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:04.906 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:19:04.906 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:19:04.906 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:19:05.401 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2023-09-10T01:19:05.401 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2023-09-10T01:19:05.402 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2023-09-10T01:19:05.402 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2023-09-10T01:19:05.402 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2023-09-10T01:19:05.403 DEBUG:tasks.check_counter:Getting stats from j 2023-09-10T01:19:05.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.j perf dump 2023-09-10T01:19:06.137 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.a/config 2023-09-10T01:19:06.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:06 smithi136 ceph-mon[133708]: pgmap v499: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 82 KiB/s wr, 26 op/s 2023-09-10T01:19:06.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:06 smithi092 ceph-mon[134067]: pgmap v499: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 82 KiB/s wr, 26 op/s 2023-09-10T01:19:06.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:06 smithi003 ceph-mon[97654]: pgmap v499: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 82 KiB/s wr, 26 op/s 2023-09-10T01:19:07.692 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2023-09-10T01:19:07.692 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:07.693 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 314, 2023-09-10T01:19:07.693 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 302, 2023-09-10T01:19:07.693 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 1713952, 2023-09-10T01:19:07.693 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 181616, 2023-09-10T01:19:07.694 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 4, 2023-09-10T01:19:07.694 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 0, 2023-09-10T01:19:07.694 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.067436857, 2023-09-10T01:19:07.695 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.023073826, 2023-09-10T01:19:07.695 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.046590897, 2023-09-10T01:19:07.695 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.006934707, 2023-09-10T01:19:07.695 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:07.696 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 301, 2023-09-10T01:19:07.696 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.039566248, 2023-09-10T01:19:07.696 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000131449 2023-09-10T01:19:07.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.697 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:07.697 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:07.697 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:07.698 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:07.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.698 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1713952, 2023-09-10T01:19:07.698 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 181616 2023-09-10T01:19:07.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.699 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:07.699 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 3, 2023-09-10T01:19:07.699 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 231, 2023-09-10T01:19:07.700 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 480, 2023-09-10T01:19:07.700 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 269760, 2023-09-10T01:19:07.700 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 6, 2023-09-10T01:19:07.701 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:07.701 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.024223272, 2023-09-10T01:19:07.701 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.016339952, 2023-09-10T01:19:07.701 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.003711566, 2023-09-10T01:19:07.702 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:07.702 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:07.702 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 228, 2023-09-10T01:19:07.702 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.031453440, 2023-09-10T01:19:07.703 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000137953 2023-09-10T01:19:07.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.703 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:07.704 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:07.704 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:07.704 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:07.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.705 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 480, 2023-09-10T01:19:07.705 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 269760 2023-09-10T01:19:07.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.705 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:07.706 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 4, 2023-09-10T01:19:07.706 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 4, 2023-09-10T01:19:07.706 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 2944, 2023-09-10T01:19:07.707 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 448, 2023-09-10T01:19:07.707 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 2, 2023-09-10T01:19:07.707 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 1, 2023-09-10T01:19:07.707 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.001308141, 2023-09-10T01:19:07.708 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.000077905, 2023-09-10T01:19:07.708 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.001183607, 2023-09-10T01:19:07.708 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:07.708 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:07.709 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 3, 2023-09-10T01:19:07.709 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.001311771, 2023-09-10T01:19:07.709 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000437257 2023-09-10T01:19:07.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.710 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:07.710 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:07.710 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:07.711 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:07.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.711 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-09-10T01:19:07.712 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 448 2023-09-10T01:19:07.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.712 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2023-09-10T01:19:07.712 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 0, 2023-09-10T01:19:07.713 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2023-09-10T01:19:07.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.713 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2023-09-10T01:19:07.713 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:07.714 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:07.714 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:07.714 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:07.715 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:07.715 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:07.715 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:07.715 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:07.716 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:07.716 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:07.716 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:07.716 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:07.717 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:07.717 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:07.717 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:07.718 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:07.718 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:07.718 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:07.718 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:07.719 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:07.719 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:07.719 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:07.719 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:07.720 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:07.720 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:07.720 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:07.720 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:07.721 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:07.721 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:07.721 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:07.722 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:07.722 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2023-09-10T01:19:07.722 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:07.723 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:07.723 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:07.723 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:07.724 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 149762, 2023-09-10T01:19:07.724 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 32, 2023-09-10T01:19:07.724 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:07.724 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:07.725 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2023-09-10T01:19:07.725 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2023-09-10T01:19:07.725 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:07.725 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:07.726 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:07.726 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:07.726 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 0, 2023-09-10T01:19:07.727 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 0, 2023-09-10T01:19:07.727 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:07.727 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:07.727 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:07.728 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2023-09-10T01:19:07.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 928, 2023-09-10T01:19:07.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 2, 2023-09-10T01:19:07.728 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:07.729 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2023-09-10T01:19:07.729 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:07.729 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2023-09-10T01:19:07.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2023-09-10T01:19:07.730 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:07.730 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2023-09-10T01:19:07.730 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2023-09-10T01:19:07.731 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2023-09-10T01:19:07.731 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 317, 2023-09-10T01:19:07.731 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 1669681, 2023-09-10T01:19:07.732 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:07.732 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 317, 2023-09-10T01:19:07.732 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2023-09-10T01:19:07.732 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2023-09-10T01:19:07.733 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 317, 2023-09-10T01:19:07.733 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 1669681, 2023-09-10T01:19:07.733 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2023-09-10T01:19:07.733 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2023-09-10T01:19:07.734 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2023-09-10T01:19:07.734 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2023-09-10T01:19:07.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:19:07.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2023-09-10T01:19:07.735 INFO:teuthology.orchestra.run.smithi003.stdout:} 2023-09-10T01:19:08.137 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-09-10T01:19:08.137 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-09-10T01:19:08.138 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2023-09-10T01:19:08.138 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2023-09-10T01:19:08.138 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2023-09-10T01:19:08.139 DEBUG:tasks.check_counter:Getting stats from b 2023-09-10T01:19:08.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.b perf dump 2023-09-10T01:19:08.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:08 smithi136 ceph-mon[133708]: pgmap v500: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 71 KiB/s wr, 22 op/s 2023-09-10T01:19:08.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:08 smithi092 ceph-mon[134067]: pgmap v500: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 71 KiB/s wr, 22 op/s 2023-09-10T01:19:08.791 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T01:19:08.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:08 smithi003 ceph-mon[97654]: pgmap v500: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 71 KiB/s wr, 22 op/s 2023-09-10T01:19:10.398 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:10 smithi092 ceph-mon[134067]: pgmap v501: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-09-10T01:19:10.398 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:10 smithi092 ceph-mon[134067]: osdmap e112: 12 total, 12 up, 12 in 2023-09-10T01:19:10.414 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2023-09-10T01:19:10.415 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:10.415 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 313270, 2023-09-10T01:19:10.415 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 434511, 2023-09-10T01:19:10.415 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 98237710, 2023-09-10T01:19:10.416 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 505513060, 2023-09-10T01:19:10.416 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 311, 2023-09-10T01:19:10.416 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:10.416 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 15.390550823, 2023-09-10T01:19:10.417 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 6.923590142, 2023-09-10T01:19:10.417 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 8.997154058, 2023-09-10T01:19:10.417 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 1.574248759, 2023-09-10T01:19:10.417 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:10.418 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 434508, 2023-09-10T01:19:10.418 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 67.972794012, 2023-09-10T01:19:10.418 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000156436 2023-09-10T01:19:10.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.419 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:10.419 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 319294, 2023-09-10T01:19:10.419 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.079085747, 2023-09-10T01:19:10.420 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000247 2023-09-10T01:19:10.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.420 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 1965120, 2023-09-10T01:19:10.420 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 252208 2023-09-10T01:19:10.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.421 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:10.421 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 185941, 2023-09-10T01:19:10.422 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 199380, 2023-09-10T01:19:10.422 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 47669994, 2023-09-10T01:19:10.422 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 270641337, 2023-09-10T01:19:10.422 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 12, 2023-09-10T01:19:10.423 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 5, 2023-09-10T01:19:10.423 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 9.733704060, 2023-09-10T01:19:10.423 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 3.529327617, 2023-09-10T01:19:10.423 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 5.406155082, 2023-09-10T01:19:10.424 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 1.487352331, 2023-09-10T01:19:10.424 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:10.424 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 199379, 2023-09-10T01:19:10.425 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 43.977174973, 2023-09-10T01:19:10.425 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000220570 2023-09-10T01:19:10.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.425 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:10.426 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 285133, 2023-09-10T01:19:10.426 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.041081995, 2023-09-10T01:19:10.426 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000144 2023-09-10T01:19:10.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.427 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 42528, 2023-09-10T01:19:10.427 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 565776 2023-09-10T01:19:10.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.427 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:10.428 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 1535708, 2023-09-10T01:19:10.428 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 1563816, 2023-09-10T01:19:10.428 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 258361923, 2023-09-10T01:19:10.429 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 427178864, 2023-09-10T01:19:10.429 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 28, 2023-09-10T01:19:10.429 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 7, 2023-09-10T01:19:10.429 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 58.136109993, 2023-09-10T01:19:10.430 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 25.384692110, 2023-09-10T01:19:10.430 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 38.215296514, 2023-09-10T01:19:10.430 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 1.432730075, 2023-09-10T01:19:10.430 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:10.431 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1563814, 2023-09-10T01:19:10.431 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 56.774862314, 2023-09-10T01:19:10.431 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000036305 2023-09-10T01:19:10.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.432 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:10.432 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 3004293, 2023-09-10T01:19:10.433 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.314569659, 2023-09-10T01:19:10.433 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000104 2023-09-10T01:19:10.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.433 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 485824, 2023-09-10T01:19:10.434 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 157344 2023-09-10T01:19:10.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.434 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2023-09-10T01:19:10.434 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 1, 2023-09-10T01:19:10.435 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2023-09-10T01:19:10.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.435 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-MDSRank": { 2023-09-10T01:19:10.436 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2023-09-10T01:19:10.436 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2023-09-10T01:19:10.436 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 133998, 2023-09-10T01:19:10.436 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 92.955462714, 2023-09-10T01:19:10.437 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000693707 2023-09-10T01:19:10.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.437 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-PurgeQueue": { 2023-09-10T01:19:10.438 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2023-09-10T01:19:10.438 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2023-09-10T01:19:10.438 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 35999, 2023-09-10T01:19:10.439 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 6.269642995, 2023-09-10T01:19:10.439 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000174161 2023-09-10T01:19:10.439 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.440 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": { 2023-09-10T01:19:10.440 INFO:teuthology.orchestra.run.smithi092.stdout: "request": 164425, 2023-09-10T01:19:10.440 INFO:teuthology.orchestra.run.smithi092.stdout: "reply": 148298, 2023-09-10T01:19:10.440 INFO:teuthology.orchestra.run.smithi092.stdout: "reply_latency": { 2023-09-10T01:19:10.441 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 148298, 2023-09-10T01:19:10.441 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 730.034291558, 2023-09-10T01:19:10.441 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.004922752 2023-09-10T01:19:10.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.442 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_reply": 0, 2023-09-10T01:19:10.442 INFO:teuthology.orchestra.run.smithi092.stdout: "forward": 16127, 2023-09-10T01:19:10.442 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_complete": 30, 2023-09-10T01:19:10.443 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_keys": 4, 2023-09-10T01:19:10.443 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_commit": 2096, 2023-09-10T01:19:10.443 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_split": 6, 2023-09-10T01:19:10.443 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_merge": 11, 2023-09-10T01:19:10.444 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes": 1426, 2023-09-10T01:19:10.444 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_top": 341, 2023-09-10T01:19:10.444 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_bottom": 366, 2023-09-10T01:19:10.444 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pin_tail": 719, 2023-09-10T01:19:10.445 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pinned": 940, 2023-09-10T01:19:10.445 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_expired": 2346, 2023-09-10T01:19:10.445 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_with_caps": 218, 2023-09-10T01:19:10.446 INFO:teuthology.orchestra.run.smithi092.stdout: "caps": 218, 2023-09-10T01:19:10.446 INFO:teuthology.orchestra.run.smithi092.stdout: "subtrees": 16, 2023-09-10T01:19:10.446 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse": 232643, 2023-09-10T01:19:10.446 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_hit": 165653, 2023-09-10T01:19:10.447 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_forward": 16110, 2023-09-10T01:19:10.447 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_discover": 59, 2023-09-10T01:19:10.447 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_dir_fetch": 1, 2023-09-10T01:19:10.447 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_remote_ino": 0, 2023-09-10T01:19:10.448 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_lock": 1211, 2023-09-10T01:19:10.448 INFO:teuthology.orchestra.run.smithi092.stdout: "load_cent": 1285, 2023-09-10T01:19:10.448 INFO:teuthology.orchestra.run.smithi092.stdout: "q": 0, 2023-09-10T01:19:10.449 INFO:teuthology.orchestra.run.smithi092.stdout: "exported": 24, 2023-09-10T01:19:10.449 INFO:teuthology.orchestra.run.smithi092.stdout: "exported_inodes": 1729, 2023-09-10T01:19:10.449 INFO:teuthology.orchestra.run.smithi092.stdout: "imported": 17, 2023-09-10T01:19:10.449 INFO:teuthology.orchestra.run.smithi092.stdout: "imported_inodes": 465, 2023-09-10T01:19:10.450 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_dir_fetch": 0, 2023-09-10T01:19:10.450 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_backtrace_fetch": 2238, 2023-09-10T01:19:10.450 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_peer_discover": 0, 2023-09-10T01:19:10.451 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rfiles": 674, 2023-09-10T01:19:10.451 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rbytes": 217420166, 2023-09-10T01:19:10.451 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rsnaps": 6, 2023-09-10T01:19:10.451 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_fetch": 2238, 2023-09-10T01:19:10.452 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_set_tag": 0, 2023-09-10T01:19:10.452 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_repaired": 0, 2023-09-10T01:19:10.452 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, 2023-09-10T01:19:10.452 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 586, 2023-09-10T01:19:10.453 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 20, 2023-09-10T01:19:10.453 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dirfrag_rstats": 566, 2023-09-10T01:19:10.453 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_file_inodes": 1652, 2023-09-10T01:19:10.454 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_inode_file_caps": 12, 2023-09-10T01:19:10.454 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_revoke": 54648, 2023-09-10T01:19:10.454 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_grant": 88970, 2023-09-10T01:19:10.454 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_trunc": 0, 2023-09-10T01:19:10.455 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flushsnap_ack": 1210, 2023-09-10T01:19:10.455 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flush_ack": 0, 2023-09-10T01:19:10.455 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps": 87637, 2023-09-10T01:19:10.455 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps_dirty": 25167, 2023-09-10T01:19:10.456 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_cap_release": 4264, 2023-09-10T01:19:10.456 INFO:teuthology.orchestra.run.smithi092.stdout: "process_request_cap_release": 76911 2023-09-10T01:19:10.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.457 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache": { 2023-09-10T01:19:10.457 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update": 10, 2023-09-10T01:19:10.457 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update_receipt": 15, 2023-09-10T01:19:10.457 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_try_discover": 10, 2023-09-10T01:19:10.458 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_send_discover": 64, 2023-09-10T01:19:10.458 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_handle_discover": 196, 2023-09-10T01:19:10.458 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays": 74, 2023-09-10T01:19:10.458 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_delayed": 0, 2023-09-10T01:19:10.459 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_enqueuing": 0, 2023-09-10T01:19:10.459 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_created": 21613, 2023-09-10T01:19:10.459 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_enqueued": 21539, 2023-09-10T01:19:10.460 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_reintegrated": 0, 2023-09-10T01:19:10.460 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_migrated": 0, 2023-09-10T01:19:10.460 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_processing": 0, 2023-09-10T01:19:10.460 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_enqueued": 0, 2023-09-10T01:19:10.461 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_prioritized": 0, 2023-09-10T01:19:10.461 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_started": 0, 2023-09-10T01:19:10.461 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_completed": 0, 2023-09-10T01:19:10.462 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_enqueue_scrub": 20, 2023-09-10T01:19:10.462 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_exportdir": 24, 2023-09-10T01:19:10.462 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_flush": 0, 2023-09-10T01:19:10.462 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragmentdir": 17, 2023-09-10T01:19:10.463 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragstats": 0, 2023-09-10T01:19:10.463 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_inodestats": 0 2023-09-10T01:19:10.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.463 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log": { 2023-09-10T01:19:10.464 INFO:teuthology.orchestra.run.smithi092.stdout: "evlrg": 0, 2023-09-10T01:19:10.464 INFO:teuthology.orchestra.run.smithi092.stdout: "evadd": 135058, 2023-09-10T01:19:10.464 INFO:teuthology.orchestra.run.smithi092.stdout: "evex": 133508, 2023-09-10T01:19:10.465 INFO:teuthology.orchestra.run.smithi092.stdout: "evtrm": 133508, 2023-09-10T01:19:10.465 INFO:teuthology.orchestra.run.smithi092.stdout: "ev": 1550, 2023-09-10T01:19:10.465 INFO:teuthology.orchestra.run.smithi092.stdout: "evexg": 0, 2023-09-10T01:19:10.465 INFO:teuthology.orchestra.run.smithi092.stdout: "evexd": 1962, 2023-09-10T01:19:10.466 INFO:teuthology.orchestra.run.smithi092.stdout: "segadd": 246, 2023-09-10T01:19:10.466 INFO:teuthology.orchestra.run.smithi092.stdout: "segex": 243, 2023-09-10T01:19:10.466 INFO:teuthology.orchestra.run.smithi092.stdout: "segtrm": 243, 2023-09-10T01:19:10.467 INFO:teuthology.orchestra.run.smithi092.stdout: "seg": 3, 2023-09-10T01:19:10.467 INFO:teuthology.orchestra.run.smithi092.stdout: "segmjr": 1, 2023-09-10T01:19:10.467 INFO:teuthology.orchestra.run.smithi092.stdout: "segexg": 0, 2023-09-10T01:19:10.467 INFO:teuthology.orchestra.run.smithi092.stdout: "segexd": 3, 2023-09-10T01:19:10.468 INFO:teuthology.orchestra.run.smithi092.stdout: "expos": 565495599, 2023-09-10T01:19:10.468 INFO:teuthology.orchestra.run.smithi092.stdout: "wrpos": 569874726, 2023-09-10T01:19:10.468 INFO:teuthology.orchestra.run.smithi092.stdout: "rdpos": 0, 2023-09-10T01:19:10.468 INFO:teuthology.orchestra.run.smithi092.stdout: "jlat": { 2023-09-10T01:19:10.469 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 68235, 2023-09-10T01:19:10.469 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 325.654266383, 2023-09-10T01:19:10.469 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.004772539 2023-09-10T01:19:10.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.470 INFO:teuthology.orchestra.run.smithi092.stdout: "replayed": 0 2023-09-10T01:19:10.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.470 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mem": { 2023-09-10T01:19:10.471 INFO:teuthology.orchestra.run.smithi092.stdout: "ino": 640, 2023-09-10T01:19:10.471 INFO:teuthology.orchestra.run.smithi092.stdout: "ino+": 23729, 2023-09-10T01:19:10.471 INFO:teuthology.orchestra.run.smithi092.stdout: "ino-": 23089, 2023-09-10T01:19:10.471 INFO:teuthology.orchestra.run.smithi092.stdout: "dir": 110, 2023-09-10T01:19:10.472 INFO:teuthology.orchestra.run.smithi092.stdout: "dir+": 232, 2023-09-10T01:19:10.472 INFO:teuthology.orchestra.run.smithi092.stdout: "dir-": 122, 2023-09-10T01:19:10.472 INFO:teuthology.orchestra.run.smithi092.stdout: "dn": 1426, 2023-09-10T01:19:10.472 INFO:teuthology.orchestra.run.smithi092.stdout: "dn+": 43091, 2023-09-10T01:19:10.473 INFO:teuthology.orchestra.run.smithi092.stdout: "dn-": 41665, 2023-09-10T01:19:10.473 INFO:teuthology.orchestra.run.smithi092.stdout: "cap": 218, 2023-09-10T01:19:10.473 INFO:teuthology.orchestra.run.smithi092.stdout: "cap+": 22901, 2023-09-10T01:19:10.473 INFO:teuthology.orchestra.run.smithi092.stdout: "cap-": 22683, 2023-09-10T01:19:10.474 INFO:teuthology.orchestra.run.smithi092.stdout: "rss": 79228, 2023-09-10T01:19:10.474 INFO:teuthology.orchestra.run.smithi092.stdout: "heap": 207152 2023-09-10T01:19:10.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.475 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_server": { 2023-09-10T01:19:10.475 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_client_request": 229174, 2023-09-10T01:19:10.475 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_server_request": 29856, 2023-09-10T01:19:10.475 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_request": 164425, 2023-09-10T01:19:10.476 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_session": 5062, 2023-09-10T01:19:10.476 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_peer_request": 59557, 2023-09-10T01:19:10.476 INFO:teuthology.orchestra.run.smithi092.stdout: "req_create_latency": { 2023-09-10T01:19:10.477 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 22302, 2023-09-10T01:19:10.477 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 58.160581350, 2023-09-10T01:19:10.477 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002607863 2023-09-10T01:19:10.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.478 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getattr_latency": { 2023-09-10T01:19:10.478 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 31469, 2023-09-10T01:19:10.478 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 230.980620648, 2023-09-10T01:19:10.478 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.007339941 2023-09-10T01:19:10.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.479 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getfilelock_latency": { 2023-09-10T01:19:10.479 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.480 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.480 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.480 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { 2023-09-10T01:19:10.481 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.481 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.481 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.482 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookup_latency": { 2023-09-10T01:19:10.482 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 27047, 2023-09-10T01:19:10.482 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 120.884621232, 2023-09-10T01:19:10.483 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.004469428 2023-09-10T01:19:10.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.483 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookuphash_latency": { 2023-09-10T01:19:10.483 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.484 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.484 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.485 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupino_latency": { 2023-09-10T01:19:10.485 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1, 2023-09-10T01:19:10.485 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000205906, 2023-09-10T01:19:10.486 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000205906 2023-09-10T01:19:10.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.486 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupname_latency": { 2023-09-10T01:19:10.487 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.487 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.487 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.488 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupparent_latency": { 2023-09-10T01:19:10.488 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.488 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.488 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.489 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupsnap_latency": { 2023-09-10T01:19:10.489 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.490 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.490 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.491 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lssnap_latency": { 2023-09-10T01:19:10.491 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 16, 2023-09-10T01:19:10.491 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.004098801, 2023-09-10T01:19:10.492 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000256175 2023-09-10T01:19:10.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.492 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mkdir_latency": { 2023-09-10T01:19:10.493 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 119, 2023-09-10T01:19:10.493 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.630270293, 2023-09-10T01:19:10.493 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.005296389 2023-09-10T01:19:10.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.494 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mknod_latency": { 2023-09-10T01:19:10.494 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.494 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.495 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.495 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mksnap_latency": { 2023-09-10T01:19:10.495 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 16, 2023-09-10T01:19:10.496 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 6.487341909, 2023-09-10T01:19:10.496 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.405458869 2023-09-10T01:19:10.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.496 INFO:teuthology.orchestra.run.smithi092.stdout: "req_open_latency": { 2023-09-10T01:19:10.497 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 9, 2023-09-10T01:19:10.497 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.013031459, 2023-09-10T01:19:10.497 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001447939 2023-09-10T01:19:10.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.498 INFO:teuthology.orchestra.run.smithi092.stdout: "req_readdir_latency": { 2023-09-10T01:19:10.498 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 38639, 2023-09-10T01:19:10.498 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 233.720633878, 2023-09-10T01:19:10.499 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.006048827 2023-09-10T01:19:10.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.499 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rename_latency": { 2023-09-10T01:19:10.500 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 5748, 2023-09-10T01:19:10.500 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 12.962136942, 2023-09-10T01:19:10.500 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002255069 2023-09-10T01:19:10.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.501 INFO:teuthology.orchestra.run.smithi092.stdout: "req_renamesnap_latency": { 2023-09-10T01:19:10.501 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.501 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.501 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.502 INFO:teuthology.orchestra.run.smithi092.stdout: "req_snapdiff_latency": { 2023-09-10T01:19:10.502 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.503 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.503 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.503 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmdir_latency": { 2023-09-10T01:19:10.504 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1, 2023-09-10T01:19:10.504 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000811989, 2023-09-10T01:19:10.504 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000811989 2023-09-10T01:19:10.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.505 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmsnap_latency": { 2023-09-10T01:19:10.505 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 10, 2023-09-10T01:19:10.505 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.118533146, 2023-09-10T01:19:10.506 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.011853314 2023-09-10T01:19:10.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.506 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmxattr_latency": { 2023-09-10T01:19:10.507 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.507 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.507 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.508 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setattr_latency": { 2023-09-10T01:19:10.508 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 17, 2023-09-10T01:19:10.508 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.398456257, 2023-09-10T01:19:10.508 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.023438603 2023-09-10T01:19:10.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.509 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setdirlayout_latency": { 2023-09-10T01:19:10.509 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.509 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.510 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.510 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setfilelock_latency": { 2023-09-10T01:19:10.511 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1284, 2023-09-10T01:19:10.511 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 1.177712521, 2023-09-10T01:19:10.511 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000917221 2023-09-10T01:19:10.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.512 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setlayout_latency": { 2023-09-10T01:19:10.512 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.512 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.512 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.513 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setxattr_latency": { 2023-09-10T01:19:10.513 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 7, 2023-09-10T01:19:10.514 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.020787997, 2023-09-10T01:19:10.514 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002969713 2023-09-10T01:19:10.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.514 INFO:teuthology.orchestra.run.smithi092.stdout: "req_symlink_latency": { 2023-09-10T01:19:10.515 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.515 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.515 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.516 INFO:teuthology.orchestra.run.smithi092.stdout: "req_unlink_latency": { 2023-09-10T01:19:10.516 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 21613, 2023-09-10T01:19:10.516 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 64.474447230, 2023-09-10T01:19:10.517 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002983132 2023-09-10T01:19:10.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.517 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_revoke_eviction": 0, 2023-09-10T01:19:10.517 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_acquisition_throttle": 0, 2023-09-10T01:19:10.518 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getvxattr_latency": { 2023-09-10T01:19:10.518 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.518 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.519 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.519 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.519 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessions": { 2023-09-10T01:19:10.520 INFO:teuthology.orchestra.run.smithi092.stdout: "session_count": 2, 2023-09-10T01:19:10.520 INFO:teuthology.orchestra.run.smithi092.stdout: "session_add": 6, 2023-09-10T01:19:10.520 INFO:teuthology.orchestra.run.smithi092.stdout: "session_remove": 4, 2023-09-10T01:19:10.520 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_open": 2, 2023-09-10T01:19:10.521 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_stale": 0, 2023-09-10T01:19:10.521 INFO:teuthology.orchestra.run.smithi092.stdout: "total_load": 13217, 2023-09-10T01:19:10.521 INFO:teuthology.orchestra.run.smithi092.stdout: "average_load": 6608, 2023-09-10T01:19:10.521 INFO:teuthology.orchestra.run.smithi092.stdout: "avg_session_uptime": 942, 2023-09-10T01:19:10.522 INFO:teuthology.orchestra.run.smithi092.stdout: "mdthresh_evicted": 0 2023-09-10T01:19:10.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.522 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2023-09-10T01:19:10.523 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 240, 2023-09-10T01:19:10.523 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 240, 2023-09-10T01:19:10.523 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:10.523 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:10.524 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:10.524 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:10.524 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:10.525 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:10.525 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:10.525 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:10.525 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:10.526 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:10.526 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:10.526 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:10.526 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:10.527 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:10.527 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:10.527 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:10.529 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:10.529 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:10.529 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:10.530 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:10.530 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:10.530 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:10.530 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:10.531 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:10.531 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:10.531 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:10.532 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:10.532 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:10.532 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:10.532 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2023-09-10T01:19:10.533 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:10.533 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:10.533 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:10.533 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:10.534 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 336874, 2023-09-10T01:19:10.534 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 605, 2023-09-10T01:19:10.534 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 88, 2023-09-10T01:19:10.535 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 1, 2023-09-10T01:19:10.535 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2023-09-10T01:19:10.535 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2023-09-10T01:19:10.535 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:10.536 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:10.536 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:10.536 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:10.536 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 9800, 2023-09-10T01:19:10.537 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 189, 2023-09-10T01:19:10.537 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:10.537 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:10.537 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:10.538 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2023-09-10T01:19:10.538 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 2969499, 2023-09-10T01:19:10.538 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 39379, 2023-09-10T01:19:10.538 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:10.539 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2023-09-10T01:19:10.539 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:10.539 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2023-09-10T01:19:10.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.540 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter": { 2023-09-10T01:19:10.540 INFO:teuthology.orchestra.run.smithi092.stdout: "op_active": 0, 2023-09-10T01:19:10.540 INFO:teuthology.orchestra.run.smithi092.stdout: "op_laggy": 0, 2023-09-10T01:19:10.541 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send": 143432, 2023-09-10T01:19:10.541 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send_bytes": 591452823, 2023-09-10T01:19:10.541 INFO:teuthology.orchestra.run.smithi092.stdout: "op_resend": 31, 2023-09-10T01:19:10.541 INFO:teuthology.orchestra.run.smithi092.stdout: "op_reply": 143432, 2023-09-10T01:19:10.542 INFO:teuthology.orchestra.run.smithi092.stdout: "op_latency": { 2023-09-10T01:19:10.542 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 143432, 2023-09-10T01:19:10.542 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 4594.751153641, 2023-09-10T01:19:10.543 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.032034351 2023-09-10T01:19:10.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.543 INFO:teuthology.orchestra.run.smithi092.stdout: "op_inflight": 0, 2023-09-10T01:19:10.543 INFO:teuthology.orchestra.run.smithi092.stdout: "oplen_avg": { 2023-09-10T01:19:10.544 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 143432, 2023-09-10T01:19:10.544 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 185964 2023-09-10T01:19:10.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.544 INFO:teuthology.orchestra.run.smithi092.stdout: "op": 143432, 2023-09-10T01:19:10.545 INFO:teuthology.orchestra.run.smithi092.stdout: "op_r": 2958, 2023-09-10T01:19:10.545 INFO:teuthology.orchestra.run.smithi092.stdout: "op_w": 140474, 2023-09-10T01:19:10.545 INFO:teuthology.orchestra.run.smithi092.stdout: "op_rmw": 0, 2023-09-10T01:19:10.545 INFO:teuthology.orchestra.run.smithi092.stdout: "op_pg": 0, 2023-09-10T01:19:10.546 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_stat": 1906, 2023-09-10T01:19:10.546 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_create": 24698, 2023-09-10T01:19:10.546 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_read": 633, 2023-09-10T01:19:10.547 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_write": 68848, 2023-09-10T01:19:10.547 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writefull": 607, 2023-09-10T01:19:10.547 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writesame": 0, 2023-09-10T01:19:10.547 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_append": 0, 2023-09-10T01:19:10.548 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_zero": 0, 2023-09-10T01:19:10.548 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_truncate": 6, 2023-09-10T01:19:10.548 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_delete": 43894, 2023-09-10T01:19:10.548 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_mapext": 0, 2023-09-10T01:19:10.549 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_sparse_read": 0, 2023-09-10T01:19:10.549 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_clonerange": 0, 2023-09-10T01:19:10.549 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_getxattr": 2282, 2023-09-10T01:19:10.549 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_setxattr": 37343, 2023-09-10T01:19:10.550 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_cmpxattr": 0, 2023-09-10T01:19:10.550 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_rmxattr": 0, 2023-09-10T01:19:10.550 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_resetxattrs": 0, 2023-09-10T01:19:10.551 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_call": 0, 2023-09-10T01:19:10.551 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_watch": 0, 2023-09-10T01:19:10.551 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_notify": 0, 2023-09-10T01:19:10.551 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_src_cmpxattr": 0, 2023-09-10T01:19:10.552 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls": 0, 2023-09-10T01:19:10.552 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls_filter": 0, 2023-09-10T01:19:10.552 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_other": 0, 2023-09-10T01:19:10.553 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_active": 0, 2023-09-10T01:19:10.553 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_send": 0, 2023-09-10T01:19:10.553 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_resend": 0, 2023-09-10T01:19:10.553 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_ping": 0, 2023-09-10T01:19:10.554 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_active": 0, 2023-09-10T01:19:10.554 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_send": 0, 2023-09-10T01:19:10.554 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_resend": 0, 2023-09-10T01:19:10.554 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_active": 0, 2023-09-10T01:19:10.555 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_send": 0, 2023-09-10T01:19:10.555 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_resend": 0, 2023-09-10T01:19:10.555 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_active": 0, 2023-09-10T01:19:10.556 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_send": 0, 2023-09-10T01:19:10.556 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_resend": 0, 2023-09-10T01:19:10.556 INFO:teuthology.orchestra.run.smithi092.stdout: "command_active": 0, 2023-09-10T01:19:10.556 INFO:teuthology.orchestra.run.smithi092.stdout: "command_send": 0, 2023-09-10T01:19:10.557 INFO:teuthology.orchestra.run.smithi092.stdout: "command_resend": 0, 2023-09-10T01:19:10.557 INFO:teuthology.orchestra.run.smithi092.stdout: "map_epoch": 112, 2023-09-10T01:19:10.557 INFO:teuthology.orchestra.run.smithi092.stdout: "map_full": 0, 2023-09-10T01:19:10.557 INFO:teuthology.orchestra.run.smithi092.stdout: "map_inc": 31, 2023-09-10T01:19:10.558 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_sessions": 12, 2023-09-10T01:19:10.558 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_open": 12, 2023-09-10T01:19:10.558 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_close": 0, 2023-09-10T01:19:10.559 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_laggy": 0, 2023-09-10T01:19:10.559 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_wr": 4508, 2023-09-10T01:19:10.559 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_rd": 101, 2023-09-10T01:19:10.559 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_del": 1138 2023-09-10T01:19:10.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.560 INFO:teuthology.orchestra.run.smithi092.stdout: "oft": { 2023-09-10T01:19:10.560 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_objs": 1, 2023-09-10T01:19:10.560 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_kv_pairs": 0, 2023-09-10T01:19:10.561 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_updates": 0, 2023-09-10T01:19:10.561 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_removes": 0 2023-09-10T01:19:10.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.561 INFO:teuthology.orchestra.run.smithi092.stdout: "osd-slow-ops": { 2023-09-10T01:19:10.562 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_ops_count": 0 2023-09-10T01:19:10.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.562 INFO:teuthology.orchestra.run.smithi092.stdout: "purge_queue": { 2023-09-10T01:19:10.563 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops": 0, 2023-09-10T01:19:10.563 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops_high_water": 16, 2023-09-10T01:19:10.563 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing": 0, 2023-09-10T01:19:10.563 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_high_water": 8, 2023-09-10T01:19:10.564 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed": 21539, 2023-09-10T01:19:10.564 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_item_in_journal": 0 2023-09-10T01:19:10.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.564 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:10.565 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:10.565 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2023-09-10T01:19:10.565 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:10.565 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 2034915, 2023-09-10T01:19:10.566 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 225265263, 2023-09-10T01:19:10.566 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:10.566 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 2034915, 2023-09-10T01:19:10.567 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:10.567 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:10.567 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 2034915, 2023-09-10T01:19:10.568 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 225265263, 2023-09-10T01:19:10.568 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:10.568 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.568 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.569 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.569 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.570 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_bytes": { 2023-09-10T01:19:10.570 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:10.570 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2023-09-10T01:19:10.570 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:10.571 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:10.571 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:10.571 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:10.572 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:10.572 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 143432, 2023-09-10T01:19:10.572 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 593802052, 2023-09-10T01:19:10.572 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 143432, 2023-09-10T01:19:10.573 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 593802052, 2023-09-10T01:19:10.573 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:10.573 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.574 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.574 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.574 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.575 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_ops": { 2023-09-10T01:19:10.575 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:10.575 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 1024, 2023-09-10T01:19:10.576 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:10.576 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:10.576 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:10.576 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:10.577 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:10.577 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 143432, 2023-09-10T01:19:10.577 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 143432, 2023-09-10T01:19:10.577 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 143432, 2023-09-10T01:19:10.578 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 143432, 2023-09-10T01:19:10.578 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:10.578 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.579 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.579 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.579 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.580 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle": { 2023-09-10T01:19:10.580 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:10.580 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2023-09-10T01:19:10.580 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:10.581 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 21539, 2023-09-10T01:19:10.581 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 2350687, 2023-09-10T01:19:10.581 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:10.581 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 21539, 2023-09-10T01:19:10.582 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:10.582 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:10.582 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 613, 2023-09-10T01:19:10.582 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 2350687, 2023-09-10T01:19:10.583 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:10.583 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.583 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.583 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:10.584 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle-0x55d209d58640": { 2023-09-10T01:19:10.585 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:10.585 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2023-09-10T01:19:10.585 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:10.585 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 135058, 2023-09-10T01:19:10.586 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 565680422, 2023-09-10T01:19:10.586 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:10.586 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 135058, 2023-09-10T01:19:10.587 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:10.587 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:10.587 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 68235, 2023-09-10T01:19:10.587 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 565680422, 2023-09-10T01:19:10.588 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:10.588 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:10.588 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:10.588 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:10.589 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.589 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:10.589 INFO:teuthology.orchestra.run.smithi092.stdout:} 2023-09-10T01:19:10.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:10 smithi136 ceph-mon[133708]: pgmap v501: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-09-10T01:19:10.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:10 smithi136 ceph-mon[133708]: osdmap e112: 12 total, 12 up, 12 in 2023-09-10T01:19:10.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:10 smithi003 ceph-mon[97654]: pgmap v501: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-09-10T01:19:10.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:10 smithi003 ceph-mon[97654]: osdmap e112: 12 total, 12 up, 12 in 2023-09-10T01:19:11.774 INFO:tasks.check_counter:Daemon mds.b mds.exported=24 2023-09-10T01:19:11.774 INFO:tasks.check_counter:Daemon mds.b mds.imported=17 2023-09-10T01:19:11.775 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2023-09-10T01:19:11.775 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=16 2023-09-10T01:19:11.775 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=10 2023-09-10T01:19:11.775 DEBUG:tasks.check_counter:Getting stats from e 2023-09-10T01:19:11.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.e perf dump 2023-09-10T01:19:12.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:12 smithi136 ceph-mon[133708]: pgmap v503: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 97 KiB/s wr, 27 op/s 2023-09-10T01:19:12.697 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T01:19:12.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:12 smithi092 ceph-mon[134067]: pgmap v503: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 97 KiB/s wr, 27 op/s 2023-09-10T01:19:12.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:12 smithi003 ceph-mon[97654]: pgmap v503: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 97 KiB/s wr, 27 op/s 2023-09-10T01:19:14.088 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2023-09-10T01:19:14.088 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:14.089 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 2576, 2023-09-10T01:19:14.089 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 2543, 2023-09-10T01:19:14.089 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 178651852, 2023-09-10T01:19:14.089 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 874580, 2023-09-10T01:19:14.090 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 7, 2023-09-10T01:19:14.090 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:14.090 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.286874666, 2023-09-10T01:19:14.091 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.074257800, 2023-09-10T01:19:14.091 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.142409670, 2023-09-10T01:19:14.091 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.064919166, 2023-09-10T01:19:14.091 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:14.092 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 2543, 2023-09-10T01:19:14.092 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.111659119, 2023-09-10T01:19:14.092 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000043908 2023-09-10T01:19:14.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.093 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:14.093 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.093 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.093 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.094 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 1840176, 2023-09-10T01:19:14.094 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 187888 2023-09-10T01:19:14.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.095 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:14.095 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 832, 2023-09-10T01:19:14.095 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 1061, 2023-09-10T01:19:14.096 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 150657482, 2023-09-10T01:19:14.096 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 811540, 2023-09-10T01:19:14.096 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 8, 2023-09-10T01:19:14.096 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:14.097 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.146519975, 2023-09-10T01:19:14.097 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.020607381, 2023-09-10T01:19:14.097 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.098648285, 2023-09-10T01:19:14.097 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.019777281, 2023-09-10T01:19:14.098 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:14.098 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1058, 2023-09-10T01:19:14.098 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.068019859, 2023-09-10T01:19:14.099 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000064290 2023-09-10T01:19:14.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.099 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:14.099 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.100 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.100 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.100 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 320, 2023-09-10T01:19:14.100 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 553344 2023-09-10T01:19:14.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.101 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:14.101 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 1377, 2023-09-10T01:19:14.102 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 1379, 2023-09-10T01:19:14.102 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 239425856, 2023-09-10T01:19:14.102 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 428254, 2023-09-10T01:19:14.103 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 6, 2023-09-10T01:19:14.103 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 5, 2023-09-10T01:19:14.103 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.226337334, 2023-09-10T01:19:14.103 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.021905102, 2023-09-10T01:19:14.104 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.161029397, 2023-09-10T01:19:14.104 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.034434322, 2023-09-10T01:19:14.104 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:14.104 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1377, 2023-09-10T01:19:14.105 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.065635904, 2023-09-10T01:19:14.105 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000047665 2023-09-10T01:19:14.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.106 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:14.106 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.106 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.106 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.107 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 0, 2023-09-10T01:19:14.107 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 0 2023-09-10T01:19:14.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.108 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2023-09-10T01:19:14.108 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 1, 2023-09-10T01:19:14.108 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2023-09-10T01:19:14.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.109 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-MDSRank": { 2023-09-10T01:19:14.109 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2023-09-10T01:19:14.109 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2023-09-10T01:19:14.110 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 7313, 2023-09-10T01:19:14.110 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.392342495, 2023-09-10T01:19:14.110 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000053650 2023-09-10T01:19:14.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.111 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-PurgeQueue": { 2023-09-10T01:19:14.111 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2023-09-10T01:19:14.112 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2023-09-10T01:19:14.112 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.112 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.112 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.113 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": { 2023-09-10T01:19:14.113 INFO:teuthology.orchestra.run.smithi092.stdout: "request": 0, 2023-09-10T01:19:14.114 INFO:teuthology.orchestra.run.smithi092.stdout: "reply": 0, 2023-09-10T01:19:14.114 INFO:teuthology.orchestra.run.smithi092.stdout: "reply_latency": { 2023-09-10T01:19:14.114 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.115 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.115 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.115 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_reply": 0, 2023-09-10T01:19:14.116 INFO:teuthology.orchestra.run.smithi092.stdout: "forward": 0, 2023-09-10T01:19:14.116 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_complete": 0, 2023-09-10T01:19:14.116 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_keys": 0, 2023-09-10T01:19:14.116 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_commit": 0, 2023-09-10T01:19:14.117 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_split": 0, 2023-09-10T01:19:14.117 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_merge": 0, 2023-09-10T01:19:14.117 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes": 21622, 2023-09-10T01:19:14.117 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_top": 14571, 2023-09-10T01:19:14.118 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_bottom": 7045, 2023-09-10T01:19:14.118 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pin_tail": 6, 2023-09-10T01:19:14.118 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pinned": 805, 2023-09-10T01:19:14.119 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_expired": 856, 2023-09-10T01:19:14.119 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_with_caps": 0, 2023-09-10T01:19:14.119 INFO:teuthology.orchestra.run.smithi092.stdout: "caps": 0, 2023-09-10T01:19:14.119 INFO:teuthology.orchestra.run.smithi092.stdout: "subtrees": 16, 2023-09-10T01:19:14.120 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse": 0, 2023-09-10T01:19:14.120 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_hit": 0, 2023-09-10T01:19:14.120 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_forward": 0, 2023-09-10T01:19:14.121 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_discover": 0, 2023-09-10T01:19:14.121 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_dir_fetch": 0, 2023-09-10T01:19:14.121 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_remote_ino": 0, 2023-09-10T01:19:14.121 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_lock": 0, 2023-09-10T01:19:14.122 INFO:teuthology.orchestra.run.smithi092.stdout: "load_cent": 0, 2023-09-10T01:19:14.122 INFO:teuthology.orchestra.run.smithi092.stdout: "q": 0, 2023-09-10T01:19:14.122 INFO:teuthology.orchestra.run.smithi092.stdout: "exported": 0, 2023-09-10T01:19:14.122 INFO:teuthology.orchestra.run.smithi092.stdout: "exported_inodes": 0, 2023-09-10T01:19:14.123 INFO:teuthology.orchestra.run.smithi092.stdout: "imported": 0, 2023-09-10T01:19:14.123 INFO:teuthology.orchestra.run.smithi092.stdout: "imported_inodes": 0, 2023-09-10T01:19:14.123 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_dir_fetch": 0, 2023-09-10T01:19:14.124 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_backtrace_fetch": 0, 2023-09-10T01:19:14.124 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_peer_discover": 0, 2023-09-10T01:19:14.124 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rfiles": 674, 2023-09-10T01:19:14.124 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rbytes": 217420166, 2023-09-10T01:19:14.125 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rsnaps": 6, 2023-09-10T01:19:14.125 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_fetch": 0, 2023-09-10T01:19:14.125 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_set_tag": 0, 2023-09-10T01:19:14.125 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_repaired": 0, 2023-09-10T01:19:14.126 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, 2023-09-10T01:19:14.126 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 0, 2023-09-10T01:19:14.126 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2023-09-10T01:19:14.127 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dirfrag_rstats": 0, 2023-09-10T01:19:14.127 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_file_inodes": 0, 2023-09-10T01:19:14.127 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_inode_file_caps": 0, 2023-09-10T01:19:14.127 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_revoke": 0, 2023-09-10T01:19:14.128 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_grant": 0, 2023-09-10T01:19:14.128 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_trunc": 0, 2023-09-10T01:19:14.128 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-09-10T01:19:14.128 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flush_ack": 0, 2023-09-10T01:19:14.129 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps": 0, 2023-09-10T01:19:14.129 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps_dirty": 0, 2023-09-10T01:19:14.129 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_cap_release": 0, 2023-09-10T01:19:14.130 INFO:teuthology.orchestra.run.smithi092.stdout: "process_request_cap_release": 0 2023-09-10T01:19:14.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.130 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache": { 2023-09-10T01:19:14.130 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update": 0, 2023-09-10T01:19:14.131 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update_receipt": 0, 2023-09-10T01:19:14.131 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_try_discover": 0, 2023-09-10T01:19:14.131 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_send_discover": 0, 2023-09-10T01:19:14.131 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_handle_discover": 0, 2023-09-10T01:19:14.132 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays": 0, 2023-09-10T01:19:14.132 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_delayed": 0, 2023-09-10T01:19:14.132 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_enqueuing": 0, 2023-09-10T01:19:14.132 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_created": 0, 2023-09-10T01:19:14.133 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_enqueued": 0, 2023-09-10T01:19:14.133 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_reintegrated": 0, 2023-09-10T01:19:14.133 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_migrated": 0, 2023-09-10T01:19:14.133 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_processing": 0, 2023-09-10T01:19:14.134 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_enqueued": 0, 2023-09-10T01:19:14.134 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_prioritized": 0, 2023-09-10T01:19:14.134 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_started": 0, 2023-09-10T01:19:14.135 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_completed": 0, 2023-09-10T01:19:14.135 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_enqueue_scrub": 0, 2023-09-10T01:19:14.135 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_exportdir": 0, 2023-09-10T01:19:14.135 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_flush": 0, 2023-09-10T01:19:14.136 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragmentdir": 0, 2023-09-10T01:19:14.136 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragstats": 0, 2023-09-10T01:19:14.136 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_inodestats": 0 2023-09-10T01:19:14.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.137 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log": { 2023-09-10T01:19:14.137 INFO:teuthology.orchestra.run.smithi092.stdout: "evlrg": 0, 2023-09-10T01:19:14.137 INFO:teuthology.orchestra.run.smithi092.stdout: "evadd": 0, 2023-09-10T01:19:14.137 INFO:teuthology.orchestra.run.smithi092.stdout: "evex": 0, 2023-09-10T01:19:14.138 INFO:teuthology.orchestra.run.smithi092.stdout: "evtrm": 0, 2023-09-10T01:19:14.138 INFO:teuthology.orchestra.run.smithi092.stdout: "ev": 135058, 2023-09-10T01:19:14.138 INFO:teuthology.orchestra.run.smithi092.stdout: "evexg": 0, 2023-09-10T01:19:14.138 INFO:teuthology.orchestra.run.smithi092.stdout: "evexd": 0, 2023-09-10T01:19:14.139 INFO:teuthology.orchestra.run.smithi092.stdout: "segadd": 0, 2023-09-10T01:19:14.139 INFO:teuthology.orchestra.run.smithi092.stdout: "segex": 0, 2023-09-10T01:19:14.139 INFO:teuthology.orchestra.run.smithi092.stdout: "segtrm": 0, 2023-09-10T01:19:14.140 INFO:teuthology.orchestra.run.smithi092.stdout: "seg": 3, 2023-09-10T01:19:14.140 INFO:teuthology.orchestra.run.smithi092.stdout: "segmjr": 69, 2023-09-10T01:19:14.140 INFO:teuthology.orchestra.run.smithi092.stdout: "segexg": 0, 2023-09-10T01:19:14.140 INFO:teuthology.orchestra.run.smithi092.stdout: "segexd": 0, 2023-09-10T01:19:14.141 INFO:teuthology.orchestra.run.smithi092.stdout: "expos": 565495599, 2023-09-10T01:19:14.141 INFO:teuthology.orchestra.run.smithi092.stdout: "wrpos": 569874726, 2023-09-10T01:19:14.141 INFO:teuthology.orchestra.run.smithi092.stdout: "rdpos": 569867375, 2023-09-10T01:19:14.141 INFO:teuthology.orchestra.run.smithi092.stdout: "jlat": { 2023-09-10T01:19:14.142 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.142 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.142 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.143 INFO:teuthology.orchestra.run.smithi092.stdout: "replayed": 135058 2023-09-10T01:19:14.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.143 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mem": { 2023-09-10T01:19:14.144 INFO:teuthology.orchestra.run.smithi092.stdout: "ino": 732, 2023-09-10T01:19:14.144 INFO:teuthology.orchestra.run.smithi092.stdout: "ino+": 23358, 2023-09-10T01:19:14.144 INFO:teuthology.orchestra.run.smithi092.stdout: "ino-": 22626, 2023-09-10T01:19:14.144 INFO:teuthology.orchestra.run.smithi092.stdout: "dir": 83, 2023-09-10T01:19:14.145 INFO:teuthology.orchestra.run.smithi092.stdout: "dir+": 199, 2023-09-10T01:19:14.145 INFO:teuthology.orchestra.run.smithi092.stdout: "dir-": 116, 2023-09-10T01:19:14.145 INFO:teuthology.orchestra.run.smithi092.stdout: "dn": 21622, 2023-09-10T01:19:14.145 INFO:teuthology.orchestra.run.smithi092.stdout: "dn+": 24146, 2023-09-10T01:19:14.146 INFO:teuthology.orchestra.run.smithi092.stdout: "dn-": 2524, 2023-09-10T01:19:14.146 INFO:teuthology.orchestra.run.smithi092.stdout: "cap": 0, 2023-09-10T01:19:14.146 INFO:teuthology.orchestra.run.smithi092.stdout: "cap+": 0, 2023-09-10T01:19:14.147 INFO:teuthology.orchestra.run.smithi092.stdout: "cap-": 0, 2023-09-10T01:19:14.147 INFO:teuthology.orchestra.run.smithi092.stdout: "rss": 77340, 2023-09-10T01:19:14.147 INFO:teuthology.orchestra.run.smithi092.stdout: "heap": 207152 2023-09-10T01:19:14.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.148 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_server": { 2023-09-10T01:19:14.148 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_client_request": 0, 2023-09-10T01:19:14.148 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_server_request": 0, 2023-09-10T01:19:14.148 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_request": 0, 2023-09-10T01:19:14.149 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_session": 0, 2023-09-10T01:19:14.149 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_peer_request": 0, 2023-09-10T01:19:14.149 INFO:teuthology.orchestra.run.smithi092.stdout: "req_create_latency": { 2023-09-10T01:19:14.150 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.150 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.150 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.151 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getattr_latency": { 2023-09-10T01:19:14.151 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.151 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.151 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.152 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getfilelock_latency": { 2023-09-10T01:19:14.152 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.153 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.153 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.153 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { 2023-09-10T01:19:14.154 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.154 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.154 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.155 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookup_latency": { 2023-09-10T01:19:14.155 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.155 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.155 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.156 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookuphash_latency": { 2023-09-10T01:19:14.156 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.156 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.157 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.157 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupino_latency": { 2023-09-10T01:19:14.157 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.158 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.158 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.158 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupname_latency": { 2023-09-10T01:19:14.159 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.159 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.159 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.160 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupparent_latency": { 2023-09-10T01:19:14.160 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.160 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.161 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.161 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupsnap_latency": { 2023-09-10T01:19:14.161 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.162 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.162 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.163 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lssnap_latency": { 2023-09-10T01:19:14.163 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.163 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.163 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.164 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mkdir_latency": { 2023-09-10T01:19:14.164 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.164 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.165 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.165 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mknod_latency": { 2023-09-10T01:19:14.165 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.166 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.166 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.167 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mksnap_latency": { 2023-09-10T01:19:14.167 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.167 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.167 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.168 INFO:teuthology.orchestra.run.smithi092.stdout: "req_open_latency": { 2023-09-10T01:19:14.168 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.168 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.169 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.169 INFO:teuthology.orchestra.run.smithi092.stdout: "req_readdir_latency": { 2023-09-10T01:19:14.169 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.170 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.170 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.171 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rename_latency": { 2023-09-10T01:19:14.171 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.171 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.171 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.172 INFO:teuthology.orchestra.run.smithi092.stdout: "req_renamesnap_latency": { 2023-09-10T01:19:14.172 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.172 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.173 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.173 INFO:teuthology.orchestra.run.smithi092.stdout: "req_snapdiff_latency": { 2023-09-10T01:19:14.173 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.174 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.174 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.174 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmdir_latency": { 2023-09-10T01:19:14.175 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.175 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.175 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.176 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmsnap_latency": { 2023-09-10T01:19:14.176 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.176 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.177 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.177 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmxattr_latency": { 2023-09-10T01:19:14.177 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.178 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.178 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.178 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setattr_latency": { 2023-09-10T01:19:14.179 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.179 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.179 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.180 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setdirlayout_latency": { 2023-09-10T01:19:14.180 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.180 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.180 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.181 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setfilelock_latency": { 2023-09-10T01:19:14.181 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.181 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.182 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.182 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setlayout_latency": { 2023-09-10T01:19:14.183 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.183 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.183 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.184 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setxattr_latency": { 2023-09-10T01:19:14.184 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.184 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.184 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.185 INFO:teuthology.orchestra.run.smithi092.stdout: "req_symlink_latency": { 2023-09-10T01:19:14.185 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.186 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.186 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.186 INFO:teuthology.orchestra.run.smithi092.stdout: "req_unlink_latency": { 2023-09-10T01:19:14.187 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.187 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.187 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.188 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_revoke_eviction": 0, 2023-09-10T01:19:14.188 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_acquisition_throttle": 0, 2023-09-10T01:19:14.188 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getvxattr_latency": { 2023-09-10T01:19:14.189 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.189 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.189 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.190 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessions": { 2023-09-10T01:19:14.190 INFO:teuthology.orchestra.run.smithi092.stdout: "session_count": 2, 2023-09-10T01:19:14.191 INFO:teuthology.orchestra.run.smithi092.stdout: "session_add": 6, 2023-09-10T01:19:14.191 INFO:teuthology.orchestra.run.smithi092.stdout: "session_remove": 4, 2023-09-10T01:19:14.191 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_open": 4, 2023-09-10T01:19:14.191 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_stale": 0, 2023-09-10T01:19:14.192 INFO:teuthology.orchestra.run.smithi092.stdout: "total_load": 0, 2023-09-10T01:19:14.192 INFO:teuthology.orchestra.run.smithi092.stdout: "average_load": 0, 2023-09-10T01:19:14.192 INFO:teuthology.orchestra.run.smithi092.stdout: "avg_session_uptime": 4647, 2023-09-10T01:19:14.192 INFO:teuthology.orchestra.run.smithi092.stdout: "mdthresh_evicted": 0 2023-09-10T01:19:14.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.193 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2023-09-10T01:19:14.193 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:14.193 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:14.194 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:14.194 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:14.194 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:14.195 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:14.195 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:14.195 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:14.195 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:14.196 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:14.196 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:14.196 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:14.196 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:14.197 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:14.197 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:14.197 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:14.197 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:14.198 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:14.198 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:14.198 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:14.198 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:14.199 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:14.199 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:14.199 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:14.199 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:14.200 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:14.200 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:14.200 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:14.200 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:14.201 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:14.201 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:14.201 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2023-09-10T01:19:14.201 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:14.202 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:14.202 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:14.202 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:14.203 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 261191, 2023-09-10T01:19:14.203 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 953, 2023-09-10T01:19:14.203 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:14.203 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:14.204 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2023-09-10T01:19:14.204 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2023-09-10T01:19:14.204 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:14.204 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:14.205 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:14.205 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:14.205 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 10056, 2023-09-10T01:19:14.205 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 196, 2023-09-10T01:19:14.206 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:14.206 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:14.206 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:14.207 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2023-09-10T01:19:14.207 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 17196868, 2023-09-10T01:19:14.207 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 115559, 2023-09-10T01:19:14.207 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:14.208 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2023-09-10T01:19:14.208 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:14.208 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2023-09-10T01:19:14.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.209 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter": { 2023-09-10T01:19:14.209 INFO:teuthology.orchestra.run.smithi092.stdout: "op_active": 0, 2023-09-10T01:19:14.209 INFO:teuthology.orchestra.run.smithi092.stdout: "op_laggy": 0, 2023-09-10T01:19:14.210 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send": 4415, 2023-09-10T01:19:14.210 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send_bytes": 26, 2023-09-10T01:19:14.210 INFO:teuthology.orchestra.run.smithi092.stdout: "op_resend": 4, 2023-09-10T01:19:14.210 INFO:teuthology.orchestra.run.smithi092.stdout: "op_reply": 4415, 2023-09-10T01:19:14.211 INFO:teuthology.orchestra.run.smithi092.stdout: "op_latency": { 2023-09-10T01:19:14.211 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 4415, 2023-09-10T01:19:14.211 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 7.206795960, 2023-09-10T01:19:14.211 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001632343 2023-09-10T01:19:14.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op_inflight": 0, 2023-09-10T01:19:14.212 INFO:teuthology.orchestra.run.smithi092.stdout: "oplen_avg": { 2023-09-10T01:19:14.212 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 4415, 2023-09-10T01:19:14.213 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 4416 2023-09-10T01:19:14.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.213 INFO:teuthology.orchestra.run.smithi092.stdout: "op": 4415, 2023-09-10T01:19:14.214 INFO:teuthology.orchestra.run.smithi092.stdout: "op_r": 4415, 2023-09-10T01:19:14.214 INFO:teuthology.orchestra.run.smithi092.stdout: "op_w": 0, 2023-09-10T01:19:14.214 INFO:teuthology.orchestra.run.smithi092.stdout: "op_rmw": 0, 2023-09-10T01:19:14.214 INFO:teuthology.orchestra.run.smithi092.stdout: "op_pg": 0, 2023-09-10T01:19:14.215 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_stat": 2401, 2023-09-10T01:19:14.215 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_create": 0, 2023-09-10T01:19:14.215 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_read": 2011, 2023-09-10T01:19:14.215 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_write": 0, 2023-09-10T01:19:14.216 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writefull": 0, 2023-09-10T01:19:14.216 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writesame": 0, 2023-09-10T01:19:14.216 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_append": 0, 2023-09-10T01:19:14.217 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_zero": 0, 2023-09-10T01:19:14.217 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_truncate": 0, 2023-09-10T01:19:14.217 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_delete": 0, 2023-09-10T01:19:14.217 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_mapext": 0, 2023-09-10T01:19:14.218 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_sparse_read": 0, 2023-09-10T01:19:14.218 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_clonerange": 0, 2023-09-10T01:19:14.218 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_getxattr": 2, 2023-09-10T01:19:14.218 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_setxattr": 0, 2023-09-10T01:19:14.219 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_cmpxattr": 0, 2023-09-10T01:19:14.219 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_rmxattr": 0, 2023-09-10T01:19:14.219 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_resetxattrs": 0, 2023-09-10T01:19:14.220 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_call": 0, 2023-09-10T01:19:14.220 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_watch": 0, 2023-09-10T01:19:14.220 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_notify": 0, 2023-09-10T01:19:14.220 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_src_cmpxattr": 0, 2023-09-10T01:19:14.221 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls": 0, 2023-09-10T01:19:14.221 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls_filter": 0, 2023-09-10T01:19:14.221 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_other": 0, 2023-09-10T01:19:14.222 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_active": 0, 2023-09-10T01:19:14.222 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_send": 0, 2023-09-10T01:19:14.222 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_resend": 0, 2023-09-10T01:19:14.222 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_ping": 0, 2023-09-10T01:19:14.223 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_active": 0, 2023-09-10T01:19:14.223 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_send": 0, 2023-09-10T01:19:14.223 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_resend": 0, 2023-09-10T01:19:14.223 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_active": 0, 2023-09-10T01:19:14.224 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_send": 0, 2023-09-10T01:19:14.224 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_resend": 0, 2023-09-10T01:19:14.224 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_active": 0, 2023-09-10T01:19:14.225 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_send": 0, 2023-09-10T01:19:14.225 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_resend": 0, 2023-09-10T01:19:14.225 INFO:teuthology.orchestra.run.smithi092.stdout: "command_active": 0, 2023-09-10T01:19:14.225 INFO:teuthology.orchestra.run.smithi092.stdout: "command_send": 0, 2023-09-10T01:19:14.226 INFO:teuthology.orchestra.run.smithi092.stdout: "command_resend": 0, 2023-09-10T01:19:14.226 INFO:teuthology.orchestra.run.smithi092.stdout: "map_epoch": 112, 2023-09-10T01:19:14.226 INFO:teuthology.orchestra.run.smithi092.stdout: "map_full": 0, 2023-09-10T01:19:14.226 INFO:teuthology.orchestra.run.smithi092.stdout: "map_inc": 24, 2023-09-10T01:19:14.227 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_sessions": 10, 2023-09-10T01:19:14.227 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_open": 10, 2023-09-10T01:19:14.227 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_close": 0, 2023-09-10T01:19:14.228 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_laggy": 0, 2023-09-10T01:19:14.228 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_wr": 0, 2023-09-10T01:19:14.228 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_rd": 2, 2023-09-10T01:19:14.228 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_del": 0 2023-09-10T01:19:14.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.229 INFO:teuthology.orchestra.run.smithi092.stdout: "oft": { 2023-09-10T01:19:14.229 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_objs": 0, 2023-09-10T01:19:14.229 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_kv_pairs": 0, 2023-09-10T01:19:14.230 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_updates": 0, 2023-09-10T01:19:14.230 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_removes": 0 2023-09-10T01:19:14.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.231 INFO:teuthology.orchestra.run.smithi092.stdout: "osd-slow-ops": { 2023-09-10T01:19:14.231 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_ops_count": 0 2023-09-10T01:19:14.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.231 INFO:teuthology.orchestra.run.smithi092.stdout: "purge_queue": { 2023-09-10T01:19:14.232 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops": 0, 2023-09-10T01:19:14.232 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops_high_water": 0, 2023-09-10T01:19:14.232 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing": 0, 2023-09-10T01:19:14.233 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_high_water": 0, 2023-09-10T01:19:14.233 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed": 0, 2023-09-10T01:19:14.233 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_item_in_journal": 0 2023-09-10T01:19:14.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.234 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:14.237 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:14.237 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2023-09-10T01:19:14.238 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:14.238 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 4781, 2023-09-10T01:19:14.238 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 568283506, 2023-09-10T01:19:14.238 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:14.239 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 4781, 2023-09-10T01:19:14.239 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:14.239 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:14.240 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 4781, 2023-09-10T01:19:14.240 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 568283506, 2023-09-10T01:19:14.240 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:14.240 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.241 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.241 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.242 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_bytes": { 2023-09-10T01:19:14.242 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:14.242 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2023-09-10T01:19:14.243 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:14.243 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:14.243 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:14.243 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:14.244 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:14.244 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 4415, 2023-09-10T01:19:14.244 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 565680432, 2023-09-10T01:19:14.245 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 4415, 2023-09-10T01:19:14.245 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 565680432, 2023-09-10T01:19:14.245 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:14.245 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.246 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.246 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.246 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.247 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_ops": { 2023-09-10T01:19:14.247 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:14.247 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 1024, 2023-09-10T01:19:14.248 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:14.248 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:14.248 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:14.248 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:14.249 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:14.249 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 4415, 2023-09-10T01:19:14.249 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 4415, 2023-09-10T01:19:14.250 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 4415, 2023-09-10T01:19:14.250 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 4415, 2023-09-10T01:19:14.250 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:14.250 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.251 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.251 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.251 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.252 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle": { 2023-09-10T01:19:14.252 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:14.252 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2023-09-10T01:19:14.253 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:14.253 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:14.253 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:14.253 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:14.254 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:14.254 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:14.254 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:14.254 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 0, 2023-09-10T01:19:14.255 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 0, 2023-09-10T01:19:14.255 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:14.255 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.255 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.256 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.256 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:14.257 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle-0x55b7e5b3c960": { 2023-09-10T01:19:14.257 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:14.257 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2023-09-10T01:19:14.257 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:14.258 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:14.258 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:14.258 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:14.258 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:14.259 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:14.259 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:14.259 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 0, 2023-09-10T01:19:14.260 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 0, 2023-09-10T01:19:14.260 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:14.260 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:14.260 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:14.261 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:14.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:14.261 INFO:teuthology.orchestra.run.smithi092.stdout:} 2023-09-10T01:19:14.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:14 smithi136 ceph-mon[133708]: pgmap v504: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 97 KiB/s wr, 27 op/s 2023-09-10T01:19:14.804 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:14 smithi092 ceph-mon[134067]: pgmap v504: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 97 KiB/s wr, 27 op/s 2023-09-10T01:19:14.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:14 smithi003 ceph-mon[97654]: pgmap v504: 129 pgs: 129 active+clean; 809 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 97 KiB/s wr, 27 op/s 2023-09-10T01:19:15.457 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-09-10T01:19:15.458 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-09-10T01:19:15.458 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2023-09-10T01:19:15.458 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2023-09-10T01:19:15.459 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2023-09-10T01:19:15.459 DEBUG:tasks.check_counter:Getting stats from h 2023-09-10T01:19:15.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.h perf dump 2023-09-10T01:19:16.170 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T01:19:16.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:16 smithi136 ceph-mon[133708]: pgmap v505: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 787 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-09-10T01:19:16.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:16 smithi092 ceph-mon[134067]: pgmap v505: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 787 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-09-10T01:19:16.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:16 smithi003 ceph-mon[97654]: pgmap v505: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 787 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-09-10T01:19:17.826 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2023-09-10T01:19:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 144866, 2023-09-10T01:19:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 209739, 2023-09-10T01:19:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 47468165, 2023-09-10T01:19:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 247351752, 2023-09-10T01:19:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 48, 2023-09-10T01:19:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 6.793318761, 2023-09-10T01:19:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 3.162882654, 2023-09-10T01:19:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 4.774377886, 2023-09-10T01:19:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.311502029, 2023-09-10T01:19:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 209738, 2023-09-10T01:19:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 29.597682948, 2023-09-10T01:19:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000141117 2023-09-10T01:19:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 166663, 2023-09-10T01:19:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.041369479, 2023-09-10T01:19:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000248 2023-09-10T01:19:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 1853024, 2023-09-10T01:19:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 209568 2023-09-10T01:19:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 1531700, 2023-09-10T01:19:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 1502750, 2023-09-10T01:19:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 268147899, 2023-09-10T01:19:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 311861428, 2023-09-10T01:19:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 13, 2023-09-10T01:19:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 5, 2023-09-10T01:19:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 56.577867431, 2023-09-10T01:19:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 24.165726655, 2023-09-10T01:19:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 39.833456161, 2023-09-10T01:19:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.393381222, 2023-09-10T01:19:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1502750, 2023-09-10T01:19:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 17.850597283, 2023-09-10T01:19:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000011878 2023-09-10T01:19:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 2996698, 2023-09-10T01:19:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.316598438, 2023-09-10T01:19:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000105 2023-09-10T01:19:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 4928, 2023-09-10T01:19:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 130880 2023-09-10T01:19:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 20569, 2023-09-10T01:19:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 20766, 2023-09-10T01:19:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 6636559, 2023-09-10T01:19:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 121120949, 2023-09-10T01:19:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 13, 2023-09-10T01:19:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 7, 2023-09-10T01:19:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 1.700056608, 2023-09-10T01:19:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.499655229, 2023-09-10T01:19:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.303794165, 2023-09-10T01:19:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.805735140, 2023-09-10T01:19:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 20762, 2023-09-10T01:19:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 2.711815242, 2023-09-10T01:19:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000130614 2023-09-10T01:19:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 4, 2023-09-10T01:19:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000398, 2023-09-10T01:19:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000099 2023-09-10T01:19:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 160, 2023-09-10T01:19:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 426288 2023-09-10T01:19:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2023-09-10T01:19:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 1, 2023-09-10T01:19:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2023-09-10T01:19:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-MDSRank": { 2023-09-10T01:19:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2023-09-10T01:19:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2023-09-10T01:19:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 35052, 2023-09-10T01:19:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 39.161674978, 2023-09-10T01:19:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001117245 2023-09-10T01:19:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-PurgeQueue": { 2023-09-10T01:19:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2023-09-10T01:19:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2023-09-10T01:19:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 19821, 2023-09-10T01:19:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 2.716691992, 2023-09-10T01:19:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000137061 2023-09-10T01:19:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:17.852 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.852 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": { 2023-09-10T01:19:17.852 INFO:teuthology.orchestra.run.smithi092.stdout: "request": 90000, 2023-09-10T01:19:17.853 INFO:teuthology.orchestra.run.smithi092.stdout: "reply": 87651, 2023-09-10T01:19:17.853 INFO:teuthology.orchestra.run.smithi092.stdout: "reply_latency": { 2023-09-10T01:19:17.853 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 87651, 2023-09-10T01:19:17.854 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 232.823032877, 2023-09-10T01:19:17.854 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002656250 2023-09-10T01:19:17.854 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.855 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_reply": 0, 2023-09-10T01:19:17.855 INFO:teuthology.orchestra.run.smithi092.stdout: "forward": 2349, 2023-09-10T01:19:17.855 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_complete": 3, 2023-09-10T01:19:17.855 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_keys": 0, 2023-09-10T01:19:17.856 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_commit": 44, 2023-09-10T01:19:17.856 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_split": 4, 2023-09-10T01:19:17.856 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_merge": 1, 2023-09-10T01:19:17.856 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes": 1561, 2023-09-10T01:19:17.857 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_top": 248, 2023-09-10T01:19:17.857 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_bottom": 1295, 2023-09-10T01:19:17.857 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pin_tail": 18, 2023-09-10T01:19:17.857 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pinned": 1207, 2023-09-10T01:19:17.858 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_expired": 359, 2023-09-10T01:19:17.858 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_with_caps": 316, 2023-09-10T01:19:17.858 INFO:teuthology.orchestra.run.smithi092.stdout: "caps": 316, 2023-09-10T01:19:17.859 INFO:teuthology.orchestra.run.smithi092.stdout: "subtrees": 15, 2023-09-10T01:19:17.859 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse": 143836, 2023-09-10T01:19:17.859 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_hit": 79794, 2023-09-10T01:19:17.859 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_forward": 2338, 2023-09-10T01:19:17.860 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_discover": 54, 2023-09-10T01:19:17.860 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_dir_fetch": 2, 2023-09-10T01:19:17.860 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_remote_ino": 0, 2023-09-10T01:19:17.861 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_lock": 896, 2023-09-10T01:19:17.861 INFO:teuthology.orchestra.run.smithi092.stdout: "load_cent": 2716, 2023-09-10T01:19:17.861 INFO:teuthology.orchestra.run.smithi092.stdout: "q": 0, 2023-09-10T01:19:17.861 INFO:teuthology.orchestra.run.smithi092.stdout: "exported": 12, 2023-09-10T01:19:17.862 INFO:teuthology.orchestra.run.smithi092.stdout: "exported_inodes": 303, 2023-09-10T01:19:17.862 INFO:teuthology.orchestra.run.smithi092.stdout: "imported": 12, 2023-09-10T01:19:17.862 INFO:teuthology.orchestra.run.smithi092.stdout: "imported_inodes": 846, 2023-09-10T01:19:17.863 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_dir_fetch": 0, 2023-09-10T01:19:17.863 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_backtrace_fetch": 1004, 2023-09-10T01:19:17.863 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_peer_discover": 0, 2023-09-10T01:19:17.863 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rfiles": 675, 2023-09-10T01:19:17.864 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rbytes": 211566833, 2023-09-10T01:19:17.864 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rsnaps": 6, 2023-09-10T01:19:17.864 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_fetch": 1004, 2023-09-10T01:19:17.865 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_set_tag": 0, 2023-09-10T01:19:17.865 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_repaired": 0, 2023-09-10T01:19:17.865 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, 2023-09-10T01:19:17.865 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 93, 2023-09-10T01:19:17.866 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2023-09-10T01:19:17.866 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dirfrag_rstats": 93, 2023-09-10T01:19:17.866 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_file_inodes": 911, 2023-09-10T01:19:17.867 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_inode_file_caps": 3, 2023-09-10T01:19:17.867 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_revoke": 9075, 2023-09-10T01:19:17.867 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_grant": 52474, 2023-09-10T01:19:17.867 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_trunc": 0, 2023-09-10T01:19:17.868 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flushsnap_ack": 1334, 2023-09-10T01:19:17.868 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flush_ack": 0, 2023-09-10T01:19:17.868 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps": 30981, 2023-09-10T01:19:17.869 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps_dirty": 14267, 2023-09-10T01:19:17.869 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_cap_release": 3267, 2023-09-10T01:19:17.869 INFO:teuthology.orchestra.run.smithi092.stdout: "process_request_cap_release": 72679 2023-09-10T01:19:17.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.870 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache": { 2023-09-10T01:19:17.870 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update": 54, 2023-09-10T01:19:17.870 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update_receipt": 160, 2023-09-10T01:19:17.871 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_try_discover": 88, 2023-09-10T01:19:17.871 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_send_discover": 219, 2023-09-10T01:19:17.871 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_handle_discover": 117, 2023-09-10T01:19:17.872 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays": 0, 2023-09-10T01:19:17.872 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_delayed": 0, 2023-09-10T01:19:17.872 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_enqueuing": 0, 2023-09-10T01:19:17.872 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_created": 11617, 2023-09-10T01:19:17.873 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_enqueued": 11617, 2023-09-10T01:19:17.873 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_reintegrated": 0, 2023-09-10T01:19:17.873 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_migrated": 0, 2023-09-10T01:19:17.873 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_processing": 0, 2023-09-10T01:19:17.874 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_enqueued": 0, 2023-09-10T01:19:17.874 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_prioritized": 0, 2023-09-10T01:19:17.874 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_started": 0, 2023-09-10T01:19:17.875 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_completed": 0, 2023-09-10T01:19:17.875 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_enqueue_scrub": 0, 2023-09-10T01:19:17.875 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_exportdir": 16, 2023-09-10T01:19:17.875 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_flush": 0, 2023-09-10T01:19:17.876 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragmentdir": 5, 2023-09-10T01:19:17.876 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragstats": 0, 2023-09-10T01:19:17.876 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_inodestats": 0 2023-09-10T01:19:17.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.877 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log": { 2023-09-10T01:19:17.877 INFO:teuthology.orchestra.run.smithi092.stdout: "evlrg": 0, 2023-09-10T01:19:17.877 INFO:teuthology.orchestra.run.smithi092.stdout: "evadd": 90406, 2023-09-10T01:19:17.878 INFO:teuthology.orchestra.run.smithi092.stdout: "evex": 0, 2023-09-10T01:19:17.878 INFO:teuthology.orchestra.run.smithi092.stdout: "evtrm": 0, 2023-09-10T01:19:17.878 INFO:teuthology.orchestra.run.smithi092.stdout: "ev": 90406, 2023-09-10T01:19:17.878 INFO:teuthology.orchestra.run.smithi092.stdout: "evexg": 0, 2023-09-10T01:19:17.879 INFO:teuthology.orchestra.run.smithi092.stdout: "evexd": 0, 2023-09-10T01:19:17.879 INFO:teuthology.orchestra.run.smithi092.stdout: "segadd": 115, 2023-09-10T01:19:17.879 INFO:teuthology.orchestra.run.smithi092.stdout: "segex": 0, 2023-09-10T01:19:17.880 INFO:teuthology.orchestra.run.smithi092.stdout: "segtrm": 0, 2023-09-10T01:19:17.880 INFO:teuthology.orchestra.run.smithi092.stdout: "seg": 115, 2023-09-10T01:19:17.880 INFO:teuthology.orchestra.run.smithi092.stdout: "segmjr": 9, 2023-09-10T01:19:17.880 INFO:teuthology.orchestra.run.smithi092.stdout: "segexg": 0, 2023-09-10T01:19:17.881 INFO:teuthology.orchestra.run.smithi092.stdout: "segexd": 0, 2023-09-10T01:19:17.881 INFO:teuthology.orchestra.run.smithi092.stdout: "expos": 4194304, 2023-09-10T01:19:17.881 INFO:teuthology.orchestra.run.smithi092.stdout: "wrpos": 229701497, 2023-09-10T01:19:17.881 INFO:teuthology.orchestra.run.smithi092.stdout: "rdpos": 0, 2023-09-10T01:19:17.882 INFO:teuthology.orchestra.run.smithi092.stdout: "jlat": { 2023-09-10T01:19:17.882 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 18366, 2023-09-10T01:19:17.882 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 111.682441375, 2023-09-10T01:19:17.883 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.006080934 2023-09-10T01:19:17.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.885 INFO:teuthology.orchestra.run.smithi092.stdout: "replayed": 0 2023-09-10T01:19:17.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.885 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mem": { 2023-09-10T01:19:17.885 INFO:teuthology.orchestra.run.smithi092.stdout: "ino": 970, 2023-09-10T01:19:17.886 INFO:teuthology.orchestra.run.smithi092.stdout: "ino+": 12821, 2023-09-10T01:19:17.886 INFO:teuthology.orchestra.run.smithi092.stdout: "ino-": 11851, 2023-09-10T01:19:17.886 INFO:teuthology.orchestra.run.smithi092.stdout: "dir": 123, 2023-09-10T01:19:17.887 INFO:teuthology.orchestra.run.smithi092.stdout: "dir+": 240, 2023-09-10T01:19:17.887 INFO:teuthology.orchestra.run.smithi092.stdout: "dir-": 117, 2023-09-10T01:19:17.887 INFO:teuthology.orchestra.run.smithi092.stdout: "dn": 1570, 2023-09-10T01:19:17.888 INFO:teuthology.orchestra.run.smithi092.stdout: "dn+": 24850, 2023-09-10T01:19:17.888 INFO:teuthology.orchestra.run.smithi092.stdout: "dn-": 23280, 2023-09-10T01:19:17.888 INFO:teuthology.orchestra.run.smithi092.stdout: "cap": 223, 2023-09-10T01:19:17.888 INFO:teuthology.orchestra.run.smithi092.stdout: "cap+": 12805, 2023-09-10T01:19:17.889 INFO:teuthology.orchestra.run.smithi092.stdout: "cap-": 12582, 2023-09-10T01:19:17.889 INFO:teuthology.orchestra.run.smithi092.stdout: "rss": 71904, 2023-09-10T01:19:17.889 INFO:teuthology.orchestra.run.smithi092.stdout: "heap": 198960 2023-09-10T01:19:17.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.890 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_server": { 2023-09-10T01:19:17.890 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_client_request": 130291, 2023-09-10T01:19:17.890 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_server_request": 4, 2023-09-10T01:19:17.891 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_request": 90000, 2023-09-10T01:19:17.891 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_session": 1812, 2023-09-10T01:19:17.891 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_peer_request": 8874, 2023-09-10T01:19:17.891 INFO:teuthology.orchestra.run.smithi092.stdout: "req_create_latency": { 2023-09-10T01:19:17.892 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 11640, 2023-09-10T01:19:17.892 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 40.923292185, 2023-09-10T01:19:17.892 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.003515746 2023-09-10T01:19:17.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.893 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getattr_latency": { 2023-09-10T01:19:17.893 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 30, 2023-09-10T01:19:17.893 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.058652655, 2023-09-10T01:19:17.894 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001955088 2023-09-10T01:19:17.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.894 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getfilelock_latency": { 2023-09-10T01:19:17.895 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.895 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.895 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.896 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { 2023-09-10T01:19:17.896 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.896 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.897 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.897 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookup_latency": { 2023-09-10T01:19:17.898 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 18217, 2023-09-10T01:19:17.898 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 22.469819276, 2023-09-10T01:19:17.898 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001233453 2023-09-10T01:19:17.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.899 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookuphash_latency": { 2023-09-10T01:19:17.899 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.899 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.900 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.900 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupino_latency": { 2023-09-10T01:19:17.900 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.901 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.901 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.901 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupname_latency": { 2023-09-10T01:19:17.902 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.902 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.902 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.903 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupparent_latency": { 2023-09-10T01:19:17.903 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.903 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.904 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.904 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupsnap_latency": { 2023-09-10T01:19:17.904 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.905 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.905 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.906 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lssnap_latency": { 2023-09-10T01:19:17.906 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.906 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.906 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.907 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mkdir_latency": { 2023-09-10T01:19:17.907 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.908 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.908 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.908 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mknod_latency": { 2023-09-10T01:19:17.909 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.909 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.909 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.910 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mksnap_latency": { 2023-09-10T01:19:17.910 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.910 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.911 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.911 INFO:teuthology.orchestra.run.smithi092.stdout: "req_open_latency": { 2023-09-10T01:19:17.912 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.912 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.912 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.913 INFO:teuthology.orchestra.run.smithi092.stdout: "req_readdir_latency": { 2023-09-10T01:19:17.913 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 43823, 2023-09-10T01:19:17.913 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 129.953068507, 2023-09-10T01:19:17.913 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002965407 2023-09-10T01:19:17.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.914 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rename_latency": { 2023-09-10T01:19:17.914 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1974, 2023-09-10T01:19:17.915 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 7.337282744, 2023-09-10T01:19:17.915 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.003716961 2023-09-10T01:19:17.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.916 INFO:teuthology.orchestra.run.smithi092.stdout: "req_renamesnap_latency": { 2023-09-10T01:19:17.916 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.916 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.916 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.917 INFO:teuthology.orchestra.run.smithi092.stdout: "req_snapdiff_latency": { 2023-09-10T01:19:17.917 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.917 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.918 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.918 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmdir_latency": { 2023-09-10T01:19:17.919 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.919 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.919 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.920 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmsnap_latency": { 2023-09-10T01:19:17.920 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.920 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.920 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.921 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmxattr_latency": { 2023-09-10T01:19:17.921 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.922 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.922 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.922 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setattr_latency": { 2023-09-10T01:19:17.923 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 22, 2023-09-10T01:19:17.923 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.355731570, 2023-09-10T01:19:17.923 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.016169616 2023-09-10T01:19:17.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.924 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setdirlayout_latency": { 2023-09-10T01:19:17.924 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.924 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.925 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.925 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setfilelock_latency": { 2023-09-10T01:19:17.926 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 328, 2023-09-10T01:19:17.926 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.484067491, 2023-09-10T01:19:17.926 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001475815 2023-09-10T01:19:17.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.927 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setlayout_latency": { 2023-09-10T01:19:17.927 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.927 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.928 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.928 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setxattr_latency": { 2023-09-10T01:19:17.928 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.929 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.929 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.929 INFO:teuthology.orchestra.run.smithi092.stdout: "req_symlink_latency": { 2023-09-10T01:19:17.930 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.930 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.930 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.931 INFO:teuthology.orchestra.run.smithi092.stdout: "req_unlink_latency": { 2023-09-10T01:19:17.931 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 11617, 2023-09-10T01:19:17.931 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 31.241118449, 2023-09-10T01:19:17.932 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002689258 2023-09-10T01:19:17.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.932 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_revoke_eviction": 0, 2023-09-10T01:19:17.933 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_acquisition_throttle": 0, 2023-09-10T01:19:17.933 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getvxattr_latency": { 2023-09-10T01:19:17.933 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.933 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.934 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.934 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:17.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.935 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessions": { 2023-09-10T01:19:17.935 INFO:teuthology.orchestra.run.smithi092.stdout: "session_count": 1, 2023-09-10T01:19:17.935 INFO:teuthology.orchestra.run.smithi092.stdout: "session_add": 3, 2023-09-10T01:19:17.936 INFO:teuthology.orchestra.run.smithi092.stdout: "session_remove": 2, 2023-09-10T01:19:17.936 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_open": 1, 2023-09-10T01:19:17.936 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_stale": 0, 2023-09-10T01:19:17.936 INFO:teuthology.orchestra.run.smithi092.stdout: "total_load": 19589, 2023-09-10T01:19:17.937 INFO:teuthology.orchestra.run.smithi092.stdout: "average_load": 19589, 2023-09-10T01:19:17.937 INFO:teuthology.orchestra.run.smithi092.stdout: "avg_session_uptime": 888, 2023-09-10T01:19:17.937 INFO:teuthology.orchestra.run.smithi092.stdout: "mdthresh_evicted": 0 2023-09-10T01:19:17.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.938 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2023-09-10T01:19:17.938 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:17.938 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:17.939 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:17.939 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:17.939 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:17.939 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:17.940 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:17.940 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:17.940 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:17.940 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:17.941 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:17.941 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:17.941 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:17.942 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:17.942 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:17.942 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:17.943 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:17.943 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:17.943 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:17.943 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:17.944 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:17.944 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:17.944 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:17.944 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:17.945 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:17.945 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:17.945 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:17.946 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:17.946 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:17.946 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:17.946 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:17.947 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2023-09-10T01:19:17.947 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:17.947 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:17.947 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:17.948 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:17.948 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 367521, 2023-09-10T01:19:17.948 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 850, 2023-09-10T01:19:17.949 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 352, 2023-09-10T01:19:17.949 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 4, 2023-09-10T01:19:17.949 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2023-09-10T01:19:17.949 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2023-09-10T01:19:17.950 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:17.950 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:17.950 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:17.951 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:17.951 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 10056, 2023-09-10T01:19:17.951 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 196, 2023-09-10T01:19:17.951 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:17.952 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:17.952 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:17.952 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2023-09-10T01:19:17.953 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 3856398, 2023-09-10T01:19:17.953 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 44077, 2023-09-10T01:19:17.953 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:17.954 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2023-09-10T01:19:17.954 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:17.954 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2023-09-10T01:19:17.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.955 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter": { 2023-09-10T01:19:17.955 INFO:teuthology.orchestra.run.smithi092.stdout: "op_active": 0, 2023-09-10T01:19:17.955 INFO:teuthology.orchestra.run.smithi092.stdout: "op_laggy": 0, 2023-09-10T01:19:17.956 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send": 44018, 2023-09-10T01:19:17.956 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send_bytes": 227661036, 2023-09-10T01:19:17.956 INFO:teuthology.orchestra.run.smithi092.stdout: "op_resend": 27, 2023-09-10T01:19:17.956 INFO:teuthology.orchestra.run.smithi092.stdout: "op_reply": 44018, 2023-09-10T01:19:17.957 INFO:teuthology.orchestra.run.smithi092.stdout: "op_latency": { 2023-09-10T01:19:17.957 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 44018, 2023-09-10T01:19:17.957 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 444.394464131, 2023-09-10T01:19:17.958 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.010095744 2023-09-10T01:19:17.958 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.958 INFO:teuthology.orchestra.run.smithi092.stdout: "op_inflight": 0, 2023-09-10T01:19:17.958 INFO:teuthology.orchestra.run.smithi092.stdout: "oplen_avg": { 2023-09-10T01:19:17.959 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 44018, 2023-09-10T01:19:17.959 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 44380 2023-09-10T01:19:17.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.959 INFO:teuthology.orchestra.run.smithi092.stdout: "op": 44018, 2023-09-10T01:19:17.960 INFO:teuthology.orchestra.run.smithi092.stdout: "op_r": 1429, 2023-09-10T01:19:17.960 INFO:teuthology.orchestra.run.smithi092.stdout: "op_w": 42589, 2023-09-10T01:19:17.960 INFO:teuthology.orchestra.run.smithi092.stdout: "op_rmw": 0, 2023-09-10T01:19:17.960 INFO:teuthology.orchestra.run.smithi092.stdout: "op_pg": 0, 2023-09-10T01:19:17.961 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_stat": 0, 2023-09-10T01:19:17.961 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_create": 10, 2023-09-10T01:19:17.961 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_read": 413, 2023-09-10T01:19:17.962 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_write": 18779, 2023-09-10T01:19:17.962 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writefull": 271, 2023-09-10T01:19:17.962 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writesame": 0, 2023-09-10T01:19:17.962 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_append": 0, 2023-09-10T01:19:17.963 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_zero": 0, 2023-09-10T01:19:17.963 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_truncate": 4, 2023-09-10T01:19:17.963 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_delete": 23307, 2023-09-10T01:19:17.963 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_mapext": 0, 2023-09-10T01:19:17.964 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_sparse_read": 0, 2023-09-10T01:19:17.964 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_clonerange": 0, 2023-09-10T01:19:17.964 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_getxattr": 1007, 2023-09-10T01:19:17.964 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_setxattr": 20, 2023-09-10T01:19:17.965 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_cmpxattr": 0, 2023-09-10T01:19:17.965 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_rmxattr": 0, 2023-09-10T01:19:17.965 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_resetxattrs": 0, 2023-09-10T01:19:17.966 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_call": 0, 2023-09-10T01:19:17.966 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_watch": 0, 2023-09-10T01:19:17.966 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_notify": 0, 2023-09-10T01:19:17.966 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_src_cmpxattr": 0, 2023-09-10T01:19:17.967 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls": 0, 2023-09-10T01:19:17.967 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls_filter": 0, 2023-09-10T01:19:17.967 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_other": 0, 2023-09-10T01:19:17.967 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_active": 0, 2023-09-10T01:19:17.968 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_send": 0, 2023-09-10T01:19:17.968 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_resend": 0, 2023-09-10T01:19:17.968 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_ping": 0, 2023-09-10T01:19:17.969 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_active": 0, 2023-09-10T01:19:17.969 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_send": 0, 2023-09-10T01:19:17.969 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_resend": 0, 2023-09-10T01:19:17.969 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_active": 0, 2023-09-10T01:19:17.970 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_send": 0, 2023-09-10T01:19:17.970 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_resend": 0, 2023-09-10T01:19:17.970 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_active": 0, 2023-09-10T01:19:17.971 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_send": 0, 2023-09-10T01:19:17.971 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_resend": 0, 2023-09-10T01:19:17.971 INFO:teuthology.orchestra.run.smithi092.stdout: "command_active": 0, 2023-09-10T01:19:17.972 INFO:teuthology.orchestra.run.smithi092.stdout: "command_send": 0, 2023-09-10T01:19:17.972 INFO:teuthology.orchestra.run.smithi092.stdout: "command_resend": 0, 2023-09-10T01:19:17.972 INFO:teuthology.orchestra.run.smithi092.stdout: "map_epoch": 112, 2023-09-10T01:19:17.973 INFO:teuthology.orchestra.run.smithi092.stdout: "map_full": 0, 2023-09-10T01:19:17.973 INFO:teuthology.orchestra.run.smithi092.stdout: "map_inc": 24, 2023-09-10T01:19:17.973 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_sessions": 12, 2023-09-10T01:19:17.973 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_open": 12, 2023-09-10T01:19:17.974 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_close": 0, 2023-09-10T01:19:17.974 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_laggy": 0, 2023-09-10T01:19:17.974 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_wr": 418, 2023-09-10T01:19:17.974 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_rd": 15, 2023-09-10T01:19:17.975 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_del": 136 2023-09-10T01:19:17.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.975 INFO:teuthology.orchestra.run.smithi092.stdout: "oft": { 2023-09-10T01:19:17.975 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_objs": 1, 2023-09-10T01:19:17.976 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_kv_pairs": 0, 2023-09-10T01:19:17.976 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_updates": 0, 2023-09-10T01:19:17.976 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_removes": 0 2023-09-10T01:19:17.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.977 INFO:teuthology.orchestra.run.smithi092.stdout: "osd-slow-ops": { 2023-09-10T01:19:17.977 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_ops_count": 0 2023-09-10T01:19:17.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.978 INFO:teuthology.orchestra.run.smithi092.stdout: "purge_queue": { 2023-09-10T01:19:17.978 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops": 0, 2023-09-10T01:19:17.978 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops_high_water": 16, 2023-09-10T01:19:17.978 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing": 0, 2023-09-10T01:19:17.979 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_high_water": 8, 2023-09-10T01:19:17.979 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed": 11617, 2023-09-10T01:19:17.979 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_item_in_journal": 0 2023-09-10T01:19:17.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.980 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:17.980 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:17.980 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2023-09-10T01:19:17.980 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:17.981 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 1697131, 2023-09-10T01:19:17.981 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 171571810, 2023-09-10T01:19:17.981 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:17.982 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 1697131, 2023-09-10T01:19:17.982 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:17.982 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:17.982 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 1697131, 2023-09-10T01:19:17.983 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 171571810, 2023-09-10T01:19:17.983 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:17.983 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.983 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.984 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.984 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:17.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.984 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_bytes": { 2023-09-10T01:19:17.985 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:17.985 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2023-09-10T01:19:17.985 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:17.986 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:17.986 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:17.986 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:17.986 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:17.987 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 44018, 2023-09-10T01:19:17.987 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 228930038, 2023-09-10T01:19:17.987 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 44018, 2023-09-10T01:19:17.987 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 228930038, 2023-09-10T01:19:17.988 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:17.988 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.988 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.988 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.989 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:17.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.989 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_ops": { 2023-09-10T01:19:17.989 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:17.990 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 1024, 2023-09-10T01:19:17.990 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:17.990 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2023-09-10T01:19:17.990 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2023-09-10T01:19:17.991 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:17.991 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:17.991 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 44018, 2023-09-10T01:19:17.992 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 44018, 2023-09-10T01:19:17.992 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 44018, 2023-09-10T01:19:17.992 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 44018, 2023-09-10T01:19:17.992 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:17.993 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.993 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.993 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.993 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:17.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.994 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle": { 2023-09-10T01:19:17.994 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:17.994 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2023-09-10T01:19:17.995 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:17.995 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 11617, 2023-09-10T01:19:17.995 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 1269317, 2023-09-10T01:19:17.995 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:17.996 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 11617, 2023-09-10T01:19:17.996 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:17.996 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:17.996 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 413, 2023-09-10T01:19:17.997 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 1269317, 2023-09-10T01:19:17.997 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:17.997 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:17.997 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:17.998 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:17.998 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:17.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:17.998 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle-0x55b9beaa2640": { 2023-09-10T01:19:17.999 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 385704, 2023-09-10T01:19:17.999 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2023-09-10T01:19:17.999 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:17.999 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 90406, 2023-09-10T01:19:18.000 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 225507193, 2023-09-10T01:19:18.000 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:18.000 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 90406, 2023-09-10T01:19:18.000 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:18.001 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:18.001 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 18366, 2023-09-10T01:19:18.001 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 225121489, 2023-09-10T01:19:18.001 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:18.002 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:18.002 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:18.002 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:18.003 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:18.003 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:18.003 INFO:teuthology.orchestra.run.smithi092.stdout:} 2023-09-10T01:19:18.146 INFO:tasks.check_counter:Daemon mds.h mds.exported=12 2023-09-10T01:19:18.146 INFO:tasks.check_counter:Daemon mds.h mds.imported=12 2023-09-10T01:19:18.147 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2023-09-10T01:19:18.147 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2023-09-10T01:19:18.147 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2023-09-10T01:19:18.148 DEBUG:tasks.check_counter:Getting stats from k 2023-09-10T01:19:18.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.k perf dump 2023-09-10T01:19:18.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:18 smithi092 ceph-mon[134067]: pgmap v506: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 787 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-09-10T01:19:18.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:18 smithi136 ceph-mon[133708]: pgmap v506: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 787 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-09-10T01:19:18.737 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.b/config 2023-09-10T01:19:18.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:18 smithi003 ceph-mon[97654]: pgmap v506: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 787 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-09-10T01:19:19.963 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2023-09-10T01:19:19.963 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:19.964 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 318, 2023-09-10T01:19:19.964 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 306, 2023-09-10T01:19:19.964 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 1715040, 2023-09-10T01:19:19.964 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 184096, 2023-09-10T01:19:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 4, 2023-09-10T01:19:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 0, 2023-09-10T01:19:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.050074781, 2023-09-10T01:19:19.966 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.014459906, 2023-09-10T01:19:19.966 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.035925535, 2023-09-10T01:19:19.966 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.007536956, 2023-09-10T01:19:19.967 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:19.967 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 305, 2023-09-10T01:19:19.967 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.012128482, 2023-09-10T01:19:19.967 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000039765 2023-09-10T01:19:19.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.968 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:19.968 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:19.969 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:19.969 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:19.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.969 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 1715040, 2023-09-10T01:19:19.970 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 184096 2023-09-10T01:19:19.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.970 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:19.971 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 7, 2023-09-10T01:19:19.971 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 236, 2023-09-10T01:19:19.971 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 3424, 2023-09-10T01:19:19.971 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 270736, 2023-09-10T01:19:19.972 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 6, 2023-09-10T01:19:19.972 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:19.972 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.011614421, 2023-09-10T01:19:19.972 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.007052037, 2023-09-10T01:19:19.973 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.002367550, 2023-09-10T01:19:19.973 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:19.973 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:19.974 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 234, 2023-09-10T01:19:19.974 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.024628543, 2023-09-10T01:19:19.974 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000105250 2023-09-10T01:19:19.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.975 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:19.975 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:19.975 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:19.976 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:19.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.976 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-09-10T01:19:19.976 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 270736 2023-09-10T01:19:19.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.977 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:19.977 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 0, 2023-09-10T01:19:19.978 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 2, 2023-09-10T01:19:19.978 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 0, 2023-09-10T01:19:19.978 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 0, 2023-09-10T01:19:19.978 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 2, 2023-09-10T01:19:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 1, 2023-09-10T01:19:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.000402646, 2023-09-10T01:19:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.000000000, 2023-09-10T01:19:19.980 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.000336888, 2023-09-10T01:19:19.980 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:19.980 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:19.980 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:19.981 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:19.981 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:19.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.982 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:19.982 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:19.982 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:19.982 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:19.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.983 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 0, 2023-09-10T01:19:19.983 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 0 2023-09-10T01:19:19.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.984 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2023-09-10T01:19:19.984 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 0, 2023-09-10T01:19:19.984 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2023-09-10T01:19:19.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:19.985 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2023-09-10T01:19:19.985 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:19.985 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:19.986 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:19.986 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:19.986 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:19.987 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:19.987 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:19.987 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:19.987 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:19.988 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:19.988 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:19.988 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:19.989 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:19.989 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:19.989 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:19.989 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:19.990 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:19.990 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:19.990 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:19.991 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:19.991 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:19.991 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:19.991 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:19.992 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:19.992 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:19.992 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:19.992 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:19.993 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:19.993 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:19.993 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2023-09-10T01:19:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:19.995 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:19.995 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:19.995 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 148754, 2023-09-10T01:19:19.996 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 31, 2023-09-10T01:19:19.996 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:19.996 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:19.996 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2023-09-10T01:19:19.997 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2023-09-10T01:19:19.997 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:19.997 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:19.998 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:19.998 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:19.998 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 0, 2023-09-10T01:19:19.998 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 0, 2023-09-10T01:19:19.999 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:19.999 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:19.999 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:19.999 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2023-09-10T01:19:20.000 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 928, 2023-09-10T01:19:20.000 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 2, 2023-09-10T01:19:20.000 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:20.001 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2023-09-10T01:19:20.001 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:20.001 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2023-09-10T01:19:20.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-09-10T01:19:20.002 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:20.002 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2023-09-10T01:19:20.002 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2023-09-10T01:19:20.002 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2023-09-10T01:19:20.003 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 321, 2023-09-10T01:19:20.003 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 1670201, 2023-09-10T01:19:20.003 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:20.004 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 321, 2023-09-10T01:19:20.004 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2023-09-10T01:19:20.004 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2023-09-10T01:19:20.004 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 321, 2023-09-10T01:19:20.005 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 1670201, 2023-09-10T01:19:20.005 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2023-09-10T01:19:20.005 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2023-09-10T01:19:20.006 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2023-09-10T01:19:20.006 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2023-09-10T01:19:20.006 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:20.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-09-10T01:19:20.007 INFO:teuthology.orchestra.run.smithi092.stdout:} 2023-09-10T01:19:20.394 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2023-09-10T01:19:20.394 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2023-09-10T01:19:20.395 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2023-09-10T01:19:20.395 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2023-09-10T01:19:20.395 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2023-09-10T01:19:20.396 DEBUG:tasks.check_counter:Getting stats from c 2023-09-10T01:19:20.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.c perf dump 2023-09-10T01:19:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:20 smithi136 ceph-mon[133708]: pgmap v507: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 785 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 88 KiB/s wr, 17 op/s 2023-09-10T01:19:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:20 smithi136 ceph-mon[133708]: osdmap e113: 12 total, 12 up, 12 in 2023-09-10T01:19:20.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:20 smithi092 ceph-mon[134067]: pgmap v507: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 785 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 88 KiB/s wr, 17 op/s 2023-09-10T01:19:20.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:20 smithi092 ceph-mon[134067]: osdmap e113: 12 total, 12 up, 12 in 2023-09-10T01:19:20.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:20 smithi003 ceph-mon[97654]: pgmap v507: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 785 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 88 KiB/s wr, 17 op/s 2023-09-10T01:19:20.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:20 smithi003 ceph-mon[97654]: osdmap e113: 12 total, 12 up, 12 in 2023-09-10T01:19:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T01:19:22.331 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2023-09-10T01:19:22.331 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:22.331 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 1946, 2023-09-10T01:19:22.332 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 1915, 2023-09-10T01:19:22.332 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 55897490, 2023-09-10T01:19:22.332 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 679771, 2023-09-10T01:19:22.332 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 7, 2023-09-10T01:19:22.333 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:22.333 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.222778761, 2023-09-10T01:19:22.333 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.054419674, 2023-09-10T01:19:22.334 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.104296039, 2023-09-10T01:19:22.334 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.059070854, 2023-09-10T01:19:22.334 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:22.334 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1914, 2023-09-10T01:19:22.335 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.110163432, 2023-09-10T01:19:22.335 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000057556 2023-09-10T01:19:22.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.336 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:22.336 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.336 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.336 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.337 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1838256, 2023-09-10T01:19:22.337 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 188080 2023-09-10T01:19:22.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.338 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:22.338 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 649, 2023-09-10T01:19:22.338 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 876, 2023-09-10T01:19:22.338 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 58890500, 2023-09-10T01:19:22.339 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 840812, 2023-09-10T01:19:22.339 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 9, 2023-09-10T01:19:22.339 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:22.340 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.093119471, 2023-09-10T01:19:22.340 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.018312905, 2023-09-10T01:19:22.340 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.051257074, 2023-09-10T01:19:22.340 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.016955170, 2023-09-10T01:19:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 874, 2023-09-10T01:19:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.050791436, 2023-09-10T01:19:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000058113 2023-09-10T01:19:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-09-10T01:19:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 641088 2023-09-10T01:19:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 1768, 2023-09-10T01:19:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 1770, 2023-09-10T01:19:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 113716561, 2023-09-10T01:19:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 549855, 2023-09-10T01:19:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 6, 2023-09-10T01:19:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 5, 2023-09-10T01:19:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.176291162, 2023-09-10T01:19:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.034581344, 2023-09-10T01:19:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.087457153, 2023-09-10T01:19:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.044334078, 2023-09-10T01:19:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1768, 2023-09-10T01:19:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.096290917, 2023-09-10T01:19:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000054463 2023-09-10T01:19:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 0, 2023-09-10T01:19:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 0 2023-09-10T01:19:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2023-09-10T01:19:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 1, 2023-09-10T01:19:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2023-09-10T01:19:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-MDSRank": { 2023-09-10T01:19:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2023-09-10T01:19:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2023-09-10T01:19:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 6792, 2023-09-10T01:19:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.163326738, 2023-09-10T01:19:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000024046 2023-09-10T01:19:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-PurgeQueue": { 2023-09-10T01:19:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2023-09-10T01:19:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2023-09-10T01:19:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2023-09-10T01:19:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: "request": 0, 2023-09-10T01:19:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: "reply": 0, 2023-09-10T01:19:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: "reply_latency": { 2023-09-10T01:19:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_reply": 0, 2023-09-10T01:19:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "forward": 0, 2023-09-10T01:19:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_complete": 0, 2023-09-10T01:19:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_keys": 0, 2023-09-10T01:19:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_commit": 0, 2023-09-10T01:19:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_split": 0, 2023-09-10T01:19:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_merge": 0, 2023-09-10T01:19:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes": 12735, 2023-09-10T01:19:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_top": 153, 2023-09-10T01:19:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_bottom": 8354, 2023-09-10T01:19:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pin_tail": 4228, 2023-09-10T01:19:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pinned": 12516, 2023-09-10T01:19:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_expired": 12, 2023-09-10T01:19:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_with_caps": 0, 2023-09-10T01:19:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "caps": 0, 2023-09-10T01:19:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "subtrees": 16, 2023-09-10T01:19:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse": 0, 2023-09-10T01:19:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_hit": 0, 2023-09-10T01:19:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_forward": 0, 2023-09-10T01:19:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_discover": 0, 2023-09-10T01:19:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_dir_fetch": 0, 2023-09-10T01:19:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_remote_ino": 0, 2023-09-10T01:19:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_lock": 0, 2023-09-10T01:19:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "load_cent": 0, 2023-09-10T01:19:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "q": 0, 2023-09-10T01:19:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "exported": 0, 2023-09-10T01:19:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "exported_inodes": 0, 2023-09-10T01:19:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "imported": 0, 2023-09-10T01:19:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "imported_inodes": 0, 2023-09-10T01:19:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_dir_fetch": 0, 2023-09-10T01:19:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_backtrace_fetch": 0, 2023-09-10T01:19:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_peer_discover": 0, 2023-09-10T01:19:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rfiles": 0, 2023-09-10T01:19:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rbytes": 0, 2023-09-10T01:19:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rsnaps": 0, 2023-09-10T01:19:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_fetch": 0, 2023-09-10T01:19:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_set_tag": 0, 2023-09-10T01:19:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_repaired": 0, 2023-09-10T01:19:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_inotable_repaired": 0, 2023-09-10T01:19:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_inodes": 0, 2023-09-10T01:19:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_base_inodes": 0, 2023-09-10T01:19:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dirfrag_rstats": 0, 2023-09-10T01:19:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_file_inodes": 0, 2023-09-10T01:19:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_inode_file_caps": 0, 2023-09-10T01:19:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_revoke": 0, 2023-09-10T01:19:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_grant": 0, 2023-09-10T01:19:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_trunc": 0, 2023-09-10T01:19:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-09-10T01:19:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flush_ack": 0, 2023-09-10T01:19:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps": 0, 2023-09-10T01:19:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps_dirty": 0, 2023-09-10T01:19:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_cap_release": 0, 2023-09-10T01:19:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "process_request_cap_release": 0 2023-09-10T01:19:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache": { 2023-09-10T01:19:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update": 0, 2023-09-10T01:19:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update_receipt": 0, 2023-09-10T01:19:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_try_discover": 0, 2023-09-10T01:19:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_send_discover": 0, 2023-09-10T01:19:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_handle_discover": 0, 2023-09-10T01:19:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays": 0, 2023-09-10T01:19:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_delayed": 0, 2023-09-10T01:19:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_enqueuing": 0, 2023-09-10T01:19:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_created": 0, 2023-09-10T01:19:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_enqueued": 0, 2023-09-10T01:19:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_reintegrated": 0, 2023-09-10T01:19:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_migrated": 0, 2023-09-10T01:19:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_processing": 0, 2023-09-10T01:19:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_enqueued": 0, 2023-09-10T01:19:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_prioritized": 0, 2023-09-10T01:19:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_started": 0, 2023-09-10T01:19:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_completed": 0, 2023-09-10T01:19:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_enqueue_scrub": 0, 2023-09-10T01:19:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_exportdir": 0, 2023-09-10T01:19:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_flush": 0, 2023-09-10T01:19:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragmentdir": 0, 2023-09-10T01:19:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragstats": 0, 2023-09-10T01:19:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_inodestats": 0 2023-09-10T01:19:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log": { 2023-09-10T01:19:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "evlrg": 0, 2023-09-10T01:19:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "evadd": 0, 2023-09-10T01:19:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "evex": 0, 2023-09-10T01:19:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "evtrm": 0, 2023-09-10T01:19:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "ev": 90410, 2023-09-10T01:19:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "evexg": 0, 2023-09-10T01:19:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "evexd": 0, 2023-09-10T01:19:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "segadd": 0, 2023-09-10T01:19:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "segex": 0, 2023-09-10T01:19:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "segtrm": 0, 2023-09-10T01:19:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "seg": 115, 2023-09-10T01:19:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "segmjr": 9, 2023-09-10T01:19:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "segexg": 0, 2023-09-10T01:19:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "segexd": 0, 2023-09-10T01:19:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "expos": 4194304, 2023-09-10T01:19:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "wrpos": 229706380, 2023-09-10T01:19:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "rdpos": 229703219, 2023-09-10T01:19:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "jlat": { 2023-09-10T01:19:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "replayed": 90410 2023-09-10T01:19:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mem": { 2023-09-10T01:19:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: "ino": 885, 2023-09-10T01:19:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "ino+": 12772, 2023-09-10T01:19:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "ino-": 11887, 2023-09-10T01:19:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "dir": 81, 2023-09-10T01:19:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "dir+": 117, 2023-09-10T01:19:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "dir-": 36, 2023-09-10T01:19:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "dn": 12735, 2023-09-10T01:19:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "dn+": 13048, 2023-09-10T01:19:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "dn-": 313, 2023-09-10T01:19:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "cap": 0, 2023-09-10T01:19:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "cap+": 0, 2023-09-10T01:19:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "cap-": 0, 2023-09-10T01:19:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "rss": 68336, 2023-09-10T01:19:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "heap": 207152 2023-09-10T01:19:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server": { 2023-09-10T01:19:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_client_request": 0, 2023-09-10T01:19:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_server_request": 0, 2023-09-10T01:19:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_request": 0, 2023-09-10T01:19:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_session": 0, 2023-09-10T01:19:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_peer_request": 0, 2023-09-10T01:19:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "req_create_latency": { 2023-09-10T01:19:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getattr_latency": { 2023-09-10T01:19:22.395 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.395 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.395 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.396 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getfilelock_latency": { 2023-09-10T01:19:22.396 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.396 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.396 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "req_link_latency": { 2023-09-10T01:19:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookup_latency": { 2023-09-10T01:19:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookuphash_latency": { 2023-09-10T01:19:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupino_latency": { 2023-09-10T01:19:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupname_latency": { 2023-09-10T01:19:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupparent_latency": { 2023-09-10T01:19:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupsnap_latency": { 2023-09-10T01:19:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.406 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lssnap_latency": { 2023-09-10T01:19:22.406 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.406 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.407 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.407 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mkdir_latency": { 2023-09-10T01:19:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mknod_latency": { 2023-09-10T01:19:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mksnap_latency": { 2023-09-10T01:19:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "req_open_latency": { 2023-09-10T01:19:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "req_readdir_latency": { 2023-09-10T01:19:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rename_latency": { 2023-09-10T01:19:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "req_renamesnap_latency": { 2023-09-10T01:19:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "req_snapdiff_latency": { 2023-09-10T01:19:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmdir_latency": { 2023-09-10T01:19:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmsnap_latency": { 2023-09-10T01:19:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmxattr_latency": { 2023-09-10T01:19:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setattr_latency": { 2023-09-10T01:19:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setdirlayout_latency": { 2023-09-10T01:19:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setfilelock_latency": { 2023-09-10T01:19:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setlayout_latency": { 2023-09-10T01:19:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setxattr_latency": { 2023-09-10T01:19:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "req_symlink_latency": { 2023-09-10T01:19:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "req_unlink_latency": { 2023-09-10T01:19:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_revoke_eviction": 0, 2023-09-10T01:19:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_acquisition_throttle": 0, 2023-09-10T01:19:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getvxattr_latency": { 2023-09-10T01:19:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessions": { 2023-09-10T01:19:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "session_count": 1, 2023-09-10T01:19:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "session_add": 3, 2023-09-10T01:19:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "session_remove": 2, 2023-09-10T01:19:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_open": 2, 2023-09-10T01:19:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_stale": 0, 2023-09-10T01:19:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "total_load": 0, 2023-09-10T01:19:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "average_load": 0, 2023-09-10T01:19:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "avg_session_uptime": 4678, 2023-09-10T01:19:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "mdthresh_evicted": 0 2023-09-10T01:19:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2023-09-10T01:19:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2023-09-10T01:19:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 272184, 2023-09-10T01:19:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 1070, 2023-09-10T01:19:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 88, 2023-09-10T01:19:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 1, 2023-09-10T01:19:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2023-09-10T01:19:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2023-09-10T01:19:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 9816, 2023-09-10T01:19:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 190, 2023-09-10T01:19:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2023-09-10T01:19:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 11317202, 2023-09-10T01:19:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 73926, 2023-09-10T01:19:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2023-09-10T01:19:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2023-09-10T01:19:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter": { 2023-09-10T01:19:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "op_active": 0, 2023-09-10T01:19:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "op_laggy": 0, 2023-09-10T01:19:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send": 3991, 2023-09-10T01:19:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send_bytes": 21, 2023-09-10T01:19:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "op_resend": 3, 2023-09-10T01:19:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "op_reply": 3991, 2023-09-10T01:19:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "op_latency": { 2023-09-10T01:19:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3991, 2023-09-10T01:19:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 6.089864137, 2023-09-10T01:19:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001525899 2023-09-10T01:19:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "op_inflight": 0, 2023-09-10T01:19:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "oplen_avg": { 2023-09-10T01:19:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3991, 2023-09-10T01:19:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 3992 2023-09-10T01:19:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "op": 3991, 2023-09-10T01:19:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "op_r": 3991, 2023-09-10T01:19:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "op_w": 0, 2023-09-10T01:19:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "op_rmw": 0, 2023-09-10T01:19:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "op_pg": 0, 2023-09-10T01:19:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_stat": 2164, 2023-09-10T01:19:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_create": 0, 2023-09-10T01:19:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_read": 1825, 2023-09-10T01:19:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_write": 0, 2023-09-10T01:19:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writefull": 0, 2023-09-10T01:19:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writesame": 0, 2023-09-10T01:19:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_append": 0, 2023-09-10T01:19:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_zero": 0, 2023-09-10T01:19:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_truncate": 0, 2023-09-10T01:19:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_delete": 0, 2023-09-10T01:19:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_mapext": 0, 2023-09-10T01:19:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_sparse_read": 0, 2023-09-10T01:19:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_clonerange": 0, 2023-09-10T01:19:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_getxattr": 1, 2023-09-10T01:19:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_setxattr": 0, 2023-09-10T01:19:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_cmpxattr": 0, 2023-09-10T01:19:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_rmxattr": 0, 2023-09-10T01:19:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_resetxattrs": 0, 2023-09-10T01:19:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_call": 0, 2023-09-10T01:19:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_watch": 0, 2023-09-10T01:19:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_notify": 0, 2023-09-10T01:19:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_src_cmpxattr": 0, 2023-09-10T01:19:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls": 0, 2023-09-10T01:19:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls_filter": 0, 2023-09-10T01:19:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_other": 0, 2023-09-10T01:19:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_active": 0, 2023-09-10T01:19:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_send": 0, 2023-09-10T01:19:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_resend": 0, 2023-09-10T01:19:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_ping": 0, 2023-09-10T01:19:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_active": 0, 2023-09-10T01:19:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_send": 0, 2023-09-10T01:19:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_resend": 0, 2023-09-10T01:19:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_active": 0, 2023-09-10T01:19:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_send": 0, 2023-09-10T01:19:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_resend": 0, 2023-09-10T01:19:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_active": 0, 2023-09-10T01:19:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_send": 0, 2023-09-10T01:19:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_resend": 0, 2023-09-10T01:19:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "command_active": 0, 2023-09-10T01:19:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "command_send": 0, 2023-09-10T01:19:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "command_resend": 0, 2023-09-10T01:19:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "map_epoch": 113, 2023-09-10T01:19:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "map_full": 0, 2023-09-10T01:19:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "map_inc": 25, 2023-09-10T01:19:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_sessions": 10, 2023-09-10T01:19:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_open": 10, 2023-09-10T01:19:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_close": 0, 2023-09-10T01:19:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_laggy": 0, 2023-09-10T01:19:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_wr": 0, 2023-09-10T01:19:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_rd": 2, 2023-09-10T01:19:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_del": 0 2023-09-10T01:19:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "oft": { 2023-09-10T01:19:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_objs": 0, 2023-09-10T01:19:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_kv_pairs": 0, 2023-09-10T01:19:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_updates": 0, 2023-09-10T01:19:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_removes": 0 2023-09-10T01:19:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "osd-slow-ops": { 2023-09-10T01:19:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_ops_count": 0 2023-09-10T01:19:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "purge_queue": { 2023-09-10T01:19:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops": 0, 2023-09-10T01:19:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops_high_water": 0, 2023-09-10T01:19:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing": 0, 2023-09-10T01:19:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_high_water": 0, 2023-09-10T01:19:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed": 0, 2023-09-10T01:19:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_item_in_journal": 0 2023-09-10T01:19:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2023-09-10T01:19:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 4359, 2023-09-10T01:19:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 228090759, 2023-09-10T01:19:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 4359, 2023-09-10T01:19:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 4359, 2023-09-10T01:19:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 228090759, 2023-09-10T01:19:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.481 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.481 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.482 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_bytes": { 2023-09-10T01:19:22.482 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:22.482 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2023-09-10T01:19:22.483 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:22.483 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:22.483 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:22.483 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:22.484 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:22.484 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 3991, 2023-09-10T01:19:22.484 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 225560381, 2023-09-10T01:19:22.484 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 3991, 2023-09-10T01:19:22.485 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 225560381, 2023-09-10T01:19:22.485 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:22.485 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.485 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.486 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.486 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.487 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_ops": { 2023-09-10T01:19:22.487 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:22.487 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 1024, 2023-09-10T01:19:22.487 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:22.488 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:22.488 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:22.488 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:22.488 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:22.489 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 3991, 2023-09-10T01:19:22.489 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 3991, 2023-09-10T01:19:22.489 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 3991, 2023-09-10T01:19:22.489 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 3991, 2023-09-10T01:19:22.490 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:22.490 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.490 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.490 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.491 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.491 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle": { 2023-09-10T01:19:22.491 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:22.492 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2023-09-10T01:19:22.492 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:22.492 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:22.493 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:22.493 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:22.493 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:22.493 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:22.494 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:22.494 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2023-09-10T01:19:22.494 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2023-09-10T01:19:22.494 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:22.495 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.495 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.495 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.495 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:22.496 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle-0x5625a790a960": { 2023-09-10T01:19:22.496 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:22.497 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2023-09-10T01:19:22.497 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:22.497 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:22.497 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:22.498 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:22.498 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:22.498 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:22.498 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:22.499 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2023-09-10T01:19:22.499 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2023-09-10T01:19:22.499 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:22.500 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:22.500 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:22.500 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:22.500 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.501 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:22.501 INFO:teuthology.orchestra.run.smithi136.stdout:} 2023-09-10T01:19:22.607 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:22 smithi136 ceph-mon[133708]: pgmap v509: 129 pgs: 129 active+clean; 774 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 267 KiB/s wr, 40 op/s 2023-09-10T01:19:22.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:22 smithi092 ceph-mon[134067]: pgmap v509: 129 pgs: 129 active+clean; 774 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 267 KiB/s wr, 40 op/s 2023-09-10T01:19:22.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:22 smithi003 ceph-mon[97654]: pgmap v509: 129 pgs: 129 active+clean; 774 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 267 KiB/s wr, 40 op/s 2023-09-10T01:19:24.054 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2023-09-10T01:19:24.054 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2023-09-10T01:19:24.055 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2023-09-10T01:19:24.055 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2023-09-10T01:19:24.055 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2023-09-10T01:19:24.056 DEBUG:tasks.check_counter:Getting stats from f 2023-09-10T01:19:24.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.f perf dump 2023-09-10T01:19:24.617 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T01:19:24.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:24 smithi136 ceph-mon[133708]: pgmap v510: 129 pgs: 129 active+clean; 774 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 267 KiB/s wr, 40 op/s 2023-09-10T01:19:24.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:24 smithi136 ceph-mon[133708]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:19:24.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:24 smithi092 ceph-mon[134067]: pgmap v510: 129 pgs: 129 active+clean; 774 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 267 KiB/s wr, 40 op/s 2023-09-10T01:19:24.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:24 smithi092 ceph-mon[134067]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:19:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:24 smithi003 ceph-mon[97654]: pgmap v510: 129 pgs: 129 active+clean; 774 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 267 KiB/s wr, 40 op/s 2023-09-10T01:19:24.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:24 smithi003 ceph-mon[97654]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-09-10T01:19:25.945 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2023-09-10T01:19:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 1123, 2023-09-10T01:19:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 1123, 2023-09-10T01:19:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 100531794, 2023-09-10T01:19:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 348521, 2023-09-10T01:19:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 7, 2023-09-10T01:19:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.138199980, 2023-09-10T01:19:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.023394744, 2023-09-10T01:19:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.074044485, 2023-09-10T01:19:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.032107858, 2023-09-10T01:19:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1122, 2023-09-10T01:19:25.950 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.073598538, 2023-09-10T01:19:25.950 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000065595 2023-09-10T01:19:25.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.950 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:25.951 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:25.951 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:25.951 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:25.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.952 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-09-10T01:19:25.952 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 448 2023-09-10T01:19:25.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 1284, 2023-09-10T01:19:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 1514, 2023-09-10T01:19:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 117776060, 2023-09-10T01:19:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 953811, 2023-09-10T01:19:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 8, 2023-09-10T01:19:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.149385118, 2023-09-10T01:19:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.033992819, 2023-09-10T01:19:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.072921941, 2023-09-10T01:19:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.032365752, 2023-09-10T01:19:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1511, 2023-09-10T01:19:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.091425368, 2023-09-10T01:19:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000060506 2023-09-10T01:19:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 320, 2023-09-10T01:19:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 555104 2023-09-10T01:19:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 2036, 2023-09-10T01:19:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 2004, 2023-09-10T01:19:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 82052225, 2023-09-10T01:19:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 707628, 2023-09-10T01:19:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 5, 2023-09-10T01:19:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 4, 2023-09-10T01:19:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.243922196, 2023-09-10T01:19:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.053829231, 2023-09-10T01:19:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.124950071, 2023-09-10T01:19:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.061363980, 2023-09-10T01:19:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 2003, 2023-09-10T01:19:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.120568920, 2023-09-10T01:19:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000060194 2023-09-10T01:19:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1843568, 2023-09-10T01:19:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 188880 2023-09-10T01:19:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2023-09-10T01:19:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 1, 2023-09-10T01:19:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2023-09-10T01:19:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-MDSRank": { 2023-09-10T01:19:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2023-09-10T01:19:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2023-09-10T01:19:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 6907, 2023-09-10T01:19:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.175087395, 2023-09-10T01:19:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000025349 2023-09-10T01:19:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-PurgeQueue": { 2023-09-10T01:19:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2023-09-10T01:19:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2023-09-10T01:19:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2023-09-10T01:19:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "request": 0, 2023-09-10T01:19:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "reply": 0, 2023-09-10T01:19:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "reply_latency": { 2023-09-10T01:19:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_reply": 0, 2023-09-10T01:19:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "forward": 0, 2023-09-10T01:19:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_complete": 0, 2023-09-10T01:19:25.975 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_keys": 0, 2023-09-10T01:19:25.975 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_commit": 0, 2023-09-10T01:19:25.975 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_split": 0, 2023-09-10T01:19:25.975 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_merge": 0, 2023-09-10T01:19:25.976 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes": 24334, 2023-09-10T01:19:25.976 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_top": 1780, 2023-09-10T01:19:25.976 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_bottom": 9552, 2023-09-10T01:19:25.977 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pin_tail": 13002, 2023-09-10T01:19:25.977 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pinned": 21791, 2023-09-10T01:19:25.977 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_expired": 126, 2023-09-10T01:19:25.977 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_with_caps": 0, 2023-09-10T01:19:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: "caps": 0, 2023-09-10T01:19:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: "subtrees": 9, 2023-09-10T01:19:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse": 0, 2023-09-10T01:19:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_hit": 0, 2023-09-10T01:19:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_forward": 0, 2023-09-10T01:19:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_discover": 0, 2023-09-10T01:19:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_dir_fetch": 0, 2023-09-10T01:19:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_remote_ino": 0, 2023-09-10T01:19:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_lock": 0, 2023-09-10T01:19:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: "load_cent": 0, 2023-09-10T01:19:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: "q": 0, 2023-09-10T01:19:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: "exported": 0, 2023-09-10T01:19:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: "exported_inodes": 0, 2023-09-10T01:19:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: "imported": 0, 2023-09-10T01:19:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: "imported_inodes": 0, 2023-09-10T01:19:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_dir_fetch": 0, 2023-09-10T01:19:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_backtrace_fetch": 0, 2023-09-10T01:19:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_peer_discover": 0, 2023-09-10T01:19:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rfiles": 0, 2023-09-10T01:19:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rbytes": 0, 2023-09-10T01:19:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rsnaps": 0, 2023-09-10T01:19:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_fetch": 0, 2023-09-10T01:19:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_set_tag": 0, 2023-09-10T01:19:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_repaired": 0, 2023-09-10T01:19:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_inotable_repaired": 0, 2023-09-10T01:19:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_inodes": 0, 2023-09-10T01:19:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_base_inodes": 0, 2023-09-10T01:19:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dirfrag_rstats": 0, 2023-09-10T01:19:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_file_inodes": 0, 2023-09-10T01:19:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_inode_file_caps": 0, 2023-09-10T01:19:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_revoke": 0, 2023-09-10T01:19:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_grant": 0, 2023-09-10T01:19:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_trunc": 0, 2023-09-10T01:19:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-09-10T01:19:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flush_ack": 0, 2023-09-10T01:19:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps": 0, 2023-09-10T01:19:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps_dirty": 0, 2023-09-10T01:19:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_cap_release": 0, 2023-09-10T01:19:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "process_request_cap_release": 0 2023-09-10T01:19:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache": { 2023-09-10T01:19:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update": 0, 2023-09-10T01:19:25.990 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update_receipt": 0, 2023-09-10T01:19:25.990 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_try_discover": 0, 2023-09-10T01:19:25.990 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_send_discover": 0, 2023-09-10T01:19:25.990 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_handle_discover": 0, 2023-09-10T01:19:25.991 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays": 0, 2023-09-10T01:19:25.991 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_delayed": 0, 2023-09-10T01:19:25.991 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_enqueuing": 0, 2023-09-10T01:19:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_created": 0, 2023-09-10T01:19:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_enqueued": 0, 2023-09-10T01:19:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_reintegrated": 0, 2023-09-10T01:19:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_migrated": 0, 2023-09-10T01:19:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_processing": 0, 2023-09-10T01:19:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_enqueued": 0, 2023-09-10T01:19:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_prioritized": 0, 2023-09-10T01:19:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_started": 0, 2023-09-10T01:19:25.994 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_completed": 0, 2023-09-10T01:19:25.994 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_enqueue_scrub": 0, 2023-09-10T01:19:25.994 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_exportdir": 0, 2023-09-10T01:19:25.995 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_flush": 0, 2023-09-10T01:19:25.995 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragmentdir": 0, 2023-09-10T01:19:25.995 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragstats": 0, 2023-09-10T01:19:25.995 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_inodestats": 0 2023-09-10T01:19:25.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:25.996 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log": { 2023-09-10T01:19:25.996 INFO:teuthology.orchestra.run.smithi136.stdout: "evlrg": 0, 2023-09-10T01:19:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: "evadd": 0, 2023-09-10T01:19:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: "evex": 0, 2023-09-10T01:19:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: "evtrm": 0, 2023-09-10T01:19:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: "ev": 110679, 2023-09-10T01:19:25.998 INFO:teuthology.orchestra.run.smithi136.stdout: "evexg": 0, 2023-09-10T01:19:25.998 INFO:teuthology.orchestra.run.smithi136.stdout: "evexd": 0, 2023-09-10T01:19:25.998 INFO:teuthology.orchestra.run.smithi136.stdout: "segadd": 0, 2023-09-10T01:19:25.998 INFO:teuthology.orchestra.run.smithi136.stdout: "segex": 0, 2023-09-10T01:19:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segtrm": 0, 2023-09-10T01:19:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: "seg": 130, 2023-09-10T01:19:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segmjr": 11, 2023-09-10T01:19:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: "segexg": 0, 2023-09-10T01:19:26.000 INFO:teuthology.orchestra.run.smithi136.stdout: "segexd": 0, 2023-09-10T01:19:26.000 INFO:teuthology.orchestra.run.smithi136.stdout: "expos": 21986127, 2023-09-10T01:19:26.000 INFO:teuthology.orchestra.run.smithi136.stdout: "wrpos": 301565391, 2023-09-10T01:19:26.001 INFO:teuthology.orchestra.run.smithi136.stdout: "rdpos": 301565330, 2023-09-10T01:19:26.001 INFO:teuthology.orchestra.run.smithi136.stdout: "jlat": { 2023-09-10T01:19:26.001 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.001 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.002 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.002 INFO:teuthology.orchestra.run.smithi136.stdout: "replayed": 110679 2023-09-10T01:19:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mem": { 2023-09-10T01:19:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: "ino": 964, 2023-09-10T01:19:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: "ino+": 24309, 2023-09-10T01:19:26.004 INFO:teuthology.orchestra.run.smithi136.stdout: "ino-": 23345, 2023-09-10T01:19:26.004 INFO:teuthology.orchestra.run.smithi136.stdout: "dir": 80, 2023-09-10T01:19:26.004 INFO:teuthology.orchestra.run.smithi136.stdout: "dir+": 95, 2023-09-10T01:19:26.005 INFO:teuthology.orchestra.run.smithi136.stdout: "dir-": 15, 2023-09-10T01:19:26.005 INFO:teuthology.orchestra.run.smithi136.stdout: "dn": 24436, 2023-09-10T01:19:26.005 INFO:teuthology.orchestra.run.smithi136.stdout: "dn+": 24730, 2023-09-10T01:19:26.005 INFO:teuthology.orchestra.run.smithi136.stdout: "dn-": 294, 2023-09-10T01:19:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: "cap": 0, 2023-09-10T01:19:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: "cap+": 0, 2023-09-10T01:19:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: "cap-": 0, 2023-09-10T01:19:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "rss": 74120, 2023-09-10T01:19:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "heap": 207152 2023-09-10T01:19:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server": { 2023-09-10T01:19:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_client_request": 0, 2023-09-10T01:19:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_server_request": 0, 2023-09-10T01:19:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_request": 0, 2023-09-10T01:19:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_session": 0, 2023-09-10T01:19:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_peer_request": 0, 2023-09-10T01:19:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "req_create_latency": { 2023-09-10T01:19:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getattr_latency": { 2023-09-10T01:19:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getfilelock_latency": { 2023-09-10T01:19:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "req_link_latency": { 2023-09-10T01:19:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookup_latency": { 2023-09-10T01:19:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookuphash_latency": { 2023-09-10T01:19:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupino_latency": { 2023-09-10T01:19:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupname_latency": { 2023-09-10T01:19:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupparent_latency": { 2023-09-10T01:19:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupsnap_latency": { 2023-09-10T01:19:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lssnap_latency": { 2023-09-10T01:19:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mkdir_latency": { 2023-09-10T01:19:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mknod_latency": { 2023-09-10T01:19:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mksnap_latency": { 2023-09-10T01:19:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "req_open_latency": { 2023-09-10T01:19:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "req_readdir_latency": { 2023-09-10T01:19:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rename_latency": { 2023-09-10T01:19:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "req_renamesnap_latency": { 2023-09-10T01:19:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "req_snapdiff_latency": { 2023-09-10T01:19:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmdir_latency": { 2023-09-10T01:19:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmsnap_latency": { 2023-09-10T01:19:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmxattr_latency": { 2023-09-10T01:19:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setattr_latency": { 2023-09-10T01:19:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setdirlayout_latency": { 2023-09-10T01:19:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setfilelock_latency": { 2023-09-10T01:19:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setlayout_latency": { 2023-09-10T01:19:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setxattr_latency": { 2023-09-10T01:19:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "req_symlink_latency": { 2023-09-10T01:19:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "req_unlink_latency": { 2023-09-10T01:19:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_revoke_eviction": 0, 2023-09-10T01:19:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_acquisition_throttle": 0, 2023-09-10T01:19:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getvxattr_latency": { 2023-09-10T01:19:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessions": { 2023-09-10T01:19:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "session_count": 1, 2023-09-10T01:19:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "session_add": 1, 2023-09-10T01:19:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "session_remove": 0, 2023-09-10T01:19:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_open": 1, 2023-09-10T01:19:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_stale": 0, 2023-09-10T01:19:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "total_load": 0, 2023-09-10T01:19:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "average_load": 0, 2023-09-10T01:19:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "avg_session_uptime": 2826, 2023-09-10T01:19:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "mdthresh_evicted": 0 2023-09-10T01:19:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2023-09-10T01:19:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2023-09-10T01:19:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 257470, 2023-09-10T01:19:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 1056, 2023-09-10T01:19:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2023-09-10T01:19:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2023-09-10T01:19:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 9816, 2023-09-10T01:19:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 190, 2023-09-10T01:19:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2023-09-10T01:19:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 19471552, 2023-09-10T01:19:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 120264, 2023-09-10T01:19:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2023-09-10T01:19:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:26.073 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2023-09-10T01:19:26.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.073 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter": { 2023-09-10T01:19:26.074 INFO:teuthology.orchestra.run.smithi136.stdout: "op_active": 0, 2023-09-10T01:19:26.074 INFO:teuthology.orchestra.run.smithi136.stdout: "op_laggy": 0, 2023-09-10T01:19:26.074 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send": 4069, 2023-09-10T01:19:26.074 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send_bytes": 21, 2023-09-10T01:19:26.075 INFO:teuthology.orchestra.run.smithi136.stdout: "op_resend": 3, 2023-09-10T01:19:26.075 INFO:teuthology.orchestra.run.smithi136.stdout: "op_reply": 4069, 2023-09-10T01:19:26.075 INFO:teuthology.orchestra.run.smithi136.stdout: "op_latency": { 2023-09-10T01:19:26.076 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 4069, 2023-09-10T01:19:26.076 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 7.535831766, 2023-09-10T01:19:26.076 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001852010 2023-09-10T01:19:26.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.077 INFO:teuthology.orchestra.run.smithi136.stdout: "op_inflight": 0, 2023-09-10T01:19:26.077 INFO:teuthology.orchestra.run.smithi136.stdout: "oplen_avg": { 2023-09-10T01:19:26.077 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 4069, 2023-09-10T01:19:26.077 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 4070 2023-09-10T01:19:26.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.078 INFO:teuthology.orchestra.run.smithi136.stdout: "op": 4069, 2023-09-10T01:19:26.078 INFO:teuthology.orchestra.run.smithi136.stdout: "op_r": 4069, 2023-09-10T01:19:26.079 INFO:teuthology.orchestra.run.smithi136.stdout: "op_w": 0, 2023-09-10T01:19:26.079 INFO:teuthology.orchestra.run.smithi136.stdout: "op_rmw": 0, 2023-09-10T01:19:26.079 INFO:teuthology.orchestra.run.smithi136.stdout: "op_pg": 0, 2023-09-10T01:19:26.079 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_stat": 2172, 2023-09-10T01:19:26.080 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_create": 0, 2023-09-10T01:19:26.080 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_read": 1895, 2023-09-10T01:19:26.080 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_write": 0, 2023-09-10T01:19:26.080 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writefull": 0, 2023-09-10T01:19:26.081 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writesame": 0, 2023-09-10T01:19:26.081 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_append": 0, 2023-09-10T01:19:26.081 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_zero": 0, 2023-09-10T01:19:26.082 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_truncate": 0, 2023-09-10T01:19:26.082 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_delete": 0, 2023-09-10T01:19:26.082 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_mapext": 0, 2023-09-10T01:19:26.082 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_sparse_read": 0, 2023-09-10T01:19:26.083 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_clonerange": 0, 2023-09-10T01:19:26.083 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_getxattr": 1, 2023-09-10T01:19:26.083 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_setxattr": 0, 2023-09-10T01:19:26.084 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_cmpxattr": 0, 2023-09-10T01:19:26.084 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_rmxattr": 0, 2023-09-10T01:19:26.084 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_resetxattrs": 0, 2023-09-10T01:19:26.084 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_call": 0, 2023-09-10T01:19:26.085 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_watch": 0, 2023-09-10T01:19:26.085 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_notify": 0, 2023-09-10T01:19:26.085 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_src_cmpxattr": 0, 2023-09-10T01:19:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls": 0, 2023-09-10T01:19:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls_filter": 0, 2023-09-10T01:19:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_other": 0, 2023-09-10T01:19:26.086 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_active": 0, 2023-09-10T01:19:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_send": 0, 2023-09-10T01:19:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_resend": 0, 2023-09-10T01:19:26.087 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_ping": 0, 2023-09-10T01:19:26.088 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_active": 0, 2023-09-10T01:19:26.088 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_send": 0, 2023-09-10T01:19:26.088 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_resend": 0, 2023-09-10T01:19:26.088 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_active": 0, 2023-09-10T01:19:26.089 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_send": 0, 2023-09-10T01:19:26.089 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_resend": 0, 2023-09-10T01:19:26.089 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_active": 0, 2023-09-10T01:19:26.089 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_send": 0, 2023-09-10T01:19:26.090 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_resend": 0, 2023-09-10T01:19:26.090 INFO:teuthology.orchestra.run.smithi136.stdout: "command_active": 0, 2023-09-10T01:19:26.090 INFO:teuthology.orchestra.run.smithi136.stdout: "command_send": 0, 2023-09-10T01:19:26.091 INFO:teuthology.orchestra.run.smithi136.stdout: "command_resend": 0, 2023-09-10T01:19:26.091 INFO:teuthology.orchestra.run.smithi136.stdout: "map_epoch": 114, 2023-09-10T01:19:26.091 INFO:teuthology.orchestra.run.smithi136.stdout: "map_full": 0, 2023-09-10T01:19:26.091 INFO:teuthology.orchestra.run.smithi136.stdout: "map_inc": 26, 2023-09-10T01:19:26.092 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_sessions": 9, 2023-09-10T01:19:26.092 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_open": 9, 2023-09-10T01:19:26.092 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_close": 0, 2023-09-10T01:19:26.093 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_laggy": 0, 2023-09-10T01:19:26.093 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_wr": 0, 2023-09-10T01:19:26.093 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_rd": 2, 2023-09-10T01:19:26.093 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_del": 0 2023-09-10T01:19:26.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.094 INFO:teuthology.orchestra.run.smithi136.stdout: "oft": { 2023-09-10T01:19:26.094 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_objs": 0, 2023-09-10T01:19:26.095 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_kv_pairs": 0, 2023-09-10T01:19:26.095 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_updates": 0, 2023-09-10T01:19:26.095 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_removes": 0 2023-09-10T01:19:26.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.096 INFO:teuthology.orchestra.run.smithi136.stdout: "osd-slow-ops": { 2023-09-10T01:19:26.096 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_ops_count": 0 2023-09-10T01:19:26.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.097 INFO:teuthology.orchestra.run.smithi136.stdout: "purge_queue": { 2023-09-10T01:19:26.097 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops": 0, 2023-09-10T01:19:26.097 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops_high_water": 0, 2023-09-10T01:19:26.097 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing": 0, 2023-09-10T01:19:26.098 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_high_water": 0, 2023-09-10T01:19:26.098 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed": 0, 2023-09-10T01:19:26.098 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_item_in_journal": 0 2023-09-10T01:19:26.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.099 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:26.099 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:26.099 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2023-09-10T01:19:26.099 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:26.100 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 4439, 2023-09-10T01:19:26.100 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 299938988, 2023-09-10T01:19:26.100 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:26.101 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 4439, 2023-09-10T01:19:26.101 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:26.101 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:26.101 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 4439, 2023-09-10T01:19:26.102 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 299938988, 2023-09-10T01:19:26.102 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:26.102 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.103 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.103 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:26.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.104 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_bytes": { 2023-09-10T01:19:26.104 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:26.104 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2023-09-10T01:19:26.104 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:26.105 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:26.105 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:26.105 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:26.106 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:26.106 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 4069, 2023-09-10T01:19:26.106 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 297389403, 2023-09-10T01:19:26.106 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 4069, 2023-09-10T01:19:26.107 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 297389403, 2023-09-10T01:19:26.107 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:26.107 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.107 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.108 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:26.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.109 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_ops": { 2023-09-10T01:19:26.109 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:26.109 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 1024, 2023-09-10T01:19:26.109 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:26.110 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:26.110 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:26.110 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:26.110 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:26.111 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 4069, 2023-09-10T01:19:26.111 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 4069, 2023-09-10T01:19:26.111 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 4069, 2023-09-10T01:19:26.112 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 4069, 2023-09-10T01:19:26.112 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:26.112 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.112 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.113 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:26.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.114 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle": { 2023-09-10T01:19:26.114 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:26.114 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2023-09-10T01:19:26.114 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:26.115 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:26.115 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:26.115 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:26.116 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:26.116 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:26.116 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:26.116 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2023-09-10T01:19:26.117 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2023-09-10T01:19:26.117 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:26.117 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.118 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.118 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:26.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:26.119 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle-0x55de65a42960": { 2023-09-10T01:19:26.119 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:26.119 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2023-09-10T01:19:26.119 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:26.120 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:26.120 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:26.120 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:26.121 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:26.121 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:26.121 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:26.121 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2023-09-10T01:19:26.122 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2023-09-10T01:19:26.122 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:26.122 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:26.123 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:26.123 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:26.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:26.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:26.124 INFO:teuthology.orchestra.run.smithi136.stdout:} 2023-09-10T01:19:26.374 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2023-09-10T01:19:26.374 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2023-09-10T01:19:26.374 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2023-09-10T01:19:26.375 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2023-09-10T01:19:26.375 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2023-09-10T01:19:26.375 DEBUG:tasks.check_counter:Getting stats from i 2023-09-10T01:19:26.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.i perf dump 2023-09-10T01:19:26.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:26 smithi136 ceph-mon[133708]: pgmap v511: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 45 op/s 2023-09-10T01:19:26.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:26 smithi136 ceph-mon[133708]: osdmap e114: 12 total, 12 up, 12 in 2023-09-10T01:19:26.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:26 smithi092 ceph-mon[134067]: pgmap v511: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 45 op/s 2023-09-10T01:19:26.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:26 smithi092 ceph-mon[134067]: osdmap e114: 12 total, 12 up, 12 in 2023-09-10T01:19:26.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:26 smithi003 ceph-mon[97654]: pgmap v511: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 45 op/s 2023-09-10T01:19:26.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:26 smithi003 ceph-mon[97654]: osdmap e114: 12 total, 12 up, 12 in 2023-09-10T01:19:26.930 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T01:19:28.243 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2023-09-10T01:19:28.243 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:28.244 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 187070, 2023-09-10T01:19:28.244 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 314453, 2023-09-10T01:19:28.244 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 58723034, 2023-09-10T01:19:28.244 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 313689142, 2023-09-10T01:19:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 49, 2023-09-10T01:19:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 10.211931739, 2023-09-10T01:19:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 4.690855167, 2023-09-10T01:19:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 5.742586248, 2023-09-10T01:19:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.886312190, 2023-09-10T01:19:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 314452, 2023-09-10T01:19:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 23.194001931, 2023-09-10T01:19:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000073760 2023-09-10T01:19:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 234452, 2023-09-10T01:19:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.054290877, 2023-09-10T01:19:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000231 2023-09-10T01:19:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-09-10T01:19:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 448 2023-09-10T01:19:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 32581, 2023-09-10T01:19:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 33086, 2023-09-10T01:19:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 10095715, 2023-09-10T01:19:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 124183307, 2023-09-10T01:19:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 7, 2023-09-10T01:19:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 2.259396687, 2023-09-10T01:19:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.776224082, 2023-09-10T01:19:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.631316847, 2023-09-10T01:19:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.772385510, 2023-09-10T01:19:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 33085, 2023-09-10T01:19:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 3.256390213, 2023-09-10T01:19:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000098424 2023-09-10T01:19:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 9027, 2023-09-10T01:19:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.001658691, 2023-09-10T01:19:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000183 2023-09-10T01:19:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1855344, 2023-09-10T01:19:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 212160 2023-09-10T01:19:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 172628, 2023-09-10T01:19:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 159513, 2023-09-10T01:19:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 54726342, 2023-09-10T01:19:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 129102259, 2023-09-10T01:19:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 16, 2023-09-10T01:19:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2023-09-10T01:19:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 8.401991887, 2023-09-10T01:19:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 3.236651492, 2023-09-10T01:19:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 5.355317892, 2023-09-10T01:19:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.689235008, 2023-09-10T01:19:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 159510, 2023-09-10T01:19:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 6.211041059, 2023-09-10T01:19:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000038938 2023-09-10T01:19:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 283891, 2023-09-10T01:19:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.038342771, 2023-09-10T01:19:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000135 2023-09-10T01:19:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 320, 2023-09-10T01:19:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 559136 2023-09-10T01:19:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2023-09-10T01:19:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 1, 2023-09-10T01:19:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2023-09-10T01:19:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-MDSRank": { 2023-09-10T01:19:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2023-09-10T01:19:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2023-09-10T01:19:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 55717, 2023-09-10T01:19:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 49.461987565, 2023-09-10T01:19:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000887736 2023-09-10T01:19:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-PurgeQueue": { 2023-09-10T01:19:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2023-09-10T01:19:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2023-09-10T01:19:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 39798, 2023-09-10T01:19:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 4.426233699, 2023-09-10T01:19:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000111217 2023-09-10T01:19:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2023-09-10T01:19:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "request": 120112, 2023-09-10T01:19:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "reply": 118640, 2023-09-10T01:19:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "reply_latency": { 2023-09-10T01:19:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 118640, 2023-09-10T01:19:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 255.685700704, 2023-09-10T01:19:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002155139 2023-09-10T01:19:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_reply": 0, 2023-09-10T01:19:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "forward": 1472, 2023-09-10T01:19:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_complete": 2, 2023-09-10T01:19:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_keys": 0, 2023-09-10T01:19:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_commit": 30, 2023-09-10T01:19:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_split": 2, 2023-09-10T01:19:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_merge": 1, 2023-09-10T01:19:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes": 1678, 2023-09-10T01:19:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_top": 354, 2023-09-10T01:19:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_bottom": 1303, 2023-09-10T01:19:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pin_tail": 21, 2023-09-10T01:19:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pinned": 1172, 2023-09-10T01:19:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_expired": 223, 2023-09-10T01:19:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_with_caps": 227, 2023-09-10T01:19:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "caps": 227, 2023-09-10T01:19:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "subtrees": 15, 2023-09-10T01:19:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse": 241436, 2023-09-10T01:19:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_hit": 118587, 2023-09-10T01:19:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_forward": 1469, 2023-09-10T01:19:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_discover": 27, 2023-09-10T01:19:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_dir_fetch": 0, 2023-09-10T01:19:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_remote_ino": 0, 2023-09-10T01:19:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_lock": 3197, 2023-09-10T01:19:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "load_cent": 9218, 2023-09-10T01:19:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "q": 0, 2023-09-10T01:19:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "exported": 5, 2023-09-10T01:19:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "exported_inodes": 162, 2023-09-10T01:19:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "imported": 14, 2023-09-10T01:19:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "imported_inodes": 981, 2023-09-10T01:19:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_dir_fetch": 0, 2023-09-10T01:19:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_backtrace_fetch": 1038, 2023-09-10T01:19:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_peer_discover": 0, 2023-09-10T01:19:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rfiles": 675, 2023-09-10T01:19:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rbytes": 211566833, 2023-09-10T01:19:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rsnaps": 6, 2023-09-10T01:19:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_fetch": 1038, 2023-09-10T01:19:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_set_tag": 0, 2023-09-10T01:19:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_repaired": 0, 2023-09-10T01:19:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_inotable_repaired": 0, 2023-09-10T01:19:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_inodes": 97, 2023-09-10T01:19:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_base_inodes": 0, 2023-09-10T01:19:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dirfrag_rstats": 97, 2023-09-10T01:19:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_file_inodes": 941, 2023-09-10T01:19:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_inode_file_caps": 4, 2023-09-10T01:19:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_revoke": 8891, 2023-09-10T01:19:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_grant": 85228, 2023-09-10T01:19:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_trunc": 0, 2023-09-10T01:19:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flushsnap_ack": 1151, 2023-09-10T01:19:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flush_ack": 0, 2023-09-10T01:19:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps": 24852, 2023-09-10T01:19:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps_dirty": 10757, 2023-09-10T01:19:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_cap_release": 2894, 2023-09-10T01:19:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "process_request_cap_release": 107602 2023-09-10T01:19:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache": { 2023-09-10T01:19:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update": 69, 2023-09-10T01:19:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update_receipt": 126, 2023-09-10T01:19:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_try_discover": 70, 2023-09-10T01:19:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_send_discover": 137, 2023-09-10T01:19:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_handle_discover": 106, 2023-09-10T01:19:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays": 0, 2023-09-10T01:19:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_delayed": 0, 2023-09-10T01:19:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_enqueuing": 0, 2023-09-10T01:19:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_created": 23216, 2023-09-10T01:19:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_enqueued": 23216, 2023-09-10T01:19:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_reintegrated": 0, 2023-09-10T01:19:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_migrated": 0, 2023-09-10T01:19:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_processing": 0, 2023-09-10T01:19:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_enqueued": 0, 2023-09-10T01:19:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_prioritized": 0, 2023-09-10T01:19:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_started": 0, 2023-09-10T01:19:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_completed": 0, 2023-09-10T01:19:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_enqueue_scrub": 0, 2023-09-10T01:19:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_exportdir": 6, 2023-09-10T01:19:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_flush": 0, 2023-09-10T01:19:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragmentdir": 3, 2023-09-10T01:19:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragstats": 0, 2023-09-10T01:19:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_inodestats": 0 2023-09-10T01:19:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log": { 2023-09-10T01:19:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "evlrg": 0, 2023-09-10T01:19:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "evadd": 110679, 2023-09-10T01:19:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "evex": 14339, 2023-09-10T01:19:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "evtrm": 12291, 2023-09-10T01:19:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "ev": 98388, 2023-09-10T01:19:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "evexg": 0, 2023-09-10T01:19:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "evexd": 2048, 2023-09-10T01:19:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "segadd": 144, 2023-09-10T01:19:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "segex": 16, 2023-09-10T01:19:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "segtrm": 14, 2023-09-10T01:19:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "seg": 130, 2023-09-10T01:19:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "segmjr": 9, 2023-09-10T01:19:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "segexg": 0, 2023-09-10T01:19:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "segexd": 2, 2023-09-10T01:19:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "expos": 21986127, 2023-09-10T01:19:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "wrpos": 301565391, 2023-09-10T01:19:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "rdpos": 0, 2023-09-10T01:19:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "jlat": { 2023-09-10T01:19:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 28575, 2023-09-10T01:19:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 127.134620322, 2023-09-10T01:19:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.004449155 2023-09-10T01:19:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "replayed": 0 2023-09-10T01:19:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mem": { 2023-09-10T01:19:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "ino": 995, 2023-09-10T01:19:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "ino+": 24385, 2023-09-10T01:19:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "ino-": 23390, 2023-09-10T01:19:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "dir": 104, 2023-09-10T01:19:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "dir+": 172, 2023-09-10T01:19:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "dir-": 68, 2023-09-10T01:19:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "dn": 1678, 2023-09-10T01:19:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "dn+": 37437, 2023-09-10T01:19:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "dn-": 35759, 2023-09-10T01:19:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "cap": 227, 2023-09-10T01:19:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "cap+": 24032, 2023-09-10T01:19:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "cap-": 23805, 2023-09-10T01:19:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "rss": 72900, 2023-09-10T01:19:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "heap": 198960 2023-09-10T01:19:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server": { 2023-09-10T01:19:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_client_request": 208083, 2023-09-10T01:19:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_server_request": 0, 2023-09-10T01:19:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_request": 120112, 2023-09-10T01:19:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_session": 726, 2023-09-10T01:19:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_peer_request": 20990, 2023-09-10T01:19:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "req_create_latency": { 2023-09-10T01:19:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 23178, 2023-09-10T01:19:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 58.426209798, 2023-09-10T01:19:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002520761 2023-09-10T01:19:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getattr_latency": { 2023-09-10T01:19:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 395, 2023-09-10T01:19:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 3.151619144, 2023-09-10T01:19:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.007978782 2023-09-10T01:19:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getfilelock_latency": { 2023-09-10T01:19:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "req_link_latency": { 2023-09-10T01:19:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookup_latency": { 2023-09-10T01:19:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 30233, 2023-09-10T01:19:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 32.901193771, 2023-09-10T01:19:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001088254 2023-09-10T01:19:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookuphash_latency": { 2023-09-10T01:19:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupino_latency": { 2023-09-10T01:19:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1, 2023-09-10T01:19:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000146628, 2023-09-10T01:19:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000146628 2023-09-10T01:19:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupname_latency": { 2023-09-10T01:19:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupparent_latency": { 2023-09-10T01:19:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupsnap_latency": { 2023-09-10T01:19:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lssnap_latency": { 2023-09-10T01:19:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mkdir_latency": { 2023-09-10T01:19:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mknod_latency": { 2023-09-10T01:19:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mksnap_latency": { 2023-09-10T01:19:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "req_open_latency": { 2023-09-10T01:19:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "req_readdir_latency": { 2023-09-10T01:19:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 37269, 2023-09-10T01:19:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 108.758035592, 2023-09-10T01:19:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002918190 2023-09-10T01:19:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rename_latency": { 2023-09-10T01:19:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 4137, 2023-09-10T01:19:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 11.861095086, 2023-09-10T01:19:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.002867076 2023-09-10T01:19:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "req_renamesnap_latency": { 2023-09-10T01:19:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "req_snapdiff_latency": { 2023-09-10T01:19:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmdir_latency": { 2023-09-10T01:19:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmsnap_latency": { 2023-09-10T01:19:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmxattr_latency": { 2023-09-10T01:19:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setattr_latency": { 2023-09-10T01:19:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 11, 2023-09-10T01:19:28.338 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.223575476, 2023-09-10T01:19:28.338 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.020325043 2023-09-10T01:19:28.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.338 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setdirlayout_latency": { 2023-09-10T01:19:28.339 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.339 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.339 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.340 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setfilelock_latency": { 2023-09-10T01:19:28.340 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 200, 2023-09-10T01:19:28.340 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.276790109, 2023-09-10T01:19:28.341 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001383950 2023-09-10T01:19:28.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.341 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setlayout_latency": { 2023-09-10T01:19:28.341 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.342 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.342 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.342 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setxattr_latency": { 2023-09-10T01:19:28.343 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.343 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.343 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.344 INFO:teuthology.orchestra.run.smithi136.stdout: "req_symlink_latency": { 2023-09-10T01:19:28.344 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.344 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.344 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.345 INFO:teuthology.orchestra.run.smithi136.stdout: "req_unlink_latency": { 2023-09-10T01:19:28.345 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 23216, 2023-09-10T01:19:28.345 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 40.087035100, 2023-09-10T01:19:28.346 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.001726698 2023-09-10T01:19:28.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.346 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_revoke_eviction": 0, 2023-09-10T01:19:28.347 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_acquisition_throttle": 0, 2023-09-10T01:19:28.347 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getvxattr_latency": { 2023-09-10T01:19:28.347 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.347 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.349 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.349 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.350 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessions": { 2023-09-10T01:19:28.350 INFO:teuthology.orchestra.run.smithi136.stdout: "session_count": 1, 2023-09-10T01:19:28.350 INFO:teuthology.orchestra.run.smithi136.stdout: "session_add": 1, 2023-09-10T01:19:28.351 INFO:teuthology.orchestra.run.smithi136.stdout: "session_remove": 0, 2023-09-10T01:19:28.351 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_open": 1, 2023-09-10T01:19:28.351 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_stale": 0, 2023-09-10T01:19:28.352 INFO:teuthology.orchestra.run.smithi136.stdout: "total_load": 21086, 2023-09-10T01:19:28.352 INFO:teuthology.orchestra.run.smithi136.stdout: "average_load": 21086, 2023-09-10T01:19:28.352 INFO:teuthology.orchestra.run.smithi136.stdout: "avg_session_uptime": 993, 2023-09-10T01:19:28.352 INFO:teuthology.orchestra.run.smithi136.stdout: "mdthresh_evicted": 0 2023-09-10T01:19:28.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.353 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2023-09-10T01:19:28.353 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:28.354 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:28.354 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:28.354 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:28.355 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:28.355 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:28.355 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:28.356 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:28.356 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:28.356 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:28.356 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:28.357 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:28.357 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:28.357 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:28.357 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:28.358 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:28.358 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:28.358 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:28.359 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:28.359 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:28.359 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:28.359 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:28.360 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:28.360 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:28.360 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:28.361 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:28.361 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:28.361 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:28.363 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2023-09-10T01:19:28.363 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:28.363 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:28.364 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:28.364 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:28.364 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 354997, 2023-09-10T01:19:28.364 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 869, 2023-09-10T01:19:28.365 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:28.365 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:28.365 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2023-09-10T01:19:28.366 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2023-09-10T01:19:28.366 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:28.366 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:28.366 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:28.367 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:28.367 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 9816, 2023-09-10T01:19:28.367 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 190, 2023-09-10T01:19:28.368 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:28.368 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:28.368 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:28.368 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2023-09-10T01:19:28.369 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 3949352, 2023-09-10T01:19:28.369 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 36937, 2023-09-10T01:19:28.369 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:28.369 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2023-09-10T01:19:28.370 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:28.370 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2023-09-10T01:19:28.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.371 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter": { 2023-09-10T01:19:28.371 INFO:teuthology.orchestra.run.smithi136.stdout: "op_active": 0, 2023-09-10T01:19:28.371 INFO:teuthology.orchestra.run.smithi136.stdout: "op_laggy": 0, 2023-09-10T01:19:28.371 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send": 78194, 2023-09-10T01:19:28.372 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send_bytes": 301608422, 2023-09-10T01:19:28.372 INFO:teuthology.orchestra.run.smithi136.stdout: "op_resend": 28, 2023-09-10T01:19:28.372 INFO:teuthology.orchestra.run.smithi136.stdout: "op_reply": 78194, 2023-09-10T01:19:28.373 INFO:teuthology.orchestra.run.smithi136.stdout: "op_latency": { 2023-09-10T01:19:28.373 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 78194, 2023-09-10T01:19:28.373 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 583.200983759, 2023-09-10T01:19:28.373 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.007458385 2023-09-10T01:19:28.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.374 INFO:teuthology.orchestra.run.smithi136.stdout: "op_inflight": 0, 2023-09-10T01:19:28.374 INFO:teuthology.orchestra.run.smithi136.stdout: "oplen_avg": { 2023-09-10T01:19:28.374 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 78194, 2023-09-10T01:19:28.375 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 78604 2023-09-10T01:19:28.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.375 INFO:teuthology.orchestra.run.smithi136.stdout: "op": 78194, 2023-09-10T01:19:28.376 INFO:teuthology.orchestra.run.smithi136.stdout: "op_r": 1764, 2023-09-10T01:19:28.376 INFO:teuthology.orchestra.run.smithi136.stdout: "op_w": 76430, 2023-09-10T01:19:28.376 INFO:teuthology.orchestra.run.smithi136.stdout: "op_rmw": 0, 2023-09-10T01:19:28.376 INFO:teuthology.orchestra.run.smithi136.stdout: "op_pg": 0, 2023-09-10T01:19:28.377 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_stat": 2, 2023-09-10T01:19:28.377 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_create": 21, 2023-09-10T01:19:28.377 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_read": 723, 2023-09-10T01:19:28.377 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_write": 29298, 2023-09-10T01:19:28.378 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writefull": 374, 2023-09-10T01:19:28.378 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writesame": 0, 2023-09-10T01:19:28.378 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_append": 0, 2023-09-10T01:19:28.379 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_zero": 0, 2023-09-10T01:19:28.379 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_truncate": 2, 2023-09-10T01:19:28.379 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_delete": 46526, 2023-09-10T01:19:28.379 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_mapext": 0, 2023-09-10T01:19:28.380 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_sparse_read": 0, 2023-09-10T01:19:28.380 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_clonerange": 0, 2023-09-10T01:19:28.380 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_getxattr": 1040, 2023-09-10T01:19:28.380 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_setxattr": 42, 2023-09-10T01:19:28.381 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_cmpxattr": 0, 2023-09-10T01:19:28.381 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_rmxattr": 0, 2023-09-10T01:19:28.381 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_resetxattrs": 0, 2023-09-10T01:19:28.381 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_call": 0, 2023-09-10T01:19:28.382 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_watch": 0, 2023-09-10T01:19:28.382 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_notify": 0, 2023-09-10T01:19:28.382 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_src_cmpxattr": 0, 2023-09-10T01:19:28.383 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls": 0, 2023-09-10T01:19:28.383 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls_filter": 0, 2023-09-10T01:19:28.383 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_other": 0, 2023-09-10T01:19:28.383 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_active": 0, 2023-09-10T01:19:28.384 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_send": 0, 2023-09-10T01:19:28.384 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_resend": 0, 2023-09-10T01:19:28.384 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_ping": 0, 2023-09-10T01:19:28.385 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_active": 0, 2023-09-10T01:19:28.385 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_send": 0, 2023-09-10T01:19:28.385 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_resend": 0, 2023-09-10T01:19:28.385 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_active": 0, 2023-09-10T01:19:28.386 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_send": 0, 2023-09-10T01:19:28.386 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_resend": 0, 2023-09-10T01:19:28.386 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_active": 0, 2023-09-10T01:19:28.387 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_send": 0, 2023-09-10T01:19:28.387 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_resend": 0, 2023-09-10T01:19:28.387 INFO:teuthology.orchestra.run.smithi136.stdout: "command_active": 0, 2023-09-10T01:19:28.387 INFO:teuthology.orchestra.run.smithi136.stdout: "command_send": 0, 2023-09-10T01:19:28.388 INFO:teuthology.orchestra.run.smithi136.stdout: "command_resend": 0, 2023-09-10T01:19:28.388 INFO:teuthology.orchestra.run.smithi136.stdout: "map_epoch": 114, 2023-09-10T01:19:28.388 INFO:teuthology.orchestra.run.smithi136.stdout: "map_full": 0, 2023-09-10T01:19:28.389 INFO:teuthology.orchestra.run.smithi136.stdout: "map_inc": 26, 2023-09-10T01:19:28.389 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_sessions": 12, 2023-09-10T01:19:28.389 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_open": 12, 2023-09-10T01:19:28.389 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_close": 0, 2023-09-10T01:19:28.390 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_laggy": 0, 2023-09-10T01:19:28.390 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_wr": 401, 2023-09-10T01:19:28.390 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_rd": 5, 2023-09-10T01:19:28.391 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_del": 170 2023-09-10T01:19:28.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.391 INFO:teuthology.orchestra.run.smithi136.stdout: "oft": { 2023-09-10T01:19:28.392 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_objs": 1, 2023-09-10T01:19:28.392 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_kv_pairs": 0, 2023-09-10T01:19:28.392 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_updates": 0, 2023-09-10T01:19:28.393 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_removes": 0 2023-09-10T01:19:28.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.393 INFO:teuthology.orchestra.run.smithi136.stdout: "osd-slow-ops": { 2023-09-10T01:19:28.393 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_ops_count": 0 2023-09-10T01:19:28.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.394 INFO:teuthology.orchestra.run.smithi136.stdout: "purge_queue": { 2023-09-10T01:19:28.394 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops": 0, 2023-09-10T01:19:28.395 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops_high_water": 16, 2023-09-10T01:19:28.395 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing": 0, 2023-09-10T01:19:28.395 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_high_water": 8, 2023-09-10T01:19:28.395 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed": 23216, 2023-09-10T01:19:28.395 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_item_in_journal": 0 2023-09-10T01:19:28.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.396 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:28.396 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:28.396 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2023-09-10T01:19:28.397 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:28.397 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 392275, 2023-09-10T01:19:28.397 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 90764744, 2023-09-10T01:19:28.397 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:28.398 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 392275, 2023-09-10T01:19:28.398 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:28.398 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:28.399 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 392275, 2023-09-10T01:19:28.399 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 90764744, 2023-09-10T01:19:28.399 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:28.399 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.400 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.400 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.400 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.401 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_bytes": { 2023-09-10T01:19:28.401 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:28.401 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2023-09-10T01:19:28.402 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:28.402 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:28.402 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:28.403 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:28.403 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:28.403 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 78194, 2023-09-10T01:19:28.403 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 304141813, 2023-09-10T01:19:28.404 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 78194, 2023-09-10T01:19:28.404 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 304141813, 2023-09-10T01:19:28.404 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:28.405 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.405 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.405 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.405 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.406 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_ops": { 2023-09-10T01:19:28.406 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:28.407 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 1024, 2023-09-10T01:19:28.407 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:28.407 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2023-09-10T01:19:28.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2023-09-10T01:19:28.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:28.408 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2023-09-10T01:19:28.408 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 78194, 2023-09-10T01:19:28.409 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 78194, 2023-09-10T01:19:28.409 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 78194, 2023-09-10T01:19:28.409 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 78194, 2023-09-10T01:19:28.410 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:28.410 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.410 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.410 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.411 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.411 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle": { 2023-09-10T01:19:28.412 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:28.412 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2023-09-10T01:19:28.412 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:28.412 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 23216, 2023-09-10T01:19:28.413 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 2533456, 2023-09-10T01:19:28.413 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:28.413 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 23216, 2023-09-10T01:19:28.414 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:28.414 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:28.414 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 723, 2023-09-10T01:19:28.414 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 2533456, 2023-09-10T01:19:28.415 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:28.415 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.415 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.415 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.416 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:28.416 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle-0x5596ad336640": { 2023-09-10T01:19:28.417 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:28.417 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2023-09-10T01:19:28.417 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:28.417 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 110679, 2023-09-10T01:19:28.418 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 297371087, 2023-09-10T01:19:28.418 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:28.418 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 110679, 2023-09-10T01:19:28.419 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:28.419 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:28.419 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 28575, 2023-09-10T01:19:28.420 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 297371087, 2023-09-10T01:19:28.420 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:28.420 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:28.420 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:28.421 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:28.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:28.422 INFO:teuthology.orchestra.run.smithi136.stdout:} 2023-09-10T01:19:28.584 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:28 smithi136 ceph-mon[133708]: pgmap v513: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 317 KiB/s wr, 45 op/s 2023-09-10T01:19:28.586 INFO:tasks.check_counter:Daemon mds.i mds.exported=5 2023-09-10T01:19:28.587 INFO:tasks.check_counter:Daemon mds.i mds.imported=14 2023-09-10T01:19:28.587 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2023-09-10T01:19:28.587 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2023-09-10T01:19:28.587 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2023-09-10T01:19:28.588 DEBUG:tasks.check_counter:Getting stats from l 2023-09-10T01:19:28.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9670cca5bc8e17916865dc84939edfc10b1f2c00 shell --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da -- ceph daemon mds.l perf dump 2023-09-10T01:19:28.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:28 smithi092 ceph-mon[134067]: pgmap v513: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 317 KiB/s wr, 45 op/s 2023-09-10T01:19:28.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:28 smithi003 ceph-mon[97654]: pgmap v513: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 317 KiB/s wr, 45 op/s 2023-09-10T01:19:29.146 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/mon.c/config 2023-09-10T01:19:30.400 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2023-09-10T01:19:30.401 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2023-09-10T01:19:30.401 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 0, 2023-09-10T01:19:30.401 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 2, 2023-09-10T01:19:30.402 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 0, 2023-09-10T01:19:30.402 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 0, 2023-09-10T01:19:30.402 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 4, 2023-09-10T01:19:30.403 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 0, 2023-09-10T01:19:30.403 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.001228583, 2023-09-10T01:19:30.403 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.000000000, 2023-09-10T01:19:30.403 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.000302303, 2023-09-10T01:19:30.404 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:30.404 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:30.404 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:30.404 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:30.405 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:30.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.405 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:30.406 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:30.406 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:30.406 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:30.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.407 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 0, 2023-09-10T01:19:30.407 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 0 2023-09-10T01:19:30.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.407 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2023-09-10T01:19:30.408 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 325, 2023-09-10T01:19:30.408 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 311, 2023-09-10T01:19:30.408 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 1718800, 2023-09-10T01:19:30.409 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 186416, 2023-09-10T01:19:30.409 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 2, 2023-09-10T01:19:30.409 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 1, 2023-09-10T01:19:30.409 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.052480028, 2023-09-10T01:19:30.410 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.015961843, 2023-09-10T01:19:30.410 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.037380293, 2023-09-10T01:19:30.410 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.007718530, 2023-09-10T01:19:30.411 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:30.411 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 311, 2023-09-10T01:19:30.411 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.019907701, 2023-09-10T01:19:30.411 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000064011 2023-09-10T01:19:30.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.412 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:30.412 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:30.413 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:30.413 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:30.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.413 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1718800, 2023-09-10T01:19:30.414 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 186416 2023-09-10T01:19:30.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.414 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2023-09-10T01:19:30.414 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 3, 2023-09-10T01:19:30.415 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 237, 2023-09-10T01:19:30.415 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 480, 2023-09-10T01:19:30.415 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 270816, 2023-09-10T01:19:30.416 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 6, 2023-09-10T01:19:30.416 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 3, 2023-09-10T01:19:30.416 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.012661703, 2023-09-10T01:19:30.416 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.008227092, 2023-09-10T01:19:30.417 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.001816924, 2023-09-10T01:19:30.417 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-09-10T01:19:30.417 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2023-09-10T01:19:30.417 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 234, 2023-09-10T01:19:30.418 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.023016583, 2023-09-10T01:19:30.418 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000098361 2023-09-10T01:19:30.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.418 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2023-09-10T01:19:30.419 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:30.419 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:30.419 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:30.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.420 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 480, 2023-09-10T01:19:30.420 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 270816 2023-09-10T01:19:30.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.420 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2023-09-10T01:19:30.421 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 0, 2023-09-10T01:19:30.421 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2023-09-10T01:19:30.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.422 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2023-09-10T01:19:30.422 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2023-09-10T01:19:30.422 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2023-09-10T01:19:30.422 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2023-09-10T01:19:30.423 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2023-09-10T01:19:30.423 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2023-09-10T01:19:30.423 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2023-09-10T01:19:30.424 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2023-09-10T01:19:30.424 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2023-09-10T01:19:30.424 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2023-09-10T01:19:30.424 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2023-09-10T01:19:30.425 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2023-09-10T01:19:30.425 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2023-09-10T01:19:30.425 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2023-09-10T01:19:30.425 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2023-09-10T01:19:30.426 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2023-09-10T01:19:30.426 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2023-09-10T01:19:30.426 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2023-09-10T01:19:30.427 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2023-09-10T01:19:30.427 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2023-09-10T01:19:30.427 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2023-09-10T01:19:30.427 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2023-09-10T01:19:30.428 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2023-09-10T01:19:30.428 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2023-09-10T01:19:30.428 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2023-09-10T01:19:30.428 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2023-09-10T01:19:30.429 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2023-09-10T01:19:30.429 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2023-09-10T01:19:30.429 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2023-09-10T01:19:30.429 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2023-09-10T01:19:30.430 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2023-09-10T01:19:30.430 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2023-09-10T01:19:30.430 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2023-09-10T01:19:30.430 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2023-09-10T01:19:30.431 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2023-09-10T01:19:30.431 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2023-09-10T01:19:30.431 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2023-09-10T01:19:30.432 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 147970, 2023-09-10T01:19:30.432 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 29, 2023-09-10T01:19:30.432 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2023-09-10T01:19:30.432 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2023-09-10T01:19:30.433 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2023-09-10T01:19:30.433 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2023-09-10T01:19:30.433 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2023-09-10T01:19:30.433 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2023-09-10T01:19:30.434 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2023-09-10T01:19:30.434 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2023-09-10T01:19:30.434 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 0, 2023-09-10T01:19:30.435 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 0, 2023-09-10T01:19:30.435 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2023-09-10T01:19:30.435 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2023-09-10T01:19:30.435 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2023-09-10T01:19:30.436 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2023-09-10T01:19:30.436 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 928, 2023-09-10T01:19:30.436 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 2, 2023-09-10T01:19:30.436 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2023-09-10T01:19:30.437 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2023-09-10T01:19:30.437 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2023-09-10T01:19:30.437 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2023-09-10T01:19:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2023-09-10T01:19:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-09-10T01:19:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2023-09-10T01:19:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2023-09-10T01:19:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2023-09-10T01:19:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 324, 2023-09-10T01:19:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 1670591, 2023-09-10T01:19:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2023-09-10T01:19:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 324, 2023-09-10T01:19:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2023-09-10T01:19:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2023-09-10T01:19:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 324, 2023-09-10T01:19:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1670591, 2023-09-10T01:19:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2023-09-10T01:19:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2023-09-10T01:19:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2023-09-10T01:19:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2023-09-10T01:19:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2023-09-10T01:19:30.443 INFO:teuthology.orchestra.run.smithi136.stdout:} 2023-09-10T01:19:30.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:30 smithi136 ceph-mon[133708]: pgmap v514: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 77 KiB/s wr, 15 op/s 2023-09-10T01:19:30.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:30 smithi092 ceph-mon[134067]: pgmap v514: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 77 KiB/s wr, 15 op/s 2023-09-10T01:19:30.790 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2023-09-10T01:19:30.791 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2023-09-10T01:19:30.791 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2023-09-10T01:19:30.791 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2023-09-10T01:19:30.792 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2023-09-10T01:19:30.792 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-09-10T01:19:30.803 INFO:tasks.kclient:Unmounting kernel clients... 2023-09-10T01:19:30.803 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T01:19:30.803 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2023-09-10T01:19:30.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:30 smithi003 ceph-mon[97654]: pgmap v514: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 77 KiB/s wr, 15 op/s 2023-09-10T01:19:30.856 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-09-10T01:19:30.856 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2023-09-10T01:19:30.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:19:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 -- 172.21.15.3:0/109713323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 msgr2=0x7f6554106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 --2- 172.21.15.3:0/109713323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 0x7f6554106a20 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6548009990 tx=0x7f6548009ca0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 -- 172.21.15.3:0/109713323 shutdown_connections 2023-09-10T01:19:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 --2- 172.21.15.3:0/109713323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6554100ff0 0x7f655410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 --2- 172.21.15.3:0/109713323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 0x7f6554106a20 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 --2- 172.21.15.3:0/109713323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100630 0x7f6554100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 -- 172.21.15.3:0/109713323 >> 172.21.15.3:0/109713323 conn(0x7f65540fc140 msgr2=0x7f65540fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:31.155 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 -- 172.21.15.3:0/109713323 shutdown_connections 2023-09-10T01:19:31.156 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.152+0000 7f655afff700 1 -- 172.21.15.3:0/109713323 wait complete. 2023-09-10T01:19:31.157 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 Processor -- start 2023-09-10T01:19:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 -- start start 2023-09-10T01:19:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6554100630 0x7f65541966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.158 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100ff0 0x7f6554196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 0x7f6554199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655419a5c0 con 0x7f6554100ff0 2023-09-10T01:19:31.159 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655419a730 con 0x7f6554106640 2023-09-10T01:19:31.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f655afff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655419a8f0 con 0x7f6554100630 2023-09-10T01:19:31.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f6553fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100ff0 0x7f6554196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.160 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f6553fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100ff0 0x7f6554196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37364/0 (socket says 172.21.15.3:37364) 2023-09-10T01:19:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f6553fff700 1 -- 172.21.15.3:0/2043617512 learned_addr learned my addr 172.21.15.3:0/2043617512 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.154+0000 7f6558d9b700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6554100630 0x7f65541966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f655959c700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 0x7f6554199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6553fff700 1 -- 172.21.15.3:0/2043617512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6554100630 msgr2=0x7f65541966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6553fff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6554100630 0x7f65541966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6553fff700 1 -- 172.21.15.3:0/2043617512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 msgr2=0x7f6554199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6553fff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 0x7f6554199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6553fff700 1 -- 172.21.15.3:0/2043617512 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540009520 con 0x7f6554100ff0 2023-09-10T01:19:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f655959c700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 0x7f6554199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:19:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6553fff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100ff0 0x7f6554196c10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f654000eb30 tx=0x7f654000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6551ffb700 1 -- 172.21.15.3:0/2043617512 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540009880 con 0x7f6554100ff0 2023-09-10T01:19:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65480095f0 con 0x7f6554100ff0 2023-09-10T01:19:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6551ffb700 1 -- 172.21.15.3:0/2043617512 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65400044e0 con 0x7f6554100ff0 2023-09-10T01:19:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.155+0000 7f6551ffb700 1 -- 172.21.15.3:0/2043617512 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540003d80 con 0x7f6554100ff0 2023-09-10T01:19:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.156+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65541a3350 con 0x7f6554100ff0 2023-09-10T01:19:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.156+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6554067480 con 0x7f6554100ff0 2023-09-10T01:19:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.158+0000 7f6551ffb700 1 -- 172.21.15.3:0/2043617512 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f6540003f20 con 0x7f6554100ff0 2023-09-10T01:19:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.160+0000 7f6551ffb700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f65440a07b0 0x7f65440a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.160+0000 7f6551ffb700 1 -- 172.21.15.3:0/2043617512 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f6540014070 con 0x7f6554100ff0 2023-09-10T01:19:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.160+0000 7f6558d9b700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f65440a07b0 0x7f65440a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.161+0000 7f6558d9b700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f65440a07b0 0x7f65440a2c70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f654800b2e0 tx=0x7f6548005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.164+0000 7f6551ffb700 1 -- 172.21.15.3:0/2043617512 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f6540095ce0 con 0x7f6554100ff0 2023-09-10T01:19:31.378 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.376+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6554110610 con 0x7f6554100ff0 2023-09-10T01:19:31.379 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.378+0000 7f6551ffb700 1 -- 172.21.15.3:0/2043617512 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+25078 (secure 0 0 0) 0x7f6540099300 con 0x7f6554100ff0 2023-09-10T01:19:31.384 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:19:31.385 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":114,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:19:25.328137+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/3516755706":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/1056299694":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6801/91094391":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6800/91094391":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/2575312228":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/2296857090":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:19:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f65440a07b0 msgr2=0x7f65440a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f65440a07b0 0x7f65440a2c70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f654800b2e0 tx=0x7f6548005fb0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100ff0 msgr2=0x7f6554196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100ff0 0x7f6554196c10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f654000eb30 tx=0x7f654000ee40 comp rx=0 tx=0).stop 2023-09-10T01:19:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 shutdown_connections 2023-09-10T01:19:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f65440a07b0 0x7f65440a2c70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6554100630 0x7f65541966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6554106640 0x7f6554199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 --2- 172.21.15.3:0/2043617512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6554100ff0 0x7f6554196c10 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.380+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 >> 172.21.15.3:0/2043617512 conn(0x7f65540fc140 msgr2=0x7f65540fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.381+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 shutdown_connections 2023-09-10T01:19:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.381+0000 7f655afff700 1 -- 172.21.15.3:0/2043617512 wait complete. 2023-09-10T01:19:31.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-09-10T00:56:36.373564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-09-10T01:00:41.489409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-09-10T01:00:42.983230+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-09-10T01:00:46.555744+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-09-10T01:19:31.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-09-10T01:19:31.615 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/594427950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44106660 msgr2=0x7f9a44106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/594427950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44106660 0x7f9a44106a40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009990 tx=0x7f9a40009ca0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/594427950 shutdown_connections 2023-09-10T01:19:31.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/594427950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a44101010 0x7f9a4410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/594427950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44106660 0x7f9a44106a40 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/594427950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a44100650 0x7f9a44100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.617 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/594427950 >> 172.21.15.3:0/594427950 conn(0x7f9a440fc160 msgr2=0x7f9a440fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:31.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/594427950 shutdown_connections 2023-09-10T01:19:31.618 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.614+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/594427950 wait complete. 2023-09-10T01:19:31.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.615+0000 7f9a4c5ff700 1 Processor -- start 2023-09-10T01:19:31.620 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.615+0000 7f9a4c5ff700 1 -- start start 2023-09-10T01:19:31.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4c5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a44100650 0x7f9a4419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4c5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44101010 0x7f9a4419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4c5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a44106660 0x7f9a44198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.621 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4c5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a44198dc0 con 0x7f9a44101010 2023-09-10T01:19:31.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4c5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a44198f30 con 0x7f9a44106660 2023-09-10T01:19:31.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4c5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a441990f0 con 0x7f9a44100650 2023-09-10T01:19:31.622 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a49b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44101010 0x7f9a4419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4ab9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a44106660 0x7f9a44198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a49b9a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44101010 0x7f9a4419ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37384/0 (socket says 172.21.15.3:37384) 2023-09-10T01:19:31.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a49b9a700 1 -- 172.21.15.3:0/842341550 learned_addr learned my addr 172.21.15.3:0/842341550 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:31.623 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.616+0000 7f9a4a39b700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a44100650 0x7f9a4419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4ab9c700 1 -- 172.21.15.3:0/842341550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a44100650 msgr2=0x7f9a4419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4ab9c700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a44100650 0x7f9a4419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.624 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4ab9c700 1 -- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44101010 msgr2=0x7f9a4419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4ab9c700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44101010 0x7f9a4419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4ab9c700 1 -- 172.21.15.3:0/842341550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a400095f0 con 0x7f9a44106660 2023-09-10T01:19:31.625 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a49b9a700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44101010 0x7f9a4419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:19:31.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4ab9c700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a44106660 0x7f9a44198750 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00efc0 tx=0x7f9a3c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:31.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a3b7fe700 1 -- 172.21.15.3:0/842341550 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c018070 con 0x7f9a44106660 2023-09-10T01:19:31.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a44199370 con 0x7f9a44106660 2023-09-10T01:19:31.626 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a3b7fe700 1 -- 172.21.15.3:0/842341550 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a3c00f040 con 0x7f9a44106660 2023-09-10T01:19:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a3b7fe700 1 -- 172.21.15.3:0/842341550 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00cd40 con 0x7f9a44106660 2023-09-10T01:19:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.617+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a441a56d0 con 0x7f9a44106660 2023-09-10T01:19:31.627 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.618+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a44067480 con 0x7f9a44106660 2023-09-10T01:19:31.628 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.620+0000 7f9a3b7fe700 1 -- 172.21.15.3:0/842341550 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f9a3c003a00 con 0x7f9a44106660 2023-09-10T01:19:31.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.622+0000 7f9a3b7fe700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9a300a07b0 0x7f9a300a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:31.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.622+0000 7f9a3b7fe700 1 -- 172.21.15.3:0/842341550 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f9a3c0cc270 con 0x7f9a44106660 2023-09-10T01:19:31.629 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.622+0000 7f9a4a39b700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9a300a07b0 0x7f9a300a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:31.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.623+0000 7f9a4a39b700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9a300a07b0 0x7f9a300a2c70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000b2e0 tx=0x7f9a40005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:31.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.624+0000 7f9a3b7fe700 1 -- 172.21.15.3:0/842341550 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f9a3c0956d0 con 0x7f9a44106660 2023-09-10T01:19:31.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.835+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a44199c60 con 0x7f9a44106660 2023-09-10T01:19:31.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.836+0000 7f9a3b7fe700 1 -- 172.21.15.3:0/842341550 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a3c098cf0 con 0x7f9a44106660 2023-09-10T01:19:31.837 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2023-09-10T01:19:31.839 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9a300a07b0 msgr2=0x7f9a300a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9a300a07b0 0x7f9a300a2c70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000b2e0 tx=0x7f9a40005fd0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.840 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a44106660 msgr2=0x7f9a44198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:31.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a44106660 0x7f9a44198750 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00efc0 tx=0x7f9a3c00c3a0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 shutdown_connections 2023-09-10T01:19:31.841 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f9a300a07b0 0x7f9a300a2c70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a44100650 0x7f9a4419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a44106660 0x7f9a44198750 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.842 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 --2- 172.21.15.3:0/842341550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a44101010 0x7f9a4419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:31.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.838+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 >> 172.21.15.3:0/842341550 conn(0x7f9a440fc160 msgr2=0x7f9a4410b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:31.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.839+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 shutdown_connections 2023-09-10T01:19:31.843 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:31.839+0000 7f9a4c5ff700 1 -- 172.21.15.3:0/842341550 wait complete. 2023-09-10T01:19:31.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-09-10T01:19:32.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 -- 172.21.15.3:0/1771789049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c106650 msgr2=0x7f7c3c106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1771789049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c106650 0x7f7c3c106a30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009990 tx=0x7f7c30009ca0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 -- 172.21.15.3:0/1771789049 shutdown_connections 2023-09-10T01:19:32.067 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1771789049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c106650 0x7f7c3c106a30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1771789049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c101000 0x7f7c3c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1771789049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c100640 0x7f7c3c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.068 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 -- 172.21.15.3:0/1771789049 >> 172.21.15.3:0/1771789049 conn(0x7f7c3c0fc170 msgr2=0x7f7c3c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:32.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 -- 172.21.15.3:0/1771789049 shutdown_connections 2023-09-10T01:19:32.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.065+0000 7f7c431ff700 1 -- 172.21.15.3:0/1771789049 wait complete. 2023-09-10T01:19:32.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.066+0000 7f7c431ff700 1 Processor -- start 2023-09-10T01:19:32.071 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.066+0000 7f7c431ff700 1 -- start start 2023-09-10T01:19:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c431ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c100640 0x7f7c3c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c431ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c101000 0x7f7c3c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c431ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c106650 0x7f7c3c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c431ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c198e80 con 0x7f7c3c106650 2023-09-10T01:19:32.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c431ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c198ff0 con 0x7f7c3c100640 2023-09-10T01:19:32.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c431ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1991b0 con 0x7f7c3c101000 2023-09-10T01:19:32.073 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c101000 0x7f7c3c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c40f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c100640 0x7f7c3c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c101000 0x7f7c3c19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41474/0 (socket says 172.21.15.3:41474) 2023-09-10T01:19:32.074 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c3bfff700 1 -- 172.21.15.3:0/1974864562 learned_addr learned my addr 172.21.15.3:0/1974864562 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:32.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.067+0000 7f7c4179c700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c106650 0x7f7c3c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.075 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c40f9b700 1 -- 172.21.15.3:0/1974864562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c101000 msgr2=0x7f7c3c19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c40f9b700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c101000 0x7f7c3c19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c40f9b700 1 -- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c106650 msgr2=0x7f7c3c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.076 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c40f9b700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c106650 0x7f7c3c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c40f9b700 1 -- 172.21.15.3:0/1974864562 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300095f0 con 0x7f7c3c100640 2023-09-10T01:19:32.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c4179c700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c106650 0x7f7c3c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:19:32.077 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c3bfff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c101000 0x7f7c3c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:19:32.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c40f9b700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c100640 0x7f7c3c19e660 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7c300059d0 tx=0x7f7c30005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:32.078 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c39ffb700 1 -- 172.21.15.3:0/1974864562 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3001d070 con 0x7f7c3c100640 2023-09-10T01:19:32.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c199430 con 0x7f7c3c100640 2023-09-10T01:19:32.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c39ffb700 1 -- 172.21.15.3:0/1974864562 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c30022470 con 0x7f7c3c100640 2023-09-10T01:19:32.079 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c39ffb700 1 -- 172.21.15.3:0/1974864562 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30017650 con 0x7f7c3c100640 2023-09-10T01:19:32.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.068+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c1a55e0 con 0x7f7c3c100640 2023-09-10T01:19:32.080 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.069+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c067480 con 0x7f7c3c100640 2023-09-10T01:19:32.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.072+0000 7f7c39ffb700 1 -- 172.21.15.3:0/1974864562 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7c3000fad0 con 0x7f7c3c100640 2023-09-10T01:19:32.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.073+0000 7f7c39ffb700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c2c0a07b0 0x7f7c2c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.081 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.073+0000 7f7c39ffb700 1 -- 172.21.15.3:0/1974864562 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f7c300cd9d0 con 0x7f7c3c100640 2023-09-10T01:19:32.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.073+0000 7f7c3bfff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c2c0a07b0 0x7f7c2c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.074+0000 7f7c3bfff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c2c0a07b0 0x7f7c2c0a2c70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c199ee0 tx=0x7f7c28009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:32.082 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.075+0000 7f7c39ffb700 1 -- 172.21.15.3:0/1974864562 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7c30096e30 con 0x7f7c3c100640 2023-09-10T01:19:32.289 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.288+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c3c199be0 con 0x7f7c3c100640 2023-09-10T01:19:32.290 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.288+0000 7f7c39ffb700 1 -- 172.21.15.3:0/1974864562 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c3009a450 con 0x7f7c3c100640 2023-09-10T01:19:32.290 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:19:32.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c2c0a07b0 msgr2=0x7f7c2c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.292 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c2c0a07b0 0x7f7c2c0a2c70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c199ee0 tx=0x7f7c28009260 comp rx=0 tx=0).stop 2023-09-10T01:19:32.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c100640 msgr2=0x7f7c3c19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c100640 0x7f7c3c19e660 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7c300059d0 tx=0x7f7c30005720 comp rx=0 tx=0).stop 2023-09-10T01:19:32.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 shutdown_connections 2023-09-10T01:19:32.293 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f7c2c0a07b0 0x7f7c2c0a2c70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c3c106650 0x7f7c3c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c3c100640 0x7f7c3c19e660 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.294 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 --2- 172.21.15.3:0/1974864562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c3c101000 0x7f7c3c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 >> 172.21.15.3:0/1974864562 conn(0x7f7c3c0fc170 msgr2=0x7f7c3c10b970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:32.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 shutdown_connections 2023-09-10T01:19:32.295 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.291+0000 7f7c431ff700 1 -- 172.21.15.3:0/1974864562 wait complete. 2023-09-10T01:19:32.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-09-10T01:19:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2327770656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc106650 msgr2=0x7f0bcc106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2327770656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc106650 0x7f0bcc106a30 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009990 tx=0x7f0bc0009ca0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2327770656 shutdown_connections 2023-09-10T01:19:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2327770656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc101000 0x7f0bcc10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2327770656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc106650 0x7f0bcc106a30 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2327770656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bcc100640 0x7f0bcc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2327770656 >> 172.21.15.3:0/2327770656 conn(0x7f0bcc0fc170 msgr2=0x7f0bcc0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:32.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2327770656 shutdown_connections 2023-09-10T01:19:32.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.518+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2327770656 wait complete. 2023-09-10T01:19:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 Processor -- start 2023-09-10T01:19:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 -- start start 2023-09-10T01:19:32.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bcc100640 0x7f0bcc196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc101000 0x7f0bcc196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc106650 0x7f0bcc19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc19a600 con 0x7f0bcc101000 2023-09-10T01:19:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc19a770 con 0x7f0bcc106650 2023-09-10T01:19:32.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.519+0000 7f0bd2bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc19a930 con 0x7f0bcc100640 2023-09-10T01:19:32.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc101000 0x7f0bcc196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.526 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bd099b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bcc100640 0x7f0bcc196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc101000 0x7f0bcc196c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37428/0 (socket says 172.21.15.3:37428) 2023-09-10T01:19:32.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 -- 172.21.15.3:0/2697685910 learned_addr learned my addr 172.21.15.3:0/2697685910 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:32.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bd119c700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc106650 0x7f0bcc19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.527 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 -- 172.21.15.3:0/2697685910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bcc100640 msgr2=0x7f0bcc196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bcc100640 0x7f0bcc196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 -- 172.21.15.3:0/2697685910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc106650 msgr2=0x7f0bcc19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.528 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc106650 0x7f0bcc19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 -- 172.21.15.3:0/2697685910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00095f0 con 0x7f0bcc101000 2023-09-10T01:19:32.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bd099b700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bcc100640 0x7f0bcc196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:19:32.529 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.520+0000 7f0bcbfff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc101000 0x7f0bcc196c50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800e9a0 tx=0x7f0bb800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:32.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.521+0000 7f0bc9ffb700 1 -- 172.21.15.3:0/2697685910 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8009790 con 0x7f0bcc101000 2023-09-10T01:19:32.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.521+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc19ac10 con 0x7f0bcc101000 2023-09-10T01:19:32.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.521+0000 7f0bc9ffb700 1 -- 172.21.15.3:0/2697685910 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb8010430 con 0x7f0bcc101000 2023-09-10T01:19:32.530 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.521+0000 7f0bc9ffb700 1 -- 172.21.15.3:0/2697685910 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8018680 con 0x7f0bcc101000 2023-09-10T01:19:32.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.521+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc1a3570 con 0x7f0bcc101000 2023-09-10T01:19:32.531 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.522+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc067480 con 0x7f0bcc101000 2023-09-10T01:19:32.532 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.524+0000 7f0bc9ffb700 1 -- 172.21.15.3:0/2697685910 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f0bb80105a0 con 0x7f0bcc101000 2023-09-10T01:19:32.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.525+0000 7f0bc9ffb700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0bbc0a07b0 0x7f0bbc0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.525+0000 7f0bc9ffb700 1 -- 172.21.15.3:0/2697685910 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f0bb8014070 con 0x7f0bcc101000 2023-09-10T01:19:32.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.526+0000 7f0bd099b700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0bbc0a07b0 0x7f0bbc0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.533 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.526+0000 7f0bd099b700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0bbc0a07b0 0x7f0bbc0a2c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000b2e0 tx=0x7f0bc0005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:32.534 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.530+0000 7f0bc9ffb700 1 -- 172.21.15.3:0/2697685910 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0bb8051e10 con 0x7f0bcc101000 2023-09-10T01:19:32.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:32 smithi136 ceph-mon[133708]: pgmap v515: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 66 KiB/s wr, 15 op/s 2023-09-10T01:19:32.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:32 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2043617512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:19:32.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:32 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/842341550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:19:32.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:32 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1974864562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:19:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:32 smithi092 ceph-mon[134067]: pgmap v515: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 66 KiB/s wr, 15 op/s 2023-09-10T01:19:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:32 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2043617512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:19:32.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:32 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/842341550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:19:32.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:32 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1974864562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:19:32.739 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.738+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0bcc1105b0 con 0x7f0bcc101000 2023-09-10T01:19:32.740 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.739+0000 7f0bc9ffb700 1 -- 172.21.15.3:0/2697685910 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0bb8095940 con 0x7f0bcc101000 2023-09-10T01:19:32.740 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2023-09-10T01:19:32.742 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0bbc0a07b0 msgr2=0x7f0bbc0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0bbc0a07b0 0x7f0bbc0a2c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000b2e0 tx=0x7f0bc0005fb0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc101000 msgr2=0x7f0bcc196c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.743 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc101000 0x7f0bcc196c50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800e9a0 tx=0x7f0bb800ecb0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 shutdown_connections 2023-09-10T01:19:32.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0bbc0a07b0 0x7f0bbc0a2c70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bcc100640 0x7f0bcc196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.744 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc106650 0x7f0bcc19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 --2- 172.21.15.3:0/2697685910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bcc101000 0x7f0bcc196c50 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 >> 172.21.15.3:0/2697685910 conn(0x7f0bcc0fc170 msgr2=0x7f0bcc104150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:32.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 shutdown_connections 2023-09-10T01:19:32.745 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.741+0000 7f0bd2bff700 1 -- 172.21.15.3:0/2697685910 wait complete. 2023-09-10T01:19:32.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:32 smithi003 ceph-mon[97654]: pgmap v515: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 66 KiB/s wr, 15 op/s 2023-09-10T01:19:32.759 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:32 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2043617512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:19:32.759 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:32 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/842341550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-09-10T01:19:32.759 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:32 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1974864562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-09-10T01:19:32.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-09-10T01:19:32.966 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.965+0000 7fca47fff700 1 -- 172.21.15.3:0/1922425685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 msgr2=0x7fca4010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.965+0000 7fca47fff700 1 --2- 172.21.15.3:0/1922425685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca4010f2f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fca3800b0a0 tx=0x7fca3800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.967 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.966+0000 7fca47fff700 1 -- 172.21.15.3:0/1922425685 shutdown_connections 2023-09-10T01:19:32.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.966+0000 7fca47fff700 1 --2- 172.21.15.3:0/1922425685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca40072a30 0x7fca40072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.966+0000 7fca47fff700 1 --2- 172.21.15.3:0/1922425685 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca4010f2f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.966+0000 7fca47fff700 1 --2- 172.21.15.3:0/1922425685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca40073350 0x7fca4010c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.968 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.966+0000 7fca47fff700 1 -- 172.21.15.3:0/1922425685 >> 172.21.15.3:0/1922425685 conn(0x7fca400fc170 msgr2=0x7fca400fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:32.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.966+0000 7fca47fff700 1 -- 172.21.15.3:0/1922425685 shutdown_connections 2023-09-10T01:19:32.969 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.966+0000 7fca47fff700 1 -- 172.21.15.3:0/1922425685 wait complete. 2023-09-10T01:19:32.971 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 Processor -- start 2023-09-10T01:19:32.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 -- start start 2023-09-10T01:19:32.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca40072a30 0x7fca4019e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca40073350 0x7fca4019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.972 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca40198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40198ee0 con 0x7fca4010ce80 2023-09-10T01:19:32.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40199050 con 0x7fca40073350 2023-09-10T01:19:32.973 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca47fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40199210 con 0x7fca40072a30 2023-09-10T01:19:32.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.967+0000 7fca4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca40198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca40198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37454/0 (socket says 172.21.15.3:37454) 2023-09-10T01:19:32.974 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4559a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca40073350 0x7fca4019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 -- 172.21.15.3:0/3959136269 learned_addr learned my addr 172.21.15.3:0/3959136269 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:32.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca45d9b700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca40072a30 0x7fca4019e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.975 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 -- 172.21.15.3:0/3959136269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca40072a30 msgr2=0x7fca4019e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca40072a30 0x7fca4019e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 -- 172.21.15.3:0/3959136269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca40073350 msgr2=0x7fca4019ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:32.976 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca40073350 0x7fca4019ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:32.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 -- 172.21.15.3:0/3959136269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38009d00 con 0x7fca4010ce80 2023-09-10T01:19:32.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca4659c700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca40198870 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fca38009fd0 tx=0x7fca3800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:32.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca36ffd700 1 -- 172.21.15.3:0/3959136269 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3800e040 con 0x7fca4010ce80 2023-09-10T01:19:32.977 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca36ffd700 1 -- 172.21.15.3:0/3959136269 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca380094c0 con 0x7fca4010ce80 2023-09-10T01:19:32.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca40199490 con 0x7fca4010ce80 2023-09-10T01:19:32.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.968+0000 7fca36ffd700 1 -- 172.21.15.3:0/3959136269 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca380052a0 con 0x7fca4010ce80 2023-09-10T01:19:32.978 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.969+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca401a57f0 con 0x7fca4010ce80 2023-09-10T01:19:32.979 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.970+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca401077a0 con 0x7fca4010ce80 2023-09-10T01:19:32.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.971+0000 7fca36ffd700 1 -- 172.21.15.3:0/3959136269 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fca38003a00 con 0x7fca4010ce80 2023-09-10T01:19:32.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.974+0000 7fca36ffd700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fca2c0a07b0 0x7fca2c0a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:32.980 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.974+0000 7fca36ffd700 1 -- 172.21.15.3:0/3959136269 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7fca380cda30 con 0x7fca4010ce80 2023-09-10T01:19:32.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.974+0000 7fca45d9b700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fca2c0a07b0 0x7fca2c0a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:32.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.975+0000 7fca45d9b700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fca2c0a07b0 0x7fca2c0a2c70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009c60 tx=0x7fca3c009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:32.981 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:32.976+0000 7fca36ffd700 1 -- 172.21.15.3:0/3959136269 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fca38096e90 con 0x7fca4010ce80 2023-09-10T01:19:33.189 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.189+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fca401127f0 con 0x7fca4010ce80 2023-09-10T01:19:33.190 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.189+0000 7fca36ffd700 1 -- 172.21.15.3:0/3959136269 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v114) v1 ==== 103+0+11 (secure 0 0 0) 0x7fca38003cb0 con 0x7fca4010ce80 2023-09-10T01:19:33.191 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2023-09-10T01:19:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fca2c0a07b0 msgr2=0x7fca2c0a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fca2c0a07b0 0x7fca2c0a2c70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009c60 tx=0x7fca3c009260 comp rx=0 tx=0).stop 2023-09-10T01:19:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 msgr2=0x7fca40198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca40198870 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fca38009fd0 tx=0x7fca3800bce0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 shutdown_connections 2023-09-10T01:19:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fca2c0a07b0 0x7fca2c0a2c70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.194 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca40072a30 0x7fca4019e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fca4010ce80 0x7fca40198870 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 --2- 172.21.15.3:0/3959136269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca40073350 0x7fca4019ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 >> 172.21.15.3:0/3959136269 conn(0x7fca400fc170 msgr2=0x7fca401048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:33.195 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 shutdown_connections 2023-09-10T01:19:33.196 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.192+0000 7fca47fff700 1 -- 172.21.15.3:0/3959136269 wait complete. 2023-09-10T01:19:33.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-09-10T01:19:33.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.418+0000 7f13743ff700 1 -- 172.21.15.3:0/3915683701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c106650 msgr2=0x7f136c106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:33.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.418+0000 7f13743ff700 1 --2- 172.21.15.3:0/3915683701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c106650 0x7f136c106a30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1368009990 tx=0x7f1368009ca0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.420 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.419+0000 7f13743ff700 1 -- 172.21.15.3:0/3915683701 shutdown_connections 2023-09-10T01:19:33.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.419+0000 7f13743ff700 1 --2- 172.21.15.3:0/3915683701 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f136c101000 0x7f136c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.419+0000 7f13743ff700 1 --2- 172.21.15.3:0/3915683701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c106650 0x7f136c106a30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.421 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.419+0000 7f13743ff700 1 --2- 172.21.15.3:0/3915683701 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136c100640 0x7f136c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.419+0000 7f13743ff700 1 -- 172.21.15.3:0/3915683701 >> 172.21.15.3:0/3915683701 conn(0x7f136c0fc170 msgr2=0x7f136c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:33.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.419+0000 7f13743ff700 1 -- 172.21.15.3:0/3915683701 shutdown_connections 2023-09-10T01:19:33.422 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.419+0000 7f13743ff700 1 -- 172.21.15.3:0/3915683701 wait complete. 2023-09-10T01:19:33.424 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 Processor -- start 2023-09-10T01:19:33.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 -- start start 2023-09-10T01:19:33.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c100640 0x7f136c072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:33.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f136c101000 0x7f136c072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:33.425 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136c106650 0x7f136c1a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:33.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c073670 con 0x7f136c100640 2023-09-10T01:19:33.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c1a4e10 con 0x7f136c101000 2023-09-10T01:19:33.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.420+0000 7f13743ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c1a4f50 con 0x7f136c106650 2023-09-10T01:19:33.426 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c100640 0x7f136c072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:33.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137299c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136c106650 0x7f136c1a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:33.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c100640 0x7f136c072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37476/0 (socket says 172.21.15.3:37476) 2023-09-10T01:19:33.427 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 -- 172.21.15.3:0/2665304638 learned_addr learned my addr 172.21.15.3:0/2665304638 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:33.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137199a700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f136c101000 0x7f136c072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:33.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 -- 172.21.15.3:0/2665304638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136c106650 msgr2=0x7f136c1a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:33.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136c106650 0x7f136c1a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.428 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 -- 172.21.15.3:0/2665304638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f136c101000 msgr2=0x7f136c072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:33.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f136c101000 0x7f136c072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 -- 172.21.15.3:0/2665304638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13680095f0 con 0x7f136c100640 2023-09-10T01:19:33.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.421+0000 7f137219b700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c100640 0x7f136c072900 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1368005910 tx=0x7f1368005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:33.429 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.422+0000 7f13637fe700 1 -- 172.21.15.3:0/2665304638 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136801d070 con 0x7f136c100640 2023-09-10T01:19:33.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.422+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c1a5150 con 0x7f136c100640 2023-09-10T01:19:33.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.422+0000 7f13637fe700 1 -- 172.21.15.3:0/2665304638 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1368022470 con 0x7f136c100640 2023-09-10T01:19:33.430 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.422+0000 7f13637fe700 1 -- 172.21.15.3:0/2665304638 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368017650 con 0x7f136c100640 2023-09-10T01:19:33.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.422+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136c1a5630 con 0x7f136c100640 2023-09-10T01:19:33.431 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.423+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136c067480 con 0x7f136c100640 2023-09-10T01:19:33.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.425+0000 7f13637fe700 1 -- 172.21.15.3:0/2665304638 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f136800fad0 con 0x7f136c100640 2023-09-10T01:19:33.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.426+0000 7f13637fe700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f13580a07b0 0x7f13580a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:33.432 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.426+0000 7f13637fe700 1 -- 172.21.15.3:0/2665304638 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f13680cd990 con 0x7f136c100640 2023-09-10T01:19:33.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.427+0000 7f137199a700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f13580a07b0 0x7f13580a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:33.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.427+0000 7f137199a700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f13580a07b0 0x7f13580a2c70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f136c0740f0 tx=0x7f135c00b410 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:33.433 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.430+0000 7f13637fe700 1 -- 172.21.15.3:0/2665304638 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f1368096f10 con 0x7f136c100640 2023-09-10T01:19:33.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:33 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2697685910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:19:33.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:33 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3959136269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:19:33.661 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.660+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f136c073e30 con 0x7f136c100640 2023-09-10T01:19:33.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:33 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2697685910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:19:33.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:33 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3959136269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:19:33.663 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.661+0000 7f13637fe700 1 -- 172.21.15.3:0/2665304638 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v114) v1 ==== 81+0+919 (secure 0 0 0) 0x7f1368027020 con 0x7f136c100640 2023-09-10T01:19:33.663 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/3516755706 2023-09-11T01:03:12.189996+0000 2023-09-10T01:19:33.663 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1056299694 2023-09-11T01:03:12.189996+0000 2023-09-10T01:19:33.664 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/91094391 2023-09-11T01:03:12.189996+0000 2023-09-10T01:19:33.664 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/91094391 2023-09-11T01:03:12.189996+0000 2023-09-10T01:19:33.664 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/2575312228 2023-09-11T01:03:12.189996+0000 2023-09-10T01:19:33.665 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/2296857090 2023-09-11T01:03:12.189996+0000 2023-09-10T01:19:33.665 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/1916209618 2023-09-11T00:53:55.752989+0000 2023-09-10T01:19:33.665 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/70910038 2023-09-11T00:53:28.332636+0000 2023-09-10T01:19:33.665 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/71687535 2023-09-11T00:53:28.332636+0000 2023-09-10T01:19:33.666 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/3193301911 2023-09-11T00:53:28.332636+0000 2023-09-10T01:19:33.666 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1972995090 2023-09-11T00:53:55.752989+0000 2023-09-10T01:19:33.666 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/4153284543 2023-09-11T00:53:28.332636+0000 2023-09-10T01:19:33.666 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1689603322 2023-09-11T00:53:55.752989+0000 2023-09-10T01:19:33.667 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/1916209618 2023-09-11T00:53:55.752989+0000 2023-09-10T01:19:33.667 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/70910038 2023-09-11T00:53:28.332636+0000 2023-09-10T01:19:33.667 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/886409476 2023-09-11T00:53:55.752989+0000 2023-09-10T01:19:33.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f13580a07b0 msgr2=0x7f13580a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:33.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f13580a07b0 0x7f13580a2c70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f136c0740f0 tx=0x7f135c00b410 comp rx=0 tx=0).stop 2023-09-10T01:19:33.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c100640 msgr2=0x7f136c072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:33.669 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c100640 0x7f136c072900 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1368005910 tx=0x7f1368005740 comp rx=0 tx=0).stop 2023-09-10T01:19:33.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 shutdown_connections 2023-09-10T01:19:33.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f13580a07b0 0x7f13580a2c70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.670 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136c106650 0x7f136c1a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f136c100640 0x7f136c072900 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 --2- 172.21.15.3:0/2665304638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f136c101000 0x7f136c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:33.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 >> 172.21.15.3:0/2665304638 conn(0x7f136c0fc170 msgr2=0x7f136c0fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:33.671 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 shutdown_connections 2023-09-10T01:19:33.672 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:33.664+0000 7f13743ff700 1 -- 172.21.15.3:0/2665304638 wait complete. 2023-09-10T01:19:33.672 INFO:teuthology.orchestra.run.smithi003.stderr:listed 16 entries 2023-09-10T01:19:33.683 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-09-10T01:19:33.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:19:33.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-09-10T01:19:33.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:33 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2697685910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-09-10T01:19:33.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:33 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3959136269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-09-10T01:19:33.892 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:19:33.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:19:33.893 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-09-10T01:19:34.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:19:34.023 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2023-09-10T01:19:34.067 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-09-10T01:19:34.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:19:34.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-09-10T01:19:34.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-10T01:19:34.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2023-09-10T01:19:34.285 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-09-10T01:19:34.297 INFO:tasks.cephadm:Teardown begin 2023-09-10T01:19:34.297 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-09-10T01:19:34.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:19:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.508+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/1913517988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14100650 msgr2=0x7f0b14100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.508+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/1913517988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14100650 0x7f0b14100ad0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0b040098f0 tx=0x7f0b04009c00 comp rx=0 tx=0).stop 2023-09-10T01:19:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.509+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/1913517988 shutdown_connections 2023-09-10T01:19:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.509+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/1913517988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b14101010 0x7f0b1410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.509+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/1913517988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14106660 0x7f0b14106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.509+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/1913517988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14100650 0x7f0b14100ad0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.509+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/1913517988 >> 172.21.15.3:0/1913517988 conn(0x7f0b140fc160 msgr2=0x7f0b140fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.509+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/1913517988 shutdown_connections 2023-09-10T01:19:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.509+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/1913517988 wait complete. 2023-09-10T01:19:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 Processor -- start 2023-09-10T01:19:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 -- start start 2023-09-10T01:19:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14100650 0x7f0b1419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14101010 0x7f0b1419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b14106660 0x7f0b141965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14196c40 con 0x7f0b14100650 2023-09-10T01:19:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14196db0 con 0x7f0b14101010 2023-09-10T01:19:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.510+0000 7f0b1c3ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14196f70 con 0x7f0b14106660 2023-09-10T01:19:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14100650 0x7f0b1419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a99c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b14106660 0x7f0b141965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:34.517 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a99c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b14106660 0x7f0b141965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:41568/0 (socket says 172.21.15.3:41568) 2023-09-10T01:19:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1999a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14101010 0x7f0b1419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a99c700 1 -- 172.21.15.3:0/580372960 learned_addr learned my addr 172.21.15.3:0/580372960 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14100650 0x7f0b1419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37496/0 (socket says 172.21.15.3:37496) 2023-09-10T01:19:34.518 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 -- 172.21.15.3:0/580372960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b14106660 msgr2=0x7f0b141965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b14106660 0x7f0b141965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 -- 172.21.15.3:0/580372960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14101010 msgr2=0x7f0b1419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:34.519 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14101010 0x7f0b1419caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 -- 172.21.15.3:0/580372960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100095f0 con 0x7f0b14100650 2023-09-10T01:19:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.511+0000 7f0b1a19b700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14100650 0x7f0b1419c560 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000eb90 tx=0x7f0b1000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.512+0000 7f0b0b7fe700 1 -- 172.21.15.3:0/580372960 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b10009970 con 0x7f0b14100650 2023-09-10T01:19:34.520 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.512+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b04009520 con 0x7f0b14100650 2023-09-10T01:19:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.512+0000 7f0b0b7fe700 1 -- 172.21.15.3:0/580372960 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b100044e0 con 0x7f0b14100650 2023-09-10T01:19:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.512+0000 7f0b0b7fe700 1 -- 172.21.15.3:0/580372960 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b10003e50 con 0x7f0b14100650 2023-09-10T01:19:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.512+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b14197410 con 0x7f0b14100650 2023-09-10T01:19:34.521 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.513+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b14067480 con 0x7f0b14100650 2023-09-10T01:19:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.515+0000 7f0b0b7fe700 1 -- 172.21.15.3:0/580372960 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f0b10004000 con 0x7f0b14100650 2023-09-10T01:19:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.516+0000 7f0b0b7fe700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b000a0830 0x7f0b000a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.516+0000 7f0b0b7fe700 1 -- 172.21.15.3:0/580372960 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f0b10014070 con 0x7f0b14100650 2023-09-10T01:19:34.523 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.516+0000 7f0b1999a700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b000a0830 0x7f0b000a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:34.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.517+0000 7f0b1999a700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b000a0830 0x7f0b000a2cf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0b04005e90 tx=0x7f0b04019040 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:34.524 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.520+0000 7f0b0b7fe700 1 -- 172.21.15.3:0/580372960 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f0b10095b50 con 0x7f0b14100650 2023-09-10T01:19:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:34 smithi003 ceph-mon[97654]: pgmap v516: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 66 KiB/s wr, 15 op/s 2023-09-10T01:19:34.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:34 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2665304638' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:19:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:34 smithi136 ceph-mon[133708]: pgmap v516: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 66 KiB/s wr, 15 op/s 2023-09-10T01:19:34.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:34 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2665304638' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:19:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:34 smithi092 ceph-mon[134067]: pgmap v516: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 66 KiB/s wr, 15 op/s 2023-09-10T01:19:34.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:34 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2665304638' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-09-10T01:19:34.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.773+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b14197ae0 con 0x7f0b14100650 2023-09-10T01:19:34.774 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.774+0000 7f0b0b7fe700 1 -- 172.21.15.3:0/580372960 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+11298 (secure 0 0 0) 0x7f0b10099170 con 0x7f0b14100650 2023-09-10T01:19:34.775 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:19:34.775 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:19:23.109006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:19:34.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b000a0830 msgr2=0x7f0b000a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:34.777 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b000a0830 0x7f0b000a2cf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0b04005e90 tx=0x7f0b04019040 comp rx=0 tx=0).stop 2023-09-10T01:19:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14100650 msgr2=0x7f0b1419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14100650 0x7f0b1419c560 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000eb90 tx=0x7f0b1000eea0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.778 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 shutdown_connections 2023-09-10T01:19:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f0b000a0830 0x7f0b000a2cf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b14106660 0x7f0b141965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b14100650 0x7f0b1419c560 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.779 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 --2- 172.21.15.3:0/580372960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b14101010 0x7f0b1419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 >> 172.21.15.3:0/580372960 conn(0x7f0b140fc160 msgr2=0x7f0b140fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 shutdown_connections 2023-09-10T01:19:34.780 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.776+0000 7f0b1c3ff700 1 -- 172.21.15.3:0/580372960 wait complete. 2023-09-10T01:19:34.781 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 42 2023-09-10T01:19:34.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-09-10T01:19:35.000 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 -- 172.21.15.3:0/3857278688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 msgr2=0x7f00fc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 --2- 172.21.15.3:0/3857278688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc100aa0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009990 tx=0x7f00e8009ca0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 -- 172.21.15.3:0/3857278688 shutdown_connections 2023-09-10T01:19:35.001 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 --2- 172.21.15.3:0/3857278688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc100fe0 0x7f00fc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 --2- 172.21.15.3:0/3857278688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00fc106630 0x7f00fc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 --2- 172.21.15.3:0/3857278688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc100aa0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.002 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 -- 172.21.15.3:0/3857278688 >> 172.21.15.3:0/3857278688 conn(0x7f00fc0fc110 msgr2=0x7f00fc0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:35.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 -- 172.21.15.3:0/3857278688 shutdown_connections 2023-09-10T01:19:35.003 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:34.999+0000 7f0102bff700 1 -- 172.21.15.3:0/3857278688 wait complete. 2023-09-10T01:19:35.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.000+0000 7f0102bff700 1 Processor -- start 2023-09-10T01:19:35.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.000+0000 7f0102bff700 1 -- start start 2023-09-10T01:19:35.005 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f0102bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f0102bff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00fc100fe0 0x7f00fc072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f0102bff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc106630 0x7f00fc073380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f0102bff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1a9130 con 0x7f00fc100620 2023-09-10T01:19:35.006 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f0102bff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1a9270 con 0x7f00fc100fe0 2023-09-10T01:19:35.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f0102bff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1a93b0 con 0x7f00fc106630 2023-09-10T01:19:35.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010099b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010099b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49380/0 (socket says 172.21.15.3:49380) 2023-09-10T01:19:35.007 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010099b700 1 -- 172.21.15.3:0/1187639627 learned_addr learned my addr 172.21.15.3:0/1187639627 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:35.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f00fbfff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00fc100fe0 0x7f00fc072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010119c700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc106630 0x7f00fc073380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010099b700 1 -- 172.21.15.3:0/1187639627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc106630 msgr2=0x7f00fc073380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.008 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010099b700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc106630 0x7f00fc073380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010099b700 1 -- 172.21.15.3:0/1187639627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00fc100fe0 msgr2=0x7f00fc072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.001+0000 7f010099b700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00fc100fe0 0x7f00fc072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.002+0000 7f010099b700 1 -- 172.21.15.3:0/1187639627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f0009c40 con 0x7f00fc100620 2023-09-10T01:19:35.009 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.002+0000 7f010099b700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc072900 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c370 tx=0x7f00f000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:35.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.002+0000 7f00f9ffb700 1 -- 172.21.15.3:0/1187639627 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f0017070 con 0x7f00fc100620 2023-09-10T01:19:35.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.002+0000 7f00f9ffb700 1 -- 172.21.15.3:0/1187639627 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00f000f040 con 0x7f00fc100620 2023-09-10T01:19:35.010 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.002+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e80095f0 con 0x7f00fc100620 2023-09-10T01:19:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.002+0000 7f00f9ffb700 1 -- 172.21.15.3:0/1187639627 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f000e050 con 0x7f00fc100620 2023-09-10T01:19:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.002+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc1a9a00 con 0x7f00fc100620 2023-09-10T01:19:35.011 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.003+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc067480 con 0x7f00fc100620 2023-09-10T01:19:35.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.005+0000 7f00f9ffb700 1 -- 172.21.15.3:0/1187639627 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f00f0004440 con 0x7f00fc100620 2023-09-10T01:19:35.012 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.007+0000 7f00f9ffb700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f00ec0a0830 0x7f00ec0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.007+0000 7f00f9ffb700 1 -- 172.21.15.3:0/1187639627 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f00f00cbca0 con 0x7f00fc100620 2023-09-10T01:19:35.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.007+0000 7f00fbfff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f00ec0a0830 0x7f00ec0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.008+0000 7f00fbfff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f00ec0a0830 0x7f00ec0a2cf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0740f0 tx=0x7f00e8005fd0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:35.013 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.011+0000 7f00f9ffb700 1 -- 172.21.15.3:0/1187639627 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f00f0095070 con 0x7f00fc100620 2023-09-10T01:19:35.266 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.266+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00fc073e30 con 0x7f00fc100620 2023-09-10T01:19:35.267 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.267+0000 7f00f9ffb700 1 -- 172.21.15.3:0/1187639627 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+11298 (secure 0 0 0) 0x7f00f0098690 con 0x7f00fc100620 2023-09-10T01:19:35.268 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:19:35.268 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14574,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/2642590493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":2642590493},{"type":"v1","addr":"172.21.15.3:6835","nonce":2642590493}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/3588843640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":3588843640},{"type":"v1","addr":"172.21.15.3:6837","nonce":3588843640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/640397108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":640397108},{"type":"v1","addr":"172.21.15.3:6839","nonce":640397108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/1239897501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":1239897501},{"type":"v1","addr":"172.21.15.3:6841","nonce":1239897501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24562,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3536147716","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3536147716},{"type":"v1","addr":"172.21.15.136:6833","nonce":3536147716}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24581,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1626913605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1626913605},{"type":"v1","addr":"172.21.15.92:6833","nonce":1626913605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-09-10T01:00:43.531603+0000","modified":"2023-09-10T01:19:23.109006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24698,"mds_1":24647,"mds_2":24634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24604":{"gid":24604,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6835/662717057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":662717057},{"type":"v1","addr":"172.21.15.136:6835","nonce":662717057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24610":{"gid":24610,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3841438021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3841438021},{"type":"v1","addr":"172.21.15.136:6837","nonce":3841438021}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24620":{"gid":24620,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/790897161","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":790897161},{"type":"v1","addr":"172.21.15.92:6835","nonce":790897161}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24634":{"gid":24634,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.136:6839/1259564848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1259564848},{"type":"v1","addr":"172.21.15.136:6839","nonce":1259564848}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24647":{"gid":24647,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.92:6837/12010426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":12010426},{"type":"v1","addr":"172.21.15.92:6837","nonce":12010426}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24698":{"gid":24698,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/149607042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":149607042},{"type":"v1","addr":"172.21.15.92:6839","nonce":149607042}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-09-10T01:19:35.269 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f00ec0a0830 msgr2=0x7f00ec0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f00ec0a0830 0x7f00ec0a2cf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0740f0 tx=0x7f00e8005fd0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 msgr2=0x7f00fc072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.270 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc072900 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c370 tx=0x7f00f000c680 comp rx=0 tx=0).stop 2023-09-10T01:19:35.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 shutdown_connections 2023-09-10T01:19:35.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f00ec0a0830 0x7f00ec0a2cf0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.271 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc106630 0x7f00fc073380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00fc100620 0x7f00fc072900 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 --2- 172.21.15.3:0/1187639627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00fc100fe0 0x7f00fc072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 >> 172.21.15.3:0/1187639627 conn(0x7f00fc0fc110 msgr2=0x7f00fc0fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:35.272 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 shutdown_connections 2023-09-10T01:19:35.273 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.269+0000 7f0102bff700 1 -- 172.21.15.3:0/1187639627 wait complete. 2023-09-10T01:19:35.273 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 42 2023-09-10T01:19:35.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-10T01:19:35.497 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 -- 172.21.15.3:0/600971629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f124410ce70 msgr2=0x7f124410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 --2- 172.21.15.3:0/600971629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f124410ce70 0x7f124410f2e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f123c00b0a0 tx=0x7f123c00b3b0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 -- 172.21.15.3:0/600971629 shutdown_connections 2023-09-10T01:19:35.498 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 --2- 172.21.15.3:0/600971629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f124410ce70 0x7f124410f2e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 --2- 172.21.15.3:0/600971629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1244072a30 0x7f1244072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 --2- 172.21.15.3:0/600971629 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244073350 0x7f124410c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.499 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 -- 172.21.15.3:0/600971629 >> 172.21.15.3:0/600971629 conn(0x7f12440fc140 msgr2=0x7f12440fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:35.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 -- 172.21.15.3:0/600971629 shutdown_connections 2023-09-10T01:19:35.500 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.496+0000 7f124c5ff700 1 -- 172.21.15.3:0/600971629 wait complete. 2023-09-10T01:19:35.502 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.497+0000 7f124c5ff700 1 Processor -- start 2023-09-10T01:19:35.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.497+0000 7f124c5ff700 1 -- start start 2023-09-10T01:19:35.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124c5ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244072a30 0x7f124419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124c5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1244073350 0x7f124419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.503 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124c5ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f124410ce70 0x7f12441988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124c5ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244198f20 con 0x7f1244073350 2023-09-10T01:19:35.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124c5ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244199090 con 0x7f124410ce70 2023-09-10T01:19:35.504 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124c5ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244199250 con 0x7f1244072a30 2023-09-10T01:19:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124a39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244072a30 0x7f124419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124a39b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244072a30 0x7f124419e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:32948/0 (socket says 172.21.15.3:32948) 2023-09-10T01:19:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124a39b700 1 -- 172.21.15.3:0/3875275451 learned_addr learned my addr 172.21.15.3:0/3875275451 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f1249b9a700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1244073350 0x7f124419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.498+0000 7f124ab9c700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f124410ce70 0x7f12441988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124a39b700 1 -- 172.21.15.3:0/3875275451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f124410ce70 msgr2=0x7f12441988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.506 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124a39b700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f124410ce70 0x7f12441988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124a39b700 1 -- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1244073350 msgr2=0x7f124419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124a39b700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1244073350 0x7f124419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124a39b700 1 -- 172.21.15.3:0/3875275451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c009d00 con 0x7f1244072a30 2023-09-10T01:19:35.507 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124ab9c700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f124410ce70 0x7f12441988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-10T01:19:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124a39b700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244072a30 0x7f124419e840 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f124000d950 tx=0x7f124000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f123b7fe700 1 -- 172.21.15.3:0/3875275451 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1240009940 con 0x7f1244072a30 2023-09-10T01:19:35.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1244199530 con 0x7f1244072a30 2023-09-10T01:19:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f123b7fe700 1 -- 172.21.15.3:0/3875275451 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1240010460 con 0x7f1244072a30 2023-09-10T01:19:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f123b7fe700 1 -- 172.21.15.3:0/3875275451 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124000f620 con 0x7f1244072a30 2023-09-10T01:19:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.499+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12441a5830 con 0x7f1244072a30 2023-09-10T01:19:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.500+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1244067480 con 0x7f1244072a30 2023-09-10T01:19:35.510 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.502+0000 7f123b7fe700 1 -- 172.21.15.3:0/3875275451 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f12400108d0 con 0x7f1244072a30 2023-09-10T01:19:35.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.504+0000 7f123b7fe700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12300a07b0 0x7f12300a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.504+0000 7f123b7fe700 1 -- 172.21.15.3:0/3875275451 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f12400cc210 con 0x7f1244072a30 2023-09-10T01:19:35.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.504+0000 7f1249b9a700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12300a07b0 0x7f12300a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.505+0000 7f1249b9a700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12300a07b0 0x7f12300a2c70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f124410e3a0 tx=0x7f1234009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:35.512 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.506+0000 7f123b7fe700 1 -- 172.21.15.3:0/3875275451 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f12400955e0 con 0x7f1244072a30 2023-09-10T01:19:35.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:35 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/580372960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:19:35.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:35 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1187639627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:19:35.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:35 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/580372960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:19:35.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:35 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1187639627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:19:35.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:35 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/580372960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:19:35.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:35 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1187639627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-09-10T01:19:35.721 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.720+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12441127e0 con 0x7f1244072a30 2023-09-10T01:19:35.723 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.722+0000 7f123b7fe700 1 -- 172.21.15.3:0/3875275451 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+25078 (secure 0 0 0) 0x7f124007f210 con 0x7f1244072a30 2023-09-10T01:19:35.724 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-09-10T01:19:35.724 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":114,"fsid":"4260c16a-4f74-11ee-9ab7-7b867c8bd7da","created":"2023-09-10T00:52:53.067114+0000","modified":"2023-09-10T01:19:25.328137+0000","last_up_change":"2023-09-10T00:59:57.701042+0000","last_in_change":"2023-09-10T00:59:40.253179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-10T00:56:36.373564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-09-10T01:00:41.489409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-09-10T01:00:42.983230+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-09-10T01:00:46.555744+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaca1361-1d53-4fff-b909-d973a1ce185a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6803","nonce":3367767236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6805","nonce":3367767236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6809","nonce":3367767236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3367767236},{"type":"v1","addr":"172.21.15.3:6807","nonce":3367767236}]},"public_addr":"172.21.15.3:6803/3367767236","cluster_addr":"172.21.15.3:6805/3367767236","heartbeat_back_addr":"172.21.15.3:6809/3367767236","heartbeat_front_addr":"172.21.15.3:6807/3367767236","state":["exists","up"]},{"osd":1,"uuid":"e1f77daa-c19a-4eba-aebf-27ca13859ee5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6801","nonce":4007866052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6803","nonce":4007866052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6807","nonce":4007866052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4007866052},{"type":"v1","addr":"172.21.15.92:6805","nonce":4007866052}]},"public_addr":"172.21.15.92:6801/4007866052","cluster_addr":"172.21.15.92:6803/4007866052","heartbeat_back_addr":"172.21.15.92:6807/4007866052","heartbeat_front_addr":"172.21.15.92:6805/4007866052","state":["exists","up"]},{"osd":2,"uuid":"2c7848b8-ced5-42a8-b5e1-59a572c3fca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6801","nonce":781257073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6803","nonce":781257073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6807","nonce":781257073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":781257073},{"type":"v1","addr":"172.21.15.136:6805","nonce":781257073}]},"public_addr":"172.21.15.136:6801/781257073","cluster_addr":"172.21.15.136:6803/781257073","heartbeat_back_addr":"172.21.15.136:6807/781257073","heartbeat_front_addr":"172.21.15.136:6805/781257073","state":["exists","up"]},{"osd":3,"uuid":"c08fe6de-b501-43d2-bcf3-2ce57eb464f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6811","nonce":434967290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6813","nonce":434967290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6817","nonce":434967290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":434967290},{"type":"v1","addr":"172.21.15.3:6815","nonce":434967290}]},"public_addr":"172.21.15.3:6811/434967290","cluster_addr":"172.21.15.3:6813/434967290","heartbeat_back_addr":"172.21.15.3:6817/434967290","heartbeat_front_addr":"172.21.15.3:6815/434967290","state":["exists","up"]},{"osd":4,"uuid":"b1cf565e-0d0a-49ec-b39b-6dbee04e586f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6809","nonce":2892874326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6811","nonce":2892874326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6815","nonce":2892874326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2892874326},{"type":"v1","addr":"172.21.15.92:6813","nonce":2892874326}]},"public_addr":"172.21.15.92:6809/2892874326","cluster_addr":"172.21.15.92:6811/2892874326","heartbeat_back_addr":"172.21.15.92:6815/2892874326","heartbeat_front_addr":"172.21.15.92:6813/2892874326","state":["exists","up"]},{"osd":5,"uuid":"eb4bd187-8a27-479f-a3c3-018993c8d986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6809","nonce":1094245369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6811","nonce":1094245369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6815","nonce":1094245369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":1094245369},{"type":"v1","addr":"172.21.15.136:6813","nonce":1094245369}]},"public_addr":"172.21.15.136:6809/1094245369","cluster_addr":"172.21.15.136:6811/1094245369","heartbeat_back_addr":"172.21.15.136:6815/1094245369","heartbeat_front_addr":"172.21.15.136:6813/1094245369","state":["exists","up"]},{"osd":6,"uuid":"6a4f8fc3-430f-4dbc-86f7-00e9a6e1d8b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6819","nonce":989795778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6821","nonce":989795778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6825","nonce":989795778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":989795778},{"type":"v1","addr":"172.21.15.3:6823","nonce":989795778}]},"public_addr":"172.21.15.3:6819/989795778","cluster_addr":"172.21.15.3:6821/989795778","heartbeat_back_addr":"172.21.15.3:6825/989795778","heartbeat_front_addr":"172.21.15.3:6823/989795778","state":["exists","up"]},{"osd":7,"uuid":"23003010-ab55-4574-8464-0f56aec3ddbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6817","nonce":4114652830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6819","nonce":4114652830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6823","nonce":4114652830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":4114652830},{"type":"v1","addr":"172.21.15.92:6821","nonce":4114652830}]},"public_addr":"172.21.15.92:6817/4114652830","cluster_addr":"172.21.15.92:6819/4114652830","heartbeat_back_addr":"172.21.15.92:6823/4114652830","heartbeat_front_addr":"172.21.15.92:6821/4114652830","state":["exists","up"]},{"osd":8,"uuid":"77f3d09f-2450-4861-9c06-172488429b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6817","nonce":1553122381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6819","nonce":1553122381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6823","nonce":1553122381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1553122381},{"type":"v1","addr":"172.21.15.136:6821","nonce":1553122381}]},"public_addr":"172.21.15.136:6817/1553122381","cluster_addr":"172.21.15.136:6819/1553122381","heartbeat_back_addr":"172.21.15.136:6823/1553122381","heartbeat_front_addr":"172.21.15.136:6821/1553122381","state":["exists","up"]},{"osd":9,"uuid":"8b3bd058-2c37-4910-b1b9-661e888ef9aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6827","nonce":2161930524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6829","nonce":2161930524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6833","nonce":2161930524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2161930524},{"type":"v1","addr":"172.21.15.3:6831","nonce":2161930524}]},"public_addr":"172.21.15.3:6827/2161930524","cluster_addr":"172.21.15.3:6829/2161930524","heartbeat_back_addr":"172.21.15.3:6833/2161930524","heartbeat_front_addr":"172.21.15.3:6831/2161930524","state":["exists","up"]},{"osd":10,"uuid":"b312bb63-21fa-4f9e-b676-c0cfca04749a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6825","nonce":2881136596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6827","nonce":2881136596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6831","nonce":2881136596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2881136596},{"type":"v1","addr":"172.21.15.92:6829","nonce":2881136596}]},"public_addr":"172.21.15.92:6825/2881136596","cluster_addr":"172.21.15.92:6827/2881136596","heartbeat_back_addr":"172.21.15.92:6831/2881136596","heartbeat_front_addr":"172.21.15.92:6829/2881136596","state":["exists","up"]},{"osd":11,"uuid":"6d479a7b-430c-4707-89c8-d20e7b47d68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6825","nonce":2854427876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6827","nonce":2854427876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6831","nonce":2854427876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2854427876},{"type":"v1","addr":"172.21.15.136:6829","nonce":2854427876}]},"public_addr":"172.21.15.136:6825/2854427876","cluster_addr":"172.21.15.136:6827/2854427876","heartbeat_back_addr":"172.21.15.136:6831/2854427876","heartbeat_front_addr":"172.21.15.136:6829/2854427876","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:55:51.889183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:12.988370+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:33.310409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:56:58.060062+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:19.330479+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:57:39.548091+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:05.065362+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:26.895129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:58:47.671160+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:13.995481+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:35.355346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-10T00:59:55.895538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.26","mappings":[{"from":1,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/3516755706":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/1056299694":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6801/91094391":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6800/91094391":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/2575312228":"2023-09-11T01:03:12.189996+0000","172.21.15.3:0/2296857090":"2023-09-11T01:03:12.189996+0000","172.21.15.3:6801/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6801/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/71687535":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/3193301911":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1972995090":"2023-09-11T00:53:55.752989+0000","172.21.15.3:0/4153284543":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/1689603322":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/1916209618":"2023-09-11T00:53:55.752989+0000","172.21.15.3:6800/70910038":"2023-09-11T00:53:28.332636+0000","172.21.15.3:0/886409476":"2023-09-11T00:53:55.752989+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-10T01:19:35.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12300a07b0 msgr2=0x7f12300a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.726 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12300a07b0 0x7f12300a2c70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f124410e3a0 tx=0x7f1234009260 comp rx=0 tx=0).stop 2023-09-10T01:19:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244072a30 msgr2=0x7f124419e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244072a30 0x7f124419e840 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f124000d950 tx=0x7f124000dc60 comp rx=0 tx=0).stop 2023-09-10T01:19:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 shutdown_connections 2023-09-10T01:19:35.727 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f12300a07b0 0x7f12300a2c70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1244072a30 0x7f124419e840 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f124410ce70 0x7f12441988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 --2- 172.21.15.3:0/3875275451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1244073350 0x7f124419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 >> 172.21.15.3:0/3875275451 conn(0x7f12440fc140 msgr2=0x7f124410efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:35.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 shutdown_connections 2023-09-10T01:19:35.729 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.725+0000 7f124c5ff700 1 -- 172.21.15.3:0/3875275451 wait complete. 2023-09-10T01:19:35.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-09-10T01:19:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 -- 172.21.15.3:0/609797961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100520 msgr2=0x7f734c1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 --2- 172.21.15.3:0/609797961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100520 0x7f734c1009a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f733c009920 tx=0x7f733c009c30 comp rx=0 tx=0).stop 2023-09-10T01:19:35.951 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 -- 172.21.15.3:0/609797961 shutdown_connections 2023-09-10T01:19:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 --2- 172.21.15.3:0/609797961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f734c1065e0 0x7f734c1069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 --2- 172.21.15.3:0/609797961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c100ee0 0x7f734c10d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 --2- 172.21.15.3:0/609797961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100520 0x7f734c1009a0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 -- 172.21.15.3:0/609797961 >> 172.21.15.3:0/609797961 conn(0x7f734c077a00 msgr2=0x7f734c077e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:35.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 -- 172.21.15.3:0/609797961 shutdown_connections 2023-09-10T01:19:35.953 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.949+0000 7f73547ff700 1 -- 172.21.15.3:0/609797961 wait complete. 2023-09-10T01:19:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.950+0000 7f73547ff700 1 Processor -- start 2023-09-10T01:19:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f73547ff700 1 -- start start 2023-09-10T01:19:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f73547ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f734c100520 0x7f734c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f73547ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100ee0 0x7f734c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f73547ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c1065e0 0x7f734c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.956 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f73547ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c198d50 con 0x7f734c1065e0 2023-09-10T01:19:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f73547ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c198ec0 con 0x7f734c100ee0 2023-09-10T01:19:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f73547ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c199080 con 0x7f734c100520 2023-09-10T01:19:35.957 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f7352d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c1065e0 0x7f734c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f7352d9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c1065e0 0x7f734c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49426/0 (socket says 172.21.15.3:49426) 2023-09-10T01:19:35.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f7352d9c700 1 -- 172.21.15.3:0/465584759 learned_addr learned my addr 172.21.15.3:0/465584759 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:35.958 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f7351d9a700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100ee0 0x7f734c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.951+0000 7f735259b700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f734c100520 0x7f734c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f7352d9c700 1 -- 172.21.15.3:0/465584759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f734c100520 msgr2=0x7f734c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f7352d9c700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f734c100520 0x7f734c19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.959 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f7352d9c700 1 -- 172.21.15.3:0/465584759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100ee0 msgr2=0x7f734c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f7352d9c700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100ee0 0x7f734c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f7352d9c700 1 -- 172.21.15.3:0/465584759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7344009d00 con 0x7f734c1065e0 2023-09-10T01:19:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f7352d9c700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c1065e0 0x7f734c1986e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f734400e3c0 tx=0x7f734400e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f73437fe700 1 -- 172.21.15.3:0/465584759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73440090d0 con 0x7f734c1065e0 2023-09-10T01:19:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c009520 con 0x7f734c1065e0 2023-09-10T01:19:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f73437fe700 1 -- 172.21.15.3:0/465584759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f734400f040 con 0x7f734c1065e0 2023-09-10T01:19:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f73437fe700 1 -- 172.21.15.3:0/465584759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344007c20 con 0x7f734c1065e0 2023-09-10T01:19:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.952+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c199550 con 0x7f734c1065e0 2023-09-10T01:19:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.953+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734c0678d0 con 0x7f734c1065e0 2023-09-10T01:19:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.955+0000 7f73437fe700 1 -- 172.21.15.3:0/465584759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f7344003a00 con 0x7f734c1065e0 2023-09-10T01:19:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.957+0000 7f73437fe700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f73380a07b0 0x7f73380a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.958+0000 7f73437fe700 1 -- 172.21.15.3:0/465584759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 9483+0+0 (secure 0 0 0) 0x7f73440cc1e0 con 0x7f734c1065e0 2023-09-10T01:19:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.958+0000 7f735259b700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f73380a07b0 0x7f73380a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.958+0000 7f735259b700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f73380a07b0 0x7f73380a2c70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7348009c60 tx=0x7f7348009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:35.962+0000 7f73437fe700 1 -- 172.21.15.3:0/465584759 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f7344095640 con 0x7f734c1065e0 2023-09-10T01:19:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.221+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f734c199550 con 0x7f734c1065e0 2023-09-10T01:19:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.224+0000 7f73437fe700 1 -- 172.21.15.3:0/465584759 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7f734c199550 con 0x7f734c1065e0 2023-09-10T01:19:36.227 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.226+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f73380a07b0 msgr2=0x7f73380a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:36.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.226+0000 7f73547ff700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f73380a07b0 0x7f73380a2c70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7348009c60 tx=0x7f7348009260 comp rx=0 tx=0).stop 2023-09-10T01:19:36.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.226+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c1065e0 msgr2=0x7f734c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:36.228 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.226+0000 7f73547ff700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c1065e0 0x7f734c1986e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f734400e3c0 tx=0x7f734400e6d0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 shutdown_connections 2023-09-10T01:19:36.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f73380a07b0 0x7f73380a2c70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f734c100520 0x7f734c19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.229 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f734c1065e0 0x7f734c1986e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 --2- 172.21.15.3:0/465584759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f734c100ee0 0x7f734c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 >> 172.21.15.3:0/465584759 conn(0x7f734c077a00 msgr2=0x7f734c072aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:36.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 shutdown_connections 2023-09-10T01:19:36.231 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.227+0000 7f73547ff700 1 -- 172.21.15.3:0/465584759 wait complete. 2023-09-10T01:19:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-09-10T01:19:36.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-09-10T01:19:36.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 -- 172.21.15.3:0/2663079429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 msgr2=0x7fc23c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:36.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 --2- 172.21.15.3:0/2663079429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 0x7fc23c10d510 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc23400b0a0 tx=0x7fc23400b3b0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 -- 172.21.15.3:0/2663079429 shutdown_connections 2023-09-10T01:19:36.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 --2- 172.21.15.3:0/2663079429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 0x7fc23c10d510 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 --2- 172.21.15.3:0/2663079429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c106650 0x7fc23c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 --2- 172.21.15.3:0/2663079429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c100640 0x7fc23c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 -- 172.21.15.3:0/2663079429 >> 172.21.15.3:0/2663079429 conn(0x7fc23c0fc170 msgr2=0x7fc23c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:36.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 -- 172.21.15.3:0/2663079429 shutdown_connections 2023-09-10T01:19:36.462 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.458+0000 7fc2441ff700 1 -- 172.21.15.3:0/2663079429 wait complete. 2023-09-10T01:19:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.459+0000 7fc2441ff700 1 Processor -- start 2023-09-10T01:19:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.459+0000 7fc2441ff700 1 -- start start 2023-09-10T01:19:36.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc2441ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c100640 0x7fc23c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc2441ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 0x7fc23c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc2441ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c106650 0x7fc23c19afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc2441ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c19b620 con 0x7fc23c101000 2023-09-10T01:19:36.465 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc2441ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c19b790 con 0x7fc23c100640 2023-09-10T01:19:36.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc2441ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c19b950 con 0x7fc23c106650 2023-09-10T01:19:36.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c106650 0x7fc23c19afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:36.466 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24179a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 0x7fc23c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:36.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc241f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c100640 0x7fc23c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:36.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c106650 0x7fc23c19afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.3:32986/0 (socket says 172.21.15.3:32986) 2023-09-10T01:19:36.467 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 -- 172.21.15.3:0/1961181388 learned_addr learned my addr 172.21.15.3:0/1961181388 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc241f9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c100640 0x7fc23c196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:38528/0 (socket says 172.21.15.3:38528) 2023-09-10T01:19:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 -- 172.21.15.3:0/1961181388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c100640 msgr2=0x7fc23c196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c100640 0x7fc23c196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 -- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 msgr2=0x7fc23c196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 0x7fc23c196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.460+0000 7fc24279c700 1 -- 172.21.15.3:0/1961181388 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234009d00 con 0x7fc23c106650 2023-09-10T01:19:36.469 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.461+0000 7fc24279c700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c106650 0x7fc23c19afe0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc2340062a0 tx=0x7fc234007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.461+0000 7fc232ffd700 1 -- 172.21.15.3:0/1961181388 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23400e040 con 0x7fc23c106650 2023-09-10T01:19:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.461+0000 7fc232ffd700 1 -- 172.21.15.3:0/1961181388 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2340094c0 con 0x7fc23c106650 2023-09-10T01:19:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.461+0000 7fc232ffd700 1 -- 172.21.15.3:0/1961181388 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2340052a0 con 0x7fc23c106650 2023-09-10T01:19:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.461+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c19bbd0 con 0x7fc23c106650 2023-09-10T01:19:36.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.461+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c1a3470 con 0x7fc23c106650 2023-09-10T01:19:36.471 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.462+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c067480 con 0x7fc23c106650 2023-09-10T01:19:36.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.464+0000 7fc232ffd700 1 -- 172.21.15.3:0/1961181388 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fc23401d470 con 0x7fc23c106650 2023-09-10T01:19:36.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.465+0000 7fc232ffd700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc2280a0830 0x7fc2280a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:36.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.465+0000 7fc232ffd700 1 -- 172.21.15.3:0/1961181388 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 9697+0+0 (secure 0 0 0) 0x7fc2340cd9a0 con 0x7fc23c106650 2023-09-10T01:19:36.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.465+0000 7fc241f9b700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc2280a0830 0x7fc2280a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:36.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.466+0000 7fc241f9b700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc2280a0830 0x7fc2280a2cf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc238009be0 tx=0x7fc238009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:36.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.469+0000 7fc232ffd700 1 -- 172.21.15.3:0/1961181388 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fc234096d20 con 0x7fc23c106650 2023-09-10T01:19:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-a[97630]: 2023-09-10T01:19:36.222+0000 7f766c353700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-09-10T01:19:36.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: pgmap v517: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-09-10T01:19:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3875275451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:19:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/465584759' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-09-10T01:19:36.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-09-10T01:19:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-09-10T01:19:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/465584759' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-09-10T01:19:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: osdmap e115: 12 total, 12 up, 12 in 2023-09-10T01:19:36.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:36 smithi003 ceph-mon[97654]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-09-10T01:19:36.662 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-c[171170]: ignoring --setuser ceph since I am not root 2023-09-10T01:19:36.662 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-c[171170]: ignoring --setgroup ceph since I am not root 2023-09-10T01:19:36.663 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mds[171194]: -- 172.21.15.136:0/3051272412 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5561062bd1e0 con 0x556106292800 2023-09-10T01:19:36.663 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-c[171170]: starting mds.c at 2023-09-10T01:19:36.664 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-f[170830]: ignoring --setuser ceph since I am not root 2023-09-10T01:19:36.664 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-f[170830]: ignoring --setgroup ceph since I am not root 2023-09-10T01:19:36.664 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mds[170854]: -- 172.21.15.136:0/617717256 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f86a4511e0 con 0x55f86a426800 2023-09-10T01:19:36.664 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-f[170830]: starting mds.f at 2023-09-10T01:19:36.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: pgmap v517: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-09-10T01:19:36.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3875275451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:19:36.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/465584759' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-09-10T01:19:36.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-09-10T01:19:36.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-09-10T01:19:36.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/465584759' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-09-10T01:19:36.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: osdmap e115: 12 total, 12 up, 12 in 2023-09-10T01:19:36.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-mon[133708]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-09-10T01:19:36.667 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-i[171516]: ignoring --setuser ceph since I am not root 2023-09-10T01:19:36.668 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-i[171516]: ignoring --setgroup ceph since I am not root 2023-09-10T01:19:36.668 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:19:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-i[171516]: starting mds.i at 2023-09-10T01:19:36.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: pgmap v517: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-09-10T01:19:36.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3875275451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-09-10T01:19:36.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/465584759' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-09-10T01:19:36.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-09-10T01:19:36.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-09-10T01:19:36.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/465584759' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-09-10T01:19:36.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: osdmap e115: 12 total, 12 up, 12 in 2023-09-10T01:19:36.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mon[134067]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-09-10T01:19:36.719 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: ignoring --setuser ceph since I am not root 2023-09-10T01:19:36.719 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: ignoring --setgroup ceph since I am not root 2023-09-10T01:19:36.719 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mds[171478]: -- 172.21.15.92:0/1871047804 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bcb65351e0 con 0x55bcb650b000 2023-09-10T01:19:36.720 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: starting mds.b at 2023-09-10T01:19:36.720 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-e[170771]: ignoring --setuser ceph since I am not root 2023-09-10T01:19:36.720 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-e[170771]: ignoring --setgroup ceph since I am not root 2023-09-10T01:19:36.721 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mds[170795]: -- 172.21.15.92:0/4103882175 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b6f96751e0 con 0x55b6f964b000 2023-09-10T01:19:36.721 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-e[170771]: starting mds.e at 2023-09-10T01:19:36.721 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-h[171112]: ignoring --setuser ceph since I am not root 2023-09-10T01:19:36.722 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-h[171112]: ignoring --setgroup ceph since I am not root 2023-09-10T01:19:36.722 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-mds[171136]: -- 172.21.15.92:0/3066754635 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c50277d1e0 con 0x55c502753000 2023-09-10T01:19:36.722 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:19:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-h[171112]: starting mds.h at 2023-09-10T01:19:36.728 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:36.728+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc23c110660 con 0x7fc23c106650 2023-09-10T01:19:37.230 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.230+0000 7fc232ffd700 1 -- 172.21.15.3:0/1961181388 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc23409a340 con 0x7fc23c106650 2023-09-10T01:19:37.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc2280a0830 msgr2=0x7fc2280a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:37.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc2280a0830 0x7fc2280a2cf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc238009be0 tx=0x7fc238009260 comp rx=0 tx=0).stop 2023-09-10T01:19:37.234 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c106650 msgr2=0x7fc23c19afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:37.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c106650 0x7fc23c19afe0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc2340062a0 tx=0x7fc234007870 comp rx=0 tx=0).stop 2023-09-10T01:19:37.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 shutdown_connections 2023-09-10T01:19:37.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fc2280a0830 0x7fc2280a2cf0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.235 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c106650 0x7fc23c19afe0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c100640 0x7fc23c196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 --2- 172.21.15.3:0/1961181388 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc23c101000 0x7fc23c196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 >> 172.21.15.3:0/1961181388 conn(0x7fc23c0fc170 msgr2=0x7fc23c0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 shutdown_connections 2023-09-10T01:19:37.237 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.233+0000 7fc2441ff700 1 -- 172.21.15.3:0/1961181388 wait complete. 2023-09-10T01:19:37.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-09-10T01:19:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 -- 172.21.15.3:0/349305012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4106650 msgr2=0x7f5da4106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 --2- 172.21.15.3:0/349305012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4106650 0x7f5da4106a30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009990 tx=0x7f5d94009ca0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 -- 172.21.15.3:0/349305012 shutdown_connections 2023-09-10T01:19:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 --2- 172.21.15.3:0/349305012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da4101000 0x7f5da410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 --2- 172.21.15.3:0/349305012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4106650 0x7f5da4106a30 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 --2- 172.21.15.3:0/349305012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5da4100640 0x7f5da4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.474 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 -- 172.21.15.3:0/349305012 >> 172.21.15.3:0/349305012 conn(0x7f5da40fc170 msgr2=0x7f5da40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:37.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 -- 172.21.15.3:0/349305012 shutdown_connections 2023-09-10T01:19:37.475 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.471+0000 7f5da8fff700 1 -- 172.21.15.3:0/349305012 wait complete. 2023-09-10T01:19:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.472+0000 7f5da8fff700 1 Processor -- start 2023-09-10T01:19:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.472+0000 7f5da8fff700 1 -- start start 2023-09-10T01:19:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da8fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5da4100640 0x7f5da419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da8fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4101000 0x7f5da419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:37.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da8fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da4106650 0x7f5da4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:37.481 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da8fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4198dd0 con 0x7f5da4101000 2023-09-10T01:19:37.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da8fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4198f40 con 0x7f5da4106650 2023-09-10T01:19:37.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da8fff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4199100 con 0x7f5da4100640 2023-09-10T01:19:37.482 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5d9bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4101000 0x7f5da419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:37.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da4106650 0x7f5da4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:37.483 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da259c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5da4100640 0x7f5da419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da4106650 0x7f5da4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.3:38548/0 (socket says 172.21.15.3:38548) 2023-09-10T01:19:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5da2d9d700 1 -- 172.21.15.3:0/3346048900 learned_addr learned my addr 172.21.15.3:0/3346048900 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5d9bfff700 1 -- 172.21.15.3:0/3346048900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5da4100640 msgr2=0x7f5da419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5d9bfff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5da4100640 0x7f5da419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5d9bfff700 1 -- 172.21.15.3:0/3346048900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da4106650 msgr2=0x7f5da4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5d9bfff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da4106650 0x7f5da4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.473+0000 7f5d9bfff700 1 -- 172.21.15.3:0/3346048900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940095f0 con 0x7f5da4101000 2023-09-10T01:19:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.474+0000 7f5d9bfff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4101000 0x7f5da419ec30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00d950 tx=0x7f5d8c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.474+0000 7f5d9b7fe700 1 -- 172.21.15.3:0/3346048900 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c00f890 con 0x7f5da4101000 2023-09-10T01:19:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.474+0000 7f5d9b7fe700 1 -- 172.21.15.3:0/3346048900 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d8c010460 con 0x7f5da4101000 2023-09-10T01:19:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.474+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da41993e0 con 0x7f5da4101000 2023-09-10T01:19:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.474+0000 7f5d9b7fe700 1 -- 172.21.15.3:0/3346048900 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c00e610 con 0x7f5da4101000 2023-09-10T01:19:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.474+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da41a56e0 con 0x7f5da4101000 2023-09-10T01:19:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.475+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da4067480 con 0x7f5da4101000 2023-09-10T01:19:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.477+0000 7f5d9b7fe700 1 -- 172.21.15.3:0/3346048900 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7f5d8c0105d0 con 0x7f5da4101000 2023-09-10T01:19:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.478+0000 7f5d9b7fe700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5d900a0830 0x7f5d900a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.478+0000 7f5d9b7fe700 1 -- 172.21.15.3:0/3346048900 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 9697+0+0 (secure 0 0 0) 0x7f5d8c0cc060 con 0x7f5da4101000 2023-09-10T01:19:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.479+0000 7f5da259c700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5d900a0830 0x7f5d900a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.479+0000 7f5da259c700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5d900a0830 0x7f5d900a2cf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400b5c0 tx=0x7f5d94005fb0 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.483+0000 7f5d9b7fe700 1 -- 172.21.15.3:0/3346048900 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7f5d8c095390 con 0x7f5da4101000 2023-09-10T01:19:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1961181388' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T01:19:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T01:19:37.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-09-10T01:19:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-09-10T01:19:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: Cluster is now healthy 2023-09-10T01:19:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-09-10T01:19:37.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6835/3618164024,v1:172.21.15.136:6838/3618164024] up:boot 2023-09-10T01:19:37.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6835/1674328998,v1:172.21.15.92:6839/1674328998] up:boot 2023-09-10T01:19:37.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6836/3926141503,v1:172.21.15.136:6839/3926141503] up:boot 2023-09-10T01:19:37.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.136:6834/3550377468,v1:172.21.15.136:6837/3550377468] up:boot 2023-09-10T01:19:37.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6834/1729734856,v1:172.21.15.92:6837/1729734856] up:boot 2023-09-10T01:19:37.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: mds.? [v2:172.21.15.92:6836/2111580908,v1:172.21.15.92:6838/2111580908] up:boot 2023-09-10T01:19:37.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: fsmap 12 up:standby 2023-09-10T01:19:37.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:19:37.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:19:37.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:19:37.563 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:19:37.564 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:19:37.564 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:37 smithi003 ceph-mon[97654]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:19:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1961181388' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T01:19:37.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T01:19:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-09-10T01:19:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-09-10T01:19:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: Cluster is now healthy 2023-09-10T01:19:37.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-09-10T01:19:37.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6835/3618164024,v1:172.21.15.136:6838/3618164024] up:boot 2023-09-10T01:19:37.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6835/1674328998,v1:172.21.15.92:6839/1674328998] up:boot 2023-09-10T01:19:37.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6836/3926141503,v1:172.21.15.136:6839/3926141503] up:boot 2023-09-10T01:19:37.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.136:6834/3550377468,v1:172.21.15.136:6837/3550377468] up:boot 2023-09-10T01:19:37.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6834/1729734856,v1:172.21.15.92:6837/1729734856] up:boot 2023-09-10T01:19:37.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: mds.? [v2:172.21.15.92:6836/2111580908,v1:172.21.15.92:6838/2111580908] up:boot 2023-09-10T01:19:37.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: fsmap 12 up:standby 2023-09-10T01:19:37.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:19:37.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:19:37.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:19:37.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:19:37.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:19:37.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:37 smithi136 ceph-mon[133708]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:19:37.696 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:37.696+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5da4199c70 con 0x7f5da4101000 2023-09-10T01:19:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1961181388' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T01:19:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-09-10T01:19:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-09-10T01:19:37.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-09-10T01:19:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: Cluster is now healthy 2023-09-10T01:19:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-09-10T01:19:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6835/3618164024,v1:172.21.15.136:6838/3618164024] up:boot 2023-09-10T01:19:37.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6835/1674328998,v1:172.21.15.92:6839/1674328998] up:boot 2023-09-10T01:19:37.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6836/3926141503,v1:172.21.15.136:6839/3926141503] up:boot 2023-09-10T01:19:37.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.136:6834/3550377468,v1:172.21.15.136:6837/3550377468] up:boot 2023-09-10T01:19:37.718 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6834/1729734856,v1:172.21.15.92:6837/1729734856] up:boot 2023-09-10T01:19:37.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: mds.? [v2:172.21.15.92:6836/2111580908,v1:172.21.15.92:6838/2111580908] up:boot 2023-09-10T01:19:37.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: fsmap 12 up:standby 2023-09-10T01:19:37.719 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-09-10T01:19:37.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-09-10T01:19:37.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-09-10T01:19:37.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-09-10T01:19:37.720 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-09-10T01:19:37.721 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:37 smithi092 ceph-mon[134067]: from='mgr.25667 172.21.15.3:0/547243643' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-09-10T01:19:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.363+0000 7f5d9b7fe700 1 -- 172.21.15.3:0/3346048900 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5d8c0989b0 con 0x7f5da4101000 2023-09-10T01:19:38.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5d900a0830 msgr2=0x7f5d900a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:38.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5d900a0830 0x7f5d900a2cf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400b5c0 tx=0x7f5d94005fb0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.369 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4101000 msgr2=0x7f5da419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:38.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4101000 0x7f5da419ec30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00d950 tx=0x7f5d8c00dc60 comp rx=0 tx=0).stop 2023-09-10T01:19:38.370 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 shutdown_connections 2023-09-10T01:19:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7f5d900a0830 0x7f5d900a2cf0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5da4100640 0x7f5da419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5da4106650 0x7f5da4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 --2- 172.21.15.3:0/3346048900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5da4101000 0x7f5da419ec30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.368+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 >> 172.21.15.3:0/3346048900 conn(0x7f5da40fc170 msgr2=0x7f5da410ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.369+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 shutdown_connections 2023-09-10T01:19:38.372 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.370+0000 7f5da8fff700 1 -- 172.21.15.3:0/3346048900 wait complete. 2023-09-10T01:19:38.373 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2023-09-10T01:19:38.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-09-10T01:19:38.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.644+0000 7fa6085ff700 1 -- 172.21.15.3:0/1284473740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 msgr2=0x7fa60010c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:38.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.644+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1284473740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 0x7fa60010c940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00098f0 tx=0x7fa5f0009c00 comp rx=0 tx=0).stop 2023-09-10T01:19:38.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.645+0000 7fa6085ff700 1 -- 172.21.15.3:0/1284473740 shutdown_connections 2023-09-10T01:19:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.645+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1284473740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa600072a30 0x7fa600072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.645+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1284473740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa60010ce80 0x7fa60010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.645+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1284473740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 0x7fa60010c940 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.645+0000 7fa6085ff700 1 -- 172.21.15.3:0/1284473740 >> 172.21.15.3:0/1284473740 conn(0x7fa6000fc170 msgr2=0x7fa6000fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.645+0000 7fa6085ff700 1 -- 172.21.15.3:0/1284473740 shutdown_connections 2023-09-10T01:19:38.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.645+0000 7fa6085ff700 1 -- 172.21.15.3:0/1284473740 wait complete. 2023-09-10T01:19:38.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 Processor -- start 2023-09-10T01:19:38.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 -- start start 2023-09-10T01:19:38.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa600072a30 0x7fa6001989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:38.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 0x7fa600198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60010ce80 0x7fa60019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60019d940 con 0x7fa600072a30 2023-09-10T01:19:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60019dab0 con 0x7fa600073350 2023-09-10T01:19:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa6085ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60019dc70 con 0x7fa60010ce80 2023-09-10T01:19:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.646+0000 7fa60639b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa600072a30 0x7fa6001989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa60639b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa600072a30 0x7fa6001989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49496/0 (socket says 172.21.15.3:49496) 2023-09-10T01:19:38.653 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa60639b700 1 -- 172.21.15.3:0/1401337665 learned_addr learned my addr 172.21.15.3:0/1401337665 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:38.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa605b9a700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 0x7fa600198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:38.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa606b9c700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60010ce80 0x7fa60019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:38.654 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa605b9a700 1 -- 172.21.15.3:0/1401337665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60010ce80 msgr2=0x7fa60019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa605b9a700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60010ce80 0x7fa60019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa605b9a700 1 -- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa600072a30 msgr2=0x7fa6001989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa605b9a700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa600072a30 0x7fa6001989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:38.655 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa605b9a700 1 -- 172.21.15.3:0/1401337665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8009d00 con 0x7fa600073350 2023-09-10T01:19:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa60639b700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa600072a30 0x7fa6001989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:19:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.647+0000 7fa605b9a700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 0x7fa600198ee0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009810 tx=0x7fa5f000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.648+0000 7fa5f77fe700 1 -- 172.21.15.3:0/1401337665 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f001c070 con 0x7fa600073350 2023-09-10T01:19:38.656 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.648+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f0009520 con 0x7fa600073350 2023-09-10T01:19:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.648+0000 7fa5f77fe700 1 -- 172.21.15.3:0/1401337665 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5f000bba0 con 0x7fa600073350 2023-09-10T01:19:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.648+0000 7fa5f77fe700 1 -- 172.21.15.3:0/1401337665 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0003970 con 0x7fa600073350 2023-09-10T01:19:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.648+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6001a5620 con 0x7fa600073350 2023-09-10T01:19:38.657 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.649+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa600067480 con 0x7fa600073350 2023-09-10T01:19:38.658 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.651+0000 7fa5f77fe700 1 -- 172.21.15.3:0/1401337665 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7fa5f0020030 con 0x7fa600073350 2023-09-10T01:19:38.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.652+0000 7fa5f77fe700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa5ec0a0830 0x7fa5ec0a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:38.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.652+0000 7fa5f77fe700 1 -- 172.21.15.3:0/1401337665 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 9143+0+0 (secure 0 0 0) 0x7fa5f00cd3a0 con 0x7fa600073350 2023-09-10T01:19:38.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.652+0000 7fa60639b700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa5ec0a0830 0x7fa5ec0a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:38.659 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.653+0000 7fa60639b700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa5ec0a0830 0x7fa5ec0a2cf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc011550 tx=0x7fa5fc009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:38.660 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.654+0000 7fa5f77fe700 1 -- 172.21.15.3:0/1401337665 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7fa5f0096950 con 0x7fa600073350 2023-09-10T01:19:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:38 smithi136 ceph-mon[133708]: pgmap v519: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-09-10T01:19:38.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:38 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3346048900' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:38.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:38 smithi092 ceph-mon[134067]: pgmap v519: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-09-10T01:19:38.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:38 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3346048900' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:38.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:38 smithi003 ceph-mon[97654]: pgmap v519: 129 pgs: 129 active+clean; 775 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-09-10T01:19:38.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:38 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3346048900' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:38.870 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:38.869+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa600199cf0 con 0x7fa600073350 2023-09-10T01:19:39.385 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.384+0000 7fa5f77fe700 1 -- 172.21.15.3:0/1401337665 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa5f0099f70 con 0x7fa600073350 2023-09-10T01:19:39.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.388+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa5ec0a0830 msgr2=0x7fa5ec0a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.388+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa5ec0a0830 0x7fa5ec0a2cf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc011550 tx=0x7fa5fc009260 comp rx=0 tx=0).stop 2023-09-10T01:19:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 msgr2=0x7fa600198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:39.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 0x7fa600198ee0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009810 tx=0x7fa5f000fa00 comp rx=0 tx=0).stop 2023-09-10T01:19:39.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 shutdown_connections 2023-09-10T01:19:39.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7fa5ec0a0830 0x7fa5ec0a2cf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60010ce80 0x7fa60019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa600072a30 0x7fa6001989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 --2- 172.21.15.3:0/1401337665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa600073350 0x7fa600198ee0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 >> 172.21.15.3:0/1401337665 conn(0x7fa6000fc170 msgr2=0x7fa60010f000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 shutdown_connections 2023-09-10T01:19:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.389+0000 7fa6085ff700 1 -- 172.21.15.3:0/1401337665 wait complete. 2023-09-10T01:19:39.394 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2023-09-10T01:19:39.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-09-10T01:19:39.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 -- 172.21.15.3:0/2191072818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00101000 msgr2=0x7efd0010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:39.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 --2- 172.21.15.3:0/2191072818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00101000 0x7efd0010d510 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7efcf800b0a0 tx=0x7efcf800b3b0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 -- 172.21.15.3:0/2191072818 shutdown_connections 2023-09-10T01:19:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 --2- 172.21.15.3:0/2191072818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00106650 0x7efd00106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 --2- 172.21.15.3:0/2191072818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00101000 0x7efd0010d510 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 --2- 172.21.15.3:0/2191072818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd00100640 0x7efd00100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 -- 172.21.15.3:0/2191072818 >> 172.21.15.3:0/2191072818 conn(0x7efd000fc170 msgr2=0x7efd000fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:39.640 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 -- 172.21.15.3:0/2191072818 shutdown_connections 2023-09-10T01:19:39.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.637+0000 7efd051ff700 1 -- 172.21.15.3:0/2191072818 wait complete. 2023-09-10T01:19:39.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 Processor -- start 2023-09-10T01:19:39.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 -- start start 2023-09-10T01:19:39.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00100640 0x7efd00196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:39.643 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd00101000 0x7efd00196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:39.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00106650 0x7efd00199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:39.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0019a580 con 0x7efd00106650 2023-09-10T01:19:39.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0019a6f0 con 0x7efd00101000 2023-09-10T01:19:39.644 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.638+0000 7efd051ff700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0019a8b0 con 0x7efd00100640 2023-09-10T01:19:39.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00106650 0x7efd00199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:39.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcfed9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00100640 0x7efd00196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:39.645 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcfe59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd00101000 0x7efd00196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:39.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00106650 0x7efd00199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49508/0 (socket says 172.21.15.3:49508) 2023-09-10T01:19:39.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 -- 172.21.15.3:0/2236853717 learned_addr learned my addr 172.21.15.3:0/2236853717 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-09-10T01:19:39.646 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 -- 172.21.15.3:0/2236853717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00100640 msgr2=0x7efd00196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:39.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00100640 0x7efd00196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 -- 172.21.15.3:0/2236853717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd00101000 msgr2=0x7efd00196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:39.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd00101000 0x7efd00196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:39.647 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 -- 172.21.15.3:0/2236853717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf8009d00 con 0x7efd00106650 2023-09-10T01:19:39.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcfe59c700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd00101000 0x7efd00196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-10T01:19:39.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.639+0000 7efcff59e700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00106650 0x7efd00199f80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009850 tx=0x7efcf80077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:39.648 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.640+0000 7efce7fff700 1 -- 172.21.15.3:0/2236853717 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf800e040 con 0x7efd00106650 2023-09-10T01:19:39.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.640+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0019ab30 con 0x7efd00106650 2023-09-10T01:19:39.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.640+0000 7efce7fff700 1 -- 172.21.15.3:0/2236853717 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcf80079e0 con 0x7efd00106650 2023-09-10T01:19:39.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.640+0000 7efce7fff700 1 -- 172.21.15.3:0/2236853717 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf80052a0 con 0x7efd00106650 2023-09-10T01:19:39.649 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.640+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd001a34f0 con 0x7efd00106650 2023-09-10T01:19:39.650 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.641+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd00067480 con 0x7efd00106650 2023-09-10T01:19:39.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.643+0000 7efce7fff700 1 -- 172.21.15.3:0/2236853717 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 27) v1 ==== 135389+0+0 (secure 0 0 0) 0x7efcf8003a00 con 0x7efd00106650 2023-09-10T01:19:39.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.644+0000 7efce7fff700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efce80a07b0 0x7efce80a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-10T01:19:39.651 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.645+0000 7efce7fff700 1 -- 172.21.15.3:0/2236853717 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 8510+0+0 (secure 0 0 0) 0x7efcf80cd670 con 0x7efd00106650 2023-09-10T01:19:39.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.645+0000 7efcfed9d700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efce80a07b0 0x7efce80a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-10T01:19:39.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.645+0000 7efcfed9d700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efce80a07b0 0x7efce80a2c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009be0 tx=0x7efcf0009260 comp rx=0 tx=0).ready entity=mgr.25667 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-10T01:19:39.652 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.646+0000 7efce7fff700 1 -- 172.21.15.3:0/2236853717 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179712 (secure 0 0 0) 0x7efcf8096f30 con 0x7efd00106650 2023-09-10T01:19:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:39 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/3346048900' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-09-10T01:19:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:39 smithi136 ceph-mon[133708]: osdmap e116: 12 total, 12 up, 12 in 2023-09-10T01:19:39.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:39 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/1401337665' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:39.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:39 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:39.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:39 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/3346048900' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-09-10T01:19:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:39 smithi092 ceph-mon[134067]: osdmap e116: 12 total, 12 up, 12 in 2023-09-10T01:19:39.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:39 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/1401337665' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:39.717 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:39 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:39 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/3346048900' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-09-10T01:19:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:39 smithi003 ceph-mon[97654]: osdmap e116: 12 total, 12 up, 12 in 2023-09-10T01:19:39.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:39 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/1401337665' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:39.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:39 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:39.877 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:39.877+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efd00110610 con 0x7efd00106650 2023-09-10T01:19:40.384 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.383+0000 7efce7fff700 1 -- 172.21.15.3:0/2236853717 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v118) v1 ==== 178+0+0 (secure 0 0 0) 0x7efcf8004e60 con 0x7efd00106650 2023-09-10T01:19:40.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efce80a07b0 msgr2=0x7efce80a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:40.389 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efce80a07b0 0x7efce80a2c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009be0 tx=0x7efcf0009260 comp rx=0 tx=0).stop 2023-09-10T01:19:40.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00106650 msgr2=0x7efd00199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-10T01:19:40.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00106650 0x7efd00199f80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009850 tx=0x7efcf80077f0 comp rx=0 tx=0).stop 2023-09-10T01:19:40.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 shutdown_connections 2023-09-10T01:19:40.390 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:6800/3336040255,v1:172.21.15.3:6801/3336040255] conn(0x7efce80a07b0 0x7efce80a2c70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00100640 0x7efd00196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd00106650 0x7efd00199f80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:40.391 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 --2- 172.21.15.3:0/2236853717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd00101000 0x7efd00196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-10T01:19:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 >> 172.21.15.3:0/2236853717 conn(0x7efd000fc170 msgr2=0x7efd000fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-10T01:19:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 shutdown_connections 2023-09-10T01:19:40.392 INFO:teuthology.orchestra.run.smithi003.stderr:2023-09-10T01:19:40.388+0000 7efd051ff700 1 -- 172.21.15.3:0/2236853717 wait complete. 2023-09-10T01:19:40.393 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data_ec' removed 2023-09-10T01:19:40.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-09-10T01:19:40.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-09-10T01:19:40.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-09-10T01:19:40.563 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-09-10T01:19:40.563 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-09-10T01:19:40.607 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-09-10T01:19:40.624 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-09-10T01:19:40.641 INFO:tasks.cephadm:Stopping all daemons... 2023-09-10T01:19:40.642 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-09-10T01:19:40.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a 2023-09-10T01:19:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:40 smithi136 ceph-mon[133708]: pgmap v521: 97 pgs: 97 active+clean; 277 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:19:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:40 smithi136 ceph-mon[133708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-09-10T01:19:40.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:40 smithi136 ceph-mon[133708]: osdmap e117: 12 total, 12 up, 12 in 2023-09-10T01:19:40.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:40 smithi136 ceph-mon[133708]: from='client.? 172.21.15.3:0/2236853717' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:40.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:40 smithi003 ceph-mon[97654]: pgmap v521: 97 pgs: 97 active+clean; 277 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:19:40.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:40 smithi003 ceph-mon[97654]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-09-10T01:19:40.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:40 smithi003 ceph-mon[97654]: osdmap e117: 12 total, 12 up, 12 in 2023-09-10T01:19:40.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:40 smithi003 ceph-mon[97654]: from='client.? 172.21.15.3:0/2236853717' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:40.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:40 smithi092 ceph-mon[134067]: pgmap v521: 97 pgs: 97 active+clean; 277 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2023-09-10T01:19:40.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:40 smithi092 ceph-mon[134067]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-09-10T01:19:40.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:40 smithi092 ceph-mon[134067]: osdmap e117: 12 total, 12 up, 12 in 2023-09-10T01:19:40.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:40 smithi092 ceph-mon[134067]: from='client.? 172.21.15.3:0/2236853717' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-09-10T01:19:41.059 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:40 smithi003 systemd[1]: Stopping Ceph mon.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:41.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:41 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-a[97630]: 2023-09-10T01:19:41.173+0000 7f7671b5e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:19:41.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:41 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-a[97630]: 2023-09-10T01:19:41.173+0000 7f7671b5e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-09-10T01:19:42.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Sep 10 01:19:41 smithi003 bash[190180]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-a 2023-09-10T01:19:42.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.a.service' 2023-09-10T01:19:42.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:19:42.142 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-09-10T01:19:42.142 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-09-10T01:19:42.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.b 2023-09-10T01:19:42.465 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:42 smithi092 systemd[1]: Stopping Ceph mon.b for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:42.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-b[134043]: 2023-09-10T01:19:42.542+0000 7f17dc05e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:19:42.866 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-b[134043]: 2023-09-10T01:19:42.542+0000 7f17dc05e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-09-10T01:19:43.090 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.b.service' 2023-09-10T01:19:43.115 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:42 smithi092 bash[179004]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-b 2023-09-10T01:19:43.116 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:43 smithi092 systemd[1]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.b.service: Succeeded. 2023-09-10T01:19:43.116 INFO:journalctl@ceph.mon.b.smithi092.stdout:Sep 10 01:19:43 smithi092 systemd[1]: Stopped Ceph mon.b for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da. 2023-09-10T01:19:43.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:19:43.138 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-09-10T01:19:43.138 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-09-10T01:19:43.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.c 2023-09-10T01:19:43.530 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:43 smithi136 systemd[1]: Stopping Ceph mon.c for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:43.899 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-c[133684]: 2023-09-10T01:19:43.529+0000 7fdaa31b4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:19:43.899 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-c[133684]: 2023-09-10T01:19:43.529+0000 7fdaa31b4700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-09-10T01:19:44.162 INFO:journalctl@ceph.mon.c.smithi136.stdout:Sep 10 01:19:43 smithi136 bash[179692]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mon-c 2023-09-10T01:19:44.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mon.c.service' 2023-09-10T01:19:44.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:19:44.352 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-09-10T01:19:44.352 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-09-10T01:19:44.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.x 2023-09-10T01:19:44.808 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:19:44 smithi003 systemd[1]: Stopping Ceph mgr.x for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:45.278 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:19:45 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:19:45.005+0000 7f16af8ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:19:45.278 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:19:45 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x[97902]: 2023-09-10T01:19:45.005+0000 7f16af8ec700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-09-10T01:19:45.559 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Sep 10 01:19:45 smithi003 bash[190322]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-x 2023-09-10T01:19:45.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.x.service' 2023-09-10T01:19:45.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:19:45.892 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-09-10T01:19:45.892 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-09-10T01:19:45.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.y 2023-09-10T01:19:46.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:19:45 smithi092 systemd[1]: Stopping Ceph mgr.y for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:46.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:19:46 smithi092 bash[179145]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-y 2023-09-10T01:19:46.784 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Sep 10 01:19:46 smithi092 systemd[1]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-09-10T01:19:46.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.y.service' 2023-09-10T01:19:46.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:19:46.826 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-09-10T01:19:46.826 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-09-10T01:19:46.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.z 2023-09-10T01:19:47.162 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:19:46 smithi136 systemd[1]: Stopping Ceph mgr.z for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:47.532 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:19:47 smithi136 bash[179842]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mgr-z 2023-09-10T01:19:47.532 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Sep 10 01:19:47 smithi136 systemd[1]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-09-10T01:19:47.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mgr.z.service' 2023-09-10T01:19:47.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:19:47.759 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-09-10T01:19:47.759 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-09-10T01:19:47.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.0 2023-09-10T01:19:48.289 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 01:19:47 smithi003 systemd[1]: Stopping Ceph osd.0 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:48.559 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 01:19:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-0[119797]: 2023-09-10T01:19:48.288+0000 7f1e551b6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:19:48.559 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 01:19:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-0[119797]: 2023-09-10T01:19:48.288+0000 7f1e551b6700 -1 osd.0 118 *** Got signal Terminated *** 2023-09-10T01:19:48.560 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 01:19:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-0[119797]: 2023-09-10T01:19:48.288+0000 7f1e551b6700 -1 osd.0 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:19:54.809 INFO:journalctl@ceph.osd.0.smithi003.stdout:Sep 10 01:19:54 smithi003 bash[190468]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-0 2023-09-10T01:19:56.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.0.service' 2023-09-10T01:19:56.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:19:56.328 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-09-10T01:19:56.328 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-09-10T01:19:56.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.1 2023-09-10T01:19:56.673 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 01:19:56 smithi092 systemd[1]: Stopping Ceph osd.1 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:19:56.965 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 01:19:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-1[139135]: 2023-09-10T01:19:56.671+0000 7fe69e5ab700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:19:56.966 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 01:19:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-1[139135]: 2023-09-10T01:19:56.671+0000 7fe69e5ab700 -1 osd.1 118 *** Got signal Terminated *** 2023-09-10T01:19:56.966 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 01:19:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-1[139135]: 2023-09-10T01:19:56.671+0000 7fe69e5ab700 -1 osd.1 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:02.716 INFO:journalctl@ceph.osd.1.smithi092.stdout:Sep 10 01:20:02 smithi092 bash[179294]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-1 2023-09-10T01:20:03.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.1.service' 2023-09-10T01:20:03.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:03.851 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-09-10T01:20:03.851 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-09-10T01:20:03.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.2 2023-09-10T01:20:04.162 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 01:20:03 smithi136 systemd[1]: Stopping Ceph osd.2 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:04.662 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 01:20:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-2[139502]: 2023-09-10T01:20:04.226+0000 7ff539c29700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:04.662 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 01:20:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-2[139502]: 2023-09-10T01:20:04.226+0000 7ff539c29700 -1 osd.2 118 *** Got signal Terminated *** 2023-09-10T01:20:04.663 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 01:20:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-2[139502]: 2023-09-10T01:20:04.226+0000 7ff539c29700 -1 osd.2 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:10.162 INFO:journalctl@ceph.osd.2.smithi136.stdout:Sep 10 01:20:10 smithi136 bash[179988]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-2 2023-09-10T01:20:11.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.2.service' 2023-09-10T01:20:11.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:11.390 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-09-10T01:20:11.391 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-09-10T01:20:11.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.3 2023-09-10T01:20:11.809 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:11 smithi003 systemd[1]: Stopping Ceph osd.3 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:12.309 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:11 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T01:20:11.901+0000 7fc8a855a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:12.309 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:11 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T01:20:11.901+0000 7fc8a855a700 -1 osd.3 118 *** Got signal Terminated *** 2023-09-10T01:20:12.309 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:11 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T01:20:11.901+0000 7fc8a855a700 -1 osd.3 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:14.559 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:14 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T01:20:14.252+0000 7fc8a4b50700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.440218+0000 front 2023-09-10T01:19:52.440312+0000 (oldest deadline 2023-09-10T01:20:14.140439+0000) 2023-09-10T01:20:15.559 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:15 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T01:20:15.230+0000 7fc8a4b50700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.440218+0000 front 2023-09-10T01:19:52.440312+0000 (oldest deadline 2023-09-10T01:20:14.140439+0000) 2023-09-10T01:20:16.559 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:16 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3[127791]: 2023-09-10T01:20:16.183+0000 7fc8a4b50700 -1 osd.3 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.440218+0000 front 2023-09-10T01:19:52.440312+0000 (oldest deadline 2023-09-10T01:20:14.140439+0000) 2023-09-10T01:20:17.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:17.430+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:18.062 INFO:journalctl@ceph.osd.3.smithi003.stdout:Sep 10 01:20:17 smithi003 bash[190753]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-3 2023-09-10T01:20:18.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:18 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:18.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:18.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:18.479+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:19.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:19 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:19.138+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:19.559 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:19 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:19.264+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:19.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.3.service' 2023-09-10T01:20:19.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:19.813 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-09-10T01:20:19.814 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-09-10T01:20:19.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.4 2023-09-10T01:20:19.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:19.438+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:20.099 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:20 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:20.098+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:20.100 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 01:20:19 smithi092 systemd[1]: Stopping Ceph osd.4 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:20.465 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 01:20:20 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4[147180]: 2023-09-10T01:20:20.174+0000 7f5eb7c19700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:20.466 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 01:20:20 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4[147180]: 2023-09-10T01:20:20.174+0000 7f5eb7c19700 -1 osd.4 118 *** Got signal Terminated *** 2023-09-10T01:20:20.466 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 01:20:20 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4[147180]: 2023-09-10T01:20:20.174+0000 7f5eb7c19700 -1 osd.4 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:20.559 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:20 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:20.279+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:20.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:20 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:20.455+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:21.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:21 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:21.070+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:21.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:21 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:21.311+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:21.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:21 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:21.481+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:22.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:22 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:22.049+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:22.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:22 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:22.355+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:22.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:22 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:22.489+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:23.018 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:22 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:22.734+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:23.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:23 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:23.017+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:23.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:23 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:23.017+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:23.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:23 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:23.384+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:23.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:23 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:23.384+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:23.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:23 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:23.523+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:23.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:23 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:23.516+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:23.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:23 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:23.516+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:24.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:23 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:23.784+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:24.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:23 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:23.977+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:24.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:23 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:23.977+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:24.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:24 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:24.366+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:24.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:24 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:24.366+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:24.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:24 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:24.491+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:24.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:24 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:24.484+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:24.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:24 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:24.484+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:25.215 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 01:20:24 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4[147180]: 2023-09-10T01:20:24.967+0000 7f5eb1a0a700 -1 osd.4 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:01.115967+0000 front 2023-09-10T01:20:01.115992+0000 (oldest deadline 2023-09-10T01:20:24.616234+0000) 2023-09-10T01:20:25.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:24 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:24.832+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:25.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:24 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:24.955+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:25.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:24 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:24.955+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:25.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:25.035+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:25.809 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:25 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:25.463+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:25.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:25 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:25.413+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:25.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:25 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:25.413+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:25.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:25.455+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:25.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:25 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:25.455+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:26.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:25 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:25.841+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:26.216 INFO:journalctl@ceph.osd.4.smithi092.stdout:Sep 10 01:20:25 smithi092 bash[179564]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-4 2023-09-10T01:20:26.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:25 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:25.951+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:26.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:25 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:25.951+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:26.412 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:26 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:26.074+0000 7fbd6d0f0700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.514064+0000 front 2023-09-10T01:19:57.514103+0000 (oldest deadline 2023-09-10T01:20:25.114846+0000) 2023-09-10T01:20:26.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:26 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:26.044+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:26.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:26.438+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:26.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:26.438+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:26.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:26.491+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:26.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:26 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:26.431+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:26.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:26 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:26.431+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:27.106 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:26 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:26.839+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:27.107 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:26 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:26.963+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:27.107 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:26 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:26.963+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:27.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.4.service' 2023-09-10T01:20:27.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:27.154 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-09-10T01:20:27.154 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-09-10T01:20:27.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.5 2023-09-10T01:20:27.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:27.079+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:27.413 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:27.076+0000 7fbd6d0f0700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.514064+0000 front 2023-09-10T01:19:57.514103+0000 (oldest deadline 2023-09-10T01:20:25.114846+0000) 2023-09-10T01:20:27.413 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:27 smithi136 systemd[1]: Stopping Ceph osd.5 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:27.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:27 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:27.440+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:27.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:27 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:27.440+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:27.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:27 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:27.462+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:27.912 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:27.503+0000 7fbd71afc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:27.912 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:27.503+0000 7fbd71afc700 -1 osd.5 118 *** Got signal Terminated *** 2023-09-10T01:20:27.913 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:27.503+0000 7fbd71afc700 -1 osd.5 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:27.913 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:27.475+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:27.913 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:27 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:27.475+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:28.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:27.829+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:28.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:27.924+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:28.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:27 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:27.924+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:28.412 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:28.049+0000 7fbd6d0f0700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.514064+0000 front 2023-09-10T01:19:57.514103+0000 (oldest deadline 2023-09-10T01:20:25.114846+0000) 2023-09-10T01:20:28.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:28.085+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:28.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:28 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:28.408+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:28.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:28 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:28.408+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:28.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:28 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:28.458+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:28.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:28.481+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:28.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:28 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:28.481+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:29.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:28 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:28.814+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:29.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:28 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:28.877+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:29.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:28 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:28.877+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:29.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:29 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:29.123+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:29.413 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:29 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:29.083+0000 7fbd6d0f0700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.514064+0000 front 2023-09-10T01:19:57.514103+0000 (oldest deadline 2023-09-10T01:20:25.114846+0000) 2023-09-10T01:20:29.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:29 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:29.395+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:29.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:29 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:29.395+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:29.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:29 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:29.473+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:29.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:29 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:29.492+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:29.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:29 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:29.492+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:30.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:29 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:29.804+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:30.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:29 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:29.863+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:30.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:29 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:29.863+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:30.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:30.120+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:30.413 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:30.120+0000 7fbd6d0f0700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.514064+0000 front 2023-09-10T01:19:57.514103+0000 (oldest deadline 2023-09-10T01:20:25.114846+0000) 2023-09-10T01:20:30.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:30 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:30.415+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:30.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:30 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:30.415+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:30.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:30 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:30.415+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:30.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:30 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:30.484+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:30.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:30 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:30.484+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:30.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:30.528+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:30.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:30 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:30.528+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:31.215 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:30.815+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:31.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:30.815+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:31.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:30.841+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:31.217 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:30 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:30.841+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:31.412 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:31 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:31.170+0000 7fbd6d0f0700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.514064+0000 front 2023-09-10T01:19:57.514103+0000 (oldest deadline 2023-09-10T01:20:25.114846+0000) 2023-09-10T01:20:31.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:31 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:31.093+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:31.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:31 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:31.093+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:31.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:31 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:31.404+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:31.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:31 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:31.404+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:31.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:31 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:31.404+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:31.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:31 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:31.503+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:31.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:31 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:31.503+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:31.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:31 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:31.552+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:31.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:31 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:31.552+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:32.215 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:31.829+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:32.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:31.829+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:32.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:31.829+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:32.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:31.883+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:32.217 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:31 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:31.883+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:32.412 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:32 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5[148191]: 2023-09-10T01:20:32.186+0000 7fbd6d0f0700 -1 osd.5 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.514064+0000 front 2023-09-10T01:19:57.514103+0000 (oldest deadline 2023-09-10T01:20:25.114846+0000) 2023-09-10T01:20:32.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:32 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:32.057+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:32.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:32 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:32.057+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:32.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:32 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:32.386+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:32.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:32 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:32.386+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:32.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:32 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:32.386+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:32.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:32 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:32.539+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:32.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:32 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:32.539+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:32.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:32 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:32.591+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:32.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:32 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:32.591+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:33.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:32.899+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:33.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:32.899+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:33.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:32.782+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:33.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:32.782+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:33.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:32 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:32.782+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:33.291 INFO:journalctl@ceph.osd.5.smithi136.stdout:Sep 10 01:20:33 smithi136 bash[180252]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-5 2023-09-10T01:20:33.292 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:33 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:33.040+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:33.292 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:33 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:33.040+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:33.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:33.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:33.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:33.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:33.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:33.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:33.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:33.498+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:33.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:33 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:33.498+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:33.849 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:33 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:33.595+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:33.850 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:33 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:33.595+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:34.162 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:34 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:34.090+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:34.163 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:34 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:34.090+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:34.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:33 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:33.894+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:34.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:33 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:33.894+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:34.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:33 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:33.793+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:34.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:33 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:33.793+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:34.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:33 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:33.793+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:34.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.5.service' 2023-09-10T01:20:34.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:34.699 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-09-10T01:20:34.699 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-09-10T01:20:34.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.6 2023-09-10T01:20:34.767 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:34 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:34.449+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:34.768 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:34 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:34.449+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:34.768 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:34 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:34.378+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:34.769 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:34 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:34.378+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:34.769 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:34 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:34.378+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:34.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:34 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:34.593+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:34.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:34 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:34.593+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:34.913 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:34 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:34.593+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:35.059 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:34 smithi003 systemd[1]: Stopping Ceph osd.6 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:35.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:34 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:34.878+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:35.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:34 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:34.878+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:35.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:34 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:34.775+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:35.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:34 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:34.775+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:35.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:34 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:34.775+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:35.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:35 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:35.122+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:35.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:35 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:35.122+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:35.559 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:35.339+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:35.559 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:35.339+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:35.560 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:35.339+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:35.560 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:35.214+0000 7fa75f790700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:35.561 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:35.214+0000 7fa75f790700 -1 osd.6 118 *** Got signal Terminated *** 2023-09-10T01:20:35.561 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:35.214+0000 7fa75f790700 -1 osd.6 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:35.561 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:35.452+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:35.562 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:35 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:35.452+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:35.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:35 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:35.612+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:35.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:35 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:35.612+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:35.913 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:35 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:35.612+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:36.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:35 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:35.859+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:36.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:35 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:35.859+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:36.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:35 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:35.823+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:36.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:35 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:35.823+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:36.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:35 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:35.823+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:36.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:36.107+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:36.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:36.107+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:36.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:36 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:36.383+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:36.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:36 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:36.383+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:36.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:36 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:36.383+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:36.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:36 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:36.409+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:36.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:36 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:36.409+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:36.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:36.653+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:36.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:36.653+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:36.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:36 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:36.653+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:37.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:36.878+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:37.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:36.878+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:37.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:36.813+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:37.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:36.813+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:37.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:36 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:36.813+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:37.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:37 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:37.089+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:37.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:37 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:37.090+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:37.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:37 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:37.412+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:37.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:37 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:37.412+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:37.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:37 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:37.412+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:37.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:37 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:37.443+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:37.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:37 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:37.443+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:38.087 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:37 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:37.696+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:38.088 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:37 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:37.696+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:38.088 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:37 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:37.696+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:38.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:37 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:37.909+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:38.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:37 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:37.909+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:38.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:37 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:37.832+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:38.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:37 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:37.832+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:38.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:37 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:37.832+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:38.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:38 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:38.086+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:38.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:38 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:38.086+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:38.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:38 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:38.439+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:38.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:38 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:38.439+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:38.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:38 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:38.439+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:38.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:38 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:38.408+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:38.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:38 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:38.408+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:39.043 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:38 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:38.704+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:39.044 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:38 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:38.704+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:39.044 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:38 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:38.704+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:39.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:38 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:38.949+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:39.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:38 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:38.949+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:39.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:38 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:38.949+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:39.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:38 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:38.828+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:39.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:38 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:38.828+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:39.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:38 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:38.828+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:39.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:39 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:39.042+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:39.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:39 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:39.042+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:39.809 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:39 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:39.380+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:20:00.456557+0000 front 2023-09-10T01:20:00.456590+0000 (oldest deadline 2023-09-10T01:20:22.756747+0000) 2023-09-10T01:20:39.810 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:39 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6[136402]: 2023-09-10T01:20:39.380+0000 7fa75b585700 -1 osd.6 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:09.159185+0000 front 2023-09-10T01:20:09.159414+0000 (oldest deadline 2023-09-10T01:20:30.259338+0000) 2023-09-10T01:20:39.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:39 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:39.406+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:39.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:39 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:39.406+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:39.812 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:39 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:39.406+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:40.076 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:39 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:39.725+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:40.077 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:39 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:39.725+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:40.077 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:39 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:39.725+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:40.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:39 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:39.947+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:40.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:39 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:39.947+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:40.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:39 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:39.947+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:40.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:39 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:39.848+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:40.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:39 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:39.848+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:40.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:39 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:39.848+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:40.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:39 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:39.848+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:40.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:40.075+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:40.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:40.075+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:40.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:40.075+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:40.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:40 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:40.414+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:40.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:40 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:40.414+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:40.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:40 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:40.414+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:41.038 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:40.728+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:41.039 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:40.728+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:41.039 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:40 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:40.728+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:41.109 INFO:journalctl@ceph.osd.6.smithi003.stdout:Sep 10 01:20:41 smithi003 bash[191032]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-6 2023-09-10T01:20:41.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:40 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:40.987+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:41.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:40 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:40.987+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:41.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:40 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:40.987+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:41.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:40 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:40.880+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:41.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:40 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:40.880+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:41.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:40 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:40.880+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:41.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:40 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:40.880+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:41.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:41.037+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:41.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:41.037+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:41.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:41.037+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:41.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:41 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:41.441+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:41.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:41 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:41.441+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:41.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:41 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:41.441+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:41.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:41 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:41.441+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:41.995 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:41.688+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:41.995 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:41.688+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:41.995 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:41.688+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:42.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:41.987+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:42.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:41.987+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:42.217 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:41.987+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:42.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:41.920+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:42.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:41.921+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:42.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:41.921+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:42.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:41.921+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:42.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:41.993+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:42.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:41.993+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:42.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:41 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:41.993+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:42.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:42 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:42.426+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:42.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:42 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:42.426+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:42.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:42 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:42.426+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:42.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:42 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:42.426+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:42.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.6.service' 2023-09-10T01:20:42.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:42 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:42.639+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:42.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:42 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:42.639+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:42.913 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:42 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:42.639+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:42.953 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:42.954 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-09-10T01:20:42.954 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-09-10T01:20:42.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.7 2023-09-10T01:20:43.303 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:42.966+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:43.303 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:42.966+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:43.304 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:42.966+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:43.304 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:42.966+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:43.304 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:42.966+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:43.305 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:42.966+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:43.305 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:42 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:42.966+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:43.305 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:42 smithi092 systemd[1]: Stopping Ceph osd.7 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:43.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:43.030+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:43.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:43.030+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:43.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:43.030+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:43.715 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:43.302+0000 7fba6f590700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:43.716 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:43.302+0000 7fba6f590700 -1 osd.7 118 *** Got signal Terminated *** 2023-09-10T01:20:43.716 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:43.302+0000 7fba6f590700 -1 osd.7 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:43.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:43 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:43.432+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:43.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:43 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:43.432+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:43.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:43 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:43.432+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:43.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:43 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:43.432+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:43.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:43.656+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:43.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:43.656+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:43.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:43 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:43.656+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:44.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:43.988+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:44.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:43.988+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:44.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:43.988+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:44.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:43.937+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:44.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:43.937+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:44.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:43.937+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:44.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:43 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:43.937+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:44.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:44 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:44.079+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:44.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:44 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:44.079+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:44.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:44 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:44.079+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:44.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:44 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:44.476+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:44.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:44 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:44.476+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:44.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:44 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:44.476+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:44.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:44 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:44.476+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:45.122 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:44 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:44.693+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:45.122 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:44 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:44.693+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:45.123 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:44 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:44.693+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:45.215 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:44 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:44.959+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:45.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:44 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:44.959+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:45.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:44 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:44.959+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:45.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:44 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:44.955+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:45.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:44 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:44.955+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:45.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:44 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:44.955+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:45.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:44 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:44.955+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:45.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:45 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:45.120+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:45.412 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:45 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:45.120+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:45.413 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:45 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:45.120+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:45.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:45 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:45.445+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:45.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:45 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:45.445+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:45.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:45 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:45.445+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:45.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:45 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:45.445+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:46.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:45 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:45.729+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:46.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:45 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:45.729+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:46.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:45 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:45.729+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:46.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:45.962+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:46.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:45.962+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:46.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:45.962+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:46.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:45.984+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:46.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:45.984+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:46.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:45.984+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:46.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:45.984+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:46.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:46.168+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:46.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:46.168+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:46.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:46.168+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:46.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:46 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:46.403+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:46.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:46 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:46.403+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:46.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:46 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:46.403+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:46.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:46 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:46.403+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:47.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:46.736+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:47.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:46.736+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:47.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:46.736+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:47.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:46 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:46.736+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:47.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:46.941+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:47.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:46.941+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:47.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:46.941+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:47.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:46.949+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:47.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:46.949+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:47.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:46.949+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:47.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:46.949+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:47.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:46 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:46.949+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:47.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:47 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:47.176+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:47.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:47 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:47.176+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:47.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:47 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:47.176+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:47.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:47.354+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:47.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:47.354+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:47.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:47.354+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:47.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:47.354+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:47.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:47 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:47.354+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:48.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:47 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:47.748+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:48.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:47 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:47.748+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:48.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:47 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:47.748+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:48.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:47 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:47.748+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:48.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:47.914+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.090167+0000 front 2023-09-10T01:19:57.090183+0000 (oldest deadline 2023-09-10T01:20:22.390272+0000) 2023-09-10T01:20:48.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:47.914+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.691413+0000 front 2023-09-10T01:20:07.691577+0000 (oldest deadline 2023-09-10T01:20:30.591537+0000) 2023-09-10T01:20:48.216 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:47.914+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.093702+0000 front 2023-09-10T01:20:15.093683+0000 (oldest deadline 2023-09-10T01:20:38.593841+0000) 2023-09-10T01:20:48.217 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7[155732]: 2023-09-10T01:20:47.914+0000 7fba6837f700 -1 osd.7 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.495038+0000 front 2023-09-10T01:20:21.495072+0000 (oldest deadline 2023-09-10T01:20:47.395242+0000) 2023-09-10T01:20:48.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:47.965+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:48.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:47.965+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:48.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:47.965+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:48.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:47.965+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:48.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:47 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:47.965+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:48.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:48.170+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:48.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:48.170+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:48.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:48.170+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:48.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:48.344+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:48.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:48.344+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:48.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:48.344+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:48.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:48.344+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:48.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:48 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:48.344+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:49.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:48.720+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:49.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:48.720+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:49.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:48.720+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:49.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:48 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:48.720+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:49.217 INFO:journalctl@ceph.osd.7.smithi092.stdout:Sep 10 01:20:49 smithi092 bash[179834]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-7 2023-09-10T01:20:49.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:48 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:48.942+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:49.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:48 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:48.942+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:49.218 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:48 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:48.942+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:49.219 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:48 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:48.942+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:49.219 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:48 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:48.943+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:49.559 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:49 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:49.295+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:49.559 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:49 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:49.295+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:49.559 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:49 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:49.295+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:49.560 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:49 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:49.295+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:49.560 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:49 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:49.295+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:49.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:49.173+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:49.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:49.173+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:49.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:49.173+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:49.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:49.173+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:25.056203+0000 front 2023-09-10T01:20:25.056311+0000 (oldest deadline 2023-09-10T01:20:49.156157+0000) 2023-09-10T01:20:50.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:49.745+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:50.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:49.745+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:50.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:49.745+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:50.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:49 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:49.745+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:50.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:49 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:49.951+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:50.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:49 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:49.951+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:50.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:49 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:49.951+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:50.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:49 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:49.951+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:50.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:49 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:49.951+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:50.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.7.service' 2023-09-10T01:20:50.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:50.290 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-09-10T01:20:50.291 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-09-10T01:20:50.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.8 2023-09-10T01:20:50.649 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:50.206+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:50.650 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:50.206+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:50.650 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:50.206+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:50.650 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:50.206+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:25.056203+0000 front 2023-09-10T01:20:25.056311+0000 (oldest deadline 2023-09-10T01:20:49.156157+0000) 2023-09-10T01:20:50.651 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 systemd[1]: Stopping Ceph osd.8 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:50.651 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:50.648+0000 7fde65e2a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:50.651 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:50.648+0000 7fde65e2a700 -1 osd.8 118 *** Got signal Terminated *** 2023-09-10T01:20:50.651 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:50.648+0000 7fde65e2a700 -1 osd.8 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:50.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:50.342+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:50.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:50.342+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:50.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:50.342+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:50.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:50.342+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:50.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:50 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:50.342+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:50.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:50.698+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:50.912 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:50.698+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:50.913 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:50.698+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:50.913 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:50 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:50.698+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:51.215 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:50 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:50.958+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:51.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:50 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:50.958+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:51.216 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:50 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:50.958+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:51.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:50 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:50.958+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:51.217 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:50 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:50.958+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:51.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:51.238+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:51.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:51.238+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:51.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:51.238+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:51.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:51.238+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:25.056203+0000 front 2023-09-10T01:20:25.056311+0000 (oldest deadline 2023-09-10T01:20:49.156157+0000) 2023-09-10T01:20:51.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:51.338+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:51.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:51.338+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:51.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:51.338+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:51.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:51.338+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:51.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:51 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:51.338+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:52.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:51.732+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:52.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:51.732+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:52.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:51.732+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:52.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:51 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:51.732+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:52.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:52 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:52.001+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:52.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:52 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:52.001+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:52.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:52 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:52.001+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:52.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:52 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:52.001+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:52.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:52 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:52.001+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:52.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:52.273+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:52.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:52.273+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:52.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:52.273+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:52.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:52.273+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:25.056203+0000 front 2023-09-10T01:20:25.056311+0000 (oldest deadline 2023-09-10T01:20:49.156157+0000) 2023-09-10T01:20:52.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:52.377+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:52.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:52.377+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:52.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:52.377+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:52.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:52.377+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:52.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:52 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:52.377+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:53.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:52.764+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:53.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:52.764+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:53.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:52.764+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:53.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:52 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:52.764+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:53.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:53 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:53.036+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:53.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:53 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:53.036+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:53.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:53 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:53.036+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:53.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:53 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:53.036+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:53.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:53 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:53.036+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:53.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:53.311+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:53.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:53.311+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:53.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:53.311+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:53.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:53.311+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:25.056203+0000 front 2023-09-10T01:20:25.056311+0000 (oldest deadline 2023-09-10T01:20:49.156157+0000) 2023-09-10T01:20:53.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:53.365+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:53.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:53.365+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:53.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:53.365+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:53.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:53.365+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:53.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:53 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:53.365+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:54.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:53.804+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:54.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:53.804+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:54.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:53.804+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:54.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:53 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:53.804+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:54.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:54 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:54.055+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:54.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:54 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:54.055+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:54.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:54 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:54.055+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:54.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:54 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:54.055+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:54.468 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:54 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:54.055+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:54.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:54.343+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:54.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:54.343+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:54.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:54.343+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:54.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:54.343+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:25.056203+0000 front 2023-09-10T01:20:25.056311+0000 (oldest deadline 2023-09-10T01:20:49.156157+0000) 2023-09-10T01:20:54.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:54.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:54.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:54.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:54.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:54.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:54.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:54.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:54.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:54 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:54.399+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:55.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:54.790+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:55.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:54.790+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:55.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:54.790+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:55.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:54.790+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:55.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:55 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:55.063+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:55.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:55 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:55.063+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:55.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:55 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:55.063+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:55.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:55 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:55.063+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:55.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:55 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:55.063+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:55.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:55.372+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.850396+0000 front 2023-09-10T01:19:58.850503+0000 (oldest deadline 2023-09-10T01:20:24.750456+0000) 2023-09-10T01:20:55.662 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:55.372+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:08.053119+0000 front 2023-09-10T01:20:08.053143+0000 (oldest deadline 2023-09-10T01:20:30.353430+0000) 2023-09-10T01:20:55.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:55.372+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.054825+0000 front 2023-09-10T01:20:15.054794+0000 (oldest deadline 2023-09-10T01:20:39.754789+0000) 2023-09-10T01:20:55.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:55.372+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:25.056203+0000 front 2023-09-10T01:20:25.056311+0000 (oldest deadline 2023-09-10T01:20:49.156157+0000) 2023-09-10T01:20:55.663 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8[156241]: 2023-09-10T01:20:55.372+0000 7fde5e418700 -1 osd.8 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:29.156614+0000 front 2023-09-10T01:20:29.156574+0000 (oldest deadline 2023-09-10T01:20:55.056639+0000) 2023-09-10T01:20:55.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:55 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:55.431+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:55.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:55 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:55.431+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:55.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:55 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:55.431+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:55.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:55 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:55.431+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:55.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:55 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:55.431+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:56.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:55.798+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:56.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:55.798+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:56.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:55.798+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:56.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:55.798+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:56.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:55 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:55.798+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:20:56.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:56.093+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:56.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:56.093+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:56.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:56.093+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:56.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:56.093+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:56.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:56 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:56.093+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:56.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:56 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:56.463+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:56.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:56 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:56.463+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:56.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:56 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:56.463+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:56.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:56 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:56.463+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:56.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:56 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:56.464+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:56.840 INFO:journalctl@ceph.osd.8.smithi136.stdout:Sep 10 01:20:56 smithi136 bash[180520]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-8 2023-09-10T01:20:57.124 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:56 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:56.839+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:57.125 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:56 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:56.839+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:57.126 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:56 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:56.839+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:57.126 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:56 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:56.839+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:57.127 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:56 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:56.839+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:20:57.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:57 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:57.064+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:57.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:57 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:57.064+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:57.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:57 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:57.064+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:57.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:57 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:57.064+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:57.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:57 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:57.064+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:57.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:57 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:57.466+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:57.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:57 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:57.466+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:57.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:57 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:57.466+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:57.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:57 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:57.466+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:57.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:57 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:57.466+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:57.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.8.service' 2023-09-10T01:20:57.899 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:20:57.900 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-09-10T01:20:57.900 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-09-10T01:20:57.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.9 2023-09-10T01:20:58.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:57 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:57.858+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:58.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:57 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:57.858+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:58.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:57 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:57.858+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:58.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:57 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:57.858+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:58.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:57 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:57.858+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:20:58.309 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:57 smithi003 systemd[1]: Stopping Ceph osd.9 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:20:58.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:58 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:58.035+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:58.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:58 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:58.035+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:58.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:58 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:58.035+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:58.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:58 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:58.035+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:58.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:58 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:58.035+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:58.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.410+0000 7fbca2802700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:20:58.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.410+0000 7fbca2802700 -1 osd.9 118 *** Got signal Terminated *** 2023-09-10T01:20:58.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.410+0000 7fbca2802700 -1 osd.9 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:20:58.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.515+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:58.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.515+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:58.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.515+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:58.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.515+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:58.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.515+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:58.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:58 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:58.515+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:27.624296+0000 front 2023-09-10T01:20:27.624235+0000 (oldest deadline 2023-09-10T01:20:57.625161+0000) 2023-09-10T01:20:59.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:58 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:58.842+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:20:59.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:58 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:58.842+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:20:59.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:58 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:58.842+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:20:59.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:58 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:58.842+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:20:59.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:58 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:58.842+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:20:59.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:59 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:59.015+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:20:59.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:59 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:59.015+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:20:59.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:59 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:59.015+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:20:59.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:59 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:59.015+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:20:59.468 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:59 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:59.015+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:20:59.468 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:20:59 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:20:59.015+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:20:59.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:59 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:59.528+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:20:59.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:59 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:59.528+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:20:59.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:59 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:59.528+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:20:59.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:59 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:59.528+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:20:59.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:59 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:59.528+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:20:59.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:20:59 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:20:59.528+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:27.624296+0000 front 2023-09-10T01:20:27.624235+0000 (oldest deadline 2023-09-10T01:20:57.625161+0000) 2023-09-10T01:21:00.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:59 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:59.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:00.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:59 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:59.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:00.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:59 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:59.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:00.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:59 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:59.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:00.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:20:59 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:20:59.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:00.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:00 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:00.052+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:00.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:00 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:00.052+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:00.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:00 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:00.052+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:00.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:00 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:00.052+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:00.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:00 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:00.052+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:00.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:00 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:00.052+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:00.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:00.543+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:21:00.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:00.543+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:21:00.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:00.543+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:21:00.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:00.543+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:21:00.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:00.543+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:21:00.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:00 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:00.543+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:27.624296+0000 front 2023-09-10T01:20:27.624235+0000 (oldest deadline 2023-09-10T01:20:57.625161+0000) 2023-09-10T01:21:01.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:00 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:00.886+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:01.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:00 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:00.886+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:01.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:00 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:00.886+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:01.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:00 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:00.886+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:01.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:00 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:00.886+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:01.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:01 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:01.041+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:01.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:01 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:01.041+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:01.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:01 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:01.041+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:01.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:01 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:01.041+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:01.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:01 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:01.041+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:01.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:01 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:01.041+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:01.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:01 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:01.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:21:01.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:01 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:01.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:21:01.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:01 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:01.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:21:01.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:01 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:01.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:21:01.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:01 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:01.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:21:01.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:01 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:01.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:27.624296+0000 front 2023-09-10T01:20:27.624235+0000 (oldest deadline 2023-09-10T01:20:57.625161+0000) 2023-09-10T01:21:02.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:01 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:01.905+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:02.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:01 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:01.905+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:02.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:01 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:01.905+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:02.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:01 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:01.905+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:02.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:01 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:01.905+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:02.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:02 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:02.087+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:02.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:02 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:02.087+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:02.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:02 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:02.087+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:02.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:02 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:02.087+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:02.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:02 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:02.087+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:02.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:02 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:02.087+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:02.468 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:02 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:02.087+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:02.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:02.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.516390+0000 front 2023-09-10T01:19:52.516425+0000 (oldest deadline 2023-09-10T01:20:18.416622+0000) 2023-09-10T01:21:02.809 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:02.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:58.417345+0000 front 2023-09-10T01:19:58.417580+0000 (oldest deadline 2023-09-10T01:20:22.517254+0000) 2023-09-10T01:21:02.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:02.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.119467+0000 front 2023-09-10T01:20:07.119399+0000 (oldest deadline 2023-09-10T01:20:30.019485+0000) 2023-09-10T01:21:02.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:02.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:15.920851+0000 front 2023-09-10T01:20:15.920882+0000 (oldest deadline 2023-09-10T01:20:41.221177+0000) 2023-09-10T01:21:02.810 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:02.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:21.222153+0000 front 2023-09-10T01:20:21.222319+0000 (oldest deadline 2023-09-10T01:20:46.522362+0000) 2023-09-10T01:21:02.811 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:02 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9[144904]: 2023-09-10T01:21:02.521+0000 7fbc9b5f1700 -1 osd.9 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:27.624296+0000 front 2023-09-10T01:20:27.624235+0000 (oldest deadline 2023-09-10T01:20:57.625161+0000) 2023-09-10T01:21:03.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:02 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:02.909+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:03.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:02 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:02.909+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:03.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:02 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:02.909+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:03.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:02 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:02.909+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:03.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:02 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:02.909+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:03.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:02 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:02.909+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:03.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:03.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:03.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:03.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:03.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:03.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:03.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:03.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:03.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:03.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:03.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:03.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:03.468 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:03 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:03.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:04.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:03.871+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:04.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:03.871+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:04.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:03.871+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:04.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:03.871+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:04.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:03.871+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:04.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:03 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:03.871+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:04.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:04.142+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:04.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:04.142+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:04.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:04.142+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:04.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:04.142+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:04.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:04.142+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:04.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:04.142+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:04.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:04 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:04.142+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:04.561 INFO:journalctl@ceph.osd.9.smithi003.stdout:Sep 10 01:21:04 smithi003 bash[191297]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-9 2023-09-10T01:21:05.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:04.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:05.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:04.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:05.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:04.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:05.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:04.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:05.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:04.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:05.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:04 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:04.847+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:05.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:05.144+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:05.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:05.144+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:05.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:05.144+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:05.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:05.144+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:05.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:05.144+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:05.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:05.144+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:05.468 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:05 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:05.144+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:06.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:05.897+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:06.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:05.897+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:06.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:05.897+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:06.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:05.897+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:06.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:05.897+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:06.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:05.897+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:06.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.9.service' 2023-09-10T01:21:06.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:06.444 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-09-10T01:21:06.445 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-09-10T01:21:06.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.10 2023-09-10T01:21:06.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.125+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:06.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.125+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:06.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.125+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:06.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.125+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:06.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.125+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:06.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.125+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:06.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.125+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:06.798 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 systemd[1]: Stopping Ceph osd.10 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:07.137 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.797+0000 7f0a48f18700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:07.137 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.797+0000 7f0a48f18700 -1 osd.10 118 *** Got signal Terminated *** 2023-09-10T01:21:07.138 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:06 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:06.797+0000 7f0a48f18700 -1 osd.10 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:21:07.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:06.898+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:07.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:06.898+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:07.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:06.898+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:07.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:06.898+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:07.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:06.898+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:07.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:06 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:06.898+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:07.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:07.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:07.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:07.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:07.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:07.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:07.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:07.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:07.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:07.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:07.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:07.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:07.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:07 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:07.136+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:08.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:07.869+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:08.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:07.869+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:08.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:07.869+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:08.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:07.869+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:08.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:07.869+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:08.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:07 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:07.869+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:08.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:08.176+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:08.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:08.176+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:08.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:08.176+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:08.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:08.176+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:08.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:08.176+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:08.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:08.176+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:08.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:08 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:08.176+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:09.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:08.918+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:09.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:08.918+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:09.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:08.918+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:09.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:08.918+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:09.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:08.918+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:09.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:08 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:08.918+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:09.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:09.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:09.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:09.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:09.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:09.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:09.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:09.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:09.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:09.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:09.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:09.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:09.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:09 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:09.129+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:10.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:09.941+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:10.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:09.941+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:10.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:09.941+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:10.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:09.941+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:10.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:09.941+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:10.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:09 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:09.941+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:10.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:10 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:10.165+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:10.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:10 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:10.165+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:10.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:10 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:10.165+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:10.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:10 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:10.165+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:10.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:10 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:10.165+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:10.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:10 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:10.165+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:10.468 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:10 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:10.165+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:11.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:10 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:10.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:11.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:10 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:10.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:11.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:10 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:10.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:11.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:10 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:10.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:11.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:10 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:10.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:11.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:10 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:10.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:11.465 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:11.195+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.507100+0000 front 2023-09-10T01:19:52.507040+0000 (oldest deadline 2023-09-10T01:20:17.207101+0000) 2023-09-10T01:21:11.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:11.195+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.207481+0000 front 2023-09-10T01:19:57.207499+0000 (oldest deadline 2023-09-10T01:20:22.507707+0000) 2023-09-10T01:21:11.466 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:11.195+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:07.208954+0000 front 2023-09-10T01:20:07.208884+0000 (oldest deadline 2023-09-10T01:20:31.308893+0000) 2023-09-10T01:21:11.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:11.195+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2023-09-10T01:20:14.711061+0000 front 2023-09-10T01:20:14.710897+0000 (oldest deadline 2023-09-10T01:20:39.410689+0000) 2023-09-10T01:21:11.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:11.195+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:24.612329+0000 front 2023-09-10T01:20:24.612358+0000 (oldest deadline 2023-09-10T01:20:46.912433+0000) 2023-09-10T01:21:11.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:11.195+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:26.912900+0000 front 2023-09-10T01:20:26.912976+0000 (oldest deadline 2023-09-10T01:20:58.113447+0000) 2023-09-10T01:21:11.467 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:11 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10[164307]: 2023-09-10T01:21:11.195+0000 7f0a41d07700 -1 osd.10 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:39.214366+0000 front 2023-09-10T01:20:39.214344+0000 (oldest deadline 2023-09-10T01:21:01.514403+0000) 2023-09-10T01:21:12.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:11 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:11.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:12.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:11 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:11.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:12.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:11 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:11.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:12.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:11 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:11.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:12.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:11 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:11.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:12.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:11 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:11.978+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:12.715 INFO:journalctl@ceph.osd.10.smithi092.stdout:Sep 10 01:21:12 smithi092 bash[180099]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-10 2023-09-10T01:21:13.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:12 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:12.986+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:13.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:12 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:12.986+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:13.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:12 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:12.986+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:13.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:12 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:12.986+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:13.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:12 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:12.986+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:13.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:12 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:12.986+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:13.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:12 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:12.986+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2023-09-10T01:20:47.105425+0000 front 2023-09-10T01:20:47.105359+0000 (oldest deadline 2023-09-10T01:21:12.405281+0000) 2023-09-10T01:21:13.784 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.10.service' 2023-09-10T01:21:13.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:13.864 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-09-10T01:21:13.865 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-09-10T01:21:13.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.11 2023-09-10T01:21:14.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 systemd[1]: Stopping Ceph osd.11 for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:14.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:13.991+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:14.163 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:13.991+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:14.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:13.991+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:14.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:13.991+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:14.164 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:13.991+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:14.165 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:13.991+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:14.165 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:13 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:13.991+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2023-09-10T01:20:47.105425+0000 front 2023-09-10T01:20:47.105359+0000 (oldest deadline 2023-09-10T01:21:12.405281+0000) 2023-09-10T01:21:14.662 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:14 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:14.210+0000 7f68a1b75700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:14.662 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:14 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:14.210+0000 7f68a1b75700 -1 osd.11 118 *** Got signal Terminated *** 2023-09-10T01:21:14.663 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:14 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:14.210+0000 7f68a1b75700 -1 osd.11 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-09-10T01:21:15.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:15 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:15.037+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:15.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:15 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:15.037+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:15.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:15 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:15.037+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:15.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:15 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:15.037+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:15.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:15 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:15.037+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:15.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:15 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:15.037+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:15.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:15 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:15.037+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2023-09-10T01:20:47.105425+0000 front 2023-09-10T01:20:47.105359+0000 (oldest deadline 2023-09-10T01:21:12.405281+0000) 2023-09-10T01:21:16.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:16 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:16.006+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:16.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:16 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:16.006+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:16.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:16 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:16.006+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:16.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:16 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:16.006+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:16.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:16 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:16.006+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:16.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:16 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:16.006+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:16.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:16 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:16.006+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2023-09-10T01:20:47.105425+0000 front 2023-09-10T01:20:47.105359+0000 (oldest deadline 2023-09-10T01:21:12.405281+0000) 2023-09-10T01:21:17.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:17.005+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:17.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:17.005+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:17.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:17.005+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:17.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:17.005+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:17.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:17.005+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:17.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:17.005+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:17.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:17 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:17.005+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2023-09-10T01:20:47.105425+0000 front 2023-09-10T01:20:47.105359+0000 (oldest deadline 2023-09-10T01:21:12.405281+0000) 2023-09-10T01:21:18.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:18.034+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:18.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:18.034+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:18.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:18.034+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:18.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:18.034+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:18.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:18.034+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:18.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:18.034+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:18.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:18 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:18.034+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2023-09-10T01:20:47.105425+0000 front 2023-09-10T01:20:47.105359+0000 (oldest deadline 2023-09-10T01:21:12.405281+0000) 2023-09-10T01:21:19.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2023-09-10T01:19:52.395101+0000 front 2023-09-10T01:19:52.395165+0000 (oldest deadline 2023-09-10T01:20:16.495203+0000) 2023-09-10T01:21:19.412 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-09-10T01:19:57.596693+0000 front 2023-09-10T01:19:57.596622+0000 (oldest deadline 2023-09-10T01:20:22.896772+0000) 2023-09-10T01:21:19.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2023-09-10T01:20:04.597542+0000 front 2023-09-10T01:20:04.597523+0000 (oldest deadline 2023-09-10T01:20:33.998326+0000) 2023-09-10T01:21:19.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2023-09-10T01:20:22.200417+0000 front 2023-09-10T01:20:22.200312+0000 (oldest deadline 2023-09-10T01:20:46.300239+0000) 2023-09-10T01:21:19.413 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2023-09-10T01:20:31.402489+0000 front 2023-09-10T01:20:31.402584+0000 (oldest deadline 2023-09-10T01:20:54.902700+0000) 2023-09-10T01:21:19.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2023-09-10T01:20:38.403930+0000 front 2023-09-10T01:20:38.403779+0000 (oldest deadline 2023-09-10T01:21:02.503726+0000) 2023-09-10T01:21:19.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2023-09-10T01:20:47.105425+0000 front 2023-09-10T01:20:47.105359+0000 (oldest deadline 2023-09-10T01:21:12.405281+0000) 2023-09-10T01:21:19.414 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11[165003]: 2023-09-10T01:21:19.064+0000 7f689a163700 -1 osd.11 118 heartbeat_check: no reply from 172.21.15.136:6820 osd.8 since back 2023-09-10T01:20:53.506352+0000 front 2023-09-10T01:20:53.506377+0000 (oldest deadline 2023-09-10T01:21:18.206370+0000) 2023-09-10T01:21:20.162 INFO:journalctl@ceph.osd.11.smithi136.stdout:Sep 10 01:21:19 smithi136 bash[180793]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-osd-11 2023-09-10T01:21:21.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@osd.11.service' 2023-09-10T01:21:21.334 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:21.335 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-09-10T01:21:21.336 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-09-10T01:21:21.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.a 2023-09-10T01:21:21.809 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:21:21 smithi003 systemd[1]: Stopping Ceph mds.a for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:22.309 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:21:21 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-a[153316]: 2023-09-10T01:21:21.847+0000 7fb494d71700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:22.309 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:21:21 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-a[153316]: 2023-09-10T01:21:21.847+0000 7fb494d71700 -1 mds.a *** got signal Terminated *** 2023-09-10T01:21:25.309 INFO:journalctl@ceph.mds.a.smithi003.stdout:Sep 10 01:21:25 smithi003 bash[191572]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-a 2023-09-10T01:21:25.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.a.service' 2023-09-10T01:21:25.538 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:25.539 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-09-10T01:21:25.539 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-09-10T01:21:25.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.d 2023-09-10T01:21:25.809 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:21:25 smithi003 systemd[1]: Stopping Ceph mds.d for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:26.309 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:21:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-d[152313]: 2023-09-10T01:21:26.048+0000 7f95be8b7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:26.310 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:21:26 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-d[152313]: 2023-09-10T01:21:26.048+0000 7f95be8b7700 -1 mds.d *** got signal Terminated *** 2023-09-10T01:21:28.309 INFO:journalctl@ceph.mds.d.smithi003.stdout:Sep 10 01:21:27 smithi003 bash[191714]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-d 2023-09-10T01:21:28.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.d.service' 2023-09-10T01:21:28.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:28.450 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-09-10T01:21:28.450 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-09-10T01:21:28.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.g 2023-09-10T01:21:28.809 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:21:28 smithi003 systemd[1]: Stopping Ceph mds.g for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:29.309 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:21:28 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-g[152883]: 2023-09-10T01:21:28.946+0000 7fd9eb8e9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:29.310 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:21:28 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-g[152883]: 2023-09-10T01:21:28.946+0000 7fd9eb8e9700 -1 mds.g *** got signal Terminated *** 2023-09-10T01:21:33.558 INFO:journalctl@ceph.mds.g.smithi003.stdout:Sep 10 01:21:33 smithi003 bash[191858]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-g 2023-09-10T01:21:33.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.g.service' 2023-09-10T01:21:33.807 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:33.808 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-09-10T01:21:33.808 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-09-10T01:21:33.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.j 2023-09-10T01:21:34.061 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:21:33 smithi003 systemd[1]: Stopping Ceph mds.j for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:34.559 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:21:34 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-j[151926]: 2023-09-10T01:21:34.304+0000 7f597cc67700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:34.559 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:21:34 smithi003 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-j[151926]: 2023-09-10T01:21:34.304+0000 7f597cc67700 -1 mds.j *** got signal Terminated *** 2023-09-10T01:21:37.058 INFO:journalctl@ceph.mds.j.smithi003.stdout:Sep 10 01:21:36 smithi003 bash[192001]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-j 2023-09-10T01:21:37.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.j.service' 2023-09-10T01:21:37.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:37.320 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-09-10T01:21:37.320 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-09-10T01:21:37.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.b 2023-09-10T01:21:37.675 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:21:37 smithi092 systemd[1]: Stopping Ceph mds.b for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:37.965 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:21:37 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: 2023-09-10T01:21:37.673+0000 7f75e23f4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:37.966 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:21:37 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b[171454]: 2023-09-10T01:21:37.673+0000 7f75e23f4700 -1 mds.b *** got signal Terminated *** 2023-09-10T01:21:40.645 INFO:journalctl@ceph.mds.b.smithi092.stdout:Sep 10 01:21:40 smithi092 bash[180376]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-b 2023-09-10T01:21:40.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.b.service' 2023-09-10T01:21:40.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:40.876 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-09-10T01:21:40.876 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-09-10T01:21:40.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.e 2023-09-10T01:21:41.215 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:21:40 smithi092 systemd[1]: Stopping Ceph mds.e for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:41.715 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:21:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-e[170771]: 2023-09-10T01:21:41.264+0000 7f35cdddd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:41.716 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:21:41 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-e[170771]: 2023-09-10T01:21:41.264+0000 7f35cdddd700 -1 mds.e *** got signal Terminated *** 2023-09-10T01:21:44.703 INFO:journalctl@ceph.mds.e.smithi092.stdout:Sep 10 01:21:44 smithi092 bash[180525]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-e 2023-09-10T01:21:44.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.e.service' 2023-09-10T01:21:44.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:44.745 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-09-10T01:21:44.745 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-09-10T01:21:44.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.h 2023-09-10T01:21:44.965 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:21:44 smithi092 systemd[1]: Stopping Ceph mds.h for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:45.465 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:21:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-h[171112]: 2023-09-10T01:21:45.104+0000 7fecd97fe700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:45.466 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:21:45 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-h[171112]: 2023-09-10T01:21:45.104+0000 7fecd97fe700 -1 mds.h *** got signal Terminated *** 2023-09-10T01:21:48.715 INFO:journalctl@ceph.mds.h.smithi092.stdout:Sep 10 01:21:48 smithi092 bash[180665]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-h 2023-09-10T01:21:48.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.h.service' 2023-09-10T01:21:48.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:48.831 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-09-10T01:21:48.831 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-09-10T01:21:48.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.k 2023-09-10T01:21:49.204 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:21:48 smithi092 systemd[1]: Stopping Ceph mds.k for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:49.465 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:21:49 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-k[170432]: 2023-09-10T01:21:49.203+0000 7f74f8391700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:49.469 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:21:49 smithi092 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-k[170432]: 2023-09-10T01:21:49.203+0000 7f74f8391700 -1 mds.k *** got signal Terminated *** 2023-09-10T01:21:53.672 INFO:journalctl@ceph.mds.k.smithi092.stdout:Sep 10 01:21:53 smithi092 bash[180817]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-k 2023-09-10T01:21:53.825 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.k.service' 2023-09-10T01:21:53.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:53.860 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-09-10T01:21:53.860 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-09-10T01:21:53.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.c 2023-09-10T01:21:54.162 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:21:53 smithi136 systemd[1]: Stopping Ceph mds.c for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:54.662 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:21:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-c[171170]: 2023-09-10T01:21:54.209+0000 7f5b98b03700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:54.662 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:21:54 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-c[171170]: 2023-09-10T01:21:54.209+0000 7f5b98b03700 -1 mds.c *** got signal Terminated *** 2023-09-10T01:21:56.836 INFO:journalctl@ceph.mds.c.smithi136.stdout:Sep 10 01:21:56 smithi136 bash[181072]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-c 2023-09-10T01:21:56.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.c.service' 2023-09-10T01:21:56.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:21:56.879 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-09-10T01:21:56.880 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-09-10T01:21:56.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.f 2023-09-10T01:21:57.162 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:21:56 smithi136 systemd[1]: Stopping Ceph mds.f for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:21:57.662 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:21:57 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-f[170830]: 2023-09-10T01:21:57.250+0000 7f3310b9c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:21:57.662 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:21:57 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-f[170830]: 2023-09-10T01:21:57.250+0000 7f3310b9c700 -1 mds.f *** got signal Terminated *** 2023-09-10T01:22:00.626 INFO:journalctl@ceph.mds.f.smithi136.stdout:Sep 10 01:22:00 smithi136 bash[181221]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-f 2023-09-10T01:22:00.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.f.service' 2023-09-10T01:22:00.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:22:00.849 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-09-10T01:22:00.850 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-09-10T01:22:00.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.i 2023-09-10T01:22:01.162 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:22:00 smithi136 systemd[1]: Stopping Ceph mds.i for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:22:01.662 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:22:01 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-i[171516]: 2023-09-10T01:22:01.215+0000 7f3eca421700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:22:01.662 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:22:01 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-i[171516]: 2023-09-10T01:22:01.215+0000 7f3eca421700 -1 mds.i *** got signal Terminated *** 2023-09-10T01:22:04.847 INFO:journalctl@ceph.mds.i.smithi136.stdout:Sep 10 01:22:04 smithi136 bash[181362]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-i 2023-09-10T01:22:04.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.i.service' 2023-09-10T01:22:04.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:22:04.886 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-09-10T01:22:04.886 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-09-10T01:22:04.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.l 2023-09-10T01:22:05.162 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:22:04 smithi136 systemd[1]: Stopping Ceph mds.l for 4260c16a-4f74-11ee-9ab7-7b867c8bd7da... 2023-09-10T01:22:05.662 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:22:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-l[170488]: 2023-09-10T01:22:05.256+0000 7f29cd359700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-09-10T01:22:05.662 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:22:05 smithi136 ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-l[170488]: 2023-09-10T01:22:05.256+0000 7f29cd359700 -1 mds.l *** got signal Terminated *** 2023-09-10T01:22:07.412 INFO:journalctl@ceph.mds.l.smithi136.stdout:Sep 10 01:22:07 smithi136 bash[181509]: ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da-mds-l 2023-09-10T01:22:07.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4260c16a-4f74-11ee-9ab7-7b867c8bd7da@mds.l.service' 2023-09-10T01:22:07.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-09-10T01:22:07.561 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-09-10T01:22:07.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da --force --keep-logs 2023-09-10T01:22:07.962 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T01:22:11.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da --force --keep-logs 2023-09-10T01:22:11.635 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T01:22:13.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da --force --keep-logs 2023-09-10T01:22:13.588 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T01:22:15.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-09-10T01:22:15.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-09-10T01:22:15.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-09-10T01:22:15.299 INFO:tasks.cephadm:Archiving crash dumps... 2023-09-10T01:22:15.302 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi003/crash 2023-09-10T01:22:15.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash -- . 2023-09-10T01:22:15.377 INFO:teuthology.orchestra.run.smithi003.stderr:tar: /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash: Cannot open: No such file or directory 2023-09-10T01:22:15.378 INFO:teuthology.orchestra.run.smithi003.stderr:tar: Error is not recoverable: exiting now 2023-09-10T01:22:15.381 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi092/crash 2023-09-10T01:22:15.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash -- . 2023-09-10T01:22:15.433 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash: Cannot open: No such file or directory 2023-09-10T01:22:15.433 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2023-09-10T01:22:15.435 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi136/crash 2023-09-10T01:22:15.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash -- . 2023-09-10T01:22:15.506 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/crash: Cannot open: No such file or directory 2023-09-10T01:22:15.506 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2023-09-10T01:22:15.507 INFO:tasks.cephadm:Checking cluster log for badness... 2023-09-10T01:22:15.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-09-10T01:22:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:grep: /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph.log: No such file or directory 2023-09-10T01:22:15.594 INFO:tasks.cephadm:Compressing logs... 2023-09-10T01:22:15.595 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-10T01:22:15.601 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-10T01:22:15.604 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-10T01:22:15.655 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-09-10T01:22:15.656 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-09-10T01:22:15.661 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.5.log 2023-09-10T01:22:15.665 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.2.log 2023-09-10T01:22:15.665 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.11.log 2023-09-10T01:22:15.665 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.5.log: /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.z.log 2023-09-10T01:22:15.666 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.8.log 2023-09-10T01:22:15.666 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.f.log 2023-09-10T01:22:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.c.log 2023-09-10T01:22:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.i.log 2023-09-10T01:22:15.668 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.c.log 2023-09-10T01:22:15.668 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log 2023-09-10T01:22:15.668 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.i.log: /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.l.log 2023-09-10T01:22:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log: /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.l.log: 94.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.l.log.gz 2023-09-10T01:22:15.670 INFO:teuthology.orchestra.run.smithi136.stderr: 92.2% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.z.log.gz 2023-09-10T01:22:15.671 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’gzip : No such file or directory-5 2023-09-10T01:22:15.671 INFO:teuthology.orchestra.run.smithi003.stderr:--verbose -- /var/log/ceph/cephadm.log 2023-09-10T01:22:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2023-09-10T01:22:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.0.log 2023-09-10T01:22:15.672 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.g.log 2023-09-10T01:22:15.673 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.0.log: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-09-10T01:22:15.674 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.a.log 2023-09-10T01:22:15.675 INFO:teuthology.orchestra.run.smithi003.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2023-09-10T01:22:15.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.6.log 2023-09-10T01:22:15.675 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.3.log 2023-09-10T01:22:15.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.d.log 2023-09-10T01:22:15.676 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.3.log: gzip 94.4%-5 -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.g.log.gz--verbose 2023-09-10T01:22:15.676 INFO:teuthology.orchestra.run.smithi003.stderr:-- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.x.log 2023-09-10T01:22:15.677 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.9.log 2023-09-10T01:22:15.677 INFO:teuthology.orchestra.run.smithi003.stderr: 94.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.a.log.gz 2023-09-10T01:22:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log 2023-09-10T01:22:15.679 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose : No such file or directory-- 2023-09-10T01:22:15.680 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/cephadm.log 2023-09-10T01:22:15.681 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.9.log: /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log: 94.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.d.log.gz 2023-09-10T01:22:15.682 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.j.log 2023-09-10T01:22:15.686 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.a.log 2023-09-10T01:22:15.688 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.10.log 2023-09-10T01:22:15.690 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.h.log 2023-09-10T01:22:15.690 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.e.log 2023-09-10T01:22:15.690 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.4.log 2023-09-10T01:22:15.691 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.1.log 2023-09-10T01:22:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.a.log: 90.9% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log.gz 2023-09-10T01:22:15.692 INFO:teuthology.orchestra.run.smithi003.stderr: 94.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.j.log.gz 2023-09-10T01:22:15.694 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.k.log 2023-09-10T01:22:15.694 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log 2023-09-10T01:22:15.694 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.7.log 2023-09-10T01:22:15.695 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.b.log 2023-09-10T01:22:15.695 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.b.log 2023-09-10T01:22:15.696 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.y.log 2023-09-10T01:22:15.696 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.b.log: /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.y.log: 94.5% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.k.log.gz 2023-09-10T01:22:15.699 INFO:teuthology.orchestra.run.smithi092.stderr: 92.1% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.y.log.gz 2023-09-10T01:22:15.798 INFO:teuthology.orchestra.run.smithi136.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-09-10T01:22:16.096 INFO:teuthology.orchestra.run.smithi003.stderr: 89.8% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mgr.x.log.gz 2023-09-10T01:22:16.131 INFO:teuthology.orchestra.run.smithi092.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-09-10T01:22:16.646 INFO:teuthology.orchestra.run.smithi092.stderr: 91.0% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log.gz 2023-09-10T01:22:16.764 INFO:teuthology.orchestra.run.smithi136.stderr: 90.9% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-volume.log.gz 2023-09-10T01:22:16.876 INFO:teuthology.orchestra.run.smithi136.stderr: 91.9% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.c.log.gz 2023-09-10T01:22:18.312 INFO:teuthology.orchestra.run.smithi003.stderr: 91.7% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.a.log.gz 2023-09-10T01:22:18.493 INFO:teuthology.orchestra.run.smithi092.stderr: 91.8% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mon.b.log.gz 2023-09-10T01:22:20.625 INFO:teuthology.orchestra.run.smithi136.stderr: 95.3% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.c.log.gz 2023-09-10T01:22:22.436 INFO:teuthology.orchestra.run.smithi136.stderr: 95.6% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.f.log.gz 2023-09-10T01:22:39.061 INFO:teuthology.orchestra.run.smithi092.stderr: 95.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.e.log.gz 2023-09-10T01:22:45.399 INFO:teuthology.orchestra.run.smithi003.stderr: 91.6% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.0.log.gz 2023-09-10T01:22:45.807 INFO:teuthology.orchestra.run.smithi136.stderr: 91.7% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.11.log.gz 2023-09-10T01:22:51.674 INFO:teuthology.orchestra.run.smithi003.stderr: 91.2% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.6.log.gz 2023-09-10T01:22:53.245 INFO:teuthology.orchestra.run.smithi003.stderr: 91.3% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.3.log.gz 2023-09-10T01:22:54.498 INFO:teuthology.orchestra.run.smithi136.stderr: 91.1% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.8.log.gz 2023-09-10T01:22:54.624 INFO:teuthology.orchestra.run.smithi003.stderr: 91.2% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.9.log.gz 2023-09-10T01:22:54.628 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-09-10T01:22:54.628 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m38.993s 2023-09-10T01:22:54.629 INFO:teuthology.orchestra.run.smithi003.stderr:user 2m15.010s 2023-09-10T01:22:54.629 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m10.402s 2023-09-10T01:22:59.646 INFO:teuthology.orchestra.run.smithi136.stderr: 91.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.2.log.gz 2023-09-10T01:23:12.975 INFO:teuthology.orchestra.run.smithi136.stderr: 91.1% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.5.log.gz 2023-09-10T01:23:17.059 INFO:teuthology.orchestra.run.smithi092.stderr: 91.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.10.log.gz 2023-09-10T01:23:17.655 INFO:teuthology.orchestra.run.smithi092.stderr: 91.7% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.1.log.gz 2023-09-10T01:23:38.657 INFO:teuthology.orchestra.run.smithi092.stderr: 91.3% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.7.log.gz 2023-09-10T01:23:42.194 INFO:teuthology.orchestra.run.smithi092.stderr: 91.4% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-osd.4.log.gz 2023-09-10T01:23:54.661 INFO:teuthology.orchestra.run.smithi136.stderr: 93.5% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.i.log.gz 2023-09-10T01:23:54.663 INFO:teuthology.orchestra.run.smithi136.stderr: 2023-09-10T01:23:54.663 INFO:teuthology.orchestra.run.smithi136.stderr:real 1m39.042s 2023-09-10T01:23:54.664 INFO:teuthology.orchestra.run.smithi136.stderr:user 3m54.117s 2023-09-10T01:23:54.664 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m7.990s 2023-09-10T01:24:29.415 INFO:teuthology.orchestra.run.smithi092.stderr: 94.1% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.h.log.gz 2023-09-10T01:26:05.733 INFO:teuthology.orchestra.run.smithi092.stderr: 94.1% -- replaced with /var/log/ceph/4260c16a-4f74-11ee-9ab7-7b867c8bd7da/ceph-mds.b.log.gz 2023-09-10T01:26:05.756 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-09-10T01:26:05.757 INFO:teuthology.orchestra.run.smithi092.stderr:real 3m50.134s 2023-09-10T01:26:05.757 INFO:teuthology.orchestra.run.smithi092.stderr:user 5m42.132s 2023-09-10T01:26:05.757 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m13.495s 2023-09-10T01:26:05.758 INFO:tasks.cephadm:Archiving logs... 2023-09-10T01:26:05.759 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi003/log 2023-09-10T01:26:05.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-10T01:26:28.035 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi092/log 2023-09-10T01:26:28.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-10T01:27:21.413 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi136/log 2023-09-10T01:27:21.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-10T01:27:56.308 INFO:tasks.cephadm:Removing cluster... 2023-09-10T01:27:56.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da --force 2023-09-10T01:27:56.710 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T01:27:57.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da --force 2023-09-10T01:27:58.681 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T01:27:59.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4260c16a-4f74-11ee-9ab7-7b867c8bd7da --force 2023-09-10T01:27:59.888 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 4260c16a-4f74-11ee-9ab7-7b867c8bd7da 2023-09-10T01:28:00.649 INFO:tasks.cephadm:Removing cephadm ... 2023-09-10T01:28:00.650 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2023-09-10T01:28:00.699 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2023-09-10T01:28:00.733 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2023-09-10T01:28:00.751 INFO:tasks.cephadm:Teardown complete 2023-09-10T01:28:00.752 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-10T01:28:00.762 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-10T01:28:00.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-10T01:28:00.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-10T01:28:00.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-10T01:28:00.912 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-10T01:28:00.922 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-10T01:28:00.923 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-10T01:28:00.928 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-10T01:28:00.930 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-10T01:28:00.946 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2023-09-10T01:28:00.947 INFO:teuthology.orchestra.run.smithi092.stderr:bash: ntpq: command not found 2023-09-10T01:28:00.965 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-10T01:28:00.966 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-09-10T01:28:00.966 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +138us[ +138us] +/- 79ms 2023-09-10T01:28:00.966 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -1262us[-1265us] +/- 77ms 2023-09-10T01:28:00.967 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +951us[ +951us] +/- 63ms 2023-09-10T01:28:00.967 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T01:28:00.967 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-09-10T01:28:00.974 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-10T01:28:00.974 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-09-10T01:28:00.974 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 199 +247us[ +258us] +/- 76ms 2023-09-10T01:28:00.975 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 7 377 73 -1176us[-1167us] +/- 76ms 2023-09-10T01:28:00.975 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 206 +1017us[+1044us] +/- 60ms 2023-09-10T01:28:00.975 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T01:28:00.978 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-10T01:28:00.978 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2023-09-10T01:28:00.979 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +127us[ +166us] +/- 79ms 2023-09-10T01:28:00.979 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -1251us[-1211us] +/- 77ms 2023-09-10T01:28:00.979 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +959us[ +998us] +/- 63ms 2023-09-10T01:28:00.979 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-09-10T01:28:00.981 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-10T01:28:00.992 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-10T01:28:00.992 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-10T01:28:01.022 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-10T01:28:01.208 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-10T01:28:01.370 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-10T01:28:01.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-10T01:28:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T01:28:01.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-10T01:28:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T01:28:01.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-10T01:28:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T01:28:01.642 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-10T01:28:01.654 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-10T01:28:01.681 INFO:teuthology.task.internal:Duration was 3092.194944 seconds 2023-09-10T01:28:01.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-10T01:28:01.714 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-10T01:28:01.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-10T01:28:01.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-10T01:28:01.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-10T01:28:01.781 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-10T01:28:01.808 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-10T01:28:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-10T01:28:02.343 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-10T01:28:02.344 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2023-09-10T01:28:02.344 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-10T01:28:02.404 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2023-09-10T01:28:02.405 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-10T01:28:02.449 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2023-09-10T01:28:02.449 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-10T01:28:02.500 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-10T01:28:02.501 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-10T01:28:02.506 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-10T01:28:02.509 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-10T01:28:02.606 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-10T01:28:02.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-10T01:28:02.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-10T01:28:02.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-10T01:28:02.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-10T01:28:02.797 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-10T01:28:02.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-10T01:28:02.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-10T01:28:02.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-10T01:28:02.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-10T01:28:02.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-10T01:28:02.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-10T01:28:02.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-10T01:28:02.997 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2023-09-10T01:28:03.000 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2023-09-10T01:28:03.011 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2023-09-10T01:28:03.058 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-10T01:28:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T01:28:03.100 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-10T01:28:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T01:28:03.116 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-10T01:28:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-10T01:28:03.133 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-10T01:28:03.145 INFO:teuthology.task.internal:Transferring archived files... 2023-09-10T01:28:03.147 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi003 2023-09-10T01:28:03.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-10T01:28:03.233 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi092 2023-09-10T01:28:03.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-10T01:28:03.273 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534/remote/smithi136 2023-09-10T01:28:03.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-10T01:28:03.313 INFO:teuthology.task.internal:Removing archive directory... 2023-09-10T01:28:03.313 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-10T01:28:03.318 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-10T01:28:03.320 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-10T01:28:03.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-10T01:28:03.376 INFO:teuthology.task.internal:Not uploading archives. 2023-09-10T01:28:03.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-10T01:28:03.398 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-10T01:28:03.398 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-10T01:28:03.403 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-10T01:28:03.405 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-10T01:28:03.419 INFO:teuthology.orchestra.run.smithi092.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 01:28 /home/ubuntu/cephtest 2023-09-10T01:28:03.421 INFO:teuthology.orchestra.run.smithi136.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 01:28 /home/ubuntu/cephtest 2023-09-10T01:28:03.442 INFO:teuthology.orchestra.run.smithi003.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 01:28 /home/ubuntu/cephtest 2023-09-10T01:28:03.445 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-10T01:28:03.457 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-10T01:28:03.515 INFO:teuthology.nuke:Checking targets against current locks 2023-09-10T01:28:03.543 DEBUG:teuthology.nuke:shortname: smithi003 2023-09-10T01:28:03.543 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-10T01:28:03.566 DEBUG:teuthology.nuke:shortname: smithi092 2023-09-10T01:28:03.567 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-10T01:28:03.572 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-10 00:28:12.072759', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-10T01:28:03.591 DEBUG:teuthology.nuke:shortname: smithi136 2023-09-10T01:28:03.591 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-10T01:28:03.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-10 00:28:12.075313', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-10T01:28:03.621 INFO:teuthology.orchestra.console.smithi003:Power off 2023-09-10T01:28:03.622 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-10T01:28:03.634 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-09-09_19:00:20-fs-wip-batrick-testing-20230909.145638-distro-default-smithi/7392534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-10 00:28:12.074198', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-10T01:28:03.645 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2023-09-10T01:28:03.645 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:03.666 INFO:teuthology.orchestra.console.smithi092:Power off 2023-09-10T01:28:03.666 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-10T01:28:03.679 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-09-10T01:28:03.692 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2023-09-10T01:28:03.692 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:03.707 INFO:teuthology.orchestra.console.smithi136:Power off 2023-09-10T01:28:03.708 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-10T01:28:03.719 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-09-10T01:28:03.732 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2023-09-10T01:28:03.732 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:03.756 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-09-10T01:28:07.681 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:07.720 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:07.758 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:07.792 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-09-10T01:28:07.830 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-09-10T01:28:07.868 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-09-10T01:28:11.795 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:11.831 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:11.869 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:11.904 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-09-10T01:28:11.942 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-09-10T01:28:11.980 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-09-10T01:28:15.908 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:15.943 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:15.981 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-10T01:28:16.017 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2023-09-10T01:28:16.053 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2023-09-10T01:28:16.091 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2023-09-10T01:28:16.119 INFO:teuthology.orchestra.console.smithi003:Power off completed 2023-09-10T01:28:16.154 INFO:teuthology.orchestra.console.smithi092:Power off completed 2023-09-10T01:28:16.193 INFO:teuthology.orchestra.console.smithi136:Power off completed 2023-09-10T01:28:16.300 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 533e5fc8ca061dbf84e56917eecb852edf10a398 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 3092.194943666458 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9670cca5bc8e17916865dc84939edfc10b1f2c00 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f1e42f9c98ab4631ab1218a59bf882dc status: fail success: false 2023-09-10T01:28:16.300 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-10T01:28:16.387 INFO:teuthology.run:FAIL